OPTRWS 1A 02 April 2001 NIST NFIS Reference Manual optrws optimizes the regional weights for PNN classifier. optrws <prsfile> Optrws optimizes the regional weights, each of which goes with one 2x2-vector block of the orientation array. Since that array has 28 x 30 vectors, there are (28 / 2) x (30 / 2) = 210 regional weights. Optimization consists of attempting to approximate the minimum, or at least a local minimum, of an "activation error rate" of the Probabilistic Neural Network (PNN) classifier when it is run on a set of fingerprints, using the same set as the prototypes for the PNN but leaving out of the prototype set, each time, the particular fingerprint that the network is being run on. The program first finds a reasonable value to use as an initial value for all 210 weights. Then, it uses a very simple form of a gradient descent algorithm to finish optimizing the weights. Each iteration consists of, first, estimating the gradient of the error at the current "basepoint", and second, approximately finding the minimum (or at least a local minimum) of the error along the straight line pointing in the anti-estimated-gradient direction. (Estimating the gradient seems to be sufficient, and calculating it from its definition may actually require more computation then estimating.) Because the activation error is apparently such a well-behaved function of distance along this line, for this particular problem, it seems sufficient to use a very simple algorithm for the line search: this consists of taking large equal-sized steps in the anti-gradient direction until the error ceases to decrease, then halving the stepsize and going in the opposite direction along the line until the error again ceases to decrease, etc., with the process stopping when the step size becomes smaller than a threshold. This finds a local minimum, and it appears that this function generally is unimodal along the line, so that this local minimum will be the minimum along the line. The 0th basepoint is (irw,irw,...,irw), where irw is the initial value decided upon for all regional weights. The 1st basepoint is the result of the line search that follows the gradient-estimation at the 0th basepoint; etc. Stopping of the program is controlled by specifying the number of line searches that are to be done. If this parameter is set to 1, then the program only gets as far as basepoint 1. Since optrws records each basepoint, the program can be manually stopped if it turns out that it is taking too much time, without the run being a total waste of cycles. At each of the basepoints, optrws produces the following files (in a directory specified as one of the parameters). The basepoint, as a "matrix" file of dimensions 14 x 15 (these dimensions correspond to the geometric interpretation of the basepoint as a set of regional weights); these files have names bspt_0.bin, bspt_1.bin, etc. or bspt_0.asc, bspt_1.asc, etc. The estimated gradient of the activation error rate at the basepoint, also as a "matrix" file of dimensions 14 x 15; these files have names egrad_0.bin, egrad_1.bin, etc. or egrad_0.asc, egrad_1.asc, etc. And, the activation error rate at the basepoint, as a text file; these files have names acerr_0.txt, acerr_1.txt, etc. As optrws is running (which make take several hours), these intermediate results files may be examined to find out what kind of progress the optimization is making. The acerr files obviously can be examined to find out if the reported error rate is still decreasing or has leveled off. Also, the rwpics command (see rwpics man page) can be used to make, from a set of bspt files, a set of grayscale IHead images depicting these regional weights sets in their proper geometric layout. Rwpics can also make two other kinds of pictures: grayscale pictures of a set of estimated gradients (egrad files), and grayscale-binary (i.e. 0 and 255 pixels) pictures showing the signs of the elements of estimated gradients. (The blocks whose estimated gradient elements, i.e. estimated partials, are negative, are ones whose weights will be increasing as optrws takes steps in the anti-estimated-gradient direction.) For the "optimal" set of regional weights, just use the final bspt file produced before the optrws run stops by itself (because of doing the specified number of iterations) or, if optimization appears not to be making much more progress, kill the optrws process and use the last bspt file produced. Or, it could also be interesting to do testing using various basepoints, to find out whether the decreases in the activation error during optimization correspond to error decreases on a test set, i.e. to find out whether even small improvements in the weights in the sense of training error rate, are actually significant in the sense of generalizing to other data. (The weights seem to generalize well, not too suprisingly since there are only 210 of them, hardly a large enough number for them to be capable of becoming very specifically tuned to the training data in such a way as to have little generalization value.) The parameters of optrws are specified by parameter files. The program first reads pcasys/parms/optrws.prs, which contains default values of some of its parameters; then it reads the user-provided parameters file whose name is given as the argument. Consult optrws.prs to find out what the parameters are, and as an example of the format of a parameters file. Optrws.prs specifies default values for the parameters that have defaults, and it also has a comment concerning each parameter that has no default value. The user parameters file must specify a value for each parameter that does not have a default, and it also can specify default-overriding values for one or more of the other parameters. Optrws can start several simultaneous instances of another program, optrwsgw, each time it needs to estimate the gradient, if desired. This can reduce the time needed for optimization, if there are several processors available. To use this feature, set acerror_stepped_points_nprocs in your parameters file to a value > 1 (probably should be <= number of processors available). If the operating system on your computer does not implement fork() and execl(), then the Makefile for optrws should be modified by appending -DNO_FORK_AND_EXECL to the definition of CFLAGS, so that a different subset of the code will be compiled and the linker will thereby find no unresolved references. <prsfile> A file specifying values of some or all of the parameters. Parameters not specified in this file assume default values. pcasys/parms/optrws.prs Contains default values for some of the optrws parameters. The remaining parameters, with no default values must be specified in the user prsfile. Default settings in pcasys/parms/optrws.prs n_feats_use 64 How many (first) features of each K-L feature vector to use. irw_init 0.1 Initial value for irw. irw_initstep 1.0 Initial step size for irw. irw_stepthr .01 Optimization of irw stops when step size becomes smaller than this threshold. grad_est_stepsize .001 Step size for secant-estimation of gradient. n_linesearches 2 Number of (estimate gradient, line search) iterations to do. linesearch_initstep .1 Initial step size for line search. linesearch_stepthr .01 Line search stops when its step size becomes smaller than this threshold. tablesize 1000 Size of a table used to hold pairs of values corresponding to previous computations of the error, either as a function of irw or as a function of distance along downhill-pointing line. Lookup in this table saves some cycles by avoiding repeated calculations. acerror_stepped_points_nprocs 1 How many processes to use when computing the activation error at the points stepped to from a basepoint, in order to compute the approximate gradient by secant method. If 1, optrws computes the error at all stepped points itself. If > 1, optrws starts this many child processes, each of which computes the error at an interval of the stepped points. verbose y If y, write progress messages to standard output. ascii_outfiles n Whether outputfiles are to be ascii (y) or binary (n). klfvs_file (no default, user must set) File containing K-L feature vectors to be used as prototypes set, and also as "tuning" set, for the optimization. Usually the output on the lintran function. classes_file (no default, user must set) File containing the classes that go with the feature vectors of klfvs_file. Must be a pcasys "classes" formatted file. n_klfvs_use (no default, user must set) How many of the K-L feature vectors to use (off the top). eigvecs_file (no default, user must set) File containing the eigenvectors. outfiles_dir (no default, user must set) The directory in which optrws is to produce its output files. From test/pcasys/execs/optrws/optrws.src: % optrws optrws.prs Optimizes the regional weights for a set of feature vectors based on the parameters set in the file optrws.prs. rwpics (1A) NIST/ITL/DIV894/Image Group