AFRS Performance Evaluation Tests - NIST

TEST RESULTSIn connection with the conversion of theAPRSfrom ascanner, certain special test materials were prepared.flying spotAmongscanner to a solid statethese was a recording of the"Medium",digitalscainneronAPRSNo.4 andThesefingerprintswerescannedthebyand "Heavy".the gray scale data were recorded on disk. This permitted the same gray scale data to befingerprints of a single individual using three different inking densities, "Light",directed repeatedly to thescale data shouldAPRSpreprocessor for minutiae detection.produce cm identical minutiaein the identity or position of the detectedlisteach timeitisIdeally, theprocessed.minutiae represent imperfect performance of thepreprocessor which might be caused by noise or otherfactors.This gray scale data recorded on disk was input to the preprocessor oftimes.The detected minutiae weresame grayDifferencesAPRSNo. 4 eightregistered and clipped and then recorded as eightfrom each of three cards. The data from the lightly inked card werecard 105, medium as card 203, and heavy as card 301 from system No. 4.fingers of datarecorded asThe same procedure was followedusing thesamefingerprintsonAPRSNo.2.This systemhas had decoupling capacitors installed on the printed circuit cards in the preprocessor."GROUND TRUTH- MATCH RESULTSThe "ground truth" data forThe objective wasat NBS.these three fingerprints was obtained using the Graphicto record only those minutiae thatappeared within the areacovered by the clipping box. This would provide minutiae from an area that wasAPen commontomachine read data was centeredin the field of view of the Graphic Pen in order to approximately define the clipping boxboundaries. Then the fingerprint was positioned under the magnifier lenses and adjusted tomake the minutiae agree with those on the machine-read plot. Next the outline of theclipping box was centered on the machine-read plot. Finally the minutiae of the fingerprintthat were within or even slightly beyond the boundaries of the clipping box were read bytwo different operators to develop the "ground truth" data base for the minutiae of thatthe area of the machine-read minutiae data.plot of thefinger.There may be an element of uncertainty in this process. This comes about because in themachine-read data, the clipping box is centered on a position that is found as a result ofprocessing the ridge flow data derived from the machine-read gray scale information. Ifnoise in the system affects minutiae positions,might cause the clipping box boundaries to begray scale data through the preprocessor.boundaries of the clipping boxin theitmayalso affect ridge flow data.Because ofthis,"ground truth" datasome ofsetmayThiseach pass of identicalslightly different forthe minutiae near theactuallybe outside theboundaries of the clipping box for the machine-read data on one or more of the processingpasses.reason.Candidate minutiae have been identified that might have been missed for thisThese candidates consist of "ground truth" minutiae that were located very closeMoore andR. Park, "The Graphic PenAnEconomical SemiautomaticFingerprint Reader", Proc. 1977 Carnahan Conference on Crime Countermeasures, UKYBU112, ORES, Univ of Kentucky, Lexington, KY 40506.R. T.J.4-

to the clippingbox boundarywere not detectedthatinany of the multiple passes of thegray scale data through the preprocessor.OnOfthe lightly inked fingerprint there were 83 minutiae on theinitial"ground truth"list.1, 2, 3, 4, 8, 12, 23, 56, and 78 were not detected in anypass of the data. It is reasonable to assume that they might have been outside theclipping box. A value of 74 is therefore used for T in Equation 1 for this fingerprint. Onthe fingerprint with medium inking, there were 75 minutiae on the initial list and three ofthese. Nos. 1, 49 and 53 were not detected in any pass of the data, so T is assumed to haveOn the fingerprint with the heavy inking, there were 76 minutiae on thea value of 72.Four of these. Nos. 5, 6, 19 and 51 had no detections in anyinitial "ground truth" list.pass of the machine-read data so a value of 72 is assumed for T.these, nine minutiae, Nos.The toleranceon displacement and orientationlimitsconsidered mates are 0.4With the grayscale datafrom these three fingerprints recorded on disk and then enteredas eight independent passes into the preprocessor ofEquation 1 are shownin the followingTableAPIS No. 426Tot.Score;The mean number-732/592 -1.24-582/576 -1.0157-468/576 -0.8124%of true minutiae detected on the lightly inked print was 17.5 orthe "ground truth" minutiae.OnNo.4, the values for use in1Detection Scores forOnAPRS1.TABLEtruth".for a pair of minutiae to bemm (four X, Y matcher units) and 12 degrees (Theta).Onthe heavily inked printitthemedium inkedwas 33.375, or46%printitwas23.5, or33%allof "groundof "ground truth".the lightly inked print, 34 different true minutiae were detected on one orbut only six were consistently detected onofeight passes.Onthemoremedium inkedpasses,print,46were detected on one or more passes, and only five were detectedon all eight passes. With the heavily inked print, 48 true minutiae were detected on one ormore passes and 16 were detected on all eight passes. The distribution of the numbers ofdifferent true minutiae5

minutiae and consistency of their detection with eight passes through the preprocessor ofAPRSNo. 4isshowninTable2.TABLE 2Consistency of Detections.LIGHTPigure 1 showsMEDIUMAPRSNo, 4HEAVYMinutiae detected 1 time71135536853326423651716 ’"2 times458765168344648Minutiae detected one or more timessame datathisin a different way.Thisisa cumulative distribution of thepercentage of the minutiae detections that are repeated eight times, seven times,Pigures2,etc.3 and 4 show distribution of distances that the minutiae were displaced from the"ground truth" position in each of the eight passes of the gray scale data through theNo. 4 preprocessor for thelight,medium andheavily inked image.APRSPigure 5 shows asummaryof this same information for the eight passes and all three inkings. This figureshows that the most frequent displacement is two units with the lightly inked images. Theheavily inked image shows a noticeable peak at one unit of displacement.The preprocessor ofcapacitors toitsAPRS No2.has been modified with the addition ofsome decouplingprinted circuit cards in an attempt to reduce internally generated noise.this reader as with APRS No. 4.Gray scale datawas recorded on disk and passed through the preprocessor eight times and the minutiaedetections from each pass compared with the "ground truth" data from the light, medium andheavily inked fingerprint. The results are shown in Table 3. They are very comparable tothe performance shown for APRS No. 4 in Table 1. The lightly inked print provided a meanThe same procedures were followed withof 17.5 true minutiae or24%of "ground truth".Onthemedium inkednumber of detections was 21.625 or 30% of "ground truth". Thea mean of 33.125 minutiae which is 46% of the "ground truth".Thedistribution ofnumbers of minutiae and the consistency of detection4.6print themeanheavily inked print yieldedisshowninTable

TABLESDetection Scores forAPRSNo. 51317314154548155962215155324058m452 422Tot.Score: -733/592 -1.244032 4n430-476/576 -0.83-64S/576 -1.13TABLE 4Consistencv of Detections.LIGHTAFRSNo. 2MEDIUM HEAVYMinutiae detected 1 time4116842"2 times33"31114"4222"525456473517W44”"6If "8' ""II"46 Minutiae detected one or more timesFigure 6 shows the cumulative distribution of the repeated minutiae detections shown inTable4.Figures7,8 and 9 show distribution of distances that the minutiae were displaced from the"ground truth" positioninNo. 2 preprocessor for theFigure 10 shows ainkings.each of the eight passes of the gray scale data through thelight,summarymedium andof thisAFRSheavily inked image.same information7and all threetwo units with the lightfor the eight passesThis figure shows that the most frequent displacementis

and medium inked images.one unit of displacement.Theimage again shows a noticeable peakheavily inkedonlyat"FALSE GROUND TRUTH" RESULTSnamebeen given to the"Falsegroundin (X,Y, Theta) space representing every minutiae detection, true ortruth"theismultiple passes of thedetection logic.Eachthat hassame grayfieldset of candidate "cluster" sitesisof the mates comprising the cluster.limits ofdisplacement and rotationfromand minutiaeand rotated to a "best fit"Mating minutiaepass.recalculated to be the mean position and orientationMinutiae that do not have mates within the toleranceof detected minutiaeistranslatedposition with respect to the field of minutiae detected in theare identified and the "cluster" sitefalse, resultingscale data through the preprocessorstillfirstestablish candidate sites thatmay become populatedwith subsequent passes of the data.Since the gray scale data thatisusedwould be expectedN"clusters"isa constanttonumbergray scale dataisproduceisdigital,performancein a noise-freeeach having a population ofof minutiae detections andMpassed through the detector.isThenumbertheMenvironmentmates, whereof times that theNsamefact that this is not the case is amatter of serious concern.Table 5 shows the performance ofprints.The same dataAFRSNo. 4on themedium andlight,heavily inkedshown in Fig. 11 as the cumulative distribution of the percentageThere is very small difference in the results from the light andwhile the heavily inked print produced noticeably more consistentisof multiple detections.medium inkedprint,detections.Table 6 andFig.AFRS12 show the corresponding performance ofsuperiority of the heavily inked printismuchlessNo.2.Herepronounced.TABLE 5SUMMARY OF "FALSE GROUND TRUTH" CLUSTER SIZES AFRS NO. 4-LIGHT T18Minutiae detected33"""tl"43"2 times"3"4"5"6H"time1 8"""M"Different minutiae detected150one or more times8the

TABLE 6SUMMARY OF "FALSE GROUND TRUTH" CLUSTER SIZES APRS N0.2-MEDIUM HEAVYLIGHT5853573627171411121514138988Minutiae detected"""168141512182537""mtime1"2 times"3"4"5"6"7"8"""""Different minutiae detectedone or more timesThe softwareroutines that developed the datashown aboveinTables 5 and 6 for"falseground truth" performance hst the identity of each of the minutiae that formed each ofManythe clusters.of these clusters appear to have been formed from false detectionsthat occurred repeatedly.Acomparison of the numbers of minutiae detected eight timesinTables 2 and 4 with the number of minutiae detected eight times in Tables 5 and 6 revealsinformation aboutFIRST PASShowfrequently these repeated false detections occurred.MATCH RESULTSThis comparison takes the minutiae detections resulting from thedata from thelight,themedium, and thefirstpass of the gray scaleandheavily inked fingerprinttreatsitas thereference against which the other seven passes of the data from each of these fingerprintsare compared.as toisto provide athatisAsisthe case of the "false ground truth" data therewhether the minutiae detected are true ormeasure of the displacementshumannot biased by theThefalse.in positionposition selection rules asisnodistinctionmadechief utility of this comparisonand orientation of mating minutiaeit isin the"ground truth" data.Figures 13 through 15 show the seven individual results obtained from each of the threedegrees of inking onAPRSNo.4.Figures 16 and 17 show thesummarydata for thissystem.Figures 18 through 20AFRSNo.2 andFigs. 21show the sevenindividual resultsand 22 cover the summaryfrom each of the three inkings onresults for this system.These data show that the most frequently observed displacement in position for matingminutiae is one matcher unit, although there are displacements greater than that in a fewpercent of the detections. With no noise, it would be expected that there would be nodisplacement, since the same digital data is input each time.9

CONCLUSIONSimprovements resulting from upgrading thereaders with the new sohd state scanner subsystems, there are serious problems in theAPRS. These are manifest in the form of inconsistencies in the detection of minutiae,both true and false, even when identical gray scale data is input to the preprocessor andThesetest results indicate that despite theminutiae detectioncircuitry.The performanceismoreerraticwhenlightly orinked fingerprints are used than with heavily inked, high contrast, images.moderatelyIn thesetests,only about one quarter of the true minutiae were detected in lightly inked prints, one thirdmoderately inked prints and one half in the heavily inked prints on any singlePresumably the observed inconsistencies in performanceresult from noise in the preprocessor. Minor inconsistencies in detection performance couldbe expected to be caused by the recursive behavior of the ridge valley filter, but thesewould be expected to be constrained to the top few per cent of the fingerprint image area.Since several of the instances where minutiae were detected consistently (eight times) occurin the top five percent of the image area, it is not believed that the recursive attributes ofthe filter are a major contributor to the inconsistencies observed in minutiae detection.in theprocessing of the fingerprint data.Itisbelieved that the detection probability displayed in these testsissupporting an effective automated latent fingerprint identification system.that long range planning should contemplate reconversion of the files toautomatednot capable ofItissuggestedbe used for anRigorous quality control measures are suggested to insure thatthe quality of the re-converted file data is maintained at an acceptable level.It islatent system.believed thatsome ofthe software routines listed in theAppendix would be appropriatecandidates for use in support of this function.Thesetest data also strongly suggest that there is no significant difference in theperformance of APRS No. 2 and No. 4 and that the FBI made the correct choice in notadding the decoupling capacitors to the remainder of the APRS preprocessors. It isbelieved thatmuch moreextensive measures will be required to correct the problems inthese systems.show a definite trend to improved performance with increasing inkThis is in agreement with vendor claims that readers perform better withelectronically generated fingerprints which provide images with high contrast.Finally, these datadensity.10

CUM.%BY NO. OF DETECTIONS 4--ALLGROUND3INKINGSTRUTHFigure111

MINUTIAE DISPLACEMENT FROM GROUND TRUTHSAME GRAY SCALE DATA - APRS #4 - LIGHT COFNOFigure122

MINUTIAE DISPLACEMENT FROM GROUND TRUTHSAME GRAY SCALE DATA - APRS #4 - MEDIUM EOFNO.Figure133

MINUTIAE DISPLACEMENT FROM GROUND TRUTHSAME GRAY SCALE DATA - AFRS #4 - HEAVY AEOFNOFigure144

NO. OF SPLACEMENTDATA-AFRSFROMMEDIUM 4-GROUNDALLCIRCLE;3TRUTHINKINGSHEAVYFigure155

CUM.%BY NO. OF DETECTIONS 2--ALLGROUND3INKINGSTRUTHFigure166

MINUTIAE DISPLACEMENT FROMGROUND TRUTHSAME GRAY SCALE DATA - AFRS #2 - LIGHT AEOFNOfigure177

MINUTIAE DISPLACEMENT FROM GROUND TRUTHSAME GRAY SCALE DATA - APRS #2 - MEDIUM EOFNO.Figure188

MINUTIAE DISPLACEMENT FROM GROUND TRUTHSAME GRAY SCALE DATA - APRS §2 - HEAVY AEOFNOFigure199

NO. OF TRUTHINKINGSHEAVYFigure 1020

CUM.%BY NO. OF CTIONSAFRS 4--FALSEALL3GROUNDINKINGSTRUTHFigure2111

CUM.%BY NO. OF CTIONSAFRS§2--FALSEALL3GROUNDINKINGSTRUTHFigure 1222

MINUTIAE DISPLACEMENT FROM PASS #1SAME GRAY SCALE DATA - APRS #4 - LIGHT INKINGFigure 1323

MINUTIAE DISPLACEMENT FROM PASS #1SAME GRAY SCALE DATA - AFRS #4 - MEDIUM AEOfNOFigure 1424

MINUTIAE DISPLACEMENT FROM PASS #1SAME GRAY SCALE DATA - APRS #4 - HEAVY AEOFNO.Figure 1525

NO. OE ISPLACEMENT-APRSMEDIUM 4EROM-ALLPASSCIRCLE:3 1INKINGSHEAVYFigure2616

CUM.%BY NO. OF DETECTIONS 4-ALLWITH3PASSINKINGS1Figure 1727

MINUTIAE DISPLACEMENT FROM PASS #1SAME GRAY SCALE DATA - APRS #2 - LIGHT INKINGFigure2818

MINUTIAE DISPLACEMENT FROM PASS#1SAME GRAY SCALE DATA - AFRS #2 - MEDIUM INKINGFigure 1929

MINUTIAE DISPLACEMENT FROM PASS #1SAME GRAY SCALE DATA - APRS #2 - HEAVY INKINGFigure 2030

1NO. OF MINUTIAESCALETRIANGLE;DATADISPLACEMENT-APRSMEDIUM 2FROM-PASSALLCIRGLE:3 1INKINGSHEAVYFigure 2131

CUM.%BY NO. OF DETECTIONS§2--ALLWITH3PASSINKINGS1Figure 2232

APPENDIXcode for theprogram FINDTRANS and its subroutines. The program and subroutines are written inFORTRAN-77. Only two of the subroutines, FBIOPEN and FBIREAD are machine specific andonly run on a VAX/VMS system, while the remaining subroutines and program should beThis appendix contains a verbal description, flow diagrams, and the entireportable.The program FINDTRANS calculates the "best fit" transformation values for delta X, delta Yand delta THETA to be used in matching a pair of fingerprints. Additionally, the programevaluates the accuracy of a match performed using these transformation values. A "best fit"may be defined as the orientation in translation and rotation of a comparison fingerprintwith a mating base fingerprint, such that the positions of the majority of minutiae from thecomparison fingerprint are relatively close to those of the base fingerprint as evidenced byhighest matching score. A "best fit" transformation may be one of several possible "best fit"transformations because of the many combinations of transformations that can be generatedwith equal scores. In this situation the first "best fit" transformation found is thetransformation used.As input, FINDTRANS requires the name of the file containing the base fingerprint (the onewhich will not be transformed) and the quantity of other prints to be compared against thisbase fingerprint. For each comparison print, the minutiae positions are read in and theis assumed at the transformation values of 0 in X, 0 in Y, 0 in rotation.then issues four calls to the subroutine BESTFIT, which calculates newtransformation values. Each time BESTFIT is called, a constant theta difference limit of 12degrees is used. However, the straight line distance tolerance decreases with each call toBESTFIT. The values that are used for the four calls are 30, 15, 8, and 4 respectively. Onlyinitial"bestfit"FINDTRANSfour calls oflittleBESTFIT are used as thesign of change."bestWhen FINDTRANSisfit"values after the fourth iterationdone, the "bestfit"transformationdiagnostic output of the match performed using this transformationisisshow veryknown andgenerated.FINDTRANS, as it performs the majority of thesaves the assumed "best fit" values that were passed to it and asksthe user to input the beginning and ending rotational range with values stated in degrees.This range is used to test for the best transformation rotation value. This will be a valuebetween the assumed "best fit" rotation plus the beginning value, increasing in increments ofone degree, to the assumed "best fit" rotation plus the ending value. The beginning andending values may be negative but the beginning value must alwa

TableofContents Introduction 1 PerformanceMetrics 1 ComparisonWith"GROUNDTRUTH" 2 ComparisonWith"FALSEGROUNDTRUTH" 3 ComparisonWithFirstPassData 3 TestResults 4 "GROUNDTRUTH"MatchResults. 4 "FALSEGROUNDTRUTH"Results 8 FirstPassMatchResults