System Design and Performance Assessment: A Biometric Menagerie Perspective. Norman Poh Lecturer Dept of Computing

Transcription

1 System Design and Performance Assessment: A Biometric Menagerie Perspective Norman Poh n.poh@surrey.ac.uk Lecturer Dept of Computing 1

2 2

3 Biometric menagerie Terms & performance metrics Variability vs Repeatability Biometric score theory Procedures User-specific score calibration Biblio note DET confidence intervals Performance synthesis User-specific fusion Doddington s classification Yager and Dunston s Biometric Menagerie Index User-ranking User-specific fusion with modality selection Concluding remarks User-ranking on unseen data Tricks of the trade

4 Is biometric menagerie persistent? Yes Assumption: Is the template fixed? No If you update your template, you need to recompute the separability criterion Yes Will the system operate in the same environment as the development set? No You need to compute the separability criterion for the expected operating condition Yes Is the system expected to operate for less than 1 year? The menagerie is likely to persist No At some point, you will throw away the template anyway. Menu 4

5 Thank you 5

6 TERMS AND PERFORMANCE METRICS 6

7 Some basic terms Gallery Template or Template or Template reference or reference reference Query or probe Biometric matcher Score 7

8 Genuine score Template Query or probe Biometric matcher Match score Genuine score (Mated score) 8

9 Zero-effort impostor score Template Query or probe Biometric matcher Non-match Score Zero-effort impostor score (Non-mated score) 9

10 Nonzero-effort impostor score Fake/spoofed sample Template Query or probe Biometric matcher Spoof score Nonzero-effort impostor score 10

11 Biometric Menagerie scope Future research Rattani, Ajita, Norman Poh, and Arun Ross. "Analysis of userspecific score characteristics for spoof biometric attacks." Computer Vision and Pattern Recognition Workshops (CVPRW), 2012 IEEE Computer Society Conference on. IEEE,

12 likelihood p(y ω = I) p(y ω = G) pdf: probabilistic density function Score, y 12

13 likelihood p(y ω = I) p(y ω = G) Score, y 13

14 Probability likelihood Decision threshold, Δ y > Δ p(y ω = I) p(y ω = G) False rejection False acceptance 1 False match rate, FMR (False Acceptance Rate, FAR) False non-match Rate, FNMR (False Rejection Rate, FRR) FMR Δ = P(y > Δ ω = I) FNMR Δ = P(y Δ ω = G) = 1 P(y Δ ω = I) P y Δ ω = p y ω dy Δ Score, y 14

15 1) ROC FMR Δ FMR Δ FNMR Δ 2) ROC (variant) True Acceptance Rate 3) DET FAR Δ False Alarm Rate Note: Φ is the PDF of a normal distribution; Φ 1 is its inverse Φ 1 (FMR Δ ) 15

16 FNMR[%] FNMR[%] ROC versus DET Receiver s Operating Characteristic (ROC) curve EER Detection Error Trade-off (DET) curve EER FMR[%] EER: Equal Error Rate or Cross-over error rate Further reading: Biometric Testing and Statistics FMR[%] Menu 16

17 VARIABILITY AND REPEATABILITY OF BIOMETRIC PERFORMANCE 17

18 Skill of the operator Quality of the template emographics of target users Stochastic nature of the algorithms Sources of variability Template We are trying to collect the scores and measure the system performance Sensor type Sensor type Session variability Biometric matcher Match score Demographics of impostor Query or probe Environment Lighting Temperature Application scenario Covert vs overt Level of user s cooperation Mounted or mobile Biometric features of the design population Wayman, James L., Antonio Possolo, and Anthony J. Mansfield. "Modern statistical and philosophical framework for uncertainty assessment in biometric performance testing." IET biometrics 2.3 (2013):

19 Three types of biometric tests Technology test Use a (sequestered) database to test/compare algorithms Repeatable because many factors are fixed once data is collected Scenario testing Measure the overall system (sensor + algorithm) performance in a typical realworld application domain It is not repeatable because another field test will not give exactly the same result Operational testing Model the performance in a specific application using a specific biometric system Not repeatable Source: Philips, P.J., Martin, A., Wilson, C., Przybocki, M.: An introduction to the evaluation of biometric systems, IEEE Comput., 2000, 33, (2), pp Comments: Wayman, James L., Antonio Possolo, and Anthony J. Mansfield. "Modern statistical and philosophical framework for uncertainty assessment in biometric performance testing." IET biometrics 2.3 (2013):

20 Repeatability Same SOP for enrolment, same operators Same biometric system & sensor Same application scenario Similar (controlled) environment Comparable user size but different subjects Installation site A Installation site B Will the two curves be the same? How much do they vary? 20

21 Clients-based bootstrapping 200 clients 100 clients 100 clients 100 clients 100 clients Variability due to the user composition is observed under technology test Questions: How much do they vary? Why do they vary? Can we extrapolate (generalize) to another scenario? 21

22 Session variability why it matters? Queries of genuine samples are from the same day or different days Dev set Eva set Would the performance be similar? 22

23 With vs without session variability Same session Different session Biosecure DS2 score database 23

24 Quiz Should we consider session variability when testing a biometric system? A: No, we should not because we want to fix all parameters in favour of experimental repeatability B: Yes, we must do so in every biometric testing Need a hint? Explanation Do you know of any biometric system that is designed to operate only on the same day as enrolment? We must acknowledge that we can never fix all parameters and fixing all is not always desirable Menu 24

25 BIOMETRIC SCORE THEORY 25

26 likelihood What assumptions? The demographic composition of the zeroeffort impostors Nonzero-effort impostor Adverse conditions The quality of the environment may vary User interaction A myriad # of unmeasurable factors Score, y Samples are independently and identically distributed (i.i.d.) 26

27 likelihood Formation of nonmatch scores Zero effort impostors Client templates/ models Y 1 I Impostor scores Y 2 I Y 3 I score 27

28 likelihood Formation of match scores Client accesses Client template/ model Client scores Y 1 G Y 2 G Y 3 G score Scores of 3 clients 28

29 likelihood An Observed Experimental Outcome Impostor scores Y 1 I Client scores Y 1 G Strong correlation exists among scores generated by the common client model Y 2 I Y 2 G Y 3 I Y 3 G Samples are independently and identically distributed (i.i.d.) given the claimant ID score 29

30 likelihood How would i.i.d. samples look like? Impostor scores Y I Client scores Y G Samples are independently and identically distributed (i.i.d.) score 30

31 Which is correct? Y I Y G Y 1 I Y 1 G Y 2 I Y 2 G Y 3 I Y 3 G Samples are i.i.d. Samples are i.i.d. given the claimed ID 31

32 Local pdf = Claimant-specific pdf Global pdf = System-level pdf Global, p(y ω = G) Global: p(y ω = I) Local, p(y ω = I, j) Local, p(y ω = G, j) XM2VTS face system (DCTmod2,GMM) 200 users/clients 3 genuine scores per user (blue curve) 400 impostor scores per user (red curve) 32

33 How are the local and global models related? J p y ω = p y ω, j P(j ω) j=1 for ω {G, I} The class-conditional pdf associated to claimant j The prior probability of claimant j given the class label ω There is no reason why one claimant is more important than another; so, P(j ω) should be uniform: P j ω = 1 J p y ω = 1 J J j=1 p y ω, j The system-level class-conditional pdf score is simply an average of the pdf of the claimant-specific pdf. 33

34 Consequence The user composition has direct influence on the system performance even when all experimental conditions are fixed P y Δ ω = G J = 1 P y Δ ω = G, j J j=1 P y > Δ ω = I J = 1 P y > Δ ω = I, j J Menu j=1 34

35 Classification of literature BIBLIOGRAPHY NOTES 35

36 Studies of biometric menagerie Classification of research in Biometric Menagerie Measuring the phenomenon Looking at the extremes of distribution Relative ranking Doddington s zoo Yager and Dunstone s classification User discrimination Directly characterising the variability Biometric menagerie index Reducing the impact Score normalisation Fusion Parametric: Z-norm, F- norm, Group-based F-norm Training-based: MS-LLR, logistic regression 36

37 Studies of biometric menagerie Measuring the phenomenon Looking at the extremes of distribution Relative ranking Doddington s zoo Yager and Dunstone s classification User discrimination Directly characterising the variability Biometric menagerie index Reducing the impact Score normalisation Fusion Parametric: Z-norm, F- norm, Group-based F-norm Training-based: MS-LLR, logistic regression 37

38 Some references Doddington s zoo Yager and Dunstone s classification User discrimination Biometric menagerie index Parametric: Z-norm, F- norm, Group-based F-norm Training-based: MS-LLR, logistic regression Doddington, George, et al. Sheep, goats, lambs and wolves: A statistical analysis of speaker performance in the NIST 1998 speaker recognition evaluation. NATIONAL INST OF STANDARDS AND TECHNOLOGY GAITHERSBURG MD, Yager, Neil, and Ted Dunstone. "The biometric menagerie." Pattern Analysis and Machine Intelligence, IEEE Transactions on 32.2 (2010): * N. Poh and J. Kittler, A Methodology for Separating Sheep from Goats for Controlled Enrollment and Multimodal Fusion, in 6th Biometrics Symposium, pp , * N. Poh, A. Ross, W. Li, and J. Kittler, A User-Specific and Selective Multimodal Biometric Fusion Strategy by Ranking Subjects, Pattern Recognition 46(12): , N. Poh and J. Kittler, A Biometric Menagerie Index for Characterising Template/Modelspecific Variation, in Int l Conf. on Biometrics (ICB'09), * N. Poh, A. Rattani, M. Tistarelli and J. Kittler, Group-specific Score Normalization for Biometric Systems, in IEEE Computer Society Workshop on Biometrics (CVPR), pages 38-45, * N. Poh and S. Bengio, An Investigation of F-ratio Client-Dependent Normalisation on Biometric Authentication Tasks, in IEEE Int'l Conf. Acoustics, Speech, and Signal Processing (ICASSP), volume I, pages , 2005 * N. Poh and M. Tistarelli, Customizing Biometric Authentication Systems via Discriminative Score Calibration, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012 * N. Poh and J. Kittler, Incorporating Variation of Model-specific Score Distribution in Speaker Verification Systems, IEEE Trans. Audio, Speech and Language Processing, 16(3): , Menu 38

39 DODDINGTON S ZOO 39

40 likelihood Doddington s Zoo Strong impostor High FAR High FRR Good clients p y ω = I, j (the majority of p y ω = G, j the claimants) p y ω = I p y ω = G Score, y Doddington, George, et al. Sheep, goats, lambs and wolves: A statistical analysis of speaker performance in the NIST 1998 speaker recognition evaluation. NATIONAL INST OF STANDARDS AND TECHNOLOGY GAITHERSBURG MD,

41 likelihood Characterising claimant-specific pdf Need more and more data First moment: μ j I Second moment: σ j I 2 Skewness: m 3 j I Kurtosis: m 4 j I First moment: μ j G Second moment: σ j G 2 Skewness: m 3 j G Kurtosis: m 4 j G p y ω = I, j p y ω = G, j Score, y 41

42 Mean μ = E y [y] Variance σ 2 = E y [ μ y 2 ] Skewness E y y μ 3 σ 3 Kurtosis E y y μ 4 σ 4 Peakedness Source: Wikipedia entries: mean, variance, skewness,and kurtosis 42

43 Claimants What s the difference between a wolf and a lamb??? Each subject may have multiple query images p y ω = I, j = 1 p y ω = I, j = p y ω = I, j = 3 1 Y 1 G Y 1 I 2 3 Y 3 I Y 2 G Y 3 G Y 1 I Lambs are claimants whose zero-effort impostor scores are relatively high (1 template each) Goats are claimants whose genuine scores are relatively low 43

44 Claimants (1 template each) What s the difference between a wolf and a lamb??? queries 1 Y 1 G Y 2 G 3 Y 3 G Wolves are those who can generate very high ZE impostor scores Claimants can be wolves, too! 44

45 Wolves are lambs and lambs are wolves when We use a symmetric matcher and consider a closed set G Y 1 2 G Y 2 3 G Y G Y 1 2 G Y 2 3 G Y 3 A symmetric matcher is one that satisfies: M(a,b) = M(b,a) 45

46 likelihood Why do wolves matter? p y ω = I, j is a wolf p y ω = I p y ω = G Score, y Murakami, Takao, Kenta Takahashi, and Kanta Matsuura. "Towards optimal countermeasures against wolves and lambs in biometrics." Biometrics: Theory, Applications and Systems (BTAS), 2012 IEEE Fifth International Conference on. IEEE,

47 Claimants Summary: Goats and lamb detection Each subject may have multiple query images Gen Y 1 G Y 1 I Y 1 G Y 1 I 100 G Y 100 I Y 100 G Y 100 I Y

48 Summary: Goats and lamb detection Y 1 G Y 1 I G Y 100 I Y

49 Sort Sort Summary: Goats and lamb detection Goats Statistic Scores Scores Statistic μ 1 G Y 1 G The original Goat test was implemented using nonparametric Kruskal-Wallis rank sum test; we use a simplified test here Y 1 I μ 1 I G μ 100 G Y 100 I Y 100 I μ Lamb 49

50 Claimants Summary: Wolves detection Each subject may have multiple query images 1 Y 1 G Gen G Y 100 W W W W μ 1000 μ 1001 μ 4999 μ 5000 Wolves Sort Menu 50

51 YAGER AND DUNSTONE S CLASSIFICATION 51

52 Mean Genuine score Biometric Menagerie Mean Impostor Score Yager, Neil, and Ted Dunstone. "The biometric menagerie." Pattern Analysis and Machine Intelligence, IEEE Transactions on 32.2 (2010):

53 Mean Genuine score Mean Impostor Score N. Poh, A. Rattani, M. Tistarelli and J. Kittler, Group-specific Score Normalization for Biometric Systems, in IEEE Computer Society Workshop on Biometrics (CVPR), pages 38-45,

54 Group-based F-norm N. Poh, A. Rattani, M. Tistarelli and J. Kittler, Group-specific Score Normalization for Biometric Systems, in IEEE Computer Society Workshop on Biometrics (CVPR), pages 38-45, Menu 54

55 CONCLUDING REMARKS 55

56 Concluding remarks Are subjects inherently hard to match? Match score analysis: This question depends on how well we understand the environment in which biometric operates, thus requiring some generalization capability to unseen data Non-match score analysis: This can be answered! Why biometric menagerie: a conjecture 56

57 Application Scenario Are subjects inherently hard to match? Match score analysis {U j 1, U j 2, U j 3, U j 4 } U j 1 U j 2 M(U j 1, U j 2 ) U j 1 U j 2 U j 3 U j 4 U j 1 U j 2 U j 3 U j 4 template queries n-1 scores n 2 = n(n 1) scores 2 User-specific score calibration: to reduce the menagerie effect Template selection: to find a representative template Poh, Norman, and Josef Kittler. "Incorporating variation of model-specific score distribution in speaker verification systems." IEEE Transactions on Audio, Speech and Language Processing 16.3 (2008): Jain, Anil, Umut Uludag, and Arun Ross. "Biometric template selection: a case study in fingerprints." Audio-and Video-Based Biometric Person Authentication. Springer Berlin Heidelberg,

58 Sort Sort Are subjects inherently hard to match? Non-match score analysis Set A queries Set B queries Statistic μ 1 I Y 1 I A Y 1 I B Statistic μ 1 I I μ I Y 200 A I Y 200 B I μ 100 Claimants Impostor-based Uniqueness Measure Compare Impostor-based Uniqueness Measure 58

59 Non-match scores are very stable 100 impostors 1000 impostors 4000 impostors 8000 impostors Klare, Brendan F., and Anil K. Jain. "Face recognition: Impostor-based measures of uniqueness and quality." Biometrics: Theory, Applications and Systems (BTAS), 2012 IEEE Fifth International Conference on. IEEE,

60 XM2VTS score-level benchmark database 200 claimants 13 experiments Download the database here: μ j ω=i μ j ω=c Norman Poh and Samy Bengio, Database, Protocol and Tools for Evaluating Score-Level Fusion Algorithms in Biometric Authentication, Pattern Recognition, Volume 39, Issue 2, Pages , σ j ω=i σ j ω=c One data set 60

61 Generalisation ability of biometric menagerie ~ 1 month apart 25 impostors 8 samples 70 impostors 8 samples 2 genuine samples from a 3 genuine samples (from a different different session then enrolment) session then enrolment) Can we predict the statistics? N. Poh, A. Ross, W. Li, and J. Kittler, A User-Specific and Selective Multimodal Biometric Fusion Strategy by Ranking Subjects, Pattern Recognition 46(12): ,

62 One of the 13 experiments μ j ω=c σ j ω=c μ j ω=i σ j ω=i μ j ω dev σ j ω dev 62

63 How predictable are the statistics? μ j ω=i μ j ω=c σ j ω=i σ j ω=c One of the 13 experiments 63

64 L.J.P. van der Maaten and G.E. Hinton. Visualizing High-Dimensional Data Using t-sne. Journal of Machine Learning Research 9(Nov): ,

65 Conjectures Biometric menagerie is systemdependent Feature space It is related to subject representativeness Poor quality enrolment Even with well-controlled enrolment, the menagerie still exists User-specific schemes can reduce the effect Changing menagerie membership Teli, Mohammad Nayeem, et al. "Biometric zoos: Theory and experimental evidence." Biometrics (IJCB), 2011 International Joint Conference on. IEEE, Wittman et al, Empirical Studies of the Existence of the Biometric Menagerie in the FRGC 2.0 Color Image Corpus, CVPRW 2006 * Paone, Liberating the Biometric Menagerie Through Score Normalization, PhD thesis, 2013 * N. Poh and J. Kittler, Incorporating Variation of Model-specific Score Distribution in Speaker Verification Systems, IEEE Trans. Audio, Speech and Language Processing, 16(3): , Popescu-Bodorin, Nicolaie, Valentina Emilia Balas, and Iulia Maria Motoc. "The Biometric Menagerie A Fuzzy and Inconsistent Concept." Soft Computing Applications. Springer Berlin Heidelberg,

66 Scenario Should the menagerie classify the users or their templates? Classify users Classify templates Algorithm U j 1 U j 2 U j 3 U j 4 Users score characteristics persist! Number of goats overlapped out of 50 trials Teli, Mohammad Nayeem, et al. "Biometric zoos: Theory and experimental evidence." Biometrics (IJCB), 2011 International Joint Conference on. IEEE, N. Poh, A. Ross, W. Li, and J. Kittler, A User-Specific and Selective Multimodal Biometric Fusion Strategy by Ranking Subjects, Pattern Recognition 46(12): , Popescu-Bodorin, Nicolaie, Valentina Emilia Balas, and Iulia Maria Motoc. "The Biometric Menagerie A Fuzzy and Inconsistent Concept." Soft Computing Applications. Springer Berlin Heidelberg, Menu 66

67 BIOMETRIC MENAGERIE INDEX 67

68 Desirable properties of a menagerie index index = 0 0 index 1 index=1 68

69 Biometric Menagerie Index Client i Between client variance Client j Between client variance Within client variance Within client variance μ i μ μ j Total variance N. Poh and J. Kittler, A Biometric Menagerie Index for Characterising Template/Modelspecific Variation, in Int l Conf. on Biometrics (ICB'09),

70 BMI= E Mean of between client variance = Total variance Total variance 70

71 Global mean = client mean Within-client variance dominates Between client variance dominates 71

72 XM2VTS database Nonmatch comparisons Match comparisons N. Poh and J. Kittler, A Biometric Menagerie Index for Characterising Template/Modelspecific Variation, in Int l Conf. on Biometrics (ICB'09), Get the data set: 72

73 Findings Impostor BMI is insensitive to the choice of (zero-effort) impostor client BMI > Impostor BMI High client BMI values suggest that client-specific threshold or normalization is essential Menu 73

74 USER RANKING 74

75 Sort User ranking Genuine ZE Impostor 1 Y 1 G Y 1 I Calculate the statistic μ 1 G μ 1 I, σ 1 I Rank the users 100 G I Y 100 Y 100 G μ 100 I I μ 100, σ

76 Why rank the subjects? N. Poh and J. Kittler, A Methodology for Separating Sheep from Goats for Controlled Enrollment and Multimodal Fusion, in 6th Biometrics Symposium, pp , Quality control during enrollment If the newly acquired template is not representative of the person, acquire a better (more representative) sample A tool to assess the worstscenario DET curve How bad could a system perform for a group of weak (the weakest) users? The experience of each user in interacting with a biometric device matters! A modality selection criterion in fusion Use only the more representative biometric modality instead. Novel group specific decision An optimal decision threshold for each group of users N. Poh, A. Ross, W. Li, and J. Kittler, A User-Specific and Selective Multimodal Biometric Fusion Strategy by Ranking Subjects, Pattern Recognition 46(12): , N. Poh, A. Rattani, M. Tistarelli and J. Kittler, Group-specific Score Normalization for Biometric Systems, in IEEE Computer Society Workshop on Biometrics (CVPR), pages 38-45,

77 EER Criteria to rank the users Calculate the performance of each user template and rank them somehow! For each user/template! [IEEE Trans. SP, 2006] F-ratio 77

78 Many parametric discrimination criteria available Multivariate statistics 78

79 Sort User-ranking algorithm Calculate the statistics Apply separability criterion Rank the users Group their scores Plot performance Worst perf μ 1 G μ 1 I, σ 1 I C G μ 100 I I μ 100, σ 10 C Best perf N. Poh and J. Kittler, A Methodology for Separating Sheep from Goats for Controlled Enrollment and Multimodal Fusion, in 6th Biometrics Symposium, pp ,

80 Results on Face, Fingerprints & Iris Biosecure database a face example F-ratio d-prime random N. Poh and J. Kittler, A Methodology for Separating Sheep from Goats for Controlled Enrollment and Multimodal Fusion, in 6th Biometrics Symposium, pp ,

81 Performance disparity across users

82 Findings Very few weak users; but they dominate the errors F-ratio and d-prime can be used to rank the users Can we rank the subject with maximal stability (on unseen data)? B-ratio (PRJ 2013) ranks users on unseen data Menu 82

83 USER RANKING ON UNSEEN DATA 83

84 Rank the users on unseen data! Independent data sets (different sessions) C 1 C 1 Dev set Eva set C 100 C 100 Unseen data Can we predict the statistics? We designed 6 criteria and searched the best one N. Poh, A. Ross, W. Li, and J. Kittler, A User-Specific and Selective Multimodal Biometric Fusion Strategy by Ranking Subjects, Pattern Recognition 46(12): ,

85 Predictability of user ranking on unseen data F-ratio B-ratio (in F- norm domain) XM2VTS 13 systems 85

86 Sort Sort Summary Calculate the statistics μ 1 G μ 1 I, σ 1 I Apply separability criterion C 1 Rank the users We should be able to predict problematic users before they use the technology Re-enrol with new samples G μ 100 I I μ 100, σ 10 C 100 Use a different biometric modality μg I I 101 μ 101, σ 101 C 101 Recommend another authentication mechanism Menu 86

87 BIOMETRIC MENAGERIE PROCEDURE 87

88 Biometric Menagerie Procedures Definition 1: Interventions that allow us to exploit Biometric Menagerie to improve the system s future operation User ranking user tailoring Definition 2: Algorithms that exploit Biometric Menagerie to estimate the system s future operational performance Performance intervals estimation Performance synthesis Menu 88

89 PERFORMANCE CONFIDENCE ESTIMATION 89

90 DET confidence intervals Different subjects, different demographics Dev set Eva set How well is the predicted curve? Measure coverage 90

91 Coverage 91

92 likelihood Conventional bootstrap Impostor scores Client scores score 92

93 Bootstrap 1 Bootstrap 2 Bootstrap 3 93

94 A two-level bootstrap Handle between model variance by boostrapping the enrolled identities Handle within model variance by bootstrapping the within-model scores N. Poh and S. Bengio, Performance Generalization in Biometric Authentication Using Joint User-specific and Sample Bootstraps, IEEE Trans. on Pattern Analysis and Machine Intelligence, 29(3): , March

95 likelihood Subset Bootstrap Impostor scores Client scores Bolle, Ratha & Pakanti, Error analysis of pattern recognition systems: the subsets bootstrap, CVIU 2014 score 95

96 Two-level Bootstrap Bootstrap users/models U=3 times 1 2 Bootstrap samples within a model: S=4 times Plot DET Level 1 Level 2 96

97 DET confidence via bootstrapping 97

98 98

99 User-specific bootstrap Sample bootstrap DET angle Fix the users (claimants), bootstrap only the samples Bootstrap the claimants, and keep all the scores due to the chosen claimants 99

100 Effect of subject samples Increased subject Better precision 100

101 Findings User-induced variability is more important than the sample variability Increasing the subject population reduces the uncertainty Coverage achieved 70-90% on NIST 2005 Speaker Evaluation data set 101

102 Average coverage of 24 NIST 2005 systems Two-level Bootstrap Bootstrap subset S=1, U varies Conventional bootstrap, S varies Within model variability, U=1, S varies Configuration: Trained on 31 users; tested on another 62 users 102

103 Quiz How to best generate 10,000 genuine score samples in order to conduct a biometric test? Recruit 10 subjects each contribute samples Recruit 100 subjects each contribute samples Recruit 1000 subjects each contribute 10+1 samples Menu 103

104 On-going research PERFORMANCE SYNTHESIS 104

105 Synthesizing DET curves to other operating conditions Data with recorded conditions Operational data that we haven t collected Previous operational data Application specs A new installation Performance synthesis Compare prediction quality 105

106 Benefits of performance synthesis Liberates performance curves from its assessment data set Provides a framework for test reporting that promotes documentation and measurement of experimental conditions Potentially reduces cost in testing 106

107 Negative (impostor) scores Positive (genuine) scores Negative scores (cond 1) Negative scores (cond Q ) Calculate DET Positive scores (cond 1) Positive scores (cond Q ) Estimate cdf Estimate cdf Estimate cdf Estimate cdf Estimate confidence Bootstrap Bootstrap Bootstrap Bootstrap Combine Combine Set the priors of P Q ω 107

108 The underpinning theory p y ω = p y ω, Q P(Q ω) Q P y Δ ω = P y Δ ω, Q P(Q ω) Q 108

109 P y Δ ω = G J = 1 P y Δ ω = G, j J j=1 P y > Δ ω = I = 1 J J j=1 P y > Δ ω = I, j 109

110 Study 1: Fingerprint with NFIQ 4&

111 Simulated DET curves of varying quality High quality tendency [ ] Low quality tendency [ ] Equal prior quality [ ] Relative weights of the 4 curves: [Q1 Q2 Q3 Q4&5] 111

112 Study II: Cross-sensor performance Cross-sensor Sensor 1 (Biometrika) Sensor 2 (Ital) 112

113 Simulated DET curves with mixture of data Sensor 1 Sensor 2 Sensor 1 + Senso 2 + Cross-sensor [1 0 0] [0 1 0] [1 1 1] The ratio shows [ S1 S2 cross-sensor] 113

114 Assessment with DET angles 114

115 Achievable coverage Coverage achieved between 60% and 80% Menu 115

116 USER-SPECIFIC SCORE NORMALISATION 116

117 Score normalisation Original matching scores Z-norm F-norm Bayesian classifier [IEEE TASLP 08] 117

118 Menu 118

119 USER-SPECIFIC FUSION 119

120 Client-specific fusion 120

121 Some results Menu [IEEE TASLP 08] 121

122 USER-SPECIFIC & SELECTIVE FUSION 122

123 System architecture Recommend usage Compute criterion Criterion = B-norm of Fratio Modality 1 Modality 2 F-norm F-norm Fusion with selection Compute criterion Recommend usage 123

124 Rationale: The B-ratio of F-norm F-ratio = μ j G μ j I σ j G +σ j I B-ratio = μ j G μ j I σ j I The discrimination power of the user is fully captured by σ F j I σ F j G unknown The B-norm of F-ratio = μ Fj G μ F j I = 1 σ F j I σ F j I μ F j I =0 μ F j G =1 124

125 Cost savings XM2VTS database. # of users J = 200 # of system = 2 125

126 A user-specific and selective multimodal biometric fusion strategy by ranking subjects, PRJ

127 EER (%) Performance on test set Computational savings 10 of 15 fusion experiments shown here Menu 127