Pattern Recognition. Approximating class densities, Bayesian classifier, Errors in Biometric Systems

Transcription

1 htt://.cubs.buffalo.edu attern Recognton Aromatng class denstes, Bayesan classfer, Errors n Bometrc Systems B. W. Slverman, Densty estmaton for statstcs and data analyss. London: Chaman and Hall, 986. htt://.acsu.buffalo.edu/~tulyakov/aers/tulyakov_009_cybersecurty_bometrcs.df

2 htt://.cubs.buffalo.edu Bayesan classfcaton Suose e have classes and e kno robablty densty functons of ther feature vectors. Ho some ne attern should be classfed? Bayes classfcaton rule: classfy to the class hch has bggest osteror robablty >? : osteror Usng Bayes formula, e can rerte classfcaton rule: > lkelhood? : ror

3 htt://.cubs.buffalo.edu Estmatng robablty densty functon. arametrc df estmaton: model unknon robablty densty functon of class by some arametrc functon ; θ and determne arameters based on tranng samles. Eamle: Gaussan functon ; µ Non-arametrc df estmaton:. Hstogram. K nearest neghbor 3. Kernel methods arzen kernels or ndos π N ˆ ϕ N N h h 4. Other methods estmatng cumulatve dstrbuton functon frst, SVM densty estmaton, etc. l / e µ s the number of tranng samles

4 htt://.cubs.buffalo.edu Estmatng kernel dth Non-arametrc df estmaton: Fed kernels: N ˆ ϕ Adatve kernels: or h h N ˆ ϕ N h h N ˆ ϕ N h h N ˆ ϕ

5 htt://.cubs.buffalo.edu Estmatng kernel dth Recall, e used mamum lkelhood method for arametrc df estmaton: ma θ ˆ X ; θ ma θ ˆ N,,..., N θ ma ˆ k ; θ θ k Can e use same method for estmatng the kernel dth? h ˆ No, the ma s not achevable: ma h ma h N k N k ˆ k f h 0 ; h N ϕ k N h h

6 htt://.cubs.buffalo.edu Estmatng kernel dth Soluton: searate model data kernel centers from testng data - cross-valdaton technque N k h h k h N ma ϕ ˆ k h k h h N

7 htt://.cubs.buffalo.edu Estmatng kernel dth Tred mamum lkelhood cross-valdaton and stll dverges? N k h k h k h N ma ϕ Ths mght haen f data s somehat dscrete: ˆ Soluton - truly searate model data from testng data: N k k k h h h N ma ϕ

8 htt://.cubs.buffalo.edu Eamles of df estmaton arzen-ndo kernel estmates of a unvarate normal densty usng dfferent ndo dths and numbers of samles. DHS Heurstc method of dth calculaton: h h n n

9 htt://.cubs.buffalo.edu Eamles of df estmaton arzen-ndo kernel estmates of a bmodal densty usng dfferent ndo dths and numbers of samles.

10 htt://.cubs.buffalo.edu Eamles of df estmaton arzen-ndo kernel estmates of a bvarate normal densty usng dfferent ndo dths and numbers of samles.

11 htt://.cubs.buffalo.edu Error n df estmaton ˆ Dscreancy beteen true densty and ts estmaton : ˆ MSE ˆ E{ ˆ } - Mean Square Error MSE ˆ E{ ˆ } MISE ˆ [ ] { Eˆ} { Eˆ} + + E{ ˆ } { Eˆ} [ ] Eˆ + [ E{ Eˆ ˆ} ] E{ ˆ } d - Mean Integrated Square Error E } E + } Eectatons are { ˆ E{ ˆ} + { ˆ ˆ taken over the set of ossble aromatons or over the sets of tranng samles

12 htt://.cubs.buffalo.edu Bas and varance of estmaton error [ ] Eˆ + [ E{ Eˆ ˆ} ] MSE ˆ ˆ Eˆ Bas Varance Eˆ - Average aromaton y E ˆ K y dy h h Bas s the dfference beteen true densty and average aromaton Varance s the dfference beteen average aromaton and ndvdual aromatons Smaller kernel dth reduces bas, but ncreases varance.

13 htt://.cubs.buffalo.edu Bas and varance of estmaton error If some assumtons on the true densty are made e.g. then t s ossble to analytcally fnd the kernel dth hch gves smallest Slverman arzen: h ot { } / 5 { } / 5 / 5 ϕ t dt d /5 k n Otmal kernel dth gets smaller hen the number of tranng samles ncreases. For otmal kernel dth also decreases: MISE ˆ d < MISE ˆ n MISE ~ C ϕ { } / 5 4 / 5 d n Note, that s unknon. Above formulas are useful for theory, but not for ractcal alcatons. For multvarate df aromaton: MISE 4/4+ d The erformance decreases eonentally hen the number of dmensons ncreases ~ n

14 htt://.cubs.buffalo.edu Bayesan classfcaton Bayes classfcaton rule: classfy to the class hch has bggest osteror robablty >? : Bayes classfcaton rule mnmzes the total robablty of msclassfcaton. Cost of errors. Errors haen hen samles of class are ncorrectly classfed to belong to class, and samles of class are classfed to belong to class. The cost of makng these errors can be dfferent : λ - the cost of msclassfyng samles of class λ - the cost of msclassfyng samles of class

15 htt://.cubs.buffalo.edu Total cost or rsk of classfcaton Classfcaton algorthm slts feature sace nto to decson regons: R R R - samles n ths regon are classfed as beng n class R - samles n ths regon are classfed as beng n class d - the roorton of samles of class beng classfed as class d - the roorton of samles of class beng classfed as class d R R R d - the roorton of all nut samles beng class but classfed as beng n class - the roorton of all nut samles beng class but classfed as beng n class Cost λ + d λ d R - total cost

16 htt://.cubs.buffalo.edu Mnmzng total cost of classfcaton Snce and cover hole feature sace R R + R R d d Thus + } { d d Cost λ λ Cost s mnmzed f ncludes only onts here + + } { R R R d d d Cost λ λ λ λ λ R 0 < λ λ

17 htt://.cubs.buffalo.edu Bayesan classfcaton Bayesan classfer s an otmal classfer mnmzng total classfcaton cost. Such classfer s ossble only f e have full knoledge about class dstrbutons. If then classfy as class. λ λ > If then classfy as class. λ λ Alternatvely, assumng non-zero terms, the class assgnment s based on testng hether or λ λ λ λ > Decson surface searates to decson regons. λ λ - lkelhood rato λ λ < > - lkelhood rato test

18 htt://.cubs.buffalo.edu erformance of Bayesan classfcaton Denote: t λ λ - decson threshold t > t t R - decson regon of class for threshold t > t t R - decson regon of class for threshold t t t R - decson regon of class for threshold t R d t MR - msclassfcaton rate for class and threshold t t R d t MR - msclassfcaton rate for class and threshold t

19 htt://.cubs.buffalo.edu erformance of Bayesan classfcaton MR t MR t and comletely characterze the erformance of a Bayesan classfer λ,λ For a gven msclassfcaton costs and ror class robabltes e fnd λ, t λ Then the msclassfcaton cost s Cost λ MR t + λ MR t

20 htt://.cubs.buffalo.edu ROC of a Bayesan classfcaton MR and MR are used only th the the same. t t t Thus the arameter s not mortant and the erformance of a Bayesan classfer can be characterzed only by the relatonshs beteen and. MR MR t t t MR Eamle of an otmal Bayesan ROC curve and some non-otmal classfer s ROC curve. 0 MR MR MR For a gven the of a non otmal classfer should be bgger; otherse non-otmal classfer ould outerform otmal.

21 htt://.cubs.buffalo.edu Bometrc Alcaton Tyes Verfcaton System : Clam s made enrollee dentty User s bometrc s matched only th stored bometrc of clamed enrollee The decson to accet clam s made usng only one matchng score Identfcaton System :N No clam about dentty s made User s bometrc s matched th stored bometrcs of all enrolled ersons The hghest matchng score determnes the most robable enrollee The decson about accetng dentfcaton attemt s made based on the matchng score for that enrollee and otonally usng other matchng scores too Screenng Matchng aganst a atch lst Ooste of verfcaton

22 htt://.cubs.buffalo.edu erformance of Verfcaton System For bometrc matchers erson dentty verfcaton e dstngush to classes: Genune erson s clamed dentty s correct Imostor - erson s clamed dentty s n correct The decson for genune class s to accet, and the decson for the mostor class s to reject. The decson s usually done based on a sngle matchng score of nut bometrc th the enrolled bometrc temlate of clamed dentty erson. Instead of otmal > < θ use > < θ If s monotonous, these decsons are equvalent. Instead of MR t and MR t use FAR t m d - false accet rate for threshold t > t FRR t gen d - false reject rate for threshold t < t

23 htt://.cubs.buffalo.edu Errors n Verfcaton Systems Each verfcaton attemt has to ossbltes:. Genune event - nut bometrcs and stored bometrcs from clamed dentty belong to the same erson.. Imostor event - nut bometrcs s dfferent from clamed dentty bometrcs. The scores roduced by matchng algorthm ll have dstrbutons: gen s s m s s genune mostor event event

24 htt://.cubs.buffalo.edu Errors n Verfcaton Systems FAR and FRR are determned by the decson rule accet or reject results of recognton. Usually FAR and FRR are defned usng some threshold: s FAR θ s ds > θ θ m Also called: False Match Rate FMR mostor event θ FRR θ s ds s < θ gen genune Also called: False Non-Match Rate FNMR event

25 htt://.cubs.buffalo.edu Errors n Verfcaton Systems

26 htt://.cubs.buffalo.edu erformance of Bometrc Matchers.5 m robablty FAR t FRRt gen t 3 Scores FRRt FARt

27 htt://.cubs.buffalo.edu ROC Curve FAR θ FRRθ ROC curve connects and curves. Note that they both use same θ at the same tme, so e are able to construct such lot.

28 htt://.cubs.buffalo.edu Tyes of ROC Curve log FAR θ Takng and log FRR θ nstead of FARθ and FRRθ s reasonable f they are small.

29 htt://.cubs.buffalo.edu Tyes of ROC Curve

30 htt://.cubs.buffalo.edu Usng ROC Curve

31 htt://.cubs.buffalo.edu Comarng ROC Curves

32 htt://.cubs.buffalo.edu Comarng ROC Curves Area under ROC curve - FRR vs FAR reresents the robablty that random genune score s hgher than random mostor score.

33 htt://.cubs.buffalo.edu Comarng ROC Curves Comare match and non-match score denstes by d-rme method: µ µ d σ m m + n σ n

34 htt://.cubs.buffalo.edu Comarng ROC Curves EER FRR θ FAR θ FRR θ FAR θ Equal Error Rate EER: at such as θ Mnmum Total Error Rate TER: TER mn FRR θ + θ FAR θ

35 htt://.cubs.buffalo.edu Trade-offs Selecton of the oeratng ont n a artcular alcaton s a trade-off beteen securty and convenence.

36 htt://.cubs.buffalo.edu Estmatng FAR and FRR In contrast to estmatng df, FAR and FRR are easly estmated: FAR t > t m d { { > t, s s mostor mostor } } FRR t < t gen d { { < t, s s genune genune } } Tyes of ROC curves: { FRR t, FAR t} < t< { FAR t, gen FRR t + m FAR t} {log FRR t, log FAR t} < t< < t<

37 htt://.cubs.buffalo.edu Usng FAR and FRR In Bayesan frameork e ant to mnmze total cost: Cost C FA mostor s > θ mostor C FA + C m FA FR genune FAR θ + C s < θ genune FR gen FRR θ θ C, C, mostor, genune Correct settng of n verfcaton alcaton requres estmatng

38 Eamle htt://.cubs.buffalo.edu Consder the roblem of deloyng bometrc matcher for an amusement ark admsson C C FA m FR gen $0 % Cost $ 99% C cost of accetng mostor to the ark - robablty of mostor attemts - cost of rejectng genune user - robablty of genune attemts FA m FAR θ + FAR θ +.99 FAR θ +.99 C FR FRR θ gen FRR θ FRR θ

39 htt://.cubs.buffalo.edu Face matcher C better mnmzes cost Cost. FAR θ +.99 FRR θ

40 htt://.cubs.buffalo.edu m 0% If e had more mostor attemts, say, then matcher r ould get loer cost Cost FAR θ +.9 FRR θ

41 htt://.cubs.buffalo.edu Errors n Identfcaton Systems N eole are enrolled n the database. The recognton algorthm erforms N matchngs th outut scores: s > s >... > s N the scores are ordered by magntude, but not by eole d The decson algorthm usually consdered: Accet class f s > θ and θ > s >... > Reject otherse s N

42 htt://.cubs.buffalo.edu Errors n Identfcaton Systems Other tyes of decsons nvolve selectng a subset of matched classes: Threshold based: Rank based: s > s >... > s > θ k -select all classes bgger than threshold -select k classes th best scores Hybrd: -select based on threshold, f not successful select k classes based on rank

43 htt://.cubs.buffalo.edu FNMR and FMR n Identfcaton Systems FNMR False non-match rate: FNMR θ FRR θ gen s ds s < θ FMR False match rate: θ genune FMR θ ma s s < θ corresond s to all N -mostor event s s > θ corresond s to all N -mostor event < θ corresond s to one mostor event θ m > θ corresond s to one mostor event s ds FAR θ N

44 htt://.cubs.buffalo.edu FMR for dfferent N 4 3 robablty Scores

45 htt://.cubs.buffalo.edu Errors n Identfcaton Systems FMR and FNMR mght not adequately descrbe the erformance of dentfcaton systems - closed set / oen set dentfcaton - rejectng all dentfcaton results mght be a correct choce - errors are connected: mostor mght be a to choce, but genune s also hgher than the threshold Score belongng to dfferent classes are usually deendent, so FMR can not be effectvely estmated by means of FAR Stll no good standard for measurng dentfcaton system erformance ests

46 htt://.cubs.buffalo.edu Investgatng valdty of..d. assumton Eamle: Identfcaton system th classes genune and mostor Scenaro : CorrIdent< Scenaro : CorrIdent Deendence of scores nfluences erformance n dentfcaton systems