Pattern Recognition and Machine Learning. Face Detection using Color Histograms

Transcription

1 Patten Recognition and Machine Leaning James L. Cowley ENSIMAG 3 - MMIS Fall Semeste 016 Lesson 1 Octoe 016 Outline Face Detection using Colo Histogams Notation Two-Class Patten Detectos Pefomance Evaluation Metics... 5 ROC Cuves...5 Tue Positives and False Positives...5 Pecision and Recall...7 F-Measue...7 Accuacy:...8 Matthews Coelation Coefficient Detecting Skin Pixels in Colo histogams Histogams as an Estimation of Poaility of Colo... 1 Colo Skin Detection using Luminance and Chominance Bayesian Tacking of Gaussian Blos Moment Calculations fo Blos Bayesian Tacking (exta mateial... 18

2 Detecting Skin with Colo Lesson Notation x d A featue. An oseved o measued value.! X A vecto of D featues. D The nume of dimensions fo the vecto! y The tue class fo an osevation X! { X! m } {y m } Taining samples fo leaning. y m An annotation (o gound tuth function R( X! m The Detection function o Recognition function R( X! m " { P, N} M The nume of taining samples. M T The nume of taining samples in the taget class h( X! A multidimensional histogam of X! Q The nume of cells in a histogam -

3 Detecting Skin with Colo Lesson 1. Two-Class Patten Detectos A patten detecto is a classifie with K=. Class k=1: The taget patten. Class k=: Eveything else. Patten detectos ae used in compute vision, fo example to detect faces, oad signs, pulicity logos, o othe pattens of inteest. They ae also used in signal communications, data mining and many othe domains. The patten detecto is leaned as a detection function g! X ( followed y a decision ule, d(. The detection function is leaned fom a set of taining data composed of M sample osevations { X! m } whee each sample osevation is laeled with an indicato vaiale {y m } y m = P o Positive fo examples of the taget patten (class 1 y m = N o Negative fo all othe examples (class A atio of histogams can povides a disciminant function as : g( X!! h = p(c 1 X = 1 ( X! h 1 ( X! + h ( X! = h! 1( X h( X! whee h 1 ( X! is the histogam of taining samples fo the taget class and h( X! is the histogam of ALL taining samples. To simplify the algea, we can ceate a lookup tale L( X! = h 1( X! h( X! This gives a disciminant function of g(! X = L(! X Osevations fo which g(! X > 0.5 ae estimated to e memes of the taget class. This will e called POSITIVE o P. Osevations fo which g(! X " 0.5 ae estimated to e memes of the ackgound. This will e called NEGATIVE o N. -3

4 Detecting Skin with Colo Lesson We can encode this as a decision function to define ou detection function R( X! m R( X! = d(g( X! = # %! P if g( X > 0.5 N if g( X! " 0.5 Fo taining we need gound tuth (annotation. Fo each taining sample the annotation o gound tuth tells us the eal class y m y m = P! # " Taget - Class % N othewise The Classification can e TRUE o FALSE. if R(! = y m then T else F This gives R(! = y m AND R(! R(! " y m AND R(! R(! " y m AND R(! R(! = y m AND R(! = P is a TRUE POSITIVE o TP = P is a FALSE POSITIVE o FP = N is a FALSE NEGATIVE o FN = N is a TRUE NEGATIVE o TN To ette undestand the detecto we need a tool to exploe the tade-off etween making false detections (false positives and missed detections (false negatives. The Receive Opeating Chaacteistic (ROC povides such a tool -4

5 Detecting Skin with Colo Lesson. Pefomance Evaluation Metics ROC Cuves Two-class classifies have long een used fo signal detection polems in communications and have een used to demonstate optimality fo signal detection methods. The quality metic that is used is the Receive Opeating Chaacteistic (ROC cuve. This cuve can e used to descie o compae any method fo signal o patten detection. The ROC cuve is geneated y adding a vaiale Bias tem to a disciminant function. R(! X = d(g(! X + B and plotting the ate of tue positive detection vs false positive detection whee R(! is the classifie as in lesson 1. As the ias tem, B, is swept though a ange of values, it changes the atio of tue positive detection to false positives. Fo a atio of histogams, g( X! m is a poaility anging fom 0 to 1. B can ange fom less than 0.5 to moe than When B 0.5 all detections will e Negative. When B > +0.5 all detections will e Positive. Between 0.5 and +0.5 R( X! will give a mix of TP, TN, FP and FN. The ias tem, B, can act as an adjustale gain that sets the sensitivity of the detecto. The ias tem allows us to tade False Positives fo False Negatives. The esulting cuve is called a Receive Opeating Chaacteistics (ROC cuve. The ROC plots Tue Positive Rate (TPR against False Positive Rate (FNR as a function of B fo the taining data { X! m }, {y m }. Tue Positives and False Positives Fo each taining sample, the detection as eithe Positive (P o Negative (N IF g(! +B > 0.5 THEN P else N The detection can e TRUE (T o FALSE (F depending on the indicato vaiale y m IF y m = R(! THEN T else F -5

6 Detecting Skin with Colo Lesson Comining these two values, any detection can e a Tue Positive (TP, False Positive (FP, Tue Negative (TN o False Negative (FN. Fo the M samples of the taining data {! }, {y m } we can define: #P as the nume of Positives, #N as the nume of Negatives, #T as the nume of Tue and #F as the nume of False, Fom this we can define: #TP as the nume of Tue Positives, #FP as the nume of False Positives, #TN as the nume of Tue Negative, #FN as the nume of False Negatives. Note that #P = #TP + #FN And #N = #FP+ #TN The Tue Positive Rate (TPR is TPR = #TP # P = #TP #TP+# FN The False Positive Rate (FPR is FPR = # FP # N = # FP # FP+#TN The ROC plots the TPR against the FPR as a ias B is swept though a ange of values. When B is less than 0.5, all the samples ae detected as N, and oth the TPR and FPR ae 0. As B inceases oth the TPR and FPR incease. Nomally TPR should ise monotonically with FPR. If TPR and FPR ae equal, then the detecto is no ette than chance. The close the cuve appoaches the uppe left cone, the ette the detecto. y m = R( X! m d(g(! +B > 0.5 T F P Tue Positive (TP False Positive (FP N False Negative (FN Tue Negative (TN -6

7 Detecting Skin with Colo Lesson Pecision and Recall Pecision, also called Positive Pedictive Value(PPV is the faction of etieved instances that ae elevant to the polem. PP = TP TP + FP A pefect pecision scoe (PPV=1.0 means that evey esult etieved y a seach was elevant, ut says nothing aout whethe all elevant documents wee etieved. Recall, also known as sensitivity (S, hit ate, and Tue Positive Rate (TPR is the faction of elevant instances that ae etieved. S = TPR = TP T = TP TP + FN A pefect ecall scoe (TPR=1.0 means that all elevant documents wee etieved y the seach, ut says nothing aout how many ielevant documents wee also etieved. Both pecision and ecall ae theefoe ased on an undestanding and measue of elevance. In ou case, elevance coesponds to Tue. Pecision answes the question How many of the Positive Elements ae Tue? Recall answes the question How many of the Tue elements ae Positive? In many domains, thee is an invese elationship etween pecision and ecall. It is possile to incease one at the cost of educing the othe. F-Measue The F-measues comine pecision and ecall into a single value. The F measues measue the effectiveness of etieval with espect to a use who attaches times as much impotance to ecall as pecision. The F 1 scoe weights ecall highe than pecision. F 1 Scoe: -7

8 Detecting Skin with Colo Lesson F 1 = TP TP + FP + FN The F1 scoe is the hamonic mean of pecision and sensitivity. This is the geometic mean divided y the aithmetic mean. Accuacy: Accuacy is the faction of test cases that ae coectly classified (T. ACC = T M = TP +TN M whee M is the quantity of test data. Note that the tems Accuacy and Pecision have a vey diffeent meaning in Measuement theoy. In measuement theoy, accuacy is the aveage distance fom a tue value, while pecision is a measue of the epoduciility fo the measuement. Matthews Coelation Coefficient The Matthews coelation coefficient is a measue of the quality of inay (two-class classifications. This measue was poposed y the iochemist Bian W. Matthews in MCC takes into account tue and false positives and negatives and is geneally egaded as a alanced measue that can e used even if the classes ae of vey diffeent sizes. The MCC is in essence a coelation coefficient etween the oseved and pedicted inay classifications MCC esults a value etween +1 and -1, whee +1 epesents a pefect pediction, 0 no ette than andom pediction and 1 indicates total disageement etween pediction and osevation. MCC = TP "TN # FP " FN (TP + FP(TP + FN(TN + FP(TN + FN The oiginal fomula given y matthews was: -8

9 Detecting Skin with Colo Lesson M = Total quanitity of test data: M = TN +TP + FN + FP S = P = TP + FN M TP + FP M MCC = TP M " S # P PS(1" S(1" P -9

10 Detecting Skin with Colo Lesson 3. Detecting Skin Pixels in Colo histogams A atio of colo histogams can e used to constuct a simple detecto fo skin pixels in images. Colo skin pixels can e used to detect faces, hands and othe skin coloed egions in images. Conside the images in the FDDB (Face Detection Data Set and Benchmak data ase maintained at UMASS: This data-ase was constucted fo face detection and not fo skin detection. Face egions have een hand-laeled as oth oxes and ellipses. All images ae RGB with each pixel containing 3 colos: Red, Geen and Blue. We will use the ellipses as gound tuth fo skin egions. This will ceate eos in the taining data. A typical image with and annotated face egions as an ellipse looks like: Note that thee ae skin pixels that ae NOT in the ellipse (hand, eas, neck etc, and thee ae non-skin pixels that ARE in the face (hai, teeth, etc. This will lead to mino eos in the detection. You jo will e to measue the impact of these eos in uilding a face detecto using detection of skin pixels. Annotations of face egions ae epesented as an elliptical egion, denoted y a 6- tuple ( a,, θ, c x, c y, 1 whee a and efe to the half-length of the majo and mino axes, θ is the angle of the majo axis with the hoizontal axis, and c x and c y ae the column and ow image coodinates of the cente of this ellipse. Ellipse Data: 00/07/4/ig/img_

11 Detecting Skin with Colo Lesson the standad fom of an ellipse with a majo axis along the hoizontal (x axis is: ( x " c x a when otated y θ: ( + y " c y =1 ((x " c x cos(#+ (y " c y sin(# a + (x " c x sin(#+ (y " c y cos(# ( =1-11

12 Detecting Skin with Colo Lesson Histogams as an Estimation of Poaility of Colo Assume that we have a colo image, whee each pixel (i,j is a colo vecto,! p (i, j, composed of 3 integes etween 0 and 55 epesenting Red, Geen and Blue. " R%! ' p (i, j = G ' # B We can uild a colo histogam of the image y counting the nume of times that each unique value of (R, G, B occus in the image. To do this we allocate a tale h(r, G, B of 56 x 56 x 56 cells, with each cell initially set to zeo. If each colo is epesented y 8 its, then the tale h(r, G, B has Q=( 8 3 cells. Q=( 8 3 =56 3 = 4 = 4 0 = 16 Meg Cells. Using ou ule of 8 samples pe cell, we need 3 x 4 = 7 = 18 Meg Pixels. If the taining data is composed of 18x18 images, then how many images do we need fo h(x? 18 x 18 pixels = 7 x 7 = 14 = 16 K pixels. Thus we need 8000 images! What can we do? Often we can educe oth the nume, D, of featues and the nume of values, N, fo each featue. Fo example, fo many colo images, N=3 colo values ae sufficient to detect ojects. We simply divide each colo R, G, B y 8. R' = Tunc(R/8, G'=Tunc(G/8, B'=Tunc(B/8. This educes ou histogam to 8x8x8=( 3 3 = 9 =51 cells We can also use ou knowledge of physics to look fo featues that ae "invaiant". Colo Skin Detection using Luminance and Chominance. Luminance captues suface oientation (3D shape while Chominance is a signatue fo oject pigment (identity. Thus it is convenient to tansfom the (RGB colo pixels into a colo space that sepaates Luminance fom Chominance. -1

13 Detecting Skin with Colo Lesson " L % " R% ' C 1 ' ( ' G ' # # B C Nomalizing out luminance povides a popula space fo skin detection: the (,g space. Luminance: L= R+G+B Chominance : = c 1 = R R + G + B g = c = G R + G + B These ae often called "" and "g" in the liteatue. The (, g space is often used to detect skin coloed pixels. It is common to nomalize and g to natual numes coded with N values etween 0 and N 1 y : # R # G = tunc% N " ( g = tunc% N " ( R + G + B' R + G + B' Skin pigment is geneally always the same chominance value. Luminance can change with pigment density, and skin suface oientation, ut chominance will emain invaiant. Thus we can use (,g as an invaiant colo signatue fo detecting skin in at each pixel. Fom expeience, N = 3 colo values seems to wok well fo skin, ut lage values of N may wok ette with some data sets. Suppose we have a set of K taining images {! p (i, j} of size RxC whee each pixel is an RGB colo vecto. This gives a total of M=NxRxC colo pixels. Suppose that M T of these ae laeled as skin pixels. We allocate two tale : h(,g and h skin (,g of size N x N.! Fo all i,j,k in the taining set { p k (i, j } BEGIN # R # G = tunc% N " ( g = tunc% N " ( R + G + B' R + G + B' h(, g = h(, g+1 IF the pixel! p k (i, j is skin egion THEN h skin (, g = h skin (,g+1 and M skin =M skin +1 END -13

14 Detecting Skin with Colo Lesson As efoe, we can otain a lookup tale L skin (,g that gives the poaility that a pixel is skin. L skin (,g = h skin (,g h(, g Given a new RGB image! p (i, j fo each pixel (i,j # R # G = tunc% N " ( g = tunc% N " ( R + G + B' R + G + B ' g(! p (i, j = L skin ((i,j, g(i,j We can display the esulting poailities as a gay scale image, y multiplying y each value y 56. White is high poaility. Black is low poaility. (images fom a Bayesian skin tacking in eal time We can impove the detection y gouping adjacent skin pixels into skin los and tacking skin los ove time. -14

15 Detecting Skin with Colo Lesson 4. Bayesian Tacking of Gaussian Blos Rathe than epesent a skin egion as a collection of pixels, we can calculate a Gaussian Blo. A "Blo" epesents a egion of an image. Gaussian los expess a egion in tems of moments. Assume of image of poailities of the detection of a taget: T(i,j, whee fo each pixel: T(i,j = L skin ((i,j, g(i,j The zeoth moment of the poailities is the mass (sum of poailities. Aveage mass epesents confidence. The fist moment gives is the cente of gavity. This is the "position" of the lo. The second moment is the covaiance. This gives size and oientation. We typically enclose the lo in some ectangula Region of Inteest (ROI in ode to avoid "distaction" y neighoing los. The ROI is otained y some fom of estimation o a pioi knowledge. In continuous opeation the ROI e povided y tacking. Let us epesent the ROI as a ectangle : (t,l,, t - "top" - fist ow of the ROI. l - "left" - fist column of the ROI. - "ottom" - last ow of the ROI - "ight" -last column of the ROI. (t,l,, can e seen as a ounding ox, expessed y opposite cones (l,t, (, We will compute the moments within this ROI (ounding ox. Moment Calculations fo Blos Given a taget poaility image T(i,j and a ROI (t,l,,: Sum: S = " " T (i, j i=l j=t -15

16 Detecting Skin with Colo Lesson We can estimate the "confidence" as the aveage detection poaility: Confidence: CF = Fist moments: x = µ i = 1 " " T (i, j#i S i=l j=t y = µ j = 1 " " T (i, j# j S i=l j=t S ( " t( " l Position is the cente of gavity: (µ i, µ j We will use this as the position of the lo. Second Moments: " i = 1 # # T (i, j (i % µ S i i=l j=t " j = 1 # # T (i, j ( j % µ S j i=l j=t " ij = 1 # # T (i, j (i % µ S i ( j % µ j i=l j=t These compose a covaiance matix: # C = " i % " ij " ij " j ( ' The pinciple components (λ 1, λ detemine the length and width. The pinciple diection detemines the oientation of the length. We can discove these y pinciple components analysis. RCR T = " = # 1 0 ' % 0 # ( whee R = cos(" #sin(" ' % sin(" cos(" ( -16

17 Detecting Skin with Colo Lesson The length to width atio, λ 1 /λ, is an invaiant fo shape. The angle θ is a Covaiant fo oientation. We can use the eigenvalues, o chaacteistic values, λ 1, λ, to define the width and height of the lo: Fo example: w=λ 1, h=λ # x % ( This suggests a "featue vecto" fo the lo: X! % y ( = % w( % ( h % ( " ' whee x= µ i, y = µ j, w=λ 1, h=λ and CF = S (! t(! l The confidence (CF can e seen as the Likelihood that the model fo the lo is coect. Tacking allows us to continually update an estimate fo the featues of the Blo, even if the lo is tempoaily lost to occlusion o noise. The tacked oject is often efeed to as a "taget". The vecto model fo the taget lo: povides the # x % ( Blo model: X! % y ( = % w( along with CF. (Confidence % ( h % ( " ' The pecision of the lo can e epesented y a covaiance matix P = % " xx " yx " wx " hx " #x " xy " yy " wy " hy " #y " xw " yw " ww " hw " #w " xh " yh " wh " hh " #h " x# " y# " w# " h# " ## ' ( -17

18 Detecting Skin with Colo Lesson Bayesian Tacking (exta mateial (exta mateial - not essential fo the couse A Bayesian tacke is a ecusive estimato, composed of thee phases: Pedict, Detect, Update. Detection can e povided y detecting the lo using colo statistics within a taget Region of Inteest given y a ounding ox centeed on a pevious position. The size of this ox is detemined y the estimated size of the lo enlaged y the uncetainty P. The Gaussian window is the pevious covaiance fo the lo, enlaged y some "uncetainty" covaiance. The uncetainty captues the possile loss of infomation duing the time fom the most ecent osevation. Ou Gaussian lo is Position:! " µ t = µ % i ' # µ j # Size : C t = " i % " ij " ij " j ( along with CF t. ' whee the second moment of the detected pixels, C, detemine the width, height and oientation: was used to compute to so RCR T = " = # 1 0 ' = h 0 ' % 0 # ( % 0 w ( % cos(# sin(# (% C = R T "R = ' * h 0 (% cos(# sin(# ( ' * sin(# cos(# 0 w ' * sin(# cos(# R = cos(" % sin(" #sin(" ' cos(" ( Let us define the estimated lo at time t as: ˆ µ t, ˆ C t -18

19 Detecting Skin with Colo Lesson Let us define the pedicted featue vecto at time t as:! * µ t*, C t We will compute the estimated lo fom y multiplying the detected pixels y a Gaussian mask detemined fom the pedicted lo. The Covaiance is multiplied y to offset the fact that we will use mask to estimate a new covaiance. Gaussian Mask: G(! µ t *,C t * Detected taget pixels: t(i, j " L( c! (i, j# e 1 % i ' ( * µ T % % i (( %% * 1 i ' ' ** C j µ t ' ( % * µ i (( ' ' ** i j µ i whee L( is ou lookup tale fo the atio of histogams. This is a fom of oust estimation that uses the Gaussian mask to eject outlying detections. We then estimate the new position and covaiance as efoe: Fist moments: µ i = 1 " " t(i, j#i µ S j = 1 " " t(i, j# j i=l j=t S i=l j=t Second Moments: " i = 1 # # t(i, j(i % µ S i i=l j=t " j = 1 # # t(i, j( j % µ S j i=l j=t " ij = 1 # # t(i, j(i % µ S i ( j % µ j i=l j=t Position:! " ˆ µ t = µ % i ' # µ j Size : C ˆ # t = " i % " ij " ij " j ( ' Fo this we must pedict the new position, detect (oseve the lo, and the update the estimate. The following is a zeo-th ode Kalman filte. This is the simplest (almost tivial case of a Bayesian tacke. A fist ode Kalman filtes estimates paametes and thei deivatives. The math is moe complex ut the pinciples ae the same. -19