FACE DETECTION AND TRACKING USING A BOOSTED ADAPTIVE PARTICLE FILTER WENLONG ZHENG. (Under the Direction of Suchendra M. Bhandarkar) ABSTRACT

Transcription

1 FACE DETECTION AND TRACKING USING A BOOSTED ADAPTIVE PARTICLE FILTER by WENLONG ZHENG (Under he Drecon of Suchendra M. Bhandarkar) ABSTRACT Ths hess rooses a novel algorhm for negraed face deecon and face rackng based on he synhess of an adave arcle flerng algorhm and an AdaBoos face deecon algorhm. A novel Adave Parcle Fler (APF), based on a new samlng echnque, s roosed o oban accurae esmaon of he roosal dsrbuon and he oseror dsrbuon for accurae rackng n vdeo sequences. The roosed scheme, ermed a Boosed Adave Parcle Fler (BAPF), combnes he APF wh he AdaBoos algorhm. The AdaBoos algorhm s used o deec faces n nu mage frames, whle he APF algorhm s desgned o rack faces n vdeo sequences. The roosed BAPF algorhm s emloyed for face deecon, face verfcaon, and face rackng n vdeo sequences. Eermenal resuls confrm ha he roosed BAPF algorhm rovdes a means for robus face deecon and accurae face rackng under varous rackng scenaros. INDEX WORDS: Face deecon, face rackng, arcle fler, boosed learnng

2 FACE DETECTION AND TRACKING USING A BOOSTED ADAPTIVE PARTICLE FILTER by WENLONG ZHENG B.S., Wuhan Techncal Unversy of Surveyng and Mang, Chna, 993 M.S., Wuhan Techncal Unversy of Surveyng and Mang, Chna, 996 Ph.D., Shangha Insue of Techncal Physcs, Chnese Academy of Scences, Chna, 999 A Thess Submed o he Graduae Faculy of The Unversy of Georga n Paral Fulfllmen of he Requremens for he Degree MASTER OF SCIENCE ATHENS, GEORGIA 2005

3 2005 Wenlong Zheng All Rghs Reserved

4 FACE DETECTION AND TRACKING USING A BOOSTED ADAPTIVE PARTICLE FILTER by WENLONG ZHENG Major Professor: Commee: Suchendra M. Bhandarkar Eleen T. Kraemer Kang L Elecronc Verson Aroved: Maureen Grasso Dean of he Graduae School The Unversy of Georga December 2005

5 DEDICATION Ths hess s dedcaed o my wfe, Yanfen Le, and son, Jale Zheng. v

6 ACKNOWLEDGEMENTS I am deely ndebed o my major advsor, Dr. Suchendra M. Bhandarkar. I s very hard o adequaely eress my graude for hs ecellen gudance, sage advce, and boundless encouragemen durng my research n comuer vson and wrng of hs hess. Dr. Bhandarkar was always resonsve and helful o my requess. Dr. Bhandarkar advsed me no only on research-relaed ssues, bu also on rofessonal develomen. Hs dedcaon o research, eachng and servce wll benef me lfelong. I would also lke o eress my graude o my maser commee: o Dr. Eleen T. Kraemer, for her nvaluable advce on hs hess and endless encouragemen; o Dr. Kang L, for hs suggesons on hs hess and grea suor. I am also graeful o Dr. Rober Teskey and Dr. Jessca Kssnger for her generous suor of my sudy. I would also lke o hank my fellow graduae sudens and frends, esecally Yngle Song, Xngzh Luo, Yanq Su, Xn Gao, Aura Morrs, Ananda Chowdhury, Xaochuan Y, Sddharha Chaoadhyay, for her hel on my hess research and frendsh durng my sudy. I am also graeful o my famly: o my wfe, Yanfen Le, for her love, sacrfce and endless suor; o my son, Jale Zheng, your love s so moran o me; o my arens, for encouragng my sudy; and for my arens-n-law for your suor wh famly ssues. v

7 TABLE OF CONTENTS Page ACKNOWLEDGEMENTS...v LIST OF TABLES... LIST OF FIGURES... CHAPTER INTRODUCTION.... Background Research objecves Thess srucure LITERATURE REVIEW Inroducon Face deecon Vsual rackng Summary FACE DETECTION AND TRACKING USING A BOOSTED ADAPTIVE PARICLE FILTER...39 Absrac Inroducon Sascal model Face rackng usng arcle flerng...5 v

8 3.4 The boosed adave arcle fler Eermenal resuls Conclusons...88 References Conclusons...95 REFERENCES...98 v

9 LIST OF TABLES Page Table 2.: Face deecon mehods and her reresenave works...37 Table 2.2: Vsual rackng mehods and her reresenave works...38 Table 3. Summary of rackng resuls of he APF and he Condensaon...80 Table 3.2 Summary of rackng resuls of he BAPF and he APF...8 Table 3.3 Summary of rackng resuls of he BAPF wh dfferen values of he arameer L...83 Table 3.4 Summary of rackng resuls of he BAPF wh dfferen values of he arameer F...84 Table 3.5 Summary of rackng resuls of he BAPF wh dfferen values of he arameer γ...86 v

10 LIST OF FIGURES Page Fgure 2.: Haar-lke feaures used n he AdaBoos algorhm...8 Fgure 2.2: The AdaBoos learnng algorhm...8 Fgure 2.3: The cascade srucure of Vola and Jones s sysem...9 Fgure 2.4: The srucure of he mul-vew face deecon sysem of FloaBoos...20 Fgure 2.5: Vew-based deecor dagram...2 Fgure 2.6: Face deecon usng neural neworks...22 Fgure 2.7: A B-slne conour secfed by conrol ons...28 Fgure 3.: (a) Observaon rocess: he ellse s a hyoheszed conour n an mage. (b) The mage feaures on he measuremen lne...49 Fgure 3.2: The algorhm of sandard arcle fler...55 Fgure 3.3: The algorhm of adave arcle fler...59 Fgure 3.4: Inegrang APF wh AdaBoos whn a sngle feedback conrol sysem...69 Fgure 3.5: Face eamles...7 Fgure 3.6: Nonface eamles...72 Fgure 3.7: Resuls of fronal face deecon and mulvew face deecon...72 Fgure 3.8: Trackng resuls wh scale changes n es vdeo...74 Fgure 3.9: Trackng resuls wh varous llumnaons n es vdeo...74 Fgure 3.0: Trackng resuls wh mulvews and roaons n es vdeo...75 Fgure 3.: Trackng resuls wh ou-of-lane roaons n es vdeo...75

11 Fgure 3.2: Trackng resuls wh Occlusons n es vdeo...76 Fgure 3.3: Trackng resuls of wo faces n es vdeo Fgure 3.4: Trackng resuls wh he BAPF a s dfferen mes n es vdeo Fgure 3.5: Trackng resuls wh he Condensaon algorhm a same mes as n Fgure Fgure 3.6 Trackng resuls of he APF algorhm and he Condensaon algorhm...79 Fgure 3.7 Trackng resuls of he BAPF algorhm and he APF algorhm...8 Fgure 3.8 Trackng resuls of he APF algorhm wh dfferen values of he arameer L...82 Fgure 3.9 Trackng resuls of he BAPF algorhm wh dfferen values of he arameer F...84 Fgure 3.20 Trackng resuls of he BAPF algorhm wh dfferen values of he arameer γ...86 Fgure 3.2: (a) Trackng falure n case of a long-me occluson (b) Trackng falure n case of hree eole overlang...87

12 CHAPTER INTRODUCTION

13 2. Background The fas evoluon of comuer echnologes ncludng hardware and sofware has advanced he sae of comung machnery n he as wo decades o he on where human lfe has been sgnfcanly mroved by machne nellgence. Ths rend has resuled n an acve develomen n nformaon echnology and arfcal nellgence, where more frendly and effcen aroaches for human comuer neracon are develoed based on new devces. Comuer vson, whch s one asec of machne nellgence, focuses on dulcaon/emulaon of human vson. Tradonally, comuer vson sysems have been ulzed n secfc alcaons such as assembly lne nsecons and qualy conrol n auomaed manufacurng. The ever decreasng cos of comung sysems and vdeo mage acquson equmen has resuled n comuer vson sysems advancng owards more generalzed vson alcaons such as face deecon and face rackng echnques. Face deecon, whch s he frs se n any face rocessng sysem, aems o deermne wheher here are any faces n a sngle mage. If any faces es, he rocessng sysem rovdes he mage locaon and een of each face (Yang e al., 2002). Face deecon s moran n any human face relaed sysem, such as any fully auomac face recognon sysem, warnng and survellance sysem, or face rackng and human rackng sysem. Face deecon algorhms can be ycally eended o generc objec deecon and recognon (Zhao e al., 2003), whch leads o auomac arge recognon (ATR). So far, face deecon n comuer vson s sll a challengng ask, even hough s easy for humans o erform efforlessly (Hjelmås and Low, 200). The varous face deecon relaed roblems nclude face localzaon, facal eresson recognon, face recognon, face auhencaon, and face rackng. Tradonally, he soluons o he roblems are based on mage segmenaon, facal feaure eracon, and face verfcaon

14 3 n he resence of comlcaed background. The challenges assocaed wh face deecon are conrbued by changes n scale, locaon, orenaon, ose, facal eresson, occluson, and llumnaon. Face rackng ams o kee accoun of face n a vdeo sequence.e., deermne f here are any faces n a sngle frame, and connuously esmae he locaons and ossbly he orenaons of he faces n he vdeo sequence n real me (Darrell e al., 2000; Crowley and Berard, 997; Edwards e al., 998). Face rackng belongs o he larger area of vsual objec rackng ursued by he comuer vson communy, where he objec of neres s he face. Objec rackng has been suded eensvely by researchers n he cone of comuer vson because of many vson alcaons such as auonomous robos (Davson and Murray, 998), vdeo survellance (Borg e al., 2005), human eye rackng (Hansen and Hammoud, 2005) and human face rackng (Nummaro e al., 2003). Generally seakng, an mage sequence, whch s colleced n real me, does no change radly from one frame o he ne frame. Ths resuls n a large redundancy of objec nformaon over consecuve frames sannng a ceran me nerval. Ths redundancy can be ulzed o dsambguae he aearances of he vsual objecs and rack he ndvdual objecs. Snce he human vsual sysem may no dsngush a camouflaged objec from a comlcaed background, he eloaon of he redundancy n a sequence of mages s sll regarded as a challengng roblem n he comuer vson communy (Isard, 998)..2 Research objecves The rmary objecve of hs hess s o ncororae face deecon wh face rackng n vdeo sequences. Ths hess ams o resen a new scheme for robus face deecon and accurae face rackng, where face deecon and face rackng can boos each oher n real me. Ths research

15 4 wll ake a se n movng he convenonal face rackng mechansm owards he boosed hybrd face rackng mechansm. In order o address he general roblems of face deecon and face rackng, such as low deecon rae, varaons n lghng condons, and aral occlusons or comlee occlusons, we roose a novel scheme for face deecon and rackng n hs hess by combnng an AdaBoos algorhm wh a new arcle flerng scheme, ermed an adave arcle fler (APF). The new APF uses a new samlng echnque o oban much more accurae esmaon of he roosal dsrbuon and he oseror dsrbuon, whch mroves he rackng accuracy n he vdeo sequences. We defne he combnaon of he AdaBoos algorhm and he APF as a boosed adave arcle fler (BAPF). The AdaBoos algorhm s used o deec faces n he nu mages, whle he APF s used o rack he faces n he vdeo sequences. The hybrd sysem of BAPF s emloyed for face deecon, face verfcaon, and face rackng n he vdeo sequences. Face deecon and face rackng wll enhance her erformance by muual correlaon n he rocedure. Ths BAPF can rovde robus face deecon and accurae face rackng under some suaons ha he objecs are severely corrued by he occlusons..3 Thess srucure Ths hess s organzed no four chaers n manuscr syle. Chaer nroduces he background and he research objecves. Chaer 2 rovdes a comrehensve revew of he leraure relaed o face deecon and vsual objec rackng. Chaer 3 rooses a boosed adave arcle fler (BAPF) for face deecon and face rackng by combnng an AdaBoos algorhm wh a new adave arcle fler (APF). Chaer 4 resens dscussons and conclusons.

16 5 The enre hess s srucured as follows. Chaer : Inroducon Chaer 2: Leraure Revew Chaer 3: Face Deecon and Trackng Usng a Boosed Adave Parcle Fler Chaer 4: Conclusons

17 6 CHAPTER 2 LITERATURE REVIEW

18 7 Ths chaer rovdes a relavely comrehensve revew of he leraure relaed o face deecon and rackng. I frs brefly nroduces he evoluon of and he body of leraure on face deecon and rackng. Ne, hs chaer revews he leraure on face deecon. Ths s followed by a leraure on vsual objec rackng, whch s generalzaon of face rackng. Fnally, ends wh a summary of he varous aroaches. 2. Inroducon The fas evoluon of comuer echnologes ncludng hardware and sofware has advanced he sae of comung machnery n he as wo decades, o he on where human lfe has been sgnfcanly mroved by machne nellgence. Ths rend has resuled n an acve develomen n nformaon echnology and arfcal nellgence, where more frendly and effcen aroaches for human comuer neracon are develoed based on new devces. Comuer vson, whch s one asec of machne nellgence, focuses on dulcaon/emulaon of human vson. Tradonally, comuer vson sysems have been ulzed n secfc alcaons such as assembly lne nsecons and qualy conrol n auomaed manufacurng. The ever decreasng cos of comung sysems and vdeo mage acquson equmen has resuled n comuer vson sysems advancng owards more generalzed vson alcaons such as face deecon and face rackng echnques. For eamle, comuer vson sysems, whch are deloyed n desko or embedded sysems (Penland, 2000a; Penland, 2000b; Penland and Choudhury, 2000), can deec and rack he face of he user n real me. Face deecon, whch s he frs se n any face rocessng sysem, aems o deermne wheher here are any faces n a sngle mage. If any faces es, he rocessng sysem rovdes he mage locaon and een of each face (Yang e al., 2002). Face deecon s moran n any

19 8 human face relaed sysem, such as any fully auomac face recognon sysem, warnng and survellance sysem, and face rackng and human rackng sysem. The face deecon algorhms can be ycally eended o generc objec deecon and recognon (Zhao e al., 2003), whch leads o auomac arge recognon (ATR). Face deecon n comuer vson s sll a challengng ask, even hough s easy for humans o erform efforlessly (Hjelmås and Low, 200). The varous face deecon relaed roblems are face localzaon, facal eresson recognon, face recognon, face auhencaon, and face rackng. Tradonally, he soluon o he roblems s based on mage segmenaon, facal feaure eracon, and face verfcaon n he resence of comlcaed background. The challenges assocaed wh face deecon are conrbued by changes n scale, locaon, orenaon, ose, facal eresson, occluson, and llumnaon. The varous facors affecng he mages of a human face are descrbed as follows:. Pose. A change n ose relave o he camera vewon affecs he aearance of he face n he mage. 2. Facal eresson. The facal eresson deermnes he aearance of he face n he mage. 3. Occluson. In some cases an objec may occlude he face arally or comleely, hus affecng he aearance of he face n he mage. 4. Orenaon. A relave roaon abou he camera s ocal as changes he aearance of he face n he mage. 5. Lghng condons. Dfferen llumnaon condons, such as he lgh source dsrbuon and he ocal and elecronc characerscs of a camera, roduce dfferen face mages.

20 9 Face rackng ams o kee accoun of face n a vdeo sequence.e., deermne f here are any faces n a sngle frame, and connuously esmae he locaons and ossbly he orenaons of he faces n he vdeo sequence n real me (Darrell e al., 2000; Crowley and Berard, 997; Edwards e al., 998). Face rackng les whn he larger area of vsual objec rackng ursued by he comuer vson communy, where he objec of neres s he face. Objec rackng has been suded eensvely by researchers n he cone of comuer vson because of many vson alcaons such as auonomous robos (Davson and Murray, 998), vdeo survellance (Borg e al., 2005), human eye rackng (Hansen and Hammoud, 2005) and human face rackng (Nummaro e al., 2003). Generally seakng, an mage sequence, whch s colleced n real me, does no change radly from one frame o he ne frame. Ths resuls n a large redundancy of objec nformaon over consecuve frames sannng a ceran me nerval. Ths redundancy can be ulzed o dsambguae he aearances of he vsual objecs and rack he ndvdual objecs. Snce he human vsual sysem may no dsngush a camouflaged objec from a comlcaed background, he eloaon of he redundancy n a sequence of mages s sll vewed as a challengng roblem n he comuer vson communy (Isard, 998). Whle he human vsual sysem models accurae objec rackng as an nformaon-rocessng roblem assocaed wh robus and real-me comuaon, we are unaware of any curren soluons usng arfcal nellgence whch fully undersand he human soluon. The curren soluons have o make some assumons o smlfy he rackng roblem and acce less han erfec resuls o make rogress n secfc suaons. Therefore, we have o segmen he mages of he real world no he meanngful blocks on he bass of redeermned segmenaon crera. Many aroaches o hs segmenaon roblem are roosed such as layers (Baker e al., 998), n whch he world conans cardboard cuous, eures (Malk e al., 999), n whch

21 0 he world s comosed of objecs defned by homogeneous eures, conours (Blake and Isard, 998; L e al., 2003; Rah e al., 2005), n whch he world consss of objecs defned by shaes wh known geomerc roeres, and emlaes (Boccgnone e al., 2005; Luo and Bhandarkar, 2005), n whch he world consss of objecs comrsng of redefned regons wh known roeres. Wh an am o resen a comrehensve and crcal revew of face deecon and rackng mehods, hs leraure survey s organzed as follows: In Secon 2.2, we rovde a dealed revew of varous aroaches o deec faces n a sngle mage. Secon 2.3 resens a dealed survey and dscusson of echnques for vsual rackng n an mage sequence. Fnally, he summary and dscusson are resened n Secon Face Deecon In hs secon, we carefully survey esng echnques for face deecon n a sngle mage. We classfy hese echnques no hree caegores based on how hey elo he knowledge of he face: feaure-based mehods, emlae-based mehods, and mage-based mehods. Snce magebased mehods have demonsraed beer resuls recenly comared o he oher caegores, we resen a more dealed revew of he mage-based mehods n hs secon. Some face deecon mehods may clearly overla caegory boundares, and hence can be classfed no more han one caegory. For eamle, emlae-based mehods ycally use a face emlae o erac facal feaures, and hen ulze hese feaures for face deecon (Hor e al., 2004; Govndaraju, 996; Lades e al., 993); mage-based mehods also use some secfc feaures o deec a face, such as Haar-lke feaures (Vola and Jones, 200a; 200b), and Gabor feaures (Yang e al., 2004;

22 Zhang e al., 2004). The hree caegores of face deecon mehods are descrbed n he followng:. Feaure-based mehods. These mehods make elc use of he knowledge of he human face and erac srucural feaures ha reman unchanged whle he ose, facal eresson, or llumnaon vary. These feaures can be generaed from he resuls of lowlevel analyss, such as edges, gray-levels, or color. The facal feaures can also be obaned from a more global descron of he face usng nformaon derved from face geomery. 2. Temlae-based aroaches. A se of re-defned sandard face aerns or emlaes s consruced and sored. The emlaes reresen a face as a whole or he facal feaures searaely. These mehods use he correlaons beween an nu mage and he gven aerns o deec faces. 3. Image-based mehods. These aroaches use learnng algorhms o deec faces. The learnng algorhms can caure he nheren varably n facal aearance whn a se of ranng mages. Unlke he mehods n caegory and 2, he mage-based mehods acqure he knowledge of he human face mlcly hrough mang and ranng schemes Feaure-based mehods Tycally, feaure-based face deecon mehods are roosed o frs deec facal feaures, whch are nvaran over dfferen oses and lghng condons. These facal feaures, whch nclude he eyes, nose, mouh, eyebrows and so on, are hen used o deermne he esence of a face. Many mehods are roosed o erac feaures for face deecon. These mehods can be

23 2 generally dvded no broad caegores: low-level feaure analyss and hgh-level feaure analyss. The low level feaure analyss s based on he segmenaon of facal feaures usng el roeres, such as gray scale, eure, and skn color. However, he feaures obaned from he low level feaure analyss are usually ambguous and sensve o changng llumnaon. The hgh level feaure analyss emloys a global model of he face and facal feaures ha ncororaes knowledge of face geomery. Hgh level feaure analyss can erac beer facal feaures han he low level feaure analyss. One common roblem of feaure-based mehods s ha he feaures can be severely affeced due o he varaons n lghng condons Low level feaure analyss Herers e al. (996) roose a mehod for facal feaure deecon and characersc key-on deecon usng edges and lnes. I frs uses a frs and second dervave of a Gaussan based edge deecor o deec edges and lnes n he underlyng facal regon. The lne and edge deecon can be erformed effcenly a any orenaon and scale. Then uses hree basc oeraons o deec he key-ons of he face. The frs oeraon searches he edges or lnes n a redefned regon wh a redefned orenaon and scale. The orenaon of an edge or lne n a gven locaon s deermned by he second oeraon hrough he evaluaon of he mamal resonse of a roaed fler. The hrd oeraon racks an edge or lne by a small se n he known drecon. Song e al. (2002) roose a mehod o deec objecs n an edge color sace (ECDS) nsead of he mage sace. Ther mehod assumes ha he unform-color objecs and eured objecs have dfferen dsrbuon characerscs n an ECDS. Ther mehod frs measures he color of each edge on n he edge deecon hase, and hen ransforms he edge ons no he 3D ECDS by quanzng he mage sace and he color sace. Fnally he edge

24 3 ons assocaed wh dfferen objecs are segregaed saally n he 3D ECDS raher han beng deeced as overlang n he 2D mage sace. However, hs mehod erforms oorly n suaons wh sgnfcan llumnaon change. Yang and Huang (994) roose a face deecon mehod n gray scale yramd mages. Assumng ha he face mage becomes aromaely unform a lower resoluons, hs mehod searches he unform regons sarng a a lower resoluon o oban face canddaes usng a se of rules. Then hese face canddaes are furher confrmed based on he romnen facal feaures corresondng o local mnma a hgher resoluon. Graf e al. (995) elore he gray scale behavor of faces o locae facal feaures. Ther aroach frs ales morhologcal oeraons o enhance regons ha have ceran shaes. Based on he eak value of he gray scale hsogram of he rocessed mage, hen ales he adave hresholdng algorhm o generae wo bnary mages. Fnally, her aroach evaluaes he combnaons of conneced comonens n he bnary mages o deermne he esence of he face. Huang and Trved (2004) develo a framework for face deecon and rackng usng skn color and ellcal edge conours. I deecs skn blobs f he color of he mage regon s above a redefned hreshold and obans he face canddaes. In he meanme, also deecs he face canddaes by comarng he eraced edge conours wh a redefned ellse. The fnal face canddaes are generaed usng a combnaon of color and edge feaures. Fnally he face canddaes are verfed usng a dsance merc n a reduced dmensonal feaure subsace comued va rncal comonen analyss (PCA) o remove non-faces. However, mos skn color models are ycally no very robus o sgnfcan varaons of he lghng condons. To address hs roblem, McKenna e al. (998) roose an adave color mure model o rack faces n varyng lghng condons. Ther aroach uses a sochasc model o aromae he

25 4 color dsrbuon of an objec and adas he model o he changes n lghng condons. Naseem and Derche (2005) resen a color-based face deecon mehod ha avods he effec of lumnance changes. Usng he chromac or ure color sace, a Gaussan dsrbuon model for he skn colors s develoed o oban a skn color lkelhood mage. Ths lkelhood mage s hen convered no a bnary mage usng an adave hresholdng algorhm. Fnally, a emlae machng mehod s used o esmae he regons wh he desred facal roeres Hgh level analyss Huang e al. (2004) roose a face deecon mehod ha combnes mulle facal feaures. Four classfers are desgned based on four feaure-based reresenaons: nensy, graden, Gabor (Huang e al., 2003), and 2D Haar wavele (Tokunaga e al., 2002). The nensy feaures are obaned afer he rerocessng hase conssng of lnear llumnaon correcon and hsogram equalzaon. The graden drecon feaures are eraced from he local mages usng he Sobel oeraor. Then he graden vecor s decomosed no s comonens along he egh chan-code drecons. A 2D Gabor fler s used n he mage sace and he saal frequency doman o erac he feaures. Two yes of 2D Haar bass funcons are used o characerze he changes n nensy along he horzonal and vercal drecons resulng n he 2D Haar wavele feaures. A olynomal neural nework (PNN) s emloyed for each reresenave feaure model o assgn a face lkelhood score o each face canddae. The ouu scores from he four PNNs are averaged o ge a fnal score for each face canddae. Wang and ermaran (2000) roose a fler-based mehod for face deecon and facal feaure localzaon. Ther aroach frs uses mul-scale flers o oban he re-aenve feaures of he objecs n he mage. Three reresenave models are emloyed n hs mehod: a

26 5 srucure model, a eure model, and a feaure model. Usng he geomerc aerns of he underlyng facal comonens, he srucure model s used o grou he els no face canddaes. The eure and feaure models are used o evaluae he face canddaes. The eure model valdaes he gray scale or color smlares of face canddaes usng face models. The feaure model comares he regon feaures o secfc facal feaures usng he egen-eyes mehod Temlae-based aroaches Temlae-based face deecon mehods use a sandard face aern, whch s redefned or arameerzed by a funcon. The smlares beween he sandard aerns and he local mage regons are esmaed for he face canddae and s varous comonens. The decson regardng he face canddaes are made based uon he values of hese smlares. Based on he revous work of Lades e al. (993), Wsko e al. (997) roose an elasc bunch grah machng (EBGM) mehod for face recognon. In hs mehod, faces are reresened by labeled grahs usng a Gabor wavele ransform. A se of M ndvdual model grahs s combned no a sacklke srucure, called a face bunch grah (FBG). Once he nal FBG s generaed manually, he FBG of new mages can be generaed auomacally by he EBGM rocedure. A grah smlary measure beween an mage grah and he FBG corresondng o an dencal ose s comued o mach model FBG o a new mage. Afer obanng model grahs from an mage daabase and mage grahs from he robe mages, recognon s erformed by selecng model FRG corresondng o he hghes smlary value resulng from he comarson of an mage grah o all he model grahs.

27 6 Kwon and Lobo (994) resen a face deecon mehod based on snakes and emlaes. A modfed n-el snake s used o fnd and remove small curve segmens n he mage. An ellse s used o aromae each face. A Hough ransform on he remanng snakeles s emloyed o search for a redomnan ellse. Each face canddae s evaluaed by a mehod smlar o he deformable emlae machng mehod. The fnal decson for each face canddae s rovded based on he number of machng facal feaures found n he mage and her roorons. Gunn and Non (996) resen a mehod for face boundary deecon usng a dual snake confguraon based on dynamc rogrammng o locae a global energy mnmum. Ths mehod uses dynamc rogrammng o erac he nner face boundary, whereas uses a convenonal surface normaldrven echnque o erac he ouer face boundary. Samal and Iyengar (995) roose a mehod for face localzaon usng slhouees as emlaes. Prncal comonen analyss (PCA) of he face eamles s used o oban a se of bass face slhouees. These egen-slhouees n combnaon wh a Hough ransform are hen ulzed o localze he faces Image-based mehods Image-based face deecon mehods have demonsraed ecellen resuls recenly among all face deecon mehods. Image-based mehods ycally deend on echnques from machne learnng and sascal analyss o search for he dscrmnang characerscs of face and non-face mages. In general, hese characerscs are modeled usng known sascal dsrbuons or a combnaon of known dscrmnan funcons, whch are hen used for face deecon. Much research has been conduced n mage-based mehods resulng n well known echnques, such as AdaBoos (Vola and Jones, 200a; Vola and Jones, 200b; Wang e al., 2004), FloaBoos (L e al., 2002), S-AdaBoos (Jang and Loe, 2003), neural neworks (Rowley e al., 996;

28 7 Curran e al.,2005), Suor Vecor Machnes (SVM) (Osuna e al., 997; Shh and Lu, 2004), Hdden Markov Models (Rabner and Jung, 993), and he Bayes classfer (Schnederman and Kanade, 998; Schnederman, 2004) Boosng Learnng Algorhms Based on revous work of Teu e al. (2000) and Schnederman (2000), Vola and Jones (200a; 200b) roose a robus face deecon algorhm, whch can deec faces n a rad and robus manner wh a hgh deecon rae. I resens hree conrbuons for face deecon: he negral mage, a srong classfer comrsng of weak classfers based on he AdaBoos learnng algorhm, and an archecure comrsng of a cascade of a number of srong classfers. The sysem of Vola and Jones (200a; 200b) emloys an negral mage comrsng of Haar-lke feaures for effecve feaure eracon from a large feaure se. Lenhar and Mayd (2002) rovde a se of Haar-lke feaures for AdaBoos, as shown n Fgure 2.. In he boosng rocedure as shown n Fgure 2.2, AdaBoos frs learns effecve feaures from a large feaure se. Second, consrucs a se of weak classfers, each of whch s comosed of a feaure, a hreshold and a ary. Thrd, generaes a srong classfer based on he above weak classfers, as shown n Fgure 2.2. Each eraon wll generae a weak classfer. Afer all eraons, wll resul n T weak classfers. These T weak classfers are combned no a srong classfer usng a weghed lnear combnaon. The sysem of Vola and Jones (200a; 200b) uses a cascade of srong classfers o mrove he deecon rae wh effcen comuaon, as shown n Fgure 2.3. The dea s o consruc smaller and effcen classfers based on he sub-wndows whn he mage. The smler and faser classfers wll rejec he negave sub-wndows. A large number of negaves are rejeced by he nal classfer wh mnmal rocessng. Addonal negaves are

29 8 elmnaed by subsequen layers whle requrng addonal comuaon. The number of subwndows s suosed o be reduced radly afer several sages of rocessng. Fgure 2. Haar-lke feaures used n he AdaBoos algorhm (Lenhar and Mayd, 2002) Fgure 2.2 The AdaBoos learnng algorhm (Vola and Jones, 200a)

30 9 Fgure 2.3 The cascade srucure of Vola and Jones s sysem (200a) L e al. (2002a) roose he FloaBoos algorhm, an mroved verson of AdaBoos, for learnng a boosed classfer for obanng he mnmum error rae. I uses a backrackng mechansm o mrove he deecon rae afer each eraon of AdaBoos rocedure. In he boosng rocedure, FloaBoos erforms deleons of weak classfers ha are neffecve based on he error rae. Thus a srong classfer conanng a se of weak classfers s used o mrove he classfcaon error. Bu hs mehod needs more ranng me han AdaBoos snce enals an addonal search on he curren weak classfers. L e al. (2002a) also roosed a mul-vew face deecon sysem, whch s llusraed n Fgure 2.4. Ths srucure uses he coarse-o-fne sraegy and generalzes he cascade deecon sysem roosed by Vola and Jones (200a). I consss of hree levels. Each level ece he o level conans more han one deecor. The fnal resul s obaned by mergng he combnaon of he deecors a he boom level.

31 20 Fgure 2.4 The srucure of he mul-vew face deecon sysem of FloaBoos (L e al., 2002a) Jang and Loe (2003) roose S-AdaBoos, a varan of AdaBoos for handlng oulers n aern deecon and classfcaon. S-AdaBoos dvdes he nu sace no sub-saces based on he Dvde and Conquer Prncle. Dedcaed classfers are used o rocess he sub-saces. Fnally, a secfc classfer handles he combnaon of he ouus of he dedcaed classfers. Snce hs mehod uses dfferen classfers n dfferen hases, s comuaon and effecveness are no sasfacory. Zhang e al. (2004b) roose a face deecon mehod based on boosng n herarchcal feaure saces. They assume ha global feaures derved from Prncal Comonen Analyss can be used n he laer sages of boosng o furher mrove he deecon rae. However, needs more comuaon me for eracng global feaures. Wang e al. (2004) roose a real-me facal eresson recognon sysem wh AdaBoos. In he face deecon hase, hs sysem uses he AdaBoos algorhm roosed by Vola and Jones (200a; 200b). In he facal eresson recognon hase, he eressons are learned from he boosng of Haar-lke feaure-based look-u-able ye weak classfers. Lkewse, Wu e al. (2004) roose a roaon nvaran mul-vew face deecon mehod based on real AdaBoos. The faces are groued based on he aearance from dfferen vews, and hen weak

32 2 classfers are learned from he ndvdual grous o consruc a confdence-raed look-u-able for Haar-lke feaures. Ths mehod uses a vew-based deecor ha can deal wh facal rofles and 360-degree roaed faces. A nesed-srucured cascade s roosed n hs mehod, as llusraed n Fgure 2.5. I consss of common weak classfers of Vola and Jones s sysem (200a) and mulle layers of nesed weak classfers. Each layer s a lnear nework of common weak classfers and ouus a confdence value for furher rocessng n he followng layer. Yang e al. (2004) rovde a face recognon mehod wh AdaBoosed Gabor feaures. Frs, AdaBoos selecs a small se of effecve Gabor feaures from a large daabase of mages. Then a srong classfer ncororang a few hundred of weak classfers wh Gabor feaures can dsngush he dfference beween wo face mages. Zhang e al. (2004a) also roose a smlar mehod as Yang e al. (2004). Fgure 2.5 Vew-based deecor dagram (Wu e al. 2004) Neural Nework Learnng Algorhms Rowley e al. (996; 998) has done he mos sgnfcan research among all face deecon mehods based on neural neworks, as shown n Fgure 2.6. Ths mehod consss of wo major comonens: a se of mullayer neural neworks and a decson makng module. The mullayer

33 22 neural neworks are used o learn he face and non-face aerns from he ranng ses conssng of face and non-face mages, and hen aled o deec faces. The decson makng module s used o generae he fnal decson on he bass of he combnaon of mulle deecon resuls. The frs comonen receves a el mage regon and hen ouus a score from - o, where - denoes non-face and denoes face. Usng mul-resoluon rocessng, he neural nework can deec a face of sze larger han els. The decson makng module merges he overlang deecon resuls from he ouus of he mulle neworks and makes a fnal deermnaon. One drawback of hs mehod s ha only urgh fronal faces can be deeced. Alhough Rowley furher mroves he mehod o deec roaed face mages, he resul s no romsng because of s lower deecon rae. Fgure 2.6 Face deecon usng neural neworks (Rowley e al.; 998) Curran e al. (2005) eends he work of Rowley e al. (998) o address he roblem of face deecon under gross varaons. Féraud e al. (200) roose a face deecon mehod based on a neural nework model, whch s called he Consraned Generave Model (CGM). Ths aroach comues he dsance of he nu subwndow o he se of faces o esmae he robably of an nu subwndow o be a face. The dsance s obaned based on a rojecon of a el n he

34 23 nu mage sace on he se of faces. The face deecor based on CGM and Mullayer Perceron (MLP) consss of four sages, where he las fler ouus he fnal decson. The major dsadvanage of hs mehod s ha requres non-face samles o model he rojecon, whch enals more comuaon me Suor Vecor Machnes Suor Vecor Machnes (SVMs) use srucural rsk mnmzaon o mnmze he uer bound of he eeced generalzaon error (Osuna e al., 997), whle mos oher learnng mehods such as neural neworks and Bayesan neworks are based on mnmzng he ranng error. The SVM s a lnear classfer whch comues a searang hyerlane o mnmze he eeced generalzaon error. The hyerlane s defned by a weghed combnaon of a small subse of suor vecors. The omal hyerlane s aromaed by solvng a lnear consraned quadrac rogrammng roblem. The major dsadvanages of SVMs are s comuaon me and hgh memory requremen. Terrllon e al. (2000) analyze he erformance of SVMs n sac color mages and roose a face deecon mehod. Ther aroach combnes he skn color-based mage segmenaon wh he alcaon of SVMs o he nvaran feaures derved from a generalzaon of he Orhogonal Fourer and Melln Momens (OFMMs). Shh and Lu (2004) roose a face deecon mehod combnng Dscrmnang Feaure Analyss (DFA) and SVMs. Ths aroach uses boh emoral and skn color nformaon o locae he regons of neres n he nu mage. An SVM classfer and Bayesan analyss are aled o he feaures eraced by DFA for face deecon.

35 Oher Learnng Algorhms Hdden Markov Models (HMMs) assume ha face and non-face aerns can be characerzed as a aramerc random rocess. These arameers can be obaned usng a well-defned esmaon rocedure (Rabner and Jung, 993). The am of ranng an HMM s o esmae he arorae arameers n an HMM model o mamze he robably of observng he ranng daa. Schnederman and Kanade (998) resene a nave Bayes classfer, whch elos he esmaon of he jon robably of local aearance and oson of a face aern a mulle scales. However, he erformance of a nave Bayes classfer s oor. To solve hs roblem, Schnederman (2004) roose a resrced Bayesan nework for objec deecon. Ths mehod searches he srucure of a Bayesan nework-based classfer n he large sace of ossble nework srucures. The fnal srucure s comued va consraned omzaon of wo cos funcons where esmaes and evaluaon are recomued: a localzed error n he log-lkelhood rao funcon for he srucure and a global classfcaon error for he fnal choce of he srucure. Park e al. (2005) roose a Face Probably Graden Ascen (FPGA) mehod o evaluae he omal oson, scale, and roaon varans of each face. Based on he robably ha he aral mage corresonds o a face mage, he roosed FPFA aroach uses a gradenbased erave search o deermne he objecve funcon o model he underlyng robably densy funcon. 2.3 Vsual Trackng Objec rackng has been suded eensvely n he cone of comuer vson because of many vson alcaons such as auonomous robos (Davson and Murray, 998), vdeo survellance (Borg e al., 2005), human eye rackng (Hansen and Hammoud, 2005) and human face rackng

36 25 (Nummaro e al., 2003) ha use rackng algorhms eensvely. Objec rackng n comle suaons needs o deal wh uncerany and error (Inlle e al., 997). Therefore many echnques have been develoed o solve he roblem of objec rackng. We classfy vsual objec rackng mehods no hree (ossbly overlang) broad caegores.. Image-based rackng. Image-based rackng mehods erac he generc feaures and hen grou hem based on hgh-level scene nformaon. 2. Conour-based rackng. Conour-based rackng assumes ha he objec s defned by boundares wh some roeres. I usually requres shae models (conours), dynamcal conour models and oher mage measuremens durng he rackng rocess. 3. Flerng-based rackng. The Kalman fler and he arcle fler are nvesgaed n hs caegory. Kalman flerng deals wh he rackng of shae and locaon over me n lnear dynamc sysems. Parcle flerng, on he oher hand, s no resrced o lnear sysems. The basc dea of he arcle fler s o aromae he oseror densy usng a recursve Bayesan fler usng a se of arcles wh assgned weghs Image-based Trackng Many echnques have been develoed n he las decade for vsual objec rackng. Image-based rackng mehods oban generc feaures from he mages and hen combne hem based on he hgh-level scene nformaon. Inlle e al. (997) roose a blob-racker for human rackng n real me. The background s subraced o erac foreground regons. The foreground regons are hen dvded no blobs based on color. These blobs are clusered usng romy and velocy no grous such ha a sngle grou of blobs belongs o a sngle erson. Ths aroach runs fas, bu he major dsadvanage s ha merges blobs when he objecs n he scene

37 26 aroach each oher. To address hs roblem, Huang and Trved (2004) develo a framework for face deecon and rackng usng skn color and ellcal edges. Ths aroach deecs skn blobs f he color of he area s above a hreshold n a color sace, and deecs he face canddaes by comarng he deeced edges wh a redefned ellse. The face canddaes are verfed usng a dsance measure n a feaure sace deermned va rncal comonen analyss (PCA). A connuous densy Markov hdden model (CDHMM) (Rabner, 989) s used for face rackng, n whch face orenaons are esmaed va mamum a oseror (MAP) comuaon n real me. However, skn color models are no effecve n he resence of sgnfcan varaon n he lghng condons. Bhandarkar and Luo (2005) resen a mul-color model for background udang n survellance and monorng sysems. I uses mulle color clusers o reresen he background a he el level. The background udang scheme udaes he mean and varance of each color cluser wh currenly observed color values. The advanage of hs mehod s ha s robus and comuaonally effcen for real-me monorng sysems. However, hs mehod wll gve wrong resuls when he background color does no reman consan for a erod of me. Gan e al. (2005) roose an objec rackng mehod usng he level-se mehod. Ths aroach elores boh local and global feaures of he mage sequences o oban beer rackng resuls for objecs wh a non-unform energy dsrbuon. Frs, an nal segmenaon of he objecs s erformed usng a sem-auomac aroach. Second, rackng echnques, whch are based on level se mehods or geomerc aral dfferenal equaons (PDEs), are aled o segmen he objecs n oher vdeo sequences. Thrd, a gven mage s deformed accordng o he PDEs, and he desred resul s deemed o be he seady sae soluon of hs PDE. The rocess of solvng he PDE can

38 27 be regarded as ha of mnmzng a redefned energy funcon. However, hs mehod has he lmaon of eensve comuaon. Chen and Tddeman (2005) resen a facal feaure rackng mehod usng skn color flerng. Ths aroach ulzes a 3D facal feaure model o esmae he 3D ose of a human head. Skn color flerng s frs emloyed o deec a face n he normalzed YCbCr color sace and he HSI color sace. The Lucas-Kanade (LK) algorhm (Lucas and Kanade, 98) s hen aled o rack he feaure ons. The LK rackng algorhm deecs he moon based on he ocal flow. However, hs mehod only handles fronal face vews, snce he dsaearance of ceran feaures n a mul-vew face scene makes he rackng fal. Thome and Mgue (2005) roose a human rackng mehod based on he consrucon of a 2D human aearance model, whch rovdes dscrmnave feaures ha caure boh color and shae roeres of he dfferen lmbs. Ths mehod, however, erforms oorly when faced wh sgnfcan changes n he amben lghng condons Conour-based Trackng Conour-based rackng assumes ha he objec s defned by boundares wh known roeres. I reles on shae models (conours), dynamcal models and he mage measuremens. Kass e al. (987) resen a rackng echnque feaured as Snakes o erform robus segmenaon and regon rackng by modelng an objec usng he conour defnng he objec oulne whch s nsensve o lghng changes, and alyng smoohness consrans on he conour curvaure and he objec moon. Ths rackng mechansm s more general han modelng enre objecs, and also more cluer-ressan han rackng echnques based on sgnal-rocessng or smlar low level analyss aled o feaures such as corners or edges. Many researchers (Blake and Isard,

39 28 998; MacCormck, 2000) have adoed and eended hs dea of acve conour modelng n he conen of rackng. Blake and Isard (998) develo a robablsc acve conour framework for vsual rackng where objecs are reresened by B-slne curves n an mage sequence. A gven objec s defned by s conour oulne modeled as a B-slne. Secfcally, suose he coordnaes of he B- slne conrol ons are (, y )(,, y ), L, (, ) ( () s, y() s ) T y 2 2 n n, hen he B-slne s a arameerzed curve s defned on an nerval of he real lne: y s ( ) () = B r r y () s, where () s B s a 2 2n mar whose enres are olynomals n s, and, y r are n column vecors reresenng he - and y-coordnaes of he conrol ons resecvely. Any such B-slne s called a conour. Fgure 2.7 llusraes an eamle of a face-lke conour wh 3 conrol ons. Fgure 2.7 A B-slne conour secfed by conrol ons (Isard, 998) In racce, s desrable o resrc he confguraon of he slne o a shae-vecor r deermned by he confguraon vecor X and descrbed by r = W X + Q, where W s he y 2n d shae mar, confguraon vecor X s a d column vecor, Q s a d vecor called

40 29 he objec emlae. Generally, he shae sace allows affne deformaons of he emlae Q, n a sace of rgd and non-rgd deformaons as shown n Fgure 6. Isard and Blake (998) aly he B-slne reresenaon o conours of objecs and resen he Condensaon algorhm. The Condensaon algorhm uses he affne grou arameers as he sae vecor, learns a dynamcal model for hese arameers, and emloys a arcle fler o esmae hese arameers. However, hs aroach canno deal wh local deformaons of he deformng objec because only racks he affne arameers. Followng he dea of Blake and Isard (998), Wu e al. (2003) roose a generave model aroach for conour rackng n he resence of non-saonary cluer. Ths mehod uses a roosed dynamc Bayesan nework o deal wh occlusons va elc modelng and nference. However, hs mehod s comuaonally very nensve. A more recen develomen n he conour-based rackng s he use of he level se echnque (Sehan, 989), whch s an mlc reresenaon of conours. To segmen a shae usng level ses, hs echnque deforms an nal guess of he conour shae unl reaches he mnmum of an mage-based energy funconal. Some recen reresenave research n rackng usng level se echnques nclude he works of Yezz and Soao (2003), Jackson e al. (2004), and Rah e al. (2005). Yezz and Soao (2003) roose a defnon for moon and shae deformaon for a deformng and movng objec. A fne dmensonal grou acon, such as a Eucldean or Affne grou s used o arameerze he moon of he objec. The shae deformaon s defned by he oal deformaon of he objec conour (nfne-dmensonal grou) modulo he fnedmensonal moon grou. Trackng s hen descrbed by a rajecory defned on he fnedmensonal moon grou. Ths mehod deends only on he observed mages for rackng and does no make any use of he ror nformaon on he dynamcs of he grou acon or of he deformaon. Thus collases when here s an ouler observaon or when here s occluson.

41 30 To solve hs roblem, Jackson e al. (2004) roose a generc local observer o combne ror knowledge of he sysem dynamcs n he rackng framework, where a consan velocy ror s mosed on he grou acon and a zero velocy ror s mosed on he conour. A jon mnmzaon of he energy s used o acheve he observed value of he grou acon and he conour. However, hs aroach has wo dsadvanages: nensve comuaon and nsably n he case of a nonlnear sysem. A jon mnmzaon over he grou acon and he conour a each me sam s comuaonally nensve, and s hard o choose an observer o guaranee sably for nonlnear sysem. To address hese roblems, Rah e al. (2005) formulae geomerc acve conours as a arameerzaon echnque o deal wh he deformable objecs. Ths aroach combnes a ror sysem model wh an observaon model, uses a arcle fler o esmae he condonal robably dsrbuon of he grou acon and he conour a each me se. However, hs mehod sll has wo major roblems. Snce hs mehod has o nclude some knd of redefned shae nformaon, one roblem s he dffculy o rack hghly deformable objecs whose shaes are no all redefned. Anoher roblem s he oor erformance when he racked objec s comleely occluded for many frames Flerng-based Trackng Kalman fler-based rackng The rackng of objec shae and locaon over me s well handled by he Kalman fler n he case of lnear dynamc sysems (Rehg and Kanade, 994). The Eended Kalman Fler (EKF) s he eenson of he Kalman fler o a nonlnear bu unmodal rocess where non-lnear behavor s aromaed by local lnearzaon (Jebara e al., 998). Zhao e al. (2004) develo a rackng sysem usng an ellsodal model for he gross human shae. The shae arameers are racked

42 3 usng a Kalman fler. Ths mehod uses an aearance model. The rackng mask of he model s an ellse raher han a boundng recangle, however, hs model sll suffers from he drawback of wrong udaes. Grondel e al. (2004) resen a mehod for rackng mulle ersons usng Kalman flerng and face deecon. They use a regon-based sraegy smlar o ha of Zhao e al. (2004). Face deecon resrcs he alcably of he mehod o vewons where skn color-based segmenaon may be erformed. Ths mehod uses a Kalman fler o overcome he occluson roblem. However, only aral Kalman flerng s used because several mage measuremens are lkely o resul n msses. Ths aroach akes advanage of a Kalman fler only n a redcve mode, hus resrcng o a smle moon model. Luo and Bhandarkar (2005) roose a mulle objec rackng mehod combnng he Kalman fler wh elasc machng. A regon-based model s used o model he objecs n a nework of grds. Each grd encodes he color nformaon and he feaure ons of he objec. The grd nework conans he conour and he objec shae nformaon. Ths mehod uses a Kalman fler o redc he velocy of he racked objec, and an elasc machng algorhm o localze he objecs defned by he objec model. The roosed rackng model consss of hree sub-models: he objec model, he velocy esmaon model, and he velocy measuremen model. Ths aroach has he advanage of beng able o rack boh rgd and deformable objecs. Anoher advanage s ha he elasc machng algorhm can rovde good rackng when he Kalman fler resuls n wrong redcon. However, hs aroach s resrced o suaons where he occluson s relavely shor, esecally f he moon model and objec model are smle.

43 Parcle fler-based rackng Varous arcle fler-based aroaches have been develoed o mrove he rackng erformance. I s wdely acceed ha he arcle fler has rackng erformance sueror o ha of Kalman fler (Chang e al., 2005). In hs cone, arcle flerng resens a robus objec rackng framework whou beng resrced o lnear sysems. Parcle flers, also known as sequenal Mone Carlo flers, have been wdely used n vsual rackng o address lmaons arsng from non-lneary and non-normaly of he moon model (L e al., 2003; Okuma e al., 2004). The basc dea of he arcle fler s o aromae he oseror densy usng a recursve Bayesan fler based on a se of arcles wh assgned weghs. For each frame of an mage sequence n he vsual rackng framework, a arcle fler usually consss of hree ses: samlng, weghng, and selecon. A se of arcles s drawn from a roosal dsrbuon n he samlng se. In he weghng se, each arcle s hen weghed based on he rao of s rue robably o s aromaed robably usng he roosal dsrbuon. Afer ouung he arcle saes and weghs for he oseror densy esmaon, he arcles are seleced (resamled) accordng o he esmaed oseror densy o oban a unform wegh dsrbuon n he selecon se. The Condensaon algorhm, a smle arcle fler, roosed by Isard (998) s desgned o solve he rackng roblems arsng from non-lneary and non-normaly of he moon model. In he samlng se, he Condensaon algorhm uses a smle roosal dsrbuon o draw a se of arcles, whch defnes he condonal dsrbuon on he arcle sae n he revous frame. Ths roosal dsrbuon does no make use of he nformaon from he curren frame. The laes observaon s only aled n he weghng se raher han n he samlng se. As a resul, generaes only a very rough esmaon of he oseror dsrbuon and also needs a

44 33 large number of arcles o reresen he oseror dsrbuon. MacCormck and Isard (2000) resen a aroned samlng echnque o solve hs roblem, whch requres ha he saesace be slced. Douce e al. (200) resen an omal roosal dsrbuon (OPD) for sae esmaon of jum Markov lnear sysems, whch s used o recursvely comue omal sae esmaes based on he selecon of he mnmum value of he varance of he weghs. However, hs aroach s comuaonally very nensve. To address he above roblems, L e al. (2003) roose a Kalman arcle fler (KPF) and an unscened arcle fler (UPF) o mrove he arcle samlng n he cone of vsual conour rackng. Ths aroach makes use of a Kalman fler or an unscened Kalman fler o ncororae he curren observaon. The Kalman fler or he unscened Kalman fler can seer he se of arcles o regons of hgh lkelhood n he search sace, and hus reduce he number of arcles. Ths aroach also uses he local lnearzaon of he OPD wh he Gaussan dsrbuon o resul n less nensve comuaon comared o he orgnal OPD. However, hs aroach does no handle he occluson roblem, and he Kalman fler and he unscened fler may resul n wrong udaes due o comlcaed moons of he objecs n he scene. To address he occluson roblem, Wang and Cheong (2005) roose a arcle fler wh a Markov random feld (MRF) based reresenaon of he racked objec whn a dynamc Bayesan framework. Ths mehod ransforms he objec no a comose of mulle MRF regons o mrove he modelng accuracy. Each MRF regon s able o swch labels beween foreground or background, hus he occluson can be accuraely modeled by elong he flebly of he observaon model. However, hs aroach has wo man roblems: one s he nensve comuaon nvolved n he MRF modelng; he oher s ha s hard o ge he sable and comac regons n he MRF mlemenaon. Usng he daa assocaon echnques, Chang

45 34 e al. (2005) resen a kernel arcle fler o mrove samlng effcency for mulle objec rackng. Ths scheme nvokes kernels o connuously aromae he oseror densy, where he kernels for objec reresenaon and localzaon are allocaed based on he graden derved from he kernel densy. Snce hs mehod assumes ha he objecs beng racked are ndsngushable from each oher n erms of he observaon model, s dffcul o handle suaons n whch he moon aern of objecs n one grou changes drascally. Ths s a general roblem wh mos daa assocaon echnques. Rah e al. (2005) formulae geomerc acve conours as a arameerzaon echnque o deal wh he deformable objecs. Ths aroach frs ncororaes a ror sysem model wh an observaon model, and hen uses a arcle fler o esmae he condonal robably dsrbuon of he grou acon and he conour a each me se. However, hs mehod erforms oorly when he racked objec s comleely occluded for many frames. Isard and MacCormck (200) roose a Bayesan mulle-blob racker (BraMBLe), an early mlemenaon of a arcle fler n whch he number of racked objecs can vary durng rackng. Based on he heory of Bayesan correlaon, hs aroach develos a robus observaon model ha recsely reresens he lkelhood of dfferng numbers of objecs beng racked. However, hs aroach reles on modelng a fed background o denfy foreground objecs. To address hs roblem, Okuma e al. (2004) rela he assumon of a fed background o handle real mage sequences, where he background may vary. Based on he work of Vermaak e al. (2003), Okuma e al. (2004) roose a boosed arcle fler (BPF) for mulle objec deecon and rackng, whch nerleaves he AdaBoos algorhm wh a smle arcle fler (he Condensaon algorhm). Ths aroach uses he AdaBoos algorhm o learn models of he objecs, and hese models are hen used o seer he arcle fler. The roosal

46 35 dsrbuon of he arcle fler ncororaes nformaon from AdaBoos n he curren observaon, whch releves he samlng/esmaon roblem n he Condensaon algorhm. However, hs mure mehod does no resen a sysemac way of ncororang objec models o guaranee accurae aromaon of he roosal dsrbuon, and also does no address he occluson roblem. 2.4 Summary We am o rovde a comrehensve revew of he leraure relaed o face deecon and vsual rackng, and o caegorze he varous aroaches roosed n over 90 aers. The face deecon mehods are also dvded no hree major caegores, and he vsual rackng aroaches are also dvded no hree major caegores. The face deecon mehods and he reresenave research works are summarzed n Table 2.. Table 2.2 summarzes he vsual rackng aroaches and he reresenave research works. However, noe ha some mehods can be classfed no more han one caegory. From Table 2. and he survey on face deecon n Secon 2.2, we can recognze ha sgnfcan rogress has been made n face deecon n he las decade. Face deecon has evolved from mehods ha use smle feaures and heurscs o mehods ha use mulle comle feaures, robably analyss and learnng algorhms. Due o he varaon n lghng condons, orenaon, ose, facal eresson, facal har, and occluson, face deecon s sll a challengng roblem n he comuer vson research communy. Alhough vsual objec rackng has made large rogress n he las decade as seen n Table 2.2 and he leraure revew n Secon 2.3, here s sll work o be done o deal wh comle moons of scene objecs, comle backgrounds, deformable shaes, and cases of comlee occluson. Currenly many

47 36 researchers have focused on he sascal analyss of he racked objecs, resulng n sascal models for he moon and aearances of he scene objecs, o handle he above challengng roblems. Ineresngly, here are only a few works ha deal wh he nerleavng of face deecon and vsual rackng. We can eec ha he research on robus face deecon and rackng wll sll reman an acve research area, snce he research addresses several dffcul roblems dealng wh general objec deecon, rackng and recognon. To address he roblems assocaed wh face deecon and vsual rackng revewed n Secon 2.2 and Secon 2.3, we roose a novel scheme for face deecon and rackng n hs hess by combnng he AdaBoos algorhm wh a new arcle flerng scheme, called an adave arcle fler (APF). The new APF uses a novel samlng echnque o oban much more accurae esmaon of he roosal dsrbuon and he oseror dsrbuon. We erm he combnaon of AdaBoos and APF as a boosed adave arcle fler (BAPF). Frs, he AdaBoos algorhm s used o deec faces n an nu mage, and he BAPF algorhm s hen used for face verfcaon and rackng n real vdeo sequences. The BAPF algorhm can oban good rackng resuls n suaons where he objecs are severely occluded.

48 37 Table 2. Face deecon mehods and her reresenave works Caegory Characerscs Works Feaure-based mehods Temlae-based mehods Imagebased mehods Boosng Learnng Neural Nework (NN) Suor Vecor Machnes (SVM) Oher Learnng Algorhms Facal feaures wh edges and Herers e al. (996) lnes Song e al. (2002) Gray scale Yang and Huang (994) Graf e al. (995) Skn color and ellcal edges Huang and Trved (2004) McKenna e al. (998) Naseem and Derche (2005) Mulle facal feaures Huang e al. (2004) Wang and ermaran (2000) Elasc bunch grah machng Wsko e al. (997) Snakes and emlaes Kwon and Lobo (994) Gunn and Non (996) Slhouees Samal and Iyengar (995) AdaBoos Vola and Jones (200a; 200b) Lenhar and Mayd (2002) Wang e al. (2004) FloaBoos L e al. (2002a; 2002b) S-AdaBoos Jang and Loe (2003) AdaBoos and PCA Zhang e al. (2004b) AdaBoos wh look-u-able Wu e al. (2004) ye weak classfers AdaBoos wh Gabor feaures Yang e al. (2004) Mullayer neural neworks Rowley e al. (996; 998) Curran e al. (2005) NN and Consraned Generave Féraud e al. (200) Model SVM wh olynomal kernel Osuna e al. (997) SVM wh Orhogonal Fourer Terrllon e al. (2000) and Melln Momens (OFMM) SVM wh Dscrmnang Shh and Lu (2004) Feaure Analyss Hdden Markov Model (HMM) Rabner and Jung (993) Nave Bayes classfer Schnederman and Kanade (998) Resrced Bayesan nework Schnederman (2004) Face Probably Graden Ascen Park e al. (2005) (FPGA)

49 38 Table 2.2 Vsual rackng mehods and her reresenave works Caegory Characerscs Works Blob-racker Inlle e al. (997 Skn color and ellcal edges Huang and Trved (2004) Connuous densy Markov Rabner (989) hdden model (CDHMM) Image-based rackng Mul-color model Bhandarkar and Luo (2005) Level-se mehod or geomerc Gan e al. (2005) aral dfferenal equaons (PDE) Skn color flerng Chen and Tddeman (2005) 2D human aearance model Thome and Mgue (2005) Snakes Kass e al. (987) Acve conour Blake and Isard (998) Isard (998) MacCormck (2000) Conour-based rackng Level se echnque Sehan (989) Yezz and Soao (2003) Jackson e al. (2004) Rah e al. (2005) Flerngbased rackng Kalman fler-based rackng Parcle fler-based rackng Kalman fler (KF) Rehg and Kanade (994) Eended Kalman Fler (EKF) Jebara e al. (998) KF wh ellse and color Zhao e al. (2004) Grondel e al. (2004) KF wh elasc machng Luo and Bhandarkar (2005) Condensaon algorhm Isard (998) PF wh aroned samlng MacCormck and Isard (2000) PF wh omal roosal Douce e al. (200) dsrbuon (OPD) Kalman arcle fler (KPF) and L e al. (2003) unscened arcle fler (UPF) PF wh Markov random feld Wang and Cheong (2005) (MRF) Kernel arcle fler Chang e al. (2005) PF wh geomerc acve Rah e al. (2005) conours Mulle-blob racker Isard and MacCormck (200) (BraMBLe) Boosed arcle fler (BPF) Okuma e al. (2004)

50 39 CHAPTER 3 FACE DETECTION AND TRACKING USING A BOOSTED ADAPTIVE PARICLE FILTER Zheng, W. and S. Bhandarkar. To be submed o IEEE Transacons on Paern Analyss and Machne Inellgence.

51 40 Absrac: Ths aer rooses a novel algorhm for negraed face deecon and face rackng based on a combnaon of a novel adave arcle flerng algorhm and an AdaBoos face deecon algorhm. The roosed mehod rovdes a general framework for deecng and rackng faces n vdeo sequences. Usng a novel samlng echnque, an adave arcle fler (APF) s nroduced o oban accurae esmaon of he roosal dsrbuon and he oseror dsrbuon for accurae rackng n vdeo sequences. The roosed scheme, ermed a Boosed Adave Parcle Fler (BAPF), combnes he APF wh he AdaBoos algorhm. The AdaBoos algorhm s used o deec faces n he nu mages, whle he APF s used o rack faces n vdeo sequences. The roosed BAPF algorhm s emloyed for face deecon, face verfcaon, and face rackng n vdeo sequences. The ndvdual erformances of face deecon and face rackng can be muually mroved n he roosed rackng rocedure. The resuls of eermens confrm ha he roosed BAPF algorhm rovdes a means for robus face deecon and accurae face rackng under varous rackng scenaros. Keywords: Face deecon, face rackng, arcle fler, boosed learnng

52 4 3. Inroducon Face deecon s moran n any human face relaed sysem, such as any fully auomac face recognon sysem, a vdeo-based survellance and warnng sysem, or face rackng and human rackng sysem. Recenly, face deecon usng machne learnng and sascal esmaon mehods has demonsraed ecellen resuls among all esng face deecon mehods. Much research has been conduced n he area of face deecon echnques, such as AdaBoos (Vola and Jones, 200a; Vola and Jones, 200b), FloaBoos (L e al., 2002), S-AdaBoos (Jang and Loe, 2003), neural neworks (Rowley e al., 996; Curran e al., 2005), Suor Vecor Machnes (SVM) (Osuna e al., 997; Shh and Lu, 2004), Hdden Markov Models (Rabner and Jung, 993), and he Bayes classfer (Schnederman and Kanade, 998; Schnederman, 2004). Vola and Jones (200a; 200b) roose a robus AdaBoos face deecon algorhm, whch can deec faces n a rad and robus manner wh a hgh deecon rae. L e al. (2002) roose he FloaBoos algorhm, an mroved verson of AdaBoos, for learnng a boosed classfer wh mnmum error rae. I uses a backrack mechansm o mrove he deecon rae afer each eraon of he AdaBoos rocedure. However hs mehod s comuaonally more neffcen han he AdaBoos algorhm. Jang and Loe (2003) roose S-AdaBoos, a varan of AdaBoos for handlng oulers n aern deecon and classfcaon. Snce hs mehod uses dfferen classfers n dfferen hases, s comuaonal effcency and accuracy are no sasfacory. Rowley e al. (996) have done he mos sgnfcan research among all face deecon mehods based on neural neworks. They emloy a mullayer neural nework o learn he face and nonface aerns from he ranng ses conssng of face and nonface mages. One drawback of her mehod s ha only urgh fronal faces can be deeced. Alhough Rowley e al. furher mrove he mehod o deec roaed face mages, he resul s no romsng because of low

53 42 deecon rae. Suor Vecor Machnes (SVMs) uses srucural rsk mnmzaon o mnmze he uer bound of he eeced generalzaon error (Osuna e al., 997; Shh and Lu, 2004). The major dsadvanages of SVMs are nensve comuaon and hgh memory requremens. Hdden Markov Models (HMMs) assume ha face and non-face aerns can be characerzed as a aramerc random rocess. These arameers can be obaned usng a well-defned esmaon rocedure (Rabner and Jung, 993). The am of ranng an HMM s o esmae he arorae arameers n an HMM model o mamze he robably of observng he ranng daa. Schnederman and Kanade (998) resen a nave Bayes classfer, whch elos he esmaon of he jon robably of local aearance and oson of a face aern a mulle scales. However, he erformance of he nave Bayes classfer s oor. To address hs roblem, Schnederman (2004) rooses a resrced Bayesan nework for objec deecon. Ths mehod searches he srucure of a Bayesan nework-based classfer n he large sace of ossble nework srucures. Objec rackng has been suded eensvely n he cone of comuer vson because of varous vson alcaons such as auonomous robos (Davson and Murray, 998), vdeo survellance (Borg e al., 2005), human eye rackng (Hansen and Hammoud, 2005) and human face rackng (Nummaro e al., 2003) ha use rackng algorhms eensvely. Issues of uncerany and error should be consdered n objec rackng under comle suaons (Inlle e al., 997). Therefore, many echnques have been develoed o solve he roblem of objec rackng. Many echnques have been develoed n he las decade for vsual objec rackng. Imagebased rackng mehods oban generc feaures from he mages and hen combne hem based on hgh-level scene nformaon. Inlle e al. (997) roose a blob-racker for human rackng n

54 43 real me. The background s subraced o erac foreground regons. The foreground regons are hen dvded no blobs based on color. Ths aroach runs fas, bu has a major dsadvanage n erms of mergng blobs when he objecs n he scene aroach each oher. Conour-based rackng assumes ha he objec s defned by boundares wh known roeres (Blake and Isard, 998; MacCormck, 2000; Rah e al., 2005). Conour-based rackng reles on shae models (conours), dynamcal models and mage measuremens. The rackng of objec shae and locaon over me s well handled by he Kalman fler n he case of lnear dynamc sysems (Rehg and Kanade, 994). The Eended Kalman Fler (EKF) s an eenson of he Kalman fler o a nonlnear bu unmodal rocess where non-lnear behavor s aromaed by local lnearzaon (Jebara e al., 998). I s wdely acceed ha he arcle fler s sueror n rackng erformance o he Kalman fler (Chang e al., 2005), snce he arcle fler resens a robus objec rackng framework whou beng resrced o lnear sysems. Parcle flers, also known as sequenal Mone Carlo flers, have been wdely used n vsual rackng o address lmaons arsng from non-lneary and non-normaly of he moon model (L e al., 2003; Okuma e al., 2004). The basc dea of he arcle fler s o aromae he oseror densy usng a recursve Bayesan fler based on a se of arcles wh assgned weghs. The Condensaon algorhm, a smle arcle fler, roosed by Isard (998) s desgned o solve he rackng roblems arsng from non-lneary and non-normaly of he moon model. Durng he samlng se, he Condensaon algorhm uses a smle roosal dsrbuon o draw a se of arcles, whch defnes he condonal dsrbuon on he arcle sae n he revous frame. Ths roosal dsrbuon does no make use of he nformaon from he curren frame.

55 44 Varous aroaches have been develoed o mrove he rackng erformance of a arcle fler. L e al. (2003) roose a Kalman arcle fler (KPF) and an unscened arcle fler (UPF) o mrove he arcle samlng n he cone of vsual conour rackng. Ths aroach makes use of a Kalman fler or an unscened Kalman fler o ncororae he curren observaon. The Kalman fler or he unscened Kalman fler can seer he se of arcles o regons of hgh lkelhood n he search sace, and hus reduce he number of arcles. To address he occluson roblem, Wang and Cheong (2005) roose a arcle fler wh a Markov random feld (MRF) based reresenaon of he racked objec whn a dynamc Bayesan framework. Ths mehod ransforms an objec no a comose of mulle MRF regons o mrove he modelng accuracy. Usng daa assocaon echnques, Chang e al. (2005) resen a kernel arcle fler o mrove he samlng effcency for mulle objec rackng. Ths scheme nvokes kernels o connuously aromae he oseror densy, where he kernels for objec reresenaon and localzaon are allocaed based on he graden derved from he kernel densy. However, hs mehod can no handle suaons n whch he moon aern of objecs n one grou changes drascally. Rah e al. (2005) formulae geomerc acve conours as a arameerzaon echnque o deal wh he deformable objecs. Bu he erformance of her echnque s oor when he racked objec s comleely occluded over many frames. Isard and MacCormck (200) roose a Bayesan mulle-blob racker (BraMBLe), an early mlemenaon of a arcle fler, n whch he number of racked objecs can vary durng rackng. Noneheless, hs aroach reles on modelng a fed background o denfy foreground objecs. To address hs roblem, Okuma e al. (2004) rela he assumon of a fed background o handle real mage sequences, where he background may vary. Okuma e al. (2004) roose a boosed arcle fler (BPF) for mulle objec deecon and rackng, whch

56 45 nerleaves he AdaBoos algorhm wh a smle arcle fler (he Condensaon algorhm). However, hs mehod does no resen a sysemac way of ncororang objec models o guaranee accurae aromaon of he roosal dsrbuon, and also does no address he occluson roblem. In hs aer, we roose a new arcle flerng scheme, whch s ermed as an adave arcle fler (APF), o enable much more accurae esmaon of he roosal dsrbuon and of he oseror dsrbuon. Based on he revous work of Isard (998), L e al. (2003), Vermaak e al. (2003) and Okuma e al. (2004), we also roose a novel scheme for face deecon and rackng by combnng he APF algorhm wh he AdaBoos algorhm. We erm he combnaon of he APF algorhm and he AdaBoos algorhm as a boosed adave arcle fler (BAPF). The AdaBoos algorhm s used o deec faces n an nu mage, and he BAPF algorhm s desgned for face verfcaon and rackng n real vdeo sequences. The BAPF algorhm can oban good rackng resuls n suaons n whch he objecs are severely occluded. Eermenal resuls show ha he roosed BAPF mehod rovdes robus face deecon and accurae face rackng under varous rackng scenaros. 3.2 Sascal Model Mahemacal noaon: The mahemacal noaon used n he formulaon of he sascal model and he arcle flerng algorhm s descrbed below:, a sae vecor for an objec conour; y, an observaon vecor; ( y ), observaon lkelhood (or ermed as observaon densy); T, lengh of he measuremen lne;

57 46, a fne number of samle ons on a conour; s, normal o he conour (or ermed as measuremen lne); m, nde of he deeced feaures; m, number of he deeced feaures; z, edge feaure; λ, densy for he he Posson dsrbuon of cluer feaures on he measuremen lne; T ( m ), he Posson dsrbuon of cluer feaures on he measuremen lne; ( z v { }), generc lkelhood funcon of he observaon a a samle on (, 2, L,n) = q 0, robably of undeeced feaures for an objec boundary; q, robably of deeced feaures for an objec boundary;, a sae vecor for an objec a me ; {,, L } =, a sae vecor hsory u o me ; : 2, y, an observaon vecor a me ; { y, y, L y } : 2, y =, an observaon hsory vecor u o me ;, mean value of a sae vecor; ω, Gaussan nose; A, mar descrbng he deermnsc comonen of he dynamcal model; B, mar descrbng he sochasc comonen; ( ) ( y ) : ( y ) :, dynamcal model (or ermed as ranson ror);, oseror densy;, effecve ror; = ;

58 47 ( y y ) :, observaon ror; f ( ), funcon of objec sae vecor; E[ f ( )], esmae of a funcon ( ) f ; L, number of eraons of loo l n he adave arcle fler; N, number of arcles; w () (), arcle wegh;, arcle sae vecor; ( ) q, y : N, roosal dsrbuon; ( ) ( ˆ ),ˆ P ( ), Gaussan dsrbuon;, arcle sae vecor comued whn loo l; ( ),l u l ( ), roosal dsrbuon comued whn loo l used n adave arcle fler; () ξ, ξ M () 2, wo secfc values n doman D m,, wo secfc values of a connuous funcon n doman D; m,, wo secfc values of a connuous funcon n doman D; 2 M 2 Φ, a connuous funcon n doman D; ma l, mn l, wo secfc values n doman D used o mose bounds on Φ whn loo l; K l, a consan whn loo l; E [ f ( ), ˆ ( ) ], samlng error a he eraon se l wh resec o ( ) c ( f ( )) f ; E, esmae of a samled on on he conour combnng he esmaon values from he APF and he AdaBoos algorhm; γ, wegh assgned o he Adaboos deecon algorhm;

59 48 η, confdence measure for each deeced face n he mage; d, dsance beween he cener of a deeced face and he cener of a samled emlae conour; F, number of he revous frames used for he esmaon of an objec n he curren frame Observaon Model We denoe a sae vecor for an objec by, and an observaon vecor s denoed by y. I s moran for conour rackng o oban accurae esmaon of he observaon lkelhood (or ermed as observaon densy) ( y ). Blake e al. (998), Isard e al. (998), and MacCormck e al. (998, 2000) nroduce sascal models for comung he observaon densy ( y ). These models use a se of normals o a hyoheszed conour o collec secfc mage feaures. A fne number of samle ons, called conrol ons, are generaed on he hyoheszed conour. We follow he general drecon of hese models for modelng he observaon rocess, bu n arcular follow he one roosed by MacCormck (2000). Fgure 3. shows an observed conour and mage feaures eraced along a measuremen lne. We denoe a fne number of samle ons on a hyoheszed conour by a se {,, 2, L,n} and erm he normals o he conour as measuremen lnes, whch are denoed by a se {,, 2, L,n} s =. The lengh of he measuremen lnes s fed a a value T. A Canny edge deecor s aled o he measuremen lne (, 2, L,n) s =, = n order o oban he osons of he edge feaures ( m) { z,m, 2, L, m } = (m s he nde of he deeced feaures, m s he number of he deeced feaures). Obvously, each feaure s jonly generaed by he boundary of an objec and he random cluer resened n he mage.

60 49 samle on (conrol on) hyoheszed conour z 2 z 3 z 4 z 5 z T measuremen lne (a) (b) Fgure 3. (a) Observaon rocess: he ellse s a hyoheszed conour n an mage. (b) The mage feaures on he measuremen lne. Cluer feaures on he measuremen lne (, 2, L,n) dsrbuon wh densy λ : s = are assumed o obey he Posson T ( m ) = m ( λt ) λt m! e (3-) where m s he number of deeced cluer feaures. A boundary densy funcon s assumed o obey a Gaussan dsrbuon, hus he generc lkelhood funcon of he observaon a a samle on (, 2, L,n) = can be descrbed by (MacCormck, 2000): ( v { }) m ( ) ( ) m m λt q z ( ) 2 λt z = = + e q0 e - (3-2) 2 m! λ = 2πσ 2σ where q 0 s he robably of undeeced feaures for an objec boundary, and q s he robably of deeced feaures for an objec boundary. Based on he assumon of ndeenden

61 50 and dencal dsrbuon of all samle ons, he overall lkelhood funcon of he observaon ( y ) can be reresened by: m ( ) ( ) m m λt q z ( ) n 2 λt ( ) y = e q0 + e - (3-3) 2 = m! λ = 2πσ 2σ Dynamcal Model Generally, a arcle fler algorhm requres a dynamcal model o demonsrae how a rackng sysem evolves over me. An auo-regressve rocess (ARP) model has been wdely used for he urose of dynamc modelng (Lukeohl, 993; Black e al., 995; MacCormck, 2000). Blake e al. (993, 995) model objec dynamcs as a second order rocess. Isard e al. (998) and L e al. (2003) follow he dynamcal model of Blake e al. (993, 995) for objec rackng. Followng he revous work of Blake e al. (993, 995), Isard e al. (998) and L e al. (2003), hs aer emloys a second-order ARP as he dynamcal model for face rackng. I s wdely acceed ha he second-order ARP caures varous moons of neres for vsual rackng (MacCormck, 2000). The arameers for he dynamc model n a ycal real alcaon can be obaned by learnng from he nu ranng daa. The second-order ARP resens he sae a me wh a lnear combnaon of he revous wo saes and addve Gaussan nose. The dynamcal model can be reresened as a second order lnear dfference equaon: where ( ) Bω = A + (3-4) ω s Gaussan nose ha s ndeenden of he sae-vecor, and denoes he mean value of he sae vecor. A and B are marces descrbng he dynamcal model wh he deermnsc comonen and he sochasc comonen, resecvely. The sae-vecor

62 5 encodes he knowledge of he objec conour n he curren sae and he revous sae. I s reresened by: X = X. In mos real alcaons, we se some reasonable defaul values for he arameers A, B and of he dynamcal model. I s effecve and sraghforward o aromae hem hrough vdeo sequences, n whch he objec conducs ycal moons (Blake e al., 995; Reynard e al., 996). The dynamcal model can also be reresened by a emoral Markov chan (Isard e al., 998): 2 ( ) = C e B (( ) A( ) ) (3-5) 2 where C s a consan, and denoes he Eucldean norm. 3.3 Face Trackng usng Parcle Flerng 3.3. The Flerng Dsrbuon We denoe a sae vecor for an objec a me by, and s hsory u o me by {,, L } : = 2,. Lkewse, an observaon vecor a me s denoed by u o me s denoed by y { y, y, L y } y and s hsory : = 2,. The sandard roblem of arge rackng n sascal aern recognon ermnology s o esmae he sae of he objecs a me, usng a se of observaons y from a sequence of nu mages. A oseror densy ( y ) : demonsraes all he nformaon abou a me ha s deducble from he se of observaons y u o ha me.

63 52 We assume ha objec dynamcs form a emoral Markov rocess and observaons y are ndeenden. Therefore, he dynamcs are deermned by a ranson ror ( ) ranson ror ( ) and he observaon densy ( y ) ( y ) :, he oseror densy. Gven he can be comued by alyng Bayes rule (Paouls e al., 990) for nferrng he oseror sae densy from me-varyng observaons. The oseror densy s esmaed recursvely va Bayesan flerng (Isard e al., 998; Douce e al., 200): where ( y ) : ( y, y : ) ( y : ) ( y y ) : ( y ) ( y : ) ( y y ) = = (3-6) ( y ) ( ) ( y ) d (3-7) The oseror densy ( y ) : = : : udang. Frs, an effecve ror ( y ) : s generally evaluaed n wo ses, namely redcon and shown n Eq. (3-7) s redced from he oseror : densy ( y ) va he ranson ror ( ) ( y ) : by: :. Second, he oseror densy s udaed based uon new observaon y a me, whch s eressed n Eq. (3-6). The observaon ror ( y ) y whch s he denomnaor n Eq. (3-6) can be reresened : ( y ) = ( y, y ) ( y ) ( y ) y : : = : Furhermore, he observaon ror ( y ) Thus Eq. (3-6) becomes: (3-8) y can be reresened by an negraon oeraor: : ( y ) ( y ) ( y ) y : = : d (3-9)

64 ( y ) ( y ) ( y : ) ( y ) ( y ) d : 53 : = (3-0) Based on Eq. (3-7), we subsue he effecve ror ( y ) ( y ) o oban: : ( y ) ( ) ( y : ) ( y ) ( ) ( y ) d d : = (3-) d Besdes he esmae of he oseror densy ( y ), he esmae of a funcon ( ) : : f of objec sae vecor s also comued under many suaons. We can aromae s eeced value of he funcon f ( ) by: E [ f ( )] f ( ) ( y ) (3-2) = : d Eq. (3-) and Eq. (3-2) reresen an omal soluon o he sandard roblem of objec rackng. Obvously, hs soluon nvolves hgh-dmensonal negraons, non-lneary and nonnormaly of he moon model under many rackng scenaros. Hgh-dmensonal negraons usually can no be comued easly. Thus a arcle fler, also known as a sequenal Mone Carlo fler, s adoed as a raccal soluon o he roblem of objec rackng The Sandard Parcle Fler A sandard arcle fler uses Mone Carlo smulaon o oban he oseror robably ( y ) : reresened by Eq. (3-). Parcle flerng makes use of random samlng sraeges n order o model a comle oseror robably ( y ) o aromae he oseror robably ( y ) : :. I uses N weghed dscree arcles by he observaon of he daa. Each arcle consss of a sae vecor and a wegh w. The weghed arcle se s gven by () () {(,w ), =, 2, L,N}. Parcle flerng samles he sace sanned by wh N dscree

65 54 arcles and aromaes he dsrbuon wh he assocaed weghs of he ons samled by he arcles. Secfcally, we assume ha N arcles are used for samlng o oban he oseror robably ( y ), and ha dscree samle ons n he sace are gven by : resecvely. Thus we have: 2 N,,..., N ( y : ) = w ( ) δ (3-3) = Snce s nfeasble o draw samles drecly from he oseror dsrbuon, a roosal dsrbuon ( ) q, y : s used o easly draw he samles for aromaon of he oseror robables. Based on he roosal dsrbuon ( ) q, y :, a arcle fler samles ( ) from () for arcle (, 2, L,N ) = and comues he wegh for ( ) usng he followng equaon: () = ( ) ( ) ( ) ( y ) () () ( ) (, y ) w () w q : (3-4) The oseror dsrbuon ( y ) can hus be aromaed as: : N ( ) () () ( y ) w δ (3-5) : = The esmae of he funcon f ( ) of he sae vecor could be comued as: E N ( ) () () [ f ( )] w f (3-6) = The sandard arcle fler can be descrbed as conssng of four ses: nalzaon, samlng, esmaon, and selecon (L e al., 2003). In he samlng se, a se of arcles s drawn from he roosal dsrbuon, and each arcle s weghed based on he rao of s rue robably o s aromaed robably usng he roosal dsrbuon. In he esmaon se,

66 55 he sandard arcle fler aromaes he oseror densy usng he ouu of he samlng se, namely he arcles saes and weghs. The arcles are seleced accordng o he esmaed oseror densy o oban a unform wegh dsrbuon n he selecon se. The sandard arcle flerng algorhm s llusraed n Fgure Inalzaon Inalze a se of arcles from he ror ( 0 ) o oban 0 0 = 2. Samlng se a) For =, 2,, N () Samle from he roosal dsrbuon q(, y : ). b) Comue he weghs of arcles () ( ) ( ) () ( y ) ( ) () w = () () w, q(, y : ) =, 2,, N c) Normalze () () w w = N () w, =, 2,, N = 3. Esmaon se Oban a se of arcles N () ( ) {(,w ), =, 2, L,N} ( ) () () ( y : ) w = where δ () s he Drac funcon. The esmae of ( ) N () () E[ f ( )] w f. ( ) ( ) {(,w ),, 2, L,N}. The oseror dsrbuon. Le =0. δ can be aromaed usng he ouu se of arcles, = 4. Selecon se Resamle arcles () ( ) wh robably ( ) y : aromaely dsrbued wh resec o ( ) () Assgn w =, =, 2,, N. N 5. Se =+, go o se 2. f can be comued by: w o oban N..d random arcles. (), Fgure 3.2 The algorhm of a sandard arcle fler

67 The Adave Parcle Fler The Adave Parcle Fler Algorhm Recenly, one of he acve research areas n arcle flerng s o generae a good roosal dsrbuon ( ) and hus oban a more accurae esmae of he oseror q, y : dsrbuon ( y ) :. The am s o oban a close aromaon o he oseror robably dsrbuon. The Condensaon (Isard e al., 998) algorhm makes no use of knowledge obaned from he curren mage frame, whch leads o a rough esmae of he oseror dsrbuon. Douce e al. (200) rovde an omal roosal dsrbuon (OPD) for sae esmaon of jum Markov lnear sysems, and recursvely comue omal sae esmaes based on he selecon of he mnmum varance of weghs ( ) w ( =, 2,, N ). To overcome he roblem of neffcen comuaon of he OPD, L e al. (2003) roose a Kalman arcle fler (KPF) and an unscened arcle fler (UPF) o drve a se of arcles o he regons n he search sace wh hgh lkelhood. L e al. (2003) emloy a local lnearzaon of he OPD o esmae he roosal dsrbuon, whch s assumed o be a Gaussan dsrbuon. Therefore, he roosal dsrbuon can be reresened as: u l ( ) ( ) ( ) ( ) ( ) q( ) ( ), y = N ˆ,ˆ P = :k =, 2,, N. (3-7) ( ) where mean ˆ and covarance ( ) ( ) ˆP characerze he Gaussan dsrbuon ( ˆ ) ( ),ˆ P N. In hs aer, we roose a new arcle flerng scheme, ermed as an Adave Parcle Fler (APF), o enable much more accurae esmaon of he roosal dsrbuon and he oseror dsrbuon. Our mehod eends he Condensaon algorhm and he Kalman arcle fler o oban an accurae aromaon of he roosal dsrbuon and he oseror dsrbuon.

68 57 In he samlng se of he APF algorhm as shown n Fgure 3.3, a new samlng sraegy s used o mrove he accuracy of he aromaon, whch s dfferen from he samlng used n oher arcle flers. The samlng se s he mos moran se n a arcle flerng algorhm. For each dscree arcle ( ),l ( ),l based on a roosal dsrbuon u ( ) l, he adave arcle fler generaes a new arcle. We use he loo conrolled by he arameer l n he APF algorhm o mlemen he new samlng echnque. L s he fed number of eraons of loo l. L can be adjused n dfferen real alcaons. When L =, he APF s equvalen o he ure sandard arcle fler. When L >, he APF erforms more samlng eraons han he sandard arcle fler. We wll rove n Secon ha he addonal eraons oban a lower esmaon error of he roosal dsrbuon and oseror dsrbuon. In order o enable more accurae esmaon of he roosal dsrbuon, we erae he samlng rocedure wh a consran, whch s called he Adave Learnng Consran (ALC). The ALC s descrbed usng he followng equaon whch s dealed n a laer secon. K ma α K mn (3-8) l l l l where ma l = ma N () () ( ) () { M ξ,l, M 2 ξ 2,l } () () ( ) () { m ξ,l, m2 ξ 2,l } mn = mn l N K l, l K are consans, 0 < α <. If he roosed consran s sasfed, he eraon for generang new arcles n he same mage wll connue. The eraon n he samlng se wll so when he roosed consran s no sasfed or he redefned loo hreshold s reached. Theorecally and raccally, he arcles wh sae vecor and weghs obaned n he laes eraon wll resen a beer

69 58 aromaon of he roosal dsrbuon and he oseror dsrbuon. The heorecal analyss and eermenal resuls wll be resened n laer secons o confrm he sueror erformance of he APF algorhm. The oher ses of APF are smlar o hose of oher arcle flers such as KPF and UPF. The nalzaon se akes advanage of he nformaon from he resuls of he AdaBoos face deecon algorhm. The adave arcle fler algorhm s descrbed n Fgure 3.3.

70 59. Inalzaon Inalze a se of arcles from he ror ( ) 2. Samlng se ) For l =, 2,, L a) For =, 2,, N () Samle from u l,l Fgure 3.3 The algorhm of adave arcle fler 0 o ge ( ) ( ) {(,w ),, 2, L,N} 0 0 = () based on he roosal dsrbuon u ( ),l ( ) ( ) ( ) ( ) ( ) q( ) ( ), y = N ˆ,ˆ P = :k l, where. Le =0. ( ) () (). Consruc l ( ) = w,l,l N δ, = where δ () s he Drac funcon. b) If he Adave Learnng Consran s sasfed, where K l mal α K l mnl : ) Comue he weghs of arcles () ( ) = y w ) Normalze () () w,l w,l = N w ( ) ( ) ( ) ( ) ( ) w,l,l,l, l () () w, 0 = w when l =. = (),l, =, 2,, N, =, 2,, N ) Connue he loo l b) If he Adave Learnng Consran s no me, where K l mal > α K l mnl : () () ( ) ( ) ) Le w = w,l, =,l, ) Break he loo l () () () () 2) Le w = w, =, =, 2,, N.,L (),L () w 3) w =, ( ) ( ) q(, y :k ) =, 2,, N. 3. Esmaon se Oban a se of arcles () ( ),w, =, 2, L,N. The oseror dsrbuon N () () ( y ) w : = {( ) } ( ) δ can be aromaed usng he ouu se of arcles. The esmae value of ( ) E N () () [ f ( )] w f = f can be comued as: ( ). 4. Selecon se Resamle arcles () wh robably ( ) w o oban N..d random arcles y : aromaely dsrbued wh resec o ( ) () Assgn w =, =, 2,, N. N 5. Se =+, go o se 2.. (),

71 Modelng The Adave Learnng Consran A crcal se n he adave arcle fler (APF) s obanng a good aromaon o o he samlng roosal dsrbuon u ( ) urose of choosng he roosal dsrbuon ( ) l, whch s shown n he samlng se n Fgure 3.3. The u recursvely n a gven sae s o reduce he esmaon error, whch s a resul of aromang he oseror dsrbuon ( y ) l wh a : fne number of arcles. The desgn of a sngle eraon of he esmaon of he roosal dsrbuon ( ) u wll be resened n he followng dervaon. The eraon s descrbed by he l loo l: l =, 2,, L n Fgure 3.3, where L s a gven defaul value. The addonal eraons of loo l n he frs loo of se 2 could be used o reduce he esmaon error adavely. We make followng analyss o rove hs on. Frs, we rove ha he eraons resul n he convergence of he esmae of he roosal dsrbuon. The convergence shows ha he esmaon error of he roosal dsrbuon a loo se l = k+ s less han ha of he roosal dsrbuon a loo se l = k, where (, 2,, L ) k L. Ths resuls n beer aromaon of he roosal dsrbuon and he oseror dsrbuon hrough he eraons of loo l. Thus, we can oban a lower esmaon error of he roosal dsrbuon and he oseror dsrbuon. Second, we resen he Adave Learnng Consran n he dervaon, whch clarfes he APF descrbed n Fgure 3.3. We defne he error of a samlng funcon ˆ ( ) wh resec o ( ) where [ f ( ), ˆ ( ) ] f ( ) ( ( ) ˆ ( ) ) f as: E = d (3-9) ( ) = ( y ) ( y ) ( ) ( y : ) ( y ) ( ) ( y ) d : =, : d d

72 6 ( ) ( ) () () ( ) = = N : w ˆ ˆ y δ, and denoes he Eucldean norm. The roagaon of errors beween he eraons n he adave arcle fler can be analyzed as follows. Secfcally, we consder a sngle eraon se l: { },L,, l L 2. From he APF algorhm shown n Fgure 3.3, he esmae of he roosal dsrbuon n he eraon se l s gven by: ( ) () () ( ) = = N,l,l l w δ (3-20) Thus, he samlng error ( ) ( ) [ ] ˆ, f E a he eraon se l wh resec o ( ) f s comued as: ( ) ( ) [ ] ( ) ( ) ( ) ( ) = d f ˆ, f E l ( ) ( ) ( ) ( ) ( ) ( ) ( ) () () ( ) N,l,l : : d w d d d f y y y y = = δ (3-2) Based uon Eq. (3-3), we have ( ) ( ) = = N : w y δ. Subsung ( ) : y wh ( ) = N w δ n Eq. (3-2) and erformng he Drac funcon comuaon, we oban he esmaon error as: ( ) ( ) [ ] l, f E ( ) ( ) () () ( ) ( ) () () ( ) () () ( ) N,l,l N N d w d w w f y y = = = = δ

73 62 ( ) ( ) () () ( ) ( ) () () ( ) () () ( ) N,l,l N N d w d w w f y y = = = = δ (3-22) From he APF algorhm shown n Fg. 3-2, we know () () ( ) ( ) ( ) ( ) ( ) l,,l,l,l w w = y (3-23) () () () = = N,l,l,l w w w ( ) ( ) ( ) ( ) ( ) ( ) () ( ) () () ( ) () = = N,l,l,l,l,l,l w w y y (3-24) Afer combnng Eq. (3-22), Eq. (3-23), and Eq. (3-24), we oban: ( ) ( ) [ ] l, f E ( ) ( ) () () ( ) ( ) () () ( ) = = = N N d w w f y y () ( ) () () ( ) () () ( ) () () ( ) () () ( ) N,l N,l,l,l,l,l,l d w w y y = = δ ( ) ( ) () () ( ) ( ) () () ( ) N N d d w w f y y = = = () ( ) () ( ) () () ( ) () () ( ) () () ( ) () = = N,l,l,l N,l,l,l,l w w f y y ( ) ( ) () ( ) () ( ) () ( ) () N N d w d w f y y = = = () ( ) () ( ) () () ( ) () () ( ) () () ( ) () = = N,l,l,l N,l,l,l,l w w f y y (3-25) Usng he Lagrange heorem, we could oban secfc values ( ) ξ and () 2 ξ n doman D: () D ξ, () D 2 ξ, ( ) N,,2,3, L = such ha ( ) ( ) () ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) () w ) ( f d w f = y y ξ ξ ξ (3-26) ( ) () ( ) () ( ) ( ) ( ) ( ) ( ) ( ) w d w 2 2 = y y ξ ξ (3-27)

74 63 Therefore we oban: E [ f ( ), ( ) ] l N f = = N () () () () ( ξ ) ( y ξ ) ξ = () () () ( y ξ ) ξ 2 ( ) () w ( ) () w 2 N = f () () ( ) y N =,l () () ( ) ( ) () w () () () ( y ),l,l ( ) () w,l,l,l,l = N = f () () () () ( ξ ) ( y ξ ) ξ N () = ( ) () w ( y ξ () () ) ( ) () ξ w Suose ha f ( ), ( ) he followng equaon: m, M R, 2 2 y, ( ) ( ) f () () (,l ) y,l N () = ( () () ) ( ) (),l w,l ( y () () ) ( ) () w,l,l,l (3-28) are connuous funcons on doman D, hence we have m ξ f () (),l () () () () ( ξ ) ( y ξ ) ξ M ξ () (),l () () () () () ( ) ( ) ( ) ( ) () w f y w,l,l,l,l (3-29) Lkewse, we have: m, 2 M 2 R m 2 ξ N = M 2 2 () () ( y ξ 2,l ξ 2 () () ( ξ ) () () (),l ) 2 w N () () = () () ( ) ( ) () y w,l,l,l (3-30) Le () () ( ) ( ) ( ) ( ) ( y ξ )( ξ ) w ( ) F = f ξ (3-3)

75 64 F ( ) () ξ w N () () () 2 = ( y 2 ) 2 = ξ (3-32) F () = () ( ) ( ) y ( ) ( ) ( ) ( ) w ( ) f,l,l,l, l ( ) - F (3-33) = F 2 N = () () () ( ),l ( ) () w y - F 2 (3-34),l,l Thus we oban: E [ f ( ), ( ) ] = = l F () () () F + F N = F2 F2 + F2 F ( F + F ) () () ( ) ( ) F + F F N = F2 2 2 ( F + F ) = F () () F F F N 2 2 = F2 ( F2 + F2 ) N = F () () F F F F2 (3-35) Le Snce f ( ), ( ) F () y, Φ = N = () ( ) () () F F2 F2 F (3-36) are connuous funcons defned on doman D, ( ) F, F 2,, F2 are also connuous funcons on doman D. Furhermore, Φ s also a connuous funcon on doman D. Based on he roeres of connuous funcons, Eq. (3-29) and Eq. (3-30), Φ s bounded by wo secfc values, ma l and mn l. Le

76 ma mn l l = ma N = mn N ( ) () ( ) () { M,l, M 2 ξ 2,l } 65 ξ (3-37) ( ) () ( ) () { m,l, m2 ξ 2,l } ξ (3-38) We oban: where K l l ( f ( X ), l ( X )) K l mal mn E, for loo se l (3-39) K l s a consan. Lkewse, we can ge he followng equaon a loo se l-: Le K mn E ( f ( ), l ( ) ) K l mal, for loo se l- (3-40) l l K K l l ma mn l l α <, where 0 < α <, (3-4) hus we oban: E ( f ( X ) ( X )) E( f ( X ), ( X )) α, 0 < α < (3-42), l l If Eq. (3-4) s sasfed, hen Eq. (3-42) ensures ha he esmaon error for he roosal dsrbuon and oseror dsrbuon converges durng he eraons. Eq. (3-4) s only a necessary condon for he convergence of he esmae, whch can be learned from he comuaon durng he eraons. So we name as he Adave Learnng Consran (ALC). Eq. (3-4) can be reresened by: K ma α K mn (3-43) l l l l Eher Eq. (3-4) or Eq. (3-43) s ermed as he ALC. The ALC can be guaraneed by searchng he ma l and mn l from he N arcles n each eraon as follows:

77 () (). Fnd ξ, ξ 2, M, 2 (), ξ, M, 2 2 M, (, 2, L,N ) M, (, 2, L,N ) 66 = from he N arcles n loo l. Deermne ξ = from he N arcles n loo l-. 2. Search for ma l and mn l usng Eq. (3-37) and Eq. (3-38). 3. Deermne wheher or no he ALC s sasfed accordng o Eq. (3-43). Thus, we rove ha he eraons of loo l resul n he convergence of he esmae of he roosal dsrbuon. The convergence demonsraes ha he esmaon error of he roosal dsrbuon a loo se l = k+ s less han ha of he roosal dsrbuon a loo se l = k, where (, 2,, L ) k L. Ths resuls n beer aromaon of he roosal dsrbuon and he oseror dsrbuon hrough he eraons of loo l. As a resul, we can oban a lower esmaon error of he roosal dsrbuon and he oseror dsrbuon. Therefore, we confrm ha he APF algorhm wh ALC can resul n a more accurae esmae of he roosal dsrbuon and oseror dsrbuon. Generally, as more frames are rocessed durng rackng, general arcle flers wll resul n a monooncally ncreasng rackng error. However, he roosed APF algorhm s desgned o mrove he esmae of he roosal dsrbuon and he oseror dsrbuon as a rackng sysem evolves over me. ( ) 3.4 The Boosed Adave Parcle Fler The boosed adave arcle fler (BAPF) for face deecon and rackng emloys wo objec models: he conour-based model used n he adave arcle fler (APF) and he regon-based model used n face deecon. The objec models used n he cone of rackng le n hree general caegores (Luo, 2005): he conour-based models (L e al., 2003; Koller e al., 994; Terzooulos e al., 993), he regon-based models (Isard e al.,200; McKenna e al., 2000;

78 67 Nummaro e al., 2003), and he feaure on-based models (Ruckldge, 995; Malk e al., 2002; Lee e al., 2004). Snce he BAPF algorhm uses wo models n face deecon and rackng, has advanages over he general arcle fler. The ncororaon of he AdaBoos algorhm whn he APF algorhm subsanally mroves he robusness of he BAPF algorhm. The AdaBoos algorhm resens a mechansm for mananng he combned reresenaon, whch makes he BAPF algorhm more owerful han he naïve K-means cluserng mehod of Vermaak e al. (2003). The BAPF algorhm also erforms beer han he mure reresenaon roosed by Okuma e al. (2004) snce our aroach emloys a more effecve arcle flerng algorhm,.e., he APF algorhm. Secfcally, he BAPF algorhm allows us o effecvely deec faces leavng and enerng he regons of neres, and he BAPF algorhm rovdes robus face deecon and accurae face rackng under varous rackng scenaros Face Deecon hrough AdaBoos Among he varous face deecon mehods, he boosed learnng-based face deecon mehods have demonsraed ecellen resuls. Based on he revous work of Teu e al. (2000) and Schnederman (2000), Vola and Jones (200a; 200b) have roosed a robus face deecon algorhm, whch can deec faces n a rad and robus manner wh a hgh deecon rae. The face deecon echnque n AdaBoos s comrsed of hree asecs: he negral mage, a srong classfer comrsng of weak classfers based on he AdaBoos learnng algorhm, and an archecure comrsng of a cascade of a number of srong classfers. The sysem of Vola and Jones (200a; 200b) emloys an negral mage comrsng of Haar-lke feaures (Lenhar and Mayd, 2002) for effecve feaure eracon from a large

79 68 feaure se. In he boosng rocedure, AdaBoos frs learns effecve feaures from a large feaure se. Second, consrucs a se of weak classfers, each of whch s comosed of a feaure, a hreshold and a ary. Thrd, generaes a srong classfer based on he above weak classfers. Each eraon n he AdaBoos algorhm generaes a weak classfer. Afer all eraons are comleed, he resul s a se of weak classfers. These weak classfers are combned no a srong classfer usng a weghed lnear combnaon. The sysem of Vola and Jones (200a; 200b) uses a cascade of srong classfers o mrove deecon rae wh effcen comuaon. The dea s o consruc smaller and effcen classfers based on he sub-wndows whn he mage. The smler and faser classfers wll rejec he negave sub-wndows. A large number of negaves are rejeced by he nal classfer wh mnmal rocessng. Addonal negaves are elmnaed by subsequen layers whle requrng addonal comuaon. The number of sub-wndows o be rocessed reduces radly afer several sages of rocessng. We emloy he sysem of Vola and Jones (200a; 200b) for face deecon. A 25 layer cascade of boosed classfers s raned o deec mulvew faces. A se of face and nonface (ermed as background) samle mages are used for ranng. Each samle mage s croed and scaled o a resoluon of els. A se of 6230 mulvew face mages are colleced from vdeo sequences wh dfferen reflecons, llumnaons and backgrounds o make face deecon more robus n dfferen scenaros. Anoher se of 6598 nonface eamles wh he sze of els are colleced from vdeo sequences conanng no faces. The deals of AdaBoos ranng and AdaBoos face deecon resuls are resened n Secon 3.5.

80 Inegrang Adave Parcle Fler wh AdaBoos The roosed face deecon and rackng model consss of wo submodels: an AdaBoos face deecon model and an adave arcle fler face rackng model. The AdaBoos face deecon model erforms mulvew face deecon based on he raned AdaBoos algorhm. The APF model conducs vsual conour rackng usng he arcle flerng algorhm descrbed n Secon Fgure 3.4 shows he srucure of he roosed face deecon and rackng model. Sngle Frame Adave Parcle Fler Face Trackng Predcon Inalzaon Verfcaon Defnon AdaBoos Face Deecon Focus-of-aenon Fgure 3.4 Inegrang he APF wh AdaBoos whn a sngle feedback conrol sysem The rocess for face deecon and rackng conans wo hases: an nalzaon hase and a rackng hase. In he nalzaon hase of he APF, he AdaBoos face deecon model can rovde he nal arameers for he APF face rackng model based on he observaons of he mage sequences durng a ceran me nerval. Durng he rackng hase, he AdaBoos face deecon model and he APF face rackng model mrove he rackng erformance va muual neracon. The AdaBoos deecon model hels he APF model o fnd and defne new objecs, and o verfy he curren saes of he objecs beng racked. On he oher hand, he APF model rovdes focus-of-aenon regons whn he mage o seed u he AdaBoos face deecon. Afer alyng AdaBoos face deecon o one mage, we oban a confdence measure η for each deeced face n he mage from he deecon rocedure. From he APF algorhm, he

81 70 esmae of f ( ) n he adave arcle fler a each samle on along he conour s comued as: E N ( ) () () [ f ( )] w f (3-44) = We combne he resuls of he AdaBoos algorhm and he APF algorhm o oban new oson for a samled on, whch s descrbed by: where E c ( f ( )) = ( γ ) E( f ( )) + γ η d (3-45) E c reresens he esmae of a samled on n he conour combnng he esmaon values from he APF and he AdaBoos algorhm, he arameerγ s he wegh assgned o he Adaboos deecon, he arameer η s a confdence measure for each deeced face n he mage, and d s he dsance beween he cener of a deeced face and he cener of a samled emlae conour. The value of ( f ( )) E s fed back o he APF for furher rocessng. c The arameerγ can be adjused whou affecng he convergence of he adave arcle fler. Whenγ = 0, our aroach s equvalen o he ure adave arcle fler. By ncreasngγ, we emhasze he AdaBoos face deecon. Whenγ =, our aroach s equvalen o he ure AdaBoos algorhm. In realy, we could adjus he value of he arameerγ based on dfferen scene condons deermned by cluer, llumnaon and occlusons. 3.5 Eermenal Resuls 3.5. AdaBoos Face Deecon The sysem of Vola and Jones (200a; 200b) s used o deec faces n nu mages. In our eermen, we ran a 25-layer cascade of srong classfers o deec mulvew faces n vdeo sequences. A daa se s comosed of face and nonface mages of sze A se of 6230

82 7 mulvew face mages of 8 ersons are colleced from vdeo sequences wh dfferen reflecons, llumnaons and backgrounds o make face deecon more robus n dfferen scenaros. The face mages are croed and scaled o a resoluon of els. Anoher se of 6598 nonface eamles wh he sze of are colleced from vdeo sequences conanng no faces. The nonface eamles use he same sze as he one emloyed by he vdeo camera for real vdeo sequence acquson, bu hs s no a requremen. Fgure 3.5 shows some random face eamles used for he ranng, and Fgure 3.6 shows some random nonface eamles used for ranng. A larger ranng se of face and nonface eamles ycally leads o beer deecon resuls, alhough falures sll es n regons of overla and cluer. Some resuls of face deecon usng our raned AdaBoos are llusraed n Fgure 3.7. AdaBoos face deecon erforms well n mos cases, bu ofen leads o false osves n comlcaed sequences conssng of cluer or overlas. Fgure 3.5 Face eamles

83 72 Fgure 3.6 Nonface eamles Fgure 3.7 Resuls of fronal face deecon and mulvew face deecon Boosed Adave Parcle Fler The roosed boosed adave arcle fler (BAPF) s mlemened usng C++ under he Mcrosof Vsual C++.NET envronmen on a Penum M.6 GHz Comuer. Vdeo sequences are of sze els and are samled a 30 frames er second. In he begnnng, he AdaBoos face deecon model rovdes he nal saes for he adave arcle fler (APF) face rackng model for observaons of he mage sequences durng a ceran me nerval. Snce he conour defnes he aearance of he face n he vdeo sequences s roughly crcular or 2 2 ellcal, we use a smle arameerzed model o reresen he conour.e., A + By + C = 0.

84 73 Of course, our mehod can also be aled o more comle conours ha use B-slne reresenaons. The roosed BAPF algorhm has been aled o varous rackng scenaros as shown n Fgure 3.8 hrough Fgure 3.3. The rackng resuls from hree es vdeo sequences shown below are caured under varous lghng condons, scales, occlusons, and roaons. Two es vdeos (es vdeo and 3) comrse of one face n he scene, es vdeo s used for eermens n dfferen rackng scenaros, and es vdeo 3 s used o comare he BAPF algorhm wh he Condensaon Algorhm. The hrd es vdeo (es vdeo 2) ha comrses wo faces n he scene s used for mul-face rackng eermens on dfferen scenaros. All rackng resuls are obaned usng N = 000 arcles n he APF. In Fgure 3.8 hrough Fgure 3.3, a yellow ellse mles he absence of occluson, whereas a red ellse means ha occluson has occurred. Fgure 3.8 resens he snashos of sngle face rackng n es vdeo whle he scale of he face changes. I shows ha he roosed BAPF rackng algorhm can handle sgnfcan scale changes n he objec aearance. Fgure 3.9 llusraes he snashos of sngle face rackng n es vdeo under changng llumnaon. I demonsraes ha he roosed BAPF algorhm s robus o changes n varous lghng condons due o he negraon of he AdaBoos sascal learnng and he robusness of adave arcle flerng. Fgure 3.0 descrbes he snashos of sngle face rackng n es vdeo under changes n vewon and n-lane roaons. I roves ha he roosed BAPF algorhm can handle mulvew face deecon and rackng. Fgure 3. rovdes he snashos of sngle face rackng n es vdeo wh n-lane roaons. I shows ha he roosed BAPF algorhm can handle he aearance changes due o objec roaons. Fgure 3.2 llusraes he snashos of sngle face rackng n es vdeo where occlusons haen. I confrms ha he roosed BAPF algorhm erforms correcly n he resence of occlusons because of he

85 74 robusness of adave arcle flerng. Fgure 3.3 resens he snashos of wo-face rackng n es vdeo 2 under varous rackng scenaros. Fgure 3.8 Trackng resuls wh scale changes n es vdeo. From lef o rgh, he frame numbers are 98, 043, and 067. Fgure 3.9 Trackng resuls wh llumnaon changes n es vdeo. From lef o rgh, he frame numbers are 866, 954, and 969.

86 75 Fgure 3.0 Trackng resuls wh mulvews and roaons n es vdeo. Yellow ellse means ha no occluson has occurred, whereas red ellse means ha occluson has occurred. From o lef o boom rgh, he frame numbers are 55, 59, 524, 530, 533, 544, 566, 573, and 585. Fgure 3. Trackng resuls wh ou-of-lane roaons n es vdeo. From lef o rgh, he frame numbers are 04, 35, and 52.

87 76 Fgure 3.2 Trackng resuls wh occlusons n es vdeo. Yellow ellse means ha no occluson has occurred, whle red ellse means ha occluson has occurred. From o lef o boom rgh, he frame numbers are 356, 359, 362, 366, 382, and 397. Fgure 3.3 Trackng resuls of wo faces n es vdeo 2. Yellow ellse means ha no occluson has occurred, whle red ellse means occluson has occurred. From o lef o boom rgh, he frame numbers are 2, 4, 25, 38, 78, and 38.

88 77 The erformance of he BAPF algorhm has been comared o he Condensaon algorhm, a general arcle fler. Boh algorhms emloy N = 000 arcles for face rackng n he es vdeo 3. The eermenal resuls show ha rackng accuracy of he BAPF algorhm s sueror o ha of he Condensaon algorhm. The BAPF algorhm rovdes beer erformance han he Condensaon fler. However, beer erformance does no necessarly mean hgher comuaonal effcency. The BAPF algorhm acually needs more comung me han he Condensaon algorhm snce he BAPF algorhm erforms more comuaon on accoun of he addonal eraons needed o oban beer nonlnear esmaons. Some eamles of rackng resuls are resened n Fgure 3.4 for he BAPF algorhm and Fgure 3.5 for he Condensaon algorhm. Fgure 3.4 Trackng resuls wh he BAPF a s dfferen mes n es vdeo 3. From o lef o boom rgh, he frame numbers are 2, 40, 6, 36, 58, and 80.

89 78 Fgure 3.5 Trackng resuls wh he Condensaon algorhm a same mes as n Fgure 3.4. From o lef o boom rgh, he frame numbers are 2, 40, 6, 36, 58, and 80. Usng rackng accuracy and comuaon me, we quanavely analyze he erformance of he BAPF algorhm, he APF algorhm, and he Condensaon n hs secon. The rackng accuracy s defned by he dslacemen errors beween he cenrod of a ground ruh face and he cenrod of a racked face n vdeo sequences. All hree algorhms are esed on he es vdeo 3, and all algorhms emloy N = 000 arcles for face rackng. In he followng quanave analyss, he erformance of he APF algorhm s comared o he Condensaon algorhm, he erformance of he BAPF algorhm s comared o he APF algorhm, he erformance of he APF algorhm s analyzed wh dfferen values of he arameer L, he erformance of he BAPF algorhm s analyzed wh dfferen values of he arameer F, and he erformance of he BAPF algorhm s analyzed wh dfferen values of he arameerγ. We comare he erformance of he APF algorhm o he Condensaon algorhm. Boh algorhms emloy N = 000 arcles for face rackng n he es vdeo 3. In he APF algorhm, he number of he eraons of he loo l s L = 3. The eermenal resuls, as shown n Fgure

90 and Table 3., demonsrae ha rackng accuracy of he APF algorhm s beer han ha of he Condensaon algorhm. I can be seen from Table 3. ha he mean value of he dslacemen error n he APF algorhm s less han ha of he dslacemen error n he Condensaon algorhm, and he comuaon me of he APF algorhm s greaer han he Condensaon algorhm. The APF algorhm hus rovdes beer erformance han he Condensaon algorhm. The comuaon me of he APF algorhm s comarable bu greaer han ha of he Condensaon algorhm, snce he APF algorhm erforms more comuaon on accoun of he addonal eraons needed o oban beer esmaons of he roosal dsrbuon and he oseror dsrbuon. 45 APF vs. Condensaon Dslacemen (els) APF Condensaon Frame number Fgure 3.6 Trackng resuls of he APF algorhm and he Condensaon algorhm

91 80 Table 3. Summary of rackng resuls of he APF and he Condensaon algorhm APF Condensaon Mean dslacemen error (els) Sandard devaon (els) Seed (frame/sec) The erformance of he BAPF algorhm s comared o he APF algorhm. Boh algorhms emloy N = 000 arcles for face rackng n he es vdeo 3. The number of he eraons of he loo l s L = 3 for boh algorhms. In he BAPF algorhm, he wegh assgned o he AdaBoos face deecon sγ = 0. 8, and he arameer F for he number of he revous frames s. The eermenal resuls, as shown n Fgure 3.7 and Table 3.2, demonsrae ha he rackng accuracy of he BAPF algorhm s beer han ha of he APF algorhm. The BAPF algorhm rovdes beer erformance han he APF algorhm. I can be seen from Table 3. ha he mean value of he dslacemen error n he BAPF algorhm s less han ha of he dslacemen error n he APF algorhm, and he comuaon me of he BAPF algorhm s larger han he APF algorhm snce he BAPF algorhm erforms AdaBoos face deecon n each frame.

92 8 40 BAPF vs. APF Dslacemen (els) BAPF APF Frame number Fgure 3.7 Trackng resuls of he BAPF algorhm and he APF algorhm Table 3.2 Summary of rackng resuls of he BAPF and he APF BAPF APF Mean dslacemen error (els) Sandard devaon (els) Seed (frame/sec) The erformance of he APF algorhm s analyzed usng dfferen values of he arameer L. L s he number of eraons of loo l n he APF algorhm. The APF algorhm emloys N = 000 arcles for face rackng n he es vdeo 3. The number of he eraons L changes from o 4 n he eermens. The eermenal resuls, as shown n Fgure 3.8 and Table 3.3, demonsrae ha rackng accuracy of he APF algorhm s mroved as he number L ncreases.

93 82 I can be seen from Table 3.3 ha he mean value of he dslacemen error n he APF algorhm wh large L s less han ha wh small L. Thus, he APF algorhm wh large L rovdes beer erformance han wh small L. However, he comuaon me of he APF algorhm ncreases as he number L ncreases snce he APF algorhm erforms addonal eraon o esmae he oseror dsrbuon. When L s greaer han 3, we can see from Fgure 3.8 and Table 3.3 ha he esmaon accuracy of he oseror dsrbuon s mroved bu no sgnfcanly, whereas he comuaon me ncreases sgnfcanly. We have o choose a balance beween he esmaon accuracy and he comuaon me n real alcaons. In our eermens, we choose L = 3 for he APF algorhm and he BAPF algorhm. 45 APF Dslacemen (els) L=4 L=3 L=2 L= Frame number Fgure 3.8 Trackng resuls of he APF algorhm wh dfferen values of he arameer L

94 83 Table 3.3 Summary of rackng resuls of he APF wh dfferen values of he arameer L L=4 L=3 L=2 L= Mean dslacemen error (els) Sandard devaon (els) Seed (frame/sec) The erformance of he BAPF algorhm s analyzed based on dfferen values of he arameer F. In Eq. (3-45), we combne he resuls of he APF algorhm and he AdaBoos algorhm o oban he curren conour of he racked face n he curren frame. The AdaBoos face deecon s erformed for each frame n he vdeo sequences used n our eermens. Usng AdaBoos face deecon, we oban he esmaed oson of a deeced face based on he resuls of he revous F frames. We defne F as he number of he revous frames for he esmaon of he face n he curren frame. For eamle, F = 3, we average he osons of he deeced face among he revous 3 frames ncludng he curren frame o oban an esmaed oson of he face n he curren frame. Then we use Eq. (3-45) o combne he resuls of he APF algorhm and he AdaBoos algorhm. In hs eermen, he wegh assgned o he resul of AdaBoos face deecon s γ = 0. 8 n he BAPF algorhm. The number of he revous frames F s vared whn a se of values conssng of, 3, 5, and 0. The number of he eraons s L = 3. The BAPF algorhm emloys N = 000 arcles for face rackng n he es vdeo 3. The eermenal resuls, as shown n Fgure 3.9 and Table 3.4, show ha rackng accuracy of he BAPF algorhm s decreased as he number F ncreases. I can be seen from Table 3.4 ha he mean value of he dslacemen error n he BAPF algorhm wh large F s greaer han ha wh small F. Thus, he BAPF algorhm wh small F rovdes beer erformance han ha wh large F. The comuaon

95 84 me of he BAPF algorhm s same for dfferen values of he arameer F. Thus, we choose F = for he BAPF algorhm n our eermens. 25 BAPF 20 Dslacemen (els) F=0 F=5 F=3 F= Frame number Fgure 3.9 Trackng resuls of he BAPF algorhm wh dfferen values of he arameer F Table 3.4 Summary of rackng resuls of he BAPF wh dfferen values of he arameer F F=0 F=5 F=3 F= Mean dslacemen error (els) Sandard devaon (els) Seed (frame/sec)

96 85 The erformance of he BAPF algorhm s analyzed usng dfferen values of he arameerγ (ermed as gamma n Fgure 3.20), whereγ s a wegh assgned o he AdaBoos face deecon n he BAPF algorhm as shown n Eq. (3-45). The wegh γ s vared whn a se of values conssng of 0, 0.5, 0.8, and.0. Whenγ = 0, he BAPF algorhm s equvalen o he ure APF algorhm. By ncreasngγ, we emhasze he AdaBoos face deecon. Whenγ =, he BAPF algorhm s equvalen o he ure AdaBoos algorhm. The arameer F for he number of he revous frames consdered s. The number of he eraons of he loo l s L = 3. The BAPF algorhm emloys N = 000 arcles for face rackng n he es vdeo 3. The eermenal resuls, as shown n Fgure 3.20 and Table 3.5, show ha rackng accuracy of he APF algorhm wh dfferen weghs γ vares. I can be seen from Table 3.5 ha he mean value of he dslacemen error for he BAPF algorhm wh γ = 0. 8 s he smalles, and he mean value of he dslacemen error for he BAPF algorhm wh γ = 0 s he larges. The eermenal resuls show ha he erformance of he ure AdaBoos algorhm s he wors, snce he ure AdaBoos algorhm deecs false faces or msses he racked face n he vdeo sequences. In he BAPF algorhm, he APF algorhm rovdes regons of neres o he AdaBoos algorhm, and he AdaBoos rovdes he deeced face for he combnaon funcon n Eq. (3-45). Thus, he erformance of he BAPF algorhm s beer han eher he APF algorhm or he AdaBoos algorhm used ndvdually. Table 3.5 shows ha he comuaon me of he BAPF algorhm wh dfferen γ values s he same snce he APF algorhm and he AdaBoos algorhm are erformed for each frame of vdeo sequences. In our eermens, we choose γ = 0. 8 for he BAPF algorhm.

97 86 20 BAPF 00 Dslacemen (els) gamma=0 gamma=0.5 gamma=0.8 gamma=.0 Frame number Fgure 3.20 Trackng resuls of he BAPF algorhm wh dfferen values of he arameer γ Table 3.5 Summary of rackng resuls of he BAPF wh dfferen values of he arameer γ γ =0 γ =0.5 γ =0.8 γ =.0 Mean dslacemen error (els) Sandard devaon (els) Seed (frame/sec) Trackng s successful hroughou ece when comlee occluson occurs for long me duraon, as shown n Fgure 3.6. In hs case, he occluded face can no be dsngushed from he foreground or he background. In case of occluson for shor me duraon, we can assume ha he occluded face s locaed a he same lace n he mage durng he occluson. However, we can no make same assumon for occluson over a longer me erod, snce he occluded

98 87 erson may walk ou of he scene durng he occluson. When he faces of hree eole are occluded and are algned wh he ocal as of he camera as shown n Fgure 6 (b), s hard o deec and rack he faces of he wo eole n he back whch resuls n rackng falure as well. From boh cases shown n Fgure 6, s clear ha from he aearance of he face alone s no ossble o relably deermne he locaons of he occluded face. For correc rackng, we have o elo more nformaon such as he aearance of he body or he lmbs o augmen he BAPF algorhm o handle cases of comlee occluson over long me duraon. However, he augmened aearance model for accurae rackng wll ncrease he comley of a dynamcal model, whch may reduce robusness n oher cases. comlee occluson aral occluson (a) (b) Fgure 3.2 (a) Trackng falure n case of occluson for a long me duraon (b) Trackng falure n case of hree eole overlang