Human Action Recognition in Smart Classroom

Transcription

1 Huma Aco Recogo Smar Classroom Habg Re Guagyou Xu (Depe of Compuer Scece ad echology, sghua Uversy, Bejg 00084, P.R.Cha) E-mal: Absrac hs paper preses a ew sysem for eachers aural complex aco recogo smar classroom order o realze ellge camerama ad vrual mouse. Frs, he sysem proposes a hybrd huma model ad employs 2-order B-sple fuco o deec he wo shoulder jos he slhouee mage o oba he basc moo feaures cludg he elbow agles, moo parameers of he ad wo hads. he, Prmve-based Coupled Hdde Markov Model (PCHMM) s preseed for aural coex-depede aco recogo. Las, some comparso expermes show PCHMM s beer ha he radoal HMM ad coupled HMM. Keywords: aco recogo, prmve feaures, Coupled Hdde Markov Model. Iroduco I rece years, he research o ellge evrome coeco wh ubquous compug ad pervasve compug has araced more ad more aeo, such as EasyLvg(Mcrosof), Iellge Rooms(MI AI Lab) ad KdsRoom(MI Meda Lab). here arses he eed for he compuers o deec he subjec evrome, ad furher o recogze hm, udersad hs eo as well as behavors ad adap o hs habs, whch s ofe called "Lookg a People". As a resul, a umber of research areas should be volved, cludg deeco, expresso recogo, gesure recogo, huma rackg, pose esmao, body laguage udersadg, ec. he Smar classroom s a projec of ellge evromes for ele-educao. Almos all he prese ele-educao sysems requre he eachers o s dow fro of he vdeo camera, however he eacher s experece s of much dfferece from eachg a ordary classroom. he smar classroom s a vrual classroom, where here s a blackboard (meda board), sudes (sude cos ad sude vdeo) ad ohers, lke a real oe. Ad by he echology of huma compuer eraco ad augme realy, he eacher feel very comforable ad could also use speech, gesure, body laguage ad hadwrg o mprove effcecy as well. hs paper focuses o he eacher s upper-lmb aco recogo o udersad he eacher s eo for ellge camerama ad vrual mouse sysem. he framework of he recogo sysem s as followg: Recogo Coex Iformao Deeco Auo Camera Seleco If a froal uprgh exss 3D Moo Feaure Exracg Moo Recogo Fgure. Sysem framework I he smar classroom, here are a lo of cameras aroud he eacher. By he deeco module [3], he wo eghborg cameras ha he eacher are auo-seleced. herefore, oly he froal acos are cosdered hs paper. By recogo module[6], he eacher s dey s recogzed ad hs persoal formao could be rereved he daabase, whch s very mpora o ge more 3D moo feaure seco 2 ad adap hs hab furher applcao.. Relaed Work I mos prevous research o aco recogo, he acos are cofed o a predefed commad se, whch requres ha he subjecs are well raed ad he moos are uform. he feaures ad recogo algorhms are oally daa-drve whou ay Proceedgs of he Ffh IEEE Ieraoal Coferece o Auomac ad Gesure Recogo (FGR 02) /02 $ IEEE

2 formao of hgh-level feaure ad coex formao. For example, [7], mome-based feaures are exraced from mulple vews of moo eergy mages(mei) ad moo hsory mages(mhi), ad emplae machg algorhm are employed o recogze he aerobcs exercses ad he well performed moves he KdsRoom. Ad [2], wh he coordaes of he wo hads, Hdde Markov Models are used o recogze Amerca sg laguages. I [9], wh 3D rajecores of he wo hads, coupled Hdde Markov Models are preseed o recogze 3 kds of a Ch Ch ua. However, he eacher s aural aco recogo s much more dffcul ha he above. he moo s more aural, complex ad depede o he coex ad scearo. No subjec would be raed ad everyoe has hs ow hab. 2. Hybrd Huma Model I may aco recogo sysems, oly he rajecores of wo hads (or oly had) are exraced as moo feaures, hece much ambguy recogo. I coras, hs paper preses he hybrd huma model as he Fgure 2 o ge more explc 3D moo feaures. hs model cludes he, he ruk, wo s ad wo hads, whch are dspesable pars o ay perso ad are easly deeced, ad wh whch more formao ca be gve o reduce he complexy ad mprove he robusess grealy..2 Our approach I psychologcal research o aco recogo, was foud ha moo models subjecs md are o he moo parameers such as he parameers of poso ad moo speed, bu oly some characersc feaures, lke he relave poso of hads ad, he relao bewee he movg had ad he scearo, ec. Based o hs prcple, hs paper preses a ew sysem o recogze he eacher s aural complex aco he smar classroom. Ad much of s geeral ad ca be used oher areas. Frs, hs paper gves he hybrd huma model o oba he basc moo feaures cludg moo feaure of he, wo hads ad he wo elbow agles. he, a recogo algorhm, amed Prmve-based Coupled Hdde Markov Model (PCHMM), s preseed o recogze he subjec s aural complex aco. As a oally daa-drve algorhm, he radoal CHMM s o su o aural complex aco recogo because he feaure dmeso s oo large ad he wh-class scaer s oo much. Ad uforuaely, s mpossble o ge eough rag samples coag everyoe s every ype of aco. Ulke radoal CHMM, PCHMM s a approach wh hgh-level feaures ad coex formao, whch eed less rag samples ad could dmsh he wh-class scaer grealy. hs paper s arraged as follows: seco wo descrbes he hybrd huma model ad he basc moo feaure esmao; ad seco hree gves he PCHMM for upper-lmb aco recogo; seco four s he expermeal resuls ad las s he cocluso. Fgure 2. Hybrd huma model hs model s a hybrd from 2D model ad 3D model. he, wo hads are 2D ellpses 3D space ad he ruk s a 2D recagle 3D space. O he oher had, he upper ad lower s are 3D cylders. 2. Coex I hs paper, he coex s a geerc erm, cludg he eacher s persoal formao dexed by he recogo resul. hough oly a lle s useful hs paper, he eacher s model s very mpora for furher applcaos such as huma 3D modelg, more precse moo esmao, hab self-adapao, ec. he coex also cludes some scearo formao, such as he poso of some objecs o he desk. Ad wh he resul of prevous aco udersadg (akg objecs from he desk, pug back objecs, ec), he vara b objec he coex represes he objec he subjec s had. 2.2 Shoulder jos deeco Wh he eacher s slhouee mage obaed by backgroud subraco, hs paper gves a effecve approach o deec he wo shoulder jos. Frs, he ruk area s segmeed he slhouee mage as followg: Proceedgs of he Ffh IEEE Ieraoal Coferece o Auomac ad Gesure Recogo (FGR 02) /02 $ IEEE

3 ruk where he Fgure 3(a), = C( ) Slhouee () s he slhouee area as Slhouee s he area, s he wo area(cludg wo hads) ad C s a operaor o ge he maxmum coecve area. he hsogram of C( Slhouee - ) o he X coordae as Fgure 3(c) ca be looked as 5 les as Fgure 3(d). he areas covered by l ad l 5 are cosdered as he wo s area. By deleg he, he lef s ruk as Fgure 3(e). he l ( =,..., 5 ) s esmaed o sasfy parameers of he followg cosra: m s= 0 ( H ( s) F ( s) + H ( s) F ( s) ) (2) x x where s s arc legh ormalzed o 0,] H ( s) ( H x( s), H y( s)) F( s) ( F ( s), F ( s)) [, = s he hsogram fuco, = s he fuco composed of x y he 5 les. he equao (2) s a global opmzao problem wh hyper parameers. I hs paper, 2-order B-sple wh 5 segmes s employed o ge he soluo ad he resul s as he Fgure 3(d), whch s very precse, robus ad oly coss 2.56ms. Las, wo specal corer operaors are used o deeced he wo shoulder jos he ROI (rego of eres) whch s deermed by he poso ad he sze of he. he deeco of he shoulder jo s based he global formao. hough s 3D coordaes are very precse, hey are very robus. Whe a had s fro of he ruk, here may be some ambguy bewee l ad l 2 (or l 4, l 5 ). Bu hs case, l ad l 2 are very small he hsogram ad o maer how much he error of s, he devao of ruk wll be very lle. y y O Y l l2 ruk l3 l4 l5 X Shoulder Jos (d) F(x) (e) ruk 2.3 Basc moo feaures Wh he deeco of he, wo hads ad wo shoulder jos sereo mages, her 3D coordaes o could be obaed. he elbow jo θ ( 0 θ 90 ) are calculaed wh he egrao of shoulder jo 3D coordae ad he legh of he upper, lower s. I hs paper, he elbow agle s more mpora ha elbow jo 3D coordae. he reaso s: () Because huma 3D model buldg s mecosumg ad usable, s early mpossble o esmae he elbow jo 3D coordae precsely ad robusly whou ay markers. (2) For aco recogo, s ecessary o ge he elbow jo 3D coordaes because he poso of elbow jo s meagless mos huma moo. (3) he elbow agle s sgfca o represe he sae. herefore, he basc moo feaure for each had s obaed as followg: ( P had had, A had, P, θ elbow, b objec ) (3) where P had had, A had s he had 3D poso, 3D velocy ad 3D accelerao respecvely. Ad P s 3D poso ad 3D velocy, θelbow s he elbow agle ad b objec s a vara coex. From equao (3), he followg could be go: Vhad = P had Ahad = Vhad (4) V = P (5) Due o dffere eachers ad dffere acos, here s much wh-class scaer he 7-dmeso basc moo feaures. Ad could o be used drecly for aco recogo. Y ruk 3. Prmve-based Coupled Hdde Markov Model Fgure 3.(a) Slhouee (b) C( Slhouee - ) O (c) H(s) X hs paper roduces he prmve feaures o he radoal coupled-hmm ad calls Prmve-based Coupled Hdde Markov Model. 3. Prmve feaures Proceedgs of he Ffh IEEE Ieraoal Coferece o Auomac ad Gesure Recogo (FGR 02) /02 $ IEEE

4 he saes aco recogo ofe have defe meag ad much clear segmeao. hs s because each sae has some uambguous feaures, called prmve feaures or prmves hs paper. Each prmve feaure λ s represeed by a Gaussa Mxure Model (GMM) ad he dsrbuo desy s as he followg: G p λ ) = G ( p) ) (6) = where p s a prmve vara, G ( p) (=,, G ) s a Gaussa model wh meaξ ad covarace marx σ, whch are hyperparameers of he G s he Gaussa model umber, p () dsrbuo, s he wegh fuco for each Gaussa model. he parameers of G ( p) ad p () could be esmaed by Expecao Maxmum (EM) algorhm. 3.2 Represeao of he saes he saes hs paper s srcly defed by some prmve feaures ad correspodg wegh. hese prmve feaures of sae S are supposed o be depede of each oher. he observao deses fuco of S s as followg: O S) = P S) = p λ ) W ( ) (7) P = where O s he observao(basc moo feaure hs paper), P s he prmve feaure umber of S, λ s he h prmve feaure (=,, P ), W () s he correspodg wegh for λ, P p =, L, } s he prmve se of S. Ad he { p p relao of equao (7): p p ad O ca be descrbed as he f ( O, S, Coex) = (8) where Coex s he coex formao ad f s he fuco o exrac he p from O, S ad Coex. I hs paper, W () s esmaed by he followg: arg max O S) W (), W (2),..., W ( ) = (9) M = W ( ) = where W(), S are he same as equao(6), O s he h rag sample for sae S ad s he umber of he rag samples. Here, W () =,..., ) s esmaed by maxmum lkelhood. ( 3.3 Prmve-based Coupled Hdde Markov Model For each had, he basc moo feaure from me o s cosdered o be -order Markov cha. Ad suppose ha he relao bewee he wo hads sasfy PCHMM, as he Fgure 4. A P A + P + A +2 P +2 W W + W +2 L L + L +2 R R + R +2 W W + W +2 B P B + P + B +2 P +2 Fgure 4. PCHMM srucure. where he superscrp, +, + 2 he srucure mea a me, +, + 2 respecvely, L s he lef had sae a me, A s he basc moo feaure of he lef had a me, P s he prmve feaure of lef had moo, W s he wegh vecor for L, R s he rgh had sae a me, B s he basc moo feaure sequece of he rgh had a me, P s he prmve feaure of he rgh had moo, W s he wegh vecor for R. L = { L, =,..., } s he lef had sae sequece, A = { A, =,..., } s he basc moo feaure sequece of he lef had. R = { R, =,..., } s he rgh had sae sequece, B = { B, =,..., } s he basc moo feaure sequece of he rgh had. he lkelhood fuco of PCHMM wh he basc moo feaure A ad B s: A, B Θ) = L ) * R ) * A L ) A L ) * B R ) (0) * B R ) * * A L, R ) * = 2 B L, R ) Proceedgs of he Ffh IEEE Ieraoal Coferece o Auomac ad Gesure Recogo (FGR 02) /02 $ IEEE

5 where he PCHMM parameer se Θ coas he pror probables P ( L ) ad P ( R ) for he wo Markov chas, he observao deses fuco A L ) ad P ( B R ), he raso probables L L, R ) ad P ( R L, R ). he pror probables P ( L ) ad P ( R ) are supposed o be equal for each model. Wh forward-backward Verb algorhm, he parameers se Θ of he model ca be esmaed as followg: Θ S arg max A, B Θ) () where ( A, B = s s he sequece umber for rag, ) s he h rag sequece. 4. Expermeal Resuls For he eacher he smar classroom, here are oally 7 kds of aural acos o be recogzed: akg objecs from he desk Pug back objecs Pog o he sudes Pog o he blackboard (vrual mouse) Commucao wh he sudes Explag objecs Drkg waer For each aco, here are 50 samples. Comparso expermes are doe amog HMM, radoal CHMM ad PCHMM ad he resul s as followg: Fgure 6. he comparso o he elbow agles where Wh meas moo feaures clude he elbow agles ad Whou meas he reverse. he Fgure 6 shows he basc moo feaures wh he elbow agles perform much beer ha he oe whou hem. hough he elbow agles are o very precse, hey are very mpora feaures o represe he sae. A he same me, shows he ad wo hads moo feaures are suffce for complex aco recogo. 5. Cocluso hs paper preses a framework for he eacher s complex aco recogo he smar classroom. Wh he Hybrd Huma Model, basc moo feaure are exraced whch cludes he wo elbow agles ad he moo feaures of he head ad wo hads. Prmve-based Coupled-HMM are used for recogo. Ad he ecouragg experme resul show he PCHMM s very robus ad ca oba beer resul especally he case of oly less rag samples. Fally, he prooype of he smar classroom for ele-educao s as followg: Fgure 5. he comparso amog HMM, radoal CHMM ad PCHMM where he x coordae s he sze of he rag daa se ad all he esg ses are he whole daa se. I shows PCHMM s he bes amog he hree algorhms, especally wh less rag daa. Aoher comparso expermes o he elbow agles are carred ou. Ad he resul s as followg: Fgure 7.he prooype of smar classroom. I hs prooype, he framework of aural complex aco recogo performs he smar camerama ad vrual mouse very robus ad accuraely. Proceedgs of he Ffh IEEE Ieraoal Coferece o Auomac ad Gesure Recogo (FGR 02) /02 $ IEEE

6 5. Refereces [] G. Johasso. Vsual percepo of bologcal moo ad a model for s aalyss. Percepo ad Psychophyscs, 4(2):20-2,973. [2] Sarer, Weaver J, Pelad. A real-me Amerca sg laguage recogo usg desk ad wearable compuer based vdeo. IEEE rasaco o Paer Aalyss ad Mache Iellgece [J], 998, 20(2): [3] Hazhou A, Luhog Lag, Guagyou Xu, deeco emplae machg cosraed subspace. I Proceedgs, Eded by H.R. Araba, Ieraoal Coferece o Arfcal Iellgece 200 (IC-AI 200), Vol.II, pp , Las Vegas, Nevada, USA, Jue 25-28, 200 [4] Rezek, L., Sykacek, P., Robers, S.J. Coupled hdde Markov models for bosgal eraco modelg. Advaces Medcal Sgal ad Iformao Processg, Frs Ieraoal Coferece o (IEE Cof. Publ. No.476), 2000 Page(s): [5] Mahew Brad, Nura Olver ad Alex Pelad. Coupled hdde markov models for complex aco recogo. Proceedgs of IEEE Coferece o Compuer Vso ad Paer Recogo, 997: [6] Z.Y.Peg, W.Hog, L.H.Lag, G.Y.Xu ad H.J.Zhag. Deecg facal feaures o mage sequeces usg cross-verfcao mechasm. Proceedg of he Secod IEEE Pacfc-Rm Coferece o Mulmeda, 200, pp [7] James W. Davs, Aaro F. Bobc. he Represeao ad recogo of aco usg emporal emplaes. Proceedg of he Ieraoal Coferece o Compuer Vso ad Paer Recogo, 997, 928~934 [8] Chrsopher R. Wre, Bra P. Clarkso, Alex Pelad. Udersadg purposeful huma moo. Proceedg of he Fourh IEEE Ieraoal Coferece o Auomac ad Gesure Recogo, 2000,pp: [9] Mahew Brad, Nura Olver, Alex Pelad. Coupled hdde markov models for complex aco recogo. Proceedg of IEEE Socey Coferece o Compuer Vso ad Paer Recogo, 997, pp: Proceedgs of he Ffh IEEE Ieraoal Coferece o Auomac ad Gesure Recogo (FGR 02) /02 $ IEEE