Motion Feature Extraction Scheme for Content-based Video Retrieval

Transcription

1 oon Feure Exrcon Scheme for Conen-bsed Vdeo Rerevl Chun Wu *, Yuwen He, L Zho, Yuzhuo Zhong Deprmen of Compuer Scence nd Technology, Tsnghu Unversy, Bejng , Chn ABSTRACT Ths pper proposes he exrcon scheme of globl moon nd objec rjecory n vdeo sho for conen-bsed vdeo rerevl. oon s he ey feure represenng emporl nformon of vdeos. And s more objecve nd conssen compred o oher feures such s color, exure, ec. Effcen moon feure exrcon s n mporn sep for conen-bsed vdeo rerevl. Some pproches hve been en o exrc cmer moon nd moon cvy n vdeo sequences. When delng wh he problem of objec rcng, lgorhms re lwys proposed on he bss of nown objec regon n he frmes. In hs pper, whole pcure of he moon nformon n he vdeo sho hs been cheved hrough nlyzng moon of bcground nd foreground respecvely nd uomclly. 6-prmeer ffne model s ulzed s he moon model of bcground moon, nd fs nd robus globl moon esmon lgorhm s developed o esme he prmeers of he moon model. The objec regon s obned by mens of globl moon compenson beween wo consecuve frmes. Then he cener of objec regon s clculed nd rced o ge he objec moon rjecory n he vdeo sequence. Globl moon nd objec rjecory re descrbed wh PEG-7 prmerc moon nd moon rjecory descrpors nd vld smlr mesures re defned for he wo descrpors. Expermenl resuls ndce h our proposed scheme s relble nd effcen. Keywords: Vdeo rerevl, moon feure exrcon, PEG-7, moon esmon, objec rcng 1. INTRODUCTION Effcen nd quc rerevl n lrge-scle mulmed dbse s one emergen problem for conen-bsed mulmed rerevl pplcons nowdys. A sndrd descrpon of he med conen s one mporn requremen for conen reuse. The correspondng sndrd nmed s PEG-7 hs been wored on by ISO SC9 WG11 group, whch wll provde sndrd descrpon for mulmed conen d. The descrpon, long wh he mulmed conen, suppors quc rerevl nd esy ccess o he conen n whch he user s neresed. The mn elemens of PEG-7 sndrd re Descrpors ( D ), Descrpon Schemes ( DS ), Descrpon Defnon Lnguge ( DDL ) nd Sysem Tools. PEG-7 defnes color, exure, shpe nd moon descrpors for descrpon of low level feures. Whle he defnon of descrpors s specfed whn he scope of PEG-7, he exrcon of feures nd he serch engne re no nsde he scope of PEG-7. oon feure s sgnfcn for he descrpon of vdeo conen. Vdeo rerevl bsed on moon feures s one mporn pr of rerevl pplcons n vdeo dbse. For nsnce, when browsng he vdeo obned by survellnce sysem or wchng spors progrms, he user lwys hs he need o fnd ou he objec movng n some specl drecon. In our mul-feure PEG-7 vdeo rerevl projec, he pr of moon feure * Correspondence: eml: wuchun00@mls.snghu.edu.cn; phone: ; fx: Sorge nd Rerevl for ed Dbses 00, nerv. Yeung, Chung-Sheng L, Rner W. Lenhr, Edors, Proceedngs of SPIE Vol (00) 00 SPIE X/0/$15.00

2 exrcon, descrpon nd rerevl s of much mpornce. Auomc exrcon lgorhms of moon descrpors wll be much useful. Some reled wor hs been done n he spec of exrcon of moon descrpors. Jennn e. l. [1] proposed her lgorhms for exrcon of cmer moon descrpor nd moon rjecory descrpor. In her lgorhm, he exrcon of moon rjecory descrpor ws bsed on he ssumpon h he objec ws lredy segmened correcly nd hey ddn del wh he problem of objec segmenon. Kng e. l. [] proposed her lgorhm on compressed domn d, nd only dd her wor on cmer moon nlyss. Dvrn e. l. [3] focused on moon cvy descrpor, whch descrbed he cvy n vdeo sequence n whole. However, more ccure nd complee moon nformon wll be obned f foreground re cn be segmened uomclly from he frmes. Becuse n mny suons moon of bcground nd moon of foreground represen dfferen semnc nformon. Bsed on he nformon, some useful nd hgh-level nformon cn be undersood of he vdeo conen. Then conen-bsed rerevl wll become more effcen. In hs pper, we ry o exrc more hgh-level nformon n hs wy. We furher nvesge lgorhms o uomclly exrc globl moon prmeers nd moon rjecory prmeers from compressed domn, whch re effecve for he descrpon of bcground nd foreground moon respecvely. The exrcon lgorhm of prmerc moon descrpor s bsed on our former globl moon esmon lgorhm proposed for spre codng used n PEG-4 [4]. Bsed on he shos segmened correcly, we esme he globl moon wh 6- prmeer ffne model. We propose he lgorhm of globl moon compenson o exclude he bcground from frme nd ge he regon of he objec. Then we rc he objec uomclly nd cree moon rjecory descrpor. Smlry mesures of he wo descrpors re defned for rerevl. In order o es he effec of proposed lgorhms, some expermens re mde bsed on moon-bsed vdeo rerevl sysem. The res of hs pper s orgnzed s follows. In secon, we descrbe he uomc exrcon lgorhms of he wo moon feures. In secon 3, smlry mesures re defned. Expermenl resuls re presened n secon 4. Conclusons re gven n secon 5.. OTION FEATURE EXTRACTION ALGORITH.1 EXTRACTION OF PARAETRIC OTION In PEG-7 [6], prmerc moon descrpor s defned whch represens he globl moon nformon n vdeo sequences wh D moon models. Globl moon s he movemen of bcground n frme sequence nd s mnly cused by cmer moon. Globl moon nformon represens he emporl relons n vdeo sequences. Compred wh oher vdeo feures, cn represen he hgh-level semnc nformon beer. And s mporn for moonbsed objec segmenon, objec rcng, moscng, ec. oon-bsed vdeo rerevl cn be mplemened by prmerc moon descrpor on he bss of pproprely defned smlry mesure beween moon models. [Exrcon] In lrge-scle vdeo dbses, he vdeos re mosly PEG-1, PEG- compressed vdeo srems. We esme he globl moon of vdeo shos usng DC mges obned by prly decodng he compressed vdeo srem. A DC mge s composed of he DC coeffcens cqured by DCT rnsform on 8 8 blocs n frme. I s mnure of 1/8 1/8 Proc. SPIE Vol

3 he sze of orgnl frme nd represens he color dsrbuon of he orgnl pcure. So we use DC mges for wo resons: 1) I cn sve much of he me en by complee decodng of he compressed vdeo srems nd processng full mge. ) D n DC mge s he verge of ech bloc, so hs he effec of smoohng, depressng he nfluence of noses n frme. In he descrpon of our globl moon esmon lgorhm herenfer, mge represens he DC mge of frme, pxel represens pon n DC mge, nd he pxel s poson s represened by he poson of he pxel he op lef corner of he 8 8 bloc n he orgnl frme. We e sx-prmeer ffne model s he model of our globl moon esmon. We use [ x ] Τ, y o represen he poson of he pxel n curren mge.[ x beween hem cn be show s Equon 1: ] Τ 1, y 1 s he correspondng poson n he prevous mge. The relons x y 1 1 = = x dx + by + ey + + c f. (1) Le ( x, y) I represen curren mge nd I' ( x', y' ) represen he prevous mge. = (, b, c, d, e, f ) Τ s he prmeer vecor. We defne energy funcon s R ' ' ' ( ) = w[ Ι ( x 1, y 1) Ι( x, y )]. () x, y The gol of globl moon esmon s o ge he opmum prmeer vecor whch mnmzes R ( ).Guse- Newon nd Levenberg-rqurde erve mehods cn be used o solve such opmum problems. When we ge he prmeer vecor sep, we expnd he energy funcon ( ) R n Tlor seres nd drop he secondorder erms: Τ 1 Τ R( ) R( ) + g ( ) + ( ) Η ( ). (3) In he bove equon, g s grden mrx nd Η s Hessn mrx. They re defned s follows: g Η = J WJ Τ Τ = J W γ, + γ w H. (4) 98 Proc. SPIE Vol. 4676

4 Τ γ = [ r1 r LrN ] represens he resduls. J = γ.the wegh mrx W s dgonl mrx. H s he Hessn mrx of γ.if γ s smll, we cn ge he pproxmon: Η WJ. (5) J Τ Le R ( ) = 0,henwege J Τ WJ ( ) = J Wγ. (6) Τ We cn ge he ncremen of from he bove equon. Then he prmeer vecor nex sep cn be go: = By hs erve clculon, we cn fnlly ge n opmum prmeer vecor. In our lgorhm, we clcule one prmeer vecor for every pr of djcen DC mges n one vdeo sho. Snce mchng by he 6 prmeers n he globl moon model drecly does no hve much menng n rerevl pplcon, we nsed clcule he horzonl rnslon velocy T x, vercl rnslon velocy T y, ngle of roon nd scle s of globl moon from every prmeer vecor of he globl moon model. We rnsform he sxprmeer ffne model n Equon 1 no he followng: x y 1 1 = sr x y + T T x y. (7) In Equon 7, s s he scle. T x s he horzonl rnslon velocy. T y s he vercl rnslon velocy. And R s he roon mrx, defned s cos sn R = [ ]. (8) sn cos n whch s he ngle of roon. We hen clcule he new rnsformed prmeer vecor Equon 7 from (, b, c, d, e, f ) Τ n Equon 1. T ( T x, Ty, s, ) n We selec he hsogrm spce of globl moon prmeers s he feure spce nd cluser he feure pons n. We use he prmeer vecor of he pon he cener of one cluser s he cluser s represenon. Thus we ge se of prmeer vecors s represenons for hs sho. We descrbe hs se of prmeer vecors wh PEG-7 prmerc Proc. SPIE Vol

5 moon descrpor. They form he descrpon of he sho s globl moon. In our mplemenon, he prmerc descrpon of one vdeo sho conns 3-4 prmeer vecors.. EXTRACTION OF OTION TRAJECTORY PEG-7 s moon rjecory descrpor ncludes ls of eypons nd se of nerpolng funcons h descrbe he rjecory of objec beween wo eypons. The nerpolng funcon used s frs order nerpolng or second order nerpolng funcon. The exrcon of moon rjecory s sgnfcn for objec-bsed vdeo rerevl. In gven conex wh cern pror nowledge, cn be much useful n mny pplcons. For exmple, he deecon of n objec wh dngerous movng rjecory n survellnce sysem, he rerevl of specl con n spors gme, ec. [Exrcon] Our lgorhm here s lso mplemened on DC mges nd bsed on he ssumpon h he shos re segmened correcly. Before rcng of objec n sho, we should frs deec he objec regon. Afer we ge globl moon prmeers of he bcground, he bcground cn be excluded from he mge by globl moon compenson. In he dfferen mp beween ech pr of frmes, objec regon re lef wh hgh remnder vlue. Thus we ge he rough regon of he objec. The objec regon cn be furher refned wh morphologcl mge processng mehods. We hen projec he dfferences n dfference mp long he horzonl xs nd he vercl xs. We ge wo 1-D hsogrms long he wo xes. A smple cse wh one movng objec n he vdeo sequence s llusred n Fgure 1. Hs(y) y Hs(x) x Fgure 1: Dfference Projecon Along x nd y Axes We clcule he poson of he movng objec n frme wh he sscl resuls n ech hsogrm. We compue he men of he smple dsrbuon n he hsogrm long x xs s he x coordne of he cener of he movng objec, nd he men of smple dsrbuon n he hsogrm long y xs s he y coordne. When rcng he objec, we clcule he cener of he objec regon wh bove lgorhm for ech frme. We hen selec he se of eypons o be descrbed wh PEG-7 moon rjecory descrpor from he se of ceners. The rjecory we exrc s -D rjecory. And he lgorhm s ppled o ech dmenson (x nd y) respecvely. We begn wh he nervl connng he frs hree ceners n he frme sequence. When curren nervl conns N 300 Proc. SPIE Vol. 4676

6 ceners, we clcule second order nerpolng funcon s n Equon 9 o pproxme objec rjecory on hs nervl. f 1 () f + v ( ) + ) =. (9) ( Then he nex cener s en no hs nervl. We eep he nerpolng funcon s he sme nd clcule he pproxmon error. If he error s smller hn he hreshold, he new cener s dded o hs nervl nd he process repes. Oherwse, he N cener nervl s ep nd he frs nd ls ceners re en s he eypons o be descrbed n PEG-7 moon rjecory descrpor, ogeher wh he second order nerpolng funcon. The process hen srs gn wh he followng hree ceners. Fnlly, we cn ge he moon rjecory descrpor of he sho. 3. SIILARITY ATCHING ETHODS We mplemen nd es our lgorhms wh query-by-exmple mode. We exrc he descrpors of he exmple seleced by he user, nd hen mch hem wh descrpors of shos n he dbse. For prmerc moon descrpor, we defne smlry mesure on he bss of four prmeers T ( T x, Ty, s, ), clculed from he 6 prmeers of ffne moon model obned by globl moon esmon. The smlry mesure s defned s follows, W ( sho1, = X X ( sho1, + WY Y ( sho1, + W W + W + W + W X Y ( sho1, + WS S( sho1, sho ) S. (10) X Y sho1, = ( T T (. (11) X1 X ) sho1, = ( T T (. (1) Y1 Y ) ( sho1, = ( 1 ). (13) S sho1, = ( S S (. (14) 1 ) = 0,1,..., m. m s he number of represenve globl moon prmeer vecors for he sho. W X WY, W, WS, re he weghs beween 0 nd 1, seleced ccordng o dfferen pplcons. In our mplemenon, he weghs cn be djused by users n he query nerfce. For exmple, f he user cres more for queryng vdeo shos wh domnn Proc. SPIE Vol

7 horzonl moon hn oher moon, he cn gve WX lrger wegh pproxmng 1 whle gvng W Y, W, WS he weghs ner 0. We defne he followng smlry mesure for moon rjecory descrpor: Wp p ( sho1, + Ws s ( sho1, + W ( sho1, ( sho1, =. (15) W + W + W p s ( sho1, = (( x x ) + ( y y ) ) p ( sho1, = (( v v ) + ( v v ) ) s 1 1. (16) x1 x y1 y. (17) ( sho1, = (( ) + ( ) ) x1 x y1 y. (18) = 0,1,...,n. n s he number of eypons chosen o descrbe he rjecory. W p Ws, W, re he weghs beween 0nd1. p s he Eucldn mesure beween poson vecors of he eypons n he moon rjecory descrpor. s s he Eucldn mesure of velocy vecors of eypons. And s h of cceleron vecors. The smlry mesures of prmerc moon descrpor nd moon rjecory descrpor cn be combned o form one smlry mesure. We frs normlze ech smlry mesure o he sme dynmc rnge of 0 o 1. Then we clcule he lner combnon of he wo smlry mesures s our fnl smlry mesure. S = α +. (19) 1 1 α α re he weghs beween 0 nd 1., 1 1,α re clculed s n Equon 10 nd 15 respecvely. 4. EXPERIENTAL RESULTS Our expermens re mde on moon-bsed vdeo rerevl sysem s shown n Fgure. In he user nerfce, users cn choose o rereve vdeo shos wh prmerc moon feure or objec rjecory feure. Or hey cn use boh feures he sme me. The users cn lso djus he weghs n smlry mesures ccordng o her own needs. 30 Proc. SPIE Vol. 4676

8 Fgure : A Snpsho of oon-bsed Vdeo Rerevl Sysem Inerfce In our expermens o es he effec of our moon descrpor exrcon lgorhm, we consruc sho dbse of 358 shos from PEG-7 es d se. The shos re en from 40-mnue vdeo sequences, ncludng bsebll, golf, nmls, news, ec. In order o es he effcency of prmerc moon feure exrcon scheme, we choose dfferen query shos represenve of he followng cmer moon respecvely, pnnng, rcng, boomng, zoomng nd slgh moon. In order o es he effcency of objec moon rjecory exrcon scheme, we choose dfferen query shos wh cern objec movng rjecory. Queres re lso mde by combnng he wo descrpors. The rereved shos re rned ccordng o her smlry o he query shos. Prs of he expermenl resuls re llusred n Fgure 3, Fgure 4 nd Fgure 5. We dsply he bes 5 mches for ech query sho. In Fgure 3, we query shos of rcng lef. Query Sho Fve Bes ches Fgure 3: An Exmple of Trcng Query In Fgure 4, he mn cmer moon n he query sho s zoomng. Query Sho Fve Bes ches Fgure 4: An Exmple of Zoomng Query Proc. SPIE Vol

9 In Fgure 5, he objec n he query sho s movng rgh. Query Sho Fve Bes ches Fgure 5: An Exmple of ovng Rgh Query The query expermens gve good resuls. So n generl, our prmerc moon descrpor nd moon rjecory descrpor exrcon lgorhms could wor very well. 5. CONCLUSIONS AND FUTURE WORK Expermenl resuls ndce h our lgorhm s effecve. In hs pper, we focus our reserch on moon nlyss, whch s mos conssen feure mong ll low-level feures for vdeos. The moon rerevl sysem proposed n he pper s prccl emp n moon-bsed vdeo rerevl. And moon of bcground nd foreground cn be exrced wh our lgorhm. We cn furher e some mehods o ccelere he convergence of our moon esmon lgorhms. In our objec s rcng lgorhm, we ge he regon of objec moon by wpng off he bcground n frme. Algorhm o uomclly obn he precse conour of he objec s o be developed n he fuure. We wll connue our reserch n hese specs. Our rerevl s mplemened by query-by-exmple nd low-level moon feures. For prccl vdeo rerevl sysem, should suppor mul-modl nd mul-feure rerevl. Thus ll hese wor wll be exended o develop rerevl pplcons bsed on hgh-level semnc query combned wh mulple feures. REFERENCES 1. S. Jennn, B. ory, Vdeo oon Represenon for Improved Conen Access, IEEE Trnscon on Consumer Elecroncs, Vol. 46, No. 3, pp , Augus 000. H.B. Kng, Spo-Temporl Feure Exrcon from Compressed Vdeo D, TENCON 99, Proceedngs of he IEEE Regon 10 Conference, Vol., pp , A. Dvrn, A. Vero, Vdeo Browsng Sysem Bsed on Compressed Domn Feure Exrcon, IEEE Trnscon on Consumer Elecroncs, Vol. 46, No. 3, pp , Augus Y.W. He, Y.Z. Zhong, S.Q. Yng, Fs Approch of Spre Codng for Vdeo Conen, proceedngs of he SPIE Inernonl Symposum on Informon Technologes 000(ISIT 000), Bosonsschuses,USA,November Y.W. He, L. Zho, S.Q. Yng, Y.Z. Zhong, Regon-bsed Trcng n Vdeo Sequences Usng Plnr Perspecve odels, proceedngs of he 3rd Inernonl Conference on ulmodl Inerfces(ICI 000), Bejng, Ocober Y.W. He, Globl oon Esmon Algorhm nd Is Applcon n Vdeo Codng, ser s Thess, Tsnghu Unversy, P. R. Chn, Ocober Proc. SPIE Vol. 4676

10 7. T. Vlchos, Smple ehod for Esmon of Globl oon Prmeers Usng Sprse Trnslonl oon Vecor Felds, Elecroncs Leers, Vol. 34, No. 1, pp. 60-6, Jnury ISO/IEC JTC1/SC9/WG11 N375, Overvew of he PEG-7 Sndrd, L Bule, Ocober ISO/IEC JTC1/SC9/WG11/N3705, Tex of ISO/IEC /CD Informon Technology-ulmed Conen Descrpon Inerfce-Pr 5 ulmed Descrpon Schemes, L Bule, Ocober 000 Proc. SPIE Vol