An Integrated and Interactive Video Retrieval Framework with Hierarchical Learning Models and Semantic Clustering Strategy

Transcription

1 An Inegraed and Ineracve Vdeo Rereval Framewor wh Herarchcal Learnng Models and Semanc Cluserng Sraegy Na Zhao, Shu-Chng Chen, Me-Lng Shyu 2, Suar H. Rubn 3 Dsrbued Mulmeda Informaon Sysem Laboraory School of Compung and Informaon Scences Florda Inernaonal Unversy, Mam, FL 3399, USA 2 Deparmen of Elecrcal and Compuer Engneerng Unversy of Mam, Coral Gables, FL 3324, USA 3 Space and Naval Warfare Sysems Cener (SSC), San Dego, CA , USA {nzhao002, chens}@cs.fu.edu, 2 shyu@mam.edu, 3 suar.rubn@navy.ml Absrac In hs research, we propose an negraed and neracve framewor o manage and rereve large scale vdeo archves. The vdeo daa are modeled by a herarchcal learnng mechansm called HMMM (Herarchcal Marov Model Medaor) and ndexed by an nnovave semanc vdeo daabase cluserng sraegy. The cumulaed user feedbacs are reused o updae he affny relaonshps of he vdeo obecs as well as her nal sae probables. Correspondngly, boh he hgh level semancs and user percepons are employed n he vdeo cluserng sraegy. The clusered vdeo daabase s capable of provdng appealng mulmeda experence o he users because he modeled mulmeda daabase sysem can learn he user s preferences and neress neracvely.. Inroducon and Relaed Wor Wh he recen advances n mulmeda echnologes, he number of mulmeda fles and archves ncrease dramacally. Therefore, becomes an mporan research opc o mne and cluser he mulmeda daa, especally o accommodae he requremens of vdeo rereval n a dsrbued envronmen. Snce he mulmeda daabases may be dsrbued geographcally hrough he local newor or world-wde Inerne, he assocaed worloads could be que expensve when dealng wh complcaed vdeo queres. In parcular, semanc based vdeo rereval s mul-dscplnary and nvolves he negraon of vsual/audo feaures, emporal/spaal relaonshps, semanc evens/even paerns, hgh-level user percepons, ec. Therefore, s expeced o ulze a concepual daabase cluserng echnque o ndex and manage he mulmeda daabases such ha he relaed daa can be rereved ogeher and furhermore he communcaon coss n he query processng can be sgnfcanly reduced /06/$ IEEE. 438 Currenly, here exs approaches focusng on he cluserng echnques for he vdeo daa. For example, a herarchcal cluserng mehod for spors vdeo was presened []. Two levels of clusers are consruced where he op level s clusered by he color feaure and he boom level s clusered by he moon vecors. [2] descrbes a specral cluserng mehod o group vdeo shos no scenes based on her vsual smlary and emporal relaonshps. In [5], he algorhms are proposed for unsupervsed dscovery of he vdeo srucure by modelng he evens and her sochasc srucures n vdeo sequences va usng Herarchcal Hdden Marov Models (HHMM). Based on our bes nowledge, mos of he exsng researches produce he clusers manly on low-level and/or md-level feaures, and do no consder hgh-level conceps or user percepons n he cluserng procedure. Ths brngs he problem of semanc gap. Relevance feedbac s an effecve mehod o narrow down hs semanc gap. However, mos of he exsng relevance feedbac sysems are only capable of provdng real-me updaes on he rereval resuls whou any furher mprovemen of he overall sysem performance. In addon, mulmeda daabases may no be effcenly modeled n hese approaches even afer he cluserng echnque s appled. In hs paper, an negraed and neracve vdeo rereval framewor s proposed o effcenly organze, model, and rereve he conen of a large scale mulmeda daabase. The core of our proposed framewor s a learnng mechansm called HMMM (Herarchcal Marov Model Medaor) [6] and an nnovave vdeo cluserng sraegy. HMMM models he vdeo daabase; whle he cluserng sraegy groups vdeo daa wh smlar characerscs no clusers ha exhb ceran hgh level semancs. The HMMM srucure s hen exended by addng an addonal level o represen he clusers and her relaonshps. The proposed framewor s desgned o accommodae advanced queres va consderng he hgh level semanc meanng. Frs of all, s capable of searchng semanc

2 evens or even paerns consderng her populary by evaluang her access frequences n he large amoun of hsorcal queres. Second, he users can choose one or more example paerns wh her ancpaed feaures from he nal rereved resuls, and hen ssue he nex round of query. I can search and re-ran he canddae paerns whch nvolve he smlar aspecs wh he posve examples reflecng he user s neress. Thrd, vdeo cluserng can be conduced o furher reduce he searchng me especally when dealng wh he op- smlary rerevals. As he HMMM mechansm helps o raverse he mos opmzed pah o perform he rereval, he proposed framewor can only search several clusers for he canddae resuls whou raversng all he pahs o chec he whole daabase. Ths paper s organzed as follows: Secon 2 presens he overall framewor of our proposed research. In Secon 3, he dealed echnques are furher expanded by nroducng HMMM model and explanng he cluserng sraegy. Moreover, he rereval algorhm and example are also ncluded. Secon 4 analyzes he expermenal resuls. Fnally, conclusons are summarzed n Secon Overall Framewor Fgure demonsraes he overall worflow of he proposed framewor. In hs framewor, he soccer vdeos are frs segmened no dsnc vdeo shos and her lowlevel vdeo/audo feaures are exraced. A mulmeda daa mnng approach s ulzed o pre-process he vdeo shos o ge an nal canddae pool for he poenal mporan evens. Afer ha, a se of nal even labels wll be gven o some of he shos, where no all of hese labels are correc. All of hese daa and nformaon wll be fed no hs framewor for even paern searchng and vdeo rereval purposes. The vdeos ncluded n he canddae pool are modeled n he s level of MMM (Marov Model Medaor) models, whereas he vdeos are modeled n he 2 nd level. Afer nalzng he s level and 2 nd level of MMM models, he users are allowed o ssue he even or even paern queres. Furhermore, he users can selec her neresed even paerns n he nal resuls and re-ssue he query o refne he rereval resuls and her ranngs. Ths sep s also recognzed as onlne learnng. These user seleced sho sequences are sored as posve paerns for he fuure offlne ranng. Afer a ceran amoun of queres and feedbacs, he proposed framewor s able o perform he offlne ranng. The hsorcal queres and user access records are ulzed o updae he affny relaonshps of he vdeos/vdeo shos as well as her nal sae probables. Thereafer, boh he semanc evens and he hgh level user percepons are employed o consruc he vdeo clusers, whch are hen modeled by a hgher level (3 rd level) of he MMM model. In he meanwhle, he 2 nd level MMM model are dvded no a se of sub-models based on he clusered vdeo groups. The clusered daabase and he updaed HMMM mechansm are capable of provdng appealng mulmeda experence o he users because he modeled mulmeda daabase sysem learns he user s preferences and neress neracvely va reusng he hsorcal queres. Vdeo Sho Segmenaon Sho Feaure Exracon Daa Cleanng Even Deecon Even/Paern Queres Feedbac Accumulae Offlne Tranng Vdeo Cluserng 3. Vdeo Daabase Cluserng 3.. Vdeo Daabase Modelng Inalze he s level and 2 nd level MMM models Updae he s level and 2 nd level MMM models Consruc he 3 rd level MMM models and Updae he 2 nd level MMM models Provde enhanced performance for he Top- even paern rereval Fgure. Overall worflow for he proposed approach In our prevous sudes, we have successfully appled he MMM (Marov Model Medaor) model n mage daabase cluserng [4], and expanded MMM o HMMM [6] o model a vdeo daabase whch s formalzed as an 8- uple λ = ( d, S, F, A, B, Π, O, L). Le n denoe he level number, where n d and d s he number of levels n an HMMM. The n h level of HMMM may conan one or more MMM models o represen he ses of dsnc mulmeda obecs and her assocae feaures. In [6], d s se as 2. S n he lowes-level MMM represens he se of vdeo shos, and he feaure se F consss of vsual/audo feaures. Whle n he 2 nd level MMM, S 2 descrbes he se of vdeos n he daabase, and F 2 conans he semanc evens deeced n he vdeo collecon. Each of he MMM models ncorporaes a se of marces for affny relaonshps (A n ), feaure values (B n ), and nal sae probably dsrbuons (Π n ). In addon, O and L are desgned for he relaonshp descrpon beween wo adacen levels. O (O,2 ) ncludes he weghs of mporance for he low-level feaures (F ) when descrbng he hgh-level semanc evens (F 2 ). L (L,2 ) descrbes he ln condons beween he vdeos (S 2 ) and he vdeo shos (S ). HMMM model carres ou a sochasc and 439

3 dynamc process n boh search and smlary calculaon, where always res o raverse he pah wh he larges possbly. Therefore, can asss n rerevng more accurae paerns qucly wh lower compuaonal coss. The specfc desgn of HMMM helps no only he general even queres, bu also he rereval of emporal based semanc even paerns. Anoher sgnfcan advanage of HMMM s s neracve feedbac and learnng sraeges, whch can profcenly assure he connuous mprovemens of he overall performance. The users are capable of provdng her own feedbacs such ha he sysem can be raned eher onlne or offlne. The onlne learnng mechansm creaes an ndvdual MMM nsance by usng he vdeo shos ha a user prefers. All of he users feedbacs are effcenly accumulaed and ready for he offlne sysem ranng process. In hs research, he large amoun of user feedbacs wll be reused n vdeo daabase cluserng o furher mprove he overall rereval performance and reduce he searchng space and me Concepual Vdeo Cluserng Smlary Measuremen In hs proposed framewor, a vdeo s reaed as an ndvdual daabase n a dsrbued mulmeda daabase sysem, where s vdeo shos are he daa nsances n he daabase. Accordngly, a smlary measure beween wo vdeos s defned as a value ndcang he leness of hese wo vdeos wh respec o her concepual conens. I s calculaed by evaluang her posve evens and even paerns n he hsorcal queres. Tha s, f wo vdeos conss of he same even(s) and/or even paern(s) and are accessed ogeher frequenly, s consdered ha hese wo vdeos are closely relaed and her smlary score should be hgh. Assume here are H user queres ssued hrough he vdeo rereval framewor, where he se of all he query paerns s denoed as QS. In order o refne her rereved resuls n real-me, he users mar her preferred even paerns as posve before mang he nex query. By evaluang he ssued query ses and her assocaed posve paerns, he smlary measure s defned as follows. Le v and v be wo vdeos, and X={x,, x m } and Y={y,, y n } be he ses of vdeo shos belongng o v and v ( X v, Y v ), where m and n are he numbers of annoaed vdeo shos n v and v. Denoe a query wh an observaon sequence (semanc even paern) wh C semanc evens as Q = e, e,..., e }, where Q QS. Le R be he se of { 2 C G posve paerns ha a user seleced from he nal rereval resuls for query Q. Ths can be represened by a marx of sze G C, G, C. As shown n Equaon (), each row of R represens an even sho sequence ha he user mared as posve, and each column ncludes he canddae even shos whch correspond o he requesed even n he query paern. { s, s2,..., sc } { s, s2,..., sc } R =. ()... G G G { s, s2,..., sc } Based on he above assumpons, he vdeo smlary funcon s defned as below. Defnon : SV(v, v ), he smlary measure beween wo vdeos, s defned by evaluang he probables of fndng he same even paern Q from v and v n he same query for all he query paerns n QS. ( v ) P( Q v ) FA( ) SV( v, v ) = P Q H, (2) Q QS where H, and FA (H ) s an adusng facor. P ( Q ) v and ( Q ) v P represen he occurrence probables of fndng Q from v and v, where he occurrence probably can be obaned by summng he on probables over all he possble saes [3]. In order o calculae hs value, we need o selec all he subses wh C even shos from he posve paern se R, whch also belong o v or v. Tha s, X = { x, x,..., x } and Y = { y, y,..., y }, where X X, 2 C 2 C, Y Y, X R Y R. If hese paerns do no exs, hen he probably value s se as 0 auomacally. ( Q v ) = P( Q, X v ) = P( Q X, v ) P( X v ) P. (3) all X all X Assume he sascal ndependence of he observaons, and gven he sae sequence of X = { x, x2,..., xc }, Equaon (4) gves he probably of X gven v. P C C ( X v ) = P( x x ) P( x ) = A ( x, x ) π ( ) + x = = Here, ( ) +. (4) P x x + represens he probably of rerevng a vdeo sho x gven ha he curren vdeo sho s x. I + corresponds o he A ( x, x + ) enry n he relaonshp P s he nal probably for vdeo sho x, marx. ( x ).e., π ( ). Equaon (5) gves he probably of an x observaon sequence (semanc even paern) C ( X, v ) = P Q = Q. P( e x ), (5) 440

4 where P e x ) ndcaes he probably of observng a ( semanc even e from a vdeo sho x. Ths value s compued usng a smlary measure by consderng lowlevel and md-level feaures. However, n hs approach, snce he users already mared hese vdeo shos as he evens hey requesed and preferred, he probables of observng he semanc evens are smply se o Cluserng Sraegy Consderng a large scale vdeo daabase, s a sgnfcan ssue o cluser smlar vdeos ogeher o speed up he smlary search. As we saed before, a wo-level HMMM has been consruced o model vdeo and vdeo shos. Furhermore, a vdeo daabase cluserng sraegy whch s raversal-based and greedy s proposed. Y p=m? N Inpus: () D: daabase ncludng M vdeos (2) Z: Maxmum sze of a cluser (Z 2) p=0; n=0; n++; sar a new cluser: CC n {}; CC n CC n {v }; q=; p++; where v has he larges Π 2(v ) value n D D D-{v }; Add v o CC n: CC n CC n {v } ; where v has he larges value of A ( v, v ) SV ( v, v ) n D 2 v v ; D D-{v }; p++; q++; Oupus: () n: Number of vdeo clusers (2) CC ( n): vdeo clusers q=z? p=m? Fgure 2. The proposed concepual vdeo daabase cluserng procedure As llusraed n Fgure 2, he proposed vdeo daabase cluserng echnque conans he followng seps. Gven he vdeo daabase D wh M vdeos and he maxmum sze of he vdeo daabase cluser as Z (Z 2), he mechansm: a) Inalze he parameers as p=0; n=0, where p denoes he number of vdeos beng clusered, and n represens he cluser number. b) Se n=n+. Search he curren vdeo daabase D for he vdeo v wh he larges saonary probably N Y N Y Π 2 (v ), and hen sars a new cluser CC n wh hs vdeo (CC n = { }; CC n CC n {v }). Inalze he parameer as q=, where q represens he number of vdeos n he curren cluser. c) Remove v from daabase D (D D-{v }). Chec f p=m. If yes, oupu he clusers. If no, go o sep d). d) Search for v, whch has he larges A ( v, v ) SV ( v, v ) n D. Add v o he curren 2 cluser CC n (CC n CC n {v }). e) v v, where v represens he mos recen clusered vdeo. Every me when a vdeo s assgned o a cluser, s auomacally removed from D (D D {v }). f) p++ and q++. Chec f p=m. If yes, oupu he cluserng resuls. If no, chec f q=z. If yes, goes o sep b) o sar a new cluser. If no, goes o sep d) o add anoher vdeo n he curren cluser. g) If here s no un-clusered vdeo lef n he curren daabase, oupu he curren clusers Ineracve Rereval upon Vdeo Clusers Iner-Cluser Relaonshps In hs research, he HMMM model s exended by he 3 rd level MMM o mprove he overall rereval performance. In he 3 rd level MMM (d = 3), he saes (S 3 ) denoes he vdeo clusers. Marx A 3 descrbes he relaonshps beween each par of clusers. Defnon 2: Assume CC m and CC n are wo vdeo clusers n he vdeo daabase D. Ther relaonshp s denoed as an enry n he affny marx A 3, whch can be compued by Equaons (6) and (7). Here, SC s he funcon ha calculaes he smlary score beween wo vdeo clusers. SC( CCm, CCn) = ( Π2 ( v ) max ( A2 ( v, v ) SV ( v, v )))/ M, S F A B Π v CC m where CC m v CCn D, CC D. (6) SC( CCm, CCn ) A3 ( CCm, CCn ) =. (7) SC( CC, CC ) CC D Table. 3-Level HMMM Model s Level MMM 2 nd Level MMM 3 rd Level MMM Sae se of vdeo shos Low level vsual/audo feaures Temporal based sae ranson probably beween vdeo shos Formalzed feaure values Inal sae probably dsrbuon for vdeo shos n m Sae se of Vdeos Semanc evens (conceps) Affny relaonshp beween vdeos Annoaed even numbers Inal sae probably dsrbuon for vdeos Sae se of vdeo clusers - Affny relaonshp beween vdeo clusers - Inal sae probably dsrbuon for vdeo clusers 44

5 The marx Π 3 can be consruced o represen he nal sae probably of he clusers. The calculaon of Π 3 s smlar o he ones for Π and Π 2. In addon, marx L 2,3 can also be consruced o llusrae he ln condons beween he 2 nd level MMMs and he 3 rd level MMM. As demonsraed n Table, he MMM models n dfferen levels of he 3-level HMMM descrbe dsnc obecs and represen dfferen meanngs Rereval hrough Clusered Vdeo Daabase Gven an example sho sequence Q = { s, s,..., s } 2 C whch represens he even paern as e, e,..., e } such { 2 C ha s descrbes e ( C), and hey follow he emporal sequence as T T... T. Assume ha a s s 2 s C user wans o fnd op- relaed sho sequences whch follow he smlar paerns. In our proposed rereval algorhm, a recursve process s conduced o raverse he HMMM daabase model and fnd he op canddae resuls. As shown n Fgure 3, a lace based srucure for he overall vdeo daabase can be consruced. Assume he ransons are sored based on her edge weghs [6], and he rereval algorhm wll raverse he edge wh a hgher wegh each me. For example, n Fgure 4, we assume ha he edge weghs sasfy w(s, s 2 ) w(s, s 4 ) w(s, s 7 ). The rereval algorhm can be descrbed as below.. Search for he frs canddae cluser, frs canddae vdeo and frs canddae vdeo sho by checng marces Π 3, Π 2, B 2, Π and B. 2. If he paern s no complee, connue search for he nex even (vdeo sho) va compung he edge weghs by checng A. 3. If he canddae paern has been compleed, he sysem goes bac sae by sae and checs for oher possble pahs. The sysem also checs f here are already canddae paerns beng rereved. If yes, he sysem sops searchng and goes o Sep If here s no more possbles n he curren vdeo, hen mar hs vdeo wh a searched flag and chec A 2 and B 2 o fnd nex canddae vdeo. 5. If all he vdeos are searched n he curren cluser, hen mar he curren cluser as searched cluser and chec A 3 o fnd he nex canddae vdeo cluser. 6. Once paerns are rereved, or here are no more possbles n he daabase, he sysem rans he canddae paerns va calculang he smlary scores [6] and oupus he canddae paerns. As shown n he Fgure 4, he yellow cells nclude he pahs he algorhm raversed. Furhermore, we desgned a funcon o fll n he mssed cells by copyng he corresponden shos n he prevous canddae paerns. Fnally, 6 complee canddae paerns are generaed. Once canddae paerns are generaed, he sysem does no need o raverse any oher clusers or vdeos. Therefore, sgnfcanly reduces he searchng spaces and acceleraes he searchng speed. CC CC 2 Cluser Vdeo T : Even T 2: Even 2 v v 2 v 3 Canddae vdeo cluser Canddae vdeo sae Vdeo sho sae whch maches he expeced even Transon whch maches he expeced emporal even Transon whch goes o search he nex canddae vdeo Fgure 3. Lace srucure of he clusered vdeo daabase Fgure 4. Resul paerns and he raverse pah s s 9 s 0 4. EXPERIMENTAL RESULTS T 3: Even 3 We have bul up a soccer vdeo daabase wh oally 45 vdeos, whch conans 8977 vdeo shos. A rereval sysem has also been mplemened for he sysem ranng and expermenal ess. Toally 50 ses of hsorcal queres were ssued and user feedbacs were reurned wh her preferred paerns, whch cover all of he 45 vdeos and 259 dsnc vdeo shos. In he cluserng process, we defne he cluser sze as 0 and he expeced resul paern number as =60. As shown n Fgure 5, we use leers G, F, and C o represen Goal, Free c, Corner c evens, respecvely. Therefore, he x-axs represens dfferen query paerns, e.g., G means a query o search for Goal evens; FG means a query o search for he even paern where a Free c followed by a Goal ; and CGF means a query paern of a Corner c even, followed by a Goal and hen a Free c, ec. For each query paern, we ssued 0 queres o compue he average execuon me n mllseconds. As llusraed n Fgure 5, he query paerns wh fewer even numbers wll be execued n less me as expeced. In addon, he execuon me of he sysem wh clusers s less han ha of he sysem whou clusers, ndcang ha our proposed approach effecvely s 2 s 4 s 7 s 3 s 5 s 6 s 8 s s 2 s 3 R Cluser Vdeo Even Even 2 Even 3 CC v s s 2 s 3 2 CC v s s 4 s 5 3 CC v s s 4 s 6 4 CC v s s 7 s 8 5 CC v s 9 s 7 s 8 6 CC 2 v 3 s s 2 s 3 442

6 groups relevan vdeos n he vdeo clusers so ha only he relevan clusers and her member vdeos wll need o be searched. Therefore, he searchng space s dramacally decreased, and he execuon of he queres becomes faser. 200 Execuon Tme Comparson Average Execuon Tme (mllseconds) whou Clusers wh Clusers (a) Query over non-clusered soccer vdeo daabase 0 G C F FG CG GC CGF FCG FGC Query Paern Fgure 5. Comparson of he average execuon me For he query paern ( Corner c followed by a Goal ), Fgure 6(a) demonsraes he frs screen of rereval resuls over he non-clusered soccer vdeo daabase; whle Fgure 6(b) shows he query resuls over he clusered daabase. I can be clearly seen ha he query resuls n he same cluser represen he smlar vsual clues, whch are mned from he hsorcal queres and feedbacs, and accordngly represen user preferences. 5. CONCLUSIONS In hs paper, an neracve vdeo rereval sysem s proposed whch ncorporaes he concepual vdeo cluserng sraegy and he HMMM herarchcal learnng mechansm. Ths proposed framewor s able o reuse he cumulaed user feedbacs o cluser he vdeos, such ha he overall sysem no only learns he user percepons, bu also ges a good daabase srucure va adopng he cluserng echnque. The HMMM-based daabase model s consruced o suppor he concepual vdeo daabase cluserng. In he meanwhle, he cluserng echnque helps o furher mprove he daabase srucure va addng a new level o model he vdeo clusers. The expermens show ha our proposed approach helps accelerae he rereval speed wh provdng decen rereval resuls. 6. ACKOWLEDGEMENTS For Shu-Chng Chen, hs research was suppored n par by NSF EIA and HRD For Me-Lng Shyu, hs research was suppored n par by NSF ITR (Medum) IIS For Suar Rubn, hs research was suppored n par by an ONR ILIR gran. (b) Query over clusered soccer vdeo daabase Fgure 6. Soccer vdeo rereval sysem nerfaces 7. REFERENCES [] C.-W. Ngo, T.-C. Pong and H.-J. Zhang, On Cluserng and Rereval of Vdeo Shos, In Proc. of he 9 h ACM Inernaonal Conference on Mulmeda, Oawa, Canada, 200, pp [2] J.-M. Odobez, D. Gaca-Perez, and M. Gullemo, Vdeo Sho Cluserng usng Specral Mehods, In Proc. of 3rd Inernaonal Worshop on Conen-Based Mulmeda Indexng (CBMI), Rennes, France, 2003, pp [3] L. Rabner and B.-H. Juang. Fundamenals of Speech Recognon, Prence Hall, 993, ISBN: [4] M.-L. Shyu, S.-C. Chen, M. Chen, and C. Zhang, Affny Relaon Dscovery n Image Daabase Cluserng and Conenbased Rereval, In Proc. of ACM Mulmeda 2004 Conference, New Yor, USA, pp [5] L. Xe, S.-F. Chang, A. Dvaaran, and H. Sun, Unsupervsed Dscovery of Mullevel Sascal Vdeo Srucures Usng Herarchcal Hdden Marov Models, In Proc. of IEEE Inernaonal Conference on Mulmeda and Expo (ICME), vol. 3, pp , July [6] N. Zhao, S.-C. Chen and M.-L. Shyu, Vdeo Daabase Modelng and Temporal Paern Rereval Usng Herarchcal Marov Model Medaor, In Proc. of he Frs IEEE Inernaonal Worshop on Mulmeda Daabases and Daa Managemen (MDDM), n conuncon wh IEEE Inernaonal Conference on Daa Engneerng (ICDE), Alana, USA,