UC San Diego UC San Diego Previously Published Works

Transcription

1 UC San Dego UC San Dego Prevously Publshed Works Tle Modelng, cluserng, and segmenng vdeo wh mxures of dynamc exures Permalnk hps://escholarshp.org/uc/em/0851 Journal IEEE Transacons on Paern Analyss and Machne Inellgence, 30(5) ISSN Auhors Chan, Anon B. Vasconcelos, Nuno Publcaon Dae Peer revewed escholarshp.org Powered by he Calforna Dgal Lbrary Unversy of Calforna

2 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 5, MAY Modelng, Cluserng, and Segmenng Vdeo wh Mxures of Dynamc Texures Anon B. Chan, Suden Member, IEEE, and Nuno Vasconcelos, Member, IEEE Absrac A dynamc exure s a spao-emporal generave model for vdeo, whch represens vdeo sequences as observaons from a lnear dynamcal sysem. Ths work sudes he mxure of dynamc exures, a sascal model for an ensemble of vdeo sequences ha s sampled from a fne collecon of vsual processes, each of whch s a dynamc exure. An expecaon-maxmzaon (EM) algorhm s derved for learnng he parameers of he model, and he model s relaed o prevous works n lnear sysems, machne learnng, meseres cluserng, conrol heory, and compuer vson. Through expermenaon, s shown ha he mxure of dynamc exures s a suable represenaon for boh he appearance and dynamcs of a varey of vsual processes ha have radonally been challengng for compuer vson (for example, fre, seam, waer, vehcle and pedesran raffc, and so forh). When compared wh sae-of-he-ar mehods n moon segmenaon, ncludng boh emporal exure mehods and radonal represenaons (for example, opcal flow or oher localzed moon represenaons), he mxure of dynamc exures acheves superor performance n he problems of cluserng and segmenng vdeo of such processes. Index Terms Dynamc exure, emporal exures, vdeo modelng, vdeo cluserng, moon segmenaon, mxure models, lnear dynamcal sysems, me-seres cluserng, Kalman fler, probablsc models, expecaon-maxmzaon. Ç 1 INTRODUCTION ONE famly of vsual processes ha has relevance for varous applcaons of compuer vson s ha of, wha could be loosely descrbed as, vsual processes composed of ensembles of parcles subec o sochasc moon. The parcles can be mcroscopc (for example, plumes of smoke), macroscopc (for example, leaves and vegeaon blowng n he wnd), or even obecs (for example, a human crowd, a flock of brds, a raffc am, or a bee hve). The applcaons range from remoe monorng for he prevenon of naural dsasers (for example, fores fres), o background subracon n challengng envronmens (for example, oudoors scenes wh vegeaon), and varous ype of survellance (for example, raffc monorng, homeland secury applcaons, or scenfc sudes of anmal behavor). Despe her praccal sgnfcance and he ease wh whch hey are perceved by bologcal vson sysems, he vsual processes n hs famly sll pose remendous challenges for compuer vson. In parcular, he sochasc naure of he assocaed moon felds ends o be hghly challengng for radonal moon represenaons such as opcal flow [1], [], [3], [4], whch requres some degree of moon smoohness, paramerc moon models [5], [6], [7], whch assume a pecewse planar world [8], or obec rackng [9], [10], [11], whch ends o be mpraccal when he number of subecs o rack s large, and hese obecs nerac n a complex manner. The man lmaon of all hese represenaons s ha hey are nherenly local, amng o acheve undersandng of he. The auhors are wh he Deparmen of Elecrcal and Compuer Engneerng, Unversy of Calforna, San Dego, 9500 Glman Drve, Mal Code 0409, La Jolla, CA E-mal: abchan@ucsd.edu, nuno@ece.ucsd.edu. Manuscrp receved 8 Sep. 006; revsed 6 Mar. 007; acceped 16 May 007; publshed onlne 5 July 007. Recommended for accepance by S. Sefano. For nformaon on obanng reprns of hs arcle, please send e-mal o: pam@compuer.org, and reference IEEECS Log Number TPAMI Dgal Obec Idenfer no /TPAMI whole by modelng he moon of he ndvdual parcles. Ths s conrary o how hese vsual processes are perceved by bologcal vson: smoke s usually perceved as a whole, a ree s normally perceved as a sngle obec, and he deecon of raffc ams rarely requres rackng ndvdual vehcles. Recenly, here has been an effor o advance oward hs ype of holsc modelng by vewng vdeo sequences derved from hese processes as dynamc exures or, more precsely, samples from sochasc processes defned over space and me [1], [13], [14], [15], [16], [17], [18], [19]. In fac, he dynamc exure framework has been shown o have grea poenal for vdeo synhess [13], [19], mage regsraon [14], moon segmenaon [15], [18], [19], [0], [1], [], and vdeo classfcaon [16], [17]. Ths s, n sgnfcan par, due o he fac ha he underlyng generave probablsc framework s capable of 1) absracng a wde varey of complex moon paerns no a smple spao-emporal process and ) synheszng samples of he assocaed me-varyng exure. One sgnfcan lmaon of he orgnal dynamc exure model [13] s, however, s nably o provde a percepual decomposon no mulple regons, each of whch belongs o a semancally dfferen vsual process: for example, a flock of brds flyng n fron of a waer founan, hghway raffc movng n oppose drecons, vdeo conanng boh smoke and fre, and so forh. One possbly o address hs problem s o apply he dynamc exure model locally [15] by splng he vdeo no a collecon of localzed spaoemporal paches, fng he dynamc exure o each pach, and cluserng he resulng models. However, hs mehod, along wh oher recen proposals [18], [], lacks some of he aracve properes of he orgnal dynamc exure model: a clear nerpreaon as a probablsc generave model for vdeo and he necessary robusness o operae whou manual nalzaon. To address hese lmaons, we noe ha whle he holsc dynamc exure model n [13] s no suable for such scenes, he underlyng generave framework s. In /08/$5.00 ß 008 IEEE Publshed by he IEEE Compuer Socey

3 910 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 5, MAY 008 fac, co-occurrng exures can be accouned for by augmenng he probablsc generave model wh a dscree hdden varable, whch has a number of saes equal o he number of exures, and encodes whch of hem s responsble for a gven pece of he spao-emporal vdeo volume. Condoned on he sae of hs hdden varable, he vdeo volume s hen modeled as a smple dynamc exure. Ths leads o a naural exenson of he dynamc exure model, a mxure of dynamc exures [0] (or mxure of lnear dynamcal sysems (LDSs)), whch we sudy n hs work. The mxure of dynamc exures s a generave model, where a collecon of vdeo sequences (or vdeo paches) are modeled as samples from a se of underlyng dynamc exures. I provdes a naural probablsc framework for cluserng vdeo and for vdeo segmenaon hrough he cluserng of spao-emporal paches. In addon o nroducng he dynamc exure mxure as a generave model for vdeo, we repor on hree man conrbuons. Frs, an expecaon-maxmzaon (EM) algorhm s derved for maxmum-lkelhood esmaon of he parameers of a dynamc exure mxure. Second, he relaonshps beween he mxure model and varous oher models prevously proposed, ncludng mxures of facor analyzers, LDSs, and swched lnear dynamc models, are analyzed. Fnally, we demonsrae he applcably of he model o he soluon of radonally dffcul vson problems ha range from cluserng raffc vdeo sequences o segmenaon of sequences conanng mulple dynamc exures. The paper s organzed as follows. In Secon, we formalze he dynamc exure mxure model. In Secon 3, we presen he EM algorhm for learnng s parameers from ranng daa. In Secons 4 and 5, we relae o prevous models and dscuss s applcaon o vdeo cluserng and segmenaon. Fnally, n Secon 6, we presen an expermenal evaluaon n he conex of hese applcaons. MIXTURES OF DYNAMIC TEXTURES In hs secon, we descrbe he dynamc exure mxure model. For compleeness, we sar wh a bref revew of he dynamc exure model..1 Dynamc Texure A dynamc exure [1], [13] s a generave model for boh he appearance and he dynamcs of vdeo sequences. I consss of a random process conanng an observed varable y, whch encodes he appearance componen (vdeo frame a me ), and a hdden sae varable x, whch encodes he dynamcs (evoluon of he vdeo over me). The sae and observed varables are relaed hrough he LDS defned by x þ1 ¼ Ax þ v ð1þ y ¼ Cx þ w ; where x IR n and y IR m (ypcally n m). The parameer A IR nn s a sae-ranson marx, and C IR mn s an observaon marx (for example, conanng he prncpal componens of he vdeo sequence when learned wh ha n [13]). The drvng nose process v s normally dsrbued wh zero mean and covarance Q, ha s, v Nð0;QÞ, where Q S n þ s a posve-defne n n marx. The observaon nose w s also zero mean and Gaussan, wh covarance R, ha s, w Nð0;RÞ, where R S m þ. Noe ha he model adoped Fg. 1. (a) Dynamc exure. (b) Dynamc exure mxure. The hdden varable z selecs he parameers of he remanng nodes. hroughou hs work, whch suppors an nal sae x 1 of arbrary mean and covarance, ha s, x 1 Nð; SÞ, sa slgh exenson of he model orgnally proposed n [13]. 1 Ths exenson produces a rcher vdeo model ha can capure varably n he nal frame and s necessary for learnng a dynamc exure from mulple vdeo samples wh dfferen nal frames (as s he case n cluserng and segmenaon problems). The dynamc exure s specfed by he parameers ¼fA; Q; C; R; ; Sg and can be represened by he graphcal model n Fg. 1a. I can be shown [3], [4] from hs defnon ha he dsrbuons of he nal sae, he condonal sae, and he condonal observaon are pðx 1 Þ¼Gðx 1 ;;SÞ; pðx x 1 Þ¼Gðx ;Ax 1 ;QÞ; pðy x Þ¼Gðy ;Cx ;RÞ; where Gðx; ; Þ ¼ðÞ n= 1= e 1 kx k s he n-dmensonal mulvarae Gaussan dsrbuon, and kxk ¼ x T 1 x s he Mahalanobs dsance wh respec o he covarance marx. Leng x 1 ¼ðx 1; ;x Þ, and y 1 ¼ ðy 1 ; ;y Þ be a sequence of saes and observaons, he on dsrbuon s pðx 1 ;y 1 Þ¼pðx 1Þ Y ¼ pðx x 1 Þ Y ¼1 pðy x Þ: A number of mehods are avalable o learn he parameers of he dynamc exure from a ranng vdeo sequence, ncludng maxmum-lkelhood mehods (for example, expecaon-maxmzaon [5]), nonerave subspace mehods (for example, N4SID [6]) or a subopmal, bu compuaonally effcen, procedure [13].. Mxure of Dynamc Texures Under he dynamc exure mxure model, he observed vdeo sequence y 1 s sampled from one of K dynamc exures, each havng some nonzero probably of occurrence. Ths s a useful exenson for wo classes of applcaons. The frs class 1. The nal condon n [13] s specfed by a fxed nal vecor x 0 IR n or, equvalenly, x 1 NðAx 0 ;QÞ. ðþ ð3þ ð4þ ð5þ

4 CHAN AND VASCONCELOS: MODELING, CLUSTERING, AND SEGMENTING VIDEO WITH MIXTURES OF DYNAMIC TEXTURES 911 nvolves a vdeo ha s homogeneous a each me nsan bu has varyng sascs over me. For example, he problem of cluserng a se of vdeo sequences aken from a saonary hghway raffc camera. Alhough each vdeo wll depc raffc movng a homogeneous speed, he exac appearance of each sequence s conrolled by he amoun of raffc congeson. Dfferen levels of raffc congeson can be represened by K dynamc exures. The second nvolves nhomogeneous vdeo, ha s, vdeo composed of mulple process ha can be ndvdually modeled as dynamc exures of dfferen parameers. For example, n a vdeo scene conanng fre and smoke, a random vdeo pach aken from he vdeo wll conan eher fre or smoke, and a collecon of vdeo paches can be represened as a sample from a mxure of wo dynamc exures. Formally, gven componen prors ¼f 1 ;...; K g wh P K ¼1 ¼ 1 and dynamc exure componens of parameers f 1 ;...; K g, a vdeo sequence s drawn by 1. samplng a componen ndex z from he mulnomal dsrbuon parameerzed by.. Samplng an observaon y 1 from he dynamc exure componen of parameers z. The probably of a sequence y 1 under hs model s pðy 1 Þ¼XK pðy 1z ¼ Þ; ð6þ ¼1 where pðy 1z ¼ Þ s he class condonal probably of he h dynamc exure, ha s, he dynamc exure componen parameerzed by ¼fA ;Q ;C ;R ; ;S g. The sysem of equaons ha defne he mxure of dynamc exures s x þ1 ¼ A z x þ v ð7þ y ¼ C z x þ w ; where he random varable z mulnomalð 1 ; ; K Þ sgnals he mxure componen from whch he observaons are drawn, he nal condon s gven by x 1 Nð z ;S z Þ and he nose processes by v Nð0;Q z Þ and w Nð0;R z Þ. The condonal dsrbuons of he saes and observaons, gven he componen ndex z are pðx 1 zþ ¼Gðx 1 ; z ;S z Þ; pðx x 1 ;zþ¼gðx ;A z x 1 ;Q z Þ; pðy x ;zþ¼gðy ;C z x ;R z Þ; and he overall on dsrbuon s pðy 1 ;x 1 ;zþ¼pðzþpðx 1zÞ Y ¼ pðx x 1 ;zþ Y ¼1 pðy x ;zþ: ð8þ ð9þ ð10þ ð11þ The graphcal model for he dynamc exure mxure s presened n Fg. 1b. Noe ha, alhough he addon of he random varable z nroduces loops n he graph, exac nference s sll racable because z s conneced o all oher nodes. Hence, he graph s already moralzed and rangulaed [7], and he uncon ree of Fg. 1b s equvalen o ha of he basc dynamc exure, wh he varable z added o each clque. Ths makes he complexy of exac nference for a mxure of K dynamc exures K mes ha of he underlyng dynamc exure. 3 PARAMETER ESTIMATION USING EM Gven a se of ndependen and dencally dsrbued (..d.) vdeo sequences fy ðþ g N ¼1, we would lke o learn he parameers of a mxure of dynamc exures ha bes fs he daa n he maxmum-lkelhood sense [3], ha s, X N ¼ argmax log pðy ðþ ;Þ: ð1þ ¼1 When he probably dsrbuon depends on hdden varables (ha s, he oupu of he sysem s observed, bu s sae s unknown), he maxmum-lkelhood soluon can be found wh he EM algorhm [8]. For he dynamc exure mxure, he observed nformaon s a se of vdeo sequences fy ðþ g N ¼1, and he mssng daa consss of 1) he assgnmen zðþ of each sequence o a mxure componen and ) he hdden sae sequence x ðþ ha produces y ðþ. The EM algorhm s an erave procedure ha alernaes beween esmang he mssng nformaon wh he curren parameers and compung new parameers gven he esmae of he mssng nformaon. In parcular, each eraon consss of E-Sep : Qð; ^Þ ¼E X;ZY ; ^ðlog pðx; Y ; Z;ÞÞ; ð13þ M-Sep : ^ ¼ argmax Qð; ^Þ; where pðx; Y ; Z;Þ s he complee-daa lkelhood of he observaons, hdden saes, and hdden assgnmen varables, parameerzed by. To maxmze clary, we only presen here he equaons of he E and M seps for he esmaon of dynamc exure mxure parameers. Ther dealed dervaon s gven n Appendx A. The only assumpons are ha he observaons are drawn ndependenly and have zero-mean, bu he algorhm could be rvally exended o he case of nonzero means. All equaons follow he noaon n Table EM Algorhm for Mxure of Dynamc Texures Observaons are denoed by fy ðþ g N ¼1, he hdden sae varables by fx ðþ g N ¼1, and he hdden assgnmen varables by fz ðþ g N ¼1. As s usual n he EM leraure [8], we nroduce a vecor z f0; 1g K, such ha z ; ¼ 1 f and only f z ðþ ¼. The complee-daa log-lkelhood s (up o a consan) gven by (15), where P ðþ ; ¼ x ðþ ðx ðþ Þ T and P ðþ ; 1 ¼xðÞ ðx ðþ 1 ÞT. Applyng he expecaon of (13) o (15) yelds he Q funcon (16), where ðx; Y ; ZÞ ¼ X z ; log 1 X z ; log S ; ; 1 X h z ; r S 1 P ðþ 1;1 xðþ 1 T x ðþ T 1 þ T ; X z ; log R 1 X X h z ; r R 1 ; ; ¼1 y ðþ x ðþ T C T C x ðþ y ðþ T þ C P ðþ ; CT 1 X z ; log Q 1 X X h z ; r ; ; ¼ P ðþ ; 1 AT A P ðþ T ; 1 þ A P ðþ 1; 1 AT y ðþ Q 1 ; y ðþ T P ðþ ; ð15þ

5 91 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 5, MAY 008 TABLE 1 Noaon for EM for Mxure of Dynamc Texures Qð; ^Þ ¼ 1 X h r R 1 C T C T þ C C T 1 X h r S 1 T T þ ^N T ð16þ 1 X h r Q 1 A T A T þ A A T þ X ^N log log R 1 log Q 1 log S ; ^N ¼ X ^z ;; ¼ X ^z ; ^x ðþ 1 ; ¼ X ^z ðþ ; ^P ¼ X ^z ; ¼ X ^z ; ¼ X ^z ; 1;1 ; X ^P ðþ ¼ ; 1 ; X ¼1 yðþ y ðþ X ¼1 yðþ ¼ X ^z ; ¼ X ^z ; ¼ X ^z ; T ; T ^x ðþ ; are he aggregaes of he expecaons ^x ðþ ¼ E x ðþ y ðþ ;z ðþ ¼ ^P ðþ ; ¼ E x ðþ y ðþ ;z ðþ ¼ ^P ðþ ; 1 ¼ E x ðþ y ðþ ;z ðþ ¼ X ¼1 X ¼ X ¼ x ðþ P ðþ ; ^P ðþ ; ; ^P ðþ ; ; ^P ðþ 1; 1 ; ð17þ ; ð18þ ; ð19þ ; ð0þ P ðþ ; 1 and he poseror assgnmen probably s ^z ; ¼ pðz ðþ ¼ y ðþ Þ¼ pðy ðþ z ðþ ¼ Þ P K k¼1 kpðy ðþ z ðþ ¼ kþ : ð1þ Hence, he E-sep consss of compung he condonal expecaons (18)-(1) and can be mplemened effcenly wh he Kalman smoohng fler [5] (see Appendx B), whch esmaes he mean and covarance of he sae x ðþ condoned on he observaon y ðþ and z ðþ ¼ ^x ðþ ¼ E x ðþ y ðþ ;z ðþ ¼ x ðþ ; ðþ ^V ðþ ; ¼ cov x ðþ y ðþ ;z ðþ ¼ ^V ðþ ; 1 ¼ cov x ðþ y ðþ ;z ðþ ¼ x ðþ x ðþ ;x ðþ ;x ðþ 1 ; ð3þ : ð4þ The second-order momens of (19) and (0) are hen calculaed as ^P ðþ ; ¼ ^V ðþ ; þ ^xðþ ð^xðþ ÞT and ^P ðþ ; 1 ¼ ^V ðþ ; 1 þ ^x ðþ ð^xðþ 1 ÞT. Fnally, he daa lkelhood pðy ðþ z ðþ ¼ Þ s compued usng he nnovaons form of he log-lkelhood (agan, see [5] or Appendx B). In he M-sep, he dynamc exure parameers are updaed accordng o (14), resulng n he followng updae sep for each mxure componen : C ¼ ð Þ 1 ; R ¼ 1 ^N A ¼ ð Þ 1 ; Q ¼ 1 ð 1Þ ^N C ; ¼ 1^N ; S ¼ 1^N ð ÞT ; ¼ ^N N : A T ; ð5þ A summary of EM for he mxure of dynamc exures s presened n Algorhm 1. The E-sep reles on he Kalman smoohng fler o compue 1) he expecaons of he hdden sae varables x, gven he observed sequence y ðþ and he componen assgnmen z ðþ, and ) he lkelhood of observaon y ðþ gven he assgnmen z ðþ. The M-sep hen compues he maxmum-lkelhood parameer values for each dynamc exure componen by averagng over all sequences fy ðþ g N ¼1, weghed by he poseror probably of assgnng z ðþ ¼. Algorhm 1 EM for a mxure of dynamc exures. Inpu: N sequences fy ðþ g N ¼1, number of componens K. Inalze f ; g for ¼ 1 o K. repea {Expecaon Sep} for ¼f1;...;Ng and ¼f1;...;Kg do Compue he expecaons (18)-(1) wh he Kalman smoohng fler (Appendx B) on y and. end for {Maxmzaon Sep} for ¼ 1 o K do Compue aggregae expecaons (17). Compue new parameers f ; g wh (5). end for unl convergence Oupu: f ; g K ¼1 3. Inalzaon Sraeges I s known ha he accuracy of parameer esmaes produced by EM s dependen on how he algorhm s nalzed. In he remander of hs secon, we presen hree

6 CHAN AND VASCONCELOS: MODELING, CLUSTERING, AND SEGMENTING VIDEO WITH MIXTURES OF DYNAMIC TEXTURES 913 nalzaon sraeges ha we have emprcally found o be effecve for learnng mxures of dynamc exures Inal Seedng If an nal cluserng of he daa s avalable (for example, by specfcaon of nal conours for segmened regons), hen each mxure componen s learned by applyng he mehod n [13] o each of he nal clusers. 3.. Random Trals Several rals of EM are run wh dfferen random nalzaons, and he parameers ha bes f he daa, n he maxmum-lkelhood sense, are seleced. For each EM ral, each mxure componen s nalzed by applcaon of he mehod of [13] o a randomly seleced example from he daa se Componen Splng The EM algorhm s run wh an ncreasng number of mxure componens, a common sraegy n he speechrecognon communy [9]: 1. Run he EM algorhm wh K ¼ 1 mxure componens.. Duplcae a mxure componen and perurb he new componen s parameers. 3. Run he EM algorhm, usng he new mxure model as nalzaon. 4. Repea Seps and 3 unl he desred number of componens s reached. For he mxure of dynamc exures, we use he followng selecon and perurbaon rules: 1) he mxure componen wh he larges msf n he sae-space (ha s, he mxure componen wh he larges egenvalue of Q) s seleced for duplcaon and ) he prncpal componen whose coeffcens have he larges nal varance (ha s, he column of C assocaed wh he larges varance n S) s scaled by CONNECTIONS TO THE LITERATURE Alhough novel as a ool for modelng vdeo, wh applcaon o problems such as cluserng and segmenaon, he mxure of dynamc exures and he proposed EM algorhm are relaed o varous prevous works n adapve flerng, sascs and machne learnng, me-seres cluserng, and vdeo segmenaon. 4.1 Adapve Flerng and Conrol In he conrol-heory leraure, Magll [30] proposes an adapve fler based on banks of Kalman flers runnng n parallel, where each Kalman fler models a mode of a physcal process. The oupu of he adapve fler s he average of he oupus of he ndvdual Kalman flers, weghed by he poseror probably ha he observaon was drawn from he fler. Ths s an nference procedure for he hdden sae of a mxure of dynamc exures condoned on he observaon. The key dfference wh respec o hs work s ha, even hough Magll [30] focuses on nference on he mxure model wh known parameers, hs work focuses on learnng he parameers of he mxure model. The work of Magll [30] s he forefaher of oher mulple-model-based mehods n adapve esmaon and conrol [31], [3], [33], bu none address he learnng problem. 4. Models Proposed n Sascs and Machne Learnng For a sngle componen ðk ¼ 1Þ and a sngle observaon ðn ¼ 1Þ, he EM algorhm for he mxure of dynamc exures reduces o he classcal EM algorhm for learnng an LDS [5], [34], [35]. The LDS s a generalzaon of he facor analyss model [4], a sascal model ha explans an observed vecor as a combnaon of measuremens ha are drven by ndependen facors. Smlarly, he mxure of dynamc exures s a generalzaon of he mxure of facor analyzers, and he EM algorhm for a dynamc exure mxure reduces o he EM algorhm for learnng a mxure of facor analyzers [36]. The dynamc exure mxure s also relaed o swchng lnear dynamcal models, where he parameers of an LDS are seleced va a separae Markovan swchng varable as me progresses. Varaons of hese models nclude [37], [38], where only he observaon marx C swches, [39], [40], [41], where he sae parameers swch (A and Q), and [4], [43], where he observaon and sae parameers swch (C, R, A, and Q). These models have one sae varable ha evolves accordng o he acve sysem parameers a each me sep. Ths makes he swchng model a mxure of an exponenally ncreasng number of LDSs wh me-varyng parameers. In conras o swchng models wh a sngle sae varable, he swchng sae-space model proposed n [44] swches he observed varable beween he oupu of dfferen LDSs a each me sep. Each LDS has s own observaon marx and sae varable, whch evolves accordng o s own sysem parameers. The dfference beween he swchng sae-space model and he mxure of dynamc exures s ha he swchng sae-space model can swch beween LDS oupus a each me sep, whereas he mxure of dynamc exures selecs an LDS only once a me ¼ 1 and never swches from. Hence, he mxure of dynamc exures s smlar o a specal case of he swchng sae-space model, where he nal probables of he swchng varable are he mxure componen probables, and he Markovan ranson marx of he swchng varable s equal o he deny marx. The ably of he swchng sae-space model o swch a each me sep resuls n a poseror dsrbuon ha s a Gaussan mxure wh a number of erms ha ncreases exponenally wh me [44]. Alhough he mxure of dynamc exures s closely relaed o boh swchng LDS models and he model n [44], he fac ha selecs only one LDS per observed sequence makes he poseror dsrbuon a mxure of a consan number of Gaussans. Ths key dfference has consequences of sgnfcan praccal mporance. Frs, when he number of componens ncreases exponenally (as s he case for models ha nvolve swchng), exac nference becomes nracable, and he EM-syle of learnng requres approxmae mehods (for example, varaonal approxmaons) whch are, by defnon, subopmal. Second, because he exponenal ncrease n he number of degrees of freedom s no accompaned by an exponenal ncrease n he amoun of avalable daa (whch only grows lnearly wh me), he dffculy of he learnng problem ncreases wh sequence. Resrcng he LDS parameers S ¼ Q ¼ I n, ¼ 0, A ¼ 0, and R as a dagonal marx yelds he facor analyss model. Smlarly, for he mxure of dynamc exures, seng S ¼ Q ¼ I n and A ¼ 0 for each facor analyss componen, and ¼ 1 (snce here are no emporal dynamcs) yelds he mxure of facor analyzers.

7 914 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 5, MAY 008 lengh. None of hese problems affec he dynamc exure mxure, for whch exac nference s racable, allowng he dervaon of he exac EM algorhm presened above. 4.3 Tme-Seres Cluserng Alhough we apply he mxure of dynamc exures o vdeo, he model s general and can be used o cluser any ype of me seres. When compared o he leraure n hs feld [45], he mxure of dynamc exures can be caegorzed as a model-based mehod for cluserng mulvarae connuousvalued me-seres daa. Two alernave mehods are avalable for cluserng hs ype of daa, boh based on he K-Means algorhm wh dsance (or smlary) measures suable for me seres: 1) Kakzawa e al. [46] measures he dsance beween wo me-seres wh he KL dvergence or he Chernoff measure, whch are esmaed nonparamercally n he specral doman, and ) Snghal and Seborg [47] measures smlary by comparng he PCA subspaces and he means of he me-seres bu dsregards he dynamcs of he me seres. The well-known connecon beween EM and K-Means makes hese algorhms somewha relaed o he dynamc exure mxure, bu hey lack a precse probablsc nerpreaon. Fnally, he dynamc exure mxure s relaed o he ARMA mxure proposed n [48]. The man dfferences are ha he ARMA mxure 1) only models unvarae daa and ) does no ulze a hdden sae model. On he oher hand, he ARMA mxure suppors hgher-order Markov models, whereas he dynamc exure mxure s based on a frs-order Markovan assumpon. 4.4 Vdeo Segmenaon The dea of applyng dynamc exure represenaons o he segmenaon of he vdeo has prevously appeared n he vdeo leraure. In fac, some of he nspraon for our work was he promse shown for emporal exure segmenaon (for example, smoke and fre) by he dynamc exure model n [13]. For example, Doreo e al. [15] segmens a vdeo by cluserng paches of dynamc exure usng he level-ses framework and he Marn dsance. More recenly, Ghoreysh and Vdal [] clusers pxel nenses (or local exure feaures) usng auoregressve (AR) processes and level ses, and Vdal and Ravchandran [18] segmens a vdeo by cluserng pxels wh smlar raecores n me usng generalzed PCA (GPCA). Alhough hese mehods have shown promse, hey do no explo he probablsc naure of he dynamc exure represenaon for he segmenaon self. On he oher hand, he segmenaon algorhms proposed n he followng secon are sascal procedures ha leverage on he mxure of dynamc exure o perform opmal nference. Ths resuls n greaer robusness o varaons due o he sochascy of he vdeo appearance and dynamcs, leadng o superor segmenaon resuls, as wll be demonsraed n Secons 5 and 6. Fnally, s worh menonng ha we have prevously proposed a graphcal model ha represens a vdeo as a collecon of layers, where each layer s modeled as a dynamc exure [19], and a Markov random feld (MRF) pror s ncluded o guaranee spaal coherence of he segmenaon. When compared o he dynamc exure mxure, hs model has sgnfcanly larger compuaonal requremens. We have no, so far, been able o apply n he conex of expermens of he scale dscussed n he followng secons. 5 APPLICATIONS Lke any oher probablsc model, he dynamc exure has a large number of poenal applcaon domans, many of whch exend well beyond he feld of compuer vson (for example, modelng of hgh-dmensonal me seres for fnancal applcaons, weaher forecasng, and so forh). In hs work, we concenrae on vson applcaons, where mxure models are frequenly used o solve problems such as cluserng [49], [50], background modelng [51], mage segmenaon and layerng [6], [5], [53], [54], [55], [56], or rereval [56], [57]. The dynamc exure mxure exends hs class of mehods o problems nvolvng vdeo of parcle ensembles subec o sochasc moon. We consder, n parcular, he problems of cluserng and segmenaon. 5.1 Vdeo Cluserng Vdeo cluserng can be a useful ool o uncover hgh-level paerns of srucure n a vdeo sream, for example, recurrng evens, evens of hgh and low probably, oulyng evens, and so forh. These operaons are of grea praccal neres for some classes of parcle-ensemble vdeo such as hose ha nvolve undersandng vdeo acqured n crowded envronmens. In hs conex, vdeo cluserng has applcaon o problems such as survellance, novely deecon, even summarzaon, or remoe monorng of varous ypes of envronmens. I can also be appled o he enres of a vdeo daabase n order o auomacally creae a axonomy of vdeo classes ha can hen be used for daabase organzaon or vdeo rereval. Under he mxure of dynamc exures represenaon, a se of vdeo sequences s clusered by frs learnng he mxure ha bes fs he enre collecon of sequences and hen assgnng each sequence o he mxure componen wh he larges poseror probably of havng generaed, ha s, by labelng sequence y ðþ wh ¼ argmax log pðy ðþ z ðþ ¼ Þþlog : ð6þ 5. Moon Segmenaon Vdeo segmenaon addresses he problem of decomposng a vdeo sequence no a collecon of homogeneous regons. Alhough hs has long been known o be solvable wh mxure models and he EM algorhm [6], [7], [50], [5], [53], he success of he segmenaon operaon depends on he ably of he mxure model o capure he dmensons along whch he vdeo s sascally homogeneous. For spaoemporal exures (for example, vdeo of smoke and fre), radonal mxure-based moon models are no capable of capurng hese dmensons due o her nably o accoun for he sochasc naure of he underlyng moon. The mxure of dynamc exures exends he applcaon of mxure-based segmenaon algorhms o vdeo composed of spao-emporal exures. As n mos prevous mxurebased approaches o vdeo segmenaon, he process consss of wo seps. The mxure model s frs learned, and he vdeo s hen segmened by assgnng vdeo locaons o mxure componens. In he learnng sage, he vdeo s frs represened as a bag of paches. For spao-emporal exure segmenaon, a pach of dmensons p p q s exraced from each locaon n he

8 CHAN AND VASCONCELOS: MODELING, CLUSTERING, AND SEGMENTING VIDEO WITH MIXTURES OF DYNAMIC TEXTURES 915 vdeo sequence (or along a regularly spaced grd), 3 where p and q should be large enough o capure he dsngushng characerscs of he varous componens of he local moon feld. Noe ha hs s unlke mehods ha model he changes of appearance of sngle pxels (for example, [18] and he AR model n []) and, herefore, have no ably o capure he spaal coherence of he local moon feld. If he segmenaon boundares are no expeced o change over me, q can be se o he lengh of he vdeo sequence. The se of spao-emporal paches s hen clusered wh recourse o he EM algorhm of Secon 3. The second sep, segmenaon, scans he vdeo locaons sequenally. A each locaon, a pach s exraced and assgned o one of he mxure componens, accordng o (6). The locaon s hen declared o belong o he segmenaon regon assocaed wh ha cluser. I s neresng o noe ha he mxure of dynamc exures can be used o effcenly segmen very long vdeo sequences by frs learnng he mxure model on a shor ranng sequence (for example, a clp from he long sequence) and hen segmenng he long sequence wh he learned mxure. The segmenaon sep only requres compung he pach log-lkelhoods under each mxure componen, ha s, (73). Snce he condonal covarances ^V 1 and ^V and he Kalman fler gans K do no depend on he observaon y, hey can be precompued. The compuaonal seps requred for he daa-lkelhood of a sngle pach herefore reduce o compung, 8 ¼f1;...; g ^x 1 ¼ A^x 1 1 ; ^x ¼ ^x 1 þ K y C ^x 1 ð7þ ; ð8þ pðy y 1 1 Þ¼ 1 log M m logðþ ð9þ 1 ðy C ^x 1 Þ T M 1 ðy C ^x 1 Þ; where M ¼ C ^V 1 C T þ R. Hence, compung he daalkelhood under one mxure componen of a sngle pach requres Oð5Þ marx-vecor mulplcaons. For a mxure of K dynamc exures and N paches, he compuaon s Oð5KNÞ marx-vecor mulplcaons. 6 EXPERIMENTAL EVALUATION The performance of he mxure of dynamc exures was evaluaed wh respec o varous applcaons: 1) cluserng of me-seres daa, ) cluserng of hghway raffc vdeo, and 3) moon segmenaon of boh synhec and real vdeo sequences. In all cases, performance was compared o a represenave of he sae-of-he-ar for hese applcaon domans. The nalzaon sraeges from Secon 3. were used. The observaon nose was assumed o be ndependen and dencally dsrbued (ha s, R ¼ I m ), he nal sae covarance S was assumed o be dagonal, and he covarance marces Q, S, and R were regularzed by enforcng a lower bound on her egenvalues. Vdeos of he resuls from all expermens are avalable from a companon Web se, accessble n [58]. 6.1 Tme-Seres Cluserng We sar by presenng resuls of a comparson of he dynamc exure mxure wh several mulvarae 3. Alhough overlappng paches volae he ndependence assumpon, hey work well n pracce and are commonly adoped n compuer vson. me-seres cluserng algorhms. To enable an evaluaon based on cluserng ground ruh, hs comparson was performed on a synhec me-seres daa se, generaed as follows. Frs, he parameers of K LDSs, wh sae-space dmenson n ¼ and observaon-space dmenson m ¼ 10, were randomly generaed accordng o U n ð 5; 5Þ; S WðI n ;nþ; C N m;n ð0; 1Þ; Q WðI n ;nþ; 0 U 1 ð0:1; 1Þ; A 0 N n;n ð0; 1Þ; Wð1; Þ; R ¼ I m ; A ¼ 0 A 0 = max ða 0 Þ; where N m;n ð; Þ s a dsrbuon on IR mn marces wh each enry dsrbued as Nð; Þ, Wð;dÞ s a Wshar dsrbuon wh covarance and d degrees of freedom, U d ða; bþ s a dsrbuon on IR d vecors wh each coordnae dsrbued unformly beween a and b, and max ða 0 Þ s he magnude of he larges egenvalue of A 0. Noe ha A s a random scalng of A 0 such ha he sysem s sable (ha s, he poles of A are whn he un crcle). A me-seres daa se was generaed by samplng 0 me-seres of lengh 50 from each of he K LDSs. Fnally, each me-seres sample was normalzed o have zero emporal mean. The daa was clusered usng he mxure of dynamc exures (DyexMx), and hree mulvarae-seres cluserng algorhms from he me-seres leraure. The laer are based on varaons of K-means for varous smlary measures: PCA subspace smlary (Snghal) [47], he KL dvergence (KakKL) [46], and he Chernoff measure (KakCh) [46]. As a baselne, he daa was also clusered wh K-means [50] usng he eucldean dsance (K-Means) and he cosne smlary (K-means-c) on feaure vecors formed by concaenang each me seres. The correcness of a cluserng s measured quanavely usng he Rand ndex [59] beween he cluserng and he ground ruh. Inuvely, he Rand ndex corresponds o he probably of parwse agreemen beween he cluserng and he ground ruh, ha s, he probably ha he assgnmen of any wo ems wll be correc wh respec o each oher (n he same cluser or n dfferen clusers). For each algorhm, he average Rand ndex was compued for each value of K ¼ f; 3;...; 8g by averagng over 100 random rals of he cluserng expermen. We refer o hs synhec expermen seup as SynhecA. The cluserng algorhms were also esed on wo varaons of he expermen based on meseres ha were more dffcul o cluser. In he frs (SynhecB), he K random LDSs were forced o share he same observaon marx ðcþ, herefore forcng all he me seres o be defned n smlar subspaces. In he second (SynhecC), he LDSs had large observaon nose, ha s, Wð16; Þ. Noe ha hese varaons are ypcally expeced n vdeo cluserng problems. For example, n applcaons where he appearance componen does no change sgnfcanly beween clusers (for example, hghway vdeo wh varyng levels of raffc), all he vdeo wll span smlar mage subspaces. Fg. presens he resuls obaned wh he sx cluserng algorhms on he hree expermens, and Table shows he overall Rand ndex, compued by averagng over K. In SynhecA (Fg. a), Snghal, KakCh, and DyexMx acheved comparable performance (overall Rand of 0.995, 0.991, and 0.991) wh Snghal performng slghly beer. On he oher hand, s clear ha he wo baselne algorhms are no suable for cluserng me-seres daa, albe here s sgnfcan mprovemen when usng K-Means-c (0.831)

9 916 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 5, MAY 008 Fg.. Tme-seres cluserng resuls on hree synhec problems: (a) SynhecA, (b) SynhecB, and (c) SynhecC. Plos show he Rand ndex versus he number of clusers ðkþ for sx me-seres cluserng algorhms. raher han K-Means (0.585). In SynhecB (Fg. b), he resuls were dfferen: he DyexMx performed bes (0.995), closely followed by KakCh and KakKL (0.985 and 0.977). On he oher hand, Snghal dd no perform well (0.865) because all he me seres have smlar PCA subspaces. Fnally, n SynhecC (Fg. c), he DyexMx repeaed he bes performance (0.993), followed by KakKL and KakCh (0.960 and 0.958), wh Snghal performng he wors agan (0.858). In hs case, he dfference beween he performance of DyexMx and hose of KakKL and KakCh was sgnfcan. Ths can be explaned by he robusness of he former o observaon nose, a propery no shared by he laer due o he fragly of nonparamerc esmaon of specral marces. In summary, hese resuls show ha he mxure of dynamc exures performs smlarly o sae-of-he-ar mehods such as [47] and [46], when clusers are well separaed. I s, however, more robus agans occurrences ha ncrease he amoun of cluser overlap, whch proved dffcul for he oher mehods. Such occurrences nclude 1) me seres defned n smlar subspaces and ) me seres wh sgnfcan observaon nose and are common n vdeocluserng applcaons. 6. Vdeo Cluserng To evaluae s performance n problems of praccal sgnfcance, he dynamc exure mxure was used o cluser vdeo of vehcle hghway raffc. Cluserng was performed on 53 vdeo sequences colleced by he Washngon Deparmen of Transporaon (WSDOT) on nersae I-5 n Seale, Washngon [60]. Each vdeo clp s 5 seconds long, and he collecon spanned abou 0 hours over wo days. The vdeo sequences were convered o gray scale and normalzed o have sze pxels, zero mean, and un varance. TABLE Overall Rand Index on he Three Synhec Expermens for Sx Cluserng Algorhms The mxure of dynamc exures was used o organze hs daa se no fve clusers. Fg. 3a shows sx ypcal sequences for each of he fve clusers. These examples, and furher analyss of he sequences n each cluser, reveal ha he clusers are n agreemen wh classes frequenly used n he percepual caegorzaon of raffc: lgh raffc (spannng wo clusers), medum raffc, slow raffc, and sopped raffc ( raffc am ). Fgs. 3b, 3c, and 3d show a comparson beween he emporal evoluon of he cluser ndex and he average raffc speed. The laer was measured by he WSDOT wh an elecromagnec sensor (commonly known as a loop sensor) embedded n he hghway asphal, near he camera. The speed measuremens are shown n Fg. 3b, and he emporal evoluon of he cluser ndex s shown for K ¼ (Fg. 3c) and K ¼ 5 (Fg. 3d). Unforunaely, a precse comparson beween he speed measuremens and he vdeo s no possble because he daa orgnae from wo separae sysems, and he vdeo daa s no me samped wh fne enough precson. Noneheless, s sll possble o examne he correspondence beween he speed daa and he vdeo cluserng. For K ¼, he algorhm forms wo clusers ha correspond o fas-movng and slow-movng raffc. Smlarly, for K ¼ 5, he algorhm creaes wo clusers for fasmovng raffc and hree clusers for slow-movng raffc (whch correspond o medum, slow, and sopped raffc). 6.3 Moon Segmenaon Several expermens were conduced o evaluae he usefulness of dynamc exure mxures for vdeo segmenaon. In all cases, wo nalzaon mehods were consdered: The manual specfcaon of a rough nal segmenaon conour (referred o as DyexMxIC) and he componen splng sraegy of Secon 3. (DyexMxCS). The segmenaon resuls are compared wh hose produced by several segmenaon procedures prevously proposed n he leraure: he level-ses mehod n [] usng Isng models (Isng), generalzed PCA (GPCA) [18], and wo algorhms represenave of he sae-of-he-ar for radonal opcal-flow-based moon segmenaon. The frs mehod (NormCus) s based on normalzed cus [61] and he moon profle represenaon proposed n [61] and [6]. 4 The second (OpFlow) represens each pxel as a feaure-vecor conanng he average opcal flow over a 5 5 wndow and clusers he feaure-vecors usng he mean-shf algorhm [63]. Prelmnary evaluaon revealed varous lmaons of he dfferen echnques. For example, he opcal flow mehods canno deal wh he sochascy of mcroscopc 4. For hs represenaon, we used a pach of sze and a moon profle neghborhood of 5 5.

10 CHAN AND VASCONCELOS: MODELING, CLUSTERING, AND SEGMENTING VIDEO WITH MIXTURES OF DYNAMIC TEXTURES 917 Fg. 3. Cluserng of raffc vdeo: (a) sx ypcal sequences from he fve clusers, (b) speed measuremens from he hghway loop sensor over me, and he cluserng ndex over me for (c) wo clusers, and (d) fve clusers. exures (for example, waer and smoke), performng much beer for exures composed of nonmcroscopc prmves (for example, vdeo of crowded scenes composed of obecs, such as cars or pedesrans, movng a a dsance). On he oher hand, level ses and GPCA perform bes for mcroscopc exures. Due o hese lmaon, and for he sake of brevy, we lm he comparsons ha follow he mehods ha performed bes for each ype of vdeo under consderaon. In some cases, he expermens were also resrced by mplemenaon consrans. For example, he avalable mplemenaons of he level-ses mehod can only be appled o vdeo composed of wo exures. In all expermens, he vdeo are gray scale, and he vdeo paches are eher 5 5 or 7 7 pxels, dependng on he mage sze (see Table 3 for more deals). The paches are normalzed o have zero emporal mean and un varance. Unless oherwse specfed, he dmenson of he sae-space s n ¼ 10. Fnally, all segmenaons are posprocessed wh a 5 5 maory smoohng fler Segmenaon of Synhec Vdeo We sar wh he four synhec sequences suded n [15]: 1.. seam over an ocean background (ocean-seam), ocean wh wo regons roaed by 90 degrees, ha s, regons of dencal dynamcs bu dfferen appearance (ocean-appearance), TABLE 3 Vdeos n Moon Segmenaon Expermen K s he number of clusers. ocean wh wo regons of dencal appearance bu dfferen dynamcs (ocean-dynamcs), and 4. fre supermposed on an ocean background (ocean-fre). The segmenaons of he frs hree vdeos are shown n Fg. 4. Qualavely, he segmenaons produced by DyexMxIC and Isng ðn ¼ Þ are smlar, wh Isng performng slghly beer wh respec o he localzaon of he segmenaon borders. Alhough boh of hese mehods mprove on he segmenaons n [15], GPCA can only segmen ocean-seam, falng on he oher wo sequences. We nex consder he segmenaon of a sequence conanng ocean and fre (Fg. 5a). Ths sequence s more challengng because he boundary of he fre regon s no saonary (ha s, he regon changes over me as he flame evolves). Once agan, DyexMxIC ðn ¼ Þ and Isng ðn ¼ Þ produce comparable segmenaons (Fgs. 5b and 5c) and are capable of rackng he flame over me. We were no able o produce any meanngful segmenaon wh GPCA on hs sequence. In summary, boh DyexMxIC and Isng perform qualavely well on he sequences n [15] bu GPCA does no. Secon 6.3. wll compare hese algorhms quanavely usng a much larger daabase of synhec vdeo Segmenaon of Synhec Vdeo Daabase The segmenaon algorhms were evaluaed quanavely on a daabase of 300 synhec sequences. These conss of hree groups of 100 vdeos, wh each group generaed from a common ground-ruh emplae, for K ¼ f; 3; 4g. The segmens were randomly seleced from a se of 1 exures, whch ncluded grass, sea, bolng waer, movng escalaor, fre, seam, waer, and plans. Examples from he daabase can be seen n Fgs. 7a and 8a, along wh he nal conours provded o he segmenaon algorhms. All sequences were segmened wh DyexMxIC, DyexMxCS, and GPCA [18]. Due o mplemenaon lmaons, he algorhms n [], Isng, AR, and AR0 (AR whou mean) were appled only for K ¼. All mehods were run wh dfferen orders of he moon models ðnþ, whereas he remanng parameers for each algorhm were fxed hroughou he expermen. Performance was evaluaed wh he Rand ndex [59] beween segmenaon and ground ruh. In addon, wo baselne segmenaons were ncluded 1) Baselne Random, whch randomly assgns pxels and ) Baselne In, whch s he nal segmenaon (ha s, nal

11 918 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 5, MAY 008 Fg. 4. Segmenaon of synhec vdeo n [15]: (a) ocean-seam, (b) ocean-appearance, and (c) ocean-dynamcs. The frs column shows a vdeo frame and he nal conour, and he remanng columns show he segmenaons from DyexMxIC, Isng [], GPCA [18], and Marn dsance (from ha n [15]). Fg. 5. Segmenaon of ocean-fre [15]: (a) vdeo frames and he segmenaon usng (b) DyexMxIC and (c) Isng []. conour) provded o he algorhms. The daabase and segmenaon resuls are avalable n [58]. Fg. 6a shows he average Rand ndex resulng from he segmenaon algorhms for dfferen orders n, and Table 4 presens he bes resuls for each algorhm. For all K, DyexMxIC acheved he bes overall performance, ha s, he larges average Rand ndex. For K ¼, Isng also performs well (0.879) bu s nferor o DyexMxIC (0.916). The remanng algorhms performed poorly, wh GPCA performng close o random pxel assgnmen. Alhough he average Rand ndex quanfes he overall performance on he daabase, does no provde nsgh on he characerscs of he segmenaon falures, ha s, wheher here are many small errors or a few gross ones. To overcome hs lmaon, we have also examned he segmenaon precson of all algorhms. Gven a hreshold, segmenaon precson s defned as he percenage of segmenaons deemed o be correc wh respec o he hreshold, ha s, he percenage wh Rand ndex larger han. Fg. 6b plos he precson of he segmenaon algorhms for dfferen hreshold levels. One neresng observaon can be made when K ¼. In hs case, for a Rand hreshold of 0.95 (correspondng o olerance of abou 1 pxel error around he border), he precsons of DyexMxIC and Isng are, respecvely, 73 percen and 51 percen. On he oher hand, for very hgh hresholds (for example, 0.98), Isng has a larger precson (31 percen versus 14 percen of DyexMxIC). Ths suggess ha Isng s very good a fndng he exac boundary when nearly perfec segmenaons are possble bu s more prone o dramac segmenaon falures. On he oher hand, DyexMxIC s more robus bu no as precse near borders. An example of hese properes s shown n Fg. 7. The frs column presens a sequence for whch boh mehods work well, whereas he second column shows an example where Isng s more accurae near he border and where DyexMxIC sll fnds a good segmenaon. The hrd and fourh columns presen examples where DyexMxIC succeeds and Isng fals (for example, n he fourh column, Isng confuses he brgh escalaor edges wh he wave rpples). Fnally, boh mehods fal n he ffh column, due o he smlary of he background and foreground waer. Wh respec o nalzaon, s clear n Fg. 6 and Table 4 ha, even n he absence of an nal conour, he mxure of dynamc exures (DyexMxCS) ouperforms all oher mehods consdered, ncludng hose ha requre an nal conour (for example, Isng) and hose ha do no (for

12 CHAN AND VASCONCELOS: MODELING, CLUSTERING, AND SEGMENTING VIDEO WITH MIXTURES OF DYNAMIC TEXTURES 919 Fg. 6. Resuls on he synhec daabase: (a) average Rand ndex versus he order of he moon model ðnþ and (b) segmenaon precson for he bes n for each algorhm. Each column presens he resuls for, 3, or 4 segmens n he daabase. example, GPCA). Agan, hs ndcaes ha vdeo segmenaon wh he dynamc exure mxure s que robus. Comparng DyexMxIC and DyexMxCS, he wo mehods acheve equvalen performance for K ¼ wh DyexMxIC performng slghly beer for K ¼ 3 and K ¼ 4. Wh mulple exures, manual specfcaon of he nal conour reduces possble ambgues due o smlares beween pars of exures. An example s gven n he hrd column n Fg. 8, where he wo foreground exures are smlar, whereas he background exure could percepually be spl no wo regons (parcles movng upward and o he rgh, and hose movng upward and o he lef). In he absence of nal conour, DyexMxCS prefers hs alernave nerpreaon. Fnally, he frs wo columns n Fg. 8 show examples where boh mehods perform well, whereas n he fourh and ffh columns boh fal (for example, when wo waer exures are almos dencal). In summary, he quanave resuls on a large daabase of synhec vdeo exures ndcae ha he mxure of dynamc exures (boh wh or whou nal conour specfcaon) s superor o all oher vdeo exure segmenaon algorhms consdered. Alhough Isng also acheves TABLE 4 The Bes Average Rand Index for Each Segmenaon Algorhm on he Synhec Daabase The order of he model ðnþ s shown n parenhess. accepable performance, has wo lmaons: 1) requres an nal conour and ) can only segmen vdeo composed of wo regons. Fnally, AR and GPCA do no perform well on hs daabase Segmenaon of Real Vdeo We fnsh wh segmenaon resuls on fve real vdeo sequences, depcng a waer founan, hghway raffc, and pedesran crowds. Alhough a precse evaluaon s no possble, because here s no segmenaon ground ruh, he resuls are suffcenly dfferen o suppor a qualave rankng of he dfferen approaches. Fg. 9 llusraes he performance on he frs sequence, whch depcs a waer founan wh hree regons of dfferen dynamcs: waer flowng down a wall, fallng waer, and urbulen waer n a pool. Alhough DyexMxCS separaes he hree regons, GPCA does no produce a sensble segmenaon. 5 Ths resul confrms he observaons obaned wh he synhec daabase of he prevous secon. We nex consder macroscopc dynamc exures, n whch case segmenaon performance s compared agans ha of radonal moon-based soluons, ha s, he combnaon of normalzed-cus wh moon profles (NormCus) or opcal flow wh mean-shf cluserng (OpFlow). Fg. 10a shows a scene of hghway raffc, for whch DyexMxCS s he only mehod ha correcly segmens he vdeo no regons of raffc ha move away from he camera (he wo large regons on he rgh) and raffc ha move oward he camera (he regons on he lef). The only error s he spl of he ncomng raffc no wo regons and can be explaned by he srong perspecve effecs nheren o car moon oward he camera. Ths could be avoded by applyng an nverse perspecve mappng o he scene, bu we have no consdered any such geomerc ransformaons. Fg. 10b shows anoher example of segmenaon of vehcle raffc on a brdge. Traffc lanes of oppose drecon are correcly segmened near he camera 5. The oher mehods could no be appled o hs sequence because conans hree regons.

13 90 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 5, MAY 008 Fg. 7. Examples of segmenaon of synhec daabase ðk ¼ Þ: (a) a vdeo frame and he nal conour, segmenaon wh (b) DyexMxIC, and (c) Isng []. The Rand ndex ðrþ of each segmenaon s shown above he mage. Fg. 8. Examples of segmenaon of synhec daabase (K ¼ 3 and K ¼ 4): (a) a vdeo frame and he nal conour, (b) segmenaon usng DyexMxIC, and (c) DyexMxCS. The Rand ndex ðrþ of each segmenaon s shown above he mage. bu merged furher down he brdge. The algorhm also segmens he waer n he boom rgh of he mage bu assgns o he same cluser as he dsan raffc. Alhough no perfec, hese segmenaons are sgnfcanly beer han hose produced by he radonal represenaons. For boh NormCus and OpFlow, segmened regons end o exend over mulple lanes, ncomng and ougong raffc are merged, and he same lane s frequenly broken no a number of subregons. The fnal wo vdeos are of pedesran scenes. The frs scene, shown n Fg. 11a, conans sparse pedesran raffc, ha s, wh large gaps beween pedesrans. DyexMxCS Fg. 9. Segmenaon of a founan scene usng DyexMxCS and GPCA. (Fg. 11b) segmens people movng up he walkway from people movng down he walkway. The second scene (Fg. 11c) conans a large crowd movng up he walkway wh only a few people movng n he oppose drecon. Agan, DyexMxCS (Fg. 11d) segmened he groups movng n dfferen drecons, even n nsances where only one person s surrounded by he crowd and movng n he oppose drecon. Fgs. 11e and 11f show he segmenaons produced by he radonal mehods. The segmenaon produced by NormCus conans gross errors, for example, frame a he far end of he walkway. Alhough he segmenaon acheved wh OpFlow s more comparable o ha of DyexMxCS, ends o oversegmen he people movng down he walkway. Fnally, we llusrae he pon ha he mxure of dynamc exures can be used o effcenly segmen very long vdeo sequences. Usng he procedure dscussed n Secon 5., a connuous hour of pedesran vdeo was segmened wh he mxure model learned from ha n Fg. 11c and s avalable for vsualzaon n he companon Web se [58]. I should be noed ha hs segmenaon requred no renalzaon a

14 CHAN AND VASCONCELOS: MODELING, CLUSTERING, AND SEGMENTING VIDEO WITH MIXTURES OF DYNAMIC TEXTURES 91 Fg. 10. Segmenaon of raffc scenes: (a) hghway raffc and (b) vehcle raffc on brdge. The lef column shows a frame from he orgnal vdeos, whereas he remanng columns show he segmenaons. Fg. 11. Segmenaon of wo pedesran scenes: (a) pedesran scene wh sparse raffc, (b) he segmenaon by DyexMxCS, (c) pedesran scene wh heavy raffc and he segmenaons by (d) DyexMxCS, (e) NormCus, and (f) OpFlow. any pon or any oher ype of manual supervson. We consder hs a sgnfcan resul, gven ha hs sequence conans a far varably of raffc densy, varous oulyng evens (for example, bcycles, skaeboarders, or even small vehcles ha pass hrough, pedesrans ha change course or sop o cha, and so forh), and varable envronmenal condons (such as varyng clouds shadows). None of hese facors appear o nfluence he performance of he DyexMxCS algorhm. To he bes of our knowledge, hs s he frs me ha a coheren vdeo segmenaon, over a me span of hs scale, has been repored for a crowded scene. 7 CONCLUSIONS In hs work, we have suded he mxure of dynamc exures, a prncpled probablsc exenson of he dynamc exure model. Alhough a dynamc exure models a sngle vdeo sequence as a sample from a lnear dynamc sysem, a

15 9 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 5, MAY 008 mxure of dynamc exures models a collecon of sequences as samples from a se of lnear dynamc sysems. We derved an exac EM algorhm for learnng he parameers of he model from a se of ranng vdeo and explored he connecons beween he model and oher lnear sysem models, such as facor analyss, mxures of facor analyzers, and swchng lnear sysems. Through exensve vdeo cluserng and segmenaon expermens, we have also demonsraed he effcacy of he mxure of dynamc exures for modelng vdeo, boh holscally and locally (pach-based represenaons). In parcular, has been shown ha he mxure of dynamc exures s a suable model for smulaneously represenng he localzed moon and appearance of a varey of vsual processes (for example, fre, waer, seam, cars, and people), and ha he model provdes a naural framework for cluserng such processes. In he applcaon of moon segmenaon, he expermenal resuls ndcae ha he mxure of dynamc exures provdes beer segmenaons han oher sae-of-he-ar mehods, based on eher dynamc exures or on radonal represenaons. Some of he resuls, namely, he segmenaon of pedesran scenes, sugges ha he dynamc exure mxure could be he bass for he desgn of compuer vson sysems capable of acklng problems, such as he monorng and survellance of crowded envronmens, whch currenly have grea soceal neres. There are also some ssues ha we leave open for fuure work. One example s how o ncorporae, n he dynamc exure mxure framework, some recen developmens n asympocally effcen esmaors based on nonerave subspace mehods [64]. By usng such esmaors n he M-sep of he EM algorhm, may be possble o reduce he number of hdden varables requred n he E-sep and consequenly mprove he convergence properes of EM. I s currenly unclear f he opmaly of hese esmaors s compromsed when he nal sae has arbrary covarance or when he LDS s learned from mulple sample pahs, as s he case for dynamc exure mxures. Anoher open queson s ha of he denfably of an LDS, when he nal sae has arbrary covarance. I s well known, n he sysem denfcaon leraure, ha he parameers of an LDS can only be denfed from a sngle spao-emporal sample f he covarance of he nal condon sasfes a Lyapunov condon. In he absence of denfably, he soluon may no be unque, may no exs, learnng may no converge, or he esmaes may no be conssen. I s mporan o noe ha none of hese problems are of grea concern for he mehods dscussed n hs paper. For example, s well known ha EM s guaraneed o converge o a local maxmum of he lkelhood funcon and produces asympocally conssen parameer esmaes. These properes are no conngen on he denfably of he LDS componens. Alhough he local maxma of he lkelhood could be rdges (ha s, no lmed o pons bu suppored by manfolds), n whch case, he opmal componen parameers would no be unque, here would be no consequence for segmenaon or cluserng as long as all compuaons are based on lkelhoods (or oher probablsc dsance measures such as he Kullback-Lebler dvergence). Nonunqueness could, neverheless, be problemac for procedures ha rely on drec comparson of he componen parameers (for example, based on her eucldean dsances), whch we do no advse. In any case, would be neresng o nvesgae more horoughly he denfably queson. The fac ha we have no experenced convergence speed problems, n he exensve expermens dscussed above, ndcaes ha lkelhood rdges are unlkely. In fuure work, we wll aemp o undersand hs queson more formally. APPENDIX A EM ALGORITHM FOR THE MIXTURE OF DYNAMIC TEXTURES Ths appendx presens he dervaon of he EM algorhm for he mxure of dynamc exures. In parcular, he complee-daa log-lkelhood funcon, he E-sep, and he M-sep are derved n he remander of he appendx. A.1 Log-Lkelhood Funcons We sar by obanng he log-lkelhood of he compleedaa (see Table 1 for noaon). Usng (11) and he ndcaor varables z ;, he complee-daa log-lkelhood s ðx; Y ; ZÞ ¼ XN ¼1 ¼ XN ¼1 ¼ X Y ¼ ¼ X ; þ X þ X ¼ ¼1 log p x ðþ ;y ðþ ;z ðþ ð30þ log YK h p x ðþ ;y ðþ ;z ðþ z; ¼ ð31þ ¼1 " z ; log p p! x ðþ 1 zðþ ¼! Y x ðþ x ðþ 1 ;zðþ ¼ p ¼1 y ðþ h z ; log þ log p x ðþ 1 zðþ ¼ log p x ðþ 1 ;zðþ ¼ x ðþ # log p y ðþ x ðþ ;z ðþ ¼ : x ðþ ;z ðþ ¼ ð3þ!# ð33þ Noe ha, from (8)-(10), he sums of he log-condonal probably erms are of he form X X 1 a ; ; ¼ 0 log Gðb ;c ; ;M Þ¼ a ; d ð 1 0 þ 1Þ log X ; 1 X X 1 a ; kb c ; k M ; ¼ þ 1 X a ; log M : ; ð34þ Snce he frs erm on he rgh-hand sde of hs equaon does no depend on he parameers of he dynamc exure mxure, does no affec he maxmzaon performed n he M-sep and can, herefore, be dropped. Subsung he approprae parameers for b, c ;, and M, we have

16 CHAN AND VASCONCELOS: MODELING, CLUSTERING, AND SEGMENTING VIDEO WITH MIXTURES OF DYNAMIC TEXTURES 93 ðx; Y ; ZÞ ¼ X z ; log 1 X z ; log S ; ; 1 X z ; x ðþ 1 S 1 X z ; log Q ; ; 1 X X z ; ðþ x A x ðþ 1 Q ; ¼ 1 X X z ; ðþ y C x ðþ R X z ; log R : ; ¼1 ; ð35þ Defnng he random varables P ðþ ; ¼ x ðþ ðx ðþ Þ T and P ðþ ; 1 ¼ x ðþ ðx ðþ 1 ÞT and expandng he Mahalanobs dsance erms, he log-lkelhood becomes (15). A. E-Sep The E-sep of he EM algorhm s o ake he expecaon of (15) condoned on he observed daa and he curren parameer esmaes ^, as n (13). We noe ha each erm of ðx; Y ; ZÞ s of he form z ; fðx ðþ ;y ðþ Þ, for some funcons f of x ðþ and y ðþ, and s expecaon s E X;ZY z ; fðx ðþ ;y ðþ Þ ð36þ ¼ E ZY E XY;Z z ; fðx ðþ ;y ðþ Þ ð37þ ¼ E z ðþ y E ðþ x ðþ y ðþ ;z z ;fðx ðþ ;y ðþ Þ ð38þ ðþ ¼ pðz ; ¼ 1y ðþ ÞE x ðþ y ðþ ;z ðþ ¼ fðx ðþ ;y ðþ Þ ; ð39þ where (38) follows from he assumpon ha he observaons are ndependen. For he frs erm of (39), pðz ; ¼ 1y ðþ Þ s he poseror probably of z ðþ ¼ gven he observaon y ðþ ^z ; ¼ p z ; ¼ 1y ðþ ¼ p z ðþ ¼ y ðþ ð40þ pðy ðþ z ðþ ¼ Þ ¼ P K k¼1 kpðy ðþ z ðþ ¼ kþ : ð41þ The funcons fðx ðþ ;y ðþ Þ are a mos quadrac n x ðþ. Hence, he second erm of (39) only depends on he frs and second momens of he saes condoned on y ðþ and componen (18)-(0) and are compued, as descrbed n Secon 3.1. Fnally, he Q funcon (16) s obaned by frs replacng he random varables z ;, ðz ; x ðþ Þ, ðz ; P ðþ ; Þ, and ðz ;P ðþ ; 1Þ n he complee-daa log-lkelhood (15) wh he correspondng expecaons ^z ;, ð^z ; ^x ðþ Þ, ð^z ðþ ; ^P ; Þ, and ð^z ðþ ; ^P ; 1Þ, and hen defnng he aggregaed expecaons (17). A.3 M-Sep In he M-sep of he EM algorhm (14), he reparameerzaon of he model s obaned by maxmzng he Q funcon by akng he paral dervave wh respec o each parameer and seng o zero. The maxmzaon problem wh respec o each parameer appears n wo common forms. The frs s a maxmzaon wh respec o a square marx X X ¼ argmax X 1 r X 1 A b log X: ð4þ Maxmzng by akng he dervave and seng o zero yelds he followng r b X 1 A log X ¼0; ð43þ 1 X T A T X T b X T ¼ 0; ð44þ A T bx T ¼ 0; ð45þ ) X ¼ 1 A: ð46þ b The second form s a maxmzaon problem wh respec o a marx X of he form X ¼ argmax 1 X r Dð BXT XB T þ XCX T Þ ; ð47þ where D and C are symmerc and nverble marces. The maxmum s gven r Dð BXT XB T þ XCX T Þ ¼ 0; ð48þ 1 ð DB DT B þ D T XC T þ DXCÞ ¼0; ð49þ DB DXC ¼ 0; ð50þ ) X ¼ BC 1 : ð51þ The opmal parameers are found by collecng he relevan erms n (16) and maxmzng. A.3.1 Observaon Marx C ¼ argmax 1 h r R 1 C T C C T þ C C T : Ths s of he form n (47), hence, he soluon s gven by C ¼ ð Þ 1. A.3. Observaon Nose Covarance R ¼ argmax ^N R log R 1 ð5þ h r R 1 C T C T þ C C T : Ths s of he form n (4), hence, he soluon s R ¼ 1 ^N C T C T þ C C T ð53þ ¼ 1 ^N C T ; ð54þ where (54) follows from subsung for he opmal value C. A.3.3 Sae-Transon Marx A ¼ argmax 1 A h r Q 1 A T A T þ A A T Ths s of he form n (47); hence, A ¼ ð Þ 1. :

17 94 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 5, MAY 008 A.3.4 Sae Nose Covarance Q ¼ argmax Q h 1 r Q 1 ð 1Þ ^N log Q A T A T þ A A T : ð55þ Ths s of he form n (4), hence, he soluon can be compued as Q ¼ 1 ð 1Þ ^N A T A 1 ¼ ð 1Þ ^N A T ; T þ A A T ð56þ where (56) follows from subsung for he opmal value A. A.3.5 Inal Sae Mean ¼ argmax 1 h r S 1 T T þ ^N T : Ths s of he form n (47), hence, he soluon s gven by ¼ 1^N. A.3.6 Inal Sae Covarance S ¼ argmax ^N S log S 1 h r S 1 T T þ ^N T : ð57þ Ths s of he form n (4), hence, he soluon s gven by S ¼ 1^N T T þ ^N T ð58þ ¼ 1^N ð ÞT : ð59þ where (59) follows from subsung for he opmal value. A.3.7 Class Probables A Lagrangan mulpler s used o enforce ha f g sum o 1 X ¼ argmax ^N log þ X! 1 ; ð60þ ; where ¼f 1 ; ; K g. The opmal value s ¼ ^N N. APPENDIX B KALMAN SMOOTHING FILTER The Kalman smoohng fler [4], [5] esmaes he mean and covarance of he sae x of an LDS, condoned on he enre observed sequence fy 1 ;...;y g. I can also be used o effcenly compue he log-lkelhood of he observed sequence. Defne he expecaons condoned on he observed sequence from me ¼ 1 o ¼ s ^x s ¼ E xy 1 ;...;y s ðx Þ; ð61þ ^V s ¼ E xy1;...;y s ðx ^x s Þðx ^x s ÞT ; ð6þ ^V ; 1 s ¼ E xy 1 ;...;y s ðx ^x s Þðx 1 ^x s 1 ÞT ; ð63þ hen he mean and covarances condoned on he enre observed sequence are ^x, ^V, and ^V ; 1. The esmaes are calculaed usng a se of recursve equaons: for ¼ 1;...; ^V 1 ¼ A ^V 1 1 AT þ Q; ð64þ K ¼ ^V 1 C T C ^V 1 C T 1; þ R ð65þ ^V ^x 1 ¼ ^V 1 ¼ A^x 1 1 ; ^x ¼ ^x 1 K C ^V 1 ; ð66þ þ K y C ^x 1 ð67þ ; ð68þ where he nal condons are ^x 0 1 ¼ and ^V 1 0 ¼ S. The esmaes ^x and ^V are obaned wh he backward recursons. For ¼ ;...; 1 J 1 ¼ ^V 1 1 AT ð ^V 1 Þ 1 ; ð69þ ^x 1 ¼ ^x 1 1 þ J 1ð^x A^x 1 1 Þ; ð70þ ^V 1 ¼ ^V 1 1 þ J 1ð ^V ^V 1 ÞJ 1 T : ð71þ The covarance ^V ; 1 s compued recursvely, for ¼ ;...; ^V 1; ¼ ^V 1 1 JT þ J 1ð ^V ; 1 A ^V 1 1 ÞJT ð7þ wh nal condon ^V ; 1 ¼ðI K CÞA ^V 1 1. Fnally, he daa log-lkelhood can also be compued effcenly usng he nnovaons form [5] log pðy 1 Þ¼X log pðy y1 1 Þ ð73þ ¼1 ¼ X ¼1 log Gðy ;C^x 1 ;C^V 1 C T þ RÞ: ð74þ If R s an..d. or dagonal covarance marx (for example, R ¼ ri m ), hen he fler can be compued effcenly usng he marx nverson lemma. ACKNOWLEDGMENTS The auhors hank Ganfranco Doreo and Sefano Soao for he synhec sequences n [15], [16], he Washngon Sae DOT [60] for he vdeos of hghway raffc, Danel Daley for he loop-sensor daa, Rene Vdal, Dheera Sngarau, and Ayeh Ghoreysh for code n [18], [], Pedro Moreno for helpful dscussons, and he anonymous revewers for nsghful commens. Ths work was funded by he US Naonal Scence Foundaon Award IIS and NSF IGERT Award DGE REFERENCES [1] B.K.P. Horn, Robo Vson. McGraw-Hll Book, [] B. Horn and B. Schunk, Deermnng Opcal Flow, Arfcal Inellgence, vol. 17, pp , [3] B. Lucas and T. Kanade, An Ierave Image Regsraon Technque wh an Applcaon o Sereo Vson, Proc. DARPA Image Undersandng Workshop, pp , [4] J. Barron, D. Flee, and S. Beauchemn, Performance of Opcal Flow Technques, In l J. Compuer Vson, vol. 1, pp , 1994.

18 CHAN AND VASCONCELOS: MODELING, CLUSTERING, AND SEGMENTING VIDEO WITH MIXTURES OF DYNAMIC TEXTURES 95 [5] P. Anandan, J. Bergen, K. Hanna, and R. Hngoran, Herarchcal Model-Based Moon Esmaon, Moon Analyss and Image Sequence Processng, pp. 1-, [6] J. Wang and E. Adelson, Represenng Movng Images wh Layers, IEEE Trans. Image Processng, vol. 3, no. 5, pp , [7] H. Sawhney and S. Ayer, Compac Represenaons of Vdeos hrough Domnan and Mulple Moon Esmaon, IEEE Trans. Paern Analyss and Machne Inellgence, vol. 18, no. 8, pp , Aug [8] M. Hansen, P. Anandan, K. Dana, G. Wal, and P. Bur, Real-Tme Scene Sablzaon and Mosac Consrucon, Proc. DARPA Image Undersandng Workshop, pp , [9] M. Isard and A. Blake, Condensaon Condonal Densy Propagaon for Vsual Trackng, In l J. Compuer Vson, vol. 9, no. 1, pp. 5-8, [10] M. Iran, B. Rousso, and S. Peleg, Deecng and Trackng Mulple Movng Obecs Usng Temporal Inegraon, Proc. European Conf. Compuer Vson, pp. 8-87, 199. [11] D. Comancu, V. Ramesh, and P. Meer, Kernel-Based Obec Trackng, IEEE Trans. Paern Analyss and Machne Inellgence, vol. 5, no. 5, pp , May 003. [1] S. Soao, G. Doreo, and Y.N. Wu, Dynamc Texures, Proc. IEEE In l Conf. Compuer Vson, pp , 001. [13] G. Doreo, A. Chuso, Y.N. Wu, and S. Soao, Dynamc Texures, In l J. Compuer Vson, vol. 51, no., pp , 003. [14] A.W. Fzgbbon, Sochasc Rgdy: Image Regsraon for Nowhere-Sac Scenes, Proc. In l Conf. Compuer Vson, vol. 1, pp , 001. [15] G. Doreo, D. Cremers, P. Favaro, and S. Soao, Dynamc Texure Segmenaon, Proc. In l Conf. Compuer Vson, vol., pp , 003. [16] P. Sasan, G. Doreo, Y. Wu, and S. Soao, Dynamc Texure Recognon, Proc. Compuer Vson and Paern Recognon, vol., pp , 001. [17] A.B. Chan and N. Vasconcelos, Probablsc Kernels for he Classfcaon of Auo-Regressve Vsual Processes, Proc. IEEE Conf. Compuer Vson and Paern Recognon, vol. 1, pp , 005. [18] R. Vdal and A. Ravchandran, Opcal Flow Esmaon and Segmenaon of Mulple Movng Dynamc Texures, Proc. IEEE Conf. Compuer Vson and Paern Recognon, vol., pp , 005. [19] A.B. Chan and N. Vasconcelos, Layered Dynamc Texures, Advances of Neural Informaon Processng (NIPS) 18, pp , 006. [0] A.B. Chan and N. Vasconcelos, Mxures of Dynamc Texures, Proc. IEEE In l Conf. Compuer Vson, vol. 1, pp , 005. [1] L. Cooper, J. Lu, and K. Huang, Spaal Segmenaon of Temporal Texure Usng Mxure Lnear Models, Proc. IEEE In l Conf. Compuer Vson Dynamcal Vson Workshop, 005. [] A. Ghoreysh and R. Vdal, Segmenng Dynamc Texures wh Isng Descrpors, ARX Models and Level Ses, Proc. European Conf. Compuer Vson Dynamcal Vson Workshop, 006. [3] S.M. Kay, Fundamenals of Sascal Sgnal Processng: Esmaon Theory. Prence Hall, [4] S. Rowes and Z. Ghahraman, A Unfyng Revew of Lnear Gaussan Models, Neural Compuaon, vol. 11, no., pp , [5] R.H. Shumway and D.S. Soffer, An Approach o Tme Seres Smoohng and Forecasng Usng he EM Algorhm, J. Tme Seres Analyss, vol. 3, no. 4, pp , 198. [6] P.V. Overschee and B.D. Moor, N4SID: Subspace Algorhms for he Idenfcaon of Combned Deermnsc-Sochasc Sysems, Auomaca, vol. 30, pp , [7] F.V. Jensen, Bayesan Neworks and Decson Graphs. Sprnger, 001. [8] A.P. Dempser, N.M. Lard, and D.B. Rubn, Maxmum Lkelhood from Incomplee Daa va he EM Algorhm, J. Royal Sascal Soc. B, vol. 39, pp. 1-38, [9] S. Young, G. Evermann, M. Gales, T. Han, D. Kershaw, X.Y. Lu, G. Moore, J. Odell, D. Ollason, D. Povey, V. Valchev, and P. Woodland, The HTK Book. Cambrdge Unv. Eng. Dep., 006. [30] D.T. Magll, Opmal Adapve Esmaon of Sampled Sochasc Processes, IEEE Trans. Auomac Conrol, vol. 10, no. 4, pp , [31] D.G. Lanos, Paronng: A Unfyng Framework for Adapve Sysems, I: Esmaon; II: Conrol, Proc. IEEE, vol. 64, no. 8, pp , [3] K.S. Narendra and J. Balakrshnan, Adapve Conrol Usng Mulple Models, IEEE Trans. Auomac Conrol, vol. 4, no., pp , [33] R.G. Brown, A New Look a Magll Adapve Fler as a Praccal Means of Mulple Hypohess Tesng, IEEE Trans. Crcus and Sysems, vol. 30, no. 10, pp , [34] V. Dgalaks, J.R. Rohlcek, and M. Osendorf, ML Esmaon of a Sochasc Lnear Sysem wh he EM Algorhm and Is Applcaon o Speech Recognon, IEEE Trans. Speech and Audo Processng, vol. 1, no. 4, pp , [35] Z. Ghahraman and G. Hnon, Parameer Esmaon for Lnear Dynamcal Sysems, Techncal Repor CRG-TR-96-, Dep. of Compuer Scence, Unv. of Torono, [36] Z. Ghahraman and G. Hnon, The EM Algorhm for Mxures of Facor Analyzers, Techncal Repor CRG-TR-96-1, Dep. of Compuer Scence, Unv. of Torono, [37] R. Shumway and D. Soffer, Dynamc Lnear Models wh Swchng, J. Am. Sascal Assoc., vol. 86, pp , [38] Y. Wu, G. Hua, and T. Yu, Swchng Observaon Models for Conour Trackng n Cluer, Proc. Compuer Vson and Paern Recognon, pp , 003. [39] M. Isard and A. Blake, A Mxed-Sae Condensaon Tracker wh Auomac Model-Swchng, Proc. In l Conf. Compuer Vson, pp , [40] V. Pavlovc, B.J. Frey, and T.S. Huang, Tme-Seres Classfcaon Usng Mxed-Sae Dynamc Bayesan Neworks, Proc. IEEE Conf. Compuer Vson and Paern Recognon, [41] V. Pavlovc, J. Rehg, and J. MacCormck, Learnng Swchng Lnear Models of Human Moon, Advances n Neural Informaon Processng Sysems 13, 000. [4] C.-J. Km, Dynamc Lnear Models wh Markov-Swchng, J. Economercs, vol. 60, pp. 1-, [43] S.M. Oh, J.M. Rehg, T. Balch, and F. Dellaer, Learnng and Inference n Paramerc Swchng Lnear Dynamc Sysems, Proc. IEEE In l Conf. Compuer Vson, vol., pp , 005. [44] Z. Ghahraman and G.E. Hnon, Varaonal Learnng for Swchng Sae-Space Models, Neural Compuaon, vol. 1, no. 4, pp , 000. [45] T.W. Lao, Cluserng of Tme Seres Daa A Survey, Paern Recognon, vol. 38, pp , 005. [46] Y. Kakzawa, R.H. Shumway, and M. Tanguch, Dscrmnaon and Cluserng for Mulvarae Tme Seres, J. Am. Sascal Assoc., vol. 93, no. 441, pp , [47] A. Snghal and D.E. Seborg, Cluserng of Mulvarae Tme- Seres Daa, Proc. Am. Conrol Conf., vol. 5, pp , 00. [48] Y. Xong and D.-Y. Yeung, Tme Seres Cluserng wh ARMA Mxures, Paern Recognon, vol. 37, pp , 004. [49] D.A. Forsyh and J. Ponce, Compuer Vson: A Modern Approach. Prence Hall, 00. [50] R. Duda, P. Har, and D. Sork, Paern Classfcaon. John Wley & Sons, 001. [51] C. Sauffer and E. Grmson, Learnng Paerns of Acvy Usng Real-Tme Trackng, IEEE Trans. Paern Analyss and Machne Inellgence, vol., no. 8, pp , Aug [5] A. Jepson and M. Black, Mxure Models for Opcal Flow Compuaon, Proc. Compuer Vson and Paern Recognon, pp , [53] Y. Wess, Smoohness n Layers: Moon Segmenaon Usng Nonparamerc Mxure Esmaon, Proc. In l Conf. Compuer Vson, pp , [54] N. Vasconcelos and A. Lppman, Emprcal Bayesan Moon Segmenaon, IEEE Trans. Paern Analyss and Machne Inellgence, vol. 3, no., pp. 17-1, Feb [55] B. Frey and N. Joc, Esmang Mxure Models of Images and Inferrng Spaal Transformaons Usng he EM Algorhm, Proc. IEEE Conf. Compuer Vson and Paern Recognon, pp , [56] C. Carson, S. Belonge, H. Greenspan, and J. Malk, Blobworld: Color- and Texure-Based Image Segmenaon Usng EM and Is Applcaon o Image Queryng and Classfcaon, IEEE Trans. Paern Analyss and Machne Inellgence, vol. 4, no. 8, pp , Aug. 00. [57] N. Vasconcelos, Mnmum Probably of Error Image Rereval, IEEE Trans. Sgnal Processng, vol. 5, no. 8, pp , 004. [58] Mxures of Dynamc Texures, hp:// proecs/moondyex, 008.

19 96 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 5, MAY 008 [59] L. Huber and P. Arabe, Comparng Parons, J. Classfcaon, vol., pp , [60] Washngon Sae Dep. of Transporaon, hp:// wa.gov, 008. [61] J. Sh and J. Malk, Normalzed Cus and Image Segmenaon, IEEE Trans. Paern Analyss and Machne Inellgence, vol., no. 8, pp , Aug [6] J. Sh and J. Malk, Moon Segmenaon and Trackng Usng Normalzed Cus, Proc. In l Conf. Compuer Vson, pp , [63] D. Comancu and P. Meer, Mean Shf: A Robus Approach oward Feaure Space Analyss, IEEE Trans. Paern Analyss and Machne Inellgence, vol. 4, no. 5, pp , May 00. [64] D. Bauer, Comparng he CCA Subspace Mehod o Pseudo Maxmum Lkelhood Mehods n he Case of No Exogenous Inpus, J. Tme Seres Analyss, vol. 6, pp , 005. Anon B. Chan receved he BS and MEng degrees n elecrcal engneerng from Cornell Unversy n 000 and 001, respecvely. He s currenly workng oward he PhD degree a he Unversy of Calforna, San Dego. In 005, he was a summer nern a Google, New York. From 001 o 003, he was a vsng scens n he Vson and Image Analyss Lab a Cornell. He s a suden member of he IEEE. Nuno Vasconcelos receved he lcencaura n elecrcal engneerng and compuer scence from he Unversdade do Poro, Porugal, n 1988, and he MS and PhD degrees from he Massachuses Insue of Technology n 1993 and 000, respecvely. From 000 o 00, he was a member of he research saff a he Compaq Cambrdge Research Laboraory, whch n 00 became he HP Cambrdge Research Laboraory. In 003, he oned he Elecrcal and Compuer Engneerng Deparmen, Unversy of Calforna, San Dego, where he heads he Sascal Vsual Compung Laboraory. He s he recpen of a US Naonal Scence Foundaon CAREER Award, a Hellman Fellowshp, and has auhored more han 50 peer-revewed publcaons. Hs work spans varous areas, ncludng compuer vson, machne learnng, sgnal processng and compresson, and mulmeda sysems. He s a member of he IEEE.. For more nformaon on hs or any oher compung opc, please vs our Dgal Lbrary a