Layered Dynamic Textures

Transcription

1 Layered Dynamc Texures Anon B Chan and Nuno Vasconcelos Deparmen of Elecrcal and Compuer Engneerng Unversy of Calforna, San Dego abchan@ucsdedu, nuno@eceucsdedu Absrac A dynamc exure s a generave model for vdeo ha reas he vdeo as a sample from spao-emporal sochasc process One problem assocaed wh he dynamc exure s ha canno model vdeo where here are regons of moon wh dfferen dynamcs, eg a scene wh smoke and fre In hs work, we nroduce he layered dynamc exure model, whch addresses hs problem by nroducng a separae sae process for each regon of moon We derve he EM algorhm for learnng he parameers of he model, and demonsrae he effcacy of he proposed model for he asks of segmenaon and synhess of vdeo Inroducon Tradonal moon represenaons, based on opc flow, are nherenly local and have sgnfcan dffcules when faced wh aperure problems and nose The classcal soluon o hs problem s o regularze he opcal flow feld [ 4], bu hs nroduces undesrable smoohng across moon edges or regons where he moon s, by defnon, no smooh (eg vegeaon n oudoors scenes) More recenly, here have been varous aemps o model vdeo as a superposon of layers subec o homogeneous moon Whle layered represenaons exhbed sgnfcan promse n erms of combnng he advanages of regularzaon (use of global cues o deermne local moon) wh he flexbly of local represenaons (lle undue smoohng), hs poenal has so far no fully maeralzed One of he man lmaons s her dependence on paramerc moon models, such as affne ransforms, whch assume a pece-wse planar world ha rarely holds n pracce [5, 6] In fac, layers are usually formulaed as cardboard models of he world ha are warped by such ransformaons and hen sched o form he frames n a vdeo sream [5] Ths severely lms he ypes of vdeo ha can be synheszed: whle layers showed mos promse as models for scenes composed of ensembles of obecs subec o homogeneous moon (eg leaves blowng n he wnd, a flock of brds, a pcke fence, or hghway raffc), very lle progress has so far been demonsraed n acually modelng such scenes Recenly, here has been more success n modelng complex scenes as dynamc exures or, more precsely, samples from sochasc processes defned over space and me [7 0] Ths work has demonsraed ha modelng boh he dynamcs and appearance of vdeo as sochasc quanes leads o a much more powerful generave model for vdeo han ha of a cardboard fgure subec o paramerc moon In fac, he dynamc exure model has shown a surprsng ably o absrac a wde varey of complex paerns of moon and appearance no a smple spao-emporal model One maor curren lmaon of he dynamc exure framework, however, s s nably o accoun for vsual processes conssng of mulple, cooccurrng, dynamc exures For example, a flock of brds flyng n fron of a waer founan, hghway raffc movng a dfferen speeds, vdeo conanng boh rees n he background and people n he foreground, and so forh In such cases, he exsng dynamc exure model s nherenly ncorrec, snce mus represen mulple moon felds wh a sngle dynamc process In hs work, we address hs lmaon by nroducng a new generave model for vdeo, whch we denoe by he layered dynamc exure (LDT) Ths consss of augmenng he dynamc exure wh a dscree hdden varable, ha enables he assgnmen of dfferen dynamcs o dfferen regons of he vdeo Condoned on he sae of hs hdden varable, he vdeo s hen modeled as a smple dynamc exure By nroducng a shared dynamc represenaon for all he pxels n he same regon, he new model s a layered represenaon When compared wh radonal layered models, replaces he process of layer formaon based on warpng of cardboard fgures wh one based on samplng from he generave model (for boh dynamcs and appearance) provded by he dynamc exure Ths enables a much rcher vdeo represenaon Snce each layer s a dynamc exure, he model can also be seen as a mul-sae dynamc exure, whch s capable of assgnng dfferen dynamcs and appearance o dfferen mage regons Recenly, some of he lmaons of he dynamc ex-

2 y Z z z 2 z 3 z 4 x ( ) z 5 z 6 z 7 z 8 x x 2 x 3 x 4 x ( K ) y z 9 z 0 z z 2 y y 2 y 3 y 4 y N z 3 z 4 z 5 z 6 (a) (b) (c) Fgure (a) The graphcal model for he dynamc exure x s he hdden sae a me, and y s he observed frame a me ; (b) The graphcal model for he layered dynamc exure y s an observed pxel process and s a hdden sae process Z s he collecon of layer assgnmen varables z ha assgns each pxels o one of he sae processes, and s modeled as an MRF; (c) An example of a 4 4MRF used for layer assgnmen ure were also addressed n [] The layered formulaon now proposed has varous dfferences wh respec o hs work Frs, enables probablsc pxel assgnmens, as opposed o he hard assgnmens of [] Second, he learnng mehod of [] s exac only when he nose erm of he appearance componen of he dynamc exure s zero As usual n compuer vson, allowng hs erm o be dfferen han zero s mporan no only because enables he processng of vdeo wh nose, bu also because nroduces flexbly wh respec o model msmaches Fnally, he model of [] does no enforce spaal conssency of he assgnmens of pxels o regons We show ha hs can be naurally done wh he layered dynamc exure model, and can lead o sgnfcan mprovemens of segmenaon accuracy when he dfferen regons have smlar dynamcs The paper s organzed as follows In Secon 2, we nroduce he layered dynamc exure model In Secon 3 we presen he EM algorhm for learnng he model from ranng daa Fnally, n Secon 4 we presen an expermenal evaluaon n he conex of segmenaon and vdeo synhess 2 Layered dynamc exures We sar wh a bref revew of dynamc exures, and hen nroduce he layered dynamc exure model 2 Dynamc exure A dynamc exure [7] s a generave model for vdeo, whch reas he vdeo as a sample from a lnear dynamcal sysem The model, shown n Fgure (a), separaes he vsual componen and he underlyng dynamcs no wo sochasc processes The dynamcs of he vdeo are represened as a me-evolvng sae process x R n, and he appearance of he frame y R m s a lnear funcon of he curren sae vecor wh some observaon nose Formally, he sysem s descrbed by { x = Ax + Bv y = Cx + rw () where A R n n s a ranson marx, C R m n a ransformaon marx, Bv d N(0, Q,) and rw d N(0, ri m ) he sae and observaon nose processes parameerzed by B R n n and r R, and he nal sae x 0 R n s a consan One nerpreaon of he dynamc exure model s ha he columns of C are he prncpal componens of he vdeo frames, and he sae vecors are he PCA coeffcens for each vdeo frame Ths s he case when he model s learned wh he mehod of [7] An alernave nerpreaon consders a sngle pxel as evolves over me Each coordnae of he sae vecor x defnes a one-dmensonal random raecory n me A pxel s hen represened as a weghed sum of random raecores, where he weghng coeffcens are conaned n he correspondng row of C Ths s analogous o he dscree Fourer ransform n sgnal processng, where a sgnal s represened as a weghed sum of complex exponenals alhough, for he dynamc exure, he raecores are no necessarly orhogonal Ths nerpreaon llusraes he ably of he dynamc exure o model he same moon under dfferen nensy levels (eg cars movng from he shade no sunlgh) by smply scalng he rows of C Regardless of nerpreaon, he smple dynamc exure model has only one sae process, whch resrcs he effcacy of he model o vdeo where he moon s homogenous 22 Layered dynamc exures We now nroduce he layered dynamc exure (LDT), whch s shown n Fgure (b) The model addresses he lmaons of he dynamc exure by relyng on a se of sae processes X = { } K = o model dfferen vdeo dynamcs The layer assgnmen varable z assgns pxel y o one of he sae processes (layers), and condoned on he layer assgnmens, he pxels n he same layer are modeled as a dynamc exure In addon, he collecon of layer assgnmens Z = {z } N = s modeled as a Markov random feld (MRF) o ensure spaal layer conssency (an example s shown n Fgure (c)) The lnear sysem equaons for

3 he layered dynamc exure are { = A () + B() v () {,,K} y, = C (z) x (z) + r (z) w, {,,N} where C () R n s he ransformaon from he hdden sae o he observed pxel doman for each pxel y and each layer, he nose parameers are B () R n n and r () R, he d nose processes are w, d N(0, ) and v () d N(0, I n ), and he nal saes are drawn from N(µ (), S () ) As a generave model, he layered dynamc exure assumes ha he sae processes X and he layer assgnmens Z are ndependen, e he layer moon s ndependen of layer locaon, and vce versa Gven he layer assgnmens, he LDT s a collecon of dynamc exures over dfferen regons of he vdeo As a resul, learnng he LDT reduces o learnng several dynamc exures, when gven he segmenaon of he vdeo no regons of dsnc moon For he more general case, he segmenaon and he dynamcs can be learned smulaneously usng he EM algorhm 23 Modelng layer assgnmens An MRF s used o model he layer assgnmens o ensure spaal conssency of he layer (see Fgure (c) for an example of he grd) The MRF has he followng dsrbuon p(z) = ψ (z ) Z (,) E ψ, (z, z ) (3) where E s he se of edges n he MRF grd, Z a normalzaon consan (paron funcon), and ψ and ψ, poenal funcons of he form α, z = ψ (z ) = (4) α K, z = K { γ, z ψ, (z, z ) = = z (5) γ 2, z z The poenal funcon ψ defnes a pror lkelhood for each layer, whle ψ, arbues hgher probably o confguraons where neghborng pxels are n he same layer Raher han learn he parameers of he poenal funcons for each model, we wll rea he MRF as a pror on Z ha regularzes he smoohness of he layers 3 Parameer esmaon usng EM The parameers of he layered dynamc exure are learned usng he Expecaon-Maxmzaon (EM) algorhm [2], whch eraes beween esmang hdden sae (2) N τ K y y, z number of pxels n a frame lengh of he observed vdeo sequence number of sae processes ndex over he pxel sequences ndex over he sae processes me ndex of a sequence he h pxel sequence he observaon a me of y he h sae sequence he sae a me of he layer assgnmen varable for y he ndcaor varable ha y s labeled Table Noaon for EM for layered dynamc exures varables X and hdden layer assgnmens Z from he curren parameers, and updang he parameers gven he curren hdden varable esmaes One eraon of he EM algorhm conans he followng wo seps E-Sep: Q(Θ; ˆΘ) = E X,Z Y ;ˆΘ(log p(x, Y, Z; Θ)) M-Sep: ˆΘ = argmax Θ Q(Θ; ˆΘ) In he remander of hs secon, we derve he on loglkelhood of he model, followed by he dervaons of he E-sep and M-sep of he learnng algorhm See Table for noaon 3 Log-lkelhood The sae processes X and layer assgnmens Z are ndependen, and hence he on log-lkelhood facors as l(x, Y, Z) = log p(x, Y, Z) (6) = log p(y X, Z) + log p(x) + log p(z) (7) =, =, + log p(y, z = ) (8) log p( ) + log p(z) τ = ( τ + =2 + log p(z) log p(y,, z = ) (9) ) log p( ) + log p(x() ) where s he ndcaor varable ha z = Subsung for he probably dsrbuons and droppng he consan erms yelds he log-lkelhood gven n (20)

4 32 E-Sep Takng he condonal expecaon of (20), he E-sep requres he compuaon of he followng erms: ˆ = E X Y ( ) (0) ˆ, = E Z,X Y ( ˆP () ), = E X Y ( ( ) T ) ˆP (), = E Z,X Y ( ˆP (), = E X Y ( ( )T ) ( ) T ) ẑ () = E Z Y ( ) = p(z = Y ) These expecaons are nracable o compue n closedform snce s no known o whch sae process each of he pxels y s assgned, and hence s necessary o margnalze over all confguraons of Z Ths problem also appears for he compuaon of he poseror layer assgnmen probably p(z = Y ) Whle oher nference approxmaon mehods, eg varaonal mehods or belef propagaon, could be used, he curren mehod ha we adop for approxmang hese expecaons s o smply average over draws from he poseror p(x, Z Y ) usng a Gbbs sampler (see Appendx for deals) 33 M-Sep The opmzaon n he M-Sep s obaned by akng he paral dervave of he Q funcon wh respec o each of he parameers For convenence, we frs defne he followng quanes, φ () = τ () = ˆP, Φ () = τ = ψ () = τ =2 ˆN = ẑ() ˆP (), ˆP (), φ () 2 = τ () =2 ˆP, Φ () Γ () Λ () = τ = ˆP (), = τ = y,ˆ, = τ = ẑ() y, 2 () Takng he paral dervave wh respec o each parameer and seng o zero yelds he parameer updaes: A () = ψ () (φ () Q () = τ (φ() 2 A () (ψ () ) T ) µ () = ˆ S () = () ˆP, µ() (µ () ) T = (Γ () ) T (Φ () N C () r () = τ ˆN ) (2) = (Λ () ) C () Γ () ) The M-sep parameer updaes are analogous o hose requred o learn a regular lnear dynamcal sysem [3, 4], wh mnor modfcaons for ransformaon marces and observaon nose 4 Expermens In hs secon, we show he effcacy of he proposed model for segmenaon and synhess of several vdeos wh mulple regons of dsnc moon Fgure 2 (a) shows he hree vdeo sequences used n esng The frs (op) s a compose of hree dsnc vdeo exures of waer, smoke, and fre The second (mddle) s of laundry spnnng n a dryer The laundry n he boom lef of he vdeo s spnnng n place n a crcular moon, and he laundry around he ousde s spnnng faser The fnal vdeo (boom) s of a hghway [5] where he raffc n each lane s ravelng a a dfferen speed The frs, second and fourh lanes (from lef o rgh) move faser han he hrd and ffh All hree vdeos have mulple regons of moon and are herefore properly modeled by he models proposed n hs paper, bu no by a regular dynamc exure A layered dynamc exure (LDT) was f o each of he hree vdeos For comparson, a layered dynamc exure wh he layer assgnmens z dsrbued as d mulnomals (LDT-d) was also learned In all he expermens, he dmenson of he sae space was n = 0 The MRF grd was based on he egh-neghbor sysem (wh clques of sze 2), and he parameers of he poenal funcons were γ = 099, γ 2 = 00, and α = /K The expecaons requred by he EM algorhm were approxmaed usng Gbbs samplng We frs presen segmenaon resuls, o show ha he models can effecvely separae layers wh dfferen dynamcs, and hen dscuss resuls relave o vdeo synhess from he learned models 4 Segmenaon The vdeos were segmened by assgnng each of he pxels o he mos probable layer condoned on he observed vdeo, e z = argmax p(z = Y ) Anoher possbly would be o assgn he pxels by maxmzng he poseror of all he pxels p(z Y ) Whle hs maxmzes he rue poseror, n pracce we obaned smlar resuls wh he wo mehods The former mehod was chosen because he ndvdual poseror dsrbuons are already compued durng he E-sep of EM Fgures 2 (b) and (c) show he segmenaon resuls obaned usng he LDT and LDT-d models, respecvely The segmened vdeo s also avalable a [6] From he segmenaons produced by LDT-d, can be concluded ha he laundry vdeo can be reasonably well segmened whou he MRF pror The segmenaon of he compose vdeo usng LDT-d s slghly worse, and conans several regons of nose Noneheless, hs confrms he nuon ha he varous vdeo regons conan very dsnc dynamcs ha can only be modeled wh separae sae processes Oherwse, he pxels should be eher randomly assgned among he varous layers, or unformly assgned o one of

5 hem The segmenaons of he raffc vdeo usng LDTd are poor Whle he dynamcs are dfferen, he dfferences are sgnfcanly more suble, and segmenaon requres sronger enforcemen of layer conssency As expeced, he nroducon of he MRF pror mproves he segmenaons for all hree vdeos For example, n he compose sequence all erroneous segmens n he waer regon are removed, and n he raffc sequence, mos of he speckled segmenaon also dsappears In erms of he overall segmenaon qualy, he LDT s able o segmen he compose vdeo perfecly The segmenaon of he laundry vdeo s plausble, as he laundry umblng around he edge of he dryer moves faser han ha spnnng n place The model also produces a reasonable segmenaon of he raffc vdeo, wh he segmens roughly correspondng o he dfferen lanes of raffc Much of he errors correspond o regons ha eher conan nermen moon (eg he regon beween he lanes) or almos no moon (eg ruck n he upper-rgh corner and fla-bed ruck n he hrd lane) Some of hese errors could be elmnaed by flerng he vdeo before segmenaon, bu we have aemped no pre or pos-processng Fnally, we noe ha he laundry and raffc vdeos are no rval o segmen wh sandard compuer vson echnques, namely mehods based on opcal flow Ths s parcularly rue n he case of he raffc vdeo where he abundance of sragh lnes and fla regons makes compung he correc opcal flow dffcul due o he aperure problem 42 Synhess The layered dynamc exure s a generave model, and hence a vdeo can be synheszed by drawng a sample from he learned model A synheszed compose vdeo comparng he LDT and he normal dynamc exure can be found a [6] When modelng a vdeo wh mulple moons, he regular dynamc exure wll average dfferen dynamcs Ths s noceable n he synheszed vdeo, where he fre regon does no flcker a he same speed as n he orgnal vdeo Furhermore, he moons n dfferen regons are coupled, eg when he fre begns o flcker faser, he waer regon ceases o move smoohly In conras, he vdeo synheszed from he layered dynamc exure s more realsc, as he fre regon flckers a he correc speed, and he dfferen regons follow her own moon paerns 5 Conclusons and Fuure Work In hs paper we have nroduced a new model, he layered dynamc exure, ha can model vdeo ha conans regons of moon wh dfferen dynamcs For hs class of vdeo, we showed ha he layered dynamc exure s more approprae for synhess han he regular dynamc exure In addon, he model provdes a naural framework for segmenng vdeo no regons of moon One dsadvanage of he model s ha he curren mplemenaon of he E-sep n he learnng algorhm requres samplng mehods, whch are compuaonally nensve Fuure work wll be dreced owards faser approxmaon mehods, such as varaonal approxmaon or belef propagaon Appendx A sample from he layered dynamc exure can be obaned usng he Gbbs sampler [7], whch s a mehod for samplng from complcaed probably dsrbuons Nong ha s much easer o sample condonally from he collecon of varables X and Z han on any ndvdual or, he Gbbs sampler s frs nalzed wh X p(x), followed by alernang beween samplng from Z p(z X, Y ) and samplng from X p(x Y, Z) The layer assgnmen dsrbuon p(z X, Y ) s gven by p(z X, Y ) = p(y X, Z)p(X Z)p(Z) p(y X)p(X) (3) p(y X, Z)p(Z) (4) p(z) p(y X, z )) (5) If he z are modeled as ndependen mulnomals, hen samplng z nvolves samplng from he poseror of he mulnomal p(z X, y ) p(y X, z )p(z ) If Z s modeled as an MRF, hen he p(y X, z ) erms are absorbed no he self poenals ψ of he MRF, and samplng can be done usng he MCMC algorhm [8] The sae processes are ndependen of each oher when condoned on he vdeo and he pxel assgnmens, e p(x Y, Z) = p( Y, Z) = p( Y ) (6) where Y = {y z = } are all he pxels ha are assgned o layer Usng he Markovan srucure of he sae process, he on probably facors no he condonals probables, p(,, x() τ Y ) = p( Y ) τ =2 p(, Y ) (7) The parameers of each condonal Gaussan s obaned wh he condonal Gaussan heorem [9], E(, Y ) = (8) µ () + Σ (), (Σ(), ) ( µ() ) cov(, Y ) = (9) Σ (), Σ (), (Σ(), ) Σ (), where he margnal mean, margnal covarance, and one-sep covarance are µ () = E( Y ), Σ (), =

6 l(x, Y, Z) = τ 2,, τ = ( r () y 2, 2C() ( y, + r C () ( ( r (S () ) (x() )T (µ() ) T µ () ( )T + µ () (µ () ) T)) τ =2 ( r (Q () ) ( log r () τ 2 ( ) T ( )T (A () ) T A () log Q () log S () + p(z) 2 ( ) T (C () ) T)) (20) (x() ) T + A () (x() )T (A () ) T)) (a) (b) (c) Fgure 2 Frames from he es vdeo sequences (a): (op) compose of waer, smoke, and fre vdeo exures; (mddle) spnnng laundry n a dryer; and (boom) hghway raffc wh lanes ravelng a dfferen speeds Segmenaon resuls for each of he es vdeos usng: (b) he layered dynamc exure, and (c) he layered dynamc exure whou MRF, Y ), whch can cov( Y ), and Σ (), = cov(x() be obaned usng he Kalman smoohng fler [3, 4] The sequence Y s hen sampled by drawng Y, followed by drawng 2 x(), Y, and so on References [] B K P Horn Robo Vson McGraw-Hll, New York, 986 [2] B Horn and B Schunk Deermnng Opcal Flow Arfcal Inellgence, vol 7, 98 [3] B Lucas and T Kanade An Ierave Image Regsraon Technque wh an Applcaon o Sereo Vson Proc DARPA Image Undersandng Workshop, 98 [4] J Barron, D Flee, and S Beauchemn Performance of Opcal Flow Technques Inernaonal Journal of Compuer Vson, vol 2, 994 [5] J Wang and E Adelson Represenng Movng Images wh Layers IEEE Trans on Image Processng, vol 3, Sepember 994 [6] B Frey and N Joc Esmang Mxure Models of Images and Inferrng Spaal Transformaons Usng he EM Algorhm In IEEE Conference on Compuer Vson and Paern Recognon, 999 [7] G Doreo, A Chuso, Y N Wu, and S Soao Dynamc exures Inernaonal Journal of Compuer Vson, vol 2, pp 9-09, 2003 [8] G Doreo, D Cremers, P Favaro, S Soao Dynamc exure segmenaon In IEEE ICCV, vol 2, pp , 2003 [9] P Sasan, G Doreo, Y Wu, and S Soao Dynamc exure recognon In IEEE CVPR, vol 2, pp 58-63, 200 [0] A B Chan and N Vasconcelos Probablsc kernels for he classfcaon of auo-regressve vsual processes In CVPR, 2005 [] R Vdal and A Ravchandran Opcal flow esmaon & segmenaon of mulple movng dynamc exures In CVPR, 2005 [2] AP Dempser, NM Lard, and DB Rubn Maxmum lkelhood from ncomplee daa va he EM algorhm Journal of he Royal Sascal Socey B, vol 39, pp -38, 977 [3] RH Shumway and DS Soffer An approach o me seres smoohng and forecasng usng he EM algorhm Journal of Tme Seres Analyss, vol 3(4), pp , 982 [4] S Rowes and Z Ghahraman A unfyng revew of lnear Gaussan models Neural Compuaon, vol, pp , 999 [5] hp://wwwwsdowagov [6] hp://wwwsvclucsdedu [7] DJC MacKay Inroducon o Mone Carlo Mehods In Learnng n Graphcal Models, pp , Kluwer Academc Press, 998 [8] S Geman and D Geman Sochasc relaxaon, Gbbs dsrbuon, and he Bayesan resoraon of mages IEEE PAMI, vol 6(6), 984 [9] S M Kay Fundamenals of Sascal Sgnal Processng: Esmaon Theory Prence-Hall Sgnal Processng Seres, 993