arxiv: v1 [cs.gr] 13 Jan 2019

Transcription

1 Noname manuscrp No. wll be nsered by he edor A Fully Bayesan Infne Generave Model for Dynamc exure Segmenaon Sahar Yousef M.. Manzur Shalman Anon B. Chan Receved: dae / Acceped: dae arxv: v1 cs.gr 13 Jan 2019 Absrac Generave dynamc exure models GDMs are wdely used for dynamc exure D segmenaon n he vdeo sequences. GDMs represen Ds as a se of lnear dynamcal sysems LDSs. A major lmaon of hese models concerns he auomac selecon of a proper number of Ds. Drchle process mxure DPM models whch have appeared recenly as he cornersone of he non-paramerc Bayesan sascs, s an opmsc canddae oward resolvng hs ssue. Under hs movaon o resolve he aforemenoned drawback, we propose a novel non-paramerc fully Bayesan approach for D segmenaon, formulaed on he bass of a jon DPM and GDM consrucon. hs neracon causes he algorhm o overcome he problem of auomac segmenaon properly. We derve he Varaonal Bayesan xpecaon- Maxmzaon VBM nference for he proposed model. Moreover, n he - sep of nference, we apply Rauch-ung-Srebel smooher RSS algorhm on Varaonal Bayesan LDSs. Ulmaely, expermens on dfferen vdeo sequences are performed. xpermen resuls ndcae ha he proposed algorhm ouperforms he prevous mehods n effcency and accuracy noceably. Keywords Dynamc exure Segmenaon Generave Dynamc exure model Nonparamerc models Drchle process Varaonal Bayesan approxmaon Sahar Yousef Sharf Unversy of echnology, ehran, Iran -mal: syousef@ce.sharf.edu M.. Manzur Shalman Sharf Unversy of echnology, ehran, Iran -mal: manzur@sharf.edu Anon B. Chan Cy Unversy of Hong Kong, Hong Kong -mal: abchan@cyu.edu.hk

2 2 Sahar Yousef M.. Manzur Shalman Anon B. Chan 1 Inroducon Dynamc exure D segmenaon has receved consderable aenon over he pas decade 1-2. Ds are composed of ensembles of parcles subjec o her sochasc moon. In fac, Ds represen mporan characerscs of vdeo sequences whch have saonary properes n appearance,.e. spaal doman, and moon,.e. me doman. From he appearance pon of vew, exures can be dvded no: srucural and sochasc. he former s mosly arfcal and perodc and can be descrbed by exon, he puave uns of pre-aenve human exure percepon 3, such as aran, Chessboard, ec. On he conrary, he laer s naural and quas-perodc such as ocean waves, grass feld, ec. In he leraure, Ds from he perspecve of her parcles are dvded no hree caegores: mcroscopc, such as plumes of smoke, waer flow, macroscopc, such as blowng leaves n wnd, objecs such as crowd of people, vehcles n raffc jams 4. Snce D Segmenaon reles on boh appearance and moon changes, effecve exure segmenaon n vdeos s one of he mos complcaed ssues n sudes of vdeo processng. here s a grea deal of effors ongong for hs purpose 5, 6-7. Alhough 8 menons fve caegores of approaches for Ds recognon, he Ds segmenaon approaches can be grouped no one of hree caegores: moon-based mehods 5, 9, 10, spaoemporal feaure based mehods 11, 12, 13, 14 and generave dynamc exure models GDMs 4, 15, 16. he moon-based mehods are one of he mos commonly used approaches for D segmenaon. In hese approaches, Ds are represened by esmang a sequence of moon paerns 13, where opcal flows are frequenly used for hs purpose. Amaz e al. presened an opcal flow based mehod n order o segmen vdeos no sac and dynamc exure regons 10. hs approach uses a varan of he brghness consancy assumpon. Afer ha, hey expanded her work by usng graden consancy and color consancy assumpons for he goal 17. hese approaches deec dynamc regons bu do no consder any dfference beween co-occurrng Ds. Vdal and Ravchandran proposed a hree-seps moon-based mehod for D modelng and segmenaon 9. Whle good resuls can be acheved n hese approaches, accurae moon analyss self s a challengng ask due o he dffcules rased by aperure problem, occluson, and vdeo nose. Moreover, he opcal flow mehods are based on he well-known brghness consancy assumpon 18. Hence, any varaon n he lghng whn he scene volaes he brghness consancy consran. Gonçalves and Bruno proposed a memory-based approach usng parally self-avodng walks on hree orhogonal planes 13, n whch he segmenaon s performed by cluserng appearance and moon feaures. In hs approach, he feaures exracon process s llumnaon-sensve, memory-consumng and suffers from heavy compuaon burden. Chen e al. proposed an approach based on local descrpors and opcal flows 5,12. In hs mehod, for consderng appearance mode, spaal exure descrpor,.e. local bnary paern LBP and Weber local descrpor

3 A Fully Bayesan Infne Generave Model for Dynamc exure Segmenaon 3 WLD, and for moon mode opcal flow and local emporal exure descrpor are used. he LBP 14 s a gray-scale nvaran exure descrpor whch descrbes he exure wh codng he exons no bnary paerns. Chen e al. proposed volume based LBP, n whch for each voxel, a bnary code s produced by hresholdng s neghborhood wh he value of he cener voxel. Usng he exraced feaures, he mehod performs he segmenaon by herarchcal splng, agglomerave mergng and pxel-wse classfcaon. Hence, he segmenaon resuls exhb obvous jaggedness of he boundares. Furhermore, LBPs are sensve o nose and llumnaon varaons. Snce LBP descrbes he exons, assumes ha he exures are srucural and canno acheve a proper resul for sochasc exures wh he quas-perodc componens, such as ocean waves. Moreover, snce dfferen dynamc exures are composed of varyng szes of exons, fndng he bes neghbor cube sze s a challengng ssue. On he oher hand, as he neghbor cube ges much larger, more bnary paerns are needed. In he leraure, GDMs are nroduced as a convenen mean for D analyss. Fgure 1 caegorzes some of he recen GD models for dfferen applcaons. here are a wde varey of works whch consder GDMs for D segmenaon 15, 7, 16, 19, 20. he GDM poses Ds as a sochasc vsual process over me and space. In hese approaches, Ds are modeled by lnear dynamcal sysems LDSs 16. Demonsrang a wde varey of complex paerns of moon and spaoemporal model, GDMs are able o reach an admssble soluon whch can model complex senses 4, 20. he GDM suffers from he resrcve assumpon ha he vdeo sequences do no conan co-occurrng Ds. For addressng hs lmaon, varous exensons of GDMs have been proposed 1, 15. Dynamc exure mxure DM 4 and layered Dynamc exure LD 15 are wo of hese effors. he DM supposes ha a collecon of spaoemporal vdeo paches are modeled as samples from a se of underlyng dynamc exures. As Ds are globally homogenous and locally nhomogeneous, pach-based approaches lead o poor resuls. Moreover, he DM, lke all cluserng models, s no a global generave model for vdeo of co-occurrng exures 21.On he conrary, LD s a global generave model whch supposes vdeos are modeled as a superposon of Ds. In hs mehod, n order o esmae he parameers of he model, xpecaon-maxmzaon M algorhm for maxmum-lkelhood s used. Moreover, Gbbs sampler s used for nference. he DM uses an nal paron and LD s nalzed by he resuls of he DM segmenaon. o our knowledge, despe he promsng resuls acheved by he recen mehods, all of hem need an mporan prerequse o segmen he Ds: deermnaon of he opmal number of Ds n vdeo sequences. As he dfferen frames of vdeo sequences may conan a dfferen number of Ds, usng exper knowledge for dervng regon segmens may be an mporan resrcon for sysemac approaches. In hs paper, a non-paramerc fully Bayesan generave approach based on Drchle process DP wll be nroduced for addressng hs lmaon. DP s a dsrbuon over dsrbuons whch commonly used as a pror on he parameers of he mxure model wh counably

4 4 Sahar Yousef M.. Manzur Shalman Anon B. Chan Regsraon Segmenaon Regsraon Segmenaon9, Classfcaon Recognon22, Synhess GD modelng Synhess & Recognon16, Segmenaon MDM15,LDM15,Varaonal LD Segmenaon Fg. 1: Hsory for Generave Dynamc exure Models nfne componens. In oher words, DP s an nfne mxure of dsrbuon wh gven paramerc dsrbuon 28. Alhough he model s defned for nfne mxures, he nference s racable because parameers of a fne se of mxures are needed o be deermned. For hs purpose, Mone Carlo echnque 29 or Varaonal Bayesan approxmaon 30 can be used, where he laer ouperforms he former n speed. he dea of usng DPs o defne mxure models wh an nfne number of componens for mage segmenaon, nfne hdden Markov random feld HMRF, has been prevously explored n 31. Due o he moon mode n Ds, he HMRF s nadequae for D segmenaon. Beal e al. proposed a model known as he nfne hdden Markov model HMM, n whch he number of hdden saes of he hdden Markov model s allowed o be counably nfne 32. Due o he connuy of he hdden saes n LDSs, HMM s no proper for D segmenaon. Beal e al. also, proposed Varaonal Kalman Smooher for one layer LDSs 33.Under hs movaon, we nroduce a novel non-paramerc generave Bayesan model for D segmenaon. Combnng GDM and DP allows us o nroduce a novel model ha has he ably o se he number of Ds auomacally. In our approach, he pror probables of he model are jonly affeced by DP and GDM wh counably nfne Ds. For nference, we use Varaonal Bayesan xpecaon Maxmzaon VBM approxmaon whch s faclaed by means of mean-feld approxmaon. In VB sep, 1s-order and 2nd-order expeced values of he hdden saes are needed. For hs purpose, we perform Rauch-ung-Srebel smooher RSS on Varaonal Bayesan LDSs.

5 A Fully Bayesan Infne Generave Model for Dynamc exure Segmenaon 5 x 1 x 2 x 3... x y 1 y 2 y 3 y Fg. 2: Graphcal model of generave dynamc exure model he wo man conrbuons of hs paper are: a novel non-paramerc formulaon of GDM based on DP for unsupervsed co-occurrng D segmenaon whch resolves he problem of deermnng he proper number of Ds. a fully Bayesan generave model whch resolves he sensvy of he GDM o he nalzaon of parameers. hs paper s organzed as follows: n Secon 2, a bref overvew of generave dynamc exure model and Drchle process s provded. In secon 3 he proposed nfne generave dynamc exure model s nroduced and an elegan runcaed Varaonal Bayesan nference algorhm for he model s derved. Secon 4 descrbes mplemenaon of he proposed mehod comprehensvely. he evaluaon of he effcency of he proposed model hrough hree ypes of aforemenoned dynamc exures.e. mcroscopc, macroscopc and objecs n comparson wh he prevous works s presened n secon 5. Fnally, he concluson s obaned n secon 6. 2 heorecal Background 2.1 Dynamc exure Model GDM s a probablsc approach whch models he me-varyng exure n he vdeos usng a LDS 23. In hs mehod, a sae space model based on auo-regressve movng average ARMA models s used. Due o he movng average propery, he srucure models he non-deermnsc daa. Moreover, Due o auo-regressve propery, he srucure models daa dependen o he pas occurrences. Fgure 2 llusraes he graphcal model of GDMs. GDMs are defned by a se of me-evolvng hdden saes, x R N, and a sequence of observaons, y R M, as below { x = Ax 1 + ζ 1 y = Cx + ξ, 1,, n whch A R N N s he ranson marx, C R M N s he observaon marx, s he emporal lengh of he vdeo, ζ R N and ξ R M are he sae nose and he observaon nose respecvely whch are ndependen and dencally dsrbued..d. sequences drawn from a known dsrbuon such as Gaussan. he nal sae s dsrbued from a normal dsrbuon as x 1 Nx 1 δ, S 1.

6 6 Sahar Yousef M.. Manzur Shalman Anon B. Chan 2.2 Drchle Process he Drchle process pror s a measure on measures 28. DPs assume ha for he random varables {θ }L =1, he random measures are drawn from a DP G 0, α, where G 0 s a base dsrbuon and α s a posve scalng parameer.e. G {G 0, α} DP G 0, α θ G G = 1,..., L. 2 Le {θj }K j=1 be he se of dsnc values aken by varables {θ }L 1 =1. Denong as f L 1 j he number of values n {θ }L 1 =1 ha equal o θ j, he condonal dsrbuon of θl gven {θ n} Π 1 n=1 has he form p θl {θ } L 1 =1, G 0, α α = α + L 1 G 0 + K j=1 f L 1 j α + L 1 δ θ j 3 n whch δ θj denoes he dsrbuon concenraed a a sngle pon θ j 34. he parameer α s a radeoff beween samplng a new parameer from he base dsrbuon and sharng a prevously sampled parameer. herefore α ndcaes a key role n deermnng he number of dsnc parameers. As α ges larger and larger G converges o G 0. On he conrary, as α ges smaller and smaller, all he values {θ }L =1 end o a sngle random varable. Drawng samples from DP can be regarded n erms of sck-breakng consrucon 35. A sck breakng pror on he space has he form where and G = j 1 π j ν = ν j j =1 π j νδ θj, 4 j=1 1 ν j, π j ν 0, 1, 5 π j ν = 1, 6 j=1 n whch {ν j } j=1 and {θ j} j=1 are he random varables drawn from ν j..d. Bea1, α and θ j G 0 respecvely. Suppose y = {y } L =1 be he se of observaons whch s modeled by DP. ach observaon y s assumed o be drawn from s relave condonal probably densy funcon p y θj whch s paramerzed by θ j. Inroducng a dscree random varable z = {z } L =1, n whch z = j denoes ha y s drawn from θj, Drchle process mxure model wh DP can be defned as..d.

7 A Fully Bayesan Infne Generave Model for Dynamc exure Segmenaon 7 y z = j; θ j py θ j, z πν Mulπν, ν j α Bea1, α, θ j G 0 G 0, n whch πν = π j ν j=1 s gven by 7, and Mul. denoes Mulnomal dsrbuon. 7 3 Infne Generave Dynamc exure Model IGDM In hs secon, we propose a novel probablsc model named nfne generave dynamc exure model IGDM. In hs model, s supposed ha he vdeo sequences are composed of nfne number of Ds and each D can be modeled by a LDS. In oher words, he model consders vdeos as a superposon of he oupu of counable nfne dsjon GDMs. In hs approach he fully Bayesan reasonng s used for reasonng whch endeavors o esmae parameers of an underlyng dsrbuon based on he observed dsrbuon. hs requres us o specfy pror dsrbuons on parameers. For hs purpose, he exponenal famles are used. he graphcal model of he proposed mehod llusraed n fgure 3. In hs model, he j h D s modeled by a separae LDS conans a se of hdden saes, x j = {x j x j R N } =1, n whch s he emporal lengh of he vdeo. Moreover, he model conans a se of observed varables, y = {y y R M, y = {y } L =1 } =1 where y deermnes he h pxel, 1, L, on he h frame, 1,, and a lace of ses, z = {z } L =1, whch represens a Markov random feld. he lnear dynamcal equaons of IGDM are as { j x = A j x j 1 + ζ, j 1, 8 y = C z xz + ξ z, 1,, where A j s he N N sae ranson marx where a a 1N A j =....., 9 a N1... a NN d n whch a nn N0, σ A, for n, n 1, N, and C j s he L N observaon marx where c c 1N C z=j =., 10 c L1... c LN d where for h row c ln N0, Σ C. he nal sae of he j h LDS s as x j 1 Nx j 1 δ j, S 1 j, n whch δ j R N and S j ℵ N + are he mean vecor and he precson marx of he

8 8 Sahar Yousef M.. Manzur Shalman Anon B. Chan w 2, λ 2, w 2, Ψ 2 m 3, λ 3, w 3, Ψ 3 w 1, Ψ 1 σ A δ j, S j Q j A j η 1, η 2 µ j, r j x j 1 x j 2 x j... 3 x j α π j C j Σ C y 1 y 2 y 3 y L Z Fg. 3: Graphcal model of IGDM j h nal sae of he LDS respecvely ℵ N + denoes all symmerc posve defne N N marces. Moreover, he sae nose process of he j h LDS n he model s gven by ζ Nζ 0, Q 1 j, 11 where Q j ℵ N + s he precson marx of he sae nose. Furhermore, he observaon nose process of he j h LDS n he model s gven by ξ z Nξ z 0, r 1 j, 12 where r j s he precson of he observaon nose. As IGDM s he superposon of nfne dsjon Ds, undersandng he vdeos s requred o undersandng he Ds. he sequence of saes n he j h LDS n IGDM s a Gauss-Markov process whch s defned as px j = px j 1 =2 px j n whch he dsrbuon of each sae s defned by x j 1, 13

9 A Fully Bayesan Infne Generave Model for Dynamc exure Segmenaon 9 f Fg. 4: Markov neghbourhood sysem x j x j 1, Aj, Q j Nx j A j x j 1, Q 1 j. 14 he dsrbuon of he observaons s defned as below: y x j, z = j, r j, µ j Ny C j x j + µ j, r 1 j, 15 where µ j R s he mean of he j h observaons of he LDS. he dsrbuon of he ses on he label feld, z, are defned by z ẑ δ, πν pz = j, z δ, βpz = j πν, 16 whle pz = j z δ, β s defned as pz = j z δ, β = c V c z j β K j =1 exp c V c z j β, 17 whch s he approxmae pon wse probables of he {z } L =1 n order o mpose MRF. In equaon 17, z j z = j, β s he nverse emperaure of he model, V c z j β are he clque poenals, c s he a member of he se of he clque ncluded n he neghbourhood sysem, C, z δ s he neghbour se of he ce z, and V c z j β s comprsed of he sngleon poenal funcon ς 1, j = 1 V z =. 18 ς K, j = K, and he doubleon poenal funcon z z δ : V, z, z = { γ1, z = z, z, z f γ 2, z z, z, z f, 19 where ς 1,..., ς K, γ 1, γ 2 are consans and f means he h frame. Fgure 4 llusraes neghbourhoods sysems. In hs fgure, he he red ses ndcae z, he green sdes ndcae z f. In hs approach, we use he fully Bayesan reasonng n whch a pror over all he hdden varables and unknown parameers, pψ, s nroduced and he compuaon of poseror dsrbuon gven all he observaons, y, and hyper-parameers, Ξ,.e. p Ψ Ξ, y s neresed. For smplcy, we use he conjugae prors. In order o perform Bayesan nference, we apply varaonal approxmaon whch descrbed n he nex secon.

10 10 Sahar Yousef M.. Manzur Shalman Anon B. Chan 3.1 Inference for IGDM As menoned before, because of beer scalably n erms of compuaonal burden, VBM s used. In hs secon, we dscuss hs echnque. In order o apply Varaonal nference, a famly of varaonal dsrbuons mus be found ha approxmaes he dsrbuon of he nfne-dmensonal random measure G n equaon 4. I can be done by consderng he runcaed sck-breakng represenaon. For hs goal, he mxure proporon,π j ν, s supposed o be zero for j > K, where K s a fxed neger value 36. Bayesan nference echnque nroduces a se of approprae pror dsrbuon over he parameers of he model. For smplcy 16, n hs paper, he conjugae exponenal prors are defned. Hence, we mpose Wshar dsrbuon over he covarance marx of he sae nose, Q j, as defned n equaons 20. Q j w 1, Ψ 1 W Q j w 1, Ψ 1, 20 n whch w 1 s he mean vecors and Ψ 1 s he covarance marx of Q j. Addonally, a jon one-dmensonal Normal-Wshar dsrbuon s mposed as he pror of he mean and covarance marx of he observaons, µ j, r j, and he mean and covarance of he nal sae, σ j, S j as defned n equaons 21 and 22 respecvely. µ j, r j m 2, λ 2, w 2, Ψ 2 NW 1 µ j, r j m 2, λ 2, w 2, Ψ 2, 21 δ j, S j m 3, λ 3, w 3, Ψ 3 NW N δ j, S j m 3, λ 3, w 3, Ψ 3, 22 n whch m 1, m 3 are he mean vecor, λ 2, λ 3, w 2, w 3 are four real numbers, and Ψ 2, Ψ 3 are he scale marces. We use Gamma pror for he Scalng parameer n DP, as α η 1, η 2 Gam α η 1, η 2, 23 where η 1, η 2 are he rae parameers. Le Ψ = {x 1:, z, ν j, α, δ j, S j, Q j, µ j, r j } L,K =1,j=1 be he se of all hdden varables and unknown parameers and Ξ = {w 1, Ψ 1, m 2, λ 2, w 2, Ψ 2, m 3, λ 3, w 3, Ψ 3, η 1, η 2, σ j A, Σj C } be he se of all hyper-parameers of he mposed prors. he jon dsrbuon pψ, y Ξ s defned as: p Ψ, y Ξ = L K P =1 j=1 =1 K P =2 j=1 y C j K P j=1 x j A j x j 1, Q j x j K P Q j w 1, Ψ 1 j=1 + µ j, z = j, r j Iz=j K P µ j, r j m 2, λ 2, w 2, Ψ 2 j=1 x j 1 δ j, S j P δ j, S j m 3, λ 3, w 3, Ψ 3 K P v j α P A j 0, σ A P C j 0, Σ C P α η 1, η 2 P Z, 24 j=1

11 A Fully Bayesan Infne Generave Model for Dynamc exure Segmenaon 11 n whch and L P C j 0, Σ C = N P Z = L =1 j=1 =1 C j 0, Σ C, 25 K pz = j z δ, βpz = j vπ j, 26 where pz = j z δ, β are he pon-wse pror probables of he MRF, gven by 17, and pz = j vπ j are he pror probables of he model saes semmng from he mposed DP, gven by 5 and 7. Varaonal Bayesan nference s used for approxmang nracable negrals arsng n Bayesan nference and machne learnng. In hs mehod, he acual poseror over he se of all hdden varables and unknown parameers,.e. p Ψ Ξ, y, s approxmaed by a Varaonal dsrbuon, known as q Ψ Ψ. Mean-feld varaonal Bayesan s he mos common ype of Varaonal Bayes, whch uses he Kullback Lebler dvergence KL dvergence of p Ψ Ξ, y from q Ψ as he dssmlary funcon. Under hs assumpon, he log margnal lkelhood of model yelds n whch log py = lq Ψ + KLq Ψ p, 27 pψ Ξ, y lq Ψ = q Ψ Ψlog dψ, 28 q Ψ Ψ where KL. sands for KL-dvergence. Snce KL dvergence s non-negave, lq Ψ forms a src lower bound of he log margnal lkelhood and wll be equal when KLq Ψ p = 0 or q Ψ Ψ = pψ Ξ, y. As log margnal lkelhood s consan by maxmzng lq Ψ, KLq Ψ p wll be mnmzed. As we defned he conjugae exponenal prors, he Varaonal poseror q Ψ Ψ s expeced o be he same dsrbuon of pψ Ξ, y37, herefore s expeced ha q Ψ Ψ facorzed as below K K qψ = q x x 1: q z z q ν ν j q α α q δ,s δ j, S j j=1 j=1 j=1 K K q Q Q j q µ,r µ j, r j qaqc, 29 j=1 where and L q x x 1: = qx, q z z = q z z, 30 qa = K N j=1 n=1 =1 qa j n, qc = =1 K N j=1 =1 qc j 31

12 12 Sahar Yousef M.. Manzur Shalman Anon B. Chan able 1: Random varables and Parameers n IGDM Random varable/ Name Dmenson Dsrbued from Parameer x j A j x j 1, Q 1 j x j Saes x j R N N µ j, r j Covarance marx of r j R NW 1 µ j, r j m 2, λ 2, w 2, Ψ 2 he sae nose Q j Covarance marx of Q j ℵ N + W N Q j w 1, Ψ 1 he observaon nose marx of observaons δ j, S j Mean and covarance δ j R N, S j ℵ N + NW N δ j, S j m 3, λ 3, w 3, Ψ 3 marx of he nal sae α Scalng parameer n DP α R K Gamα η 1, η 2 ν j Parameer of DP ν j 0, Beaν 1, α {z } L =1 Label feld {z } 1,..., Mulz πν Makng a full mean feld assumpon n whch q x x 1: = =1 q xx loses crucal nformaon abou he hdden sae chan needed for accurae nference. hus, we employ RSS algorhm whch compues he expeced sascs of he hdden saes n me O. By replacng qψ n 27, we wll have q x x 1: L =1 q zz K j=1 q νν j q α α K j=1 q δ,sδ j, S j K j=1 q QQ j K j=1 q µ,rµ j, r j K N j=1 n=1 qaj n K N j=1 =1 qcj lq Ψ = dψ ln P Ψ, y Ξ ln q x x 1: L =1 ln q zz. 32 K j=1 ln q νν j ln q α α K j=1 ln q δ,sδ j, S j K j=1 ln q QQ j K j=1 ln q µ,rµ j, r j K j=1 N n=1 qaj n K j=1 N =1 qcj Varaonal Bayesan can be seen as an exenson of he expecaon maxmzaon M algorhm whch conans VB- sep and VB-M sep and successvely converges on opmum parameer values 30. In M-Sep, by maxmzng of lq Ψ over each of he facor q Ψ Ψ n urn, holdng he oher fxed, he Varaonal poseror dsrbuon q Ψ Ψ s derved. he followng secon wll descrbe he compuaon of he approxmaed Varaonal dsrbuons n M-sep. able 1 and 2 descrbe he random varables/parameers and hyperparameers of he proposed mehod n deal, respecvely. Varaonal Bayesan M-sep VBM: In hs secon, he VBM s descrbed, see Appendx A for dervaons. Accordng o he mean feld Varaonal Bayesan, defnng qz = j = Iz = j ; j 1, K

13 A Fully Bayesan Infne Generave Model for Dynamc exure Segmenaon 13 able 2: Hyper Parameers n IGDM Hyper Parameer Name Random Varable/ Properes/ Parameer Dmenson w 1 Degree of freedom Q j w 1 > N 1 Ψ 1 Scale marx Q j Ψ 1 ℵ N + m 2 Mean µ j m 2 R λ 2 A real number µ j λ 2 > 0 w 2 A real number r j w 2 > 0 Ψ 2 1 Scale marx r j Ψ 2 R m 3 Mean δ j m 3 R N λ 3 A real number δ j λ 3 > 0 w 3 A real number S j w 3 > N 1 Ψ 3 Scale marx S j Ψ 3 ℵ N + η 1 Shape parameer α η 1 > 0 η 2 Rae parameer α η 2 > 0 σ j A varance A σ j A R Σ j C covarance marx C j ; 1, L Σ j C ℵN + he dsrbuon of covarance marx of he observaon nose s approxmaed by q Q Q j = W Q j ŵ 1, ˆΨ 1, n whch ˆΨ 1 1 = Ψ =2 ŵ 1 = w \Qj x j A j x j 1 x j A j x j 1 34 he dsrbuon of Mean and covarance marx of observaons s approxmaed by q µ,r µ j, r j = NW ˆm 2, ˆλ 2, ŵ 2, ˆΨ 2 35 n whch, accordng o he below noaons N 2 j = L qz = j, 36 =1 we wll have ȳ j = 1 N 2 j L qz = jy, 37 =1 ŵ 2 = w 2 + N 2 j, 38 ˆλ 2 = λ 2 + N 2 j, 39 2 λ2m2+n j =1 ˆm 2 =, ˆλ 2 40

14 14 Sahar Yousef M.. Manzur Shalman Anon B. Chan 1 ˆλ 2 λ 2 m 2 m 2 + =1 ˆΨ 1 2 = Ψ2 1 λ 2 2m 2 m =1 ȳjn j m 2 λ 2 + m 2 λ 2 N 2 j =1 ȳ j + =1 x jn 2 j N 2 j =1 j ȳ + L =1 qz = jy y L =1 =1 qz = jˆµ j C x j y L =1 =1 qz = jy L =1 =1 qz = j N N ˆΣj n=1 n C + µj nn C n µ j C n =1 x j n xj n 41 he dsrbuon of he mean and he covarance marx of nal sae s approxmaed by n whch, accordng o he below noaons we wll have 3 j = q δ,s δ j, S j = NW ˆm 3, ˆλ 3, ŵ 3, ˆΨ 3 42 N 3 j = x 3 j = 1 N 3 j L =1 L qz = j, 43 =1 L =1 qz = j x j 1 qz = j x j 1 x 3 j x j 1 x 3 j, ŵ 3 = w 3 + N 3 j, 46 ˆλ 3 = λ 3 + N 3 j, 47 ˆm 3 = λ 3m 3 + N 3 j x 3 j, 48 ˆλ 3 ˆΨ 3 1 = Ψ j + λ 3N 3 j m 3 x j m 3 x j. 49 ˆλ 3 he dsrbuon of Drchle process scalng parameer s approxmaed by q α α = Gamα ˆη 1, ˆη x j ˆµ j C + where ˆη 1 = η 1, 51 K 1 ˆη 2 = η 2 \α ln 1 π j. 52 j=1 he dsrbuon of Drchle process parameer s descrbed by q ν ν j = Bea ν j ˆβ j,1, ˆβ j,2 53

15 A Fully Bayesan Infne Generave Model for Dynamc exure Segmenaon 15 where L ˆβ j,1 = q z > j + 1, 54 =1 L ˆβ j,2 = α + q z = j. 55 =1 he dsrbuon of qa j s descrbed by where and ˆσ j A n = qa j = ŵ 1 ˆΨ1nn =2 N N n=1 a j n ˆµ j A n, ˆσ j A n x j 1 n x j 1 n + σ j A qn, n ˆµ j A n = ˆσ j 1 A + N ŵ 1 ˆΨ1nn n =1 =2 ŵ 1 ˆΨ1nn n =1 =2 x j 1 n x j n x j 1 n x j 1 n a j n + N n =1 σ j A qn, n a j n 58 For qc j q C j = N C j ˆµ j j C, ˆΣ C 59 where ˆΣ j C = x j r j y r j µ j 60 and ˆµ j C = ˆΣ j 1 C =1 qz = j x j x j r j + Σ j C 61

16 16 Sahar Yousef M.. Manzur Shalman Anon B. Chan Fnally, he dsrbuon of label feld s approxmaed by lnq z z = j = ln pz = j z δ, β + + x j K q z = j j=1 1 S j x j 1 δ j S jx j 1 x j 1 S j δ j + δ j S jδ j =2 x j Q j x j x j 1 Aj Q j x j x j Q j A j x j 1 + xj 1 Aj Q j A j x j 1 + =1 y R jy x j C j R j y y R j C j x j + x j C j R j Cx j + =1 y Σy µ j Σ jy y Σ jµ j + µ j Σ jµ j ln S j 1 ln Q j + ln ν j ln Rj ln Σ j + =1 j=1 K q z > j ln1 ν j. 62 In each eraon of Varaonal Bayesan nference, afer updang equaons??-62, he esmaon of ẑ mus be updaed as he las sep. For hs purpose we updae ẑ by maxmzaon of q z z = j over j as ẑ = arg max j=1:k q zz = j. 63 In Varaonal Bayesan -sep, he expeced values of he parameers are compued. he equaons of hs sep wll be explaned n he nex secon. Varaonal Bayesan -sep VB: In hs sep, he below suffcen sascs are requred qq Q j = ŵ 1 ˆΨ1, 64 qµ,r r j = ŵ 2 ˆΨ2, 65 qδ,s S j = ŵ 3 ˆΨ3, 66 qµ,r r j µ j = ŵ 2 ˆΨ2 ˆm 2, 67 qδ,s S j δ j = ŵ 3 ˆΨ3 ˆm 3, 68 qα ln α = ψ ˆη 1 ψ ˆη 2, 69 qπ ln1 π j = ψ ˆη 1 ψ ˆη 1 + ˆη 2, 70 qq ln Q j = N ln 2 + ln ˆΨ N ŵ1 n ψ, 71 2 n=1 qµ,r ln r j = ln 2 + ln ˆΨ ŵ2 2 + ψ, 72 2

17 A Fully Bayesan Infne Generave Model for Dynamc exure Segmenaon 17 qδ,s ln S j = N ln 2 + ln ˆΨ N ŵ3 n ψ, 73 2 qν ln ν j = ψ ˆβj,1 ψ n=1 ˆβj,1 + ˆβ j,2, 74 qν ln 1 ν j = ψ ˆβj,2 ψ ˆβj,1 + ˆβ j,2, 75 where ψ. denoes he dgamma funcon. Moreover, we need he expeced value of, 76 x j x j x j x j x j 1, For hs purpose, we use RSS algorhm on Varaonal Bayesan Lnear Dynamcal sysem VBLDS model, whch wll be descrbed n he nex secon. 3.2 Rauch-ung-Srebel smooher RSS algorhm In hs secon, RSS algorhm for IGDM s descrbed. RSS mehod conans wo seps: Forward recurson and backward recurson. In forward recurson we defne α x j o be as he poseror over he hdden saes a me gven observed daa up o and ncludng me,.e.: n whch, α x j px j ȳ j 1:, 79 and ȳ j = L =1 qz = jy Ñ j ; 1,, j 1, K, 80 Ñ j = L qz = j. 81 =1 I can be shown ha α x j = Nx j 1 µ, Σ, n whch µ and Σ are he mean and covarance marx respecvely and are defned as µ = Σ 1 1 Ñ j j =1 qz = jc j r j ȳ j µ j + Q j Aj Σ Σ 1 1 µ 1, 82

18 18 Sahar Yousef M.. Manzur Shalman Anon B. Chan Σ = Q j + 1 Ñ j j =1 qz = jc j R j 1 Ñ j j qz = jc j =1 Q j Aj Σ 1 Aj Q j, 83 n whch denoes he nverse ranspose of marces, and Σ s defend as Σ 1 = Σ 1 + A j Q j A j 84 he dervaons can be found n he Appendx B. In he backward recurson we defne β 1 x j 1 o be β 1 x j 1 = pȳj : xj In he Appendx B, I s shown ha β 1 x j 1 = Nxj 1 η 1, ψ 1 1, n whch ψ 1 = A j Q j A j A j Q j ψ Q j A j, 86 η 1 = ψ 1 1 Aj Q j ψ =1 ψ η + 1 Ñ j L =1 where ψ s defned as: ψ = Q j + ψ + 1 L 1 qz = jc j r j Ñ j Ñ j Accordng o he below noaons I s easy o show ha n whch Γ +1,+1 = x j x j 1 Γ, = Q j + 1 Ñ j L =1 x j x j x j qz = jc j r j ȳ j µ j L =1 qz = jc j, = ω, 89 = Γ, 1 + ω ω, 90 = A j Γ 1, ω 1ω 1, 91 Σ + A j Q j A j + ψ, 92 qz = jc j R j 1 L Ñ j =1 qz = jc j, 93 Γ +1, = Q j A j, 94 Γ,+1 = A j Q j, 95 X = Γ 1,+1 Γ, Γ 1 +1,+1 Γ +1,, 96

19 A Fully Bayesan Infne Generave Model for Dynamc exure Segmenaon 19 ω = X 1 Γ,+1 1 Σ µ + ψ η Γ 1 +1,+1 ω +1 = X 1 Γ, 1 Σ µ + ψ η Γ 1 +1, 1 Ñ j 1 Ñ j L =1 L =1 qz = jc j r j ȳ j +1 qz = jc j r j ȳ j +1, he dervaons can be found n Appendx B. 4 Implemenaon of he model We explaned he nfne generave dynamc exure model IGDM n he prevous secons. In hs secon, consderng he ables 1 and 2 and compued equaons were descrbed prevously, we explan he seps of he proposed mehod. he process dagram of our mehod s represened n fgure 5. Frs of all, he npu vdeo sequences are de-nerlaced usng an algorhm such as he spaoemporal medan fler. hen, he IGDM s appled on he feaures and he label fled s resuled as he oupu. he flowchar of he IGDM, wh respec o he derved equaons n prevous secons, s demonsraed n fgure 6. Frsly, he hyper-parameers are nalzed randomly. hen, whle he convergence condon s no sasfed, for all of he LDSs, he Bayesan varaonal M algorhm, whch composed of VB-sep, RSS algorhm, and VBM-sep, s performed. he VBM seps are surrounded by he red recangle. Fnally, he label feld s compued by he seps are surrounded by he blue recangle. Fgure 7 llusraes he process of IGDM segmenaon for a frame vdeo conans four segmens for he nal value of K = 7. 5 xpermenal Resul Here he resuls of he IGDM, he DM 4 and he LD 15 and SAW 13 are compared. We evaluaed he proposed mehod on hree ypes of exures: mcroscopc, macroscopc and objecs. For hs goal, we used UCSD Synhdb 4, Dynex 38 and UCSD Pedesran 4 daases. UCSD Synhdb conans synhec vdeo sequences conss of mcroscopc and macroscopc exures such as smoke, fre, sea waer, vegeaon, escalaor ec. Dynex s a comprehensve daabase of dynamc exures provdng a large and dverse daabase of hghqualy dynamc exures, whch have been denerlaced wh a spaoemporal

20 20 Sahar Yousef M.. Manzur Shalman Anon B. Chan Inpu: Vdeo sequence Denerlacng Oupu: Lable feld IGDM Segmenaon Fg. 5: Process dagram of he IGDM able 3: Please wre your able capon here Daase Resoluon exures UCSD Synhdb Mulple co-occurrng exures n a sngle vdeo Dynex Sea, Grass, rees, Vegeaon, Calm waer, Founans, Smoke, raffc, ec. UCSD Pedesran Pedesrans on UCSD walkways medan fler. UCSD Pedesran conans vdeo of pedesrans on UCSD walkways, from wo vewpons aken wh a saonary camera. able 3 llusraes he properes of he daases n deal. As before menoned, he DM uses an nal conour and LD uses he resul DM as he nal paron. Fgure 8 llusraes he nal conour of DM for UCSD Synhdb segmenaon. Moreover, he ground ruh of he UCSD Synhdb for vdeos wh 2 and 3 segmens are shown n fgure 9. Fgure 10 llusraes he resuls of IGDM, DM 4 and LD 15 on he UCSD Synhdb daase for wo dynamc exures. Rand ndex r-ndex of he segmenaon resuls are gven below he mages. he dynamc exure numbers, whch are esmaed by IGDM s defned by IGDM Seg no ag, are gven below he IGDM resuls. Fgure 11 llusraes he resuls of IGDM, DM and LD on he UCSD Synhdb daase for hree dynamc exures. Smlar o he prevous fgure, he Rand ndex of he segmenaon resuls are gven below he mages.

21 A Fully Bayesan Infne Generave Model for Dynamc exure Segmenaon 21 Sar Read mages and exrac observaons Inalze hyper parameers randomly Compue {q zz = j} L,K =1,j=1 Yes No Converged? j = 1 ẑ = arg max j=1:k q zz = j No j < K? nd Done for all j 1, K Varaonal Baysan xpecaon for j h LDS; eq Yes Forward-Backward algorhm for j h LDS 1. Compue forward messages; eq Compue backward messages; eq Compue 1 h -order and 2 nd -order expecaon of saes; eq Varaonal Baysan Maxmzaon for j h LDS; eq.??-62, j = j + 1 Fg. 6: Flowchar of he IGDM

22 22 Sahar Yousef M.. Manzur Shalman Anon B. Chan IGDM'LDSs nalzaon x 1 1 x x 1 IGDM segmenaon resul x 1 1 x x 1 y 1 1 y 1 2 y 1 y 1 1 y 1 2 y 1 x 2 1 x x 2 x 2 1 x x 2 y 2 1 y 2 2 y 2 y 2 1 y 2 2 y 2 x 3 1 x x 3 x 3 1 x x 3 y 3 1 y 3 2 y 3 y 3 1 y 3 2 y 3 x 4 1 x x 4 x 4 1 x x 4 y 4 1 y 4 2 y 4 y 4 1 y 4 2 y 4 x 5 1 x x 5 x 5 1 x x 5 y 5 1 y 5 2 y 5 y 5 1 y 5 2 y 5 x 6 1 x x 6 x 6 1 x x 6 y 6 1 y 6 2 y 6 y 6 1 y 6 2 y 6 x 7 1 x x 7 x 7 1 x x 7 y 7 1 y 7 2 y 7 y 7 1 y 7 2 y 7 Orgnal mage IGDM segmenaon resul Fg. 7: IGDM segmenaon for K = 7, Red plae: he parameers of K LDSs and her correspondng label feld s nalzed randomly, Gree plae: afer performng IGDM he number of exures s deermned auomacally, he non-black label felds llusrae her correspondng LDSs whch are consdered for segmenaon.

23 A Fully Bayesan Infne Generave Model for Dynamc exure Segmenaon 23 a b Fg. 8: Inal conours of he DM4, a Inal conour of synhdb 2K, b Inal conour of synhdb 3K a b Fg. 9: Ground ruh of UCSD Synhdb daase, a Ground ruh of synhdb 2K, b Ground ruh of synhdb 3K able 4: Rand ndex comparson for some approaches Mehod Synhdb 2K Synhdb 3K LD DM IGDM As menoned before, he segmenaon process n DM and LD s done subjecvely usng exper knowledge whch s a drasc lmaon for sysemac daabase approaches. he IGDM elmnaes he menoned resrcon. herefore, he mehod fnds he opmum exures number auomacally. Fgure 12 and 11 llusraes wo resuls whch IGDM over segmened he vdeo sequence. hese vdeos are segmened by one more exure number. able 4 shows he resul of segmenaon for IGDM, LD and DM on Synhdb daase quanavely based on average R-ndex. Fgure 14 llusraes he segmenaon resuls of IGDM n comparson wh he proposed mehod n 17 and 13. As he resuls show, despe he elmnaon of he nal conour and exper knowledge, IGDM overcomes he segmenaon problem properly. Fgure 13 llusraes he segmenaon resuls of IGDM on some of he vdeo sequences of Dynex daase qualavely. Ulmaely, we presen segmenaon resuls of some objec-based-dynamcexures. Fgure 16 shows he segmenaon resuls of wo objec-based dynamc exures from he Dynex daase. he IGDM segmenaon resuls s compared wh he proposed mehod n 39. As resuls show, he IGDM

24 24 Sahar Yousef M.. Manzur Shalman Anon B. Chan r= r= r=1 IGDM Seg no=2 r= r= r= IGDM Seg no=2 r= r= r= IGDM Seg no=2 r= r= r= IGDM Seg no=2 r= r= r= IGDM Seg no=2 Fg. 10: Resuls on he UCSD synhdb daabase synhdb 2K, lef o rgh: a frame of he orgnal vdeo, he segmenaon resul of he DM 4, he segmenaon resul of he LD 15, he segmenaon resul of he IGDM; he r-ndex and IGDM Seg no of he segmenaon s gven below he mages. produces he segmenaon resuls wh smooher boundares. Moreover, fgure 17 llusraes he segmenaon resuls for some of he vdeo sequences from UCSD Pedesran daase n compared wh he ground ruh avalable n he daase. We se he value of K o 20.

25 A Fully Bayesan Infne Generave Model for Dynamc exure Segmenaon 25 r= r= r= IGDM Seg no=3 r= r= r= IGDM Seg no=3 r= r= r= IGDM Seg no=3 Fg. 11: Resuls on he UCSD synhdb daabase synhdb 3K, lef o rgh: a frame of he orgnal vdeo, he segmenaon resul of he DyexMcIC 4, he segmenaon resul of he LD 15, he segmenaon resul of he IGDM, he r-ndex and IGDM Seg no of he segmenaon s gven below he mages. 6 Conclusons he am of hs paper was o propose a proper dynamc exure segmenaon approach whch elmnaes he exper knowledge abou he number of he dynamc exures and nal paronng n he prevous mehods. For hs goal, we nroduced a novel fully Bayesan non-paramerc formulaon of generave dynamc exure models for robus unsupervsed dynamc exure segmenaon. By dervng a suable poseror dsrbuon over he number of exures, he proposed model resolved he problem of deermnng he proper number of exure segmenaon auomacally. Because of a beer performance of Varaonal Bayesan approxmaon mehods compared wh Mone Carlo echnques, we used he Varaonal Bayesan M for nference. In order o compue he 1s order and 2nd order suffcen sascs of he hdden saes, Varaonal Bayesan Rauch-ung-Srebel smooher RSS s appled. Fnally, a comprehensve comparson of he proposed mehod wh he sae of he ar mehods, for he hree dynamc exure caegores.e. mcroscopc, macroscopc and objec based dynamc exure, was presened.

26 26 Sahar Yousef M.. Manzur Shalman Anon B. Chan a r= d r= b r= c r= IGDM Seg no=3 e Fg. 12: Segmenaon resuls for a vdeo sequence of UCSD synhdb 2K, a a vdeo frame, segmenaon wh: b Isng nalzed usng DM 1, c DM, d LD, e IGDM; he r-ndex and IGDM Seg no of he segmenaon s gven below he mages. r= r= r= r= IGDM Seg no=4 Fg. 13: Segmenaon resuls for a vdeo sequence of UCSD synhdb 3K, a a vdeo frame of UCSD Synhdb 3K, segmenaon wh: b GPCA 9, c DM, d LD, e IGDM; he r-ndex and IGDM Seg no of he segmenaon s gven below he mages.

27 A Fully Bayesan Infne Generave Model for Dynamc exure Segmenaon 27 IGDM Seg no=2 IGDM Seg no=2 IGDM Seg no=2 Fg. 14: Segmenaon resul for Dynex daases, from lef o rgh: a frame of he vdeo, segmenaon wh: Proposed mehod n 17, SAW 13, IGDM, D's number esmaed by IGDM. A Varaonal Bayesan M-sep As menoned before, he Varaonal poseror s expeced o be he same dsrbued from pψ Ξ, y 37. Snce we defned he conjugae exponenal prors, he Varaonal poseror wll be dsrbued from he exponenal famly. I s shown ha he bes dsrbuon for qj, for each of he facors qj n erms of he dsrbuon mnmzng he KL measuremen as descrbed before, equaons 27,28 s expressed as ln qψ Ψj y, Ξ = \j ln p Ψ, Ξ, y + cons, 99 where \j. s he expecaon aken over all varables no n he paron 37. In hs appendx, he dsrbuon of qj, for each of facors are compued. A.1 smaon of he dsrbuon qz0? Accordng o wha explaned before, for esmang qz0 we can wre: P z x1:, z6=, π1:k, α, δ1:k, S1:K, Q1:K, µ1:k, Σ1:K, R1:K, Ξ K Y = j N x1 j=1 Y K Y =1 j=1 δj, Sj 1 Iz =j Y j N x j Iz =j Ax, Q 1 j =2 Iz=j j πj ν N y0 C j x, Rj 1 N y µj, Σj 1 100

28 28 Sahar Yousef M.. Manzur Shalman Anon B. Chan IGDM Seg no=2 IGDM Seg no=2 IGDM Seg no=3 IGDM Seg no=3 IGDM Seg no=4 Fg. 15: Segmenaon resuls of IGDM for Dynex daases, lef o rgh: a frame of vdeo sequence, segmenaon resul of IGDM, D's number esmaed by IGDM s shown below of he resuls I s easy o show ha by replacng 100 no 99 and dong some calculaons, he dsrbuon of qz s obaned va 43. In whch, by defnng p,j = z = j, we can derve q z > j = K k=j+1 p,k. A.2 smaon of he dsrbuon qν j he dervaon of qν j s smlar o wha s seen before. We can wre:

29 A Fully Bayesan Infne Generave Model for Dynamc exure Segmenaon 29 IGDM Seg no=2 IGDM Seg no=2 IGDM Seg no=3 Fg. 16: Segmenaon resul for Dynex daases, from lef o rgh: a frame of he vdeo, segmenaon wh: he proposed mehod n 39, IGDM, D's number esmaed by IGDM s shown below of he resuls P ν j v \j, x 1:, Z 1:L, α, δ 1:K, S 1:K, Q 1:K, µ 1:K, r 1:K, A 1:K, C 1:K, Ξ = L P z = j π j ν Iz=j P ν π j α. 101 =1 Accordng o he equaon 99, we can derve: L L q νν j exp α + q z = j 1 ln ν j + q z > j ln 1 ν j =1 =1 102 I s easy o show ha by smplfyng 102 he dsrbuon of q ν j wll be Bea dsrbuon and can be rewren as A.3 smaon of he dsrbuon q δ j, S j Smlar o he prevous secons, we can wre P δ j, S j x 1:, z 1:L, π 1:K, α, δ \j, S \j, Q 1:K, µ 1:K, r 1:K, A 1:K, C 1:K = N herefore wh respec o he equaon 99, we can derve: x j 1 δ j, S 1 j NW δ j, S j m 3, λ 3, w 3, Ψ q δ j, S j = exp \δj,s j ln N x j 1 δ j, S 1 j + ln NW δ j, S j m 3, λ 3, w 3, Ψ Accordng o wha descrbed before, he dsrbuon of q δ j, S j wll be Normal- Wshar. By smplfyng equaon 104, he dsrbuon of q δ j, S j s obaned va

30 30 Sahar Yousef M.. Manzur Shalman Anon B. Chan IGDM Seg no=2 IGDM Seg no=3 IGDM Seg no=2 IGDM Seg no=2 IGDM Seg no=3 Fg. 17: Segmenaon resul for UCSD Pedesran daases, from op o down: a frame of he vdeo, ground ruh, segmenaon wh IGDM, D's number esmaed by IGDM. A.4 smaon of he dsrbuon q µj, rj Smlar o he prevous secons, we can derve: P µj, rj x1:, z1:l, π1:k, α, δ1:k, S1:K, Q1:K, µ\j, r\j, A1:K, C 1:K, Ξ = Y L Y =1 =1 Iz =j j j N y C x + µj, rj 1 N W µj, rj m2, λ2, w2, Ψ2. 105

31 A Fully Bayesan Infne Generave Model for Dynamc exure Segmenaon 31 Usng equaons 99 and 105, he dsrbuon of qµ j, r j can be wren: L q µ j, r j = exp \µj,r j I z = j ln N y c j =1 =1 x j + µ j, r 1 j + ln NW Q j m 2, λ 2, w 2, Ψ By smplfyng equaon 106, he dsrbuon of s derved as Wshar and s gven by A.5 smaon of he dsrbuon q Q j In order o compue he dsrbuon of q Q j we can wre: P Q j x 1:, z 1:L, π 1:K, α, δ 1:K, S 1:K, Q \j, µ 1:K, r 1:K, A 1:K, C 1:K, Ξ = =2 N x j A j x j 1, Q 1 j W Q j w 1, Ψ So, wh respec o he equaon 99 and smplfyng he equaon 107, he dsrbuon of qq j s from Wshar dsrbuon qq j = W Q j ŵ 1, ˆΨ 1 n whch he mean vecor and covarance marx s gven by??-33. A.6 smaon of he dsrbuon q α Smlar o he prevous secons, we have: P α x 1:, z 1:L, π 1:K, δ 1:K, S 1:K, Q 1:K, µ 1:K, r 1:K, A 1:K, C 1:K, Ξ K 1 = Bea π j 1, α Gamα η 1, η Accordng o he equaons??, 74 and wha descrbed before, he dsrbuon of qα can be wren as below: K 1 qα exp α η 2 ln 1 π j + η 1 1 ln α. 109 j=1 By smplfyng he above equaon, he dsrbuon of qα s from Gamma and s obaned va A.7 smaon of he dsrbuon q a j n j=1 Smlar o he prevous secons, we have: P a j n x 1:, z 1:L, δ 1:K, S 1:K, Q 1:K, µ 1:K, r 1:K, a \j \n, C1:K, Ψ = =2 N x j A j x j 1, N Q 1 j N a j n n=1 0, σ j A, 110

32 32 Sahar Yousef M.. Manzur Shalman Anon B. Chan herefore q a j n exp \a j n =2 ln N x j A j x j 1, Q 1 j + n=1 ln N a n j 0, σ j A 111 q a j n exp j \a n n=1 N a j n n=1 =2 n ŵ 1 ˆΨ1nn a j n ŵ 1 ˆΨ1nn =2 x j 1 n x j a j n n + x j 1 n x j 1 n + σ j A qn, n ŵ 1 ˆΨ1nn n =2 a j n x j 1 n x j a j n n + Accordng o he equaon 9 and expandng equaon 112, he equaons of 56 o 58 are derved. n =1 σ j A qn, n a j n 112 A.8 smaon of he dsrbuon q Smlar o he prevous secons, we have: C j P C j x 1:, Z 1:L, π 1:K, δ j, S j, Q 1:K, µ 1:K, r 1:K, a 1:K, C \j = =1 N y C j x j Iz =j + µ j, r j 1 N C j 0, Σ j C 113 herefore, qc j qz = j C j =1 ŵ 2 ˆΨ2 ˆm 2 x j x j y ŵ2 ˆΨ 2 ˆm 2ŵ2 ˆΨ 2 ŵ 2 ˆΨ2 C j x j C j x j C j 1 2 C j ŵ 2 ˆΨ2 y Σ j C C j 114 Accordng o he equaon 9 and expandng equaon 114, he equaons of 59 o 61 are derved. B Inference of he Rauch-ung-Srebel smooher RSS As menoned before, RSS algorhm conans wo seps: Forward recurson and Backward recurson. In he Forward sep, forward messages go along he LDSs. he forward messages are defned as α x j p x j ȳ j 1:, where can be wren: where ȳ j α x j = p x j 1 ȳj 1: 1 p p x j x j 1 ȳ j ȳ j 1: 1 p ȳ j x j dx j 1, 115 s defned by equaon 52. Suppose Φ ȳ j p ȳ j ȳ j 1: 1, where

33 A Fully Bayesan Infne Generave Model for Dynamc exure Segmenaon 33 1 α x = Φ ȳ j α 1 x j 1 p x j x j 1 p ȳ j x j dx 1 1 x = N j 1 µ 1 1, Σ 1 N x j A j x j 1, Q 1 j Φȳ j N ȳ j C j x j, R 1 j N ȳ j µ j, Σ 1 j he equaon 77 conans a Gaussan dsrbuon N x j 1 µ 1, 1 µ 1 = Σ 1 Σ 1 µ 1 + A j Q j x j Σ 1 = Σ 1 + A j Q j A j. So, wh respec o he above dsrbuon, he equaon 77 can be wren as α x N ȳ j exp 1 2 x j ȳ j µ j, Σ 1 j +ȳ j Q j + C j R j C j x j x j C j R j ȳ j R j C j x j R j ȳ j + µ Σ 1 1 µ 1 N µ 1 Σ 1 µ 1 dx j Σ 1, where 117 x j 1 µ 1 1, Σ 1 dx j he dsrbuon of he forward messages can be obaned from Normal dsrbuon by negraon of equaon 79 as below: α x = N x j µ, Σ 1 µ = Σ Σ = C j R j ȳ j + Q j A j Σ Σ 1 1 µ 1 Q j + C j R j C j Q j Aj Σ 1 Aj Q j. 119 In Backward sep of RSS, backward messages go back along he LDSs. he backward messages are compued as β x j = p ȳ j +1: xj, where can be wren as β 1 x j = p x x 1 p ȳ j x β x j dx j, 120 n whch β x j = 1. he equaon 82 can be wren: x β 1 x j 1 = N j A j x j 1, Q 1 j N ȳ j C j x j, R 1 j N ȳ j µ j, Σ 1 j N x j η, ψ 1 dx j 121 where conans a Gaussan dsrbuon N x j η, ψ 1, whch η = ψ 1 Q j A j x j 1 + ψη + Cj R j ȳ j ψ = Q j + ψ + C j R j C j. 122 So we can wre equaon 84 as: x j 1 A j Q j A j A j Q j ψ Q j A j x j 1 η ψ + ȳj R j Cj ψ Q j A j x j 1 1 x j 1 A j Q j exp 2 ψ ψ η + C j R j ȳ j +η ψη + ȳj R j ȳ j η ψ + ȳj R j Cj ψ ψ η η ψ + ȳj R j Cj ψ C j R j ȳ j 123

34 34 Sahar Yousef M.. Manzur Shalman Anon B. Chan By smplfyng he equaon 86, he dsrbuon of backward messages s from N x j 1 η 1, ψ 1 1, where η 1 = ψ 1 ψ 1 = 1 Aj Q j ψ ψ η + C R j ȳ j A j Q j A j A j Q j ψ Q j A j 124 In order o compue he 1 s order and 2 nd order expeced values of he hdden saes we can wre: p x j ȳ j 1: p x j ȳ j 1: p ȳ j +1: xj = α x j β x j 125 = N x j 1 µ, Σ N x j η, ψ 1 Smplfyng he equaon 89 yelds: p x j ȳ j 1: = N x j ω Γ 1 Γ = Σ + ψ ω = Γ 1 Σ µ + ψ η 126 In order o compue he 1 s order and 2 nd order expeced values of he hdden saes, we should compue he jon probably of p x j, x j +1 ȳj 1:. For hs goal we wll have: p x j, x j +1 ȳj 1: = α x j p x j +1 x p ȳ j +1 x +1 β +1 x j Smplfyng he equaon 90 yelds: p x j, x j +1 x j ȳj 1: = N x j +1 where ω ω +1, Γ, = Σ + A j Q j A j + ψ Γ +1,+1 = Q j + C j R j C j Γ +1, = Q j A j Γ,+1 = A j Q j Γ, Γ +1, Γ,+1 Γ +1, Accordng o he equaon 91, he 1 s order and 2 nd order expeced values of he hdden saes are compued as equaons A VB µ j r jµ j = 1ˆλ2 + ˆm 2 ˆΨ 2ŵ C j r j C j = Σ j C ŵ 2 ˆΨ2 131 δ j S jδ j = n =1 n=1 C j x j x j C j = n=1 n =1 1 + ˆm 3n ˆm 3n ŵ 3 ˆΨ3nn ˆλ 3 ˆΣ j C + µ j nn C µ j n C x j n n xj n

35 A Fully Bayesan Infne Generave Model for Dynamc exure Segmenaon 35 x j Ajx j 1 x j Ajx j 1 x j = x j 1 x j x j { µ j A + µ j A x j 1 xj 1 a j n x j 1 n a j n }nn 134 where {.} nn ndcaes he nn -h elemen of marx. x j A j x j 1 Qj x j A j x j 1 = N x j C j r j C j x j N = ŵ 2 ˆΨ2 n=1 n =1 Σ j C x j nn 1 x j 1 n n 136 y C j x j µ j rj y C j x j µ j = y ˆΨ ŵ2 2 y x j ˆµ j C ŵ 2 ˆΨ2 y ˆm 2ŵ2 ˆΨ 2 y y ˆΨ ŵ2 2 ˆµ j C x j + ŵ 2 ˆΨ2 N n=1 n =1 Σ j C x j nn 1 n xj 1 n + ˆm 2ŵ2 ˆΨ 2 ˆµ j C x j y ˆΨ ŵ2 2 ˆm 2 + x j ˆµ j C ŵ 2 ˆΨ2 ˆm 2 + 1ˆλ2 + ˆm 2 2ŵ 2 ˆΨ2 137 µ j m 2 λ 2 r j µ j m 2 = λ 2 ˆλ 2 + λ 2 ˆm 2 2ŵ 2 ˆΨ2 m 2 λ 2ŵ 2 ˆΨ2 ˆm 2 λ 2 ˆm 2ŵ2 ˆΨ 2 m 2 + m 2 λ 2ŵ 2 ˆΨ2 m x j 1 S j x j 1 = N ˆλ 3 + ˆm2 3 δ j S jx j 1 = x j 1 S j δ j = n =1 n=1 {ŵ 3 ˆΨ3 } nn 139 δ j S jδ j = N 2 + ˆm ˆΨ 3 3 ˆm n =1 n=1 n =1 n=1 x j 1 ˆm n 3n ŵ 3 ˆΨ3n 141 x j 1 n ŵ 3 ˆΨ3n ˆm 3n 142 x j 1 δ j Sj x j 1 δ j = N ˆλ 3 + ˆm2 3 n =1 n=1 x j 1 n ˆm 3n ŵ 3 ˆΨ3n n =1 n=1 {ŵ 3 ˆΨ3 } nn n =1 x j 1 n ŵ 3 ˆΨ3n ˆm 3n + N 2 + ˆm ˆΨ 3 3 ˆm δ j m 3 λ 3 S j δ j m 3 = λ 3 N n =1 n=1 1 + ˆm 3n ˆm 3n ŵ 3 ˆΨ3nn ˆλ 3 m 3 λ 3ŵ 3 ˆΨ3 ˆm 3 ˆm 3ŵ3 ˆΨ 3 λ 3 m 3 + m 3 λ 3ŵ 3 ˆΨ3 m 3 144

36 36 Sahar Yousef M.. Manzur Shalman Anon B. Chan a n j σ j A aj n = σ j A ˆσ j A n + ˆµ j2 A n C j Σ j C Cj = {Σ j C } nn {Σ j C } nn {ˆµ j C } n{ˆµ j C } n n=1 n = x j 1 µ x j j 1 Σ x 1 x j 1 µ j x 1 = n=1 n =1 {Σ x j2 1 } nn {Σ x j 1 } nn {µ j x 1 } 147 x j 1 µ j j x 1 Σ x 1 x j 1 µ j x 1 = 2 n=1 n =1 {Σ n j2 1 } nn {Σ x j 1 } nn {µ j x 1 } n{µ x j 1 } n n=1 n =1 {Σ x j 1 } nn {µ x j 1 } n x j 1 n + n=1 n =1 {Σ j x 1 } nn {µ j x 1 } n{µ j x 1 } n 148 x j µ j j x Σ x x j µ j x = n=1 n =1 2 µ x j n Σ xnn x j n n=1 n =1 Σ x j nn x j n x j n + n=1 n =1 µ j x n Σ xnn µj x n 149 {A j } n {ˆµ j A }n j {ˆσ A }n {A j } n {ˆµ j A }n = {ˆσ j A }n {ˆσ j A }n {ˆµj2 A }n {ˆµ j A }n{ˆσj A }n{ˆµj A }n 150 C j ˆµ j ˆΣj C C C j ˆµ j C = n=1 n =1 j { ˆΣ C } nn j { ˆΣ C } nn {ˆµ j C } n{ˆµ j C } n {ˆµ j C } n{ n=1 n =1 ˆΣ j C } nn {ˆµ j C } n 151 µ j ˆm 2 ˆλ2 r j µ j ˆm 2 = 1 + ˆλ 2 ˆm 2 ˆΨ 2ŵ2 2 ˆm ˆλ 2 2 ŵ 2 ˆΨ2 ˆm 2 152

37 A Fully Bayesan Infne Generave Model for Dynamc exure Segmenaon 37 References 1. Ayeh Ghoreysh and René Vdal. Segmenng dynamc exures wh sng descrpors, arx models and level ses. In Dynamcal Vson, pages Sprnger, Ynhu Zhang, Mohamed Abdel-Moaleb, and Zfen He. Unsupervsed segmenaon of hghly dynamc scenes hrough global opmzaon of mulscale cues. Paern Recognon, 4811: , Bela Julesz. exons, he elemens of exure percepon, and her neracons. Naure, :91, Anon B Chan and Nuno Vasconcelos. Modelng, cluserng, and segmenng vdeo wh mxures of dynamc exures Je Chen, Guoyng Zhao, Mkko Salo, sa Rahu, and Ma Pekanen. Auomac dynamc exure segmenaon usng local descrpors and opcal flow. I ransacons on Image Processng, 221: , Vjay Badrnarayanan, Ignas Budvys, and Robero Cpolla. Sem-supervsed vdeo segmenaon usng ree srucured graphcal models. I ransacons on paern analyss and machne nellgence, 3511: , Adeel Mumaz, Wechen Zhang, and Anon B Chan. Jon moon segmenaon and background esmaon n dynamc scenes. In Proceedngs of he I Conference on Compuer Vson and Paern Recognon, pages , Dmry Cheverkov and Renaud Péer. A bref survey of dynamc exure descrpon and recognon. In Compuer Recognon Sysems, pages Sprnger, Rene Vdal and Avnash Ravchandran. Opcal flow esmaon & segmenaon of mulple movng dynamc exures. In Compuer Vson and Paern Recognon, CVPR I Compuer Socey Conference on, volume 2, pages I, omer Amaz, Sándor Fazekas, Dmry Cheverkov, and Nahum Krya. Deecng regons of dynamc exure. In Inernaonal conference on scale space and varaonal mehods n compuer vson, pages Sprnger, Wefeng Ge, Zhenhua Guo, Yuhan Dong, and Youbn Chen. Dynamc background esmaon and complemenary learnng for pxel-wse foreground/background segmenaon. Paern Recognon, 59: , Je Chen, Guoyng Zhao, and Ma Pekanen. Unsupervsed dynamc exure segmenaon usng local spaoemporal descrpors. In Paern Recognon, ICPR h Inernaonal Conference on, pages 1 4. I, Wesley Nunes Gonçalves and Odemr Marnez Bruno. Dynamc exure segmenaon based on deermnsc parally self-avodng walks. Compuer Vson and Image Undersandng, 1179: , mo Ojala, Ma Pekanen, and op Maenpaa. Mulresoluon gray-scale and roaon nvaran exure classfcaon wh local bnary paerns. I ransacons on paern analyss and machne nellgence, 247: , Anon B Chan and Nuno Vasconcelos. Layered dynamc exures. I ransacons on Paern Analyss and Machne Inellgence, 3110: , Ganfranco Doreo, Alessandro Chuso, Yng Nan Wu, and Sefano Soao. Dynamc exures. Inernaonal Journal of Compuer Vson, 512:91 109, Sándor Fazekas, omer Amaz, Dmry Cheverkov, and Nahum Krya. Dynamc exure deecon based on moon analyss. Inernaonal journal of compuer vson, 821:48, Mchael A Genner and Shahrar Negahdarpour. Relaxng he brghness consancy assumpon n compung opcal flow. echncal repor, MASSACHUSS INS OF CH CAMBRIDG ARIFICIAL INLLIGNC LAB, Adn Ramrez Rvera and Oksam Chae. Spaoemporal dreconal number ransonal graph for dynamc exure recognon. I ransacons on paern analyss and machne nellgence, 3710: , Anon B Chan and Nuno Vasconcelos. Probablsc kernels for he classfcaon of auo-regressve vsual processes. In Compuer Vson and Paern Recognon, CVPR I Compuer Socey Conference on, volume 1, pages I, 2005.

38 38 Sahar Yousef M.. Manzur Shalman Anon B. Chan 21. Anon B Chan and Nuno Vasconcelos. Varaonal layered dynamc exures. In Compuer Vson and Paern Recognon, CVPR I Conference on, pages I, Payam Sasan, Ganfranco Doreo, Yng Nan Wu, and Sefano Soao. Dynamc exure recognon. In Compuer Vson and Paern Recognon, CVPR Proceedngs of he 2001 I Compuer Socey Conference on, volume 2, pages II II. I, Sefano Soao, Ganfranco Doreo, and Yng Nan Wu. Dynamc exures. In Compuer Vson, ICCV Proceedngs. ghh I Inernaonal Conference on, volume 2, pages I, Ganfranco Doreo. Dynamc exure modelng. PhD hess, Unversy of Calforna, Los Angeles, Ganfranco Doreo and Sefano Soao. dable dynamc exures. In Compuer Vson and Paern Recognon, Proceedngs I Compuer Socey Conference on, volume 2, pages II 137. I, Avnash Ravchandran, Paolo Favaro, and René Vdal. A unfed approach o segmenaon and caegorzaon of dynamc exures. In Asan Conference on Compuer Vson, pages Sprnger, Feng Yang, Gu-Song Xa, Gang Lu, Langpe Zhang, and Xn Huang. Dynamc exure recognon by aggregang spaal and emporal feaures va ensemble svms. Neurocompung, 173: , homas S Ferguson. A bayesan analyss of some nonparamerc problems. he annals of sascs, pages , Zoubn Ghahraman and Carl Rasmussen. Bayesan mone carlo. In Advances n neural nformaon processng sysems, pages , Mahew James Beal e al. Varaonal algorhms for approxmae Bayesan nference. unversy of London London, Soros P Chazs and Gabrel sechpenaks. he nfne hdden markov random feld model. I ransacons on Neural Neworks, 216: , Mahew J Beal, Zoubn Ghahraman, and Carl Rasmussen. he nfne hdden markov model. In Advances n neural nformaon processng sysems, pages , Mahew J Beal and Zoubn Ghahraman. he varaonal kalman smooher. Gasby Compuaonal Neuroscence Un, Unversy College London, ech. Rep. GCNU R, 3:2001, Davd Blackwell, James B MacQueen, e al. Ferguson dsrbuons va pólya urn schemes. he annals of sascs, 12: , Jayaram Sehuraman. A consrucve defnon of drchle prors. Sasca snca, pages , Davd M Ble, Mchael I Jordan, e al. Varaonal nference for drchle process mxures. Bayesan analyss, 11: , C.M. Bshop. Paern recognon and Machne Learnng. 38. Renaud Péer, Sándor Fazekas, and Mark J Huskes. Dynex: A comprehensve daabase of dynamc exures. Paern Recognon Leers, 3112: , Prak Soygaonkar, Shlpa Paygude, and Vbha Vyas. Dynamc exure segmenaon usng exure descrpors and opcal flow echnques. In Proceedngs of he 3rd Inernaonal Conference on Froners of Inellgen Compung: heory and Applcaons FICA 2014, pages Sprnger, 2015.