Video Layer Extraction and Reconstruction

Transcription

1 Vdeo Layer Extracton and Reconstructon Sylvan Pelleter 1, Françose Dbos 2 and Georges Koepfler 1 1 Unversty Pars Descartes, MAP5 (UMR CNRS 8145) 2 Unversty Pars Nord, LAGA (UMR CNRS 7539) MVA G. Koepfler (Unv. Pars Descartes, MAP5) Vdeo Layer Extracton and Reconstructon MVA / 19 Plan: 1 Introducton 2 Layer model defnton 3 Layer deformaton model 4 Varatonal model 5 Results. Concluson G. Koepfler (Unv. Pars Descartes, MAP5) Vdeo Layer Extracton and Reconstructon MVA / 19

2 Introducton Layer decomposton An mage of a natural scene s obtaned by projecton of the 3D scene. Ths s modeled by the superposton of several layers. An occluson appears f a layer s projected upon another. = + + mage = occludng person + occluded person + background G. Koepfler (Unv. Pars Descartes, MAP5) Vdeo Layer Extracton and Reconstructon MVA / 19 Introducton Problem: extract and reconstruct layers from a vdeo sequence; even f total occluson occurs durng several frames. n front of Example of sequence (occludng, occluded, hdden). G. Koepfler (Unv. Pars Descartes, MAP5) Vdeo Layer Extracton and Reconstructon MVA / 19

3 Layer model defnton Defnton: mage I defned on the doman D; layer of 3d movng object: (Ω, o) - Ω D regon where the object s projected f no occlusons; - o gray level functon defned on Ω, gvng the object s gray level. Consder N + 1-frame vdeo sequence (I ) =0,...,N wth: a fxed background, an occluded movng object and an occludng movng object. Three layers: background layer (D, B); occluded object layer (Ω, o Ω ); occludng object layer (O, o O ). G. Koepfler (Unv. Pars Descartes, MAP5) Vdeo Layer Extracton and Reconstructon MVA / 19 Layer model defnton Accordng to the layer model, the mage I at pxel x D s gven by: I (x) o O (x) f x O ; o Ω (x) f x Ω \ O ; B(x) else. The notaton allows for unknown nose effects. I (x) o O (x)i O (x) + o Ω (x)i Ω (x) (1 I O (x)) + B(x) ( 1 I Ω (x) ) (1 I O (x)). G. Koepfler (Unv. Pars Descartes, MAP5) Vdeo Layer Extracton and Reconstructon MVA / 19

4 Layer deformaton model Idea: 3d object moton model yelds deformaton functons, T for the occluded object and T for the occludng object. T s such that: pxel x 0 I 0 and pxel x = T (x 0 ) I are the projecton of the same movng 3d pont X. T and T take care of the perspectve deformaton of the objects. G. Koepfler (Unv. Pars Descartes, MAP5) Vdeo Layer Extracton and Reconstructon MVA / 19 Layer deformaton model Assumng no occluson n I 0 : I (x) I 0 (T 1 (x)) f T 1 (x) O 0 ; I 0 (T 1 (x)) f T 1 (x) Ω 0 ; B(x) else. Image I wth occluson.use T and Ω 0 (= the car).occluson of Ω can be restored. G. Koepfler (Unv. Pars Descartes, MAP5) Vdeo Layer Extracton and Reconstructon MVA / 19

5 Layer deformaton model Hypotheses: fxed camera and known background B; no occluson n frst mage; movng objects are rgd; movement s a unform translaton n 3d space. Thus: consder only short sequences; parametrc deformaton deduced from 3d moton and not from 2d. G. Koepfler (Unv. Pars Descartes, MAP5) Vdeo Layer Extracton and Reconstructon MVA / 19 Layer deformaton model Constructon of layer deformaton functon T : movng 3d pont X(t) = (X(t), Y (t), Z (t)); unform translaton Ẋ(t) = (A, B, C); projecton x onto I by pn-hole camera model: [ ] ] ([ x(t ) 1 x0 x = = = y(t ) c t + 1 [ X Z Y Z y 0 ] + t [ a b ]). wth a = A/Z (0), b = B/Z (0) and c = C/Z (0). x = c (x 0 + ta) = T (x 0 ). G. Koepfler (Unv. Pars Descartes, MAP5) Vdeo Layer Extracton and Reconstructon MVA / 19

6 Varatonal model I (x) o O (x) f x O ; o Ω (x) f x Ω \ O ; B(x) else. wth O = T (O 0), Ω = T (Ω 0 ) and Ω O = T ({ x 0 Ω 0 / T 1 T (x 0 ) O 0 }). Terms used n the global energy: the statc dfference: 0 (x) = I (x) B(x) ; the warp moton dfference for the occluded object: 1 (x) = I (x) I 0 (T 1 x) ; the warp moton dfference for the occludng object: 2 (x) = I (x) I 0 (T 1 x) ; the boundary detecton for the movng objects: g ( I 0 ) = 1/(1 + I 0 2 ). G. Koepfler (Unv. Pars Descartes, MAP5) Vdeo Layer Extracton and Reconstructon MVA / 19 Varatonal model Compute Ω 0, O 0 and (a, b, c, a, b, c ) from the sequence. Energy E = ) ρ ( 2 (x) dx O + ) ρ ( 1 (x) dx Ω \O + ) ρ ( 0 (x) dx D\(O Ω ) + λ g ( I 0 ) ds + λ g ( I 0 ) ds, Ω 0 O 0 wth robust estmator, e.g. ρ(s) = ǫ 2 + s 2. Energy (after change of varables) G. Koepfler (Unv. Pars Descartes, MAP5) Vdeo Layer Extracton and Reconstructon MVA / 19

7 Varatonal model: Mnmzaton Intalsaton get regons R 1 and R 2 : compare I 0 and B; parameters (a, b, c, a, b, c ): use parametrc optcal flow between I 0 and I 1. Depth order: test (Ω 0, O 0 ) = (R 1, R 2 ) versus (Ω 0, O 0 ) = (R 2, R 1 ) and keep the one wth the lowest energy E. Iterate: Mnmze on regons Ω 0 and O 0 wth I.C.M. ; Mnmze on parameters (a, b, c, a, b, c ) wth smplex method. G. Koepfler (Unv. Pars Descartes, MAP5) Vdeo Layer Extracton and Reconstructon MVA / 19 Varatonal model: Mnmzaton on regons E(Ω 0, O 0 ) = O 0 f (x 0)dx 0 + Ω 0 f (x 0 )V (x 0, O ) + λ g ( I 0 ) ds + λ Ω 0 g ( I 0 ) ds, O 0 ICM wth V (x 0, O ) = 1, f pxel x 0 s vsble n I. For two regons: compute φ : ˆD { 1, 0, 1}, such that Ω 0 = {φ = 1} and O 0 = {φ = 1}. G. Koepfler (Unv. Pars Descartes, MAP5) Vdeo Layer Extracton and Reconstructon MVA / 19

8 Varatonal model: Mnmzaton on regons E(Ω 0, O 0 ) = O 0 Ê(φ) = ˆF (s, φ s ) + s ˆD s ˆD f (x 0 )dx 0 + Ω 0 f (x 0 )V (x 0, O ) + λ g ( I 0 ) ds + λ Ω 0 g ( I 0 ) ds, O 0 ( ) ˆF s, φs, φ O + ˆL ( ) s, φ s, {φ t } s t, s ˆD wth O the set of pxels occludng pxel s n I. Ê(φ s ) depends on: φ s, other pxels are fxed; {φ t } s t, the neghborhood of s (length term); (O ) =1,...,N, the occludng pxels (non local term). G. Koepfler (Unv. Pars Descartes, MAP5) Vdeo Layer Extracton and Reconstructon MVA / 19 Varatonal model: Mnmzaton on parameters E(a, b, c, a, b, c ) = O 0 f (a, b, c, x 0 )dx 0 + Ω 0 f (a, b, c, x 0 )V (x 0, O )dx 0, wth V (x 0, O ) = 1, f pxel x 0 s vsble n I. Smplex Method Apply the method to the contnuous parameters (a, b, c, a, b, c ). G. Koepfler (Unv. Pars Descartes, MAP5) Vdeo Layer Extracton and Reconstructon MVA / 19

9 Results on synthetc sequence orgnal sequence restored sequence G. Koepfler (Unv. Pars Descartes, MAP5) Vdeo Layer Extracton and Reconstructon MVA / 19 Results on real sequences Offce Sequence Outdoor Sequence. G. Koepfler (Unv. Pars Descartes, MAP5) Vdeo Layer Extracton and Reconstructon MVA / 19

10 Concluson extract layers of movng objects from a sequence; reconstruct occluded parts; 3d moton model takes nto account deformatons; varatonal formulaton allows to recover depth nformaton. restore sequences, nverson of layers. G. Koepfler (Unv. Pars Descartes, MAP5) Vdeo Layer Extracton and Reconstructon MVA / 19