A Background Layer Model for Object Tracking through Occlusion

Transcription

1 A Background Layer Model for Object Tracking through Occlusion Yue Zhou and Hai Tao UC-Santa Cruz Presented by: Mei-Fang Huang CSE252C class, November 6, 2003

2 Overview Object tracking problems Dynamic layer model Estimating model parameters Implementation and results Conclusion and discussion

3 Tracking Problems Goal: Estimate 2D or 3D positions of foreground and background objects over time Approach: Model-based e.g. Condensation Layer-based e.g. Dynamic layer model e.g. Flexible sprites

4 Dynamic Layer Model Layer model Represent of moving objects with different motion into different layers Layered representation for motion analysis -Wang & Adelson, 1993 CVPR Dynamic Layer Model Dynamic? Components? Estimation? Dynamically Motion update layer model EM algorithm Segmentation for Allow layer order alteration and maximum Appearance a posteriori layer deletion and creation

5 Contributions of DLM Object tracking with bayesian estimation of dynamic layer representation - H. Tao, H. S. Sawhney, R. Kumar, 2002 IEEE transactions on pattern analysis and machine intelligence Complete representation Dynamic estimation Insufficient in objects with occlusion New ideas in this paper Introduce Ordering Information Z-depth -Foreground / background layer ordering -Allow multiple background layers

6 New Layer Representation Scene representation with Depth Ordering Information Moving objects are described as Foreground Layers Others belong to Background Occluding Layers Interlace foreground and background layers

7 Difference from previous work Use multiple background layers instead of only one background layer Try to solve complicated occlusion problems

8 How does it work? Video frame(t-1) Video frame(t) Layer Representation Estimation Optimize object state Object state(t-1) Layer ordering Motion Shape Object state(t)? Optimal object state(t) Layer ordering Motion Shape Appearance Appearance

9 Goal We want to estimate: Foreground layer ordering Background layer Motion layer parameters Achieved by: MAP framework (Maximum A Posteriori) arg max P( Λ Λ, I,..., I ) Λ t t t 1 t 0

10 The MAP estimation arg max P( Λ Λ, I,..., I ) Λ t t t 1 t 0 Λ t t : the state of the tracker at time t I : the image observation at time t

11 The MAP estimation Using Bayes rule & HMM P( Λ Λ, I,..., I ) = P( I Λ ) P( Λ Λ ) t t 1 t 0 t t t t 1 Λ t t : the state of the tracker at time t I : the image observation at time t PY ( X, E) = PX ( YEPY, ) ( E) PX ( E) arg max P( Λ Λ I,..., I ) Λ t t t 1, t 0 = arg max Λ t P( It,..., I0 Λ t, Λ t 1) P( Λ t Λ t 1) P( I,..., I Λ ) 0 t 1 = arg max P( I Λ ) P( Λ Λ ) Λ t t t t t t 1

12 Estimation Recall that we have P( Λ Λ, I,..., I ) = P( I Λ ) P( Λ Λ ) t t 1 t 0 t t t t 1 Likelihood Prior

13 Apply the motion layer models Prior function P( Λ Λ ) = P P P P P where t t 1 order fg _ shape bg _ shape motion appearance P = P ( o o ) order t t 1 L P = P ( τ ( x ) τ ( x )) fg _ sh a p e t, j i t 1, j i j = 1 i = 1 L + 1 P = P ( π ( x ) π ( x )) P bg _ shape t, j i t 1, j i j = 1 i = 1 = L P ( θ θ ) motion t, j t 1, j j = 1 L N N N j j j P = P ( A ( x ) A ( x )) appearance t, j i t 1, j i j = 1 i = 1

14 Apply the motion layer models Likelihood function N P ( I Λ ) = ( P ( x ) + P ( x )) t t bgo i fgo i i = 1 One background layer/ multiple foreground layer Background observation probability Probability in one background layer P ( x ) = P( I( x ) B( x )) P ( x ) bgo i i i B i L P ( x ) = [ P( I ( x ) A ( x )) P ( x )] fgo i j i j i j i j = 1 jth foreground is visible

15 Estimate Sub-Problems Approximate solution: Divide it into sub-problems

16 Motion Layer Analysis Model the shape and appearance of each object Some approaches Gaussian distributions Markov Random Fields Mixture models (Use EM to get weight) Gaussian segmentation prior function Shape map/mask

17 Motion Layer Parameters Foreground Layer Motion / Shape / Appearance models Parameters: Position, Orientation, Scale Background Layer Shape / Appearance models Shared a single motion Depth ordering Background Layer

18 What to do if there is occlusion? Depth order back front Layer ordering Layer model Optimal model EM

19 Implementation of DLM Motion models Shape models Layer visibility Shape dynamics Appearance model

20 Motion Models Foreground Position Translation + rotation Constant velocity Background Planar projective ω t ( u, v ) t t P( θ θ ) = N( θ : Φθ, Q) t t 1 t t 1 Φ : standard transition matrix for a constant velocity model Θ = [ µ, ω, s, µ, ω, s ]

21 Implementation of DLM Motion models Shape models Layer visibility Shape dynamics Appearance model

22 Shape Models Shape map (a priori) Foreground layers Background layers τ i, j( xi) π i, j( xi) Gaussian segmentation prior function Assumption Each pixel only belongs to ONE background layer π i, j( xi) = 1 j

23 Implementation of DLM Motion models Shape models Layer visibility Shape dynamics Appearance model

24 Layer Visibility See j-th foreground j j 1 P ( x ) = τ ( x )(1 π ( x )) [1 τ ( x )] j i j i l i s i l= 1 s= 1 τ i, j( xi) π i, j( xi) foreground background In j-th foreground layer shape * not in a background layer * not in 1~(j-1)-th foreground layer See j-th background j 1 P ( x ) = π ( x ) (1 τ ( x )) B, j i j i k i k = 1 In j-th background layer shape * not in 1~(j-1)-th foreground layer Observe one background L + 1 j 1 PB ( x i ) = π j( x i ) (1 τ k ( x i )) j= 1 k = 1 Sum all the possible background layers

25 Implementation of DLM Motion models Shape models Layer visibility Shape dynamics Appearance model

26 Shape Dynamics Assumption Shapes don t change dramatically Use constant velocity model Constant value Gaussian model P( τ ( x ) τ ( x )) t, j i t 1, j i = γ + N τ x τ R ω x µ s στ 2 ( t, j( i) : t 1, j( ( t, j)( i t, j) / t, j), ) Shape map alignment

27 Implementation of DLM Motion models Shape models Layer visibility Shape dynamics Appearance model

28 Appearance Models A t, j (x i ) Constant value over time Image model is a Gaussian distribution with A t The temporal change of A t is also a Guassian distribution PI x A x NI x A x 2 ( t( i) t, j( i)) = ( t( i): t, j( i), σ I ) PA x A x NA x A x 2 ( t, j( i) t 1, j( i)) = ( t, j( i): t 1, j( i), A) σ

29 Summary: State estimation P( Λ Λ, I,..., I ) t t 1 t 0 = P P P P P ( P ( x ) + P ( x )) order fg _ shape bg _ shape motion appearance bgo i fg o i i = 1 STEP 1: Find layer ordering Go through all possible orderings and maximize the posterior probability STEP 2: Motion estimation Relaxing problem to ( P ( x ) + P ( x )) P i= 1 STEP 3: Foreground shape STEP 4: Background shape STEP 5: Appearance estimation n bgo i fgo i motion Search appearance value between current observation and the previous estimate N

30 Results Foreground shapes Background shapes Foreground appearance

31 Results

32 Conclusion Achievement Handling difficult occlusion problem Solving occlusion caused by the foreground and background objects Future work Efficient optimization algorithms for optimal foreground layer ordering Flexible shape and motion model

33 Comparison Moving Objects Background Model parameters Estimation Flexible Sprites Sprite layer One layer Sprite mask Appearance EM algorithm Background Layer Model Foreground layer Multiple layer Depth order, motion shape, appearance EM for MAP + HMM

34 Discussion Why use background layers? Can we just take the background layers as foreground layers? When will this approach fail? Under which condition? Observation noisy (Synthetic video) Motion blur Does it work when there are multiple views?