Digital Matting. Outline. Introduction to Digital Matting. Introduction to Digital Matting. Compositing equation: C = α * F + (1- α) * B

Transcription

1 Digital Matting Outline. Introduction to Digital Matting. Bayesian Matting 3. Poisson Matting 4. A Closed Form Solution to Matting Presenting: Alon Gamliel,, Tel-Aviv University, May 006 Introduction to Digital Matting Goal: Separate a foreground object from the background Introduction to Digital Matting Problem: Intricate boundaries hair strands and fur. Original Photo Object with new background Introduction to Digital Matting Compositing equation: Solution: Opacity Alpha Matte Image = * + * C = α * F + (- * B

2 Compositing equation: Blue Screen Matting C= αf + (- B CR = α * FR + ( α ) * BR CG = α * FG + ( α ) * BG CB = α * FB + ( α ) * BB Solve: Given C, find (F, for each pixel The Problem: 3 equations, 7 unknowns Chroma Keying, Blue Screen Matting Statistical Methods Berman et al. 000, Ruzon and Tomasi 000, Chuang et al. 00 Known Background Controlled Scene Input User Supplied Trimap Alpha Matte Image Compositing Statistical Methods Color Model Acquisition - Summary. gather statistical information from user trimap Given image Calculate B) Calculate F). solve for F, α in the unknown region Unknown region Rely on good separation in color space Blue Screen Geometric Model Natural Matting Geometric Model Natural Matting Probabilistic Model Natural Matting Bayesian Model

3 Bayesian Matting Bayesian Matting Overview Video Matting of Complex Scenes,, SIGGRAPH 00 A A Bayesian Approach to Digital Matting,, CVPR 00. Start with a user trimap Chuang, Agarwala, Curless, Salesin, Szeliski. Solve for boundaries of the unknown region Estimate F,α using probabilistic framework, relying on nearest pixels from trimap 3. Refine trimap 4. Back to () Bayesian Matting Bayesian Matting Details ( F, = arg max F, α C) F, α = arg max C F, F) B) α ) / C) F, α = arg max C F, F) B) F, α = arg max L( C F, α ) + L( F) + L( B) + L( F, α Find most likely (F, ) values for each pixel. Apply Bayes Rule C) is constant with F, α Use log-likelihood likelihood Estimating L(C C F, F, This log-likelihood models error in the measurement of C and corresponds to a gaussian probability distribution centered at μ=αf+(- B with standard deviation σ C Estimating Color Correspondence The probability that color c agree with gaussian color model, parameterized with μ, Σ ( π ) T ( c μ ) Σ ( c μ ) f( c μ, Σ ) = exp d det( ) Σ Taking the maximum logarithm (in respect to c) T ( ) ( ) Lc ( μ, Σ ) = c μ Σ c μ Bayesian Matting Details Estimating L(F),L(B) How well estimated F,B correspond to F/B color model Nearest known F points (weighted) L(F) 3

4 Bayesian Matting Details Bayesian Matting Eliminating L( With no other good assumption, assume L( ) is constant for every possible α Solving: Derive from argmax(l( L(C C F,+L(F)+L(B)) two sets of linear equations Solve iteratively, alternating the sets, assuming that α or F/B are constant each iteration Video Matting Problem Bayesian matting requires a trimap for each frame Drawing trimaps is labor-intensive Solution Draw trimaps only for key-frames Interpolate trimaps between key-frames How to interpolate? Optical Flow Optical Flow C i + ( x ) = Ci ( x + u) C(x) ) : color at pixel coord.. x u(x) ) : velocity of the pixel at x called flow field Video Result Poisson Matting Poisson Matting,, SIGGRAPH 004 Sun, Jia,, Tang, Shum 4

5 Poisson Matting Overview Initializing Iterative Process Assumption: The matting gradients mimic the image gradients. Start with a user trimap. Estimate alpha values in unknown area, based on image gradients and known alpha values 3. Refine trimap 4. Back to () Start from a user-defined trimap For each p in Ω,, measure F&B from the nearest pixels in ΩF & ΩB Ω ΩF ΩB Estimating Matting Field I = ( F B) + α F + ( Assume smooth foreground & background We get: I = αf + (-B α F + ( << ( F B) I F B Calculating α Ω unknown area ΩF trimap foreground area ΩB trimap background area Solve: F p /B p are taken from nearest pixels at ΩF / ΩB Calculating α Minimizing is equivalent to solving the Poisson equation: α α I Δα = + = div x y F B Assuming Dirichlet boundary condition Refinement ΩF+ = { p in Ω, s.t. αp > 0.95 (IP ~ FP) F ) } ΩB+ = { p in Ω, s.t. αp < 0.05 (IP ~ BP) B ) } Generate a new trimap F, B and I are measured in grayscale Solve linear system Iterate for few iterations 5

6 Bayesian Poisson Given Image Error Effects I = ( F B) + α F + ( α F + ( << ( F B) α = ) F B ( I α F ( α ) = A( I D) Calculating α in the local case Ω unknown area ΩF trimap foreground area ΩB trimap background area ΩL user selected area Calculating α in the local case Assuming Dirichlet boundary condition Solve: A = F B D = α F + ( αg was calculated at the global case No local discontinuity can be seen 6

7 A Closed Form Solution to Matting A A Closed Form Solution to Natural Image Matting,, CVPR 006 Levin, Lischinski,, Weiss Recall Poisson Matting Estimating Matting Field I = ( F B) + α F + ( Assume smooth foreground & background We get: I = αf + (-B α F + ( << ( F B) I F B I = αf + (-B Assume smooth foreground & background For every window W α i ai i + b for each i in W W I I I3 I4 I5 I6 I7 I8 I9 a = /(F-B) b = -B/(F-B) B) Estimating Matting Field Minimize J: Closed Form Solution It can be proven that: Where L is NxN matrix, and computed directly: ε is a regulation term δ delta function μ, σ window parameters 7

8 Moving to Color Images For every window W α i a R I R i + a G I G i + a B I B i + b for each i in W W I I I3 I4 I5 I6 I7 I8 I9 Derive a R, a G, a B, b from : F = βf + (- β)f B = γb + (- γ)b This relaxes the assumption that foreground & background are smooth Moving to Color Images L is still NxN matrix, but computed differently: δ delta function μ, Σ window parameters Solving for α,, F, B Solving for α,, based on user constraints (scribbles) Can be transformed to Aα = B linear problem Solving for α,, F, B To find F, solve a set of linear equations Correspondence term (for each pixel) ( αf + ( B I) Smoothness terms (for each pixel) α F( x, y) F( x+, y) + α F( x, y) F( x, y+ ) + ( Bxy (, ) Bx ( +, y) + ( Bxy (, ) Bx ( +, y) Scribbles Scribbles Bayesian Poisson This Scribbles This 8

9 More details We wish to solve: Which is like: = * T α arg min( α L i i α = s i S arg min α T Lα + λ ( α i s i ) λ i S Differentiate: L+ λd α = λb ( ) D is a diagonal matrix, where d ii == iff i in S B is a vector, where B i ==known alpha value iff i in S Cholesky Decomposition: T RR α = λ B Future Work (my thesis) Interactive Matting Incorporate additional user scribbles directly into Aα = B linear problem Integrating global color model Statistical analysis of user scribbles Based on this analysis add more alpha constraints Solving for video Goal: Keep a minimal user interface framework Suggestion: Feature-based expansion of user scribbles Goal: Retrieving coherent results between frames Suggestion: Globally solve for frame pairs Live presentation Summary Bayesian Poisson Closed Form Hints Local color analysis Image gradients Image windows Assumptions User Data Color separation, camera quality Trimap Locally smooth foreground and background Trimap Locally linear foreground and background Scribbles Solving Method Iterative Iterative Direct Computation Time (640x480) ~30 secs ~5 secs ~60 secs 9