Convex Optimization in Computer Vision:

Transcription

1 Convex Optimization in Computer Vision: Segmentation and Multiview 3D Reconstruction Yiyong Feng and Daniel P. Palomar The Hong Kong University of Science and Technology (HKUST) ELEC Convex Optimization Fall , HKUST, Hong Kong

2 Outline of Lecture 1 Problem Setup Motivation General Problem Formulation 2 Convex Relaxation and Optimality 3 Applications: Image Segmentation and 3D Reconstruction Segmentation Multiview 3D Reconstruction Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

3 Outline of Lecture Problem Setup Motivation Problem Formulation 1 Problem Setup Motivation General Problem Formulation 2 Convex Relaxation and Optimality 3 Applications: Image Segmentation and 3D Reconstruction Segmentation Multiview 3D Reconstruction Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

4 Motivation: Segmentation Motivation Problem Formulation Consider an image, can we partition it into different regions? Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

5 Motivation Problem Formulation Motivation: Multiview Reconstruction Can we reconstruct 3D objective from multiview 2D images? Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

6 Motivation > Formulation? Motivation Problem Formulation How can we formulate mathematically the above two problems? Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

7 Basic Concepts Problem Setup Motivation Problem Formulation Denote space of interest as Ω R d 2D image segmentation d = 2 3D reconstruction d = 3 Find partition (Caccioppoli sets) with k + 1 regions: Ω i, i = 0,..., k, k i=0 Ω i = Ω, and Ω s Ω t = s t Weighted perimeter: Per g (Ω i ; Ω) Ω g (x) D1 {x Ωi }. D is some derivative called distributional derivative. Roughly speaking, if Ω i is in 2D, then it is the total weighted LENGTH of Ω i s boundary CURVE! if Ω i is in 3D, then it is the total weighted AREA of Ω i s boundary SURFACE! Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

8 Estimated Parameter Functions Motivation Problem Formulation g: nonnegative inhomogeneous metric. One example: 1 g (x) = point x 1+ I(x), where I (x) is the intensity or color value of f i : nonnegative weight functions, denote how likely one point belongs to the region Ω i. One example: f i (x) = log P i (x): negative log-likelihood for observing a certain point x belonging to Ω i Note that the smaller f i (x) is, the more likely x Ω i! Usually, f i and g functions can be estimated differently by using different information Here, we focus on the optimization part and assume f i s and g already given! Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

9 General Problem Formulation Motivation Problem Formulation Define regularized energy (with trade-off parameter ν > 0) as E (Ω i ) = k i=0 Ω i f i (x) dx + ν 1 2 k Per g (Ω i ; Ω) i=0 where first term: prior likelihood second term: regularity with weighted perimeter of partition {Ω i } Goal: minimize the regularized energy minimize {Ω i } subject to E (Ω i ) k i=0 Ω i = Ω, Ω s Ω t = s t Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

10 Outline of Lecture Problem Setup 1 Problem Setup Motivation General Problem Formulation 2 Convex Relaxation and Optimality 3 Applications: Image Segmentation and 3D Reconstruction Segmentation Multiview 3D Reconstruction Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

11 Convex Relaxation Part As we pointed out f i s and g are assumed to be given We focus on the Convex Relaxation Part (red dash region) Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

12 Implicit Representation, Case k = 1 (2 Regions) Consider now only TWO regions Clearly we just need one set Ω 1, and Ω 0 will simply be the complement set Represent the partition implicitly by function θ : Ω {0, 1} of Ω 1, i.e., θ = 1 Ω1 and 1 θ = 1 Ω0 Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

13 Implicit Reformulation, Case k = 1 (2 Regions) The regularized energy minimization amounts to minimize Ω Ω (f 1 (x) 1 Ω1 + f 0 (x) (1 1 Ω1 )) dx + νper g (Ω 1 ; Ω) 1 subject to Ω 1 Ω Per g (Ω 1 ; Ω) = Ω g (x) D1 {x Ω1 } = Ω g (x) Dθ (x) Equivalent implicit representation minimize θ subject to E (θ) Ω (f 1 (x) θ (x) + f 0 (x) (1 θ (x))) dx θ Θ +ν Ω g (x) Dθ (x) where Θ {θ : Ω {0, 1}} is a set of mappings! Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

14 Convexity, Case k = 1 (2 Regions) For any α [0, 1], and for any θ 1, θ 2 Θ, D ( αθ 1 + (1 α) θ 2) = αdθ 1 + (1 α) Dθ 2 Then we must have α Dθ 1 + (1 α) Dθ 2 E ( αθ 1 + (1 α) θ 2) θe ( θ 1) + (1 θ) E ( θ 2) Objective E (θ) is convex in θ! Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

15 Convexity, Case k = 1 (2 Regions) Recall that Θ = {θ θ : Ω {0, 1}} If any θ 1, θ 2 Θ, obviously for any α (0, 1) αθ 1 + (1 α) θ 2 / Θ Thus, Θ is nonconvex! Problem is nonconvex! Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

16 Convex Relaxation, Case k = 1 (2 Regions) Relax Θ to the convex set Θ {θ θ : Ω [0, 1]} Regularized energy minimization relaxation: minimize θ Ω (f 1 (x) θ (x) + f 0 (x) (1 θ (x))) dx +ν Ω g (x) Dθ (x) subject to θ Θ Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

17 Optimality, Case k = 1 (2 Regions) Proposition Thresholding a minimizer θ (x) of the relaxed problem leads to another optimal solution: { θ opt 1, if θ (x) µ (x) = 1 θ (x) µ = 0, if θ (x) < µ for any threshold µ (0, 1). But now observe that 1 θ (x) µ is also a minimizer of the original binary problem, why? Thus, the convex relaxation is tight! Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

18 Pre-Proof, Case k = 1 (2 Regions) Rewrite: E (θ) = = = Ω Ω Ω (f 1 (x) θ (x) + f 0 (x) (1 θ (x))) dx + ν g (x) Dθ (x) Ω ((f 1 (x) f 0 (x)) θ (x) + f 0 (x)) dx + ν g (x) Dθ (x) Ω f (x) θ (x) dx + ν g (x) Dθ (x) + C where f (x) f 1 (x) f 0 (x) and C Ω f 0 (x) dx Ignoring the constant terms, we will simply denote E (θ) f (x) θ (x) dx + ν g (x) Dθ (x) Ω Ω Ω Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

19 Proof, Case k = 1 (2 Regions) I Proof. 1 Layer cake formula: θ (x) = θ(x) µ dµ 2 Co-area formula and the fact θ (x) [0, 1]: Ω g (x) Dθ (x) = = = Per g ({x : θ (x) µ} ; Ω) dµ Per g ({x : θ (x) µ} ; Ω) dµ Ω g (x) D1 θ(x) µ dµ Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

20 Proof, Case k = 1 (2 Regions) II 3 Then E (θ) = = = = = Ω Ω f (x) θ (x) dx + ν f (x) 1 0 Ω 1 θ(x) µ dµdx + ν g (x) Dθ (x) f (x) 1 θ(x) µ dxdµ + ν Ω ( f (x) 1 θ(x) µ dx + ν Ω E ( 1 θ(x) µ ) dµ Ω 0 Ω Ω g (x) D1 θ(x) µ dµ g (x) D1 θ(x) µ dµ g (x) D1θ(x) µ ) dµ Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

21 Proof, Case k = 1 (2 Regions) III 4 Since θ is a minimizer of the relaxed problem, for any µ (0, 1), we have E (θ (x)) E ( ) 1 θ (x) µ 5 Assume µ 0 (0, 1) such that 1 θ (x) µ 0 is NOT a minimizer of the relaxed problem, we must have 6 Hence, E (θ (x)) = 1 0 E (θ (x)) < E ( 1 θ (x) µ 0 ) E (θ (x)) dµ < which leads to a contradiction! 1 0 E ( 1 θ (x) µ) dµ = E (θ (x)) Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

22 Proof, Case k = 1 (2 Regions) IV 7 Therefore, we can conclude that for any µ (0, 1), E (θ (x)) = E ( ) 1 θ (x) µ holds, i.e.,1 θ (x) µ is a minimizer of the relaxed problem, and it must be also a minimizer of the original nonconvex problem! Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

23 From Case k = 1 to k 2 Consider now more than TWO regions How to represent a partition implicitly and obtain the equivalent implicit reformulation? Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

24 Implicit Representation, Case k 2 ( 3 Regions) labeling function u : Ω {0, 1,..., k} u (x) = l x Ω l k binary functions θ (x) = (θ 1 (x),..., θ k (x)): { 1 u (x) i θ i (x) = 0 otherwise and 1 θ 1 (x) θ k (x) 0 u (x) and θ (x) are one-to-one correspondence via u (x) = k θ i (x) i=1 Then we can use k binary functions θ (x) to denote a partition {Ω i } k i=0 implicitly! Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

25 Implicit Representation, Case k 2 ( 3 Regions) Example k = 2, i.e., 3 regions: Relationship between θ (x) and a partition {Ω i } k i=0 1 {x Ω0 } = 1 θ 1 (x) 1 {x Ωi } = θ i (x) θ i+1 (x) i = 1,..., k 1 1 {x Ωk } = θ k (x) This implicit representation contains Case k = 1! Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

26 Implicit Representation, Case k 2 ( 3 Regions) Recall that the regularized energy is E (Ω i ) = k i=0 Ω i f i (x) dx + ν 1 2 k Per g (Ω i ; Ω) i=0 For simplicity, denote θ 0 (x) 1, θ k+1 (x) 0 First term can be rewritten as k i=0 Ω i f i (x) dx = k i=0 Ω (θ i (x) θ i+1 (x)) f i (x) dx Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

27 Implicit Representation, Case k 2 ( 3 Regions) Second term (involves some tough proof, omitted here) is 1 2 k i=0 Per g (Ω i ; Ω) = sup Ξ K k i=1 θ i (x) divξ i (x) dx Ω where Ξ (ξ 1,..., ξ k ) is the dual variable with ξ i : Ω R 2 Ξ is constrained to the set K Ξ : Ω Rd k, ξ i (x) g (x), x Ω, 1 i1 i2 k i 1 i i 2 K is convex since it is the intersection of balls. Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

28 Implicit Reformulation, Case k 2 ( 3 Regions) Equivalent implicit reformulation minimize θ subject to E (θ) = k i=0 Ω (θ i (x) θ i+1 (x)) f i (x) dx +ν sup k Ξ K i=1 Ω θ i (x) divξ i (x) dx θ Θ where { } Θ θ : Ω {0, 1} k, 1 θ 1 (x) θ k (x) 0, x Ω Again, the objective is convex, but the feasible set is nonconvex! Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

29 Convex Relaxation, Case k 2 ( 3 Regions) Relax Θ to { } Θ θ : Ω [0, 1] k, 1 θ 1 (x) θ k (x) 0, x Ω Convex relaxation problem minimize θ subject to E (θ) = k i=0 Ω (θ i (x) θ i+1 (x)) f i (x) dx +ν sup k Ξ K i=1 Ω θ i (x) divξ i (x) dx θ Θ We don t have Coarea formula for the second term, thus we cannot have the similar conclusion as Case k = 1 that the relaxation is tight. Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

30 Optimality, Case k 2 ( 3 Regions) Proposition Let θ (x) Θ be a minimizer of relaxed problem, we construct the threshold binary solution 1 {θ (x) µ} Θ. Assume θ gopt (x) Θ is the true global minimizer of the original problem, then: E ( ) ( 1 {θ (x) µ} E θ gopt (x) ) E ( ) 1 {θ (x) µ} E (θ (x)) Proof. The bound follows directly: E (θ (x)) E ( θ gopt (x) ) E ( ) 1 {θ (x) µ} Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

31 Optimality, Case k 2 ( 3 Regions) The choice of µ might be slightly different for different problems But usually it is around 0.5 The experiment examples show that the regularized energy E ( 1 {θ (x) µ}) is somehow robust in µ In many real world experiments, the bound is actually zero or near zero, such that for these examples the solutions are essentially optimal! Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

32 Summary on Convex Relaxation 2 regions: tight 3 regions: bounded Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

33 Outline of Lecture Problem Setup Segmentation Multiview 3D Reconstruction 1 Problem Setup Motivation General Problem Formulation 2 Convex Relaxation and Optimality 3 Applications: Image Segmentation and 3D Reconstruction Segmentation Multiview 3D Reconstruction Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

34 Applications Problem Setup Segmentation Multiview 3D Reconstruction Focus on two applications in computer vision: Image segmentation Multiview 3D reconstruction Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

35 Segmentation: Examples Segmentation Multiview 3D Reconstruction Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

36 Segmentation: Notations Segmentation Multiview 3D Reconstruction Basic notations Ω R 2 : plane that contains the scene of interest Partition: Ω i, i = 0,..., k Estimated parameter functions, for a pixel x: f i (x) 0: denote how likely one point belongs to the region Ω i ; two examples f i (x) = (I (x) c i ) 2 : squared error of input image I (x) to some mean intensity c i, where I (x) is the intensity or color value of x f i (x) = log P i (I (x)): negative log likelihood for observing a certain intensity or color value g (x) 0: inhomogeneous metric; one example g (x) = 1 1+ I(x) Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

37 Segmentation: Problem Formulation Segmentation Multiview 3D Reconstruction Minimize the regularized energy minimize {Ω i } subject to k i=0 Ω i f i (x) dx + ν 1 2 k i=1 Per g (Ω i ; Ω) k i=0 Ω i = Ω, Ω s Ω t = s t It is the same as the general formulation introduced before Optimality if there are two regions, i.e. case k = 1, the convex relaxation is tight! more than two regions, i.e. case k 2, the convex relaxation is not tight but closely bounded! Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

38 Segmentation: Numerical Results Segmentation Multiview 3D Reconstruction Sunflower, Input: 2D image = output: 10-label segmentation Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

39 Segmentation: Numerical Results Segmentation Multiview 3D Reconstruction Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

40 Reconstruction: Example I Segmentation Multiview 3D Reconstruction Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

41 Reconstruction: Example II Segmentation Multiview 3D Reconstruction Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

42 Reconstruction: Notations I Segmentation Multiview 3D Reconstruction Basic notations Ω R 3 : volume that contains the scene of interest S: a certain surface Robj S : the interior of S Rbck S : the exterior of S, i.e., the background V = R S obj RS bck Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

43 Reconstruction: Notations II Segmentation Multiview 3D Reconstruction Estimated parameter functions, for a voxel x: ρ obj (x): describes likelihood of voxel x in Robj S, the smaller value indicates the larger probability; one example ρ obj (x) = log P obj (x) ρ bck (x): describes likelihood of voxel x in Rbck S, the smaller value indicates the larger probability; one example ρ bck (x) = log P bck (x) ρ (x): photoconsistency measure, the smaller value indicates the voxel x is more likely to be on the surface; one example ( ρ (x) = exp τ ) j V OT E j (x), τ > 0 is a rate of decay parameter (say τ = 0.15), V OT E j denotes the vote from view (i.e., camera) j whether the voxel x is on the surface or not Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

44 Segmentation Multiview 3D Reconstruction Reconstruction: Problem Formulation Regularized energy (with trade-off parameter ν > 0): E (S) = ρ obj (x) dx + ρ bck (x) dx + ν ρ (x) ds R S obj R S bck S first two terms: prior likelihood last term: smoothness and photoconsistency Regularized energy minimization problem: S = arg min S Ω E (S) Robj S and RS bck are Ω 0 and Ω 1 ; ρ obj and ρ bck are f i s; ρ is g Note: this is Case k = 1 with 2 regions!!! Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

45 Reconstruction: Reformulation Segmentation Multiview 3D Reconstruction Represent S implicitly by function θ : Ω {0, 1} of Rbck S, i.e., θ = 1 R S and 1 θ = 1 bck R S obj Equivalent formulation (binary problem): minimize θ subject to E (θ) = Ω ρ obj (x) (1 θ (x)) + ρ bck (x) θ (x) dx θ Θ +ν Ω ρ (x) Dθ where Θ {u u : Ω {0, 1}} Convex relaxation Θ {θ θ : Ω [0, 1]} Optimality: thresholding the minimizer of the convex relaxation will give us the minimizer of the original problem! Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

46 Reconstruction: Numerical Results Segmentation Multiview 3D Reconstruction Input: multiview 2D images = output: reconstructed 3D model Y. Feng and D. Palomar (HKUST) CvxOpt in Comp. Vision ELEC / 48

47 For Further Reading Daniel Cremers, Thomas Pock, Kalin Kolev, and Antonin Chambolle, Convex Relaxation Techniques for Segmentation, Stereo and Multiview Reconstruction In Markov Random Fields for Vision and Image Processing. MIT Press, Antonin Chambolle, Daniel Cremers, and Thomas Pock, A convex approach to minimal partitions SIAM Journal on Imaging Sciences. 5(4): , Kalin Kolev, Maria Klodt, Thomas Brox, and Daniel Cremers, Continuous global optimization in multiview 3d reconstruction, International Journal of Computer Vision. 84(1):80 96, 2009.

48 Thanks For more information visit: palomar