Lecture 4 Colorization and Segmentation

Transcription

1 Lecture 4 Colorization and Segmentation Summer School Mathematics in Imaging Science University of Bologna, Itay June 1st 2018 Friday 11:15-13:15 Sung Ha Kang School of Mathematics Georgia Institute of Technology 1

2 Outline Lecture 1 & 2 (May 30): Variational/PDE based Image Restoration models Total Variation, Anisotropic Diffusion, TVL1, Image decomposition, Color models, Deblurring inpainting Lecture 3 (May 31): Image segmentation Mumford-Shah, Chan-Vese model Multiphase segmentation and Phase transition model Lecture 4 (June 1): Related topics Colorization Unsupervised multiphase segmentation 2

3 Colorization M. Fornasier, Nonlinear projection digital image inpainting and restoration methods, Journal of Mathematical Imaging and Vision, Volume 24, Number 3, pages ,

4 Related research - colorization The term colorization was introduced by Wilson Markle who first processed the gray scale moon image from the Apollo mission. This term was used to describe the process of adding color to grayscale movies or TV broadcasting program [Encyclopedia of television, 1997] Variational/PDE approaches: [Sapiro 2005] inpaint colors by minimizing the difference between the gradient of luminance and the gradient of color. [Yatziv and Sapiro 2006] utilized Dijkstra s shortest path algorithm for fast computation. [Fornasier 2006] related this problem to inpainting literature, used nonlinear distortion function for fitting the grayscale date µ Ω\D u(x) ū(x) 2 + λ D L(u(x)) ū(x) 2 dx + Ω u(x) p dx. [Fornasier and March 2007] Mathematical analysis, existence of solution using Γ-convergence of this model. [Fonseca, Leoni, Maggi and Morini 2009] using calibration method for minimizer of a colorization model. [K. and March 2007] couple of variational Models via Chromaticity and Brightness decomposition. [Ha Quang, K and Le 2010], many more.. 4

5 TV Colorization with S 2 penalization For a color image C S 2 and D the missing color information [K and March, 2007] E α (C) = C + λ C C o 2 dx + α (1 C ) 2 dx, Ω D c Ω and the corresponding minimization problem (P α ) min { E α (C) : C BV (Ω; R 3 ), C(x) 1 for a.e. x Ω }.. 5

6 Given 6

7 Result 7

8 Original 8

9 Colorization via weighted harmonic map For a color image C S 2 and D the missing color information [K and March, 07] F (C) = g( B ) C 2 dx + λ C C o 2 dx + α (1 C ) 2 dx, Ω D c Ω and the corresponding minimization problem (P) min { F (C) : C W 1,2 (Ω; S 2 ) }. If C is a solution of (P), the colorized image f can be again defined by setting f = B o C. Here g : R + R + is a monotone decreasing function such that g(0) = 1, g(t) > 0 for any t > 0, and lim t + g(t) = 0. e.g. g(t) = (t/a) 2, or g(t) = e (t/a)2, with a > 0. The value of the function g( B ) is close to one in regions where B o is slowly varying, while it is small at the edges of brightness. 9

10 Given 10

11 Result 11

12 Given 12

13 Result 13

14 Given 14

15 Result 15

16 True 16

17 Image decomposition [Meyer, 01] introduced an image decomposition model based on the Rudin-Osher-Fatemi s total variation minimization (TV) model. A given image f is separated into f = u + v by minimizing the following functional, inf (u,v) BV G/f =u+v Ω u + α v G. The Banach space G contains signals with large oscillations. A distribution v belongs to G if v can be written as v = 1 g g 2 = Div(g) g 1, g 2 L. The G-norm v G is defined as the infimum of all g L = sup x Ω g(x), where v = Div(g) and g(x) = g g 2 2 (x). G-norm approaches: [Vese, Sole, Osher 03], [Tadmor, Nezzar, Vese, 04], [Le, Vese, 04], [Daubechies and Teschke,04], [Aujol and Chambolle, 05], [Aujol, Aubert, Blanc-Féraud and Chambolle, 05], [Aubert and Aujol, 05],[Starck, Elad and Donoho, 05], [Garnett, Le Vese, 07], [Lieu, Vese, 08], [Kim and Vese, 09]... 17

18 Texture Colorization Decompose the image brightness B o into two components B o = B u + B v where B u is the BV component of the original image B o, and B v is the oscillatory part of image which represents noise or texture. [joint work with J-F. Aujol, 2006] Then, use B = G σ B u to minimize the same functional, F α (C) = g( B ) C 2 dx + λ C C o 2 dx + α Ω D c Ω (1 C ) 2 dx 18

19 Given 19

20 Result 20

21 True 21

22 Colorization via Reproducing Kernel Hilbert Spaces (RKHS) [Ha Quang, K. and Le 2010] Let Ω R be the image domain, and D Ω be a nonempty subset of Ω. This gray scale image as g : Ω R. Where the color is given be the domain D, and f be the given color, i.e. f : D R 3. To find F : Ω R 3 such that F D f. RGB 3 dimensional vector. In machine learning, reproducing kernel Hilbert spaces (RKHS) became a powerful paradigm, both from algorithmic and theoretical perspectives. e.g. [Schőkopf and Smola 2002] learning with kernels,[shawe-taylor and Cristianini 2004] Kernel methods for Pattern Analysis, [Vapnik 1998]. The goal of machine learning is to make inferences and generalizations based on limited sampled data. [Coifman and Lafon 2006] is one recent RKHS-based approach. Colorization : to find an extension from f : D R 3 to F : Ω R 3. 22

23 Reproducing Kernel Hilbert Space [Aronszajn 1950] Let D be an arbitrary nonempty set. Let K : D D R be a symmetric function, a positive definite kernel on D. There exists a unique Hilbert space H K of functions f : D R satisfying: 1. K x H K for all x D, where K x (t) = K(x, t); 2. span{k x } x X is dense in H K ; 3. the inner product, HK of H K satisfies: f(x) = f, K x HK (reproducing property), for all f H K and all x D. On the dense set span{k x }, the inner product is defined by i a ik xi, j b jk yj HK = i,j a ib j K(x i, y j ). The Hilbert space H K is called the Reproducing Kernel Hilbert Space with reproducing kernel K, with norm HK. 23

24 RKHS for Vector-valued function Let W D denote the vector space of all functions f : D W. A function K : D D L(W) is said to be an operator-valued positive definite kernel if for each pair (x, y) D D, K(x, y) L(W) is a self-adjoint operator and N w i, K(x i, x j )w j W 0 i,j=1 for every finite set of points {x i } N i=1 [Carmeli, De Vito, and Toigo. 2006], [Micchelli and Pontil. 2005] in D. cf. [Caponnetto, Pontil, Micchelli, and Ying. 2008], Colorization: [Ha Quang, K. and Le 2010] Given an f L 2 µ (D; W), inf f L K F 2 L F H K (Ω) 2 (D;W) + γ F 2 µ H K (Ω), for some γ > 0. A standard least square Tikhonov regularization problem in Hilbert spaces, which has a unique minimizer F γ satisfying the normal equation (L K L K + γi)f γ = L K f F γ = (L K L K + γi) 1 L K f. 24

25 Numerical Algorithm - regularized least-square The explicit solution can be computed as m F γ = K(x, x i )a i i=1 where a i s are the solutions of m K(x i, x j )a j + mγa i = f(x i ). j=1 K D (x, y), where (x, y) D D, for solving the system of linear equations, and K cd (x, y), where (x, y) Ω D, for evaluating the result. Let (2r + 1) (2r + 1) be the size of a square patch for each x Ω and a positive integer l = (2r + 1) 2, x = (x 1,..., x l ) R l. (r = 0,only the intensity value). ) ) g( x) g( y) p x y p k(x, y) = exp ( exp (, 2σ 1 (2r + 1) p σ 2 ρ p here ρ is N 2 + M 2. We experimented with 0 < p 2 and various σ 1 and σ 2 values. 25

26 Given less than 3 % color given. 26

27 Result r = 3, p = 2 σ 1 = 0.05, σ 2 = 10 27

28 Given less than 0.5 % color given. 28

29 Result p = 1, r = 2, σ 1 = 0.5, σ 2 = 10 29

30 30

31 31

32 32

33 33

34 34

35 35

36 Supervised and Transductive multi-class segmentation using p-laplacians and RKHS methods Let A R νn N denote the matrix corresponding to the linear mapping ( ( ) ) N x w 1/p i,j (x(i) x(j)), Let A U R νn, U and A L R νn, L denote the matrices containing the columns of A corresponding to the indices in U and L. The minimization problem (p-laplacians) becomes 1 arg min u U p c k=1 j N i=1 A U u k U + A Ll k L p p subject to u U S U c. Use the primal dual hybrid gradient algorithm with modified (extrapolated) primal variable (PDHGMp). PDHGMp was proved to converge for our setting if the parameters γ and τ are chosen such that γτ 1/ M U 2. 36

37 RKHS 1-Lap. 37

38 1-Lap 2-Lap RKHS 38

39 39

40

41 input truth truth RKHS p=1 combined p=2 40

42 Other problems.. Multi-phase segmentation via Modica-Mortola Phase Transition Model E ɛ [z, C k u] = Ω [ ɛ z ɛ sin2 πz ] dx + λ K 1 k=0 Ω u o C k 2 sinc 2 (z k) dx 41

43 Other problems.. Multiphase image segmentation via equally distanced multiple well potential [ Eϕ ɛ (u) = ɛ α ϕ( u ) + W (u) ] dx + λ ( c, u f) 2 dx, ɛ Ω Ω 42

44 Effect of Weighted Length 43

45 Effect of True Length 44

46 Comparison with Sin-Sinc model sin-sinc model True length TV 45

47 Other problems.. Infinite Perimeter Segmentation model - relaxed length E = Ω dist(x, Γ) f( )dx + ɛ 2 i=1 χ i u o c i 2, 46

48 Infinite Perimeter Segmentation Model Chan-Vese Proposed Denoising δ cv = 2 λ cv δ = δ cv Cornering δ cv = 1 λ cv δ = 1 δ cv λ Resolution δ cv = 2 λ cv δ = c δ +2 cv λ Oscillatory boundaries δ cv = c λ cv any δ if λ 2 Comparison with 1 λ cv c ɛ λ. 47

49 Other problems.. Unsupervised multiphase segmentation model ( K ) P (χ i ) E[K, χ i, c i u o ] = µ H 1 (Γ) + χ i i=1 K i=1 χ i u o c i 2, µ = 0.1 µ = 10 48

50 Corner Smoothing: Mumford and Shah B ε B ε χ 2 P α χ 1 B ε/2 P Λ 2 α Λ 1 The length term is bounded by ε while other terms are bounded by ε 2 or ε 2π/α. Then the change of the energy can be computed as E MS (Λ) E MS (χ) c ( ε 2 + ε 2π 2π α + ε ( sin α 2 1 )) for some constant c. Then for a sufficiently small ε, the main change is govern by ( sin α 2 1) which is always a negative value for any 0 < α < π. Let q i = S(χ) + 2 P (χ), i 1,..., K. χ i Then, the energy change E becomes S(Λ)P (Λ) S(χ)P (χ) = ε 2 ( 1 + sin α 2 )(q 1 + q 2 ) + O(ε 2 ) using (S(Λ) S(χ)) O(ε). Therefore, for any 0 < α < π, this is also negative, i.e. it reduces the energy as it cuts the corner as in the case of Mumford-Shah. 49

51 Optimal Angle: Mumford-Shah 120 o P 1 B α χ 3 r χ 1 P 3 P 1 Λ 3 α/2 2Π/3 P 3 Λ 1 Λ 2 χ 2 P 2 P 2 For any angle α the energy reduces as it becomes 2π 3. Let i = P (Λ i ) P (χ i ). Then the energy change can be computed as the following (S(Λ) S(χ))P (χ) + S(Λ)(P (Λ) P (χ)) = 1 q i i + O( 2 ) 2 The energy decreases if q i i < 0, therefore, the optimal angle depends on the values of q i, and NOT necessarily 120 o. The minimum of unsupervised model can have multiple junctions not only triple. Related to the work by Morgan and others on the immiscible ow in R 2 - there are different possibilities when the length is weighted. 50

52 Other problems.. Scale Segmentation: A regularized k-means data clustering ( k ) 1 k E[k, I i, c i D] = λ + d j c i 2. n i=1 i i=1 d j I i intensity segmentation scale segmentation 51

53 Summary * Restoration - Gray vs. Vector-Valued denoising, deblurring - Still vs. Video inpainting, colorization, dejittering... - Surface, High dimensional - Texture, Texture synthesis * Segmentation Two phase, multiphase, - Shape analysis, Object recognition Image registration - Learning: Supervise, unsupervised - Data clustering * Compression wavelet applications * Fast Numerical Computation - Data Analysis - Signal Processing - Computer vision, Computer graphics New devices, new technologies, new imaging challenges. Mathematical model - applied analysis, stable model, theory Numerical challenges/advances, always a need for fast computation 52

54 Thank you Sung Ha Kang kang/ 53