Image Alignment Computer Vision (Kris Kitani) Carnegie Mellon University

Transcription

1 Lucas Kanade Image Alignment Comuter Vision (Kris Kitani) Carnegie Mellon University

2 htt://

3

4 How can I find in the image?

5 Idea #1: Temlate Matching Slow, combinatory, global solution

6 Idea #: Pyramid Temlate Matching Faster, combinatory, locally otimal

7 Idea #3: Model refinement (when you have a good initial solution) Fastest, locally otimal

8 Some notation before we get into the math D image transformation W(; ) D image coordinate = y Translation W(; ) = = + 1 y transform 4 y coordinate Parameters of the transformation = { 1,..., N } Wared image I( 0 )=I(W(; )) Piel value at a coordinate Affine W(; ) = = 1 + y y affine transform y coordinate can be written in matri form when linear affine war matri can also be 33 when last row is [0 0 1]

9 W(; ) takes a as inut and returns a W(; ) is a function of variables W(; ) returns a of dimension = { 1,..., N } where N is for an affine model I( 0 )=I(W(; )) this war changes iel values?

10 Image alignment (roblem definition) min [I(W(; )) T ()] wared image temlate image Find the war arameters such that the SSD is minimized

11 Find the war arameters such that the SSD is minimized T () I() W(; )

12 Image alignment (roblem definition) min [I(W(; )) T ()] wared image temlate image Find the war arameters such that the SSD is minimized How could you find a solution to this roblem?

13 This is a non-linear (quadratic) function of a non-arametric function! min (Function I is non-arametric) [I(W(; )) T ()] Hard to otimize What can you do to make it easier to solve?

14 This is a non-linear (quadratic) function of a non-arametric function! min (Function I is non-arametric) [I(W(; )) T ()] Hard to otimize What can you do to make it easier to solve? assume good initialization, linearized objective and udate incrementally

15 (retty strong assumtion) If you have a good initial guess [I(W(; )) T ()] can be written as [I(W(; + )) T ()] (a small incremental adjustment) (this is what we are solving for now)

16 This is still a non-linear (quadratic) function of a non-arametric function! (Function I is non-arametric) [I(W(; + )) T ()] How can we linearize the function I for a really small erturbation of? Hint: Taylor series aroimation!

17 This is still a non-linear (quadratic) function of a non-arametric function! (Function I is non-arametric) [I(W(; + )) T ()] How can we linearize the function I for a really small erturbation of? Taylor series aroimation!

18 [I(W(; + )) T ()] Multivariable Taylor Series Eansion (First order aroimation) f(, y) f(a, b)+f (a, b)( a) f y (a, b)(y b) Linear aroimation I(W(; )) + T () Is this a linear function of the unknowns?

19 Multivariable Taylor Series Eansion (First order aroimation) f(, y) f(a, b)+f (a, b)( a) f y (a, b)(y b) Recall: 0 = W(; ) I(W(; + )) I(W(; )) + chain rule short-hand = I(W(; )) ) = I(W(; )) 0 short-hand

20 [I(W(; + )) T ()] Multivariable Taylor Series Eansion (First order aroimation) f(, y) f(a, b)+f (a, b)( a) f y (a, b)(y b) Linear aroimation I(W(; )) + T () Now, the function is a linear function of the unknowns

21 I(W(; )) + T () outut of W is a of dimension is a of dimension is a of dimension I( ) is a function of variables

22 I(W(; )) + T () ri is a of is a of dimension is a of dimension (I haven t elained this yet)

23 The (A matri of artial derivatives) = y Affine transform W(; ) = y y + 6 W = W (, y) W y (, 1 = 6 y N Rate of change of the war 3 7 y 1 = 0 y y 0 1

24 I(W(; )) + T ()

25 I(W(; )) + T () iel coordinate ( 1)

26 I(W(; )) + T () iel coordinate ( 1) image intensity (scalar)

27 war function ( 1) I(W(; )) + T () iel coordinate ( 1) image intensity (scalar)

28 war function ( 1) war arameters (6 for affine) I(W(; )) + T () iel coordinate ( 1) image intensity (scalar)

29 war function ( 1) war arameters (6 for affine) I(W(; )) + T () image gradient (1 ) iel coordinate ( 1) image intensity (scalar)

30 war function ( 1) Partial derivatives of war function ( 6) war arameters (6 for affine) I(W(; )) + T () image gradient (1 ) iel coordinate ( 1) image intensity (scalar)

31 war function ( 1) Partial derivatives of war function ( 6) war arameters (6 for affine) I(W(; )) + T () image gradient (1 ) incremental war (6 1) iel coordinate ( 1) image intensity (scalar)

32 war function ( 1) war arameters (6 for affine) Partial derivatives of war function ( 6) temlate image intensity (scalar) I(W(; )) + T () image gradient (1 ) incremental war (6 1) iel coordinate ( 1) image intensity (scalar) When you imlement this, you will comute everything in arallel and store as matri don t loo over!

33 Summary (of Lucas-Kanade Image Alignment) Problem: min [I(W(; )) T ()] wared image temlate image Difficult non-linear otimization roblem Strategy: [I(W(; + )) T ()] I(W(; )) + T () Assume known aroimate solution Solve for increment Taylor series aroimation Linearize then solve for

34 OK, so how do we solve this? min I(W(; )) + T ()

35 Another way to look at it min I(W(; )) + min vector of constants (moving terms around) T () {T () I(W(; ))} vector of variables constant Have you seen this form of otimization roblem before?

36 Another way to look at it min I(W(; )) + T () min {T () I(W(; ))} constant variable constant Looks like A b How do you solve this?

37 Least squares aroimation ˆ = arg min A b is solved by =(A > A) 1 A > b Alied to our tasks: min {T () I(W(; ))} is otimized when = H 1 > [T () I(W(; ))] after alying =(A > A) 1 A > b where H = > A > A

38 Solve: min [I(W(; )) T ()] wared image temlate image Difficult non-linear otimization roblem Strategy: [I(W(; + )) T ()] I(W(; )) + T () Assume known aroimate solution Solve for increment Taylor series aroimation Linearize Solution: = H 1 H = > > [T () I(W(; ))] Solution to least squares aroimation Hessian

39 This is called Gauss-Newton gradient decent non-linear otimization!

40 Lucas Kanade (Additive alignment) 1. War image. Comute error image 3. Comute gradient I(W(; )) [T () I(W(; ))] ri( 0 ) coordinates of the wared image (gradients of the wared image) 4. Evaluate 5. Comute Hessian H H = > 6. Comute = H 1 > [T () I(W(; ))] 7. Udate arameters + Just 8 lines of code!