Efficient & Robust LK for Mobile Vision

Transcription

1 Efficient & Robust LK for Mobile Vision Instructor - Simon Lucey Designing Comuter Vision As

2 Direct Method (ours) Indirect Method (ORB+RANSAC) H. Alismail, B. Browning, S. Lucey Bit-Planes: Dense Subixel Alignment of Binary Descritors. ACCV 2016.

3 Direct Method (ours) Indirect Method (ORB+RANSAC) H. Alismail, B. Browning, S. Lucey Bit-Planes: Dense Subixel Alignment of Binary Descritors. ACCV 2016.

4 Direct Method (ours) Indirect Method (ORB+RANSAC) H. Alismail, B. Browning, S. Lucey Bit-Planes: Dense Subixel Alignment of Binary Descritors. ACCV 2016.

5 Today Review - LK Algorithm Inverse Comosition Robust Extensions

6 Review - LK Algorithm Lucas & Kanade (1981) realized this and roosed a method for estimating war dislacement using the rinciles of gradients and satial coherence. Technique alies Taylor series aroximation to any satially coherent area governed by the war W(x; ). I( + T 4

7 Review - LK Algorithm Lucas & Kanade (1981) realized this and roosed a method for estimating war dislacement using the rinciles of gradients and satial coherence. Technique alies Taylor series aroximation to any satially coherent area governed by the war W(x; ). I( + T 5

8 Review - LK Algorithm Lucas & Kanade (1981) realized this and roosed a method for estimating war dislacement using the rinciles of gradients and satial coherence. Technique alies Taylor series aroximation to any satially coherent area governed by the war W(x; ). I( + T We consider this image to always be static... 5

9 Review - LK Algorithm Lucas & Kanade (1981) realized this and roosed a method for estimating war dislacement using the rinciles of gradients and satial coherence. Technique alies Taylor series aroximation to any satially coherent area governed by the war W(x; ). I( + T 6

10 Review - LK Algorithm Lucas & Kanade (1981) realized this and roosed a method for estimating war dislacement using the rinciles of gradients and satial coherence. Technique alies Taylor series aroximation to any satially coherent area governed by the war W(x; ). I( + @ T = T T N 0 T N T... 7 N T x 0 = W(x; ) 7

11 Reminder - Temlate Coordinates W(x 1 ; ) W(x 1 ; 0) I Source Image T Temlate 8

12 Reminder - Temlate Coordinates x 0 1 x 1 I Source Image T Temlate 8

13 Review - T = T T N 0 T N N T r x I r y I 9

14 Which Way? Ste 1: War Image I I() Ste 2: Estimate Gradients Horizontal r x I() I() Vertical r y I() 10

15 Which Way? Ste 1: War Image I I() Ste 2: Estimate Gradients 0 I() Vertical 10

16 Which Way? Ste 1: War Image I I() Ste 2: Estimate Gradients 0 I() Vertical 10

17 Which Way? Ste 1: Estimate Gradients Horizontal r x I I Vertical Ste 2: War Gradients r y I r x I() r y I() 11

18 Which Way? Ste 1: Estimate Gradients Horizontal r x I I Vertical Ste 2: War Gradients r y

19 Review - T = T T N 0 T N N T

20 ) T For an affine war, W(x; ) = ale x 4y5 T = ale x y x y 1 13

21 ) T For an homograhy war, 1 W(x; ) = 7 x + 8 y + 9 ale x 4y5 T = ale x y x y 1 14

22 Review - LK Algorithm Lucas & Kanade (1981) realized this and roosed a method for estimating war dislacement using the rinciles of gradients and satial coherence. Technique alies Taylor series aroximation to any satially coherent area governed by the war W(x; ). I( + N 1 N 1 N P N = number of ixels P = number of war arameters 15

23 Review - LK Algorithm Often just refer to, J T Also refer to, as the Jacobian matrix. H I = J T I J I as the seudo-hessian. Finally, we can refer to, T (0) =I( + ) as the temlate. 16

24 LK Algorithm Actual algorithm is just the alication of the following stes, Ste 1: Ste 2: = H 1 I JT I [T (0) I()] + kee alying stes until converges. 17

25 Examles of LK Alignment 18

26 Examles of LK Alignment 18

27 Gauss-Newton Algorithm LK is essentially an alication of the Gauss-Newton algorithm, arg min x y F(x) 2 2 s.t. F : R N! R M Ste 1: arg min x T x 2 2 Carl Friedrich Gauss Ste 2: x x + x kee alying stes until x converges. Isaac Newton 19

28 Otimization Interretation The otimization emloyed with the LK algorithm can be interreted as Gauss-Newton otimization. Other non-linear least-squares otimization strategies have been investigated (Baker et al. 2003) Levenberg-Marquadt Newton Steeest-Descent Gauss-Newton in emirical evaluations has aeared to be the most robust (Baker et al.). 20

29 LK Event Horizon Initial war estimate has to be suitably close to the groundtruth for gradient-search methods to work. Kind of like a black hole s event horizon. You got to be inside it to be sucked in!

30 Exanding the Event Horizon Best strategy is to exand the neighborhood Nacross which gradients are estimated. Simlest to do this in ractice with a blur. Often aly what is known as coarse-to-fine alignment. I(x + x 2 N

31 Questions Why is LK sensitive to the initial guess? Why can t you erform the otimization in a single shot?

32 Today Review - LK Algorithm Inverse Comosition Robust Extensions

33 ? Efficient search is essential on mobile and deskto!!

34 Comutation Concerns Unfortunately, the LK algorithm can be comutationally exensive. Requires the re-comutation of the Jacobian matrix at each iteration. J I = T T N 0 T N T... 7 N T With the additional inversion of the Hessian matrix at each iteration. H I = J T I J I Is there any way we can re-comute any of this? 26

35 Linearizing the Temlate? I( + 27

36 Linearizing the Temlate? T (0) T 27

37 Linearizing the Temlate? T (0) T T T I() Why is this useful if the temlate must be static? 27

38 Simle Examle arg min N X n=1 W(x n ; n; T W(x n ; 0) 2 2

39 Simle Examle arg min N X n=1 W(x n ; n; ) x 0 T W(x n ; 0) 2 2 x 0 3 where: x 0 = W(x; ) x = W(x; 0) + x 0 2 x 1 x 3 x 2

40 Simle Examle arg min NX n=1 W(x n ; ) W(x n ; n ; T 2 2

41 Simle Examle arg min NX n=1 W(x n ; ) W(x n ; 0) x 0 n ; T 2 2 x 0 3 where: x 0 = W(x; ) 0 + x = W(x; 0) x 0 2 x 1 x 3 x 2

42 Simle Examle arg min NX n=1 W(x n ; ) W(x n ; 0) x 0 n ; T 2 2 Static x 0 3 where: x 0 = W(x; ) 0 + x = W(x; 0) x 0 2 x 1 x 3 x 2

43 Relating and In general one can show, W(x; ) =W(x; ) x 0 1 x 0 3 where: x 0 = W(x; ) x = W(x; 0) 0 + x 1 x 0 2 x 3 x 2 30

44 Relating and In general one can show, W(x; ) W(x; ) for non-linear war x 0 1 x 0 3 where: x 0 = W(x; ) x = W(x; 0) 0 + x 1 x 0 2 x 3 x 2 30

45 Relating and Secifically for an affine war one can show, W(x; ) =W(x; ) 31

46 Relating and Secifically for an affine war one can show, M() x = M( ) x where, M() = x = ale x 1 31

47 Relating and Secifically for an affine war one can show, M( ) 1 M() x = x x 0 1 x 0 3 where: x 0 = W(x; ) x = W(x; 0) 0 + x 0 2 x 1 x 3 x 2

48 Relating and Secifically for an affine war one can show, M( ) 1 M() x = x x 0 1 x 0 3 where: x 0 = W(x; ) x = W(x; 0) 1 x 0 2 x 1 x 3 x 2

49 Relating and Secifically for an affine war one can show, M( ) 1 M() x = x also, M( + )x = x x 0 1 x 0 1 x 0 3 where: x 0 = W(x; ) x = W(x; 0) + x 0 3 where: x 0 = W(x; ) x = W(x; 0) 1 x 0 2 x 1 x 0 2 x 1 x 3 x 2 x 3 x 2

50 Relating and Secifically for an affine war one can show, M( ) 1 M() x = x also, M( + )x = x therefore, M( + ) = M( ) 1 M() = I x 0 1 x 0 1 x 0 3 where: x 0 = W(x; ) x = W(x; 0) + x 0 3 where: x 0 = W(x; ) x = W(x; 0) 1 x 0 2 x 1 x 0 2 x 1 x 3 x 2 x 3 x 2

51 Relating and Extending the affine war to the general form, M( ) 1 M() x = W(x; 1 ) x 0 1 x 0 3 where: x 0 = W(x; ) x = W(x; 0) 1 x 0 2 x 1 x 3 x 2 33

52 Inverse Comosition Algorithm Actual algorithm is just the alication of the following stes, Ste 1: Ste 2: arg min T T I() 2 2! 1 kee alying stes until converges. 34

53 Inverse Comosition Algorithm Actual algorithm is just the alication of the following stes, Ste 1: Ste 2: = H 1 T JT T [I() T(0)]! 1 kee alying stes until converges. J T T H T = J T T J T 35

54 Inverse Comosition Algorithm Actual algorithm is just the alication of the following stes, Ste 1: Ste 2: = H 1 T JT T [I() T(0)] Static! 1 Inverse Comosition kee alying stes until converges. J T T H T = J T T J T 35

55 Linearizing T = ;) T... 0 T N N ;) T Constantly changing T... 7 N T = 2 6 (x 1 T T (x N T N Static T... 7 N T 36

56 Linearizing T = ;) T... 0 T N N ;) T Constantly changing T... 7 N T = 2 6 (x 1 T T (x N T N Static T... 7 N T 36

57 Rules for Comositional Wars Not just alicable to affine wars, can be alied to any set of wars that: have an identity war (i.e. ) W(x; 0) =x set of wars must be closed under comosition, W(W(x; 1 ); 2 )!W(x; 2 1 ) for examle a homograhy, where, 3 = 2 1 det( )=+1 37

58 Inverse Comositional Performance Temlate area Intensity (fs) Table 2. Planar temlate tracking run Planar temlate tracking runtime in frames er second on single core Intel Core i7 2.8 Ghz. H. Alismail, B. Browning, S. Lucey Bit-Planes: Dense Subixel Alignment of Binary Descritors. ACCV

59 Today Review - LK Algorithm Inverse Comosition Robust Extensions

60 Robust Error Functions Least squares criterion is not robust to outliers (e.g. occlusion, illumination) For examle, the two outliers here cause the fitted line to be quite wrong. Robust error functions can be helful here.

61 Robust Error Functions Least squares criterion is not robust to outliers (e.g. occlusion, illumination) For examle, the two outliers here cause the fitted line to be quite wrong. Robust error functions can be helful here.

62 x Robust Error Functions arg min {T T I()} for examle, {x} = x 1! L 1 error L 2 {x} Robust Error L 1

63 Robust Error Functions Comutational inefficiencies cree into inverse comosition when one dearts from the classical L2 distance. Much better results for fast tracking have been achieved recently using robust reresentations (e.g. Bristow et al.).

64 Robust Reresentations 1. Comute image gradients 2. Pool into local histograms 3. Concatenate histograms 4. Normalize histograms! R N : R N K

65 Robust IC Algorithm Actual algorithm is just the alication of the following stes, Ste 1: Ste 2: arg min {T (0)} {T T {I()} 2 2! 1 kee alying stes until converges. 44

66 Descritor Performance BitPlanes Raw Pixels Temlate area Intensity BitPlanes le 2. Planar temlate tracking runtime in fram Planar temlate tracking runtime in frames er second on single core Intel Core i7 2.8 Ghz.

67 More to read Parameterizing Homograhies CMU-RI-TR Simon Baker, Ankur Datta, and Takeo Kanade The Robotics Institute Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA The motion of a lane can be described by a homograhy. We study how to arameterize homograhies to maximize lane estimation erformance. We comare the usual 3 3 matrix arameterization with a arameterization that combines 4 fixed oints in one of the images with 4 variable oints in the other image. We emirically show that this 4t arameterization is far suerior. We also comare both arameterizations with a variety of direct arameterizations. In the case of unknown relative orientation, we comare with a direct arameterization of the lane equation, and the rotation and translation of the camera(s). We show that the direct arameterization is both less accurate and far less robust than the 4-oint arameterization. We exlain the oor erformance using a measure of indeendence of the Jacobian images. In the fully calibrated setting, the direct arameterization just consists of 3 arameters of the lane equation. We show that this arameterization is far more robust than the 4-oint arameterization, but only aroximately as accurate. In the case of a moving stereo rig we find that the direct arameterization of lane equation, camera rotation and translation erforms very well, both in terms of accuracy and robustness. This is in contrast to the corresonding direct arameterization in the case of unknown relative orientation. Finally, we illustrate the use of lane estimation in 2 automotive alications. Baker & Matthews, Lucas-Kanade 20 Years On: A Unifying Framework, IJCV Bristow & Lucey, In Defense of Gradient-Based Alignment on Densely Samled Sarse Features, Sringer Book on Dense Registration Methods Baker, Datta & Kanade Parameterizing Homograhies, CMU Tech Reort In Defense of Gradient-Based Alignment on Densely Samled Sarse Features Hilton Bristow and Simon Lucey Abstract In this chater, we exlore the surrising result that gradientbased continuous otimization methods erform well for the alignment of image/object models when using densely samled sarse features (HOG, dense SIFT, etc.). Gradient-based aroaches for image/object alignment have many desirable roerties inference is tyically fast and exact, and diverse constraints can be imosed on the motion of oints. However, the resumtion that gradients redicted on sarse features would be oor estimators of the true descent direction has meant that gradient-based otimization is often overlooked in favour of grah-based otimization. We show that this intuition is only artly true: sarse features are indeed oor redictors of the error surface, but this has no imact on the actual alignment erformance. In fact, for general object categories that exhibit large geometric and aearance variation, sarse features are integral to achieving any convergence whatsoever. How the descent directions are redicted becomes an imortant consideration for these descritors. We exlore a number of strategies for estimating gradients, and show that estimating gradients via regression in a manner that exlicitly handles outliers imroves alignment erformance substantially. To illustrate the general alicability of gradient-based methods to the alignment of challenging object categories, we erform unsuervised ensemble alignment on a series of non-rigid animal classes from ImageNet. Hilton Bristow Queensland University of Technology, Australia. hilton.bristow@gmail.com Simon Lucey The Robotics Institute, Carnegie Mellon University, USA. slucey@cs.cmu.edu 1