Iterative Image Registration: Lucas & Kanade Revisited. Kentaro Toyama Vision Technology Group Microsoft Research

Transcription

1 Iterative Image Registration: Lucas & Kanade Revisited Kentaro Toyama Vision Technology Group Microsoft Research

2 Every writer creates his own precursors. His work modifies our conception of the past, as it will modify the future. Jorge Luis Borges

3 History Lucas & Kanade (IUW 1981) Bergen, Anandan, Hanna, Hingorani (ECCV 1992) Shi & Tomasi (CVPR 1994) Szeliski & Coughlan (CVPR 1994) Szeliski (WACV 1994) Black & Jepson (ECCV 1996) Hager & Belhumeur (CVPR 1996) Bainbridge-Smith & Lane (IVC 1997) Gleicher (CVPR 1997) Sclaroff & Isidoro (ICCV 1998) Cootes, Edwards, & Taylor (ECCV 1998) LK BAHH ST SC S BJ HB BL G SI CET

4 Image Registration

5 Applications

6 Applications Stereo LK BAHH ST SC S BJ HB BL G SI CET

7 Applications Stereo Dense optic flow LK BAHH ST SC S BJ HB BL G SI CET

8 Applications Stereo Dense optic flow Image mosaics LK BAHH ST SC S BJ HB BL G SI CET

9 Applications Stereo Dense optic flow Image mosaics Tracking LK BAHH ST SC S BJ HB BL G SI CET

10 Applications Stereo Dense optic flow Image mosaics Tracking Recognition? LK BAHH ST SC S BJ HB BL G SI CET

11 Lucas & Kanade Derivation #1

12 L&K Derivation 1 I 0 (x)

13 L&K Derivation 1 h I 0 (x) I 0 (x+h)

14 L&K Derivation 1 h I 0 (x) I(x)

15 L&K Derivation 1 h I 0 (x) I(x)

16 L&K Derivation 1 I 0 (x) I(x) R

17 L&K Derivation 1 I 0 (x) I(x)

18 L&K Derivation 1 h 0 I 0 (x) I(x)

19 L&K Derivation 1 I 0 (x+h 0 ) I(x)

20 L&K Derivation 1 I 0 (x+h 1 ) I(x)

21 L&K Derivation 1 I 0 (x+h k ) I(x)

22 L&K Derivation 1 I 0 (x+h f ) I(x)

23 Lucas & Kanade Derivation #2

24 L&K Derivation 2 Sum-of-squared-difference (SSD) error E(h) = Σ [ I(x) - I 0 (x+h) ] 2 x ε R E(h) Σ [ I(x) - I 0 (x) - hi 0 (x) ] 2 x ε R

25 L&K Derivation 2 = 0 Σ 2[I 0 (x)(i(x) - I 0 (x) ) - hi 0 (x) 2 ] x ε R h Σ I 0 (x)(i(x) - I 0 (x)) x ε R Σ I 0 (x) 2 x ε R

26 Comparison h Σx w(x)[i(x) - I 0 (x)] I 0 (x) Σ w(x) x h Σ I 0 (x)[i(x) - I 0 (x)] x Σ I 0 (x) 2 x

27 Comparison h Σx w(x)[i(x) - I 0 (x)] I 0 (x) Σ w(x) x h Σ I 0 (x)[i(x) - I 0 (x)] x Σ I 0 (x) 2 x

28 Generalizations

29 Original Σ E ( h ) = [I( x + h ) - I (x)] 2 0 x ε R

30 Original Dimension of image Σ E ( h ) = [I( x + h ) - (x)] 2 x ε R I 0 1-dimensional LK BAHH ST SC S BJ HB BL G SI CET

31 Generalization 1a Dimension of image Σ E ( h ) = [I( x + h ) - (x) ] 2 x ε R I 0 2D: LK BAHH ST SC S BJ HB BL G SI CET

32 Generalization 1b Dimension of image Σ E ( h ) = [I( x + h ) - (x) ] 2 x ε R I 0 Homogeneous 2D: LK BAHH ST SC S BJ HB BL G SI CET

33 Problem A Does the iteration converge? LK BAHH ST SC S BJ HB BL G SI CET

34 Local minima: Problem A

35 Local minima: Problem A

36 Problem B Zero gradient: h -Σ I 0 (x)(i(x) - I 0 (x)) x ε R Σ I 0 (x) 2 x ε R h is undefined if Σ I 0 (x) 2 is zero x ε R LK BAHH ST SC S BJ HB BL G SI CET

37 Problem B Zero gradient:?

38 Problem B Aperture problem: h y -Σ (x)(i(x) - I 0 (x)) x ε R Σ x ε R 2 LK BAHH ST SC S BJ HB BL G SI CET

39 Problem B No gradient along one direction:?

40 Solutions to A & B Possible solutions: Manual intervention LK BAHH ST SC S BJ HB BL G SI CET

41 Solutions to A & B Possible solutions: Manual intervention Zero motion default LK BAHH ST SC S BJ HB BL G SI CET

42 Solutions to A & B Possible solutions: Manual intervention Zero motion default Coefficient dampening LK BAHH ST SC S BJ HB BL G SI CET

43 Solutions to A & B Possible solutions: Manual intervention Zero motion default Coefficient dampening Reliance on good features LK BAHH ST SC S BJ HB BL G SI CET

44 Solutions to A & B Possible solutions: Manual intervention Zero motion default Coefficient dampening Reliance on good features Temporal filtering LK BAHH ST SC S BJ HB BL G SI CET

45 Solutions to A & B Possible solutions: Manual intervention Zero motion default Coefficient dampening Reliance on good features Temporal filtering Spatial interpolation / hierarchical estimation LK BAHH ST SC S BJ HB BL G SI CET

46 Solutions to A & B Possible solutions: Manual intervention Zero motion default Coefficient dampening Reliance on good features Temporal filtering Spatial interpolation / hierarchical estimation Higher-order terms LK BAHH ST SC S BJ HB BL G SI CET

47 Original Σ E ( h ) = [I( x + h ) - I (x) ] 2 0 x ε R

48 Original Transformations/warping of image Σ E ( h ) = [I( x + h ) - I 0 (x) ] 2 x ε R Translations: LK BAHH ST SC S BJ HB BL G SI CET

49 Problem C What about other types of motion?

50 Generalization 2a Transformations/warping of image Σ E ( A, h) = [I(Ax +h) - (x) ] 2 x ε R I 0 Affine: LK BAHH ST SC S BJ HB BL G SI CET

51 Generalization 2a Affine:

52 Generalization 2b Transformations/warping of image Σ E ( A ) = [I( A x ) - (x) ] 2 x ε R I 0 Planar perspective: LK BAHH ST SC S BJ HB BL G SI CET

53 Generalization 2b Affine + Planar perspective:

54 Generalization 2c Transformations/warping of image Σ E ( h ) = [I(f(x, h)) - (x) ] 2 x ε R I 0 Other parametrized transformations LK BAHH ST SC S BJ HB BL G SI CET

55 Generalization 2c Other parametrized transformations

56 Problem B h -Σ I 0 (x)(i(x) - I 0 (x)) x ε R Σ I 0 (x) 2 x ε R Generalized aperture problem: h ~ -(J T J) -1 J (I(f(x,h)) - I 0 (x)) LK BAHH ST SC S BJ HB BL G SI CET

57 Problem B Generalized aperture problem:?

58 Original Σ E ( h ) = [I( x + h ) - I (x) ] 2 0 x ε R

59 Original Image type Σ E ( h ) = [I( x + h ) - (x) ] 2 x ε R I 0 Grayscale images LK BAHH ST SC S BJ HB BL G SI CET

60 Generalization 3 Image type Σ E ( h ) = I( x + h ) - I 0 (x) 2 x ε R Color images LK BAHH ST SC S BJ HB BL G SI CET

61 Original Σ E ( h ) = [I( x + h ) - I (x) ] 2 0 x ε R

62 Original Constancy assumption Σ E ( h ) = [I( x + h ) - I 0 (x) ] 2 x ε R Brightness constancy LK BAHH ST SC S BJ HB BL G SI CET

63 Problem C What if illumination changes?

64 Generalization 4a Constancy assumption )=Σ E ( h,α,β [I( x + h ) - αi 0 (x)+β] 2 x ε R Linear brightness constancy LK BAHH ST SC S BJ HB BL G SI CET

65 Generalization 4a

66 Generalization 4b Constancy assumption Σ E ( h,λ) = [I( x + h ) - λ Τ B(x] ) 2 x ε R Illumination subspace constancy LK BAHH ST SC S BJ HB BL G SI CET

67 Problem C What if the texture changes?

68 Generalization 4c Constancy assumption Σ E ( h,λ) = [I( x + h ) - λ Τ B(x) ] 2 x ε R Texture subspace constancy LK BAHH ST SC S BJ HB BL G SI CET

69 Problem D Convergence is slower as #parameters increases.

70 Solutions to D Faster convergence: Coarse-to-fine, filtering, interpolation, etc. LK BAHH ST SC S BJ HB BL G SI CET

71 Solutions to D Faster convergence: Coarse-to-fine, filtering, interpolation, etc. Selective parametrization LK BAHH ST SC S BJ HB BL G SI CET

72 Solutions to D Faster convergence: Coarse-to-fine, filtering, interpolation, etc. Selective parametrization Offline precomputation LK BAHH ST SC S BJ HB BL G SI CET

73 Solutions to D Faster convergence: Coarse-to-fine, filtering, interpolation, etc. Selective parametrization Offline precomputation Difference decomposition LK BAHH ST SC S BJ HB BL G SI CET

74 Solutions to D Difference decomposition

75 Solutions to D Difference decomposition

76 Solutions to D Faster convergence: Coarse-to-fine, filtering, interpolation, etc. Selective parametrization Offline precomputation Difference decomposition Improvements in gradient descent LK BAHH ST SC S BJ HB BL G SI CET

77 Solutions to D Faster convergence: Coarse-to-fine, filtering, interpolation, etc. Selective parametrization Offline precomputation Difference decomposition Improvements in gradient descent Multiple estimates of spatial derivatives LK BAHH ST SC S BJ HB BL G SI CET

78 Solutions to D Multiple estimates / state-space sampling

79 Generalizations Modifications made so far: Σ x ε R [I( x + h ) - I (x)] 2 0

80 Original Error norm Σ E ( h ) = [I( x + h ) - I 0 (x) ] 2 x ε R Squared difference: LK BAHH ST SC S BJ HB BL G SI CET

81 Problem E What about outliers?

82 Generalization 5a Error norm Σ E ( h ) = ρ(i( x + h ) - I 0 (x) ) x ε R Robust error norm: LK BAHH ST SC S BJ HB BL G SI CET

83 Original Σ E ( h ) = [I( x + h ) - I (x) ] 2 0 x ε R

84 Original Image region / pixel weighting Σ E ( h ) = [I( x + h ) - I 0 (x) ] 2 x ε R Rectangular: LK BAHH ST SC S BJ HB BL G SI CET

85 Problem E What about background clutter?

86 Generalization 6a Image region / pixel weighting Σ E ( h ) = [I( x + h ) - I 0 (x) ] 2 x ε R Irregular: LK BAHH ST SC S BJ HB BL G SI CET

87 Problem E What about foreground occlusion?

88 Generalization 6b Image region / pixel weighting Σ E ( h ) = [I( x + h ) - I 0 (x) ] 2 w(x) x ε R Weighted sum: LK BAHH ST SC S BJ HB BL G SI CET

89 Generalizations Modifications made so far: Σ x ε R [I( x + h ) - I (x)] 2 0

90 Generalizations: Summary Σ E ( h ) = [I( x + h ) - I (x)] 2 0 x ε R Σ E ( h ) = ρ(i( f(x, h) ) - λβ(x )) w(x) x ε R

91 Foresight Lucas & Kanade (IUW 1981) Bergen, Anandan, Hanna, Hingorani (ECCV 1992) Shi & Tomasi (CVPR 1994) Szeliski & Coughlan (CVPR 1994) Szeliski (WACV 1994) Black & Jepson (ECCV 1996) Hager & Belhumeur (CVPR 1996) Bainbridge-Smith & Lane (IVC 1997) Gleicher (CVPR 1997) Sclaroff & Isidoro (ICCV 1998) Cootes, Edwards, & Taylor (ECCV 1998) LK BAHH ST SC S BJ HB BL G SI CET

92 Summary Generalizations Dimension of image Image transformations / motion models Pixel type Constancy assumption Error norm Image mask L&K? Y Y n Y n Y

93 Summary Common problems: Local minima Aperture effect Illumination changes Convergence issues Outliers and occlusions L&K? Y maybe Y Y n

94 Summary Mitigation of aperture effect: Manual intervention Zero motion default Coefficient dampening Elimination of poor textures Temporal filtering Spatial interpolation / hierarchical Higher-order terms L&K? n n n n Y Y n

95 Summary Better convergence: Coarse-to-fine, filtering, etc. Selective parametrization Offline precomputation Difference decomposition Improvements in gradient descent Multiple estimates of spatial derivatives L&K? Y n maybe maybe maybe maybe

96 Hindsight Lucas & Kanade (IUW 1981) Bergen, Anandan, Hanna, Hingorani (ECCV 1992) Shi & Tomasi (CVPR 1994) Szeliski & Coughlan (CVPR 1994) Szeliski (WACV 1994) Black & Jepson (ECCV 1996) Hager & Belhumeur (CVPR 1996) Bainbridge-Smith & Lane (IVC 1997) Gleicher (CVPR 1997) Sclaroff & Isidoro (ICCV 1998) Cootes, Edwards, & Taylor (ECCV 1998)