Motion estimation. Digital Visual Effects Yung-Yu Chuang. with slides by Michael Black and P. Anandan

Transcription

1 Motion estimation Digital Visual Effects Yung-Yu Chuang with slides b Michael Black and P. Anandan

2 Motion estimation Parametric motion image alignment Tracking Optical flow

3 Parametric motion direct method for image stitching

4 Tracking

5 Optical flow

6 Three assumptions Brightness consistenc Spatial coherence Temporal persistence

7 Brightness consistenc mage measurement e g brightness in a small region mage measurement e.g. brightness in a small region remain the same although their location ma change.

8 Spatial coherence Neighboring points in the scene tpicall belong to the same surface and hence tpicall have similar motions. Since the also project to nearb piels in the image we epect spatial coherence in image flow.

9 Temporal persistence The image motion of a surface patch changes graduall over time.

10 mage registration Goal: register a template image T and an input image where = T. warp so that it matches T mage alignment: and T are two images Tracking: T is a small patch around a point p in the image at t. is the image at time t+1. Optical flow: T and are patches th of images at t and t+1. T fied warp

11 Simple approach for translation Minimize brightness difference E u v u v T 2

12 Simple SSD algorithm For each offset u v compute Euv; Choose u v which minimizes Euv; Problems: Not efficient No sub-piel accurac

13 Lucas-Kanade algorithm

14 Newton s method Root finding for f=0 March and test signs Determine Δ small slow; large miss

15 Newton s method Root finding for f=0

16 Newton s method Root finding for f=0 T l i Talor s epansion: 2 '' 1 ' f f f f '' 2 ' f f f f ' f f f f f f f ' n n n f f ' 1 n n n f f n f

17 Newton s method Root finding for f=0 n f n f ' n n 2 1 0

18 Newton s method pick up = 0 iterate compute f f ' update b +Δ until converge Finding root is useful for optimization because Minimize g find root for f=g =0

19 Lucas-Kanade algorithm T v u v u E 2 u v v u u v T 2 v u T u E 2 0 v u T v E 2 0 v

20 Lucas-Kanade algorithm v u T E 2 0 u v u T E 2 0 v u T v 2 0 T v u T v u 2 2 T v u 2 T T v u 2 2 T v

21 Lucas-Kanade algorithm iterate shift with u v shift with uv compute gradient image compute error image T compute error image T- compute Hessian matri solve the linear sstem solve the linear sstem uv=uv+ u v til until converge T 2 T v u 2

22 Parametric model T v u v u E 2 W;p p 2 T E Our goal is to find p to minimize Ep W;p p T E d p to minimize Ep for all in T s domain T d d p d d W;p translation d d d 1 d A W;p affine T d d d 1 1 A d W;p affine T d d d d d d p

23 Parametric model minimize with respect to Δp W;p Δp T W W W;p Δp W;p Δp p W W;p Δp W;p Δp p W W;p Δp pp minimize W Δp W;p ; p T p 2 2

24 Parametric model image gradient warped image target image W 2 image gradient Δp p W p W; T Jacobian of the warp p W p W p W p W W n n p W p W p W p p p p W p p W n p p p p 2 1

25 Jacobian matri The Jacobian matri is the matri of all first-order partial derivatives of a vector-valued function partial derivatives of a vector-valued function. m n F R R : 2 1 n F n m n n f f f f f or 2 1 n F J n f f or 2 1 m f f f m m n f f n n m m 1 Δ Δ F J F F

26 Jacobian matri : R R F t r sin cos cos sin sin r v r u v u t r F t t t r cos v u u u t t r t v v v u u r u r J F v v r v cos sin sin cos sin sin sin sin cos cos cos sin r r r r 0 sin cos cos s s cos s s r

27 Parametric model image gradient warped image target image W 2 image gradient Δp p W p W; T Jacobian of the warp p W p W p W p W W n n p W p W p W p p p p W p p W n p p p p 2 1

28 Jacobian of the warp W W W W n W W W p p p W p p W 2 1 For eample for affine n p p p p 2 1 For eample for affine d d d d d d 1 1 W;p d d d d d d W;p p W p d d d d d d

29 Parametric model W W;p Δp T p arg min Δp T W W 0 W;p Δp T p p T 1 W Δp H T W;p p 2 Approimated Hessian H W W p p W T

30 Lucas-Kanade algorithm iterate 1 warp with W;p 2 compute error image T-Wp 3 compute gradient image with Wp W 4 evaluate Jacobian at ;p p W 5 compute p 6 compute Hessian W W 7 compute T T W;p p 8 solve Δp 9 update p b p+ Δp until converge T 1 W Δp H T W;p p

31 1 W Δp H T W;p p T

32 Coarse-to-fine strateg J pramid construction a in J warp J w refine + a J warp J w refine + a a pramid construction J J w warp refine a + a out

33 Application of image alignment

34 Direct vs feature-based Direct methods use all information and can be ver accurate but the depend on the fragile brightness constanc assumption. terative approaches require initialization. Not robust to illumination change and noise images. n earl das direct method is better. Feature based methods are now more robust and potentiall faster. Even better it can recognize panorama without initialization.

35 Tracking

36 Tracking t u v +u+vt+1

37 Tracking 0 1 t t v u brightness constanc 0 t t t v t u t t 0 t t v t u t 0 v u optical flow constraint equation 0 t v u optical flow constraint equation

38 Optical flow constraint equation

39 Multiple constraints

40 Area-based method Assume spatial smoothness

41 Area-based method Assume spatial smoothness E u v u v t 2

42 Area-based method must be invertible

43 Area-based method The eigenvalues tell us about the local image structure. The also tell us how well we can estimate the flow in both directions. Link to Harris corner detector.

44 Tetured area

45 Edge

46 Homogenous area

47 KLT tracking Select features b min 1 2 Monitor features b measuring dissimilariti il i

48 Aperture problem

49 Aperture problem

50 Aperture problem

51 Demo for aperture problem eathing_square.htm htm rberpole_llusion.htm

52 Aperture problem Larger window reduces ambiguit but easil violates spatial smoothness assumption

53

54

55 KLT tracking clemson edu/~stb/klt/

56 KLT tracking clemson edu/~stb/klt/

57 SFT tracking matching actuall Frame 0 Frame 10

58 SFT tracking Frame 0 Frame 100

59 SFT tracking Frame 0 Frame 200

60 KLT vs SFT tracking KLT has larger accumulating error; partl because our KLT implementation doesn t have affine transformation? SFT is surprisingl ii robust Combination of SFT and KLT eample

61 Rotoscoping Ma Fleischer

62 Tracking for rotoscoping

63 Tracking for rotoscoping

64 Waking life 2001

65 A Scanner Darkl 2006 Rotoshop a proprietar software. Each minute of animation required 500 hours of work.

66 Optical flow

67 Single-motion assumption Violated b Motion discontinuiti i Shadows Transparenc Specular reflection

68 Multiple motion

69 Multiple motion

70 Simple problem: fit a line

71 Least-square fit

72 Least-square fit

73 Robust statistics Recover the best fit for the majorit of the data Detect and reject outliers

74 Approach

75 Robust weighting Truncated quadratic

76 Robust weighting Geman & McClure

77 Robust estimation

78 Fragmented occlusion

79

80

81 Regularization and dense optical flow Neighboring points in the scene tpicall belong to the same surface and hence tpicall have similar motions. Since the also project to nearb piels in the image we epect spatial coherence in image flow.

82

83

84

85

86

87

88 nput image Horizontal motion Vertical motion

89

90

91 Application of optical flow video matching

92

93 nput for the NPR algorithm

94 Brushes

95 Edge clipping

96 Gradient

97 Smooth gradient

98 Tetured brush

99 Edge clipping

100 Temporal artifacts Frame-b-frame application of the NPR algorithm

101 Temporal coherence

102 References B.D. Lucas and T. Kanade An terative mage Registration Technique with an Application to Stereo Vision Proceedings of the 1981 DARPA mage Understanding di Workshop 1981 pp Bergen J. R. and Anandan P. and Hanna K. J. and Hingorani R. Hierarchical Model-Based Motion Estimation ECCV 1992 pp J. Shi and C. Tomasi Good Features to Track CVPR 1994 pp Michael Black and P. Anandan The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields Computer Vision and mage Understanding 1996 pp S. Baker and. Matthews Lucas-Kanade 20 Years On: A Unifing Framework nternational Journal of Computer Vision pp Peter Litwinowicz Processing mages and Video for An mpressionist Effects SGGRAPH Aseem Agarwala Aaron Hertzman David Salesin and Steven Seitz Keframe-Based Tracking for Rotoscoping and Animation SGGRAPH 2004 pp