Problems & Techniques

Transcription

1 Vsual Moton Estmaton Problems & Technques Prnceton Unversty COS 429 Lecture Oct. 11, 2007 Harpreet S. Sawhney

2 Outlne 1. Vsual moton n the Real World 2. The vsual moton estmaton problem 3. Problem formulaton: Estmaton through model-based algnment 4. Coarse-to-fne drect estmaton of model parameters 5. Progressve complexty and robust model estmaton 6. Mult-modal algnment 7. Drect estmaton of parallax/depth/optcal flow 8. Glmpses of some applcatons

3 Types of Vsual Moton n the Real World

4 Smple Camera Moton : Pan & Tlt Camera Does Not Change Locaton

5 Apparent Moton : Pan & Tlt Camera Moves a Lot

6 ndependent Object Moton Objects are the Focus Camera s more or less steady

7 ndependent Object Moton wth Camera Pan Most common scenaro for capturng performances

8 General Camera Moton Large changes n camera locaton & orentaton

9 Vsual Moton due to Envronmental Effects Every pxel may have ts own moton

10 The Works! General Camera & Object Motons

11 Why s Analyss and Estmaton of Vsual Moton mportant?

12 Vsual Moton Estmaton as a means of extractng nformaton Content n Dynamc magery...extract nformaton behnd pxel data... Foreground Vs. Background

13 nformaton Content n Dynamc magery...extract nformaton behnd pxel data... Foreground Vs. Background Extended Scene Geometry

14 nformaton Content n Dynamc magery...extract nformaton behnd pxel data... Foreground Vs. Background Temporal Persstence Extended Scene Geometry Layers & Mosacs Segment,Track,Fngerprnt Movng Objects Layers wth 2D/3D Scene Models Layered, Moton, Structure & Appearance Analyss provdes Compact Representaton for Manpulaton & Recognton of Scene Content

15 An Example Pn-hole camera model A Pannng Camera Pure rotaton of the camera Multple mages related through a 2D projectve transformaton: also called a homography n the specal case for camera pan, wth small frame-to-frame rotaton, and small feld of vew, the frames are related through a pure mage translaton

16 Y Pn-hole Camera Model y f Z y = f Y Z p fp

17 Camera Rotaton (Pan) Y y Z f Y y = f p f P Z P = R P p R p

18 Y Camera Rotaton (Pan) f y y Y Z = f p f P P p Z = R P R p

19 mage Moton due to Rotatons does not depend on the depth / structure of the scene Verfy the same for a 3D scene and 2D camera

20 Pn-hole Camera Model Y y f Z y = f Y Z p fp

21 Camera Translaton (Ty) Y y X X X X f Z Y y = f p f P P = P + T Z

22 Translatonal Dsplacement Y y = f Z Y + Ty y = f Z Ty y - y = f Z y y = f = f Y Z Y Z + Tz Tz y - y = -y Z mage Moton due to Translaton s a functon of the depth of the scene

23

24 Sample Dsplacement Felds Render scenes wth varous motons and plot the dsplacement felds

25 Moton Feld vs. Optcal Flow Moton Feld : 2D projectons of 3D dsplacement vectors due to camera and/or object moton Ty Y wy wx P p p Tx X P wz Z Tz Optcal Flow : mage dsplacement feld that measures the apparent moton of brghtness patterns

26 Moton Feld vs. Optcal Flow Lambertan ball rotatng n 3D Moton Feld? Optcal Flow? Courtesy : Mchael Brown.edu mage:

27 Moton Feld vs. Optcal Flow Statonary Lambertan ball wth a movng pont lght source Moton Feld? Optcal Flow? Courtesy : Mchael Brown.edu mage :

28 A Herarchy of Models Taxonomy by Bergen,Anandan et al. 92 Parametrc moton models 2D translaton, affne, projectve, 3D pose [Bergen, Anandan, et.al. 92] Pecewse parametrc moton models 2D parametrc moton/structure layers [Wang&Adelson 93, Ayer&Sawhney 95] Quas-parametrc 3D R, T & depth per pxel. [Hanna&Okumoto 91] Plane+parallax [Kumar et.al. 94, Sawhney 94] Pecewse quas-parametrc moton models 2D parametrc layers + parallax per layer [Baker et al. 98] Non-parametrc Optc flow: 2D vector per pxel [Lucas&Kanade 81, Bergen,Anandan et.al. 92]

29 Sparse/Dscrete Correspondences & Dense Moton Estmaton

30 Dscrete Methods Feature Correlaton & RANSAC

31 Vsual Moton through Dscrete Correspondences p p mages may be separated by tme, space, sensor types n general, dscrete correspondences are related through a transformaton

32 Dscrete Methods Feature Correlaton & RANSAC

33 Dscrete Correspondences Select corner-lke ponts Match patches usng Normalzed Correlaton Establsh further matches usng moton model

34 Drect Methods for Vsual Moton Estmaton Employ Models of Moton and Estmate Vsual Moton through mage Algnment

35 Characterzng Drect Methods The What Vsual nterpretaton/modelng nvolves spatotemporal mage representatons drectly Not explctly represented dscrete features lke corners, edges and lnes etc. Spato-temporal mages are represented as outputs of symmetrc or orented flters. The output representatons are typcally dense, that s every pxel s explaned, Optcal flow, depth maps. Model parameters are also computed.

36 Drect Methods : The How Algnment of spato-temporal mages s a means of obtanng : Dense Representatons, Parametrc Models

37 Drect Method based Algnment

38 Formulaton of Drect Model-based mage Algnment [Bergen,Anandan et al. 92] 1 (p ) p u(p) 2 (p) p Model mage transformaton as : p) = ( p u( p; Θ)) = ( ) 2( 1 1 p Brghtness Constancy mages separated by tme, space, sensor types

39 Formulaton of Drect Model-based mage Algnment 1 (p ) p u(p) Model mage transformaton as : 1 2(p) = (p u(p;θ)) 2 (p) p mages separated by tme, space, sensor types Reference Coordnate System

40 Formulaton of Drect Model-based mage Algnment 1 (p ) p u(p) Model mage transformaton as : 1 2(p) = (p u(p;θ)) 2 (p) p mages separated by tme, space, sensor types Reference Coordnate System Generalzed pxel Dsplacement

41 Formulaton of Drect Model-based mage Algnment 1 (p ) p u(p) Model mage transformaton as : 1 2(p) = (p u(p;θ)) 2 (p) p mages separated by tme, space, sensor types Reference Coordnate System Generalzed pxel Dsplacement Model Parameters

42 Formulaton of Drect Model-based mage Algnment 1 (p ) p u(p) Model mage transformaton as : 1 2(p) = (p u(p;θ)) 2 (p) p mages separated by tme, space, sensor types Reference Coordnate System Generalzed pxel Dsplacement Model Parameters

43 1 (p ) p u(p) Formulaton of Drect Model-based mage Algnment 2 (p) p Compute the unknown parameters and correspondences whle algnng mages usng optmzaton : Θ mn ρ(r ;σ), r p ) ( p u( p ; )) = 2( 1 Θ What all can be vared? Fltered mage Representatons (to account for llumnaton changes, Mult-modaltes)

44 1 (p ) p u(p) Formulaton of Drect Model-based mage Algnment 2 (p) p Compute the unknown parameters and correspondences whle algnng mages usng optmzaton : Θ mn ρ(r ;σ), r p ) ( p u( p ; )) = 2( 1 Θ Model Parameters What all can be vared? Fltered mage Representatons (to account for llumnaton changes, Mult-modaltes)

45 1 (p ) p u(p) Formulaton of Drect Model-based mage Algnment 2 (p) p Compute the unknown parameters and correspondences whle algnng mages usng optmzaton : Θ mn ρ(r ;σ), r p ) ( p u( p ; )) = 2( 1 Θ Measurng msmatches (SSD, Correlatons) Model Parameters What all can be vared? Fltered mage Representatons (to account for llumnaton changes, Mult-modaltes)

46 1 (p ) p u(p) Formulaton of Drect Model-based mage Algnment 2 (p) p Compute the unknown parameters and correspondences whle algnng mages usng optmzaton : Θ mn ρ(r ;σ), r p ) ( p u( p ; )) = 2( 1 Θ Optmzaton Functon Measurng msmatches (SSD, Correlatons) Model Parameters What all can be vared? Fltered mage Representatons (to account for llumnaton changes, Mult-modaltes)

47 1 (p ) p u(p) Formulaton of Drect Model-based mage Algnment 2 (p) p Compute the unknown parameters and correspondences whle algnng mages usng optmzaton : Θ mn ρ(r ;σ), r p ) ( p u( p ; )) = 2( 1 Θ Optmzaton Functon Measurng msmatches (SSD, Correlatons) Model Parameters What all can be vared? Fltered mage Representatons (to account for llumnaton changes, Mult-modaltes)

48 A Herarchy of Models Taxonomy by Bergen,Anandan et al. 92 Parametrc moton models 2D translaton, affne, projectve, 3D pose [Bergen, Anandan, et.al. 92] Pecewse parametrc moton models 2D parametrc moton/structure layers [Wang&Adelson 93, Ayer&Sawhney 95] Quas-parametrc 3D R, T & depth per pxel. [Hanna&Okumoto 91] Plane+parallax [Kumar et.al. 94, Sawhney 94] Pecewse quas-parametrc moton models 2D parametrc layers + parallax per layer [Baker et al. 98] Non-parametrc Optc flow: 2D vector per pxel [Lucas&Kanade 81, Bergen,Anandan et.al. 92]

49 Plan : Ths Part Frst present the generc normal equatons. Then specalze these for a projectve transformaton. Sdebar nto backward mage warpng. SSD and M-estmators.

50 An teratve Soluton of Model Parameters [Black&Anandan 94 Sawhney 95] Θ mn ρ(r ;σ), r p ) ( p u( p ; )) = 2( 1 Θ Gven a soluton (m) Θ at the mth teraton, fnd δθ by solvng : &( ρ r ) r r & ρ( r ) θ = r l r θ θ r r θ l k l k k w w s a weght assocated wth each measurement.

51 An teratve Soluton of Model Parameters Θ mn ρ(r ;σ), r p ) ( p u( p ; )) = 2( 1 Θ n partcular for Sum-of-Square Dfferences : We obtan the standard normal equatons: ρ SSD = 2 r 2σ 2 r θ r θl θ = r r θ l k l k k Other functons can be used for robust M-estmaton

52 How does ths work for mages? (1) mn Θ r, r p ) ( p u( p ; )) = 2( 1 Θ Let ther be a 2D projectve transformaton between the two mages: p Ρ p p Gven an ntal guess (k) Ρ Frst, warp ( p ) towards ( p ) 1 2

53 How does ths work for mages? (2) mn Θ r, r p ) ( p u( p ; )) = 2( 1 Θ p w w 1 (p ) = 1(p ) = 1 (Ρ ( k) p w ) 1 (p ) w 1 (p w ) 2 (p) p u(p) ( ) w Ρ k p p

54 How does ths work for mages? (3) mn Θ r, r p ) ( p u( p ; )) w w 1 (p ) = 1(p ) = = 2( 1 Θ 1 (Ρ ( k) p w ) p w 1 (p w ) : 1 (p ) Represents mage 1 warped towards the reference mage 2, Usng the current set of parameters w 1 (p w ) 2 (p) p u(p) ( ) w Ρ k p p

55 How does ths work for mages? (4) The resdual transformaton between the warped mage and the reference mage s modeled as: r = 2 (p ) - w 1 (p w - δp w (p w ; δθ)) Where p [ + w D]p d11 d12 d13 D = d21 d22 d23 d31 d32 0

56 How does ths work for mages? (5) The resdual transformaton between the warped mage and the reference mage s modeled as: r = 2 w (p ) - (p w - δp 1 w (p w ;D)) w T w w w p 2(p ) - (p (p ;0)) - D= 0 d d p w = (1 + d11)x + d12y + d d31x + d32y + 1 d21x + (1+ d22 )y + d d31x + d32y w p D x = 0 y x xy xy y 2 D= x 0 y 0 1

57 How does ths work for mages? (6) ) (p - r δ = d p 0 D w D w T Θ r, 2 1 mn 2 ) (p δ = w T D w D w w w T D p d p p T T Hd = g D] [ Ρ Ρ (k) 1) (k + + So now we can solve for the model parameters whle algnng mages teratvely usng warpng and Levenberg-Marquat style optmzaton

58 Sdebar : Backward Warpng Note that we have used backward warpng n the drect algnment of mages. Backward warpng avods holes. mage gradents are estmated n the warped coordnate system. Source mage : Flled Target mage : Empty Blnear Warp w p = Ρ p

59 Sdebar : Backward Warpng Note that we have used backward warpng n the drect algnment of mages. Backward warpng avods holes. mage gradents are estmated n the warped coordnate system. Source mage : Flled Target mage : Empty Bcubc Warp w p = Ρ p

60 teratve Algnment : Result (t) estmate global shft d r (t) d r (t) delay (t 1) warp ˆ ( t 1)

61 How to handle Large Transformatons? [Burt,Adelson 81] Gaussan Pyramd Processng Laplacan A herarchcal framework for fast algorthms A wavelet representaton for compresson, enhancement, fuson A model of human vson

62 teratve Coarse-to-fne Model-based mage Algnment Prmer d(p) - Warper (p) (p+ u(p; Θ)) mn ( Θ 1 2 ) p { R, T, d(p) } { H, e, k(p) } { dx(p), dy(p) } 2

63 Pyramd-based Drect mage Algnment Prmer Coarse levels reduce search. Models of mage moton reduce modelng complexty. mage warpng allows model estmaton wthout dscrete feature extracton. Model parameters are estmated usng teratve nonlnear optmzaton. Coarse level parameters gude optmzaton at fner levels.

64 Applcaton : mage/vdeo Mosacng Drect frame-to-frame mage algnment. Select frames to reduce the number of frames & overlap. Warp algned mages to a reference coordnate system. Create a sngle mosac mage. Assumes a parametrc moton model.

65 Vdeo Mosac Example VdeoBrush 96 Prnceton Chapel Vdeo Sequence 54 frames

66 Unblended Chapel Mosac

67 mage Mosacs Chps are mages. May or may not be captured from known locatons of the camera.

68 Output Mosac

69 Handlng Movng Objects n 2D Parametrc Algnment & Mosacng

70 Θ Generalzed M-Estmaton mn ρ(r ;σ), r p ) ( p u( p ; )) = 2( 1 Θ Gven a soluton (m) Θ at the mth teraton, fnd δθ by solvng : &( ρ r ) r r & ρ( r ) θ = r l r θ θ r r θ l k l k k w Choces of the ρ( r; σ ) ρ& w s a weght assocated wth each measurement. Can be vared to provde robustness to outlers. (r) r SS = functon: 1 σ 2 ρ = SS ρ& 2 r 2σ (r) r GM 2 = ρ GM 2 2 r σ = 2 1+ r σ 2 2σ 2 (σ + r 2 ) 2 2

71 Optmzaton Functons & ther Correspondng Weght Plots Geman-Mclure Sum-of-squares

72 Wth Robust Functons Drect Algnment Works for Non-domnant Movng Objects Too Orgnal two frames Background Algnment

73 Object Deleton wth Layers Orgnal Vdeo Vdeo Stream wth deleted movng object

74 Optc Flow Estmaton r = 2 (p ) w - (p w - δp 1 w (p w ;D)) w w w 2(p ) - (p (p ;0)) - δp δx [ ] w w w w (p ) - (p ) = δ x y δy 2 T Gradent Drecton Flow Vector

75 Normal Flow Constrant At a sngle pxel, brghtness constrant: x u + yv + t = 0 - t Normal Flow

76

77

78

79

80 Computng Optcal Flow: Dscretzaton Look at some neghborhood N: ( ) 0 0 ), ( ), ( want want N ), ( T = + = + b Av v j j j t ( ) 0 0 ), ( ), ( want want N ), ( T = + = + b Av v j j j t = = ), ( ), ( ), ( ), ( ), ( ), ( n n t t t n n j j j j j j M M b A = = ), ( ), ( ), ( ), ( ), ( ), ( n n t t t n n j j j j j j M M b A

81 Computng Optcal Flow: Least Squares n general, overconstraned lnear system Solve by least squares b A A A v b A v A A b Av T 1 T T T want ) ( ) ( 0 = = = + b A A A v b A v A A b Av T 1 T T T want ) ( ) ( 0 = = = +

82 Computng Optcal Flow: Stablty Has a soluton unless C = A T A s sngular [ ] = = = N y N y x N y x N x n n n n j j j j j j T ), ( ), ( ), ( ), ( ), ( ), ( C C A A C M L [ ] = = = N y N y x N y x N x n n n n j j j j j j T ), ( ), ( ), ( ), ( ), ( ), ( C C A A C M L

83 Computng Optcal Flow: Stablty Where have we encountered C before? Corner detector! C s sngular f constant ntensty or edge Use egenvalues of C: to evaluate stablty of optcal flow computaton to fnd good places to compute optcal flow (fndng good features to track) [Sh-Tomas]

84 Example of Flow Computaton

85 Example of Flow Computaton

86 Example of Flow Computaton But ths n general s not the moton feld

87 Moton Feld = Optcal Flow? From brghtness constancy, normal flow: v n T ( E )v = = - E Et E Moton feld for a Lambertan scene: E dn dt T = ρ n = ωxn E T v + E t = ρ T ( ωxn ) Δv = ρ T ( ωxn ) E Ponts wth hgh spatal gradent are the locatons At whch the moton feld can be best estmated By brghtness constancy (the optcal flow)

88 Moton llusons n Human Vson

89 Aperture Problem Too bg: confused by multple motons Too small: only get moton perpendcular to edge

90 Ouch lluson The Ouch lluson, llustrated above, s an lluson named after ts nventor, Japanese artst Hajme Ouch. n ths lluson, the central dsk seems to float above the checkered background when movng the eyes around whle vewng the fgure. Scrollng the mage horzontally or vertcally gve a much stronger effect. The lluson s caused by random eye movements, whch are ndependent n the horzontal and vertcal drectons. However, the two types of patterns n the fgure nearly elmnate the effect of the eye movements parallel to each type of pattern. Consequently, the neurons stmulated by the dsk convey the sgnal that the dsk jtters due to the horzontal component of the eye movements, whle the neurons stmulated by the background convey the sgnal that movements are due to the ndependent vertcal component. Snce the two regons jtter ndependently, the bran nterprets the regons as correspondng to separate ndependent objects (Olveczky et al. 2003).

91 Aksha Ktakao

92 Rotatng Snakes

93