Announcements Motion II No class Wednesda (Happ Thanksgiving) HW4 will be due Frida 1/8 Comment on Non-maximal supression CSE5A Lecture 15 Shi-Tomasi Corner Detector Filter image with a Gaussian. Compute the gradient everwhere. Move window over image and construct C over the window. # I x I x I & C(x, ) = % ( $ % I x I I '( Use linear algebra to ind λ 1 and λ. I the are both large, we have a corner. 1. Let e(x,) = min(λ 1 (x,), λ (x,)). (x,) is a corner i it s local maximum o e(x,) and e(x,) > τ Parameters: Gaussian std. dev, window size, threshold Non-maximum suppression Local max o magnitude o the gradient in the direction o the gradient Using normal at q, ind two points p and r on adjacent rows (or columns). We have a maximum i the value is larger than those at both p and at r. Interpolate to get values. Motion undertsanding From two or more images, motion understanding allows us to 1. Segment objects. Track moving objects 3. Estimate egomotion 4. Estimate 3D scene structure 5. Estimate multiple motions and non-rigid motion Tpical -rame wide-baseline SFM approach 1. Detect corners/eature points in each image. Hpothesize matches between pairs based on image descriptors or SSD 3. From collection o nois pairs, estimate Essential (Fundamental) matrix using RANSAC 4. Compute Epipolar geometr rom inliers. 5. Estimate 3D structure as sparse (or dense) stereo vision Two imaging situations Continuous motion (small, ininitesimal) Large Motion (discrete, widebaseline) 1
Small Motion Motion When objects move at equal speed, those more remote seem to move more slowl. - Euclid, 300 BC The Motion Field Where in the image did a point move? The Motion Field Down and let What causes a motion ield? THE MOTION FIELD 1. Camera moves (translates, rotates). Object s in scene move rigidl 3. Objects articulate (pliers, humans, animals) 4. Objects bend and deorm (ish) 5. Blowing smoke, clouds 6. Multiple movements The instantaneous velocit o all points in an image LOOMING The Focus o Expansion (FOE) Intersection o velocit vector with image plane With just this inormation it is possible to calculate: 1. Direction o motion. Time to collision
Rigid Motion: General Case p! P General Motion Let (x,,z) be unctions o time (x(t), (t), z(t)). The velocit [du/dt, dv/dt] T o the projection o (x,,z) is: Position and orientation o a rigid bod Rotation Matrix & Translation vector!p = T + Ω p Rigid Motion Velocit Vector: T Angular Velocit Vector: ω (or Ω) Ω : Speed o rotation ˆΩ : Direction o rotation Substituting where p=(x,,z) T ields: Motion Field Equation Pure Translation T u zu Tx ω uv ω x u! = ω + ωzv + Tzv T ωuv ωxv v! = + ωx ωzu T: Components o 3-D linear motion ω: Angular velocit vector (u,v): Image point coordinates : depth : ocal length For pure translation, there is no rotation or ω = 0 T u zu Tx ω uv ω x u! = ω + ωzv + Tzv T ωuv ωxv v! = + ωx ωzu Forward Translation & Focus o Expansion [Gibson, 1950] Pure Translation Radial about FOE Parallel ( FOE point at ininit) T = 0 Motion parallel to image plane 3
Sidewas Translation [Gibson, 1950] Pure Rotation: T=0 T u zu Tx ω uv ω x u! = ω + ωzv + Tzv T ωuv ωxv v! = + ωx ωzu Parallel motion vectors ( FOE point at ininit) T = 0 Motion parallel to image plane Independent o T x T T z Independent o Onl unction o (u,v), and ω Rotational MOTION FIELD Pure Rotation: Motion Field on Sphere The instantaneous velocit o points in an image PURE ROTATION ω = (0,0,1) T Motion Field Equation: Estimating Depth u = T zu T x ω +ω z v + ω xuv ω u v = T zv T +ω x ω z u ω uv ω xv Motion I T, ω, and are known or measured, then or each image point (u,v), one can solve or the depth given measured motion (du/dt, dv/dt) at (u,v). = T z u T x u +ω ω z v ω uv x + ω u 4
Problem Deinition: Optical Flow Estimating the motion ield rom images 1. Feature-based (Sect. 8.4. o Trucco & Verri) 1. Detect Features (corners) in an image. Search or the same eatures nearb (Feature tracking). How to estimate pixel motion rom image H to image I? Find pixel correspondences Given a pixel in H, look or nearb pixels o the same color in I. Dierential techniques (Sect. 8.4.1) Ke assumptions color constanc: a point in H looks the same in image I For grascale images, this is brightness constanc small motion: points do not move ver ar Optical Flow Optical Flow Constraint Equation Motion Field ( x + u δt, + v δt ) ( x, ) ( x, ) time t + δt time t Optical Flow: Velocities (u, v) Displacement: (δx, δ ) = (u δt, v δt ) Assume brightness o patch remains same in both images: I(x + u δt, + v δt,t + δt) = I(x,,t) Assume small motion: (Talor expansion o LHS up to irst order) Motion ield exists but no optical low No motion ield but shading changes I(x,,t) + δx + δ + δt = I(x,,t) x t Optical Flow Constraint Equation Elements o Optical Flow Equation dx d + + =0 dt x dt t ( x + u δt, + v δt ) ( x, ) time t ( x, ) time t + δt Optical Flow: Velocities (u, v) Displacement: (δx, δ ) = (u δt, v δt ) Subtracting I(x,,t) rom both sides and dividing b dx d, is the velocit vector o point (x,), e.g., dt dt (u, v) = optical low at (x, ) δt x corner detection Assume small interval, this becomes: I =, is the image gradient at (x,), as used in edge and δx δ + + =0 δt x δt t dx d + + =0 dt x dt t is the dierence between successive images at (x,) t 5
Solving or low Optical low constraint equation : We can measure We want to solve or x,, t dx d, dt dt One equation, two unknowns Measurements I x = x, I =, I t = t Flow vector dx u =, dt d v = dt The component o the optical low in the direction o the image gradient. Optical Flow Constraint Normal Flow Illusion Works Barber Pole Illusion Apparentl an aperture problem Resolving the aperture problem 1. Global consistenc. Compute over window 6
What is the correspondence o P & P Two was to get low 1. Think globall, and regularize over image Contour plots o image intensit in two images Brightness is same along C and along C. Look over window and assume constant motion in the window Horn & Schunck algorithm Additional smoothness constraint : es (( ux + u ) + ( vx + v )) dxd, = besides OF constraint equation term ec ( I xu + I v + = I ) dxd, t de( u, v) = du de( u, v) = dv I I ( I u + I v + I ) x x ( I u + I v + I ) x t t = 0 = 0 minimize es+λec 7
Coarse-to-ine optical low estimation run iterative L-K warp & upsample run iterative L-K... image J H Gaussian pramid o image H image I Gaussian pramid o image I 8