Visul motion Mn slides dpted from S. Seitz, R. Szeliski, M. Pollefes
Outline Applictions of segmenttion to video Motion nd perceptul orgniztion Motion field Opticl flow Motion segmenttion with lers
Video A video is sequence of frmes cptured over time Now our imge dt is function of spce (, ) nd time (t)
Applictions of segmenttion to video Bckground subtrction A sttic cmer is observing scene Gol: seprte the sttic bckground from the moving foreground
Applictions of segmenttion to video Bckground subtrction Form n initil bckground estimte For ech frme: Updte estimte using moving verge Subtrct the bckground estimte from the frme Lbel s foreground ech piel where the mgnitude of the difference is greter thn some threshold Use medin filtering to clen up the results
Applictions of segmenttion to video Bckground subtrction Shot boundr detection Commercil video is usull composed of shots or sequences showing the sme objects or scene Gol: segment video into shots for summriztion nd browsing (ech shot cn be represented b single kefrme in user interfce) Difference from bckground subtrction: the cmer is not necessril sttionr
Applictions of segmenttion to video Bckground subtrction Shot boundr detection For ech frme Compute the distnce between the current frme nd the previous one» Piel-b-piel differences» Differences of color histogrms» Block comprison If the distnce is greter thn some threshold, clssif the frme s shot boundr
Applictions of segmenttion to video Bckground subtrction Shot boundr detection Motion segmenttion Segment the video into multiple coherentl moving objects
Motion nd perceptul orgniztion Sometimes, motion is the onl cue
Motion nd perceptul orgniztion Sometimes, motion is the onl cue
Motion nd perceptul orgniztion Even impoverished motion dt cn evoke strong percept
Motion nd perceptul orgniztion Even impoverished motion dt cn evoke strong percept
Motion nd perceptul orgniztion Even impoverished motion dt cn evoke strong percept
Uses of motion Estimting 3D structure Segmenting objects bsed on motion cues Lerning dnmicl models Recognizing events nd ctivities Improving video qulit (motion stbiliztion)
Motion estimtion techniques Direct methods Directl recover imge motion t ech piel from sptio-temporl imge brightness vritions Dense motion fields, but sensitive to ppernce vritions Suitble for video nd when imge motion is smll Feture-bsed methods Etrct visul fetures (corners, tetured res) nd trck them over multiple frmes Sprse motion fields, but more robust trcking Suitble when imge motion is lrge (10s of piels)
Motion field The motion field is the projection of the 3D scene motion into the imge
Motion field nd prll P(t) is moving 3D point Velocit of scene point: V = dp/dt p(t) = ((t),(t)) is the projection of P in the imge Apprent velocit v in the imge: given b components v = d/dt nd v = d/dt These components re known s the motion field of the imge P(t) V p(t) v P(tdt) p(tdt)
Motion field nd prll V = ( V, V, VZ ) p = f P Z P(t) V P(tdt) To find imge velocit v, differentite p with respect to t (using quotient rule): v = f Z V V 2 Z z P v p(tdt) v = f V V Z z v = f V V Z z p(t) Imge motion is function of both the 3D motion (V) nd the depth of the 3D point (Z)
Motion field nd prll Pure trnsltion: V is constnt everwhere Z V V f v z = Z V V f v z = ), ( 1 v 0 p v z V Z = ( ) V f V f, v 0 =
Motion field nd prll Pure trnsltion: V is constnt everwhere V z is nonzero: v = 1 Z v 0 = ( v 0 V p), Ever motion vector points towrd (or w from) v 0, the vnishing point of the trnsltion direction z ( f V, f ) V
Motion field nd prll Pure trnsltion: V is constnt everwhere V z is nonzero: Ever motion vector points towrd (or w from) v 0, the vnishing point of the trnsltion direction V z is zero: v = 1 Z v 0 = ( v 0 V p), Motion is prllel to the imge plne, ll the motion vectors re prllel The length of the motion vectors is inversel proportionl to the depth Z z ( f V, f ) V
Opticl flow Definition: opticl flow is the pprent motion of brightness ptterns in the imge Idell, opticl flow would be the sme s the motion field Hve to be creful: pprent motion cn be cused b lighting chnges without n ctul motion Think of uniform rotting sphere under fied lighting vs. sttionr sphere under moving illumintion
Estimting opticl flow I(,,t 1) I(,,t) Given two subsequent frmes, estimte the pprent motion field u(,) nd v(,) between them Ke ssumptions Brightness constnc: projection of the sme point looks the sme in ever frme Smll motion: points do not move ver fr Sptil coherence: points move like their neighbors
Brightness Constnc Eqution: ), ( 1),, ( ),, ( ), ( t v u I t I = ), ( ), ( ),, ( 1),, ( v I u I t I t I Linerizing the right side using Tlor epnsion: The brightness constnc constrint I(,,t 1) I(,,t) 0 t I v I u I Hence,
The brightness constnc constrint I u v How mn equtions nd unknowns per piel? One eqution, two unknowns I = 0 Intuitivel, wht does this constrint men? I The component of the flow perpendiculr to the grdient (i.e., prllel to the edge) is unknown I ( u, v) I = t t 0
The brightness constnc constrint I u v How mn equtions nd unknowns per piel? One eqution, two unknowns I = 0 Intuitivel, wht does this constrint men? I The component of the flow perpendiculr to the grdient (i.e., prllel to the edge) is unknown I ( u, v) I = t t 0 If (u, v) stisfies the eqution, so does (uu, vv ) if I ( u', v') = 0 grdient (u,v) (u,v ) (uu,vv ) edge
The perture problem Perceived motion
The perture problem Actul motion
The brber pole illusion http://en.wikipedi.org/wiki/brberpole_illusion
The brber pole illusion http://en.wikipedi.org/wiki/brberpole_illusion
The brber pole illusion http://en.wikipedi.org/wiki/brberpole_illusion
Solving the perture problem How to get more equtions for piel? Sptil coherence constrint: pretend the piel s neighbors hve the sme (u,v) If we use 55 window, tht gives us 25 equtions per piel B. Lucs nd T. Knde. An itertive imge registrtion technique with n ppliction to stereo vision. In Proceedings of the Interntionl Joint Conference on Artificil Intelligence, pp. 674 679, 1981.
Solving the perture problem Lest squres problem: When is this sstem solvble? Wht if the window contins just single stright edge? B. Lucs nd T. Knde. An itertive imge registrtion technique with n ppliction to stereo vision. In Proceedings of the Interntionl Joint Conference on Artificil Intelligence, pp. 674 679, 1981.
Conditions for solvbilit Bd cse: single stright edge
Conditions for solvbilit Good cse
Lucs-Knde flow Overconstrined liner sstem Lest squres solution for d given b The summtions re over ll piels in the K K window B. Lucs nd T. Knde. An itertive imge registrtion technique with n ppliction to stereo vision. In Proceedings of the Interntionl Joint Conference on Artificil Intelligence, pp. 674 679, 1981.
Conditions for solvbilit Optiml (u, v) stisfies Lucs-Knde eqution When is this solvble? A T A should be invertible A T A entries should not be too smll (noise) A T A should be well-conditioned
Eigenvectors of A T A Recll the Hrris corner detector: M = A T A is the second moment mtri The eigenvectors nd eigenvlues of M relte to edge direction nd mgnitude The eigenvector ssocited with the lrger eigenvlue points in the direction of fstest intensit chnge The other eigenvector is orthogonl to it
Interpreting the eigenvlues Clssifiction of imge points using eigenvlues of the second moment mtri: λ 2 Edge λ 2 >> λ 1 Corner λ 1 nd λ 2 re lrge, λ 1 ~ λ 2 λ 1 nd λ 2 re smll Flt region Edge λ 1 >> λ 2 λ 1
Edge grdients ver lrge or ver smll lrge λ 1, smll λ 2
Low-teture region grdients hve smll mgnitude smllλ 1, smll λ 2
High-teture region grdients re different, lrge mgnitudes lrge λ 1, lrge λ 2
Wht re good fetures to trck? Recll the Hrris corner detector Cn mesure qulit of fetures from just single imge
Motion models Trnsltion Affine Perspective 3D rottion 2 unknowns 6 unknowns 8 unknowns 3 unknowns
Substituting into the brightness constnc eqution: v u 6 5 4 3 2 1 ), ( ), ( = = 0 t I v I u I Affine motion
0 ) ( ) ( 6 5 4 3 2 1 t I I I Substituting into the brightness constnc eqution: v u 6 5 4 3 2 1 ), ( ), ( = = Ech piel provides 1 liner constrint in 6 unknowns [ ] 2 = t I I I Err ) ( ) ( ) ( 6 5 4 3 2 1 r Lest squres minimiztion: Affine motion
Errors in Lucs-Knde The motion is lrge (lrger thn piel) Itertive refinement, corse-to-fine estimtion A point does not move like its neighbors Motion segmenttion Brightness constnc does not hold Do ehustive neighborhood serch with normlized correltion
Itertive Refinement Estimte velocit t ech piel using one itertion of Lucs nd Knde estimtion Wrp one imge towrd the other using the estimted flow field Refine estimte b repeting the process
Deling with lrge motions
Reduce the resolution!
Corse-to-fine opticl flow estimtion u=1.25 piels u=2.5 piels u=5 piels imge H1 u=10 piels imge I imge 2 Gussin prmid of imge 1 Gussin prmid of imge 2
Corse-to-fine opticl flow estimtion run itertive L-K run itertive L-K wrp & upsmple... imge J1 imge I imge 2 Gussin prmid of imge 1 Gussin prmid of imge 2
Motion segmenttion How do we represent the motion in this scene? J. Wng nd E. Adelson. Lered Representtion for Motion Anlsis. CVPR 1993.
Lered motion Brek imge sequence into lers ech of which hs coherent motion J. Wng nd E. Adelson. Lered Representtion for Motion Anlsis. CVPR 1993.
Wht re lers? Ech ler is defined b n lph msk nd n ffine motion model J. Wng nd E. Adelson. Lered Representtion for Motion Anlsis. CVPR 1993.
v u 6 5 4 3 2 1 ), ( ), ( = = Locl flow estimtes Motion segmenttion with n ffine model J. Wng nd E. Adelson. Lered Representtion for Motion Anlsis. CVPR 1993.
Motion segmenttion with n ffine model v u 6 5 4 3 2 1 ), ( ), ( = = Eqution of plne (prmeters 1, 2, 3 cn be found b lest squres) J. Wng nd E. Adelson. Lered Representtion for Motion Anlsis. CVPR 1993.
Motion segmenttion with n ffine model u(, v(, ) ) = = 1 4 2 5 3 6 1D emple Eqution of plne (prmeters 1, 2, 3 cn be found b lest squres) u(,) True flow Locl flow estimte Foreground Bckground Segmented estimte Line fitting Occlusion J. Wng nd E. Adelson. Lered Representtion for Motion Anlsis. CVPR 1993.
How do we estimte the lers? Compute locl flow in corse-to-fine fshion Obtin set of initil ffine motion hpotheses Divide the imge into blocks nd estimte ffine motion prmeters in ech block b lest squres Eliminte hpotheses with high residul error Perform k-mens clustering on ffine motion prmeters Merge clusters tht re close nd retin the lrgest clusters to obtin smller set of hpotheses to describe ll the motions in the scene Iterte until convergence: Assign ech piel to best hpothesis Piels with high residul error remin unssigned Perform region filtering to enforce sptil constrints Re-estimte ffine motions in ech region J. Wng nd E. Adelson. Lered Representtion for Motion Anlsis. CVPR 1993.
Emple result J. Wng nd E. Adelson. Lered Representtion for Motion Anlsis. CVPR 1993.