Tracking. Announcements

Transcription

1 Tracking Tuesday, Nov 24 Krisen Grauman UT Ausin Announcemens Pse 5 ou onigh, due 12/4 Shorer assignmen Auo exension il 12/8 I will no hold office hours omorrow 5 6 pm due o Thanksgiving 1

2 Las ime: Moion Ouline Moion field and parallax Opical flow, brighness consancy Aperure problem Today: Tracking Tracking as inference Linear models of dynamics Kalman filers General challenges in racking Moion esimaion echniques Direc mehods Direcly recover image moion a each pixel from spaio-emporal image brighness variaions Dense moion fields bu sensiive o appearance variaions Dense moion fields, bu sensiive o appearance variaions Suiable for video and when image moion is small 2

3 Direc mehods: Esimaing opical flow I(x,y, 1 I(x,y, Given wo subsequen frames, esimae he apparen moion field beween hem. Key assumpions Brighness consancy: projecion of he same poin looks he same in every frame Small moion: poins do no move very far Spaial coherence: poins move like heir neighbors The aperure problem Perceived moion 3

4 The aperure problem Acual moion Solving he aperure problem (grayscale image How o ge more equaions for a pixel? Spaial coherence consrain: preend he pixel s neighbors have he same (u,v If we use a 5x55 window, ha gives us 25 equaions per pixel 4

5 Las ime: Moion Ouline Moion field and parallax Opical flow, brighness consancy Aperure problem Today: Tracking Tracking as inference Linear models of dynamics Kalman filers General challenges in racking Tracking: some applicaions Body pose racking, aciviy recogniion Censusing a ba populaion Video-based inerfaces Medical apps Surveillance 5

6 Opical flow for racking? If we have more han jus a pair of frames, we could compue flow from one o he nex: Bu flow only reliable for small moions, and we may have occlusions, exureless regions ha yield bad esimaes anyway Moion esimaion echniques Direc mehods Direcly recover image moion a each pixel from spaio-emporal image brighness variaions Dense moion fields, bu sensiive o appearance variaions Suiable for video and when image moion is small Feaure-based mehods Exrac visual feaures (corners, exured areas and rack hem over muliple frames Sparse moion fields, bu more robus racking Sparse moion fields, bu more robus racking Suiable when image moion is large (10s of pixels 6

7 Feaure-based maching for moion Ineresing poin Bes maching neighborhood Search window Time Time +1 Search window is cenered a he poin where we las saw he feaure, in image I1. Bes mach = posiion where we have he highes normalized cross-correlaion value. Feaure-based maching for moion For a discree maching search, wha are he radeoffs of he chosen search window size? Which paches o rack? p Selec ineres poins e.g. corners Where should he search window be placed? Near mach a previous frame More generally, aking ino accoun he expeced dynamics of he objec 7

8 Deecion vs. racking =1 =2 =20 =21 Deecion vs. racking Deecion: We deec he objec independenly in each frame and can record is posiion over ime, e.g., based on blob s cenroid or deecion window coordinaes 8

9 Deecion vs. racking Tracking wih dynamics: We use image measuremens o esimae posiion of objec, bu also incorporae posiion prediced by dynamics, i.e., our expecaion of objec s moion paern. Deecion vs. racking Tracking wih dynamics: We use image measuremens o esimae posiion of objec, bu also incorporae posiion prediced by dynamics, i.e., our expecaion of objec s moion paern. 9

10 Tracking wih dynamics Use model of expeced moion o predic where objecs will occur in nex frame, even before seeing he image. Inen: Do less work looking for he objec, resric he search. Ge improved esimaes since measuremen noise is empered by smoohness, dynamics priors. Assumpion: coninuous moion paerns: Camera is no moving insanly o new viewpoin Objecs do no disappear and reappear in differen places in he scene Gradual change in pose beween camera and scene Tracking as inference The hidden sae consiss of he rue parameers we care abou, denoed X. The measuremen is our noisy observaion ha resuls from he underlying sae, denoed Y. 10

11 Sae vs. observaion Hidden sae : parameers of ineres Measuremen : wha we ge o direcly observe Tracking as inference The hidden sae consiss of he rue parameers we care abou, denoed X. The measuremen is our noisy observaion ha resuls from he underlying sae, denoed Y. A each ime sep, sae changes (from X -1 o X and we ge a new observaion Y. Our goal: recover mos likely sae X given All observaions seen so far. Knowledge abou dynamics of sae ransiions. 11

12 Noaion reminder x ~ N( μ, Σ Random variable wih Gaussian probabiliy disribuion ha has he mean vecor μ and covariance marix Σ. x and μ are d-dimensional, Σ is d x d. d=2 d=1 If x is 1-d, we jus have one Σ parameer - he variance: σ 2 Tracking as inference: inuiion measuremen Belief: predicion Belief: predicion Correced predicion old belief Time Time +1 12

13 Sandard independence assumpions Only immediae pas sae influences curren sae Measuremens a ime i only depend on he curren sae Predicion: Tracking as inference Given he measuremens we have seen up o his poin, wha sae should we predic? P( X y0, K, y 1 Correcion: Now given he curren measuremen, wha sae should we predic? P ( X y, K 0, y 13

14 Quesions How o represen he known dynamics ha govern he changes in he saes? How o represen relaionship beween sae and measuremens, plus our uncerainy in he measuremens? How o compue each cycle of updaes? Represenaion: We ll consider he class of linear dynamic models, wih associaed Gaussian pdfs. Updaes: via he Kalman filer. Linear dynamic model Describe he a priori knowledge abou Sysem dynamics model: represens evoluion of sae over ime, wih noise. x ~ N( Dx 1; Σd n x 1 n x n n x 1 Measuremen model: a every ime sep we ge a noisy measuremen of he sae. y ~ N( Mx; Σm m x 1 m x n n x 1 14

15 Example: randomly drifing poins x ~ N( Dx 1; Σd Consider a saionary objec, wih sae as posiion Posiion is consan, only moion due o random noise erm. Sae evoluion is described by ideniy marix D=I Example: Consan velociy (1D poins 1 d posiion 1 d posiion measuremens saes ime 15

16 Example: Consan velociy (1D poins x ~ N( Dx 1; Σd y ~ N( Mx; Σm Sae vecor: posiion p and velociy v x x p = v = D x p = p + ( Δ v = v ξ Measuremen is posiion only v 1 + noise = 0 1 Δ p 1 v + ε y p = Mx + noise = 0 v [ 1 ] noise + noise (greek leers denoe noise erms Example: Consan acceleraion (1D poins 16

17 Example: Consan acceleraion (1D poins p = p 1 + ( Δ v 1 + ε v = v + ( Δ a + ξ x ~ N( Dx 1; Σd y ~ N( Mx; Σm Sae vecor: posiion p, velociy v, and acceleraion a. p x = v a a = a ζ 1 1 Δ 0 p 1 x = D x noise v 1 + = 0 1 Δ 1 + noise a 1 1 Measuremen is posiion only p y = Mx + noise = 0 a [ 1 0 ] v noise + (greek leers denoe noise erms Quesions How o represen he known dynamics ha govern he changes in he saes? How o represen relaionship beween sae and measuremens, plus our uncerainy in he measuremens? How o compue each cycle of updaes? Represenaion: We ll consider he class of linear dynamic models, wih associaed Gaussian pdfs. Updaes: via he Kalman filer. 17

18 The Kalman filer Mehod for racking linear dynamical models in Gaussian noise The prediced/correced sae disribuions are Gaussian Only need o mainain he mean and covariance The calculaions are easy (all he inegrals can be done in closed form Kalman filer Know correced sae from previous ime sep, and all measuremens up o he curren one Predic disribuion over nex sae. Time updae ( Predic P ( X y K y 0,, y 1 Receive measuremen Know predicion of sae, and nex measuremen Updae disribuion over curren sae. Measuremen updae ( Correc ( X y, P, K 0 y Mean and sd. dev. of prediced sae: μ, σ Time advances: ++ Mean and sd. dev. of correced sae: + + μ, σ 18

19 Noaion reminder x ~ N( μ, Σ Random variable wih Gaussian probabiliy disribuion ha has he mean vecor μ and covariance marix Σ. x and μ are d-dimensional, Σ is d x d. d=2 d=1 If x is 1-d, we jus have one Σ parameer - he variance: σ 2 1D Kalman filer: Predicion Have linear dynamic model defining prediced sae evoluion, wih noise 2 X N( σ ~ dx d 1, d Wan o esimae prediced disribuion for nex sae P Updae he mean: 2 ( X y, K, y N( μ,( σ Updae he variance: 0 1 = μ dμ ( σ + = = σ d + ( dσ

20 1D Kalman filer: Correcion Have linear model defining he mapping of sae o measuremens: Y 2 (, σ ~ N mx m Wan o esimae correced disribuion given laes meas.: P X y, K, y N μ,( σ Updae he mean: ( ( μ Updae he variance: 0 = 2 + μ σ m + my ( σ = σ m + m ( σ σ ( σ m ( σ = 2 2 σ m + m ( σ 2 2 Predicion vs. correcion μ σ + my ( σ σ ( σ m + 2 m μ = ( σ = σ m + m ( σ σ m + m ( σ Wha if here is no predicion uncerainy + μ μ ( 2 = 0 + = σ The measuremen is ignored! ( σ = 0? Wha if here is no measuremen uncerainy ( = 0? y + + μ = ( σ 2 = 0 m The predicion is ignored! σ m 20

21 Consan velociy model posiion Kalman filer processing o sae x measuremen * prediced mean esimae + correced mean esimae bars: variance esimaes before and afer measuremens ime Time Time +1 Consan velociy model posiion Kalman filer processing o sae x measuremen * prediced mean esimae + correced mean esimae bars: variance esimaes before and afer measuremens ime Time Time +1 21

22 Consan velociy model posiion Kalman filer processing o sae x measuremen * prediced mean esimae + correced mean esimae bars: variance esimaes before and afer measuremens ime Time Time +1 Consan velociy model posiion Kalman filer processing o sae x measuremen * prediced mean esimae + correced mean esimae bars: variance esimaes before and afer measuremens ime Time Time +1 22

23 h/bas/ hp:// 23

24 Las ime: Moion Ouline Moion field and parallax Opical flow, brighness consancy Aperure problem Today: Tracking Tracking as inference Linear models of dynamics Kalman filers General challenges in racking Tracking: issues Iniializaion Ofen done manually Background subracion, deecion can also be used Daa associaion, muliple racked objecs Occlusions, cluer 24

25 Tracking: issues Iniializaion Ofen done manually Background subracion, deecion can also be used Daa associaion, muliple racked objecs Occlusions, cluer Which measuremens go wih which racks? Daa associaion Simple sraegy: only pay aenion o he measuremen ha is closes o he predicion Slide credi: Lana Lazebnik 25

26 Daa associaion Simple sraegy: only pay aenion o he measuremen ha is closes o he predicion Tracking: issues Iniializaion Ofen done manually Background subracion, deecion can also be used Daa associaion, muliple racked objecs Occlusions, cluer Deformable and ariculaed objecs 26

27 Tracking via deformable conours 1. Use final conour/model exraced a frame as an iniial soluion for frame Evolve iniial conour o fi exac objec boundary a frame Repea, iniializing wih mos recen frame. Tracking Hear Venricles (muliple frames Tracking via deformable conours Visual Dynamics Group, Dep. Engineering Science, Universiy of Oxford. Applicaions: Traffic monioring Human-compuer ineracion Animaion Surveillance Compuer assised diagnosis in medical imaging 27

28 Iniializaion Ofen done manually Tracking: issues Background subracion, deecion can also be used Daa associaion, muliple racked objecs Occlusions, cluer Deformable and ariculaed objecs Consrucing accurae models of dynamics E.g., Fiing parameers for a linear dynamics model Drif Accumulaion of errors over ime Drif D. Ramanan, D. Forsyh, and A. Zisserman. Tracking People by Learning heir Appearance. PAMI

29 Summary Tracking as inference Goal: esimae poserior of objec posiion given measuremen Linear models of dynamics Represen sae evoluion and measuremen models Kalman filers Recursive predicion/correcion updaes o refine measuremen General racking challenges 29