Announcements. Computer Vision I. Foreground/Background Segmentation. Visual Tracking. CSE 252A Lecture 17

Size: px

Start display at page:

Download "Announcements. Computer Vision I. Foreground/Background Segmentation. Visual Tracking. CSE 252A Lecture 17"

Harvey Gibson
5 years ago
Views:

Visual Trackig CSE 5A Lecture 17 Aoucemets Read Chapter 11 of Forsyth & Poce Homework 3 is due today by 11:59 PM Homework 4 is due Dec 18, 11:59 PM Please complete evaluatios Course TA

1 Visual Trackig CSE 5A Lecture 17 Aoucemets Read Chapter 11 of Forsyth & Poce Homework 3 is due today by 11:59 PM Homework 4 is due Dec 18, 11:59 PM Please complete evaluatios Course TA Foregroud/Backgroud Segmetatio Iput Image Backgroud Image Backgroud Image Iitially, B0( mediai ( 0 N Thereafter, updated i a sequetial estimatio of the mea maer (1 ) B 1( I ( B ( B1( if I ( backgroud otherwise 1

Differece Image Segmetatio Image Statistic-based threshold operatio: assumes idepedet Gaussia distributios Sequetial estimatio of the mea of squares (1 ) J 1( I ( J ( J 1( if I( backgroud otherwise

Illumiatio variatio 4. Camera jitter 5. Expressio variatio etc. [ Ho, Lee, Kriegma ] Mai trackig otios State: usually a fiite umber of parameters (a vector) that characterizes the state (e.g., locatio, size, pose, deformatio of thig beig tracked.

2 Differece Image Segmetatio Image Statistic-based threshold operatio: assumes idepedet Gaussia distributios Sequetial estimatio of the mea of squares (1 ) J 1( I ( J ( J 1( if I( backgroud otherwise Variace estimatio (approximatio) D ( I ( B ( 1 Threshold ( J ( B ( 1 if D( T 1( F ( 0 otherwise Segmetatio Image Visual Trackig Mai Challeges 1. 3-D Pose Variatio. Occlusio of the target 3. Illumiatio variatio 4. Camera jitter 5. Expressio variatio etc. [ Ho, Lee, Kriegma ] Mai trackig otios State: usually a fiite umber of parameters (a vector) that characterizes the state (e.g., locatio, size, pose, deformatio of thig beig tracked. Dyamics: How does the state chage over time? How is that chaged costraied? Represetatio: How do you represet the thig beig tracked Predictio: Give the state at time t-1, what is a estimate of the state at time t? Correctio: Give the predicted state at time t, ad a measuremet at time t, update the state. Iitializatio what is the state at time t=0? What is state? -D image locatio, Φ=(u,v) Image locatio + scale Φ=(u,v, Image locatio + scale + orietatio Φ=(u,v,s,θ) Affie trasformatio 3-D pose 3-D pose plus iteral shape parameters (some may be discrete). e.g., for a face, 3-D pose +facial expressio usig FACS + eye state (ope/closed). Collectios of cotrol poits specifyig a splie Above, but for multiple objects (e.g. trackig a formatio of airplae. Augmet above with temporal derivatives (, )

State Examples: object is ball, state is 3D positio+velocity, measuremets are stereo pairs object is perso, state is body cofiguratio, measuremets are frames What is state here?

Clea up biary image usig morphological operators Perform coected compoet exploratio to fid blobs. coected regios.

3 State Examples: object is ball, state is 3D positio+velocity, measuremets are stereo pairs object is perso, state is body cofiguratio, measuremets are frames What is state here? Example: Blob Tracker From iput image I(u,v) (color?) at time t, create a biary image by applyig a fuctio f(i(u,v)). Clea up biary image usig morphological operators Perform coected compoet exploratio to fid blobs. coected regios. Compute their momets (mea ad covariace of coordiates of regio), ad use as state Usig state estimate from t-1 ad perform data associatio to idetify state i from t. Blob Trackig i IR Images Trackig: Probabilistic framework Very geeral model: We assume there are movig objects, which have a uderlyig state X There are measuremets Y, some of which are fuctios of this state There is a clock at each tick, the state chages: X t-1, X t, X t+1 at each tick, we get a ew observatio: Y t-1, Y t, Y t+1 Threshold about body temperature Coected compoet aalysis Positio, scale, orietatio of regios Temporal coherece Trackig State Three mai steps X 0 X 1 X t-1 X t X t+1 Y 0 Y 1 Y t-1 Y t Y t+1 Istead of kowig state at each istat, we treat the state as radom variables X t characterized by a pdf P(X t ) or perhaps coditioed o other Radom Variables e.g., P(X t X t-1 ), etc. The observatio (measuremet) Y t is a radom variable coditioed o the state P(Y t X t ) Geerally, we do t observe the state it s hidde. We ca try to express these coditioal distributios parametrically, sample the distributio, or estimate the mode. 3

4 Simplifyig Assumptios P(Y i Y 1,, Y i-1, X i ) = P(Y i X i ) X 0 X 1 X t-1 X t X t+1 Trackig as iductio Assume data associatio is doe Sometimes challegig i cluttered scees. See work by Christopher Rasmusse o Joit Probabilistic Data Associatio Filters (JPDAF). Do correctio for the 0 th frame Assume we have corrected estimate for i th frame show we ca do predictio for i+1 frame, correctio for i+1 frame Y 0 Y 1 Y t-1 Y t Y t+1 Base case P(y x) is our observatio model The probability of y give x. For example, P(y x) might be a Gaussia with mea x. Prior distributio of iitial state Iductio step: State Predictio Give Ad, we make a measuremet y 0 Iductio step: State Correctio I predictio, we estimated the state X i give the measuremets up to i-1. Now we get the measure at time i called y i. How is this formulatio used 1. It s igored. At each time istat, the state is estimated (perhaps a maximum likelihood estimate or somethig oprobabilistic). The coditioal distributios are represeted by some coveiet parametric form (e.g., Gaussia). 3. The PDF s are represeted oparametrically, ad samplig techiques are used. 4

5 Liear dyamic models Use otatio ~ to mea has the pdf of, N(a, b) is a ormal distributio with mea a ad covariace b. A liear dyamic model has the form x i N D i1 x i1 ; di y i N M i x i ; mi Examples Poits movig with costat velocity Periodic motio Etc. Poits movig with costat acceleratio Poits movig with costat velocity We have Poits movig with costat acceleratio We have Positio Velocity (the Greek letters deote oise term Stack (u, v) ito a sigle state vector (the Greek letters deote oise term Stack (u, v) ito a sigle state vector which is the form we had above D i which is the form we had above The Kalma Filter The Kalma Filter i 1D Key ideas: Liear models iteract uiquely well with Gaussia oise - make the prior Gaussia, everythig else Gaussia ad the calculatios are easy Gaussias are really easy to represet Just a mea vector mea ad covariace matrix. Dyamic Model Notatio Corrected mea Predicted mea 5

6 Predictio for 1-D Kalma filter The ew state is obtaied by multiplyig old state by kow costat addig zero-mea oise Therefore, predicted mea for ew state is costat times mea for old state Predicted variace is sum of costat^ times old state variace ad oise variace Because: Old state is ormal radom variable, Multiplyig ormal rv by costat implies mea is multiplied by a costat variace is multiplied by square of costat Addig zero mea oise adds zero to the mea, Addig rv s adds variace Correctio for 1D Kalma filter Multi-variate Kalma Filter Patter match to idetities give i book basically, guess the itegrals, get: Notice: if measuremet oise is small, we rely maily o the measuremet, if it s large, maily o the predictio Color Histogram [Birchfield 1998; Bradski 1998] Volume [Wre et al., 1995; Bregler, 1997; Darrell, 1998] Shape Deformable curve [Kass et al.1988] Template[Blake etal. 1993; Birchfield 1998] Example-based [Cootes et al., 1993; Baumberg & Hogg, 1994] Appearace Correlatio [Lucas & Kaade, 1981; Shi & Tomasi, 1994] Photometric variatio [Hager & Belhumeur, 1998] Outliers [Black et al., 1998; Hager & Belhumeur, 1998] Norigidity [Black et al., 1998; Sclaroff & Isidoro, 1998] Trackig Modalities (Defie the features Y i ) Motio Backgroud model [Wre et al., 1995; Rosales & Sclaroff, 1999; Stauffer & Grimso, 1999] Optical flow [Cutler & Turk] Egomotio [Sawhey & Ayer, 1996; Irai & Aada, 1998] Stereo Blob correlatio [Azarbayejai & Petlad, 1996] Disparity map [Kaade et al., 1996; Koolige, 1997; Darrell et al., 1998] Color Blob trackig Color-based tracker gets lost o white kight: Same Color 6

Sakes: Active Cotours Cotour C: cotiuous curve o smooth surface i Sake S: projectio of C to image Curve types Edge betwee regios o surface with cotrastig properties Lie that cotrasts with surface

closed cotour U( r( q t B( or r ( U( Q x x y y Q ( q 0.. q, q 0.

7 Sakes: Active Cotours Cotour C: cotiuous curve o smooth surface i Sake S: projectio of C to image Curve types Edge betwee regios o surface with cotrastig properties Lie that cotrasts with surface properties o both side Silhouette of surface agaist cotrastig backgroud Geeral Algorithm: Perform edge detectio Fit parametric or o-parametric curve to data 3 Sakes: Basic Approach Parameterize a closed cotour U( r( q t B( or r ( U( Q x x y y Q ( q 0.. q, q 0.. q ) Give a predicted state q, search radially for edges Solve a least squares problem for ew state B( t 0 0 B( t Tracker Compositio: Oly Shape (Sake Tracker Compositio Tracker 1 State 1 Video stream Tracker State Estimator State Combied state Geometry-based tracker gets lost o black paw: Same shape Tracker Compositio: Color ad Shape Visual Trackig usig regios I 0 I t p t Variability model: I t = g(i 0, p t ) Icremetal Estimatio: From I 0, I t+1 ad p t compute p t+1 Combiig Trackers => Robustess I 0 -g(i t+1, p t+1 ) ==> mi Trackers i video, IR ad rage 7

Trackig usig Textured Regios Mea itesity differece betwee I ad affie warp of template image [Shi & Tomasi, 1994] I R I

Warpig J(u) = I(A u) Note that we ca uroll the loop to avoid the matrix multiply tregio Template I Tracked state I R c

well Template trackig: Plaar Case Hager/Toyama: Trackig Cycle Plaar Object => Affie motio model: u i = A u i + d

error from omial Image Warpig p - I t = g(p t, I 0 ) State itegratio Apply correctio to state p Model Iverse SSD

8 Trackig usig Textured Regios Mea itesity differece betwee I ad affie warp of template image [Shi & Tomasi, 1994] I R I C Warpig is a chage of coordiates: J(u,v) = I(f(u,v,p),g(u,v,p)) Always prefer to warp to destiatio to avoid gaps Image Warpig J(u) = I(A u) Note that we ca uroll the loop to avoid the matrix multiply tregio Template I Tracked state I R c ( ( I ( I ( ) ( ' W R C I R IC Two iterpolatio schemes earest eighbor biliear For much of trackig, earest eighbor works well Template trackig: Plaar Case Hager/Toyama: Trackig Cycle Plaar Object => Affie motio model: u i = A u i + d Predictio Prior states predict ew appearace Referece Warpig Image warpig Geerate a ormalized view Model iverse Compute error from omial Image Warpig p - I t = g(p t, I 0 ) State itegratio Apply correctio to state p Model Iverse SSD Trackig XVisio: A trackig System Face Eyes Mouth Eye Eye BestSSD BestSSD BestSSD Image Processig Compositio of Primitive Trackers 8

HUMAN COMPUTER INTERACTION

HUMAN COMPUTER INTERACTION. VISION-BASED INTERACTION A BASIC FEATURE EXTRACTION AND TRACKING I-Che Li, Natioal Chiao Tug Uiversity, Taiwa Goals Lear the basic feature etractio from visio tech. Efficiet