HUMAN COMPUTER INTERACTION

Transcription

1 HUMAN COMPUTER INTERACTION. VISION-BASED INTERACTION A BASIC FEATURE EXTRACTION AND TRACKING I-Che Li, Natioal Chiao Tug Uiversity, Taiwa

2 Goals Lear the basic feature etractio from visio tech. Efficiet ad simple? approaches for real-time user iterfaces.

3 Outlie Basic feature etractio ad trackig Color matchig Groupig or clusterig Silhouette & foregroud Filterig ad predictio Robust features ad trackig Key poits ad scale-ivariat features 3D positio estimatio

4 Visio-based Iterface Poitig gesture recogitio Eye trackig Had Gesture Recogitio, Caltech Iteractive Wall, CSAIL, MIT

5 Visio-based Iterface cot. A lot of delicate ad advaced visio tech. for various targets. Our goals Not focusig o developig brad-ew algorithms. Utilizig efficiet ad simple algorithms for friedly iterfaces ad attractive applicatios. How about a simpler eviromet or coditio? Make the thig easier!!!

6 Visio-based Iterface cot. E.g. Laser poiter E.g. color markers Blue scree, Star War III cospicuous color features

7 Visio-based Iterface cot. Silhouette images, from J. Carraza et al., Free-Viewpoit Video of Huma Actors

8 Color Features RGB image array Digitizatio Effects Noise, etc. How to assig your target colors? predefied i your system? Iitializatio stages? How to estimate the target positio? A costraied coditio will make your computatio easier ad more reliable.

9 Color Features cot. How about two or more targets? Number of objects with or without Coected compoet labelig Noise effects

10 Coected Compoet Labelig 4-eighbor or 8 eighbor. Give a ew labelig umber to those without labeled eighbors. Merge the label umbers equivalet classes. You ca easily fid classic algorithms.

11 Clusterig K-meas algorithms are popularly used. E K j i j i m j Make iitial guesses for the meas m, m,..., m k Util there are o chages i ay mea Use the estimated meas to classify the samples ito clusters For i from to k Replace m i with the mea of all of the samples for cluster i ed_for ed_util

12 Lloyd s Algorithm

13 Color Features Eyetoy games, PS/PS3

14 Feature Etractios How about these applicatios? Eyetoy games, PS

15 Foregroud Etractio Uder a eviromet with fied lightig ad static scees, the detectio of ew objects ca be more reliable.

16 Foregroud Etractio cot. Noise should be take ito accout Noise suppressio T. Horprasert et al., Visio lab, UMD

17 Motio Trackig How to distiguish multiple objects or targets i a scee? The problem of occlusio? T = T = T = 3 T = 4

18 Feature Matchig Fid correspodig poits i image video sequece oe simple techique: fid two patches with miimal summed squared error or dot product. w w k w y w y l y l k I v l u k I v u E,,,,

19 Feature Matchig cot. Block matchig ca be ureliable due to: Noise Occlusio Projectio of differet view directios Reflectio of specular lights, etc. Deformatio of objects Other approaches: optic flow, etc.

20 Filterig t Simply filterig by Gaussia filters.

21 Kalma Filterig A liear system: fa+b = fa + fb. Noisy data i hopefully less oisy out. Does t have to store all previous measuremets or re-evaluate all data each time step But delay is the price for filterig... Ref: G. Welch, G. Bishop, SIGGRAPH course otes A Itroductio to Kalma Filter S.M. Bozic, Digital ad Kalma Filterig, Edward Arold

22 Kalma Filterig cot. What is it used for? Trackig missiles Trackig heads/hads/drumsticks/ Etractig lip motio from video Fittig Bezier patches to poit data Lots of computer visio applicatios Ecoomics Navigatio

23 Measuremet & State Measuremet z, z State ˆ z z Measuremet z, z State? ˆ??

24 Combie the Measuremets The d estimatio state from measuremet + : ˆ ˆ ˆ z K z K z Product of two Gaussia PDF a b a b b a a b b a e p Gaussia distributio

25 Suppose There re Motios Not all the differece is error Some may be motio Kalma filter ca iclude a motio model Estimate velocity ad positio

26 Process Model Describes how state chages over time The state for the first eample was scalar The process model was othig chages A better model might be State is a -vector [ positio, velocity ] positio + = positio + velocity * time velocity + = velocity

27 From Predictio to Correctio KF operates by Predictig the ew state ad its ucertaity Correctig with the ew measuremet Predict Correct

28 Kalma Filter P C K P P p p ˆ ˆ ˆ k CA z K A where Q A AP P R C CP C P K T p T p T p v C z w A Estimator: Whe the state equatios are Filter gai: Error covariace matri Optimal estimate = Predictio + Kalma Gai * Measuremet Predictio Variace of estimate = Variace of predictio * Kalma Gai Q = Eww t Process oise covariace R = Evv t Measuremet oise covariace

29 Eample: D Pos. Oly Apparatus: D Tablet w w A - w- v v C C z z z C v X: iteral state e.g. at scree coordiate A: state trasitio matri Z: measuremet e.g. at tablet coordiate C: measuremet matri W: oise V: oise

30 Preparatio A } { Q Q ww E Q T } { R R vv E R T State trasitio Process Noise Covariace Measuremet Noise Covariace

31 Predictio & Correctio ˆ ˆ Q A AP P A T p p ˆ ˆ ˆ P C K P P C z K R C CP C P K p p p p T p T p

32 XY Track

33 Y Track: Movig the Still

34 Motio-depedet Performace

35 Eample: D PV Model Positio-velocity model u u dt dt v v C C z z u: chage i velocity v: measuremet error

36 Preparatio & Iitializatio v E v R v v T w E w Q u u T dt z z dt z z z z P the co-variace matri of iitial guess

37 Performace of PV Model

38 May Applicatios Egieerig Robotics, spacecraft, aircraft, automobiles Computer Trackig, real-time graphics, computer visio Ecoomics Forecastig ecoomic idicators

39 Movig Object Etractio Thresholdig o Itesity differece Optical flow

40 Optical flow The optical flow slides are modified from the Motio slides, Steve Seitz, U. Washigto.

41 Problem defiitio: optical flow How to estimate piel motio from image H to image I? Solve piel correspodece problem give a piel i H, look for earby piels of the same color i I Key assumptios color costacy: a poit i H looks the same i I For grayscale images, this is brightess costacy small motio: poits do ot move very far This is called the optical flow problem

42 Optical flow costraits grayscale images Let s look at these costraits more closely brightess costacy: Q: what s the equatio? small motio: u ad v are less tha piel suppose we take the Taylor series epasio of I:

43 Optical flow equatio Combiig these two equatios I the limit as u ad v go to zero, this becomes eact

44 Cosiderig more tha oe piel How to get more equatios for a piel? Basic idea: impose additioal costraits most commo is to assume that the flow field is smooth locally oe method: preted the piel s eighbors have the same u,v If we use a 55 widow, that gives us 5 equatios per piel!

45 RGB versio How to get more equatios for a piel? Basic idea: impose additioal costraits most commo is to assume that the flow field is smooth locally oe method: preted the piel s eighbors have the same u,v If we use a 55 widow, that gives us 5*3 equatios per piel! Note that RGB is ot eough to disambiguate because R, G & B are correlated Just provides better gradiet

46 Lucas-Kaade flow Prob: we have more equatios tha ukows Solutio: solve least squares problem miimum least squares solutio give by solutio i d of: The summatios are over all piels i the K K widow This techique was first proposed by Lucas & Kaade 98

47 Coditios for solvability Optimal u, v satisfies Lucas-Kaade equatio Whe is This Solvable? A T A should be ivertible A T A should ot be too small due to oise eigevalues λ ad λ of A T A should ot be too small A T A should be well-coditioed λ / λ should ot be too large λ = larger eigevalue Does this look familiar? A T A is the Harris matri

48 Errors i Lucas-Kaade What are the potetial causes of errors i this procedure? Suppose A T A is easily ivertible Suppose there is ot much oise i the image Whe our assumptios are violated Brightess costacy is ot satisfied The motio is ot small A poit does ot move like its eighbors widow size is too large what is the ideal widow size?

49 Iterative Refiemet Iterative Lucas-Kaade Algorithm. Estimate velocity at each piel by solvig Lucas-Kaade equatios. Warp H towards I usig the estimated flow field - use image warpig techiques 3. Repeat util covergece

50 Optical Flow: D Case Brightess Costacy Assumptio:,, dt t dt t I t t I t f t t t I t I I v I t t I I v { t f Because o chage i brightess with time Slides from Optical Flow, Feature Trackig, Normal Flow, Gary Bradski, Sebastia Thru

51 Trackig i the D case: I, t I, t p v? Slides from Optical Flow, Feature Trackig, Normal Flow, Gary Bradski, Sebastia Thru

52 Trackig i the D case: I, t I, t Temporal derivative p I t v I I I t I t I t Spatial derivative v p I I t Assumptios: Brightess costacy Small motio Slides from Optical Flow, Feature Trackig, Normal Flow, Gary Bradski, Sebastia Thru

53 Trackig i the D case: Iteratig helps refiig the velocity vector I, t I, t Temporal derivative at d iteratio p I t I v v previous I I t Ca keep the same estimate for spatial derivative Coverges i about 5 iteratios Slides from Optical Flow, Feature Trackig, Normal Flow, Gary Bradski, Sebastia Thru