Moving Object Tracking

Transcription

1 Moving Objec Tracing Princeon Universiy COS 49 Lecure Dec Harpree S. Sawhney hsawhney@sarnoff.com

2 Recapiulaion : Las Lecure Moving objec deecion as robus regression wih oulier deecion Simulaneous muliple surface/moving objec esimaion Epecaion-Maimizaion EM) algorihm as a formal mechanism for muliple model esimaion & is applicaion o muliple moion deecion and esimaion

3 The Tracing Problem Mainain ideniy across ime of designaed or auomaically deeced objecs

4 The Tracing Problem : Issues Objec Deecion : Designaed or auomaically deeced Objec Sae Insaniaion: Objec represenaion Posiion Velociy Shape Color Appearance Templae Sae Predicion: Having seen { y0y Kyi-} wha sae do hese measuremens predic for he ne ime insan i? Need a represenaion for X Y = y KY y ) P i 0 0 i- = i- Daa Associaion: Which of he measuremens a he i-h insan correspond o he prediced sae a ha insan? Use o esablish he correspondence PXi Y0 = y0 KYi- = yi-) Sae Updae: Wih he corresponding measuremen y i esablished for insan i compue an esimae of he opimal new sae hrough PXi Y0 = y0 KYi = y i)

5 Objec Deecion Designaed Objec User specifies a emplae The sysem convers ha emplae ino an appropriae represenaion o be raced

6 Objec Deecion Fied Cameras Model he bacground using a reference image or a reference disribuion Deec objecs as changes wih respec o he reference

7 Objec Deecion Moving Cameras Align consecuive frames using he now well-nown echniques sudied in his class Use frame differencing beween aligned frames o deec changes designaed as new objecs

8 Simple Tracer : Blob racer Change-based racer: Approach Align video images Deec regions of change Trac change blobs Problem wih his approach is ha i uses no appearance informaion difficul o deal wih salled or close-by objecs

9 Moving Blobs

10 Simple Tracer - Correlaion Based Correlaion-based racer: Approach Iniialize he emplaes and he suppors of foreground objecs Esimae moion by correlaion The problem wih his approach is ha i does no simulaneously compue he segmenaion and appearance No accurae segmenaion or region of suppor may drif over ime. Ge confused by cluered bacgrounds

11 Problems wih he simple racers They lac he wo ey ingrediens for opimal racing: Sae predicion Opimal sae updaion Since measuremens are never perfec --- each has some uncerainy associaed wih i --- opimal sae predicion and updaion need o ae he uncerainies ino accoun Furhermore he objec represenaion needs o be richer No jus a change blob or fied emplae Opimal mehod for updaing he sae

12 Kalman Filering Assume ha resuls of eperimen i.e. opical flow) are noisy measuremens of sysem sae Model of how sysem evolves Predicion / correcion framewor Opimal combinaion of sysem model and observaions Rudolf Emil Kalman Acnowledgmen: much of he following maerial is based on he SIGGRAPH 00 course by Greg Welch and Gary Bishop UNC)

13 Simple Eample A poin whose posiion remains consan : Say a emperaure reading Noisy measuremen of ha single poin z Variance σ uncerainy σ ) Bes esimae of rue posiion ˆ = z Uncerainy in bes esimae ˆ σ = σ σ = σ

14 Simple Eample Second measuremen z variance σ Bes esimae of rue posiion Uncerainy in bes esimae ) ˆ ˆ ˆ z z z + = + + = +σ σ σ σ σ σ σ ) ˆ ˆ ˆ z z z + = + + = +σ σ σ σ σ σ σ ˆ ˆ σ σ σ + = ˆ ˆ σ σ σ + =

15 Online Weighed Average Combine successive measuremens ino consanlyimproving esimae Uncerainy decreases over ime Only need o eep curren measuremen las esimae of sae and uncerainy We have essenially compued he Leas Squares OR Minimum Variance OR Maimum Lielihood esimae of X given a number of noisy measuremens Z hrough an incremenal mehod

16 Terminology In his eample posiion is sae in general any vecor Sae evolves over ime according o a dynamic model or process model in his eample nohing changes ) Measuremens are relaed o he sae according o a measuremen model possibly incomplee possibly noisy) Bes esimae of sae ˆˆ wih covariance P

17 Very general model: Tracing Framewor We assume here are moving objecs which have an underlying sae X There are measuremens Z some of which are funcions of his sae There is a cloc a each ic he sae changes a each ic we ge a new observaion Eamples objec is ball sae is 3D posiion+velociy measuremens are sereo pairs objec is person sae is body configuraion measuremens are frames cloc is in camera 30 fps)

18 Bayesian Graphical Model Sae Variables: Those ha ell us abou objecs & heir saes Bu hey are hidden canno be direcly observed Dynamic Model K X - X X + Measuemen Model K Z - Z Z + Measuremens: Can be direcly observed Are noisy uncerain

19 Bayesian Formulaion p z ) = κpz ) p - )p - z - ) d - p z ) Poserior probabiliy afer laes measuremen pz ) Lielihood of he curren measuremen p -) p- z-) κ Temporal prior from he dynamic model Poserior probabiliy afer previous measuremen Normalizing consan

20 Key ideas: The Kalman Filer Linear models inerac uniquely well wih Gaussian noise mae he prior Gaussian everyhing else Gaussian and he calculaions are easy Gaussians are really easy o represen once you now he mean and covariance you re done

21 Linear Models For sandard Kalman filering everyhing mus be linear Sysem / Dynamical model: ξ = Φ + ξ The mari Φ is sae ransiion mari The vecor ξ represens addiive noise assumed o have covariance Q : N0;Q) ~ N Φ - ;Q )

22 Linear Models Measuremen model / Lielihood model: z z ~ NH ;R Mari H is measuremen mari = H + μ ) The vecor μ is measuremen noise assumed o have covariance R : N0; μ)

23 Posiion-Velociy Model Poins moving wih consan velociy We wish o esimae heir PV sae a every ime insan Φ H = = = d 0 d Δ [ 0 ] Posiion-Velociy Sae Consan Velociy Dynamic Model Mari Only posiion is direcly observable

24 Predicion/Correcion Predic new sae Correc o ae new measuremens ino accoun T ˆ + Φ Φ = Φ = Q P P T ˆ + Φ Φ = Φ = Q P P ) ) P H K I P H z K = + = ˆ ) ) P H K I P H z K = + = ˆ

25 Kalman Gain Weighing of process model vs. measuremens T T ) K P H H P H + R = Compare o wha we saw earlier: σ σ + + σ

26 Opimal Linear Esimae Opimal Linear Filer ˆ + ) = K ˆ -) + K Prediced sae Measuremen Under Gaussian assumpions linear esimae is he opimal Esimaion Error: + ) = + + ) ~ ˆ + ) ~ = [K + K H ~ - I] z + K ~ K = I - -) + K For an unbiased esimae: E[ + )] = 0 K H v ˆ + ) = ˆ -) + K [z - H ˆ -)] K Is obained by minimizing he variance of he sae esimae

27 [Welch & Bishop] Resuls: Posiion-Only Model Moving Sill

28 [Welch & Bishop] Resuls: Posiion-Velociy Model Moving Sill

29 Eension: Muliple Models Simulaneously run many KFs wih differen sysem models Esimae probabiliy each KF is correc Final esimae: weighed average

30 Resuls: Muliple Models [Welch & Bishop]

31 Resuls: Muliple Models [Welch & Bishop]

32 Resuls: Muliple Models [Welch & Bishop]

33 Eension: SCAAT H be differen a differen imes Differen sensors ypes of measuremens Someimes measure only par of sae Single Consrain A A Time SCAAT) Incorporae resuls from one sensor a once Alernaive: wai unil you have measuremens from enough sensors o now complee sae MCAAT) MCAAT equaions ofen more comple bu someimes necessary for iniializaion

34 UNC HiBall 6 cameras looing a LEDs on ceiling LEDs flash over ime [Welch & Bishop]

35 Eension: Nonlineariy EKF) HiBall sae model has nonlinear degrees of freedom roaions) Eended Kalman Filer allows nonlineariies by: Using general funcions insead of marices Linearizing funcions o projec forward Lie s order Taylor series epansion Only have o evaluae Jacobians parial derivaives) no inver process/measuremen funcions

36 Oher Eensions On-line noise esimaion Using nown sysem inpu e.g. acuaors) Using informaion from boh pas and fuure Non-Gaussian noise and paricle filering

37 Daa Associaion Neares Neighbors choose he measuremen wih highes probabiliy given prediced sae popular bu can lead o caasrophe Probabilisic Daa Associaion combine measuremens weighing by probabiliy given prediced sae gae using prediced sae

38 Video based Tracing : Compleiies In addiion o posiion and velociy objec sae may include: Appearance shape specific objec models : people vehicles ec. Camera may move in addiion o he objec Trac bacground as well as he foreground Measuremen model and he associaed lielihood compuaion is more comple: Compue he lielihood of he presence of a head-n-shoulders person model a a given locaion in he image Muliple objecs need o be raced simulaneously Measuremens need o be opimally associaed wih a se of models raher han a single model as in he previous eamples

39 Applicaion - Tracing vehicles in aerial videos The goals of a racing sysem are o deec new moving objecs mainain ideniy of objecs handle muliple objecs and ineracions beween hem. e.g. passing sopped ec. provide informaion regarding he objecs e.g. shape appearance and moion. Tracing Sysem Video Sream Resuls

40 Tracing as a coninuous moion segmenaion problem Tracing problem coninuous moion segmenaion problem: esimaion of a complee represenaion of foreground and bacground objecs over ime. Complee represenaion Layer) includes: moion of objecs and bacground shape of objecs and suppor appearance of objecs Key: consrains

41 Layer based moion analysis mehod Simulaneously achieve moion and segmenaion esimaion EM algorihm) Esimae segmenaion based on moion consisency Esimae moion based on segmenaion

42 Moion layer represenaions - models/consrains Local consrains Global consrains Muli-frame consisency Moion Smooh dense flow: Weiss 97 D affine: Darrell9 Wang93 Hsu94 Sawhney96 Weiss 96 Vasconcelos97 D roaion and ranslaion & consan velociy: This paper 3D planar: Torr99 Segmenaion MRF segmenaion prior: Weiss96 Vasconcelos97 Bacground+Gaussian segmenaion prior: This paper - Secion. Consan segmenaion prior: This paper - Ellipical shape prior Appearance Consan appearance: This paper

43 Dynamic Layer Represenaion Spaial and emporal consrains on he layer segmenaion moion and appearance EM algorihm for maimum a poseriori esimaion Layer ownership is consrained by a parameric shape disribuion insead of a local smoohness consrain. I prevens he layer evolving ino arbirary shapes and enables racable esimaion over ime.

44 Represenaion and consrains - segmenaion and appearance Segmenaion prior model bacground + ellipical shapes consan value over ime Φ = { l s} Bacground layer Layer j β γ Appearance model - consan value over ime A

45 Represenaion and consrains - moion Moion model moion foreground ranslaion + roaion consan velociy model Θ = u u ω ) ω bacground planar surface

46 MAP esimaion P moion image appearance moion shape _ appearance prior shape image _ prior ) moion appearance shape prior moion appearance shape prior moion appearance shape prior image - image image +

47 MAP esimaion - formulaion Noaion curren image is. Curren sae is. Esimaion I ] A [ Φ Θ Λ = ) ) ma arg ) ma arg = P P P I I I I I Λ Λ Λ Λ Λ Λ Λ Λ prior lielihood

48 Opimizaion using EM algorihm The general Epecaion Maimizaion algorihm observaion and parameer objecive funcion: equivalen o ieraively improving condiional epecaion For he dynamic layer racer: Opimize over ) log ] ) [log ) θ θ θ θ θ P y y P E Q + = ) ) arg ma θ θ θ P y P y θ ) log ] ) [log + = P I I I z I P E Q Λ Λ Λ Λ Λ Λ Λ Q

49 Opimizaion - 3 seps Opimizaion over moion segmenaion and appearance correspond o he following hree seps: layer moion esimaion based on curren segmenaion and appearance weighed correlaion or direc mehod layer segmenaion esimaion compeiion beween moion layers layer appearance esimaion Kalman filering of appearance

50 Opimizaion - flow char frame - frame updae ownership h i j esimae moion Θ updae ownership h i j esimae shape prior Φ updae ownership h i j esimae appearance A frame +

51 Opimizaion - illusraion moion - moion shape prior - moion esimaion shape esimaion appearance esimaion shape prior appearance - appearance frame

52 Opimizaion - equaions Moion esimaion weighed SSD Ownership esimaion - gradien mehod Appearance esimaion ) / / ) / )) / )) I j i A I i j i A i j j i j j h I h T A T A σ σ σ σ + + = 3. 0 ) / / ) ) ) ) )) ) ls j j j y j i n i i j i i j i j i j i j s s s y L D L L D h s f σ γ = = 3. 0 ) / / ) ) ) ) )) ) ls j j j j i n i i j i i j i j i j i j l l l y L D L L D h l f σ γ = =

53 Inference of objec saus A sae ransiion graph is designed o rigger evens such as objec iniializaion objec eliminaion infer objec saes such as moving sopping wo objecs ha are close o each oher ec.

54 Inference of Objec Saus NM {!NM&!NS} NM & SI occluded NB!NM OB LT!NM&NS new!nm &!OB moving!nm&ns OB disappear NM&!SI&!ZM NM &!SI&ZM NM OB NM&NS sop!ns OB LT Condiions NS = normal SSD score OB = ou of scope LT = NM for a long ime ZM= zero moion esimaion NB = new blob no objec covering a blob NM = no moion blob covering he objec SI = significan increase of SSD

55 Implemenaion - Sarnoff Layer Tracer Airborne Video Surveillance Sysem racing componen) Performance: Video Sream Sarnoff VFE 00 SGI Ocane Originally developed on a PC pored o SGI Ocane. 0-5 Hz for one objec over a single processor.

56 Turning Resuls

57 Resuls Turning a) b) c)

58 Resuls Passing - opposie direcions

59 Resuls Passing - opposie direcions a) b) c)

60 Resuls Passing - he same direcion

61 Resuls Passing - he same direcion a) b) c)

62 Sop Passing Resuls

63 Resuls Sop Passing a) b) c)

64 Implemenaion - Sarnoff Layer Tracer Moion esimaion: 95% of compuaion is for moion esimaion. Currenly weighed SSD correlaion is used. Searching in a 33 window a half resoluion for 3 differen angles. The size of he objec is around 4040 piels. Ownership esimaion change image is inegraed ino he formulaion o furher improve he robusness. Appearance esimaion appearance model for he bacground is no compued insead he previous image is used.

65 An Alernaive Appearance Model In he previous model appearance ges incremenally averaged over ime since i is par of he sae vecor A more sophisicaed appearance model allows for averaging as well as eeping up wih frame-o-frame appearance changes: Jepson e al. s WSL model A miure model of appearance Esimaed incremenally using online EM

66 WSL Adapive Model in D Wandering Process: consan variance σ w d μ s Sable Process: variance σ s Los Process d [ ] d Miure model for curren daa 4 dof): p d q m d ) m p d q ) + m p d d ) + = s s w w ml pl d ) sable parameers q = μ s σ s ) miing probabiliies m = ms mw ml)

67 On-Line Approimae EM One E One E-Sep: Sep: Compue daa ownerships only a curren ime ) ) ) = j j j d d p d d p m d o m q q } { l s w j One M One M-Sep: Sep: Updae weighed i h -order daa momens ) 0) j j M d m = 0) ) s s s M M = μ 0) ) s s s s M M σ μ = Updaed miing probabiliies 0 h order momens): Updaed mean and variance of sable process: ) ) ) ) i j i j i j M d d o M + = α α } { l s w j } { l s w j

68 Esimaion of Moion Parameers To esimae he moion model parameers we maimize he sum of log lielihood and log prior : O u) = log L D u A D ) + log p u u) where: warp parameers: daa a ime -: appearance model: parameric moion: lielihood u D = { d )} R - A = q m ) = w ; u) prior

69 Opimizaion Deails Daa Lielihood: daa from ime is warped bac o - and compared o predicions from he racing region a ime -. = )) ) ) ; ) - d A d p D A D L R u w u Moion Prior: ) ) ) 0 σ σ = G G p u u u u u slow smooh Fiing process for is similar o fiing miure models for flow Jepson & Blac 993). u

70 Real-Time Tracing

71 Major Componens of a Tracer Filered Represenaion Adapive Bacground Modeling Deeced change objecs Foreground Deecion Frame-oframe Tracing Filered Bacground Model Sae Machine

72 Tracer Bloc Diagram Sysem sae a ime A ime +) Sysem sae a ime +) Objec green bo) as seen a ime. laes model of appearance) Laes appearance model Objec appearance as learn from recen pas. learn model of appearance) Probabilisic visibiliy mas brigher he piel more liely ha i belongs o he objec Velociy esimae Deph if available moion esimaion appearance esimaion Occlusion handling visibiliy esimaion Updaed learn model Updaed visibiliy mas Velociy esimae Deph if available

73 Sample Progress of he Tracer Occlusion is deeced a his frame Noe learn model is much more immune o occlusions han he laes model. The appearance models and visibiliy mas are sill frozen o =8 because of occlusion The objec reappears afer occlusion and he models and visibiliy mas are updaed Sysem sae a ime = Sysem sae a ime =8 Sysem sae a ime =6 Sysem sae a ime =7

74 Tracer Feaures Non-parameric disribuion based bacground represenaion. Resilien o environmenal effecs lie wind-induced moion heainduced scinillaion ec. Foreground eracion based on pyramid filers and flow. Tunable for differen scenarios: oudoors indoors. Comprehensive racing based on appearance moion and shape. Auomaically adaps o smooh and sudden changes of appearance. Auomaically weighs appearance and shape maching. Precise moion esimaion based on opical flow. Sae machine ha eplois appearance moion and shape. Handles occlusions and confusing evens wih muliple objecs.

75 Eample: Oudoors

76 Eample: Indoor Overhead

77 Eample: Airpor Overhead

78 Eample: Airpor Ligh Traffic)

79 Eample: Airpor Sequence

80 Eample: Hallway Sequence

81 Eample: Hallway Sequence

82 3D Tracing wih Presence of Cluer and Muli-Camera Handoff Camera Camera Video of Camera and Camera Handing-off from camera o camera

83 3D Tracing in Oudoor Scenarios Original video Video wih enire mob being raced simulaneously Each color represens a differen person in he image Noe he 3D racer can disinguish beween people and heir shadows Deph Map Video

84 3D Tracing in Oudoor Scenarios Original video Video wih people and vehicles being raced simulaneously Each color represens a differen person/vehicle in he image Deph Map Video

85