Jorge Marques, Image motion

Transcription

1 Image motion

2 finding a temlate Suose we wish to find a known temlate ) in a given image I). his roblem is known as temlate matching. temlate image FC Barcelona the temlate can be small or large

3 alignment MRI images) atlas test slice Adated from Kubic Unser 2003

4 temlate matching I uv) temlate ) = 0... B - image I ) = 0... N - dislacement t = u v ) he solution is based on two stes: define a matching criterion M e.g. cross correlation) find local maima/minima e.g. ehaustive search)

5 object detection matching criterion: M nonlinear otimization Non-minimum suression: M t 0 ) < M t) for all t in a e vicinit of radius r of d hresholding: M t 0 ) < λ λ threshold

6 matching criteria ) ) ) 0 v u I v u R B + + = = cross-correlation B 2 0 )] ) [ ) v u I v u E B + + = = sum of square differences SSD) l 2 norm squared) sum of absolute differences SAD) l norm) ) ) ) 0 v u I v u E B + + = = Non integer dislacements can be considered. Image interolation is required in this case.

7 cross-correlation SSD SAD here are better was to detect faces e.g. Viola & Jones)!!

8 limitations emlate matching has weaknesses: not invariant to rotations and scaling not invariant to illumination changes time consuming more general transformations modif matching criteria to imrove robustness temlate adatation is trick

9 roblem formulation Image alignment Given 2 or more) images I we wish to estimate a transformation which mas the first into the second ) ' ') ' ') = W ; θ) according to some criterion. his can be done using: feature based methods: based on the alignment of feature oints marks) Matlab image based methods: based on the alignment of image intensit or color

10 What geometric transformations can we use?

11 translation & rigid bod translation W ;θ) = + t θ = t 2 degrees of freedom rigid bod 2 2 W;θ) = R + t 3 degrees of freedom θ = Rt) rotation matri RR detr) = = R R = I cosθ R = sinθ -sinθ cosθ

12 afinne and rojective transformations affine transformation q r q r W; θ) = A + t 6 degrees of freedom θ = At) rojective transformation homograh) r q r s s q Wθ) = θ=...9 ) 8 degrees of freedom

13 rojective and olnomial transformations ~ ~ ~ ~ = = ' ' ] [ ] [ ] [ ] [ ~ = = = = rojective contd.) olnomial = + + q q q q a b n q q n q q : : Wθ) others. e.g. free form deformations he estimation of coefficients is numericall ill conditioned

14 roerties DoF Preserves lines? Preserves Paralelism? Preserves Angles? Preserves length? translation 2 Yes Yes Yes Yes Rigid bod 3 Yes Yes Yes Yes Affine 6 Yes Yes X X Projective 8 Yes X X X Polnomial n+2)n+)/2 X X X X

15 can we align images using intensit?

16 image based methods Problem: Given two images I we wish to find a geometric transformation W) which mas oints of the first image into oints of the second such that IW)) ). color constanc Most oular criterion SSD) 2 E θ) = [ ) I W ; θ))] Note: the sum is for all the oints in which both images ) IW)) overla. he minimization of E is a non linear roblem!!

17 Lucas-Kanade translation motion) Criterion E u v) = [ ) I + t)] 2 Parameter udate t = t 0 + t First order aroimation of the image I + t) = I + t0) + I + t0) t Lucas Kanade algorithm recursion) 2 I I I I I t 2 I = ) I + t0)) I ) I + t0)) I R t=r t t 0 + t t t 0 + t I I are the artial derivatives of I at +t 0.

18 convergence from several starting oints SSD criterion he SSD criterion is not elicitl comuted in the L-K algorithm.

19 roof 0 ) t t] ) t ) t ) [ = = I I I t d de ] ) t ) t ) [ t I I E + + = Let us minimize A necessar condition is ) t )] t ) [ t ) t ) t = + + I I I I Defining + + = + ) t ) t ) t I I I We obtain + + = I I I I I I I I I I )) t ) )) t ) t

20 discussion L-K strong oints uses all the available information It is simle aroriate for tracking can be etended to deal with general motion models L-K weak oints no guarantee that the otimal solution is obtained the solution deends on the initialization convergence is difficult if the number of arameters is high solution deends on the illumination use multile scales Illumination can be estimated

21 can we align images from sarse rototes?

22 feature based matching Problem: Given two sets of oints { i } { j } detected in the images I we wish to find a geometric transformation W that mas the oints { i } into the oints { j }. 2 2 i i = ' i i = i' i' n I n we assume that the corresondence is known i ' i

23 aroach Define a matching criterion e.g. E θ) = ' i 2 i Wi;θ) SSD criterion Minimize the criterion with resect to θ using a closed form or a numeric algorithm. Note: there are other matching e.g. l norm.

24 eamle inut outut alignment using a rojective transform

25 estimation of an homograh Homograh '= f) ' ' = = = is a nonlinear function of the unknown arameters. he minimization of the SSD criterion is difficult!! 2 E ) = ' i fi) i Idea: use another simler) criterion instead ) ' 9) ' = ) = ) algebraic error e = ) ) ) ' ) ' 2 E' ) = ei i =

26 estimation of the rojective transform 2) M M ' = E = minimize with restriction ' ' ' ' ' ' = n n n n n n n M M M M M M M M M M M M M M his roblem can be easil solved using Lagrange multiliers: is the eigenvector of matri M M associated to the smallest eigenvalue. ' ' n n n n n n n M M M M M M M he whole algorithm can be written in long) line of Matlab!

27 roof Lagrangian function L = E' λ ) = M M λ ) dl d = 0 M M λ = 0 M M = λ is na eigen vector of matri M M which one? E = M M = λ = λ choose λ min

28 other transformations? he other transformations translation affine olnomial) are easil estimated b the minimization of the SSD criterion E. Onl the rigid bod transformation is a bit more difficult because matri R is not free. It is a rotation matri: R R=RR =I and the SSD criterion must be oimized under this restriction. his roblem can be solved using the singular vector decomosition of the data.

29 unknown corresondence his is a difficult roblem! We need to estimate a ermutation matri = which minimizes the matching criterion E. tough! See the aer b Maciel & Costeira PAMI03 subotimal aroaches are used instead!

30 ransac RANSAC stands for Random Samle Consensus Fischler Bolles 98 ) It is based on hothesis generation and classification of data oints as inliers and outliers. estimate translation onl 2 oints are matched! bad attemt!

31 ransac 2) Objective: to estimate a transform Wq) with 2n degrees of freedom. Algorithm Hotheses generation randoml select n airs of oints i k ) estimate the geometric transformation Wθ) Comute the number of oints which were correctl aligned suort) i.e. such that ' k Wiθ) < ε Model selection: choose the transformation with largest suort Refinement: imrove the estimate of θ b aling the least squares method to the subset of oints which are well aligned.

32 eamle - registration Afine transform 3 marks) Matlab demo)

33 eamle - mosaicing homograh 4 marks)

34 eemlo cont.)

35 mosaicing mosaicing alignment + fusion

36 3D ultrasound without alignment with alignment

37 non-rigid alignment Kbic Unser 2003

38 region tracking

39 wo stes region detection region tracking

40 Region detection

41 roblem goal: detect all moving objects assumtions static camera static ti background show illumination changes

42 Evolution of iel color iel t=0 t=000

43 Background subtraction background foreground Piel classification background image If I)-B) < ε the iel is classified as background iel. Otherwise it is classified as active.

44 Basic background subtraction he basic background subtraction classifies a iel I) as active if A ) = if I ) B ) > λ A ) = 0 otherwise Image A) is ver nois. It has man small regions classified as active and some true objects aear fragmented in several regions. Morhologinal ost-rocessing is usuall done. icall we comute all conected comonents and eliminate all the small regions.

45 Eamle I B eliminação de regiões equenas

46 How to deal with time-varing illumination? Illumination changes can be comensated b the adatation of the background image. Onl the iels belonging to the background regiion should be adated. B t) = αb t ) + α) I t) background iels B t) = B t ) foreground iels

47 Gaussian background model )) ) ~ ) R µ N I Cackground iels are corruted b noise. We can model each iel as a random variable with Gaussian distribution iel classification B see Wren et al. 997 ) λ I λ I < )) )) background iel foreground iel µ R G B )) ) )) ) ) det ) / 2 3/ )) µ I R µ I R π e I =

48 Estimation of the Gaussian model batch = = = = t t µ t I µ t I R t I µ )) ) )) ) ) ) ) t µ t I t µ t I α t αr t R t I α t αµ t µ )) ) )) ) ) ) ) ) ) ) ) + = + = adative = t Onl background iels should be used

49 region tracking

50 region tracking occlusion false alarm new track Goal: find the trajector of each object along multile frames Dificulties: misdetections false alarms occlusions object slits and merges new tracks

51 oint tracking t- t t+ Data D = { t t i )} t i osition of the i-th region at frame t rack is a sequence of oints detected at different usuall consecutive) frames = { t ) t2 2 )... tn n )} ti i ) D ti < ti + t i+ =t i +)

52 oint association t- t t+ available methods: Statistical: roagate uncertaint and assume a dnamic model for the target trajectories e.g. Kalman or PDA filter) Deterministic: based on assignment costs and do not require dnamic models e.g. grah based methods)

53 hotheses tical assumtions a) onl regions detected in consecutive frames can be associated b) regions should corresond to a single target and vice-versa) c) new objects ma aear track birth) d) objects can disaear or be occluded track death) b ) objects can overla and form grous

54 statistical methods Statistical methods assume we know a set of tracks and wish to etend them in new frames. Involve 3 stes: rediction data association udate observations frame t+ current estimate frame t rediction frame t+ Difficulties: data association roblem initialization of new tracks Methods: nearest neighbor Kalman filter robabilistic data association filter joint robabilistic data association filter article filter

55 methods based on grahs 3 2 Nodes cooresond to the detected objects in each frame and the links define a solution for the association roblem Each admissible link has a cost C t ij) unconnected nodes also have a cost).

56 Veenman et al PAMI 200) M tracks m targets his method deals with airs of frames and formulates the association of targets to eisting tracks as an assignment roblem if M=m.

57 assignment roblem Problem: there are M agents and m tasks M=m); we wish to assign one agent to one task minimizing the total cost m C = a ij c ij i i = agents tasks Restrictions m a = i = ij m j = a ij = a ij {0} c ij is the cost of assigning agent i to task j and a ij is a binar variable wich is equal to if and onl if agent i is assigned to task j. he minimization of C under these restrictions is a linear rogramming roblem for which there are ver efficient algorithms e.g. Hungarian method.

58 Eamle = C = A tasks total cost: 0+2+2=4 agents

59 cost matri In tracking the association cost can be defined in different was. wo oular choices are t t distance criterion aij = i j t t t rediction error aij = i + vi j t v dislacement vector comuted from a revious i assignment. cannot be used in track initialization)

60 Birth and death of tracks he revious method does not account for new tracks but it has been etended to allow birth and death of tracks Consider a roblem in which all the targets are new. In this case all the M tracks should die are all the m targets corresond to new tracks. How can we do this in the revious framework? solution: add M virtual targets and m virtual tracks m M C cij = chigh if i > M or j > m

61 Eamle d frame costs were comuted using the rediction error ecet at the beginning of each track.

62 How can we deal with grous?

63 tracking of edestrians in grous work of Pedro Jorge

64 Dealing with grous

65 Problem Goal: track all edestrians in the resence of occlusions and grous

66 Bottom u aroach Hothesis: ) low level algorithms erform well most of the time 2) difficult cases should be solved b a higher level module e.g. occlusions grou merging and slitting) region detection region association conflict management

67 Low level rocessing bakground subtraction + region matching mutual favorite airing) column time how do we assign a color to each track stroke)?

68 Problem formulation: labeling Detected strokes Sace Baesian net ime 8 7 Phsical constrains: causalit and ma. occlusion time and seed

69 What is a Baesian network? It is a robabilistic model for a set of variables n Drect deendences are reresented b a grah i are the arents of node i Hothesis Markov roert)... ) = i i i i ) P ) = P 4 3 ) P 3 2 ) P 2 ) P ) Factorization n... n) = i i ) i=

70 Baesian network generation 2 each node is associated to a stroke links are created between nodes which are close and could be associated to the same object 8 7 We comute the set of admissible labels and robabilit distribution for each node.

71 Admissible Labels Each node has a set of admissible labels {34} {2} {2} grous {3)4)23)24)234} {3)4)23)24)234} {3)4)23)24)234}

72 Basic Blocks hree basic blocks occlusion merging merging How can we deal more difficult cases merge-slit)? virtual node

73 Conditional robabilit tables i j P occ P j i ) = Pnew j j = i = new i j N P slit /2 P j i ) = Pocc P new 2) j P i ) \ { i i } j j = i = new j N P j Pocc... N ) = Pnew Pmerge / L j = or...or j j = new otherwise = N L number of different grou merges

74 Observations Each label node has an observation 4 node associated In this work we etract the 3 most significant colors of the image region and comare with the most significant colors of the model

75 Inference What is the label distribution in each model? his is the role of inference! Inference can be done using the junction tree algorithm. We have used Kevin Murh s toolbo for Matlab.

76 Eamle

77 Eamle 2)

78 Eamle

79 Eamle2)

80 Eamles