Elaborazione delle Immagini Informazione multimediale - Immagini. Raffaella Lanzarotti

Transcription

1 Elaborazione delle Immagini Informazione multimediale - Immagini Raffaella Lanzarotti

2 OPTICAL FLOW Thanks to prof. Mubarak Shah,UCF 2

3 Video Video: sequence of frames (images) catch in the time Data: function of space (x,y) and time (t) 3

4 Motion field The motion field is the projection of the 3D movement in a scene on the image plane 4

5 Optical flow DEFINITION: the optical flow is the apparent movement of patterns of given brightness in the image GOAL: approximate the mo-on field by means of the op-cal flow, es-mated at each pixel (dense mo-on es-ma-on) NOTE: the optical flow does NOT coincide with the motion field 5

6 Optical flow ~= Motion field 1. Homogeneous objects generate zero optical flow. 2. Fixed sphere. Changing light source 6

7 OPTICAL FLOW f(x, y, t) f(x, y, t + 1) GOAL: Given two subsequent frames, estimate the apparent motion field between them ASSUMPTIONS: Brightness constancy: projection of the same point looks the same in every frame Small motion: points do not move very far Spatial coherence: points move like their neighbors 7 Steve Seitz

8 8

9 9

10 10

11 Representation: different colors according to the motion direction 11

12 12

13 Optical flow Brightness constancy assumption: f(x, y, t) =f(x + x, y + y, t + t) the point (x,y) moves of a quantity but it does not change appearance ( x, y) 13

14 Taylor s series Let f :[a, b]! R a function, and let x 0 2 (a, b). If exist f (1) (x 0 ),f (2) (x 0 ),...,f (n) (x 0 ), Taken h s.t. f is defined in [x 0 h, x 0 + h] f(x 0 + h) =f(x 0 )+ f (1) (x 0 ) 1! h + f (2) (x 0 ) 2! (h) f (n) (x 0 ) n! (h) n + R n+1 (h) 14

15 In compact form: Taylor s series f(x 0 + h) = P n i=0 f (i) (x 0 ) i! h i + R n+1 (h) Rn+1: residue of order n+1 x0: evaluation point 15

16 Optical flow Brightness constancy assumption: f(x, y, t) =f(x + x, y + y, t + t) Taylor series, with [h = ( )]: f(x + x, y + y, t + t) f(x, t 16

17 Optical flow Putting together the two equivalence Rewritten: t 0 f x x + f y y + f t t 0 And divided by : f x t x t + f y y t + f t 0 f x u + f y v + f t 0 17

18 f x u + f y v + f t =0 Interpretation f t f x v v = f x fy u f t f y (û, ˆv) p d u Equation of the line in the space (u,v) d = f t p f 2 x +f 2 y f t f y 18

19 The aperture problem Real motion 19

20 Aperture problem Perceived motion 20

21 Open problem Barberpole_illusion#/media/File:Barberpole-01.gif Barberpole_illusion#/media/ File:Barberpole_illusion_animated.gif 21

22 Lucas Kanade, 1981 Initially proposed as a technique for matching between stereo images Simple and intuitive Very effective Based on least square fit 22

23 Lucas Kanade, 1981 Based on the Brightness constancy assumption: f x u + f y v = f t Add another constraint: the Spatial coherence constraint: the motion is constant in small window (e.g. 3x3 or 5x5) 23

24 Lucas Kanade, 1981 Spatial coherence constraint: f x u + f y v = f x1 u + f y1 v = f t f t f x1. f x9 f y1. f y apple u v = f t1. f t f x9 u + f y9 v = f t9 Au = f t 24

25 Lucas Kanade, 1981 Au = f t A T Au = A T f t u =(A T A) 1 A T f t Solution: solve Least Squares problem

26 Condition for solvability 2 P f 2 xi 4 P fxi f yi P 3 fxi f yi 5 P f 2 yi apple u v = 2 4 P 3 fxi f ti 5 P fxi f ti A T A When is this solvable? A T A should be invertible A T A should not be too small ( 1, 2) eigenvalues should not be too small A T A should be well-conditioned 1/ 2 should not be too large, where 1 > 2 26

27 Condition for solvability 27

28 Edges gradients, and thus A T A, very large or very small large 1, small 2 28

29 Low-texture region gradients have small magnitude small 1, 2 29

30 High-texture region gradients (and thus A T A) are large large 1, 2 30

31 Procedurally 2 P apple f 2 u xi v = 4 P fxi f yi P 3 fxi f yi 5 P f 2 yi P 3 fxi f ti 5 P fyi f ti P P 2 f 2 P P f 2 xi f 2 yi ( P yi fxi f yi f xi f yi ) 2 apple u v = P fxi f yi P f 2 xi 3 54 P 3 fxi f ti 5 P fyi f ti 31

32 Procedurally 32

33 Recap Lucas- Kanade op-cal method to es-mate the mo-on field Assump-ons: brightness constancy between consecu-ve frames Spa-al coherence constraint: Fails when the displacement is large (typical opera-ng range is mo-on of 1 pixel) Solution: adopt multi-scale representations (pyramids) 33

34 Optical flow with rapid motion Fails in the areas with large motion/ variation 34

35 Solution: reduce the resolution Image pyramid 35

36 Pyramid 36

37 Optical flow Coarse-to-fine Progressive image downsampling 37

38 Gaussian Pyramid 38

39 Optical flow Coarse-to-fine compute LK Compute LK and refine Interpolation and doubling... 39

40 Pyramidal LK, Algorithm Compute LK at the highest level At level i take the OF at level i-1: (ui-1, vi-1) Bilinear interpolation to determine (ui *, vi * ) (ui *, vi * ) = 2 x (ui *, vi * ) Refine the OF computing ui, vi (flow correction) ui = ui * + ui vi = vi * + vi 40

41 Pyramidal Lucas Kanade 41