Optic Flow Computation with High Accuracy

Transcription

1 Cognitive Computer Vision Colloquium Prague, January 12 13, 2004 Optic Flow Computation with High ccuracy Joachim Weickert Saarland University Saarbrücken, Germany joint work with Thomas Brox ndrés Bruhn Nils Papenberg partially funded by DFG

2 The Optic Flow Problem Introduction (1) given: image sequence I(x) where x = (x, y, t) can be Gaussian-smoothed: I = K σ I 0 wanted: displacement field (optic flow) w = (u, v, 1) w matches object at location (x, y) at time t to its location (x+u, y+v) at time t + 1. What is Optic Flow Good for? extracting motion information e.g. in robotics compact coding of image sequences related correspondence problems in computer vision: e.g. stereo reconstruction and medical image registration

3 Introduction (2) Deformation analysis of plastic foam using an optic flow method. (a) Top left: Frame 1 of a deformation sequence. (b) Top right: Frame 2. (c) Bottom left: Colour-coded displacement field. (d) Bottom right: Vector plot of the displacement field.

4 Introduction (3) Pair of stereo images.

5 Introduction (4) Four views of a stereo reconstruction algorithm based on optic flow ideas. uthors: lvarez/deriche/sánchez/weickert (2002)

6 Variational Optic Flow Methods Introduction (5) optic flow as minimiser of a suitable energy functional: data constraints plus smoothness constraints clear formalism without hidden model assumptions rotationally invariant continuous formulations possible create dense flow fields first model due to Horn and Schunck (1981), but many improvements in the meantime: modified data and smoothness constraints (Nagel 1983, Cohen 1993, lvarez et al. 1999, W./Schnörr 2000) theoretical foundation (Snyder 1991, W./Schnörr 2000) efficient numerical algorithms (Glazer 1984, Terzopoulos 1986, Ghosal/Vaněk 1996, Bruhn et al. 2003) competitive performance

7 Introduction (6) Open Problems further improvements possible? some very good methods use strategies that lack theoretical foundation Goals presentation of an optic flow algorithm with very good performance theoretical justification of widely used warping technique Some Related Work L. lvarez, J. Weickert, and J. Sánchez, IJCV M. Lefébure and L. D. Cohen, JMIV E. Mémin and P. Pérez, ICCV 1998.

8 Outline Variational Model lgorithmic spects Relations to Warping Evaluation Conclusions Outline

9 Outline Variational Model lgorithmic spects Relations to Warping Evaluation Conclusions Outline

10 Basic assumptions Greyvalue constancy Gradient constancy Spatio-temporal smoothness Robustness Energy to minimise: E (u, v) = Ω Variational Model (1) I w := I(x + w) I(x) = 0 I xw := x I(x + w) x I(x) = 0 I yw := y I(x + w) y I(x) = 0 u 2 + v 2 = 0 = ( x, y, t ) Ψ ( s 2) = s 2 + ɛ 2 Ψ ( Iw 2 + γ (I xw 2 + Iyw 2 ) ) dx + α Ω Ψ ( u 2 + v 2) dx

11 Variational Model (2a) Minimiser has to fulfill the Euler-Lagrange equations α div ( Ψ ( u 2 + v 2 ) u ) = Ψ ( I 2 w + γ(i 2 xw + I 2 yw) ) (I x I w + γ ( I xx I xw + I xy I yw ) ) α div ( Ψ ( u 2 + v 2 ) v ) = Ψ ( I 2 w + γ(i 2 xw + I 2 yw) ) (I y I w + γ ( I xy I xw + I yy I yw ) ) where the indices denote differences or partial derivatives: I w := I(x + w) I(x) I x := x I(x + w) I y := y I(x + w) I xw := x I(x + w) x I(x) I xx := xx I(x + w) I yy := yy I(x + w) I yw := y I(x + w) y I(x) I xy := xy I(x + w)

12 Variational Model (2b) Minimiser has to fulfill the Euler-Lagrange equations α div ( Ψ ( u 2 + v 2 ) u ) = Ψ ( I 2 w + γ(i 2 xw + I 2 yw) ) (I x I w + γ ( I xx I xw + I xy I yw ) ) α div ( Ψ ( u 2 + v 2 ) v ) = Ψ ( I 2 w + γ(i 2 xw + I 2 yw) ) (I y I w + γ ( I xy I xw + I yy I yw ) ) where the indices denote differences or partial derivatives: I w := I(x + w) I(x) I x := x I(x + w) I y := y I(x + w) I xw := x I(x + w) x I(x) I xx := xx I(x + w) I yy := yy I(x + w) I yw := y I(x + w) y I(x) I xy := xy I(x + w) S ( w ) = D (w)

13 Outline Variational Model lgorithmic spects Relations to Warping Evaluation Conclusions Outline

14 Problem 1: Local Minima energy functional E(u, v) is not convex reason: terms involving I(x + w) lgorithmic spects (1) should not be linearised for large displacements numerical algorithms may yield suboptimal local minima of E(u, v), if initialisation is not chosen properly Solution: Initialisation by Coarse-to-Fine Strategy downsample problem in a full pyramid start with zero displacement at coarsest scale solve Euler-Lagrange equations S(w) = D(w) use resulting flow field as initialisation at next finer scale

15 lgorithmic spects (2) Problem 2: Solve Euler-Lagrange Equations discretise S(w) = D(w) by finite differences yields large nonlinear system of equations nonlinearity caused by I(x + w) and nonlinear penaliser Ψ Solution nonlinear system is simplified by two nested fixed point iterations linearisation of I(x + w) leads to large linear system of equations can be solved by iterative methods such as SOR

16 lgorithmic spects (3) Detailed Structure on the Linear System Resulting linear system for du k,l+1, dv k,l+1 : α div ( (Ψ ) k,l Smooth (u k + du k,l+1 ) ) ( = (Ψ ) k,l Data Ix k ( I k z + Ixdu k k,l+1 + Iy k dv k,l+1) + γ ( I k xx(i k xz + I k xxdu k,l+1 + I k xydv k,l+1 ) + I k xy(i k yz + I k xydu k,l+1 + I k yydv k,l+1 ) )) α div ( (Ψ ) k,l Smooth (v k + dv k,l+1 ) ) ( = (Ψ ) k,l Data Iy k ( I k z + Ixdu k k,l+1 + Iy k dv k,l+1) + γ ( I k xy(i k xz + I k xxdu k,l+1 + I k xydv k,l+1 ) + I k yy(i k yz + I k xydu k,l+1 + I k yydv k,l+1 ) ))

17 Outline Variational Model lgorithmic spects Relations to Warping Evaluation Conclusions Outline

18 Warping Relations to Warping (1) widely used for optic flow computation with large displacements (e.g. nandan 1989, Black/nandan 1996, Mémin/Pérez 1998) downsample image data solve problem at coarse scale use this flow field at next finer scale: warp image in order to compensate for this estimated motion solve modified problem (with other image data) at finer scale continue until finest scale reached sum up optic flow contributions from all scales successful in practice, but no theoretical justification!

19 We have proven equivalence between Relations to Warping (2) our numerical method for minimising a simplified energy E(u, v) by coarse-to-fine flow initialisations and nested fixed point iterations warping method (nested problems with motion-compensated image data) of Mémin / Pérez They lead to the same linear system of equations. This explains the success of warping: Warping has a sound theory as a numerical algorithm for minimising a single energy functional!

20 Outline Variational Model lgorithmic spects Relations to Warping Evaluation Conclusions Outline

21 Sequence Ground Truth Evaluation (1) Yosemite Sequence Synthetic sequence ( ) Known ground truth between frame 8 and frame 9 Computed Flow

22 Qualitative Evaluation Evaluation (2) Vector plot of the optic flow field for the Yosemite sequence with clouds. (a) Left: Ground truth. (b) Right: Computed flow.

23 Evaluation (3) Vector plot of the optic flow field for the Yosemite sequence without clouds. (a) Left: Ground truth. (b) Right: Computed flow.

24 Quantitative Evaluation Evaluation (4) Comparison to the best results from literature verage angular errors (E) and standard deviations (STD) for the Yosemite sequence with coluds: Yosemite with clouds Technique E STD nandan Nagel Horn/Schunck, mod Uras et al lvarez et al Weickert et al Mémin/Pérez our method

25 Evaluation (5) For the Yosemite sequence without clouds, even better results are possible: Yosemite without clouds Technique E STD Black/nandan Ju et al Bab-Hadiashar/Suter Lai/Vemuri Mémin/Pérez Weickert et al Farnebäck our method

26 Robustness under Noise Evaluation (6) dded Gaussian noise with zero mean and different standard deviations σn. Results for Yosemite sequence with clouds: σ n E STD verage angular error for σn = 40 outperforms all other methods with σn = 0!

27 Evaluation (7) Frame 8 of the Yosemite sequence with clouds. (a) Left: Original. (b) Right: Gaussian noise with standard deviation σ n = 40 added.

28 Evaluation (8) Robustness under Parameter Variations Three intuitive parameters: σ: Gaussian presmoothing of the input data α: weight of smoothness term γ: weight of gradient constancy term Parameter variation for the Yosemite sequence with clouds: σ α γ E Deviations from the optimum by a factor 2 hardly influence the result.

29 Real-World Data Evaluation (9) Real-world image sequence Ettlinger Tor by Nagel ( ) Sequence Computed Flow

30 Outline Variational Model lgorithmic spects Relation to Warping Evaluation Conclusions Outline

31 Summary Conclusions (1) novel model gradient constancy assumption within energy functional combines many successful features in a single functional novel theory postpone all linearisations to the numerical scheme numerical scheme based on two nested iterations warping theoretically justified as a special numerical approximation excellent results angular errors belong to smallest in the literature robust under parameter variations highly robust under noise

32 Ongoing Work alternative data terms Conclusions (2) correpondence between data and smoothness terms automatised selection of smoothing parameters σ, α more efficient numerics: PCG, multigrid, domain decomposition novel warpings inspired from suitable numerics? Message It is advantageous to combine transparent continuous modelling with consistent numerics. Good performance and deeper theoretical understanding are not contradictive: They are two sides of the same medal.

33 Thanks Thank you very much! more informations:

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51