Multigrid Acceleration of the Horn-Schunck Algorithm for the Optical Flow Problem

Multigrid Acceleration of the Horn-Schunck Algorithm for the Optical Flow Problem El Mostafa Kalmoun kalmoun@cs.fau.de Ulrich Ruede ruede@cs.fau.de Institute of Computer Science 10 Friedrich Alexander University Germany

Overview Introduction and Related Works Optical Flow Constraints Regularization Horn Schunck Algorithm Multigrid Scheme A Simple Illustration Variational Multigrid Experimental Results Slide 2 of 23

Introduction and Related Works Frame at time t y x Slide 3 of 23

Introduction and Related Works Frame at time t Frame at time t + dt y dy dx y+dy x x+dx Slide 3 of 23

Introduction and Related Works Frame at time t Frame at time t + dt y dy dx y+dy x x+dx The optical flow at the pixel (x,y) is the 2D-velocity vector (u,v) = ( dx dt, dy dt ) Slide 3 of 23

Introduction and Related Works Frame at time t Frame at time t + dt y dy dx y+dy x x+dx The optical flow at the pixel (x,y) is the 2D-velocity vector (u,v) = ( dx dt, dy dt ) An approximation of the 2D-motion Slide 3 of 23

Introduction and Related Works Term originates with James Gibson in 1979 Quadratic smoothness Horn Schunck (1981) Registration technique with local constraints Lucas Kanade (1981) Oriented smoothness Nagel Enkelmann (1983 86) Multigrid relaxation Terzopolous (1986) Performance evaluation of popular algorithms Barron Fleet Beauchemin (1994) General anisotropic smoothness Weickert (1996) Optimal control framework Borzi Ito Kunisch (2002) Slide 4 of 23

Optical Flow Constraints I(x,y,t) : The image intensity of the pixel (x,y) at time t. I x,i y,i t : Spatial and temporal derivatives of I. Slide 5 of 23

Optical Flow Constraints I(x,y,t) : The image intensity of the pixel (x,y) at time t. I x,i y,i t : Spatial and temporal derivatives of I. Assumption : Objects keep the same intensity over time I(x,y,t) = I(x + dx,y + dy,t + dt) Slide 5 of 23

Optical Flow Constraints An equivalent form of the (OFCE) : I. w = I t DI.( w,1) = 0 where I = (I x,i y ), DI = (I x,i y,i t ) and w = (u,v) Slide 6 of 23

Optical Flow Constraints An equivalent form of the (OFCE) : u I. w = I t DI.( w,1) = 0 where I = (I x,i y ), DI = (I x,i y,i t ) and w = (u,v) v Slide 6 of 23

Optical Flow Constraints An equivalent form of the (OFCE) : u I. w = I t DI.( w,1) = 0 where I = (I x,i y ), DI = (I x,i y,i t ) I and w = (u,v) v Slide 6 of 23

Optical Flow Constraints An equivalent form of the (OFCE) : I. w = I t DI.( w,1) = 0 I t / I u where I = (I x,i y ), DI = (I x,i y,i t ) I and w = (u,v) v Slide 6 of 23

Optical Flow Constraints An equivalent form of the (OFCE) : I. w = I t DI.( w,1) = 0 I t / I u where I = (I x,i y ), DI = (I x,i y,i t ) I and w = (u,v) v OFC line Slide 6 of 23

Optical Flow Constraints An equivalent form of the (OFCE) : I. w = I t DI.( w,1) = 0 I t / I u where I = (I x,i y ), DI = (I x,i y,i t ) w 1 I and w = (u,v) v OFC line Slide 6 of 23

Optical Flow Constraints An equivalent form of the (OFCE) : where I. w = I t DI.( w,1) = 0 I t / I w 2 u I = (I x,i y ), DI = (I x,i y,i t ) w 1 I and w = (u,v) v OFC line Slide 6 of 23

Optical Flow Constraints An equivalent form of the (OFCE) : where I. w = I t DI.( w,1) = 0 I t / I w 2 u I = (I x,i y ), DI = (I x,i y,i t ) w 1 I and w = (u,v) v OFC line We can only calculate the normal component of the velocity w and not the tangent flow Slide 6 of 23

Regularization The (OFCE) is replaced by min (u,v) x y E(u,v)dxdy (P) Slide 7 of 23

Regularization The (OFCE) is replaced by where min (u,v) x y E(u,v)dxdy (P) E = E d + αe r, E d (u,v) = (I x u + I y v + I t ) 2 (the data term) E r is the regularization term Slide 7 of 23

Regularization The (OFCE) is replaced by where min (u,v) x y E(u,v)dxdy (P) E = E d + αe r, E d (u,v) = (I x u + I y v + I t ) 2 (the data term) E r is the regularization term and α is a positive scalar for adjustment between E d and E r Slide 7 of 23

Horn-Schunck Algorithm A standard choice of E r is the isotropic stabilizer : E r (u,v) = u 2 + v 2 Slide 8 of 23

Horn-Schunck Algorithm A standard choice of E r is the isotropic stabilizer : From calculus of variations, we get E r (u,v) = u 2 + v 2 α u I x (I x u + I y v + I t ) = 0 α v I y (I x u + I y v + I t ) = 0 Slide 8 of 23

Horn-Schunck Algorithm A standard choice of E r is the isotropic stabilizer : From calculus of variations, we get E r (u,v) = u 2 + v 2 α u I x (I x u + I y v + I t ) = 0 α v I y (I x u + I y v + I t ) = 0 We discretize the Laplacien by the standard 5 point stencil 1 1 4 1 1 and use w = w w where w is the average of the neighbors. Slide 8 of 23

Horn-Schunck Algorithm Spatial and Temporal Image Derivatives Masks Slide 9 of 23

Horn-Schunck Algorithm Spatial and Temporal Image Derivatives Masks M x = 1 4 ( 1 1 1 1 ) ( 1 1 M y = 1 4 1 1 ) M t = 1 4 ( 1 1 1 1 ) Slide 9 of 23

Horn-Schunck Algorithm Spatial and Temporal Image Derivatives Masks M x = 1 4 ( 1 1 1 1 ) ( 1 1 M y = 1 4 1 1 ) M t = 1 4 ( 1 1 1 1 I x = M x (I 1 + I 2 ) I y = M y (I 1 + I 2 ) I t = M t (I 2 I 1 ) ) Coupled Gauss-Seidel relaxation u k+1 = ū k I x I x ū k + I y v k + I t α + I 2 x + I 2 y v k+1 = v k I y I x ū k + I y v k + I t α + I 2 x + I 2 y Slide 9 of 23

Multigrid Scheme Multigrid Components Slide 10 of 23

Multigrid Scheme Multigrid Components Vertex-centered grid and standard coarsening Slide 10 of 23

Multigrid Scheme Multigrid Components Vertex-centered grid and standard coarsening Coupled lexicographic point Gauss-Seidel smoother Slide 10 of 23

Multigrid Scheme Multigrid Components Vertex-centered grid and standard coarsening Coupled lexicographic point Gauss-Seidel smoother Full weighting and bilinear interpolation Slide 10 of 23

Multigrid Scheme Multigrid Components Vertex-centered grid and standard coarsening Coupled lexicographic point Gauss-Seidel smoother Full weighting and bilinear interpolation Discretization coarse grid approximation (DCA approach) Slide 10 of 23

A Simple Illustration Intensity value : I(x,y,t) = x + y +t Slide 11 of 23

A Simple Illustration Intensity value : I(x,y,t) = x + y +t Corresponding system of PDE s : α u = u + v + 1 α v = u + v + 1 Slide 11 of 23

A Simple Illustration Intensity value : I(x,y,t) = x + y +t Corresponding system of PDE s : α u = u + v + 1 α v = u + v + 1 Method α = 1 and u 0 v 0 Convergence rate Horn-Schunck 0.998 V(1,0) 0.370 V(1,1) 0.183 V(2,1) 0.116 V(3,3) 0.056 Slide 11 of 23

Variational Multigrid Write the system of the two PDE s as ( u L v ) = ( f g ) Slide 12 of 23

Variational Multigrid Write the system of the two PDE s as ( u L v ) = ( f g ) where f = I x I t, g = I y I t ( α 0 L = L d + L r, L r = 0 α and L d is a 2x2 block diagonal matrix with entries ) ( I 2 x I x I y I x I y I 2 y ) Slide 12 of 23

Variational Multigrid Variational Minimization Form Lw = F w = argmin a(z) Ω Slide 13 of 23

Variational Multigrid Variational Minimization Form Lw = F w = argmin a(z) Ω a(z) = 1 2 (Lz,z) (F,z) Slide 13 of 23

Variational Multigrid Variational Minimization Form Lw = F w = argmin a(z) Ω a(z) = 1 2 (Lz,z) (F,z) Galerkin Approach Let I h H : ΩH Ω h be a full rank linear mapping. Slide 13 of 23

Variational Multigrid Variational Minimization Form Lw = F w = argmin a(z) Ω a(z) = 1 2 (Lz,z) (F,z) Galerkin Approach Let I h H : ΩH Ω h be a full rank linear mapping. An optimal coarse grid correction I h H w H of w h is characterized by ((I h H) T L h I h H)w H = (I h H) T (F L h w h ) Slide 13 of 23

Variational Multigrid The CGO is chosen then as follows L H = I H h L h I h H and I H h = (I h H) T Slide 14 of 23

Variational Multigrid The CGO is chosen then as follows L H = I H h L h I h H and I H h = (I h H) T For our system, we get L H = = ( R 0 0 R )( L 1 h L 2 h L 2 h ( RL 1 h P RLh 2 P RLh 2 P RL3 h P ) L 3 h )( P 0 0 P ) Slide 14 of 23

Variational Multigrid Multigrid Components Slide 15 of 23

Variational Multigrid Multigrid Components Vertex-centered grid and standard coarsening Coupled lexicographic point Gauss-Seidel smoother Full weighting and bilinear interpolation Slide 15 of 23

Variational Multigrid Multigrid Components Vertex-centered grid and standard coarsening Coupled lexicographic point Gauss-Seidel smoother Full weighting and bilinear interpolation Galerkin coarse grid approximation (GCA approach) The General Scheme READ FRAMES COMPUTE SPATIAL and TEMPORAL FRAMES DERIVATIVES CALCULATE the FINEST GRID OPERATOR and the RHS Slide 15 of 23

Experimental Results Slide 16 of 23

Experimental Results A rotating sphere (128x128) Slide 16 of 23

Experimental Results A rotating sphere (128x128) the flow field Slide 16 of 23

Experimental Results the flow field after scaling Slide 17 of 23

Experimental Results The marble sequence (512x512) Slide 18 of 23

Experimental Results The marble sequence (512x512) the flow field Slide 18 of 23

Experimental Results the flow field after scaling Slide 19 of 23

Experimental Results The Hamburg taxi sequence (256x190) Slide 20 of 23

Experimental Results The Hamburg taxi sequence (256x190) the flow field Slide 20 of 23

Experimental Results the flow field after scaling Slide 21 of 23

Experimental Results α = 5 and u 0 = v 0 = 0 on P4 2.4GHz Sphere Marble Taxi ρ CPU ρ CPU ρ CPU Horn-Schunck 0.98 3.9 0.98 106 0.98 20.8 VMG V(2,1) 0.15 1.1 0.43 23.8 0.45 4.3 Slide 22 of 23

Experimental Results α = 5 and u 0 = v 0 = 0 on P4 2.4GHz Sphere Marble Taxi ρ CPU ρ CPU ρ CPU Horn-Schunck 0.98 3.9 0.98 106 0.98 20.8 VMG V(2,1) 0.15 1.1 0.43 23.8 0.45 4.3 + Robust (but not yet optimal convergence) * need matrix dependent transfer grid operators Slide 22 of 23

Future work Application in medicine imaging Consider other regularization (e.g. by Nagel, Weickert...) Slide 23 of 23

Future work Application in medicine imaging Consider other regularization (e.g. by Nagel, Weickert...) PLEASE WAKE UP! Slide 23 of 23

Future work Application in medicine imaging Consider other regularization (e.g. by Nagel, Weickert...) PLEASE WAKE UP! THANKS FOR YOUR ATTENTION Slide 23 of 23