Calculus of Variations and Computer Vision Sharat Chandran Page 1 of 23 Computer Science & Engineering Department, Indian Institute of Technology, Bombay. http://www.cse.iitb.ernet.in/ sharat January 8, 2002
Page 2 of 23 The Top Ten Algorithms of the Century Metropolis Algorithm for Monte Carlo Simplex Method for Linear Programming Krylov Subspace Iteration Methods The Decompositional Approach to Matrix Computations QR Algorithm for Computing Eigenvalues The Fortran Optimizing Compiler Quicksort Algorithm for Sorting Fast Fourier Transform Integer Relation Detection Fast Multipole Method
Overview Page 3 of 23 1. Sample problems from computer vision (fade out computer vision gurus) The problem of shape The problem of obtaining 3D information The problem of motion 2. The beauty of mathematics (fade out math gurus) The method of the calculus of variations Allied mathematical methods: Singular Value Decomposition, and the Method of Lagrange Multipliers 3. Sample solution to problems posed (or how computer scientists teach/cheat) 4. Concluding remarks
Shape Page 4 of 23 When placed out of context, images can be quite involved What you get is not what you see (WYGINWYS) What you want to compute: A contour (useful for obtaining properties such as size) Key mathematical concept: parametric curve x(s), y(s) Idealized image
Motion Page 5 of 23 Optical flow is an intensitybased approximation to image motion from sequential timeordered images Key mathematical concept: Two functions: u(x, y) and v(x, y) What does it look like? How do we compute optical flow?
A short video segment Specifications to help. Application of Optical Flow Page 6 of 23
Two Problems Page 7 of 23 Determine the equation of the curve joining two points (x 1, y 1 ) and (x 2, y 2 ) on the plane such that the length of the curve joining them is minimum Goal: Prove that your answer is correct (x 1, y 1 ) (x 2, y 2 ) Similar problem: Determine the equation of the support so that a ball placed at (x 1, y 1 ) reaches (due to gravity) (x 2, y 2 ) in the least possible time
Page 8 of 23 Functions and Functionals Differential segment length is ( dx 2 + dy 2 ) 1/2 where dx and dy are infinitesimal lengths in the x and y directions along the curve. Therefore the length of the curve joining the two end points (x 1, y 1 ) and (x 2, y 2 ) is J = = x2 x 1 x2 x 1 (1 + y ) dx F (x, y, y ) dx For the first problem, what should the function y be so that J is minimized? Compare: Find x 0 such that y = f(x) is minimum. J is a functional, a quantity that depends on functions rather than dependent variables.
Solution to the Shortest Length Problem The Euler-Lagrange equation is a necessary condition for minimising J F y d dx F y = 0 (1) Page 9 of 23 In our problem, F = 1 + y. Applying Equation 1 we get Solution: y is a constant. d dx ( y 1 + y ) = 0 The shortest distance curve is a straight line.
Detour: Why the Euler equation Page 10 of 23 J = x 2 x 1 F (x, f, f ) dx with boundary conditions f(x 1 ) = f 1 and f(x 2 ) = f 2. Let η(x) be a test function. Deform f by ɛη(x). Then, dj/ dɛ = 0 at the right place. This is true for all test functions η(x). The boundary conditions assert η(x 1 ) = η(x 2 ) = 0 If f(x) is replaced by f(x) + ɛη(x) then f (x) will be replaced by f (x) + ɛη (x). The integral then becomes J = x2 x 1 F (x, f + ɛη, f + ɛη ) dx
Page 11 of 23 Detour: Why the Euler equation If F is differentiable, expand the integrand in a Taylor series Differentiate with respect to ɛ and set to zero Apply integration by parts to get x2 x 1 η (x)f f dx = [η(x)f f] x 2 x 1 x2 The first term is zero due to the boundary conditions Hence x2 η(x)(f f df f x 1 dx )dx = 0 x 1 η(x) d(f f) dx dx,
Bottom Line Euler Equations Various generalizations are possible Higher order derivatives: J = x 2 x 1 F (x, f, f, f,...)dx Integrand may depend on several functions instead of only one Finding functions that have two independent variables J = F (x, y, f, f x, f y )dxdy Page 12 of 23 The bottom line Function to optimize F (x, ux )dx F (x, ux, u xx )dx F (x, ux, v x )dx F (x, y, ux, u y )dxdy The Euler-Lagrange equations F u d F dx u x = 0 F u d F dx u x d2 F dx 2 uxx = 0 F u d F dx u x = 0 F v d F dx v x = 0 F u d F dx u x d F dy u y = 0
The Snake Formulation Page 13 of 23 Start with any old curve (a snake ). Mathematically deform it to be the desired shape (based on image data). Curve is v(s) = (x(s), y(s)). Define energy of the curve to be E(v(s)) = 1 where E snake = E int + E ext 0 E snake (v(s)) ds (2) Find the curve that minimizes Equation 2 Remember, actual image is not so clean!
Page 14 of 23 More on Formulation The energy functional is given by E = 1 0 (E int(v(s)) + E img (v(s)) + E con (v(s))) ds E int = (α(s) v s (s) 2 + β(s) v ss (s) 2 )/2 E con = k(v(s) x) 2 E img = w line E line (v(s)) + w edge E edge (v(s)) + w term E term (v(s)) Line energy is E line (v(s)) = I(x, y) Edge energy is E edge (v(s)) = I(x, y) 2
Applying Euler-Lagrange to Snakes We minimize J 1 = (α(s)x 2 s + β(s)x 2 ss)/2 + E x (s)ds = X(s, x, x, x )ds Page 15 of 23 and J 2 = (α(s)y 2 s + β(s)y 2 ss)/2 + E y (s)ds = Y (s, y, y, y )ds For X(s, x, x, x ) the Euler equation is X x d ds X x + d2 ds 2X x = 0
More on Application Page 16 of 23 X x = E s, X x = α(s)x (s), and X x = β(s)x (s) d ds X x = α x + αx and d ds (X (s)) = β x + βx and d2 (X (s)) = β x + 2β x + βx ds 2 For purpose of illustration use only two terms: E s α x αx = 0 Numerically approximating we have E x E i+1 E i at location i α (s) α i+1 α i x (s) x i+1 x i α x = (α i+1 α i )(x i+1 x i ) = (α i+1 α i )x i+1 (α i+1 α i )x i x (s) x i+1 2x i + x i 1
Page 17 of 23 Reducing Snakes To A Matrix Equation 1 1... 0 E 1 0 1... 0 E..... 2.. 0 0... 1 E n (α 2 α 1 ) (α 2 α 1 ) 0... 0 0 (α 3 α 2 ) (α 3 α 2 )... 0....... 0 0 0... (α n+1 α n ) T α 1 2 1 0... 0 0 x 1 α 2 1 2 1... 0 0 x....... 2 = 0... α n 0 0 0... 2 1 x n This can be written as Ax = B, and solved using SVD. x 1 x 2. x n
Page 18 of 23 The Method of Lagrange Multipliers Minimize a function f(x, y) subject to g(x, y) = 0. For example, find a point on the line x sin θ y cos θ + p = 0 closest to origin That is, minimize x 2 + y 2 subject to the given constraint. In MOLM, we generate a new function E = f + λg and set the partial derivatives with respect to x, y and λ to zero. In the example The solution is the obvious one 2x + λ sin θ = 0 2y + λ cos θ = 0 x = p sin θ y = +p cos θ x sin θ y cos θ + p = 0 The method can be generalized to larger number of constraints.
Page 19 of 23 Formulation for Motion Let E(x, y, t) be the intensity value at point (x, y) in the image Due to motion, let this point correspond to a point (x + δx, y + δy) at time instance t + δt Key assumption: E(x, y, t) = E(x + δx, y + δy, t + δt) Two parts to determine u(x, y) and v(x, y) Use Taylor s theorem to get E x u + E y v + E t = 0 Introduce a smoothness constraint: f(u, v, u x, v x, u y, v y ) = u x2 + u y2 + v x2 + v y 2 Using MOLM we have a functional that needs to be minimized f(u, v, u x, v x, u y, v y ) + λg(u, v, t) 2 dxdy u x 2 + u y 2 + v x 2 + v y 2 + λ(e x u + E y v + E t ) 2 dxdy
Page 20 of 23 Applying Euler-Lagrange for Optical Flow We have F u x u x F y u y = 0 F v x v x F y v y = 0 Applying these we get F u = 2λE x (E x u + E y v + E t ) F ux = 2u x, x F u x = 2u xx F uy = 2u y, And F v = 2λE x (E x v + E y v + E t ) F vx = 2v x, x F v x = 2v xx F vy = 2v y, This implies a coupled system of equations 2 u = λ(e x u + E y v + E t )E x 2 v = λ(e x u + E y v + E t )E y y F u y y F v y = 2u yy = 2v yy
Page 21 of 23 Discretizing we have Solution using Euler Lagrange u k,l, u k+1,l u k,l+1 u k 1,l u k,l 1 : u component of the velocity at points (k, l), (k + 1, l), (k, l + 1), (k 1, l) and (k, l 1) respectively. v k,l, v k+1,l v k,l+1 v k 1,l v k,l 1 : v component of the velocity at points (k, l), (k + 1, l), (k, l + 1), (k 1, l) and (k, l 1) respectively. u k,l = u k,l λe x E t 1 + λe x 2 v k,le x E t 1 + λe x 2 v k,l = v k,l λe x E t 1 + λe x 2 u k,le x E t 1 + λe x 2
Uses of Euler Lagrange Page 22 of 23 P roblem Regularization principle Contours Esnake (v(s))ds Area based Optical flow 2 [(ux + u 2 y + v 2 x + v 2 y ) + λ(e xu + E yv + i t) 2 )]dxdy Edge detection [(Sf i) 2 + λ (f xx) 2 ]dx Contour based Optical flow [(V N V N ) 2 + λ( δv δx )2 ] Surface reconstruction [(S f d 2 + λ(f xx + 2f 2 xy + f 2 yy )]dxdy Spatiotemporal approximation [(S f i) 2 + λ( f V + ft) 2 ]dxdydt Colour I y Ax 2 + λ P z 2 Shape from shading [(E R(f, g)) 2 + λ(f 2 x + f 2 y + g 2 x + g 2 y )]dxdy Stereo {[ 2 G (L(x, y) R(x + d(x, y), y))] 2 + λ( d) 2 }dxdy
Page 23 of 23 Concluding Remarks Two interesting problems have been described Defined and derived the Euler Lagrange equations Used mathematics to solve the computer vision problem