TRACKING and DETECTION in COMPUTER VISION

Transcription

1 Technischen Universität München Winter Semester 2013/2014 TRACKING and DETECTION in COMPUTER VISION Template tracking methods Slobodan Ilić

2 Template based-tracking Energy-based methods The Lucas-Kanade(LK) algorithm Compositional Algorithm Inverse Compositional Algorithm (IC) Efficient Second Order Method (ESM) Learning methods Learning linear predictors(hyperplane template matching of Jurie-Dhome) 2

3 Motivation template image Follow a template image in an image sequence by estimating the warp. 3

4 Template warping 4

5 Template warping Given the template T which we want to track: Template image T (x) 4

6 Template warping Given the template T which we want to track: Take all pixels x form the template and Template image T (x) 4

7 Template warping Given the template T which we want to track: Take all pixels x form the template and Warp them using the function W (x; p) parameterized in terms of parameters p to the input image I Template image Input image warp T (x) W (x; p) 4

8 Template warping Given the template T which we want to track: Take all pixels x form the template and Warp them using the function W (x; p) parameterized in terms of parameters p to the input image I Assign the pixel intensity values of the input image at the warped location to the template image I(W (x; p)) Template image Input image warp W (x; p) T (x) 4 I(W (x; p))

9 The tracking task The goal of template-based tracking is to find the parameters p such that: p by minimizing the norm of: I(W(x; p )) = T (x) Using a prediction as an approximation of the estimation p we can reformulate the goal: Find the parameter increments p p p + p such that: This is a non-linear minimization problem. (1) 5

10 Assumptions 6

11 Assumptions No errors in the template image boundaries: only the appearance of the object to be tracked appears in the template image. 6

12 Assumptions No errors in the template image boundaries: only the appearance of the object to be tracked appears in the template image. No large occlusions: the entire template is visible in the input image. 6

13 Assumptions No errors in the template image boundaries: only the appearance of the object to be tracked appears in the template image. No large occlusions: the entire template is visible in the input image. Brightness constancy assumption: the intensity of the object appearance is always the same. 6

14 The Lucas-Kanade Uses the Gauss-Newton method for minimization: Applies a first-order approximation Minimizes iteratively The warp has to be differentiable. 7

15 The derivation of Lucas-Kanade Linearize the error function of Eq. (1) on slide 5 by using a first-order Taylor series approximation of warped source image : = And minimize the following function: (2) 8

16 The derivation of Lucas-Kanade Linearize the error function of Eq. (1) on slide 5 by using a first-order Taylor series approximation of warped source image : = And minimize the following function: (2) Image gradient evaluated at W(x; p) 8

17 The derivation of Lucas-Kanade Linearize the error function of Eq. (1) on slide 5 by using a first-order Taylor series approximation of warped source image : = And minimize the following function: (2) Image gradient evaluated at W(x; p) Jacobian of the warp 8

18 Jacobian of the warp If warp is a function: then its Jacobian is: 9

19 Typical warps Translation, like in optical-flow: Affine, like in template tracking: 10

20 LK derivation Minimizing the Eq. 2 from slide 8 is a least-square problem and has a closed form solution. The partial derivatives of Eq.2 in respect with the parameter update p is: setting equation to zero gives closed-form solution: where H is nxn (Gauss-Newton) approximation of the Hessian matrix: 11

21 The LK alg. summary Iterate until p < 12

22 Alternative approaches Lucas-Kanade approximately minimizes: with respect to pand updates the parameters is step 9 like p p + p BUT, these are not the only ways to minimize this function, thus 3 other approaches are introduced: Compositional Image Alignment Inverse Compositional Image Alignment Inverse Additive Image Alignment 13

23 Compositional Image Alignment The warp is composed of two warps: which can be written as: and the solution is sought for the warp instead of additive update of the parameters p Thus the problem to minimize becomes: (3) 14

24 Compositional Affine Warp Affine warp: Compositional affine wrap: W(W(x; p); p)) = 1+p 1 p3 p5 p2 1+p4 p6 1+ p 1 p3 p5 p2 1+ p4 p x y 1 15

25 Derivation After applying first order Taylor approximation to I(W(x; p); p)) the Eq. 3 from slide 14 becomes: where I(W(x)) denotes the warped image and in order to proceed we assume that the warp W(x; 0) =x simplifies to: is identity warp, thus the above eq. I(W(x; p)) 16

26 Compositional vs. Additive Approach Forward additive approach: Compute image gradient and warp it with W(x; p) Compositional approach: Jacobian of the warp is evaluated at (x; p) Compute image gradient of the warped image I(W) Jacobian of the warp is evaluated at (x; 0) and is constant 17

27 Compositional vs. Additive Differences The gradient of I(W(x; p)) should be used in step (3) of the algorithm from slide 12. The Jacobian of the warp can be pre-computed because it is evaluated at (x;0) rather then being recomputed at each iteration in step (4). The warp is updated in step 9 and not warp parameters. 18

28 Inverse Compositional (IC) Approach (4) W(x; p) W(x; p) T(x) Template image W(x; p) W(x; p) W(x; p) 1 Source image 19

29 Derivation After first order Taylor expansion Eq. 4 becomes: and assuming that solution is: W(x; 0) =x is the identity warp the where the Hessian of image is replaced by the one of the template: since the there is nothing in H which depends on p it is constant and can be pre-computed 20

30 IC Algorithm 21

31 ESM (Efficient Second-order Method) (1) I = T + J p=0 dp + dp T H p=0 dp [second-order Taylor expansion] (2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt dp] (3) dp T H p=0 = ½(J p=dp - J p=0 ) [from Equation (2)] (4) I = T + J p=0 + ½(J p=dp - J p=0 )dp [by injecting (3) in (1)] (5) dp = [½(J p=0 + J p=dp )] + (I - T) [from Equation (4)] Like Gauss-Newton but replace J p=0 by ½(J p=0 + J p=dp ). Need to compute J p=dp at each iteration, and a pseudo-inverse at each iteration, but need much less iterations. 22

32 ESM Convergence Gauss-Newton ESM 23

33 Learning a Linear Predictor [Jurie&Dhome PAMI02] From the IC approach we had: Where we can denote J = T W to be p Jacobian of the error function we minimize Thus we can rewrite it: p =(J T (0)J (0)) 1 J T (0) or p = A 24

34 Hyperplane approximation If for each pixel we write an error function: y i ( p) =T(W(x i ; p)) I(W(x i ; p) and the total error function is with the solution being: we can rewrite it like: p = A E = i y i ( p) 2 we see that a 11,...,a ij...a nn are coefficients of n hyperplanes that can be estimated using linear least-squares N is a number of image sample points, e.g number of pixels in the template 25 p 1 p j p N

35 Training Can we learn A and make computation even faster? A is computed offline by regression: {(, ), (, ),...,(, )} p 0 p 1 y 1 (0) y 2 (0) image differences by minimizing: By writing N t k=1 ( p k Ay k (0)) 2 y Nt (0) p Nt H =(y i (0),...,y Nt (0)) Y =( p 1,..., p Nt ) and assuming N t >N= Y [n Nt ] = A h[nxn] H [N Nt ] A h = YH T (HH T ) 1 26

36 Advantages Can "jump" over local minimums. Handle faster motion. O p 27

37 Active Shape Models For deformable objects: p = x t u t v θ s with x = u i v i (u i, v i ) (t u, t v ): 2D translation; ϑ: 2D rotation; s: scale. 28

38 Dimensionality Reduction Training set: { } x i Covariance matrix of the training data: Cov = λ 1 λ SVD of the covariance matrix: Cov = U 2 Keep only the first columns of U : U = ( R x ). x = x + R x c 29 i ( x i x )( x i x ) T λ n U T U 2 c U 1

39 First Modes x = x + R x c Can be considered as a linear warp: W(x; c) = x + R x c Can constrain with c i < 3 λ i c are parameter of the warp we are searching 30

40 Fit Function min c,t u,t v,θ,s i dist( Tr ( tu,t v,θ,s x ),y ) i i Optimum can be found with linear algebra operations. 31

41 Convergence 32

42 Active Appearance Models g = g + R g c g 33

43 Fit Function min c,t u,t v,θ,s Tr ( tu,t v,θ,s g( c) ) I 2 = min c,t u,t v,θ,s ([ Tr ( tu,t v,θ,s g( c) )] I ) i i Non-linear least-squares. Optimization can be performed with the Gauss-Newton algorithm. i 2 34

44 Convergence 35

45 Example 36