Classification of Optical Flow by Constraints

Transcription

1 Classification of Optical Flow by Constraints Yusuke Kameda 1 and Atsushi Imiya 2 1 School of Science and Technology, Chiba University, Japan 2 Institute of Media and Information Technology, Chiba University, Japan Abstract In this paper, we analyse mathematical properties of spatial optical-flow computation algorithm First by numerical analysis, we derive the convergence property on variational optical-flow computation method used for cardiac motion detection From the convergence property of the algorithm, we clarify the condition for the scheduling of the regularisation parameters This condition shows that for the accurate and stable computation with scheduling the regularisation coefficients, we are required to control the sampling interval for numerical computation 1 Introduction In this paper, we analyse mathematical properties of three-dimensional opticalflow computation algorithm, since three-dimensional optical flow is a fundamental method for the non-invasive cardiac motion analysis [1,2] Furthermore, classification and separation of the regions on the heart wall using optical flow derives cardiac diagnosis features [10] Optical flow is a well established method in computer vision [3,4,5,6,7] For variational cardiac optical-flow computation, additional consistencies and constraints to image motion-analysis are used [1] The gradient consistency, and thin plate deformation constraint and the divergence-free constraint are typical additional consistency and constraints, respectively The gradient consistency is used for enhancement of the region boundary The deformable constraint, which is equivalent to thin plate deformation constraint [8] is used for physical model of heart-wall deformation The divergence-free constraint is based on the masspreservation requirement for physical objects Therefore, for accurate and stable computation, the scheduling of the regularisation parameters [9] is a fundamental problem There are two types of evaluation method for inverse problems for non-invasive diagnosis First one is analysis the accuracy of the solution using normalised phantoms, that is, evaluate the difference between phantom, which is used the ground truth, and the solution derived by the algorithm The second one is mathematics-based evaluation, that is, clarification of the convergence and stability of the algorithm employing numerical analysis In this paper, from the view point of mathematical-based evaluation, we first derive the convergence property on variational optical-flow computation method used for cardiac motion detection Secondly, we prove that the divergence-free constraint is dependent to the first order smoothness constraint, which is called the Horn-Schunck regulariser [4] and the vector spline constraint [16,13,14,15] is WG Kropatsch, M Kampel, and A Hanbury (Eds): CAIP 2007, LNCS 4673, pp 61 68, 2007 c Springer-Verlag Berlin Heidelberg 2007

2 62 Y Kameda and A Imiya dependent to the second order smoothness constraint, which is equivalent to thin plate deformation constraint [11,12] Thirdly, from the convergence property of the algorithm, we clarify the condition for the scheduling of the regularisation parameters This condition shows that for the accurate and stable computation with scheduling the regularisation coefficients, we are required to control the sampling interval for numerical computation Furthermore, using the first and second order derivatives, we show that it is possible to classify the motions of points in an image using the orders of differential- constraints [10] 2 Vector Spline Constraints For matching of the deformable boundary between a pair of images [11], the minimisation of J TP (u, v) = (f(x, y) f(x u, y v)) 2 dx R 2 {( +λ u 2 xx +2u 2 xy + ) ( u2 yy + v 2 xx +2vxy 2 + )} v2 yy dxdy (1) R 2 produces a promising result, assuming that each element of the vector (u, v) is four-times differentiable For the interpolation of the vector field on a plane, the solutions u which minimise J 1st (u) = m,n i=1,j=1 for twice differentiable function u, and J 2nd (u) = m,n i=1,j=1 u(x ij ) x ij 2 ( + γ1 divu 2 + γ 2 rotu 2) dx (2) R 2 u(x ij ) x ij 2 ( + γ1 divu 2 + γ 2 rotu 2) dx, (3) R 2 for four-times differentiable function u, respectively, where {x ij } m,n i=1,j=1, are called the first-order vector-spline and the second-order vector-spline, respectively [14,15] Here, the gradient operation in the second term of the regulariser in eq (3) is computed as the vector gradient for the higher dimensional problems Equations (1) and (3) are typical minimisation problems with the second order constraints which are used in computer-vision problems 3 Optical Flow Computation For a spatio-temporal image f(x, y, z, t), the total derivative is given as d dt f = f dx x dt + f dy y dt + f dz z dt + f dt t dt (4)

3 Classification of Optical Flow by Constraints 63 where u =(ẋ, ẏ, ż) =( dx dt, dz dt ) is the motion of each point x =(x, y, z) Optical flow consistency [4,5,6] d f =0 (5) dt implies that the motion of the point u =(u, v, w) =(ẋ, ẏ, ż) is the solution of the singular equation, R 3 dt, dy f x u + f y v + f z w + f t =0 (6) The optical-flow computation with the second order constraint is { } J αβ (u) = f u + f t 2 + αtr u u + βtrhuh u dx, (7) for trhuh u = trh uh u + trh vh v + trh wh w, (8) where H f is the Hessian matrix of the function f The Euler-Lagrange equation of the variational problem is Δu β α Δ2 u = 1 α ( f u + f t ) (9) Let (x 1,x 2,x 3 ) =(x, y, z) Setting shu = n (a i )2 +(a + i )2 (10) where a i = x i u i i=1 u (i+1),a + i = u i + x i+1 x i for u 4 = u 1 and x 4 = x 1, there is a relation, for x i+1 u (i+1), (11) tr( u u )= 1 2 (div2 u + rotu 2 + sh 2 u) (12) Furthermore, for trhh,wehavetherelation where trhh = divu 2 + rotu 2 + h 2 (u)+k 2 (u) (13) h 2 (u) = v xy u xy Δ xy v Δ xy u + w xy v xy Δ yz w Δ yz v + u xy w xy Δ zx Δ zx w (14) ( k 2 v (u) =divs, s = y w y v z w z, w z u z w x u x u x v x u y v y ) (15) Δ αβ = 2 α β 2, for α, β {xy,yz,zy} These relations imply the next properties Δ αβ = 2 α 2 2 β 2 (16)

4 64 Y Kameda and A Imiya Assertion 1 The first order smoothness constraint tr u u involves the divergence-free constraint divu Therefore, the first order smoothness constraint involves the divergence-free condition Assertion 2 The second order minimum-deformation constraint trhuh u involves the second order vecror spline constraint γ 1 divu 2 + γ 2 rotu 2 Therefore, considering the regularisation term trhuh u is equivalent to solve vector spline minimisation [12,13,14,15,16] for optical-flow computation 4 Numerical Scheme and Its Stability In this paper, by embedding images in a rectangular region which encircles images, we adopt Dirichlet condition f = 0 on the boundary Furthermore, for the sampled optical flow vector u ijk =(u ijk,v ijk,w ijk ), we set the vectorisation of sampled function as u := u 111 u 112 u MMM, u ijk = u ijk v ijk w ijk Then, we have the discrete version of Euler-Lagrange equation (17) Lu β α L2 u = 1 α (Su + s), s = f f f (18) where L is the discrete Laplacian operation In eqs (9) and (18), the operation to u are split to both side of equation Therefore, both operations are non-singular and spectra of one operation is less than one and the other is larger than one It is possible to derive a converging numerical scheme However, the operation S is singular In this section, we derive a converging iteration scheme from the equation u + Lu β α L2 u = u + 1 α Su + 1 α s, S = f f t op (19) since operations in both side of the equation are non-singular The second order numerical derivative is u(i + h) 2u(i)+u(i h) 2 u = h 2 (20) Setting D 2 to be the second order discrete differential operator derived by eq (20), that is, D 2 = , (21)

5 Classification of Optical Flow by Constraints 65 for Dirichlet boundary condition, the fourth order operator is D 4 = D 2 D 2 Furthermore, the numerical operations L and L 2 which stand for Δ and Δ 2,in R 3, respectively, are given as L = D 2 I I + I D 2 I + I I D 2, (22) L 2 = D 4 I I + I D 4 I + I I D 4 +2(D 2 D 2 I + I D 2 D 2 + D 2 I D 2 ), (23) where A B is the Kronecker product of matrices A and B, andi is the identity matrix For an appropriate positive number Δτ, we have a splitting expression, [ ( Diag I + Δτ 1 )] [{ α S ijk u = P Diag I + Δτ (L βα )}] L2 Pu Δτ α s, (24) for permutation matrix which transforms u of eq (17) to From this, we have the iterative form u 111 u 112 v := vec = Pu (25) u MMM Au n+1 = Bu n + c (26) for ( A = Diag I + Δτ ) α S ijk, S = f(i, j, k) f(i, j, k) (27) ( B = P I + ΔτL Δτ β ) α L2 P, c = Δτ α s, s = f t f(i, j, k) (28) Both A and B are non-singular and ρ(a) > 1, where ρ(a) isthemaximumof the absolute value of the eigenvalues of the matrix A Therefore, if ρ(b) < 1 then the iterative form eq (26) is stable and converges to the solution of the original equation We derive the condition for α, β, and Δτ to guarantee the condition ρ(b) < 1 Setting U and Λ to be the discrete Fourier transform matrix and the diagonal matrix respectively, we have the relations where L = UΛU, L 2 = UΛ 2 U (29) Λ = Diag (λ ijk ), λ ijk = μ i + μ j + μ k, μ k = ( ) π 1 cos M +1 k, (30)

6 66 Y Kameda and A Imiya Substituting these relations to ρ (I + ΔτL Δτ βα ) L2 < 1 (31) finally, we have the relation ρ (I + ΔτΛ Δτ βα ) Λ2 < Δτ h 2 16 β Δτ 32 α h 4 (32) From eq (32), we have the next theorem Theorem 1 If the inequality Δτ h 2 16 β Δτ 32 α h 4 < 1 (33) is satisfied, the iterative form is stable and converges to the solution of the original PDE The iteration is terminated if max x R un+1 (x) u n (x) < u n (x) 10 m, (34) 3 where u is the Euclidean norm in R 3 5 Numerical Examples Let u (αβγ) be the optical flow vector computed using α-th, β-th, and γ-th constraints For each point x, we can have many optical-flow vectors u (α),where α is a string of positive integers, for example, 1, 2, 12, 123, and so on The vector u (2) is the deformation vector of the deformable boundary Therefore, if u 2 is sufficiently small at point x, andu (1) u (2) themotioninthe neighbourhood of this point x is homogeneous This geometrical property of optical-flow computed by the variational method implies that the operation D (i) = {x u (i) u (j),i j} (35) derives segments in R n using the orders of constraints for the computation of variational problems Generally, it is possible to adopt many constraints for the classification of moving points in a space with optical-flow vectors For example, two operations D h = {x u (1) u (12) }, D d = {x u (12) u (1) } (36) classify a scene into a deforming part and a homogeneously moving part, since the regularisers tr u u and trd 2 ud 2 u minimise smoothness and elastic energy of the solution u, respectively Furthermore, the operation u = { u (1), if u 12 1onx, u (12), otherwise, (37)

7 Classification of Optical Flow by Constraints 67 z z z y x (a) Optical flow for 10 5, 0 for frames 2 and 3from20frames 50 y (b) Optical flow for 10 5, 10 5 for frames 2 and 3 from 20 frames x y (c) Optical flow for 10 5, 10 4 for frames 2 and 3 from 20 frames x Fig 1 Optical Flow Classification in Three-Dimensional Euclidean Space (a), (b), and (c), are the optical-flow field images of the beating heart for the regularisation parameters α, β are 10 5, 0, 10 5, 10 5,and 10 5, 10 4, respectively The Horn-Schunck constraint extracts the smooth motion u (1) of the whole heart The second order constraint extracts the deformbale motion u (12) The combination of the Horn-Schunck constraint and deformable constraint allows us to separate and classify the motions of points in images to smooth motion and deformable motion allows us to express multi-modal motion simultaneously, using variational computation methods These properties imply that the orders of the constraints in variational problems act as the scale of scale space analysis Figure 1 shows cardiac optical flow u (1) and u (12) extracted for some combinations of α, β Here α, β are 10 5, 0, 10 5, 10 5,and 10 5, 10 4 InFigure1, (a), (b), and (c) are the optical-flow field images of the beating heart for the regularisation parameters α, β are 0 5, 0, 10 5, 10 5,and 10 5, 10 4, respectively These sequence show that, as expected, the flow sequence obtained by the Horn- Schunck constraint extracts smooth motion of the whole heart Furthermore, with the second order constraint, the deformbale motion on the wall of beating heart is extracted The extracted dominate deformable part is the atriums These results indicate that the combination of the Horn-Schunck constraint and deformable constraint allows us to separate and classify the motions of points in images to the smooth motion u (1) and the deformable motion u (12) 6 Conclusions In this paper, we first derived the convergence property on variational opticalflow computation method used for volumetric cardiac motion detection From this convergence property of the algorithm, we clarified the condition for the scheduling of the regularisation parameters Images for our experiment are provided from Roberts Research Institute at the University of Western Ontario through Professor John Barron We express thanks to Professor John Barron for allow us to use his data set The research was supported by No , Grant-in-Aid for Scientific Research of Japan Society for the Promotion of Sciences

8 68 Y Kameda and A Imiya References 1 Zhou, Z, Synolakis, CE, Leahy, RM, Song, SM: Calculation of 3D internal displacement fields from 3D X-ray computer tomographic images In: Proceedings of Royal Society: Mathematical and Physical Sciences, vol 449, pp (1995) 2 Song, SM, Leahy, RM: Computation of 3-D velocity fields from 3-D cine images of a human heart IEEE Transactions on Medical Imaging 10, (1991) 3 Aubert, G, Kornprobst, P: Mathematical Problems in Image Processing:Partial Differential Equations and the Calculus of Variations Springer, NewYork (2002) 4 Horn, BKP, Schunck, BG: Determining optical flow Artificial Intelligence 17, (1981) 5 Nagel, H-H: On the estimation of optical flow: Relations between different approaches and some new results Artificial Intelligence 33, (1987) 6 Barron, JL, Fleet, DJ, Beauchemin, SS: Performance of optical flow techniques International Journal of Computer Vision 12, (1994) 7 Weickert, J, Schnörr, C: Variational optic flow computation with a spatiotemporal smoothness constraint Journal of Mathematical Imaging and Vision 14, (2001) 8 Timoshenko, SP: History of Strength of Materials Dover, Mineola, NY (1983) 9 Chang, H-H: Variational approach to cardiac motion estimation for small animals in tagged magnetic resonance imaging, 2006 IEEE Pacific-Rim Image and Video Technology, (2006) 10 Chang, H-H, Moura, JMF, Yijen, L, Wu, YL, Ho, C: Early detection of rejection in cardiac MRI: A spectral graph approach, 2006 IEEE International Symposium on Biomedical Imaging, Arlington, (2006) 11 Grenander, U, Miller, M: Computational anatomy: An emerging discipline Quarterly of applied mathematics 4, (1998) 12 Sorzano, CÓS, Thévenaz, P, Unser, M: Elastic registration of biological images using vector-spline regularization IEEE Tr Biomedical Engineering 52, (2005) 13 Wahba, G, Wendelberger, J: Some new mathematical methods for variational objective analysis using cross-validation Monthly Weather Review 108, (1980) 14 Amodei, L, Benbourhim, MN: A vector spline approximation Journal of Approximation Theory 67, (1991) 15 Benbourhim, MN, Bouhamidi, A: Approximation of vectors fields by thin plate splines with tension Journal of Approximation Theory 136, (2005) 16 Suter, D: Motion estimation and vector spline In: Proceedings of CVPR 94, pp (1994)