Improved Fast Gauss Transform

Transcription

1 Improved Fast Gauss Transform Changjiang Yang Department of Computer Science University of Maryland

2 Fast Gauss Transform (FGT) Originally proposed by Greengard and Strain (1991) to efficiently evaluate the weighted sum of Gaussians: G(y j )= NX i=1 q i e ky j x i k /h, FGT is an important variant of Fast Multipole Method (Greengard & Rolklin 1987, Greengard and Strain 1990). FGT similar to fast Fourier transform: j =1,...,M. FFT is for the data on regular grid; FGT is for data on arbitrary positio FFT is arithmetic based; FGT is approximate analysis based Widely used in mathematical physics, computational chemistry, mechanics. Recently applied in our group to RBF interpolation, acoustic scattering EM scattering, non uniform FFT and computer vision.

3 Fast Gauss Transform (cont d) The weighted sum of Gaussians G(y j )= NX i=1 q i e ky j x i k /h, Targets Sources is equivalent to the matrix-vector product: j =1,...,M. G(y 1 ) e k x 1 y 1 k /h e kx y 1 k /h e kx N y 1 k /h G(y ).. = e k x 1 y k /h e kx y k /h e kx N y k /h G(y M ) e kx 1 y M k /h e kx y M k /h e kx N y M k /h Direct evaluation of the sum of Gaussians requires O(N ) operations. Fast Gauss transform reduces the cost to O(N logn) operations. q 1 q.. q N

4 Fast Gauss Transform (cont d) Three key components of the FGT: The factorization or separation of variables Space subdivision scheme Error bound analysis

5 Factorization of Gaussian Factorization is one of the key parts of the FGT. Factorization breaks the entanglements between the sources and targets. The contributions of the sources are accumulated at the centers, then distributed to each target. Center Sources Targets Sources Targets Sources Targets

6 Factorization in FGT: Hermite Expansion The Gaussian kernel is factorized into Hermite functions e ky x ik /h = p 1 X n=0 where Hermite function h n (x) is defined by Exchange the order of the summations G y j 1 n! µ xi x h h n (x) =( 1) n dn dx n NX p X q i i n p X NX n n n i n h n µ y x µ e x. h µ xi x n µ yj x hn h µ xi x q i h n h h n µ yj x h +²(p), ² p, ² p a.k.a. Moment, A n

7 The FGT Algorithm Step 0: Subdivide the space into uniform boxes. Step 1: Assign sources and targets into boxes. Step : For each pair of source and target boxes, compute the interactions using one of the four possible ways: N B N F, M C M L : Direct evaluation N B N F, M C > M L : Taylor expansion N B > N F, M C M L : Hermite expansion N B > N F, M C > M L : Hermite expansion --> Taylor expansion Step 3: Loop through boxes evaluating Taylor series for boxes with more than M L targets. N B : # of sources in Box B M C : # of targets in Box C N F : cutoff of Box B M L : cutoff of Box C

8 Too Many Terms in Higher Dimensions The higher dimensional Hermite expansion is the Kronecker product of d univariate Hermite expansions. Total number of terms is O(p d ), p is the number of truncation terms. The number of operations in one factorization is O(p d ). h 0 h 0 h 0 h 1 h 0 h h 0 h 1 h h 1 h 0 h 1 h 1 h 1 h h h 0 h h 1 h h d=1 d= d=3 d>3

9 Too Many Boxes in Higher Dimensions The FGT subdivides the space into uniform boxes and assigns the source points and target points into boxes. For each box the FGT maintain a neighbor list. The number of the boxes increases exponentially with the dimensionality.

10 FGT in Higher Dimensions The FGT was originally designed to solve the problems in mathematical physics (heat equation, vortex methods, etc), where the dimension is up to 3. The higher dimensional Hermite expansion is the product of univariate Hermite expansion along each dimension. Total number of terms is O(p d ). The space subdivision scheme in the original FGT is uniform boxes. The number of boxes grows exponentially with dimension. Most boxes are empty. The exponential dependence on the dimension makes the FGT extremely inefficient in higher dimensions.

11 r x /h The Decay of Gaussian Functions The decay of the Gaussian kernel function is very fast. The effect of Gaussian kernel outside of certain range can be safely ignored. The time consuming translation operators in original FGT can be safely removed f(r x /h) E E-11.3E E- 1.6E E E

12 Improved Fast Gauss Transform Multivariate Taylor expansion Multivariate Horner s Rule Space subdivision based on k-center algorithm Error bound analysis

13 C α = α α! NX µ qi e kx i x k /h x i x α. h Multivariate Taylor Expansions The Taylor expansion of the Gaussian function: e ky j x i k /h = e ky j x k /h e kx i x k /h e (y j x ) (x i x )/h, The first two terms can be computed independently. The Taylor expansion of the last term is: e (y j x ) (x i x )/h = X α 0 α where α=(α 1, L, α d ) is multi-index. The multivariate Taylor expansion about center x * : G(y j )= X α 0 α! where coefficients C α are given by µ xi x h Cαe ky j x k /h µ y j x h α µ yj x α, h α.

14 Reduction from Taylor Expansions The idea of Taylor expansion of Gaussian kernel is first introduced in the course CMSC878R by Ramani and Nail. The number of terms in multivariate Hermite expansion is O(p d ). The number of terms in multivariate Taylor expansion ³ is p+d 1, asymptotically O(d p ), a big reduction for large d. d Number of terms Number of terms Hermite Expansion Taylor Expansion Number of terms Number of terms Hermite Expansion Taylor Expansion d Dimension d p Truncation order p

15 Improved Fast Gauss Transform Multivariate Taylor expansion Multivariate Horner s Rule Space subdivision based on k-center algorithm Error bound analysis

16 Monomial Orders Let α=(α 1, L, α n ), β=(β 1, L, β n ), then three standard monomial orders: Lexicographic order, or dictionary order: α lex β iff the leftmost nonzero entry in α - β is negative. Graded lexicographic order (Veronese map, a.k.a, polynomial embedding): α grlex β iff 1 i n α i < 1 i n β i or ( 1 i n α i = 1 i n β i and α lex β ). Graded reverse lexicographic order: α grevlex β iff 1 i n α i < 1 i n β i or ( 1 i n α i = 1 i n β i and the rightmost nonzero entry in α - β is positive). Example: Let f(x,y,z) = xy 5 z + x y 3 z 3 + x 3, then w.r.t. lex f is: f(x,y,z) = x 3 + x y 3 z 3 + xy 5 z ; w.r.t. grlex f is: f(x,y,z) = x y 3 z 3 + xy 5 z + x 3 ; w.r.t. grevlex f is:f(x,y,z) = xy 5 z + x y 3 z 3 + x 3.

17 The Horner s Rule The Horner s rule (W.G.Horner 1819) is to recursively evaluate the polynomial p(x) = a n x n + L + a 1 x + a 0 as: p(x) = ((L(a n x + a n-1 )x+l)x + a 0. It costs n multiplications and n additions, no extra storage. We evaluate the multivariate polynomial iteratively using the graded lexicographic order. It costs n multiplications and n additions, and n storage. 1 a b c a b c x 0 x 1 x=(a,b,c) a b c a ab ac b bc c x a b c a 3 a b a c ab abc ac b 3 b c bc c 3 x 3

18 An Example of Taylor Expansion Suppose x = (x 1, x, x 3 ) and y = (y 1, y, y 3 ), then stant ps ct: 9ops ps ct:9ops 1 4/3 1 x 1 x 1 x x x 1 x x 1 x e x y X α α xα y α 4 4 x 3 x 1 x 3 x 1 x 3 4 x x 1 x 4 8 α x x 3 x 1 x x 3 4 x 3 x 1 x 3 4/3 x x x 3 4 x x 3 4/3 x ps ct: 9 ops y 1 y 1 3 y y 1 y y 3 y 1 y 3 y y y 3 y 3 3 3

19 3 3 3 An Example of Taylor Expansion (Cont d) G y NX q i e kx i yk NX q i e kx ik e kyk X i i α α α xα i y α 1 e kyk X α NX α α yα i q i e kx ik x α i Constant 19 ops Direct: 9ops q i e x x x e y y y 4/3 1 x 1 x 1 x y 1 y 1 4 x x 1 x x 1 x y y 1 y 4 x 3 x 1 x 3 x 1 x 3 y 3 y 1 y 3 4 x x 1 x y 8 x x 3 x 1 x x 3 y y 3 4 x 3 x 1 x 3 y 3 4/3 x 3 4 x x 3 4 x x 3 4/3 1N ops Direct:31Nops x ops Direct:30ops

20 Improved Fast Gauss Transform Multivariate Taylor expansion Multivariate Horner s Rule Space subdivision based on k-center algorithm Error bound analysis

21 Space Subdivision Scheme The space subdivision scheme in the original FGT is uniform boxes. The number of boxes grows exponentially with the dimensionality. The desirable space subdivision should adaptively fit the density of the points. The cell should be as compact as possible. The algorithm is better to be a progressive one, that is the current space subdivision is from the previous one. Based on the above considerations, we model the task as k-center problem.

22 k-center Algorithm The k-center problem is defined to seek the best partition of a set of points into clusters (Gonzalez 1985, Hochbaum and Shmoys 1985, Feder and Greene 1988). Given a set of points and a predefined number k, k-center clustering is to find a partition S = S 1 S L S k that minimizes max 1 i k radius(s i ), where radius(s i ) is the radius of the smallest disk that covers all points in S i. The k-center problem is NP-hard but there exists a simple -approximation algorithm. Smallest circles

23 Farthest-Point Algorithm The farthest-point algorithm (a.k.a. k-center algorithm) is a -approximation of the optimal solution (Gonzales 1985). The total running time is O(kn), n is the number of points. It can be reduced to O(n log k) using a slightly more complicated algorithm (Feder and Greene 1988). 1. Initially randomly pick a point v 0 as the first center and add it to the center set C.. For i =1 tok 1do For every point v V, compute the distance from v to the current center set C = {v 0,v 1,...,v i 1 }: d i (v, C) =min c C kv ck. From the points V C find a point v i that is farthest away from the current center set C, i.e. d i (v i,c)=max v min c C kv ck. Addv i to the center set C. 3. Return the center set C = {v 0,v 1,...,v k 1 } as the solution to k-center

24 A Demo of k-center Algorithm k = 4

25 Results of k-center Algorithm The results of k-center algorithm. 40,000 points are divided into 64 clusters in 0.48 sec on a 900MHZ PIII PC.

26 More Results of k-center Algorithm The 40,000 points are on the manifolds.

27 Results of k-center Algorithm The computational complexity of k-center algorithm is O(n logk). Points are generated using uniform distribution. (Left) Number of points varies from 1000 to for 64 clusters; (Right) The number of clusters k varies from 10 to 500 for points. Time(s) Time(s) Time(s) Time (s) n x Log k log k

28 Improved Fast Gauss Transform Algorithm Step 1 Assign the N sources into K clusters using the farthest-point clustering algorithm such that the radius is less than r x. Step Choose p sufficiently large such that the error estimate is less than the desired precision ². Step 3 For each cluster S k with center c k, compute the coefficien ts: C k α = α X µ α q i e k xi ckk /h xi c k. α! h x i S k Collect the contributions from sources to centers Step 4 Repeat for each target y j, find its neighbor clusters whose centers lie within the range r y. Then the sum of Gaussians can be evaluated by the expression: Control far field cutoff error G(y j )= X ky j c k k<hρ y X α <p Control series truncation error C k αe ky j c k k /h µ yj c k h α.

29 Improved Fast Gauss Transform Multivariate Taylor expansion Multivariate Horner s Rule Space subdivision based on k-center algorithm Error bound analysis

30 Error Bound of IFGT The total error from the series truncation and the cutoff outside of the neighborhood of targets is bounded by E(y) X µ p ³ rx p ³ ry p q i + e (r y /h). p! h h Truncation error Cutoff error r x r y r x Sources Targets

31 Error Bound Analysis Increase the number of truncation terms p, the error bound will decrease. With the progress of k-center algorithm, the radius of the source points r x will decrease, until the error bound is less than a given precision. The error bound first decreases, then increases with respect to the cutoff radius r y Real max abs error Estimated error bound Real max abs error Estimated error bound Real max abs error Estimated error bo Error Error Error Error Error p rx ry

32 Experimental Result The speedup of the fast Gauss transform in 4, 6, 8, 10 dimensions (h=1.0) direct method, 4D fast method, 4D direct method, 6D fast method, 6D direct method, 8D fast method, 8D direct method, 10D fast method, 10D D 6D 8D 10D CPU time Max abs error N N

33 Programming Issues Streaming SIMD Extensions (SSE) and SSE (Intel)

34 Example of SSE/SSE Codes _asm{ mov eax, DWORD PTR [ebp+8] ;h mov edi, DWORD PTR [ebp+1] ;w mov ebx, DWORD PTR [ebp+0] ;img mov ecx, DWORD PTR [ebp+4] ;ii xor edx, edx nextcol1: movdqa xmm0, XMMWORD PTR [ebx] movdqa XMMWORD PTR [ecx], xmm0 mov esi, 1 add ebx, 16 nextrow1: paddd xmm0, XMMWORD PTR [ebx] add ecx, 16 movdqa XMMWORD PTR [ecx], xmm0 add ebx, 16 inc esi cmp esi, eax jl nextrow1 add ecx, 16 inc edx cmp edx, edi jl nextcol1 }

35 Some Applications Kernel density estimation Mean-shift based segmentation Object tracking Robot navigation

36 Kernel Density Estimation (KDE) Kernel density estimation (a.k.a Parzen method, Rosenblatt 1956, Parzen 196) is an important nonparametric technique. KDE is the keystone of many algorithms: Radial basis function networks Support vector machines Mean shift algorithm Regularized particle filter The main drawback is the quadratic computational complexity. Very slow for large dataset.

37 Kernel Density Estimation Given a set of observations {x 1, L, x n }, an estimate of density function is Kernel function Ã! nx kx xi k ˆf n (x) = 1 nh d Dimension i=1 Some commonly used kernel functions K h Bandwidth Rectangular Triangular Epanechnikov Gaussian The computational requirement for large datasets is O(N ), for N points.

38 Efficient KDE and FGT In practice, the most widely used kernel is the Gaussian K N (x)=(π) d/ e 1 kxk. The density estimate using the Gaussian kernel: ˆp n (x) =c N X N e k x x i k /h i=1 Fast Gauss transform can reduce the cost to O(N logn) in low-dimensional spaces. Improved fast Gauss transform accelerates the KDE in both lower and higher dimensions.

39 Experimental Result Image segmentation results of the mean-shift algorithm with the Gaussian kernel. Size: 43X94 Time: s Direct evaluation: more than hours Size: 481X31 Time: s Direct evaluation: more than hours

40 Some Applications Kernel density estimation Mean-shift based segmentation Object tracking Robot Navigation

41 Object Tracking Goal of object tracking: find the moving objects between consecutive frames. A model image or template is given for tracking. Usually a feature space is used, such as pixel intensity, colors, edges, etc. Usually a similarity measure is used to measure the difference between the model image and current image. Temporal correlation assumption: the change between two consecutive frames is small. Similarity function

42 d Proposed Similarity Measures Our similarity measure Directly computed from the data points. Well scaled into higher dimensions. Ready to be speeded up by fast Gauss transform, if Gaussian kernels are employed. Interpreted as the expectation of the density estimations over the model and target images.

43 Experimental results Pure translation Feature space: RGB + D spatial space Average processing time per frame: s. Our method Histogram tracking using Bhattacharyya distance

44 Experimental results Pure translation Feature space: RGB + D gradient + D spatial space Average processing time per frame: s.

45 Experimental results Pure translation Feature space: RGB + D spatial space The similarity measure is robust and accurate to the small target, compared with the other methods, such as histogram.

46 Experimental Results Translation + Scaling Feature space: RGB + D spatial space

47 Conclusions Improved fast Gauss transform is presented for higher dimensional spaces. Kernel density estimation is accelerated by the IFGT. A similarity measure in joint feature-spatial space is proposed for robust object tracking. With different motion models, we derived several tracking algorithms. The improved fast Gauss transform is applied to speed up the tracking algorithms to real-time performance.