Fast Multipole Methods: Fundamentals & Applications. Ramani Duraiswami Nail A. Gumerov

Transcription

1 Fast Multipole Methods: Fundamentals & Applications Ramani Duraiswami Nail A. Gumerov

2 Week 1. Introduction. What are multipole methods and what is this course about. Problems from physics, mathematics, computer vision, statistics, etc. that can be solved with fast multipole methods. Historical review of the fast multipole method. Review of Complexity Homework.

3 Course Mechanics Background Needed Linear Algebra Matrices, Vectors, Linear Systems, LU Decomposition, Eigenvalue problem, SVD Numerical Analysis (Interpolation, Approximation, Finite Differences, Taylor Series, etc.) Programming Matlab, C/C++ and/or FORTRAN For particular applications in your domain you will need to know the background material there E.g., familiarity with partial differential equations or integral equations, or interpolation, or Participation essential!

4 Ideally Course Requirements you are an applied student in CS, engineering, physics or industry, who has a problem in mind and would like to explore how the FMM could be applied to it or, you are Applied Math/Scientific Computation student who wants to learn this important algorithm At the end of the course you will have a familiarity with Application of FMM to different problems Data structures for FMM, and analysis related to it. Course focus is on applying the methods to achieve solution. Theory will be developed as needed to proceed

5 Homework Will try to have it every week Account for 35 % of your grade Will not be excessive Essential for learning --- must do as opposed to just read. Homework handed out last class of a week. Due last class of next week Thanksgiving week no homework (you can turn in the previous homework after thanksgiving)

6 Projects & Exams There will be a final project that will require you to implement an FMM algorithm in a field of your choice, account for 25% of the grade. Project to be chosen latest by October 17. Implementation (reimplementation) of an FMM algorithm Use an existing FMM algorithm to solve an applied or mathematical problem you can discuss these with us, and we will help you choose one, and over the project If you already have a project in mind you can discuss it with us Exams intermediate exam worth 15%, week of October 16 final exam worth 25%. Finals Week

7 Class web & Mailing List Discussions, announcements Homework will be posted on the web No late homework (except by timely prior arrangement) Hardcopy (no web or submissions) Solutions will be posted after homework collected Links to papers etc.

8 Introductions addresses What are your interests? What do you want us to cover? Last three years outline is posted at the course web site

9 Scientific Computing Do science on the computer Accepted paradigm in many fields Computational physics Computational chemistry Computational biology Computational fluid dynamics Computational solid mechanics Stellar dynamics Molecular Modeling Machine Learning Data Mining Meteorology Computational Statistics Computational Electromagnetics Medical Imaging & Tomography Computational Acoustics Geology and Ocean Engineering Circuit design Photonics Nanotechnology Model the problem as best as we can Then represent the physical system discretely

10 Problem Size As problem size grows, in general fidelity of representation grows On the other hand cost of solving the problem Memory required to store the discrete representation also grow Way the cost creases with size is the memory or run time complexity of an algorithm Need a language to talk about fast and low memory algorithms Asymptotic complexity

11 Asymptotic Equivalence Let n the problem size Two algorithms with costs f(n) and g(n) are said to be equivalent: f(n) ~ g(n) f( n) lim = n g( n) 1

12 Little Oh Asymptotically smaller: f(n) = o(g(n)) f( n) lim = n g( n) 0

13 Big Oh Asymptotic Order of Growth: f(n) = O(g(n)) limsup f( n) n g( n) <

14 The Oh s If f = o(g) or f ~ g then f = O(g) lim = 0 lim = 1 lim <

15 The Oh s If f = o(g), then g O(f) f g lim = 0 lim = g f

16 Big Oh Equivalently, f(n) = O(g(n)) c,n 0 0 n n 0 f(n) c g(n)

17 Big Oh f(x) = O(g(x)) log scale blue stays below red c g(x) ln c f(x)

18 Complexity we will specify the function g(n) The most common complexities are O(1) - not proportional to any variable number, i.e. a fixed/constant amount of time O(N) - proportional to the size of N (this includes a loop to N and loops to constant multiples of N such as 0.5N, 2N, 2000N - no matter what that is, if you double N you expect (on average) the program to take twice as long) O(N 2 ) - proportional to N squared (you double N, you expect it to take four times longer - usually two nested loops both dependent on N). O(log N) - this is tricker to show - usually the result of binary splitting. O(N log N) this is usually caused by doing log N splits but also doing N amount of work at each "layer" of splitting.

19 Theta Same Order of Growth: f(n) = Θ(g(n)) f(n) = O(g(n)) and g(n) = O(f(n))

20 Log complexity If you half data at each stage then number of stages until you have a single item is given (roughly) by log 2 N. => binary search takes log 2 N time to find an item. All logs grow a constant amount apart (homework) So we normally just say log N not log 2 N. Log N grows very slowly

21 What is the Fast Multipole Method? An algorithm for achieving fast products of particular dense matrices with vectors Similar to the Fast Fourier Transform For the FFT, matrix entries are uniformly sampled complex exponentials For FMM, matrix entries are Derived from particular functions Functions satisfy known translation theorems Name is a bit unfortunate What is a multipole? We will return to this Why is this important?

22 Vectors and Matrices d dimensional column vector x and its transpose n d dimensional matrix M and its transpose M t

23 Matrix vector product s 1 = m 11 x 1 + m 12 x m 1d x d s 2 = m 21 x 1 + m 22 x m 2d x d s n = m n1 x 1 + m n2 x m nd x d Matrix vector product is identical to a sum s i = j=1d m ij x j So algorithm for fast matrix vector products is also a fast summation algorithm d products and sums per line N lines Total Nd products and Nd sums to calculate N entries If d N then matrix multiplication cost is O(N 2 ) Storage of the matrix entries is also O(N 2 )

24 Linear Systems Solve a system of equations Mx=s M is a N N matrix, x is a N vector, s is a N vector Direct solution (Gauss elimination, LU Decomposition, SVD, ) all need O(N 3 ) operations Iterative methods typically converge in k steps with each step needing a matrix vector multiply O(N 2 ) if properly designed, k<< N A fast matrix vector multiplication algorithm (O(N log N) operations) will speed all these algorithms

25 Is this important? Argument: Moore s law: Processor speed doubles every 18 months If we wait long enough the computer will get fast enough and let my inefficient algorithm tackle the problem Is this true? Yes for algorithms with same asymptotic complexity No!! For algorithms with different asymptotic complexity For a million variables, we would need about 16 generations of Moore s law before a O(N 2 ) algorithm was comparable with a O(N) algorithm Similarly, clever problem formulation can also achieve large savings.

26 Memory complexity Sometimes we are not able to fit a problem in available memory Don t care how long solution takes, just if we can solve it To store a N N matrix we need N 2 locations 1 GB RAM = =1,073,741,824 bytes => largest N is 32,768 Out of core algorithms copy partial results to disk, and keep only necessary part of the matrix in memory FMM allows reduction of memory complexity as well Elements of the matrix required for the product can be generated as needed

27 Parallel Processing Another way to tackle large problems is to throw more hardware (or more sophisticated hardware) at the problem If the work can be distributed across several processors (say P of them), then it can reduce the cost by some factor (ideally P). Usually requires reformulation of the solution algorithm (unless it is trivially parallel) Was hot in the 80s. Interest in this area again as purpose built parallel processing engines such as Graphical Processing Units, Multicore processors, FPGAs, etc. become widely available

28 The need for fast algorithms Grand challenge problems in large numbers of variables Simulation of physical systems Electromagnetics of complex systems Stellar clusters Protein folding Turbulence Learning theory Kernel methods Support Vector Machines Graphics and Vision Light scattering

29 General problems in these areas can be posed in terms of millions (10 6 ) or billions (10 9 ) of variables Recall Avogadro s numer ( molecules/mole

30 Dense and Sparse matrices Operation estimates are for dense matrices. Majority of elements of the matrix are non-zero However in many applications matrices are sparse A sparse matrix (or vector, or array) is one in which most of the elements are zero. If storage space is more important than access speed, it may be preferable to store a sparse matrix as a list of (index, value) pairs. For a given sparsity structure it may be possible to define a fast matrix-vector product/linear system algorithm

31 Can compute a x 1 a 1 x 1 0 a x 2 a 2 x a3 0 0 x3 = a3x a 4 0 x 4 a 4 x a 5 x 5 a 5 x 5 In 5 operations instead of 25 operations Sparse matrices are not our concern here

32 Structured matrices Fast algorithms have been found for many dense matrices Typically the matrices have some structure Definition: A dense matrix of order N N is called structured if its entries depend on only O(N) parameters. Most famous example the fast Fourier transform

33 Fourier Analysis Def.: mathematical techniques for breaking up a signal into its components (sinusoids) Jean Baptiste Joseph Fourier ( ) can represent any continuous periodic signal as a sum of sinusoidal waves = + +

34 Notation For f, periodic with period p Fourier transform f(t) -> F[k] F[k] = 1/p 0 p f (t) e -2 π i k t/p d t = 1/p 0 p f (t) cos(2 π k t/p) dt i / p 0 p f (t) sin(2 π k t/p) dt Inverse Fourier transform F[k] -> f(t) f(t) = Σ k in Z F[k] e 2 π i k t/p

35 DFT N-1 Σ p f (t) e -2 π i k t/p φ[n] e d t 0 n =0-2 π i k nh /p f(t) φ[n] = f(n h) h

36 DFT and its inverse for periodic discrete data Φ[k] = N-1 Σ n =0 φ[n] e -2 π i k n h /p p = N h N-1 = Σ n=0 φ[n] e -2 π i k n / N This is automatically periodic in k with period N Inverse is like Fourier series, but with only p terms

37 Discrete time Numerical Fourier Analysis DFT is really just a matrix multiplication! F [m] = 1/Ν F[0] F[1] F[2]... F[N-1] = Freq. index N-1 Σ k =0 e -2 π i k m/n f[k] time index f[0] f[1] f[2]... f[n-1] F = F f N

38 Fourier Matrices

39 i w 4 = Primitive Roots of Unity Example: The complex number e 2pi/n is a primitive n-th root of unity, where w 3 1 A number ω is a primitive n-th root of unity, for n>1, if ω n = 1 The numbers 1, ω, ω 2,, ω n-1 are all distinct 2πi w 2 1 n Imaginary 45 o cos(π/4) w 1 =cos(π/4) + i w 8 Real Check: if properties are satisfied 1. ω = e 1 n 2 i n π n 2πi 2. ω = e = e = cos2π + isin 2π = 1 n 1 jp 0 j 3. S= ω = ω + ω 2 j 3 j + ω + ω j( n 1) ω = 0 p= 0 w 5 w 6 w 7 n complex roots of unity equally spaced around the circle of unit radius centered at the origin of the complex plane.

40 Roots of Unity: Properties Property 1: Let ω be the principal n th root of unity. If n > 0, then ω n/2 = -1. Proof: ω = e 2π i / n ω n/2 = e π i = -1. (Euler's formula) Reflective Property: Corollary: ω k+n/2 = -ω k. Property 2: Let n > 0 be even, and let ω and ν be the principal n th and (n/2) th roots of unity. Then (ω k ) 2 = ν k. Proof:(ω k ) 2 = e (2k)2π i / n = e (k) 2π i / (n / 2) = ν k. Reduction Property: If ω is a primitive (2n)-th root of unity, then ω 2 is a primitive n-th root of unity.

41 L3: Let n > 0 be even. Then, the squares of the n complex n th roots of unity are the n/2 complex (n/2) th roots of unity. Proof: If we square all of the n th roots of unity, then each (n/2) th root is obtained exactly twice since: L1 ω k + n / 2 = - ω k thus, (ω k + n / 2 ) 2 = (ω k ) 2 L2 both of these = ν k ω k + n / 2 and ω k have the same square Inverse Property: If ω is a primitive root of unity, then ω -1 =ω n-1 Proof: ωω n-1 =ω n =1

42 Fast Fourier Transform Presented by Cooley and Tukey in 1965, but invented several times, including by Gauss (1809) and Danielson & Lanczos (1948) Danielson Lanczos lemma

43 So far we have seen what happens on the right hand side How about the left hand side? When we split the sums in two we have two sets of sums with N/2 quantities for N points. So the complexity is N 2 /2 + N 2 /2 =N 2 So there is no improvement Need to reduce the sums on the right hand side as well We need to reduce the number of sums computed from 2N to a lower number Notice that the values corresponding to k and k+n/2 will be the same. The transforms F ek and F ok are periodic in k with length N/2. So we need only compute half of them!

44 FFT So DFT of order N can be expressed as sum of two DFTs of order N/2 evaluated at N/2 points Does this improve the complexity? Yes (N/2) 2 +(N/2) 2 =N 2 /2< N 2 But we are not done. Can apply the lemma recursively Finally we have a set of one point transforms One point transform is identity

45 FFT Algorithm FFT (n, a 0, a 1, a 2,..., a n-1 ) if (n == 1) // n is a power of 2 return a 0 ω e 2π i / n (e 0,e 1,e 2,...,e n/2-1 ) FFT(n/2, a 0,a 2,a 4,...,a n-2 ) (d 0,d 1,d 2,...,d n/2-1 ) FFT(n/2, a 1,a 3,a 5,...,a n-1 ) for k = 0 to n/2-1 y k e k + ω k d k y k+n/2 e k - ω k d k O(n) complex multiplies if we pre-compute ω k. return (y 0,y 1,y 2,...,y n-1 ) T ( n) = 2T ( n / 2) + O( n) T ( n) = O( nlogn)

46 Complexity Each F k is a sum of log 2 N transforms and (factors) There are N F k s So the algorithm is O(N log 2 N) This is a recursive algorithm

47 function y = ffttx(x) %FFTTX Textbook Fast Finite Fourier Transform. % FFTTX(X) computes the same finite Fourier transform as FFT(X). % The code uses a recursive divide and conquer algorithm for % even order and matrix-vector multiplication for odd order. % If length(x) is m*p where m is odd and p is a power of 2, the % computational complexity of this approach is O(m^2)*O(p*log2(p)). x = x(:); n = length(x); omega = exp(-2*pi*i/n); FFTtx if rem(n,2) == 0 % Recursive divide and conquer k = (0:n/2-1)'; w = omega.^ k; u = ffttx(x(1:2:n-1)); v = w.*ffttx(x(2:2:n)); y = [u+v; u-v]; else % The Fourier matrix. j = 0:n-1; k = j'; F = omega.^ (k*j); y = F*x; end

48 Scrambled Output of the FFT a 0, a 1, a 2, a 3, a 4, a 5, a 6, a 7 a 0, a 2, a 4, a 6 a 1, a 3, a 5, a 7 a 0, a 4 a 2, a 6 a 1, a 5 a 3, a 7 a 0 a 4 a 2 a 6 a 1 a 5 a 3 a 7 "bit-reversed" order

49 FFT and IFFT The FFT has revolutionized many applications by reducing the complexity by a factor of almost n Can relate many other matrices to the Fourier Matrix

50 Circulant Matrices Toeplitz Matrices Hankel Matrices Vandermonde Matrices

51 Modern signal processing very strongly based on the FFT One of the defining inventions of the 20 th century

52 Fast Multipole Methods (FMM) Introduced by Rokhlin & Greengard in 1987 Called one of the 10 most significant advances in computing of the 20 th century Speeds up matrix-vector products (sums) of a particular type Above sum requires O(MN) operations. For a given precision ε the FMM achieves the evaluation in O(M+N) operations.

53 Can accelerate matrix vector products Convert O(N 2 ) to O(N log N) However, can also accelerate linear system solution Use iterative methods that take k steps and use a matrix vector product at each step Convert O(N 3 ) to O(kN log N)

54 A very simple algorithm Not FMM, but has some key ideas Consider S(x i )= j=1n α j (x i y j ) 2 i=1,,m Naïve way to evaluate the sum will require MN operations Instead can write the sum as S(x i )=( j=1n α j )x i2 + ( j=1n α j y j2 ) -2x i ( j=1n α j y j ) Can evaluate each bracketed sum over j and evaluate an expression of the type S(x i )=β x i2 + γ -2x i δ Requires O(M+N) operations Key idea use of analytical manipulation of series to achieve faster summation

55 Approximate evaluation FMM introduces another key idea or philosophy In scientific computing we almost never seek exact answers At best, exact means to machine precision So instead of solving the problem we can solve a nearby problem that gives almost the same answer If this nearby problem is much easier to solve, and we can bound the error analytically we are done. In the case of the FMM Manipulate series to achieve approximate evaluation Use analytical expression to bound the error FFT is exact FMM can be arbitrarily accurate

56 Some FMM algorithms Molecular and stellar dynamics Computation of force fields and dynamics Interpolation with Radial Basis Functions Solution of acoustical scattering problems Helmholtz Equation Electromagnetic Wave scattering Maxwell s equations Fluid Mechanics: Potential flow, vortex flow Laplace/Poisson equations Fast nonuniform Fourier transform

57 Applications I Interpolation Given a scattered data set with points and values {x i, f i } Build a representation of the function f(x) That satisfies f(x i )=f i Can be evaluated at new points One approach use radial-basis functions f(x)= in α i R(x-x i ) + p(x) f j = in α i R(x j -x i ) + p(x j ) Two problems Determining α i Knowing α i determine the product at many new points x j Both can be solved via FMM (Cherrie et al, 2001; Gumerov & Duraiswami, 2006)

58 RBF interpolation Applications 2 Cherrie et al 2001 Original unstructured data Gumerov & Duraiswami, 2006 RBF interpolation to regular spatial grid

59 Applications 3 Sound scattering off rooms and bodies Need to know the scattering properties of the head and body (our interest) 2 2 P+ k P = py ( ) Gxyk (, ; ) ( ) ( ) = (, ; ) ( ) Γ ( ) y G xy, Cxpx Gxyk py d ny ny Γy 0 P + i σ P = g n P lim r ikp = 0 r r ik e x y = 4π x y

60 BEM/FMM computations of acoustic scattering from complex shape objects Incident plane wave exp(ikz) Mesh: elements vertices db Hz (kd=0.96) 2.5 khz (kd=9.6) 25 khz (kd=96) -9

61 Sound Pressure kd=0.96

62 Sound Pressure kd=9.6

63 Sound Pressure kd=96

64 Potential 125 randomly oriented ellipsoids vertices and elements (1016 vertices and 2028 elements per ellipsoid)

65 1000 randomly oriented ellipsoids vertices and elements (488 vertices and 972 elements per ellipsoid) Potential

66

67 EM wave scattering Similar to acoustic scattering Send waves and measure scattered waves Attempt to figure out object from the measured waves Need to know Radar cross-section Many applications Light scattering Radar Antenna design. Darve 2001

68 Molecular and stellar dynamics Many particles distributed in space Particles exert a force on each other Simplest case force obeys an inverse-square law (gravity, coulombic interaction) d x dt = ( x x ) 2 N i i j = F, 2 i Fi = qq i j 3 j 1 xi x j j i

69 Fluid mechanics Incompressible Navier Stokes Equation u = 0 u ρ + t + 2 ( u ) u p = µ u u = φ + A Laplace equation for potential and Poisson equation for vorticity Solved via particle methods

70 Vortex Methods

71 Applications Digital Signal Processing: analyzing and manipulating real-world signals using a computer Toys and consumer electronics Speech recognition Audio/video compression Medical imaging Communications Radar