Chapter 6: Fast Fourier Transform and Applications

Chapter 6: Fast Fourier Transform and Applications Michael Hanke Mathematical Models, Analysis and Simulation, Part I Read: Strang, Ch. 4. Fourier Sine Series In the following, every function f : [,π] C will be identified with the periodic continuation onto R. A function f : [,π] C is called odd if f(x) f( x) for all x [π,π]. A function f is called even if f(x) f( x) for all x [π,π]. If f is even, f is odd. Similarly, if f is odd, f is even. Most important odd functions: sin(nx). Fourier sine series: S(x) b n sinnx Q: Which functions f can be represented by a sine series? A: Very many (if representation is understood in the correct way). Michael Hanke, NADA, November 6, 28 Michael Hanke, NADA, November 6, 28 Computation of the Coefficients A First Example: Square Wave From sinnxsinkx /2cos(n k)x /2cos(n+k)x it follows {, if n k, sinnxsinkxdx π, if n k. The functions sinnx are orthogonal to each other in L 2 (,π). Assume that f(x) b n sinnx. Multiply through by sinkx and integrate: So f(x)sinkxdx b n b k π sinnxsinkxdx b k f(x) sinkxdx. sin 2 kxdx πb k. Assumptions: The series must converge in such a sense that integration is possible after multiplication by sinkx. Summation and integration must be exchangable., if x (,), SW(x), if x (,π),, if x,,π. Fourier sine series: SW(x) 4 [ sinx π + sin3x + sin5x + sin7x ] + 3 5 7 2.5.5.5.5 2 3 2 2 3 Michael Hanke, NADA, November 6, 28 2 Michael Hanke, NADA, November 6, 28 3

Gibbs Phenomenon Fourier Cosine Series 2.5.5.5.5 2 3 2 2 3 In the case of even functions, the prototypes are cosines, cosnx. Fourier cosine series: C(x) a + a n cosnx. Again, we have orthogonality:, if n k, cosnxcoskxdx, if n k, π, if n k >..8.6.4 Let f : [,π] C be an even function. Assume f(x) a + a n cosnx..2 SW N9 N5 N2 2.2 2.4 2.6 2.8 3 3.2 3.4 Gibbs phenomenon: Partial sums overshoot near jumps. Then: a k { R π R π π f(x)dx, if k, f(x)coskxdx, if k >. Michael Hanke, NADA, November 6, 28 4 Michael Hanke, NADA, November 6, 28 5 Examples An Observation: Decay of Coefficients Repeating Ramp RR is obtained by integrating SW: RR(x) x. Fourier cosine series: RR(x) π 2 π [ cosx + cos3x + cos5x + cos7x ] + 4 2 3 2 5 2 7 2 Note: The coefficients are equal to those obtained by termwise intergration of the sine series for SW. Up-Down UD is obtained as the derivative of SW: coefficients functions no decay Delta functions /k decay Step functions (with jumps) /k 2 decay Ramp functions (with corners) /k 4 decay Spline functions (jumps in f ) r k decay (r < ) Analytic functions The partial sums for analytical functions converge exponentially fast! This is the basis for fast solution methods for certain partial differential equations. Details will follow later. Fourier cosine series: UD(x) 2δ(x) 2δ(x π). UD(x) 4 [cosx+cos3x+cos5x+cos7x+ ].?? π Q: What about convergence? The terms are not a zero sequence! Michael Hanke, NADA, November 6, 28 6 Michael Hanke, NADA, November 6, 28 7

Fourier Series For Dirac s Delta Functional Dirac s Delta Functional (cont) Definition: For every continuous function f on [, π], δ(x) f(x)dx f(). δ is not a usual function. It is a functional: δ : C[,π] C. A simple calculation gives: Partial sums: δ(x) + π [cosx+cos2x+cos3x+ ]. δ N [+2cosx+ +2cosNx] 3.5 3 2.5 2.5.5.5 3 2 2 3 Q: In which sense converges δ N against δ? A: For every continuous function f C[,π], it holds δ N (x) f(x)dx δ(x) f(x)dx f(). Notation: Weak convergence. Michael Hanke, NADA, November 6, 28 8 Michael Hanke, NADA, November 6, 28 9 Fourier Series: General Periodic Functions Fouries Series: The Complex Version Let f : [,π] C be any (nice) function. Then f f even + f odd f even /2( f(x)+ f( x)) f odd /2( f(x) f( x)) Hence, f can be written as a sum of sine and cosine series: f f even + f odd a + a n cosnx+ b n sinnx Moivre s theorem: e iα cosα+isinα. Define c k (a k ib k )/2, c k (a k + ib k )/2. Then: c k e ikx + c k e ikx c k (coskx+isinkx)+c k (coskx isinkx) (c k + c k )coskx+i(c k c k )sinkx a k coskx+b k sinkx. The Fourier series can be equivalently written as f(x) n c n e inx. In what follows we will always use the complex notation! Michael Hanke, NADA, November 6, 28 Michael Hanke, NADA, November 6, 28

Properties Further Useful Properties Orthogonality in L 2 (,π): Let φ k (x) exp(ikx) for k..., 2,,,,2,... Every f L 2 (,π) (complex!) has a representation f(x) + k ˆf k e ikx, with ˆf k f 2( f,φ k) f(x)e ikx dx in the sense of L 2 (,π). (The partial sums converge towards f in the means square norm.) Since SW L 2 (,π), we conclude that pointwise convergence cannot always be expected. Since e ikx coskx+isinkx, the convergence will be the better the more periodic u is. Parseval s identity: Consequently, (φ k,φ j ) e ikx e i jx dx e i(k j)x dx δ k j For the derivatives, we have f(x) 2 dx f 2 + ˆf k 2 k f L 2 + (,π) ˆf k 2 <. k f (p) + (x) (ik) p ˆf k e ikx k Let H p per {v H p (,π) v is -periodic}. f Hper p + k 2p ˆf k 2 <. k This is the generalization of the decay property for Fourier coefficients. (Strang, p. 32, 327) Michael Hanke, NADA, November 6, 28 2 Michael Hanke, NADA, November 6, 28 3 The Discrete Fourier Transform (DFT) Read: Strang, Ch. 4.3 Without loss of generality assume the basic interval to be [,]. Let [, ] be subdivided into N equidistant intervals, h N, x j jh. For a periodic function, f() f() f(x N ) such that the trapezoidal rule reads Z Here, we used f(x)e ikx dx h w e ih f j e ikx j N N ( f ) j e ih jk f j w jk : c k The discrete version of the inverse transformation is, Theorem. f j k c k e ikx j k c k w k j One transformation is the inverse of the other, f j f j Proof. Compute: The Proof f j k k N c k w k j N l f l w lk w k j l f l w ( j l)k k Since { w ( j l)n w ( j l)k w, if j l, k N, if j l, the result follows. Some common notation: w w 2 w F w 2 w 4 w 2() w w 2() w ()2 Then it holds: f Fc, c N Ff, F N F Note: F is symmetric, but not Hermitian. Michael Hanke, NADA, November 6, 28 4 Michael Hanke, NADA, November 6, 28 5

The Fast Discrete Fourier Transform (FFT) FFT (cont.) The naive application of the discrete Fourier transform has complexity O(N 2 ). (matrix-vector multiplication) Example: N 2 2. The naive approach requires 2 24.7 7 complex multiplications. The FFT requires only 6 2 2 2.4 4 multiplications. The basic idea: Let N be a power of 2 and M N/2. f j k M k c k w k j c k w k j + c k w k j k even k odd c 2k (w 2 ) k j } {{ } : f j c 2k +(w 2 ) k j k } {{ } : f j +w j M f j and f j are discrete Fourier transform of half the original size M! This formula can be simplified: For j,...,m : Take it as it stands. For j M,...,N : Let j j M. It holds w M and w N : This gives the identities w M+ j w M w j, (w 2 ) k j (w 2 ) k j. f j f j + w j f j f j+m f j w j f j } j,...,m This recursion gives rise to a divide-and-conquer strategy. Computational complexity: O(N logn) Michael Hanke, NADA, November 6, 28 6 Michael Hanke, NADA, November 6, 28 7 Assumptions: FFT: Computational Complexity The exponentials w j are precomputed. Let W(N) be the number of complex operations for a FFT of length N. W(2M) 2W(M)+4M,W(). Denote w j W(2 j ) and N 2 n : w,w j 2w j + 2 2 j. Multiply the equation by 2 N j and sum up: Consequently, n j 2 n j w j 2 n j 2 n j( w j + 2 j) 2n2 n n + 2 n j w j W(N) w n 2n2 n 2N ld N Shifted DFT Using the base interval [,] leads to the standard DFT. What happens if we use [,π] instead? (Let N be even.) c k N N N N N ik jh f j e f j e ik jh + e i ik jh N f j e jn/2 f j e ik jh + ik( j N)h N f j e jn/2 l N/2 f j e ik jh + N f l e iklh l N/2 Similarly, for the inverse DFT it holds, f l Order of coefficients: f l e iklh c k e iklh, l N/2,...,N/2. DFT: (c,c,...,c ) shifted DFT: (c N/2,c N/2+,...,c,c,...,c ) This is what matlab s fftshift does. Michael Hanke, NADA, November 6, 28 8 Michael Hanke, NADA, November 6, 28 9

Fourier Integrals Fourier Integrals: The Key Rules Read: Strang, p. 367 37 Fourier series are convenient to describe periodic functions. Equivalently, f must be defined on a finite interval. Q: What happens if the function is not periodic? Consider f : R C. The Fourier transform ˆf F ( f) is given by Z ˆf(k) f(x)e ikx dx, k R. Examples: d f/dx ik ˆf(k) Z f(x)dx ˆf(k)/(ik) f( d) e ikd ˆf(k) ê ic f ˆf(k c) Here, f should be in L (R). Inverse transformation: f(x) Z ˆf(k)e ikx dk Note: Often, you will se a more symmetric version by using a different scaling. Theorem of Plancherel: f L 2 (R) ˆf L 2 (R) and Z f(x) 2 dx Z ˆf(k) 2 dk. Delta functional ˆδ(k) for all k R. Centred square pulse Let {, if L x L, f(x), if x > L. Then ˆf(k) 2 sinkl 2LsinckL, k where sinct sint/t is the Sinus cardinalis function. Michael Hanke, NADA, November 6, 28 2 Michael Hanke, NADA, November 6, 28 2 Sampling The Sampling Theorem Read: Strang, p. 69 693 By using the Fourier transform F and its inverse, any function can be reconstructed. Q: Can a function be reconstructed by using only discrete samples? Obviously, no. The questions becomes the general interpolation problem which does not have a unique solution. Often, an interpolation problem gets a unique solution if the class of possible interpolants is restricted. Q: What is the correct class if we stick to Fourier transforms? Consider one period of a simple harmonic f(t) ae i(ωt+φ). Obviously, one needs (at least) two samples in [, ω/()) for determining the two parameters. Using an a-priori bound on the Fourier transform ˆf of a function f, this function can be reconstructed by discrete sampling. Theorem: (Shannon-Nyquist) Assume that f is band-limited by W, i.e., ˆf(k) for all k W. Let T π/w be the Nyquist rate. Then it holds f(x) f(nt)sincπ(x/t n) where sinct sint/t is the Sinus cardinalis function. Note: The sinc function is band-limited: {, if k π, ŝinc(k), elsewhere. Use now equidistant sampling with step size T. Definition: Nyquist sampling rate: T π/ω. For a given sampling rate T, frequencies higher than the Nyquist frequency ω N π/t cannot be detected. A higher frequency harmonic is mapped to a lower frequency one. This effect is called aliasing. Michael Hanke, NADA, November 6, 28 22 Michael Hanke, NADA, November 6, 28 23

Sampling Theorem: Proof Proof (cont) Assume for simplicity W π. By the inverse Fourier transform, Define f(x) f(k) Z ˆf(k)e ikx dk ˆf(k)e ikx dk. { ˆf(k), if < x < π, periodic continuation, if k π. f can be represented as a Fourier series: where ˆ f n Hence, for < x < π, f(k) n f(k)e ink dk ˆf(k) f(k) n ˆ f n e ink, ˆf(k)e ink dk f( n). f( n)e ink. Therefore, f(x) n n n ˆf(k)e ikx dk ( f( n)e )e ink ikx dk n f( n) e ik(x+n) dk f( n) sinπ(x+n) π(x+n) sinπ(x n) f(n) π(x n) Michael Hanke, NADA, November 6, 28 24 Michael Hanke, NADA, November 6, 28 25 Spectral Interpolation Spectral Interpolation (cont.) Read: Strang, p. 448 45 For the DFT (on [,]) we know f(x j ) k Consider the function Π f, Π f(x) c k e ikx j, j,...,n. k c k e ikx, x R. This is an interpolating trigonometric polynomial, the so-called spectral interpolant. Note: Even for real f, Π f is in general complex (with the exception of the grid points x j, of course). Q: How can one obtain a real interpolant for a real-valued function?. Replace Π f by the shifted interpolant, Π c f(x) c k e ikx, x R. Note: Π f Π c f with the exception of the grid points. 2. Replace Π c f by its symmetrized variant, P f(x) N/2 c k e ikx, x R. Here, N km 2 c M + c M+ + + c + 2 c N. This interpolation can be explicitely written down, P f(x) p(x) psinc is the periodic sinc function, f j psinc(x jh). psinc(x) N N/2 e ikx sin(πx/h) (/h) tan(x/2) Michael Hanke, NADA, November 6, 28 26 Michael Hanke, NADA, November 6, 28 27

Spectral Methods: Differentiation Idea: Given a function u at discrete points, interpolate by a suitable smooth function p(x) and set u (x j ) p (x). Examples:. Piecewise linear interpolation: u (x j ) u j+ u j h 2. Piecewise quadratic interpolation: u (x j ) u j+ u j 2h Let s now use spectral interpolation: Remarks: p(x) u j psinc(x jh) u (x j ) p (x j ) u j d dx psinc(x j jh) Repeat: The Finite Element Method Start with a differential equation. Derive the weak formulation a(u,v) L(v) for all v V and, if possible, the minimization formulation P(u) a(u,u) L(u) min! 2 (M) Find an approximating (finite dimensional) space V h V and solve V and M, respectively: Galerkin and Ritz methods. The method yields up to a constant the best approximation of u in V h. (V) Piecewise polynomial interpolation uses only local informations. Spectral differentiation uses all gridpoints for evaluating one derivative. Spectral differentiation leads to full matrices DP while standard differences give rise to sparse matrices. Computational complexity: Polynomial: O(N) Spectral via FFT: O(N logn) Michael Hanke, NADA, November 6, 28 28 Michael Hanke, NADA, November 6, 28 29 FEM: Pros and Cons Possible Alternatives Advantages with finite elements: Very flexible, easy to adapt to complex domains and/or solutions; Fast algorithms for its implementation, fast solvable by, e.g., multigrid methods; In principle, good convergence: With Pp elements, e h O(h p+ ). Drawbacks with finite elements: Many degrees of freedom necessary for obtaining a good approximation (especially in 3D); Very hard to construct Pp elements in higher dimensions. Q: Are there alternatives? Q: Are there alternatives? A: Use basis functions which are closely adapted to the problem at hand. Advantages: We will obtain exponential convergence, that is, faster than any power of h. Only very few degrees of freedom needed for high accuracy. Drawbacks: The stiffness matrix will be full. Every problem needs its own set of ansatz functions. Q: May it be efficient in practice? A: Fast transformation algorithms. Michael Hanke, NADA, November 6, 28 3 Michael Hanke, NADA, November 6, 28 3

An Example Example: The Analytical Solution Find the -periodic solutions of with a constant r >. u + ru f(x),x (,) Weak formulation, with V H per(,): Z a(u,v) (u v + ru v)dx, L(v) Z f vdx. Insert the Fourier expansion and test against all basis functions φ j e ikx : Z a(u,φ j ) ( + k + Z û k k û j ( j 2 + r). ikû k e ikx i je i jx + + k (ik ( i j)+r)e i(k j)x dx û k e ikx e i jx )dx Analogously, Hence, a(u,φ j ) û j ( j 2 + r). Z L(v) f e i jx dx ˆf j. û j j 2 + r ˆf j, j,±,±2,... The solution seems even ok if r is not a negative square of an integer. If even f H p per(,), then u H p+2 per (,): + k k 2p+4 û k 2 + k k 2p+4 ˆf k 2 + (k 2 + r) < k 2p ˆf 2 k 2 < k Michael Hanke, NADA, November 6, 28 32 Michael Hanke, NADA, November 6, 28 33 Example: Galerkin s Method Applied Apply now Galerkin s method with V N {v v +N/2 ˆv k e ikx }: Since R e ikx e i jx dx for i j, the solution is easily seen to be: Error estimation: Theorem u N N/2 û hk e ikx with û hk û k. e N (x) u(x) u N (x) û k e ikx For all square integrable functions f, e N 6 N 2 f For f L 2 (,), Galerkin s Method: Proofs e N 2 If even f H p per(,): e N 2 û k 2 (N 2 /4+r) 2 f 2 6 N4 f 2 ˆf k 2 (k 2 + r) 2 (N/2) 2p (N 2 /4+r) 2 ˆf k 2 (k 2 + r) 2 k 2p ˆf k 2 k 2p (k 2 + r) 2 k 2p ˆf k 2 C(p)2 N 2p+2 (quadratic convergence). If even f H p per(,): O ( N (p+)) If f is infinitely often differentiable, we have exponential convergence. Michael Hanke, NADA, November 6, 28 34 Michael Hanke, NADA, November 6, 28 35

What Is Behind It? Q: Why on earth does this method work that good?? A: The exponentials e ikx are eigenfunctions of the differential operator. (Here: u + ru) Later on, we will see that the discrete versions of the exponentials are eigenfunctions of the finite difference discretizations of certain differential equations. This makes it clear why they will be important for analyzing numerical schemes. (Pseudo-)Spectral Methods Read: Strang, p. 45 453 Consider again the equation u + ru f. Ansatz as before u N c k e ikx. Collocation: Use test functions v j (x) δ(x x j ) for x j jh π, h /N. Equivalently, u N(x j )+ru N (x j ) f(x j ), j N/2,...,N/2. Some computations: The solution becomes c k ( (ik) 2 e ikx j + re ikx j ) [c k (k 2 + r) ˆf k ]e ikx j, all j c k (k 2 + r) ˆf k, all k c k ˆf k k 2 + r. ˆf k e ikx j This is the same solution as obtained by the Galerkin method. Michael Hanke, NADA, November 6, 28 36 Michael Hanke, NADA, November 6, 28 37 Pseudo-Spectral Methods Other Applications Of FFT Fourier series are only well-suited for periodic boundary conditions. In case of Dirichlet boundary conditions, Chebyshev polynomials T k (x) are a viable alternative. (Strang, p. 336 338) Chebyshev polynomials are eigenfunctions of the equation d ( ) dt λwt, < x < dx w dx Digital signal processing Digital image processing (encoding [JPEG, MPEG, DVB], denoising, reconstruction,...) Analysis of random processes Stability analysis of numerical schemes with w(x) x 2. The corresponding scalar product is ( f,g) w R w(x) f(x)g(x)dx. Michael Hanke, NADA, November 6, 28 38 Michael Hanke, NADA, November 6, 28 39