Lecture 7 January 26, 2016

Size: px

Start display at page:

Download "Lecture 7 January 26, 2016"

Bethanie Bishop
5 years ago
Views:

1 MATH 262/CME 372: Applied Fourier Analysis and Winter 26 Elements of Modern Signal Processing Lecture 7 January 26, 26 Prof Emmanuel Candes Scribe: Carlos A Sing-Long, Edited by E Bates Outline Agenda: Discrete Fourier transform umerical accuracy of the trapezoidal rule 2 Fourier transform of finite signals 3 Introduction to Fast Fourier Transforms Last Time: We defined discrete convolutions for signals defined on Z Following a similar approach to that used for the continuous-time Fourier transform, we deduced that time-invariant maps on discrete signals are precisely the maps that can be expressed as convolutions by a fixed function We also showed that complex exponentials with again diagonalize these operators In turn, this motivated the definition of the Fourier series of a discrete signal, which is a periodic function The natural question is, is every periodic function a Fourier series? For square-integrable functions on the circle S, the answer is yes We showed that complex exponentials are a complete orthonormal set in this space, which immediately yields expressions for the inverse transform, as well as the Parseval-Plancherel theorem Furthermore, we saw why Shannon s sampling theorem can be interpreted as a restatement of the completeness of this orthonormal sequence 2 umerical accuracy of the trapezoidal rule In many applications one wants to estimate the integral of a function over a finite interval Under proper normalizations, we can state this problem as estimating the integral I f(t) dt This is typically done by using quadrature rules, that is, by using a finite sequence of nodes {t n } n [, ] and weights {w n} n such that I n w n f(t n ),

2 is a good approximation of I The numerical accuracy of the quadrature rule is characterized by the behavior of the error I I as increases One would like to choose nodes and weights so that the error decays as quickly as possible with The simplest method of quadrature is the trapezoidal rule, which uses uniformly spaced nodes with (almost) uniform weights: I ( ( ) ( ) 2 f() + f + + f + 2 ) f() Despite its simplicity, it is extraordinarily accurate when f is smooth and periodic (for instance, see Fig ) Let us see why this is the case Working on [, ], assume f is -periodic In particular, f() f() so that I n f ( n ) How close is I to I? To learn the answer, we express f as a linear combination of the basis functions we identified in Lecture 6: Let e k (t) e i2πkt and write f(t) k Z e k, f e k (t), which yields I e k, f e i2πk n n k Z k Z e k, f ( n e i2πk n ) ow, e i2πk/ is an th root of unity, equal to if and only if k is a multiple of We shall then use the following fact: If ξ is an th root of unity, then Indeed, ξ is a root of x, and so it is also a root of Consequently, We then have + ξ + ξ ξ () x x + x + x2 + + x { e i2πk n k is a multiple of otherwise n I ( ) k, f δ(k m) k Z e m Z e, f + I + m Z, m m Z, m 2 e m, f e m, f

3 If the Fourier coefficients of f can be computed, we could even have an explicit expression for the remainder I I e m, f m Z, m To determine how fast it decays as a function of, consider f that is p-times differentiable and -periodic In this case, integration by parts tells us e k, f f(t)e i2πkt dt i2πk f (t)e i2πkt dt (i2πk) p f (p) (t)e i2πkt dt The periodicity is necessary for the boundary terms to vanish during integration by parts We now see that if p is large and f (p) is small, then this Fourier coefficient will be negligible More precisely, f(t)e i2πkt dt f (p) 2πk p, In this case, I I m Z, m f (p) 2πm p 2 f (p) (2π) p m> m p 2 f (p) (2π) p ζ(p), where ζ(p) tends to as p ow, when f is smooth, this bound holds for every p, meaning the error decays with faster than p for any p Such behavior is called spectral accuracy We noted that if f is smooth but not periodic (ie f() f()), then the argument fails as f(t)e i2πkt f() f() dt + i2πk i2πk f (t)e i2πkt dt And the failure is seemingly quite dramatic, since now the argument from above would stop with p, giving O( ) error The reality is only slightly better: standard numerical analysis would show O( 2 ) error The failure in accuracy can be understood by considering the periodic domain S R/Z of period The Fourier series we are considering are functions on S If f() f(), then the periodic extension of f to all of R has a discontinuity at ( mod Z) So even though f is smooth on (, ), it is not even continuous on S, hence the loss of accuracy One workaround to this problem is to window the function f Again suppose f is smooth on (, ) but not periodic For any δ >, we can construct a smooth function χ such that χ(t) for t [δ, δ] and decays to at the boundaries t and t Then we can write I f(t) dt χ(t)f(t) dt + χ(t)f(t) dt + δ ( χ(t))f(t) dt ( χ(t))f(t) dt + ( χ(t))f(t) dt δ The function χf is smooth and periodic, and consequently the trapezoidal rule will be extremely accurate in estimating the first term For the second and third terms, we can use a more sophisticated quadrature rule Moreover, for small δ, the expense of having a fine mesh (ie large ) on the short intervals [, δ] and [ δ, δ] is comparatively small 3

4 f(t) e cos2 (t) Figure : Here on [, π] instead of [, ], the trapezoidal rule with only 8 points (shown in red) yields an estimate I of impressive accuracy: I I (ie 8 digits of accuracy) 2 An example Consider f(t) sin(πt) over [, ] A direct computation gives We thus have the bound m Z, m e k, f e m, f 4 π sin(πt)e i2πkt dt 2 π m> 4k 2 4(m) 2 π 2 m> m 2 π 6 2, showing the trapezoidal quadrature has O( 2 ) accuracy For instance, see Fig 2 to see that 4 gives a quadrature error on the order 2 evertheless, the same quadrature performs perfectly when f(t) sin(2πt) The difference is that when extended periodically, sin(πt) has a discontinuity in its first derivative while sin(2πt) is smooth 3 Fourier transform of finite signals Here we consider finite signals {f[n]} n, for which we use brackets instead of parentheses to indicate their argument Here is the length of the signal We define the discrete Fourier transform (DFT) of f as ˆf[k] n f[n]e i2πkn, k,, ote that the DFT of f is also a finite signal of the same length We can naturally represent f as a vector f C with f n f[n] and similarly for ˆf Then the DFT is just the matrix multiplication ˆf F f where F kn e i2πkn, k, n,, 4

5 (a) f(t) sin(πt) (b) f(t) sin(2πt) Figure 2: (a) Integrating sin(πt) on [, ] using the trapezoidal rule always underestimates the area With 4 points we obtain I whereas the true value is I π/ We achieve digit of accuracy (b) Integrating sin(2πt) on [, ] using the trapezoidal rule underestimates the area over [, 2 ] and overestimates it over [ 2, ] The two errors exactly cancel each other to give I 4 I We use roots of unity to simplify the notation Let z e i2π Then ˆf k n z kn f n, and, in matrix notation ˆf ˆf z z 2 z ˆf 2 z 2 z ( 2)( 2) z ( 2)() ˆf z z ()( 2) z ()() }{{}}{{} ˆf F f f f 2 f } {{ } f This shows that F is a Vandermonde matrix (a matrix in which each row is a geometric sequence) Furthermore, the DFT (when suitable rescaled) is unitary, meaning F F F F I We verify this directly: Since z z, we compute { m n, l F ml F ln where we have again used () l z ml zln This yields the discrete inversion formula l z (n m)l F F or f F ˆf otherwise, 5

6 In function notation, f[n] k ˆf[k]e i2πkn In what follows we will not make distinction between vectors and finite signals 3 DFT as a quadrature Consider a continuous-time signal f(t) on S The Fourier transform of f is given by ˆf(ω) f(t)e i2πωt dt In practice, this integral may be difficult to compute analytically Suppose we estimated the integral by a left-hand sum with intervals This would give ˆf(ω) n f ( n ) 2πωn i e When ω {,,, }, this estimate is exactly the DFT of the finite discretization f[n] f ( ) n That is, when the finite signal consists of uniformly spaced samples of an analog signal, the DFT is an estimate of the first (nonnegative) Fourier coefficients And we have seen that when f(t) is smooth and periodic, this estimate is astonishingly accurate Furthermore, when the signal is real-valued, the positive-frequency coefficients immediately give the negative-frequency ones: ˆf( k) ˆf(k) 32 Circular convolution For infinite, discrete signals, one has a well-defined convolution (f g)(n) m Z f(m)g(n m) We would like a notion of convolution for finite signals as well However, finite signals {f[n]} n and {g[n]} n are only defined on {,,, }, meaning the sum above would include terms that are not defined Consequently, we extend the signals in the natural way: f[n] : f[n mod ] and g[n] g[n mod ], n Z ow f[n] and g[n] can be regarded as periodic signals on Z We can now define circular convolution as (f g)[n] m f[m]g[n m] m f[m mod ]g[n m mod ] In analogy with the infinite case, the discrete Fourier transform diagonalizes circular convolutions (this is an indication that we have made a good definition of convolution) That is, h g f DFT ĥ ˆfĝ 6

7 Continuing the analogy, we have circular translations and the circular impulse f f τ [n] f[n τ mod ], δ[n] { n mod, otherwise A translation-invariant map is a linear map L satisfying Lf τ (Lf) τ By the same arguments as given in Lectures and 6, a map is translation-invariant if and only if it is a circular convolution In particular, the corresponding convolution kernel is the response to the impulse, h : Lδ Consequently, we have the diagram DFT f[n] ˆf[k] L ĥ[k] (f h)[n] DFT ˆf[k] ĥ[k] If we denote by L the matrix representing L, this result can be interpreted as a factorization of the form L F diag(ĥ)f, where ĥ[] ĥ[] diag(ĥ) ĥ[ 2] ĥ[ ] We thus have a simple method of applying L to a signal f More important, we have a fast method Multiplication by a fixed transfer function ĥ[k] is certainly fast, and we will see in the next section how the DFT and inverse DFT are carried out efficiently 4 Introduction to Fast Fourier Transforms The DFT of a finite signal x of size is given by ˆx[k] n x[n]e i2πkn How do we compute it? The direct approach is to assemble the DFT matrix, which is dense The computational cost of this method is O( 2 ) This has many drawbacks For instance: We can only take DFTs of relatively small signals The problem with this is that one typically wants a large number of samples from a continuous signal to obtain an accurate estimate of its frequency content When the accuracy is proportional to, we are performing a computation that is four times more expensive to be only twice as accurate 7

8 It is not possible to compute the DFT in (near) real-time The problem with this is the DFT allows us to extract information contained on a prescribed frequency band by filtering This is the case in wireless communications, where a delay in extracting the relevant information amounts, for instance, to a significant delay between receiving a signal on a cellphone and actually listening to the message For these reasons and others, having an efficient algorithm to compute the DFT is extremely important The de facto algorithm to compute the DFT is the Fast Fourier Transform (FFT) This algorithm was conceived by Gauss (Gauss, circa 85) and rediscovered and popularized by Cooley & Tukey (Cooley & Tukey, 965) It allows us to compute the DFT of a signal of length in O( log ) operations Here we will discuss the radix-2 algorithm which assumes is a power of two The main idea behind it is the following When is even, a direct computation shows that ˆx[k] n /2 n /2 n x[n]e i2π kn /2 x[2n]e i2π k(2n) + n x[2n]e i2π /2 kn + e i2π x[2n + ]e i2π k(2n+) /2 k n x[2n + ]e i2π /2 kn Consequently, we can decompose the DFT of a signal of size into two DFTs of signals of size /2 To formalize this idea, let z e i2π, be the th root of unity and define the even and odd parts of x as x even [k] x[2k] and x odd [k] x[2k + ], for k,, /2, that is, the even and odd parts are signals of half the length of the original With this notation, the expression above can be written as {ˆxeven [k] + z k ˆx[k] ˆx odd[k], k /2, ˆx even [k /2] z k /2 ˆx odd [k /2], /2 k, and the DFT of a signal of size can be decomposed into two DFTs of signals of size /2 Furthermore, if we knew ˆx even and ˆx odd, then the above shows how to reconstruct ˆx Suppose that T is the cost of computing a DFT of a signal of size Then the above shows T 2T /2 }{{} computing ˆx even and ˆx odd + /2 }{{} computing z k for k,, /2 + 3/2 }{{} forming the upper and lower part of ˆx For a historical account we refer to Heideman, M, Johnson, D H, Burrus, C S, Gauss and the history of the Fast Fourier Transform, IEEE ASSP Magazine, (4), 4-2 (984) 8

9 If 2 M for some M > then or T 2 M 2T 2 M + 2 M+ T 2 M 2 M T + 2M2 M, T T + 2 log 2, which asymptotically yields a complexity of O( log ) ote the FFT can be used to compute the inverse DFT efficiently too This is because x[n] k ˆx[k]e i2πkn k ˆx[ k]e i2π ( k)n e i2π n k ˆx[ k]e i2πkn, and consequently we only need to take the DFT of a reversed ˆx and then multiply by the factor above It is clear this has complexity O( log ) too In general, does not need to be a power of two, although FFTs are most efficient when it is Many techniques have been developed to reduce the DFT of a signal of size to DFTs of signals of smaller size, for instance, by attempting to use the prime factors of to achieve such a decomposition References [] Cooley, J W, Tukey, J W, An algorithm for the machine calculation of complex Fourier series, Mathematics of Computation, 9(2), 297-3, 965 9

Lecture 6 January 21, 2016

Lecture 6 January 21, 2016 MATH 6/CME 37: Applied Fourier Analysis and Winter 06 Elements of Modern Signal Processing Lecture 6 January, 06 Prof. Emmanuel Candes Scribe: Carlos A. Sing-Long, Edited by E. Bates Outline Agenda: Fourier