The moral of the story regarding discontinuities: They affect the rate of convergence of Fourier series

Transcription

1 Lecture 7 Inner product spaces cont d The moral of the story regarding discontinuities: They affect the rate of convergence of Fourier series As suggested by the previous example, discontinuities of a function fx create problems for its Fourier series expansion by slowing down its rate of convergence. At a jump discontinuity, the convergence may be quite slow, with the partial sums demonstrating Gibbs ringing. Another way to look at this situation is as follows: Generally, a higher number of terms in the Fourier series expansion or higher frequencies are needed in order to approximate a function fx near points of discontinuity. But, in fact, it doesn t stop there the existence of points of discontinuity actually affects the rate convergence at other regions of the interval of expansion. To see this, let s return to the two examples studied above, i.e., the functions x, π < x, f 1 x = x = x, < x π, 1 and 1, π < x, f 2 x = 1, < x π, 2 ote that we have subscripted them for convenience. Recall that the function f 1 x is continuous on [ π,π] and its 2π-extension is continuous for all x R. On the other hand f 2 x has a discontinuity at x = and its 2π-extension has discontinuities at all points kπ. We noticed how well a rather low number of terms i.e., 5 in the Fourier expansion of f 1 x approximated it over the interval [ π,π]. On the other hand, we saw how the discontinuities of f 2 x affected the performance of the Fourier expansion, even for a much larger number of terms i.e., 5. This is not so surprising when we examine the decay rates of the Fourier series coefficients for each function: 1. For f 1 x, the coefficients a k decay as O1/k 2 as k. 2. For f 2 x, the coefficients b k decay as O1/k as k. 65

2 The coefficients for f 1 are seen to decay more rapidly than those of f 2. As such, you don t have to go to such high k values which multiply sine and cosine functions, of maximum absolute value 1 for the coefficients a k to become negligible to some prescribed accuracy ǫ. Of course, there is the infinite tail of the series to worry about, but the above reasoning is still valid. The other important point is that the rate of decay of the coefficients affects the convergence over the entire interval, not just around points of discontinuity. This has been viewed as a disadvantage of Fourier series expansions: that a bad point, p, i.e. a point of discontinuity, even near or at the end of an interval will affect the convergence of a Fourier series over the entire interval, even if the function fx is very nice on the other side of the interval. We illustrate this situation in the sketch on the left in the figure below. Researchers in the signal/image processing community recognized this problem years ago and came up with a clever solution: If the convergence of the Fourier series over the entire interval [a,b] is being affected by such a bad point p, why not split the interval into two subintervals, say A = [a, c] and B = [c, b] and perform separate Fourier series expansions over each subinterval. Perhaps in this way, the number of coefficients saved by the niceness of fx over [a, c] might exceed the number of coefficients needed to accomodate the bad point p. The idea is illustrated in the sketch on the right in the figure below. The above discussion is, of course, rather simplified, but it does describe the basic idea behind block coding, i.e., partitioning a signal or image into subblocks and Fourier coding each subblock, as opposed to coding the entire signal/image. Block coding is the basis of the JPEG compression method for images as well as for the MPEG method for video sequences. More on this later. y = fx y = fx a p b a c p b nice region of smoothness of fx bad point of discontinuity Fourier series on [a, c] Fourier series on [c, d] Fourier series on [a, b]. 66

3 Greater degree of smoothness implies faster decay of Fourier series coefficients The effect of discontinuities on the rate of convergence of Fourier series expansions does not end with the discussion above. Recall that the Fourier series for the continuous function f 1 x given above demonstrated quite rapid convergence. But it is possible that series will demonstrate even more rapid convergence due to the fact that the Fourier series coefficients a k and b k decay even more rapidly than 1/k 2. Recall that the function f 1 x is continuous, but that its derivative f x is only piecewise continuous, having discontinuities at x = and x = ±π. Functions with greater degrees of smoothness, i.e., higher-order continuous derivatives will have Fourier series with more rapid convergence. We simply state the following result without proof: Theorem: Suppose that fx is 2π-periodic and C n [ π,π], for some n > that is, its nth derivative and all lower order derivatives is continuous. Then the Fourier series coefficients a k and b k decay as 1 a k,b k = O k n+1, as k. An idea of the proof is as follows. To avoid complications, suppose that f is piecewise continuous, corresponding to n = above, the coefficients must decay at least as quickly as 1/k, since they comprise a square-summable sequence in l 2. ow consider the function gx = x fs ds, 3 which is a continuous function of x Exercise. The Fourier series coefficients of gx may be obtained by termwise integration of the coefficients of fx AMATH 231. This implies that the series coefficients of gx will decay at least as quickly as 1/k 2. Integrate again, etc.. In other words, the more regular or smooth a function fx is, the faster the decay of its Fourier series coefficients, implying that you can generally approximate fx to a desired accuracy over the interval with a fewer number of terms in the Fourier series expansion. Conversely, the more irregular a function fx is, the slower the decay of its FS coefficients, so that you ll need more terms in the FS expansion to approximate it to a desired accuracy. This feature of regularity/approximability is very well-known and appreciated in the signal and image processing field. In fact, it is a very important, and still ongoing, field of research in analysis. The above discussion may seem somewhat handwavy and imprecise. Let s look at the problem 67

4 in a little more detail. And we ll consider the more general case in which a function fx is expressed in terms of a set of of functions, {φ k x} k=1, which form a complete and orthonormal basis on an interval [a,b], i.e., fx = c k φ k x, c k = f,φ k. 4 k=1 Here, the equation is understood in the L 2 sense, i.e., the sequence of partial sums, S n x, defined as follows, converges to f in L 2 norm/metric, i.e., S n x = n c k φ k x, 5 k=1 f S n 2 as n. 6 The expression in the above equation is the magnitude of the error associated with the approximation fx = S n x, which we shall simply refer to as the error in the approximation. This error may be expressed in terms of the Fourier coefficients c k. First note that fx S n x = c k φ k. 7 Therefore the L 2 -squared error is given by Thus, k=n+1 f S n 2 2 = f S n,f S n = c k φ k, = f S n 2 = k=n+1 k=n+1 [ k=n+1 l=n+1 c l φ l c k 2. 8 c k 2 ] 1/2. 9 Recall that for the above sum of an infinite series to be finite, the coefficients c k must tend to zero sufficiently rapidly. The above summation of coefficients starting at k = n + 1 may be viewed as involving the tail of the infinite sequence of coefficients c k, as sketched schematically below. For a fixed n >, the greater the rate of decay of the coefficients c k, the smaller the area under the curve that connects the tops of these lines representing the coefficient magnitudes, i.e., the smaller the magnitude of the term on the right of Eq. 9, hence the smaller the error in the approximation. From a signal processing point of view, more of the signal is concentrated in the first n coefficients c k. 68

5 c k 2 vs. k n + 1 k tail of infinite sequence From the examples presented earlier, we see that singularities in the function/signal, e.g., discontinuities of the function, will generally reduce the rate of decay of the Fourier coefficients. As such, for a given n, the error of approximation by the partial sum S n will be larger. This implies that in order to achieve a certain accuracy in our approximation, we shall have to employ more coefficients in our expansion. In the case of the Fourier series, this implies the use of functions sin kx and cos kx with higher k, i.e., higher frequencies. Unfortunately, such singularities cannot be avoided, especially in the case of images. Images are defined by edges, i.e., sharp changes in greyscale values, which are precisely the points of discontinuity in an image. However, singularities are not the only reason that the rate of decay of Fourier coefficients may be reduced, as we ll see below. Higher variation means higher frequencies are needed In the previous discussion, we saw how the irregularity or lack of smoothness of a function fx for example, points of discontinuity in fx or its derivatives affects the convergence of its Fourier series expansion. This phenomenon is very important in signal and image processing, particularly in the field of signal/image compression, where we wish to store approximations to the signal fx to a prescribed accuracy with as few coefficients as possible. In addition to smoothness, however, the rate of change of f, as measured by the magnitude of its derivative, f x, or gradient f, also affects the convergence. Contrast the two functions sketched 69

6 below. The function on the left, gx, has little variation over the interval [a,b] whereas the one on the right, hx, has significant variation. gx hx a b a b In order to accomodate the more rapid change in fx, i.e., in order to approximate such a function better, sine and cosine functions of higher frequencies, i.e., higher oscillation, are required. In other words, we expect that the Fourier series coefficients of gx will decay more rapidly than those of hx. Example 1: We can illustrate this point with the help of the following analytical example. Consider the normalized Gaussian function, g σ x = 1 x 2 2πσ 2 e 2σ 2, 1 which you have probably encountered in a course on probability or statistics. The variance of this function is σ 2 and its standard deviation is σ. As σ decreases toward zero, the graph of g σ x becomes more peaked higher and narrower as shown in the figure below. In what follows, we ll consider the function g σ x as defined only over the interval [ π,π] so that we may examine its Fourier series. 2 Gaussian functions 1.5 sigma =.25 Gt 1.5 sigma =.5 sigma = t 7

7 Clearly, the magnitude of the derivative of g σ x is increasing near x =. Let us now observe the effect of this increase on the Fourier coefficients of g σ x. Since it is an even function, its Fourier series will be composed only of cosine functions, i.e., g σ x = c + where we are using the orthonormal cosine basis set see earlier notes, c k φ k, 11 k=1 φ x = 1 2π, φ k x = 1 π cos kx, k Technically, the computation of the integrals of the Gaussian function is rather complicated since we are integrating only over the finite interval [ π,π]. For sufficiently large σ, the tail of g σ x lying outside this interval is very small in fact, it is exponentially small, therefore negligible. To a good approximation, therefore, a = = = π π g σ xφ xdx 1 2π g σ xdx 1 2π, 13 and a k = 1 π π π = 1 π = 1 π g σ x cos kxdx g σ x cos kxdx e x2 2σ 2 cos kxdx = 1 π e σ2 k These results can be derived from the following formula that can be found in integral tables, You let a 2 = 1 and then do some algebra. 2σ2 e a2 x 2 cos bxdx = π b 2 2a e 4a ote that the distribution of a k values with respect to k > we don t even have to square them since they are all positive is a Gaussian distribution with variance 1 σ. As we let σ +, 71

8 Profile of a k coefficients 1 σ k the distribution spreads out, in complete opposition to the function g σ x getting more concentrated at x =. We ll return to this theme the complementarity of space and frequency later in this course. Example 2: This is a numerical version of the previous example. For < a < π, let g a x denote the function, g a x = 3 2a 3 2a 1 x a, x a, 1 + x a, a x,, a < x π. 16 A sample graph of this function is sketched in the figure below. y q 3 2a y = g ax π a a π The multiplicative factor 3/2a was chosen so that g a 2 = 1, 17 for all a >, a kind of normalization condition. ote that as a approaches zero, the peak becomes more pronounced, since the magnitudes of the slopes of the peak are given by g ax = 3/2a 3/2. 72

9 Since the function g a x is even, it will admit a Fourier cosine series i.e., the coefficients b k of all sine terms are zero. Here we consider the expansion of g a x in terms of the orthonormal cosine basis, Then where e 1 x = 1 2π, e k x = 1 π cos kx, k g a x = c + c k e k, 19 k=1 c k = g a,e k. 2 For example, c = 1 2 π 2 a g a x dx = 1 2 3a π. 21 Since g a L 2 [ π,π], the sequence of Fourier coefficients c = c,c 1,c 2, is square summable, i.e., c l 2 sequence space. Moreover, from a previous lecture, implying that g a L 2 = c l 2 = 1, 22 [c k ] 2 = k=1 In the figure below are plotted the coefficients c n, n 2, for a values 1.,.5,.25,.1,.5. The coefficients were computed using MAPLE. The plots clearly show that the rate of decay of the coefficients decreases as a is decreased. For a = 1., the coefficients c n appear to be negligible for n > 5, at least to the resolution of the plot. This would suggest that the partial sum function S 5 x, composed of cosine terms with coefficients c to c 5 would provide an excellent approximation to g a x over the interval. On the other hand, for a =.5, it appears that we would have to use the partial sum S 1 x, and so on. In order to understand this more quantitatively, the partial sums S 2 x were computed for the a- values shown in the above figure. From these partial sums, the L 2 distances g a S 2 2 were computed using MAPLE. These distances represent the L 2 error in approximating g a with S 2. The results are presented in the table below. Clearly, as a is decreased, the error in approximation by the partial sums S 2 increases. There appears to be a dramatic increase between a =.25 and a =.1. Improvement by block coding. In light of the earlier discussion on block coding, let us see if we can improve the approximation to the above triangular peak function by dividing up the interval 73

10 .5.4 a= a=.5.1 a=.25 a=.1 a= n Coefficients c n of Fourier cosine series expansion of the triangular peak-function g a x defined in Eq. 16, for a = 1.,.5,.25,.1,.5. As a decreases, the rate of decay of the Fourier coefficients c n is seen to decrease. a g a S Error in approximation to g a x afforded by partial sum functions S 2 x comprised of Fourier coefficients c to c 2. and coding the function separately over the subintervals. In the following experiment, the interval I = [ π,π] was partitioned into the three subintervals, I 1 = [ π, π/3], I 2 = [ π/3,π/3], I 3 = [π/3,π]. 24 For a 1, the approximation of g a x over intervals I 1 and I 3 is trivial since g a x =. As such we don t even have to supply any Fourier coefficients but we should record the use of the first coefficient c =. After all, the function g a x is constant on these intervals, and we should specify the value of the constant. Since 21 coefficients were used in the previous experiment S 2 x uses c k, k 2, we shall use 19 coefficients to code the function g a x over interval I 2. It remains to construct the Fourier series approximation to g a x over interval I 2 = [ π/3,π/3]. 74

11 From Lecture 7, we must employ the basis set {e k } = { 1 2a, 1 a cos πx, a 1 a sin πx, a 1 2a cos 2πx a }, 25 where a = π/3. Once again, the sine functions are discarded since g a x is an even function. This was easily done in MAPLE: For each a value, the necessary integrals were computed actually only the integrals over [,a] were computed, followed by the L 2 distance between g a and the S 18 x partial sum functions. The results are presented in the table below. We can see an improvement for all a a g a S Error in approximation to g a x afforded by partial sums S 18 of Fourier cosine series over interval [ π/3, π/3] employing Fourier coefficients c to c 18, along with the trivial Fourier expansions c = on [ π, π/3 and π/3, π]. values a roughly five-fold decrease in the error for a = 1 and about a three-fold decrease for a =.5. This very simple implementation of block coding has achieved the goal of decreasing the error with a given number of coefficients. Question: The fact that the Fourier series over [ π/3, π/3] works better to approximate the function g a x might appear rather magical. Can you come up with a rather rather simple explanation for the improvement in accuracy? That being said, the improvement is rather impressive in this case because we know the function essentially to infinte accuracy, i.e., we have its formula. If we had only a finite set of discrete data points representing sampled values of the function, the improvement would not be so dramatic. We ll return to this matter after looking at discrete Fourier transforms. 75

12 Fourier series on the interval [ a, a], even and odd extensions In a previous lecture, it was mentioned that the following functions comprise an orthonormal set on the interval [ a,a], where a > : e = 1 2a, e 1 = 1 a cos πx, e 2 = 1 sin a a πx, e 3 = 1 cos a a 2πx a,. 26 Moreover, this set serves as a complete orthonormal basis for the space L 2 [ a,a] of square-integrable functions on [ a,a]. Thus, for an f L 2 [ a,a], f = f,e k e k. 27 k= This may be translated to the following standard unnormalized Fourier series expansion having the form where fx = a + k=1 [ a k cos kπx a + b k sin kπx a ], 28 a = 1 2a a k = 1 a b k = 1 a a a a a a a fx dx kπx fxcos dx a kπx fxsin dx. 29 a We use the term unnormalized since the coefficients a k, b k are multiplying the unnormalized functions coskπx/a and sinkπx/a. The normalization factors, which involve a factors that become a upon squaring, are swept into the a k and b k coefficients, which accounts for the factors appearing in front of the above integrals. Once again, in the special case a = π, the above formulas become the standard formulas for Fourier series on [ π,π], cf. Eq. 1, Lecture 1 of these notes. Fourier cosine series on [ a, a] and periodic extensions In the case that fx is even, i.e., fx = f x, then all coefficients b k =, so that the expansion in 28 becomes a Fourier cosine series expansion. Moreover, since fx is even, it need only be defined on the interval [,a], and the expressions for the coefficients a k become a = 1 a fx dx, a k = 2 a kπx fxcos dx, k 1. 3 a a a 76

13 ow suppose that we are given a function fx defined on the interval [,a] as input data. From this data, we may construct the a k coefficients these coefficients define a Fourier cosine series that converges to to the even 2a-extension of fx, constructed from fx by means of two steps, illustrated schematically in the figure below, 1. A flipping of the graph of fx with respect to the y-axis to produce an even function on [ a,a]. 2. Copying this graph on the intervals [a,3a], [3a,5a], etc. and [ 3a, a], [ 5a, 3a], etc.. y y = fx 5a 3a 2a a a 2a 3a 4a x original data 2a-extension even extension of data 2a-extension 2a-even extension of fx, x a ote that the resulting 2a-extension is continuous at all patch points, i.e., x = 2k 1a, k Z. For this reason, Fourier cosine series are usually employed in the coding of signals and images. The JPEG/MPEG standards are based on versions of the discrete cosine transform. Fourier sine series on [ a, a] and periodic extensions In the case that fx is odd, i.e., fx = f x, then all coefficients a k =, so that the expansion in 28 becomes a Fourier sine series expansion. Moreover, since fx is odd, it need only be defined on the interval [,a] as well. The expression for the coefficients b k becomes b k = 2 a kπx fxsin dx, k a a Once again, suppose that we are given a function fx defined on the interval [,a] as input data. From this data, we may construct the b k coefficients these coefficients define a Fourier sine series that converges to to the odd 2a-extension of fx, constructed from fx by means of two steps, illustrated schematically in the figure below, 77

14 1. An inversion of the graph of fx with respect to the origin produce an odd function on [ a, a]. If f, then one of the points, ±f will have to be deleted for f to be single-valued at x =. 2. Copying this graph on the intervals [a, 3a], [3a, 5a], etc. and [ 3a, a], [ 5a, 3a], etc.. Once again, some endpoints of the pieces of the graph will have to be deleted to make f single-valued. y original data y = fx 5a 3a 2a a a 2a 3a 4a x 2a-extension odd extension of data 2a-extension 2a-of extension of fx, x a ote that the resulting 2a-extension need not be continuous at the patch points, i.e., x = 2k 1a, k Z. Indeed, if f, then the odd extension of fx will not even be continuous at, ±2a, ±4a, etc.. The two-dimensional case: image functions ote: The discussion in the first two paragraphs is slightly more general than that presented in class. We now examine briefly the Fourier analysis of two-dimensional functions, which will be used primarily to represent images. We shall consider an image function fx,y to be defined over a suitable rectangular region D R 2. For the moment, let D be defined as the rectangular region a x a, b y b, centered at the origin. A suitable function space for the representation of images will be the space of square-integrable functions on D, i.e., L 2 D: L 2 D = {f : D R fx,y 2 da < } 32 ow let D 78

15 1. {e k x} 1 denote the orthonormal set of sine and cosine functions on the space L 2 [ a,a]. 2. {o k y} 1 denote the orthonormal set of sine and cosine functions in the space L 2 [ b,b]. Theorem: The set of all product functions {φ kl x,y = e k xo l y} k = 1,2,, l = 1,2,, form an orthonormal basis in L 2 D. For simplicity, we now assume that our images are defined on square regions, i.e., a = b, and further assume that a = b = 1. In this case the basis functions e k and o k have the same functional form: {e k } 1 = { 1 2,cosπx,sinπx,cos2πx,sin2πx, } 33 The set of all products e k xe l y will lead to a complicated mixture of sine and cosine functions. It is convenient to assume that the image function fx,y is an even function with respect to both x and y, implying that we use only the cosine functions in our basis. In essence, this amounts to the assumption that the actual image being analyzed lies in the region [,1] [,1]. Analogous to the one-dimensional case, the use of only cosine functions will perform an even 2π-periodic extension of this image, both in the x and y directions. Let us examine this further. 1. Even w.r.t. x: fx,y = f x,y. 2. Even w.r.t. y: fx,y = fx, y. 3. From 1 and 2: f x,y = fx, y, implying that fx,y = f x, y, i.e., symmetry w.r.t. inversion about,. This means that the graph of fx,y in the first quadrant, i.e., [,a] [,a], i.e., the input image, is flipped w.r.t. the y-axis, then flipped w.r.t. the x-axis, and finally flipped w.r.t. the point,. The result is an even 2π-extension of the function fx,y. The process is illustrated below. The advantage of an even extension in both directions is that no discontinuities are introduced. The function fx,y is continuous at all points on the x and y-axes. As such, no complications regarding convergence of the Fourier series are introduced artificially. The net result is that the input image function fx,y defined on the region [,1] [,1] will admit a Fourier cosine series expansion of the form, fx,y = a + a kl coskπxcoslπy. 34 k=1 l=1 79

16 y 1 original image -1 1 x -1 Input image fx, y, x, y 1, and its even 2π-extension in x and y directions via Fourier cosine transform. The series coefficients a kl could be obtained from the expansion for f in terms of the orthonormal basis functions or by simply multiplying both sides of 34 with the function cosmπxcosnπy and integrating x and y over [, 1], and exploiting the orthogonality of the cosine functions. The net result is a = a l = 2 a k = 2 a kl = fx,y dxdy, fx,ycoslπy dxdy, l 1, fx,ycoskπx dxdy, k 1, fx,ycoskπxcoslπy dxdy, k,l

17 Lecture 8 The Discrete Fourier Transform We now turn to the analysis of discrete data, e.g., sets of measurements, y k,k =,1,2,, as opposed to signals in continuous time, e.g., ft. We also assume that the measurements are evenly spaced in time/space, i.e., there is a fixed time interval T > between each measurement. This is necessary for the basic theory to be presented below. That being said, it is very often the procedure employed in scientific experiments, e.g., measuring the temperature at a particular location at hourly intervals. At this time, we shall simply assume that the measurements correpond to the values of a function ft at discrete times, t n = nt. In the signal processing literature, the usual notation for such a sampling is as follows, f[n] := fnt, n {,1,2, } or n {, 1,,1, }. 36 The square brackets are rather cumbersome some authors employ the notation f n, but we shall reserve this notation for other purposes. The idea is sketched below. y o f[] f[1] o f[2] f[3] f[4] f[5] f[6] o o o o o f[n] o y = ft T 2T 3T 4T 5T 6T nt t We now assume that we are working with a set of such consecutive data points which will comprise an -vector, indexed as follows, f = f[],f[1],,f[ 1]. 37 These measurements could be complex-valued, so that f C. Furthermore, we assume that this set of measurements is then periodized, i.e., extended into the future and backwards into the past, so that f[k + ] = f[k], k Z

18 This represents a periodic extension of the data, a discrete analogy to the periodization of functions produced by Fourier series representations. A derivation of the DFT from Fourier expansion in terms of complex exponentials Let us first assume that we are working with a function ft that is a-periodic, i.e., ft + a = ft, t R. 39 We now use the fact that the following doubly-infinite set of functions, e k t = 1 a e i2πkt/a, k {, 2, 1,,1,2, }, 4 forms an orthonormal basis for the space of functions L 2 [,a]. It is a good exercise to verify the orthonormality of these functions over the interval [, a]. In Lecture 6, we introduced a set of complex exponential functions that were orthonormal over the interval [ a,a]. We now expand ft in terms of this basis over [,a], ft = k= where the Fourier coefficients c k are given by the complex scalar product c k = f,e k = a fte k tdt c k e k, 41 = 1 a fte i2πkt/a dt 42 a Ignoring the constant, we now construct Riemann sum approximations to the above integral following the usual procedure from first-year Calculus. Let 1 be a fixed integer. Construct an equipartition of the interval [,a] in the usual way, i.e., let and define the partition points, t = a, 43 t n = n t = na, n =,1,2,,. 44 We use the Riemann sum that is produced by evaluating ft at the left-endpoints of each of the subintervals I n = [t n,t n+1 ],n =,1,2,, 1, i.e., a ft exp i2πkt dt a 1 82 ft n exp i2πkt n t a

19 = = 1 ft n exp i2πkn a 1 a 1 ft n exp i2πkn. 45 We now ignore the constant factor a/ and focus on the remaining summation. The ft n are viewed as discrete samples of the function ft so we define, as before, f[n] = ft n = fnt, 46 where the sampling time is given by T = t = a/. The summation in Eq. 45 may then be written as follows, c[k] := 1 1 f[n] exp i2πkn. 47 This has the form of a complex scalar product between the -vector of sampled data points, defined earlier and the complex -vector e k, with components e k [n] = 1 exp f = f[],f[1],,f[ 1], 48 e k = e k [1],e k [2],,e k [ 1], 49 i2πkn, n =,1,, 1. 5 We ll show below that the vectors e k are orthonormal. The index n plays the role of the time or spatial variable and k is the index of the frequency. We haven t said anything about the frequency k so far. In the continuous formulation, we required all integer values of k. From Eq. 47, it is easily shown we ll do it later that i.e., the complex -vector, c, defined as c[k + ] = c[k], 51 c = c[],c[1],,c[ 1], 52 is -periodic, as is the -vector of sampled data, f. As such, the frequency index k may be constrained to the values,1,, 1. The complex -vector c defined in Eq. 52 is known as a discrete Fourier transform DFT of the discrete -vector f in Eq. 48. Let us now investigate this DFT in terms of complex periodic -vectors. 83

20 An orthonormal periodic basis in C The goal is to provide a representation of a set of data in terms of periodic basis vectors in C. First of all, the following inner product will be used in C : f,g = 1 where the bar once again denotes complex conjugation. f[n]g[n], 53 Of course, any orthogonal set of complex -vectors will serve as a basis for C, but we wish to use a set of periodic vectors. The family u k C discovered in the previous section will do the trick: For k =,1,, 1, define the vector u k = u k [1],u k [2],,u k [ 1], 54 with components i2πkn u k [n] = exp, n =,1,, Once again, the index n plays the role of the time or spatial variable and k is the index of the frequency. ote that in the special case k =, all elements u k [n] = 1. In other words, for all 2, the -vector u C is a row of 1 s: This will have important implications in for the DFT. u = 1,1,1,,1. 56 Let us now show that the vectors u k are -periodic. First consider a given k {,1,, 1}. Then consider a given component u k [n], n {,1,, 1}, in the vector u k. From Eq. 55, i2πkn + u k [n + ] = exp i2πkn i2πk = exp exp i2πkn = exp expi2πk i2πkn = exp = u k [n]

21 We claim that the set of -vectors {u k } forms an orthogonal set in C. To prove this, consider the inner product between two elements, u k and u l : u k,u l = = 1 1 i2πkn exp i2πk ln exp exp i2πln. 58 Case 1: k = l. In this case, the above inner product reduces to u k,u l = 1 1 =. 59 Case 2: k l. First let p = k l, an integer. Then the inner product in 58 becomes u k,u l = = 1 1 i2πpn exp [ ] i2πp n exp = 1 + r + + r 1, 6 i2πp where r = exp. The sum of this finite geometric series is S = 1 r 1 r = 1 ei2πp 1 r = r =. 61 Therefore, u k,u l = δ kl, 62 i.e., the set {u k } is an orthogonal set. Therefore it is a basis in C. In particular, it is the desired basis because of its internal periodicity. Once again, we may view the n index as a spatial index in fact, n/ plays the role of t or x. From this orthogonal basis set {u k }, we construct the orthonormal basis vectors, e k = 1 u k, k =,1,, 1, 63 with components e k [n] = 1 exp i2πkn, n =,1,,

22 Once again, the case k = is special. For 2, e = 1 1,1,,1. 65 Examples: 1. = 2: In this very simple case, one can probably guess the vectors that are generated. First of all, from Eq. 65, for k =, e = 1 2 1,1. 66 For k = 1, Therefore, e 1 [1] = 1 i2π 1 1 exp 2 2 = 1 2 exp iπ = e 1 = 1 2 1, = 3: Once again, the case k = is simple. From Eq. 65, e = 1 3 1,1,1. 69 For k = 1, using Eq. 64, a n = : b n = 1: c n = 2: e 1 [] = 1 3 exp = e 1 [1] = 1 [ i2π exp = 1 1 ] i. 71 e 1 [2] = 1 [ i4π exp = 1 1 ] i. 72 In summary, For k = 2, using Eq. 64, e 1 = 1 1, i, i 73 86

23 a n = : b n = 1: c n = 2: In summary, e 2 [] = 1 3 exp = e 2 [1] = 1 [ i4π exp = 1 1 ] i. 75 e 2 [2] = 1 [ i8π exp = 1 1 ] i. 76 e 2 = 1 1, i, i 77 Discrete Fourier Transform, Version 1 We now employ the orthonormal basis developed above to construct our first version of the DFT. Any element f C will have an expansion of the form In component form, 1 f = f,e k e k. 78 k= 1 f[n] = f,e k e k [n] = k= 1 k= c[k]e k [n], 79 where the c[k] = f,e k denote the Fourier coefficients of f in the e k basis. Let us now examine these coefficients: or c[k] = f,e k = 1 f[n]e k [n], 8 c[k] = 1 1 f[n]exp i2πkn, k =,1,, This relation defines a discrete Fourier transform DFT of f. The components of the vector c = c[1],c[2],,c[ 1] comprise the DFT of the vector f = f[1],f[2],,f[ 1]. Mathematically, we can write c = Ff, 82 87

24 where F : C C denotes the discrete Fourier transform operator on complex -vectors. Important comment: ote the choice of a instead of the before discrete Fourier transform. Unfortunately, there are several closely-related definitions, and it is important to recognize this fact. For this reason, we refer to the above DFT as DFT, Version 1. Let us return to Eq. 81 to show that, indeed, the DFT vector c with components c[k] is - periodic: c[k + ] = = = = f[n]exp i2πk + n f[n]exp i2πkn f[n]exp i2πkn 1 1 f[n]exp i2πkn exp i2πn exp i2πn = c[k]. 83 Eq. 81 is the definition of the discrete Fourier transform implemented in the MAPLE programming language. In MAPLE, the relation would be written as c = FourierTransformf f = InverseFourierTransformc, where we still have to define the inverse DFT. Mathematically, the above formula is elegant because of the following result, f 2 = c 2, 84 where 2 denotes the L 2 norm defined by the complex inner product in C. To see this: f 2 2 = f,f = 1 f[n]f[n] 88

25 = = = = 1 1 k= 1 [ 1 k= 1 l= 1 k= l= 1 k= c[k]c[k] c[k]e k [n] c[k]c[l] [ 1 ][ 1 c[k]c[l] e k,e l ] c[l]e l [n] l= e k [n]e l [n] = c 2 2, 85 ] from which 84 follows. This means that the DFT operator F is norm-preserving, i.e., the norm of c is the norm of f. Inverse DFT, Version 1 Let us now see if we can find a result for the inverse discrete Fourier transform, i.e., given the DFT c, how can we find f, written mathematically as f = F 1 c. 86 In order to invert relation 81, we shall utilize the orthonormality of the e k vectors in 64. For a 1 i2πkm particular value of m {, 1,, 1}, multiply both sides of Eq. 81 by exp and then sum over k: 1 1 i2πkm c[k] exp = 1 k= 1 1 f[n] k= i2πkm n exp. 87 We have already seen earlier that the final summation is δ mn. Thus, for each m, only the term n = m from the sum over n contributes. As a result, we have f[m] = 1 1 i2πkm c[k] exp. 88 k= This relation is true for each m =,1,, 1. It is customary to let n denote the spatial or time variable, so we rewrite the above result as f[n] = 1 1 i2πkn c[k] exp, n =,1,, k= 89

26 These relations comprise the inverse discrete Fourier transform IDFT associated with the DFT in Eq. 81. A closer look at Eq. 89 shows that, in fact, the inverse DFT is nothing more than the expansion of the discrete vector f in terms of the orthonormal basis {e k }. the DFT coefficients c[k] are used to construct the signal elements f[n]. We now summarize the results obtained above: DFT and IDFT, Version 1 c[k] = f[n] = 1 1 f[n]exp i2πkn, k =,1,, 1, 1 1 i2πkn c[k] exp, n =,1,, 1. 9 k= DFT and IDFT, Version 2 A second version of the DFT and its inverse is employed in many mathematics books e.g., the book by Kammler. Unlike the first version, it is not symmetric. But there is a legitimate reason for its definition, since it arises naturally from a discretization of the integrals used to compute Fourier series coefficients. We shall postpone the discussion of this result to another lecture. For the moment, we simply state the second version of the DFT. The DFT, Version 2 is defined as follows, F[k] = 1 1 f[n]exp i2πkn, k =,1,, ote the use of F to denote the DFT: It is customary to let capital letters denote the FT/DFTs of functions. The only difference between this version and Version 1 in Eq. 81 is that the factor in front is 1/ instead of 1/. In the same manner as was done for Version 1, the inverse DFT associated with the above DFT is given by f[n] = 1 k= i2πkn F[k]exp, n =,1,,

27 DFT and IDFT, Version 3 This is the version that appears in most of the signal processing literature e.g. Mallat as well as mathematics books that deal with signal processing applications e.g., Boggess and arcowich. It appears to be the version that is most widely used by research workers in signal and image processing, as witnessed by the fact that it is the version implemented in MATLAB. As such, unless specified otherwise, this will be the version used in this course. The DFT, Version 3 is defined as follows, F[k] = 1 f[n]exp i2πkn, k =,1,, There is no factor in front of the summation. The inverse DFT associated with the this DFT is given by f[n] = 1 1 k= i2πkn F[k]exp, n =,1,, In MATLAB, the DFT and IDFT are denoted as follows, F = fftf, f = ifftf. Using the orthogonality property of the complex exponential functions established earlier, i.e., u k,u l = δ kl, it can be shown a simple modification of the derivation for DFT, Version 1, Eq. 85 that this particular version of the DFT satisfies the relation, f 2 2 = 1 F 2 2, 95 where 2 denotes the L 2 /Euclidean norm on C. From this point onward, we shall omit the subscript 2 from the norm and write, with the understanding that it represents the L 2 norm. Matrix form of DFT You ll note that all the coefficients multiplying the f[n] elements in the DFT of Eq. 93 involve powers of the complex number ω = exp i2π = cos 91 2π isin 2π. 96

28 A closer examination shows that if f and F are written as column -vectors, f and F, respectively, then the DFT relation in 93 may be written in matrix form as F = Ff, 97 where F is an complex matrix having the form ω ω 2 ω 1 1 ω 2 ω 4 ω ω 1 ω 2 2 ω 1 1 The kth entry of the vector F is given by 98 F[k] = f[] + ω k f[1] + ω 2k f[2] + + ω 1k f[ 1] = f[] + f[1]ω k + f[2]ω k 2 + f[ 1]ω k 1, 99 where the second line indicates that F[k] is a polynomial in ω k. This suggests that it may be evaluated recursively, as opposed to computing the terms separately and adding them up. The following is a pseudocode version of Horner s algorithm for computing the entire vector F: z:=1 ω = e i2π/ for k=,1,...,-1 do S:=f[-1] for l=2,3,..., do S:=z*S+f[-l] od F[k]:=S z:=z*ω od In this form, the computation of the DFT F requires 2 complex operations, which translates to 4 2 real operations. For special values of, the procedure can be optimized, utilizing the fact that ω is a root of unity. This is the basis of the fast Fourier transform FFT which we may discuss a little later in the course. 92

29 Lecture 9 Discrete Fourier Transform cont d We now examine the DFT a little further, with the help of some examples. As mentioned earlier, we shall be using the DFT, Version 3 the MATLAB formula summarized again below: DFT and IDFT, Version 3 F[k] = f[n] = 1 1 f[n]exp i2πkn, k =,1,, k= i2πkn F[k]exp, n =,1,, An important note: As we know from before, the DFT and IDFT may be viewed as inner products between appropriate -vectors. In the special case k =, all of the complex exponentials in Eq. 1 are equal to 1. This is because the particular element F[] is the inner product between the -vector f and the unnormalized -vector u = 1,1,,1. As such, F[] = f,u = 1 f[n]. 12 Some examples: For = 4: 1. f = 1,1,1,1, F = 4,,, This illustrates Eq. 12 above. 2. g =,1,,1, G = 2,, 2, 3. h = 1,2,1,2, H = 6,, 2,. 4. a = 1,2,3,4, A = 1, 2 + 2i, 2, 2 2i. Comments: 1. In 1, the signal f is a constant signal, i.e., no variation. This means that the only frequency component is zero frequency, i.e., k =. This is why the first element, k =, corresponding to the constant vector u is the only nonzero component of F. The signal f is orthogonal to all other vectors u k, k. 93

30 2. In 2, the signal g has period 2, i.e., it oscillates with twice the periodicity of signal f period 4. This accounts for the nonzero entry G[2] = ote that the third result is in accordance with the linearity of the DFT: h = f + g implies that H = Fh = Ff + g = Ff + Fg = F + G The result in 4 shows that a real-valued signal can have a complex-valued DFT. 5. Each of the four results above demonstrates the modified Parseval equality for the DFT, Version 3 mentioned earlier, i.e., f 2 = 1 F Some more complicated examples: 5. We consider the function fx = cos2x defined on the interval x 2π. From this function we construct = 256 equally-spaced samples, f[n] = fx n = cos2x n, x n = 2πn, n =,1,, The samples are plotted on the left in the figure below Fk x k Sampled signal f[n] = cos2x n, n =,1,,255 and magnitudes F[k] of its DFT. umerically, we find that all DFT coefficients F[k] are zero, except for two elements: F[2] = 128, F[254] =

31 A plot of the magnitudes F[k] of the DFT coefficients is presented on the right in the figure below. The nonzero entry for F[2] picks out the k = 2 frequency of the signal. We ll see later that the F[254] component does likewise. umerically, we also find that and Thus, Eq. 95 is satisfied. 255 f 2 = f[n] 2 = 128 to two decimals, 17 1 F 2 = F[2]2 + F[4] 2 = = ow consider the function fx = sin2x defined on the interval x 2π. From this function we construct = 256 equally-spaced samples, f[n] = fx n = sin2x n, x n = 2πn, n =,1,, The samples are plotted on the left in the figure below f[n] Fk x k Sampled signal f[n] = sin2x n, n =,1,,255 and magnitudes F[k] of its DFT. umerically, we find that all DFT coefficients F[k] are zero, except for two elements: F[2] = 128i, F[254] = 128i. 11 Of course, there is a similarity between this spectrum and that of Example 5 in that the peaks coincide at k = 2 and 254, corresponding to the common frequency k = 2. As such, a plot of the magnitudes F[k] of the DFT coefficients, presented on the right in the figure below, is identical to the corresponding plot of Example 5. 95

32 On the other hand, the DFT coefficients for the sin2x function are complex. In fact, they are purely imaginary. The coefficient F[2] for the sinx function is obtained from the coefficient of the cosx function by multiplication by i = e iπ/2. This might have something to do with the fact that the sinx function is a shifted version of the cosx version. More on this later. The moral of the story is that the magnitudes F[k] do not contain all of the information about a signal. If we write F[k] = F[k] e iφ k, 111 then the phases φ k also contain information about the signal and cannot be ignored. umerically, we also find that 255 f 2 = f[n] 2 = 128 to two decimals, 112 and 1 F 2 = = Thus, Eq. 95 is satisfied. 7. We now consider the function fx = sin2x+5sin5x defined on the interval x 2π. We have added a higher-frequency term to the function of Example 5. From this function we again construct = 256 equally-spaced samples, f[n] = fx n = sin2x n + 5sin5x n, x n = 2πn, n =,1,, The samples are plotted on the left in the figure below. umerically, we find that all DFT f[n] Fk x k Sampled signal f[n], n =,1,,255 and magnitudes F[k] of the DFT, for Example 7. 96

33 coefficients F[k] are zero, except for four elements: F[2] = 128i, F[254] = 128i, 115 as expected, corresponding to the sin2x component, and F[5] = 64i, F[251] = 64i, 116 corresponding to the sin5x component. From the linearity of the DFT, the DFT of the sum of these two functions is the sum of their DFTs. Also note that the ratio of amplitudes of these two sets follows the 1 : 5 ratio of the sin2x and sin5x components. A plot of the magnitudes F[k] of the DFT coefficients is presented on the right in the figure below. umerically, we also find that and once again in accordance with Eq f 2 = f[n] 2 = F 2 = = 3328, Let us now generalize the results from the previous two examples. Suppose that we have the complex-valued function fx = expik x, defined on the interval [,2π], with k an integer. For the moment, we assume that k {,1,2,, 1}. From this function, we extract equally-spaced samples at the sample points x n = 2πn/, n =,1,, 1, i.e, i2πk n f[n] = fx n = expik x n = exp,n =,1,, The function fx is 2π-periodic. But what is its DFT? By definition, its DFT is given by F[k] = = 1 1 f[n]exp i2πkn i2πk n exp exp i2πkn. 12 ow recall from the previous lecture that the discrete exponential functions u k, u k [n] = 1 97 exp i2πkn, 121

34 an orthogonal set on C, i.e., u k,u l = δ kl. 122 This means that the F[k] = in Eq. 12 unless k = k. In other words, the -point DFT of the exponential function expik x sampled on [,2π] is given by F[k] = δ kk. 123 The DFT consists of a single peak of magnitude at k = k. But wait just one minute! The above result applies to the case k {,1,2,, 1}. What happens if it is not in the set of frequencies {,1,2,, 1} covered by the DFT? For example, if = 256, what happens if k = 26? Will the function be oscillating too quickly to be detected? The answer is o, it will be detected. Somehow, one gets the feeling that everything here happens modulo, because of the periodicity of the vectors. And that is what happens in frequency space as well. Let us replace k in Eq. 12 with k +. Then the RHS becomes 1 i2πk + n exp exp i2πkn = = = i2πk n exp i2πk n exp i2πk n exp exp i2πkn exp i2πkn exp exp i2πkn i2πkn exp i2πkn. 124 In other words, the result in Eq. 12 is unchanged. Therefore the same result holds for k +p, where p is an integer. This implies that a peak will show up at k = k mod. So the final result is: The -point DFT of the sampled function expik x, x 2π, i.e., f[n] given in Eq. 119, is given by a single peak:, k = k mod, F[k] =, otherwise

35 ote: We actually proved this result earlier, when we showed that the discrete Fourier coefficient vectors c = c[1],c[2],,c[ 1] associated with the orthonormal basis e k are -periodic. But it doesn t hurt to revisit this result. From this property, we may now go back and verify that the calculations of Examples 5-7 of the previous lecture, involving sine and cosine functions, are correct. 1. Since cosk x = 1 2 eik x e ik x, 126 it follows, from the linearity of the DFT, and the modulo property derived above, that the -point DFT of cosk x consists of two peaks of height /2, i.e., 2, k = k, F[k] = 2, k = k,, otherwise. 127 The peak at k = k comes from the modulo property. The second exponential in Eq. 126 would produce a peak at k = k which, in turn, because of the -periodicity of the DFT, produces a peak at k = k + = k. ote that this result is in agreement with the computation in Example 5 above. 2. sink x = 1 2i eik x 1 2i e ik x, 128 it follows, once again from the linearity of the DFT, and the modulo property derived above, that the -point DFT of sink x consists of the following two peaks, 2i = 2 i, k = k, F[k] = 2i = 2 i, k = k,, otherwise. 129 ote that these peaks have the same magnitudes as for the cosine case, but that they are now complex. Moreover, the two peaks of the DFT of the sine function are complex conjugates of each other. More on this later. For the moment, we note that this result is in agreement with the computation in Example 6 above. 99

36 3. Finally, notice that if we add up the DFTs of the sine and cosine function appropriately, we retrieve the DFT of the exponential, i.e., Fcosk x + isink x = Fexpik x. 13 The peaks of the cosine and sine at k = k constructively interfere whereas their peaks at k = k destructively interfere. The result is a single peak of height at k = k. We now consider a slightly perturbed version of Example 6 of the previous lecture, namely the function fx = sin2.1x defined on the interval x 2π. From this function we construct = 256 equallyspaced samples, f[n] = fx n = sin2.1x n, x n = 2πn, n =,1,, The samples are plotted on the left in the figure below. ote that this signal is not 2π-periodic, but the sampling and resulting DFT produces a 2π-periodic extension. As such, there is a significant jump between f[255] and f[256] = f[]. This time, we find that the DFT spectrum of coefficients F[k] is not as simple as in the first two examples. First of all, with the exception of F[] , all DFT coefficients are complex, i.e., have nonzero imaginary part. A plot of the magnitudes F[k] of the DFT coefficients is presented on the right in the figure below. There is still a dominant peak at k = 2, but it is not a singular peak it is somewhat diffuse f[n] Fk x k Sampled signal f[n] = sin2.1x n, n =,1,,255 and magnitudes F[k] of the DFT, for Example 9. In order to show the diffuseness of the DFT spectrum, the coefficients are plotted on a different scale so that the enormous peaks at k = 2 and 254 do not mask their behaviour as in the previous plot. 1

37 2 15 Fk k Plot of magnitudes Fk of DFT of sin2.1x n signal of Example 9, magnified to show the diffuse structure around the dominant peaks at k = 2 and 254. Ideally, the DFT would like to place a peak at the frequency k = 2.1, but it doesn t exist. As such, the dominant peaks are found at k = 2 and 254. But all other frequencies are need to accomodate this nonexistent or irregular frequency note that their contribution decreases as we move away from the peaks. If this appears to be a rather bizarre phenomenon, just go back and think about the Fourier sine series of this function, i.e., sin2.1x = b k sinkx. 132 k=1 In fact, the coefficients b k can be computed rather easily, and one observes that they produce a somewhat diffuse Fourier spectrum that peaks at k = 2. The reader may wish to examine the effect of further perturbing the frequency of the sampled signal, i.e., the function fx = sin2 + ǫx as ǫ is increased. For example, will the k = 3 and 253 components of the DFT increase in magnitude. And for ǫ >.5, does k = 3 take over in magnitude? 11