Communication Technology Laboratory Prof. Dr. H. Bölcskei Sternwartstrasse 7 CH-809 Zürich Mathematics of Information Spring semester 08 Solution to Homework Problem Overcomplete expansion in R a) Consider the vectors e, e 0 0, e 3 e e. Our goal is to find vectors ẽ, ẽ, ẽ 3 such that ] x x, ẽ e + x, ẽ e + x, ẽ 3 e 3 [e e e 3 A ẽt ẽ T ẽ T 3 In order to find these vectors, we are looking for a right inverse of the matrix A. One possible right inverse can be found by noting that A A T (AA T ) I. right inverse x. First we calculate and the inverse and finally The vectors ẽ, ẽ, ẽ 3 AA T 0 0 0 0 (AA T ) 3 [ ] ẽ T ẽ T A T (AA T ). ẽ T 3 3 are given by ẽ /3, ẽ /3 /3, ẽ /3 3. /3 /3
Comparing to the given set of vectors ẽ, ẽ, ẽ 3 we find ẽ e, ẽ 0 e 3, ẽ 3 e. 0 It should be emphasized that the right inverse is not unique: the system of equations a b 0 0 c d 0 0 e f B has infinitely many solutions of the form λ γ B λ + γ λ γ for any λ, γ R. Any such matrix B is a valid right inverse of A, which generates in general a different set of vectors ẽ, ẽ, ẽ 3. b) Assume that x can be written in the form x x, e ẽ + x, e ẽ ] [ẽ [ e ẽ T 0 e T x ] } {{ } A e T e T x. A is a square, non-singular matrix and has, therefore, a unique inverse A [ ] [ẽt ] 0 0 / / ẽ T e e, e e. We conclude that x can be represented in the form and this expansion is unique. x x, e ẽ + x, e ẽ () Problem Equality in the Cauchy-Schwarz inequality Let us assume that x y and x, y. Then from x y, y x, y y 0, we conclude that x y is orthogonal to y. Applying Pythagoras theorem to x (x y) + y, we can write x x y + y,
which implies that x y 0 and hence x y. ow, if x and y, we can re-write x, y x y as x x, y y Let us define We then have x φ arg x, y. y e iφ x x, y, y and by defining x e iφ x/ x and ỹ y/ y. We recover the first case, where x ỹ and x, ỹ. We then have x ỹ, which gives where c e iφ x / y. iφ x x e y y cy, Problem 3 Parallelogramm law a) Let x, y X. We have the following: Likewise, we have x + y x + y + x, y + y, x x + y + x, y + x, y By adding (??) and (??), we obtain b) Let us assume that x + y + Re { x, y }. () x y x + y Re { x, y }. (3) x + y + x y x + y. x + y + x y x + y (4) holds for all x, y X. As suggested in the problem statement, we suppose that X is a real normed space and for all x, y X, we set x, y x + y x y. Let us show that, defines an inner product on X. Symmetry: We can directly see from the definition of, that x, y y, x holds for all x, y X. Bilinearity: Let x, y, z X. By definition of,, we have x + y, z x + y + z x + y z. (5)
Using (??), it holds that x + y + z + x y x + z x + z + x x + z y + z + y y + z Subtracting (??) and (??) to (??), we obtain the equality + y + z (6) + z (7) + z. (8) x + y + z x + z + y + z x + y + x + y z, which, combined with (??), gives x + y, z x + z z + y + z z x + y x y + x + y x,z + x y,z + y x, z + y, z x + y x y + x + y 0 thanks to (??) x, z + y, z. Thus, we have x + y, z x, z + y, z. Positivity: For all x X, we have x, x x x x x 0. Since is a norm on X, it is then true that x, x 0 implies x 0. Homogeneity: Let x, y X. We can show by induction that nx, y n x, y holds for all n. Indeed, both hand sides equal 0 for n 0, and if nx, y n x, y holds for n, we can use the bilinearity of, previously shown to write that In addition, we have that (n + )x, y nx, y + x, y (n + ) x, y. x, y x + y x y x y x + y x y x, y, x + y x y x + y x y which implies that nx, y n x, y holds for all n Z. For r p/q Q, we then have q rx, y px, y p x, y, which amounts to rx, y r x, y. The norm is continuous on X (because it is Lipschitz continuous with Lipschitz constant ). Hence, by density of Q in R, we can
conclude that λx, y λ x, y for all λ R. We can verify that any two elements of the Banach space l (Z) satisfy the parallelogram law: for all sequences u {u k } k Z and v {v k } k Z, we have u + v + u v + k + k u k + v k + + k u k v k u k + v k + Re{u k v k } + u k + v k Re{u k v k } u + v. From., we can conclude that the norm defined on l (Z) is induced by an inner product and that l (Z) is hence a Hilbert space. To show that l (Z) is not an inner product space, we define the two sequences u {u k } k Z and v {v k } k Z such that u k {, k 0 0, otherwise and v k {, k 0, otherwise. We have that u + v u v and u v. Consequently, the parallelogram law does not hold, implying that l (Z) is not an inner product space (using.), and thus, it is not a Hilbert space. Problem 4 Projection on orthogonal subspaces a) Let y, y S such that x y x y d min x z. z S The parallelogram law applied to (x y ) and (x y ) implies that 4d x y + x y x y y + y y 4 x (y + y )/ + y y 4d + y y, (9) where we used the fact that (y + y )/ S in the last step. Therefore, y y 0, which is only possible if y y. We showed the proposition for a real vector space, but it also holds for a complex vector space.
b) Let y, y S such that (x y ) S and (x y ) S. Then, y y S, and y y, z y, z y, z y, z y, z + x, z x, z x y, z x y, z 0, z S. In particular, we obtain for z y y that y y y y, y y 0, which is only possible if y y. Problem 5 Unconditional convergence The set S {x k k } is orthogonal. Therefore, the sum x k k converges unconditionally if and only if the sum Since it converges unconditionally. x k <. k x k k k <, k Problem 6 Define the basis functions Discrete Fourier Transform (DFT) as a signal expansion They form an orthonormal system as shown by e k, e l e k (n) e iπ k n, k 0,,...,. e k (n)e l (n) n0 e iπ(k l) e i(k l) π k0 n0 e iπ k n e iπ l n 0, k l, k l. k0 e i π (k l)n
We have functions in C that form an orthonormal system, thus, they form an orthonormal basis. Therefore any signal can be expressed as where f, e k n0 f(n) k0 f, e k e k (n) f(n)e k (n) f(n)e iπ k n f(k). n0 Therefore we see that the inverse of the DFT is given by f(n) k0 f(k)e iπ k n.