EECS 6A Fall 7 Designing Information Devices and Systems I Discussion 3B. Orthogonal Matching Pursuit Lecture Orthogonal Matching Pursuit (OMP) algorithm: Inputs: A set of m songs, each of length n: S = { S, S,..., S m } An n-dimensional received signal vector: r The sparsity level k of the signal Some threshold, th. When the norm of the signal is below this value, the signal contains only noise. Outputs: A set of songs that were identified, F, which will contain at most k elements A vector x containing song messages (a,a,...), which will be of length k or less An n-dimensional residual y Procedure: Initialize the following values: y = r, j =, k, A = [ ], F = {/} while (( j k) and ( y th)) : (a) Cross-correlate y with the shifted versions of all songs. Find the song index i and the shifted version of the song, S i N, with which the received signal has the highest correlation value. (b) Add i to the set of song indices F. (c) Column concatenate matrix A with the correctly shifted version of the song: A = [ ] A S i N (d) Use least squares to obtain the message value: x = (A T A) A T r (e) Update the residual value y by subtracting: y = r A x (f) Update the counter: j = j + EECS 6A, Fall 7, Discussion 3B, Last Updated: 7--6 6:
. Orthogonal Matching Pursuit Let s work through an example of the OMP algorithm. Suppose that we have a vector x R 4. We take 3 measurements of it, b = m T x = 4, b = m T x = 6, and b 3 = m T 3 x = 3, where m, m and m 3 are some measurement vectors. We are given that x is sparse and only has non-zero entries. In particular, M x b m T m T x b m T 3 x x x 3 x 4 where exactly of x to x 4 are non-zero. Use Orthogonal Matching Pursuit to estimate x to x 4. (a) Why can we not solve for x directly? We cannot solve for x directly because we have three measurements (or equations) but four unknowns. Since our system is underdetermined, we cannot solve for the unique x directly. (b) Why can we not apply the least squares process to obtain x? Recall the least squares solution: ˆx = (M T M) M T b. M T M is only invertible if it has a trivial null space, i.e., if M has a trivial null space. However, in this case, M is a 3 4 matrix, so there is at least one free variable, which means that its null space is non-trivial. Therefore, M T M is not invertible, and we cannot use least squares to solve for x. (c) Compute the inner product of every column with the b vector. Which column has the largest inner product? This will be the first column of the matrix A. Why are we using the inner product instead of the correlation? Does it make sense to shift the columns of A?, =, = 3, =, = 6 EECS 6A, Fall 7, Discussion 3B, Last Updated: 7--6 6:
The third column has the largest inner product with, so A =. We still column-concatenate A with the original column vector since the received signal contains a scalar multiple of the original unscaled column vector. We are using the inner product here instead of the correlation because there is no delay here; it does not make sense to shift A s columns because the columns of M are the only possible vectors we receive. Notice that each of the column vectors has different norm, so we have to take this into consideration. Therefore, when we want to find the column with the largest inner product, we have to normalize each of the column vectors, so that the inner products are independent of the norms of the column vectors., =, = 3, = 6, = 6 The first column has the largest inner product with, so A =. (d) Now find the projection of b onto the columns of A. Use this to update the residual. ˆx = (A T A) A T b = [ ] [ ] = 8 = 3 proj Col(A) b = A ˆx = 3 3 = 3 3 r = b proj Col(A) b = 3 = EECS 6A, Fall 7, Discussion 3B, Last Updated: 7--6 6: 3
ˆx = (A T A) A T b = [ ] [ ] = proj Col(A) b = A ˆx = = r = b proj Col(A) b = = (e) Now compute the inner product of every column with the new residual vector. Which column has the largest inner product? This will be the second column of A., =, =, =, = 3 The fourth column has the largest inner product with, so A =. Again, the column vectors all have a different norm, so we have to normalize them when we are finding the column vector with the largest inner product with the residual vector., =, = 3, = EECS 6A, Fall 7, Discussion 3B, Last Updated: 7--6 6: 4
, = The second column has the largest inner product with, so A =. (f) Project b onto the columns of A to find x. [ ] ˆx = (A T A) A T b = [ ] = Therefore, x 3 = and x 4 =, so x =. [ ] ˆx = (A T A) A T b = [ ] [ ] = 3 Therefore, x = and x = 3, so x = 3. [ ] EECS 6A, Fall 7, Discussion 3B, Last Updated: 7--6 6:
3. One Magical Procedure (Fall Final) Suppose that we have a vector x R and an N measurement matrix M defined by column vectors c,..., c, such that: M x = c c x b We can treat the vector b R N as a noisy measurement of the vector x, with measurement matrix M and some additional noise in it as well. You also know that the true x is sparse it only has two non-zero entries and all the rest of the entries are zero in reality. Our goal is to recover this original x as best we can. However, your intern has managed to lose not only the measurements b but the entire measurement matrix M as well! Fortunately, you have found a backup in which you have all the pairwise inner products c i, c j between the columns of M and each other as well as all the inner products c i, b between the columns of M and the vector b. Finally, you also know the inner product b, b of b with itself. All the information you have is captured in the following table of inner products. (These are not the vectors themselves.), c c c 3 c 4 c b c c c 3 c 4 6 c b 9 (So, for example, if you read this table, you will see that the inner product c, c 3 =, that the inner product c 3, b =, and that the inner product b, b = 9. By symmetry of the real inner product, c 3, c = as well.) Your goal is to find which entries of x are non-zero and what their values are. (a) Use the information in the table above to answer which of the c,..., c has the largest magnitude inner product with b. Reading off the table, c 4 has the largest inner product with b. (b) Let the vector with the largest magnitude inner product with b be c a. Let b p be the projection of b onto c a. Write b p symbolically as an expression only involving c a, b, and their inner products with themselves and each other. The magnitude of the projection is c a, b c a, and the direction of the projection is c a c a. Thus: c a, b b p = c a, c a c a EECS 6A, Fall 7, Discussion 3B, Last Updated: 7--6 6: 6
(c) Use the information in the table above to find which of the column vectors c,..., c has the largest magnitude inner product with the residue b b p. Hint: The linearity of inner products might prove useful. The inner product of b b p with a vector c i is: c a, b b b p, c i = b, c i c a, c a c a, c i Finding the numerical values of the inner products: b b p, c b b p, c b b p, c 3 b b p, c 4 b b p, c 4 Thus the vector with the highest inner product with the residue is: c. (d) Suppose that the vectors we found in parts (a) and (c) are c a and c c. These correspond to the components of x that are non-zero, that is, b x a c a +x c c c. However, there might be noise in the measurements ] [ xa b, so we want to find the linear least squares estimates x a and x c. Write a matrix expression for x c in terms of appropriate matrices filled with the inner products of c a, c c, b. ] [ xa We use least squares to solve for. Let A = [ ] c x a c c. Using the least-squares formula, c [ xa x c ] = (A T A) A T b [ ] ca, c = a c a, c c c a, b c c, c a c c, c c c c, b (e) Compute the numerical values of x a and x c using the information in the table. Substituting the previous expression with values from the table, we get: x = 3,x 4 = 4 3. ] [ x4 = x = [ c4, c 4 c 4, c c, c 4 c, c [ ] [ 6 ] c 4, b c, b ] = [ ][ ] 6 3 = [ 3 3 8 3 ] EECS 6A, Fall 7, Discussion 3B, Last Updated: 7--6 6: 7