We will now focus on underdetermined systems of equations: data acquisition system

Transcription

1 l 1 minimization We will now focus on underdetermined systems of equations: # samples = resolution/ bandwidth data acquisition system unknown signal/image Suppose we observe y = Φx 0, and given y we attempt to estimate x 0 by applying the pseudo-inverse of Φ to y we are implicitly solving the program min x x 2 subject to Φx = y. Of course, we will recover x 0 exactly only under very special circumstances, namely that x 0 is in Range(Φ ); if x 0 = Φ α for some α R M, then Φ (ΦΦ ) 1 y = Φ (ΦΦ ) 1 Φx 0 = Φ (ΦΦ ) 1 ΦΦ α = Φ α = x 0. So if x 0 lies in a particular M-dimensional subspace of R N, it can be recovered from observations through the M N matrix Φ. 1

2 Yesterday, we hinted that a different variational framework, one based on l 1 minimization instead of l 2 minimization, would allow us to recover sparse vectors. Given y, we solve min x x 1 subject to Φx = y. (1) It is our goal today to formalize the conditions under which this recovery procedure is effective. Necessary and sufficient conditions for l 1 recovery Let x 0 be a vector supported on a set Γ {1, 2,..., N}, and let y = Φx 0. It is clear that x 0 is a solution to (1) if and only if It is always true that x 0 + h 1 x 0 1 h with Φh = 0. (2) x 0 + h 1 x 0 1 = γ Γ γ Γ ( x 0 [γ] + h[γ] x 0 [γ] ) + γ Γ c h[γ] sgn(x 0 [γ])h[γ] + γ Γ c h[γ] since Thus (2) holds when a + b a sgn(a)b for all a, b R. γ Γ sgn(x 0 [γ])h[γ] γ Γ c h[γ] for all h Null(Φ). (3) This condition is also necessary, since if there is an h Null(Φ) with h[γ] < sgn(x 0 [γ])h[γ], γ Γ c γ Γ 2

3 then the same is true for ɛh for all ɛ >, and Φ(x 0 + ɛh) = y, and x 0 + ɛh 1 = γ Γ < γ Γ γ Γ x 0 [γ] + ɛh[γ] + γ Γ c ɛh[γ] x 0 [γ] + ɛh[γ] ɛ sgn(x 0 [γ])h[γ] x 0 [γ] for some small enough ɛ > 0 = x 0 1 It is interesting to note that given the observation matrix Φ, our ability to recover a vector x 0 is determined only by 1. the set Γ on which x 0 is supported (the locations of its active elements ), and 2. the signs of the elements on this set. The magnitudes of the entries are not involved at all. Geometrically, the support set coupled with the sign sequence specifies the facet of the l 1 ball on which x 0 lives:!"#$l 1!%&'(,)$L 1 (%-,.*%+ 01&(2(.$(%-,.*%+$*3 random orientation dimension N-M 3

4 Duality and optimality We can get a more workable sufficient condition for exact recovery of x 0 by looking at the dual of the convex program min x x 1 subject to Φx = y. (4) Let s start by considering a general optimization program with linear equality constraints: min x f(x) subject to Φx = y. (5) The Lagrangian for this problem is L(x, v) = f(x) + v (Φx y) If f is differentiable, then x is a solution to (5) if it is feasible, Φx = y, and there exists a v R M such that x L(f, v) = x f(x) + Φ v = 0. We are interested in the particular functional f(x) = x 1, which is not differentiable but is convex. Fortunately, there is an easy modification to the condition above in this case x is a solution to (5) if it is feasible and there exists a v R M such that Φ v is a subgradient of f at x. A vector u R N is a subgradient of a function f at x 0 if it is normal to a supporting hyperplane ; that is if f(x) f(x 0 ) + u (x x 0 ). If f is differentiable at x 0, then f(x 0 ) is the only subgradient. 4

5 ubgradients efinition To make this concept more concrete, consider the (very relevant) one dimensional example f(t) = t. is a subgradient of f at x 0 if for all x f (x) f (x 0 )+u (x x 0 ) f is differentiable at x 0, the only subgradient is rf (x 0 ) ubgradients of f (t) = t, t 2 R ( {subgradients} = {sgn(t)} t 6= 0 {subgradients} =[ 1, 1] t = 0 In this case, the set of subgradients is simply the derivative (±1) for t away from the origin, and all slopes between 1 and +1 at the origin: { sgn(t) t 0 {subgradients} = [ 1, 1] t = 0. In R N, for f(x) = x 1, the subgradients of f at x 0 collection of vectors u such that are the u[γ] = sgn(x 0 [γ]), γ Γ u[γ] 1, γ Γ c, where Γ is the support set of x 0. 5

6 Given an M N matrix Φ and a vector x 0 R N supported on Γ, set y = Φx 0. Then x 0 is a solution to (4) if there exits a v R M such that (Φ v)[γ] = sgn(x 0 [γ]), γ Γ (Φ v)[γ] 1, γ Γ c. In addition, if Φ is injective on the set of all vectors supported on Γ and we can find a v such that the second condition is (Φ v)[γ] < 1, γ Γ c then x 0 is the unique solution. Choosing a particular dual vector Our recovery conditions boxed above simply ask that we be able to find one such v (there could be many). Many of the results in the fields of sparse recovery and compressed sensing have narrowed down the condition down by simply testing a prescribed vector. Let Φ Γ = the M Γ submatrix containing the columns of Φ indexed by Γ and let We set z 0 R Γ contain the signs of x 0 on Γ. u 0 = Φ Φ Γ (Φ ΓΦ Γ ) 1 z 0. (6) Now sufficient conditions for x 0 to be the unique minimizer of (4) are 6

7 1. Φ ΓΦ Γ is invertible. If this is true, then the expression above for u 0 is well-behaved, and b construction we will have u 0 [γ] = sgn(x 0 [γ]); 2. If 1) holds, then we need u 0 [γ] < 1. Note on complex vectors: Almost everything we have said so far about l 1 minimization extends in a straightforward manner to complex-valued vectors. First, it is worth mentioning that if x C N, then x 1 = N x[n] = n=1 N n=1 Re {x[n]} 2 + Im {x[n]} 2. In this case, the l 1 minimization program can no longer be recast as a linear program, but rather is what is called a sum of norms program (which is a particular type of second order cone program ). This type of problem, however, is not too much more difficult to solve from a practical perspective. The sufficient conditions for recovery are the same, but now we take sgn(z) to be the phase of the complex number z. That is, if z = Ae jθ, then sgn(z) = θ. 7

8 Example: Spikes and Sines Dictionary Let us return to the Φ created by concatenating the spikes orthobasis and the Fourier orthobasis: Φ = [ I F ]. Here the matrix is n 2n. Suppose that α 0 C 2n is supported on a set Γ = T Ω where as before T denotes the spike support and Ω the sine support. Yesterday, in establishing the Donoho-Starck uncertainty principle, we saw that we could write Thus, if Φ ΓΦ Γ = I + G, where G for some 0 < δ < 1, we have T Ω δn, (Φ ΓΦ Γ ) δ. T Ω n. So our first sufficient condition holds when T Ω < n, which is implied by Γ = T + Ω < 2 n. For the second condition, construct u 0 as in (6), where now z 0 are the phases of α 0 on Γ. Then for a particular entry γ Γ c, u 0 [γ] = Φ γφ Γ (Φ ΓΦ Γ ) 1 z 0 = w γ, z 0 8

9 where Φ γ is the column of Φ corresponding to γ, and w γ is the vector w γ = (Φ ΓΦ Γ ) 1 Φ ΓΦ γ. Using Cauchy-Schwarz, u 0 [γ] w γ z 0 (Φ ΓΦ Γ ) 1 Φ ΓΦ γ 2 T + Ω. The entries in the vector Φ ΓΦ γ are either equal to zero or 1/ n, and so u 0 [γ] 1 1 δ T + Ω T + Ω. n When can we guarantee that the expression on the right less than 1? We know that we will at least need Γ = T + Ω n. In this case, T Ω n/4, so we can take δ = 1/4, and so if then Γ = T + Ω 3 n 4 max γ Γ c u 0[γ] < 1 and α 0 will be the minimizer to (4). 9

10 Example: Dictionaries with small coherence Now suppose that Φ is an M N matrix with normalized columns, and coherence Φ γ 2 = 1, γ = 1, 2,..., N µ = max 1 γ 1,γ 2 N γ 1 γ 2 Φ γ1, Φ γ2. The quantity µ is essentially a measure of how closely aligned any two columns of Φ are. Let Γ be a fixed subset of {1, 2,..., N}, and x 0 be a vector supported on Γ with sign sequence z 0. For the first optimality condition note that we can write Φ ΓΦ Γ as Φ ΓΦ Γ = I + G where each entry of the Γ Γ matrix G is less than or equal to µ. We have G G F = G j,k 2 j,k µ Γ, and so we can guarantee Φ ΓΦ Γ is invertible when Γ 1 µ. To check the second condition, we have for γ Γ c u 0 [γ] (Φ ΓΦ Γ ) 1 Φ ΓΦ γ 2 z µ Γ Φ ΓΦ γ 2 Γ. 10

11 Since γ Γ, all the entries of Φ ΓΦ γ will have magnitude less than µ, and so Φ ΓΦ γ 2 µ Γ, and so Thus u 0 [γ] µ Γ 1 µ Γ. Γ < 1 2µ ensures that x 0 will be recoverable from observations y = Φx 0. 11