One condition for all: solution uniqueness and robustness of l 1 -synthesis and l 1 -analysis minimizations

Transcription

1 One condition for all: solution uniqueness and robustness of l 1 -synthesis and l 1 -analysis minimizations Hui Zhang Ming Yan Wotao Yin June 9, 2013 The l 1-synthesis and l 1-analysis models recover structured signals from their undersampled measurements. Abstract The solution of the former model is often a sparse sum of dictionary atoms, and that of the latter model often makes sparse correlations with dictionary atoms. This paper addresses the question: when can we trust these models to recover specific signals? We answer the question with a necessary and sufficient condition that guarantees the recovery to be unique and exact and that also guarantees the recovery is robust in presence of measurement noise. The condition is one-for-all in the sense that it applies to both of the l 1-synthesis and l 1-analysis models, and to both of their constrained and unconstrained formulations. Furthermore, a convex infinity-norm program is introduced for numerically verifying the condition. The comparison with related existing conditions are included. Keywords: exact recovery, robust recovery, l 1 -analysis, l 1 -synthesis, sparse optimization, compressive sensing 1 Introduction Let x R n be a signal of interest. This paper studies when l 1 minimization can uniquely and robustly recover x from its linear measurements b = Φx + w, (1) where Φ R m n is a certain matrix and w R m is noise. We focus on the compressive setting m n. The results of this paper cover the following l 1 minimization formulations: minimize Ψ T x 1, subject to Φx = b, (2a) x minimize Φx b λ Ψ T x 1, x (2b) minimize Ψ T x 1, subject to Φx b 2 δ, (2c) x where δ, λ are positive parameters. When Ψ = Id, the identify matrix, models in (2) are referred to as the l 1 (or more generally, l 1 -synthesis) models. When Ψ Id, they are referred to as the l 1 -analysis models (see [6] for a recent overview). Department of Mathematics and Systems Science, College of Science, National University of Defense Technology, Changsha, Hunan, China, hhuuii.zhang@gmail.com Department of Computational and Applied Mathematics, Rice University, Houston, Texas 77005, USA. s: yanm@ucla.edu and wotao.yin@rice.edu 1

2 In synthesis models, the signal of interest is synthesized as x = Dc, where D is a certain dictionary and c has the sparse coefficients. The analysis model recently attracts a lot of attention. It is assumed in [3, 14, 13] that the underlying signal makes sparse correlations with the columns (atoms) in a dictionary ˉD, i.e., ˉDT x is sparse. The signal recover model is (2b) with Ψ set to ˉD. For both synthesis and analysis models, one is interested in when the recovery is successful, that is, the solution is unique and the solution error is proportional to the amount of noise. There are several nonuniversal (applied to specific sparse signals) and universal (applied to all sparse signals) conditions addressing questions in various forms for l 1 -synthesis minimization; examples include the non-universal dual certificate condition [7] and the RIPless property [4], and universal conditions such as the restricted isometry principle [2], the null space condition [5], the spherical section property [20], and others. Since l 1 -analysis minimization takes a more general form than l 1 -synthesis minimization, some of the above conditions have been extended to the analysis case; recent works [9, 18, 11, 16, 15, 12] have made significant contributions. Regarding a specific signal x, this paper establishes a necessary and sufficient condition that guarantees the unique solution of any model in (2) and also that the solutions of models (2b) and (2c) are robust to any noise added to b. A method based on l minimization that verifies the condition is presented. In addition, the proposed condition is compared to other conditions in the literature, most of which are stronger than ours and are thus sufficient but not necessary. Certain parts of our proofs are inspired by [1, 9, 10, 19]. Notation. We equip R n with the canonical scalar product, and Euclidean norm 2. We let return the cardinality if the input is a set or the absolute value if the input is a number. For any x R n, supp(x) = {k : 1 k n, x k 0} is the index set of the non-zero entries of x. sign(x) is the vector whose ith entry is the sign of x i, taking a value among +1, 1, and 0. For any p 1, the l p -norm of x R n is ( n ) 1/p x p = x i p, i=1 its l 0 - norm is x 0 = supp(x), and its l -norm is x = max{ x i : i = 1,, n}. For x R n and I {1, 2,, n}, x I denotes the vector formed by the entries x i of x for i I, and I c is the complement of I. Similarly, A I is the submatrix formed by the columns of A indexed by I. A T is the transpose of A. We use A T I for the transpose of submatrix A I, not a submatrix of A T. For square matrix A, λ max (A) and λ min (A) denote its largest and smallest eigenvalues, respectively, Cond(A) denotes its condition number, and A denotes its spectral norm. The null and column spaces of A are denoted by Ker(A) and Im(A), respectively. Outline. The rest of the paper is organized as follows. Section 2 states the main results of this paper. Section 3 reviews several related results. Section 4 discusses condition verification. Proofs for the main results are given in sections 5 and 6. 2 Main condition and results 2.1 Main condition Condition 1. Given ˉx R n, index sets I = supp(ψ T ˉx) {1,, l} and J = I c satisfy (1) Ker(Ψ T J ) Ker(Φ) = {0}; (2) There exists y R l such that Ψy Im(Φ T ), y I = sign(ψ T I ˉx), and y J < 1. Part (1) of the condition says that there does not exist any nonzero Δx satisfying both Ψ T J ˉx = ΨT J (ˉx+Δx) and Φˉx = Φ(ˉx+Δx). Otherwise, there exists a nonempty interval I = [ˉx αδx, ˉx+αΔx] for some sufficiently small α > 0 so that Φx = Φˉx and Ψ T x 1 is linear for x I; hence ˉx cannot be the unique minimizer. Part 2

3 (2) states the existence of a strictly-complementary dual certificate y. To see this, let us check a part of the optimality conditions of (2a): 0 Ψ 1 (Ψ T x) Φ T β, where vector β is the Lagrangian multipliers; we can rewrite the condition as 0 = Ψy Φ T β, with y 1 (Ψ T x), which translates to y I = sign(ψ T I x) and y J 1. This y certifies the optimality of ˉx. For solution uniqueness and/or robustness, we shall later show that y J < 1 strictly is needed. A variant of Condition 1 is given as follows, which is equivalent to Condition 5 below from [9]. Condition 2. Given ˉx R n, let I = supp(ψ T ˉx). There exists a nonempty index set J I c such that the index sets I, J and K = (I J) c satisfy (1) Ker(Ψ T J ) Ker(Φ) = {0}; (2) There exists y R l such that Ψy Im(Φ T ), y I = sign(ψ T I ˉx), y J < 1, and y K 1. In Condition 2, a smaller J relaxes part (2) but gives a larger Ker(Ψ T J ) and thus tightens part (1). Although Condition 2 allows a more flexible J than Condition 1, we shall show that they are equivalent. 2.2 Main results Depending on the specific models in (2), we need the following assumptions: Assumption 1. Matrix Φ has full row-rank. Assumption 2. λ max (ΨΨ T ) = 1. Assumption 3. Matrix Ψ has full row-rank. Assumptions 1 and 3 are standard. Assumption 2 is non-essential. We can scale a general Ψ by multiplying 1 it with. Below we state our main results, whose proofs are given in sections 5 and 6. λmax(ψψ T ) Theorem 1 (Uniqueness). Under Assumption 1, let ˆx be a solution to problem (2a) or (2b), or under Assumptions 1 and 3, let ˆx be a solution to problem (2c). The followings are equivalent: 1) Solution ˆx is unique; 2) Condition 1 holds for ˉx = ˆx; 3) Condition 2 holds for ˉx = ˆx. This theorem states that Conditions 1 and 2 are equivalent, and they are necessary and sufficient for a solution ˆx to problem (2a), or to problem (2b), or to problem (2c) to be unique. To state our next result on robustness, we let r(j) := u 2 sup. u Ker(Ψ T J )\{0} Φu 2 Part (1) of Condition 1 ensures that 0 < r(j) < +. If Ψ = I, then u Ker(Ψ T J )\{0} is a sparse nonzero vector with maximal support J c, so r(j) is the inverse of the minimal singular value of the submatrix Φ J c. Below we claim Condition 1 ensures the robustness of problems (2b) and (2c) to arbitrary noise in b. Theorem 2 (Robustness). Under Assumptions 1-3, given an original signal x R n, let I = supp(ψ T x ) and J = I c. For arbitrary noise w, let b = Φx + w. If Condition 1 is met for ˉx = x and w 2 δ, then 1) For any C 0 > 0, there exists constant C 1 > 0 such that every minimizer x δ,λ of problem (2b) using parameter λ = C 0 δ satisfies Ψ T (x δ,λ x ) 1 C 1 δ; 3

4 2) Every minimizer x δ of problem (2c) satisfies Moreover, defining Ψ T (x δ x ) 1 C 2 δ. we can let β = (ΦΦ T ) 1 Ψy, C 3 = r(j) I and C 4 = 1 + Cond(Ψ) Φ C 3 1 y J, C 1 = 2C 3 + C 0 β 2 + (1 + C 0 β 2 /2) 2 C 4 C 0, C 2 = 2C 3 + 2C 4 β 2. Remark 1. From the results of Theorem 2, it is straightforward to derive l 1 or l 2 bounds for (x δ,λ x ) and (x δ x ) under Assumption 2. Remark 2. Since C 0 is free to choose, one can choose the optimal C 0 = and simplify C 1 to 4C 4 4 β 2 +C 4 β 2 2 C 1 = 2C 3 + C 4 β 2 + C4 2 β C 4 β 2 2C 3 + 2C 4 β 2 + 2, which becomes very similar to C 2. This reflects the equivalence between problems (2b) and (2c) in the sense that given λ, one can find δ so that they have the same solution, and vice versa. Remark 3. Both C 1 and C 2 are the sum of 2C 3 and other terms. 2C 3 alone bounds the error when Ψ T x δ,λ (or Ψ T x δ ) and Ψ T x have matching signs. Since C 3 does not depend on y J, part (2) of Condition 1 does not play any role, whereas part (1) plays the major role. When the signs of Ψ T x δ,λ (or Ψ T x δ ) and Ψ T x do not match, the remaining terms in C 1 and C 2 are involved, and they are affected part (2) of Condition 1; in particular, y J < 1 plays a big role as C 4 is inversely proportional to 1 y. Also, since there is no knowledge about the support of Ψ T x δ,λ, which may or may not equal to that of Ψ T x, C 4 inevitably depends the global properties of Ψ and Φ. In contrast, C 3 only depends on the restricted property of Φ. 3 Related works In the case of Ψ = Id, Condition 1 is well known in the literature for l 1 (or l 1 -synthesis) minimization. It is initially proposed in [7] as a sufficient condition for the l 1 solution uniqueness. For problems (2b) and (2c), [8, 17] present sufficient but non-necessary conditions for solutions uniqueness. Later, its necessity is established in [10] for model (2b) and then in [19] for all models in (2), assuming Ψ = Id or equal to an orthogonal basis. The solution robustness of model (2b) is given under the same condition in [10]. Below we restrict our literature review to results for the l 1 -analysis model. 3.1 Previous uniqueness conditions Papers [9, 14, 18, 11] cover the uniqueness of the l 1 -analysis model and use stronger conditions than ours. The following condition in [14] guarantees the solution uniqueness for problem (2a): Condition 3. Given ˉx, let Q be a basis matrix of Ker(Φ), and I = supp(ψ T ˉx). The followings are met: (1) Ψ T IcQ is full column rank; (2) (Q T Ψ I c) + Q T Ψ I sign(ψ T I ˉx) < 1. 4

5 Paper [18] proposes the following condition for the solution uniqueness and robustness for problems (2a) and (2b) (the robustness requires the non-zero entries of Ψ T I x to be sufficient large compared to noise). Condition 4. For a given ˉx, index sets I = supp(ψ T ˉx) and J = I c satisfy: (1)Ker(Ψ T J ) Ker(Φ) = {0}; (2)Let A [J] = U(U T Φ T ΦU) 1 U T and Ω [J] = Ψ + J (ΦT ΦA [J] Id)Ψ I, where U is a basis matrix of Ker(Ψ T J ). Then IC(sign(Ψ T I ˉx)) := min u Ker(Ψ J ) Ω[J] sign(ψ T I ˉx) u < 1. According to [18], Conditions 3 and 4 do not contain each other. The following example shows that Conditions 3 and 4 are both stronger than Conditions 1 and 2. Let ( ) ( ) Ψ =, Φ =, ˆx =, b = It is straightforward to verify that Conditions 1 and 2 hold. However, Conditions 3 and 4 fail to hold. Indeed, we have Ψ T x = (10.5, 0, 0) T and I = {1}. Q = (1, 0, 0) T is a basis matrix of Ker(Φ). Thus ( (Q T Ψ I c) + Q T Ψ I sign(ψ T 10.5 I x) = 101, 105 ) T = (3) Hence, Condition 3 does not hold. Furthermore, U = (1, 1, 10) T is a basis matrix of Ker(Ψ T J ), and the definition of Ω [J] gives us Ω [J] = ( , )T. Therefore, IC(sign(Ψ T 105 I x)) = 101 > 1, so Condition 4 does not hold either. Paper [18] also presents sufficient conditions for solution uniqueness, which are reviewed in [19] and shown to be not necessary. 3.2 Previous robustness conditions Turning to solution robustness, [9, 11] have studied the robustness of problems (2b) and (2c) in the Hilbertspace setting. Translating to the finite dimension, the conditions in [9] read: Condition 5. Given ˉx, the following two statements hold: (1) Ker(Ψ T J ) Ker(Φ) = {0}; (2) There exists y 1 (Ψ T ˉx) such that Ψy Im(Φ T ) and J = {i : y i < 1} =. This condition is equivalent to Condition 2. Under Condition 5, [9] shows the existence of constant C (not explicitly given) such that the solution x δ,λ to (2b) obeys Ψ T (x δ,λ x ) 2 Cδ when λ is set proportional to the noise level δ. In order to obtain an explicit formula for C, [11] introduces the following: Condition 6. Let ˆΨ = (ΨΨ T ) 1 Ψ. Given ˉx, the following two statements hold: (1) There exists some y 1 (Ψ T ˉx) such that Ψy Im(Φ T ); (2) For some t (0, 1), letting I(t) = {i : y i > t}, the mapping ˆΦ := Φ Span{ ˆΨi :i I(t)} is injective. Under this condition, the solutions to (2b) and (2c) are subject to error bounds whose constants depend on t, ˆΦ, the lower frame bound of Ψ, and other quantities. Proposition 1. Condition 6 is stronger than Condition

6 Proof. Let J = I(t) c ; then we have y J t < 1 from the definition of I(t). It remains to show that Ker(Ψ T J ) Ker(Φ) = {0}. For any x Ker(Ψ T J ), we have x = (ΨΨ T ) 1 ΨΨ T x = (ΨΨ T ) 1 Ψ J Ψ T J x + (ΨΨ T ) 1 Ψ J cψ T J cx = (ΨΨT ) 1 Ψ I(t) Ψ T I(t) x. Since Φ restricted to Span{ ˆΨ i : i I(t)} = Im((ΨΨ T ) 1 Ψ I(t) ) is injective, we have what we need. Paper [18] provides a much stronger condition below that strengthens Condition 4 by dropping the dependence on the Ψ-support (see the definition of RC(I) below). Condition 7. Given ˉx, index sets I = supp(ψ T ˉx) and J = I c satisfy: (1) Ker(Ψ T J ) Ker(Φ) = {0}; (2) Letting Ω [J] be given as in Condition 4, RC(I) := max p R I, p 1 min u Ker(Ψ J ) Ω[J] p u < 1. Under this condition, a nice error bound and a certain kind of weak sign consistency (between Ψ T x δ,λ and Ψ T x ) are given provided that problem (2b) is solved with the parameter λ = ρ w 2c J 2(1 RC(I)) for some ρ > 1, where c J = Ψ + J ΦT (ΦA [J] Φ T Id) 2,. When 1 RC(I) gets close to 0, this λ can become too large than it should be. 4 Verifying the conditions In this section, we present a method to verify Condition 1. Our method includes two steps: (Step 1:) Let Φ = UΣV T be the singular value decomposition of Φ. Assume V = [v 1,, v n ]. Since Φ has full row-rank, we have Ker(Φ) = Span{v m+1,, v n } and Q = [v m+1,, v n ] as a basis of Ker(Φ). We verify that Ψ T J Q has full row-rank, ensuring part (1) of Condition 1. (Step 2:) Let u 1 = Q T Ψ I sign(ψ T I ˉx) and A = QT Ψ J. Solve the convex problem minimize u, subject to Au = u 1. (4) u R J If the optimal objective of (4) is strictly less than 1, then part (2) of Condition 1 holds. In fact, we have: Proposition 2. Part (2) of Condition 1 holds if and only if (4) has an optimal objective < 1. Proof. Let û be a minimizer of (4). Assume û < 1. We consider the vector y composed by y I = sign(ψ T I ˉx) and y J = û. To show part (2) of Condition 1, it suffices to prove Ψy Im(Φ T ), or equivalently, Q T Ψy = 0. Indeed, The converse is obvious. Q T Ψy = Q T Ψ J y J + Q T Ψ I y I = Q T Ψ J û + Q T Ψ I y I = 0. Convex program (4) is similar in form to one in [18] though they are used to verify different conditions. 5 Proof of Theorem 1 We establish Theorem 1 in two steps. Our first step proves the theorem for problem (2a) only. The second step proves Theorem 1 for problems (2b) and (2c). 6

7 5.1 Proof of Theorem 1 for problem (2a) The equivalence of the three statements is shown in the following orders: 3) = 1) = 2) = 3). 3) = 1). Consider any perturbation ˆx + h where h Ker(Φ)\{0}. Take a subgradient g 1 (Ψ T ˆx) obeying g I = sign(ψ T I ˆx) = y I, g K = y K, and g J 1 such that g J, Ψ T J h = ΨT J h 1. Then, Ψ T (ˆx + h) 1 Ψ T ˆx 1 + Ψg, h (5a) = Ψ T ˆx 1 + Ψg Ψy, h (5b) = Ψ T ˆx 1 + g y, Ψ T h (5c) = Ψ T ˆx 1 + g J y J, Ψ T J h (5d) Ψ T ˆx 1 + Ψ T J h 1 (1 y J ), where (5b) follows from Ψy Im(Φ T ) = Ker(Φ) and h Ker(Φ), (5d) follows from the setting of g, and (5e) is an application of the inequality x, y x 1 y and g J, Ψ T J h = ΨT J h 1. Since h Ker(Φ)\{0} and Ker(Ψ T J ) Ker(Φ) = {0}, we have Ψ T J h 1 > 0. Together with the condition y J < 1, we have Ψ T (ˆx + h) 1 > Ψ T ˆx 1 for every h Ker(Φ)\{0} which implies that ˆx is the unique minimizer of (2a). 1) = 2). For every h Ker(Φ)\{0}, we have Φ(ˆx + th) = Φˆx and can find t small enough around 0 such that sign(ψ T I (ˆx + th)) = sign(ψt I ˆx). Since ˆx is the unique solution, for small and nonzero t we have Therefore, for any h Ker(Φ)\{0}, we have Ψ T (ˆx) 1 < Ψ T (ˆx + th) 1 = Ψ T I (ˆx + th) 1 + Ψ T I c(ˆx + th) 1 (5e) (6a) = Ψ T I (ˆx + th), sign(ψ T I (ˆx + th)) + tψ T I ch 1 (6b) = Ψ T I ˆx + tψ T I h, sign(ψ T I ˆx) + tψ T I ch 1 (6c) = Ψ T I ˆx, sign(ψ T I ˆx) + t Ψ T I h, sign(ψ T I ˆx) + tψ T I ch 1 (6d) = Ψ T (ˆx) 1 + t Ψ T I h, sign(ψ T I ˆx) + tψ T I ch 1. (6e) Ψ T I h, sign(ψ T I ˆx) < Ψ T I ch 1. (7) If the condition Ker(Ψ T I c) Ker(Φ) = {0} does not hold, we can choose a nonzero vector h Ker(Ψ T I c) Ker(Φ). We also have h Ker(Ψ T I c) Ker(Φ). Then we have Ψ T I h, sign(ψt I ˆx) < 0 and ΨT I h, sign(ψt I ˆx) < 0, which is a contradiction. It remains to show the existence of y in item (2) of Condition 1. This part is in spirit of the methods in papers [10] and [19], which are based on linear programming strong duality. We take ŷ with restrictions ŷ I = sign(ψ T I ˆx) and ŷ I c = 0. If such ŷ satisfies Ψŷ Im(ΦT ), then the existence has been shown. If Ψŷ / Im(Φ T ) = Ker(Φ), then we shall construct a new vector to satisfy part (2) of Condition 1. Let Q be a basis matrix of Ker(Φ). We have that a := Q T Ψŷ must be a nonzero vector. Consider the following problem minimize z subject to Q T Ψz = a and z I = 0. (8) z R l For any minimizer ẑ of problem (8), we have Ψ(ŷ + ẑ) Ker(Φ) = Im(Φ T ) and (ŷ + ẑ) I = ŷ I = sign(ψ T I ˆx). Thus, we shall show that the objective of problem (8) is strictly less than 1. To this end, we rewrite problem (8) in an equivalent form as: whose Lagrange dual problem is minimize z I c subject to Q T Ψ I cz I c = a, (9) z maximize p, a subject to Ψ T I cqp 1 1. (10) p 7

8 Note that Qp Ker(Φ) and p, a = p, Q T Ψŷ = p, Q T Ψ I sign(ψ T I ˆx) = ΨT I Qp, sign(ψt I ˆx). By using (7), for any p we have { Ψ T I p, a = Qp, sign(ψt I ˆx) = 0, if Qp = 0; Ψ T I Qp, sign(ψt I ˆx) < ΨT I cqp (11) 1 1, otherwise. Hence, problem (10) is feasible, and its objective value is strictly less than 1. By the linear programming strong duality property, problems (8) and (9) also have solutions, and their the objective value is strictly less than 1, too. This completes the proof. 2) = 3). Let J = I c and K = ; then Condition 2 follows. The proof of 3) = 1) is a standard technique in compressed sensing community. 5.2 Proof of Theorem 1 for problems (2b) and (2c) Lemma 1. Let γ > 0. If γ Φx b Ψ T x 1 is constant on a convex set Ω, then both Φx b and Ψ T x 1 are constant on Ω. Proof. It suffices to prove the case where the convex set has more than one point. Suppose x 1 and x 2 are arbitrary two different points in Ω. Consider the line segment L connecting x 1 and x 2. By the convexity of set Ω, we know L Ω. Thus, ĉ = γ Φx b Ψ T x 1 is a constant on L. If Φx 1 b Φx 2 b, then for any 0 < α < 1, we have γ Φ(αx 1 + (1 α)x 2 ) b Ψ T (αx 1 + (1 α)x 2 ) 1 (12a) = γ α(φx 1 b) + (1 α)(φx 2 b) α(ψ T x 1 ) + (1 α)(ψ T x 2 ) 1 (12b) < α(γ Φx 1 b Ψ T x 1 1 ) + (1 α)(γ Φx 2 b Ψ T x 2 1 ) (12c) = αĉ + (1 α)ĉ = ĉ, (12d) where the strict inequality follows from the strict convexity of γ 2 2 and the convexity of Ψ T x 1. This means that the points αx 1 + (1 α)x 2 on L attain a lower value than ĉ, which is a contradiction. Therefore, we have Φx 1 b = Φx 2 b, from which it is easy to see Ψ T x 1 1 = Ψ T x 2 1. We let X λ and Y δ denote the sets of solutions to problems (2b) and (2c), respectively; moreover, we assume that these two sets are nonempty. Then, from Lemma 1, we have the following result. Corollary 1. In problem (2b), Φx b and Ψ T x 1 are constant on X λ ; in problem (2c), Φx b and Ψ T x 1 are constant on Y δ. Proof. Since Φx b λ Ψ T x 1 is constant over X λ, the result follows directly from Lemma 1 for problem (2b). For problem (2c), if 0 Y δ, then we have Y δ = {0} because of the full row-rankness of Ψ. The result holds trivially. Suppose 0 Y δ. Since the optimal objective Ψ T x 1 is constant for all x Y δ, we have to show that Φx b 2 2 = δ for all x Y δ. If there exist a nonzero ˆx Y δ such that Φˆx b 2 2 < δ, we can find a non-empty ball B centered at ˆx with a sufficiently small radius ρ > 0 such that Φ x b 2 2 < δ for all x B. Let α = min{ ρ 2 ˆx 2, 1 2 } (0, 1). We have (1 α)ˆx B and (1 α)ψt ˆx 1 < Ψ T ˆx 1, which is a contradiction. Proof of Theorem 1 for problems (2b) and (2c). This proof exploits Corollary 1. Since the results of Corollary 1 are identical for problems (2b) and (2c), we present the proof for problem (2b) only. 8

9 By assumption, X λ is nonempty so we pick ˆx X λ. Let b = Φˆx, which is independent of the choice of ˆx according to Corollary 1. We introduce the following problem minimize Ψ T x 1, subject to Φx = b, (13) x and let X denote its solution set. Now, we show that X λ = X. Since Φx = Φˆx and Ψ T x 1 = Ψ T ˆx 1 for all x X λ and conversely any x obeying Φx = Φˆx and Ψ T x 1 = Ψ T ˆx 1 belongs to X λ, it suffices to show that Ψ T x 1 = Ψ T ˆx 1 for any x X. Assuming this does not hold, then since problem (13) has ˆx as a feasible solution and has a finite objective, we have a nonempty X and there exists x X satisfying Ψ T x 1 < Ψ T ˆx 1. But, Φ x b 2 = b b 2 = Φˆx b 2 and Ψ T x 1 < Ψ T ˆx 1 mean that x is a strictly better solution to problem (2b) than ˆx, contradicting the assumption ˆx X λ. Since X λ = X, ˆx is the unique solution to problem (2b) if and only if it is the unique solution to problem (13). Since problem (13) is in the same form of problem (2a), applying the part of Theorem 1 for problem (2a), which is already proved, we conclude that ˆx is the unique solution to problem (2b) if and only if Condition 1 or 2 holds. 6 Proof of Theorem 2 Lemma 2. Assume that vectors ˉx and y satisfy Condition 1. Let I = supp(ψ T ˉx) and J = I c. We have Ψ T x Ψ T ˉx 1 C 3 Φ(x ˉx) 2 + C 4 d y (x, ˉx), x, (14) where d y (x, ˉx) := Ψ T x 1 Ψ T ˉx 1 Ψy, x ˉx is the Bregman distance of function Ψ T 1, the absolute constants C 3, C 4 are given in Theorem 2. Proof. This proof is divided into two parts. They are partially inspired by [9]. 1. this part shows that for any u Ker(Ψ T J ), ( ) Ψ T x Ψ T C 3 Φ ˉx Ψ T (x u) 1 + C 3 Φ(x ˉx) 2. (15) λmin (ΨΨ T ) 2. this part shows that f(x) := min { Ψ T (x u) 1 : u Ker(Ψ T J ) } (1 y J ) 1 d y (x, ˉx). (16) Using the definition of C 4, combining (15) and (16) gives (14). Part 1. Let u Ker(Ψ T J ). By the triangle inequality of norms, we get Ψ T x Ψ T ˉx 1 Ψ T (x u) 1 + Ψ T (u ˉx) 1. (17) Since ˉx Ker(Ψ T J ), we have u ˉx Ker(ΨT J ) and thus u ˉx 2 r(j) Φ(u ˉx) 2, where r(j) < + follows from part (1) of Condition 1. Using the fact that supp(ψ T (u ˉx)) = I, we derive that Ψ T (u ˉx) 1 I Ψ T (u ˉx) 2 I u ˉx 2 I r(j) Φ(u ˉx) 2 (18a) (18b) (18c) = C 3 Φ(u ˉx) 2, (18d) 9

10 where we have used the assumption λ max (ΨΨ T ) = 1 and the definition C 3 = r(j) I. Furthermore, Φ(u ˉx) 2 Φ(x u) 2 + Φ(x ˉx) 2 Φ x u 2 + Φ(x ˉx) 2 (19a) (19b) Φ ΨT (x u) 2 λmin (ΨΨ T ) + Φ(x ˉx) 2 (19c) Φ ΨT (x u) 1 λmin (ΨΨ T ) + Φ(x ˉx) 2. (19d) Therefore, we get (15) after combining (17), (18), and (19). Part 2. Since Ψy, ˉx = Ψ T ˉx 1 implies d y (x, ˉx) = Ψ T x 1 Ψy, x, it is equivalent to proving f(x) (1 y J ) 1 ( Ψ T x 1 Ψy, x ). (20) Since u Ker(Ψ T J ) is equivalent to ΨT J u = 0, the Lagrangian of the minimization problem in (16) is L(u, v) = Ψ T (x u) 1 + v, Ψ T J u = Ψ T (x u) 1 + Ψ J v, u x + Ψ J v, x. (21) Then, f(x) = min u max v L(u, v). Following the minimax theorem, we derive that Let f(x) = max v min u L(u, v) = max min{ Ψ T (x u) 1 + Ψ J v, u x + Ψ J v, x } (22a) v u = max min w u { ΨT (x u) 1 + w, u x + w, x : w Im(Ψ J )} (22b) = max w { w, x : w ΨT 1 (0) Im(Ψ J )} (22c) = max w { cψy + w, x : w ΨT 1 (0) Im(Ψ J )} cψy, x, c > 0 (22d) =c max w { Ψy + w, x : w c 1 Ψ T 1 (0) Im(Ψ J )} c Ψy, x, c > 0. (22e) c = (1 y J ) 1 and Z J = {z R l : z I = 0}. Since y J < 1 from part (2) of Condition 1, we have c < + and get ( y + c 1 1 (0) Z J ) 1 (0), (23) from which we conclude ( Ψy + c 1 Ψ T 1 (0) Im(Ψ J ) ) Ψ T 1 (0). (24) Hence, for any w c 1 Ψ T 1 (0) Im(Ψ J ), it holds Ψy + w Ψ T 1 (0), which by the convexity of Ψ T 1 implies Ψy + w, x Ψ T x 1. Therefore, f(x) c( Ψ T x 1 Ψy, x ). Lemma 3. ([1], Theorem 3; [10], Lemma 3.5) Suppose that x R n is a fixed vector obeying supp(ψ T x ) = I and that there are vectors satisfying y (Ψ T x ) and Ψy = Φ T β. Then for every δ > 0 and every data vector b satisfying Φx b 2 δ, the following two statements hold: 1)Every minimizer x δ,λ of problem (2b) satisfies d y (x δ,λ, x ) (δ+λ β 2/2) 2 λ and Φx δ,λ b 2 δ+λ β 2 ; 2)Every minimizer x δ of problem (2c) satisfies d y (x δ, x ) 2δ β 2. From Ψy = Φ T β and the full-rankness of Φ, we have β = (ΦΦ T ) 1 Ψy. 10

11 Proof of Theorem 2. Firstly, we derive that Ψ(x δ,λ x ) 1 C 3 Φ(x δ,λ x ) 2 + C 4 d y (x δ,λ, x ) C 3 Φx δ,λ b 2 + C 3 Φx b 2 + C 4 d y (x δ,λ, x ) (25a) (25b) (δ + λ β 2 /2) 2 C 3 (δ + λ β 2 ) + C 3 δ + C 4, (25c) λ where the first and the third inequalities follow from Lemmas 2 and 3, respectively. Substituting λ = C 0 δ and collecting like terms in (25c), we obtain the first part of Theorem 2. The second part can be proved in the same way. Acknowledgements The work of H. Zhang is supported by China Scholarship Council during his visit to Rice University, and in part by Graduate School of NUDT under Funding of Innovation B110202, Hunan Provincial Innovation Foundation for Postgraduate CX2011B008, and NSFC Grants No and No The work of M. Yan is supported in part by the Center for Domain-Specific Computing (CDSC) funded by NSF grants CCF and ARO/ARL MURI grant FA The work of W. Yin is supported in part by NSF grants DMS and ECCS They thank Rachel Ward and Xiaoya Zhang for their discussions. References [1] M. Burger and S. Osher, Convergence rates of convex variational regularization, Inverse Problems, 20 (2004), p [2] E. Candes, The restricted isometry property and its implications for compressed sensing, Comptes Rendus Mathematique, 346 (2008), pp [3] E. J. Candès, Y. C. Eldar, D. Needell, and P. Randall, Compressed sensing with coherent and redundant dictionaries, Applied and Computational Harmonic Analysis, 31 (2011), pp [4] E. J. Candes and Y. Plan, A probabilistic and RIPless theory of compressed sensing, IEEE Trans. Inf. Theor., 57 (2011), pp [5] A. Cohen, W. Dahmen, and R. Devore, Compressed sensing and best k-term approximation, J. Amer. Math. Soc, (2009), pp [6] M. Elad, P. Milanfar, and R. Rubinstein, Analysis versus synthesis in signal priors, Inverse Problems, 23 (2007), p [7] J. J. Fuchs, On sparse representations in arbitrary redundant bases, IEEE Trans. Inf. Theor., 50 (2004), pp [8] J. J. Fuchs, Recovery of exact sparse representations in the presence of bounded noise, IEEE Trans. Inf. Theor., 51 (2005), pp [9] M. Grasmair, Linear convergence rates for Tikhonov regularization with positively homogeneous functionals, Inverse Problems, 27 (2011), p

12 [10] M. Grasmair, M. Haltmeier, and O. Scherzer, Necessary and sufficient conditions for linear convergence of l 1 -regularization, Comm. Pure Appl. Math., 64 (2011), pp [11] M. Haltmeier, Stable signal reconstruction via l 1 -minimization in redundant, non-tight frames, Signal Processing, IEEE Transactions on, 61 (2013), pp [12] F. Krahmer and R. Ward, Compressive imaging: stable and robust recovery from variable density frequency samples, arxiv: , (2012). [13] Y. Liu, T. Mi, and S. Li, Compressed sensing with general frames via optimal-dual-based l 1 -analysis, Information Theory, IEEE Transactions on, 58 (2012), pp [14] S. Nam, M. E. Davies, M. Elad, and R. Gribonval, The cosparse analysis model and algorithms, Applied and Computational Harmonic Analysis, 34 (2013), pp [15] D. Needell and R. Ward, Near-optimal compressed sensing guarantees for total variation minimization, arxiv: , (2012). [16] D. Needell and R. Ward, Stable image reconstruction using total variation minimization, arxiv: , (2012). [17] R. J. Tibshirani, The lasso problem and uniqueness, arxiv: , (2012). [18] S. Vaiter, G. Peyre, C. Dossal, and J. Fadili, Robust sparse analysis regularization, Information Theory, IEEE Transactions on, 59 (2013), pp [19] H. Zhang, W. Yin, and L. Cheng, Necessary and sufficient conditions of solution uniqueness in l 1 minimization, arxiv: , (2012). [20] Y. Zhang, Theory of compressive sensing via l 1 -minimization: a non-rip analysis and extensions, Journal of the Operations Research Society of China, 1 (2013), pp