Phase Transition Phenomenon in Sparse Approximation

Transcription

1 Phase Transition Phenomenon in Sparse Approximation University of Utah/Edinburgh L1 Approximation: May 17 st 2008

2 Convex polytopes Counting faces Sparse Representations via l 1 Regularization Underdetermined system, infinite number of solutions Ax = b, A R n N, n < N

3 Convex polytopes Counting faces Sparse Representations via l 1 Regularization Underdetermined system, infinite number of solutions Ax = b, A R n N, n < N Seek sparsest solution, x l 0 := # nonzero elements min x l 0 subject to Ax = b

4 Convex polytopes Counting faces Sparse Representations via l 1 Regularization Underdetermined system, infinite number of solutions Ax = b, A R n N, n < N Seek sparsest solution, x l 0 := # nonzero elements min x l 0 subject to Ax = b Efficient, non-combinatorial, solution via convex l 1 relaxation min x l 1 subject to Ax = b

5 Convex polytopes Counting faces Sparse Representations via l 1 Regularization Underdetermined system, infinite number of solutions Ax = b, A R n N, n < N Seek sparsest solution, x l 0 := # nonzero elements min x l 0 subject to Ax = b Efficient, non-combinatorial, solution via convex l 1 relaxation min x l 1 subject to Ax = b How sparse is necessary such that l 1 recovers l 0?

6 Convex polytopes Counting faces Sparse Representations via l 1 Regularization Underdetermined system, infinite number of solutions Ax = b, A R n N, n < N Seek sparsest solution, x l 0 := # nonzero elements min x l 0 subject to Ax = b Efficient, non-combinatorial, solution via convex l 1 relaxation min x l 1 subject to Ax = b How sparse is necessary such that l 1 recovers l 0? Matrix family A: Gaussian iid entries, random ortho-projector [Baryshnikov and Vitale]

7 Convex polytopes Counting faces Sparse Representations via l 1 Regularization Underdetermined system, infinite number of solutions Ax = b, A R n N, n < N Seek sparsest solution, x l 0 := # nonzero elements min x l 0 subject to Ax = b Efficient, non-combinatorial, solution via convex l 1 relaxation min x l 1 subject to Ax = b How sparse is necessary such that l 1 recovers l 0? Matrix family A: Gaussian iid entries, random ortho-projector [Baryshnikov and Vitale] Gaussian ensemble: Comp. Sens. without prior basis selection min Φx l 1 subject to AΦx b ǫ

8 Convex polytopes Counting faces Geometry of l 1 recovering l 0 Sparsity: x R N with k < n nonzeros on k 1 face of l 1 ball. Matrix A projects face of l 1 ball either onto or into conv(a). l 1 ball R N edge onto conv(a) edge into conv(a)

9 Convex polytopes Counting faces Geometry of l 1 recovering l 0 Sparsity: x R N with k < n nonzeros on k 1 face of l 1 ball. Matrix A projects face of l 1 ball either onto or into conv(a). l 1 ball R N edge onto conv(a) edge into conv(a) Survived faces are sparsity patterns in x where l 1 l 0 Faces which fall inside conv(a) are not solutions to l 1

10 Convex polytopes Counting faces Geometry of l 1 recovering l 0 min x 1 subject to Ax = b, A R n N Grow conv(αa) from α = 0 until intersects b R n

11 Convex polytopes Counting faces Geometry of l 1 recovering l 0 min x 1 subject to Ax = b, A R n N Grow conv(αa) from α = 0 until intersects b R n l 1 l 0 l 1 l 0

12 Convex polytopes Counting faces Expected number of faces, random ortho-projector f k (Q) Ef k (AQ) = 2 β(f,g)γ(g,q) s 0 F F k (Q) G F n+1+2s(q) where β and γ are internal and external angles respectively [Affentranger, Schneider]

13 Convex polytopes Counting faces Expected number of faces, random ortho-projector f k (Q) Ef k (AQ) = 2 β(f,g)γ(g,q) s 0 F F k (Q) G F n+1+2s(q) where β and γ are internal and external angles respectively [Affentranger, Schneider] γ(f l,t N 1 ) = l + 1 π 0 e (l+1)x2 ( 2 π x 0 e y2 dy) N l 1 dx. Large deviation analysis, Q = T N 1, C N : exponential behavior in N

14 Convex polytopes Counting faces Counting Faces of Random Polytopes: Strong Strong recovery phase transition, ρ S (δ): With overwhelming probability on the selection of A, l 1 recovers l 0 for every x 0 with x 0 l 0 (ρ S (δ) ǫ) n For l (ρ S (δ) ǫ) n, f l (C N ) Ef l (A n,n C N ) π N e ǫn Prob{f l (AC N ) = f l (C N ), l (ρ S (δ) ǫ)n } 1, as N k n δ = n/n

15 Exponentiality Finite dimensional bounds Weak phase transitions Exponentiality of phase transition Lower and upper bounds on the expected number faces lost f l (C N ) Ef l (A n,n C N ) < (N + 3) 5 e N ψ(k/n,n/n) f l (C N ) Ef l (A n,n C N ) > N 3/2 e N ψ(k/n,n/n)

16 Exponentiality Finite dimensional bounds Weak phase transitions Exponentiality of phase transition Lower and upper bounds on the expected number faces lost f l (C N ) Ef l (A n,n C N ) < (N + 3) 5 e N ψ(k/n,n/n) f l (C N ) Ef l (A n,n C N ) > N 3/2 e N ψ(k/n,n/n) ψ(k/n, n/n) 8 0 k n n/n ρ S (k/n) is the zero level curve: ψ(k/n,ρ S (k/n)) = 0.

17 Exponentiality Finite dimensional bounds Weak phase transitions For k n (1 θ)ρ S (n/n), f k (Q) Ef k (AQ) < (N + 3) 5 e nθω S (n/n) with Q = C N,T N 1 for = ±, Ω * (n/n) for * = + (solid), and * = ± (dashed) S n/n Ω S (n/n) 1/2.

18 Exponentiality Finite dimensional bounds Weak phase transitions For k n (1 θ)ρ S (n/n), f k (Q) Ef k (AQ) < (N + 3) 5 e nθω S (n/n) with Q = C N,T N 1 for = ±, Ω * (n/n) for * = + (solid), and * = ± (dashed) S n/n Ω S (n/n) 1/2. Level curves converge to ρ S as n 1.

19 Exponentiality Finite dimensional bounds Weak phase transitions Phase transition for small N: Strong k n n/n Left: Q = T N 1, Right: Q = C N. P{f k (AQ) = f k (Q)} > 0: k neighborliness existence N = 200,1000,5000, and N

20 Exponentiality Finite dimensional bounds Weak phase transitions Phase transition for small N: Strong k n n/n Left: Q = T N 1, Right: Q = C N. P{f k (AQ) = f k (Q)} > 0: k neighborliness existence N = 200,1000,5000, and N How do the phase transitions change when requiring only successful recovery of the k sparse vector most of the time?

21 Exponentiality Finite dimensional bounds Weak phase transitions Counting Faces of Random Polytopes: Weak Weak recovery phase transition, ρ W (δ): For l (ρ W (δ) ǫ) n, Ef l (AC N ) (1 ǫ)f l (C N ), With overwhelming probability on the selection of A, l 1 recovers l 0 for most x 0 with x 0 l 0 (ρ W (δ) ǫ) n k n n/n Strong (all x) and Weak (most x) transition

22 Exponentiality Finite dimensional bounds Weak phase transitions Exponentiality of phase transition, weak For k n (1 θ)ρ W (n/n), f k (Q) Ef k (AQ) f k (Q) with Q = C N,T N 1 for = ±,+. < (N + 3) 6 e nθ2 Ω W (n/n) 1 Ω * (n/n) for * = + (solid), and * = ± (dashed) W n/n Ω W (n/n) 1/4.

23 Exponentiality Finite dimensional bounds Weak phase transitions Exponentiality of phase transition, weak For k n (1 θ)ρ W (n/n), f k (Q) Ef k (AQ) f k (Q) with Q = C N,T N 1 for = ±,+. < (N + 3) 6 e nθ2 Ω W (n/n) 1 Ω * (n/n) for * = + (solid), and * = ± (dashed) W n/n Ω W (n/n) 1/4. Level curves converge to ρ W as n 1/2.

24 Exponentiality Finite dimensional bounds Weak phase transitions Phase transition for small N: Weak k n n/n Left: Ef k (AC N ) f k(c N ): 99% face survive N = 200,1000,5000, and N For k n (1 θ)ρ W (n/n), Ef k (AC N )/f k (C N ) > 1 (N + 2) 6 e nθ2 /4

25 Exponentiality Finite dimensional bounds Weak phase transitions Phase transition for small N: Weak k n n/n Left: Ef k (AC N ) f k(c N ): 99% face survive N = 200,1000,5000, and N For k n (1 θ)ρ W (n/n), Ef k (AC N )/f k (C N ) > 1 (N + 2) 6 e nθ2 /4 Good empirical agreement for N = 200. Right: Phase transitions also known for x 0 What happens in the asymptotic regime n N?

26 Exponentiality Finite dimensional bounds Weak phase transitions The Compressed Sensing Regime: n N Sub-exponential growth of n with respect to N n, δ := n/n n 0, log(n n ) n 0, N n.

27 Exponentiality Finite dimensional bounds Weak phase transitions The Compressed Sensing Regime: n N Sub-exponential growth of n with respect to N n, log(n n ) δ := n/n n 0, 0, N n. n Strong Threshold, Nonnegative ρ + S (δ) 2e log(δ2 π) 1, δ 0 Strong Threshold, Signed ρ ± S (δ) 2e log(δ π) 1, δ 0 Weak Thresholds ρ W (δ) 2log(δ) 1, δ 0

28 Exponentiality Finite dimensional bounds Weak phase transitions The Compressed Sensing Regime: n N Sub-exponential growth of n with respect to N n, log(n n ) δ := n/n n 0, 0, N n. n Strong Threshold, Nonnegative ρ + S (δ) 2e log(δ2 π) 1, δ 0 Strong Threshold, Signed ρ ± S (δ) 2e log(δ π) 1, δ 0 Weak Thresholds ρ W (δ) 2log(δ) 1, δ 0 Principle Difference: e-times less strict sparsity requirement for recover of most threshold compared with for all threshold Gaussian setting fully characterized for l 1 without noise

29 Exponentiality Finite dimensional bounds Weak phase transitions The Compressed Sensing Regime: n N Sub-exponential growth of n with respect to N n, log(n n ) δ := n/n n 0, 0, N n. n Strong Threshold, Nonnegative ρ + S (δ) 2e log(δ2 π) 1, δ 0 Strong Threshold, Signed ρ ± S (δ) 2e log(δ π) 1, δ 0 Weak Thresholds ρ W (δ) 2log(δ) 1, δ 0 Principle Difference: e-times less strict sparsity requirement for recover of most threshold compared with for all threshold Gaussian setting fully characterized for l 1 without noise Empirical evidence indicates a notion of universality for l 1

30 Universality result: non-negativity Comparisons Weak Phase Transitions: Black: Weak phase transition: x 0 (top) x, signed (bot.) Overlaid empirical evidence of 50% success rate for Gaussian, Uniform 0 & 1, and Fourier k n n/n

31 Universality result: non-negativity Comparisons Towards Universality: Sign Constraints - Not l 1 Let x 0 be k-sparse and form b = Ax. Are there other y R N such that Ay = b, y 0, y x?

32 Universality result: non-negativity Comparisons Towards Universality: Sign Constraints - Not l 1 Let x 0 be k-sparse and form b = Ax. Are there other y R N such that Ay = b, y 0, y x? As n,n, Typically No provided k/n < ρ H W (δ)

33 Universality result: non-negativity Comparisons Towards Universality: Sign Constraints - Not l 1 Let x 0 be k-sparse and form b = Ax. Are there other y R N such that Ay = b, y 0, y x? As n,n, Typically No provided k/n < ρ H W (δ) Universal: A an ortho-complement of B R N n N with entries selected i.i.d. from a symmetric distribution For k/n < ρ H W (δ) and x 0, use any method you like, the answer is unique.

34 Universality result: non-negativity Comparisons Towards Universality: Sign Constraints - Not l 1 Let x 0 be k-sparse and form b = Ax. Are there other y R N such that Ay = b, y 0, y x? As n,n, Typically No provided k/n < ρ H W (δ) Universal: A an ortho-complement of B R N n N with entries selected i.i.d. from a symmetric distribution Good empirical agreement for N = 200. Overlaid empirical evidence of 50% success rate for Gaussian, Uniform 0,-1,+1, and Sparse 0,-1,+1

35 The Geometry of l 1 Regularization Universality result: non-negativity Comparisons Comparisons with other work Don t we already have universality, the RIP (1 δ k ) x 2 2 Ax 2 2 (1 + δ k ) x 2 2 RIP l 1 l 0 condition, δ 2k < 2 1 k n n/n Strong (all x), RIP transition RIP: robustness, pessimistic, hard to check, not necessary Polytope: no robustness (yet), iff results, hard to check

36 The Geometry of l 1 Regularization Universality result: non-negativity Comparisons Comparisons with other work Don t we already have universality, the RIP (1 δ k ) x 2 2 Ax 2 2 (1 + δ k ) x 2 2 RIP l 1 l 0 condition, δ 2k < 2 1 k n n/n Strong (all x), Weak, RIP transition RIP: robustness, pessimistic, hard to check, not necessary Polytope: no robustness (yet), iff results, hard to check Coherence: highly pessimistic, easy to check, not necessary Phase portrait implied by coherence? k/n < Const/log(n)

37 Universality result: non-negativity Comparisons References Donoho, T (2005) Sparse Nonnegative Solution of Underdetermined Linear Systems. PNAS Donoho, T (2005) Neighborliness of Randomly Projected Simplices in High Dimensions. PNAS Donoho (2005) High-Dimensional Centrosymmetric Polytopes with Neighborliness Proportional to Dimension. Discrete and Computational Geometry. Online, Dec Donoho (2006) Compressed Sensing. IEEE Info Thry. April. Donoho, T (2006) Counting faces of randomly-projected polytopes when the projection radically lowers dimension, J. AMS.