Spin Glass Approach to Restricted Isometry Constant

Transcription

1 Spin Glass Approach to Restricted Isometry Constant Ayaka Sakata 1,Yoshiyuki Kabashima 2 1 Institute of Statistical Mathematics 2 Tokyo Institute of Technology 1/29

2 Outline Background: Compressed sensing Problem setup: Restricted isometry constant (RIC) Spin glass approach Replica symmetric analysis Improvement by replica symmetry breaking Summary 2/29

3 Compressed sensing Reconstruct original signal X from its undersampled measurement Y. N M Y = M A X N Undersampled measurement Original signal 3/29

4 Compressed sensing Reconstruct original signal X from its undersampled measurement Y. N zero component M Y = M A X N Undersampled measurement Original signal Reconstruction is possible when X is sufficiently sparse. 4/29

5 Practical relevance The problem is related to various technologies of modern signal processing Many application domains Refraction seismic survey (mine examination) Tomography(X-ray CT,MRI) Single pixel camera Noise removal of image Data streaming computing Group testing etc. 5/29

6 Simulation of tomography LT:Original(Logan-Shepp Phantom) #512x512 RT:Sampling 512 points of 2D FT from 22 directions. LB:Recovery of pseudo-inverse (standard approach) RB:Recovery utilizing the sparseness of spatial variations. Original is perfectly recovered. Perfect recovery is realized by only 2% samples of what Nyquist-Shannon s theory requires. Breaking of the conventional limit! EJ Candes J Romberg and T. Tao, IEEE Trans. IT Vol. 52, (2006) 6/29

7 Two major reconstruction methods l 0 reconstruction xˆ = argmin x,subject toy X 0 = Ax l 1 reconstruction ˆx = argmin X x 1,subject toy = Ax Sufficient conditions for the reconstruction are given by Restricted Isometriy Constant (RIC). 7/29

8 Restricted Isometry Constant (RIC) Definition (Candès and Tao (2006)). Let A be a column-wisely normalized M N matrix. Let x R N be an arbitrary S-sparse vector, whose number of non-zero components is smaller than S. Then, if there exists such a constant d S = max{d S min, d S max } that satisfies (1 δ S min ) x F 2 Ax F 2 (1+ δ S max ) x F 2, A is said to satisfy the S-restricted isometry property (RIP) with the restricted isometry constant (RIC) d S. M µ=1 2 A µi = 1 for i Intuitively, d s quantifies how A deviates from orthogonal transforms in terms of Frobenius (L 2 ) norm for S-sparse vectors. 8/29

9 Sufficient conditions for l 0, l 1 reconstruction 1. l 0 reconstruction gives the unique S-sparse solution when d 2S < l 1 reconstruction gives the same unique S-sparse solution as l 0 reconstruction when d 2S < 2 1/2 1. [Candes and Tao, IEEE Trans. Inform. Theory (2005)] 3. (Improvement of 2.) l 1 reconstruction gives the same unique S-sparse solution as l 0 reconstruction when min max (4 2 3) δ2s + δ2s < 4( 2 1). [Foucart and Lai, Appl. Comput. Harmon. Anal. (2009)] 9/29

10 Computational difficulty RIC plays a key role in theoretical analysis and performance guarantee of compressed sensing. On top of this, it is purely of great interest as a fundamental problem of linear algebra. Unfortunately, there are no known large (systematic) matrices with bounded RICs (and computing RICs is strongly NP-hard),.! Wikipedia" On the other hand, many random matrices have been shown to remain bounded. Therefore, much effort has been paid for improving the bounds. " Our study is along this line. 10/29

11 Why computationally difficult? Suppose a situation where non-zero components are fixed. N = 5, V={1,2,3,4,5}, S = 2, T = {2,4} A a 1 a 2 a 3 a 4 a = A T a 2 a 4 X T X λ T 2 2 T 2 min ( ATAT ) xt ATxT λmax( ATAT ) x F F T F The change of norm can be easily characterized by eigenvalues of sub-matrix A T. 11/29

12 Why computationally difficult? l In evaluation of RIC, the inequalities must hold for all possible combinations of the positions of non-zeros. δ l This causes a combinatorial difficulty, yielding an exact expression of RIC as S = max{1 min T: T = S, T V λ min ( A T T A T ), max T: T = S, T V λ max ( A T T A T ) 1}. All possible column choices => Combinatorial difficulty Structures of the problem and the difficulty are analogous to those of spin glass problems. 12/29

13 Analogy to spin glasses The choice of T are represented by binary vector {c i } {0, 1} N. N = 5, S = 2, T = {2,4} c = {0, 1, 0, 1, 0} A a 1 a 2 a a 4 a 5 a 2 a 4 a 2 a 4 T and max T can be regarded as `energy functions of c given A. 0 diag( c) = T λ ( A T A ) λ T ( A A T ) min A T 13/29

14 Analogy to spin glasses δ S = max{ 1 c: min N ci i= 1 = S λ min ( c A), c: max N ci i= 1 = S λ max ( c A) 1} RIC is given by minimum and maximum energy of c. l Energy of 0-1 spin c Λ + (c A) = λ min (c, A) Λ (c A) = λ max (c, A) l Canonical distribution of c N P(c A;µ) exp( µnλ sgn(µ) (c A) )δ c i S l Quenched randomness ( ) = 2π M 1 P A ( ) MN /2 exp M 2 µ,i 2 A µi i=1 14/29

15 Free entropy (free energy) φ µ A Spin glass approach N ( )δ c i S i=1 ( ) = 1 N log exp µnλ sgn(µ)(c A) { } N c 0,1 Z ( µ A) : Partition function Typical properties can be assessed by the replica method. φ ( µ ) = φ ( µ A) A = lim n 0 n log Z n ( µ A) A Energy (min/max eigenvalues) ( ) = φ ( µ ) λ µ µ Entropy (# of T producing λ) ω ( µ ) = φ ( µ ) µ φ ( µ ) µ 15/29

16 ω + Possible minimum and maximum eigenvalues ( λ ) min ω ( λ ) max 0 Possible minimum eigenvalue λ min λ min 0 Possible maximum eigenvalue λ max λ max δ S min = max{1 λ, λ max 1} 16/29

17 φ( µ ) + Replica symmetric analysis α = log 2 2 αµ q { α + χ + µ (1 q) } { α + χ + µ (1 q) } Dz log 1+ e K ( qˆ qˆ y + Dy exp 1 0 2Qˆ α = M / N (compression rate) ρ = S / N (fraction of non-zero components) q, χ, Qˆ, qˆ 1, qˆ, K} { 0 + Qˆ 2 α + log( α + χ) 2 qˆ 1 χ qˆ q Kρ 2 µ 2 are determined by saddle point equations. qˆ 0 z) 2 17/29

18 α = 0.5,ρ = 0.1 ω + Replica symmetric entropy α = M N,ρ = S N ( λ ) min ω ( λ max ) MP MP λ min δ S λ min min = max{1 λ, λ max 1} λ max λ max 18/29

19 Replica symmetric RIC d S at α = 0.5 α = M N,ρ = S N Comparison with prior works Bah-Tanner (2010) d S Our result Lower bound (Dossal et al, 2010) ρ /α 19/29

20 l 0, l 1 reconstruction limit α = M N,ρ = S N l 0 limit ρ /α l 1 limit α Dashed lines are Bah Tanner (2010) 20/29

21 Improvement by RSB l The RS RIC estimate is lower than any existing upper bounds, being consistent with a known lower bound. l On the other hand, there are non-negligible deviations from the experimental data. l In fact, detailed analysis shows that the replica symmetry is broken for the left and right edges of the entropy curves. l However, physical interpretation of RSB indicates that the RS estimates still serve as upper bounds of RIC, and the bounds are improved as the higher RSBs are taken into account. 21/29

22 Replica symmetric entropy (again) α = 0.5,ρ = 0.1 ω + α = M N,ρ = S N ( λ ) min ω ( λ max ) MP MP λ min δ S λ min min = max{1 λ, λ max 1} λ max λ max 22/29

23 Physical interpretation of RSB State space { energy :λ entrropy : s Single dominant cluster RS µ d µ c RS, 1RSB degenerated 1RSB µ Complexity = entropy of pure states ( ) = 1 N Σ λ,s Total entropy log (#pure states specified by ( λ,s ) ) Non-negativity constraint: Must be non-negative. ω ( λ) = 1 N ( ( ) ) ( ( ))! max log ds exp N s + Σ λ,s s { s + Σ( λ,s) } Cf) Montanari and Ricci-Tersenghi (2003), Krzakala et al (2007), 23/29

24 Physical interpretation of RSB State space { energy :λ entrropy : s Single dominant cluster RS µ d µ c RS, 1RSB degenerated 1RSB µ RS evaluation: the complexity constraint is ignored. ω RS ( λ) = max s { s + Σ( λ,s) } 1RSB evaluation: the complexity constraint is taken into account. ω 1RSB ( λ) = max{ s + Σ( λ,s) Σ( λ,s) 0} The non-negativity constraint of the 1RSB evaluation means s ω RS ( λ) ω ( λ). 1RSB 24/29

25 State space Single dominant cluster Physical interpretation of RSB µ d k { energy :λ entrropy : s krsb krsb, (k+1)1rsb (k+1)rsb degenerated If necessary, a similar argument may be applied for the dominant cluster of 1RSB, which yields ω RS µ c k ( λ) ω ( λ) ω ( λ). 1RSB 2RSB µ Applying the similar argument repeatedly concludes ω RS ( λ) ω ( λ) ω ( λ). 1RSB 2RSB 25/29

26 ω RS Monotonic improvement by RSB The series of inequalities ( λ) ω ( λ) ω ( λ). 1RSB 2RSB indicates that bounds are monotonically improved by incorporating the higher RSB. ω + ( λ min ) ω ( λ max ) λ min 0 0 λmax higher RSB!! higher RSB 26/29

27 1RSB results l The estimates are actually improved by the 1RSB solution! l Two scenarios for the RSB transition l λ min : Random first order transition (RFOT) l λ max : de Almeida-Thouless instability (full RSB) α = 0.5,ρ = 0.1 tightened tightened 27/29

28 Summary Evaluation of the restricted isometry constant (RIC) can be formulated as a spin glass problem. We provided a replica based-framework for accurate evaluation of RICs. Replica evaluation provides the current best accuracy. Although the RS solution is not thermodynamically stable, the physical interpretation of RSB implies that the RS estimates still have the meaning of upper-bounds. The bounds are monotonically improved by taking the higher RSB into account. Future work Application to other matrix ensembles Mathematical justification 28/29

29 Thank you for your attention. Acknowledgments RIKEN special postdoctoral researcher (SPDR) fellowship, KAKENHI No (AS) KAKENHI No (YK) 29/29