Estimating Unknown Sparsity in Compressed Sensing

Transcription

1 Estimating Unknown Sparsity in Compressed Sensing Miles Lopes UC Berkeley Department of Statistics CSGF Program Review July 16, 2014 early version published at ICML 2013 Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

2 Overview of compressed sensing (CS) Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

3 Traditional A/D conversion is wasteful (graphic by Mishali and Eldar, 2011) Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

4 Traditional A/D conversion is wasteful (graphic by Mishali and Eldar, 2011) Compressed sensing: Do acquisition and compression in one step! Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

5 Natural signals are compressible We should acquire the relevant 25k numbers, rather than acquire 1M numbers. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

6 A real example of compressed sensing single pixel camera, Wakin et al., 2006 Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

7 A real example of compressed sensing single pixel camera, Wakin et al., 2006 x = scene (signal vector in R p ) a i = grid of mirrors (binary vector in R p, i = 1,..., n) y i = a i, x + ɛ i, (diode output / linear measurement) Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

8 Mathematical setup Compressed sensing is a framework for estimating a sparse vector from a small number of linear measurements, In matrix notation, if a i y i = a i, x + ɛ i, i = 1,..., n. is ith row of A R n p then, y = Ax + ɛ. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

9 Mathematical setup Compressed sensing is a framework for estimating a sparse vector from a small number of linear measurements, In matrix notation, if a i y i = a i, x + ɛ i, i = 1,..., n. is ith row of A R n p then, y = Ax + ɛ. Goal: Design a matrix A and use noisy measurements y to accurately recover x. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

10 Mathematical setup Compressed sensing is a framework for estimating a sparse vector from a small number of linear measurements, In matrix notation, if a i y i = a i, x + ɛ i, i = 1,..., n. is ith row of A R n p then, y = Ax + ɛ. Goal: Design a matrix A and use noisy measurements y to accurately recover x. Obstacle: Dimension of x R p is large, and number of measurements is small: n p. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

11 Mathematical setup Compressed sensing is a framework for estimating a sparse vector from a small number of linear measurements, In matrix notation, if a i y i = a i, x + ɛ i, i = 1,..., n. is ith row of A R n p then, y = Ax + ɛ. Goal: Design a matrix A and use noisy measurements y to accurately recover x. Obstacle: Dimension of x R p is large, and number of measurements is small: n p. Structure: The signal x is sparse in some known basis (e.g. wavelet) Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

12 Mathematical setup Compressed sensing is a framework for estimating a sparse vector from a small number of linear measurements, In matrix notation, if a i y i = a i, x + ɛ i, i = 1,..., n. is ith row of A R n p then, y = Ax + ɛ. Goal: Design a matrix A and use noisy measurements y to accurately recover x. Obstacle: Dimension of x R p is large, and number of measurements is small: n p. Structure: The signal x is sparse in some known basis (e.g. wavelet) Note: The matrix A is singular; short and wide If x is sparse in an orthonormal basis U R p p, this can be ignored abstractly by choosing A := ÃU, giving observations y = Ã(Ux) + ɛ. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

13 How do we recover x from compressed measurements? Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

14 The fundamental theorem of compressed sensing Suppose we have y = Ax + ɛ with noise bounded by ɛ 2 ɛ 0. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

15 The fundamental theorem of compressed sensing Suppose we have y = Ax + ɛ with noise bounded by ɛ 2 ɛ 0. Let x be a solution to the convex optimization problem minimize v 1 subject to Av y 2 ɛ 0, v R p. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

16 The fundamental theorem of compressed sensing Suppose we have y = Ax + ɛ with noise bounded by ɛ 2 ɛ 0. Let x be a solution to the convex optimization problem minimize v 1 subject to Av y 2 ɛ 0, v R p. Define the sparsity x 0 := #{j : x j 0}. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

17 The fundamental theorem of compressed sensing Suppose we have y = Ax + ɛ with noise bounded by ɛ 2 ɛ 0. Let x be a solution to the convex optimization problem minimize v 1 subject to Av y 2 ɛ 0, v R p. Define the sparsity x 0 := #{j : x j 0}. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

18 The fundamental theorem of compressed sensing Suppose we have y = Ax + ɛ with noise bounded by ɛ 2 ɛ 0. Let x be a solution to the convex optimization problem minimize v 1 subject to Av y 2 ɛ 0, v R p. Define the sparsity x 0 := #{j : x j 0}. Theorem If A R n p is randomly drawn from a suitable ensemble, and ( ) n x 0 log p x 0, then with high probability x x 2 ɛ 0. cf. Candès, Romberg, and Tao, 06, Donoho 06 Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

19 The issue of unknown sparsity Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

20 unknown sparsity Many aspects of CS depend on x 0 = #{j : x j 0}, but this number is typically unknown. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

21 unknown sparsity Many aspects of CS depend on x 0 = #{j : x j 0}, but this number is typically unknown. How do we know if x is sparse at all (in a given basis)? Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

22 unknown sparsity Many aspects of CS depend on x 0 = #{j : x j 0}, but this number is typically unknown. How do we know if x is sparse at all (in a given basis)? How do we know if we have taken enough measurements?, i.e. n x 0 log(p/ x 0 )? Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

23 unknown sparsity Many aspects of CS depend on x 0 = #{j : x j 0}, but this number is typically unknown. How do we know if x is sparse at all (in a given basis)? How do we know if we have taken enough measurements?, i.e. n x 0 log(p/ x 0 )? If we knew x 0, we could take just enough measurements. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

24 unknown sparsity Many aspects of CS depend on x 0 = #{j : x j 0}, but this number is typically unknown. How do we know if x is sparse at all (in a given basis)? How do we know if we have taken enough measurements?, i.e. n x 0 log(p/ x 0 )? If we knew x 0, we could take just enough measurements. Realistic signals x R p do not have coordinates exactly equal to 0. In this case, the value x 0 = p is not descriptive of x. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

25 unknown sparsity Many aspects of CS depend on x 0 = #{j : x j 0}, but this number is typically unknown. How do we know if x is sparse at all (in a given basis)? How do we know if we have taken enough measurements?, i.e. n x 0 log(p/ x 0 )? If we knew x 0, we could take just enough measurements. Realistic signals x R p do not have coordinates exactly equal to 0. In this case, the value x 0 = p is not descriptive of x. = Even if we could estimate x 0 perfectly, it wouldn t help in practice. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

26 unknown sparsity Many aspects of CS depend on x 0 = #{j : x j 0}, but this number is typically unknown. How do we know if x is sparse at all (in a given basis)? How do we know if we have taken enough measurements?, i.e. n x 0 log(p/ x 0 )? If we knew x 0, we could take just enough measurements. Realistic signals x R p do not have coordinates exactly equal to 0. In this case, the value x 0 = p is not descriptive of x. = Even if we could estimate x 0 perfectly, it wouldn t help in practice. = x 0 is the wrong measure of sparsity to estimate We want to estimate a more realistic measure of sparsity! Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

27 An alternative view of sparsity Idea: For any non-zero vector in x R p, we can put a distribution on {1,..., p}. Consider the probability vector (π 1 (x),..., π p (x)) given by π j (x) = xj x 1. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

28 An alternative view of sparsity Idea: For any non-zero vector in x R p, we can put a distribution on {1,..., p}. Consider the probability vector (π 1 (x),..., π p (x)) given by π j (x) = xj x 1. Suppose we draw a random index J π(x). J is most likely to fall on the indices where x j is large. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

29 An alternative view of sparsity Idea: For any non-zero vector in x R p, we can put a distribution on {1,..., p}. Consider the probability vector (π 1 (x),..., π p (x)) given by π j (x) = xj x 1. Suppose we draw a random index J π(x). J is most likely to fall on the indices where x j is large. effective states of J = effective coordinates of x Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

30 Connecting entropy and sparsity Fact: If H(J) is the Shannon entropy of J, then # of effective states of J exp(h(j)). Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

31 Connecting entropy and sparsity Fact: If H(J) is the Shannon entropy of J, then # of effective states of J exp(h(j)). A more general measure of entropy is the Rényi entropy H q (J) = 1 1 q log p i=1 πq i, where π i = P(J = i) = xi x 1, and q [0, ]. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

32 Connecting entropy and sparsity Fact: If H(J) is the Shannon entropy of J, then # of effective states of J exp(h(j)). A more general measure of entropy is the Rényi entropy H q (J) = 1 1 q log p i=1 πq i, where π i = P(J = i) = xi x 1, and q [0, ]. Recall x q q = i x i q. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

33 Connecting entropy and sparsity Fact: If H(J) is the Shannon entropy of J, then # of effective states of J exp(h(j)). A more general measure of entropy is the Rényi entropy H q (J) = 1 1 q log p i=1 πq i, where π i = P(J = i) = xi x 1, and q [0, ]. Recall x q q = i x i q. We now define the numerical sparsity: ( ) q x q 1 q s q (x) := exp(h q (J)) =. x 1 It can be checked that as q 0, we obtain s q (x) x 0. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

34 s 2 (x) = x 2 1 x 2 2 = effective number of coordinates vectors in R 100 and s 2 (x) values coordinate value s 2 (x) = 16.4, x 0 = 100 s 2 (x) = 32.7, x 0 = 45 s 2 (x) = 66.6, x 0 = coordinate index Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

35 s 2 (x) = x 2 1 x 2 2 = effective number of coordinates x sub-level set s 2 (x) 1.1 in R x 1 x sub-level set s 2 (x) 1.9 in R x 1 Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

36 ( ) q x q 1 q Properties of s q (x) := x 1. 1 Continuous function of x away from origin whenever q > 0. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

37 ( ) q x q 1 q Properties of s q (x) := x 1. 1 Continuous function of x away from origin whenever q > 0. 2 For all non-zero x, we have 1 s q (x) p. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

38 ( ) q x q 1 q Properties of s q (x) := x 1. 1 Continuous function of x away from origin whenever q > 0. 2 For all non-zero x, we have 1 s q (x) p. 3 When q 0, it matches hard sparsity, s q (x) x 0. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

39 ( ) q x q 1 q Properties of s q (x) := x 1. 1 Continuous function of x away from origin whenever q > 0. 2 For all non-zero x, we have 1 s q (x) p. 3 When q 0, it matches hard sparsity, s q (x) x 0. 4 Scale-invariance, s q (x) = s q (cx) Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

40 ( ) q x q 1 q Properties of s q (x) := x 1. 1 Continuous function of x away from origin whenever q > 0. 2 For all non-zero x, we have 1 s q (x) p. 3 When q 0, it matches hard sparsity, s q (x) x 0. 4 Scale-invariance, s q (x) = s q (cx) 5 General lower bound for all non-zero x, s q (x) x 0 Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

41 How does s 2 (x) relate to signal recovery? Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

42 A link between s 2 (x) and signal recovery Theorem (L., 2014) Suppose the compressed sensing model holds with x 0, n p, and ɛ 2 ɛ 0. Also assume the matrix A R n p is randomly drawn from a suitable ensemble. Let x argmin{ v 1 : Av y 2 ɛ 0, v R p }. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

43 A link between s 2 (x) and signal recovery Theorem (L., 2014) Suppose the compressed sensing model holds with x 0, n p, and ɛ 2 ɛ 0. Also assume the matrix A R n p is randomly drawn from a suitable ensemble. Let x argmin{ v 1 : Av y 2 ɛ 0, v R p }. Then, there are absolute constants c 1, c 2, c 3 > 0 such that for any n 1, the event x x 2 ɛ x 2 c 0 s 1 x 2 + c 2(x) pe 2 n log( n ) occurs with probability at least 1 2 exp( c 3 n). Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

44 A link between s 2 (x) and signal recovery Theorem (L., 2014) Suppose the compressed sensing model holds with x 0, n p, and ɛ 2 ɛ 0. Also assume the matrix A R n p is randomly drawn from a suitable ensemble. Let x argmin{ v 1 : Av y 2 ɛ 0, v R p }. Then, there are absolute constants c 1, c 2, c 3 > 0 such that for any n 1, the event x x 2 ɛ x 2 c 0 s 1 x 2 + c 2(x) pe 2 n log( n ) occurs with probability at least 1 2 exp( c 3 n). essential points: s 2(x) offers a meaningful way of measuring sample complexity. This result applies to any non-zero x (hard sparsity is not required). Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

45 How do we estimate s q (x)? Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

46 How do we estimate s q (x)? Recall the relation ( ) q x q 1 q s q (x) =. x 1 = It s enough just to estimate x q for general q. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

47 Estimation scheme for q = 2 and generalizations Let x R p be a fixed signal. The case q = 2. If a i R p has i.i.d. N(0, 1) coordinates, then a i, x x 2 N(0, 1) Hence, if we had noiseless measurements y i = a i, x, then y 1,..., y n, is a sample from N(0, x 2 2). Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

48 Estimation scheme for q = 2 and generalizations Let x R p be a fixed signal. The case q = 2. If a i R p has i.i.d. N(0, 1) coordinates, then a i, x x 2 N(0, 1) Hence, if we had noiseless measurements y i = a i, x, then y 1,..., y n, is a sample from N(0, x 2 2). idea: Estimating x 2 2 amounts to estimating the scale parameter (variance) Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

49 Estimation scheme for q = 2 and generalizations Let x R p be a fixed signal. The case q = 2. If a i R p has i.i.d. N(0, 1) coordinates, then a i, x x 2 N(0, 1) Hence, if we had noiseless measurements y i = a i, x, then y 1,..., y n, is a sample from N(0, x 2 2). idea: Estimating x 2 2 amounts to estimating the scale parameter (variance) Note: This doesn t depend on dimension or sparsity of x only the norm. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

50 Estimation scheme for q = 2 and generalizations Let x R p be a fixed signal. The case q = 2. If a i R p has i.i.d. N(0, 1) coordinates, then a i, x x 2 N(0, 1) Hence, if we had noiseless measurements y i = a i, x, then y 1,..., y n, is a sample from N(0, x 2 2). idea: Estimating x 2 2 amounts to estimating the scale parameter (variance) Note: This doesn t depend on dimension or sparsity of x only the norm. The general case q (0, 2]. If a i R p has i.i.d. coordinates drawn from a stable q (0, 1) distribution then, a i, x x q stable q (0, 1) = estimate x q by viewing it as a scale parameter of an i.i.d. sample! Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

51 Summary Derived s q (x) as a generalization of x 0 Showed that s q (x) measures sample complexity for recovery of x. Derived an estimator for s q (x) that does not rely on sparsity assumptions and is dimension free Proved a CLT for the estimator and obtained exact formulas for asymptotic variance = confidence intervals/hypothesis tests related to sparsity Validated asymptotic calculations with simulations. Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21

52 Thank you for your attention. Special thanks to: Thesis advisor, Peter Bickel at UC Berkeley Practicum Advisor, Philip Kegelmeyer at Sandia Livermore CSGF fellowship, grant DE-FG02-97ER25308 Everyone at the Krell Institute for making the fellowship possible Miles Lopes ( UC Berkeley ) estimating unknown sparsity July 16, / 21