Compressive Sensing with Random Matrices

Transcription

1 Compressive Sensing with Random Matrices Lucas Connell University of Georgia 9 November 017 Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

2 Overview We will discuss compressive sensing, and, in particular, the viability of using random matrices. An outline: 1 Introduce some vector notation and terminology Discuss Compressive Sensing 3 Review probability 4 Define random matrices and concentration of measure 5 Define an ɛ-covering number, and prove a short lemma 6 Use these tools to prove our main result Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November 017 / 18

3 Vector Notation The l N p norm of a vector x R N is defined as ( N ) 1 i=1 x l N p := x i p p if 0 < p < max x i if p = i=1,...,n Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

4 Vector Notation The l N p norm of a vector x R N is defined as ( N ) 1 i=1 x l N p := x i p p if 0 < p < max x i if p = i=1,...,n Definition (Sparsity) We say a vector x R N is k-sparse if #{i : x i 0} k Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

5 Vector Notation The l N p norm of a vector x R N is defined as ( N ) 1 i=1 x l N p := x i p p if 0 < p < max x i if p = i=1,...,n Definition (Sparsity) We say a vector x R N is k-sparse if #{i : x i 0} k In R N, with a fixed k N and given a set T = {i 1, i,..., i k } {1,,..., N} of indices, we define X T := {[x 1, x,..., x N ] T : x i = 0 for i T } which is a k-dimensional subspace of R N We similarly define x T Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

6 Examples Example (Sparse Vectors) Let 3 4 x = and y = x is 3-sparse. y is 1-sparse. Example (X T Subspace) Consider R 5, and set T = {1,, 4}. X T := {[x 1, x, 0, x 4, 0] T : x 1, x, x 4 R} Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

7 Motivation In many applications, one wants to solve Φx = b where x R N is k - sparse, Φ R n N, and n << N Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

8 Motivation In many applications, one wants to solve Φx = b where x R N is k - sparse, Φ R n N, and n << N This can be quite hard, and we instead solve: argmin x l N, such that Φx = b (1) 1 Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

9 Motivation In many applications, one wants to solve Φx = b where x R N is k - sparse, Φ R n N, and n << N This can be quite hard, and we instead solve: argmin x l N, such that Φx = b (1) 1 Theorem (Candes and Tao, Decoding by linear programming, 005) Suppose b = Φx, with x k-sparse. If Φ has the Restricted Isometry Property of order k, then x is the unique solution to (1) Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

10 Motivation In many applications, one wants to solve Φx = b where x R N is k - sparse, Φ R n N, and n << N This can be quite hard, and we instead solve: argmin x l N, such that Φx = b (1) 1 Theorem (Candes and Tao, Decoding by linear programming, 005) Suppose b = Φx, with x k-sparse. If Φ has the Restricted Isometry Property of order k, then x is the unique solution to (1) Theorem (Candes and Tao, Near-optimal signal recovery from random projections: Universal encoding strategies?, 006) If Φ R n N is a subgaussian random matrix with n cklog ( ) N k, then Φ satisfies the RIP of order k with overwhelming probability. Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

11 Probability Review Definition (Random Variable) A (real) random variable returns a real number with some probability determined by a probability distribution function. Example: Let X N (0, 1) (standard normal distribution) Figure: Pr [X 1] = Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

12 Probability Review Subgaussian Random Variable: A subgaussian random variable X is a random variable whose distribution function s tails decay exponentially fast. Or, more precisely, for any t R: Pr [ X µ t] e t σ Figure: Blue is standard normal; orange is subgaussian. Notice the red area (subgaussian) is less than the green area (standard normal) Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

13 Probability Review Union Bounds: Given random variables X 1, X, we know Pr [X 1 t or X t] Pr [X 1 t] + Pr [X t]. The same inequality holds for any finite number of random variables. Example Let X 1, X,..., X m N (0, 1) be independent. Using a finite union bound: Pr [X i t for at least one i = 1,,..., m] m Pr [X i t] = mpr [X 1 t] i=1 Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

14 Concentration of Measure The sum of (independent) random variables concentrates around the mean. Figure: Blue is standard normal; orange is where X 1,..., X 5 N (0, 1) 5 X i i=1 5 Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

15 Random Matrix Definition (Random Matrix) A random matrix, Φ R n N, is a matrix whose entries, φ i,j, are each (independent) random variables. Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

16 Random Matrix Definition (Random Matrix) A random matrix, Φ R n N, is a matrix whose entries, φ i,j, are each (independent) random variables. Theorem (Subgaussian Random Matrix) Letting q R N, ɛ (0, 1), and Φ be an n N matrix with entries φ i,j N (0, 1 n ), this concentration results in c 0(ɛ) > 0 with: [ Pr (1 ɛ) q l N Φq l n (1 + ɛ) q l N ] fails e nc 0(ɛ) Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

17 Covering Numbers Definition ( -Covering Number -Packing Number) 1 -covering number: N(U, ) = min n such that {q1, q,..., qn } with the property n [ U B (qi ) i=1 -packing number: P(U, ) = max m such that {q1, q,..., qm } with the property d(qi, qj ) >, i 6= j Figure: A space, U Lucas Connell (University of Georgia) Figure: A space, V Compressive Sensing with Random Matrices 9 November / 18

18 Covering Numbers Definition ( -Covering Number -Packing Number) 1 -covering number: N(U, ) = min n such that {q1, q,..., qn } with the property n [ U B (qi ) i=1 -packing number: P(U, ) = max m such that {q1, q,..., qm } with the property d(qi, qj ) >, i 6= j Figure: An -cover of our space U Lucas Connell (University of Georgia) Figure: An -packing in our space V Compressive Sensing with Random Matrices 9 November / 18

19 An Upper Bound for the ɛ-covering Number of a Unit Ball Lemma (Upper Bound for Covering of a Unit Ball in R k ) If U is the unit ball in R k and ɛ (0, 1), N(U, ɛ) ( 3 ɛ )k Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

20 An Upper Bound for the ɛ-covering Number of a Unit Ball Lemma (Upper Bound for Covering of a Unit Ball in R k ) If U is the unit ball in R k and ɛ (0, 1), N(U, ɛ) ( 3 ɛ )k It is known that N(U, ɛ) P(U, ɛ), so we bound the ɛ-packing number. Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

21 An Upper Bound for the ɛ-covering Number of a Unit Ball Lemma (Upper Bound for Covering of a Unit Ball in R k ) If U is the unit ball in R k and ɛ (0, 1), N(U, ɛ) ( 3 ɛ )k It is known that N(U, ɛ) P(U, ɛ), so we bound the ɛ-packing number. Let {q 1, q,..., q m } be an optimal ɛ-packing. Hence m B ɛ (q i) B 1+ ɛ (0) i=1 Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

22 An Upper Bound for the ɛ-covering Number of a Unit Ball Lemma (Upper Bound for Covering of a Unit Ball in R k ) If U is the unit ball in R k and ɛ (0, 1), N(U, ɛ) ( 3 ɛ )k It is known that N(U, ɛ) P(U, ɛ), so we bound the ɛ-packing number. Let {q 1, q,..., q m } be an optimal ɛ-packing. Hence m B ɛ (q i) B 1+ ɛ (0) i=1 Since d(q i, q j ) > ɛ, i j, these balls are actually disjoint. Taking volume [ m ] m [ ] ( ɛ ) k Vol B ɛ (q i) = Vol B ɛ (q i) = mc k i=1 i=1 [ ] ( Vol B 1+ ɛ (0) = C k 1 + ɛ ) k Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

23 An Upper Bound for the ɛ-covering Number of a Unit Ball Lemma (Upper Bound for Covering of a Unit Ball in R k ) If U is the unit ball in R k and ɛ (0, 1), N(U, ɛ) ( 3 ɛ )k It is known that N(U, ɛ) P(U, ɛ), so we bound the ɛ-packing number. Let {q 1, q,..., q m } be an optimal ɛ-packing. Hence m B ɛ (q i) B 1+ ɛ (0) i=1 Since d(q i, q j ) > ɛ, i j, these balls are actually disjoint. Taking volume [ m ] m [ ] ( ɛ ) k Vol B ɛ (q i) = Vol B ɛ (q i) = mc k i=1 i=1 [ ] ( Vol B 1+ ɛ (0) = C k 1 + ɛ ) k Rearranging these results ultimately yields: m ( ) +ɛ k ( ɛ 3 ) k ɛ Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

24 The Restricted Isometry Property (RIP) Definition We say a matrix Φ satisfies the RIP of order k if δ k (0, 1) such that, for any T with #(T ) k (1 δ k ) x T l N Φx T l n (1 + δ k) x T l N () Our goal is now to show that a random matrix, Φ with entries from a subgaussian distribution, satisfies the RIP of order k with overwhelming probability. Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

25 Major Theorem (See also 1 ) Theorem (Candes and Tao, Near-optimal signal recovery from random projections: Universal encoding strategies?, 006) Given δ (0, 1), c 1, c R + such that Φ satisfies the RIP of order k with probability greater than 1 e cn, provided n c 1 k log ( ) N k 1 Baraniuk et al., A Simple Proof of the Restricted Isometry Property for Random Matrices. Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

26 Major Theorem (See also 1 ) Theorem (Candes and Tao, Near-optimal signal recovery from random projections: Universal encoding strategies?, 006) Given δ (0, 1), c 1, c R + such that Φ satisfies the RIP of order k with probability greater than 1 e cn, provided n c 1 k log ( ) N k Fix a k satisfying the hypothesis, let T be a set of indices with #(T ) = k. Note here that there are ( ) N k choices for T. Now consider B T := X T B 1 (0) 1 Baraniuk et al., A Simple Proof of the Restricted Isometry Property for Random Matrices. Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

27 Major Theorem (See also 1 ) Theorem (Candes and Tao, Near-optimal signal recovery from random projections: Universal encoding strategies?, 006) Given δ (0, 1), c 1, c R + such that Φ satisfies the RIP of order k with probability greater than 1 e cn, provided n c 1 k log ( ) N k Fix a k satisfying the hypothesis, let T be a set of indices with #(T ) = k. Note here that there are ( ) N k choices for T. Now consider B T := X T B 1 (0) By our Lemma on covering numbers, can pick δ 4 -covering Q T = {q 1,..., q α } with α ( ) 1 k δ 1 Baraniuk et al., A Simple Proof of the Restricted Isometry Property for Random Matrices. Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

28 Via concentration of measure, for a given q i Q T : [( Pr 1 δ ) ( q i l N Φq i l n 1 + δ ) ] q i l N fails e nc 0( δ ) Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

29 Via concentration of measure, for a given q i Q T : [( Pr 1 δ ) ( q i l N Φq i l n 1 + δ ) ] q i l N fails e nc 0( δ ) Using a union bound on our finite number of points: [( Pr 1 δ ) ( q l N Φq l n 1 + δ ) ] q l fails for a q Q T N ( ) 1 k e nc 0( δ ) δ Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

30 Via concentration of measure, for a given q i Q T : [( Pr 1 δ ) ( q i l N Φq i l n 1 + δ ) ] q i l N fails e nc 0( δ ) Using a union bound on our finite number of points: [( Pr 1 δ ) ( q l N Φq l n 1 + δ ) ] q l fails for a q Q T N ( ) 1 k e nc 0( δ ) δ From this, one can show: [ Pr (1 δ) x l N Φx l n (1 + δ) x l N fails for an x X T ] ( ) 1 k e nc 0( δ ) δ Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

31 Since there are ( ) ( N k en ) k k such k-dimensional subspaces, use a union bound to see [ Pr (1 δ) x l N Φx l n (1 + δ) x l N fails for an x ] X T T ( 1 δ ) k ( ) en k e nc 0( δ ) = e nc 0( δ )+k[log( en k )+log( 1 δ )] k Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

32 Since there are ( ) ( N k en ) k k such k-dimensional subspaces, use a union bound to see [ Pr (1 δ) x l N Φx l n (1 + δ) x l N fails for an x ] X T T ( 1 δ ) k ( ) en k e nc 0( δ ) = e nc 0( δ )+k[log( en k )+log( 1 δ )] k By picking some constant c and taking the complement, [ Pr x ] X T, (1 δ) x l N Φx l n (1 + δ) x l N T 1 e nc Which is to say: Φ satisfies the RIP of order k specified in the theorem with at least the above probability. Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

33 Summary Want to compress a high-sparse vector, in a way where we can recover it with little-to-no error. How do we compress it? Compressing with a subgaussian matrix Φ allows for unique l 1 -minimization; an accurate recovery How small can we compress? We can pick n logarithmic to N. Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18

34 References Baraniuk, Richard et al. A Simple Proof of the Restricted Isometry Property for Random Matrices. In: Constructive Approximation 8.3 (Dec. 008), pp issn: doi: /s x. url: Candes, Emmanuel J and Terence Tao. Decoding by linear programming. In: IEEE transactions on information theory 51.1 (005), pp Near-optimal signal recovery from random projections: Universal encoding strategies?. In: IEEE transactions on information theory 5.1 (006), pp Lucas Connell (University of Georgia) Compressive Sensing with Random Matrices 9 November / 18