Compressed Sensing: Lecture I. Ronald DeVore

Transcription

1 Compressed Sensing: Lecture I Ronald DeVore

2 Motivation Compressed Sensing is a new paradigm for signal/image/function acquisition

3 Motivation Compressed Sensing is a new paradigm for signal/image/function acquisition To understand the motivation for this new area we shall consider the classical approach to signal and image acquisition

4 Motivation Compressed Sensing is a new paradigm for signal/image/function acquisition To understand the motivation for this new area we shall consider the classical approach to signal and image acquisition This will expose some deficiencies in the classical approach

5 Motivation Compressed Sensing is a new paradigm for signal/image/function acquisition To understand the motivation for this new area we shall consider the classical approach to signal and image acquisition This will expose some deficiencies in the classical approach The goal of Compressed Sensing (CS) is to remove these deficiencies

6 Motivation Compressed Sensing is a new paradigm for signal/image/function acquisition To understand the motivation for this new area we shall consider the classical approach to signal and image acquisition This will expose some deficiencies in the classical approach The goal of Compressed Sensing (CS) is to remove these deficiencies We shall see that the remedy is a merging of ideas from functional analysis, approximation, and probability

7 Traditional Signal Processing Model signals as bandlimited functions x(t) Support of ˆx contained in [ Ωπ, Ωπ] Shannon-Nyquist says x can be exactly represented in terms of translates S(t nh) of a Sinc function S. Coefficients in this representation are uniform time samples x(nh) with spacing h 1 Ω These samples allow for exact reconstruction (Nyquist rate) A/D Converters: quantize these samples Problem: If Ω is too large we cant build circuitry to sample faithfully at the desired rate

8 Traditional Image Compression Models images as bivariate functions F(x, y) Pixel values are averages of F over small squares Sensors generate an array of quantized pixel values giving the piecewise constant function F Compression: Transform pixel values to coefficients in some basis representation: Fourier, wavelets, curvelets, etc. Compute all coefficients c λ of F in this basis Retain only quantized versions c λ of only the largest of these coefficients Problem: We take many samples, compute many coefficients, but retain only a few. Is it possible to take and store only a few samples from the very beginning?

9 Why there is hope The belief is that real world signals/images are sparse or very compressible in a suitable basis Thus the information content in the real world signals is small CS models signals by means of this sparsity or compressibility This is in contrast to the Shannon model of bandlimited CS will try to take advantage of this sparsity by designing new ways to sample signals and therefore new sensors Real world signals are analog but we will develop the mathematics in the discrete setting Some massaging must be done to move from discrete to analog

10 Discrete Compressed Sensing x IR N with N large

11 Discrete Compressed Sensing x IR N with N large We are able to ask n questions about x

12 Discrete Compressed Sensing x IR N with N large We are able to ask n questions about x Question means inner product v x with v IR N - called sample

13 Discrete Compressed Sensing x IR N with N large We are able to ask n questions about x Question means inner product v x with v IR N - called sample What are the best questions to ask??

14 Discrete Compressed Sensing x IR N with N large We are able to ask n questions about x Question means inner product v x with v IR N - called sample What are the best questions to ask?? Any such sampling is given by Φx where Φ is an n N matrix

15 Discrete Compressed Sensing x IR N with N large We are able to ask n questions about x Question means inner product v x with v IR N - called sample What are the best questions to ask?? Any such sampling is given by Φx where Φ is an n N matrix We are interested in the good / best matrices Φ

16 Discrete Compressed Sensing x IR N with N large We are able to ask n questions about x Question means inner product v x with v IR N - called sample What are the best questions to ask?? Any such sampling is given by Φx where Φ is an n N matrix We are interested in the good / best matrices Φ Here good means the samples y = Φx contain enough information to approximate x well

17 Discrete Compressed Sensing x IR N with N large We are able to ask n questions about x Question means inner product v x with v IR N - called sample What are the best questions to ask?? Any such sampling is given by Φx where Φ is an n N matrix We are interested in the good / best matrices Φ Here good means the samples y = Φx contain enough information to approximate x well How can we make this problem precise??

18 Encoder/Decoder We view Φ as an encoder

19 Encoder/Decoder We view Φ as an encoder Since Φ : IR N IR n many x are encoded with same y

20 Encoder/Decoder We view Φ as an encoder Since Φ : IR N IR n many x are encoded with same y N := {η : Φη = 0} the null space of Φ

21 Encoder/Decoder We view Φ as an encoder Since Φ : IR N IR n many x are encoded with same y N := {η : Φη = 0} the null space of Φ F(y) := {x : Φx = y} = x 0 + N for any x 0 F(y)

22 Encoder/Decoder We view Φ as an encoder Since Φ : IR N IR n many x are encoded with same y N := {η : Φη = 0} the null space of Φ F(y) := {x : Φx = y} = x 0 + N for any x 0 F(y) The hyperplanes F(y) with y IR n stratify IR N

23 ) 1 ( y F 2 ) ( y F k ) ( y F The sets F(y)

24 Encoder/Decoder We view Φ as an encoder Since Φ : IR N IR n many x are encoded with same y N := {η : Φη = 0} the null space of Φ F(y) := {x : Φx = y} = x 0 + N for any x 0 F(y) The hyperplanes F(y) with y IR n stratify IR N Decoder is any (possibly nonlinear) mapping from IR n IR N

25 Encoder/Decoder We view Φ as an encoder Since Φ : IR N IR n many x are encoded with same y N := {η : Φη = 0} the null space of Φ F(y) := {x : Φx = y} = x 0 + N for any x 0 F(y) The hyperplanes F(y) with y IR n stratify IR N Decoder is any (possibly nonlinear) mapping from IR n IR N x := (Φ(x)) is our approximation to x from the information extracted

26 Encoder/Decoder We view Φ as an encoder Since Φ : IR N IR n many x are encoded with same y N := {η : Φη = 0} the null space of Φ F(y) := {x : Φx = y} = x 0 + N for any x 0 F(y) The hyperplanes F(y) with y IR n stratify IR N Decoder is any (possibly nonlinear) mapping from IR n IR N x := (Φ(x)) is our approximation to x from the information extracted Let A := A n,n := {(Φ, ) : Φ is n N}

27 Measuring Sparsity We can say nothing about the performance of an encoder/decoder without some knowledge about x

28 Measuring Sparsity We can say nothing about the performance of an encoder/decoder without some knowledge about x CS assumes the signal is either sparse or compressible

29 Measuring Sparsity We can say nothing about the performance of an encoder/decoder without some knowledge about x CS assumes the signal is either sparse or compressible Measuring Sparsity

30 Measuring Sparsity We can say nothing about the performance of an encoder/decoder without some knowledge about x CS assumes the signal is either sparse or compressible Measuring Sparsity The support of x is supp(x) := {i : x i 0}

31 Measuring Sparsity We can say nothing about the performance of an encoder/decoder without some knowledge about x CS assumes the signal is either sparse or compressible Measuring Sparsity The support of x is supp(x) := {i : x i 0} Σ k := {x : #supp(x) k}

32 Measuring Sparsity We can say nothing about the performance of an encoder/decoder without some knowledge about x CS assumes the signal is either sparse or compressible Measuring Sparsity The support of x is supp(x) := {i : x i 0} Σ k := {x : #supp(x) k} Measuring Compressibility

33 Measuring Sparsity We can say nothing about the performance of an encoder/decoder without some knowledge about x CS assumes the signal is either sparse or compressible Measuring Sparsity The support of x is supp(x) := {i : x i 0} Σ k := {x : #supp(x) k} Measuring Compressibility Let X be a sequence space norm: typical examples are l p norms

34 Measuring Sparsity We can say nothing about the performance of an encoder/decoder without some knowledge about x CS assumes the signal is either sparse or compressible Measuring Sparsity The support of x is supp(x) := {i : x i 0} Σ k := {x : #supp(x) k} Measuring Compressibility Let X be a sequence space norm: typical examples are l p norms x lp := { N i=1 x i p } 1/p

35 Measuring Sparsity We can say nothing about the performance of an encoder/decoder without some knowledge about x CS assumes the signal is either sparse or compressible Measuring Sparsity The support of x is supp(x) := {i : x i 0} Σ k := {x : #supp(x) k} Measuring Compressibility Let X be a sequence space norm: typical examples are l p norms x lp := { N i=1 x i p } 1/p σ k (x) X := inf z Σk x z X

36 Measuring Sparsity We can say nothing about the performance of an encoder/decoder without some knowledge about x CS assumes the signal is either sparse or compressible Measuring Sparsity The support of x is supp(x) := {i : x i 0} Σ k := {x : #supp(x) k} Measuring Compressibility Let X be a sequence space norm: typical examples are l p norms x lp := { N i=1 x i p } 1/p σ k (x) X := inf z Σk x z X σ k (x) X measures compressibility of x

37 Measuring Sparsity We can say nothing about the performance of an encoder/decoder without some knowledge about x CS assumes the signal is either sparse or compressible Measuring Sparsity The support of x is supp(x) := {i : x i 0} Σ k := {x : #supp(x) k} Measuring Compressibility Let X be a sequence space norm: typical examples are l p norms x lp := { N i=1 x i p } 1/p σ k (x) X := inf z Σk x z X σ k (x) X measures compressibility of x If x l 1 then σ k (x) x k 1/2

38 I. First performance measure for Φ Given k,n we can ask for the smallest value of n for which there is an n N matrix such that each vector in Σ k is captured exactly by the information y = Φx

39 I. First performance measure for Φ Given k,n we can ask for the smallest value of n for which there is an n N matrix such that each vector in Σ k is captured exactly by the information y = Φx There is a such that (Φx) = x for all x Σ k

40 I. First performance measure for Φ Given k,n we can ask for the smallest value of n for which there is an n N matrix such that each vector in Σ k is captured exactly by the information y = Φx There is a such that (Φx) = x for all x Σ k Key to understanding when this happens are the following matrices Φ T

41 I. First performance measure for Φ Given k,n we can ask for the smallest value of n for which there is an n N matrix such that each vector in Σ k is captured exactly by the information y = Φx There is a such that (Φx) = x for all x Σ k Key to understanding when this happens are the following matrices Φ T Given any set T contained in {1,...,N}, the matrix Φ T is the submatrix of Φ obtained from the columns T

42 I. First performance measure for Φ Given k,n we can ask for the smallest value of n for which there is an n N matrix such that each vector in Σ k is captured exactly by the information y = Φx There is a such that (Φx) = x for all x Σ k Key to understanding when this happens are the following matrices Φ T Given any set T contained in {1,...,N}, the matrix Φ T is the submatrix of Φ obtained from the columns T Similarly x T is the restriction of x to T

43 Simple Solution to this problem Theorem: If Φ is any n N matrix and 2k n, then the following are equivalent: (i) There is a decoder such that (Φ(x)) = x, for all x Σ k, (ii) Σ 2k N(Φ) = {0}, (iii) For all sets T with #T = 2k, the matrix Φ T has rank 2k. (iv) For all sets T with #T = 2k, the 2k 2k positive definite matrix Φ T Φ T is invertible and therefore all its eigenvalues are positive.

44 Simple Proof The equivalence of (ii), (iii), (iv) is linear algebra. (i) (ii). Suppose (i) holds and x Σ 2k N. We can write x = x 0 x 1 where both x 0,x 1 Σ k. Since Φ(x 0 ) = Φ(x 1 ), we have, by (i), that x 0 = x 1 and hence x = x 0 x 1 = 0. (ii) (i). Given any y IR n, we define (y) to be any element in F(y) with smallest support. This defines the decoder. We need to check that each for each x Σ k, the vector y = Φ(x) is uniquely decoded as x. To see this, suppose x 1,x 2 Σ k with y = Φ(x 1 ) = Φ(x 2 ), then x 1 x 2 N Σ 2k. From (ii), this means that x 1 = x 2. Hence, if x Σ k then (Φ(x)) = x as desired.

45 Optimal Matrices for I. n = 2k is the solution to our first problem

46 Optimal Matrices for I. n = 2k is the solution to our first problem Given k we can construct matrices Φ of size 2k N with the properties of the theorem?

47 Optimal Matrices for I. n = 2k is the solution to our first problem Given k we can construct matrices Φ of size 2k N with the properties of the theorem? Vandermonde matrix. Choose x 1 < x 2 < < x N x 1 x 2 x N Φ := x n 1 1 x n 1 2 x n 1 N

48 Optimal Matrices for I. n = 2k is the solution to our first problem Given k we can construct matrices Φ of size 2k N with the properties of the theorem? Vandermonde matrix. Choose x 1 < x 2 < < x N x 1 x 2 x N Φ := x n 1 1 x n 1 2 x n 1 N Problem with these matrices is they are not stable in computation: Φ t T Φ T has large condition number

49 Second measure of optimality Optimality on classes

50 Second measure of optimality Optimality on classes Let X be a sequence norm on IR N which will be used to measure error

51 Second measure of optimality Optimality on classes Let X be a sequence norm on IR N which will be used to measure error Let K be a compact set in IR N

52 Second measure of optimality Optimality on classes Let X be a sequence norm on IR N which will be used to measure error Let K be a compact set in IR N Best performance on the class K in norm X E n,n (K) X := inf sup (Φ, ) A n,n x K x (Φ(x)) X

53 Second measure of optimality Optimality on classes Let X be a sequence norm on IR N which will be used to measure error Let K be a compact set in IR N Best performance on the class K in norm X E n,n (K) X := inf sup (Φ, ) A n,n x K x (Φ(x)) X Typical example is X = l p = l N p and K = U(l q ) with q < p

54 Gelfand widths E n,n (K) X is closely related to Gelfand widths d n (K) X := inf codim(y )=n sup x K Y x X

55 . The set K Y N Y =

56 Gelfand widths E n,n (K) X is closely related to Gelfand widths d n (K) X := inf codim(y )=n sup x K Y x X Theorem: If K = K and K + K CK, then d n (K) X E n,n (K) X Cd n (K) X

57 Gelfand widths E n,n (K) X is closely related to Gelfand widths d n (K) X := inf codim(y )=n sup x K Y x X Theorem: If K = K and K + K CK, then d n (K) X E n,n (K) X Cd n (K) X Easy to prove: Y N

58 Gelfand widths E n,n (K) X is closely related to Gelfand widths d n (K) X := inf codim(y )=n sup x K Y x X Theorem: If K = K and K + K CK, then d n (K) X E n,n (K) X Cd n (K) X Easy to prove: Y N Gelfand widths of U(l q ) in l p are all known

59 Gelfand widths E n,n (K) X is closely related to Gelfand widths d n (K) X := inf codim(y )=n sup x K Y x X Theorem: If K = K and K + K CK, then d n (K) X E n,n (K) X Cd n (K) X Easy to prove: Y N Gelfand widths of U(l q ) in l p are all known Example C 1 log(n/n) n d n (U(l 1 )) l2 C 2 log(n/n) n

60 Gelfand widths E n,n (K) X is closely related to Gelfand widths d n (K) X := inf codim(y )=n sup x K Y x X Theorem: If K = K and K + K CK, then d n (K) X E n,n (K) X Cd n (K) X Easy to prove: Y N Gelfand widths of U(l q ) in l p are all known Example C 1 log(n/n) n d n (U(l 1 )) l2 C 2 log(n/n) n Kashin 1977 (Improvement in logarithm by Gluskin-Garnaev)

61 What are optimal matrices for II Kashin; Gluskin; Candes-Romberg-Tao

62 What are optimal matrices for II Kashin; Gluskin; Candes-Romberg-Tao Restricted Isometry Property (RIP) of order k: There exists 0 < δ = δ k < 1 such that (1 δ) x 2 l N 2 Φ(x) 2 l n 2 (1 + δ) x 2 l N 2, x Σ k

63 What are optimal matrices for II Kashin; Gluskin; Candes-Romberg-Tao Restricted Isometry Property (RIP) of order k: There exists 0 < δ = δ k < 1 such that (1 δ) x 2 l N 2 Φ(x) 2 l n 2 (1 + δ) x 2 l N 2, x Σ k Equivalently the eigenvalues of Φ T Φ T are in [1 δ, 1 + δ] if #(T) k

64 What are optimal matrices for II Kashin; Gluskin; Candes-Romberg-Tao Restricted Isometry Property (RIP) of order k: There exists 0 < δ = δ k < 1 such that (1 δ) x 2 l N 2 Φ(x) 2 l n 2 (1 + δ) x 2 l N 2, x Σ k Equivalently the eigenvalues of Φ T Φ T are in [1 δ, 1 + δ] if #(T) k Here the matrix norm is the spectral norm

65 What are optimal matrices for II Kashin; Gluskin; Candes-Romberg-Tao Restricted Isometry Property (RIP) of order k: There exists 0 < δ = δ k < 1 such that (1 δ) x 2 l N 2 Φ(x) 2 l n 2 (1 + δ) x 2 l N 2, x Σ k Equivalently the eigenvalues of Φ T Φ T are in [1 δ, 1 + δ] if #(T) k Here the matrix norm is the spectral norm The larger k the better Φ will encode

66 What are optimal matrices for II Kashin; Gluskin; Candes-Romberg-Tao Restricted Isometry Property (RIP) of order k: There exists 0 < δ = δ k < 1 such that (1 δ) x 2 l N 2 Φ(x) 2 l n 2 (1 + δ) x 2 l N 2, x Σ k Equivalently the eigenvalues of Φ T Φ T are in [1 δ, 1 + δ] if #(T) k Here the matrix norm is the spectral norm The larger k the better Φ will encode Recall our earlier condition for Problem I which only requires invertibility for each T but no uniform bound on norms of inverses

67 RIP Implies Optimal Performance If Φ has RIP for k then for any q 1 there is a decoder such that x (Φ(x)) l2 Ck 1/q+1/2 x lq

68 RIP Implies Optimal Performance If Φ has RIP for k then for any q 1 there is a decoder such that x (Φ(x)) l2 Ck 1/q+1/2 x lq The constant C > 0 depends only on δ in RIP

69 RIP Implies Optimal Performance If Φ has RIP for k then for any q 1 there is a decoder such that x (Φ(x)) l2 Ck 1/q+1/2 x lq The constant C > 0 depends only on δ in RIP This can be proved rather easily

70 RIP Implies Optimal Performance If Φ has RIP for k then for any q 1 there is a decoder such that x (Φ(x)) l2 Ck 1/q+1/2 x lq The constant C > 0 depends only on δ in RIP This can be proved rather easily Given n,n what is the largest value of k for which we can find Φ satisfying RIP?

71 RIP Implies Optimal Performance If Φ has RIP for k then for any q 1 there is a decoder such that x (Φ(x)) l2 Ck 1/q+1/2 x lq The constant C > 0 depends only on δ in RIP This can be proved rather easily Given n,n what is the largest value of k for which we can find Φ satisfying RIP? There is a constant c 0 such that whenever k there is a Φ satisfying RIP for this value of k c 0n log N/n

72 RIP Implies Optimal Performance If Φ has RIP for k then for any q 1 there is a decoder such that x (Φ(x)) l2 Ck 1/q+1/2 x lq The constant C > 0 depends only on δ in RIP This can be proved rather easily Given n,n what is the largest value of k for which we can find Φ satisfying RIP? There is a constant c 0 such that whenever k there is a Φ satisfying RIP for this value of k c 0n log N/n For example, this gives a rather simple proof of the optimal upper bound for widths d n (U(l 1 )) l2 C log(n/n) n

73 Building matrices How can we construct Φ with RIP for the above range of k?

74 Building matrices How can we construct Φ with RIP for the above range of k? We choose at random N vectors from the unit sphere in IR n and use these as the columns of Φ

75 Building matrices How can we construct Φ with RIP for the above range of k? We choose at random N vectors from the unit sphere in IR n and use these as the columns of Φ We choose each entry of Φ independently and at random from the Gaussian distribution with mean 0 and variance n 1/2

76 Building matrices How can we construct Φ with RIP for the above range of k? We choose at random N vectors from the unit sphere in IR n and use these as the columns of Φ We choose each entry of Φ independently and at random from the Gaussian distribution with mean 0 and variance n 1/2 Can use Bernouli process and create a matrix with entries ±1/ n

77 Building matrices How can we construct Φ with RIP for the above range of k? We choose at random N vectors from the unit sphere in IR n and use these as the columns of Φ We choose each entry of Φ independently and at random from the Gaussian distribution with mean 0 and variance n 1/2 Can use Bernouli process and create a matrix with entries ±1/ n With high probability any of the above constructions result in a matrix which satisfies RIP for the large range of k

78 Building matrices How can we construct Φ with RIP for the above range of k? We choose at random N vectors from the unit sphere in IR n and use these as the columns of Φ We choose each entry of Φ independently and at random from the Gaussian distribution with mean 0 and variance n 1/2 Can use Bernouli process and create a matrix with entries ±1/ n With high probability any of the above constructions result in a matrix which satisfies RIP for the large range of k Problem: None of these are constructive. In this sense we are back to the 1970 s and Kashin

79 Verification of RIP Let Φ(ω) = (φ i,j (ω)), ω Ω, be random matrices

80 Verification of RIP Let Φ(ω) = (φ i,j (ω)), ω Ω, be random matrices Each φ i,j is an independent realization of some fixed random variable r with mean zero and variance 1/n

81 Verification of RIP Let Φ(ω) = (φ i,j (ω)), ω Ω, be random matrices Each φ i,j is an independent realization of some fixed random variable r with mean zero and variance 1/n Trivially we have E( Φ(ω)x 2 l ) = x 2 n 2 l N 2

82 Verification of RIP Let Φ(ω) = (φ i,j (ω)), ω Ω, be random matrices Each φ i,j is an independent realization of some fixed random variable r with mean zero and variance 1/n Trivially we have E( Φ(ω)x 2 l n 2 ) = x 2 l N 2 For many processes we have concentration inequality Prob( Φ(ω)x 2 l n 2 x 2 l N 2 δ x 2 l N 2 ) Ce c(δ)n

83 Verification of RIP Let Φ(ω) = (φ i,j (ω)), ω Ω, be random matrices Each φ i,j is an independent realization of some fixed random variable r with mean zero and variance 1/n Trivially we have E( Φ(ω)x 2 l n 2 ) = x 2 l N 2 For many processes we have concentration inequality Prob( Φ(ω)x 2 l n 2 x 2 l N 2 δ x 2 l N 2 ) Ce c(δ)n For example, this condition is well-known and not difficult to check for Gaussian, Bernouli, etc. Theorem (Baraniuk-Davenport-DeVore-Wakin) Given c 0 there is a c 1 > 0 such that with probability 1 e c 1n, the matrix Φ(ω) satisfies RIP of order k with constant δ for all k c 0 n/ log(n/n)

84 Sketch Proof of Theorem We find a net of points P which cover the unit sphere in Σ k to accuracy δ/4: #(P) ( N) k (12/δ) k e c 2n Using the concentration inequality, we see that Φ = Φ(ω) satisfies (1 δ/2) q l N 2 Φ(q) l n 2 (1 + δ/2) q l N 2, q P with probability 1 Ce (c 2 c(δ/2))n Extend this to all x Σ k by a boot strapping estimate: Let M be the norm of Φ on Σ k. Given x with x l N 2 = 1 find q such that x q l N 2 δ/4 Φ(x) l n 2 Φ(x q) l n 2 + Φ(q) l n 2 Mδ/ δ/2 Hence M Mδ/ δ/2 and so M 1 + δ

85 Johnson-Lindenstrauss Lemma Matrices Φ satisfying RIP are closely related to the following Johnson-Lindenstrauss Lemma

86 Johnson-Lindenstrauss Lemma Matrices Φ satisfying RIP are closely related to the following Johnson-Lindenstrauss Lemma Given a set of points Q in IR N, then for any n cǫ 2 log[#(q)], there is a linear mapping Φ from IR N into IR n such that for all x,y Q (1 ǫ)dist(x,y) dist(φ(x), Φ(y)) (1 + ǫ)dist(x,y)

87 Johnson-Lindenstrauss Lemma Matrices Φ satisfying RIP are closely related to the following Johnson-Lindenstrauss Lemma Given a set of points Q in IR N, then for any n cǫ 2 log[#(q)], there is a linear mapping Φ from IR N into IR n such that for all x,y Q (1 ǫ)dist(x,y) dist(φ(x), Φ(y)) (1 + ǫ)dist(x,y) This lemma is easily proved from the Concentration of Measure Inequalities

88 Johnson-Lindenstrauss Lemma Matrices Φ satisfying RIP are closely related to the following Johnson-Lindenstrauss Lemma Given a set of points Q in IR N, then for any n cǫ 2 log[#(q)], there is a linear mapping Φ from IR N into IR n such that for all x,y Q (1 ǫ)dist(x,y) dist(φ(x), Φ(y)) (1 + ǫ)dist(x,y) This lemma is easily proved from the Concentration of Measure Inequalities A Random draw of a matrix satisfying CMI will satisfy JL

89 Johnson-Lindenstrauss Lemma Matrices Φ satisfying RIP are closely related to the following Johnson-Lindenstrauss Lemma Given a set of points Q in IR N, then for any n cǫ 2 log[#(q)], there is a linear mapping Φ from IR N into IR n such that for all x,y Q (1 ǫ)dist(x,y) dist(φ(x), Φ(y)) (1 + ǫ)dist(x,y) This lemma is easily proved from the Concentration of Measure Inequalities A Random draw of a matrix satisfying CMI will satisfy JL It is also easy to prove RIP from JL

90 Road Map to Kashin-Gluskin Step 1: Show Random families satisfying CMI satisfy RIP

91 Road Map to Kashin-Gluskin Step 1: Show Random families satisfying CMI satisfy RIP We have shown an easy proof of this

92 Road Map to Kashin-Gluskin Step 1: Show Random families satisfying CMI satisfy RIP We have shown an easy proof of this Step 2: Show that RIP for Φ of order k implies x Φ(x) l1 x l1 k 1/2

93 Road Map to Kashin-Gluskin Step 1: Show Random families satisfying CMI satisfy RIP We have shown an easy proof of this Step 2: Show that RIP for Φ of order k implies x Φ(x) l1 x l1 k 1/2 This is also easy and will be shown in the next lecture

94 Road Map to Kashin-Gluskin Step 1: Show Random families satisfying CMI satisfy RIP We have shown an easy proof of this Step 2: Show that RIP for Φ of order k implies x Φ(x) l1 x l1 k 1/2 This is also easy and will be shown in the next lecture Taking k = cn/ log(n/n) these two results give d n (U(l 1 )) l1 sup x l1 x Φ(x) l1 Ck 1 C log(n/n) n

95 Road Map to Kashin-Gluskin Step 1: Show Random families satisfying CMI satisfy RIP We have shown an easy proof of this Step 2: Show that RIP for Φ of order k implies x Φ(x) l1 x l1 k 1/2 This is also easy and will be shown in the next lecture Taking k = cn/ log(n/n) these two results give d n (U(l 1 )) l1 sup x l1 x Φ(x) l1 Ck 1 C log(n/n) n This is the Kashin-Gluskin result in sharp form

96 Implementable Φ Say we want to build a sensor using Compressed Sensing ideas

97 Implementable Φ Say we want to build a sensor using Compressed Sensing ideas It is a very very significant problem to understand what we can build and retain the performance promised by the CS theory

98 Implementable Φ Say we want to build a sensor using Compressed Sensing ideas It is a very very significant problem to understand what we can build and retain the performance promised by the CS theory We cannot generate a random matrix

99 Implementable Φ Say we want to build a sensor using Compressed Sensing ideas It is a very very significant problem to understand what we can build and retain the performance promised by the CS theory We cannot generate a random matrix We could try to replace random by pseudo-random

100 Implementable Φ Say we want to build a sensor using Compressed Sensing ideas It is a very very significant problem to understand what we can build and retain the performance promised by the CS theory We cannot generate a random matrix We could try to replace random by pseudo-random We can also try to use parity check codes or other constructions of 0, 1 matrices

101 Implementable Φ Say we want to build a sensor using Compressed Sensing ideas It is a very very significant problem to understand what we can build and retain the performance promised by the CS theory We cannot generate a random matrix We could try to replace random by pseudo-random We can also try to use parity check codes or other constructions of 0, 1 matrices How close can we get to best performance with implementable algorithms?

102 Implementable Φ Say we want to build a sensor using Compressed Sensing ideas It is a very very significant problem to understand what we can build and retain the performance promised by the CS theory We cannot generate a random matrix We could try to replace random by pseudo-random We can also try to use parity check codes or other constructions of 0, 1 matrices How close can we get to best performance with implementable algorithms?

103 An example There is an n N matrix Φ with entries 0, 1 such that each row is a circulant shift of the previous row by N/n and Φ is instance-optimal for k c 0 n/ log(n/n)

104 An example There is an n N matrix Φ with entries 0, 1 such that each row is a circulant shift of the previous row by N/n and Φ is instance-optimal for k c 0 n/ log(n/n) Construction uses polynomials over finite fields (DeVore)

105 An example There is an n N matrix Φ with entries 0, 1 such that each row is a circulant shift of the previous row by N/n and Φ is instance-optimal for k c 0 n/ log(n/n) Construction uses polynomials over finite fields (DeVore) This is the best known range of k for non-probabilistic constructions

106 An example There is an n N matrix Φ with entries 0, 1 such that each row is a circulant shift of the previous row by N/n and Φ is instance-optimal for k c 0 n/ log(n/n) Construction uses polynomials over finite fields (DeVore) This is the best known range of k for non-probabilistic constructions

107 An example There is an n N matrix Φ with entries 0, 1 such that each row is a circulant shift of the previous row by N/n and Φ is instance-optimal for k c 0 n/ log(n/n) Construction uses polynomials over finite fields (DeVore) This is the best known range of k for non-probabilistic constructions