Sparse Interactions: Identifying High-Dimensional Multilinear Systems via Compressed Sensing

Transcription

1 Sparse Interactions: Identifying High-Dimensional Multilinear Systems via Compressed Sensing Bobak Nazer and Robert D. Nowak University of Wisconsin, Madison Allerton 10/01/10

2 Motivation: Virus-Host Interaction virus fruit fly from Paul Ahlquist and Yoshihiro Kawaoka s Labs at UW - Madison.

3 Motivation: Virus-Host Interaction virus c Add dsrna of the Drosophila RNAi library (targeting to 13,071 Drosophila genes) to each well of 384-well microplates 0 h Add DL1 cells to the plates fruit fly GFP sensors 48 h Infect with FVG-R virus 72 h Measure Renilla luciferase activity to assess the efficiency of virus replication from Paul Ahlquist and Yoshihiro Kawaoka s Labs at UW - Madison.

4 Motivation: Virus-Host Interaction virus c Add dsrna of the Drosophila RNAi library (targeting to 13,071 Drosophila genes) to each well of 384-well microplates 0 h Add DL1 cells to the plates fruit fly GFP sensors 48 h Infect with FVG-R virus 72 h Measure Renilla luciferase activity to assess the efficiency of virus replication from Paul Ahlquist and Yoshihiro Kawaoka s Labs at UW - Madison. Which genes does the virus hijack for reproduction?

5 Basic Model Genome Hijacked Proteins Discovered Hijacked Genes

6 Basic Model Genome Knockdown Hijacked Proteins Discovered Hijacked Genes

7 Basic Model Genome Knockdown Hijacked Proteins Discovered Hijacked Genes

8 Basic Model Genome Knockdown Hijacked Proteins Discovered Hijacked Genes

9 Basic Model Genome Knockdown Hijacked Proteins Discovered Hijacked Genes

10 Basic Model Genome Knockdown Hijacked Proteins Discovered Hijacked Genes 13,071 genes to check in a fruit fly.

11 Redundant Model Genome Hijacked Proteins Discovered Hijacked Genes

12 Redundant Model Genome Knockdown Hijacked Proteins Discovered Hijacked Genes

13 Redundant Model Genome Knockdown Knockdown Hijacked Proteins Discovered Hijacked Genes

14 Redundant Model Genome Knockdown Knockdown Hijacked Proteins Discovered Hijacked Genes

15 Redundant Model Genome Knockdown Hijacked Proteins Discovered Hijacked Genes

16 Redundant Model Genome Knockdown Knockdown Hijacked Proteins Discovered Hijacked Genes

17 Redundant Model Genome Knockdown Knockdown Hijacked Proteins Discovered Hijacked Genes Millions of pairs of genes to check in a fruit fly.

18 Sparse Linear Systems Genome Knockdown y = i a i x i Can model output (virus expression) as a sparse linear system. x i a i corresponds to whether gene is involved or not. corresponds to knockdowns.

19 Sparse Multilinear Systems Genome Knockdown Knockdown y = i a i x i + i<j a i a j x ij Can model output (virus expression) as a sparse multilinear system. x i a i corresponds to whether gene is involved or not. corresponds to knockdowns.

20 Sparse Multilinear Systems Genome Knockdown y = i a i x i + i<j a i a j x ij Can model output (virus expression) as a sparse multilinear system. x i a i corresponds to whether gene is involved or not. corresponds to knockdowns.

21 This Talk How many measurements are needed to infer the coefficients for a sparse multilinear system? y = i 1 a i1 x i1 + i 1 <i 2 a i1 a i2 x i1 i i 1 <i 2 < <i D a i1 a i2 a id x i1 i 2 i D

22 This Talk How many measurements are needed to infer the coefficients for a sparse multilinear system? y = i 1 a i1 x i1 + i 1 <i 2 a i1 a i2 x i1 i i 1 <i 2 < <i D a i1 a i2 a id x i1 i 2 i D Sufficient to consider order D interactions y = i 1 <i 2 < <i D a i1 a i2 a id x i1 i 2 i D

23 Vectorized Multilinear System y k = i 1 <i 2 < <i D a ki1 a ki2 a kid x i1 i 2 i D ( Take inputs to be binary symmetric for simplicity. )

24 Vectorized Multilinear System y k = i 1 <i 2 < <i D a ki1 a ki2 a kid x i1 i 2 i D ( Take inputs to be binary symmetric for simplicity. ) Write coefficients as a vector {x i1 i 2 i } x D

25 Vectorized Multilinear System y k = i 1 <i 2 < <i D a ki1 a ki2 a kid x i1 i 2 i D ( Take inputs to be binary symmetric for simplicity. ) Write coefficients as a vector {x i1 i 2 i } x D Write measurements as a vector {a ki1 a ki2 a kid } a k

26 Vectorized Multilinear System y k = i 1 <i 2 < <i D a ki1 a ki2 a kid x i1 i 2 i D ( Take inputs to be binary symmetric for simplicity. ) Write coefficients as a vector {x i1 i 2 i } x D Write measurements as a vector {a ki1 a ki2 a kid } a k Back to a linear problem: y = Ax with potentially dependent measurements.

27 Vectorized Multilinear System y k = i 1 <i 2 < <i D a ki1 a ki2 a kid x i1 i 2 i D ( Take inputs to be binary symmetric for simplicity. ) Write coefficients as a vector {x i1 i 2 i } x D Write measurements as a vector {a ki1 a ki2 a kid } a k Back to a linear problem: y = Ax with potentially dependent measurements. Can we get down to S log (N/S) measurements?

28 Compressed Sensing Candes-Romberg-Tao ʼ06, Candes-Tao ʼ06, Donoho ʼ06 Linear measurements: Coefficients are S-sparse: y k = i y = Ax a ki x i = a T k x x 0 = S x R N

29 Compressed Sensing Candes-Romberg-Tao ʼ06, Candes-Tao ʼ06, Donoho ʼ06 Linear measurements: Coefficients are S-sparse: y k = i y = Ax a ki x i = a T k x x 0 = S x R N If the measurement matrix satisfies the restricted isometry property (RIP) with δ 2S < 2 1 for all S-sparse vectors: (1 δ S )x 2 2 Ax 2 2 (1 + δ S )x 2 2

30 Compressed Sensing Candes-Romberg-Tao ʼ06, Candes-Tao ʼ06, Donoho ʼ06 Linear measurements: Coefficients are S-sparse: y k = i y = Ax a ki x i = a T k x x 0 = S x R N If the measurement matrix satisfies the restricted isometry property (RIP) with δ 2S < 2 1 for all S-sparse vectors: (1 δ S )x 2 2 Ax 2 2 (1 + δ S )x 2 2 Then we can recover the coefficients through an 1 optimization: min ˆx 1 subject to Aˆx = y

31 Compressed Sensing Candes-Romberg-Tao ʼ06, Candes-Tao ʼ06, Donoho ʼ06 Linear measurements: Coefficients are S-sparse: y k = i y = Ax a ki x i = a T k x x 0 = S x R N If the measurement matrix satisfies the restricted isometry property (RIP) with δ 2S < 2 1 for all S-sparse vectors: (1 δ S )x 2 2 Ax 2 2 (1 + δ S )x 2 2 Then we can recover the coefficients through an 1 optimization: min ˆx 1 subject to Aˆx = y Can get RIP with K cs log(n/s) measurements.

32 Norm Preservation The restricted isometry property guarantees that the norms of all sparse vectors are approximately preserved. Example: M = 3 inputs, D = 2 interactions a =[a 1 a 2 a 1 a 3 a 2 a 3 ]

33 Norm Preservation The restricted isometry property guarantees that the norms of all sparse vectors are approximately preserved. Example: M = 3 inputs, D = 2 interactions a =[a 1 a 2 a 1 a 3 a 2 a 3 ] Our measurements preserve the norm in expectation: E y 2 = x T E A T A x Each diagonal element is the sum of products of squared terms. Each off-diagonal element is the sum of products, with at least one unique term. E A T A = I E y 2 = x 2

34 Example: Best Case Sparsity Pattern If each index is involved in at most one interaction, then the measurements concentrate very quickly. Example: y = a 1 a 2 x 12 + a 3 a 5 x 35 + a 4 a 7 x 47

35 Example: Best Case Sparsity Pattern If each index is involved in at most one interaction, then the measurements concentrate very quickly. Example: y = a 1 a 2 x 12 + a 3 a 5 x 35 + a 4 a 7 x 47 Relabel y =ã 1 x 1 +ã 2 x 2 +ã 3 x 3

36 Example: Best Case Sparsity Pattern If each index is involved in at most one interaction, then the measurements concentrate very quickly. Example: y = a 1 a 2 x 12 + a 3 a 5 x 35 + a 4 a 7 x 47 Relabel y =ã 1 x 1 +ã 2 x 2 +ã 3 x 3 Best case concentration is subgaussian: inf T P( y2 k 1 >t) exp( ct)

37 Example: Worst Case Sparsity Pattern If each a subset of indices is involved in event interaction, then the measurements concentrate more slowly. Example: y = a i a j P y 2 S 2 S 1 i j S Worst case concentration is subexponential: sup T P( y 2 k 1 >t) exp( ct 1/D )

38 Eigenvalue Experiment Look at minimum eigenvalues of subsets of Gram matrix for order D = 1, 2, 3 interactions. Minimum Eigenvalue D=1 D=2 D= Size Obviously, there is some cost incurred by dependencies within the vector.

39 Gershgorinʼs Disc Theorem Approach inspired by Haupt-Bajwa-Raz-Nowak 08: i) Control each element of the Gram matrix G R = A T RA R via Hoeffding s inequality. Bound probability that Gram matrix is at worst: δ 1 SS δ SS δ SS δ 1 SS G R = δ SS δ SS 1

40 Gershgorinʼs Disc Theorem Approach inspired by Haupt-Bajwa-Raz-Nowak 08: i) Control each element of the Gram matrix G R = A T RA R via Hoeffding s inequality. Bound probability that Gram matrix is at worst: δ 1 SS δ SS δ SS δ 1 SS G R = δ SS δ SS 1 ii) Gershgorin s Disc Theorem guarantees that eigenvalues lie in the range: g ii j=i g ij λ i (G R ) g ii + j=i g ij

41 Gershgorinʼs Disc Theorem Approach inspired by Haupt-Bajwa-Raz-Nowak 08: i) Control each element of the Gram matrix G R = A T RA R via Hoeffding s inequality. Bound probability that Gram matrix is at worst: δ 1 SS δ SS δ SS δ 1 SS G R = δ SS δ SS 1 ii) Gershgorin s Disc Theorem guarantees that eigenvalues lie in the range: g ii j=i g ij λ i (G R ) g ii + j=i g ij N iii) Union bound over all S sparse patterns. Get RIP with constant long as number of measurements is at least: δ S K = cs 2 log N

42 Heavy-Tailed Restricted Isometries Rudelson-Vershynin 08, Vershynin 10: Assume the expected Gram matrix is identity,, and the rows of the measurement matrix are independent, then if the number of measurements is at least E A T A = I K = c S 2 log S 2 3 log N with expected RIP constant E[δ S ]

43 Heavy-Tailed Restricted Isometries Rudelson-Vershynin 08, Vershynin 10: Assume the expected Gram matrix is identity,, and the rows of the measurement matrix are independent, then if the number of measurements is at least E A T A = I K = c S 2 log S 2 3 log N with expected RIP constant E[δ S ] One of the key points in the proof is that it avoids using a union bound and insteads bounds the RIP constants of all sparsity patterns simultaneously.

44 Heavy-Tailed Restricted Isometries Rudelson-Vershynin 08, Vershynin 10: Assume the expected Gram matrix is identity,, and the rows of the measurement matrix are independent, then if the number of measurements is at least E A T A = I K = c S 2 log S 2 3 log N with expected RIP constant E[δ S ] One of the key points in the proof is that it avoids using a union bound and insteads bounds the RIP constants of all sparsity patterns simultaneously. This corresponds exactly to our linearized problem.

45 Rademacher Chaos of order D: How exactly do the tails behave? y = i 1 <i 2 < <i D a i1 a i2 a id x i1 i 2 i D a i are independent and binary symmetric (Rademacher).

46 Rademacher Chaos of order D: How exactly do the tails behave? y = i 1 <i 2 < <i D a i1 a i2 a id x i1 i 2 i D a i are independent and binary symmetric (Rademacher). The combinatorial dimension measures the level of dependence introduced by the sparsity pattern. A chaos has combinatorial dimension if T L T L (A 1 A 2... A D ) sup A 1,A 2,...,A (max D 1 j D A j ) α C 1 α T L C 2 L α Takes values between 1 α D

47 Chaos Tail Bounds Blei-Janson 04: A Rademacher chaos with combinatorial dimension satisfies: exp c 1 t 2/α sup L P ( u >t) exp c 2 t 2/α α

48 Chaos Tail Bounds Blei-Janson 04: A Rademacher chaos with combinatorial dimension satisfies: exp c 1 t 2/α sup L P ( u >t) exp c 2 t 2/α α Using this bound it can be shown that: P y 2 1 exp c max Kt/S, K 1/α t 1/α from which we can derive the number of measurements needed to get RIP for a single pattern.

49 Chaos Tail Bounds Blei-Janson 04: A Rademacher chaos with combinatorial dimension satisfies: exp c 1 t 2/α sup L P ( u >t) exp c 2 t 2/α α Using this bound it can be shown that: P y 2 1 exp c max Kt/S, K 1/α t 1/α from which we can derive the number of measurements needed to get RIP for a single pattern. Union bound technique from Baraniuk-Devore-Davenport-Wakin 08: K min cs 2 log(n/s), c S α log α (N/S)

50 Summary of Bounds Gershgorin S 2 log N Second Moment Union Bound Rudelson-Vershynin S(log 3 S)(log N) Second Moment No Union Bound Rademacher Chaos S α log α (N/S) Tail Bounds Union Bound

51 Conclusions and Future Directions Can recover sparse vectors from multilinear systems despite heavytailed behavior and dependencies. Number of measurements may depend on the pattern of sparsity. What is possible for polynomial systems?

52 Acknowledgments Thanks to Paul Ahlquist for introducing us to virus replication. Thanks to Ben Recht for introducing us to the work of Rudelson and Vershynin.