Structured Sparsity. group testing & compressed sensing a norm that induces structured sparsity. Mittagsseminar 2011 / 11 / 10 Martin Jaggi

Transcription

1 Structured Sparsity group testing & compressed sensing a norm that induces structured sparsity (ariv.org/abs/ Obozinski, G., Jacob, L., & Vert, J.-P., October 2011) Mittagssear 2011 / 11 / 10 Martin Jaggi

2 Sparse Solutions to underdetered Linear Systems b = A b 2 R m 2 R d m d kk 0 apple k

3 Group Testing b 2 R m 2 R d kk 0 apple k m d Dorfman, R. (1943). The Detection of Defective Members of Large Populations. Annals of Mathematical Statistics b A Kainkaryam et al. (2010). poolmc: Smart pooling of mrna samples in microarray eperiments. BMC Bioinformatics Coding theory interpretation: A is the parity check matri of a linear code

4 Group Testing b 2 R m 2 R d kk 0 apple k m d b A

5 Compressed sensing b 2 R m 2 R d kk 0 apple k m d b A Theorem m = O(k log d) is enough. k-sparse signals are recovered by taking the solution of smallest `1 -norm Donoho, D. L. (2006). Compressed sensing. IEEE Transactions on Information Theory (3400 citations) Candes, E. J., & Tao, T. (2005). Decoding by Linear Programg. IEEE Transactions on Information Theory (1200 citations) hard `0 ka bk 2 + kk 0 easy `1 ka bk 2 + kk 1 Santosa and Symes (1983)

6 b 2 Rm 2 Rd Phase transition kk0 k hard `0 ka m d easy `1 bk2 + kk0 bk2 + kk1 ka CHAPTER 3. PERFORMANCE ANALYSIS: EMPIRICAL PHASE DIAGRAMS32 Stepwise with FDR threshold, z~n(0,16), Normalized L2 error, p=200 combinatorial search relative error #non-zeros in 0.6 ρ=k/n k m `1 solves ` δ=n/p m d #rows of A = #measurements Figure 3.8: Empirical Phase diagram for Forward Stepwise-FDR Thresholding: each 2 color indicates a different median normalized!2 error of the coefficients β β over β Stodden, V.2 (2006). Model Selection When The Number Of Variables 10 realizations. A term is added to the model if it has the largest t-statistic of all Eceeds The Number Of Observations. PhD thesis. stanford.edu. candidate terms and its corresponding p-value is less than the FDR value, defined as (.25 (number of terms currently in the model)/(total number of variables)). The number of variables is fied at 200, and model noise z N(0, 16). This version of

7 Why does easy `1 ka bk 2 + kk 1 have sparse solutions? A = b kk 1 apple t

8 How to solve easy `1 ka bk 2 + kk 1 in practice? Linear Program kk 1 A b =0 f() Frank-Wolfe (Sparse Greedy) ka bk 2 kk 1 apple t D R n D =

9 Single Piel Camera b i ka bk 2 + kk 1 A i Duarte et al. Single-Piel Imaging via Compressive Sampling. IEEE Signal Processing Magazine

10 Computer Vision Background Subtraction kb k + kk 1 B B Candes, E. J. et al. (2011). Robust principal component analysis Journal of the ACM

11 A Structured Norm Obozinski, G., Jacob, L., & Vert, J.-P. (October 2011). Group Lasso with Overlaps: the Latent Group Lasso approach. ariv stat.ml. G is a collection of subsets g [d] [ g =[d] kk 1 := (v i ) v i i2[d] = i2[d] v i kk G := (v g ) kv g k g v g = supp(v i )={i} supp(v g ) g G = {{1}, {2}, {3}}

12 A Structured Norm Obozinski, G., Jacob, L., & Vert, J.-P. (October 2011). Group Lasso with Overlaps: the Latent Group Lasso approach. ariv stat.ml. G is a collection of subsets g [d] [ g =[d] kk 1 := (v i ) v i i2[d] = i2[d] v i kk G := (v g ) kv g k g v g = supp(v i )={i} supp(v g ) g G = {{1}, {2}, {3}} k.k G Lemma: the unit ball of is the conve hull of the union of disks D g = n v 2 R d supp(v g ) g, kv g k g apple1 o G = {{1, 2}, {2, 3}}

13 A Structured Norm kkg := g (v ) Obozinski, G., Jacob, L., & Vert, J.-P. (October 2011). Group Lasso with Overlaps: the Latent Group Lasso approach. ariv stat.ml. G = {{1}, {2}, {3}} G = {{1, 2}, {3}} = kv g kg vg G = {{1, 2}, {2, 3}} Lemma: the unit ball of k.kg is the conve hull of a union of disks G = {{1, 2}, {1, 3}, {2, 3}} Figure 2: Unit balls for k k`1 /`2 (left), proposed by Jenatton et al. (2009), and G[ (m proposed in this paper, for the groups G = {{1, 2}, {2, 3}}. w2 is represe the vertical coordinate. We note that singularities eist in both cases, bu at di erent positions: for k k`1 /`2 they correspond to situations where o or only w2 is nonzero, i.e., where all covariates of one group are shrunk t w G[, they correspond to situations where only w1 or only w3 2is equal to where all covariates of one group are nonzero. For comparison, we show right the unit ball of both norms for the partition G = {{1, 2}, {3}}, whe {1,2} both reduce to the classical group Lasso penalty.

14 Optimizing with the Structured Norm kk G := (v g ) kv g k g v g = f()+ kk G Lemma: the unit ball of k.k G is the conve hull of a union of disks Frank-Wolfe (Sparse Greedy) f() kk G apple t f() D R n

15 Relation to Set-Cover G is a collection of subsets g [d] [ g =[d] kk 1 := (v i ) v i i2[d] v i = i2[d] kk G := (v g ) kv g k g v g = supp(v i )={i} supp(v g ) g kk 0 := (v i ) 1 vi 6=0 i2[d] v i = i2[d] kk G set cover := (v g ) 1 v g 6=0 v g = supp(v i )={i} supp(v g ) g

16 Open Questions kk G := (v g ) kv g k g v g = More applications (related to set-cover?) Phase transition phenomenon when applied to the combinatorial set-cover? Is it the closest conve function to set-cover? G = {{1, 2}, {2, 3}}