Quantized Iterative Hard Thresholding:

Transcription

1 Quantized Iterative Hard Thresholding: ridging 1-bit and High Resolution Quantized Compressed Sensing Laurent Jacques, Kévin Degraux, Christophe De Vleeschouwer Louvain University (UCL), Louvain-la-Neuve, elgium SampTA2013, Jacobs University remen

2 Compressive Sampling and Quantization Compressed sensing theory says: Linearly sample a signal at a rate function of its intrinsic dimensionality R M? Φ Information theory and sensor designer say: Okay, but I need to quantize/digitize my measurements! (e.g., in ADC) SampTA2013, Jacobs University remen

3 (this talk) Focus on scalar quantization Turning measurements into bits scalar quantization Q Out q i = Q[(Φx) i ]=Q[φ i, x] Ω R q i In q = Q Φx Ω = Ω M, τ i τ i+1 e.g., Lloyd-Max Ω = {q i R :1 i 2 }, (levels) T = {τ i R :1 i 2 +1, τ i τ i+1 } (thresholds) Important conventions: Definition of Φ independent of M bits per measurement Total bit budget: R = M No further encoding (e.g., entropic) (e.g., Φ ij iid N (0, 1)) preserves measurement dynamic! 5 SampTA2013, Jacobs University remen

4 1. First extreme: CS and high-resolution scalar quantization SampTA2013, Jacobs University remen 6

5 Former solutions Quantization is like a noise q = Q Φx = Φx + n quantization distortion SampTA2013, Jacobs University remen 7

6 Former solutions Quantization is like a noise q = Q Φx = Φx + n CS is robust, e.g., with... asis Pursuit DeNoise (PDN): ˆx = argmin u R N u 1 s.t. Φu q Iterative Hard Thresholding (IHT): ˆx = argmin u R N Φu y 2 s.t. u 0 K thresholding gradient descent iterating: x (n+1) = H K x (n) + µφ (y Φx (n) ) SampTA2013, Jacobs University remen 8

7 Former solutions Quantization is like a noise q = Q Φx = Φx + n CS is robust (e.g., with PDN or IHT) If n and 1 M Φ is RIP(δ,aK)withδ δ 0,then ˆx x C M + De(K), for some C, D > 0 and e(k) = deviation to K-sparsity. e(k) =x x K 1 / K or x x K 2 + x x K 1 / K [Candès, 08] (a = 2, δ 0 < 2 1) [lumensath, Davies, 09] (a =3, δ 0 < 1/32) SampTA2013, Jacobs University remen 9

8 Former solutions Quantization is like a noise q = Q Φx = Φx + n CS is robust (e.g., with PDN or IHT) If n and 1 M Φ is RIP(δ,aK)withδ δ 0,then ˆx x C M + De(K), for some C, D > 0 and e(k) = deviation to K-sparsity. Question: finding a for QCS? e.g., High-Resolution ounds! M = O(α) (unif.q bin width α) or O(2 ) for -bit Q SampTA2013, Jacobs University remen 10

9 2. Second extreme: CS and 1-bit scalar quantization SampTA2013, Jacobs University remen 11

10 1-bit Compressed Sensing q Φ x Oversampling in M = sign [oufounos, M M N N araniuk, 08] M-bits! ut, which information inside q? SampTA2013, Jacobs University remen 12

11 1-bit Compressed Sensing Computational bits matter! q Φ x Oversampling in M = sign [oufounos, M M N N araniuk, 08] M-bits! ut, which information inside q? SampTA2013, Jacobs University remen 13

12 Oversampling in M 1-bit Compressed Sensing Computational bits matter! q Φ x = sign M M N N Warning 1: signal amplitude is lost! Amplitude is arbitrarily fixed Examples : x = 1 or Φx 1 =1 Warning 2:!forbidden sensing! (e.g. ernoulli + canonical sparsity) [Plan, Vershynin, 11] SampTA2013, Jacobs University remen 14

13 Why does it work? x on S 2 M vectors: {ϕi : 1 i M } iid Gaussian ϕ5 ϕ1 1-bit Measurements ϕ1, x > 0 ϕ2, x > 0 ϕ3, x 0 ϕ4, x > 0 ϕ5, x > 0... ϕ2 x Smaller and smaller when M increases ΣK {u : sign (Φu) = sign (Φx)} ϕ4 ϕ3 Lower bound on this width? SampTA2013, Jacobs University remen 15

14 1-bit CS bounds Let A( ) := sign (Φ ) with Φ N M N (0, 1). If M = O( 1 K log N ), then, w.h.p, for any two unit K-sparse vectors x and s, if only b bits are different between A(x) and A(s) x s K+b K [LJ, Laska, oufounos, araniuk, 13] (b = 0) [LJ, Degraux, De Vleeschouwer, 13] (b = 0) If M = O( 2 K log N) inary Stable Embedding (SE): d ang (x, s) d H (A(x),A(s)) d ang (x, s)+ =RIPlike [LJ, Laska, oufounos, araniuk, 13] [Plan, Vershynin, 11, 12] Lesson: possible recovery should be as consistent as possible. SampTA2013, Jacobs University remen 17

15 Easiest 1-bit reconstruction Fact: If M = O( 2 K log N/K) (for x Σ K fixed, s Σ K ) π/2 M sign (Φx), Φs x, s (w.h.p) Let x Σ K S N 1 and q = sign(φx). Compute ˆx = π 2M H K(Φ q) [Plan, Vershynin, 12] Then, if previous property holds, x ˆx 2. [LJ, Degraux, De Vleeschouwer, 13] arg max u R N q T Φu s.t. u 0 K, u =1 PV-L0 problem [ahmani, oufounos, Raj, 13] SampTA2013, Jacobs University remen 18

16 inary Iterative Hard Thresholding (IHT) ϕ j ϕ k x J (x )=0 x J (x ) = 0 Idea: greedily forcing consistency forcing sparsity M J (x )= sign (ϕ j, x) ϕ j, x j=1 q j with (λ) =(λ λ )/2 or J (x )=[ diag(q)(φx )] 1 SampTA2013, Jacobs University remen 20

17 inary Iterative Hard Thresholding (IHT) ϕ j ϕ k x J (x )=0 x J (x ) = 0 Idea: greedily forcing consistency forcing sparsity M J (x )= sign (ϕ j, x) ϕ j, x j=1 q j with (λ) =(λ λ )/2 or J (x )=[ diag(q)(φx )] 1 Objective: argmin u J (u) s.t. u 0 K Given q = A(x) and K, setl = 0, x 0 = 0: a l+1 = x l + τ 2 ΦT q A(x l ), ( gradient towards consistency) (τ > 0 controls gradient descent) x l+1 = H K (a l+1 ), l l +1 (proj. K-sparse signal set) SampTA2013, Jacobs University remen 21

18 3. ridging 1-bit & -bit CS? SampTA2013, Jacobs University remen 22

19 ridging 1-bit & -bit CS? -bit quantizer defined with thresholds: λ q i τ i τ i+1 R + + q i = Q[λ] sign (λ τ i ) sign (λ τ i+1 ) λ R i =[τ i, τ i+1 ) sign (λ τ i ) = +1 & sign (λ τ i+1 )= 1 Can we combine multiple thresholds in 1-bit CS? SampTA2013, Jacobs University remen 23

20 ridging 1-bit & -bit CS? Given T = {τ j } and Ω = {q j } ( T =2 +1= Ω + 1), let s define J(ν, λ) = 2 j=2 with w j = q j q j 1. w j sign (λ τj )(ν τ j ), Remark: for symmetric Q, Q(λ) = 2 j=2 w j sign (λ τ j ) SampTA2013, Jacobs University remen 24

21 ridging 1-bit & -bit CS? Given T = {τ j } and Ω = {q j } ( T =2 +1= Ω + 1), let s define J(ν, λ) = 2 j=2 with w j = q j q j 1. w j sign (λ τj )(ν τ j ), Illustration: λ [τ j 1, τ j ), ν [τ j, τ j+1 ) J(ν, λ) = sign (λ τ j )(ν τ j ) =(ν τ j ) delocalized IHT 1 -sided norm λ ν τ j 1 τ j τ j+1 (for w j = 1) SampTA2013, Jacobs University remen 25

22 ridging 1-bit & -bit CS? Given T = {τ j } and Ω = {q j } ( T =2 +1= Ω + 1), let s define J(ν, λ) = 2 j=2 with w j = q j q j 1. w j sign (λ τj )(ν τ j ), Illustration: λ [τ j 1, τ j ), ν [τ j+1, τ j+2 ) J(ν, λ) =(ν τ j )+(ν τ j+1 ) (ν τ j ) (ν τ j+1 ) λ ν τ j 1 τ j τ j+1 (for w j = 1) SampTA2013, Jacobs University remen 26

23 ridging 1-bit & -bit CS? Given T = {τ j } and Ω = {q j } ( T =2 +1= Ω + 1), let s define J(ν, λ) = 2 j=2 with w j = q j q j 1. w j sign (λ τj )(ν τ j ), Illustration: (ν τ j )+(ν τ j+1 ) λ (ν τ j ) τ j 1 τ j τ j+1 (for w j = 1) SampTA2013, Jacobs University remen 27

24 ridging 1-bit & -bit CS? Given T = {τ j } and Ω = {q j } ( T =2 +1= Ω + 1), let s define J(ν, λ) = 2 j=2 with w j = q j q j 1. Illustration: more bins, w j sign (λ τj )(ν τ j ), λ R (λ ν)2 J(λ, ν) q 5 SampTA2013, Jacobs University remen 28

25 ridging 1-bit & -bit CS? Given T = {τ j } and Ω = {q j } ( T =2 +1= Ω + 1), let s define J(ν, λ) = 2 j=2 with w j = q j q j 1. w j sign (λ τj )(ν τ j ), For u, v R M : J (u, v) := M k=1 J(u k,v k ) (component wise) Remarks: J is convex in For =1(j = 2 only): J (u, v) (sign (v) u) 1 1 -sided 1-bit energy For 1: J(ν, λ) 1 2 (ν λ)2 and J (u, v) 1 2u v2 (quadratic energy) SampTA2013, Jacobs University remen 29

26 ridging 1-bit & -bit CS? Let s define an inconsistency energy: E (u) :=J (Φu, q) withq = Q [Φx] and E (x) =0 Idea: Minimize it in Σ K (as for Iterative Hard Thresholding) min u R N E (u) s.t.u 0 K, [lumensath, Davies, 08] SampTA2013, Jacobs University remen 30

27 ridging 1-bit & -bit CS? Let s define an inconsistency energy: E (u) :=J (Φu, q) withq = Q [Φx] and E (x) =0 Idea: Minimize it in Σ K (as for Iterative Hard Thresholding) min u R N E (u) s.t.u 0 K, combinatorial but greedy solution (as for IHT): [lumensath, Davies, 08] x (n+1) = H K [x (n) µ E (x (n) )] and x (0) = 0. (sub) gradient (µ >0, see after) Φ (sign (Φu) sign (Φx)) IHT! E (u) =Φ (Q (Φu) q) =1 1 Quantized IHT (QIHT) all that.. for this!? ;-) Φ (Φu q) IHT! SampTA2013, Jacobs University remen 31

28 ridging 1-bit & -bit CS? So, QIHT reads x (n+1) = H K x (n) + µ Φ q Q (Φx (n) ) and x (0) = 0. Setting μ? for = 1, μ has no impact for high, IHT solution: [lumensath, Davies, 08] If Φ/ M is RIP, µ< 1 (1+δ 2K )M for any, heuristic: µ = 1 M (1 ck/m) for some c>0. Validated by extensive simulations (c = 3) SNR: R = M R = M R = M SampTA2013, Jacobs University remen 33

29 QIHT simulations N = 1024, K= 16, R= M {64, 128,, 1280}, 100 trials Note: entropy could be computed instead of (e.g., for further efficient coding) (+ Lloyd-Max Gauss. Q.) Oracle * * R = M R = M *: almost 6d per bit gain SampTA2013, Jacobs University remen 34

30 QIHT simulations N = 1024, K= 16, R= M {64, 128,, 1280}, 100 trials Note: entropy could be computed instead of (e.g., for further efficient coding) (+ Lloyd-Max Gauss. Q.) Oracle * * R R QIHT stops if x (n+1) x (n) x (n) or n = 500. < 10 4 oosting at low Easiest algo *: almost 6d per bit gain R = M SampTA2013, Jacobs University remen 35

31 QIHT simulations N = 1024, K= 16, R= M {64, 128,, 1280}, 100 trials Note: entropy could be computed instead of (e.g., for further efficient coding) Running time: (+ Lloyd-Max Gauss. Q.) R = M R = M R = M Consistency? R = M R = M R = M SampTA2013, Jacobs University remen 36

32 Conclusions IHT framework can integrate quantization Lot of theoretical works to do/come... IHT/QIHT convergence/optimality guarantees? New embedding results? considering Q(λ) = 2 j=2 w j sign (λ τ j ) = sum of 1-bit costs + thresholds? lend of RIP/εSE? as in 1 δ x x dist(q(φx), Q(Φx )) 1+δ x x +? for sparse x, x and with,m 0 and δ M 0? Coming: Application to -bit compressive sensors (RMPI) SampTA2013, Jacobs University remen 38

33 Further Reading (this work) L. Jacques, K. Degraux, C. De Vleeschouwer, Quantized Iterative Hard Thresholding: ridging 1-bit and High-Resolution Quantized Compressed Sensing", SAMPTA 2013, T. lumensath, M.E. Davies, Iterative thresholding for sparse approximations. Journal of Fourier Analysis and Applications, 14(5-6), pp , 2008 P. T. oufounos and R. G. araniuk, 1-it compressive sensing, Proc. Conf. Inform. Science and Systems (CISS), Princeton, NJ, March 19-21, oufounos, P. T. (2009, November). Greedy sparse signal reconstruction from sign measurements. In Conference Record of the Forty-Third Asilomar Conference on Signals, Systems and Computers, 2009 Y. Plan, R. Vershynin, Dimension reduction by random hyperplane tessellations, arxiv: , Y. Plan, R. Vershynin, Robust 1-bit compressed sensing and sparse logistic regression: a convex programming approach, IEEE Trans. Info. Theory, arxiv: , J. N. Laska, R. G. araniuk, Regime change: it-depth versus measurement-rate in compressive sensing, IEEE Trans. Signal Processing, 60(7), pp , L. Jacques, J. N. Laska, P. T. oufounos, and R. G. araniuk, Robust 1-it Compressive Sensing via inary Stable Embeddings of Sparse Vectors, IEEE Trans. Info. Theory, 59(4), S. ahmani, P.T. oufounos,. Raj, Robust 1-bit Compressive Sensing via Gradient Support Pursuit, SampTA2013, Jacobs University remen 39