Topics in MMSE Estimation for Sparse Approximation

Transcription

1 for Spare Approximatio * Michael Elad The Computer Sciece Departmet The Techio Irael Ititute of techology Haifa 3, Irael Workhop: Sparity ad Computatio Haudorff ceter of Mathematic Uiverity of Bo Jue 7-, Joit work with Irad Yaveh Mata Protter Javier Turek The CS Departmet, The Techio

2 Part I - Motivatio Deoiig By Averagig Several Spare Repreetatio

3 Spare Repreetatio Deoiig Spare repreetatio modelig: K x Aume that we get a oiy meauremet vector N y = x + v = D+ v where i AWGN Our goal recovery of x (or ). A fixed Dictioary D The commo practice Approximate the olutio of mi.t. D y ε 3

4 Orthogoal Matchig Puruit OMP fid oe atom at a time for approximatig the olutio of mi.t. D y ε Mai Iteratio =, r Iitializatio ad = y S = D = { } = y = + i z.t. i K, E(i ). Compute E(i) = mi z d r for i. Chooe i E(i) 3. Update S : S = S {i } 4. LS : = mi y.t. up p = S { } K 5. Update D Re idual: r = y D No r ε Ye Stop 4

5 Uig everal Repreetatio Coider the deoiig problem mi.t. D ad uppoe that we ca fid a group of J cadidate olutio j { } J j j= uch that D j j y << y N ε ε Baic Quetio: What could we do with uch a et of competig olutio i order to better deoie y? Why hould thi help? How hall we practically fid uch a et of olutio? Relevat work: [Leug & Barro ( 6)] [Laro & Sele ( 7)] [Schitter et. al. (`8)] [Elad ad Yaveh ( 8)] [Giraud ( 8)] [Protter et. al. ( )] 5

6 Geeratig May Repreetatio Our * Awer: Radomizig the OMP Mai Iteratio =, r Iitializatio ad = y S = D = { } = y = + i z.t. i K, E(i ). Compute E(i) = mi z d r for i. Chooe i E(i) 3. Update S : S = S {i } 4. LS : = mi y.t. up p = S { } K 5. Update D Re idual: r = y D No r ε * Laro ad Schitter propoe a more complicated Ye ad Stop determiitic tree pruig method 6

7 Geeratig May Repreetatio Our * Awer: Radomizig the OMP Iitializatio =, = r = y D = y ad S = { } = + Mai Iteratio z i. Compute E(i) = mi z d r for i K. Chooe i with.t. probability i K, E(i ) exp E(i) c E(i) 3. Update S : S = S {i } 4. LS : = mi D y.t. up p = 5. Update Re idual : r = y D { } For ow, let et the parameter c maually for bet performace. { } S Later we hall defie a way to et it automatically No r ε Ye Stop 7

8 Let Try Propoed Experimet : Form a radom D. Multiply by a pare vector ( = ). Add Gauia iid oie (σ=) ad obtai y. Solve the problem mi.t. D y OMP uig OMP, ad obtai. { } j Ue RadOMP ad obtai RadOMP. Let look at the obtaied repreetatio j= D + = v y 8

9 Some Obervatio Hitogram Hitogram 5 5 Radom-OMP cardialitie OMP cardiality 3 4 Cadiality ˆ Radom-OMP deoiig OMP deoiig D ˆ D Noie Atteuatio y D Hitogram Noie Atteuatio Radom-OMP error OMP error Repreetatio Error Dˆ y Radom-OMP deoiig OMP deoiig Cardiality We ee that The OMP give the paret olutio Neverthele, it i ot the mot effective for deoiig. The cardiality of a repreetatio doe ot reveal it efficiecy. 9

10 The Surprie (to ome of u) Let propoe the average RadOMP = ˆ j = j a our repreetatio 3 Averaged Rep. Origial Rep. OMP Rep. value Thi repreetatio IS NOT SPARSE AT ALL but it oie atteuatio i:.6 (OMP give.6) idex

11 Repeat thi Experimet Dictioary (radom) of ize N=, K= True upport of i σ x = ad ε= We ru OMP for deoiig. We ru RadOMP J= time ad average J RadOMP ˆ = j J j= Deoiig i aeed by Dˆ D y D RadOMP Deoiig Factor OMP veru RadOMP reult Mea Poit Cae of zero olutio, where y OMP Deoiig Factor

12 Part II - Explaatio It i Time to be More Precie

13 Our Sigal Model K x D i fixed ad kow. Aume that i built by: N A fixed Dictioary D Chooig the upport with probability P() from all the K poibilitie Ω. Let aume that P(i S)=P i are draw idepedetly. Chooig the coefficiet uig iid Gauia etrie N, σ x. ( ) The ideal igal i x=d=d. The p.d.f. P() ad P(x) are clear ad kow 3

14 Addig Noie K x Noie Aumed: N A fixed Dictioary D v + y The oie v i additive white Gauia vector with probability P v (v) P ( y x) = C exp x σ y The coditioal p.d.f. P(y ), P( y), ad eve P(y ), P( y), are all clear ad well-defied (although they may appear aty). 4

15 The Key The Poterior P( y) We have acce to P ( y) MAP MAP* ˆ = ArgMax P( y) Oracle kow upport oracle ˆ MMSE MMSE { } ˆ = E y * Actually, there i a delicate problem with thi defiitio, due to the uavoidable mixture of cotiuou ad dicrete PDF. The olutio i to etimate the MAP upport S. 5

16 6 Let Start with The Oracle ( ) ( ) ( ) ( ) ( ) y P P y P y P y, P = = ( ) σ x exp P ( ) σ y exp y P D ( ) σ σ x y exp y P D T x T h y ˆ = σ σ + σ = Q D I D D * Whe i kow * Commet: Thi etimate i both the MAP ad MMSE. The oracle etimate of x i obtaied by multiplicatio by D.

17 The MAP Etimatio MAP P(y )P() ŝ = ArgMax P( y) = ArgMax Ω Ω P(y) P(y ) = P(y, )P( )d =... T hq h log(det( Q)) σx exp Baed o our prior for geeratig the upport ( ) = i ( j) P P P i j ŝ h h log(det( Q)) = ArgMax exp Ω T MAP Q i Pi σ x ( P ) j j 7

18 The MAP Etimatio ŝ MAP T hq h log(det( Q)) = ArgMax Ω P i + log log( P ) + j i σx j Implicatio: The MAP etimator require to tet all the poible upport for the maximizatio. For the foud upport, the oracle formula i ued. I typical problem, thi proce i impoible a there i a combiatorial et of poibilitie. Thi i why we rarely ue exact MAP, ad we typically replace it with approximatio algorithm (e.g., OMP). 8

19 The MMSE Etimatio ˆ MMSE P( y) P() P(y ) =... = E { y} = P( y) E{ y,} T h Q h log( det( Q)) Pi exp i σ Ω Thi i the oracle for, a we have ee before E { } y, = ˆ = Q h x ( P ) j j ˆ MMSE = Ω P( y) ˆ 9

20 The MMSE Etimatio ˆ MMSE Implicatio: = E { y} = P( y) E{ y,} Ω ˆ MMSE = Ω P( y) ˆ The bet etimator (i term of L error) i a weighted average of may pare repreetatio!!! A i the MAP cae, i typical problem oe caot compute thi expreio, a the ummatio i over a combiatorial et of poibilitie. We hould propoe approximatio here a well.

21 The Cae of = ad P i =P P( y) Thi i our c i the Radom-OMP T hq h log(det( Q)) exp σx exp (y d ) σ σ x +σ T i Baed o thi we ca propoe a greedy algorithm for both MAP ad MMSE: i Pi σ x ( Pj ) j The i-th atom i D MAP chooe the atom with the larget ier product (out of K), ad do o oe at a time, while freezig the previou oe (almot OMP). MMSE draw at radom a atom i a greedy algorithm, baed o the above probability et, gettig cloe to P( y) i the overall draw (almot RadOMP).

22 Comparative Reult The followig reult correpod to a mall dictioary ( 6), where the combiatorial formula ca be evaluated a well. Parameter: N,K: 6 P=. (varyig cardiality) σ x = J=5 (RadOMP) Averaged over experimet Relative Repreetatio Mea-Squared-Error Oracle MMSE MAP OMP Rad-OMP σ

23 Part III Divig I A Cloer Look At the Uitary Cae DD T = D T D = I 3

24 Few Baic Obervatio Let u deote β = D T y σ +σ Q DD I I T x = + = σ σx σσx T h = D y = β σ σ σ ˆ = = β = β oracle x Q h c σ +σx (The Oracle) 4

25 Back to the MAP Etimatio T h Q h log(det( Q )) P i σ ( ) exp i ( P ) j P S y x j T Q = β h h c σ log(det( Q)) = S log c ( ) σ x ( ) i exp β i = qi P S y i c σ P c P i i 5

26 The MAP Etimator q i = exp β c Pi i σ Pi c Ŝ MAP i obtaied by maximizig the expreio ( ) qi P S y i Thu, every i uch that q i > hould be i the upport, which lead to σ P i MAP c βi β i > log c i i ˆ = P c Otherwie i MAP P=. σ=.3 σ x = β 6

27 The MMSE Etimatio q i = exp β c Pi i σ Pi c Some algebra ad we get that 3 P=. σ=.3 σ x = q ˆ = β + MMSE i c i qi i i MMSE β g i Thi reult lead to a dee repreetatio vector. The curve i a moothed verio of the MAP oe. 7

28 What About the Error? { } ˆ { Q } oracle i= E = trace =... = c σg { } ( ) { Q } MMSE MMSE oracle Ω E ˆ = P y trace + ˆ ˆ E ( ) 4 i i i i i= i= =... = c σ g + c β g g { } ˆ = ˆ ˆ + E{ ˆ } MAP MAP MMSE MMSE i ( ) 4 MAP i i i i i i= i= =... = c σ g + c β g + I ( g ) g i qi = + q i 8

29 A Sythetic Experimet The followig reult correpod to a dictioary of ize ( ).5. Relative Mea-Squared-Error Empirical Oracle Theoretical Oracle Empirical MMSE Theoretical MMSE Empirical MAP Theoretical MAP Parameter:,K:.5 P=. σ x = Averaged over experimet. The average error are how relative to σ σ 9

30 Part IV - Theory Etimatio Error 3

31 Ueful Lemma Let (a k,b k ) k=,,, be pair of poitive real umber. Let m be the idex of a pair uch that a b k m k. k= m The. k= a b k k a b k m a b Equality i obtaied oly if all the ratio a k /b k are equal. m We are itereted i thi reult becaue : { } ˆ oracle E = c σg i= { } ˆ = σ + β ( ) MMSE 4 i i i i i= i= E c g c g g i { } ˆ = σ + β ( + ) MAP 4 MAP i i i i i i= i= E c g c g I ( g ) Thi lead to 3

32 Theorem MMSE Error P c k Defie Gk =. Chooe m uch that k, Gm Gk. Pk Pk = P = << K { } e MMSE E ˆ + log Gm 4Gm 4 { } oracle E ˆ + Otherwie Gme E E thi error ratio boud become { ˆ } { ˆ } MMSE oracle Cot logk 3

33 Theorem MAP Error P c k Defie Gk =. Chooe m uch that k, Gm Gk. Pk Pk = P = << K { } MAP E ˆ + log Gm e Gm { } oracle E ˆ + Otherwie Ge m E E thi error ratio boud become { ˆ } { ˆ } MMSE oracle Cot logk 33

34 The Boud Factor v. P Parameter: P=[,] σ x = σ=.3 Notice that the tedecy of the two etimator to alig for P i ot reflected i thee boud. Boud Factor e + log Gm 4Gm 4 + Otherwie Gme + log Gm e Gm + Otherwie Ge m P 34

35 Part V We Are Doe Summary ad Cocluio 35

36 Today We Have See that Sparity ad Redudacy are ued for deoiig of igal/image How? By fidig the paret repreetatio ad uig it to recover the clea igal MAP ad MMSE ejoy a cloed-form, exact ad cheap formulae. Their error i bouded ad tightly related to the oracle error Uitary cae? Ye! Averagig everal rep lead to better deoiig, a it approximate the MMSE Ca we do better? More o thee (icludig the lide ad the relevat paper) ca be foud i 36