Lecture 21: Signal Subspaces and Sparsity

Transcription

1 ECE 830 Fall 00 Statistical Sigal Prcessig istructr: R. Nwak Lecture : Sigal Subspaces ad Sparsity Sigal Subspaces ad Sparsity Recall the classical liear sigal mdel: X = H + w, w N(0, where S = H, is a liear-parametric mdel fr the sigal ad w is ise. Here H is a kw k matrix, whse clums spa the sigal subspace, ad R k are the sigal parameters. The MLE f is: ad the MLE f the sigal is: b MLE =(H T H) H T X wi) bs = H b = H(H T H) H T X P H where P H = H(H H H) H T is the rthgal prjecti peratr t the sigal subspace. The Bayes MMSE estimatr based a prir N (0, I) is the Wieer filter (psterir mea ad MAP estimatr): b Wieer = H T HH T + w I X = + w bmle This fllws directly frm the Gauss-Markv Therem. Ad as the SNR grws H T HH T + w I! (H T H) H T as S i the high SNR situati, the Wieer filter acts essetially the same as the MLE; it prjects X t the sigal subspace. At lw SNR the Wieer filter shriks the MLE tward zer t balace the trade betwee bias ad variace. Sparsity I the classic set-up: X = H + w we assume that we kw the lw-dimesial sigal subspace. I may prblems we may t have this ifrmati, but we might kw that the sigal lies i e f may subspaces i a certai trasfrm dmai. Example Narrwbad Cmmuicatis. The cmmuicati sigal lies i e f may arrw frequecy bads, but we may t kw which bad it will be i (e.g. frequecy hppig cmmuicati). If x is the sigal ad U is the DFT, the = U T x is a sparse vectr (i.e., there are just a few -zer frequecies), but it is t kw which frequecies will have -zer ce ciets. Example Wavelet-based Image Prcessig. The discrete wavelet trasfrm (DWT) is very e ective at cmpressig atural images. I fact it is the basis f the JPEG-000 stadard. The DWT f images teds t be sparse i the fllwig sese. If x is a image ad U detes the DWT, the the DWT ce ciets = U T x ted t be mstly zer (r very early zer). The lcatis f the relatively few -zer (r sigificat) ce ciets i the vectr deped x i a cmplicated way. S, while images d apprximately lie i a subspaces f the wavelet dmai, the subspace is di eret fr each di eret image. w!

2 Lecture : Sigal Subspaces ad Sparsity 3 Sparse Sigal Mdels Let U be a matrix whes clums frm a rthbasis fr R. Fr example, U culd be the DFT r DWT. The sigal f iterest is represeted i this dmai as s = U. Csider the bservati mdel A equivalet bservati mdel is: x = U + w, w N(0, U T x = U T U + u T w = + w 0 wi). where w 0 N (0, w U T U). Sice the clums f U are rthrmal, U T U = I, ad s w 0 N (0, w I). Thus, after trasfrmig the sigal by U T we have a direct bservati f plus GWN. The prblem f estimatig is called deisig. If we make assumpti abut the, the we culd use the MLE: b MLE = U T x. The MLE f the sigal is the bs MLE = U b MLE = UU T x = x, siceuu T = I. If we suppse that the ce ciets ted t have a certai eergy, the we culd use the prir: N (0, I) ad the Wieer filter: b Wieer = U T UU T + w I x Sice U is a rthrmal trasfrm UU T = I ad the wieer filter simplifies t: b Wieer = U T I + x = = + w + w w U T x bmle, ad we see that the Wieer filter is simply shrikig the MLE accrdig t the SNR. Nw suppse ur prir kwledge abut is that it is sparse; i.e. may r mst f the ce ciets are zer (r ear zer). This is t captured by the Gaussia prir, which mdels every ce ciet as a Gaussia radm variable with pwer. If may ce ciets are zer, the may shuld have apprximately zer pwer! S we wuld like t desig a prir prbability desity that reflects ur belief that mst f the ce ciets are zer r ear zer i magitude. Example 3 Gaussia mixture. Let,..., dete the ce ciets ad mdel them as fllws: i iid p N (0, 0 ) + ( p) N (0, ), fr i =,..., with 0 << ad p. I wrds this prir is sayig that a large fracti p f the ce ciets ted t be very small i magitude (i.e. i 0 )ad p ted t be large. A example is depicted i Figure. Example 4 Laplacia prir. Let,..., dete the ce ciets ad mdel them as fllws: i e i, i =,..., We will fcus the Laplacia prir because it leads t very simple ad ituitive slutis t the deisig prblem ad it is lg-ccave, which makes it cmputatiablly tractable whe used i iverse prblems such as decvluti. A example is depicted i Figure.

3 Lecture : Sigal Subspaces ad Sparsity 3 Figure : Example f Gaussia-mixture prir 4 Laplacia prirs fr sparsity Figure : Example f Laplacia prir Assume the prir ad bservati mdel r equivaletly Recall that U T w N (0, Y p() = p( i ) = i= Y e i i= x = U + w, w N ((0, I) U T x = + U T w I). Defiig y = U T x,wehavethemdel y = + w, w N (0, I) The likelihd f give y is Y p(y ) = p e (y i i ) i=

4 Lecture : Sigal Subspaces ad Sparsity 4 The psterir distributi f is Csider the MAP estimatr p( y) / p(y )p() Y = p e (y i i ) e i i= b = arg max p( x) = arg max lg(p( x)) = arg max = arg max If i 6= 0, the we ca di eretiate t btai X h (y i i ) i i + cstat i= X h (yi i ) i + i i= (y i i ) + sig( i )=0 ) i = y i sig( i ) ad clearly the miimizer must have the same sig as y i, ad s b i = y i sig(y i ) Pluggig this it the argumet f the miimizati yields (y i b i ) + b i = = y i sig(y i ) + y i sig(y i ) () O the ther had if b i = 0, the the bjective fucti s value is Observe that (y i b i ) + b i = y i () () < (), whe y i > () > (), whe y i apple Therefre, the ptimal sluti is b i = 8 < : 0 if y i apple y i sig(y i ) if y i > This is called a sft-threshld fucti. It ca be writte cmpactly as b i = sig(y i ) max( y i, 0)

5 Lecture : Sigal Subspaces ad Sparsity 5 Figure 3: Plt f sft-threshld. It is depicted graphically i Figure 3. The sft-threshld estimatr is: 3 b 6 7 b = 4. 5, bs = U b = X b i u i b i: b i 6=0 where u i is the ith clum (basis vectr) f U. Nte that the sft-threshld estimatr autmatically selects a sigal subspace based the magitude/eergy f the bserved data i each -dimesial subspace. 5 Summary We studied the sigal plus ise mdel y = U T x + w, w N (0, I) The MLE is b MLE x u x u = U T x = y. The Wieer filter (based a Gaussia prir) is give by b Wieer = + r b Wieer,i = i i + y, N (0, y i, i N (0, I) ii)

6 Lecture : Sigal Subspaces ad Sparsity 6 The sft-threshldig estimatr based the Laplacia prir i iid e i has the frm y b + b i = sig(y i ) max( y i, 0)! Data-adaptive shrikage t trade bias ad variace. Example 5 Csider the fllwig bservati MLE: =, y = apple 0 apple 0 b MLE = y =. prbably just ise full dimesi Wieer filter: Sft-threshd: b ST = apple b Wieer = 0 + apple max(0, 0) max(, 0) = = / apple 9 0 apple 0 full dimesi shrik t -dimesi 6 Iverse prblems Suppse we bserve a distrted sigal s i ise: x = As + w () = AU + w, w N (0, A is a kw matrix, suppse s is sparse i basis U, ad write s = U. Wieer Filter (with Gaussia Prir): N (0, I) ) liear, -adaptive. b Wieer Sparse Sluti (Laplacia Prir): ) -liear, adaptive. b L I) x AU + Lass x AU + Lass Bth are cvex ptimizatis. The Wieer filter, which is liear, has a liear-algebraic sluti. The Lass (Least abslute shrikage ad selecti peratr) is liear, ad des t have a simple clsed-frm sluti (except whe A = I). The EM algrithm ca be used t slve the Lass ptimizati. Recall Example 6 frm Lecture 6. That shws hw t cmpute the fucti Q(, (t) ) fr the bservati mdel x = H + w. Abve we are simply usig A istead f H. The EM algrithm ca be used t maximize the lg likelihd plus lg prir simply by chagig the M-step t = arg max Q(, (t) ) + lg p(). P I the Lass prblem, lg p() = = i i. I this case, the M-step ivlves a simple crdiatewise sft-threshldig perati. (t+)

7 Lecture : Sigal Subspaces ad Sparsity 7 7 The Lass Csider the bservati mdel ad the fllwig estimatr f x = A + w, w N (0, I), b x A +. This is called the lass estimatr, stadig fr least abslute shrikage ad selecti peratr, rigially prpsed i Tibshirai, R. (996). Regressi shrikage ad selecti via the lass. J. Ryal. Statist. Sc B., Vl. 58, N., pages 67-88). As discussed abve, the lass ca be derived as a MAP estimatr fr a Gaussia likelihd ad Laplacia prir. Based this bservati, the lass estimatr ca be cmputed usig a EM algrithm. The EM algrithm is a bud ptimizati algrithm; at each step we ptimize a bud the bjective fucti that is easier t slve tha the rigial prblem. Here we derive a bud ptimizati algrithm fr the lass. The bud ptimizati algrithm will prduce a sequece f iterates (), (),..., ad the iitial iterate () ca be arbitrary (e.g., radm, equal t zer, equal t x). T derive the algrithm, suppse we have a curret iterate (t). We ca write the bjective fucti as fllws. L() = kx Ak + kk = kx A (t) + A (t) Ak + kk = kx A (t) k + [A T (x A (t) )] T ( (t) )+ka (t) Ak + kk = C + [A T (x A (t) )] T ( (t) )+ka (t) Ak + kk where C = kx A (t) k is a cstat idepedet f the variable. Ctiuig, we have L() = C + [A T (x A (t) )] T ( (t) )+( (t) ) T A T A( (t) )+ kk apple C + [A T (x A (t) )] T ( (t) )+ k (t) k + kk where ay value greater tha r equal t the largest eigevalue f A T A. Our ext iterate will be the value f that miimizes this upper bud. The miimizer is ua ected by multiplyig r addig cstats, s we have (t+) C + [A T (x A (t) )] T ( (t) )+ k (t) k + kk C + [A T (x A (t) )] T ( (t) )+k (t) k + kk C 0 + [A T (x A (t) )] T ( (t) )+k (t) k + 0 kk where C 0 = C ad 0 =. Let z := [A T (x A (t) )] a cstat vectr idepedet f. Nw write (t+) C 0 +z T ( (t) )+k (t) k + 0 kk C 00 + kz + (t) k + 0 kk kz + (t) k + 0 kk

8 Lecture : Sigal Subspaces ad Sparsity 8 where C 00 = C 0 t kzk, ather cstat idepedet f. Ntice that the fial ptimizati is equivelet (t+) kx (t) k + 0 kk where x (t) := (t) + [A T (x A (t) )]. The sluti t this ptimizati is the sft-threshldig estimatr, easily cmputed as fllws. (t+) i = sig(x (t) i ) max( x (t) i 0, 0). T sum up, the iterative algrithm starts with a iitial guess () ad cmputes fr t =,,... x (t) = (t) + [A T (x A (t) )] (t+) i = sig(x (t) i ) max( x (t) i where 0 < apple, the largest eigevalue f A T A., 0),i=,..., 8 Statistical Aalysis f Sft-Threshldig Csider the direct bservati mdel where y R is give by y = + w, w N (0, I). Suppse that may f the ce ciets i are equal t zer. The MLE f is simply x, ad its MSE is. The sft-threshldig estimatr b i = sig(y i ) max( y i t, 0),t>0 ca perfrm much better, especially if is sparse. Befre we aalyze the sft-threshldig estimatr, let us csider a ideal threshldig estimatr. Suppse that a racale tells us the magitude f each i.theracle estimatr is b O i = yi if i 0 if i < I ther wrds, we estimate a ce The MSE f this estimatr is ciet if ad ly if the sigal pwer is at least as large as the ise pwer. E X ( b i O i ) = i= X mi( i, Ntice that the MSE f the racle estimatr is always less tha r equal t the MSE f the MLE. If is sparse, the the MSE f the racle estimatr ca be much smaller. If all but k<ce ciets are zer, the the MSE f the racle estimatr is at mst k. Remarkably, the sft-threshldig estimatr cmes very clse t achievig the perfrmace f the racle, ad shw by the fllwig therem (Therem i Ideal Spatial Adaptati by Wavelet Threshldig, by Dh ad Jhste). Therem Assume the direct bservati mdel abve ad let i= b i = sig(y i ) max( y i t, 0) ) with t = p lg. The Ek b k apple ( lg + ) ( + X mi( i, i= ) )

9 Lecture : Sigal Subspaces ad Sparsity 9 The therem shws that the sft-threshldig estimatr mimics the MSE perfrmace f the racle estimatr t withi a factr f rughly lg. Fr example, if is k-sparse (with -zer ce ciets larger tha i magitude), the the MSE f the racle is k ad the MSE f the sft-threshldig estimatr is at mst ( lg + )(k + ) k lg whe is large. This als crrespds t a huge imprvemet ver the MLE if k lg. Prf: T simplify the aalysis, assume that =. The geeral result fllws directly. It su ce t shw that E[( b i i ) ] apple ( lg + ) +mi( i, ) fr each i. Sletx N (µ, ) ad let t (x) = sig(x) max( x t, 0). We will shw that with t = p lg E[( t (x) µ) ] apple ( lg + ) +mi(µ, ). First te that t (x) =x sig(x)( x ^ t), where a ^ b is shrthad tati fr mi(a, b). It fllws that E[( t (x) µ) ] = E[(x µ) ] E[sig(x)( x ^ t)(x µ)] + E[x ^ t ] = E[sig(x)( x ^ t)(x µ)] + E[x ^ t ] The expected value i the secd term is equal t P( x <t), which is verified as fllws. The expectati ca be split it itegrals ver fur itervals, (, t], ( t, 0], (0,t], ad (t, ). Each itegrad is a liear r quadratic fucti f x times the Gaussia desity fucti. Let (x) := p e x / ad (x) := R (y)dy, the cumulative distributi fucti f (x), ad csider the fllwig idefiite x Gaussia itegral frms: (x) dx = (x), by defiiti f, x (x) dx = p xe x / dx = p e u du = p e u = (x), u= x / x (x) dx = (x) x (x). The last frm is verified as fllws. Let u = x ad dv = x (x)dx. The itegrati by parts R R R udv = uv vdu ad x (x)dx = (x) shw that x (x) dx = x x (x)dx x (x)dx = x (x)+ (x) = (x) x (x). The Gaussia distributi we are csiderig has mea µ s the shifted itegral frms belw, which fllw immediately frm the derviatis abve by variable substituti, will be used i ur aalysis: R (i) (x µ)dx = (x µ) R (ii) x (x µ)dx = µ (x µ) (x µ) R (iii) x (x µ)dx = (+µ ) (x µ) (x + µ) (x µ)

10 Lecture : Sigal Subspaces ad Sparsity 0 Usig these frms we cmpute E[sig(x)( x ^ t)(x µ)] = = t sig(x)( x ^ t)(x µ) (x µ) dx t(x µ) (x µ) dx + t t ( t µ) x(x µ) (x µ)dx + 0 (t µ) ( µ) t (t µ) 0 x(x µ) (x µ)dx t ( µ) ( t µ) t ( t µ) t(x µ) (x µ)dx t t (t µ) = (t µ) ( t µ) =P( x <t) S we have shw that E[( t (x) µ) ] = P( x <t)+e[x ^ t ] Nte first that sice x ^ t apple t we have E[( t (x) µ) ] apple +t = + lg <(lg + )(/ + ). O the ther had, sice x ^ t apple x we als have E[( t (x) µ) ] apple P( x <t)+µ + = ( P( x <t)) + µ = P( x t)+µ. The prf will be fiished if we shw that P( x t) apple ( lg + )/ + ( lg )µ. Defie g(µ) :=P( x have t) ad te that g is symmetric abut 0. Usig a Taylr s series with remaider we g(µ) apple g(0) + sup g00 µ, where g 00 is the secd derivative f g. Nte that g(µ) =[ P(z apple t µ) P(z apple t µ)], where z N (0, ). Usig the Gaussia tail bud P(z >t) apple e t / ad pluggig i t = p lg we btai g(0) apple /. Nte that g 0 (µ) = [ (t µ)+ ( t µ)] ad g 0 (0) = 0. The itegral (ii) abve shws that the derivative f (t µ)withrespecttµ is equal t (t µ) (t µ). S we have g 00 (µ) = [(t µ) (t µ)+( t µ) ( t µ)]. It is easy t check that g 00 (µ) < s it fllws that sup µ g 00 (µ) apple 4 lg fr all. 8. The Lass Csider the bservati mdel ad the fllwig estimatr f x = A + w, w N (0, I), b x A +. This is called the lass estimatr, stadig fr least abslute shrikage ad selecti peratr, rigially prpsed i Tibshirai, R. (996). Regressi shrikage ad selecti via the lass. J. Ryal. Statist. Sc B., Vl.

11 Lecture : Sigal Subspaces ad Sparsity 58, N., pages 67-88). The lass has received a ermus amut f atteti i recet years. If A = I, thethisreducest the sft-threshldig estimatr with threshld. Mre geerally if A acts like the idetity peratr fr all sparse vectrs, the statistical errr buds similar t thse fr sft-threshldig ca be btaied. Fr example, if x is k-sparse ad A satisfies the s-called Restricted Ismetry Prperty ( )ku vk apple ka(u v)k apple ( + )ku vk fr all k sparse vectrs u, v R fr a > 0, the the MSE f the lass estimatr is early equal t that f a racle estimatr that kws the lcatis f the -zer cmpets f x. A cmprehesive verview f lass errr buds is give i this paper va de Geer, S. ad Buhlma, P. (009). O the cditis used t prve racle results fr the Lass. Electric Jural f Statistics 3, Wavelet Deisig Wavelets are lcally supprted, scillatig fuctis that itegrate t zer. The Daubechies wavelets are special fuctis that satisfy the s-called vaishig mmet cditis. If (t) is the Daubechies wavelet with k vaishig mmets, the t` (t) dt = 0, ` =0,,...,k. The discrete-time aalgs defied t {0,,..., } satisfy X t=0 t` (t) = 0, ` =0,,...,k. The vaishig mmets prperty implies that if we itegrate such a wavelet agaist a plymial f degree k r less, the the itegral (i.e., ier prduct betwee the wavelet ad the plymial) is zer. Figure 4: Daubechies wavelet with 4 vaishig mmets. Wavelets als ca be used t geerate a rthrmal basis fr L fuctis the uit iterval [0, ) r vectrs i R (based respectively the ctiuus r discrete wavelets, abve). A rthmal basis is geerated by scalig ad shiftig the argumet f the basic wavelet fucti. This prduces cmpressed ad dilated versis f the wavelet at di eret lcatis i the iterval. I the case f R, the Daubechies wavelet basis csists f wavelet fuctis at scales, /, /4,...,/ plus e cstat r lw frequecy vectr t cmplete the basis. The scale idicates the apprximate fracti

12 Lecture : Sigal Subspaces ad Sparsity f the set {0,,..., } that each wavelet fucti is supprted. At scale j each wavelet is supprted O( j ) pits ad there are j wavelet fuctis spaces uifrmly ver the set {0,,..., }, fr j =0,,...,lg /. Figure 5: Daubechies wavelet basis fuctis with vaishig mmets. Dete the wavelet fuctis by { i } i=. There are fuctis ad this set frms a rthrmal basis fr R. That is, ay vectr x R ca be represeted exactly as x = X i ( 0 ix) i. Defie the wavelet ce ciets f x t be the ier prducts i = 0 ix, i =,..., ad let W dete the matrix with clums equal t the wavelet basis fuctis. The the vectr f wavelet ce ciets = W 0 x ad x = W. The matrix W 0 is called the wavelet trasfrm ad W is the iverse wavelet trasfrm. Let us recrd a few key facts abut wavelet trasfrms. Because f the lcal supprt f the wavelet fuctis, the value f x(t) fr ay pit t {0,,..., } is a liear cmbiati f abut lg () wavelet fuctis (rughly e at each scale). I ther wrds, all but abut lg () wavelet fuctis are equal t zer at ay give pit t. Als te that if the values f x are determied by a plymial, the mst the wavelet ce ciets (the ier prducts betwee x ad the wavelet basis fuctis) will be zer by the vaishig mmets prperty. Puttig these tw facts tgether we have the fllwig result. If the values f x R are piecewise plymial with m pieces, the all but abut m lg wavelet ce ciets will be zer (i.e., the ce ciets will be sparse). Nw suppse that x is a vectr f samples f a Hölder -smth fucti [0, ]. Let R dete the wavelet trasfrm f this vectr based a wavelet with at least k = d e vaishig mmets. Sice the samples are apprximately plymial i ay small subiterval, we cclude less tha Cmlg ce ciets will be large i magitude, where C>0isacstat. The rest f the ce ciets will be less tha Cm i magitude. Suppse that we make isy measuremets f x. Each sample is crrupted with idepedet stadard Gaussia ise. Sice the wavelet trasfrm is rthrmal, the trasfrm f the isy samples is give by iid where w i N (0, ). We ca deise these ce b satisfyig y i = i + w i,i=,...,, Ek b k apple ( lg + ) ciets accrdig t Therem abve t btai a estimate ( + ) X mi( i, ) i=

13 Lecture : Sigal Subspaces ad Sparsity 3 Sice few ce t ciets are large ad mst are very small we have that the average MSE is buded accrdig /( +), If we take m = lg the m Ekb k apple C lg lg + m. Ekb k apple C lg /( +) lg. This is withi a ply-lgarithmic factr f the MSE based piecewise plymial fittig. The wavelet deisig apprach als has sigificat advatages. First f all, tice that wavelet deisig is autmatically adaptive t the smthess f the uderlyig fucti. Usig a wavelet basis with k vaishig mmets, it achieves the ptimal rate f cvergece (up t a plylg factr) fr ay Hölder class with d e applek +. Wavelet deisig requires prir kwledge f, ulike classical methds that eed t kw i rder t chse the umber f plymial pieces t fit t the data. Secdly, wavelet deisig ca eve hadle piecewise smth fuctis with disctiuities betwee pieces. Suppse that the uderlyig fucti is ly piecewise Hölder smth (e.g., it may have a few pits f disctiuity). The the wavelet deisig rate stays the same as abve, but the piecewise plymial rate degrades t /. Figure 6: Piecewise smth fucti (tp). Ce mmets (bttm). Ntice that mst f the ce ciets f Daubechies wavelet trasfrm with 4 vaishig ciets are zer. Figure 7: Deisig i acti. Origial fucti (tp). Nisy versi (middle). Deised estimate btaied by threshldig isy wavelet ce ciets (bttm).