Bayesia Estimati fr Ctiuus-Time Sparse Stchastic Prcesses Arash Amii, Ulugbek S Kamilv, Studet, IEEE, Emrah Bsta, Studet, IEEE, Michael User, Fellw, IEEE Abstract We csider ctiuus-time sparse stchastic prcesses frm which we have ly a fiite umber f isy/iseless samples Our gal is t estimate the iseless samples (deisig) ad the sigal i-betwee (iterplati prblem) By relyig tls frm the thery f splies, we derive the jit a priri distributi f the samples ad shw hw this prbability desity fucti ca be factrized The factrizati eables us t tractably implemet the maximum a psteriri ad miimum mea-square errr (MMSE) criteria as tw statistical appraches fr estimatig the ukws We cmpare the derived statistical methds with well-kw techiques fr the recvery f sparse sigals, such as the ` rm ad Lg (`-` relaxati) regularizati methds The simulati results shw that, uder certai cditis, the perfrmace f the regularizati techiques ca be very clse t that f the MMSE estimatr Idex Terms Deisig, Iterplati, Lévy Prcess, MAP, MMSE, Statistical Learig, Sparse Prcess I INTODUCTION THE recet ppularity f regularizati techiques i sigal ad image prcessig is mtivated by the sparse ature f real-wrld data It has resulted i the develpmet f pwerful tls fr may prblems such as deisig, decvluti, ad iterplati The emergece f cmpressed sesig, which fcuses the recvery f sparse vectrs frm highly uder-sampled sets f measuremets, is playig a key rle i this ctext [], [], [] Assume that the sigal f iterest {s[i]} m i is a fiite-legth discrete sigal als represeted by s as a vectr) that has a sparse r almst sparse represetati i sme trasfrm r aalysis dmai (eg, wavelet r DCT) Assume mrever that we ly have access t isy measuremets f the frm m s[i] s[i]+[i], where [i] m detes a additive i i white Gaussia ise The, we wuld like t estimate {s[i]} i The cmm sparsity-prmtig variatial techiques rely pealizig the sparsity i the trasfrm/aalysis dmai [], [5] by impsig ŝ[i] m arg mi i ks sk` + J sparse (s), () {s[i]} where s is the vectr f isy measuremets, J sparse ( ) is a pealty fucti that reflects the sparsity cstrait i the trasfrm/aalysis dmai ad is a weight that is usually set based the ise ad sigal pwers The chice f Mauscript received December 8, ; revised July 7, The authrs are with the Bimedical Imagig Grup (BIG), Écle plytechique fédérale de Lausae (EPFL), Lausae, Switzerlad Emails: {arashamii,ulugbekkamilv,emrahbsta,michaeluser}@epflch This wrk was supprted by the Eurpea esearch Ceter (EC) uder FUN-SP grat J sparse ( ) is e f the favrite es i cmpressed k k` sesig whe {s[i]} m i is itself sparse [6], while the use f J sparse (s) TV(s), where TV stads fr ttal variati, is a cmm chice fr piecewise-smth sigals that have sparse derivatives [7] Althugh the estimati prblem fr a give set f measuremets is a determiistic prcedure ad ca be hadled withut recurse t statistical tls, there are beefits i viewig the prblem frm the stchastic perspective Fr istace, e ca take advatage f side ifrmati abut the ubserved data t establish prbability laws fr all r part f the data Mrever, a stchastic framewrk allws e t evaluate the perfrmace f estimati techiques ad argue abut their distace frm the ptimal estimatr The cvetial stchastic iterpretati f the variatial methd i () leads t the fidig that {ŝ[i]} m i is the maximum a psteriri (MAP) estimate f {s[i]} m i I this iterpretati, the quadratic data term is assciated with the Gaussia ature f the additive ise, while the sparsifyig pealty term crrespds t the a priri distributi f the sparse iput Fr example, the pealty J sparse ( ) k k` is assciated with the MAP estimatr with Laplace prir [8], [9] Hwever, ivestigatis f the cmpressible/sparse prirs have revealed that the Laplace distributi cat be csidered as a sparse prir [], [], [] ecetly i [], it is argued that () is better iterpreted as the miimum mea-square errr (MMSE) estimatr f a sparse prir Thugh the discrete stchastic mdels are widely adpted fr sparse sigals, they ly apprximate the ctiuus ature f real-wrld sigals The mai challege fr emplyig ctiuus mdels is t traspse the cmpressibility/sparsity ccepts i the ctiuus dmai while maitaiig cmpatibility with the discrete dmai I [], a exteded class f piecewise-smth sigals is prpsed as a cadidate fr ctiuus stchastic sparse mdels This class is clsely related t sigals with a fiite rate f ivati [5] Based ifiitely divisible distributis, a mre geeral stchastic framewrk has bee recetly itrduced i [6], [7] There, the ctiuus mdels iclude Gaussia prcesses (such as Brwia mti), piecewise-plymial sigals, ad -stable prcesses as special cases I additi, a large prti f the itrduced family is csidered as cmpressible/sparse with respect t the defiiti i [] which is cmpatible with the discrete defiiti I this paper, we ivestigate the estimati prblem fr the samples f the ctiuus-time sparse mdels itrduced i [6], [7] We derive the a priri ad a psteriri prbability desity fuctis (pdf) f the iseless/isy samples
We preset a practical factrizati f the prir distributi which eables us t perfrm statistical learig fr deisig r iterplati prblems I particular, we implemet the ptimal MMSE estimatr based the message-passig algrithm The implemetati ivlves discretizati ad cvluti f pdfs, ad is i geeral, slwer tha the cmm variatial techiques We further cmpare the perfrmace f the Bayesia ad variatial deisig methds Amg the variatial methds, we csider quadratic, TV, ad Lg regularizati techiques Our results shw that, by matchig the regularizer t the statistics, e ca almst replicate the MMSE perfrmace The rest f the paper is rgaized as fllws: I Secti II, we itrduce ur sigal mdel which relies the geeral frmulati f sparse stchastic prcesses prpsed i [6], [7] I Secti IV, we explai the techiques fr btaiig the prbability desity fuctis ad, we derive the estimati methds i Secti III We study the special case f Lévy prcesses which is f iterest i may applicatis i Secti V, ad preset simulati results i Secti VI Secti VII ccludes the paper II SIGNAL MODEL I this secti, we adapt the geeral framewrk f [6] t the ctiuus-time stchastic mdel studied i this paper We fllw the same tatial cvetis ad write the iput argumet f the ctiuus-time sigals/prcesses iside parethesis (eg, s( )) while we emply brackets (eg, s[ ]) fr discrete-time es Mrever, the tilde diacritic is used t idicate the isy sigal Typically, s[ ] represets discrete isy samples I Figure, we give a sketch f the mdel The tw mai parts are the ctiuus-time ivati prcess ad the liear peratrs The prcess s( ) is geerated by applyig the shapig peratr L the ivati prcess w It ca be whiteed back by the iverse peratr L (Sice the whiteig peratr is f greater imprtace, it is represeted by L while L refers t the shapig peratr) Furthermre, the discrete bservatis s[ ] are frmed by the isy measuremets f s( ) The ivati prcess ad the liear peratrs have distict implicatis the resultat prcess s Our mdel is able t hadle geeral ivati prcesses that may r may t iduce sparsity/cmpressibility The disticti betwee these tw cases is idetified by a fucti f(!) that is called the Lévy expet, as will be discussed i Secti II-A The sparsity/cmpressibility f s ad, csequetly, f the measuremets s, is iherited frm the ivatis ad is bserved i a trasfrm dmai This dmai is tied t the peratr L I this paper, we deal with peratrs that we represet by all-ple differetial systems, tued by actig up the ples Althugh the mdel i Figure is rather classical fr Gaussia ivatis, the ivestigati f -Gaussia ivatis is trivial While the trasiti frm Gaussia t - Gaussia ecessitates the reivestigati f every defiiti ad result, it prvides us with a mre geeral class f stchastic prcesses which icludes cmpressible/sparse sigals A Ivati Prcess Of all white prcesses, the Gaussia ivati is udubtedly the e that has bee ivestigated mst thrughly Hwever, it represets ly a tiy fracti f the large family f white prcesses, which is best explred by usig Gelfad s thery f geeralized radm prcesses I his apprach, ulike with the cvetial pit-wise defiiti, the stchastic prcess is characterized thrugh ier prducts with test fuctis Fr this purpse, e first chses a fucti space E f test fuctis (eg, the Schwartz class S f smth ad rapidly decayig fuctis) The, e csiders the radm variable give by the ier prduct hw, 'i, where w represets the ivati prcess ad ' E [8] Defiiti : A stchastic prcess is called a ivati prcess if ) it is statiary, ie, the radm variables hw, ' i ad hw, ' i are idetically distributed, prvided ' is a shifted versi f ', ad ) it is white i the sese that the radm variables hw, ' i ad hw, ' i are idepedet, prvided ',' Eare -verlappig test fuctis (ie, ' ' ) The characteristic frm f w( ) is defied as 8 ' E: ˆ P w (') E e jhw,'i, () where E{ } represets the expected-value peratr The characteristic frm is a pwerful tl fr ivestigatig the prperties f radm prcesses Fr istace, it allws e t easily ifer the prbability desity fucti f the radm variable hw, 'i, r the jit desities f hw, ' i,,hw, ' i Further details regardig characteristic frms ca be fud i Appedix A The key pit i Gelfad s thery is t csider the frm ˆ P w (') exp f '(x) dx () ad t prvide the ecessary ad sufficiet cditis f(!) (the Lévy expet) fr w t defie a geeralized ivati prcess ver S (dual f S) The class f admissible Lévy expets is characterized by the Lévy-Khitchie represetati therem [9], [] as f(!) jµ! + \{}! e ja! j!a ],[ (a) v(a)da, () where B(a) fr a Bad therwise, ad v( ) (the Lévy desity) is a real-valued desity fucti that satisfies mi(,a )v(a)da< (5) \{} I this paper, we csider ly symmetric real-valued Lévy expets (ie, µ ad v(a) v( a)) Thus, the geeral frm f () is reduced t f(!)! + (cs(a!) ) v(a) da (6) \{} Next, we discuss three particular cases f (6) which are f special iterest i this paper
Shapig Op (Liear) L White Ivati w(x) { } Sparse Prcess Discrete Measuremets + s(x) Samplig L{ } [i] Whiteig Op Fig s [i] AWGN Cecti betwee the white ise w(x), the sparse ctiuus sigal s(x), ad the discrete measuremets s [i] ) Gaussia Ivati: The chice v turs (6) it fg (!)!, (7) which implies P wg (') e k'k (8) This shws that the radm variable hwg, 'i has a zer-mea Gaussia distributi with variace k'k ) Impulsive Piss: Let ad v(a) pa (a), where pa is a symmetric prbability desity fucti The crrespdig white prcess is kw as the impulsive Piss ivati By substituti i (6), we btai fip (!) (cs(a!) ) pa (a) da \{} p a (!) (9), where p a detes the Furier trasfrm f pa Let [,] represets the test fucti that takes [, ] ad therwise Thus, if X hwip, [,] i, the fr the pdf f X we kw that (see Appedix A) px (x) F! e (p a (!) ) (x) X I [], this type f ivati is itrduced as a ptetial cadidate fr sparse prcesses, sice all the ier prducts have a mass prbability at x ) Symmetric -Stable: The stable laws are prbability desity fuctis that are clsed uder cvluti Mre precisely, the pdf f a radm variable X is said t be stable if, fr tw idepedet ad idetical cpies f X, amely, X, X, ad fr each pair f scalars c, c, there exists c such that c X + c X has the same pdf as cx Fr stable laws, it is kw that c c +c fr sme < []; this is the reas why the law is idexed with A -stable law which crrespds t a symmetric pdf is called symmetric -stable It is pssible t defie symmetric -stable white prcesses fr < < by csiderig ad v(a) a c+, where < c Frm (6), we get fs (!) si a! cs(a!) da c da + a a + \{} si x c! dx c!, () + x c where c is a psitive cstat This yields P ws (') e c k'k (), pa pa (x)() {z } which cfirms that every radm variable f the frm hws, 'i has a symmetric -stable distributi [] The fat-tailed distributis icludig -stables fr < < are kw t geerate cmpressible sequeces [] Meawhile, the Gaussia distributis are als stable laws that crrespd t the extreme value ad have classical ad well-kw prperties that differ fudametally frm -Gaussia laws The key message f this secti is that the ivati prcess is uiquely determied by its Le vy expet f (!) We shall explai i Secti II-C hw f (!) affects the sparsity ad cmpressibility prperties f the prcess s where ( ) stads fr the Dirac distributi, xk k is a sequece f radm Piss pits with parameter, ad {ak }k is a idepedet ad idetically distributed (iid) sequece with prbability desity pa idepedet f {xk } The sequece {xk }k is a Piss pit radm sequece with parameter if, fr all real values a < b < c < d, the radm variables N {xk } \ [a, b] ad N {xk } \ [c, d] are idepedet ad N (r N ) fllws the Piss distributi with mea (b a) (r (d c)), which ca be writte as B Liear Operatrs The secd imprtat cmpet f the mdel is the shapig peratr (the iverse f the whiteig peratr L) that determies the crrelati structure f the prcess Fr the geeralized stchastic defiiti f s i Figure, we expect t have e e F! (x) + i X i p a (!) i! i e i! i (x) i times It is t hard t check (see Appedix II i [] fr a prf) that this distributi matches the e that we btai by defiig X w(x) ak (x xk ), () k Prb{N } e (b a) (b! a) () hs, 'i hl w, 'i hw, L 'i, (5) where L represets the adjit peratr f L It shws that L ' ught t defie a valid test fucti fr the
equalities i (5) t remai valid I tur, this sets cstraits L The simplest chice fr L wuld be that f a peratr which frms a ctiuus map frm S it itself, but the class f such peratrs is t rich eugh t cver the desired mdels i this paper Fr this reas, we take advatage f a result i [6] that exteds the chice f shapig peratrs t thse L peratrs fr which L frms a ctiuus mappig frm S it Lp fr sme p ) Valid Iverse Operatr L : I the sequel, we first explai the geeral requiremets the iverse f a give whiteig peratr L The, we fcus a special class f peratrs L ad study the implicatis fr the assciated shapig peratrs i mre details We assume L t be a give whiteig peratr, which may r may t be uiquely ivertible The miimum requiremet the shapig peratr L is that it shuld frm a rightiverse f L (ie, LL I, where I is the idetity peratr) Furthermre, sice the adjit peratr is required i (5), L eeds t be liear This implies the existece f a kerel h(x, ) such that L w(x) h(x, )w( )d (6) Liear shift-ivariat shapig peratrs are special cases that crrespd t h(x, ) h(x ) Hwever, sme f the L peratrs csidered i this paper are t shift-ivariat We require the kerel h t satisfy the fllwig three cditis: (i) Lh(x, ) (x ), where L acts the parameter x ad is the Dirac fucti, (ii) h(x, ), fr > max(, x), h(x, ) (iii) ( + p ) + x p dx is buded fr all p Cditi (i) is equivalet t LL I, while (iii) is a sufficiet cditi studied i [6] t establish the ctiuity f the mappig L : S 7! Lp, fr all p Cditi (ii) is a cstrait that we impse i this paper t simplify the statistical aalysis Fr x, its implicati is that the radm variable s(x) L w(x) is fully determied by w( ) with x, r, equivaletly, it is idepedet f w( ) fr > x Frm Pw, we fcus differetial peratrs L f dthe frm i i Di, where D is the first-rder derivative ( dx ), D is the idetity peratr (I), ad i are cstats With Gaussia ivatis, these peratrs geerate the autregressive prcesses A equivalet represetati f L, which helps us i the aalysis, Q is its decmpsiti it first-rder factrs as L i (D ri I) The scalars ri are the rts f the characteristic plymial ad crrespd t the ples f the iverse liear system Here, we assume that all the ples are i the left half-plae <ri This assumpti helps us assciate the peratr L t a suitable kerel h, as shw i Appedix B Every differetial peratr L has a uique causal Gree fucti L [] The liear shift-ivariat system defied by h(x, ) L (x ) satisfies cditis (i)-(ii) If all the ples strictly lie i the left half-plae (ie, <ri < ), due t abslute itegrability f L (stability f the system), h(x, ) satisfies cditi (iii) as well The defiiti f L give thrugh the (x ) L { } Ld,T { } L,T (x ) (a) w(x) ut (x) L,T (b) Fig (a) Liear shift-ivariat peratr Ld,T L ad its impulse respse L,T (L-splie) (b) Defiiti f the auxiliary sigal ut (x) kerels i Appedix B achieves bth liearity ad stability, while lsig shift-ivariace whe L ctais ples the imagiary axis It is wrth metiig that the applicati f tw differet right-iverses f L a give iput prduces results that differ ly by a expetial plymial that is i the ull space f L ) Discretizati: Apart frm qualifyig as whiteig peratrs, differetial peratrs have ther appealig prperties such as the existece f fiite-legth discrete cuterparts T explai this ccept, let us first csider the first-rder ctiuus-time derivative peratr D that is assciated with the fiite-differece filter H(z) z This discrete cuterpart is f fiite legth (FI filter) Further, fr ay right iverse f D such as D, the system Dd,T D is shift ivariat ad its impulse respse is cmpactly supprted Here, Dd,T is the discretized peratr crrespdig t the samplig perid T with impulse respse ( ) ( T ) It shuld be emphasized that the discrete cuterpart H(z) is a discrete-dmai peratr, while the discretized peratr acts ctiuus-dmai sigals It is easy t check that this impulse respse cicides with the causal B-splie f degree Q ( [,[ ) I geeral, the discrete cuterpart f L ri I) is defied thrugh its factrs Each D ri I i (D is assciated with its discrete cuterpart Hi (z) eri z ad a discretized peratr give by the impulse respse ( ) eri T ( T ) The cvluti f such impulse respses gives rise t the impulse respse f Ld,T (up t the scalig factr ), which is the discretized peratr f L fr the sampligp perid T By expadig the cvluti, we btai the frm i dt [k] ( kt ) fr the impulse respse f Ld,T It is w evidet that Ld,T crrespds t a FI filter f legth ( + ) represeted by {dt [k]}k with dt [] 6 esults i splie thery cfirm that, fr ay right iverse L f L, the peratr Ld,T L is shift ivariat ad the supprt f its impulse respse is ctaied i [, T ) [] The cmpactly supprted impulse respse f Ld,T L, which we dete by L,T ( ), is usually referred t as the L-splie We defie the geeralized fiite differeces by ut (x) L,T (Ld,T w (x) (Ld,T L w)(x) X s)(x) dt [k]s(x kt ) (7) k We shw i Figures (a), (b) the defiitis f ad ut (x), respectively L,T (x)
5 C Sparsity/Cmpressibility The ivati prcess ca be thught f as a ccateati f idepedet atms A csequece f this idepedece is that the prcess ctais redudacies Therefre, it is icmpressible uder uique represetati cstrait I ur framewrk, the rle f the shapig peratr L is t geerate a specific crrelati structure i s by mixig the atms f w Cversely, the whiteig peratr L udes the mixig ad returs a icmpressible set f data, i the sese that it maximally cmpresses the data Fr a discretizati f s crrespdig t a samplig perid T, the peratr L d,t mimics the rle f L It efficietly ucuples the sequece f samples ad prduces the geeralized differeces u T, where each term depeds ly a fiite umber f ther terms Thus, the rle f L d,t ca be cmpared t that f cvertig discrete-time sigals it their trasfrm dmai represetati As we explai w, the sparsity/cmpressibility prperties f u T are clsely related t the Lévy expet f f w The ccept is best explaied by fcusig a special class kw as Lévy prcesses that crrespd t L,T (x) [,T [(x) (see Secti V fr mre details) By usig () ad (5), we ca check that the characteristic fucti f the radm variable u T is give by e Tf(!) Whe the Lévy fucti f is geerated thrugh a zer desity v(a) that is abslutely itegrable (ie, impulsive Piss), the pdf f u T assciated with e Tf(!) ecessarily ctais a mass at the rigi [] (Therems 8 ad ) This is iterpreted as sparsity whe csiderig a fiite umber f measuremets It is shw i [], [] that the cmpressibility f the measuremets depeds the tail f their pdf I simple terms, if the pdf decays at mst iverse-plymially, the, it is cmpressible i sme crrespdig L p rm The iterestig pit is that the iverse-plymial decay f a ifiitedivisible pdf is equivalet t the iverse-plymial decay f its Lévy desity v( ) with the same rder [] (Therem 7) Therefre, a ivati prcess with a fat-tailed Lévy desity results i prcesses with cmpressible measuremets This idicates that the slwer the decay rate f v( ), the mre cmpressible the measuremets f s D Summary f the Parameters f the Mdel We w briefly review the degrees f freedm i the mdel ad hw the depedet variables are determied The ivati prcess is uiquely determied by the Lévy triplet (µ,, v) The sparsity/cmpressibility f the prcess ca be determied thrugh the Lévy desity v (see Secti II-C) The values ad { i } i, r, equivaletly, the ples {r i} i f the system, serve as the free parameters fr the whiteig peratr L As explaied i Secti II-B, the taps f the FI filter d T [i] are determied by the ples III PIO DISTIBUTION T derive statistical estimati methds such as MAP ad MMSE, we eed the a priri distributis I this secti, by usig the geeralized differeces i (7), we btai the prir distributi ad factrize it efficietly This factrizati is fudametal, sice it makes the implemetati f the MMSE ad MAP estimatrs tractable The geeral prblem studied i this paper is t estimate s(x) at x i m m s m s i fr arbitrary m s N (values f the ctiuus prcess s at a uifrm samplig grid with T m m s ), give a fiite umber (m + ) f isy/iseless measuremets { s[k]} m k f s(x) at the itegers Althugh we are aware that this is t equivalet t estimatig the ctiuus prcess, by icreasig m s we are able t make the estimati grid as fie as desired Fr piecewise-ctiuus sigals, the limit prcess f refiig the grid ca give us access t the prperties f the ctiuus dmai T simplify the tatis let us defie st [i] s(x) xit, (8) u T [i] u T (x) xit, where u T (x) is defied i (7) Our gal is t derive the jit distributi f s T [i] (a priri distributi) Hwever, the s T [i] are i geeral pairwise-depedet, which makes it difficult t deal with the jit distributi i high dimesis This crrespds t a large umber f samples Meawhile, as will be shw i Lemma, the sequece {u T [i]} i frms a Markv chai f rder ( ) that helps i factrizig the jit prbability distributis, whereas {s T [i]} i des t The leadig idea f this wrk is the that each u T [i] depeds a fiite umber f u T [j], j 6 i It the becmes much simpler t derive the jit distributi f {u T [i]} i ad lik it t that f {s T [i]} i Lemma helps us t factrize the jit pdf f {u T [i]} i Lemma : Fr N ad i, where is the differetial rder f L, ) the radm variables {u T [i]} i are idetically distributed, ) the sequece {u T [i]} i is a Markv chai f rder, ad ) the sample u T [i+n] is statistically idepedet f u T [i] ad s T [i] Prf First te that u T [i] L,T w (x) xit hw, L,T (it )i (9) Sice L,T (it ) fuctis are shifts f the same fucti fr varius i ad w is statiary (Defiiti ), {u T [i]} i are idetically distributed ecallig that L,T ( ) is supprted [,T), we kw that L,T (it ) ad L,T (i + N)T have cmm supprt fr N Thus, due t the whiteess f w cf Defiiti ), the radm variables hw, L,T (it )i ad hw, L,T (i + N)T i are idepedet Csequetly, we ca write p u u T [i + ] u T [i + ],u T [i + ], p u u T [i + ] u T [i + ],,u T [i + ], () which cfirms that the sequece {u T [i]} i frms a Markv chai f rder ( ) Nte that the chice f N is due t the supprt f L,T If the supprt f L,T was ifiite, it wuld be impssible t fid j such that u T [i] ad u T [j] were idepedet i the strict sese
6 T prve the secd idepedece prperty, we recall that s T [i] s(x) xit h(it, )w( )d hw, h(it, )i () Cditi (ii) i Secti II-B implies that h(it, ) fr >max(,it) Hece, h(it, ) ad L,T (i+n)t have disjit supprts Agai due t whiteess f w, this implies that u T [i + N] ad s T [i] are idepedet We w explit the prperties f u T [i] t btai the a priri distributi f s T [i] Therem, which is prved i Appedix C summarizes the mai result f Secti III Therem : Usig the cvetis f Lemma, fr k we have p s s T [k],,s T [] ky d T [] p u u T [ ] u T [ i] i p s s T [ ],,s T [] () I the defiiti f L prpsed i Secti II-B, except whe all the ples are strictly icluded i the left half-plae, the peratr L fails t be shift-ivariat Csequetly, either s(x) r s T [i] are statiary A iterestig csequece f Therem is that it relates the prbability distributi f the -statiary prcess s T [i] t that f the statiary prcess u T [i] plus a miimal set f trasiet terms Next, we shw i Therem hw the cditial prbability f u T [i] ca be btaied frm a characteristic frm T maitai the flw f the paper, the prf is pstped t Appedix D Therem : The prbability desity fucti f u T [ ] cditied ( ) previus u T [i] is give by where F {! i } F {! i } p u u T [ ] u T [ i] e I w, L,T (!,,! ) e I w, L,T (,!,,! ) f w i {u T [ i]} i {u T [ i]} i, () I w, L,T (!,,! ): P i! i L,T (x it ) dx () IV SIGNAL ESTIMATION MMSE ad MAP estimati are tw cmm statistical paradigms fr sigal recvery Sice the ptimal MMSE estimatr is rarely realizable i practice, MAP is fte used as the ext best thig I this secti, i additi t applyig the tw methds t the prpsed class f sigals, we settle the questi f kwig whe MAP is a gd apprximati f MMSE Fr the estimati purpse it is cveiet t assume that the samplig istaces assciated with s[i] are icluded i the uifrm samplig grid fr which we wat t estimate the values f s I ther wrds, we assume that T m m T, where T is the samplig perid i the defiiti f s T [ ] ad T is a psitive iteger This assumpti des impse lwer bud the resluti f the grid because we ca set T arbitrarily clse t zer by icreasig T T simplify mathematical frmulatis, we use vectrial tatis We idicate the vectr f isy/iseless measuremets { s[i]} m i by s The vectr s T stads fr the hypthetical realizati {s T [k]} mt k f the prcess the csidered grid, ad s T,T detes the subset {s T [i T ]} m i that crrespds t the pits at which we have a sample Fially, we represet the vectr f estimated values {ŝ T [k]} mt k by ŝ T A MMSE Deisig It is very cmm t evaluate the quality f a estimati methd by meas f the mea-square errr, r SN I this regard, the best estimatr, kw as MMSE, is btaied by evaluatig the psterir mea, r ŝ T E{s T s} Fr Gaussia prcesses, this expectati is easy t btai, sice it is equivalet t the best liear estimatr [] Hwever, there are ly a few -Gaussia cases where a explicit clsed frm f this expectati is kw I particular, if the additive ise is white ad Gaussia ( restricti the distributi f the sigal) ad pure deisig is desired (T ), the the MMSE estimatr ca be refrmulated as ŝ MMSE s + r lg p s (x) x s, (5) where ŝ MMSE stads fr ŝ T,MMSE with T, ad is the variace f the ise [5], [6], [7] Nte that the pdf p s, which is the result f cvlvig the a priri distributi p s with the Gaussia pdf f the ise, is bth ctiuus ad differetiable B MAP Estimatr Searchig fr the MAP estimatr amuts t fidig the maximizer f the distributi p(s T s) It is cmmplace t refrmulate this cditial prbability i terms f the a priri distributi The additive discrete ise ñ is white ad Gaussia with the variace Thus, p s T s p s s T p s (s T ) p s ( s) e k s s T,T k ( ) m+ p s (s T ) p s ( s) (6) I MAP estimati, we are lkig fr a vectr s T that maximizes the cditial a psteriri prbability, s that (6) leads t ŝ T,MAP arg max s T p s T s arg max s T e k s s T,T k p s (s T ) ( ) m+ p s ( s) arg max e k s s T,T k p s (s T ) (7) s T The last equality is due t the fact that either ( ) m+ r p s ( s) deped the chice f s T Therefre, they play rle i the maximizati
7 If the pdf f s T is buded, the cst fucti (7) ca be replaced with its lgarithm withut chagig the maximizer The equivalet lgarithmic maximizati prblem is give by ŝ T,MAP arg mi s T ks T,T sk lg p s (s T ) (8) By usig the pdf factrizati prvided by Therem, (8) is further simplified t ŝ T,MAP arg mi ks T,T sk s T m X T k lg p u u T [k] u T [k i] i lg p s s T [ ],,s T [], (9) where each u T [i] is prvided by the liear cmbiati f the elemets i s T fud i (7) If we dete the term lg p u u T [k] u T [k i] by ŝ T,MAP arg mi s T i T u T [k],,u T [k + ], the MAP estimati becmes equivalet t the miimizati prblem ks T,T sk + m X T k T u T [k],,u T [k + ] lg p s s T [ ],,s T [], () where The iterestig aspect is that the MAP estimatr has the same frm as () where the sparsity-prmtig term T i the cst fucti is determied by bth L ad the distributi f the ivati The well-kw ad successful TV regularizer crrespds t the special case where T ( ) is the uivariate fucti ad the FI filter d T [ ] is the fiitedifferece peratr I Appedix E, we shw the existece f a ivati prcess fr which the MAP estimati cicides with the TV regularizati C MAP vs MMSE T have a rugh cmparis f MAP ad MMSE, it is beeficial t refrmulate the MMSE estimatr i (5) as a variatial prblem similar t (), thereby, expressig the MMSE sluti as the miimizer f a cst fucti that csists f a quadratic term ad a sparsity-prmtig pealty term I fact, fr sparse prirs, it is shw i [] that the miimizer f a cst fucti ivlvig the `-rm pealty term apprximates the MMSE estimatr mre accurately tha the cmmly csidered MAP estimatr Here, we prpse a differet iterpretati withut gig it techical details Frm (5), it is clear that ŝ MMSE s + r lg p s ŝ MMSE b s, () where b s r lg p s s We ca check that ŝ MMSE i () sets the gradiet f the cst J(s) ks sk lg p s (s b s ) t zer It suggests that ŝ MMSE arg mi s ks sk lg p s (s b s ) () which is similar t (8) The latter result is ly valid whe the cst fucti has a uique miimizer Similarly t [], it is pssible t shw that, uder sme mild cditis, this cstrait is fulfilled Nevertheless, fr the qualitative cmparis f MAP ad MMSE, we ly fcus the lcal prperties f the cst fuctis that are ivlved The mai disticti betwee the cst fuctis i () ad (8) is the required pdf Fr MAP, we eed p s, which was shw t be factrizable by the itrducti f geeralized fiite differeces Fr MMSE, we require p s ecall that p s is the result f cvlvig p s with a Gaussia pdf Thus, irrespective f the disctiuities f p s, the fucti p s is smth Hwever, the latter is lger separable, which cmplicates the miimizati task The ther differece is the ffset term b s i the MMSE cst fucti Fr heavy-tail ivatis such as -stables, the cvluti by the Gaussia pdf f the ise des t greatly affect p s I such cases, p s ca be apprximated by p s fairly well, idicatig that the MAP estimatr suffers frm a bias (b s ) The effect f cvlvig p s with a Gaussia pdf becmes mre evidet as p s decays faster I the extreme case where p s is Gaussia, p s is als Gaussia (cvluti f tw Gaussias) with a differet mea (which itrduces ather type f bias) The fact that MAP ad MMSE estimatrs are equivalet fr Gaussia ivatis idicates that the tw biases act i ppsite directis ad cacel ut each ther I summary, fr super-expetially decayig ivatis, MAP seems t be csistet with MMSE Fr heavy-tail ivatis, hwever, the MAP estimatr is a biased versi f the MMSE, where the effect f the bias is bservable at high ise pwers The sceari i which MAP diverges mst frm MMSE might be the expetially decayig ivatis, where we have bth a mismatch ad a bias i the cst fucti, as will be cfirmed i the experimetal part f the paper V EXAMPLE: LÉVY POCESS T demstrate the results ad implicatis f the thery, we csider Lévy prcesses as special cases f the mdel i Figure Lévy prcesses are rughly defied as prcesses with statiary ad idepedet icremets that start frm zer The prcesses are cmpatible with ur mdel by settig L d dx (ie, ad r ), L x, r h(x, ) [,[(x ) [,[( ) which impses the budary cditi s() w( )d The crrespdig discrete FI filter has the tw taps d T [] ad d T [] The impulse respse f L d,t L is give by, L,T (x) u(x) u(x T ) A MMSE Iterplati apple x<t, therwise () A classical prblem is t iterplate the sigal usig iseless samples This crrespds t the estimati f s T [ ] where T - ( T des t divide ) by assumig s[i] s T [i T ] ( apple i apple m) Althugh there is ise i this sceari, we ca still emply the MMSE criteri t estimate s T [ ] We shw that the MMSE iterplatr f a Lévy prcess is the simple liear iterplatr, irrespective f the type f
8 ivati T prve this, we assume l T < <(l + ) T ad rewrite s T [ ] as s T [ ] s T [l T ]+ This eables us t cmpute X kl T + s T [k] s T [k {z ] () } u T [k] ŝ T [ ] E s T [ ] {s T [i T ]} m i X s T [l T ]+E u T [k] {s T [k T ]} i (5) kl T + Sice the mappig frm the set {s T [i T ]} i t {s T []} [ {u [i] s T [(i + ) T ] s T [i T ]} i is e t e, the tw sets ca be used iterchageably fr evaluatig the cditial expectati Thus, ŝ T [ ] s T [l T ]+ X kl T + E u T [k] {s T []}[{u [i]} i (6) Accrdig t Lemma, u T [k] (fr k > ) is idepedet f s T [] ad u T [k ], where k 6 k By rewritig u [i] as P (i+)t k i T + u T [k ], we cclude that u [i] is idepedet f u T [k] uless i T +apple k apple (i + ) T Hece, ŝ T [ ] s T [l T ]+ X kl T + E u T [k] u [l + ] (7) Sice u [l + ] P (l+) T il T + u T [i] ad {u T [i]} i is a sequece f iid radm variables, the expected mea f u T [i] cditied u [l + ] is the same fr all i with l T +apple i apple (l + ) T, which yields E u T [k] u [l + ] T (l+) X T By applyig (8) t (7), we btai il T + E u T [i] u [l + ] (l+)t X E u T [i] u [l + ] T il T + u [l + ] (8) T ŝ T [ ] s T [l T ]+ l T T u [l + ] ( ) s T [l T ]+ s T [(l + ) T ], (9) where lt T Obviusly, (9) idicates a liear iterplati betwee the samples B MAP Deisig Sice, the fiite differeces u T [i] are idepedet Therefre, the cditial prbabilities ivlved i Therem ca be replaced with the simple pdf values p s s T [k],,s T [] p s s T [] ky p u u T [ ] () I additi, sice f w (), frm Therem, we have Iw, L,T (!) Tf w(!) p u u T [ ] F! e Tfw(!) u T [ ] () I the case f impulsive Piss ivatis, as shw i (), the pdf f u T [i] has a sigle mass prbability at x Hece, the MAP estimatr will chse u T [i] fr all i, resultig i a cstat sigal I ther wrds, accrdig t the MAP criteri ad due t the budary cditi s(), the ptimal estimate is thig but the trivial all-zer fucti Fr the ther types f ivatis where the pdf f the icremets u T [i] is buded, r, equivaletly, whe the Lévy desity v( ) is sigular at the rigi [], e ca refrmulate the MAP estimati i the frm f () as ŝ T [] ad m T X m ŝ T [k] arg mi s[i] s k T [i T ] s T [k] i + T s T [] + m X T k T s T [k] s T [k ],() where T ( ) lg p u ( ) ad Because shiftig the fucti T with a fixed additive scalar des t chage the miimizer f (), we ca mdify the fucti t pass thrugh the rigi (ie, T () ) After havig applied this mdificati, the fucti T presets itself as shw i Figure fr varius ivati prcesses such as ) Gaussia ivati:, ad v(a), which implies p u (x) e x p () ) Laplace-type ivati:, v(a) e p e a a, which implies (see Appedix E) p u (x) r e e p e x () The Lévy prcess f this ivati is kw as the variace gamma prcess [8] ) Cauchy p ivati ( -stable with ):, v(a) e 8 a, which implies q 8e p u (x) e+8 x (5) The parameters f the abve ivatis are set such that they all lead t the same etrpy value lg( e) The egative lg-likelihds f the first tw ivati types resemble the ` ad ` regularizati terms Hwever, the curve f T fr the Cauchy ivati shws a cvex lg-type fucti
9 s [] pu s [] (x) T 8 pu s [] s [] pu s [] s [] pu s [m] x C MMSE Deisig As discussed i Secti IV-C, the MMSE estimatr, either i the expectati frm r as a miimizati prblem, is t separable with respect t the iputs This is usually a critical restricti i high dimesis Frtuately, due t the factrizati f the jit a priri distributi, we ca lift this restricti by emplyig the pwerful message-passig algrithm The methd csists f represetig the statistical depedecies betwee the parameters as a graph ad fidig the margial distributis by iteratively passig the estimated pdfs alg the edges [9] The trasmitted messages alg the edges are als kw as beliefs, which give rise t the alterative ame f belief prpagati I geeral, the margial distributis (beliefs) are ctiuus-dmai bjects Hece, fr cmputer simulatis we eed t discretize them I rder t defie a graphical mdel fr a give jit prbability distributi, we eed t defie tw types f des: variable des that represet the iput argumets fr the jit pdf ad factr des that prtray each e f the terms i the factrizati f the jit pdf The edges i the graph are draw ly betwee des f differet type ad idicate the ctributi f a iput argumet t a give factr Fr the Le vy prcess, we csider the jit cditial pdf p {st [k]}k {s [i]}i factrized as m p {st [k]}m k {s [i]}i Qm i G (s [i] st [i]; ) QmT k G s [] s [] ; pu (st [k] st [k ]) s [m] G s [] s [] ; G s [m] ] cditial prbability s [m] ; ise distributi 5 Fig The T ( ) lg pu ( ) fuctis at T fr Gaussia, Laplace, ad Cauchy distributis The parameters f each pdf are tued such that they all have the same etrpy ad the curves are shifted t efrce them t pass thrugh the rigi s [m prir distributi s [] 6 5 s [] ; s [] Gaussia Laplace Cauchy G s [], (6) where is a rmalizati cstat that depeds ly the isy measuremets ad G is the Gaussia fucti defied as x G x; p e (7) ps ({s [i]}m i ) Nte that, by defiiti, we have st [] Fr illustrati purpses, we csider the special case f pure deisig crrespdig t T We give i Figure the bipartite graph G (V, F, E) assciated t the jit pdf (6) The variable des V {,, m} depicted i the middle f the graph stad fr the iput argumets {st [k]}m k The factr des F {,, m} are placed at the right ad Fig Factr graph fr the MMSE deisig f a Le vy prcess There are m variable des (circles) ad m factr des (squares) left sides f the variable des depedig whether they represet the Gaussia factrs r the pu ( ) factrs, respectively The set f edges E {(i, a) V F } als idicates a participati f the variable des i the crrespdig factr des The message-passig algrithm csists f iitializig the des f the graph with prper D fuctis ad updatig these fuctis thrugh cmmuicatis ver the graph It is desired that we evetually btai the margial pdf p s [k] {s [i]}m i the kth variable de, which eables us t btai the mea The details f the messages set ver the edges ad updatig rules are give i [], [] VI S IMULATION ESULTS Fr the experimets, we csider the deisig f Le vy prcesses fr varius types f ivati, icludig thse itrduced i Secti II-A ad the Laplace-type ivati discussed i Appedix E Amg the heavy-tail -stable ivatis, we chse the Cauchy distributi crrespdig t The fur implemeted deisig methds are ) Liear miimum mea-square errr (LMMSE) methd r quadratic regularizati (als kw as smthig splie []) defied as m X arg mi ks s k` + s[i] s[i ], (8) s[i] i where shuld be ptimized Fr fidig the ptimum fr give ivati statistics ad a give additive-ise variace, we search fr the best fr each realizati by cmparig the results with the racle estimatr prvided by the iseless sigal The, we average ver a umber f realizatis t btai a uified ad realizatiidepedet value This prcedure is repeated each time the statistics (either the ivati r the additive ise) chage Fr Gaussia prcesses, the LMMSE methd cicides with bth the MAP ad MMSE estimatrs ) Ttal-variati regularizati represeted as m X arg mi ks s k` + s[i] s[i ], (9) s[i] i where shuld be ptimized The ptimizati prcess fr is similar t the e explaied fr the LMMSE methd
9 8 7 MMSE LMMSE MAP Lg TV 9 8 MMSE Lg TV LMMSE 6 7 SN [db] 5 SN [db] 6 5 AWGN AWGN Fig 5 SN imprvemet vs variace f the additive ise fr Gaussia ivatis The deisig methds are: MMSE estimatr (which is equivalet t MAP ad LMMSE estimatrs here), Lg regularizati, ad TV regularizati Fig 6 SN imprvemet vs variace f the additive ise fr Gaussia cmpud Piss ivatis The deisig methds are: MMSE estimatr, Lg regularizati, TV regularizati, ad LMMSE estimatr ) Lgarithmic (Lg) regularizati described by arg mi ks sk ` + s[i] mx i lg + (s[i] s[i ]), (5) where shuld be ptimized The ptimizati prcess is similar t the e explaied fr the LMMSE methd I ur experimets, we keep fixed thrughut the miimizati steps (eg, i the gradietdescet iteratis) Ufrtuately, Lg is t ecessarily cvex, which might result i a cvex cst fucti Hece, it is pssible that gradiet-descet methds get trapped i lcal miima rather tha the desired glbal miimum Fr heavy-tail ivatis (eg, -stables), the Lg regularizer is either the exact, r a very gd apprximati f, the MAP estimatr ) Miimum mea-square errr deiser which is implemeted usig the message-passig techique discussed i Secti V-C The experimets are cducted i MATLAB We have develped a graphical user iterface that facilitates the prcedures f geeratig samples f the stchastic prcess ad deisig them usig MMSE r the variatial techiques We shw i Figure 5 the SN imprvemet f a Gaussia prcess after deisig by the fur methds Sice the LMMSE ad MMSE methds are equivalet i the Gaussia case, ly the MMSE curve btaied frm the messagepassig algrithm is pltted As expected, the MMSE methd utperfrms the TV ad Lg regularizati techiques The cuter ituitive bservati is that Lg, which icludes a cvex pealty fucti, perfrms better tha TV Ather advatage f the Lg regularizer is that it is differetiable ad quadratic arud the rigi A similar sceari is repeated i Figure 6 fr the cmpud- Piss ivati with 6 ad Gaussia amplitudes (zer-mea ad ) As metied i Secti V-B, sice the pdf f the icremets ctais a mass prbability at x, the MAP estimatr selects the all-zer sigal as the mst prbable chice I Figure 6, this trivial estimatr is excluded SN [db] 5 5 5 5 MMSE Lg MAP TV LMMSE AWGN Fig 7 SN imprvemet vs variace f the additive ise fr Cauchy ( stable with ) ivatis The deisig methds are: MMSE estimatr, Lg regularizati (which is equivalet t MAP here), TV regularizati, ad LMMSE estimatr frm the cmparis It ca be bserved that the perfrmace f the MMSE deiser, which is csidered t be the gld stadard, is very clse t that f the TV regularizati methd at lw ise pwers where the surce sparsity dictates the structure This is csistet with what was predicted i [] Meawhile, it perfrms almst as well as the LMMSE methd at large ise pwers There, the additive Gaussia ise is the dmiat term ad the statistics f the isy sigal is mstly determied by the Gaussia cstituet, which is matched t the LMMSE methd Excludig the MMSE methd, e f the ther three utperfrms ather e fr the etire rage f ise Heavy-tail distributis such as -stables prduce sparse r cmpressible sequeces With high prbability, their realizatis csist f a few large peaks ad may isigificat samples Sice the cvluti f a heavy-tail pdf with a Gaussia pdf is still heavy-tail, the isy sigal lks sparse eve at large ise pwers The pr perfrmace f the LMMSE methd bserved i Figure 7 fr Cauchy distributis cfirms this characteristic The pdf f the Cauchy distributi, give by (+x ), is i fact the symmetric -stable distributi with
SN [db] 7 6 5 MMSE Lg TV MAP LMMSE Lg regularizati appraches Simulati results shwed that we ca almst achieve the MMSE perfrmace by tuig the regularizati techique t the type f ivati ad the pwer f the ise We have als develped a graphical user iterface i MATLAB which geerates realizatis f stchastic prcesses with varius types f ivati ad allws the user t apply either the MMSE r variatial methds t deise the samples AWGN Fig 8 SN imprvemet vs variace f the additive ise fr Laplace-type ivatis The deisig methds are: MMSE estimatr, Lg regularizati, TV regularizati (which is equivalet t MAP here), ad LMMSE estimatr The Lg regularizer crrespds t the MAP estimatr f this distributi while there is direct lik betwee the TV regularizer ad the MAP r MMSE criteria The SN imprvemet curves i Figure 7 idicate that the MMSE ad Lg (MAP) deisers fr this sparse prcess perfrm similarly (specially at small ise pwers) ad utperfrm the crrespdig `-rm regularizer (TV) I the fial sceari, we csider ivatis with ad v(a) e a a This results i fiite differeces btaied at T that fllw a Laplace distributi (see Appedix E) Sice the MAP deiser fr this prcess cicides with TV regularizati, smetimes the Laplace distributi has bee csidered t be a sparse prir Hwever, it is prved i [], [] that the realizatis f a sequece with Laplace prir are t cmpressible, almst surely The curves preseted i Figure 8 shw that TV is a gd apprximati f the MMSE methd ly i light-ise cditis Fr mderate t large ise, the LMMSE methd is better tha TV VII CONCLUSION I this paper, we studied ctiuus-time stchastic prcesses where the prcess is defied by applyig a liear peratr a white ivati prcess Fr specific types f ivati, the prcedure results i sparse prcesses We derived a factrizati f the jit psterir distributi fr the isy samples f the brad family ruled by fixed-cefficiet stchastic differetial equatis The factrizati allws us t efficietly apply statistical estimati tls A csequece f ur pdf factrizati is that it gives us access t the MMSE estimatr It ca the be used as a gld stadard fr evaluatig the perfrmace f regularizati techiques This eables us t replace the MMSE methd with a mre-tractable ad cmputatially efficiet regularizati techique matched t the prblem withut cmprmisig the perfrmace We the fcused Lévy prcesses as a special case fr which we studied the deisig ad iterplati prblems usig MAP ad MMSE methds We als cmpared these methds with the ppular regularizati techiques fr the recvery f sparse sigals, icludig the ` rm (eg, TV regularizer) ad the APPENDIX A CHAACTEISTIC FOMS I Gelfad s thery f geeralized radm prcesses, the prcess is defied thrugh its ier prduct with a space f test fuctis, rather tha pit values Fr a radm prcess w ad a arbitrary test fucti ' chse frm a give space, the characteristic frm is defied as the characteristic fucti f the radm variable X hw, 'i ad give by ˆ P w (') E e jhw,'i p X (x)e jx dx F x p X (x) () (5) As a example, let w G be a rmalized white Gaussia ise ad let ' be a arbitrary fucti i L () It is well-kw that hw G,'i is a zer-mea Gaussia radm variable with variace k'k L Thus, i this example we have that ˆ P wg (') e k'k L (5) A iterestig prperty f the characteristic frms is that they help determie the jit prbability desity fuctis fr arbitrary fiite dimesis as ˆ P w kx i! i ' i E e jhw,p k i!i'ii E e j P k i!ixi F x p X (x) (!,,! k ), (5) where {' i } i are test fuctis ad {! i } are scalars Equati (5) shws that a iverse Furier trasfrm f the characteristic frm ca yield the desired pdf Beside jit distributis, characteristic frms are useful fr geeratig mmets t: @ + + k ˆ @! @! k k P w kx! i' i!! k i ( j) + + k x x kpx(x,,xk)dx dxk k k ( j) + + k E x x k k (5) Nte that the defiiti f radm prcesses thrugh characteristic frms icludes the classical defiiti based the pit values by chsig Diracs as the test fuctis (if pssible) Except fr the stable prcesses, it is usually hard t fid the distributis f liear trasfrmatis f a prcess Hwever, there exists a simple relati betwee the characteristic frms: The GUI is available at http://bigwwwepflch/amii/matlab cdes/ SSS GUIzip
Let w be a radm prcess ad defie Lw, where L is a liear peratr Als dete the adjit f L by L The, e ca write ˆ P (') E e jhlw,'i E e jhw,l 'i ˆ P w (L ') (55) Nw it is easy t extract the prbability distributi f frm its characteristic frm APPENDIX B SPECIFICATION OF TH-ODE SHAPING KENELS T shw the existece f Q a kerel h fr the th-rder differetial peratr L i (D r ii), we defie 8 < e ri(x ) [,[(x ) h i (x, ) : [,[( x i ), <r i, (56) e ri(x ) [,[(x ), <r i <, where x i are psitive fixed real umbers It is t hard t check that h i satisfies the cditis (i)-(iii) fr the peratr L i D r i I Next, we cmbie h i t frm a prper kerel fr L as h(x, ) Y Y h i ( i+, i ) d i (57) +x i By relyig the fact that the h i satisfy cditis (i)-(iii), it is pssible t prve by iducti that h als satisfies (i)- (iii) Here, we ly prvide the mai idea fr prvig (i) We use the factrizati L L L ad sequetially apply every L i h The startig pit i yields Q i L h(x, ) (x ) h i( i+, i )d i h ( +, ) Y i i Y h i ( i+, i ) i d i (58) x Thus, L h(x, ) has the same frm as h with replaced by By ctiuig the same prcedure, we fially arrive at L h (x, t), which is equal t (x ) APPENDIX C POOF OF THEOEM Fr the sake f simplicity i the tatis, fr defie u T [ ] u T [ ] 7 u[ ] 6 u T [] s T [ ] s T [ ] s T [] we (59) 7 5 Sice the u T [i] are liear cmbiatis f s T [i], the ( + ) vectr u[ ] ca be liearly expressed i terms f s T [i] as u[ ] D ( +) ( +) 6 s T [ ] s T [ ] s T [] 7 5, (6) where D ( +) ( +) is a upper-triagular matrix defied by d T [] d T [] d T [] d T [] d T [ ] d T [] 6 d T [] d T [] 7 5 ( + ) I (6) Sice d T [] 6, e f the diagal elemets f the upper-triagular matrix D ( +) ( +) is zer Thus, the matrix is ivertible because det D ( +) ( +) (d T []) + Therefre, we have that p s,u u[ ] p s s T [ ],,s T [] d T [] + (6) A direct csequece f Lemma is that, fr, we btai p s,u u T [ ] u[ ] p u u T [ ] u T [ i] which, i cjucti with Bayes rule, yields p s,u u[ ] p s,u u[ ] p s,u u T [ ] u[ ] p u u T [ ] u T [ i] i (6) (6) i By multiplyig equatis f the frm (6) fr,,k, we get p s,u u[k] p s,u u[ ] ky p u u T [ ] u T [ i] i (65) It is w easy t cmplete the prf by substitutig the umeratr ad demiatr f the left-had side i (65) by the equivalet frms suggested by (6) APPENDIX D POOF OF THEOEM As develped i Appedix A, the characteristic frm ca be used t geerate the jit prbability desity fuctis T use (5), we eed t represet u T [i] as ier-prducts with the white prcess This is already available frm (9) This yields p u {u T [ i]} k i F ˆ P k {! i } Pw i!i L,T ( T it ) {u T [ i]} k i (66)
Frm (), we have kx Pˆ w! i L,T ( T it ) i P k e fw i!i L,T ( T it x) dx P k e fw i!i L,T (x it ) dx (67) Usig (), it is w easy t verify that Pˆ P w i!i L,T ( T it ) ei w, (! L,T,,! ), Pˆ P w i!i L,T ( T it ) ei w, (,! L,T,,! ) (68) The ly part left t meti befre cmpletig the prf is that p u u T [ i] i p u u T [ ] u T [ i] (69) i p u u T [ i] i APPENDIX E WHEN DOES TV EGULAIATION MEET MAP? The TV-regularizati techique is e f the successful methds i deisig Sice the TV pealty is separable with respect t first-rder fiite differeces, its iterpretati as a MAP estimatr is valid ly fr a Lévy prcess Mrever, the MAP estimatr f a Lévy prcess cicides with TV regularizati ly if T (x) lg p u (x) x +, where ad are cstats such that > This cditi implies that p u is the Laplace pdf p u (x) e x This pdf is a valid distributi fr the first-rder fiite differeces f the Lévy prcess characterized by the ivati with µ ad v(a) e a a because f l (!) e j!a e a da a cs(!a) e a da (7) a Thus, we ca write d d! f l(!) si(!a)e a da! +! (7) By itegratig (7), we btai that f l (!) lg( +! )+, where is a cstat The key pit i fidig this cstat is the fact that f l (), which results i f l (!) lg +! Nw, fr the samplig perid T, Equati () suggests that p u (x) F! e Tfl(!) (x) F! ( T ) +! (x) x T K T ( x ) p, (7) T (T ) where K t ( ) is the mdified Bessel fucti f the secd kid The latter prbability desity fucti is kw as symmetric variace-gamma r symm-gamma It is t hard t check that we btai the desired Laplace distributi fr T (x) lg p d (x) 8 6 T 5 T T T 5 5 Fig 9 The fucti T lg p u fr differet values f T after efrcig the curves t pass thrugh the rigi by applyig a shift Fr T, the desity fucti p u fllws a Laplace law Therefre, the crrespdig T is the abslute-value fucti T Hwever, this value f T is the ly e fr which we bserve this prperty Shuld the samplig grid becme fier r carser, the MAP estimatr wuld lger cicide with TV regularizati We shw i Figure 9 the shifted T fuctis fr varius T values fr the afremetied ivati where x EFEENCES [] E Cadès, J mberg, ad T Ta, bust ucertaity priciples: Exact sigal recstructi frm highly icmplete frequecy ifrmati, IEEE Tras Ifrm Thery, vl 5,, pp 89 59, Feb 6 [] E Cadès ad T Ta, Near ptimal sigal recvery frm radm prjectis: Uiversal ecdig strategies, IEEE Tras Ifrm Thery, vl 5,, pp 56 55, Dec 6 [] D L Dh, Cmpressed sesig, IEEE Tras Ifrm Thery, vl 5,, pp 89 6, Apr 6 [] J L Starck, M Elad, ad D L Dh, Image decmpsiti via the cmbiati f sparse represetatis ad a variatial apprach, IEEE Tras Image Prc, vl,, pp 57 58, Oct 5 [5] S J Wright, D Nwak, ad M A T Figueired, Sparse recstructi by separable apprximati, IEEE Tras Sig Prc, vl 57, 7, pp 79 9, Jul 9 [6] E Cadès ad T Ta, Decdig by liear prgrammig, IEEE Tras Ifrm Thery, vl 5,, pp 5, Dec 5 [7] L udi, S Osher, ad E Fatemi, Nliear ttal variati based ise remval algrithms, Phys D: Nli Phem, vl 6, pp 59 68, Nv 99 [8] D P Wipf ad B D a, Sparse Baysia learig fr basis selecti, IEEE Tras Sig Prc, vl 5, 8, pp 5 6, Aug [9] T Park ad G Casella, The Bayesia LASSO, J f the Amer Stat Assc, vl, pp 68 686, Ju 8 [] V Cevher, Learig with cmpressible prirs, i Prc Neural Ifrmati Prcessig Systems (NIPS), Vacuver BC, Caada, Dec 8, 8 [] A Amii, M User, ad F Marvasti, Cmpressibility f determiistic ad radm ifiite sequeces, IEEE Tras Sig Prc, vl 59,, pp 59 5, Nv [] Gribval, V Cevher, ad M Davies, Cmpressible distributis fr high-dimesial statistics, IEEE Tras Ifrm The, vl 58, 8, pp 56 5, Aug [] Gribval, Shuld pealized least squares regressi be iterpreted as maximum a psteriri estimati? IEEE Tras Sig Prc, vl 59, 5, pp 5, May [] M User ad P Tafti, Stchastic mdels fr sparse ad piecewisesmth sigals, IEEE Tras Sig Prc, vl 59,, pp 989 6, Mar [5] M Vetterli, P Marzilia, ad T Blu, Samplig sigals with fiite rate f ivati, IEEE Tras Sig Prc, vl 5, 6, pp 7 8, Ju