Sparse stochastic processes

Transcription

1 Sparse stochastic processes Part II: Functional characterization of sparse stochastic processes Prof. Michael Unser, LIB Ecole CIMPA, Analyse et Probabilité, Abidjan 204 oad map for theory of sparse processes 2 Specification of inverse operator Functional analysis solution of SDE 3 Characterization of generalized stochastic process White noise w L Mixing operator Whitening operator L s =L w Very easy! (after solving. & 2.) Multi-scale wavelet analysis Characterization of continuous-domain white noise Higher mathematics: generalized functions (Schwartz) measures on topological vector spaces Gelfand s theory of generalized stochastic processes Infinite divisibility (Lévy-Khintchine formula) 4 Characterization of transform-domain statistics Easy when: i =L i 2

2 CONTENT! 4.4 Specification of white noise (innovation)! 7.2 General abstract characterization! 7.3 Non-Gaussian stationary processes! 7.4 Lévy processes and extension Mixed processes! 7.5 Self-similar processes Stable fractal processes fbm through the looking glass Scale-invariant Poisson processes 3 Generalized innovation process Difficulty : w 6= w(x) is too rough to have a pointwise interpretation Difficulty 2: w is an infinite-dimensional random entity; its pdf can be formally specified by a measure P w (E) where E S 0 ( d ) Axiomatic definition (Gelfand-Vilenkin 964) w is a generalized innovation process (or continuous-domain white noise) over S 0 ( d ) if. Observability : X = h',wi is a well-defined random variable for any test function ' 2 S( d ). 2. Stationarity : X x0 = h'( x 0 ),wi is identically distributed for all x 0 2 d. 3. Independent atoms : X = h',wi and X 2 = h' 2,wi are independent whenever ' and ' 2 have non-intersecting support. X = h, i X 2 = h, i Characteristic functional (!! ' ) dp w (') =E{e jh',wi } = e jh',gi P w (dg) S 0 4

3 Sparser Infinite divisibility and Lévy exponents Definition: A random variable X with generic pdf p id (x) is infinitely divisible (id) iff., for any N 2 +, there exist i.i.d. random variables X,...,X N such that X d = X + +X N. Criterion for infinite divisibility (necessary and sufficient) f(!) = log bp id (!) = e j!x p id (x)dx is a valid Lévy exponent Construction from infinitesimal atoms: white noise analysis i.i.d. Proposition X = hw, recti = h, i = h, i + + h, i w is a generalized innovation process, X = hw, recti is infinitely divisible. N N (Amini-U., IEEE-IT 204) Bottom line One-to-one correspondence between infinitely-divisible distributions and Lévy exponents and, by extension, innovation processes. 5 Examples of infinitely divisible laws p id (x) (a) Gaussian f(!) = 2 0 2! (b) Laplace f( ) = log +! (c) Compound Poisson f(!) = (ejx! )p A (x)dx (d) Cauchy (stable) f(!) = s 0! Characteristic function: bp id (!) = p id (x)e j!x dx =e f(!) 6

4 Specification of innovations in S 0 ( d ) Definition The function f :! C is a valid Lévy-Schwartz exponent iff. ˆp X (!) =e f(!) is the characteristic function of an infinitely divisible random variable X id with the property that E{ X id } < for some > 0. For instance: =2if f 00 (0) = 2 w < (finite variance scenario) Theorem (Fageot-Amini-U., arxiv) Let f be a valid Lévy-Schwartz exponent. Then, the characteristic functional dp w (') =exp f '(x) dx = E{e jh',wi } d is continuous, positive-definite S( d )! C with P d w (0) =. It uniquely specifies a generalized innovation process w over S 0 ( d ) (white Lévy noise) that satisfies the three required axiomatic properties. Equivalent statement of Lévy-Schwartz admissibility: E{ h',wi } < with > GENEAL ABSTACT CHAACTEIATION white noise w with Lévy exponent f L s =L w Whitening operator L Generalized innovation model s =L w, 8' 2 S, h',si = h', L wi = hl ' {z },wi ) P c s (') =E{e jhs,'i } = P d w (L ')=exp f L '(r) dr d Mathematical conditions on L and f? 8

5 Sufficient conditions for existence Theorem (see Chap 4) (U.-Tafti-Sun, IEEE-IT 204) Let s =L w where w is an innovation process with Lévy exponent f. Then, s is a generalized process over S 0 ( d ) with characteristic form E{e jh',si } = P c s (') = exp f L '(r) dr d provided that any of the following conditions is met. Condition S: L is a continuous operator S( d )! S( d ) and f is Lévy-Schwartz admissible; Condition : L is a continuous operator S( d )! ( d ) and f is Lévy-Schwartz admissible; Condition L p : L is a continuous operator S( d )! L p ( d ) and f is p-admissible for some p 2 [, 2]. The couple (L,f) is then said to be admissible. Lévy-Schwartz admissibility: E{ h',wi } < with > 0 p-admissiblility: f(!) +! f 0 (!) apple C! p 9 Hierarchy of spaces S( d ) ( d ) L p ( d ) S 0 ( d ) for any p>0 ( d ): space of rapidly decaying functions ( d )= \ 0 L, ( d )= \ n2 + L,n ( d ) (f i ) converges in ( d ), (f i ) converges in all L, ( d ) spaces Family of semi-norms: k'k n =ess sup x2 d x n '(x) for all n 2 N d. 0

6 Correlation form B s (', ' 2 )=E{hs, ' i hs, ' 2 i} = ' (r ) s (r, r 2 )' 2 (r 2 )dr dr 2 d d c Ps (! ' +! 2 ' 2! =0,! 2 =0 (by Kernel theorem) Second order assumption: f 0 (0) = 0 and f 00 (0) = w 2 < Special case of white noise: B w (', ' 2 )= w 2 h', ' 2 i ) B s (', ' 2 )=B L w(', ' 2 )=B w (L ', L ' 2 ) = whl 2 L ', ' 2 i, Autocorrelation function s (r, r 2 )=E{s(r ) s(r 2 )} = B s ( ( r ), ( r 2 )) = 2 wl L { ( r )}(r 2 ) Operators: fundamental invariance properties Definition An operator T is shift-invariant iff., for any function ' in its domain and any r 0 2 d, T{'( r 0 )}(r) =T{'}(r r 0 ). Definition An operator T is scale-invariant of order iff., for any function ' in its domain, T{'}(r/a) = a T{'( /a)}(r), where a 2 + is the dilation factor. T{'(a )}(r) = a T{'}(ar) Definition An operator T is scalarly rotation-invariant iff., for any function ' in its domain, T{'}( T r)=t{'( T )}(r), where is any orthogonal matrix in d d. 2

7 Fractional-order operators operators Liouville s fractional derivative j!r d! D '(r) = (j!) ˆ'(!)e 2 Proposition [U.-Blu, 2007] The complete family of -D scale-invariant convolution operators of order reduces to the fractional whose Fourier-based definition '(r) = (j!) 2 + ( j!) 2 j!r d! ˆ'(!)e 2 2 Order of differentiation: Phase factor: , for 0, (, +) and, 0 2 Special cases: D /2, H 0 (fractional Hilbert transform) 3 Fractional derivatives and generalized functions Theorem The differential is continuous from S() to L p () for > p. For ' 2 ' remains infinitely differentiable, but its decay is constrained '(r) apple Const + r +. Impulse response: D { }(r) =F{(j!) } (r) = r +, for /2 ( ) Generalized function h', r + ( +) (tempered singularity at r =0for < 0) r + ( + ) i = h', Dn { = hd n ', = ( ) n ( + n + ) r +n + ( + n + ) }i r +n + ( + n + ) i 0 r +n ' (n) (r)dr. 4

8 Invariance properties: definitions for any ' 2 S( d ) Translation of 2 S 0 ( d ) by r 0 : h', ( r 0 )i = h'( + r 0 ), i A generalized stochastic process s is stationary if it has the same probability laws as its translated version s( r 0 ) for any r 0 2 d., Ps c (') = P c s '( + r 0 ) Affine transformation of 2 S 0 ( d ): h', (T )i = det(t) h'(t ), i A generalized stochastic process s is isotropic if it has the same probability laws as its rotated version s( T ) for any (d d) rotation matrix., Ps c (') = P c s '( ) A generalized stochastic process s is self-similar of scaling order H if it has the same probability laws as any of its scaled and renormalized version a H s( /a)., c Ps (') = c P s a H+d '(a ) Duality relation: h',a H s( /ai = ha H+d s(a ), i H: Hurst exponent 5 2nd-order invariance properties A generalized stochastic process s is wide-sense stationary (WSS) if E{h',si} = E{h'( + r 0 ),si} B s ', ' 2 = B s ' ( + r 0 ), ' 2 ( + r 0 ) for any ', ', ' 2 2 S( d ) and any r 0 2 d., E{s(r)} = Const. and E{s(r )s(r 2 )} = s (r r 2 ) If, in addition, s (r r 2 )= s ( r r 2 ), then the process is WSS isotropic. A generalized stochastic process s (with zero mean) is wide-sense self-similar with scaling order H if it has the same second-order moments as a H s( /) with a>0., B s ', ' 2 = a 2H+2d B s ' (a ), ' 2 (a ), a 2H s r a, r 2 a = s (r, r 2 ) Note: self-similarity is incompatible with stationarity! 6

9 Invariance properties of innovation model Theorem The high-level statistical properties of s = L w are tightly linked to the invariance properties of L (or, equivalently, L ) described by its generalized impulse response h(, r 0 )=L { ( r 0 )} 2 S 0 ( d d ).. If L is linear shift-invariant, then s is stationary and h(r, r 0 )=h(r r 0, 0) = L (r r 0 ) where L =L { } is the Green s function of L. 2. If L is translation- and rotation-invariant, then s is stationary isotropic and h(r, r 0 )= L ( r r 0 ) where L ( r ) =L { }(r) is a purely radial function. 3. If L is scale-invariant of order ( ) and w 2 = f 00 (0) <, then s is wide-sense self-similar with Hurst exponent H = d/2. 4. If L is scale-invariant of order ( ) and f is homogeneous of degree 0 < apple 2, then s is self-similar with Hurst exponent H = d + d/. 7 Invariance properties: Sketch of proof White noise (or innovation) processes is stationary (by change of variable) log P c w '( + r 0 ) = f '( + r 0 )}(r) dr = f '(x) dx d d Scale-invariance of characterization functional: log P c s a H+d '( ) = f a H+d L {'(a )}(r) dr (by linearity of L ) d = f a H+d a L {'}(ar) dr (by scale invariance) d = a d f a H+d L {'}(x) dx (change of variable) d = a H+ d {z d } f L '(r) dr ( -homogeneity of f) = d Homogeneous Lévy exponents = -stable ones; symmetric members of the family: f (!; s 0 )= s 0!. Single finite-variance member: f 2 (!) = b 2! 2 (Gaussian) 8

10 Interpretation as conventional stochastic process white noise w with Lévy exponent f L s =L w L Generalized impulse response: h(r, r 0 )=L { ( r 0 )}(r) =L { ( r)}(r 0 ) Kernel theorem: X = h', L wi = hl ',wi = h h(r, )'(r)dr,wi d Proposition If h is an ordinary function of d d with h(r, ) 2 ( d ) resp., h(r, ) 2 L p ( d ) when f is p-admissible for r 2 d fixed, then s admits the pointwise representation s(r) =hh(r, ),wi. Link with classical formulation (stochastic integral) s(r) = h(r, r 0 )w(r 0 )dr 0 = h(r, r 0 )dw (r 0 ) d d where W (E) =h E,wi for any Borel set E d NON-GAUSSIAN STATIONAY POCESSES LSI filtering of white noise: s = h w with h 2 ( d ) resp., L ( d ) Adjoint map ' 7! h _ ' : S( d )! S( d ) resp., L p ( d ) h _ (r) =h( r) (reversal) kh _ 'k Lp = kh ' _ k Lp applekhk L k'k Lp Stochastic process s is stationary with characteristic form cp s (') =exp f (h _ ')(r) dr d and non-gaussian unless f(!) = 2 w 2! 2. 20

11 Differential operators with stable inverses First-order operator: P =D Id with e( ) 6= 0 8 < (r) =F j! (r) = e r [0,)(r) if e( ) < 0, : e r (,0](r) if e( ) > 0. Inverse operator: S()! S() P ' = ' Adjoint: P = P ) P ' = ' = _ ' Higher-order operators with e( n ) 6= 0and N>M P P N {s}(r) =q M (D){w}(r) Inverse operator L : S()! S() L : S()! S() L =P N P q M (D) L ' = L ' with L 2 () (exponential decay) 2 Autocorrelation and power spectrum h(r) = L (r) =F n bl(!) o (r) Proposition The autocorrelation function of the stationary stochastic process s(r) =( L w)(r), where w is a white Lévy noise with variance w 2, is given by s (r) =E{s( + r) s( )} = 2 w ( L _ L)(r) F! s(!) = 2 w b L(!) 2 Gaussian stochastic process (by Parseval) cp s (') =exp f (h _ ')(r) dr =exp d! ˆ'(!) 2 2 w d! d L(!) b 2 (2 ) 2 22

12 Generalized increment process u(r) =L d s(r) = P k2 d d L [k]s(r k) s =L w L d : discrete approximation of whitening operator Corollary Let s =L w be a generalized stochastic process (possibly non-stationary) where the whitening operator L is spline-admissible with generalized B-spline L 2 ( d ) such that L( r 0 )=L d L ( r 0 ) for all r 0 2 d. Then, the generalized increment process u =L d s is stationary with characteristic form cp u (') =exp f ( L _ ')(r) dr. In the case of a 2nd-order process, its autocorrelation function is E{u( + r) u( )} = 2 w( L _ L)(r). Proof: u =L d L w = L w LÉVY POCESSES AND EXTENSIONS Classical definition The stochastic process W = {W (t) :t 2 + } is a Lévy process if it fulfills the following requirements:. W (t) =0almost surely. 2. Given 0 apple t <t 2 <...<t n, the increments W (t 2 ) W (t ), W (t 3 ) W (t 2 ),...,W(t n ) W (t n ) are mutually independent. 3. For any given step T, the increment process T W (t), where T is the operator that associates W (t) to W (t) W (t T ), is stationary. Equivalent generalized process: solution of unstable SDE DW = Ẇ = w subject to boundary condition W (0) = 0 24

13 Stabilizing the anti-derivative operator D: scale-invariant operator with =... but the system is no longer BIBO stable (t) t Adjoint inverse operator (LSI): D '(t) = + t '( )d =( _ + ')(t) /2 L p () D (t) t D (t) = ( )d t Modified anti-derivative operators: I 0'(t) =D '(t) (D ')( ) _ +(t) I 0 (t) I 0 (t) = t 0 ( )d I 0 : continuous operator S()! () t I 0 : imposes vanishing boundary condition at t =0 25 Distributional characterization of Lévy process For all ' 2 S(), h',wi = h', I 0 wi = hi 0',wi I 0 : S()! () Stabilized anti-derivative operators I 0 '(t) = t '( )d I 0'(t) = 0 t + '( )d, h(t, ) =I 0 { ( )}(t) = + (t ) +( ) ) W (t) =h (0,t],wi Characteristic form cp W (') = P c w (I 0') =exp f(i 0'(r))dr ˆp W (t) (!) =E{e j!w (t) } = P c w (! (0,t] )=exp Increment process f! (0,t] ( ) d f(0) = 0 =e tf(!) u(t) =D d W (t) =W (t) W (t ) = h (t,t],wi ˆp u (!) =E{e j!u } = c P w (! (t,t] )=e f(!) ) {u(k)} k2 i.i.d. Infinitely divisible 26

14 From Brownian motion to Lévy flights (a): Gaussian Nobert Wiener (b): Laplace (c): Poisson (d): Cauchy Paul Lévy Stabilized inverses (generalization) Signal-domain formulas Pole: =j! 0 with! 0 2 Stabilized left inverse : I! 0 '(r) =( _ j! 0 ')(r) ˆ'(! 0 ) _ j! 0 (r) Stabilized right inverse : I!0 '(r) =( j!0 ')(r) e j! 0r ( j!0 ')(0) Theorem I! 0 is continuous from S() to () and extends continuously to a linear operator ()! (). It is the dual of I!0 in the sense that hi!0, 'i = h, I! 0 'i for all ', 2 S(). Moreover, I!0 '(0) = 0 (zero boundary condition) I! 0 (D j! 0 Id) ' = ' (left-inverse property) (D j! 0 Id)I!0 = (right-inverse property) 28

15 Higher-order extensions of Lévy processes K poles on imaginary axis: N K k =j! k, apple k apple K P =D Id Factorized Nth-order SDE: (P P N K ) (P j! P j!k ) s = q M (D) w Stabilized inversion s = (I!K I! ) (P N K P )q M (D)w = I (!K :! )T LSI w bt LSI (!) = Q N q M (j!) K n= (j! n) 8 >< >: s (0) = 0 P j!k s (0) = 0. P j!2 P j!k s (0) = 0, Characteristic form: Ps c (') =exp f T LSII (! :! K ) '(t) dt L =T LSI I (! :! K ) : S()! () 29 Examples: lowpass processes L=D L=D 2 30

16 Examples: bandpass processes L=(D Id)(D 2 Id) with =(j3 /4, j3 /4) L=(D Id)(D 2 Id) with =( j /2, 0.05 j /2) 3 Application: signal modeling (Audio) Sparse, bandpass processes L= dn dt n + a n dt n + + a d dt + a 0I d n (a) Gaussian (b) Alpha stable α=.2 Mixed sparse processes: s mix = s + + s M cp smix (') = MY m= cp sm (') =exp MX m= f m Lm '(t) dt! Gaussian (Am) generalized Lévy (Am, SαS) 32

17 7.5 SELF-SIMILA POCESSES Stable fractal processes L=( ) 2 F! k!k (scale-invariant of order ) f (!) = s 0! (homogeneous of order, p-admissible with p = ) fractional SDE: ( ) 2 s,h = w (S S noise) Stabilized right and left inverses (see Chapter 5) I and I : S( d )! L ( d ) ) s,h = I H+d d w Hurst exponent: H = d + d with 2 (0, 2] cp s,h (') = P c w (I H+d d ') for H 2 + \ + 33 (f) Brownian motion through the looking glass DW = w (unstable SDE!) D B H = w W =I 0 w, 8' 2 S, h',wi = hi 0',wi L 2 -stable anti-derivative: I 0'(t) = ˆ'(!) ˆ'(0) e j!t d! j! 2 Characteristic form of Brownian motion (a.k.a. Wiener process) cp W (') = exp 2 ki 0'k 2 L 2! ˆ'(!) ˆ'(0) 2 d! = exp (by Parseval) 2 j! 2 Stabilization non-stationary behavior Characteristic form of fractional Brownian motion! ˆ'(!) ˆ'(0) cp 2 d! BH (') = exp 2 ( j!) 2 with H = 2 2 (0, ) 34

18 Causal scale-invariant operators Definition Fractional derivatives of order 2, 6=, 2, 3,... D {'}(ø) = ˆ'(!)(j!) j!ø d! e 2º D {'}(ø) = ˆ'(!)( j!) j!ø d! e 2º Fractional integrators of order 2 ( p,+), + p 6=,2,3,... I 0,p {'}(ø) = e j!ø P b + p c (j!) k k=0 k! (j!) ˆ'(!) d! 2º I 0,p {'}(ø) = ˆ'(!) P b + p c k=0 ˆ' (k) (0)! k ( j!) k! e j!ø d! 2º Properties causal, LSI anti-causal, LSI I 0,p {'}(0) = 0 L p -stable Table 7. Causal scale-invariant operators and their adjoint. I 0,2 {'}( ) = =(D ˆ'(!) ˆ'(0) j! d! e ( j!) 2 + ) I 0{'}( ) ) B H =I 0 D H+/2 w 35 Stabilized fractional integrators Theorem The fractional integration = 2 Pb + p c k=0 b' (k) (0)! k k! b'(!) e j!r d! ( j!) 2 (j!) 2 + continuously maps S() into L p () for p>0, 2, and 2 +, subject to the restriction + p 6=, 2,... It is a linear operator that is scale-invariant and is a left inverse The adjoint is given '(r) = e j!r Pb + p c (j!r) k k=0 k! b'(!) d! 2 ( j!) 2 (j!) 2 + and is the proper scale-invariant right inverse functions. to be applied to generalized Compare with (I 0) n and I n 0 36

19 fbm as a stochastic integral Generalized impulse response L { ( )}(t) =L e j!t j! d! { ( t)}( ) =h H+/2,2 (t, ) = e ( j!) 2 = (t ) + ( ) ( ) + As functions of : h H+/2,2 (t, ) 2 L 2 () for t 2 (fixed) h,2 (t, ) =I 0{ ( t )}( ) = (0,t ]( ) fbm: Classical representation B H (t) =hh H+/2,2 (t, ),wi = (t ) + ( ) + dw ( ) ( ) = 0 (t ) ( ) dw ( )+ ( ) t 0 (t ) dw ( ) 37 Kernels of fractional integral operators h,p (,t) p =2, =0.8,.8 p =2, =, 2 p =2, =.2, 2.2 Proposition The kernel of the operator I 0,p is given by h,p (t, ) =I 0,p { ( t)}( ) = ( ) (t ) + Moreover, h,p (t, ) 2 L p () for any fixed value t = t and b +/pc X k=0 t k k! ( ) k + ( k). + p 2 + \N. Extended family of self-similar processes: s =I 0, w s w (see Chap 5) 38

20 Examples of self-similar processes L F! (j!) H+ 2 ) L : fractional integrator fbm; H = 0.50 Poisson; H = 0.50 fbm; H = 0.75 fbm; H =.25 fbm; H =.50 H=.5 H=.75 H=.25 H=.5 Poisson; H = 0.75 Poisson; H =.25 Poisson; H =.50 Gaussian Sparse (generalized Poisson) Fractional Brownian motion (Mandelbrot, 968) (U.-Tafti, IEEE-SP 200) 39 Scale-invariant Poisson processes Compound Poisson process X A k r 0 (r r k ) k2 Fractional extension X A k (r r k ) k2 + L=D Wide sense self-similar Poisson field Impulsive noise: w Poisson (r) = X k2 A k (r r k ) A k : i.i.d. with pdf p A cp wpoisson (') =exp d e ja'(r) p A (a) da dr ) s Poisson (r) =p 0 (r)+ X k2 A k L (r r k ) 40

21 Fractional integrators in multiple dimensions Theorem [Sun-U., 202] Let I p with p and > 0 be the isotropic fractional integral operator defined by Ip '(r) = d ˆ'(!) X k appleb + d p c k!k k ˆ'(0)! k k! e jh!,ri d! (2 ) d Then, under the condition that 6= 2, 4,... and + d p 6=, 2, 3,..., Ip is the unique left inverse of ( ) 2 that continuously maps S( d ) into L p ( d ) for p and is scaleinvariant. The adjoint operator I p, which is the proper scale-invariant right inverse of ( ) 2, is given by I p '(r) = d e jh!,ri X k appleb + d p c k!k d j k r k! k k! ˆ'(!) d! (2 ) d. 4 Self-similar and isotropic processes Stochastic partial differential equation : ( ) H+ 2 s(r) =w(r) Gaussian H=.5 H=.75 H=.25 H=.75 Sparse (generalized Poisson) (U.-Tafti, IEEE-SP 200) 42

22 Equations of a screen saver Mondrian 2 = X k A k (x x k ) ) s(x) =a 0 + X k A k (x x k ) 0 + where x k are Poisson distributed with rate and A k i.i.d. Gaussian with characteristic function ˆp A. Complete mathematical description (characteristic form) 8' 2 S( 2 ) (Schwartz s space of smooth and rapidly-decaying test functions): E{e jhs,'i } =exp ˆp A '(x 0,x 0 2)dx 0 dx 0 2 dx dx 2 x 2 with ˆp A (!) =e!2 2 x Gaussian L=D r D r2 F! (j! )(j! 2 ) L=( ) /2 F! k!k