A Unified Formulation of Gaussian Versus Sparse Stochastic Processes

Transcription

1 A Unified Formulation of Gaussian Versus Sparse Stochastic Processes Michael Unser, Pouya Tafti and Qiyu Sun EPFL, UCF Appears in IEEE Trans. on Information Theory, 2014 Presented by Liming Wang M. Unser, P. Tafti, Q. Sun (EPFL, UCF) Generalized Stochastic Processes 1 / 15

2 Motivations Most of current formulations of compressed sensing and sparse signal recovery are fundamentally deterministic. It is likely to achieve further progress by adopting statistical point of view for the description of sparse signal. It has been observed that certain class of processes admits a sparse representation in a matched wavelet-like basis where the Karhunen-Loève transform is sub-optimal. The treatment of the paper relies on the notion of Generalized Function and Generalized Stochastic Processes proposed by Gelfand and Vilenkin. M. Unser, P. Tafti, Q. Sun (EPFL, UCF) Generalized Stochastic Processes 2 / 15

3 Outline of the paper Motivated by the Gaussian AR model, the generalized stochastic processes and generalized white noise have been considered. The proposed innovation model can capture both non-gaussian stationary and non-stationary processes. It has been suggested that the a matched wavelet-like transform on the generalized stochastic process obtained by the innovation model would possess a sparse representation for the wavelet coefficients. M. Unser, P. Tafti, Q. Sun (EPFL, UCF) Generalized Stochastic Processes 3 / 15

4 Motivation: Gaussian vs. Non-Gaussian AR(1) Processes A continuous time Gaussian AR(1) process can be formulated as s α (t) = (ρ α w)(t), where ρ α (t) = e αt u(t) with u(t) being a unit step function and Re(α) < 0. w(t) is a Gaussian white noise. Equivalently, it can be interpreted as (D αid)s α (t) = w(t), where D := d dt, Id are the differential and identity operators (in distributional sense), respectively. M. Unser, P. Tafti, Q. Sun (EPFL, UCF) Generalized Stochastic Processes 4 / 15

5 M. Unser, P. Tafti, Q. Sun (EPFL, UCF) Generalized Stochastic Processes 5 / 15

6 Sparsity in Continuous-time Domain The sparsity in continuous-time domain may refer to two distinct notions (or mixing of two). A finite rate of innovation. (M. Vetterli, TSP, 2002) Large or even infinite rate of innovation, but with the property that a few innovations dominate the overall behavior. In this case, the histogram of observation is distinguished by its heavy tail. (M. Unser, TSP, 2011; R. Gribonval. TIT, 2012) M. Unser, P. Tafti, Q. Sun (EPFL, UCF) Generalized Stochastic Processes 6 / 15

7 Generalized Function Instead of defining a function in a traditional sense, the idea of generalized function is to first define a function by evaluating its effect on a collection of nice-behaved test functions S. Namely, manifest the function as a functional on S, which can be extended later to whole range of functions of interests by continuation argument. M. Unser, P. Tafti, Q. Sun (EPFL, UCF) Generalized Stochastic Processes 7 / 15

8 Generalized Function We first define the S, which is called the Schwartz s space. Recall that the L p norm of a function f (t) is defined as f p := ( R f (t) p dt) 1/p for 1 p < and f := ess sup t R f (t). We define the weighted L p,α with α > 0 as L p,α = {f L p : (1 + t α )f (t) p < } S := {φ C : D n φ,m <, m, n Z + }. We define a generalized function of tempered distribution as a f S, where S denotes the dual of S, i.e., f is a linear functional on S. For example, the Dirac function δ can be defined as δ(φ) := φ(0), or more commonly expressed in engineering as δ(t)φ(t)dt = φ(0). By the dense property of S, f then can be uniquely extended to enlarged topological space such as L p, which is called the generalized function (distribution). M. Unser, P. Tafti, Q. Sun (EPFL, UCF) Generalized Stochastic Processes 8 / 15

9 Generalized Stochastic Processes The idea of generalized stochastic processes proposed by Gelfand and Vilenkin is to replace the point measurements {s(t n )} by a series of scalar product {< s, φ n >} with test function φ n S. In other words, this translates into defining a generalized stochastic process as a linear random functional. The foundation of generalized stochastic processes is that one can deduce all the statistical information about the process from the knowledge of its characteristic form ˆP s (φ) = E[e j<s,φ> ]. Any finite dimensional version can be recovered by substituting φ = ω 1 φ ω n φ n and ˆP s (φ) can be viewed as the characteristic function of X 1 =< s, φ 1 >,..., X n =< s, φ n >. M. Unser, P. Tafti, Q. Sun (EPFL, UCF) Generalized Stochastic Processes 9 / 15

10 Generalized Stochastic Processes In fact, Galfand s theory rests upon the principle that spacifying an admissible functional ˆP s (φ) is equivalent to defining the underlying generalized stochastic process (Bochner-Minlos Theorem). Definition A complex-valued function f (ω) of real variable ω is called positive-definite iff N m=1 n=1 N f (ω m ω n )ξ m ξ m 0, ω 1,..., ω N R, ξ 1,..., ξ N C and N Z +. Bochner s Theorem states that a bounded, continuous function f : R C is a characteristic function of some R.V. iff f is positive definite, and f (0) = 1. M. Unser, P. Tafti, Q. Sun (EPFL, UCF) Generalized Stochastic Processes 10 / 15

11 Generalized Stochastic Processes Similarly, we can extend previous notion of positive-definiteness to the case where f is a functional and we can have the following Bochner-Minlos Theorem. Theorem (Bochner-Minlos) Given a functional ˆP s (φ) on a nuclear space X that is continuous, positive-definite and ˆP s (0) = 1, there exists a unique probability measure P s on dual space X such that ˆP s (φ) = E[e j<s,φ> ] = e j<s,φ> dp s (s) X M. Unser, P. Tafti, Q. Sun (EPFL, UCF) Generalized Stochastic Processes 11 / 15

12 Generalized White Noise Processes We define a generalized white noise as a generalized random process that is stationary and measurements for non-overlapping test functions are independent. Gelfand and Vilenkin considered the generic class of functional ˆP(φ) = exp( f (φ(t))dt), where f is a continuous function on the real line and φ is a test function. Note that this functional specifies an independent white noise if ˆP is continuous and positive definite and ˆP(φ 1 + φ 2 ) = ˆP(φ 1 ) ˆP(φ 2 ), whenever φ 1 and φ 2 have non-overlapping support. They proved that the complete class of functionals of above form with the required properties is obtained by choosing f to be the a Lévy exponent. M. Unser, P. Tafti, Q. Sun (EPFL, UCF) Generalized Stochastic Processes 12 / 15

13 Generalized White Noise Processes Definition A complex-valued continuous function f (ω) is a valid Lévy exponent iff f (0) = 0 and g τ (ω) = e τf (w) is positive definite of ω for all τ > 0. They actually established an injective correspondence between the characteristic form of an independent noise process to the family of infinite-divisible laws whose characteristic function takes form ˆp X (ω) = e f (ω) = E[e jωx ]. Recall that there is an injective relationship between the infinite-divisible RVs to the Lévy processes. Thus the Lévy exponent is fully characterized by the famous Lévy Khinchine Formula. Theorem (Lévy-Khinchine) f (ω) is a valid Lévy exponent iff it can be written as f (ω) = jb 1 ω b 2ω [e jaω 1 jaωχ a <1 (a)]dv (a). R\{0} M. Unser, P. Tafti, Q. Sun (EPFL, UCF) Generalized Stochastic Processes 13 / 15

14 Stochastic Differential Equations Previously we considered the generalized process with whitening operator L : S > S Ls = w. This definition only makes sense if we can construct an inverse operator T = L 1 and then the stochastic process s can be expressed formally as s = L 1 w. The paper showed the existence of the inverse operator for generic stochastic linear differential equation N n=0 a nd n s = M m=0 b md m w, by requiring the Lévy exponent f to be p-admissible, i.e., with 1 p 2, if there exists a positive constant C such that f (ω) + ω f (ω) C ω p for all ω R. M. Unser, P. Tafti, Q. Sun (EPFL, UCF) Generalized Stochastic Processes 14 / 15

15 Sparsification in a Wavelet-Like Basis Assume that we have a wavelet-like basis ψ i,k (t) = ψ i (t 2 i k) with scale and location indices (i, k). ψ i = L φ i is some normalized reference wavelet and φ i is an appropriate scale-dependent smoothing kernel. We have the following property on the wavelet coefficients v i [k] =< s, ψ i,k >. Theorem Let v i (t) =< s, ψ i ( t) > with ψ i = L φ i be the i-th channel of the continuous wavelet transform of a generalized stochastic process s with whitening operator L and p-admissible Lévy exponent f. Then, v i (t) us a generalized stationary process with characteristic functional ˆP vi (ϕ) = ˆP s (φ i ϕ). Moreover, the characteristic function of the wavelet coefficient v i [k] = v i (2 i k) is given by ˆp vi (ω) = e f i (ω) and is infinitely divisible with f i (ω) = f (ωφ i (t))dt. The infinite divisibility suggests that the representation is sparse. M. Unser, P. Tafti, Q. Sun (EPFL, UCF) Generalized Stochastic Processes 15 / 15