Weiss-Weinstein Bounds for Various Priors

Transcription

1 Weiss-Weinstein Bounds for Various Priors Florian Xaer, Christoph F. Mecklenbräuker, Peter Gerstoft, and Gerald Matz Abstract We address analytic solutions of the Weiss-Weinstein bound (WWB), which lower bounds the mean squared error of Bayesian inferrers. The bound supports discrete, absolutely continuous, and singular continuous probability distributions, the latter corresponding to joint estimation and detection. We present new analytical solutions for truncated Gaussian, Laplace, categorical, uniform, and Bernoulli distributions. We focus on sparse signals modeled by a Laplace prior as used in Bayesian LASSO methods, priors of truncated Gaussian densities, and uninformatie priors. In general, finding the tightest WWB of a model is a non-conex optimization problem. Hence, we show numerical examples of known and new WWBs to gain additional insight. Index Terms Weiss-Weinstein lower bound, Bayesian inference, mean squared error, LASSO, categorical, uniform, Laplace, truncated Gaussian I. INTRODUCTION In this paper we address discrete-time stochastic models in a Bayesian context. Random parameters are inferred from random obserations. Estimation and detection (classification) are types of inference for continuous and discrete random parameters. They are designed by minimizing an expected loss function. If the loss function is the mean squared error, then the optimization result is the minimum mean squared error (MMSE) inferrer. Hence, the mean squared error of an inferrer is a performance indicator. The study of lower bounds on the mean squared error of a Bayesian estimator [] [4] entails arious bounds [5] [8]. The most prominent bound is the Bayesian Cramér-Rao bound (CRB) for linear models, where the prior and the likelihood function are Gaussian. In this case, the mean squared error of the MMSE estimator coincides with the CRB. We are interested in Bayesian lower bounds for the inference of parameters, which are jointly applicable to discrete and continuous random state ariables. Additionally, the bound shall be alid for probability densities with finite support. It turns out that the regularity conditions for the applicability of the Bayesian CRB are too restrictie for discrete parameters [9], []. We want bounds with relaxed regularity conditions, which are applicable to discrete parameters, especially to discrete parameters that stems from quantized continuous states. This requirement guides us to the Weiss-Weinstein bounds (WWB) [5], [7], [], [2] and their sequential ersions [], [3] [5]. Florian Xaer florian@xaer.me, Gerald Matz gmatz@nt.tuwien.ac.at, and Christoph Mecklenbräuker cfm@nt.tuwien.ac.at are with the Institute of Telecommunications, Vienna Uniersity of Technology, Gusshausstrasse 25/E389, 4 Vienna, Austria. Peter Gerstoft gerstoft@ucsd.edu is with Scripps Institution of Oceanography, Uniersity of California, San Diego, 95 Gilman Dr, La Jolla, CA , California. The WWB is parametrized by a test point. The optimal test point proides the tightest WWB. For linear models and sample spaces R N, the WWB depends on the Bhattacharyya coefficient (BC) []. If the sample space is a subset of R N, then this leads to a generalized Bhattacharyya coefficient (GBC). Using analytic solutions of the (G)BC for popular distributions, numerical computations gie insight into the (G)BC and WWB. We open with a summary of WWBs in Section II, adding singular-continuous distributions that are inducing joint estimation and detection (e.g. in [5]). In Section III we present the BC stemming from Gaussian distributions and derie the GBC from categorical, Bernoulli, Laplace, uniform, and truncated Gaussian distributions. The newly deried WWBs are used as priors in Sections V to VII. A Laplacian prior induces sparsity and leads to the Bayesian least absolute shrinkage and selection operator (LASSO). The truncated Gaussian prior introduces a finite support as in practical problems, whereas a uniform prior is uninformatie, and links to frequentist models. II. WEISS-WEINSTEIN BOUNDS Consider the probability space (R N, B, P x ) with the N- dimensional sample space R N, the induced Borel algebra B, and the probability measure P x (B) = P (x B) [6], [7]. Due to Lebesgue decomposition, a probability distribution may consists of an absolutely continuous, a singular (discrete), and a singular continuous part, i.e. P x = c P c x + c 2 P d x + c 3 P s x, c + c 2 + c 3 =. Thus, the expectation of a measurable ector-alued function g(x) is E x (g(x)) := g(x)dp x (x) = c g(x)f x (x)dx R N R N + c 2 g(x)p x (x) + c 3 g(x)dpx s x C R x R N N with the probability density function (PDF) f x (x) = dpx c (x)/d λ c (x) and the probability mass function (PMF) p x (x) = dpx d (x)/d λ d C(x). If Px s = Px α... P α N x, α i {c, d}, is a product measure, then a hybrid density x (x) exists as a product of PDFs and PMFs. We call all three types simply probability density (PD) [8]. In what follows, we use the notation λ x for probability measures wheneer we assume the existence of a PD for the random ariable x. To simplify notation, we write E ( ) := E x,y ( ), f(x) := f x (x), p(x) := p x (x), and (x) := x (x). For the Bayesian estimate ˆx(y), the parameters x R N are inferred from measurements y R M. In what follows, we consider the affine measurement equation y = Cx +, x (x), (),

2 2 with measurement matrix C : R N R M and random noise R N. The resulting estimation error of estimate ˆx is defined by e := ˆx(y) x. The mean squared error of the inferrer is lower bounded by the Weiss-Weinstein matrix W parametrized by a test-point matrix H = [h,..., h N ]. The test point h a, a =,..., N, pertains to element [x] a. For the bound, we insert s i = s j = /2 and (38) into (39)-(4) of [5] and reformulate it. Hence, the mean-squared-error matrix is bounded as (left-hand side minus right-hand side is positie semi-definite.) E ( ee T) W (H) := HJ H T, () where J = J(H) depends on H with typical element [J] a,b := ϱ( h a, h b ) ϱ(h a, h b ) ρ (Ch a, )ρ x (h a, )ρ (Ch b, )ρ x (h b, ), (2) with the symmetrized ersion of the generalized non-metric Bayesian Bhattacharyya coefficient ϱ(h a, h b ) := [ρ (Ch a, Ch b )ρ x (h a, h b ) +ρ ( Ch a, Ch b )ρ x ( h a, h b )], (3) and the non-symmetric Bhattacharyya coefficient (GBC) ρ (h a, h b ) = ( + h a ) ( h b )dλ, (4) V where V = { : () > }. The GBC ρ x is similar. Due to the product measure λ, the integral in (4) is an abbreiation of sums and Lebesgue integrals. The multiplications of the GBCs in (2) and (3) are due to linearity and statistical independence. Eq. (3) simplifies for symmetric PDs. For infinite support of (), (4) becomes the BC [9] [2] ρ (h a, h b ) = ( + h a ) ( h b )dλ. (5) Obsere that if we substite := h b in (4), then the GBC is still dependent on h a and h b. We will see that the BC of the Gaussian and Laplace distributions depend only on the sum h := h a +h b and that PDs with finite support induce box constrains on the test-point ector h []. III. ANALYTIC SOLUTIONS In what follows, we derie analytic solutions of the BC of seeral distributions. Obsere from (4) that if () = ( )... N ( N ), then ρ (h a, h b ) = ρ (h a,, h b, )... ρ N (h a,n, h b,n ). Thus, we consider uniariate PDs wheneer the elements of the parameter ector are assumed to be statistically independent. A. Categorical Distribution Categorical distributions are defined by a probability mass function. If the categorical distribution stems from the equidistant quantization (discretization) of a continuous distribution, l R N, l L Z, we can use () to (4). In this case, the mean squared error makes sense. Inserting its density p() = l L p l =l with probability masses p l and indicator function, into (4) gies ρ C (h a, h b ) = [ ] /2 p l +ha=l p l hb =l. L l L l L Obsere that the GBC depends on h a and h b. The Bernoulli distribution p = + p = and the discrete uniform distribution Unif {r, s} with p l = p and h a ± h b s r + are examples of categorical distributions. Next, we address absolutely continuous distributions. B. Uniform Distribution Let us insert the continuous uniform distribution Unif {r, s}, into (4), i.e. s r f() = s r r s, s r r +ha s r hb sd. We distinguish four cases: s ha r r+hb s+h b d, h a, h b, s r h r+hb a s+h b d, h a, h b, s ha r s r+hb s+h b d, h a >, h b <, r h r+hb a s+h b d, h a <, h b >. Merging the first two and splitting the last two, the GBC becomes h s r, h a, h b or h a, h b, ha s r, h a >, h b <, h a > h b, ρ cu + h b (h a, h b ) = s r, h a >, h b <, h a h b, + ha s r, h a <, h b >, h a > h b, h b s r, h a <, h b >, h a h b,, else. with h a ± h b < s r. C. Laplace Distribution The Laplace distribution La {b, µ x } is a sparse prior. Inserting Laplace s density f() = µ e b into (5) and substituting w = µ and h = h a + h b gies e ( w+h + w ) dw. If h >, then the integral splits into three terms, w >, w + h > : w <, w + h < : w <, w + h > : h e w /b h / dw = 2 e h /, e w /b h / dw = 2 e h /, h e h / dw = h e h /. This is a reised ersion of BC in [], which contains only the first case.

3 3 Due to symmetry, for h, there are three similar cases. The composition of the terms proides the BC {( + ρ L (h a, h b ) = ρ L h (h) = ) e h /, h >, ( h ) e h/, h. D. (Truncated) Gaussian Distribution The density of Gaussian distribution N {µ, C } is () := (2π) N /2 (det C ) /2 e 2 µ 2 C. with the mean µ, the coariance matrix C, and the weighted norm h C := (h T C h) /2. Then the Bayesian BC [] is gien by ρ G (h b, h b ) = ρ G (h) := e 8 h 2 C. The truncated Gaussian N {µ, C, r, s} is a Gaussian distribution limited to the support [r, s], i.e., the density is gien by f() = c e 2 µ 2 C r s. with appropiate normalization factor c. Inserting f() into (4), substituting = µ, and use normalization factor c, we get s µ c e 4 r µ ( e 4 ( 2 2 C 2 T C h b + h b 2 C ) +2 T C h a+ h a 2 C r ha s h a r+hb s+h b d. Next, we substitute = + ha /2 h b/2 and obtain ρ tg (h a, h b ) = c ρg (h a, h b ) ) s µ + ha/2 h b/2 r µ + ha/2 h b/2 e 2 2 C r + ha/2+h b/2 s r ha/2 h b/2 sd. The two cases h a, h b and h a, h b lead to ρ G (h a, h b ) c s µ ha /2 h b /2 r µ + ha /2+ h b /2 If h a, h b >, then ρ tg c ρg for h a + h b and c ρg (h a, h b ) is s µ ha /2 h b /2 r µ + ha /2 h b /2 s µ + ha /2 h b /2 r µ + ha /2+ h b /2 e 2 2 C d. e 2 2 C d e 2 2 C d for h a + h b >. If h a >, h b, then ρ tg c ρg s µ + ha /2+ h b /2 r µ + ha /2 h b /2 (h a, h b ) is e 2 2 C d Weiss-Weinstein bound W (ha).5.5 Laplace Uniform Gauss tgauss Categorical Uniform (discr.) Bernoulli Figure. Weiss-Weinstein bound s. test point for equal prior and noise distributions. h a = h b for uniariate distributions. for h a + h b and c ρg s µ ha /2 h b /2 r µ + ha /2 h b /2 e 2 2 C d for h a + h b >. Due to the finite support, h a ± h b s r. This integral and Factor c in general can only be numerically ealuated. IV. NUMERICAL EXAMPLES In the sequel, we use the scalar measurement model with C =. Test matrix H shrinks to a scalar, i.e. h a = h b in (2). Fig. plots WWBs for Laplace La {, }, Gaussian N {, 2}, uniform Unif {, }, and truncated Gaussian N {, 2, 3, 3} prior and measurement distributions. In general, the WWB is non-concae regarding the test point and no general solution for the optimal test point h exists. The existence of an analytic solution of the tightest bound depends on the BC of the particular distributions. Howeer, due to symmetry, for unimodal densities only h is needed and the BC is quasi-concae. The optimal test point is small. In case of Gaussian distributions, the optimal test point tends to h a. The WWB approaches the CRB, cf. [] for a proof. The tightest WWB in the Gaussian case coincides with the CRB and is reached by an optimal estimator. Fig. also shows the WWB of the categorical distribution {.5,.2,.,.2}, the Bernoulli distribution {.5,.5}, and the discrete uniform distribution Unif { 3, 3} for =. The shape of WWB of discrete distributions follow that of continuous distributions. In case of Bernoulli distributions, there are only two possible test points h a = h b = ± = ±. According to [], the WWBs of continuous distributions and their quantized discrete ersions are equal for the same test points. Comparing the WWB of the discrete uniform distribution with the bound of the continuous uniform distribution, we obsere how a smaller support lowers the Weiss-Weinstein bound.

4 REFERENCES 4 4 Weiss-Weinstein bound W (ha) 3 2 Laplace/Laplace Laplace/Gaussian Gaussian/Gaussian Weiss-Weinstein Bound W (ha).5 support [r, s] [ 2σ 2, 2σ 2 ] [ σ 2, σ 2 ] [ /2σ 2, /2σ 2 ] [ 3σ 2, 3σ 2 ] Gaussian [, ] Weiss-Weintstein bound W (ha).5 Gaussian/Gaussian Uniform/Gaussian Uniform/Uniform Figure 2. WWB for a Laplace prior and Gaussian likelihood compared to pure Laplacian and pure Gaussian distributions. Figure 3. WWB for truncated Gaussian densities of support [r, s] s. test point. The Gaussian cure is for comparison (dotted). Figure 4. WWB for a uniform prior and Gaussian likelihood function compared to pure uniform and pure Gaussian case. V. SPARSE SIGNALS LAPLACE PRIOR The least absolute shrinkage and selection operator (LASSO) is a prominent estimation method for sparse signals x [22]. LASSO minimizes the squared error min x y Cx 2 2, where ectors x, y are deterministic. Assuming a sparse signal x, it makes sense to constrain the l - norm x = n x n < b R +. The associated Lagrangian becomes min x y Cx b x, b R +. In the Bayesian context x and y are random. Then the minimization is interpreted as the maximization of the posterior distribution, if the likelihood function is Gaussian and the prior is Laplacian. In Fig. 2, we show the WWB for the scalar linear measurement model for pure Laplace distributions x, La {, 2}, pure Gaussian distributions x, N {, 2} and Laplace prior / Gaussian noise x La {, 2}, N {, 2}. Obsere that the WWB of the mixed case lies in-between the pure examples. In the limit h, the bound is greater than zero. VI. THE TRUNCATED GAUSSIAN DENSITY Real-world applications that are modeled by a Gaussian prior, are usually box constrained. This leads to the truncated Gaussian distribution. Let x, N {, 2, r, s}. Fig. 3 shows the WW for different [r, s]. The Gaussian case (r and s ) is our reference. Compare the cures with that of continuous uniform distributions in Fig.. They look similarly. The bound decreases with decreasing support size s r. Compare it with the pure uniform cure in Fig. 3. The support of the truncated Gaussian density defines, whether the WWB tends to the uniform or to the Gaussian WWB. VII. UNINFORMATIVE PRIOR UNIFORM PRIOR A uniform prior is uninformatie in the sense that it only defines a minimum and maximum alue. Measurement y updates the beliefe of signal x. Due to the uninformatie prior, the noise distribution has a strong impact on the beliee update. Let us now compare uniform prior / Gaussian noise x Unif { 2.45, 2.45}, N {, 2} with uniform distributions x, Unif { 2.45, 2.45}, and Gaussian distributions x, N {, 2}. For the uniform distribution we used r = s = + /2 2σ to achiee identical ariance σ 2 = 2 for the Gaussian and uniform distribution. Next, we plot the WWB for the scalar linear measurement model in Fig. 4. Replacing the uniform by a Gaussian likelihood function increases the lower bound. Obsere that the WWB of the mixed model lies aboe the pure uniform case, but has a similar shape. Two characteristics influence the maximum WWB: the support and the flatness of the densities. The greater the support, the greater the possible error. The flatter the density, the greater the error. VIII. CONCLUSION The Weiss-Weinstein bound (WWB) is parametrized by test points leading to different leels. The WWB depends on the density s ariability and on its support. They apply to both discrete and absolutely continuous probability distributions. This is useful for quantized ariables, where the alues in the sample space hae a physical interpretation. Furthermore, the WWB supports singular-continuous multiariate probability distributions with a hybrid joint density. A hybrid joint density is the product of probability densities and probability mass functions. The WWB is applicable to probability distributions that are non-differentiable or hae finite support, e.g. uniform or truncated Gaussian distributions, and distributions defined only on real positie support, e.g. exponential distributions. For linear models, the WWB depends on the generalized Bhattacharyya coefficient. If the distribution is unimodal, then the WWB is bimodal. To exemplify its use, we discussed a linear measurement model. This illustrates the application of the WWB for sparse signal models with a Laplace-distributed prior and additie Gaussian noise. The WWB for a truncated Gaussian prior tends to the normally or uniformly distributed case depending on its support. The WWB of a model with a continuous uniform prior and a Gaussian likelihood function is greater than of the uniform-only model and has a similar shape.

5 5 REFERENCES [] S. M. Kay, Fundamentals of Statistical Signal Processing: Estimation Theory. NJ: Pearson Education, 993, ol.. [2] B. Ristic, S. Arulampalam, and N. Gordon, Beyond the Kalman filter: Particle filters for tracking applications. Boston: Artech House, 24. [3] O. Hlinka, O. Slučiak, F. Hlawatsch, P. M. Djurić and, and M. Rupp, Likelihood consensus and its application to distributed particle filtering, IEEE Trans. Signal Process., ol. 6, no. 8, pp , Aug. 22. [4] O. Hlinka, F. Hlawatsch, and P. M. Djurić, Distributed particle filtering in agent networks: A surey, classification, and comparison, IEEE Signal Process. Mag., ol. 3, no., pp. 6 8, 23. [5] A. J. Weiss and E. Weinstein, A general class of lower bounds in parameter estimation, IEEE Trans. Inf. Theory, ol. 34, no. 2, pp , 988. [6] H. L. Van Trees and K. L. Bell, Bayesian bounds for parameter estimation and nonlinear filtering/tracking. IEEE Press, 27. [7] A. Renaux, P. Forster, P. Larzabal, C. D. Richmond, and A. Nehorai, A fresh look at the Bayesian bounds of the Weiss-Weinstein family, IEEE Trans. Signal Process., ol. 56, no., pp , No. 28. [8] D. A. Koshae and O. A. Stepano, Application of the Rao-Cramer inequality in problems of nonlinear estimation, Journal of Comp. and System Sci. Int., ol. 36, pp , Apr [9] Z. Duan, V. P. Jilko, and X. R. Li, State estimation with quantized measurements: Approximate MMSE approach, in th Int. Conf. on Inform. Fusion, IEEE, 28, pp. 6. [] F. Xaer, P. Gerstoft, G. Matz, and C. Mecklenbräuker, Analytic sequential Weiss-Weinstein bounds, IEEE Trans. Signal Process., ol. 6, no. 2, pp , 23. [] N. D. Tran, A. Renaux, R. Boyer, S. Marcos, and P. Larzabal, Weiss-Weinstein bound for MIMO radar with colocated linear arrays for SNR threshold prediction, Signal Processing, ol. 92, no. 5, pp , 22. [2] A. Renaux, Weiss Weinstein bound for data-aided carrier estimation, Signal Process. Letters, IEEE, ol. 4, no. 4, pp , 27. [3] I. Rapoport and Y. Oshman, Recursie Weiss- Weinstein lower bounds for discrete-time nonlinear filtering, in 43rd IEEE Conf. on Decision and Control, ol. 3, Dec. 24, pp [4] S. Reece and D. Nicholson, Tighter alternaties to the Cramer-Rao lower bound for discrete-time filtering, in 8th Int. Conf. on Inform. Fusion, ol., Jul. 25, 6 pp. [5] F. Xaer, G. Matz, P. Gerstoft, and C. F. Mecklenbräuker, Predictie state ector encoding for decentralized field estimation in sensor networks, in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Kyoto, JP, Mar. 22, pp [6] P. Billingsley, Probability and measure, 3rd. John Wiley & Sons, Inc., New York, 22. [7] F. E. Burk, A Garden of Integrals, st ed., ser. The Dolciani Math. Expositions 3. The Math. Assoc. of Am., 27. [8] F. Xaer, Mixed discrete-continuous Bayesian inference: Censored measurements of sparse signals, IEEE Trans. Signal Process., ol. 63, no. 2, pp , No. 25. [9] Z. Ben-Haim and Y. C. Eldar, A comment on the Weiss Weinstein bound for constrained parameter sets, IEEE Trans. Inf. Theory, ol. 54, no., pp , 28. [2] A. Bhattacharyya, On a measure of diergence between two statistical populations defined by their probability distributions, Bull. Calcutta Math. Soc, ol. 35, no. 99-9, p. 4, 943. [2] T. Kailath, The diergence and Bhattacharyya distance measures in signal selection, IEEE Trans. Commun. Technol., ol. 5, no., pp. 52 6, Feb [22] T. Park and G. Casella, The Bayesian Lasso, Journal of the American Statistical Association, ol. 3, no. 482, pp , 28. eprint: 98/