Cramer-Rao Lower Bound Computation Via the Characteristic Function

Transcription

1 Cramer-Rao ower Bound Computation Via the Characteristic Function Steven Kay, Fellow, IEEE, and Cuichun Xu Abstract The Cramer-Rao ower Bound is widely used in statistical signal processing as a benchmark to evaluate unbiased estimators. However, for some random variables, the probability density function has no closed analytical form. Therefore, it is very hard or impossible to evaluate the Cramer-Rao ower Bound directly. In these cases the characteristic function may still have a closed and even simple form. In this paper, we propose a method to evaluate the Cramer-Rao ower Bound via the characteristic function. As an example, the Cramer-Rao ower Bound of the scale parameter and the shape parameter of the K-distribution is accurately evaluated with the proposed method. Finally it is shown that for probability density functions with a scale parameter, the Cramer-Rao ower Bound for the remaining parameters do not depend on the scale parameter. I. INTRODUCTION In statistical parameter estimation theory, the Cramer-Rao ower Bound (CRB) is a lower bound on the variance of any unbiased estimator []. et p(x; θ) denotes the probability density function (PDF) of a random variable X parameterized by θ, where θ [θ, θ 2,,θ ] T. et ˆθ be an unbiased estimator of θ. If p(x; θ) satisfies the regularity conditions, or [ ] lnp(x; θ) E 0 θ This work was supported by the Office of Naval Research under contracts N and N M-033 and by the Air Force Research aboratory, Sensors Directorate, Hanscom AFB, MA. Steven Kay is with the Department of Electrical Computuer and Biomedical Engineering, University of Rhode Island, Kingston, RI (Phone: , Fax: , kay@ele.uri.edu) Cuichun Xu is with the Department of Electrical Computuer and Biomedical Engineering, University of Rhode Island, Kingston, RI (Phone: , Fax: , ccxu@ele.uri.edu)

2 2 where E [ ] denotes the expectation, then ) [ ] var (ˆθi I (θ) CRB θ ii i () ) where var (ˆθi denotes the variance of the i th element of the estimator, or the estimator for θ i, [ I (θ) ] ii is the (i, i) element of the inverse Fisher information matrix, and is the CRB for θ i. The Fisher information matrix is defined as [I(θ)] ij E [ ] lnp(x; θ) lnp(x; θ). (2) θ i θ j It is seen that the key step to obtain the CRB is the evaluation of [I (θ)] ij. Compared to other variance bounds [2], [3], the CRB is usually easier to compute. Therefore it is extensively used in the signal processing literature as a benchmark to evaluate the performance of an unbiased estimator. However, there are cases in which p (x; θ) has no closed form. As a result, the CRB is hard or impossible to evaluate analytically via (2). However, in some cases in which p (x; θ) has no closed form, the characteristic function (CF) of p (x; θ) does have a closed and even simple form, for example the K-distribution [4], the α-stable distribution [5] and the Class A distribution [6]. The CF of p (x; θ) is defined as [ φ X (ω; θ) E e jωx]. (3) This is just the Fourier transform of p (x; θ). Therefore p (x; θ) can be thought of as a time domain function and φ X (ω; θ) as a frequency domain function. Since (2) is essentially an integration in the time domain, it is possible to evaluate I (θ) equivalently in the frequency domain. This idea could be applied to both univariate and multivariate distributions. In this paper, we focus on parameter CRB evaluation of univariate distributions. The extension to multivariate distributions will be explored in the future work. The paper is organized as follows. In Section II, the method to evaluate the CRB via the CF is introduced. In Section III, an example is given and Section IV offers some conclusions. II. CRB COMPUTATION VIA THE CHARACTERISTIC FUNCTION Equation (2) can be rewritten as [I (θ)] ij lnp(x; θ) lnp(x; θ) p (x; θ)dx θ i θ j p (x; θ) p (x; θ) dx. (4) θ i θ j p (x; θ) Assume there is a real positive number, such that p (x; θ) 0 for x > 2, then [I (θ)] ij /2 /2 p (x; θ) p (x; θ) dx. (5) θ i θ j p (x; θ)

3 3 et u x, [I (θ)]ij /2 et G (u; θ) p (u; θ) /2 p (u; θ) p (u; θ) θ i θ j p (u; θ) du. /2 G (u; θ) G (u; θ) [I (θ)] ij du. (6) /2 θ i θ j G (u; θ) From the property of a PDF, p (x; θ) and equivalently G (u; θ) 0, so we can consider G (u; θ) to be a discrete-time power spectral density (PSD). Therefore, G (u; θ) can be approximated by an autoregressive (AR) model for a sufficiently large model order, i.e. [7] where σ 2 u is the excitation noise variance, and G (u; θ) σ2 u A (u) 2 (7) A (u) + a[]e j2πu + a[2]e j2πu2 + + a[p] e j2πup where a[], a[2],,a[p] are the AR coefficients with p being the order of the AR model. et F denote the inverse Fourier transform. It is readily seen that F {A (u)} /2 /2 a[n] A (u)e j2πun du The proper model order p can be determined by comparing the estimated AR PSD σ2 u with the true A(u) 2 G (u; θ) or numerically calculated G (u; θ). If the fitting error is too large, then a larger p should be used. Plugging (7) into (6), we have [I (θ)] ij σ 2 u σ 2 u /2 /2 /2 /2 [ ] [ ] G (u; θ) G (u; θ) A (u) A (u) du θ i θ j [ ] [ ] G (u; θ) G (u; θ) A (u) A (u) du. (8) θ i θ j The last equation relies on the property that p (x; θ), and therefore G (u; θ) is real. By Parseval s theorem, where and [I (θ)] ij σ 2 u n { } G (u, θ) x[n] F A (u) θ i { } G (u, θ) y [n] F A (u) θ j x[n]y [n] (9)

4 4 et g [n] F {G (u; θ)}, so that g[n] can be thought of as the autocorrelation sequence associated with the PSD G (u; θ) [7]. It is given by g [n] etting x u in the above equation yields /2 /2 /2 /2 /2 G (u; θ)e j2πun du p (u; θ)e j2πun du. (0) g[n] p(x; θ)e j2π x n d x /2 E X [e j2π n x] ( ) 2πn φ X ; θ. () Since the CF is assumed to have a closed analytical form, g[n] can be easily obtained from (). With g[0], g[],,g[p] known, the Yule-Walker method can be used to determine the AR parameters { a[], a[2],,a[p], σ 2 u } [7]. Next, since the inverse Fourier transform is linear, we have { } G (u; θ) F F {G (u; θ)}. (2) θ i θ i Plugging () into (2), we have et so that from (9) where denotes the convolution. { } G (u; θ) F θ i [I (θ)] ij σ 2 u ( ) φ 2πn X ; θ θ i ( ) 2πn φ X θ i ; θ. (3) h i [n] ( ) 2πn φ X θ i ; θ n (4) (a[n] h i [n]) (a[n] h j [n]) (5) If there is a real positive integer M, such that h i [n] 0, and h j [n] 0 for n > M, then (5) can be approximated as [I (θ)] ij σ 2 u M n M (a[n] h i [n])(a[n] h j [n]). (6) After the analytical form of h i [n] and h j [n] are obtained, M can be obtained.

5 5 For an arbitrary p (x; θ), we can always find the positive number, such that p (x; θ) 0 for x > 2. This may be done directly from the expression of p (x; θ). However, since we are considering the case when p (x; θ) has no closed form, it is necessary to be able to determine from φ X (ω; θ). To do so note that the moments of X can be obtained by [8] E [X n ] d n φ X (ω) j n dω n. ω0 From the moments of X and φ X (ω; θ), can be found using probability bounds. In the example of the next section, the Chernoff bound is used to determine [9]. The steps to evaluate [I (θ)] ij via the CF φ X (ω; θ) are summarized below, ) From φ X (ω; θ), obtain such that p (x; θ) 0 for x > 2, by using a bound. 2) Select an AR model order p. 3) Calculate g[0], g[],,g[p] from (). 4) Determine a[], a[2], a[p], σ 2 u by the Yule-Walker method. 5) Obtain the expression of h i [n] and h j [n] from (4). 6) Select M from the expression of h i [n] and h j [n]. 7) Evaluate I(θ) from (6). We note that in the above procedure the PDF p (x; θ) need not be known. III. CRB COMPUTATION FOR THE K-DISTRIBUTION The K-distribution is widely used to model sea surface clutter in high-resolution radar systems and to model sea floor reverberation in high-resolution active sonar systems [4], [0], [], [2]. et X X r + jx i be a zero mean, circularly symmetric complex Gaussian random variable, i.e. [X r, X i ] T N ( 0, σ 2 I ), where I is a 2 2 identity matrix. et Y be a Gamma random variable with the shape parameter being ν + and the scale parameter being /2, or ( 2) ν+ ) Γ(ν+) p Y (y) yν exp ( 2 y y 0 0 y < 0 where Γ ( ) denotes the Gamma function and ν >. Then Y X is distributed according to a complex K-distribution [2]. Denote this variable as R Z + jt, where Z is the real part of R and T is the imaginary part. The joint CF of (Z, T) is given by [0] φ Z,T (ω, γ) E [e jωz+jγt] ( + σ 2 (ω 2 + γ 2 )) (7) ν+. (8)

6 6 Hence, the CF of Z is given by and the PDF of Z can be shown to be p Z (z) φ Z (ω) φ Z,T (ω, 0) πσ 2 Γ (ν + ) ( + σ 2 ω 2 ) ν+ (9) ( ) z ν+/2 K 2σ ν+/2 where K α ( ) is the modified Bessel function of the second kind. ( ) z σ From (20) it is seen that σ is the scale parameter and ν is the shape parameter. The moments of Z can be obtained from the CF as and E E [Z] j [Z 2] 2 φ Z (ω) ω 2 φ Z (ω) ω Therefore the variance of Z is [ var(z) E Z 2] E 2 [Z] 2 (ν + ) σ 2. (20) 0 (2) ω0 2 (ν + )σ 2. (22) ω0 It is seen that as ν decreases, the variance of Z decreases. When ν, var(z) 0, which means Z approaches the deterministic value 0. In [0] an approximate CRB for ν of the PDF of R is derived under the condition that σ 2 is known and ν 0. In the rest of this paper, we will focus on computing the CRBs for both ν and σ 2 of p Z (z) without any additional constrain on the range of ν. Considering the expression of (20), it would be a formidable task to evaluate [I (θ)] ij using (2). Therefore we evaluate [I (θ)] ij via the CF using the method introduced in the last section. A. Selection of is determined from the Chernoff bound, which is given as [9] prob (z τ) exp( sτ)e [exp(sz)] exp( sτ + log (E [exp(sz)])) (23) for any s 0. With this bound, an is found as 2 5 (ν + ) σ for ν 4 0 5σ for ν < 4 The details are given in Appendix I. In general, a simple procedure to determine is to increase it, until the results converge. (24)

7 7 B. Selection of the AR Model Order p It can be seen that from (20) as ν, p Z (z) becomes narrower, requiring p to be large. The proper value of p is also determined by. This is because as increases, the PSD G(u; θ) becomes narrower, requiring a larger p. For the selection of given by (24), an empirical formula is obtained from simulations, and is given as 200 exp( 7ν)+70 for < ν < 0.9 p 50 for ν 0.9 where denotes the smallest integer greater than. In general, a simple procedure to determine p is to increase it, until the results converge. C. Expression of g[n] and h i [n] Plugging (9) into () we have Plugging (9) into (4) yields and g[n] h σ 2 [n] ( + σ 2 ω 2 ) ν+ ω 2πn [ σ 2 ( + σ 2 ω 2 ) ν+ ω 2 ( + ν) ( + σ 2 ω 2 ) ν+2 ω 2πn h ν [n] [ ν ( + σ 2 ω 2 ) ν+ ln ( + σ 2 ω 2) ( + σ 2 ω 2 ) ν+ ] ω 2πn ] ω 2πn ω 2πn (25) (26) D. Selection of M From the derivation in Appendix II, M min(m, M 2 ) where M 2π ( ) + ν 2ν σ 2ν+4 2ν+2.

8 8 and ( min 2πσ M xν 2πσ where x ν is the greatest root of the equation, 2πσ log ( + x) ( + x) and it can be easily found by a numerical method. ( ) 2 ) 2ν ν 0 4 ν 0 <ν < 0 In general, a simple procedure to determine M is to increase it, until the results converge. E. A Corroboration Test Now we have everything to calculate I (θ) from (6). As a corroboration test of the proposed method, a special case in which ν 0 is considered first. When ν 0, it is readily shown that p Z (z) ( 2σ exp z ). (27) σ After some simple algebra, we have ( ( [I (θ)] σ2 σ E lnp x; σ 2 ) ) 2 2 σ 2 4σ4. (28) The theoretical value given by (28) and the value computed from the proposed method are shown in Fig. as circles and stars respectively. It can be seen that the match is very good and the results are precise enough for most applications. However, if higher precision is required, we can always increase, p and M. F. CRB Evaluation of ν and σ 2 for the K-PDF of (20) Even though there are two parameters in p Z (z), it is shown in Appendix III that the CRB ν is independent of σ 2, and for a fixed ν, CRB σ 2 is a linear function of σ 4. This is because σ is a scale parameter of p Z (z). Therefore, there is no need to plot the 2 dimensional figures of CRB σ 2 and CRB ν versus σ 2 and ν. In Fig. 2 the computed CRB ν is plotted versus ν and in Fig. 3 the computed CRB σ 2 σ 4 better viewing. is plotted versus ν. In both figures the computed values are connected with straight line for

9 9 7 6 ν0 Theoretic Calculated 5 [I(θ)] σ 2 σ σ 2 Fig.. Theoretical and computed [I(θ)] σ 2 σ 2 versus σ2 for the K-PDF of (20) with ν CRB ν ν Fig. 2. Evaluated CRB ν versus ν for all σ 2. IV. CONCUSION We have proposed a method to evaluate the CRB via the CF. When the PDF has no closed form, it can be very hard or impossible to evaluate the CRB directly. With the proposed method, the CRB

10 CRB σ 2/σ ν Fig. 3. Evaluated CRB σ 2 σ 4 versus ν. can be accurately evaluated when the CF is known. As an example, the CRBs of the scale parameter and the shape parameter of the K-distribution have been computed with the proposed approach. It has also been shown that for a PDF with a scale parameter, the CRB for the remaining parameters do not depend on the scale parameter. APPENDIX I SEECTION OF From (9), the moment generation function of Z is given as E [ζ z ] etting ζ s and minimizing logarithm of (23) ( σ 2 ζ 2 ) υ+. F (s) sτ (ν + ) log ( σ 2 s 2) with respect to s, we have ) 2 F (ŝ) (ν + ) 2 + τ2 ( σ 2 (ν + ) 2 + τ 2 σ (ν + ) 2 + ν + (ν + ) log σ2 τ 2.

11 If we let τ 5 (ν + ) 2 σ 2 then It can be shown that for ν < 4 F (ŝ) 3 log ( ( ( log ν. 5) 5)) prob (z τ ν < 4) prob (z τ ν 4) exp(f (ŝ; ν 4)) < Hence if we choose 5 (ν + ) σ for ν 4 τ 5 5σ for ν < 4 (29) then prob (z τ) < Since p Z (z) is symmetric we have etting 2τ, then p Z (z) 0 for z > 2. prob (z τ) < APPENDIX II SEECTION OF M Differentiating (9) with respect to σ 2 and taking the absolute value yield d dσ 2φ X (ω) ω 2 ( + ν) ( + σ 2 ω 2 ) ν+2 Hence, we have d dσ φ 2 X (ω) < 0 4, if ω > < ω2 ( + ν) (σ 2 ω 2 ) ν+2 ( + ν ) 2ν+2 σ 2ν Equivalently, h σ 2 [n] < 0 4 for all n M, if M 2π ( + ν 0 4 ( + ν) (σ 2 ) ν+2 ω 2ν+2. (30) ( ) + ν 2ν σ 2ν+4 2ν+2. ) 2ν+2 σ 2ν+4 2ν+2.

12 2 Differentiating (9) with respect to ν and taking the absolute value yield d dν φ X (ω) ln ( + σ 2 ω 2) ( + σ 2 ω 2 ) ν+ If ν > 2 then d dν φ X (ω) < 0 4 for ω σ < ln ( + σ 2 ω 2) (σ 2 ω 2 ) ν+ ( σ 2 ω 2) /2 (σ 2 ) ν+2 ω 2ν+2 (σ 2 ω 2 ) ( ) 2 2ν Or equivalently h ν [n] < 0 4 for all n M 2, if ( ) 2 2ν+ M 2 2πσ 0 4. ν+/2. (3) However, when ν 0, M 2 could be very large and this bound is not tight enough. Therefore, another bound is derived in the follows. For ν > 0 log ( + σ 2 ω 2) ( log + σ 2 ω 2) ( + σ 2 ω 2 ν+ < ) ( + σ 2 ω 2 ) solving the equation log(+x) +x 0 4 we have x , so log ( + σ 2 ω 2) ( + σ 2 ω 2 ) ν+ < 0 4 if ω σ Or equivalently h ν [n] < 0 4 for all n M 2, if M 2 2πσ Therefore for ν > 0 we have For < ν 0, the equation M 2 min ( ( ) 2 ) 2πσ ν+, 2πσ 0 4. log ( + x) ( + x) +ν 0 4 is first solved numerically. et x ν be the greatest root, then log ( + σ 2 ω 2) ( + σ 2 ω 2 ) ν+ < 0 4

13 3 if ω > xν So finally, and σ. Or equivalently h ν [n] < 0 4 for all n M 2, if M 2 xν. 2πσ ( min 2πσ M xν 2πσ, 2πσ ( ) 2 ) 2ν+ 0 4 M min(m, M 2 ). ν 0 <ν < 0 APPENDIX III INDEPENDENCE OF THE CRB OF A SCAE PARAMETER Theorem : et p Z (z; α, λ) be a PDF parameterized by α and λ. If p Z (z; α, λ) satisfies the regularity condition and λ is a positive scale parameter, or p Z (z; α, λ) λf(λz; α), then CRB α is independent of the value of λ, and for a fixed α, CRB λ is a linear function of λ 2. We note that the CRB for α will be smaller when λ is known. Also, in the Theorem there is only one remaining parameter, i.e. α. When there are more than one remaining parameters the same conclusion still holds, and it can be proved similarly. Proof: By definition, the (, ) element of the Fisher information matrix is [ ( ) ln(pz (z; α, λ)) 2 ] I αα E α ( ) (pz (z; α, λ)) 2 α p Z (z; α, λ) dz ( ) λf (λz; α) 2 dz. (32) α λf (λz; α) It is assumed that the range of the PDF is from to. For PDFs with other ranges, the theorem can be proved similarly. In the rest of the proof the limits of the integral are omitted for simplicity. et x λz, after straightforward algebra, we have from (32) ( ) f (x; α) 2 I αα dx. (33) α f (x; α) This is a function of α only, so it can be denoted as g (α).

14 4 By definition I λλ ( ) (pz (z; α, λ)) 2 λ p Z (z; α, λ) dz ( ) λf (λz; α) 2 λ λf (λz; α) dz ( f (λz; α) + λz f (u; α) u ) 2 dz. (34) uλz λf (λz; α) et x λz, after straightforward algebra, we have from (34) ( ) f (u; α) 2 I λλ f (x; α) + x u ux λ 2 dx. (35) f (x; α) This can be denoted as λ 2 g 2 (α), where g 2 (α) is a function of α only. Similarly I αλ ( ) ( ) (pz (z; α, λ)) (pz (z; α, λ)) α λ p Z (z; α, λ) dz ( ) ( ) (λf (λz; α)) f (u; α) f (λz; α) + λz α u dz. (36) uλz λf (λz; α) et x λz, after straightforward algebra, we have from (36) ( ) ( ) (f (x; α)) f (u; α) I αλ f (x; α) + x α u dx. (37) ux λf (x; α) This can be denoted as λ g 3(α), where g 3 (α) is a function of α only. Therefore, the Fisher information matrix can be expressed as I (α, λ) g (α) λ g 3 (α). λ g 3 (α) λ g 2 2 (α) As a result, I (α, λ) This expression proves the theorem. g (α) g 2 (α) g 2 3 (α) g 2 (α) λg 3 (α) λg 3 (α) λ 2 g (α) Considering the K-distribution PDF given by (20), according to Theorem, CRB ν is independent of σ. Hence CRB ν is independent of σ 2. Also CRB σ transformation it can be shown [], CRB σ 2 is a linear function of σ 4.. is a linear function of σ 2. After parameter

15 5 REFERENCES [] S. M. Kay, Fundamentals of Statistical Signal Processing: Estimation Theory, Englewood Cliffs, NJ: Prentice-Hall, 993. [2] R. J. McAulay and E. M. Hofstetter, Barankin Bounds on Parameter Estimation, IEEE Trans. Inform. Theory, vol. 7, pp , Nov. 97. [3] M. G. Kendall and A. Stuart, The Advanced Theory of Statistics, vol. 2, Macmillan, New York: 979. [4] E. Jakeman and P. N. Pusey, A Model for Non-Rayleigh Sea Echo, IEEE Trans. Antennas Propagation, vol. Ap-24, pp , Nov [5] M. Shao and C.. Nikias, Signal processing with fractional lower order moments: stable processes and their applications, Proc. IEEE, vol. 8, pp , July [6] D. Middleton, Canonical and Quasi-Canonical Probability Models of Class A Interference, IEEE Trans. Electromagnetic Compat., EMC-25, pp , May [7] S. M. Kay, Modern Spectral Estimation: Theory and Application, Englewood Cliffs, NJ: Prentice-Hall, 988. [8] S. M. Kay, Intuitive Probability and Random Processes using MATAB, New York: Springer, [9] H.. V. Trees, Detection Estimation and Modulation Theory, Part I, New York: Wiley, 968. [0] D. A. Abraham and A. P. yons, Novel physical interpretations of K-distributed reverberation, IEEE Journal of Oceanic Engineering, vol. 27, no.4, pp , Oct [] A. P. yons and D. A. Abraham, Statistical characterization of high frequency shallow-water sea-floor backscatter, J. Acoust. Soc. Amer., vol. 06, no 3, pp , 999. [2] K. D. Ward, Compound representation of high resolution sea clutter, Electron. ett., vol. 7, no.6, pp , Oct