Motivation CRLB and Regularity Conditions CRLLB A New Bound Examples Conclusions

Transcription

1 A Survey of Some Recent Results on the CRLB for Parameter Estimation and its Extension Yaakov 1 1 Electrical and Computer Engineering Department University of Connecticut, Storrs, CT A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 1 / 30

2 Motivation Is the Cramèr-Rao Lower Bound (CRLB) valid for more relaxed regularity conditions than are generally cited in the literature? Yes. Specifically, the integrability of the first two derivatives of the log-likelihood function (LLF) is not necessary, and it is also not necessary for the support of the likelihood function (LF) to be independent of the parameter to be estimated. Is there a bound for a LF with parameter dependent support (i.e., measurement noise with a finite support pdf where it is nonzero)? Yes: the Cramér-Rao-Leibniz Lower Bound (CRLLB) for likelihood functions with parameter-dependent support A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 2 / 30

3 CRLB and Regularity Conditions (i) (i ) (i ) (ii) (ii ) Table 1: Regularity Conditions for the CRLB/CRLLB LF Properties existence and absolute integrability of the first two derivatives of the LF existence and absolute integrability of the first two derivatives λ 1 (x; z) and of λ 2 (x; z) of the LLF existence (finiteness) of the expected values of the square of λ 1 (x; z) and of λ 2 (x; z) (the FI) the support of the LF be independent of x continuity of the LF at the boundary of its support (the pdf of the measurement noise is zero at the boundary of its support) discontinuity of the LF at the boundary of its support Comments Commonly cited, but not sufficient for CRLB (inverted parabola) Not necessary for CRLB (raised cosine) Combined with (ii ) N & S for CRLB Not necessary for CRLB (raised cosine) Combined with (i ) N & S for CRLB Existence of FIM E{λ 2 1 (x; z)} is N & S for CRLLB A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 3 / 30

4 Classical CRLB The classical CRLB states that the variance of an unbiased estimator ˆx(z) of a real-valued (non-random) parameter x has the following lower bound E [ (ˆx(z) x) 2] J(x) 1 = J 1 (x) 1 = { E [ λ1 (x; z) 2]} 1 = J 2 (x) 1 = {E [λ2 (x; z)]} 1 (1) where the expectations are over z, J denotes the Fisher Information (FI) and λ 1 (x; z) = ln p(z x) x λ 2 (x; z) = 2 ln p(z x) x 2 (3) are the first and second derivatives of the LLF. (2) A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 4 / 30

5 Regularity Conditions The generally held regularity conditions for the CRLB to hold are (i ) and (ii) from Table 1. (i ) [integrability of LLF derivatives] is too stringent and should be replaced by (i ) existence (finiteness) of the expected values of λ 2 1 and of λ 2 (the FI). (ii) [parameter-independent LF support] is also too stringent and should be replaced by (ii ) continuity of the LF at the boundary of its support. Both cases will be illustrated later through specific examples. Parameter-dependent support of the LF arises when an unknown parameter is observed in the presence of additive measurement noise and the measurement noise pdf has a finite support. A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 5 / 30

6 Continuity of the LF Regularity Condition The CRLB derivation starts by expressing the unbiasedness property of the estimator as follows E [(ˆx(z) x)] = h(x) g(x) (ˆx(z) x)p(z x) dz = 0 (4) where p(z x) is the LF, and the integral is over the support of the LF. Differentiating the above w.r.t. x with Leibniz integral rule, necessitated by the dependence of the support on x, yields D(x) = d dx h(x) g(x) [ˆx(z) x]p(z x) dz x = D L (x) + D I (x) = 0 x (5) A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 6 / 30

7 Continuity of the LF Regularity Condition (cont.) The first term of (5) the extra Leibniz term is where D L (x) = D L1 (x) + D L2 (x) (6) D L1 (x) = dh(x) [ˆx(z) x]p(z x) dx (7) z=h(x) D L2 (x) = dg(x) [ˆx(z) x]p(z x) dx (8) z=g(x) The second term of (5) D I (x) = h(x) g(x) p(z x) dz + h(x) g(x) [ˆx(z) x] p(z x) dz (9) x comprises the terms resulting from the interchange of the differentiation and integration. A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 7 / 30

8 Interchangibility of Differentiation and Integration If we have D L (x) = 0, then D(x) = D I (x) = 0, and we have interchangeability of the differentiation w.r.t. x and integration w.r.t. z in (5). The classical derivation of the CRLB then follows. This interchangeability is equivalent to the extra Leibniz terms either being zero or summing up to zero. For a general unbiased estimator, these do not sum to zero since this would require: the product of the derivatives of the integration limits and the LF at the limits to be the same (which is possible), and that the estimate be the same at the upper and lower limits of z (which is not possible). The only other possibility for the above interchangeability to hold is if the LF is zero at the boundary of its support i.e., if the LF is continuous (in z) at the boundary of its support. A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 8 / 30

9 CRLLB The New Bound (2014) If the extra Leibniz terms are not zero, i.e., D L (x) 0, then, using (5) and (9), we have h(x) g(x) [ˆx(z) x] p(z x) dz = 1 D L (x) (10) x Following the steps of the classical CRLB derivation yields the CRLLB E [ (ˆx(z) x) 2] [1 D L(x)] 2 J 1 (11) where the FI is J 1 (x) = E [ { [ λ 1 (x; z) 2] ] } ln p(z x) 2 = E x (12) Note that the two forms of the FI are no longer equal, and as such the CRLLB is expressed solely in terms of J 1. A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 9 / 30

10 Remarks about the CRLLB The achievability of this new bound statistical efficiency of the estimator is subject to the same collinearity condition as for the classical CRLB, i.e., ln p(z x) = J 1 (x)[ˆx(z) x] x (13) x This is equivalent to stating that the LF should belong to the exponential family (Necessary). It should be noted that the CRLLB (11), unlike the CRLB for unbiased estimators, is not independent of the estimator. In view of (6) (8) the CRLLB depends on the estimator s values at the boundary of the support of the LF. This seems to be unavoidable, similarly to the situation of the CRLB for biased estimators. A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 10 / 30

11 The Curious Case of the Inverted-Parabola LF Support p(w) w Figure 1: Measurement noise pdf p(w) = 3 [ 4a a 2 3 w 2], a = 2 (limited in magnitude, with a centrality tendency) A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 11 / 30

12 The Curious Case of the Inverted-Parabola LF Support Consider the measurement z = x + w where the measurement noise has the inverted parabola pdf with finite-interval support The LF of x is in this case p(w) = 3 4a 3 [ a 2 w 2] (14) w [ a, +a] (15) p(z x) = 3 4a 3 [ a 2 (z x) 2], z [x a, x + a] (16) centered at x, i.e., parameter dependent. It is easy to verify that this LF satisfies (i) [integrability of LF derivatives] and (ii ) [continuity of the LF at the boundary of its support], so one would expect the CRLB to hold. A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 12 / 30

13 Inverted-Parabola Example (cont.) The first derivative of the LLF is x a λ 1 (z x) = 2(z x) a 2 (z x) 2 (17) The first form of the FI is x+a [ ] 2(z x) 2 3 [ J 1 (x) = a 2 a 2 (z x) 2 4a 3 (z x) 2] dz Rewriting the above, one obtains J 1 (x) = 3 a 3 a a w 2 (18) a 2 dw = (19) w2 Since this integral is infinite, the CRLB is zero, i.e., meaningless in this case. A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 13 / 30

14 Inverted-Parabola Example (cont.) For the inverted-parabola case, (i) [the derivatives of the LF w.r.t. x are integrable w.r.t. z] holds but (i ) [finite expected values of λ 2 1 and λ 2] does not and there is no meaningful CRLB. The actual variance of the ML estimator in this case can be easily shown to be ˆx ML (z) = z (20) var[ˆx ML (z)] = a2 5 The above discussion points to the fact that requirement (i) on the LF is not sufficient for the CRLB and should be replaced by (i ) see above. (21) A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 14 / 30

15 Raised Cosine Likelihood Function p(w) w Figure 2: Measurement noise pdf p(w) = 1 2a magnitude, with a centrality tendency) [ 1 + cos π a w], a = π (limited in A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 15 / 30

16 Raised Cosine Likelihood Function In this case, the LF of x is p(z x) = 1 [1 + cos π ] 2a a (z x), z [x a, x + a] (22) This model does not satisfy (i ) [integrability of LLF derivatives] (because of ln 0) (ii) [parameter independent LF support] This model does satisfy (i ) [existence of expected values of λ 2 1 and λ 2 ] (ii ) [continuity of LF at boundary of its support] This model, therefore, should have a valid CRLB It can be easily shown that, for a single measurement, the ML estimator of x is unbiased and given by ˆx ML (z) = z (23) A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 16 / 30

17 Raised Cosine Likelihood Function CRLB The variance of (23) is (for a = π, for simplicity and with no loss of generality) x+π var[ˆx ML (z)] = 1 (z x) 2 [1 + cos(z x)] dz 2π x π = 1 [ 2π 3 /3 4π ] = π 2 /3 2 = (24) 2π The FI for (22) is given by J = 1 x+π [ ] sin(z x) 2 [1 + cos(z x)] dz 2π x π 1 + cos(z x) = 1 π [1 cos w] dw = 1 (25) 2π π The CRLB is therefore valid J 1 = 1 < var[ˆx ML (z)] = (26) A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 17 / 30

18 Uniform LF Centered at the Unknown Parameter p(w) w Figure 3: Measurement noise pdf p(w) = 1 2a, a = 2 A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 18 / 30

19 Uniform LF Centered at the Unknown Parameter Consider the measurement with uniformly distributed noise over w [ a, a] which yields the uniform LF p(z x) = 1 2a with parameter-dependent support (27) z [x a, x + a] (28) The unbiased estimator ˆx(z) = z has variance var[ˆx(z)] = (2a) 2 /12 = a 2 /3 (29) The naïve evaluation of the FI yields zero, and thus it appears that J = 0, i.e., the lower bound (1) is infinite. The reason for this apparent failure of the CRLB is that the LF (27) violates the continuity requirement (ii ) by being discontinuous at the boundary of its support. A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 19 / 30

20 The Raised Fractional Cosine LF p(w) w Figure 4: Measurement noise pdf p(w) = 1 2a [ 1 + β cos π a w], a = π, β = 0.5 A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 20 / 30

21 The Raised Fractional Cosine LF Consider a measurement which follows the raised fractional cosine pdf. The LF is (with 0 < β < 1) p(z x) = 1 [1 + β cos π ] 2a a (z x) (30) with support z [x a, x + a] (31) Clearly, the LF (30) does not satisfy (ii ) because it is discontinuous at the boundary of its support. Thus the classical CRLB does not hold for this LF. The unbiased ML estimator in this case is ˆx ML (z) = z Its variance (setting again a = π for simplicity), is which yields, e.g., for β = 0.5, var[ˆx ML (z)] = π 2 /3 2β (32) var[ˆx ML (z)] = (33) A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 21 / 30

22 Raised Fractional Cosine CRLLB The Leibniz terms (6) are, noting that the derivatives of the integration limits w.r.t. x are unity D L (x) = (z x)p(z x) z=x+a (z x)p(z x) z=x a = 1 β (34) The FI in the present case is J = 1 2π π π The CRLLB is therefore E [ (ˆx(z) x) 2] β 2 sin 2 w 1 + β cos w dw = 1 1 β 2 (35) β β 2 = β 2 (36) For β = 0.5 the CRLLB yields the valid result var[ˆx ML (z)] = = (37) A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 22 / 30

23 The Truncated Gaussian LF p(w) w Figure 5: Measurement noise pdf p(w) = 1 w2 e 2σ c 2, σ = 1, a = 2 A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 23 / 30

24 The Truncated Gaussian LF Consider a measurement with errors from a truncated Gaussian pdf. The LF is p(z x) = 1 (z x)2 e 2σ c 2, z [x a, x + a] (38) where the normalizing constant is c = [ ( a ) ( 2πσ Φ Φ a )] (39) σ σ and Φ( ) is the standard Gaussian cdf. The unbiased ML estimator for a single measurement is ˆx ML (z) = z and its variance is x+a E { (z x) 2} = 1 (z x) 2 e (z x)2 2σ c 2 dz x a ( = σ 2 1 2a ) a2 e 2σ c 2 (40) A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 24 / 30

25 Truncated Gaussian CRLLB The FI in this case is J = E { (z x) 2} σ 4 = 1 2ac a2 (1 σ 2 e The Leibniz term is 2σ 2 ) D L (x) = (z x)p(z x) z=x+a (z x)p(z x) z=x a = 2a c (41) e a2 2σ 2 (42) Using (42), the variance of the ML estimator can be written as var[ˆx ML (z)] = E { (z x) 2} = (1 D L )σ 2 (43) A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 25 / 30

26 Truncated Gaussian CRLLB (cont.) The standard CRLB is invalid larger than var[ˆx ML (z)] CRLB : var[ˆx(z)] σ2 1 D L > var[ˆx ML (z)] = (1 D L )σ 2 The CRLLB is met exactly! (44) CRLLB : var[ˆx(z)] (1 D L )σ 2 = var[ˆx ML (z)] (45) The variance (43) of the estimator is equal to the CRLLB, therefore, this estimator is statistically efficient. It should also be noted that comparison to the CRLB would lead to an apparent super-efficiency of the ML estimator for this finite-support LF. This is due to the CRLB being invalid in this case When compared to the CRLLB (which is lower than the CRLB in this case), the ML estimator is found to be efficient (because the LF is exponential). A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 26 / 30

27 Uniform LF as a Limit of the Truncated Gaussian p(w) σ = 1 σ = 3 σ = w Figure 6: Truncated Gaussian pdf for a = 2 and increasing σ. A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 27 / 30

28 The Uniform LF as a Limit of the Truncated Gaussian The truncated Gaussian pdf discussed in the previous example approaches the uniform pdf in the limit. The normalizing constant can be written as c ( ) 2a 2πσ + HOT 3 (46) 2πσ where HOT 3 refers to the higher order terms (of the order of σ 3 ) of the Taylor series expansion. The limiting form of the normalizing constant is (since HOT 3 approaches zero as σ approaches infinity) lim c = lim 2a 2πσ = 2a (47) σ σ 2πσ and the pdf, therefore, becomes lim p(z x) = 1 σ 2a (48) A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 28 / 30

29 The Uniform CRLLB as a Limit of the Truncated Gaussian The CRLLB for the uniform LF should then follow as the limit of (45) with (42) var{ˆx(z)} lim (1 D L)σ 2 σ 2ac a2 = lim (1 e σ 2σ 2 ) σ 2 = (2a)2 12 = a2 3 = E{(z x)2 } (49) This resulting CRLLB matches the variance (29) of the ML estimator for the uniform pdf, demonstrating that the ML estimator is indeed efficient in this case. A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 29 / 30

30 Conclusions In summary, the classical CRLB holds for LFs with parameter-dependent support under the following conditions: (i ) existence (finiteness) of the expected value of the square of the first derivative of the log-lf and the expected value of the second derivative of the log-lf [the derivative has to exist only a.e., e.g., Laplace LF], and (ii ) continuity of LF at the boundary of its support. For LFs that satisfy (i ) but not (ii ), the recently developed CRLLB provides the bound. The CRLLB solved the longstanding problem of the inapplicability of the CRLB to the uniformly distributed measurement noise case and shattered the myth of super-efficiency (for z U[0, x]; IEEE TAES July 2014). Extension to vector parameters: the Leibniz term is an integral on the LF support boundary of an interior product. A Survey of Some Recent Results on the CRLB for Parameter Estimation 504v and its Extension 30 / 30