Estimation Theory Fredrik Rusek. Chapters 6-7

Transcription

1 Estimation Theory Fredrik Rusek Chapters 6-7

2 All estimation problems Summary

3 All estimation problems Summary Efficient estimator exists

4 All estimation problems Summary MVU estimator exists Efficient estimator exists

5 All estimation problems Summary MVU estimator exists Efficient estimator exists Linear signal model, Gaussian noise

6 All estimation problems Summary MVU estimator exists Efficient estimator exists Linear signal model, Gaussian noise CRLB exists

7 All estimation problems Summary MVU estimator exists Efficient estimator exists CRLB exists Linear signal model, Gaussian noise Today we will place the following piece: Linear signal in non-gaussian noise

8 All estimation problems Summary MVU estimator exists Efficient estimator exists CRLB exists Linear signal model, Gaussian noise Chapter 6 deals with non-gaussian noise This is not very well pointed out Today we will place the following piece: Linear signal in non-gaussian noise

9 Chapter 6 Best linear unbiased estimators Definition of the BLUE Linear in received data Many possible estimators possible. The BLUE is the one that is: Unbiased Smallest possible variance over all {a n }

10 Chapter 6 Best linear unbiased estimators Definition of the BLUE Linear in received data Many possible estimators possible. The BLUE is the one that is: Unbiased Smallest possible variance over all {a n } Optimal when MVU is linear in the data Suboptimal otherwise

11 Chapter 6 Best linear unbiased estimators Definition of the BLUE Linear in received data Examples: DC level in white noise MVU: = BLUE Many possible estimators possible. The BLUE is the one that is: Unbiased Smallest possible variance over all {a n } Optimal when MVU is linear in the data Suboptimal otherwise

12 Chapter 6 Best linear unbiased estimators Definition of the BLUE Linear in received data Examples: DC level in white noise MVU: = BLUE Many possible estimators possible. The BLUE is the one that is: Unbiased Smallest possible variance over all {a n } German tank problem MVU: Optimal when MVU is linear in the data Suboptimal otherwise BLUE: (can be shown) BLUE MVU

13 Chapter 6 Best linear unbiased estimators Definition of the BLUE Linear in received data Examples: DC level in white noise MVU: = BLUE Many possible estimators possible. The BLUE is the one that is: Unbiased Smallest possible variance over all {a n } German tank problem MVU: Optimal when MVU is linear in the data Suboptimal otherwise BLUE: (can be shown) BLUE MVU The MVU is often not linear, hence the name best linear

14 Chapter 6 Best linear unbiased estimators The MVU is often not linear, hence the name best linear

15 Chapter 6 Best linear unbiased estimators Definition of the BLUE Sometimes LE is very bad Noise power estimation (x[n]=w[n]) MVU BLUE

16 Chapter 6 Best linear unbiased estimators Definition of the BLUE Sometimes LE is very bad Noise power estimation (x[n]=w[n]) MVU BLUE BLUE + cleverness Transform data as y[n] = x 2 [n] Apply BLUE to y[n]

17 Chapter 6 Best linear unbiased estimators Finding the BLUE Constraint

18 Chapter 6 Best linear unbiased estimators Finding the BLUE Constraint Variance

19 Chapter 6 Best linear unbiased estimators Finding the BLUE Constraint Variance

20 Chapter 6 Best linear unbiased estimators Finding the BLUE Constraint Variance Use vector notation

21 Chapter 6 Best linear unbiased estimators Finding the BLUE Constraint Variance Use vector notation

22 Chapter 6 Best linear unbiased estimators Finding the BLUE Constraint Variance

23 Chapter 6 Best linear unbiased estimators Finding the BLUE Constraint Variance

24 Chapter 6 Best linear unbiased estimators Finding the BLUE Constraint Variance

25 Chapter 6 Best linear unbiased estimators Finding the BLUE Constraint Optimization problem Variance

26 Chapter 6 Best linear unbiased estimators Finding the BLUE Constraint Optimization problem To satisfy we must have

27 Chapter 6 Best linear unbiased estimators Finding the BLUE Constraint Optimization problem To satisfy we must have Now write

28 Chapter 6 Best linear unbiased estimators Finding the BLUE Constraint Optimization problem To satisfy we must have Now write Define

29 Chapter 6 Best linear unbiased estimators Finding the BLUE Constraint Optimization problem To satisfy we must have Now write Define Which yields w[n] not Gaussian, but zero-mean This is the linear model, but with non-gaussian noise

30 Chapter 6 Best linear unbiased estimators Finding the BLUE Constraint Optimization problem To satisfy we must have Now write We have Define Which yields

31 Chapter 6 Best linear unbiased estimators Finding the BLUE Constraint Optimization problem To satisfy we must have Now write We have Define Which yields

32 Chapter 6 Best linear unbiased estimators Finding the BLUE Lagrangian Optimization problem

33 Chapter 6 Best linear unbiased estimators Finding the BLUE Lagrangian Optimization problem

34 Chapter 6 Best linear unbiased estimators Finding the BLUE Lagrangian Optimization problem To find λ, use the constraint function with

35 Chapter 6 Best linear unbiased estimators Finding the BLUE Lagrangian Optimization problem To find λ, use the constraint function with

36 Chapter 6 Best linear unbiased estimators Finding the BLUE Lagrangian Optimization problem To find λ, use the constraint function with Plug back:

37 Chapter 6 Best linear unbiased estimators Finding the BLUE Lagrangian Optimization problem To find λ, use the constraint function with Variance Plug back:

38 Chapter 6 Best linear unbiased estimators To compute the BLUE, we need Know that E(x) = sθ Know s Know C Connections to the MVU estimator for the linear model will soon be made

39 Chapter 6 Best linear unbiased estimators BLUE for vector parameter

40 Chapter 6 Best linear unbiased estimators BLUE for vector parameter Constraint

41 Chapter 6 Best linear unbiased estimators BLUE for vector parameter Constraint Which becomes

42 Chapter 6 Best linear unbiased estimators BLUE for vector parameter Constraint Which becomes

43 Chapter 6 Best linear unbiased estimators BLUE for vector parameter Constraint Which becomes

44 Chapter 6 Best linear unbiased estimators BLUE for vector parameter Constraint Cost function to optimize Which becomes

45 Chapter 6 Best linear unbiased estimators BLUE for vector parameter Optimal solution

46 Chapter 6 Best linear unbiased estimators BLUE for vector parameter Optimal solution Summary For a model y=hθ+n, a linear estimator has the same form no matter the distribution of the noise (for the same covariance C of it)

47 Chapter 6 Best linear unbiased estimators BLUE for vector parameter Optimal solution Summary For a model y=hθ+n, a linear estimator has the same form no matter the distribution of the noise (for the same covariance C of it) If n is Gaussian, then a linear estimator is MVU

48 Chapter 6 Best linear unbiased estimators BLUE for vector parameter Optimal solution Summary For a model y=hθ+n, a linear estimator has the same form no matter the distribution of the noise (for the same covariance C of it) If n is Gaussian, then a linear estimator is MVU If n is not Gaussian, the BLUE is the best linear estimator, but not MVU

49 Chapter 6 Best linear unbiased estimators BLUE for vector parameter Optimal solution Summary For a model y=hθ+n, a linear estimator has the same form no matter the distribution of the noise (for the same covariance C of it) If n is Gaussian, then a linear estimator is MVU If n is not Gaussian, the BLUE is the best linear estimator, but not MVU Performance of the BLUE for non-gaussian noise is identical to the performance with Gaussian noise

50 Chapter 6 Best linear unbiased estimators Est. Problem with Gaussian noise Est. Problem with Non-Gaussian noise Variance of linear estimator

51 Chapter 6 Best linear unbiased estimators Est. Problem with Gaussian noise Est. Problem with Non-Gaussian noise Variance of linear estimator

52 Chapter 6 Best linear unbiased estimators Est. Problem with Gaussian noise Est. Problem with Non-Gaussian noise Variance of linear estimator Estimator is Optimal Estimator is not optimal

53 Chapter 6 Best linear unbiased estimators Est. Problem with Gaussian noise Est. Problem with Non-Gaussian noise Variance of linear estimator Estimator is Optimal Estimator is not optimal Variance of nonlinear estimator The same?? But smaller than Therefore, Gaussian noise is the worst noise possible

54 Chapter 6 Best linear unbiased estimators Est. Problem with Gaussian noise Est. Problem with Non-Gaussian noise Variance of linear estimator Estimator is Optimal Estimator is not optimal Variance of nonlinear estimator The same?? But smaller than Alternative proofs 1. Based on calculus of the variations 2. Based on link between Fisher information and Kullback-Leibler divergence (information theory)

55 All estimation problems Summary MVU estimator exists Efficient estimator exists CRLB exists Linear signal model, Gaussian noise Where to put?? Linear signal in non-gaussian noise

56 We do not have explicit info Summary???? MVU estimator exists Efficient estimator exists CRLB exists With uniform noise, the CRLB doesn t exist For Laplace noise it does No efficient estimator seems to exist see my Italian proof Perhaps MVU does not always exist.?

57 We do not have explicit info Summary???? MVU estimator exists???? Efficient estimator exists CRLB exists With uniform noise, the CRLB doesn t exist For Laplace noise it does No efficient estimator seems to exist see my Italian proof Perhaps MVU does not always exist.?

58 Chapter 7 Maximum Likelihood arg

59 Chapter 7 Maximum Likelihood arg Theorem. If an efficient estimator exists, then it is given by the MLE

60 Chapter 7 Maximum Likelihood arg Theorem. If an efficient estimator exists, then it is given by the MLE Proof.

61 Chapter 7 Maximum Likelihood arg Theorem. If an efficient estimator exists, then it is given by the MLE Proof. ML ML

62 Chapter 7 Maximum Likelihood Example 7.3. DC level in white Gaussian noise with variance = DC-level

63 Chapter 7 Maximum Likelihood Example 7.3. DC level in white Gaussian noise with variance = DC-level Find MLE Check point 1: Try the CRLB

64 Chapter 7 Maximum Likelihood Example 7.3. DC level in white Gaussian noise with variance = DC-level Find MLE Check point 2: Try Neyman-Fisher

65 Chapter 7 Maximum Likelihood Example 7.3. DC level in white Gaussian noise with variance = DC-level Find MLE Check point 2: Try Neyman-Fisher If is complete, then an unbiased function of T(x) is MVU

66 Chapter 7 Maximum Likelihood Example 7.3. DC level in white Gaussian noise with variance = DC-level Find MLE Check point 2: Try Neyman-Fisher If is complete, then an unbiased function of T(x) is MVU Not clear what function to choose

67 Chapter 7 Maximum Likelihood Example 7.3. DC level in white Gaussian noise with variance = DC-level Find MLE No more strategies to find the MVU proceed to MLE

68 Chapter 7 Maximum Likelihood Example 7.3. DC level in white Gaussian noise with variance = DC-level Possible to show: As N, As N, Asymptotically efficient

69 Chapter 7 Maximum Likelihood Example 7.3. DC level in white Gaussian noise with variance = DC-level Possible to show: As N, As N, Asymptotically efficient We saw this notation before, when we claimed that as N grows, we can estimate a transformation of a variable as the transformation of the estimate

70 Chapter 7 Maximum Likelihood Example 7.3. DC level in white Gaussian noise with variance = DC-level Possible to show: As N, As N, Asymptotically efficient By the CLT, we have that is Gaussian as N grows. Asymptotically, one can show that the MLE is a linear transformation of this Gaussian variable. Thus, the estimator is Gaussian distributed Linearity is not easy to see, but possible..

71 Chapter 7 Maximum Likelihood Example 7.3. DC level in white Gaussian noise with variance = DC-level Possible to show: As N, As N, Asymptotically efficient By the CLT, we have that is Gaussian as N grows. Asymptotically, one can show that the MLE is a linear transformation of this Gaussian variable. Thus, the estimator is Gaussian distributed But since the variance of the estimator is given by the CRLB (asympt.),

72 Effect of N Chapter 7 Maximum Likelihood

73 Chapter 7 Maximum Likelihood

74 Chapter 7 Maximum Likelihood Proof of mean (conistency): Let θ 0 be the true value of θ

75 Chapter 7 Maximum Likelihood Proof of mean (conistency): Let θ 0 be the true value of θ By the CLT, we have

76 Chapter 7 Maximum Likelihood Proof of mean (conistency): Let θ 0 be the true value of θ By the CLT, we have

77 Chapter 7 Maximum Likelihood Proof of mean (conistency): Let θ 0 be the true value of θ By the CLT, we have Maximized for θ= θ 0

78 Chapter 7 Maximum Likelihood Proof of mean (conistency): Let θ 0 be the true value of θ By the CLT, we have Maximized for θ= θ 0 As N grows, the MLE is θ 0

79 Chapter 7 Maximum Likelihood Neyman-Fisher factorization MLE

80 Chapter 7 Maximum Likelihood Neyman-Fisher factorization MLE To find the ML, it is sufficient to optimize the function g(t(x),θ) Hence, MLE can be made on the basis of the sufficient statistic(s) only

81 Transformation of parameters Chapter 7 Maximum Likelihood

82 Chapter 7 Maximum Likelihood Transformation of parameters Let us tie things together The MLE of a transformed parameter is the transformed MLE of the parameter

83 Chapter 7 Maximum Likelihood Transformation of parameters Let us tie things together The MLE of a transformed parameter is the transformed MLE of the parameter The MLE is efficient if an efficient estimator exists

84 Chapter 7 Maximum Likelihood Transformation of parameters Let us tie things together The MLE of a transformed parameter is the transformed MLE of the parameter The MLE is efficient if an efficient estimator exists From before: The (non-linear) transformed estimate of an efficient estimator does not preserve efficiency Not efficient efficient

85 Chapter 7 Maximum Likelihood Transformation of parameters Let us tie things together The MLE of a transformed parameter is the transformed MLE of the parameter The MLE is efficient if an efficient estimator exists From before: The (non-linear) transformed estimate of an efficient estimator does not preserve efficiency efficient Not efficient This argument didn t say that an efficient estimator for α does not exist

86 Chapter 7 Maximum Likelihood Transformation of parameters Let us tie things together The MLE of a transformed parameter is the transformed MLE of the parameter The MLE is efficient if an efficient estimator exists From before: The (non-linear) transformed estimate of an efficient estimator does not preserve efficiency efficient Not efficient This did not say that an efficient estimator for α does not exist We can deduce: A non-linear function of a parameter that can be efficiently estimated cannot be efficiently estimated efficient Not efficient (but would have been if an efficient existed since it is the transformed MLE)

87 Chapter 7 Maximum Likelihood Transformation of parameters Let us tie things together The MLE of a transformed parameter is the transformed MLE of the parameter The MLE is efficient if an efficient estimator exists From before: The (non-linear) transformed estimate of an efficient estimator does not preserve efficiency efficient Not efficient This did not say that an efficient estimator for α does not exist We can deduce: A non-linear function of a parameter that can be efficiently estimated cannot be efficiently estimated Due to previous linearization argument, the transformed estimate is asymptotically efficient.

88 Chapter 7 Maximum Likelihood Transformation of parameters Let us tie things together The MLE of a transformed parameter is the transformed MLE of the parameter The MLE is efficient if an efficient estimator exists From before: The (non-linear) transformed estimate of an efficient estimator does not preserve efficiency efficient Not efficient This did not say that an efficient estimator for α does not exist We can deduce: A non-linear function of a parameter that can be efficiently estimated cannot be efficiently estimated Due to previous linearization argument, the transformed estimate is asymptotically efficient. This could also have been realized by the observation that transformed estimate = MLE = asymptotically efficient

89 Chapter 7 Maximum Likelihood Transformation of parameters What is this?

90 Chapter 7 Maximum Likelihood Transformation of parameters Consider the case For some value of α, there are two values of A that produces α, The likelihood of α, is the largest of the two likelihoods

91 Chapter 7 Maximum Likelihood

92 MLE (linear) Chapter 7 Maximum Likelihood

93 Chapter 7 Maximum Likelihood MLE (linear) MLE (db) due to invariance

94 Chapter 7 Maximum Likelihood Efficient? Meaningful to compare with CRLB? Efficient (Var = 2σ 4 /N=CRLB) MLE (linear) MLE (db) due to invariance

95 Chapter 7 Maximum Likelihood Efficient? No! Asymptotically: Yes! Meaningful to compare with CRLB? No! Not even unbiased Meets CRLB asymptotically Efficient (Var = 2σ 4 /N=CRLB) MLE (linear) MLE (db) due to invariance

96 Chapter 7 Maximum Likelihood Set N=10. Let us anyway check the CRLB for the db case: Measured variance (Matlab): 4.17 Var=2.21xCRLB Efficient? No! Asymptotically: Yes! Meaningful to compare with CRLB? No! Not even unbiased Meets CRLB asymptotically Efficient (Var = 2σ 4 /N=CRLB) MLE (linear) MLE (db) due to invariance

97 Chapter 7 Maximum Likelihood Set N=50. Let us anyway check the CRLB for the db case: Measured variance (Matlab): 0.77 Var=2.04xCRLB Efficient? No! Asymptotically: Yes! Meaningful to compare with CRLB? No! Not even unbiased Meets CRLB asymptotically Efficient (Var = 2σ 4 /N=CRLB) MLE (linear) MLE (db) due to invariance

98 Chapter 7 Maximum Likelihood Set N=150. Let us anyway check the CRLB for the db case: Measured variance (Matlab): 0.25 Var=2xCRLB Efficient? No! Asymptotically: Yes! Meaningful to compare with CRLB? No! Not even unbiased Meets CRLB asymptotically Efficient (Var = 2σ 4 /N=CRLB) MLE (linear) MLE (db) due to invariance

99 Chapter 7 Maximum Likelihood Extension to vector parameter: Straightforward

100 Chapter 7 Maximum Likelihood MLE for linear model in Gaussian noise With no derivations: How should we argue in order to establish the MLE?

101 Chapter 7 Maximum Likelihood MLE for linear model in Gaussian noise With no derivations: How should we argue in order to establish the MLE? In this case a linear estimator was optimal (Chapter 4)

102 Chapter 7 Maximum Likelihood MLE for linear model in Gaussian noise With no derivations: How should we argue in order to establish the MLE? In this case a linear estimator was optimal (Chapter 4) This linear estimator was also efficient

103 Chapter 7 Maximum Likelihood MLE for linear model in Gaussian noise With no derivations: How should we argue in order to establish the MLE? In this case a linear estimator was optimal (Chapter 4) This linear estimator was also efficient An efficient estimator is always the MLE

104 Chapter 7 Maximum Likelihood MLE for linear model in Gaussian noise With no derivations: How should we argue in order to establish the MLE? In this case a linear estimator was optimal (Chapter 4) This linear estimator was also efficient An efficient estimator is always the MLE MLE is the linear estimator from Chapter 4

105 Chapter 7 Maximum Likelihood Asymptotic MLE: some intuition With a stationary Gaussian process, the log-likelihood becomes Appears strange, but is not if one knows his linear algebra + Szegö s Theorem

106 Chapter 7 Maximum Likelihood Asymptotic MLE: some intuition Covariance matrix is Toeplitz Elements of C are autocorrelation values of the process (of course) Asymptotic Toeplitz matrix admits an Eigenvalue factorization as C=QSQ* (Q=DFT) PSD is the Fourier transform of the autocorrelation sequence (e.g. from the Proakis course) not exactly Szegö s Thm, but a consequence thereof: Asymptotically and very loosely speaking, the Eigenvalues of C, i.e. S, converges to Fourier transform of the autocorrelation sequence.

107 Chapter 7 Maximum Likelihood Asymptotic MLE: some intuition Covariance matrix is Toeplitz Elements of C are autocorrelation values of the process (of course) Asymptotic Toeplitz matrix admits an Eigenvalue factorization as C=QSQ* (Q=DFT) PSD is the Fourier transform of the autocorrelation sequence (e.g. from the Proakis course) not exactly Szegö s Thm, but a consequence thereof: Asymptotically and very loosely speaking, the Eigenvalues of C, i.e. S, converges to Fourier transform of the autocorrelation sequence. Consider now log det(c) det(c) is the product of eigenvalues So, log det(c) is the sum of the logarithm of the eigenvalues But: eigenvalues = Fourier transform of autocorrelation = PSD

108 Chapter 7 Maximum Likelihood Asymptotic MLE: some intuition Covariance matrix is Toeplitz Elements of C are autocorrelation values of the process (of course) Asymptotic Toeplitz matrix admits an Eigenvalue factorization as C=QSQ* (Q=DFT) PSD is the Fourier transform of the autocorrelation sequence (e.g. from the Proakis course) not exactly Szegö s Thm, but a consequence thereof: Asymptotically and very loosely speaking, the Eigenvalues of C, i.e. S, converges to Fourier transform of the autocorrelation sequence. Consider now log det(c) det(c) is the product of eigenvalues So, log det(c) is the sum of the logarithm of the eigenvalues But: eigenvalues = Fourier transform of autocorrelation = PSD

109 Chapter 7 Maximum Likelihood Asymptotic MLE: some intuition Covariance matrix is Toeplitz Elements of C are autocorrelation values of the process (of course) Asymptotic Toeplitz matrix admits an Eigenvalue factorization as C=QSQ* (Q=DFT) PSD is the Fourier transform of the autocorrelation sequence (e.g. from the Proakis course) not exactly Szegö s Thm, but a consequence thereof: Asymptotically and very loosely speaking, the Eigenvalues of C, i.e. S, converges to Fourier transform of the autocorrelation sequence. Consider now -x T C -1 x x T C -1 x = x T QS -1 Q*x Q*x is the Fourier transform of x S -1 is one divided with the PSD x T Q together with Q*x is the periodogram of x