Variations. ECE 6540, Lecture 10 Maximum Likelihood Estimation

Transcription

1 Variations ECE 6540, Lecture 10

2 Last Time BLUE (Best Linear Unbiased Estimator) Formulation Advantages Disadvantages 2

3 The BLUE A simplification Assume the estimator is a linear system For a single parameter estimator For a multi parameter estimator 3

4 The BLUE A simplification Assume the estimator is a linear system For a single parameter estimator For a multi parameter estimator Question: What is the variance / covariance of each of these for any random variable? 4

5 The BLUE A simplification Assume the estimator is a linear system For a single parameter estimator For a multi parameter estimator Question: What is the variance / covariance of each of these for any random variable? 5

6 The BLUE A simplification Assume the estimator is a linear system For a single parameter estimator (assuming an unbiased estimator) var Covariance matrix 6

7 The BLUE A simplification Assume the estimator is a linear system For a single parameter estimator (assuming an unbiased estimator) 1 Note that it s not necessarily true that for all Unbiased assumption 7

8 The BLUE A simplification Assume the estimator is a linear system For a single parameter estimator (assuming an unbiased estimator) arg min subject to 1 8

9 The BLUE A simplification Assume the estimator is a linear system For a single parameter estimator (assuming an unbiased estimator) arg min 1 Lagrange Multiplier 9

10 The BLUE A simplification Assume the estimator is a linear system For a single parameter estimator (assuming an unbiased estimator) arg min

11 The BLUE A simplification Assume the estimator is a linear system For a single parameter estimator (assuming an unbiased estimator) The Estimator: The Expected Value: E The Variance: var E 11

12 The BLUE Compute the BLUE for such that, is white noise with variance (not necessarily Gaussian) Question: What is BLUE for the above problem? 12

13 The BLUE Compute the BLUE for such that, is white noise with variance (not necessarily Gaussian) Question: What is BLUE for the above problem? The Estimator: The Expected Value: E Determination of The Variance: var 13

14 The BLUE Compute the BLUE for such that, is white noise with variance (not necessarily Gaussian) 1 The Estimator: The Expected Value: E The Variance: var E Question: Where have we seen this before? 14

15 The BLUE A simplification Assume the estimator is a linear system For a single parameter estimator For a multi parameter estimator Question: What is the variance / covariance of each of these for any random variable? 15

16 The BLUE A simplification Assume the estimator is a linear system For a multi parameter estimator (assuming an unbiased estimator) cov Covariance matrix 16

17 The BLUE A simplification Assume the estimator is a linear system For a multi parameter estimator (assuming an unbiased estimator) What is the estimator? What is the expected value of the estimator? What is the variance of the estimator? 17

18 The BLUE A simplification Assume the estimator is a linear system For a multi parameter estimator (assuming an unbiased estimator) The S matrix: The Estimator: The Expected Value: E E The Variance: cov Question: Where have we seen this before? 18

19 The BLUE Gauss Markov Theorem Consider the following expression with unknown and known, is noise with covariance (any distribution) Then the best linear unbiased estimator (BLUE) is The PDF of w is arbitrary. (note: this is not necessarily the MVU estimate). Question: Why is this powerful? 19

20 The BLUE What are the advantages of BLUE? What are the disadvantages of BLUE? 20

21 What have we learned about so far? 21

22 Minimum Variance Unbiased Estimators (MVU Estimators) Neyman Fischer Factorization & Rao Blackwell Lehmann Scheffé Cramer Rao Lower Bounds Approximate MVU Estimators BLUE 22

23 Consider this problem (DC Offset problem): is white, Gaussian noise with variances 1.Estimate. What is the MVU Estimator? What is the BLUE? 23

24 Consider this problem (DC Offset problem): is white, Gaussian noise with variances 1.Estimate. What is the MVU Estimator? What is the BLUE? 24

25 Consider this problem ( Gain problem): is white, Gaussian noise with variances 1.Estimate. What is the MVU Estimator? What is the BLUE? 25

26 Consider this problem ( Gain problem): is white, Gaussian noise with variances 1.Estimate. What is the MVU Estimator? What is the BLUE? 26

27 Consider this problem ( Gain problem): is white, Gaussian noise with variances 1.Estimate. S vector:???? The Estimator: The Expected Value: E The Variance: var 27

28 Consider this problem ( Gain problem): is white, Gaussian noise with variances 1.Estimate. S vector: 1 The Estimator: The Expected Value: E The Variance: var Result: Same as the DC Offset problem [suboptimal] 28

29 Consider this problem ( Gain problem): is white, Gaussian noise with variances 1.Estimate. What is the MVU Estimator? What is the BLUE? 29

30 Consider this problem ( Gain problem) [slightly different than problem in the book]: is white, Gaussian noise with variances 1.Estimate. Neyman Fischer Approach: Find a sufficient statistic. ; / /exp 2 30

31 Consider this problem ( Gain problem) [slightly different than problem in the book]: is white, Gaussian noise with variances 1.Estimate. Neyman Fischer Approach: Find a sufficient statistic. 1 1 ; 2 / /exp ; 2 / /exp 2 2 ; 1 2 / /exp ; 1 2 / /exp ; ; 31

32 Consider this problem ( Gain problem) [slightly different than problem in the book]: is white, Gaussian noise with variances 1.Estimate. Neyman Fischer Approach: Find a sufficient statistic. Is this a complete, sufficient statistic? Sufficient Statistic: 32

33 Consider this problem ( Gain problem) [slightly different than problem in the book]: is white, Gaussian noise with variances 1.Estimate. Neyman Fischer Approach: Find a sufficient statistic. Sufficient Statistic: Is this a complete, sufficient statistic? Can I find an unbiased estimator using this sufficient statistic? 33

34 Consider this problem ( Gain problem) [slightly different than problem in the book]: is white, Gaussian noise with variances 1.Estimate. Neyman Fischer Approach: Find a sufficient statistic. Sufficient Statistic: Is this a complete, sufficient statistic? Can I find an unbiased estimator using this sufficient statistic? Can apply for any unbiased? 34

35 Consider this problem ( Gain problem) [slightly different than problem in the book]: is white, Gaussian noise with variances 1.Estimate. Neyman Fischer Approach: Find a sufficient statistic. Sufficient Statistic: Is this a complete, sufficient statistic? Can I find an unbiased estimator using this sufficient statistic? Can apply for any unbiased? 35

36 Consider this problem ( Gain problem) [slightly different than problem in the book]: is white, Gaussian noise with variances 1.Estimate. Cramer Rao Lower Bound Approach: Find the Cramer Rao lower bound. ; / /exp 2 36

37 Consider this problem ( Gain problem) [slightly different than problem in the book]: is white, Gaussian noise with variances 1.Estimate. Cramer Rao Lower Bound Approach: Find the Cramer Rao lower bound. ; /exp / ln ; ln 2 ln ln ; ln 2 ln ; ; 37

38 Consider this problem ( Gain problem) [slightly different than problem in the book]: is white, Gaussian noise with variances 1.Estimate. Cramer Rao Lower Bound Approach: Find the Cramer Rao lower bound. ln ;

39 Consider this problem ( Gain problem) [slightly different than problem in the book]: is white, Gaussian noise with variances 1.Estimate. Cramer Rao Lower Bound Approach: Find the Cramer Rao lower bound. ln ; var 1 3 Potentially up to 3 times improvement over the BLUE 39

40 Consider this problem ( Gain problem) [slightly different than problem in the book]: is white, Gaussian noise with variances 1.Estimate. Cramer Rao Lower Bound Approach: Find the MVU Estimator. ;? I Is it possible to find? 40

41 Consider this problem ( Gain problem) [slightly different than problem in the book]: is white, Gaussian noise with variances 1.Estimate. Cramer Rao Lower Bound Approach: Find the MVU Estimator. ;? I Is it possible to find? 41

42 Where do we go now!? 42

43 Where do we go now!? We go to The most used classical statistical estimator 43

44 argmax ; Note that this is equivalent to argmax ln ; Since ln is a monotonically increasing function. Question: How do I interpret this? Why is this useful? 44

45 Consider this problem ( Gain problem) [slightly different than problem in the book]: is white, Gaussian noise with variances 1.Estimate. Maximum Likelihood Approach ; /exp / ln ; ln 2 ln ln ; ln 2 ln ; 0 45

46 Consider this problem ( Gain problem) [slightly different than problem in the book]: is white, Gaussian noise with variances 1.Estimate. Maximum Likelihood Approach Choose + to keep answer positive

47 Consider this problem ( Gain problem) [slightly different than problem in the book]: is white, Gaussian noise with variances 1.Estimate. Maximum Likelihood Approach 2 2 Is this estimator unbiased? Is this estimator minimum variance? 47

48 Consider this problem ( Gain problem) [slightly different than problem in the book]: is white, Gaussian noise with variances 1.Estimate. Maximum Likelihood Approach 2 2 Is this estimator unbiased? Nope Is this estimator minimum variance? Nope 48

49 Consider this problem ( Gain problem) [slightly different than problem in the book]: is white, Gaussian noise with variances 1.Estimate. Maximum Likelihood Approach Estimator variance Cramer-Rao Lower Bound BLUE variance 9 Variance Estimator mean True value Number of samples [N] Number of samples [N] 49

50 Properties of the MLE 50

51 Properties of Maximum Likelihood Estimators Property 1) The Maximum Likelihood Estimator will always be a function of only sufficient statistics Proof: argmax argmax argmax argmax argmax ; ln ; ln ; ln ; ln ln ; 51

52 Properties of Maximum Likelihood Estimators Property 2) Assuming the derivative of the log likelihood exists and the Fisher Information is non zero, the Maximum Likelihood Estimator converges to as In other words: E This property is known as consistency 52

53 Properties of Maximum Likelihood Estimators Property 3) Assuming the derivative of the log likelihood exists and the Fisher Information is non zero, the Maximum Likelihood Estimator converges to an efficient estimator as In other words: var This property is known as asymptotic efficiency 53

54 Properties of Maximum Likelihood Estimators Property 4) Assuming the derivative of the log likelihood exists and the Fisher Information is non zero, the Maximum Likelihood Estimator converges to a normal distribution as In other words: ~, 54

55 Properties of Maximum Likelihood Estimators Property 5) Assume is the MLE for. Now consider the one to one parameter transformation,. The MLE for the new parameter is now defined by Of the transformation is not one to one (i.e., invertible), then arg max : ; This property is known as invariance 55

56 Multi parameter MLE 56

57 argmax ; Note that this is equivalent to argmax ln ; Since ln is a monotonically increasing function. 57

58 Consider the DC offset problem is white, Gaussian noise with variances.estimate and. Maximum Likelihood Approach ; /exp / 58

59 Consider the DC offset problem is white, Gaussian noise with variances.estimate and. Maximum Likelihood Approach ; /exp / ln ; ln 2 ln ln ; ln 2 ln 2 ;

60 Consider the DC offset problem is white, Gaussian noise with variances.estimate and. Maximum Likelihood Approach ; /exp / ln ; ln 2 ln ; 0 0 let 1 60

61 Consider the DC offset problem is white, Gaussian noise with variances.estimate and. Maximum Likelihood Approach 1 Question: Is this different than the MVU estimator? 61

62 Consider the DC offset problem is white, Gaussian noise with variances.estimate exp. 62

63 Consider the DC offset problem is white, Gaussian noise with variances.estimate exp. Maximum Likelihood Approach 1 exp exp 63

64 Consider the energy estimation problem is white, Gaussian noise with variances. Estimate the signal. Maximum Likelihood Approach ; /exp / 64

65 Consider the energy estimation problem is white, Gaussian noise with variances. Estimate the signal. Maximum Likelihood Approach ; /exp / ln ; ln 2 ln ln ; ln 2 ln 2 ;

66 Consider the energy estimation problem is white, Gaussian noise with variances. Estimate the signal. Maximum Likelihood Approach ; /exp / 66

67 Consider the energy estimation problem is white, Gaussian noise with variances. Estimate the signal. Maximum Likelihood Approach ; /exp / ln ; ln 2 ln ln ; ln 2 ln 2 ;