Detection & Estimation Lecture 1 Intro, MVUE, CRLB Xiliang Luo General Course Information Textbooks & References Fundamentals of Statistical Signal Processing: Estimation Theory/Detection Theory, Steven M. Kay, Prentice Hall, 1993. Principles of Signal Detection and Parameter Estimation, Benard C. Levy, Springer, 2008. Detection, Estimation, and Modulation Theory, Part I, Harry L. Van Trees, John Wiley & Sons, Inc., 2001. Lecturer Dr. Xiliang Luo Office hour: Tuesday, Thursday, 8:00 10:00am TA Mr. Shengda Jin Office hour: TBA Grading Homework: (Weekly) 40% (due at the beginning of each lecture) Midterm: 30% [10/24 28] Final: 30% [1/2 6] 1
General Course Information You must complete the weekly HW independently Discussions among students are allowed but solutions must be your own On the 1 st page of your HW, please copy: The works submitted herein are entirely my own. I have derived every single step shown below. (sign your name and date) General Course Information Course website http://sist.shanghaitech.edu.cn/faculty/luoxl/class/2016fall_ee251 /EE251.htm Course forum https://piazza.com/class/isyla0s68imsv 2
Estimation Radar Sonar Speech Image analysis Biomedicine Communication Control Seismology Radar 3
Sonar Estimation Problem Given a data set x[0],x[1],,x[n 1] We want to determine the value of an unknown parameter as: 0,1,,1 this function is an estimator Date back to Gauss, 1795 least squares planetary movement 4
Estimation Problem The data has to be dependent on the unknown parameter pdf: 0,,1; the semicolon denotes the dependence Example: Gaussian pdf Classical vs Bayesian Classical estimation the unknown parameter is deterministic Bayesian estimation the unknown parameter is itself random we are estimating one realization of the random parameter the data are characterized by the joint pdf, : the prior pdf 5
Estimator Performance Question: How is this estimator? find the mean,variance Best estimator? topic next Unbiased Estimator On average, the estimator should yield the true value this estimator is unbiased E,, Example: An estimator is unbiased does not mean it is a good estimator 6
Minimum Variance In order to find one optimal estimator, we need to specify the criterion one natural criterion is the Mean Square Error (MSE) Example: 1 1 / Not realizable! MVUE Minimize the variance while being unbiased Question: whether MVUE exists? unbiased estimator with minimum variance for all values of the unknown parameter Example: [Example 2.3], 1, 0 0,1 1, 2, 0 1 2 0 1 3 1 0 1 2 7
MVUE No known turn the crank procedure to produce the MVUE Next, we will discuss Cramer Rao lower bound Rao Blackwell Lehmann Scheffe theorem best linear estimator Cramer Rao Lower Bound We need to place a lower bound on the variance of any unbiased estimator! Check whether our estimator is MVUE Check how far our estimator is from the optimal one even the optimal one may not exist Tells us it is impossible to find an estimator that can beat the bound 8
Likelihood Function When the pdf is views a function of the unknown parameter, it is referred to as the likelihood function Example: 0 0 0; 1 ln 0; ln 2 1 2 ln 0; 2 1 0 CRLB Regularity condition: ln ; 0, For any unbiased estimator, we have: ln ; Furthermore, one unbiased estimator achieving the bound exists iff: ln ; is the MVUE and the min variance is given by 1/ 9
Regularity Condition x[n], n=0,,n 1, IID according to U[0,], let s check the regularity condition ln ; / What is going on here? ;?0 Some Examples DC level in white noise,0,1,,1 ln; 10
Fisher Information Fisher Information ln ; ln ; nonnegative additive for independent observations Proof of CRLB Setup: 1. pdf depends on 2. we need to estimate one scalar parameter We consider all unbiased estimators for the parameter 0,1,,1 11
Proof of CRLB ; ; ln ; ln ; ; ; ; ln ; ; ln ; Proof of CRLB ; ln ; ; Equality condition (Cauchy Schwarz inequality) ln ; Furthermore, we can find 12
Example General CRLB for Signals in WGN ;,0,,1 var ; Sinusoidal Frequency Estimation ; cos 2, 0, 0.5 Example Range Estimation [Example 3.13 in Kay s book], 0, Sample at Nyquist rate (2B): Δ Δ Δ, 0,,1 Δ, 0,, 1, 0, 1 Δ,, 1,,1 M: length of signal /Δ: delay in samples 13
Example Range Estimation 1 Δ 1 /2 2 ^2 mean square BW of the signal Vector Parameter For vector parameters:,, Regularity condition: ln ;, For any unbiased estimator, we have: 0, ln ; Furthermore, one unbiased estimator achieving the bound exists iff: ln ; is the MVUE and the min variance is given by 14
Example DC Level in WGN:, are unknown,0,1,,1 ln ; ln ; / 0 0 /2 ln ; ln ; Note: typically, the more unknowns, the higher the CRLB! Asymptotic CRLB For a WSS Gaussian random process x[n] with zero mean, whose PSD depends on parameter, Fisher information matrix element can be approximated as 2 ln ; ln ; Fd=10Hz 15
Asymptotic CRLB Almost any WSS Gaussian random process x[n] can be represented as the output of a filter with white input The PSD is then Asymptotic CRLB For large N (much larger than the impulse response length, or the correlation time of ), we have 16
Asymptotic CRLB Parseval s Theorem: Fourier Transform relationship between u[n] and x[n] We have Asymptotic CRLB Asymptotic pdf is: ln ; 2 ln 2 2 To eliminate, we use the following: = 0 17
Asymptotic CRLB Asymptotic pdf: ln ; 2 ln 2 2 CRLB can be found as: ln 2 ln ; ln ; Note: periodogram spectral estimator: Center Frequency of Process PSD depends on the center frequency. Some time a want to estimate the center frequency ; 12 18
HW Week 1: Chapter 2: 2.7, 2.11 Chapter 3: 3.6, 3.11, 3.12, 3.13, 3.16, 3.18, 3.19, 3.20 Week 2: Chapter 3: 3.10, 3.14, 3.15, 3.17 Complete the proof for the asymptotic CRLB 19