Detection & Estimation Lecture 1

Similar documents
Detection & Estimation Lecture 1

EIE6207: Estimation Theory

Variations. ECE 6540, Lecture 10 Maximum Likelihood Estimation

Estimation, Detection, and Identification CMU 18752

Estimation and Detection

Advanced Signal Processing Introduction to Estimation Theory

A Few Notes on Fisher Information (WIP)

1. (Regular) Exponential Family

Detection and Estimation Theory

ECE531 Lecture 10b: Maximum Likelihood Estimation

ELEG 5633 Detection and Estimation Minimum Variance Unbiased Estimators (MVUE)

Probability and Statistics for Final Year Engineering Students

Fundamentals of Statistical Signal Processing Volume II Detection Theory

DETECTION theory deals primarily with techniques for

Estimation Theory Fredrik Rusek. Chapters

Module 2. Random Processes. Version 2, ECE IIT, Kharagpur

Chapter 8: Least squares (beginning of chapter)

Rowan University Department of Electrical and Computer Engineering

Module 1 - Signal estimation

Lecture Notes 7 Stationary Random Processes. Strict-Sense and Wide-Sense Stationarity. Autocorrelation Function of a Stationary Process

EIE6207: Maximum-Likelihood and Bayesian Estimation

Review Quiz. 1. Prove that in a one-dimensional canonical exponential family, the complete and sufficient statistic achieves the

ECE 275A Homework 6 Solutions

PATTERN RECOGNITION AND MACHINE LEARNING

Classical Estimation Topics

Linear models. x = Hθ + w, where w N(0, σ 2 I) and H R n p. The matrix H is called the observation matrix or design matrix 1.

Terminology Suppose we have N observations {x(n)} N 1. Estimators as Random Variables. {x(n)} N 1

Advanced Signal Processing Minimum Variance Unbiased Estimation (MVU)

Maximum Likelihood Estimation

Detection theory 101 ELEC-E5410 Signal Processing for Communications

Proof In the CR proof. and

Lecture 8: Information Theory and Statistics

Basic concepts in estimation

Technique for Numerical Computation of Cramér-Rao Bound using MATLAB

ECE 275B Homework # 1 Solutions Version Winter 2015

2 Statistical Estimation: Basic Concepts

Mean Likelihood Frequency Estimation

LINEAR MODELS IN STATISTICAL SIGNAL PROCESSING

covariance function, 174 probability structure of; Yule-Walker equations, 174 Moving average process, fluctuations, 5-6, 175 probability structure of

Probability Space. J. McNames Portland State University ECE 538/638 Stochastic Signals Ver

Cramer-Rao Lower Bound Computation Via the Characteristic Function

All other items including (and especially) CELL PHONES must be left at the front of the room.

Signals and Spectra - Review

ECE 275B Homework # 1 Solutions Winter 2018

Methods of evaluating estimators and best unbiased estimators Hamid R. Rabiee

A TERM PAPER REPORT ON KALMAN FILTER

ELEG 3143 Probability & Stochastic Process Ch. 6 Stochastic Process

Estimation theory. Parametric estimation. Properties of estimators. Minimum variance estimator. Cramer-Rao bound. Maximum likelihood estimators

ECE 3800 Probabilistic Methods of Signal and System Analysis

Parametric Signal Modeling and Linear Prediction Theory 1. Discrete-time Stochastic Processes

Stochastic Process II Dr.-Ing. Sudchai Boonto

EEM 409. Random Signals. Problem Set-2: (Power Spectral Density, LTI Systems with Random Inputs) Problem 1: Problem 2:

Estimation, Detection, and Identification

Large Sample Properties of Estimators in the Classical Linear Regression Model

Estimators as Random Variables

Mathematical statistics

System Identification, Lecture 4

System Identification, Lecture 4

Course content (will be adapted to the background knowledge of the class):

Frames and Phase Retrieval

Estimation Tasks. Short Course on Image Quality. Matthew A. Kupinski. Introduction

On the Cramér-Rao lower bound under model mismatch

UCSD ECE153 Handout #40 Prof. Young-Han Kim Thursday, May 29, Homework Set #8 Due: Thursday, June 5, 2011

ENEE 621 SPRING 2016 DETECTION AND ESTIMATION THEORY THE PARAMETER ESTIMATION PROBLEM

Mathematical statistics

Chapter 3: Unbiased Estimation Lecture 22: UMVUE and the method of using a sufficient and complete statistic

Introduction to Statistical Inference

ECE 275A Homework 7 Solutions

Stochastic Processes. M. Sami Fadali Professor of Electrical Engineering University of Nevada, Reno

Define characteristic function. State its properties. State and prove inversion theorem.

Spring 2014 ECEN Signals and Systems

Statistical and Adaptive Signal Processing

A Simple Phase Unwrapping Algorithm and its Application to Phase- Based Frequency Estimation

QUANTIZATION FOR DISTRIBUTED ESTIMATION IN LARGE SCALE SENSOR NETWORKS

Probability Models in Electrical and Computer Engineering Mathematical models as tools in analysis and design Deterministic models Probability models

SIGNAL STRENGTH LOCALIZATION BOUNDS IN AD HOC & SENSOR NETWORKS WHEN TRANSMIT POWERS ARE RANDOM. Neal Patwari and Alfred O.

Optimal Time Division Multiplexing Schemes for DOA Estimation of a Moving Target Using a Colocated MIMO Radar

Part III Spectrum Estimation

Two hours. To be supplied by the Examinations Office: Mathematical Formula Tables THE UNIVERSITY OF MANCHESTER. 21 June :45 11:45

Estimation Theory Fredrik Rusek. Chapters 6-7

Lecture 7 Introduction to Statistical Decision Theory

ECE 6540, Lecture 06 Sufficient Statistics & Complete Statistics Variations

A General Overview of Parametric Estimation and Inference Techniques.

Mathematical Statistics

Unbiased Estimation. Binomial problem shows general phenomenon. An estimator can be good for some values of θ and bad for others.

Frequency estimation by DFT interpolation: A comparison of methods

Tutorial: Statistical distance and Fisher information

EE482: Digital Signal Processing Applications

Optimal Mean-Square Noise Benefits in Quantizer-Array Linear Estimation Ashok Patel and Bart Kosko

FUNDAMENTAL FILTERING LIMITATIONS IN LINEAR NON-GAUSSIAN SYSTEMS

10-704: Information Processing and Learning Fall Lecture 24: Dec 7

ST5215: Advanced Statistical Theory

SIGNAL STRENGTH LOCALIZATION BOUNDS IN AD HOC & SENSOR NETWORKS WHEN TRANSMIT POWERS ARE RANDOM. Neal Patwari and Alfred O.

MAT 211, Spring 2015, Introduction to Linear Algebra.

EE 574 Detection and Estimation Theory Lecture Presentation 8

ECE 516: System Control Engineering

STAT 730 Chapter 4: Estimation

Machine Learning. Instructor: Pranjal Awasthi

SIMON FRASER UNIVERSITY School of Engineering Science

Example 1. Let be a random sample from. Please find a good point estimator for

Transcription:

Detection & Estimation Lecture 1 Intro, MVUE, CRLB Xiliang Luo General Course Information Textbooks & References Fundamentals of Statistical Signal Processing: Estimation Theory/Detection Theory, Steven M. Kay, Prentice Hall, 1993. Detection, Estimation, and Modulation Theory, Part I, Harry L. Van Trees, John Wiley & Sons, Inc., 2001. Principles of Signal Detection and Parameter Estimation, Benard C. Levy, Springer, 2008. Lecturer Dr. Xiliang Luo Office hour: TBA TA Mr. Xiaoyu Zhang Office hour: TBA Grading Homework: (Weekly) 40% (due at the beginning of each lecture) Midterm: 30% [open book] Final: 30% [open book] 1

General Course Information You must complete the weekly HW independently Discussions among students are allowed but solutions must be your own On the 1 st page of your HW, please copy: The works submitted herein are entirely my own. I have derived every single step shown below. (sign your name and date) Estimation Radar Sonar Speech Image analysis Biomedicine Communication Control Seismology 2

Radar Sonar 3

Estimation Problem Given a data set x[0],x[1],,x[n 1] We want to determine the value of an unknown parameter as: 0,1,,1 this function is an estimator Date back to Gauss, 1795 least squares planetary movement Estimation Problem The data has to be dependent on the unknown parameter pdf: 0,,1; the semicolon denotes the dependence Example: Gaussian pdf 4

Classical vs Bayesian Classical estimation the unknown parameter is deterministic Bayesian estimation the unknown parameter is itself random we are estimating one realization of the random parameter the data are characterized by the joint pdf, : the prior pdf Estimator Performance Question: How is this estimator? find the mean,variance Best estimator? topic next 5

Unbiased Estimator On average, the estimator should yield the true value this estimator is unbiased E,, Example: An estimator is unbiased does not mean it is a good estimator Minimum Variance In order to find one optimal estimator, we need to specify the criterion one natural criterion is the Mean Square Error (MSE) Example: 1 1 / Not realizable! 6

MVUE Minimize the variance while being unbiased Question: whether MVUE exists? unbiased estimator with minimum variance for all values of the unknown parameter Example: [Example 2.3], 1, 0 0,1 1, 2, 0 1 2 0 1 3 1 0 1 2 MVUE No known turn the crank procedure to produce the MVUE Next, we will discuss Cramer Rao lower bound Rao Blackwell Lehmann Scheffe theorem best linear estimator 7

Cramer Rao Lower Bound We need to place a lower bound on the variance of any unbiased estimator! Check whether our estimator is MVUE Check how far our estimator is from the optimal one even the optimal one may not exist Tells us it is impossible to find an estimator that can beat the bound Likelihood Function When the pdf is views a function of the unknown parameter, it is referred to as the likelihood function Example: 0 0 0; 1 ln 0; ln 2 1 2 ln 0; 2 1 0 8

CRLB Regularity condition: ln ; 0, For any unbiased estimator, we have: ln ; Furthermore, one unbiased estimator achieving the bound exists iff: ln ; is the MVUE and the min variance is given by 1/ Regularity Condition x[n], n=0,,n 1, IID according to U[0,], let s check the regularity condition ln ; / What is going on here? ;?0 9

Some Examples DC level in white noise,0,1,,1 ln; Fisher Information Fisher Information ln ; ln ; nonnegative additive for independent observations 10

Proof of CRLB Setup: 1. pdf depends on 2. we need to estimate one scalar parameter We consider all unbiased estimators for the parameter 0,1,,1 Proof of CRLB ; ; ln ; ln ; ; ; ; ln ; ; ln ; 11

Proof of CRLB ; ln ; ; Equality condition (Cauchy Schwarz inequality) ln ; Furthermore, we can find Example General CRLB for Signals in WGN ;,0,,1 var ; Sinusoidal Frequency Estimation ; cos 2, 0, 0.5 12

Example Range Estimation [Example 3.13 in Kay s book], 0, Sample at Nyquist rate (2B): Δ Δ Δ, 0,,1 Δ, 0,, 1, 0, 1 Δ,, 1,,1 M: length of signal /Δ: delay in samples Example Range Estimation 1 Δ 1 /2 2 ^2 mean square BW of the signal 13

HW Chapter 2: 2.7, 2.11 Chapter 3: 3.6, 3.11, 3.12, 3.13, 3.16, 3.18, 3.19, 3.20 Vector Parameter For vector parameters:,, Regularity condition: ln ;, For any unbiased estimator, we have: 0, ln ; Furthermore, one unbiased estimator achieving the bound exists iff: ln ; is the MVUE and the min variance is given by 14

Example DC Level in WGN:, are unknown,0,1,,1 ln ; ln ; / 0 0 /2 ln ; ln ; Note: typically, the more unknowns, the higher the CRLB! Asymptotic CRLB For a WSS Gaussian random process x[n] with zero mean, whose PSD depends on parameter, Fisher information matrix element can be approximated as 2 ln ; ln ; Fd=10Hz 15

Asymptotic CRLB Almost any WSS Gaussian random process x[n] can be represented as the output of a filter with white input The PSD is then Asymptotic CRLB For large N (much larger than the impulse response length, or the correlation time of ), we have 16

Asymptotic CRLB Parseval s Theorem: Fourier Transform relationship between u[n] and x[n] We have Asymptotic CRLB Asymptotic pdf is: ln ; 2 ln 2 2 To eliminate, we use the following: = 0 17

Asymptotic CRLB Asymptotic pdf: ln ; 2 ln 2 2 CRLB can be found as: ln 2 ln ; ln ; Note: periodogram spectral estimator: Center Frequency of Process PSD depends on the center frequency. Some time a want to estimate the center frequency ; 12 18