Advanced Signal Processing Introduction to Estimation Theory

Transcription

1 Advanced Signal Processing Introduction to Estimation Theory Danilo Mandic, room 813, ext: Department of Electrical and Electronic Engineering Imperial College London, UK URL: mandic c D. P. Mandic Advanced Signal Processing 1

2 Aims of this lecture To introduce the notions of an Estimator, Estimate, Estimandum To discuss the bias and variance in statistical estimation theory, asymptotically unbiased and consistent estimators Performance metric, such as the Mean Square Error (MSE) The bias variance dilemma and the MSE in this context To derive feasible MSE estimators A class of Minimum Variance Unbiased (MVU) estimators Extension to the vector parameter case Point estimators, confidence intervals, statistical goodness of an estimator, the role of noise c D. P. Mandic Advanced Signal Processing 2

3 Role of estimation in signal processing try also the function specgram in Matlab it produces the TF diagram below M aaaa tl aaa b An enabling technology in many DSP applications Radar and sonar: estimation of the range and azimuth Image analysis: motion estimation, segmentation Speech: features used in recognition and speaker verification Seismics: oil reservoirs Horizontal axis: time Vertical axis: frequency Communications: equalization, symbol detection Biomedicine: various applications c D. P. Mandic Advanced Signal Processing 3

4 From Lecture 2: Estimation of the parameters of an AR(2) process (under- vs. over-fitting) Original AR(2) process x[n] = 0.2x[n 1] 0.9x[n 2] + w[n], w[n] N (0, 1), estimated using AR(1), AR(2) and AR(20) models: 5 Original and estimated signals AR coefficients Time [sample] Original AR(2) signal AR(1), Error= AR(2), Error= AR(20), Error= Coefficient index Can we put this into a bigger estimation theory framework? Can we quantify the goodness of the estimators above (bias, variance)? c D. P. Mandic Advanced Signal Processing 4

5 Statistical estimation problem for simplicity, consider a DC level in WGN, x[n] = A + w[n], w N (0, σ 2 ) Problem statement: We seek to determine a set of parameters θ = [θ 1,..., θ p ] T from a set of data points x = [x[0],..., x[n 1]] T such that the values of these parameters would yield the highest probability of obtaining the observed data. In other words, max p(x; θ) reads : p(x) parametrised by θ span θ The unknown parameters may be seen as either deterministic or random variables There are essentially two alternatives to the statistical case No a priori distribution assumed: Maximum Likelihood A priori distribution known: Bayesian estimation Key problem to estimate a group of parameters from a discrete-time signal or dataset. c D. P. Mandic Advanced Signal Processing 5

6 Estimation of a scalar random variable Given an N-point dataset x[0], x[1],..., x[n 1] which depends on an unknown parameter θ (scalar), define an estimator as some function, g, of the dataset, that is ˆθ = g ( x[0], x[1],..., x[n 1] ) which may be used to estimate θ (single parameter case). (in our DC level estimation problem, θ = A) This defines the problem of parameter estimation We also need to determine g( ) Theory and techniques of statistical estimation are available Estimation based on PDFs which contain unknown but deterministic parameters is termed classical estimation In Bayesian estimation, the unknown parameters are assumed to be random variables, which may be prescribed a priori to lie within some range of allowable parameters (or desired performance) c D. P. Mandic Advanced Signal Processing 6

7 The stastical estimation problem First step: to model, mathematically, the data We employ a Probability Desity Function (PDF) to describe the inherently random measurement process, that is p(x[0], x[1],..., x[n 1]; θ) which is parametrised by the unknown parameter θ P(x[0]; ) Example: for N = 1, and θ denoting the mean value, a generic form of PDF for the class of Gaussian PDFs with any value of θ is given by p(x[0]; θ) = 1 2πσ 2 exp [ 1 2σ 2 (x[0] θ) 2] 1 2 x[0] Clearly, the observed value of x[0] critically impacts upon the likely value of θ. c D. P. Mandic Advanced Signal Processing 7

8 Estimator vs. Estimate The parameter to be estimated is then viewed as a realisation of the random variable θ Data are described by the joint PDF of the data and parameters: p(x, θ) = p(x θ) p(θ) }{{}}{{} (conditional P DF ) (prior P DF ) An estimator is a rule that assigns a value to the parameter θ from each realisation of x = x = [x[0],..., x[n 1]] T An estimate of the true value, θ, also called estimandum, is obtained for a given realisation of x = [x[0],..., x[n 1]] T in the form ˆθ = g(x) Example: for a noisy straight line: p(x; θ) = 1 (2πσ 2 ) N/2 e [ 1 2σ 2 N 1 n=0 (x[n] A Bn)2] Performance is critically dependent upon this PDF assumption the estimator should be robust to a slight mismatch between the measurement and the PDF assumption c D. P. Mandic Advanced Signal Processing 8

9 Example 1: Finding the parameters of a straight line specification of the PDF is critical in determining a good estimator In practice, we choose a PDF which fits the problem constraints and any a priori information; but it must also be mathematically tractable. Example: Assume that on the average data values are increasing Data: straight line embedded in random noise w[n] N (0, σ 2 ) x[n] noisy line x[n] = A + Bn + w[n] n = 0, 1,..., N 1 [ 1 p(x; A, B) = e 1 N 1 (2πσ 2 ) N/2 2σ 2 n=0 (x[n] A Bn)2] A 0 ideal noiseless line n Unknown parameters: A, B θ [A B] T Careful: what would be the effects of bias in A and B? c D. P. Mandic Advanced Signal Processing 9

10 Bias in parameter estimation Estimation theory (scalar case): estimate the value of an unknown parameter, θ, from a set of observations of a random variable described by that parameter ˆθ = g ( x[0], x[1],..., x[n 1] ) Example: given a set of observations from Gaussian distribution, estimate the mean or variance from these observations. Recall that in linear mean square estimation, when estimating the value of a random variable y from an observation of a related random variable x, the coefficients A and B within the estimator y = Ax + B depend upon the mean and variance of x and y, as well as on their correlation. The difference between the expected value of the estimate and the actual value θ is called the bias and will be denoted by B. B = E{ˆθ N } θ where ˆθ N denotes estimation over N data samples, x[0],..., x[n 1] c D. P. Mandic Advanced Signal Processing 10

11 Asymptotic unbiasedness If the bias is zero, then the expected value of the estimate is equal to the true value, that is E{ˆθ N } = θ B = E{ˆθ N } θ = 0 and the estimate is said to be unbiased. If B 0 then the estimator ˆθ = g(x) is said to be biased. Example: Consider the sample mean estimator of the DC level in WGN, x[n] = A + w[n], w N (0, 1), given by  = x = 1 N + 2 N 1 n=0 x[n] that is θ = A Is the above sample mean estimator of the true mean A biased? Observe: This estimator is biased but the bias B 0 when N lim E{ˆθ N } = θ N Such as estimator is said to be asymptotically unbiased. c D. P. Mandic Advanced Signal Processing 11

12 How about the variance? It is desirable that an estimator be either unbiased or asymptotically unbiased (think about the power of estimation error due to DC offset) For an estimate to be meaningful, it is necessary that we use the available statistics effectively, that is, var 0 as N or in other words { lim var{ˆθ N } = lim ˆθ N E{ˆθ N } 2} = 0 N N If ˆθ N is unbiased then E{ˆθ N } = θ, and from Tchebycheff inequality ɛ > 0 P r{ ˆθ N θ ɛ} var{ˆθ N } ɛ 2 if var 0 as N, then the probability that ˆθ N differs by more than ɛ from the true value will go to zero (showing consistency). In this case, ˆθ N is said to converge to θ with probability one. c D. P. Mandic Advanced Signal Processing 12

13 Mean square convergence NB: Mean square error criterion is very different from the variance criterion Another form of convergence, stronger than convergence with probability one is the mean square convergence. An estimate ˆθ N is said to converge to θ in the mean square sense, if lim E{ ˆθ N θ 2 } N }{{} mean square error = 0 For an unbiased estimator, this is equivalent to the previous condition that the variance of the estimate goes to zero An estimate is said to be consistent if it converges, in some sense, to the true value of the parameter We say that the estimator is consistent if it is asymptotically unbiased and has a variance that goes to zero as N c D. P. Mandic Advanced Signal Processing 13

14 Example 2: Assessing the performance of the Sample Mean as an estimator Consider the estimation of a DC level, A, in random noise, which can be modelled as x[n] = A + w[n] where w[n] is some zero-mean random iid process. Aim: to estimate A given {x[0], x[1],..., x[n 1]} Intuitively, the sample mean is a reasonable estimator, and has the form  = 1 N N 1 n=0 x[n] Q1: How close will  be to A? Q2: Are there better estimators than the sample mean? c D. P. Mandic Advanced Signal Processing 14

15 Mean and variance of the Sample Mean estimator x[n] = A + w[n] w[n] N (0, σ 2 ) Estimator = f( random data ) it is a random variable itself its performance must be judged statistically (1) What is the mean of Â? { } } N 1 1 E {Â = E x[n] = 1 N N n=0 (2) What is the variance of Â? N 1 n=0 E {x[n]} = A unbiased Assumption: The samples of w[n]s are uncorrelated } var {Â = E [ { Â E[Â]] 2 1 = var }{{} N variability around the mean = 1 N 2 N 1 n=0 var {x[n]} = 1 N 2N σ2 = σ2 N N 1 n=0 x[n] Notice the variance 0 as N consistent (see your P&A sets) } c D. P. Mandic Advanced Signal Processing 15

16 Minimum Variance Unbiased (MVU) estimation Aim: to establish good estimators of unknown deterministic parameters Unbiased estimator on the average yields the true value of the unknown parameter, independently of its particular value, i.e. E(ˆθ) = θ a < θ < b where (a, b) denotes the range of possible values of θ Example 3: Unbiased estimator for a DC level in white Gaussian noise (WGN), observed as x[n] = A + w[n] n = 0, 1,..., N 1 where A is the unknown, but deterministic, parameter to be estimated which lies within the interval (, ), then the sample mean can be used as an estimator of A, namely  = 1 N N 1 n=0 x[n] c D. P. Mandic Advanced Signal Processing 16

17 Careful: The estimator is parameter dependent! An estimator may be unbiased for certain values of the unknown parameter but not all, such an estimator is not unbiased Example 4: Consider another sample mean estimator of a DC level:  = 1 2N } Therefore: E { } E { N 1 n=0 x[n] = 0 when A = 0 but = A 2 when A 0 (parameter dependent) Hence  is not an unbiased estimator. A biased estimator introduces a systemic error which should not generally be present Our goal is to avoid bias if we can, as we are interested in stochastic signal properties and bias is largely deterministic c D. P. Mandic Advanced Signal Processing 17

18 Remedy: How about averaging? But do we average data segments or estimates? Also look in the CW Assignment dealing with PSD in your CW ) Several unbiased estimators of the same quantity may be averaged together. For example, given the L independent estimates {ˆθ1, ˆθ 2,..., ˆθ L } we may choose to average them, to yield ˆθ = 1 L L l=1 ˆθ l Our assumption was that the individual estimators, ˆθ l = g(x), are unbiased, with equal variances, and mutually uncorrelated. Then (NB: averaging biased estimators will not remove the bias) } E {ˆθ = θ and } var {ˆθ = 1L } } Ll=1 {ˆθl var = {ˆθl 1L var 2 Note, as L, ˆθ θ (consistent estimator) c D. P. Mandic Advanced Signal Processing 18

19 Effects of averaging for real world data Problem 3.4 from your P/A sets: heart rate estimation The heart rate, h, of a patient is automatically recorded } by a computer every 100ms. One second of the measurements {ĥ1, ĥ2,..., ĥ10 are averaged to obtain ĥ. Given } than E {ĥi = αh for some constant α and var(ĥi) = 1 for all i, determine whether averaging improves the estimator, for α = 1 and α = 1/2. ĥ = i=1 ĥ i [n], Before Averaging After Averaging α = 1 α = 1 } E {ĥ = α h = αh p(ĥi) p(ĥi) i=1 For α = 1, the estimator is unbiased. For α = 1/2 it will not be unbiased unless the estimator is formed as ĥ = i=1 ĥi[n]. α = 1 2 h ĥ i α = 1 2 h ĥ i } var {ĥ = 1 10 L 2 } var {ĥi p(ĥi) p(ĥi) i=1 h/2 h ĥ i h/2 h ĥ i c D. P. Mandic Advanced Signal Processing 19

20 Minimum variance criterion An optimality criterion is necessary to define an optimal estimator Such a criterion is the Mean Square Error (MSE), given by { ) } 2 ( MSE(ˆθ) = E (ˆθ θ E { error 2} ) which measures the average mean squared deviation of the estimate from the true value (error power). Without any constraints, this criterion leads to unrealisable estimators those which are not solely a function of the data (see Example 5). { ) ( )] } 2 MSE(ˆθ) = E [(ˆθ E(ˆθ) + E(ˆθ) θ = var(ˆθ) + E { (ˆθ) θ} 2 = var(ˆθ) + B 2 (ˆθ) MSE = VARIANCE OF THE ESTIMATOR + SQUARED BIAS c D. P. Mandic Advanced Signal Processing 20

21 Example 5: An MSE estimator with a gain factor Consider the following estimator for DC level in WGN Â = a 1 N N 1 n=0 x[n] Task: Find the value of a which results in the minimum MSE. Solution: } E {Â = aa and var(â) = a2 σ 2 N so that we have MSE(Â) = a2 σ 2 N + (a 1)2 A 2 Of course, the choice a = 1 removes the mean and minimises the variance c D. P. Mandic Advanced Signal Processing 21

22 Example 5: (continued) An MSE estimator with a gain (is a biased estimator feasible?) But, can we find a analytically? Differentiating with respect to a yields MSA a (Â) = 2aσ2 N + 2(a 1)A2 and setting the result to zero gives the optimal value a opt = A2 A 2 + σ2 N but we do not know the value of A The optimal value, a opt, depends on A which is the unknown parameter Comment: any criterion which depends on the value of an unknown parameter to be found is likely to yield unrealisable estimators Practically, the minimum MSE estimator needs to be abandoned, and the estimator must be constrained to be unbiased. c D. P. Mandic Advanced Signal Processing 22

23 Example 6: A counter-example a little bias can help (but the estimator is difficult to control) Q: Let {y[n]}, n = 1,..., N be iid Gaussian variables N (0, σ 2 ). Consider the following estimate of σ 2 ˆσ 2 = α N y 2 [n] α > 1 N n=1 Find α which minimises the MSE of ˆσ 2. S: It is straightforward to show that E{σ 2 } = ασ 2 and MSE(ˆσ 2 ) = E{ (ˆσ 2 σ 2) 2 } = E{ˆσ 4 } + σ 4 (1 2α) = α2 N 2 N n=1 N E{y 2 [n]y 2 [s]} + σ 4 (1 2α) s=1 Hint : Σ n 2 = Σ n Σ s = α2 N 2 ( N 2 σ 4 + 2Nσ 4) + σ 4 (1 2α) = σ 4[ α 4 (1 + 2 N ) + (1 2α) ] The MMSE is obtained for α min = N N+2 and has the value min MSE(ˆσ 2 ) = 2σ4 N+2. Given that the minimum variance of an unbiased estimator (CRLB, later) is 2σ 4 /N, this is an example of a biased estimator which obtains a lower MSE than the CRLB. c D. P. Mandic Advanced Signal Processing 23

24 Desired: minimum variance unbiased (MVU) estimator Minimising the variance of an unbiased estimator concentrates the PDF of the error about zero estimation error is therefore less likely to be large Existence of the MVU estimator The MVU estimator is an unbiased estimator with minimum variance for all θ, that is, ˆθ 3 on the graph. c D. P. Mandic Advanced Signal Processing 24

25 Methods to find the MVU estimator The MVU estimator may not always exist, but if it exists we need to examine if that is an optimal estimator A single unbiased estimator may not exist in which case a search for the MVU is fruitless. Some solutions are: Determine the Cramer-Rao lower bound (CRLB) and find some estimator which satisfies the MVU criteria Apply the Rao-Blackwell-Lehmann-Scheffe (RBLS) theorem Restrict the class of estimators to be not only unbiased, but also linear in both the parameters and data (BLUE) (Lecture 5) Model-based approaches, such as Least Squares and Sequential Least Squares (Lecture 6) Adaptive estimators (Lecture 7) c D. P. Mandic Advanced Signal Processing 25

26 Extensions to the vector parameter case If θ = [ˆθ1, ˆθ 2,..., ˆθ p ] T R p 1 is a vector of unknown parameters, an estimator is said to be unbiased if By defining E(ˆθ i ) = θ i where a i < θ i < b i for i = 1, 2,..., p E(θ) = E(θ 1 ) E(θ 2 ). E(θ p ) an unbiased estimator has the property E(ˆθ) = θ within the p dimensional space of parameters spanned by θ = [θ 1,..., θ p ] T. An MVU estimator has the additional property that its var( ˆθ i ), for i = 1, 2,..., p, is the minimum among all unbiased estimators. c D. P. Mandic Advanced Signal Processing 26

27 Summary and food for thoughts We are now equipped with performance metrics for assessing the goodnes of any estimator (bias, variance, MSE). Since MSE = var + bias 2, some biased estimators may yield low MSE. However, we prefer minimum variance unbiased (MVU) estimators. Even a simple Sample Mean estimator is a very rich example of the advantages of statistical estimators. The knowledge of the parametrised PDF p(data;parameters) is very important for designing efficient estimators. We have introduced statistical point estimators, would it be useful to also know the confidence we have in our point estimate? In many disciplines it is useful to design so called set membership estimates, where the output of an estimator belongs to a pre-definined bound (range) of values. We will next address linear, best linear unbiased, maximum likelihood, least squares, sequential least squares, and adaptive estimators. c D. P. Mandic Advanced Signal Processing 27

28 Homework: Check another proof for the MSE expression MSE(ˆθ) = var(ˆθ) + bias 2 (θ) Note : var(x) = E[x 2 ] [ E[x] ] 2 ( ) Idea : Let x = ˆθ θ substitute into ( ) to give var(ˆθ θ) }{{} term (1) Let us now evaluate these terms: (1) var(ˆθ θ) = var(ˆθ) (2) E[ˆθ θ] 2 = MSE = E[(ˆθ θ) 2 ] }{{} term (2) [ E[ˆθ θ] ] 2 }{{} term (3) ( ) (3) [ E[ˆθ θ] ] 2 = [ E[ˆθ] E[θ] ] 2 = [ E[ˆθ θ] ] 2 = bias 2 (ˆθ) Substitute (1), (2), (3) into (**) to give var(ˆθ) = MSE bias 2 MSE = var(ˆθ) + bias 2 (ˆθ) c D. P. Mandic Advanced Signal Processing 28

29 Recap: Unbiased estimators Due to the linearity properties of the statistical expectation operator, E { }, that is E{a + b} = E{a} + E{b} the sample mean estimator can be shown to be unbiased, i.e. } E { = 1 N 1 N n=0 E {x[n]} = 1 N 1 N n=0 A = A In some applications, the value of A may be constrained to be positive. For example, the value of an electronic component such as an inductor, capacitor or resistor would be positive (prior knowledge). For N data points in random noise, unbiased estimators generally have symmetric PDFs centred about their true value, that is  N (A, σ 2 /N) c D. P. Mandic Advanced Signal Processing 29

30 Notes c D. P. Mandic Advanced Signal Processing 30

31 Notes c D. P. Mandic Advanced Signal Processing 31

32 Notes c D. P. Mandic Advanced Signal Processing 32