13. Parameter Estimation. ECE 830, Spring 2014

Size: px

Start display at page:

Download "13. Parameter Estimation. ECE 830, Spring 2014"

Preston Norris
6 years ago
Views:

1 13. Parameter Estimation ECE 830, Spring / 18

2 Primary Goal General problem statement: We observe X p(x θ), θ Θ and the goal is to determine the θ that produced X. Given a collection of observations x 1,..., x n and a probability model p(x 1,..., x n θ) parameterized by the parameter θ, determine the value of θ that best matches the observations. 2 / 18

3 Example: Stock Market (Dow-Jones Industrial Avg.) Based on this plot we might conjecture that the data is on average increasing. Probability model: x[k] = A + Bk + w[k] A, B unknown parameters, w[k] white Gaussian noise to model fluctuations. p(x A, B) = 3 / 18

4 Example: A Classic Example Radar 4 / 18

5 Example: A Classic Example Radar (cont.) The received waveform is time-dilated and shifted version of original waveform g(t) plus noise x(t) = g(αt τ) + w(t). [ ] α The parameter of interest is θ =, where α is related to τ velocity / Doppler shift, τ is related to distance τ = 2d c, c = speed of light x ˆτ ˆd = cˆτ 2 5 / 18

6 Example: A modern Example - Imaging Image processing can involve complicated estimation problems. For example, suppose we observe a moving object with noise. This image is blurry and noisy and our goal is to restore this image by debarring and denoising. 6 / 18

7 Example: A modern Example - Imaging We can model the moving part of the observed image as x = h}{{} θ + }{{} w motion blur (convolution) noise where the parameter of interest θ is the ideal image. Observation model: X =Hθ + w, w N(0, σ 2 I) X N (Hθ, σ 2 I) Estimator: ˆθ = f(x) a function of x 7 / 18

8 Basic Ingredients of Estimation Theory Observations (data): x = [x 1,..., x n ] T A observation probability model parameterized by θ: p(x θ), θ Θ, x X A class or collection of possible parameter values Θ: e.g., R, {linear estimators}, {unbiased estimators} An estimator ˆθ : X Θ, a mapping from X to Θ A loss/error function l : Θ Θ R +. l(θ, ˆθ) measures proximity of ˆθ to θ Risk (average/expected loss) R(θ, ˆθ) = E[l(θ, ˆθ(x))] = l(θ, ˆθ(x)) p(x θ)dx X 8 / 18

9 Optimal Estimator ˆθ opt = arg min R(θ, ˆθ) ˆθ:X Θ ˆθ opt is optimal with respect to the chosen loss function. Example losses include: Squared Error (l 2 loss) l(θ, ˆθ) = θ ˆθ n 2 2 = (θ i ˆθ i ) 2 Absolute Error (l 1 loss) (penalizes large errors less than l 2 ) l(θ, ˆθ) = θ ˆθ n 1 = θ i ˆθ i 0/1 loss l(θ, ˆθ) = { 1, ˆθ θ i=1 i=1 0, otherwise R(θ, ˆθ) =E[1 {ˆθ(x) θ} ] = P (ˆθ θ) 9 / 18

10 Special Case - Hypothesis Testing: Hypothesis testing can be viewed as a special case in which the parameter θ takes one of two possible values, 0 or 1, and the loss is 0/1 loss or some weighted version of it. 10 / 18

11 Terminology in Estimation Theory Define ɛ( θ) := θ θ Recall that θ = θ(x) is a function of data = ɛ( θ) is a statistic! Mean Squared Error: E[ɛ T ɛ] := MSE( θ) Bias: Variance / Covariance: Bias( θ) := E[ θ] θ Var( θ) := E[( θ E[ θ])( θ E[ θ]) T ] 11 / 18

12 Bias-variance decomposition 12 / 18

13 Constraints Often we are interested in estimators with special properties and/or estimators that satisfy certain constraints. Example: Minimum Variance, Unbiased Estimator Let θ be an unknown parameter and let θ be an estimator of θ given the data x. Choose the estimate θ such that the Bias = 0 and the variance is as small as possible. Example: Best Linear Unbiased Estimator Often we would like to keep our estimator simple. One common constraint is to find the best estimator that is linear in the data, unbiased and minimizes variance among all linear functions of the data. 13 / 18

14 Asymptotics Estimators are often studied as a function of the number of observations: θ(x) = θ n where n = dim x θ is asymptotically unbiased if An estimator is consistent if lim Bias( θ n ) = n lim MSE( θ n ) = n A consistent estimator is at least asymptotically unbiased. Some estimators are unbiased, but inconsistent. The latter basically means that our estimation does not improve as the number of data increase. Inconsistent estimators can provide reasonable estimates when we have a small number of data. However, consistent estimators are usually favored in practice. 14 / 18

15 Prior information In the problem formulation above, we do not make any assumptions on what θ might be; we simply minimize the cost function over all allowable possibilities. But suppose that we believe that some values of θ are more likely than others. We can reflect this prior belief by viewing the parameter θ as a random variable itself with a prior density p(θ) This leads to the joint density function: p(x, θ) = p(x θ)p(θ) We can specify cost functions based on this joint density. The prior p(θ) effectively steers or directs our search in what we believe are the right directions. 15 / 18

16 Example: Stock Market We might believe that 2800 < A < We can incorporate this belief with 16 / 18

17 Example: Getting our feet wet iid X 1, X 2,, X n N (µ, 1) ˆµ n = ˆµ(X 1,..., X n ) = 1 n n i=1 X i l 2 loss: l(µ, ˆµ) = µ ˆµ n 2 2 MSE risk: bias: E[ˆµ n ] = R(µ, ˆµ) = E[(µ ˆµ n ) 2 ] = E[(µ E[ˆµ n ]) 2 ] + E[(ˆµ }{{} n E[ˆµ n ]) }{{} bias 2 variance 2 ] 17 / 18

18 Example: Getting our feet wet (cont.) variance: consistency: 18 / 18

10. Linear Models and Maximum Likelihood Estimation

10. Linear Models and Maximum Likelihood Estimation ECE 830, Spring 2017 Rebecca Willett 1 / 34 Primary Goal General problem statement: We observe y i iid pθ, θ Θ and the goal is to determine the θ that