Linear models. x = Hθ + w, where w N(0, σ 2 I) and H R n p. The matrix H is called the observation matrix or design matrix 1.

Transcription

1 Linear models As the first approach to estimator design, we consider the class of problems that can be represented by a linear model. In general, finding the MVUE is difficult. But if the linear model is valid, this task is straightforward. A model with parameters θ R p 1 and data x R n 1 is linear, if it is of the form x = Hθ + w, where w N(0, σ 2 I) and H R n p. The matrix H is called the observation matrix or design matrix

2 Linear models For example, the "DC level in WGN" problem belongs to this class with x = [x[0], x[1],..., x[n 1]] T w = [w[0], w[1],..., w[n 1]] T θ = [A] H = [1, 1,..., 1 } {{ } N times With these definitions, x[n] = A 1 + w[n] holds for all n = 0, 1,..., N 1. ] T

3 Linear models (cont.) Also fitting a straight line to a set of data belongs to this class. In this case the model is x[n] = A + Bn + w[n], n = 0, 1,..., N 1 and the problem is to find MVU estimators for A and B assuming w[n] N(0, σ 2 ).

4 Linear models (cont.) In matrix form x = Hθ + w, or 1 0 x[0] x[1] 1 1 w[0] ( ) = 1 2 A w[1] +. B.. } {{ }. x[n 1] θ w[n 1] } {{ } 1 N 1 } {{ } x } {{ } w H The matrix H is called the observation matrix.

5 Linear models: finding the MVUE The nice thing about linear models is that the MVUE given by the CRLB theorem is always found. More specifically, the factorization ln p(x;θ) θ = I(θ)(g(x) θ) can always be done. According to CRLB theorem for the vector parameter case, g(x) is then the MVUE. To see what does the factorization look like, let s calculate the derivative of the log-likelihood function.

6 Linear models: finding the MVUE (cont.) The likelihood function for each sample x[n] is now p(x[n]; θ) = [ 1 exp 1 ] 2πσ 2 2σ 2 (x[n] [Hθ] n) 2 and the joint probability for all samples p(x; θ) = N 1 n=0 [ 1 p(x[n]; θ) = exp (2πσ 2 ) N 2 1 2σ 2 ] N 1 (x[n] [Hθ] n ) 2 n=0 or in vector form 1 p(x; θ) = (2πσ 2 ) N 2 [ exp 1 ] 2σ 2 (x Hθ)T (x Hθ)

7 Linear models: finding the MVUE (cont.) After taking the logarithm and differentiating, we get ln p(x; θ) θ = [ ln(2πσ 2 ) N 1 2 θ = 1 2σ 2 ] 2σ 2 (x Hθ)T (x Hθ) ] [ x T x 2x T Hθ + θ T H T Hθ θ It can be shown that for any vector v and any symmetric matrix M the following differentiation rules hold. θ vt θ = v θ θt Mθ = 2Mθ.

8 Linear models: finding the MVUE (cont.) Using these, we can evaluate the above formula: ln p(x; θ) θ = 1 2σ 2 [ 2H T x + 2H T Hθ ] = 1 σ 2 [ H T x + H T Hθ ]

9 Linear models: finding the MVUE The MVUE g(x) is given by the following factorization: ln p(x; θ) θ = I(θ)(g(x) θ), If the square matrix H T H is invertible 2, we can cleverly multiply by the identity matrix, or I = H T H (H T H) 1 : ln p(x; θ) θ = HT H [ ] σ 2 (H T H) 1 H T x + θ = HT H [ ] σ 2 (H T H) 1 H T x θ 2 It usually is; we will return to this issue later.

10 Linear models: finding the MVUE Comparing this with the required factorization of the CRLB theorem, ln p(x; θ) θ = I(θ)(g(x) θ), we can see immediately, that the MVUE g(x) exists, and is given by ˆθ = g(x) = (H T H) 1 H T x.

11 Linear models: finding the MVUE (cont.) Furthermore, the Fisher information matrix is I(θ) = HT H σ 2, which means that the covariance matrix of the estimator is its inverse: C ˆθ = σ2 (H T H) 1.

12 Linear models: theorem MVU estimator for the linear model If the observed data can be modeled as x = Hθ + w (1) where x is an N 1 vector of observations, H is a known N p observation matrix (with N > p) and rank p, θ is a p 1 vector of parameters to be estimated, and w is an N 1 noise vector with pdf N(0, σ 2 I), then the MVU estimator is and the covariance matrix of ˆθ is ˆθ = (H T H) 1 H T x (2) C ˆθ = σ2 (H T H) 1 (3)

13 Linear models: theorem (cont.) Moreover, the MVU estimator is efficient in that it attains the CRLB. Proof. We have already proven everything except the fact that the estimator is unbiased. The unbiasedness is easily seen: E[ ˆθ] = E[(H T H) 1 H T x] = (H T H) 1 H T E[x] = (H T H) 1 H T Hθ = θ. (Here we used the fact that E[x] = Hθ + E[w] = Hθ.)

14 Examples: Line fitting In the line fitting case the equation was: 1 0 x[0] x[1] 1 1 w[0] ( ) = 1 2 A w[1] +. B.. } {{ }. x[n 1] θ w[n 1] } {{ } 1 N 1 } {{ } x } {{ } w H

15 Examples: Line fitting (cont.) Once we observe the data x and assume this model, the MVU estimator is ˆθ = (H T H) 1 H T x Writing the matrices open, we have: (Â ) ( ) ( ) = ˆB N N N 1 x[0] x[1]. x[n 1]

16 Examples: Line fitting (cont.) Now, ( ) ( H T N N 1 H = n=0 n N 1 n=0 n N 1 = n=0 n2 N N(N 1) 2 N(N 1) 2 N(N 1)(2N 1) 6 ) and we can show that the inverse is ( 2(2N 1) (H T H) 1 N(N+1) = 6 6 N(N+1) N(N+1) 12 N(N 2 1) )

17 Examples: Line fitting (cont.) Finally, ( ) = ˆB ( 2(2N 1) N(N+1) 6 6 N(N+1) N(N+1) 12 N(N 2 1) ) ( N 1 ) n=0 x[n] N 1 n=0 nx[n] Below is the result of one test run, with σ 2 = 1000, A = 1 and B = 2. In this realization, the result was  = and B =

18 Examples: Line fitting (cont.) The covariance matrix ( (or inverse of) the Fisher information matrix) is C ˆθ = This tells that the estimates  will have a lot higher variance than the estimates ˆB.

19 Examples: Line fitting (cont.) We can validate this by estimating the parameters from 1000 noise realizations. The histograms and the corresponding variances are plotted below. Estimates for A. Theoretical variance = Sample variance = Estimates for B. Theoretical variance = Sample variance =

20 Amplitude of a sinusoid So far we have considered problems, where the function was also linear (straight line or a constant). The model allows also other cases as long as the relationship between the parameters and the data is linear. These include for example estimation of the amplitude of a known sinusoid. Consider the model x[n] = A 1 cos(2πf 1 n + φ 1 ) + A 2 cos(2πf 2 n + φ 2 ) + B + w[n], for n = 0, 1,..., N 1, where f 1, f 2, φ 1, φ 2 are known and A 1, A 2 and B are the unknowns.

21 Amplitude of a sinusoid (cont.) Then the linear model is applicable with namely x = Hθ + w, x[0] cos(2πf 1 + φ 1 ) cos(2πf 2 + φ 2 ) 1 w[0] x[1] cos(4πf 1 + φ 1 ) cos(4πf 2 + φ 2 ) 1 = cos(6πf 1 + φ 1 ) cos(6πf 2 + φ 2 ) 1 A w[1] 1 A 2 +. x[n 1] B.... } {{ } w[n 1] } {{ } cos(2(n 1)πf 1 + φ 1 ) cos(2(n 1)πf 2 + φ 2 ) 1 θ } {{ } x } {{ } H w

22 Amplitude of a sinusoid (cont.) Again, the MVU estimator is ˆθ = (H T H) 1 H T x. The Matlab code for this is below. Note that now we re generating the simulated data exactly according to our model. It s interesting to see how deviations from the model affect the performance try it: Try also other curves instead of the sinusoids + lines.

23 Code % Let s generate a test case first: N = 200; n = (0:N-1) ; sigma_sq = 10; % Variance of WGN w = sqrt(sigma_sq)*randn(n,1); A = 1; % This is the unknown for the estimator B = -2; % This is the unknown for the estimator C = 10; % This is the unknown for the estimator f1 = 0.05; % This parameter the estimator knows f2 = 0.02; % This parameter the estimator knows theta = [A;B;C];

24 Code (cont.) H = [cos(2*pi*f1*n+pi/4), cos(2*pi*f2*n-pi/10), ones(n,1)]; x = H*theta + w; % This is the observed data. % Now lets try to estimate theta from the data x: % Note: Below is Matlab s preferred way for % thest = inv(h *H)*H *x thest = H \ x; plot(n,h*theta, b-, LineWidth, 2); hold on plot(n,x, go, LineWidth, 2); plot(n,h*thest, r-, LineWidth, 2); hold off

25 Amplitude of a sinusoid, results Below is the result of one example run True model Noisy data Estimated sinusoid In this case ˆθ = [1.2128, , ] T The true θ = [1, 2, 10] T.

26 Amplitude of a sinusoid, results (cont.) The covariance matrix is diagonal: C ˆθ =

27 Linear models other examples in Kay s book Curve fitting: For example the gravitational force can be modeled using a second order polynomial: x(t n ) = θ 1 + θ 2 t n + θ 3 t 2 n + w(t n ), n = 0,..., N 1 In matrix form, this is given by or x = Hθ + w

28 Linear models other examples in Kay s book (cont.) x(t 0 ) x(t 1 ). x(t N 1 ) 1 t 0 t t 1 t 2 1 =... 1 t N 1 t 2 N 1 θ 1 θ 2 + θ 3 w 0 w 1. w N 1 Notice, that for polynomial models, the matrix H has a special form, and is called Vandermonde matrix.

29 Linear models other examples in Kay s book (cont.) The nice property of the linear model is that you can try inserting whatever functions you can imagine, and let the formula decide if they are useful or not. As an example, below is an example of data with two linear models fitted into it.

30 Linear models other examples in Kay s book (cont.) 100 Model: y = 0.406*x MSE = Model: y = 0.002*x *x MSE = y(n) 50 y(n) x(n) x(n) Below are some additional models (although not very suitable ones).

31 Linear models other examples in Kay s book (cont.) 100 Model: y = 2.368*cos(2*pi*0.01*x) *(1+x) MSE = Model: y = *sqrt(x) *log(1+x)+0.000*x *x. MSE = y(n) 50 y(n) x(n) x(n) Note that the MSE is a good indicator of model suitability. We will discuss this later in the context of sequential least squares.

32 Linear models other examples in Kay s book (cont.) Fourier analysis x[n] = M k=1 a k cos ( 2πkn N with n = 0, 1,..., N 1. Now and ) + M k=1 b k sin ( 2πkn) + w[n], N θ = [a 1, a 2,, a M, b 1, b 2,, b M ] T

33 Linear models other examples in Kay s book (cont.) ( 1 ) ( 1 ) ( 1 ) ( 0 ) ( 0 ) ( 0 ) cos 2πN cos 4πN cos 2Mπ N sin 2πN sin 4πN sin 2Mπ ( ) ( ) ( ) ( ) ( ) ( N ) cos 2π 2 N cos 4π 2 N cos 2Mπ 2 N sin 2π 2 N sin 4π 2 N sin 2Mπ 2 N H = (. ) (. ) ( ) ( ) ( ) ( ) 2π(N 1) 4π(N 1) 2Mπ(N 1) 2π(N 1) 4π(N 1) 2Mπ(N 1) cos N cos N cos N sin N sin N sin N The MVU estimator results in the usual DFT coefficients, as one could expect.

34 Linear models other examples in Kay s book (cont.) System identification: Any linear process can be modeled using a FIR filter. In system identification context, we measure the input and the output of an unknown system ("black box"), and try to model its properties by a FIR filter. The problem is essentially estimating the FIR impulse response, and thus it s natural to formulate the problem as a linear model.

35 Linear models other examples in Kay s book (cont.) Denote the input by u[n], and the output by x[n], n = 0, 1,..., N 1. Also denote the FIR impulse response by h[k], k = 0, 1,..., p 1. Then our model for the measured output data is p 1 x[n] = h[k]u[n k] + w[n], n = 0, 1,..., N 1 k=0 or in matrix form u[0] 0 0 h[0] u[1] u[0] 0 h[1] x = w.. u[n 1] u[n 2] u[n p] h[p 1] } {{ }} {{ } H θ

36 Linear models other examples in Kay s book (cont.) Because this is in linear model form (assuming w[n] is WGN), the minimum variance FIR coefficient vector is ˆθ = (H T H) 1 H T x. Kay continues the discussion by asking: "What is the best selection for u[n]?" If we can select the input sequence, which one produces the smallest variance? Answer: any sequence whose covariance matrix is diagonal. That is, any (pseudo)random sequence.

37 Automatic Bacteria Counting from Microscope The next example considers automatic counting and measuring of DAPI stained bacteria from microscope image. DAPI a is a fluorescent stain molecule that binds strongly to DNA. When excited by ultraviolet light (wavelength near 358 nm), it starts to emit longer wavelengths (near 461 nm which is blue light). DAPI staining is widely used in biology and medicine for highlighting the cells for counting, tracking and other purposes. a 4,6-diamidino-2-phenylindole

38 Automatic Bacteria Counting from Microscope Traditionally (and even today) the number of cells is calculated manually. However, there are numerous automatic solutions available. At our department the software CellC was developed for this task. 3 The code is freely available at 3 J. Selinummi, J. Seppälä, O. Yli-Harja, and J. Puhakka, "Software for quantification of labeled bacteria from digital microscope images by automated image analysis," BioTechniques, Vol. 39, No 6, 2005, pp

39 CellC Operation The software consists of the following stages: Normalization of the background for variations in illumination Extraction of cells by thresholding Separation of clustered cells by marker-controlled watershed segmentation Finally, too small or large objects are discarded. The output is an excel file of cell sizes and locations together with a binary image of the segmented cells.

40 Background Correction Often the illumination is not homogeneous, but is more bright in the center. This can be corrected by fitting a two-dimensional quadratic surface and subtracting the result. Denote the image intensity at (x k, y k ) by z k. Then the quadratic model for the intensities is z = Hθ + w, or z 1 x 2 1 y 2 1 x 1 y 1 x 1 y 1 1 c 1 z 2. = x 2 2 y 2 2 x 2 y 2 x 2 y 2 1 c w. x 2 N y2 N x Ny N x N y N 1 c 6 z N

41 Background Correction Left: Blue channel with uneven illumination. Center: Fitted quadratic surface. Right: Difference image. z(x, y) = x y xy x y

42 Extension: 2D Measurements In another project we were required to model displacements on a 2D grid 4 The measurement data consisted of 2D vector displacement measurements. In other words, we know that the displacements at points (x 1, y 1 ), (x 2, y 2 ),..., (x N, y N ) are ( x 1, y 1 ), ( x 2, y 2 ),..., ( x N, y N ). 4 Manninen, T., Pekkanen, V., Rutanen, K., Ruusuvuori, P., Rönkkä, R. and Huttunen, H., "Alignment of individually adapted print patterns for ink jet printed electronics," Journal of Imaging Science and Technology, 54(5), Oct

43 Extension: 2D Measurements This case was also modeled using a 2nd order polynomial model: x 1 y 1 x 2 1 y 2 1 x 1 y 1 x 1 y 1 1 a 1 b 1 x 2 y 2.. = x 2 2 y 2 2 x 2 y 2 x 2 y 2 1 a 2 b w. x N y N x 2 N y2 N x Ny N x N y N 1 a 6 b 6 The familiar formula θ = (H T H) 1 H T x applies also in this case. Note that it would have been equivalent to separate this to two linear models; one for x k and another for y k.

44 Results Below is an example of a resulting vector field.

45 Linear Models Summary If a linear model (x = Hθ + w) can be assumed, the MVUE reaching the CRLB can be found in closed form: θ = (H T H) 1 H T x. Matlab calls it theta = H \ x, and Excel LINEST. We will continue discussion on this topic in chapter 8: Least Squares (LS). It turns out that the linear LS estimator has exactly the above formula. The difference is that LS assumes nothing about the distribution, and thus has no guarantees for optimality or unbiasedness. Additionally, LS has numerous extensions to be discussed later.