Least Squares Estimation Namrata Vaswani,

Transcription

1 Least Squares Estimation Namrata Vaswani, Least Squares Estimation 1

2 Recall: Geometric Intuition for Least Squares Minimize J(x) = y Hx 2 Solution satisfies: H T H ˆx = H T y, i.e. ˆx = (H T H) 1 H T y So H T (y H ˆx) = 0 The least error (y H ˆx) is to column space of H Think 3D: minimum error is always to plane of projection Least Squares Estimation 2

3 Weighted Least Squares y = Hx + e Minimize Solution: J(x) = (y Hx) T W (y Hx) y Hx 2 W (1) ˆx = (H T W H) 1 H T W y (2) Given that E[e] = 0 and E[ee T ] = V, Min. Variance Unbiased Linear Estimator of x: choose W = V 1 in (2) Min. Variance of a vector: variance in any direction is minimized Least Squares Estimation 3

4 Proof (skip if you want to) Given ˆx = Ly, find L, s.t. E[Ly] = E[LHx] = E[x], so LH = I Let L 0 = (H T V 1 H) 1 H T V 1 Error variance E[(x ˆx)(x ˆx) T ] E[(x ˆx)(x ˆx) T ] = E[(x LHx Le)(x LHX Le) T ] = E[Lee T L T ] = LV L T Say L = L L 0 + L 0. Here LH = I, L 0 H = I, so (L L 0 )H = 0 LV L T = L 0 V L T 0 + (L L 0 )V (L L 0 ) T + 2L 0 V (L L 0 ) T = L 0 V L T 0 + (L L 0 )V (L L 0 ) T + (H T V 1 H) 1 H T (L L 0 ) T = L 0 V L T 0 + (L L 0 )V (L L 0 ) T L 0 V L T 0 Thus L 0 is the optimal estimator (Note: for matrices) Least Squares Estimation 4

5 Regularized Least Squares Minimize J(x) = (x x 0 ) T Π 1 0 (x x 0) + (y Hx) T W (y Hx) (3) x x x 0, y y Hx 0 J(x) = x T Π 1 0 x + y T W y = z M 1 z z 0 y x M Π W I H Least Squares Estimation 5

6 Solution: Use weighted least squares formula with ỹ = 0 y, H = I H, W = M Get: ˆx = x 0 + (Π H T W H) 1 H T W (y Hx 0 ) Advantage: improves condition number of H T H, incorporate prior knowledge about distance from x 0 Least Squares Estimation 6

7 Recursive Least Squares Use in one of following situations: When number of equations much larger than number of variables: Storage problem Getting data sequentially, do not want to re-solve the full problem again The dimension of x is large, want to avoid inverting matrices Goal: At step i 1, have ˆx i 1 : minimizer of (x x 0 ) T Π 1 0 (x x 0) + H i 1 x Y i 1 2 W i 1, Y i 1 = [y 1,...y i 1 ] T Find ˆx i : minimizer of (x x 0 ) T Π 1 0 (x x 0) + H i x Y i 2 W i, Least Squares Estimation 7

8 H i = H i 1 h i (h i is a row vector), Y i = [y 1,...y i ] T (column vector) For simplicity of notation, assume x 0 = 0 and W i = I. Define H T i H i = H T i 1H i 1 + h T i h i ˆx i = (Π H T i H i ) 1 H T i Y i = (Π H T i 1H i 1 + h T i h i ) 1 (H T i 1Y i 1 + h T i y i ) P i = (Π H T i H i ) 1, P 0 = Π 0 So P i = [P 1 i 1 + ht i h i ] 1 Use Matrix Inversion identity: Least Squares Estimation 8

9 (A + BCD) 1 = A 1 + A 1 B(C 1 + DA 1 B) 1 DA 1 where Thus P i = P i 1 K i h i P i 1 K i = P i 1 h T i (1 + h i P i 1 h T i ) 1 (4) ˆx 0 = 0 ˆx i = P i H T i Y i = [P i 1 K i h i P i 1 ][H T i 1Y i 1 + h T i y i ] = ˆx i 1 + K i (y i h iˆx i 1 ) The last equality uses the facts that (i) ˆx i 1 = P i 1 H T i 1 Y i 1, (ii) [P i 1 K i h i P i 1 ]h T i y i = K i y i (expand K i, obtain this after a few manipulations). Least Squares Estimation 9

10 Here we considered the weight W i = I. If W i I, the equation for K i modifies to (replace y i by w 1/2 i y i & h i by w 1/2 i h i, where w i = (W i ) i,i ) K i = P i 1 h T i (w i 1 + h i P i 1 h T i ) 1 (5) Also, here we considered y i to be a scalar and h i to be a row vector. In general: y i can be a k-dim vector, h i will be a matrix with k rows, and the above formulae still apply, replace 1 by I everywhere RLS with Forgetting factor Weight older data with smaller weight J(x) = i j=1 (y j h j x) 2 β(i, j) Exponential forgetting: β(i, j) = λ i j, λ < 1 Moving average: β(i, j) = 0 if i j > and β(i, j) = 1 otherwise Least Squares Estimation 10

11 Summarizing Recursive LS In general can assume that y i is k dimensional and so h i has k rows. Weight matrix (W i ) i,i = w i. Solution is: ˆx 0 = x 0, P 0 = Π 0 K i = P i 1 h T i (w i 1 + h i P i 1 h T i ) 1 P i = (I K i h i )P i 1 ˆx i = ˆx i 1 + K i (y i h i x i ) (6) This is a recursive way to get the Regularized LS solution ˆx i = (Π H T i W i H i ) 1 Y i (7) with H i = [h T 1, h T 2,...h T i ]T, Y i = [y T 1, y T 2,...y T i ]T Least Squares Estimation 11

12 Connection with Kalman Filtering The above is also the Kalman filter estimate of the state for the following system model: x i = x i 1 y i = h i x i + v i, v i N (0, R i ), w i = R 1 i (8) Least Squares Estimation 12

13 Kalman Filter Motivation RLS was for static data: estimate the signal x better and better as more and more data comes in, e.g. estimating the mean intensity of an object from a video sequence RLS with forgetting factor assumes slowly time varying x Kalman filter: if the signal is time varying, and we know (statistically) the dynamical model followed by the signal: e.g. tracking a moving object x 0 N (0, Π 0 ) x i = F i x i 1 + v x,i, v x,i N (0, Q i ) The observation model is as before: y i = h i x i + v i, v i N (0, R i ) Least Squares Estimation 13

14 Goal: get the best (minimum mean square error) estimate of x i from Y i Cost: J(ˆx i ) = E[(x i ˆx i ) 2 Y i ] Minimizer: conditional mean ˆx i = E[x i Y i ] This is also the MAP estimate, i.e. ˆx i also maximizes p(x i Y i ) Least Squares Estimation 14

15 Example Applications Recursive LS: Adaptive noise cancelation Channel equalization using a training sequence Object intensity estimation: x = intensity, y i = vector of intensities of object region in frame i, h i = 1 m (column vector of m ones) Keep updating estimate of location of an object that is static, using a sequence of location observations coming in sequentially Recursive LS with forgetting factor: object not static but drifts very slowly (e.g. floating object) or object intensity changes very slowly Kalman filter: Track a moving object (estimate its location, velocity at each time), when acceleration is assumed i.i.d. Gaussian Least Squares Estimation 15

16 Material adapted from Chapters 2, 3 of Linear Estimation, by Kailath, Sayed, Hassibi Least Squares Estimation 16