Least-squares data fitting

Transcription

1 EE263 Autumn 2015 S. Boyd and S. Lall Least-squares data fitting 1

2 Least-squares data fitting we are given: functions f 1,..., f n : S R, called regressors or basis functions data or measurements (s i, g i), i = 1,..., m, where s i m n S and (usually) problem: find coefficients x 1,..., x n R so that x 1f 1(s i) + + x nf n(s i) g i, i = 1,..., m i.e., find linear combination of functions that fits data least-squares fit: choose x to minimize total square fitting error: m (x 1f 1(s i) + + x nf n(s i) g i) 2 i=1 2

3 Least-squares data fitting total square fitting error is Ax g 2, where A ij = f j(s i) hence, least-squares fit is given by (assuming A is skinny, full rank) corresponding function is applications: x = (A T A) 1 A T g f lsfit (s) = x 1f 1(s) + + x nf n(s) interpolation, extrapolation, smoothing of data developing simple, approximate model of data 3

4 Least-squares polynomial fitting problem: fit polynomial of degree < n, p(t) = a 0 + a 1t + + a n 1t n 1, to data (t i, y i), i = 1,..., m basis functions are f j(t) = t j 1, j = 1,..., n matrix A has form A ij = t j 1 i 1 t 1 t 2 1 t n t 2 t 2 2 t n 1 2 A =.. 1 t m t 2 m t n 1 m (called a Vandermonde matrix) 4

5 Vandermonde matrices assuming t k t l for k l and m n, A is full rank: suppose Aa = 0 corresponding polynomial p(t) = a a n 1t n 1 vanishes at m points t 1,..., t m by fundamental theorem of algebra p can have no more than n 1 zeros, so p is identically zero, and a = 0 columns of A are independent, i.e., A full rank 5

6 Example fit g(t) = 4t/(1 + 10t 2 ) with polynomial m = 100 points between t = 0 & t = 1 fits for degrees 1, 2, 3, 4 have RMS errors.135,.076,.025,.005, respectively degree degree degree degree

7 Growing sets of regressors consider family of least-squares problems minimize p x ia i y i=1 for p = 1,..., n (a 1,..., a p are called regressors) approximate y by linear combination of a 1,..., a p project y onto span{a 1,..., a p} regress y on a 1,..., a p as p increases, get better fit, so optimal residual decreases 7

8 Growing sets of regressors solution for each p n is given by where x (p) ls = (A T pa p) 1 A T py = Rp 1 Q T py A p = [a 1 a p] R m p is the first p columns of A A p = Q pr p is the QR factorization of A p R p R p p is the leading p p submatrix of R Q p = [q 1 q p] is the first p columns of Q 8

9 Norm of optimal residual versus p plot of optimal residual versus p shows how well y can be matched by linear combination of a 1,..., a p, as function of p residual y min x x 1a 1 y 1 min x 1,...,x 7 7 x ia i y i= p 9

10 Least-squares system identification we measure input u(t) and output y(t) for t = 0,..., N of unknown system u(t) unknown system y(t) system identification problem: find reasonable model for system based on measured I/O data u, y example with scalar u, y (vector u, y readily handled): fit I/O data with movingaverage (MA) model with n delays where h 0,..., h n R ŷ(t) = h 0u(t) + h 1u(t 1) + + h nu(t n) 10

11 System identification we can write model or predicted output as ŷ(n) u(n) u(n 1) u(0) ŷ(n + 1). = u(n + 1) u(n) u(1)... ŷ(n) u(n) u(n 1) u(n n) model prediction error is e = (y(n) ŷ(n),..., y(n) ŷ(n)) least-squares identification: choose model (i.e., h) that minimizes norm of model prediction error e... a least-squares problem (with variables h) h 0 h 1. h n 11

12 Example data used to fit model u(t) t y(t) t 12

13 Example for n = 7 we obtain MA model with (h 0,..., h 7) = (.024,.282,.418,.354,.243,.487,.208,.441) with relative prediction error e / y = y(t) actual output, ŷ(t) predicted from model 13

14 Model order selection question: how large should n be? obviously the larger n, the smaller the prediction error on the data used to form the model suggests using largest possible model order for smallest prediction error 14

15 Model order selection relative prediction error e / y n difficulty: for n too large the predictive ability of the model on other I/O data (from the same system) becomes worse 15

16 Out of sample validation evaluate model predictive performance on another I/O data set not used to develop model model validation data set check prediction error of models (developed using modeling data) on validation data plot suggests n = 10 is a good choice relative prediction error e / y n 16

17 Validation for n = 50 the actual and predicted outputs on system identification and model validation data are: t t y(t) actual output, ŷ(t) predicted from model loss of predictive ability when n too large called model overfit or overmodeling 17