Least Squares Methods

Transcription

1 Det. of Biomed. Eg. BME80: Iverse Problems i Bioegieerig Kug ee Uiv. Least Squares Methods Overdetermied liear equatios m where R ad m > More equatios tha ukows Caot solve for i most cases. Least squares solutio of overdetermied liear equatios Residual or error is r. Fid ls arg mi r R ls R( ) ad ls is closest to. ls is rojectio of o R(). ssume is full rak. From d r d. d 0, d ls he otimal residual is ls * r I. Pseudoiverse is a left iverse of ; I Projectio of o R() is P. P where rojectio matri is ls Orthogoalit ricile: r R( ), i.e., z R( ), rz, I z 0 Least squares estimatio Model: v () is what we wat to estimate. () is sesor measuremets. (3) v is ukow oise or measuremet error htt://iirc.khu.ac.kr Eug Je Woo

2 Det. of Biomed. Eg. BME80: Iverse Problems i Bioegieerig Kug ee Uiv. Least squares estimate, ˆ arg mi () If v = 0, ˆ. ubiased () Liear estimator, i.e., ˆ B for some B (3) ˆ is the best liear ubiased estimator (BLUE). Least squares data fittig Prearatio () Fuctios f,, : f S R, S R : basis fuctios or regressors m (), g s with s i S ad m > : data or measuremets i i i Problem: fid a liear combiatio of fuctios that fits data, i.e., j f j( s i) gi for i =,,, m j * Least squares solutio: () g g m () g (3) aij f j ( si) R arg mi g g m Fittig fuctio: fls () s f() s f() s Choosig the value of : lot r = g as a fuctio of Least squares olomial fittig for S R () f () s s j j () a ij (Vadermode matri) j s i licatios () Iterolatio, etraolatio, smoothig of data () Simle ad aroimate model of data Least squares sstem idetificatio Problem: fid a model for ukow sstem from iutoutut data htt://iirc.khu.ac.kr Eug Je Woo

3 Det. of Biomed. Eg. BME80: Iverse Problems i Bioegieerig Kug ee Uiv. [ ] Ukow Sstem [ ] ssume a model: for eamle, M() Collect iutoutut data with N > ( + ) [ ] w [ i] i0 i () [ ] [ ] [0] w 0 [ ] [ ] [] w w [ N] [ N] N [ ] w [ ] [ ] () [ N] * Least squares solutio: w arg mi w w Choosig the order of the model () () Larger (a) Small error for a articular set of data (smaller bias) (b) r to fit eve oise (overfit or overmodelig) (c) Poor geeralizatio ad oor redictive abilit (larger variace) () Crossvalidatio: use differet set of data ad moitor the chage of error o this data set Multiobjective least squares wo (cometig) objectives () () J J B z Weighted sum objective () J J J B z with 0 htt://iirc.khu.ac.kr 3 Eug Je Woo

4 Det. of Biomed. Eg. BME80: Iverse Problems i Bioegieerig Kug ee Uiv. () B z B z Least squares solutio: ˆ B B B z Regularized least squares With B = I ad z = 0, J J J () here is a ealt for large. () It stabilizes the algorithm. (3) It works for a. (4) ikhoov regularizatio ˆ I Least squares solutio: () Useful whe is illcoditioed. () Useful whe we kow is small. (3) Useful whe model is accurate ol for small. Uderdetermied liear equatios m where R with m < More ukows tha equatios is uder secified ad there are ma solutios. m ssumig rak() =, for each R () Set of solutios is : z:, zn( ) () Solutio has dim N( ) m degree of freedom (3) What is the best solutio? Miimum orm solutio m where R with m < ad rak() = Miimum orm solutio: For a such that =, (), 0, i.e., htt://iirc.khu.ac.kr 4 Eug Je Woo

5 Det. of Biomed. Eg. BME80: Iverse Problems i Bioegieerig Kug ee Uiv. () Orthogoalit coditio: N ( ) is rojectio of 0 o the solutio set : z:, zn( ) is seudoiverse of Derivatio () Formulatio: mi subject to () Lagrage multilier, L(, ) (3) Otimalit coditios L (a) 0 L 0 (b) (4) Comarig regularized least squares solutio, I 0 htt://iirc.khu.ac.kr 5 Eug Je Woo

6 Det. of Biomed. Eg. BME80: Iverse Problems i Bioegieerig Kug ee Uiv. Least Squares Estimatio Liear model: ad h, h,, h ihi or i h i i N N R : measuremets, kow N N R : model outut, sigal comoet, ukow h h h h h h R hn hn hn N : model structure, assumed to be kow R : model arameter, ukow N N R : measuremet error, oise comoet, ukow or kow statistics Least squares solutio ad e tr e ad e 0 ˆ G Normal equatio: ˆ or ˆ G Gram matri or Grammia, G is osigular iff ii h are liearl ideedet. Let sa ii Projectios h ad assume N R ad ˆ (a) ˆ P, P : rojectio oto htt://iirc.khu.ac.kr 6 Eug Je Woo

7 Det. of Biomed. Eg. BME80: Iverse Problems i Bioegieerig Kug ee Uiv. ˆ ˆ IN P P, P IN P : orthogoal oto (b) (c) P P ad P P : smmetric (d) P PP P ad P PP P : idemotet (e) PP PP 0: orthogoal (f) P P I : decomositio of idetit N (g) lso ote that P, P0, P,, P0, Geometr P P h : sigal subsace with sa ii sa ii P N a : orthogoal subsace with P (a) Costruct a a a N( N ) N R so that (b) ah i j 0, i,, ( N ), j,, 0 N I P P R N, I P P ˆ ˆ (a), P (b), P htt://iirc.khu.ac.kr 7 Eug Je Woo

8 Det. of Biomed. Eg. BME80: Iverse Problems i Bioegieerig Kug ee Uiv. Orthogoalit N R, I P P ˆ ˆ ˆ ˆ PP 0 N P I P ˆ ˆ : orthogoal decomositio of (a) (b) ˆ P ˆ : miimum orm, i.e., least squares P P Eamle: comle eoetial modes aalsis jit jt j j t t Model: () t ie e e e, i.e., sum of comle i eoetials From t () t () t (), take N measuremets at t k, k = 0,,,, (N ). WLOG, assume =. (0) (0) j j j () e e e () j j j () () e e e, or N ( ) j( N) ( ) j( N) j N N ( ) e e e (0) j j j j 3 e e e e () j j j j 3 e e e e 3 (), or j( N) ( ) j( N) j( N) j N 3 e e e e N ( ) htt://iirc.khu.ac.kr 8 Eug Je Woo

9 Det. of Biomed. Eg. BME80: Iverse Problems i Bioegieerig Kug ee Uiv. ˆ ˆ ˆ P Least squares solutio: ad Eamle: olomial curve fittig i Model: () t it t3t t, i.e., olomial i From t () t () t (), take N measuremets at t k assume =., k =,,, N. WLOG, () () () () (3) 3 3 (3), or N ( ) N N ( ) () () (3) N N N ( N) ˆ, or ˆ ˆ P Least squares solutio: ad Recursive least squares (RLS) t t t t h t t t h t t t t t h for t =,, 3, t t t Defie P G hh P hh. t t t t t t t t t t t h t t t t t t tt h t t htt://iirc.khu.ac.kr 9 Eug Je Woo

10 Det. of Biomed. Eg. BME80: Iverse Problems i Bioegieerig Kug ee Uiv. LS solutio is ˆ t P t t t Pt hh t t tth tt. From matri iversio lemma, P hh P P hh P ad t t t t t t t t t hp h or hp h t t t t t t t t t ˆ t Pt tpth tht Pt t t h tt t tpth t t h t t Defie k P h t t t t t t k t z t ~ h t Summar of RLS () Iitializatio: P0 I ad ˆ 0 0 () hp h t t t t (3) k P h t t t t (4) k h t t t t t t (5) Pt Pt tpthh t t Pt Weighted least squares Problem: mi W with osigular smmetric W R NN 0 W 0 ˆ Solutio: Choice of W W W htt://iirc.khu.ac.kr 0 Eug Je Woo

11 Det. of Biomed. Eg. BME80: Iverse Problems i Bioegieerig Kug ee Uiv. () If :N 0, R, W R. () w w diag,, N W where wi. SNRi htt://iirc.khu.ac.kr Eug Je Woo