SYSEM IDEIFICAIO Ali Karimpour Associa Prossor Frdowsi Univrsi o Mashhad Rrnc: Ssm Idniicaion hor For h Usr Lnnar Ljung
Lcur 7 lcur 7 Paramr Esimaion Mhods opics o b covrd includ: Guiding Principls Bhind Paramr Esimaion Mhod. Minimizing Prdicion Error. Linar Rgrssions and h Las-Suars Mhod. A Saisical Framwork or Paramr Esimaion and h Maximum Liklihood Mhod. Corrlaion Prdicion Errors wih Pas Daa. Insrumnal Variabl Mhods. Ali Karimpour Jan 04
Paramr Esimaion Mhod lcur 7 opics o b covrd includ: Guiding Principls Bhind Paramr Esimaion Mhod. Minimizing Prdicion Error. Linar Rgrssions and h Las-Suars Mhod. A Saisical Framwork or Paramr Esimaion and h Maximum Liklihood Mhod. Corrlaion Prdicion Errors wih Pas Daa. Insrumnal Variabl Mhods. 3 Ali Karimpour Jan 04
lcur 7 Ali Karimpour Jan 04 4 Guiding Principls Bhind Paramr Esimaion Mhod Paramr Esimaion Mhod D M M M H u G : u G H H M Suppos ha w hav slcd a crain modl srucur M. h s o modls dind as: For ach θ modl rprsns a wa o prdicing uur oupus. h prdicor is a linar ilr as: Suppos h ssm is:
Guiding Principls Bhind Paramr Esimaion Mhod lcur 7 Suppos ha w collc a s o daa rom ssm as: u u... u Formall w ar going o ind a map rom h daa o h s D M DM Such a mapping is a paramr simaion mhod. 5 Ali Karimpour Jan 04
Guiding Principls Bhind Paramr Esimaion Mhod lcur 7 Evaluaing h candida modl L us din h prdicion rror as: Whn h daa s is known hs rrors can b compud or = A guiding principl or paramr simaion is: Basd on w can compu h prdicion rror εθ. Slc prdicion rror W dscrib wo approachs = bcoms as small as possibl. so ha h? Form a scalar-valud cririon uncion ha masur h siz o ε. Mak uncorrlad wih a givn daa sunc. 6 Ali Karimpour Jan 04
Paramr Esimaion Mhod lcur 7 opics o b covrd includ: Guiding Principls Bhind Paramr Esimaion Mhod. Minimizing Prdicion Error. Linar Rgrssions and h Las-Suars Mhod. A Saisical Framwork or Paramr Esimaion and h Maximum Liklihood Mhod. Corrlaion Prdicion Errors wih Pas Daa. Insrumnal Variabl Mhods. 7 Ali Karimpour Jan 04
lcur 7 Ali Karimpour Jan 04 8 Minimizing Prdicion Error Clarl h siz o prdicion rror is h sam as L o ilr h prdicion rror b a sabl linar ilr L L F hn us h ollowing norm F l V Whr l. is a scalar-valud posiiv uncion. h sima is hn dind b: min arg D V M
Minimizing Prdicion Error lcur 7 F L V l F arg min D M V Gnrall h rm prdicion rror idniicaion mhods PEM is usd or h amil o his approachs. Choic o l. Paricular mhods wih spciic nams ar usd according o: Choic o L. Choic o modl srucur Mhod b which h minimizaion is ralizd 9 Ali Karimpour Jan 04
lcur 7 Ali Karimpour Jan 04 0 Minimizing Prdicion Error L F F l V min arg D V M Choic o L h c o L is bs undrsood in a runc-domain inrpraion. hus L acs lik runc wighing. S also >> 4.4 Prilring Exrcis 7-: Considr ollowing ssm H u G Show ha h c o prilring b L is idnical o changing h nois modl rom H L H
Minimizing Prdicion Error lcur 7 F L V l F arg min D M V Choic o l A sandard choic which is convnin boh or compuaion and analsis. l S also >> 5. Choic o norms: Robusnss agains bad daa On can also paramriz h norm indpndn o h modl paramrizaion. Ali Karimpour Jan 04
Paramr Esimaion Mhod lcur 7 opics o b covrd includ: Guiding Principls Bhind Paramr Esimaion Mhod. Minimizing Prdicion Error. Linar Rgrssions and h Las-Suars Mhod. A Saisical Framwork or Paramr Esimaion and h Maximum Liklihood Mhod. Corrlaion Prdicion Errors wih Pas Daa. Insrumnal Variabl Mhods. Ali Karimpour Jan 04
lcur 7 Ali Karimpour Jan 04 3 Linar Rgrssions and h Las-Suars Mhod W inroduc linar rgrssions bor as: φ is h rgrssion vcor and or h ARX srucur i is a n b u u n...... μ is a known daa dpndn vcor. For simplici l i zro in h rmindr o his scion. Las-suars cririon rror is : Prdicion ow l L= and lε= ε / hn l V F his is Las-suars cririon or h linar rgrssion
lcur 7 Ali Karimpour Jan 04 4 Linar Rgrssions and h Las-Suars Mhod Las-suars cririon l V F h las suar sima LSE is: LS V arg min R R LS
lcur 7 Ali Karimpour Jan 04 5 Linar Rgrssions and h Las-Suars Mhod W inroduc linar rgrssions bor as: V Las-suars cririon Y Y LS Y LS V arg min whi and is ois componns ar uncorlad wih rgrssors is invribl Suppos Φ Φ Undr abov assumpions h LSE is BLUE Bs linar unbiasd simaor.
Linar Rgrssions and h Las-Suars Mhod lcur 7 W inroduc linar rgrssions bor as: Las-suars cririon Suppos Φ Φ is invribl LS Y ois componns ar uncorlad wih rgrssors and is whi Undr abov assumpions h LSE is BLUE Bs linar unbiasd simaor. Linar mans Y Unbiasd mans E{ } Bs mans cov{ } minimum possibl covarianc 6 Ali Karimpour Jan 04
Linar Rgrssions and h Las-Suars Mhod lcur 7 W inroduc linar rgrssions bor as: Y Linar mans Y Condiion or linar unbiasd simaion? E{ θ} E{ Y } E{ } I =I and is uncorrlad wih hn h simaor is unbiasd. Clarl LSE is unbiasd. E{ θ} 7 Ali Karimpour Jan 04
Linar Rgrssions and h Las-Suars Mhod lcur 7 W inroduc linar rgrssions bor as: Y Linar mans Y Condiion or bs linar unbiasd simaion? cov{ θ} E{ θ θ θ θ } E{ Y E{ θ Y θ } θ θ I h simaor is unbiasd hn =I so: cov{ θ} E{ θ θ θ θ } E{ } θ θ } I h simaor is unbiasd hn is uncorrlad wih so: cov{ θ} E{ }cov E{ } W mus minimiz i? 8 Ali Karimpour Jan 04
Linar Rgrssions and h Las-Suars Mhod lcur 7 mincov{θ}...... subjc o I Exrcis 7-: Show ha h answr o abov opimizaion is: So LSE is BLUE sinc: LS Y Exrcis 7-3: Show ha b LSE in linar rgrssion on can ind an unbiasd sima o cov{} b d V LS 9 Ali Karimpour Jan 04
lcur 7 Ali Karimpour Jan 04 0 Linar Rgrssions and h Las-Suars Mhod Wighd Las Suars Dirn masurmn could b assignd dirn wighs V or V LS h rsuling sima is h sam as prvious.
Linar Rgrssions and h Las-Suars Mhod lcur 7 W show ha in a dirnc uaion Colord Euaion-rror ois a b u... a... b v i h disurbanc v is no whi nois hn h LSE will no convrg o h ru valu a i and b i. o dal wih his problm w ma incorpora urhr modling o h uaion rror v as discussd in chapr 4 l us sa n b n a u v k n n ow is whi nois bu h nw modl ak us ou rom LS nvironmn xcp in wo cass: Known nois propris b a High-ordr modls Ali Karimpour Jan 04
lcur 7 Ali Karimpour Jan 04 Linar Rgrssions and h Las-Suars Mhod Colord Euaion-rror ois Known nois propris...... v n u b b u n a a b n a n b a Suppos h valus o a i and b i ar unknown bu k is a known ilr no oo ralisic a siuaion so w hav k v Filring hrough k - givs whr Sinc is whi h LS mhod can b applid wihou problms. oic ha his is uivaln o appling h ilr L=k -. k u B A u B A u k u k
lcur 7 Ali Karimpour Jan 04 3 Linar Rgrssions and h Las-Suars Mhod Colord Euaion-rror ois ow w can appl LS mhod. o ha n A =n a +r n B =n b +r High-ordr modls...... v n u b b u n a a b n a n b a Suppos ha h nois v can b wll dscribd b k=/d whr D is a polnomial o ordr r. So w hav k v D u B A or u D B D A Ar driving AD and BD on can asil driv A and B. A B D A D B
Paramr Esimaion Mhod lcur 7 opics o b covrd includ: Guiding Principls Bhind Paramr Esimaion Mhod. Minimizing Prdicion Error. Linar Rgrssions and h Las-Suars Mhod. A Saisical Framwork or Paramr Esimaion and h Maximum Liklihood Mhod. Corrlaion Prdicion Errors wih Pas Daa. Insrumnal Variabl Mhods. 4 Ali Karimpour Jan 04
A Saisical Framwork or Paramr Esimaion and h Maximum Liklihood Mhod lcur 7 Esimaion and h Principl o Maximum Liklihood h ara o saisical inrnc dals wih h problm o xracing inormaion rom obsrvaions ha hmslvs could b unrliabl. Suppos ha obsrvaion = has ollowing probabili dnsi uncion PDF ha is: ; x x... x P ; x A ; x x A θ is a d-dimnsional paramr vcor. h propos o h obsrvaion is in ac o sima h vcor θ using. Suppos h obsrvd valu o is * hn R * * R d dx 5 Ali Karimpour Jan 04
A Saisical Framwork or Paramr Esimaion and h Maximum Liklihood Mhod lcur 7 Esimaion and h Principl o Maximum Liklihood Man such simaor uncions ar possibl. A paricular on >>>>>>>>> maximum liklihood simaor MLE. R h probabili ha h ralizaion=obsrvaion indd should ak h valu * is proporional o * his is a drminisic uncion o θ onc h numrical valu * isinsrd and i is calld Liklihood uncion. A rasonabl simaor o θ could hn b ML * arg max R d whr h maximizaion prormd or ixd *. his uncion is known as h maximum liklihood simaor MLE. 6 * Ali Karimpour Jan 04
A Saisical Framwork or Paramr Esimaion and h Maximum Liklihood Mhod lcur 7 Exampl 7-: L i i... B indpndn random variabls wih normal disribuion wih unknown mans θ 0 and known variancs λ i i 0 i A common simaor is h sampl man: SM i i o calcula MLE w sar o drmin h join PDF or h obsrvaions. h PDF or i is: x i xp i i Join PDF or h obsrvaions is: sinc i ar indpndn ; x i i xp xi i 7 Ali Karimpour Jan 04
lcur 7 Ali Karimpour Jan 04 8 A Saisical Framwork or Paramr Esimaion and h Maximum Liklihood Mhod Exampl 7-: L i i... B indpndn random variabls wih normal disribuion wih unknown mans θ 0 and known variancs λ i A common simaor is h sampl man: i SM i Join PDF or h obsrvaions is: sinc i ar indpndn i i i i x x xp ; So h liklihood uncion is: ; Maximizing liklihood uncion is h sam as maximizing is logarihm. So ; arg max log ML i i i i i log arg max i i i i ML i /
A Saisical Framwork or Paramr Esimaion and h Maximum Liklihood Mhod lcur 7 Exampl 7-: L i i... B indpndn random variabls wih normal disribuion wih unknown mans θ 0 and known variancs λ i i 0 i Suppos =5 and i is drivd rom a random gnraion normal disribuion such ha h mans is 0 bu variancs ar: 0 3 4 6 0. 0 6 9 3 5 h simad mans or 0 dirn 40 xprimns ar shown in h igur: SM ML i i i / i i i i Dirn simaors 0 0-0 SM 4 6 8 0 Dirn xprimns ML Exrcis 6-4:Do h sam procdur or anohr xprimns and draw h corrsponding igur. Exrcis 6-5:Do h sam procdur or anohr xprimns and draw h corrsponding 9igur. Suppos all variancs as 0. Ali Karimpour Jan 04
A Saisical Framwork or Paramr Esimaion and h Maximum Liklihood Mhod lcur 7 Maximum liklihood simaor MLE ML * arg max Rlaionship o h Maximum A Posriori MAP Esima h Basian approach is usd o driv anohr paramr simaion problm. In h Basian approach h paramr isl is hough o as a random variabl. * L h prior PDF or θ is: Ar som manipulaion g z P z h Maximum A Posriori MAP sima is: MAP P ; g arg max. g 30 Ali Karimpour Jan 04
A Saisical Framwork or Paramr Esimaion and h Maximum Liklihood Mhod lcur 7 } Cramr-Rao Inuali h uali o an simaor can b assssd b is man-suar rror marix: P cov{ E 0 0 ru valu o θ W ma b inrsd in slcing simaors ha mak P small. Cramr-Rao inuali giv a lowr bound or P L hn P E E M 0 0 0 M is Fishr Inormaion marix An simaor is icin i P=M - Exrcis 7-6:Proo Cramr-Rao inuali. 3 Ali Karimpour Jan 04
A Saisical Framwork or Paramr Esimaion and h Maximum Liklihood Mhod lcur 7 Asmpoic Propris o h MLE Calculaion o P E 0 0 Is no an as ask. hror limiing propris as h sampl siz nds o inini ar calculad insad. For h MLE in cas o indpndn obsrvaions Wald and Cramr obain Suppos ha h random variabl {i} ar indpndn and idnicall disribud so ha x... x i ; xi i ; x Suppos also ha h disribuion o is givn b θ 0 ;x or som valu θ 0. hn nds o θ 0 wih probabili as nds o inini and ML ML 0 convrgs in disribuion o h normal disribuion wih zro man covarianc marix givn b Cramr-Rao lowr bound M -. 3 Ali Karimpour Jan 04
A Saisical Framwork or Paramr Esimaion and h Maximum Liklihood Mhod lcur 7 Suppos Probabilisic Modls o Dnamical Ssms M : g ; is indpndn and hav h PDF x ; Rcall his kind o modl a compl probabilisic modl. Liklihood uncion or Probabilisic Modls o Dnamical Ssms W no ha h oupu is: whr has h PDF x ; ow w mus drmin h liklihood uncion ; 33 Ali Karimpour Jan 04
lcur 7 Ali Karimpour Jan 04 34 A Saisical Framwork or Paramr Esimaion and h Maximum Liklihood Mhod? ; Lmma: Suppos u is givn as a drminisic sunc and assum ha h gnraion o is dscribd b h modl is PDF o h condiional whr x g hn h join probabili dnsi uncion or givn u is: I k k g k u k k m Proo: CPDF o givn - is g x x p Using Bas s rul h join CPDF o and - givn - can b xprssd as:. g x g x. x p x x p x x p Similarl w driv I
lcur 7 Ali Karimpour Jan 04 35 A Saisical Framwork or Paramr Esimaion and h Maximum Liklihood Mhod ; : g M Suppos ; PDF is indpndn and hav h x ow w mus drmin h liklihood uncion Probabilisic Modls o Dnamical Ssms B prvious lmma g ; ; ; ; Maximizing his uncion is h sam as maximizing ; log ; log I w din ; log l
A Saisical Framwork or Paramr Esimaion and h Maximum Liklihood Mhod lcur 7 Probabilisic Modls o Dnamical Ssms Maximizing his uncion is h sam as maximizing I w din W ma wri log ML ; log l log ; arg min ; l ; h ML mhod can hus b sn as a spcial cas o h PEM. Exrcis 7-7 Find h Fishr inormaion marix or his ssm. Exrcis 7-8: Driv a lowr bound or. Cov 36 Ali Karimpour Jan 04
Paramr Esimaion Mhod lcur 7 opics o b covrd includ: Guiding Principls Bhind Paramr Esimaion Mhod. Minimizing Prdicion Error. Linar Rgrssions and h Las-Suars Mhod. A Saisical Framwork or Paramr Esimaion and h Maximum Liklihood Mhod. Corrlaion Prdicion Errors wih Pas Daa. Insrumnal Variabl Mhods. 37 Ali Karimpour Jan 04
Corrlaion Prdicion Errors wih Pas Daa lcur 7 Idall h prdicion rror εθ or good modl should b indpndn o h pas daa - I εθ is corrlad wih - hn hr was mor inormaion availabl in - abou han pickd up b o s i εθ is indpndn o h daa s - w mus chck All ransormaion o εθ his is o cours no asibl in pracic. Uncorrlad wih 0 All possibl uncion o - Insad w ma slc a crain ini-dimnsional vcor sunc {ζ} drivd rom - and a crain ransormaion o {εθ} o b uncorrlad wih his sunc. his would giv Drivd θ would b h bs sima basd on h obsrvd daa. 38 Ali Karimpour Jan 04
lcur 7 Ali Karimpour Jan 04 39 Corrlaion Prdicion Errors wih Pas Daa L F Choos a linar ilr L and l Choos a sunc o corrlaion vcors Choos a uncion αε and din F hn calcula 0 D sol M Insrumnal variabl mhod nx scion is h bs known rprsnaiv o his amil.
lcur 7 Ali Karimpour Jan 04 40 Corrlaion Prdicion Errors wih Pas Daa 0 D sol M F ormall h dimnsion o ξ would b chosn so ha is a d-dimnsional vcor. hn hr is man uaions as unknowns. Somims on us ξ wih highr dimnsion han d so hr is an ovr drmind s o uaions picall wihou soluion. so arg min D M Exrcis 7-9: Show ha h prdicion-rror sima obaind rom min arg D V M can b also sn as a corrlaion sima or a paricular choic o L ζ and α.
lcur 7 Ali Karimpour Jan 04 4 Corrlaion Prdicion Errors wih Pas Daa Psudolinar Rgrssions W saw in chapr 4 ha a numbr o common prdicion modls could b wrin as: Psudo-rgrssion vcor φθ conains rlvan pas daa i is rasonabl o ruir h rsuling prdicion rrors b uncorrlad wih φθ so: PLR sol 0 Which w rm h PLR sima. Psudo linar rgrssions sima.
Paramr Esimaion Mhod lcur 7 opics o b covrd includ: Guiding Principls Bhind Paramr Esimaion Mhod. Minimizing Prdicion Error. Linar Rgrssions and h Las-Suars Mhod. A Saisical Framwork or Paramr Esimaion and h Maximum Liklihood Mhod. Corrlaion Prdicion Errors wih Pas Daa. Insrumnal Variabl Mhods. 4 Ali Karimpour Jan 04
Insrumnal Variabl Mhods lcur 7 Considr linar rgrssion as: h las-suar sima o θ is givn b LS sol 0 So i is a kind o PEM wih L= and ξθ=φ ow suppos ha h daa acuall dscribd b 0 v0 W ound in scion 7.3 ha LSE will no nd o θ 0 in pical cass. 43 Ali Karimpour Jan 04
lcur 7 Ali Karimpour Jan 04 44 Insrumnal Variabl Mhods LS sol 0 0 0 v W ound in scion 7.3 ha LSE will no nd o θ 0 in pical cass. IV sol 0 Such an applicaion o a linar rgrssion is calld insrumnal-variabl mhod. h lmns o ξ ar hn calld insrumns or insrumnal variabls. Esimad θ is: IV
lcur 7 Ali Karimpour Jan 04 45 Insrumnal Variabl Mhods LS sol 0 in IV mhod? as Dos 0 Exrcis 7-0: Show ha will b xis and nd o θ 0 i ollowing uaions xiss. IV 0 nonsingular b 0 v Eξ E W ound in scion 7.3 ha LSE will no nd o θ 0 in pical cass. IV sol 0 IV
Insrumnal Variabl Mhods lcur 7 So w nd Considr an ARX modl E Eξ v0 b 0 nonsingular I II a... an na b u... bn u nb v a A naural ida is o gnra h insrumns so as o scur II bu also considr I K x... x n u... u n a b Whr K is a linar ilr and x is gnrad hrough a linar ssm x M u b Mos insrumns usd in pracic ar gnrad in his wa. II and I ar saisid. Wh? 46 Ali Karimpour Jan 04