References are appeared in the last slide. Last update: (1393/08/19)

Transcription

1 SYSEM IDEIFICAIO Ali Karimpour Associae Professor Ferdowsi Universi of Mashhad References are appeared in he las slide. Las updae: /08/9

2 Lecure 5 lecure 5 Parameer Esimaion Mehods opics o be covered include: Guiding Principles Behind Parameer Esimaion Mehod. Minimizing Predicion Error. Linear Regressions and he Leas-Suares Mehod. A Saisical Framework for Parameer Esimaion and he Maximum Likelihood Mehod. Correlaion Predicion Errors wih Pas Daa. Insrumenal Variable Mehods. 2 Ali Karimpour ov 204

3 Parameer Esimaion Mehod lecure 5 opics o be covered include: Guiding Principles Behind Parameer Esimaion Mehod. Minimizing Predicion Error. Linear Regressions and he Leas-Suares Mehod. A Saisical Framework for Parameer Esimaion and he Maximum Likelihood Mehod. Correlaion Predicion Errors wih Pas Daa. Insrumenal Variable Mehods. 3 Ali Karimpour ov 204

4 lecure 5 Ali Karimpour ov Guiding Principles Behind Parameer Esimaion Mehod Parameer Esimaion Mehod D M M M e H u G : u G H H M Suppose ha we have seleced a cerain model srucure M. he se of models defined as: For each θ model represens a wa of predicing fuure oupus. he predicor is a linear filer as: Suppose he ssem is:

5 Guiding Principles Behind Parameer Esimaion Mehod lecure 5 Suppose ha we collec a se of daa from ssem as: u 2 u2... u Formall we are going o find a map from he daa o he se D M DM Such a mapping is a parameer esimaion mehod. 5 Ali Karimpour ov 204

6 Guiding Principles Behind Parameer Esimaion Mehod lecure 5 Evaluaing he candidae model Le us define he predicion error as: When he daa se is known hese errors can be compued for = 2 A guiding principle for parameer esimaion is: Based on we can compue he predicion error εθ. Selec predicion error We describe wo approaches = 2 becomes as small as possible. so ha he? Form a scalar-valued crierion funcion ha measure he size of ε. Make uncorrelaed wih a given daa seuence. 6 Ali Karimpour ov 204

7 Parameer Esimaion Mehod lecure 5 opics o be covered include: Guiding Principles Behind Parameer Esimaion Mehod. Minimizing Predicion Error. Linear Regressions and he Leas-Suares Mehod. A Saisical Framework for Parameer Esimaion and he Maximum Likelihood Mehod. Correlaion Predicion Errors wih Pas Daa. Insrumenal Variable Mehods. 7 Ali Karimpour ov 204

8 lecure 5 Ali Karimpour ov Minimizing Predicion Error Clearl he size of predicion error is he same as Le o filer he predicion error b a sable linear filer L L F hen use he following norm F l V Where l. is a scalar-valued posiive funciona norm. he esimae is hen defined b: min arg D V M

9 Minimizing Predicion Error lecure 5 F L V l F arg min D M V Generall he erm predicion error idenificaion mehods PEM is used for he famil of his approaches. Choice of l. Paricular mehods wih specific names are used according o: Choice of L. Choice of model srucure Mehod b which he minimizaion is realized 9 Ali Karimpour ov 204

10 lecure 5 Ali Karimpour ov Minimizing Predicion Error L F F l V min arg D V M Choice of L he effec of L is bes undersood in a freuenc-domain inerpreaion. hus L acs like freuenc weighing. See also >> 4.4 Prefilering [Lennar Ljung 999] Exercise 5-: Consider following ssem e H u G Show ha he effec of prefilering b L is idenical o changing he noise model from H L H

11 Minimizing Predicion Error lecure 5 F L V l F arg min D M V Choice of l A sandard choice which is convenien boh for compuaion and analsis. l 2 2 See also >> 5.2 Choice of norms: Robusness agains bad daa [Lennar Ljung 999 ] One can also parameerize he norm independen of he model parameerizaion. Ali Karimpour ov 204

12 Parameer Esimaion Mehod lecure 5 opics o be covered include: Guiding Principles Behind Parameer Esimaion Mehod. Minimizing Predicion Error. Linear Regressions and he Leas-Suares Mehod. A Saisical Framework for Parameer Esimaion and he Maximum Likelihood Mehod. Correlaion Predicion Errors wih Pas Daa. Insrumenal Variable Mehods. 2 Ali Karimpour ov 204

13 lecure 5 Ali Karimpour ov Linear Regressions and he Leas-Suares Mehod We inroduce linear regressions before as: φ is he regression vecor and for he ARX srucure i is a n b u u n μ is a known daa dependen vecor. For simplici le i zero in he reminder of his secion. Leas-suares crierion error is : Predicion ow le L= and lε= ε 2 /2 hen 2 2 l V F his is Leas-suares crierion for he linear regression

14 lecure 5 Ali Karimpour ov Linear Regressions and he Leas-Suares Mehod Leas-suares crierion 2 2 l V F he leas suare esimae LSE is: LS V arg min R f f R LS

15 lecure 5 Ali Karimpour ov Linear Regressions and he Leas-Suares Mehod e We inroduce linear regressions before as: V Leas-suares crierion 2 Y Y LS Y LS V arg min whie and e is oise componens are uncorelaed wih regressors is inverible Suppose Φ Φ Under above assumpions he LSE is BLUE Bes linear unbiased esimaor.

16 Linear Regressions and he Leas-Suares Mehod lecure 5 We inroduce linear regressions before as: Leas-suares crierion Suppose Φ Φ is inverible e LS Y oise componens are uncorelaed wih regressors and e is whie Under above assumpions he LSE is BLUE Bes linear unbiased esimaor. Linear means Y Unbiased means E{ } Bes means Cov{ } minimum possible covariance 6 Ali Karimpour ov 204

17 Linear Regressions and he Leas-Suares Mehod lecure 5 We inroduce linear regressions before as: Y E Linear means Y Condiion for linear unbiased esimaion? E{ θ} E{ Y } E{ E If =I and is uncorrelaed wih E hen he esimaor is unbiased. Clearl LSE is unbiased. E{ θ} } 7 Ali Karimpour ov 204

18 Linear Regressions and he Leas-Suares Mehod lecure 5 We inroduce linear regressions before as: Y E Linear means Y Condiion for bes linear unbiased esimaion? Cov{ θ} E{ θ θ θ θ } E{ Y E{ θ Y θ E If he esimaor is unbiased hen =I so: θ } θ Cov{ θ} E{ θ E θ θ E θ } E{ E E } θ E θ } If he esimaor is unbiased and is uncorrelaed wih E hen: 2 Cov{ θ} E{ } Cov E E{ } I We mus minimize i? 8 Ali Karimpour ov 204

19 Linear Regressions and he Leas-Suares Mehod lecure 5 mincov{θ} subjec o I Exercise 5-2: Show ha he answer of above opimizaion is: So LSE is BLUE since: LS Y Exercise 5-3: Show ha b LSE in linear regression one can find an unbiased esimae of Cov{E } b 2 e 2 d V LS 9 Ali Karimpour ov 204

20 Linear Regressions and he Leas-Suares Mehod lecure 5 Summar: Suppose Φ Φ is inverible oise componens are uncorelaed wih regressors and e is whie So LS esimae is BLUE and: e E{ θ} LS Y Cov 2 Cov e 20 Ali Karimpour ov 204

21 lecure 5 Ali Karimpour ov Linear Regressions and he Leas-Suares Mehod Weighed Leas Suares Differen measuremen could be assigned differen weighs 2 V or 2 V LS he resuling esimae is he same as previous.

22 Linear Regressions and he Leas-Suares Mehod lecure 5 Summar: Suppose Φ Φ is inverible oise componens are uncorelaed wih regressors and E{e} 0 and Cov{e} So LS esimae is: LS Y e Exercise 5-4: Show ha he LS esimae is unbiasedi is no generall BLUE. Define WLS esimae as: WLS Y Exercise 5-5: Show ha he WLS esimae is BLUE. 22 Ali Karimpour ov 204

23 A Saisical Framework for Parameer Esimaion and he Maximum Likelihood Mehod lecure 5 Example 5-: Le b e e is a WG wih cove We have 2 3. Y and we wan o find b. LS i i ow suppose e be an independen random variable and known variances λ i. WLS i / i i i i 23 Ali Karimpour ov 204

24 A Saisical Framework for Parameer Esimaion and he Maximum Likelihood Mehod Example 5-: Le b e lecure 5 ow suppose e be an independen random variable and known variances λ i. Suppose =5 and ei is derived from a random generaion normal disribuion such ha he variances areb=0: he esimaed means for 0 differen 40 experimens are shown in he figure: LS 20 LS i i 0 WLS i / i i i i Differen esimaors -20 WLS Differen experimens Exercise 5-6:Do he same procedure for anoher experimens and draw he corresponding figure. Exercise 5-7:Do he same procedure for anoher experimens and draw he corresponding 24figure. Suppose all variances as 0. Ali Karimpour ov 204

25 Linear Regressions and he Leas-Suares Mehod lecure 5 We show ha in a difference euaion Colored Euaion-error oise a b u... a... b v if he disurbance v is no whie noise hen he LSE will no converge o he rue value a i and b i. o deal wih his problem we ma incorporae furher modeling of he euaion error v as discussed in chaper 4 le us sa n b n a u v k e n n ow e is whie noise bu he new model ake us ou from LS environmen excep in wo cases: Known noise properies b a High-order models 25 Ali Karimpour ov 204

26 lecure 5 Ali Karimpour ov Linear Regressions and he Leas-Suares Mehod Colored Euaion-error oise Known noise properies v n u b b u n a a b n a n b a Suppose he values of a i and b i are unknown bu k is a known filer no oo realisic a siuaion so we have e k v Filering hrough k - gives where Since e is whie he LS mehod can be applied wihou problems. oice ha his is euivalen o appling he filer L=k -. e k u B A e u B A f f u k u k f f

27 lecure 5 Ali Karimpour ov Linear Regressions and he Leas-Suares Mehod Colored Euaion-error oise ow we can appl LS mehod. oe ha n A =n a +r n B =n b +r High-order models v n u b b u n a a b n a n b a Suppose ha he noise v can be well described b k=/d where D is a polnomial of order r. So we have e k v e D u B A or e u D B D A Afer deriving AD and BD one can easil derive A and B. A B D A D B

28 Parameer Esimaion Mehod lecure 5 opics o be covered include: Guiding Principles Behind Parameer Esimaion Mehod. Minimizing Predicion Error. Linear Regressions and he Leas-Suares Mehod. A Saisical Framework for Parameer Esimaion and he Maximum Likelihood Mehod. Correlaion Predicion Errors wih Pas Daa. Insrumenal Variable Mehods. 28 Ali Karimpour ov 204

29 A Saisical Framework for Parameer Esimaion and he Maximum Likelihood Mehod lecure 5 Esimaion and he Principle of Maximum Likelihood he area of saisical inference deals wih he problem of exracing informaion from observaions ha hemselves could be unreliable. Predicion error is : L 2... f f 2... f he likelihood funcion is: L e e2... e f e f e2... f e We mus maximize he likelihood funcion or: f e ln f e2... ln f e I ln ML arg min I 29 Ali Karimpour ov 204

30 A Saisical Framework for Parameer Esimaion and he Maximum Likelihood Mehod lecure 5 Esimaion and he Principle of Maximum Likelihood f e ln f e2... ln f e I ln ML arg min I I is eas o see ha for he Gaussian noise disribuion wih zero mean we have: f e i e 2 i 2 e i 2 2 i I e e 2... e In a case of same noise covariance we have: I e 2 e e 2 I is eas o see ha for he Gaussian noise disribuion MLE is WLS or LS. 30 Ali Karimpour ov 204

31 A Saisical Framework for Parameer Esimaion and he Maximum Likelihood Mehod lecure 5 Esimaion and he Principle of Maximum Likelihood f e ln f e2... ln f e I ln ML arg min I I is eas o see ha for he double exponenial disribuion wih zero mean we have: f e i e 2 e i I e e2... e I is an especial case of PEM. Exercise 5-8:Derive I if one consider uniform disribuion for noise. 3 Ali Karimpour ov 204

32 A Saisical Framework for Parameer Esimaion and he Maximum Likelihood Mehod lecure 5 Example 5-2: Le b e ow suppose e be an independen Gaussian random variable wih zero mean and known variances λ i. If we derive he MLE wih same variances we have: ML SM i i If we derive he MLE wih differen variances we have: ML i / i i i i 32 Ali Karimpour ov 204

33 A Saisical Framework for Parameer Esimaion and he Maximum Likelihood Mehod lecure 5 Cramer-Rao Ineuali he uali of an esimaor can be assessed b is mean-suare error marix: P cov{ } E 0 0 rue value of θ We ma be ineresed in selecing esimaors ha make P small. Cramer-Rao ineuali give a lower bound for P. heorem 5-: Cramer Rao Lower bound [Rolf Johansson 200] Le Y be observaion of a sochasic variable he disribuion on which depends on an unknown vecor θ. Le LY θ denoe he likelihood funcion and le g be an arbirar unbiased esimae of θ. hen: Y P cov{ } log L E{ log L } M M is Fisher Informaion marix An esimaor is efficien if P=M - 33 Ali Karimpour ov 204

34 lecure 5 Ali Karimpour ov A Saisical Framework for Parameer Esimaion and he Maximum Likelihood Mehod heorem 5-: Cramer Rao Lower bound [Rolf Johansson 200] Le Y be observaion of a sochasic variable he disribuion on which depends on an unknown vecor θ. Le LY θ denoe he likelihood funcion and le be an arbirar unbiased esimae of θ. hen: Y g } log log { cov{ } M L L E P M is Fisher Informaion marix Proof: Le } log log { 0 0 L L E log log 0 0 Cov L E L E M Exercise 5-9:Proof Cramer-Rao ineuali. Cov I I M Exercise 5-0:Show ha if an LSE be BLUE and he noise is Gaussian hen i is also efficien.

35 A Saisical Framework for Parameer Esimaion and he Maximum Likelihood Mehod lecure 5 Asmpoic Properies of he MLE Calculaion of P E 0 0 is no an eas ask. herefore limiing properies as he sample size ends o infini are calculaed insead. For he MLE in case of independen observaions Wald and Cramer obain Suppose ha he random variable {i} are independen and idenicall disribued. Suppose also ha he disribuion of is given for some value θ 0. hen ends o θ 0 wih probabili as ends o infini and ML 0 ML converges in disribuion o he normal disribuion wih zero mean covariance marix given b Cramer-Rao lower bound M Ali Karimpour ov 204

36 Parameer Esimaion Mehod lecure 5 opics o be covered include: Guiding Principles Behind Parameer Esimaion Mehod. Minimizing Predicion Error. Linear Regressions and he Leas-Suares Mehod. A Saisical Framework for Parameer Esimaion and he Maximum Likelihood Mehod. Correlaion Predicion Errors wih Pas Daa. Insrumenal Variable Mehods. 36 Ali Karimpour ov 204

37 Correlaion Predicion Errors wih Pas Daa lecure 5 Ideall he predicion error εθ for good model should be independen of he pas daa - If εθ is correlaed wih - hen here was more informaion available in - abou han picked up b o es if εθ is independen of he daa se - we mus check All ransformaion of εθ his is of course no feasible in pracice. Uncorrelaed wih 0 All possible funcion of - Insead we ma selec a cerain finie-dimensional vecor seuence {ζ} derived from - and a cerain ransformaion of {εθ} o be uncorrelaed wih his seuence. his would give Derived θ would be he bes esimae based on he observed daa. 37 Ali Karimpour ov 204

38 lecure 5 Ali Karimpour ov Correlaion Predicion Errors wih Pas Daa L F Choose a linear filer L and le Choose a seuence of correlaion vecors Choose a funcion αε and define F f hen calculae 0 D f sol M Insrumenal variable mehod nex secion is he bes known represenaive of his famil.

39 lecure 5 Ali Karimpour ov Correlaion Predicion Errors wih Pas Daa 0 D f sol M F f ormall he dimension of ξ would be chosen so ha f is a d-dimensional vecor. hen here is man euaions as unknowns. Someimes one use ξ wih higher dimension han d so here is an over deermined se of euaions picall wihou soluion. so arg min D f M Exercise 5-: Show ha he predicion-error esimae obained from min arg D V M can be also seen as a correlaion esimae for a paricular choice of L ξ and α.

40 Parameer Esimaion Mehod lecure 5 opics o be covered include: Guiding Principles Behind Parameer Esimaion Mehod. Minimizing Predicion Error. Linear Regressions and he Leas-Suares Mehod. A Saisical Framework for Parameer Esimaion and he Maximum Likelihood Mehod. Correlaion Predicion Errors wih Pas Daa. Insrumenal Variable Mehods. 40 Ali Karimpour ov 204

41 Insrumenal Variable Mehods lecure 5 Consider linear regression as: he leas-suare esimae of θ is given b LS sol 0 So i is a kind of PEM wih L= and ξθ=φ ow suppose ha he daa acuall described b 0 v0 We found in secion 7.3 ha LSE will no end o θ 0 in pical cases. 4 Ali Karimpour ov 204

42 lecure 5 Ali Karimpour ov Insrumenal Variable Mehods LS sol v We found in secion 7.3 ha LSE will no end o θ 0 in pical cases. IV sol 0 Such an applicaion o a linear regression is called insrumenal-variable mehod. he elemens of ξ are hen called insrumens or insrumenal variables. Esimaed θ is: IV

43 lecure 5 Ali Karimpour ov Insrumenal Variable Mehods LS sol 0 in IV mehod? as Does 0 Exercise 5-2: Show ha will be exis and end o θ 0 if following euaions exiss. IV 0 nonsingular be 0 v Eξ E We found in secion 7.3 ha LSE will no end o θ 0 in pical cases. IV sol 0 IV LS ξ mus be he same as φ meanwhile uncorrelaed wih noise.

44 Insrumenal Variable Mehods lecure 5 Exercise 5-3: Consider following ssem: a bu e ce a Derive 30 b a=0.7 b=2 and c=0. b Esimae a and b b LS mehod. c Derive 30 b a=0.7 b=2 and c=. d Esimae a and b b LS mehod and compare i b c and discuss. e Le as a IV and do following algorihm and derive a and b u and discuss. Use a LS and esimae a and b. 2 Make insrumen and esimae a and b. 3 Go o Ali Karimpour ov 204

45 Insrumenal Variable Mehods lecure 5 So we need Consider an ARX model E Eξ v0 be 0 nonsingular I II a... an na b u... bn u nb v a A naural idea is o generae he insrumens so as o secure II bu also consider I K x... x n u... u n a b Where K is a linear filer and x is generaed hrough a linear ssem x M u b Mos insrumens used in pracice are generaed in his wa. II and I are saisfied. Wh? 45 Ali Karimpour ov 204

46 References lecure 5 - Ssem Modeling and Idenificaion Rolf Johansson Ssem Idenificaion heor For he User Lennar Ljung onlinear Ssem Idenificaion Oliver elles Ali Karimpour ov 204