Regression. By Jugal Kalita Based on Chapter 17 of Chapra and Canale, Numerical Methods for Engineers

Transcription

1 Regresso By Jugl Klt Bsed o Chpter 7 of Chpr d Cle, Numercl Methods for Egeers

2 Regresso Descrbes techques to ft curves (curve fttg) to dscrete dt to obt termedte estmtes. There re two geerl pproches two curve fttg: Dt exhbt sgfct degree of sctter. The strtegy s to derve sgle curve tht represets the geerl tred of the dt. Dt s very precse. The strtegy s to pss curve or seres of curves through ech of the pots. Two types of pplctos: Tred lyss. Predctg vlues of depedet vrble, my clude extrpolto beyod dt pots or terpolto betwee dt pots. Hypothess testg. Comprg exstg mthemtcl model wth mesured dt.

3 3

4 Mthemtcl Bckgroud Smple Sttstcs I course of study, f severl mesuremets re mde of prtculr qutty, ddtol sght c be ged by summrzg the dt oe or more well chose sttstcs tht covey s much formto s possble bout specfc chrcterstcs of the dt set. These descrptve sttstcs re most ofte selected to represet The locto of the ceter of the dstrbuto of the dt, The degree of spred of the dt. 4

5 Arthmetc me. The sum of the dvdul dt pots (y) dvded by the umber of pots (). y y,, Stdrd devto. The most commo mesure of spred for smple. S S y t St ( y y) 5

6 Vrce. Represetto of spred by the squre of the stdrd devto. S y ( ) y y Degrees of freedom Coeffcet of vrto. Hs the utlty to qutfy the spred of dt. c. v. S y y 00% 6

7 7

8 stdrd devto o both sdes of me: Iclude 68% of the dt, sds: 95.5, 3 sds: 99% of the dt. 8

9 Lest Squres Regresso Ler Regresso Fttg strght le to set of pred observtos: (x, y ), (x, y ),,(x, y ). y 0 + x+e - slope 0 - tercept e- error, or resdul, betwee the model d the observtos 9

10 Crter for Best Ft/ Mmze the sum of the resdul errors for ll vlble dt: totl umber of pots e ( y However, ths s dequte crtero, so s the sum of the bsolute vlues e y 0 x o x ) 0

11 Fgure 7. ) Mmze sum of resduls, b) Mmze bsolute vlues of resduls, c) Mmze the mxmum error of y dvdul pot.

12 Best strtegy s to mmze the sum of the squres of the resduls betwee the mesured y d the y clculted wth the ler model: S r e ( y,mesured y,model) ( y 0 x ) Yelds uque le for gve set of dt.

13 Lest-Squres Ft of Strght Le [ ] ) ( 0 ) ( o r o o r x x x y x y x x y S x y S ( ) ( ) x y x x y x y x y x Norml equtos, c be solved smulteously Me vlues

14 Fgure 7.3 The resdul ler regresso represets the vertcl dstce betwee dt pot d the strght le. 4

15 Fgure 7.4 ) Shows spred of dt bout the me pot b) Shows the spred of dt roud the best-ft le. The reducto the spred gog from ) to b) sys tht b) descrbes the tedecy the dt better. 5

16 Fgure 7.5 Ler Regresso wth ) smll d b) lrge resdul errors. 6

17 Goodess of our ft If Totl sum of the squres roud the me for the depedet vrble, y, s S t Sum of the squres of resduls roud the regresso le s S r S t -S r qutfes the mprovemet or error reducto due to descrbg dt terms of strght le rther th s verge vlue. r S t S t S r r -coeffcet of determto Sqrt(r ) correlto coeffcet 7

18 For perfect ft S r 0 d rr, sgfyg tht the le expls 00 percet of the vrblty of the dt. For rr 0, S r S t, the ft represets o mprovemet. 8

19 Lerzto of Noler reltoshps It s possble to lerze some o-ler reltoshps Fgure 7.9: ) Expoetl equto b) Power equto c) Sturto-growth-rte equto 9

20 Commo Noler Reltos: () Power-lke curve: y x l(y) l( ) + b l(x) b b 3..0 (3) Sturto growth-rte curve x 3 y b +x populto growth uder lmtg codtos b3 ʹ ʹ 3 +b 3 + y x x 3 3 Be creful bout the mpled dstrbuto of the errors. Alwys use the utrsformed vlues for error lyss

21 Polyoml Regresso Some dt s poorly represeted by strght le. For these cses curve s better suted to ft the dt. The lest squres method c redly be exteded to ft the dt to hgher order polyomls (Sec. 7.).

22 y z 0, z Geerl Ler Lest Squres 0, z { Y} [ Z]{ A} + { E} [ Z] mtrx of t the mesured vlues of { Y} { A} { E} resduls 0 +, z m z + re m + bss fuctos observed vlued of the clculted vlues of ukow coeffcets z + + m z m + e the depedet vrble the depedet vrble the bss fuctos S r m y j 0 j z j Mmzed by tkg ts prtl dervtve w.r.t. ech of the coeffcets d settg the resultg equto equl to zero

23 Lest-Squres Regresso Gve: dt pots: (x,y ), (x,y ), (x,y ) Obt: "Best ft" curve: f(x) 0 Z 0 (x) + Z (x) + Z (x)+ + m Z m (x) 's re ukow prmeters of model Z 's re kow fuctos of x. We wll focus o two of the my possble types of regresso models: Smple Ler Regresso Z 0 (x) & Z (x) x Geerl Polyoml Regresso Z 0 (x), Z (x) x, Z (x) x,, Z m (x) x m

24 Lest Squres Regresso (cot'd): Geerl Procedure: For the th dt pot, (x,y ) we fd the set of coeffcets for whch: y 0 Z 0 (x ) + Z (x )... + m Z m (x ) + e where e s the resdul error the dfferece betwee reported vlue d model: e y 0 Z 0 (x ) Z (x) m Z m (x ) Our "best ft" wll mmze the totl sum of the squres of the resduls: S r e

25 y mesured vlue y e modeled vlue x x Our "best ft" wll be the fucto whch mmzes the sum of squres of the resduls: m S ( ) r e y jz j(x ) j Sr y 0Z 0(x ) Z (x ) Z (x ) L mz m(x ) ( )

26 Lest Squres Regresso (cot'd): S r e S r S S 0 r r m ( y 0 Z0( x ) mzm( x )) To mmze ths expresso wth respect to the ukows 0, m tke dervtves of S r d set them to zero: Z (x )(y Z (x )... Z (x )) m m Z (x )(y Z (x )... Z (x )) 0 0 m m M Z (x )(y Z (x )... Z (x )) m 0 0 m m

27 Lest Squres Regresso (cot'd): I Ler Algebr form: {Y} [Z] {A} + {E} or {E} {Y} [Z] {A} where: {E} d {Y} --- x [Z] x (m+) {A} (m+) x # pots (m+) # ukows {E} T [e e... e ], {Y} T [y y... y ], {A} T [ 0... m ] [ Z] Zx 0 Zx 0 L Zmx Zx 0 Zx Zmx L M M O M Zx 0 Zx L Zmx

28 Lest Squres Regresso (cot'd): {E} {Y} [Z]{A} The S r {E} T {E} ({Y} [Z]{A}) T ({Y} [Z]{A}) {Y} T {Y} {A} T [Z] T {Y} {Y} T [Z]{A} + {A} T [Z] T [Z]{A} {Y} T {Y} {A} T [Z] T {Y} + {A} T [Z] T [Z]{A} Settg 0 for,..., yelds: or S r 0 [Z] T [Z]{A} [Z] T {Y} [Z] T [Z]{A} [Z] T {Y}

29 Lest Squres Regresso (cot'd): [Z] T [Z]{A} [Z] T {Y} (C&C Eq. 7.5) Ths s the geerl form of Norml Equtos. They provdes (m+) equtos (m+) ukows. (Note tht we ed up wth system of ler equtos.)

30 Smple Ler Regresso (m ): Gve: dt pots, (x,y ),(x,y ), (x,y ) wth > Obt: "Best ft" curve: f(x) 0 + x from the equtos: y 0 + x + e y 0 + x + e y 0 + x + e Or, mtrx form: [Z] T [Z] {A} [Z] T {Y} x y x 0 y L L x x L x M M x x L x M x y M

31 Smple Ler Regresso (m ): Norml Equtos [Z] T [Z] {A} [Z] T {Y} upo multplyg the mtrces become x y 0 x x xy Norml Equtos for Ler Regresso C&C Eqs. (7.4-5) (Ths form works well for spredsheets.)

32 Smple Ler Regresso (m ): Solvg for {}: [Z] T [Z] {A} [Z] T {Y} x y x y x x 0 y x y x C&C equtos (7.6) d (7.7)

33 Smple Ler Regresso (m ): [Z] T [Z] {A} [Z] T {Y} A better verso of the frst orml equto s: ( y y)( x x) ( x x) whch s eser d umerclly more stble, but the d equto rems the sme: 0 y x

34 Commo Noler Reltos: Objectve: Use ler equtos for smplcty. Remedy: Trsform dt to ler form d perform regressos. Gve: dt whch ppers s: () expoetl-lke curve: ye bx (e.g., populto growth, rdoctve decy, tteuto of trsmsso le) C lso use: l(y) l( ) + b x

35 Mjor Pots Lest-Squres Regresso:. I ll regresso models oe s solvg overdetermed system of equtos,.e., more equtos th ukows.. How good s the ft? Ofte bsed o coeffcet of determto, r

36 Egeerg Comput/o Stdrd Errpr of the es/mte 36 Precso : If the spred of the pots roud the le s of smlr mgtude log the etre rge of the dt, The oe c use s yx S r (m+ ) stdrd error of the estmte (stdrd devto y) to descrbe the precso of the regresso estmte ( whch m+ s the umber of coeffcets clculted for the ft, e.g., m+ for ler regresso)