Hadout #8 Ttle: Foudatos of Ecoometrcs Course: Eco 367 Fall/05 Istructor: Dr. I-Mg Chu Lear Regresso Model So far we have focused mostly o the study of a sgle radom varable, ts correspodg theoretcal dstrbuto, ad samplg scheme. However, very ofte we are more terested bvarate or eve multvarate relatoshps betwee/amog radom varables. We ll beg wth a bvarate case where X ad Y are theoretcally related usg co tossg example. We ll show that the codtoal mea of Y o X ca be formed usg a determstc lear fucto X. After that we ll troduce the smple lear regresso model where Y ca be learly depedet o X emprcally. I the study of ecoomcs, we are ofte terested whether oe varable Y ca be explaed by the other varable X. For example, s flato caused by the over-jecto of moey supply the log ru? Are the households spedg govered by ther dsposable come? Is employmet status for a female worker depedet o the umber of chldre she has? All of the above questos ca be aswered ad examed usg smple lear regresso model. Be otced that the causal relatoshp s establshed usg ecoomc theory ad emprcal lear regresso model s used to exame the valdty of ecoomc theory. Amog these three examples the oly dfferece s, the thrd case, the Y varable s categorcal. We ll study the thrd case later usg Probt or Logt model, a exteso of lear regresso model. As you should fd out by ow, the data type troduced Hadout# plays a mportat role to decde how we choose a approprate model to study the data. *Cosder a expermet where a far co s tossed four tmes; sample space (0, ) 4 X # of heads obtaed o the frst three tosses, Y # of heads obtaed o all four tosses Table 8. Jot Dstrbuto X\Y 0 3 4 f(x) 0 /6 /6 0 0 0 /8 0 3/6 3/6 0 0 3/8 0 0 3/6 3/6 0 3/8 3 0 0 0 /6 /6 /8 g(y) /6 /4 3/8 /4 /6 Table 8. Smulato outcome based o tossg a co four tmes ad repeat t 00 tmes. (I dd t set the seed umber, so the outcome from aother expermet wll be dfferet) x/y 0 3 4 0 0.07 0.04 0 0 0 0 0.7 0. 0 0 0 0 0. 0.7 0 3 0 0 0 0.05 0.07
What s the codtoal mea fucto of Y gve X based o the above jot dstrbuto? Aswer: It s E(Y X); expectato of Y gve X. Table 8.3 Codtoal Dstrbuto Y 0 3 4 g(y X0) / / 0 0 0 g(y X) 0 / / 0 0 g(y X) 0 0 / / 0 g(y X3) 0 0 0 / / 3 5 7 E(Y X0), E(Y X), E(Y X), E(Y X3) If we plot E(Y X) agast X a scatter dagram, t looks lke the followg: Fgure 8. Codtoal Mea Fucto E(Y X) 0.5.0.5.0.5 3.0 3.5 0.0 0.5.0.5.0.5 3.0 X There s a exact (.e., determstc) lear relatoshp betwee E(Y X) & X, so we ca wrte the followg equato: E(Y X) 0 + *X How do we fd 0 ad? Aswer: ΔY E(Y X ) E(Y X 0) ΔX 0 Whe X 0 0 E(Y X0) E(Y X) + X The slope ad tercept of the above codtoal mea fucto are kow costats gve that we kow how X ad Y are related (.e., kowg ther jot dstrbuto fucto).
Lear Regresso Model: regresso aalyss s cocered wth the study of the relatoshp betwee oe varable called the explaed, or depedet, varable ad oe or more other varables called depedet, or explaatory, varables. Y 0 + *X + () Equato () s termed smple (oe X) lear (learty X) regresso model. Y s the depedet varable, X s the depedet varable, ad s the error term. I practce, t s ulkely the relatoshp betwee X ad Y s a exact straght le lke the co tossg example we studed earler. Therefore, the error term s added to represet ucertaty ad all other potetal factors that may cotrbute to the varato of Y (.e., the capture-all effect). We wll make assumptos about the error term later for ferece purpose. Objectves of lear regresso model: () To estmate the mea value of the depedet varable, gve the value of the depedet varable(s). I other words, we assume the codtoal mea fucto s lear: E(Y X) 0 + *X () To test hypotheses about the ature of the depedece (.e., ). The sze ad magtude of beat(s) (f there are more tha oe depedet varables) tell us how the chages Xs affect Y. Ths s called margal effect. (3) To predct, or forecast, the mea value of the depedet varable, gve the value(s) of the depedet varable(s). For example, we are terested fdg the relatoshp betwee wage ad schoolg a small market ecoomy. If the populato data s avalable, equato () s called lear populato regresso fucto (PRF). However, very ofte t s too costly to get the populato data. Therefore, a small sample s draw from the populato ad our goal s to ucover the ukow parameters the lear sample regresso fucto (SRF). Let s use Y to deote the hourly wage ad X educato backgroud (measured schoolg years). The plot of Y agast X s show the followg scatter dagram (Fg. 8.). As you may otce the actual codtoal mea fucto s ot a straght le. However, a lear regresso le seems a approprate approxmato for descrbg the relatoshp betwee wage ad educato. Be otced that, oe sample s obtaed from a model (smulated populato) where people s wages are a lear fucto of schoolg. By usg a smulated data, we kow all of the correspodg parameters the populato ad the correspodg samplg scheme. There s a advatage of usg smulated data; frst, we ca see how the SRF s dfferet from the PRF. Secodly, we ca vsualze the cosequeces of assumpto volatos by chagg the model assumpto oe at a tme (.e., geerate a ew populato but wth a dfferet model assumpto). Thrdly, we ca exame the usefuless of model predctablty. 3
Fg. 8. Wage vs. Schoolg hourly wage 0 0 30 40 6 8 0 4 6 8 school years I Fg. 8., the sample codtoal mea of wage (the blue le) teds to crease wth schoolg years. Although t s ot a exact straght le, t ca be approxmated usg a lear equato (the red le) as followg: E(Wage Schoolg) 0 + *Schoolg +, where ~N(0, ) Where s a devato term that captures other factors that may affect wage. Usually we assume that t s ormally dstrbuted wth mea zero ad varace. Ths assumpto s requred for statstcal ferece purpose. Lear regresso model: Fd a estmator that best descrbes the lear relatoshp betwee Wage (depedet varable) ad Schoolg (depedet varable). I other words, we eed a method to ucover ukow parameters 0,, ad. There are usually two approaches to do that; MLE (maxmum lkelhood estmator) ad OLS (ordary least square). We ll adopt the OLS estmator because t has a ce BLUE property. I ll gve more formato about ths BLUE property later. OLS: Y 0 + *X + ( ) () Choosg 0 & to mmze ( Y 0 X ) (3) It stads for Best Lear Ubased Estmator. 4
Apply dfferetal calculus o equato (3), we ca fd (X (X X) * (Y X) Y) SXY SXX (.) 0 Y - * X (.) Ŷ 0 + *X ( Ŷ : predcted value) (.3) ê Y - Ŷ ( ê : resdual; a estmator for ) (.4) σ ê s termed resdual whch s obtaed usg Y ( 0 + ê The umerator part s called resdual sum of squares; RSS. (.5) *X ). Var( 0 ) X σ *( + ) (.6) SXX Var( ) σ * SXX (.7) Aalyss of Varace (ANOVA) (Y - Y ) ( Ŷ - Y ) + ê (.8) TSS ESS + RSS ESS R (coeffcet of determato) TSS (.9) R * SXX SYY SXY ( ) * SXX SXX SYY (SXY) SXX * SYY (.0) R SXY SXX SYY (.) We eed to solve two equatos smultaeously. Takg the frst dervatve wth respect to 0 ad respectvely, we ca obta: -* (Y - 0 - *X ) 0 & - X * (Y - 0 - *X ) 0. The above two equatos mply that ê 0 & X * ê 0 5
Hypothess Testg: a) Null vs. Alteratve H 0 : * (usually the ull s to exame whether s zero or ot (.e., * 0), but t ca be other parameter rather tha zero, whch s specfed the ecoomc model) H A : 0 b) Test statstcs (TS) ~ t -p ( : Var( ) same (.7), : umber of observatos; p: umber of explaatory varables eed to be estmated) c) Rejecto rego It s decded by data aalyst (YOU), usually %, 5% or 0% s chose. It ca be explaed as the probablty we allow for the Type I error. I other words, the probablty we reject the ull hypothess whe t s true. d) P-Value The probablty to obta a value that s as extreme as the test statstc (TS). e.g. Study tme ad exam score Tme (X) 5 3 8 0 6 3 Score (Y) 65 69 64 75 90 75 49 77 74 58 Let s compute 0,, ad σ maually ad report the outcomes as follows: 3.8383 (score/hour). the margal score creases by 3.8383 f the study hour creases by oe ut. 0 57.703 (score) σ 3.5/(0-) 40.64 Fg. 8.3 Regresso Le ad Resduals y 40 50 60 70 80 90 00 9.6 5.7838 4.7838-0.37786.4605-3.5395-8.70-7.898-3.73.593-0 4 6 8 0 x 6
I Fg 8.3, t shows that the regresso le we get wll yeld the smallest squared aggregate devatos; ( Y 0 X ). I other words, the total of (-8.70) + + (.593) wll be the smallest gve the OLS estmator 0 ad. Hypothess Testg o 0 ad Suppose we are terested whether equals certa value, say *. H 0 : * H A : * Assume the error term,, s NIID 3 (0, ) ~ N( *, ) TS ~ t - (Why s the degree of freedom -?) Let s use the umercal example from page sx to exame whether equals zero (does study tme affect exam score?) I other words, we wat to test H 0 : 0 H A : 0 ( Var( )) σ * SXX 40.64 56.9 0.7076 σ 40.64 SXX 56.9 TS 3.8383 0 0.7076 4.56 (table value, t 8.306 at 5% level of sgfcace) The above TS dcates that we ca ot reject the ull hypothess at 5% (or eve %) level of sgfcace usg two-tal test. Alteratvely, we ca calculate the p-value to decde whether we ca reject the ull. A p- value s a measure of how much evdece we have agast the ull hypothess. I ll show you how to calculate p-value usg Stata. The p-value for TS 4.56 wth 8 df s about 9.*0-4 3 Normally, Idetcally, Idepedetly, Dstrbuted. The commo assumptos about the error tem are a) E( X) 0, b)e( X). Meag, the error tem has a mea equals zero ad t s homoscedastc (.e., costat varace). 7
What ca we coclude our umercal example? If studets study oe more hour, hs (her) exam score wll be 3.8383 sgfcatly hgher. The Aalyss of Varace (Aga; you should see the patter whe F test s eeded) Let s cosder the codtoal mea of Y: Ŷ 0 + *X Y Ŷ + ê Y - Y ( Ŷ - Y) + ê (Y - Y) ( Ŷ - Y) + ê (.8) TSS 4 ESS + RSS ESS ( Ŷ - Y ) ( 0 + *X - ( 0 + * X )) * (X - X ) ESS R (coeffcet of determato) (.9) TSS We wll apply equato (.8) ad (.9) usg matrx operatos Stata to obta coeffcet of determato the ext sesso. Table ANOVA Source df SS MS F p-value --------------------------------------------------------------------------------------------------------- Regresso SSreg SSreg/ MSreg 5 / σ Resdual - RSS σ RSS/- ---------------------------------------------------------------------------------------------------------- Total - SYY ------------------------------------------ Notce: t F (see pp. 6, Hadout #6) The Resduals The resduals ca be used for dagostc check. Ths exames whether the model assumptos are volated. Kowg whether the assumptos are volated affects the hypothess testg results. I the ext hadout we wll show that how to modfy OLS estmator gve that we detect ether multcollearty, heteroscedascty, ad/or autocorrelato problems. Predctos Ŷ 0 + * X Where X s a chose value (vector). For example, gve certa study tme, X what should be the expected exam score ( Ŷ )? 4 Deote ths term by SYY. 5 SSreg/ MSreg. 8
( X Stadard error of predcto σ *( + + X) / ) SXX Stadard error of predcto matrx form { σ *[ + X T (X T X )- X ]} / Lear Regresso Model usg Matrx Operatos ALTERNATIVELY, we ca wrte equato () o page 3 to a matrx form as followg: Y X* + (.) Where Y s a x matrx, X s a x matrx ad s a x vector. Let s gore temporarly. y y Y, X y x x 0,. x Applyg matrx dfferetato accordg to equato (3) 6 o page 4, we ca obta φ 0 (X T X )- (X T Y) (.) 7 Fg. 8.4 Y ê 0 * *X Ŷ X Projecto (blue arrow) of Y o (, X) vector space 6 I matrx form eq. (3) s equvalet to select both beta estmators to mmze T. 7 The dervato of equato (.) s same as the oe I show footote o page 5. I wll brefly expla matrx dfferetato our meetg. As you ca see (.) s a geeralzato of obtag beta estmators. I other words, t ca be appled easly to lear multple (.e., more tha oe regressors) regresso model. 9
What does Fg. 8.4 mea mathematcally? y y y * 0 + x x * x (We wat to assg a weght to ad X vector ad thus results a value (dstace) that s closet to Y vector). Small Sample Property of OLS Estmator Estmated 0 ad 0 0 E( ) OLS estmator s ubased (.3) 0 Var( ) has the smallest varace amog all the lear (.4) estmators OLS estmator s effcet. Estmated Varaces of 0 ad Var( Var(φ ) 8 0 ) Cov( 0, Cov(, 0 ) Var( ) ) σ *(X T X )- σ *( X X X ) - σ * * X ( X ) X X * (.5) 9 X ------------------------- SXX (X X) (X X * X X ) X - * X * X + X X - * * X + * X ( X ) X X - * + *( ) X. 8 Var() (X T X )- ; eeds to be estmated sce t s ukow. The estmator s 9 Ths varace-covarace matrx s the most mportat estmator for statstcal ferece purpose. 0
( X ) ( X ) + ------------------------ Let s focus o the estmated varace of s: X - * X - ( X ) Var( 0 ) σ * * X X ( X ) σ * SXX ( X ) * SXX / σ * SXX * X * SXX X σ *( + ) (.6) SXX Var( ) σ * Cov( 0, ) σ * σ * * X ( X ) X ( X ) / SXX (.7) X X / X σ * - σ * - σ * * X ( X ) X ( X ) / SXX (.8) ))? How do we fd the estmate of correlato coeffcet betwee 0 & (.e., ( 0, From here, all the routes ad cocepts are the same as I show earler. We rely o maly two equatos to get the job doe. Please go over the Mata route at least oce to lear how we obta all the mportat outcomes a lear regresso model. I geeral f we have p regressors, the φ [,,...,, ] φ (X T X )- (X T Y) 0 p p T Var(φ ) σ *(X T X )-