Based on Neter, Wasserman and Whitemore: Applied Statistics, Chapter 18, pp

Size: px

Start display at page:

Download "Based on Neter, Wasserman and Whitemore: Applied Statistics, Chapter 18, pp"

Shawn Stafford
5 years ago
Views:

1 SERIES V: REGRESSION ANALYSIS Based o Neter, Wasserma ad Whtemore: Appled Statstcs, Chapter 8, pp I. Itroducto assumptos of the model ad ts developmet The regresso aalyss aother mode of the aalyss to terrogate relatoshps betwee varables. The aalyss of regresso s ot symmetrcal ad t s always ecessary to detfy the depedet varable. The, there s oly oe depedet varable (Y, whereas the umber of depedet varables ca vary (,, It should be oted that depedet varable s chose arbtrally by the aalyst basg o hs/her experece ad kowledge of the aalysed pheomeo. However, ths choce does ot determe the drecto of the relatoshp. I other words, t does ot determe whether t s that determes the Y or the other way roud. Two the most mportat ad obvous possble goals of regresso aalyss are: detfcato ad aalyss of the ature of relatoshp betwee the depedet ad depedet varables predctg values of depedet varable from the kowledge of depedet varable. The regresso aalyss s to capture statstcal relatoshps betwee varables,.e. stuato whe for dfferet values of, we have dfferet average values (codtoal meas of Y. It s due to the fact that ths type of relatoshp s the most frequet the real word. However, extreme cases, we deal wth lack of ay relatoshp or wth fuctoal relatoshp betwee varables also regresso aalyss. The, statstcal relatoshps have two ma features (please, see Fgures 8.; 8. the Textbook: a tedecy of the depedet varable Y to vary systematcally wth the depedet varable ; a scatterg of observatos aroud the le or curve of statstcal relatoshp, partly because factors addto to the depedet varable affect the depedet varable Y, ad partly because of the heret varablty Y. Regresso aalyss corporates these features of a statstcal relato by assumg the followg: for each level of the depedet varable, there s some codtoal dstrbuto of Y; codtoal meas of these codtoal dstrbutos vary a systematc fasho wth. The above features of a regresso model ca be also captured by assumg that each observato Y s the sum of the followg two compoets: a regresso compoet, reflectg the le or curve of statstcal relatoshp a radom compoet, reflectg the scatter about the le or curve of statstcal relatoshp. II. Smple lear regresso model (model regresj lowej We lmt our terest regresso aalyss to aalyss of relatoshp betwee two varables (oe depedet Y ad oe depedet ad to lear relatoshps oly. Therefore, whle developg the regresso model, we start wth the assumpto that relatoshp betwee varables s lear. The regresso model for the th observato takes the followg form: β + β represets the regresso compoet of the model ε Y + β β + ε, where: represets the radom compoet of the model or the error term of the model (składk losowy modelu. The, Y s the value of the Y the th case (for the th elemet s the value of the depedet varable the th case (for the th elemet β ad β are parameters (parametry modelu.

2 The dea of the regresso model should be, thus, uderstood the followg way. The model s to expla the varablty of the depedet varable Y. We assume that some of ths varablty ca be explaed wth the use of depedet varable (regresso compoet, whereas the remag part of the varablty s due to radom factors ad hert varablty of the Y (radom compoet. Example: Salares ad level of educato Polad. Let s take the level of salares eared as a depedet varable Y ad the level of educato (expressed years of educato as a depedet varable. The goal of the regresso model s to aalyse determats of the varablty (dffereces of the salares. The, we assume that educato s of pvotal mportace by puttg t as a depedet varable to the model ad we also assume that there s a lear relatoshp betwee the varables. The regresso fucto (fukcja regresj relates the average value (codtoal mea średą warukową or expected value of the depedet varable Y to the value of the depedet varable. Thus, the regresso fucto s the model couterpart to the le or curve of statstcal relatoshp. The regresso fucto s also called the respose fucto ad s smply deoted the followg way: ˆ β + β Y β ad β are coeffcets of regresso (współczyk regresj or regresso parameters. The graph of the regresso fucto s regresso le; β s the tercept of the regresso le ad β s the slope of the le (please see Fgure 8.5 the Textbook. The both parameters are expressed the uts of depedet varable. β says about the average value of the depedet varable Y, whe the, whereas β determes how much (by how may uts the average (expected level of the depedet varable Y would chage whe we crease the depedet varable by. Example: Salares ad level of educato Polad. I ths example, β should be terpreted as the level of salares eared whe we deal wth a complete lack of educato (certaly f t was possble Polad to have o educato. The, β should be uderstood as a amout of PLN by whch a average (expectedsalary chages (crease or decrease alogsde the crease umber of years of educato by year. It should be stressed that ths regresso fucto, amely ts parameters β ad β, s usually ukow ad t s what we observe the real world. We also dstgush so-called emprcal regresso fucto (emprycza fukcja regresj that depcts very well the dea uderlyg the regresso fuctos ad models. It relates observed codtoal meas of the depedet varable Y to correspodg observed values of the depedet varable. It s deoted the followg way: Y β + β Its graph s smply based o the emprcal values of the varables ad Y. Thus, the kowledge of parameters of the regresso fucto s ot ecessary to plot t. I the co-ordate system, the horzotal axs s for (x, whereas the vertcal axs s used for deotg Y y. III. Estmato of the regresso fucto Parameters of the regresso fucto As oted before the regresso parameters β ad β are, as a orm, ukow. Thus, they must usually be estmated from a sample (oly part of elemets belogg to the overall populato data that we have at our had. The estmators are obtaed by the method of least squares (metoda ajmejszych kwadratów ad deoted as b ad b, respectvely. The least squares le s the le for whch the sum of the vertcal squared dstaces of the observed Y from the ftted (estmated le s a mmum (please, see Fgure 8.7 the Textbook. Cosequetly, we obta the estmated regresso fucto (oszacowaa fukcja regresj deoted the followg way: yˆ b + b where b, b estmators of β, β, respectvely; ŷ ftted value (

3 (wartość teoretycza of Y The last squares crtero requres that b ad b mmze Q that s the sum of squared devatos of ŷ from Y. Q [ Y ( b + b ] Thus, we ca dfferetate the Q wth respect to b ad b that gves: Q Q ( Y b b > ( Y b b ( Y b b > ( Y b b I ths way, we obta the set of ormal equatos (układ rówań ormalych: Y b Y b + b + Normal equatos solved for b ad b gve explct formulas for estmators of β ad β. ( b ( Y ( b Y b Y > cxy b S ( x The terpretato of b ad b s tatamout wth terpretato of β ad β. It should be just bor md that b ad b are estmators that dffer from the real parameters. Resduals (reszty The resdual for the th observato s a estmate of the error term ε. It s deoted by e ad calculated as follows: e Y yˆ. The dstcto betwee e ad ε correspods wth the dfferece betwee two devatos: ( the devato of the observed Y from the ftted yˆ b + b (e ad ( the devato of the observed Y from the codtoal (emprcal mea (expected value Yˆ β + β ( ε. Thus, e ad ε wll dffer to the extet that the ŷ ad Yˆ dffer. Propertes of least squares resduals (resdual of the regresso fucto obtaed by the lest squares method. The resduals sum s e that s Y yˆ. The sum of the squared resdual s a mmum. Ths property follows because the method of least squares mmzes Q, whereas Q s smply the sum of squared resduals. e m 3. The sum of weghted resduals s zero whe each resdual s weghted by the level of the depedet varable. 3

4 e 4. A cosequece of the propertes ( ad (3 s that the sum of the weghted resduals s zero whe each resdual s weghted by the ftted value. y ˆe IV. Assessmet of the obtaed estmated regresso fucto Certaly, each estmated regresso fucto s ot oly a estmato of emprcal (ukow observed the ature fucto, but also some approxmato of the le/curve of statstcal relatoshp. Therefore, assessg the accuracy of the obtaed regresso fucto s a pvotal elemet of the regresso aalyss. We dstgush three ma types of aalyss that allow for evaluato of the estmated regresso fucto. They clude: aalyss of the accuracy of the (lear regresso model aalyss of resduals aalyss of the accuracy of estmato of parameters of the regresso fucto b ad b. Ideally, each of the above modes of the aalyss should be coducted as they address dfferet problems relatg to the regresso aalyss. Accuracy of the (lear regresso model Assessmet of the accuracy of the (lear regresso model for the gve case checks the geerally uderstood approprateess of ths mode of the aalyss for the gve case. Namely, t s to evaluate the extet to what the varablty of the depedet varable Y has bee explaed by the varablty of the chose depedet varable. To acheve ths, we dvde the overall varablty of the Y to two compoets: varablty explaed by the regresso model varablty uexplaed by the regresso model, that s determed by the radom factors. Cosequetly we obta three sums of squared devatos (see also Fgure 8. the Textbook: total sum of squares (całkowta suma kw. odchyleń error sum of squares (resztowa suma kw. odchyleń regresso sum of squares (resztowa suma kw. odchyleń SSTO ( Y Y SSE ( Y yˆ SSR ( yˆ Y ad resultg equato: SSTO SSR + SSE. They serve for costructg the coeffcet of determato (współczyk determacj: SSR SSTO that says what part of the varablty of Y s explaed by the regresso fucto (by the varablty of. It takes values from <,> ad ca be also expressed percets. The closer t s to, the better outcome of the regresso aalyss we obtaed. It equals whe the overall varablty of Y s explaed by the wth the use of the estmated (lear regresso fucto. It s the case whe we deal wth the fuctoal relatoshp betwee varables. It should be oted that the coeffcet of determato does ot asses the accuracy of the estmato of the regresso fucto but the more geeral approprateess of the usage of lear regresso model a gve case. Drectly, t measures f t s justfed to use lear model the case of the gve two or more varables; drectly t says about r 4

5 approprateess of the choce of depedet varable (varablty of Y ca be, for example, explaed by the lear regresso model, but dfferet depedet varable would be ecessary such a case (please see Fgure 8. the Textbook. The coeffcet of determato ca be used to calculate Pearso s lear correlato coeffcet. It s smply: r xy r. It should be oted that, ths way of calculatg the correlato coeffcet does ot allow for determg the drecto of the relatoshp. However, the sg of r xy has to agree wth the sg of b, as whe the average values of Y crease wth the rse values of the b s postve ad we talk about the postve correlato. Ths relato betwee the correlato coeffcet ad the coeffcet of determato shows aga the lk betwee regresso ad correlato aalyss. Aalyss of resduals Aalyss of resduals s of pvotal mportace the regresso aalyss. I geeral terms, t allows for assessg the predctve ablty of the estmated regresso model. I other words, t says how much the estmated (ftted values of the depedet varable Y dffer from the observed oes. Thus, the most mportat mode of the aalyss of resduals s aalyss of devatos of estmated values from the observed values. There are two possble strogly related approaches/measures to asses that: error mea square ad stadard error of estmate. Error mea square (waracja reszt lub waracja resztowa s smply: ( Y ˆ y SSE MSE Its square root deotes stadard error of estmate (odchylee stadardowe reszt lub śred błąd szacuku. S( e ( Y yˆ SSE The stadard error of estmate s measured uts of depedet varable. It ca be terpreted as a average fluece that factors other tha depedet varable have o depedet varable. Also, t ca be terpreted as a average error predctg a values of the depedet varable. I other words t says by how may uts of the depedet varable we are wrog estmatg values of Y wth the usage of our regresso fucto. The stadard error of estmate s easer to terpret tha error mea square eve though they are both based o squared devatos betwee ftted ad observed Y. Example: Salares ad level of educato Polad. I ths example, stadard error of estmate equalg would mea that: o the average we are wrog by PLN predctg the average level of salary wth our regresso fucto usg the kowledge about the umber of years of educato. S( e Coeffcet of radom varato (współczyk zmeośc losowej deoted as V s a relatve Y measure of stadard error of estmate. For the estmated regresso fucto to be well adjusted to emprcal data t should ot exceed.3. Graphcal aalyss of resduals Also, the ature of dstrbuto of resduals themselves matters for assessg the regresso fucto. It s ot wth the scope of ths lecture to deal wth t detal. However, t s worth metog that ther dstrbuto agast ftted values of Y should t have ay regular patter. It should be radom. Otherwse, we would deal wth some systematc error. Aalyss of estmates of regresso coeffcets Fal, also mportat, mode of the aalyss used to evaluate the regresso fucto s the assessmet of the accuracy of estmated coeffcets of regresso. Namely, t s to exame how well the 5

6 estmated coeffcets grasp/descrbe the relatoshp betwee Y ad. To remd, they are estmated wth the use of least squares method. The aalyss uses stadard errors of estmate to obta two separate measures for b ad b : S( e S( b stadard error of estmate for b (śred błąd szacuku b ( x x x S( b S( b stadard error of estmate for b (śred błąd szacuku b Geeral terpretato of the obtaed stadard errors s that they depct average errors (mstakes predctg the gve coeffcets o the bass of the sample (emprcal regresso fucto. S(b says how much, o the average, we are wrog predctg the level of depedet varable for, whereas S(b says how much, o the average, we are wrog predctg chages the level of depedet varable caused by chage of the depedet varable by ut. Example: Salares ad level of educato Polad. I ths example, S(b 5 would say that stadard error of estmate of b s 5 PLN. That s to say that we are, o the average, wrog predctg the level of salary for years of educato by 5 PLN. S(b would say that the stadard error of estmate of b s. That s to say that we are, o the average, wrog by PLN predctg that crease of years of educato by would rse the average salary by b. Relatve errors of estmate (względe błędy szacuku for regresso coeffcets are relatve measures of these errors: S( b b S( bb They should ot be too bg amely they should ot exceed.. Otherwse, we ca expect a bg error. There are two sources of bg relatve errors. Oe source s adequate sample - ether too small or urepresetatve for a gve populato. Secod reaso ca le a lack of relatoshp betwee aalyzed varables. The latter should be captured by the aalyss of the coeffcet of determato. 6

Linear Regression with One Regressor

Linear Regression with One Regressor Lear Regresso wth Oe Regressor AIM QA.7. Expla how regresso aalyss ecoometrcs measures the relatoshp betwee depedet ad depedet varables. A regresso aalyss has the goal of measurg how chages oe varable,