QA 622 AUTUMN QUARTER ACADEMIC YEAR

Transcription

1 QA 6 AUTUMN QUARTER 6-7 ACADEMIC YEAR Istructor: Offce: James J. Cochra 7A CAB Telephoe: (38) Hours: e-mal: URL: TR 8: a.m. - oo W 8: a.m. - : a.m. or by appotmet jcochra@cab.latech.edu Homes/Jcochra Requred Textbook: Draper & Smth, Appled Regresso Aalyss, Wley- Iterscece, 998 (3 rd edto)

2 Other Suggested Texts: Rudolf J. Freud & Ramo C. Lttell, SAS System for Regresso, SAS Isttute ad Joh Wley & Sos, 3 rd ed.,. Bowerma, Bruce L. & Rchard T. O'Coell, Lear Statstcal Models: A Appled Approach, Duxbury, d ed., 99. Rawlgs, Joh O., Sastry G. Patula, & Davd A. Dckey, Appled Regresso Aalyss: A Research Tool, Sprger-Verlag, st ed., 998. Kuter, Mchael H., Chrstopher J. Nachtshem, Joh Neter, & Wllam L, Appled Lear Statstcal Models, McGraw-Hll Irw, 5 th ed., 5. Cook, Des R. & Saford Wesberg, Appled Regresso Icludg Computg ad Graphcs, Wley-Iterscece Publshg, st ed., 999. Belsley, Davd A., Edw Kuh, & Roy E. Welsh, Regresso Dagostcs, Joh Wley & Sos, st ed., 98. Prerequstes: QA 39 (calculus & lear/matrx algebra) QA 43 (Itermedate Busess Statstcs) computer lteracy (ablty to lear ad use SAS) Gradg: Iterm Exam 5 pots Lterature Revew 5 pots Comprehesve Fal Exam pots 5 pots

3 A.Deftos I. Regresso Aalyss The Bascs. Respose Varable (also called the depedet varable, output varable, or y-varable) - ths s the pheomea or characterstc we wsh to estmate or predct. The depedet varable s usually deoted y.. Predctor Varable (also called a regressor, put varable, x-varables, or depedet varable) - ths s the pheomea used to estmate or predct the value of the depedet varable. Idepedet varables are usually deoted x j. 3. Fuctoal Relatoshp - The uque value of the depedet varable (usually deoted y) ca be precsely determed gve the value of the depedet varable (usually deoted x) ths s ofte referred to as a determstc relatoshp betwee x ad y. Classes of Fuctoal Relatoshps Betwee x & y y y y y=f(x) y=f(x) y=f(x) x x x Postve/Drect Relatoshp Negatve/Iverse Relatoshp No Relatoshp

4 4. Slope - Chage the value of the depedet varable that correspods to a oe-ut chage the depedet varable. The (ukow) populato slope s usually deoted β, whle the sample-estmated slope s usually deoted b. 5. Y-Itercept - Value of the depedet varable whe the depedet varable s zero. The (ukow) populato Y-tercept s usually deoted β, whle the sample-estmated Y-tercept s usually deoted b. 6. Scatter Dagram - graphcal smultaeous presetato of the values of two varables o a Cartesa coordate system Suppose you had the followg two observatos o varables x ad y: x y The resultg graphcal dsplay would look lke ths: y x The relatoshp betwee x ad y s (by defto) lear.

5 7. Lear Relatoshp - The slope s costat at all values of the depedet varable. For the prevous example we could fd the le o whch both pots le by solvg the equato y = B + B x smultaeously for our two observatos: 5 = B + (3)B -[6 = B + (4)B ] -= -B B = so 5 = B + (3)() B = 5 3 = y The resultg le s y = + x x Such a le betwee two pots wll always be lear ad fuctoal (why?). What happes f we troduce a thrd pot (3.5, 5.5) to our data set? Sce the ew observato (3.5, 5.5) satsfes our orgal formula (falls o the le through pots (3, 5) ad (4, 6)) y = + x = = 5.5 we stll have a fuctoal lear relatoshp. y x

6 O the other had, what happes f we troduce a thrd pot (3.5, 4.5) to our data set? Sce the ew observato (3.5, 4.5) does ot satsfy our orgal formula (does ot falls o the le through pots (3, 5) ad (4, 6)) y = + x = = we o loger have a fuctoal lear relatoshp. y x 8. Stochastc (Statstcal) Relatoshp - The relatoshp(s) betwee values of the respose varable ad correspodg values of the predctor varable(s) s (are) ot determstc. Thus the value of y s estmated gve the value of x. The estmated value of the depedet varable s deoted y, ^ ad the populato slope ad y- tercept are usually deoted β ad β.

7 Two Notes Regardg Stochastc Relatoshps a. The uderlyg populato relatoshp s gve by y = β + β x + where s the varato y that s uattrbutable to x b. Sources of Varato Stochastc (Statstcal) Relatoshps - Errors Specfcato ot all varables assocated wth the depedet varable have bee accouted for - Errors Measuremet lack of accuracy assessg/quatfyg the values of ay varable. - Samplg Error totally radom varato, uattrbutable to ay source. 9. Regresso Aalyss - Statstcal methods for estmatg the relatoshp betwee the depedet varable ad the depedet varable. The estmates of β ad β are usually deoted b ad b.. Lear Regresso - Idcates that the relatoshp(s) betwee the depedet varable ad the depedet varable(s).. Smple Regresso - Idcates that the relatoshp s betwee the depedet varable ad a sgle depedet varable. Thus we would have y^ = b + b x as our estmate of the value of the depedet varable y whe x = x.

8 . Regresso Error - also called the resdual, ths s the dfferece betwee our estmate of the value of the depedet varable y whe x = x (.e., y) ^ ad the actual value of the depedet varable y whe x = x (.e., y ). The resdual for the th observato s usually deoted e, so we have that whch by substtuto s e = y y ^ e = y (b + b x ) 3. Ordary Least Squares (OLS) - Crtera for fttg a estmated regresso le to sample data whch the sum of the squared dffereces betwee actual ad estmated values of the depedet varable are mmzed,.e., whch s equvalet to ( ˆ ) m y - y = m e = that s, we wsh to fd the regresso le that mmzes the squared regresso errors we would commt usg the le to estmate the values of y for the values of x the followg sample data.

9 Graphcal Iterpretato of Ordary Least Squares (OLS) for a smple lear regresso we have y. x,y y - y ^ (total ^y y - y devato) y^ -y _ y (uexplaed devato) (explaed devato) ^ y = b + b x _ x x x Example: suppose we have collected the followg sample of 6 observatos o age ad come: AGE INCOME ( $,'s) Fd the estmated regresso le for the sample of sx observatos we have collected o age ad come: Whch s the depedet varable ad whch s the depedet varable for ths problem?

10 SCATTER DIAGRAM 7 INCOME ( $,'s) AGE Note that the pots follow a postve slope but do ot le o a straght le (.e., there appears to be a drect stochastc relatoshp betwee AGE ad INCOME) SCATTER DIAGRAM 7 INCOME ( $,'s) AGE How do we ft a le that mmzes the sums of the squared errors e ( ˆ = y - y )? What s the = = geometry of ths objectve?

11 B. Ordary Least Squares Estmate of the Regresso Le - Recall that our objectve s to ( ˆ ) m e,.e., m y - y = = whch by aother substtuto ca be rewrtte as ( ) m e = m y - b + b x a lttle dfferetal calculus ca be appled to ths expresso to derve the ormal equatos: = = Partally dfferetatg ths equato wrt b ad b yelds the followg two equatos: SSE = - (y - (b + b x )) = b = SSE = - (y - (b + b x ))x = b = These partally dfferetato (wrt b ad b ) yeld the followg two equatos (whch have bee rearraged so that b ad b are solated o the rght had sdes):

12 ad We ca solve these equatos (called the Normal Equatos) smultaeously to obta the equatos ecessary to produce the estmates b ad b. y = b + b x = = x y = b x + b x = = = x y = = ( x - x) ( y - y) x y - = = b = = ( ) x - x x = = x - = ad b = y - b x Example: suppose we have collected the followg sample of 6 observatos o age ad come: AGE INCOME ( $,'s) Fd the estmated regresso le for the sample of sx observatos we have collected o age ad come: Whch s the depedet varable ad whch s the depedet varable for ths problem?

13 SCATTER DIAGRAM 7 INCOME ( $,'s) AGE Note that the pots follow a postve slope but do ot le o a straght le (.e., there appears to be a drect stochastc relatoshp betwee AGE ad INCOME) our summary calculatos are: Aual Icome Respodet Age Years (x ) x y x $'s (y ) sums y so: = = x y = 57 y = 74 = = x = 43 x = 673

14 whch results ad b = x y = = x y - x = x - = = ( 43) ( 74) 57 - = = = b = y - bx = -.75 = so our estmated regresso equato s ŷ = x What would happe to these estmates f we chage the scale of measuremet for the depedet varable y? What f we chage the scale of measuremet for the depedet varable x?

15 Let s look at what happes to the sum of squared errors as we alter our estmates b ad b : Respose Surface of Squared Errors 35 3 Sum of Squared Errors 5 The sum of squared errors appears to be mmzed aroud -6.3 b -5.6 ad.9 b b b.9 If we refe our search ad lmt our scope, we get a clearer dea of what s happeg: Respose Surface of Squared Errors 9 8 Sum of Squared Errors The sum of squared errors appears to be mmzed aroud -6. b -5.9 ad.5 b b b.9.

16 If we further refe our search ad lmt our scope, we get a eve clearer dea of what s happeg: Respose Surface of Squared Errors Sum of Squared Errors 63 6 The sum of squared errors appears to be mmzed aroud b = 5.96 ad b = b b.75.5 Of course, a extreme refemet of our search ad lmtato of our scope provdes the clearest pcture: Respose Surface of Squared Errors Sum of Squared Errors The sum of squared errors appears to be mmzed aroud b = ad b = b b.7475

17 7 6 5 y = Note that the estmated regresso le does ) cut through the sample pots o the scatter dagram ad ) goes through the pot x,y. INCOME ( $,'s) 3 ( ) SCATTER DIAGRAM x = 4.5 y - t = AGE Note also that aother otato (commoly referred to as the Pocket Calculator otato) s commoly used: ( ) ( ) S = x x y y XY = = x y = = = ( ) S = x x XX = x = = x xy ( ) S = y y YY = y y

18 so that the formulas for b ad b are: S XY b = ad b = y - b x S XX We refer to each of the followg quattes wth specal ames: = x s the Ucorrected Sum of Squares of the X's = x s the Correcto for the Mea of the X's = x = x - s the Corrected Sum of Squares of the X's = x y s the Ucorrected Sum of x = = Products of the X's ad Y's y s the Correcto for the Mea of the X's ad Y's = x y xy - = = s the Corrected Sum of Products of the X's ad Y's

19 The formula for b b = y - b x mples that the regresso le goes through x,.e., ˆ f x = x the y = b + b x = ( y - b x ) + b x = y y We ca thk of ths problem aother way: + - ( ) ( ) m d + d e = + - s.t. y -b -bx +d -d =, =, K, + - d, d, =, K, Ths olear optmzato problem yelds the same soluto as the ormal equatos.

20 Note that crtera other tha least squares have bee suggested. Partcularly, least absolute devato (LAD) has bee cosdered: ˆ m e,.e., m y - y = = whch by aother substtuto ca be rewrtte as ( ) m e = m y - b + b x = = Why has t the least absolute devato crtero gaed greater acceptace? m = The least absolute devato as a fucto of b looks lke ths: e LAD estmate b of b We would lke to use dfferetal calculus to fd the LAD estmate of b, but the dervatve of the LAD fucto wrt b does ot exst at the mmum value of of b (the dervatve of the LAD fucto from the left ad from the rght dffer at of b ). Numercal methods must be used to fd the value of of b that mmzes the LAD fucto.

21 Furthermore, cosder ths stuato: y Ether of these les (ad ay others wth the same slope that le betwee) mmzes the LAD fucto (why?). Thus the LAD crtero does ot ecessarly yeld a uque regresso le. x Ths problem also be formulated as a olear optmzato problem: + - ( ) m d + d = + - s.t. y -b -bx +d -d =, =, K, + - d, d, =, K, e Aga, ths olear optmzato problem yelds the same soluto as the prevous approach.

22 Let s look at what happes to the sum of absolute errors as we alter our estmates b ad b : Respose Surface of Absolute Errors 5 Sum of Absolute Errors 5 The sum of absolute errors appears to be mmzed aroud -9. b -8. ad.5 b b b If we refe our search ad lmt our scope, we get a clearer dea of what s happeg: Respose Surface of Absolute Errors 7 6 Sum of Absolute Errors 5 4 The sum of absolute errors appears to be mmzed aroud -8.8 b -8.7 ad.59 b b b.54.45

23 If we further refe our search ad lmt our scope, we get a eve clearer dea of what s happeg: Respose Surface of Absolute Errors Sum of Absolute Errors The sum of absolute errors appears to be mmzed aroud b = ad b = b b.5875 Of course, a extreme refemet of our search ad lmtato of our scope provdes the clearest pcture: Respose Surface of Absolute Errors Sum of Absolute Errors The sum of squared errors appears to be mmzed aroud b = ad b = b b.59

24 Let s compare the estmated values of the respose ad the resduals from the two models: OLS model LAD model ŷ = x ŷ = x Respodet Age (x ) Aual Icome OLS LAD $'s (y ) ^ ^ y e y e E What smlartes do you otce? What dffereces do you otce? m D What s the crtero for ths approach? + - s.t. y - b - b x + d - d =, =, K, + - d, d D, =, K, + - d, d, =, K,

25 m D Or ths approach? + - s.t. y - b - b x + d - d =, =, K, + - d, d D, =, K, + - d, d, =, K, Ultmately, mmzato of squared errors (OLS) s the preferred crtero for fttg regresso equatos. C. Usg the Ordary Least Squares Estmate of the Regresso Le to Estmate Values of the Depedet Varable Y - Recall that our geerc estmated regresso equato s y ^ = b + b x so to estmate the value of the depedet varable y for a correspodg value of the depedet varable x, substtute x to the estmated regresso equato ad solve.

26 Example: Recall that our estmated regresso equato for the prevous problem s ŷ = x so to estmate the value of the depedet varable y for a correspodg value of the depedet varable x, substtute x to the estmated regresso equato ad solve. Thus, for our frst observato (x = 5) our regresso estmated value of the depedet varable come s yˆ = ( 5) = 5.9.e., we estmate that a twety-fve year-old wll ear a aual salary of $5,9 ths orgazato. We could estmate the value of the depedet varable y for each correspodg value of the depedet varable x our sample: Respodet Age Years Aual Icome Estmated Aual (x ) $'s (y ) Icome $'s ( ^y ) Σ Note that the sum of the regresso estmates s exactly equal to the sum of the observed values of the depedet varable y

27 Note that we ca use ths formato to calculate our error terms or resduals e : Respodet Age Years Aual Icome Estmated Aual (x ) $'s (y ) Icome $'s ( ^y ) Resduals e Σ Note that the sum of the resduals s zero wll ths always be so? Why or why ot? Also ote that we ca estmate the value of the depedet varable Y for ay value of the depedet varable X that s wth the rage of values for X our sample For example, what s the estmated value of come (the depedet varable Y) for a ffty year-old employee ths orgazato? The value of the depedet varable X (5) s wth the rage of values for X our sample (5-6), so ŷ = ( 5) = e., we estmate that a ffty year-old wll ear a aual salary of $57,78 ths orgazato. Why ca t we estmate estmate the value of the depedet varable Y for a value of the depedet varable X that s outsde the rage of values for X our sample?

28 We have o dea what the relatoshp betwee the depedet varable Y ad the depedet varable X s outsde the rage of values for x our sample SCATTER DIAGRAM INCOME ( $,'s) A attempt to do so s called extrapolato ths s oe of the dagers of usg regresso aalyss! AGE Note that the Y-tercept s frequetly the result of extrapolato uder such crcumstaces the Y-tercept has o real meag or terpretato! Oe possble remedy move the Y-axs to the ceter of the data (.e., ceter the data o X). How? Suppose we buld ths model: ˆ y = b + b x ' where x = x - x The Y-tercept ow represets the estmated value of the respose varable Y whe X s equal to ts (sample) mea ths s certaly ot a extrapolato. '

29 Subtractg the mea of X from every observed value of X essetally moves the X-axs so that the Y-axs tersects t at x stead of at. Y x-x x X x X =X-X whch (because we are usg x -x as the regressor) ad b = x y = = xy - x = x - = = = ( ) = ( x - x) = x - x y = ( x-x) b = y - b = y

30 Cosder the algebrac ramfcatos of ths model: ' y ˆ = b + bx = b + b ( x - x) = b + b x - b x ( ) = b - b x + b x ' = b + b x Ths mples a slght modfcato to the terpretato of the regresso estmate b : b estmated value of the respose varable Y whe the regressor varable X s equal to ts mea. Note also that the terpretato of b remas uchaged (why?) Aother possble remedy ceter the data (.e., elmate the Y-tercept). How? Suppose we buld ths model: ˆ' y = b + b x ' where y = y - y ad x = x - x ' ' The Y-tercept s ow zero - why?

31 Subtractg the meas of x ad y from every observed value of X ad Y essetally moves the X ad Y axes so that they Y-axs tersect at x, y stead of at,. Y Y =Y-Y y y-y y x x-x x X X =X-X whch (because we are usg x -x as the regressor ad y - y as the respose) b = x y = = xy - x = x - = = = ( x - x) ( x - y) = ( x - x) = (whch s detcal to our orgal equato for b ) ad b = y - bx =

32 Cosder the algebrac ramfcatos of ths model: ˆ' y = b + b x = b x ' where y = y - y ad x = x - x ' ' D.Assessg the Stregth of the Relatoshp betwee the Respose ad the Regressor(s). Parttog the Varace the Respose varable Y - Aga recall that our objectve s to = = ( ˆ ) m e = y - y Cosder the followg detty: ( ˆ ) ( ˆ ) ( ˆ ) y - y = y - y + y - y = e + y - y

33 Squarg ad summg both sdes yelds: ( y ) ( ˆ ) ( ˆ ) - y = y - y + y - y = = ( ˆ ) ( ˆ ) ( ˆ ) ( ˆ ) = y -y + y -y y -y + y -y = = = ( ˆ ) ( ˆ ) = y - y + y - y = = What happeed to ( y ˆ ) ( ˆ - y y - y )? = What happeed to ( y ˆ ) ( ˆ - y y - y )? = Frst ote that from the ormal equatos we have x y = b x + b x = = = The observe that ( ˆ ) x e = x y - y = x y - x yˆ = = = = = b x + b x - = = = x ( b + b x ) = b x + b x - b x + b x = = = = =

34 Now observe that = ˆy e ( ˆ ) = yˆ y - y = = ( + ) = b b x e = b e + b x e = = ( ) ( ) = b + b = So what happeed to ( y ˆ ) ( ˆ - y y - y )? = = ( y - y) ( y - yˆ ) ( ) ˆ ( ) = y y - y - y y - y = = = y e - yˆ e = = = - = Thus ( y ˆ ) ( ˆ - y y - y ) = =

35 So we have that: ( y ) ( ˆ ) ( ˆ - y = y - y + y - y ) = = = Whch we ca express as: Sum of Squares Sum of Squares Sum of Squares = + about the Mea due to Regresso about Regresso whch s also sometmes expressed as: Total Sums of Squares Regresso Sum = + of Squares Resdual Sum of Squares Ths s ofte referred to as a parttog of the varato the respose varable. The parttoed varato the respose varable s ofte represeted a Aalyss of Varace (ANOVA) table: Aalyss of Varace (ANOVA) Table: the Basc Splt of S YY Source of Varato Degrees of Freedom (df) Sum of Squares (SS) Mea Square (MS) Due to Regresso About Regresso (Resdual or Error) About the Mea (Corrected for y ) Note that - - ( ) = y - y = SYY ˆ ( y - y) MS Reg = ( y ˆ - y) = ( y - y) = MS Res

36 The prevous example yelds the followg parttog of the varato the respose varable: Aalyss of Varace (ANOVA) Table: the Basc Splt of S YY Source of Varato Degrees of Freedom (df) Sum of Squares (SS) Mea Square (MS) Due to Regresso About Regresso (Resdual or Error) About the Mea (Corrected for y ) The varato the respose varable ca be further parttoed to accout for varato explaed by each parameter estmate: Aalyss of Varace (ANOVA) Table: Icorporatg SS(b ) Source of Varato Degrees of Freedom (df) Sum of Squares (SS) Mea Square (MS) Due to b b About Regresso (Resdual or Error) Corrected Total Correcto Factor (due to b ) - - SS = ( yˆ - y) MS Reg b b = ( y ˆ - y) = ( y - y) = y = SS b = = y MS Res Total y =

37 The varato the respose varable s ofte placed Extra (Icremetal) Sums Of Squares Order: Aalyss of Varace (ANOVA) Table: Extra SS Order b Source of Varato b b Resdual Degrees of Freedom (df) - Sum of Squares (SS) y = SS b = = y b b ( ˆ ) SS = y - y = SS SS b b b Mea Square (MS) MS Reg MS Res Total y = We have ow measured the varato about the estmated regresso le. What f we wat to assess the stregth of the relatoshp betwee the respose varable Y ad the regressor varable X?. Coeffcet of Determato - measure the proporto of the total varato the observed values of the respose varable Y that s accouted for by the varato correspodg values of the regressor X. Usually deoted r for smple regresso ad R for multple regresso, ths s gve by ( ˆy - y) = r = = = ( y - y) = SS b b SS xy SS SS SS yy xx yy ad s the rato of SS due to regresso gve b to the Total SS Corrected for the Mea. y

38 Cosder our prevous example. We ca use our prevously developed equatos to fd the Coeffcet of Determato: ( ) ( ) S = x x y y XY = = xy xy = = = ( ) S = x x XX = x = = x ( ) S = y y YY = y y For our prevous example we could easly calculate the ecessary summary values ecessary for calculato of r tabular form: Aual Icome Respodet Age Years (x (x - )(y - ) (x - ) ) x y x $'s (y ) (y - y ) Σ x= 4.5 y= Note that SS xy = 6., SS xx = 83.5, ad SS yy =

39 By drect substtuto we have: SS 6. SS SS xy r = = =.688 xx yy ( ) ( ) Thus ths regresso explas approxmately 68.8% of the varato the observed values of the respose varable Y. Notes about the Coeffcet of Determato -. r. - larger values of r dcate relatvely stroger relatoshps - t s the square of the Pearso s Correlato Coeffcet betwee y ad b + b x (ad so betwee y ad y^ ) 3. Pearso s Correlato Coeffcet - measure of the relatve stregth ad drecto of the assocato betwee two cotuous varables. Usually deoted ρ whle ts sample estmate s deoted r ad s gve by r = xy SS SS xx xy SS Notes about the Coeffcet of Correlato - t s the rato of the covarace of X ad Y to the product of ther stadard devatos - -. r. - larger absolute values of r dcate stroger relatoshps - a egatve ^ value of r dcates a verse relatoshp whle a postve value of r dcates a drect relatoshp -r y,y = r y, b +b x = sg(b ) r for a lear regresso [where sg( ) returs the sg of the argumet] yy

40 Cosder our prevous example - aga we could easly calculate the ecessary summary values ecessary for calculato of r tabular form: Aual Icome Respodet Age Years (x (x - )(y - ) (x - ) ) x y x $'s (y ) (y - y ) Σ x= 4.5 y= Note that SS xy = 6., SS xx = 83.5, ad SS yy = By drect substtuto we have: SS 6. xy r = = =.896 xx yy ( = sg ( b ) r ) ( ) ( ) SS SS But what does ths mea?

41 Note also the terestg relatoshp betwee b ad r xy - Frst ote that: b = ( x - x) ( y - y) = = = ( x - x) = ( ) = = ( ) ( ) ( x - x) ( x - x) ( y - y) ( y - y) ( x - x) = = = = = y - y x - x y - y SS SS yy xy yy = = rxy SSxx SSxxSSyy SSxx xx xy SS yy SS SS SS Now ote that: = = ( ) ( ) x x - x = - s ( ) ( ) y y - y = - s, By drect substtuto we have: s y b = r sx xy so b s r xy scaled by the dsperso of the respose varable Y dvded by the dsperso of the regressor X.

42 E. Evaluatg the Ordary Least Squares Estmate of the Regresso Le. The F-Test - It ca be show that the Mea Square due to Regresso (MS Reg ) ad the Mea Square due to Resdual Varato (MS Res or s ) have the followg expected values: ( Reg ) = σ + β ( ) = ( ) = σ E MS x - x E s If the followg codtos are met: - the errors ε ~ N(, σ ) - the errors ε are depedet MS s the t ca be show that: Reg ( dfmsreg ) ( dfms ) Res σ σ ~ χ ~ χ wth df = df wth df = df MS Res MS Reg f β = Here we eed to recall that the rato of two depedet Ch-Square Statstcs wth degrees of freedom a ad a dvded by ther respectve degrees of freedom has a F- dstrbuto wth degrees of freedom a ad a.

43 Sce the two results from the prevous slde are depedet, f β = we have: ( dfmsreg ) df σ MS MSReg Reg ( dfms ) Res df σ Ths suggests a smple test of the ull hypothess: H : β = H : β Why s ths hypothess mportat? MSRes s = MS s Reg ~ F wth df = df, df MSReg MSRes Example: from our prevous problem we have: Aalyss of Varace (ANOVA) Table Source of Varato Degrees of Freedom (df) Sum of Squares (SS) Mea Square (MS) F-Rato Due to Regresso About Regresso (Resdual or Error) About the Mea (Corrected for y )

44 We could the fd a crtcal value of F for a gve level of sgfcace α ad df =, 4: α =. F.,,4 = 4.54 α =.5 F.5,,4 = 7.7 α =. F.,,4 =. We reject H f F > F α,,4, so we reject H at α =. OR We ca compare the p-value (provded by a computer prtout) to our chose value of α - f p-value < α the reject H. p-value =.4, so reject H f the chose α s less tha.4. Note that: - We ca exted ths test to multple regresso to test the ull hypothess H : β = β = β 3 = β m = H : at least oe β j Why s ths hypothess mportat? - Ths F-test s equvalet to a t-test (as we wll see shortly)

45 . Cofdece Itervals ad Hypothess Tests for β ad β : If the followg codtos are met: - the errors ε represet a radom varable wth a mea of (.e., E(ε ) = ) ad varace of σ (.e., V(ε ) = σ ) - the errors ε are ucorrelated, j so that cov(ε, ε j ) =. Ths results E(Y ) = β + β x V(Y ) = σ Y ad Y j are ucorrelated, j so that cov(y, Y j ) = Now we have that = b = = = = ( x - x) ( y - y) = ( x - x) ( ) ( ) = = = ( x - x) ( x - x) y - y ( x - x) = = = = = ( ) ( x - x) x - x y - x - x y x - x y ( x - x)

46 Now f the Y s are parwse ucorrelated ad the a s are costats, the varace of a fucto a = a Y + a Y + a 3 Y a Y s = ( ) ( ) V a = a V Y ad f V(Y ) = σ ( ) V a = σ a = Now recall that our prevous expresso for b a = = ( x - x) ( x - x) Reducto ow yelds ( ) V b = ( x - x) The stadard devato (stadard error) of b s the square root of the varace V(b ) ( ) = = σ sd b = = Usg the estmate s place of the ukow σ yelds the estmated stadard devato (stadard error) of b : s s est. sd ( b ) = = S xx x - x σ ( x - x) = ( ) σ S xx

47 If the followg codto s also met: - the errors ε ~ N(, σ ) the t ca be show that a ( - α)% cofdece terval s gve by: b ± t - α,- ( x - x) = ( ) = b ± t est. sd b - α,- s Ths suggests a smple test of the ull hypothess: H : β = β H : β β usg the test statstc b β t = est. sd b = s ( ) ( b β ) ( x - x) = = ( b β ) ( x - x) = s

48 We the compare t to t -, α our decso rule. I our prevous example, we have Respodet Age Years (x ) Aual Icome $'s (y ) (x - x) x Σ Thus our summary data are: s =.3697 ( ) = x - x = 83.5 = b =.748 so for H : β = we have t = = ( b β ) ( x ) - x = s ( ) ( ) =

49 Note that: - ths hypothess testg approach ca be adapted to oe-taled tests - ths hypothess testg approach ca be adapted to values of β other tha zero - smple lear regresso there s a relatoshp betwee the ths t-test ad the prevously dscussed F-test for H : β = we have : MSReg F = s = = = t = = ( ) ( ) b x - x y - y s ( ) b x - x s We ca costruct cofdece tervals ad hypothess tests for β much the same maer as we dd for β : - It ca be show that ( ) sd b = = ( ) Replacg the ukow σ wth ts sample estmate s yelds: = x x - x x = est. sd ( b ) = s ( x - x) = σ

50 A ( - α)% cofdece terval for β s the gve by: x = ± -,-α b t s x - x The test statstc for the t-test of the ull hypothess H : β = β s also gve by b β t = = est. sd ( b ) = H : β β ( ) b x = = ( ) x - x β s We the compare t to t -, α our decso rule. I our prevous example, we have Respodet Age Years (x ) Aual Icome $'s (y ) (x - x ) x Σ

51 Thus our summary data are: s =.3697 = = ( ) so for H : β = we have x - x = 83.5 x = 673 b = b β t = = = x ( ) (.3697 ) = s x - x = ( ) 3. Cofdece Itervals ad Hypothess Tests for ρ xy : Fsher s Approxmato (z trasformato) gves us (approxmately) ' + rxy - - z = l tah ( rxy ) ~ N tah ρ, - rxy - 3 So a approxmate ( α)% cofdece terval for ρ xy s gve by: α - + r ρ xy + xy l ± z α = l - ρ - rxy - xy - 3

52 The test statstc for the Null Hypothess s gve by H : ρ = ρ xy xy H : ρ ρ xy xy + r ρ xy + xy z = l l - 3 ρ - rxy - xy whch s the compared to the percetles of the N(, ) that correspod to the preselected level of sgfcace α. May SAS procedures are avalable for varous types of regresso: CALIS CATMOD GENMOD GLM LIFEREG LOESS LOGISTIC NLIN PLS PROBIT REG RSREG TPSPLINE TRANSREG

53 Geerc example usg a data set drectly: DATA ame of data set goes here; INPUT varable ames ad colum locatos go here; CARDS; data go here ; PROC???/optos; RUN; For our example data we could have: DATA salary; INPUT ame $ age come; CARDS; Jackso 5 Ross Brght Stadfer 6 65 Clark 4 64 Syder 33 3 ; TITLE Aalyss of Age/Icome Data ; PROC PRINT; VAR ame age come; RUN; PROC CORR; VAR age; WITH come; PROC REG; MODEL Icome=age; RUN;

54 Output for PROC PRINT: Aalyss of Age/Icome Data : Tuesday, September, Obs ame age come Jackso 5 Ross Brght Stadfe Clark Syder 33 3 Output for PROC CORR ad PROC REG: Aalyss of Age/Icome Data : Tuesday, September, The CORR Procedure Wth Varables: come By Varables: age Smple Statstcs Varable N Mea Std Dev Sum Mmum Maxmum come age Pearso Correlato Coeffcets, N = 6 Prob > r uder H: Rho= age come.896.4

55 Aalyss of Age/Icome Data 3 : Tuesday, September, The REG Procedure Model: MODEL Depedet Varable: come Aalyss of Varace Sum of Mea Source DF Squares Square F Value Pr > F Model Error Corrected Total Root MSE R-Square.6883 Depedet Mea Adj R-Sq.63 Coeff Var Parameter Estmates Parameter Stadard Varable DF Estmate Error t Value Pr > t Itercept age SOME questos you should be able to aswer: Suppose you have a collecto of observatos o a respose Y ad a predctor varable X.. How do you use OLS to estmate the slope ad tercept for a lear regresso whch the respose varable s Y ad the predctor varable s X? How do you use these results to produce a pot estmate of a value of Y for a gve value of X? How do you use these results to produce a terval estmate of a value of Y for a gve value of X? How do you use these results to produce a terval estmate of the mea value of Y for a gve value of X? How do you terpret the results of these procedures?. How do you evaluate the qualty of the ft of the regresso le to the sample data? How do you test hypotheses about the slope ad/or tercept? How do you produce a terval estmate for each of these parameters? How do you terpret the results of these procedures? 3. How do you use SAS or EXCEL to perform these procedures?

56 SOME questos you should be able to aswer (cotued): Suppose you have a collecto of observatos o a respose Y ad a predctor varable X. 4. How would the estmated slope ad tercept be affected f you multpled the values of the respose varable Y by a costat c? How would ths affect the coeffcet of determato r or the correlato coeffcet r xy? Why? 5. How would the estmated slope ad tercept be affected f you multpled the values of the predctor varable X by a costat c? How would ths affect the coeffcet of determato r or the correlato coeffcet r xy? Why? 6. How would the estmated slope ad tercept be affected f you added a costat c to the values of the respose varable Y? How would ths affect the coeffcet of determato r or the correlato coeffcet r xy? Why? SOME questos you should be able to aswer (cotued): Suppose you have a collecto of observatos o a respose Y ad a predctor varable X. 7. How would the estmated slope ad tercept be affected f you added a costat c to the values of the predctor varable X? How would ths affect the coeffcet of determato r or the correlato coeffcet r xy? Why? 8. How would the estmated slope ad tercept be affected f you terchaged the roles of the respose varable y ad the predctor varable X? How would ths affect the coeffcet of determato r or the correlato coeffcet r xy? Why? 9. Why s OLS preferred over LAD as a crtero for fttg regresso les?

57 SOME questos you should be able to aswer (cotued): Suppose you have a collecto of observatos o a respose Y ad a predctor varable X.. How are the ormal equatos for OLS derved? How are these equatos used to fd equatos for the estmated slope ad tercept?. Why do some aalysts ceter the predctor varable X? Why do some aalysts also ceter the respose varable Y? What are the ramfcatos of each of these modelg strateges?. How ad why s the varato the respose varable Y parttoed? 3. What assumpto(s) must be met whe performg ferece o regresso estmates?