Chapter 10. What is Regression Analysis? Simple Linear Regression Analysis. Examples

Transcription

1 Chapter 10 Smple Lnear Regresson Analyss What s Regresson Analyss? A statstcal technque that descrbes the relatonshp between a dependent varable and one or more ndependent varables. Examples Consder the relatonshp between constructon permts (x) and carpet sales (y) for a company. OR Relatonshp between advertsng expendtures and sales There probably s a relatonshp......as number of permts ncreases, sales should ncrease....set advertsng expendture and we can predct sales But how would we measure and quantfy ths relatonshp? Smple Lnear Regresson Model (SLR) Assume relatonshp to be lnear Y = a + bx + Where Y = dependent varable X = ndependent varable a = y-ntercept b = slope = random error 1

2 Random Error Component () Makes ths a probablstc model... Represents uncertanty random varaton not explaned by x Determnstc Model = Exact relatonshp Example: Temperature: o F = 9/5 o C + 3 Assets = Labltes + Equty Probablstc Model = Det. Model + Error Graphcally, SLR lne s dsplayed as lne of means 10 Y X Model Parameters a and b Estmated from the data Data collected as a par (x,y) Process of Developng SLR Model Hypothesze the model: E(Y) = a + bx Estmate Coeffcents yˆ aˆ bx ˆ Specfy dstrbuton of error term How adequate s the model? When model s approprate, use t for estmaton and predcton

3 Fttng the Straght-Lne Model Ordnary Least Squares (OLS) Once t s assumed that the model s Y = a + bx + Next we must collect the data Before estmatng parameters, we must ensure that the data follows a lnear trend Use scatterplot, scattergram, scatter dagram Monthly Carpet Sales A Scatter Plot of the Data Carpet Cty Problem Monthly Constructon Permts Assessng Ft Assessng Ft (Devatons) Monthly Carpet Sales Carpet Cty Problem Monthly Constructon Permts aka errors or resduals (r, e ) Dfference between the observed value of y and the predcted value of y e r Want r to be small y yˆ 3

4 Assessng Ft (Cont.) NOTE: Sum of the resduals s 0 e ( y yˆ ) 0 Least Squares Lne Fnd the lne that mnmzes y ŷ wth respect to the parameters Can ft many dfferent lnes; whch one s best? Lne that best fts the data s the one that mnmzes the sum of squares of the errors (SSE). Ths s the least squares lne. Recall that Mnmze yˆ aˆ bˆ x y aˆ bˆ x Least Squares Lne (Cont.) Example 1 Estmated parameters yeld smallest SSE Estmated coeffcents are gven by: ˆ SS b SS aˆ y bx ˆ SS SS xy xx xy xx x xy y x x The Central Company manufactures a certan specalty tem once a month n a batch producton run. The number of tems produced n each run vares from month to month as demand fluctuates. The company s nterested n the relatonshp between the sze of the producton run (x) and the number of man-hours of labor (y) requred for the run. The company has collected the followng data for the 10 most recent runs: 4

5 Example 1 (Cont.) Example 1 (Cont.) Run Number of tems Labor (man-hours) Estmated Regresson Equaton ŷ x Interpretaton of Regresson Equaton What does ths mean? ŷ x x = # of tems produces y = # of man-hours of labor State conclusons n terms of problem Intercept: when no tems are produced, the est. # of hrs. s Does ths make sense? No! Interpretaton (Cont.) When usng regresson to predct a response, the value of the ndependent varable must fall n the range of the orgnal data. Predctons made outsde of the range of the data s called EXTRAPOLATION and may have lttle or no valdty. In our example, our ndependent varable ranges from 30 to 90, and predctons should be made n ths range. 5

6 Interpretaton (Cont.) Slope: every unt change n x, the average value of y wll change by the slope In the example,.01 mples that for every tem produced, the average # of man-hours s expected to ncrease by.01. Example The Tr-Cty Offce Equpment Corporaton sells an mported desk calculator on a franchse bass and performs preventve mantenance and repar servce on ths calculator. Data has been collected from 18 recent calls on users to perform routne preventve mantenance servce; for each call, x s the number of machnes servced and y s the total number of mnutes spent by the servce person. Example (Cont.) Obtan Estmated Regresson Equaton SS bˆ SS xy xx aˆ y bx ˆ 64 x xy y x x yˆ x , Example (Cont.) Interpretatons Intercept: When no machnes are servced, the reparman spends an avg. of -.34 mnutes; Note that x=0 s probably not n the range of the data, so ntercept makes no sense. Slope: For each machne servced, we would expect approx mnutes of servce tme spent 6

7 Model Assumptons E() = 0 Var() = s normally dstrbuted I are ndependent Y The Nature of a Statstcal Relatonshp Regresson Curve Before performng regresson analyss, these assumptons should be valdated. Probablty dstrbutons for Y at dfferent levels of X X Assumptons for Regresson Descrptve Measures of Assocaton Coeffcent of Determnaton (R ) Unknown Relatonshp Y = X 7 7

8 Y Y Error Decomposton Y (actual value) Y -Y {* } Y -Y ^ ^ } Y (estmated value) Y ^ -Y Coeffcent of Determnaton (Cont.) 0 SSE SS yy 0 R 1 Larger R, the more varablty s explaned by the regresson model yˆ aˆ bx ˆ X Coeffcent of Determnaton (Cont.) Coeffcent of Determnaton (Cont.) Y X Y X 8

9 Correlaton Coeffcent (r) Postve square root of R aka Pearson product-moment correlaton coeffcent Untless -1 r 1 Descrbes the strength of the relatonshp between x and y Correlaton Coeffcent (r) Computatonal Formula SSxy r SSxxSSyy -1 mples strong negatve relatonshp 0 mples no relatonshp +1 mples strong postve relatonshp Measures of Assocaton (Cont.) Hgh correlaton does not mply causaton. What does ths mean? Estmaton and Predcton Satsfed wth the model, we can perform: Estmaton of the mean value of y for a gven value of x Predcton of a new observaton for a gven value of x Where do we expect to have the most success? 9

10 Estmaton & Predcton (Cont.) Scatter Plot of Correct Model The ftted SLR model s yˆ aˆ bx ˆ Estmatng y at a gven value of x, say x p, yelds the same value as predctng y at a gven value of x p. Dfference s n precson of the estmate... the samplng errors Y = X R = Scatter Plot of Curvlnear Model Scatter Plot of Outler Model Y = X R = Y = X R =

11 Scatter Plot of Influental Model Verfyng Assumptons Y = X R = Examnng Resdual Plots Regresson and Excel Excel also has a bult-n tool for performng regresson that: s easer to use provdes a lot more nformaton about the problem To nstall the Regresson tool, Tools AddIns Analyss ToolPak Then to perform the analyss Data Data Analyss Regresson 43 11

12 The TREND( ) Functon TREND(Y-range, X-range, X-value for predcton) where: Y-range s the spreadsheet range contanng the dependent Y varable, X-range s the spreadsheet range contanng the ndependent X varable(s), X-value for predcton s a cell (or cells) contanng the values for the ndependent X varable(s) for whch we want an estmated value of Y. Enterng the Central Company Data (see Example 1) Note: The TREND( ) functon s dynamcally updated whenever any nputs to the functon change. However, t does not provde the statstcal nformaton provded by the regresson tool. It s best to use these two dfferent approaches to regresson n conjuncton wth one another. Important Software Note Regresson Output When usng more than one ndependent varable, all varables for the X-range must be n one contguous block of cells (that s, n adjacent columns). SUMMARY OUTPUT Regresson Statstcs Multple R R Square Adjusted R Square Standard Error Observatons 10 ANOVA df SS MS F Sgnfcance F Regresson E-10 Resdual Total Coeffcents Standard Error t Stat P-value Lower 95% Upper 95% Lower 95.0% Upper 95.0% Intercept Number of Items E

13 Regresson Plot Man-hours of Labor Central Company 00 y =.006x R = Number of Items Multple Regresson & Model Buldng Most regresson problems nvolve more than one ndependent varable. If each ndependent varable vares n a lnear manner wth y, the estmated regresson functon n ths case s: ˆ ˆ ˆ X ŷ a b1x 1 bx b k ˆ The optmal values for the b can agan be found by mnmzng the ESS. The resultng functon fts a hyperplane to our sample data. k Example Regresson Surface for Two Independent Varables Y * * * * * * ** * * * * * * * * ** * * * * * X X 1 13

14 Example Admssons data In SLR, we had x 1 = Entrance test score y = End of year GPA Suppose other factors nvolved x = HS GPA x 3 = SAT score Model becomes y = a + b 1 x 1 + b x + b 3 x 3 + Independent Varables May represent hgher-order terms x 1 = age x = age May be dummy/ndcator varables x 3 0, f female 1, f male May be functons of ndependent varables x 4 = prce x 5 = ndustry average prce x 6 = prce dfference = x 5 x 4 Steps to Developng Multple Regresson Model 1. Hypothesze the model: y = a + b 1 x b k x k +. Estmate coeffcents 3. Specfy dstrbuton of and estmate 4. Valdate model assumptons 5. Evaluate model adequacy 6. Use for estmaton and predcton Fttng the Model b represents the change n y wth respect to each unt change n x when ALL other x s are held constant Method of fttng s the same as n SLR Estmate b s to mnmze SSE Computatonally ntensve Use MS Excel 14

15 Multple Regresson Salsberry Realty Salsberry Realty sells homes along the east coast of the Unted States. One of the questons frequently asked by prospectve buyers s: If we purchase ths home, how much can we expect to pay to heat t durng the wnter? The research department at Salsberry has been asked to develop some gudelnes regardng heatng costs for sngle famly homes. Three varables are thought to relate to the heatng costs: (1) the mean daly outsde temperature, () the number of nches of nsulaton n the attc, and (3) the age of the furnace. To nvestgate, Salsberry s research department selected a random sample of 0 recently sold homes. They determned the cost to heat the home last January, as well as the mean outsde temperature durng January n the regon, the number of nches of nsulaton n the attc, and the age of the furnace. The sample nformaton s gven below. Home Multple Regresson Salsberry Realty Heatng Cost ($) Mean Outsde Temperature ( o F) Attc Insulaton (nches) Age of Furnace (years) Multple Regresson Salsberry Realty Determne the multple regresson equaton. Whch varables are the ndependent varables? Whch varable s the dependent varable? Use MS Excel to develop a regresson equaton. Dscuss the regresson coeffcents. Why does t ndcate that some are postve and some are negatve? What s the ntercept value? What s the estmated heatng cost for a home where the mean outsde temperature s 30 degrees, there are 5 nches of nsulaton n the attc, and the furnace s 10 years old? Multple Regresson Salsberry Realty The hypotheszed model s gven by y = a + b 1 x 1 + b x + b 3 x 3 + where y = heatng cost x 1 = mean outsde temp. x = attc nsulaton x 3 = age of furnace = random error 15

16 Scatterplot 1 Mean Temp. vs. Heatng Cost Scatterplot Attc Insulaton vs. Heatng Cost Heatng Cost Heatng Cost Mean Outsde Temperature Attc Insulaton Scatterplot 3 Age of Furnace vs. Heatng Cost Multple Regresson Salsberry Realty SUMMARY OUTPUT Heatng Cost Regresson Statstcs Multple R R Square Adjusted R Square Standard Error Observatons 0 ANOVA df SS MS F Sgnfcance F E-06 Regresson 3 Resdual Total Coeffcents Standard Error t Stat P-value Lower 95% Upper 95% Intercept E Mean Outsde E Temperature (F) Attc Insulaton (nches) Age of Furnace (years) Age of Furnace 16

17 Multple Regresson Salsberry Realty Estmated regresson equaton: ŷ x 14.83x 6.10 Dscusson Meanngful nterpretatons of coeffcents Check range of each ndependent varable 1 x3 Estmate the heatng cost for a mean outsde temp. of 30 0 F, there are 5 n. of nsulaton, and the furnace s 10 years old. Estmaton and Predcton Model Assumptons: Same as SLR : Estmaton of the varance, : s d SSE MSE n k 1 ~ N 0, Usng Dummy/Indcator Varables Qualtatve varables can also be used n the regresson model Dummy/ndcator or bnary (0, 1) varables denote the presence or absence of the varable of nterest Usng Dummy/Indcator Varables A qualtatve varable wth c classes wll be represented by (c-1) dummy/ndcator varables n the model, wth each takng on the values of 0 and 1. Example: Suppose we have an ndependent var. that represents type of det: Weght Watchers, Atkns, Body for Lfe, and Proten. Note we have 4 classes (c = 4) We wll need (c-1) = 3 varables n the model 17

18 Usng Dummy/Indcator Varables Types of Det could be modeled as: 1, f WW 0, otherwse x1 x x 3 1, f Atkns 0, otherwse 1, f BFL 0, otherwse Moton Pcture Industry Example A moton pcture ndustry analyst wants to estmate the gross earnngs generated by a move. The estmate wll be based on dfferent varables nvolved n the flm's producton. The ndependent varables consdered are X 1 = producton cost of the move and X = total cost of all promotonal actvtes. A thrd varable (X 3) that the analyst wants to consder s whether or not the move s based on a book publshed before the release of the move. The analyst obtans nformaton on a random sample of 0 Hollywood moves made wthn the last fve years. The data s gven n the followng table. The model could resemble: y = a + b 1 x 1 + b x + b 3 x 3 + Moton Pcture Industry Example Moton Pcture Industry Example Move Gross Earnngs, Mllons $ Producton Cost, Mllons $ Promoton Cost, Mllons $ Book No Yes Yes No Yes Yes No No Yes No Yes Yes No No Yes No Yes No Yes Yes Prepare a scatter plot of gross earnngs versus producton cost and promoton cost. Does there appear to be a lnear relatonshp between gross earnngs and ether producton cost or promoton cost. If the analyst were to use a smple lnear regresson model to predct gross earnngs, whch varable should be used? Explan. Determne the parameter estmates for the model gven by Yˆ ˆ ˆ aˆ b1 X1 b X Analyze the results. Determne the parameter estmates for the model gven by Yˆ ˆ ˆ ˆ aˆ b1 X1 b X b3 X 3 Does X 3 help explan the gross earnngs when X 1 and X are also n the model? Explan. 18

19 Gross Earnngs v. Promoton Cost Gross Earnngs v. Producton Cost Gross Earnngs Gross Earnngs Producton Cost Promoton Cost Moton Pcture Industry Example Wth smplcty n mnd, suppose we ft three smple lnear regresson functons: ŷ ŷ ŷ aˆ bˆ x 1 1 aˆ bˆ x aˆ bˆ 3x 3 Key regresson results are: Varables Adjusted Parameter n the Model R R S e Estmates X a=5.071, b 1 =5.57 X a=4.33, b =3.761 X a=35.111, b 3 = The model usng X accounts for 77.9% of the varaton n y, leavng approx. % unaccounted for. Moton Pcture Industry Example SUMMARY OUTPUT Regresson Statstcs Multple R R Square Adjusted R Square Standard Error Observatons 0 ANOVA df SS MS F Sgnfcance F Regresson E-11 Resdual Total Coeffcents Standard Error t Stat P-value Lower 95% Upper 95% Intercept Producton Cost E Promoton Cost E

20 Moton Pcture Industry Example Modelng earnngs usng producton and promoton costs yelds: ŷ x.37 1 x R = , whch mples 93.44% of the varaton n earnngs can be explaned by prod. and prom. costs. s = 5.05, whch s sgnfcantly less than ether of the SLR models Moton Pcture Industry Example SUMMARY OUTPUT Regresson Statstcs Multple R R Square Adjusted R Square Standard Error Observatons 0 ANOVA df SS MS F Sgnfcance F Regresson E-1 Resdual Total Coeffcents Standard Error t Stat P-value Lower 95% Upper 95% Intercept Producton Cost E Promoton Cost E Book Moton Pcture Industry Example Usng the full model: ŷ x 1.8x 7.17x 3 R ncreases to 96.67% and the std. error s reduced to Moton Pcture Industry Example Indcator varables revsted ŷ x 1.8x 7.17x 3 Note that x 3 takes on the values of 0 and 1. 0

21 Selectng the Model We want to dentfy the smplest model that adequately accounts for the systematc varaton n the dependent varable, y. Arbtrarly usng all of the ndependent varables may result n overfttng. Adjusted R Statstc As addtonal ndependent varables are added to a model: The R statstc can only ncrease. The Adjusted-R statstc can ncrease or decrease. R a SSE n 1 1 SSyy n k 1 Adjusted R R The R statstc can be artfcally nflated by addng any ndependent varable to the model. We can compare adjusted-r values as a heurstc to tell whether addng an addtonal ndependent varable really helps to mprove a regresson model. Moton Pcture Industry Example Key regresson results are: Varables Adjusted Parameter n the Model R R S e Estmates x a=5.071, b 1 =5.57 x 1 & x a=8.15, b 1 =3.67, b =.367 x 1, x &x a=7.836, b 1 =.848, b =.78, b 3 =7.166 The model usng x 1, x, and x 3 appears to be best: Hghest adjusted-r and hghest R Lowest s (most precse predcton ntervals) Estmaton and Predcton Same as n SLR Lke SLR, dfference les n the error of estmaton and predcton errors In multple regresson, these standard errors are complex and beyond the scope of ths class Wll rely on MS Excel output 1

22 Concerns Parameter Estmablty nablty of the model to estmate parameters because data s concentrated n one area data must nclude at least one more level of x than the hghest order of the x-varable that s ncluded n the model Multcollnearty relatonshp between two or more ndependent varables varables contrbutng the same nformaton f two or more varables are hghly correlated, then we only need one n the model Extrapolaton (already dscussed n SLR) Correlated Errors measurements on the dependent varable are correlated tme seres analyss Polynomal Regresson Sometmes the relatonshp between a dependent and ndependent varable s not lnear. Sellng Prce $175 $150 $15 $100 $75 $ Square Footage Ths graph suggests a quadratc relatonshp between square footage (X) and sellng prce (Y). Polynomal Regresson An approprate regresson functon n ths case mght be, ˆ ŷ aˆ bˆ x b or equvalently, ŷ where, aˆ bˆ x 1 1 x1 bˆ 1x1 x x 1 Sellng Prce Graph of Estmated Quadratc Regresson Functon $175 $150 $15 $100 $75 $ Square Footage

23 Fttng a Thrd Order Polynomal Model We could also ft a thrd order polynomal model, ˆ 3 Ŷ a bˆ X bˆ X b X1 or equvalently, Ŷ ˆ ˆ a b X1 bx where, X X 1 b3x 3 X X 1 ˆ Sellng Prce Graph of Estmated Thrd Order Polynomal Regresson Functon $175 $150 $15 $100 $75 $ Square Footage Polynomal Regresson Overfttng When fttng polynomal models, care must be taken to avod overfttng. The adj.-r statstc can also be used for buldng/fttng polynomal regresson models. We can gauge the amount of overfttng by Valdatng the ft, or usng a tranng sample to buld the model and a valdaton sample to examne ts estmaton or predcton accuracy. 3