Introduction to Regression. Myra O Regan Room 142 Lloyd Institute

Transcription

1 Introduction to Regression Myra O Regan Myra.ORegan@tcd.ie Room 142 Lloyd Institute 1

2 Description of module Practical module on regression Focussing on the application of multiple regression Software Lots of computer output will use R sometimes 2 labs Some Mathematics but no linear Algebra 2

3 Topics to be covered Revision of Simple linear regression Introduction to Multiple regression Use of logs and other transformations Regression Diagnostics Use of Indicator Variables Polynomial regression Building a regression model Dealing with multicollinearity Introduction to Logistic regression Other fun techniques 3

4 Notes and Books I use BlackBoard Sheather, S. J. A Modern Approach to regression with R,, New York:, Springer 2009 Neter, J., Wasserman, W. & Kutner, M.H. Applied Linear Models, 2 nd edition Boston, Irwin:1989 Kutner. M. H., Nachtsheim, C.J., Neter, J. & Li, W. Applied Linear Statistical Models, 5 th, Boston: McGraw-Hill,

5 Purpose of regression To build a model for prediction purposes Price of diamond from number of carats Price of a house Time to process invoices Measuring the volume of wood in trees To look at relationships Factors relating to cot death 5

6 Netflix competition Variables were user, movie, date of grade, grade Grade was measured from 1 to 5 100,480,507 ratings 480,189 users 17,770 movies Movie, title and year of release 6

7 7

8 8

9 308 diamnonds, price, colour, clarity and size 9

10 10

11 11

12 Initial examination of data Know the story behind the data Understand the background Understand meanings of variables Look at each variable separately Check the quality of data Summary statistics and graphs How much missing data? 12

13 Revision of simple linear regression Manager of a purchasing department of a large company would like to predict average amount of time it takes to process a given number of invoices. Data was collected over a sample of 30 days on the number of invoices and time taken in hours Three variables Time, Number of Invoices and Day 13

14 Invoices Time N N* 0 0 Mean SE Mean StDev Minimum Q Median Q Maximum

15 15

16 Model to fit Time i = α + β Invoices i + ε i Linear model Need estimates of α and β Need SE for estimates We use Minitab to calculate estimates of α and β 16

17 17

18 What is going on here? What are the lines? More importantly what are the differences 18

19 Prediction vs Confidence intervals Confidence interval For a given value of x 0 this is an interval for the average value of the dependent variable Point Estimate ± t *s Distance value t has n-(k+1) df where k = no. of predictors s= what does this measure Distance value = 1 n + (x 0 x ) 2 (x i x ) 2 19

20 Prediction vs Confidence intervals Prediction interval For a given value of x 0 this an interval for the particular value of the dependent variable Point Estimate ± t *s 1 + Distance value t has n-(k+1) df where k = no. of predictors s= what doe this measure Distance value = 1 n + (x 0 x ) 2 (x i x ) 2 20

21 Approximate intervals for reasonably large samples Confidence intervals=2*s* 1 n Prediction intervals = 2*s * n 21

22 Example Let number of invoices = 50 Where do these numbers come from roughly? 22

23 ANOVA table Total sums of squares(ss) =(Y i Y) 2 Regression SS=(Y i Y) 2 Error SS =(Y i Y i ) 2 What is R 2? 23

24 What happens if we do the following? Let Invoices=X Subtract k from each case What will change? Time = α + β X + ε original model Time=α + β*(x-k)+ε= (α- βk)+ βx+ ε Slope does not change but intercept does Intercept = expected value of Time when X=k Normally we use k=mean of the variable 24

25 The regression equation is Time = Centered invoices 25

26 26

27 Trees data Sample of 31 black cherry trees in the Allegheny national Forest in Pennsylvania Volume in cubic feet Height in feet Diameter in inches 54 inches above ground 27

28 Variable Diameter Height Volume N N* Mean SE mean StDev Minimum Q Median Q Maximum

29 29

30 30

31 31

32 32

33 33

34 34

35 What does the F-test mean? Testing a hypothesis Null hypothesis H 0 : β 1 = β 2 = 0 Alternative Hypothesis H 1 : Not all β s =0 F=254.97, df=(2,28) p<0.001 Enough evidence against the null hypothesis 35

36 Interpretation of coefficients Volume = β 0 + β 1 *Height+ β 2 *Diameter + ε E(Volume) or Predicted(Volume) or sometimes written as Y = *Height+4.71 *Diameter Constant (-58.0) is the mean response when Height=0 and Diameter=0 β 1 change in mean response per unit increase in Height when Diameter is held constant (at any value) Similarly β 2 change in mean response per unit increase in Diameter when Height is held constant (at any value) 36

37 And a little more Example let Diameter =12 E(Volume) = Height *12 = Height Intercept changes but β 1 stays the same. Effect on mean response of height does not depend on Diameter We say effects are additive or not to interact Partial regression coefficients 37

38 Changing coefficients Height by itself 1.54 (.38) Diameter by itself 5.07 (0.25) Multiple regression Height Diameter 0.34 (0.13) Diameter Height 4.71 (0.26) 38

39 Sums of squares Same calculation as before Sequential sums of squares Diameter & Height Diameter Height Sequential sums of squares Height & Diameter Height Diameter

40 Derived variables Create a new x from the given x-variables Could be a transformation or a combination Use background knowledge to create new variable Tree crudely modeled by cylinder cylinder vol = πr 2 x ht = π 4 (Diam)2 x ht ht (Diam) 2 40

41 Plot first 41

42 42

43 43

44 44

45 45

46 Transform using logs y=log b a; b y =a; 2 3 =8; log 2 8=3; b is called the base Typical bases are e and 10 We are going to use base 10 e is a mathematical number =2.71 logs to the base e are called natural logs often written as ln 46

47 Basic rules for logs using base 10 Log(10) =1 Log(10) a =a Log(1)=0 Log(0) is not defined Log(x r )=rlog(x) 10 log(a) =a Richter scale for measuring earthquake strength is on a log 10 scale 47

48 And some more Log(ab) = log(a)+log(b) log a b = log a log b 10 ab =(10 a ) b; 10 (a+b) =10 a 10 b ;10 a-b = 10a 10 b 48

49 What are we going to do with all this? Linear Model We can take logs of X; of Y; or of both; What we are interested in examining is the interpretation of the coefficients and interpret them in the original scale We will see later when it is appropriate Let us start with the model Y=α + β*log(x) + ε 49

50 50

51 51

52 Interpretation of coefficients A 1 unit increase in log(x) is associated with β increase in Y units log(x)+1 = log(x) +log(10)= log(10x) Converting to a percentage Multiplying X by 10 equivalent to (10-1)*100% change = 900% increase in x β expected change in Y when X is multiplied by 10 β expected change in Y when X increases by 900% 52

53 And more For other percentage changes p p% increase in X = β log ( 100+p ) increase in Y 100 A 10% increase in X associated with β log ( ) increase in Y 100 β *log(1.1) increase in Y β *0.041 increase in Y 53

54 What does this mean? Volume = logheight An increase in 1 in logheight will increase Volume by 262 Multiplying height by 10 will increase Volume by 262 A 10% increase in height will increase Volume by β log ( 100+p 100 ) =262*log(1.1)=

55 Next situation Log(Y)=α+β*X+ε A 1 unit increase in X is associated with β increase in log Y units Log Y + β =10 (log y +β) = Y 10 β Each 1-unit increase in X multiplies the expected value of Y by 10 β The effect of a c-unit increase in X is to multiply the expected value of Y by 10 cβ 55

56 More Calculate ch= Y 10 β Calculate (ch-1)*100 Ch=1.20 implies a 20% increase Ch=.7 implies a 30% decrease 56

57 57

58 And now.. logvolume = Height A 1 unit increase in height increase logvolume by Each unit increase of height increases Volume by a multiple of =1.055 or 5.5% increase 58

59 Last situation Log Y = α +β*log(x) +ε A 1 unit increase in log(x) is associated with β*log(y) units p% increase in X = β log ( 100+p ) increase in log Y units a= β log ( 100+p 100 ) Log Y+a =Y*10 a

60 60

61 Again some interpretation logvolume = logheight A 1 unit increase in logheight will increase logvolume by 3.98 Multiplying height by 10 multiplies Volume by A 10% increase in height multiplies Volume by 10 (3.98*log(1.1)) = 1.46 Can interpret this a 46% increase in Volume 10% increase in height associated with a 46% increase in Volume. 61

62 Interpretation logvolume = logheight Can write as 10 logvolume =10 ( logheight) Volume = * logheight This is sometimes called a multiplicative model Using the above for prediction Height = 85 remember to use log10(85)=1.929 Using Minitab we get (1.2412, ) as PI 62

63 In the original units (1.2412, ) = ( , )= (17.42, 99.38) 85 is not in the centre Will return to when to use logs 63

64 Interpret coefficients in original scale Calculate predicted Sun circulation for weekday circulation of 300,000 both predicted and CI. You can just use the approximate solution 64

65 Interpretations 10% increase in weekly circulation associated with a 10 (1.05*log(1.1)) = increase in Sunday circulation equivalent % Increase in weekly Increase in Sunday % increase in Sunday 65

66 Approximate Confidence Intervals Calculate CI s for weekly circulation of 300,000 Predicted Value= *log(300000) =5.62 on log scale N=89;s= %CI = 5.617±2*0.056* =(5.605,5.629) = 402,835 to 425,473 66

67 Great chapter on derived variables Linoff, G. S & Berry, M. J. A. Data Mining Techniques 3 rd Edition, Wiley: Indianapolis,

68 Some summary thoughts Get to know the story of your data Use simple plots and summary statistics Does it look ok? Think about derived variables Start simply Don t forget your common sense 68