Generalized Linear Models in R

Transcription

1 Generalized Linear Models in R NO ORDER Kenneth K. Lopiano, Garvesh Raskutti, Dan Yang last modified

2 Outline 1. Background and preliminaries 2. Data manipulation and exercises 3. Data structures in R 4. Inputting and reading in data 5. R basic graphics 6. Generalized linear models in R 7. Break 8. Variable selection on hurricane data 2

3 Background An implementation of the S programming language (developed at Bell labs in mid 70 s). Programming language for statistical techniques and graphics/data visualization. Free and open source. Extensive statistical capability for small to medium-sized data. Many packages. Interfaces easily with C/C++/Fortran. Interfaces easily with Hadoop (cloud computing). Other statistical software: e.g. SAS, SPSS, STATA and Minitab, commercial, not open source. Mathematical software: Maple, Mathematica, and Matlab. 3

4 Preliminaries Entering and exiting R To begin: Unix or Linux: type R in command line; Windows: click the R icon. To quit: type q(). Need help? help.start(): start the HTML help system. help(): get help for a particular function. e.g. help(sum) Commands and variables in R are case sensitive. 4

5 Data Manipulation Create data object: > y <- 10 > Y = c("a", "b", "d") View an object: > y [1] 10 > Y [1] "a" "b" "d" Matrix: > a = 1:5 > b = 11:15 > newcol = cbind(a,b) ##Appending column vectors > newcol a b [1,] 1 11 [2,] 2 12 [3,] 3 13 [4,] 4 14 [5,] 5 15

6 Data Manipulation Matrix: > M = rbind(a,b) ##Appending row vectors > M [,1] [,2] [,3] [,4] [,5] a b > M[1,] ##Accessing a row [1] > M[2,3] ##Accessing a single element [1] 13 > dimensions = dim(a) ##Finding dimensions of a matrix > dimensions [1] 2 5

7 Data Manipulation Matrix: > M = rbind(a,b) > M [,1] [,2] [,3] [,4] [,5] a b Ex 1: Use R to find the mean and variance for the first row of M. Functions: mean(), var().

8 Data Manipulation Matrix: > M = rbind(a,b) > M [,1] [,2] [,3] [,4] [,5] a b Ex 1: Use R to find the mean and variance for the first row of M. > a = M[1,]; > mean(a) [1] 3 > var(a) [1] 2.5 Ex 2: Use R to find the correlation between the first and second row of M. Function: cor(.,.)

9 Data Manipulation Matrix: > M = rbind(a,b) > M [,1] [,2] [,3] [,4] [,5] a b Ex 1: Use R to find the mean and variance for the first row of M. > a = M[1,] > mean(a) > var(a) Ex 2: Use R to find the correlation between the first and second row of M. > cor(m[1,],m[2,]) [1] 1 > plot(a,b);

10 Matrix multiplication Matrix: > M = rbind(a,b) > M [,1] [,2] [,3] [,4] [,5] a b > M*M [,1] [,2] [,3] [,4] [,5] [1,] [2,] > M%*%t(M) ##t(.) transpose of a matrix [,1] [,2] [1,] [2,] Ex: Use R to find the first diagonal element of M T M.

11 Matrix multiplication Matrix: > M = rbind(a,b) > M [,1] [,2] [,3] [,4] [,5] a b > M*M ##Element-wise multiplication [,1] [,2] [,3] [,4] [,5] [1,] [2,] > M%*%t(M) ##Matrix multiplication [,1] [,2] [1,] [2,] Ex: Use R to find the first diagonal element of M T M. > N = t(m)%*%m; > N[1,1] [1] 122

12 Data structures in R So far we have seen vectors and matrices. Other data structures: Multi-way arrays - array(), tensors and higher-dimensional arrays. Data frames - data.frame(), tables with columns of a particular type. Lists - list(), vector of generic types.

13 Data Input and Data Frame Read data into an object (data frame): read.table() >mydata=read.table(" > mydata time status group a c b a b Show the variable names: > names(mydata) [1] "time" "status" "group" Extracting the component: > mydata$time [1] > mydata[,1] [1]

14 Control Statement Beyond matrix operations. if... else... > x = 5 > if (x>1) { x = x-1 } else { x = x+1 } > x [1] 4 for... > x = 5 > for (i in 1:10) { x = x+1 } > x [1] 15 while... > x = 5 > while (x<10) { x = x+1 } > x [1] 10 14

15 Defining Functions Define a function: myfun = function(t) { b = t*sin(t) ##Other standard functions cos(), exp(), log() return(b) } Call the function: > x=c(1,5,6) > myfun(x) [1]

16 Plot Plot myfun(): > x = seq(0,10,0.1) > y = myfun(x) > plot(x, y, xlab="input", ylab="response", col="blue", + xlim=c(0,10), ylim=c(-5,8)) > lines(x, y, col="red", lwd=2) > points(1:9, -2:6, col=1:9, pch=19) > title("my Function") My Function Response Input

17 Introduction to GLMs Relationship between response y (e.g hurricane counts) and covariates (x 1, x 2,..., x p ) (e.g. amon, dm, etc.). Start with covariates (p = 1). n observations (x i, y i ), i = 1,..., n R we read in the two datasets - Gaussian and Poisson > ### Set library to location on computer where the example datasets are > setwd("k:/short_courses/samsi UG/lessons/r-datasets") > ### Import the datasets pointozone and mi > input.data.gaussian=read.csv("example.gaussian.csv") > input.data.poisson=read.csv("example.poisson.csv") 17

18 Gaussian and Poisson data > head(input.data.gaussian) x y > head(input.data.poisson) x y

19 Motivation for GLMs A general framework for modeling the relationship between two or more variables. A GLM consists of three components. Let µ = E(y): linear predictor, η = β 0 + β 1 x link function,g(µ) = η, where g is a smooth, monotonic function. The random or stochastic component specifies the distribution of the response variable y. The observations y 1,..., y n are assumed to be independent with some density from the exponential family (e.g, Poisson or Gaussian). 19

20 Example 1: Gaussian data First we plot the raw data in Figure 20. Exploratory Scatterplot Gaussian Data y x 20

21 GLM for Gaussian data y i N (µ i, σ 2 ) link functiong(.) = Id, µ i = η i. η i = β 0 + β 1 x i. > glm.gaussian= glm(y~x, family=gaussian, data=input.data.gaussian) 21

22 GLM Output for Gaussian data > summary(glm.gaussian) # summary of the fit Call: glm(formula = y ~ x, family = gaussian, data = input.data.gaussian) Deviance Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) x <2e-16 *** --- (Dispersion parameter for gaussian family taken to be ) Null deviance: on 99 degrees of freedom Residual deviance: on 98 degrees of freedom AIC: Number of Fisher Scoring iterations: 2 22

23 GLM Outputs cont. > summary(glm.gaussian)$coefficients Estimate Std. Error t value Pr(> t ) (Intercept) e-01 x e-38 > summary(glm.gaussian)$dispersion # dispersion parameter [1] > coefficients(glm.gaussian)# coefficients from the fit (Intercept) x

24 GLM line fit Finally, the raw data and the fit are plotted in Figure1. Scatterplot with Regression Line Gaussian Data y x Figure: Scatterplot of the raw data from the Gaussian example with the estimated regression line. 24

25 Example 2: Poisson data Again we plot the raw data in Figure 2. Exploratory Scatterplot Poisson Data y x Figure: Scatterplot of the raw data from the Poisson example. 25

26 GLM for Poisson data y i Poisson(µ i ) link functiong(.) = log(.), η i = log(µ i ). η i = β 0 + β 1 x i. > glm.poisson= glm(y~x, family=poisson, data=input.data.poisson) 26

27 GLM Output for Poisson data > summary(glm.poisson) Call: glm(formula = y ~ x, family = poisson, data = input.data.poisson) Deviance Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error z value Pr(> z ) (Intercept) <2e-16 *** x <2e-16 *** --- (Dispersion parameter for poisson family taken to be 1) Null deviance: on 99 degrees of freedom Residual deviance: on 98 degrees of freedom AIC: Number of Fisher Scoring iterations: 4 27

28 GLM Poisson fit Finally, the raw data and the fit are plotted. Scatterplot with Regression Line Poisson Data y x 28

29 Prediction from GLM fits Goal of this project is to predict hurricane counts. R creates predictions using predict. > x.pred = data.frame(x=seq(25,60,1)) > predict.gaussian = predict(glm.gaussian, + type="response", + newdata=x.pred, + se.fit=true) > head(predict.gaussian$fit)

30 Variable selection on hurricane data outline Recall goal is to predict hurricane counts from set of environmental features. Adapt to multiple covariate setting (sample 5 variables). Variable/model selection to reduce number of covariates. Two variable selection methods in R: Penalized residuals (AIC, BIC) Cross-validation. 30

31 GLM with multiple variables Suppose there are p predictors (p = 5 in our example): η = g(µ) = β 0 + β 1 x 1 + β 2 x β p x p. For Gaussian or Poisson GLMs, the models are as following respectively. y i N(µ i, σ 2 ) (1) µ i = β 0 + β 1 x i β 5 x i5, (2) y i Poisson(µ i ) (3) log(µ i ) = β 0 + β 1 x i β 5 x i5. (4) 31

32 Sample hurricane data We select hurricane data from region.., level.., July for... years n =..., and we select the following 5 features: Load the data by the following command: > final.data = final.data = read.csv("countscovs.csv") > final.data = final.data[,c("count","amon.csv","limsta.csv", + "dm.csv","ggst.csv","ammsst.csv")] > head(final.data) count amon.csv limsta.csv dm.csv ggst.csv ammsst.csv

33 Scatterplot for features count amon.csv limsta.csv dm.csv ggst.csv ammsst.csv Figure: Scatterplot of the raw data from the Poisson example. 33

34 Histogram of responses y Histogram of final.data$count Frequency final.data$count Figure: Scatterplot of the raw data from the Poisson example. 34

35 Model selection motivation Fit improves with more parameters (i.e, over-fitting) e Model 1: bx Y ae c Model 2: c f X g sin( ) bx Y ae dx Amsterdam04 12 For hurricane prediction, could use all covariates or a subset. Including too many variables will produce a very complicated model and overfit data. Occam s Razor: The simplest explanation is usually the best. 35

36 Model selection motivation cont. Recall that goal is to generalize to future hurricane counts. Relationship between Goodness of Fit and Generalizability Goodness of fit Overfitting Generalizability Model Complexity Amsterdam

37 Model selection by penalized residuals Need to trade off complexity of the model with goodness of fit to data. How to quantify these notions? Penalized residuals framework: n l(y i, ŷ i ) i=1 }{{} Goodness of fit +λ #covariates in model. }{{} Complexity of model y i is actual response and ŷ i is predicted response based on model. l(.,.) is some measure of closeness (e.g. l(y i, ŷ i ) = (y i ŷ i ) 2. λ tradeoff tuning parameter (for AIC λ = 2, for BIC λ = log n ). 37

38 AIC for hurricane data Use AIC in R to determine which of the 5 features to include. Start: AIC= count ~ amon.csv + limsta.csv + dm.csv + ggst.csv + ammsst.csv Df Deviance AIC - ammsst.csv limsta.csv dm.csv ggst.csv <none> amon.csv Step: AIC=

39 AIC for hurricane data Step: AIC= count ~ amon.csv Df Deviance AIC <none> amon.csv ggst.csv dm.csv ammsst.csv limsta.csv Call: glm(formula = count ~ amon.csv, family = poisson, data = final.data) Coefficients: (Intercept) amon.csv Degrees of Freedom: 62 Total (i.e. Null); Null Deviance: Residual Deviance: AIC: Residual 39

40 Cross-validation Cross-validation is alternative approach to cross-validation. Avoids having to determine complexity of model. For k-fold cross-validation: (1) Randomly split the data into k equal (or approximately equal)-sized sets. (2) Select 1 of k sets as test and the remaining k 1 as training. (3) Run GLM on all k 1 sets. (4) Determine ŷ i for each element of test set using output of GLM and determine n i=1 l(y i, ŷ i ). (5) Repeat with each of the k sets as a test set and average residuals. Cross-validation can be used to select the model by choosing the model with the smallest average residuals. 40

41 Using cross-validation for hurricane data > temp.glm1 = glm(count~.,data=final.data, + family=poisson) > temp.glm2 = glm(count~amon.csv,data=final.data, + family=poisson) > temp.glm3 = glm(count~1,data=final.data, + family=poisson) > library(boot) > cost <- function(y,yhat,eps=0.0001) mean((log(y+eps) - log(yhat+eps))^2) > cv.out1 = cv.glm(final.data, + temp.glm1, + cost, + K=10)$delta[1] > cv.out2 = cv.glm(final.data, + temp.glm2, + cost, + K=10)$delta[1] > cv.out3 = cv.glm(final.data, + temp.glm3, + cost, + K=10)$delta[1] > print(c(cv.out1,cv.out2,cv.out3)) [1]