Well-developed and understood properties

1 INTRODUCTION TO LINEAR MODELS 1 THE CLASSICAL LINEAR MODEL Most commonly used statistical models Flexible models Well-developed and understood properties Ease of interpretation Building block for more general models 1 General Linear Model 2 Generalized Linear Model 3 Generalized Estimating Equations 4 Generalized Linear Mixed Model, etc 5 Heirarchical Generalized Linear Mixed Model, etc EXAMPLES: 1 EXAMPLE 1 Simple linear regression Objective: Relate weight to blood pressure Consider a random sample of n individuals The i-th patient has weight x i and blood pressure Y i (i = 1,2,,n) where Y i is the response variable, x i is a regressor variable, Y i = β 0 + x i + ε i, β 0, are regression coefficients unknown model parameters to be estimated, ε i is an error term

2 1 INTRODUCTION TO LINEAR MODELS x y 0 5 10 15 0 200 400 600 800 Figure 1 y= x + 3*x^2 y= -123 + 49*x 2 EXAMPLE 2 Polynomial regression Objective: Same as EXAMPLE 1 Y i = β 0 + x i + β 2 x 2 i + ε i Is this still a linear model? 3 EXAMPLE 3 Multiple linear regression Objective: Relate blood pressure to weight and age For the i-th patient, x i1 = is weight and x i2 = is age Y i = β 0 + x i1 + β 2 x i2 + β 3 x i3 + ε i where x i3 = x i1 x i2 is the weight-by-age interaction term

1 INTRODUCTION TO LINEAR MODELS 3 4 EXAMPLE 4 Data Transformations Seber & Lee, Example 12 The Law of Gravity The Inverse Square Law states that the force of gravity F between two bodies a distance D apart is given by F = c D β Question: By transforming variables, how can this be viewed as a linear regression model for the paramter β? where Y i = log(f i ), x i = log(d i ), β 0 = log(c), = β, ε i is an error term Y i = β 0 + x i + ε i, Seber states and from experimental data we can estimate β and test whether β = 2

4 1 INTRODUCTION TO LINEAR MODELS MATRIX REPRESENTATION OF LINEAR MODELS The general linear model in matrix form: Y 1 x 10 x 11 x 12 x 1,p 1 Y 2 = x 20 x 21 x 22 x 2,p 1 Y n x n0 x n1 x n2 x n,p 1 Equivalent shorthand form: Y = Xβ + ε β 0 β 2 β p 1 Y (n 1) is the response vector + X (n p) is the design (or model or regression) matrix β (p 1) is the vector of regression coefficients (model parameters) ε (n 1) is the error vector (mean 0) ε 1 ε 2 ε n (a) Example 1 Y 1 Y 2 Y n 1 x 1 1 x 2 = 1 x n ( β0 ) + ε 1 ε 2 ε n

1 INTRODUCTION TO LINEAR MODELS 5 (b) Example 2 Y 1 Y 2 Y n 1 x 1 x 2 1 1 x 2 x 2 2 = 1 x n x 2 n β 0 β 2 + ε 1 ε 2 ε n (c) Example 3 Y 1 1 x 11 x 12 x 13 Y 2 = 1 x 21 x 22 x 23 Y n 1 x n1 x n2 x n3 β 0 β 2 β 3 + ε 1 ε 2 ε n (d) Example 4 log(f 1 ) log(f 2 ) = log(f n ) 1 log(d 1 ) 1 log(d 2 ) 1 log(d n ) ( β0 ) + ε 1 ε 2 ε n

6 1 INTRODUCTION TO LINEAR MODELS (e) EXAMPLE 5 One-way analysis of variance Objective: Compare two treatments for blood pressure Consider random samples of J individuals taking one of two blood pressure medications Y ij is the blood pressure for individual j from treatment group i Y ij = µ + α i + ε ij µ = overall mean blood pressure, α i = effect on blood pressure for treatment i (i = 1,2), ε ij = error term for subject j receiving treatment i Alternate Model Representation: Y ij = µ + α 1 I 1 + α 2 I 2 + ε ij where I 1 is an indicator variable for membership in treatment group 1 and I 2 is an indicator variable for membership in treatment group 2 (f) EXAMPLE 6 Two-way analysis of variance Objective: Same as EXAMPLE 5, but we now also consider a patient s sex Y ijk = µ + α i + β j + ε ijk µ = overall mean blood pressure, α i = effect on blood pressure for treatment i (i = 1,2), β j = effect on blood pressure for sex j (j = 1,2), ε ijk = error term for subject k of sex j receiving tmnt i Alternate Model Representation: Y ij = µ + α 1 I 1 + α 2 I 2 + + J 1 + β 2 J 2 + ε ijk where I 1 is an indicator variable for membership in treatment group 1, I 2 is an indicator variable for membership in treatment group 2, J 1 indicates sex 1, and J 2 indicates sex 2

1 INTRODUCTION TO LINEAR MODELS 7 (g) EXAMPLE 7 Analysis of covariance Objective: Same as EXAMPLE 5, controlling for age Y ij = µ + α i + β(x ij x ) + ε ij µ = overall mean blood pressure, α i = effect on blood pressure for treatment i (i = 1,2), β = slope parameter, x ij = age of subject j receiving treatment i, x = overall mean age, ε ij = error term for subject j receiving treatment i Note: The alternative model representations for these ANOVA and AN- COVA models make it clear that these are linear models Let s continue with matrix representation of these models 5 Example 5 Y 11 Y 1J Y 21 Y 2J = 1 1 0 1 1 0 1 0 1 1 0 1 µ α 1 α 2 + ε 11 ε 1J ε 21 ε 2J

8 1 INTRODUCTION TO LINEAR MODELS 6 Example 6 Y 111 Y 11K Y 121 Y 12K Y 211 Y 21K Y 221 Y 22K = 1 1 0 1 0 1 1 0 1 0 1 1 0 0 1 1 1 0 0 1 1 0 1 1 0 1 0 1 1 0 1 0 1 0 1 1 0 1 0 1 µ α 1 α 2 β 2 + ε 111 ε 11K ε 121 ε 12K ε 211 ε 21K ε 221 ε 22K 7 Example 7 Y 11 Y 1J Y 21 Y 2J = 1 1 0 (x 11 x ) 1 1 0 (x 1J x ) 1 0 1 (x 21 x ) 1 0 1 (x 2J x ) µ α 1 α 2 β + ε 11 ε 1J ε 21 ε 2J

1 INTRODUCTION TO LINEAR MODELS 9 In Summary: Linear models have the form Y = Xβ + ε, where Y n 1 response vector, X n p model matrix, β p 1 vector of unknown regression parameters, ε n 1 mean zero random error vector Notes: 1 Usually x i0 = 1 for all i That is, usually there is an intercept β 0 in the model and the first column of the design matrix X is all 1 s 2 x i0,x i1,,x i,p 1 are called the predictor variables or regressor variables or the covariates They are the data 3 Linear Model means the model is linear in the unknown regression coefficients β 0,,,β p 1 4 Instead of matrices, we can write the model in terms of vectors: p 1 Y = β j x j + ε, j=0 where x j = (x 1j,x 2j,,x nj ) (Also note that we will use the convention in this course that a vector is a column vector) 5 ε is the random part of the model Right now we just assume that E(ε) = 0 Later, we will make more assumptions about the distribution of ε 6 Y is random because ε is random Y inherits randomness from ε 7 We can thus evaluate p 1 E[Y] = E[ j=0 p 1 β j x j + ε] = E[ j=0 p 1 β j x j ] + E[ε] = β j x j 8 The vector E[Y] is a linear combination of the x j 9 E[Y] span(x 0,x 1,,x p 1 ) Ω, j=0

10 1 INTRODUCTION TO LINEAR MODELS FACT: We obtain least squares estimators (LSE s) of the β j, denoted ˆβ j, by projecting Y onto Ω Note that Y is n-dimensional and Ω has dimension p The projection of Y onto Ω is denoted Ŷ In Class Exercise: Model Y i = βx i + ε i (linear regression through the origin) Two datasets: 1 {(x,y) = (2,1),(0,1)} 2 {(x,y) = (1,2),(1/2,2)} For each dataset: 1 Write out the model with vectors and matrices (using the data) 2 Show the vectors y and x on a graph 3 Identify (on your graph) Ω, 4 Plot ŷ = Proj Ω (y), 5 Plot ˆε = y ŷ, identify Ω 6 What is the dimension of Ω? What is the dimension of Ω? 7 Also make a scatterplot of the data and sketch the least squares line 8 Alternatively, if we fit the model Y i = β 0 + x i + ε i, how do your answers to the previous questions change?

1 INTRODUCTION TO LINEAR MODELS 11 2 Dataset 1 2 Dataset 2 dimension 2 1 0 dimension 2 1 0 1 1 0 1 2 dimension 1 Scatterplot 2 1 1 0 1 2 dimension 1 Scatterplot 2 1 1 y y 0 0 1 1 0 1 2 x 1 1 0 1 2 x