Introduction Statistics 211 - Statistical Methods II Presented January 8, 2018 linear models Dan Gillen Department of Statistics University of California, Irvine 1.1
Logistics and Contact Information Lectures: Monday and Wednesday, 9:30-11:50, DBH 1423 Discussion: Tuesday, 11:00-11:50, SST 122 Course web site: Instructor: http://www.ics.uci.edu/ dgillen/stat211 Dan Gillen Professor and Chair Department of Statistics Office: 2038 Donald Bren Hall Telephone: 4-9862 E-mail: dgillen@uci.edu linear models Office hours: Monday 11:00-12:00, & Tuesday 12:00-1:00, or by appointment 1.2
Description and Textbooks Prerequisites: Description: Required text: Statistics 210 or equivalent, or permission of instructor This course will introduce theory and methods for analyzing non-normal outcomes. We will focus on developing a general theoretical for models and using that to address scientific questions. Agresti A.; Foundations of Linear and Generalized Linear Models linear models 1.3
Description and Textbooks Reference texts: McCullagh P. and Nelder, J.; Generalized Linear Models Hastie T, Tibshirani R, and Friedman J., Elements of Statistical Learning Casella G. and Berger R., Statistical Inference (2nd Ed) linear models 1.4
Software/Computing that are presented in class are primarily done using the R statistical package R is free software and can be installed on multiple platforms You can download R at http://www.r-project.org/ I (highly) recommend that you use R but you may choose to use any other software package that allows you to complete the assigned coursework linear models 1.5
Assignments, Exams and Grading Homework: Midterm Exam: Final Exam: There will be a total of 6-7 homework assignments. Assignments will typically be due 1-1.5 weeks from the date they are handed out. Tentatively scheduled for Wednesday, Feb 21st. The exam will be in-class (closedbook, closed-note), and will cover material through the Wednesday, Feb 14th lecture. The final exam is scheduled for Wednesday, March 21st. The final exam will be a take-, home handed out on Wednesday, March 14th and due on Wednesday, March 21st by 10am. linear models Grading: Homework: 35% Midterm: 30% Final: 35% 1.6
Stat 211 Introduction to generalized (.5 week) Goals of scientific studies Part II - Review of linear (2.5 weeks) Ordinary least squares theory Implications of assumptions Performance of estimators when assumptions fail Regression modeling strategies Prediction Association estimation/testing Weighted least squares linear models 1.7
Stat 211 Part III - Review of asymptotic theory (1 weeks) Review of theoretical tools Asymptotic results Likelihood theory Asymptotic inference Part IV - GLMs for non-normal data (5.5 weeks) models Components of GLMs Fitting GLMs Logistic Model assumptions and diagnostics Parameter interpretations Poisson Model assumptions and diagnostics Parameter interpretations linear models 1.8
Stat 211 Part IV - GLMs for non-normal data cont d (5.5 weeks) Overdispersion Quasi-likelihood Polytomous response methods Proportional odds model Multinomial logistic Part V - Bayesian estimation for GLMs (time permitting) Inclusion of prior distributions Model fitting Interpretation linear models 1.9
First Stage of Hypothesis generation Observation Measurement of existing populations or systems Disadvantages: Confounding Limited ability to establish cause and effect Further Stages of Refinement and confirmation of hypotheses Experiment Intervention Elements of experiment Overall goal and specific aims (hypotheses) Materials and methods Collection of data Analysis Interpretation; Refinement of hypotheses linear models 1.10
Common aims of data analysis A statistical analysis is often geared towards (at least) one three common objectives 1. Hypothesis generation (description, exploration) 2. Hypothesis testing (inference about associations between variables in a population) 3. Prediction (for future sampling units) linear models 1.11
Distinctions without a difference? In application, not enough attention is given to distinguishing between these objectives This lack of distinction is because the same statistical tools (ie. generalized models) can be used to address all of these objectives HOWEVER, the strategies used to address each goal should be distinct and the interpretation of results is dependent upon the strategy employed! linear models 1.12
Distinctions without a difference? Hypothesis generation requires data-driven modeling (the scientific method!) which hopefully yields a simplified model (hypothesis) Data-driven modeling implies that usual inferential statements (p-values, confidence intervals, posterior probabilities) are invalid due to multiple comparisons Hypothesis testing seeks to test a formal (typically parsimonious) hypothesis via valid inferential statements NO data-driven modeling! linear models Prediction necessitates model building and data-driven results since simple parsimonious models (eg. only linear effects) do not generally lead to a reliable prediction model Requires much stronger (generally untestable) assumptions to make probability statements 1.13
Hypothesis Generation Example: Factors that influence median life expectancy in stage II breast cancer General goal: Want to identify factors that affect prognosis Follow a cohort of newly diagnosed patients and measure survival time (may be censored) along with covariates of interest What defines the cohort? All patients from a particular healthcare plan? All patients diagnosed at a particular hospital? All patients in a certain location? Best" estimator of the median survival (??) Model building to identify most important factors Estimation of median survival for various combinations of factors linear models 1.14
Hypothesis Testing Example: Effect of serum albumin levels on risk of mortality in end-stage renal disease patients Randomly sample a cohort of ESRD patients and follow until death (or study end) Hypothesis a first-order relationship between albumin and the log-relative risk of death A priori specify additional adjustment variables: age, gender, ethnicity, duration of disease, etc. linear models 1.15
Prediction Example 1: Spam filters Goal: Use email content to classify an email as valid or spam Potential factors include: Individual words or characters Message length Attachments Likely to be a non-linear association between message length and the probability of spam May be interactions between types of words linear models May be interactions between types of words, length of message, and the presence of attachments 1.16
Prediction Gaussian kernel:! = 0.5 HRT use (per 1,000) 150 250 350 450 550 1997 1998 1999 2000 2001 2002 2003 2004 Calendar date 352 S. Haneuse; Biostat/Stat 572 Example 2: Hormone replacement therapy use Hormone replacement therapy slows the decrease of bone density in post-menopausal women Observational studies also suggest decrease in CVD Stock brokers would like to predict future use of HRT Goal: Collect monthly HRT use over the past 7 years and predict future use (if done in 1999...oops!) linear models 1.17
Required steps in an inferential data analysis 1. Aims of a data analysis Description, exploration, confirmation, prediction 2. Establish the context of the analysis Statistics produces inference about a population based upon a sample Need to understand the population sampled Understand the data collection procedure (true random sample?) linear models Understand the background science The scientific goals of the analysis/experiment 1.18
Required steps in an inferential data analysis 3. Develop a statistical model Clearly defined (measurable) outcome is essential Predictor(s) of interest If we cannot decide which parameters would be appropriate when measurements are available on the entire population then there is no chance that statistics can be of help! linear models 1.19
Required steps in an inferential data analysis 4. Evaluation of the properties of the design, model, and estimation procedure Essential that these aspects be addressed as completely as possible prior to data analysis Clear specification of outcomes and predictors Use of robust statistical methods What is the cost of planning not to plan? linear models 5. Computation Turn the handle... 1.20
Required steps in an inferential data analysis 6. Interpretation of results Present results clearly and precisely If applicable, present scientific justification for why results agree with the hypothesis (should have been done at the design stage) The most elegant experiment/data analysis is meaningless unless it can be easily explained to the scientific community it was designed to impact linear models 1.21
Summary The basic distinction in strategies comes with respect to model selection When making inferential statements, model selection should be avoided when possible Know what you want to adjust for, and adjust for it." As long as robust statistical procedures are used, this will ensure valid probability statements When performing data-driven modeling we need to clearly define our criteria for choosing the best model and be careful with probability statements How to choose covariates? linear models What functional form? What does a 95% confidence interval mean at the end of the day? 1.22
Comparing distributions across subpopulations In many situations, the scientific question to be addressed by a statistical analysis can be viewed as comparing the distribution of some random variable or vector (the response or dependent variable) across subpopulations defined by the values of other random variables (the predictors or independent variables). Within each subpopulation, there is a distribution for the response variable. The question is whether that distribution is the same for every subpopulation. linear models 1.23
General The standard methods for addressing such questions depends upon the type of response variable and the number of subpopulations being compared. In the case of a univariate response with independent observations across subjects we can make a chart of the basic type of analysis used to test hypotheses according to whether the response variable is binary, categorical, count data, continuous, or censored continuous. We can similary consider whether the number of subpopulations is one, two, several unordered, several ordered, or potentially infinite ordered. linear models 1.24
Our Setting General Response Variable Predictor Binary Categorical Discrete Continuous Censored Constant Z-test χ 2 Poisson t-test KM Binary χ 2 χ 2 Poisson t-test logrank Categorical χ 2 χ 2 log linear ANOVA k logrank Discrete logistic polytomous Poisson linear prop hzd Continuous logistic polytomous Poisson linear prop hzd (Note that the predictor measuring which subpopulation a subject belongs to is, respectively, constant, binary, categorical, discrete, or continuous.) linear models Proposition: Under appropriate assumptions, each of the lines in the above table can be regarded as special cases of the lines below it. (proof deferred) 1.25
Regression modeling The general problem we will address is one where we have for the i th subject: 1. measured variables R i1, R i2,... related to some quality to be compared across the subpopulations, and 2. measured variables (factors) P i1, P i2,... related to subpopulation membership We assume that analysis shall proceed by defining for some functions f R and f Pj, j = 1,..., p linear models 1. response variable Y i = f R (R i1, R i2,... ) 2. predictor variables X ij = f Pj (P i1, P i2,... ) 1.26
Regression modeling Basic assumptions for methods development Statistical modeling will proceed by using the X ij s to model some functional of the distribution of the Y i s. When presenting methodologic development, we will presume knowledge of the form of the response and predictors. When exploring the application of these methods and their robustness to departures from the underlying assumptions of the theory, we will regard that we are free to explore alternative formulations of the predictors and response. linear models 1.27
In generalized, we model the interrelationships among subpopulations (as defined by predictor variables) with respect to the distribution of some response variable Unlike linear, we shall not always consider modelling the mean, and even when we do, we will not necessarily model the mean as a linear function of the parameters linear models 1.28
Definition: In the general model, we typically have Response Y X F(y; h( X, β), φ) where β represents parameters and φ is some nuisance parameter necessary to know the full distribution of Y The model is fully specified by providing the form of F taking arguments y, h, and φ. If all of our parameters of interest are in β, (i) when φ is finite dimensional, we call the model parametric (ii) when φ is infinite dimensional, we call the model semiparametric If β does not contain all of our parameters of interest, (iii) when φ is infinite dimensional, we call the model nonparametric linear models 1.29
In generalized models, h( X, β) most often models some parameter of F or some functional of the distribution (e.g., the mean, the median, the hazard, etc.). It is quite often the case that we choose h( X, β) = h( X T β) that is, F depends on β only through some linear combination of its elements. (Recall X may include arbitrary transformations of the measured factors). linear models 1.30
Definition: A generalized linear model has 1. F an exponential family distribution with pmf or pdf ( ) yθ b(θ) f (y; θ, φ) = exp + c(y, φ) a(φ) for some functions a( ), b( ), c(, ). 2. E(Y ) = µ = b (θ) modelled by linear models g(µ) = η = X T β where g( ) is termed the link function. 1.31
Definition: A model shall consist of 1. θ, a functional of the probability distribution of a response variable Y, 2. η = h( X, β), a predictor function based on the predictors X and the parameters β. (Populations are defined as the set of individuals having the same value of η. 3. g( ), a link function describing the relationship between θ and η: η = g(θ) 4. φ, a nuisance parameter vector that is required to be able to fully specify the distribution of the response variable Y in every subpopulation. That is, the distribution of Y will depend upon the predictor function η and the nuisance parameters φ. linear models 1.32
of commonly used models 1. linear : F(y, h, φ) = Φ((y h)/ φ) (the normal cdf) θ = E[Y ] g( ) is the identity link a(φ) = σ 2 linear models 1.33
of commonly used models 2. logistic : F(y, h, φ) = h y (1 h) 1 y (the Bernoulli pmf) θ = logit(e[y ]) (the log-odds of success ) g( ) is the logit link a(φ) = 1 linear models 1.34
of commonly used models 3. Poisson : F(y, h, φ) = e h h y /y! (the Poisson pmf) θ = log(e[y ]) (with E[Y ] usually standardized as a rate) g( ) is the log link a(φ) = 1 linear models 1.35
Note: In order for a model to be useful, we must (at minimum) have a means of dealing with the nuisance parameter φ and estimating the parameters β. linear models 1.36
A prelude to parameter estimation Estimating equations Definition: In the abstract situation, if we have a function of the data and the unknown parameters that has expectation E[G( X, β, φ)] = 0 for all β and φ then one possible form of estimation is to use such a function as an (unbiased) estimating equation. That is, we will search for the values of ˆ β and ˆφ such that G( X, ˆ β, ˆφ) = 0. linear models 1.37
A prelude to parameter estimation Estimating equations There are many constraints that we need put on such an equation before it stands much chance of producing useful estimates: 1. We would like the estimates produced to be unique as the information about the parameters goes to infinity in our sample 2. We would like the estimates to be efficient and consistent 3. We would like the estimates to be relatively easy to find linear models 4. We would like to be able to estimate the distribution of the estimates 1.38
A prelude to parameter estimation Notes Note 1: Maximum likelihood leads to one such estimating equation (the score equation), however we may choose other estimating equations which do not necessarily correspond to a likelihood (more later) Note 2: A Bayesian estimation seeks to incorporate prior information" in order to make direct probability statements regarding the model parameters. Estimation will be carried out by specifying an appropriate likelihood and combing the likelihood with prior distributions on model parameters to form a posterior distribution for the parameters linear models 1.39