Pseudo-score confidence intervals for parameters in discrete statistical models

Save this PDF as:

Size: px
Start display at page:

Transcription

1 Biometrika Advance Access published January 14, 2010 Biometrika (2009), pp. 1 8 C 2009 Biometrika Trust Printed in Great Britain doi: /biomet/asp074 Pseudo-score confidence intervals for parameters in discrete statistical models BY ALAN AGRESTI Department of Statistics, University of Florida, Gainesville, Florida 32611, U.S.A. AND EUIJUNG RYU Division of Biomedical Statistics and Informatics, Mayo Clinic, Rochester, Minnesota 55905, U.S.A. SUMMARY We propose pseudo-score confidence intervals for parameters in models for discrete data. The confidence interval is obtained by inverting a test that uses a Pearson chi-squared statistic to compare fitted values for the working model with fitted values of the model when a parameter of interest takes various fixed values. For multinomial models, the pseudo-score method simplifies to the score method when the model is saturated and otherwise it is asymptotically equivalent to score and likelihood ratio test-based inferences. For cases in which ordinary score methods are impractical, such as when the likelihood function is not an explicit function of model parameters, the pseudo-score method is feasible. We illustrate the method for four such examples. Generalizations of the method are also presented for future research, including inference for complex sampling designs using a quasilikelihood Pearson statistic that compares fitted values for two models relative to the variance of the observations under the simpler model. Some key words: Categorical data; Complex sampling; Contingency table; Multinomial model; Pearson chi-squared statistic; Quasilikelihood. 1. INTRODUCTION For contingency tables, large-sample score-test-based inference performs well, often even for relatively small samples. For goodness-of-fit tests of generalized linear models, the score test statistic is the Pearson statistic comparing observed and fitted values (Smyth, 2003). For interval estimation of a parameter in a multinomial model, we propose a pseudo-score method based on inverting a Pearson statistic that compares the model-fitted values to the values fitted assuming a particular value of that parameter. The inferences are asymptotically equivalent alternatives to actual score and likelihood ratio test-based inferences. When score confidence intervals are difficult to construct, such as when the likelihood function is not an explicit function of the model parameters, it is still possible to conduct pseudo-score inference. After introducing the pseudo-score method and discussing the connection between ordinary score and pseudo-score inferences, we illustrate pseudo-score confidence intervals for four models for which the ordinary score method is difficult to implement. The emphasis of this article is on modelling categorical response variables having a multinomial distribution. However, we also present a generalization of the pseudo-score method for complex sampling designs and suggest future research on a quasilikelihood generalization that uses an extended Pearson statistic comparing fitted values for two nested models relative to the estimated covariance matrix of the observations under the simpler model. Such methods are useful in applications for which likelihood-based methods such as profile likelihood confidence intervals are not available.

3 Miscellanea 3 Table 1. Contingency table for illustrating pseudoscore inferences for models for ordinal data Health care Environment Total Total not practical but the pseudo-score methods can be implemented with the same level of difficulty as profile likelihood confidence intervals. The next section shows examples of four such models. Second, as 5 discusses, extensions apply to settings in which profile likelihood methods are not available. Third, research has shown that ordinary score inferences, when available, perform well for a variety of measures for discrete data, in terms of actual error probabilities being close to nominal levels. In fact, they often perform better than likelihood ratio test-based inference and much better than Wald test-based inference when sample sizes are not large. This may reflect the fact that for canonical models, the score statistic is a standardization of a sufficient statistic that is a linear combination of the observations, and uses a null rather than non-null standard error. For example, see Koehler & Larntz (1980) for testing independence in two-way contingency tables, Newcombe (1998a) for confidence intervals for binomial proportions, Newcombe (1998b) and Agresti & Min (2005) for confidence intervals for the difference of proportions and relative risk and Miettinen & Nurminen (1985) and Agresti & Min (2005) for confidence intervals for the odds ratio. Our simulations have suggested that similar good performance occurs for pseudo-score confidence intervals. 3. EXAMPLES OF PSEUDO-SCORE CONFIDENCE INTERVALS As mentioned above, for inference about a parameter in an unsaturated model, the score statistic is not the Pearson statistic and it is often impractical to find it. Even for simple models, ordinary score confidence intervals are not available with common statistical software. Using such software, it is possible to implement the pseudo-score method by fitting the model and then fitting the reduced model for various β 0 values and comparing their fitted values using X 2 (β 0 ). This section illustrates pseudo-score confidence intervals by presenting four relatively simple models for which ordinary score-test-based inference is difficult. We illustrate the models by fitting them to Table 1 from the 2006 General Social Survey in which U.S. respondents were asked, How successful do you think the government in America is nowadays in (a) Providing health care for the sick? (b) Protecting the environment? The outcome categories are 1 = successful, 2 = neither successful nor unsuccessful, 3 = unsuccessful. Table 1 shows results for subjects of ages We treat the nine cell counts as a multinomial sample. Let (y 1, y 2 ) denote the bivariate ordinal response in Table 1. One type of model analyzes the association between the variables, such as by assuming a common global log odds ratio, { } pr(y1 i, y 2 j)pr(y 1 > i, y 2 > j) log = β pr(y 1 i, y 2 > j)pr(y 1 > i, y 2 j) for all i, j. The likelihood function is a complex function of the model parameter, making ordinary score methods difficult. Pseudo-score inferences are relatively simple, by fitting the model for various fixed β 0 using methods for maximizing a multinomial likelihood function subject to constraints. To fit the model, we used the R function mph.fit written by Prof. Joseph Lang at the University of Iowa, which uses methods developed in Lang (2004, 2005). Here, the constraints equate all four of the global log odds ratios to a common unknown value for the general model and equate them to a fixed value β 0 for the special cases. The model with unspecified β fits Table 1 well, with Pearson goodness-of-fit statistic 1 75 based on 3 degrees of

4 4 ALAN AGRESTI AND EUIJUNG RYU freedom and maximum likelihood estimate ˆβ = with standard error The 95% pseudo-score confidence interval for β is (0 556, 1 796). For example, the model under the constraint β 0 = or under the constraint β 0 = has maximum likelihood fitted values for which X 2 (β 0 ) = The profile likelihood interval is (0 562, 1 809), quite similar. An alternative type of model compares the marginal distributions of this square table. The cumulative logit marginal model logit{pr(y 1 j)} =α j, logit{pr(y 2 j)} =α j + β ( j = 1, 2), is designed to detect a location shift in the two marginal distributions. The multinomial likelihood function for the nine cell probabilities cannot be explicitly expressed in terms of the model parameters, which limits the applicability of ordinary score-test-based inference. We used Lang s R function mph.fit to fit the model and special cases under the constraint that cumulative logits in the two margins differ by a fixed value β 0. This marginal model with unconstrained β fits these data adequately, with goodness-of-fit statistic X 2 = 0 01 based on 1 degree of freedom and with ˆβ = having standard error The 95% pseudo-score confidence interval for β is ( 0 617, 0 157). The profile likelihood confidence interval is ( 0 616, 0 153). An alternative way to compare the response distributions is with a random effects model. For responses (y 1s, y 2s ) by subject s, a random intercept model is logit{pr(y 1s j)} =u s + α j, logit{pr(y 2s j)} =u s + α j + β ( j = 1, 2), where {u s } are independent from a N(0,σ) distribution with unknown standard deviation σ. The marginal likelihood obtained by integrating out the random effects does not have closed form, and score confidence intervals are difficult for this model. It is straightforward to obtain a pseudo-score confidence interval for β or σ when σ>0 by fitting the model for various fixed values of the parameter of interest. The unrestricted model fits Table 1 adequately, with goodness-of-fit X 2 = 2 0 based on 4 degrees freedom and with ˆβ = having standard error and ˆσ = 1 43 having standard error The 95% pseudo-score confidence interval for β is ( 0 833, 0 200), and the profile likelihood confidence interval is ( 0 848, 0 201). The corresponding confidence intervals for σ are (0 83, 2 08) and (0 84, 2 10). Among other models for which the pseudo-score method is simpler to implement than the ordinary score method, because the likelihood function is not an explicit function of the model parameters, are models that assign scores {v j } to ordered categories and describe mean responses. For Table 1, a regression form of the model describes how the mean response for y 2 changes linearly in the score for y 1, and a marginal model compares the marginal mean responses. The first of these models is j v jpr(y 2 = v j y 1 = v i ) = α + βv i. For this model we treat each row of the table as an independent multinomial sample. Lipsitz (1992) and Lang (2005) presented algorithms for maximum likelihood fitting of such models. For Table 1, we treat the health care opinion as the response variable and use the row numbers and column numbers as the scores. The mean response model fits well, with Pearson goodness-of-fit statistic X 2 = 0 29 based on 1 degree of freedom. The 95% pseudo-score confidence interval for β is (0 143, 0 475). The profile likelihood confidence interval is (0 145, 0 479). 4. SIMULATION EXAMPLE We promoted pseudo-score inference by suggesting that it applies in cases in which ordinary score inference is impractical to implement, and for many cases ordinary score inference has been seen to perform well, even with relatively small samples. The previous section showed four examples of its use when score inference is impractical, with the pseudo-score results being similar to profile likelihood confidence intervals. To check whether the good performance may also apply to pseudo-score inference, we conducted some simulations using models mentioned in the previous section. We found that the pseudoscore method performed similarly to the profile likelihood interval and sometimes a bit better when the sample size is quite small. We illustrate typical results with the cumulative logit marginal model, logit[pr(y 1 j)] = α j and logit[pr(y 2 j)] = α j + β. We simulated pseudo-score, profile likelihood and Wald confidence intervals

5 Miscellanea 5 Table 2. Simulated coverage percentages, with percentages of underestimation and overestimation in parentheses, for interval estimation of β in marginal cumulative logit model with sample size n and underlying bivariate normal distribution with correlation 0 or 0 5 in 2 2 and 3 3 tables 2 2 Correlation, with β = 0 0 Correlation, with β = 0 5 n Method Pseudo-score 95 6 (2 2, 2 2) 95 1(2 4, 2 5) 95 1(2 4, 2 5) 95 0(2 9, 2 1) Profile likelihood 92 7 (3 7, 3 7) 91 6(4 2, 4 2) 93 6(3 6, 2 9) 93 2(3 8, 3 2) Wald 93 3 (3 3, 3 3) 92 9(3 5, 3 5) 94 6(2 4, (2 9, 3 4) 50 Pseudo-score 94 7 (2 7, 2 7) 95 0(2 5, 2 5) 95 1(2 4, 2 5) 95 1(2 6, 2 3) Profile likelihood 94 7 (2 7, 2 7) 94 3(2 8, 2 8) 94 6(2 8, 2 6) 94 7(2 9, 2 5) Wald 94 7 (2 7, 2 7) 94 5(2 8, 2 7) 94 8(2 6, 2 6) 94 6(2 6, 2 8) 3 3 Correlation, with β = 0 0 Correlation, with β = 0 5 n Method Pseudo-score 92 6 (3 7, 3 7) 91 5(4 2, 4 3) 93 7(3 7, 2 5) 94 2(3 2, 2 6) Profile likelihood 91 4 (4 4, 4 3) 87 2(6 3, 6 5) 92 1(5 1, 2 8) 90 5(6 3, 3 2) Wald 93 0 (3 5, 3 5) 90 7(4 5, 4 8) 93 9(3 4, 2 6) 93 4(3 4, 3 2) 50 Pseudo-score 94 5 (2 8, 2 7) 93 9(3 0, 3 0) 94 5(2 8, 2 7) 94 1(3 3, 2 6) Profile likelihood 94 1 (2 9, 2 9) 93 4(3 3, 3 3) 94 2(3 1, 2 7) 93 5(3 7, 2 8) Wald 94 3 (2 9, 2 8) 93 8(3 1, 3 1) 94 3(2 9, 2 8) 94 1(2 9, 3 0) for β with 2 2 and 3 3 tables having β = 0 and 0 5, n = 20 and 50, based on a joint distribution that satisfied the model with uniform values for the row marginal probabilities and for joint cell probabilities based on underlying bivariate normal distributions with correlations 0 and 0 5. Table 2 shows the estimated coverage percentages based on simulations; an estimator of a true coverage percentage of 95 0 has standard error 0 1. For each case, the table also reports the directional error rates, estimating the percentages of cases in which the interval falls below, or above, the parameter value. In the 3 3 case with β = 0 5 and n = 20, in about 2% of the simulations the table was such that the model-fitting algorithm did not converge or at least one of the methods broke down, typically because of an infinite ˆβ, so in that case results apply conditional on this not happening. We do not report results with unbalanced marginal probabilities, because then the frequency of such problematic tables was excessive. Table 2 does not show substantially different results for the three methods, but in many small-sample cases the pseudo-score method performed somewhat better than the profile likelihood method. In this simulation and in others we conducted, when the directional error rates differed substantially, the imbalance was a bit worse for the profile likelihood method than for the pseudo-score method, as illustrated for the cases for 3 3 tables with n = 20 and β = 0 5. While there is no guarantee that similar behaviour occurs for other models or cases, our simulations were promising. The pseudo-score confidence interval is simple to construct and it shows promise of being a good, general-purpose method for inference with categorical response data when the ordinary score method is not feasible. The simulation results do not suggest that methodologists should abandon profile likelihood methods in favour of the pseudo-score method. However, as discussed next, there is scope for extending the pseudo-score method to cases in which likelihood-based methods are not available. 5. GENERALIZATIONS OF PSEUDO-SCORE INFERENCE 5 1. Pseudo-score inference for other discrete distributions Suppose y 1,...,y n are independent observations assumed to have some discrete distribution other than the multinomial. A Pearson-type pseudo-score statistic for comparing models has the form X 2 = i (ˆμ i ˆμ i0 ) 2 v(ˆμ i0 ) = (ˆμ ˆμ 0 ) T ˆV 1 0 (ˆμ ˆμ 0 ),

6 6 ALAN AGRESTI AND EUIJUNG RYU where v(ˆμ i0 ) denotes the estimated variance of y i under the null distribution for y i and ˆV 0 is the diagonal matrix of such values (Lovison, 2005). For some cases, the pseudo-score confidence intervals of this paper for multinomial models extend to other models for discrete data using this generalized statistic. An example is estimating parameters of Poisson regression models. One application for which this extended statistic is especially useful is analyzing discrete data obtained with a complex sampling scheme, by replacing ˆV 0 by an appropriately inflated or nondiagonal estimated covariance matrix. To illustrate, 3 treated the General Social Survey as a simple random sample, but in reality it is a multi-stage sample that employs stratification and clustering. The codebook for the survey suggests that because of its design, sampling variances of estimates are approximately 50% larger than obtained with simple random sampling. For the examples in 3, we can obtain more relevant 95% confidence intervals from the set of β 0 with i (ˆμ i ˆμ i0 ) 2 /(1 5ˆμ i0 ) For the marginal cumulative logit model, for instance, we get the interval ( 0 708, 0 248) for β instead of ( 0 617, 0 157). For such complex sampling designs, profile likelihood confidence intervals are not usually available Quasilikelihood inference for marginal modelling An interesting problem for future research is to extend pseudo-score inference to other quasilikelihood analyses. A possible application is marginal modelling of clustered categorical responses. A popular approach for marginal modelling uses generalized estimating equations. Because of the lack of a likelihood function, Wald methods are commonly employed, together with a sandwich estimator of the covariance matrix of model parameter estimates. For binary data, let y it denote observation t in cluster i, fort = 1,...,T i and i = 1,...,n.Lety i = (y i1,...,y iti ) T and let μ i = E(y i ) = (μ i1,...,μ iti ) T.LetV i denote the T i T i covariance matrix of y i. For a particular marginal model, let ˆμ i denote an estimate of μ i, such as the maximum likelihood estimate under the naive assumption that the i T i observations are independent. Let ˆμ i0 denote the corresponding estimate under the constraint that a particular parameter β takes value β 0.Let ˆV i0 denote an estimate of the covariance matrix of y i under this null model. The main diagonal elements of ˆV i0 are ˆμ it0 (1 ˆμ it0 )(t = 1,...,T i ). Separate estimation is needed for the null covariances, which are not part of the marginal model. With categorical explanatory variables, an estimate of cov(y it, y iu ) is the sample mean value of (y at ˆμ at0 )(y au ˆμ au0 ) for the set of all clusters a that have the same values of between-cluster explanatory variables as cluster i. This is also the sample estimate of the covariance for the multinomial distribution for the 2 2 joint distribution of (y at, y au ) for all such clusters. Now, consider X 2 (β 0 ) = i (ˆμ i ˆμ i0 ) T ˆV i0 1 (ˆμ i ˆμ i0 ). With categorical explanatory variables, X 2 (β 0 ) applies to two sets of fitted marginal proportions for the contingency table obtained by cross-classifying the multivariate binary response with the various combinations of explanatory variable values. The set of β 0 values for which X 2 (β 0 ) χ1,a 2 is a confidence interval for β. Unlike the generalized estimating equations approach, this pseudo-score method does not require using the sandwich estimator, which can be unreliable unless the number of clusters is large (Firth, 1993). Even with consistent estimation of V i0, however, the limiting null distribution of X 2 (β 0 ) need not be exactly chi-squared because the fitted values result from inefficient estimates, but preliminary simulations suggest that the chi-squared often provides a good approximation Confidence intervals based on power divergence statistics For testing goodness-of-fit of a multinomial model for a contingency table with counts {n i }, Cressie & Read (1984) presented a family of power divergence statistics that have the same asymptotic null chi-squared distribution as X 2 and G 2. The power divergence statistic for testing that a particular parameter takes value β 0 is P λ (β 0 ) = 2 ˆμi {(ˆμ i / ˆμ i0 ) λ 1}, λ(λ + 1) <λ<.

8 8 ALAN AGRESTI AND EUIJUNG RYU NEWCOMBE, R. (1998b). Interval estimation for the difference between independent proportions: comparison of eleven methods. Statist. Med. 17, R DEVELOPMENT CORE TEAM (2008). R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. ISBN , URL RAO, C. R. (1961). A study of large sample test criteria through properties of efficient estimates. Sankhya A23, SMYTH, G. K. (2003). Pearson s goodness of fit statistic as a score test statistic. In Science and Statistics: A Festschrift for Terry Speed, Ed. D. R. Goldstein, IMS Lecture Notes Monograph Series 40. Hayward, CA: Institute of Mathematical Statistics. [Received December Revised June 2009]

Multinomial Logistic Regression Models

Stat 544, Lecture 19 1 Multinomial Logistic Regression Models Polytomous responses. Logistic regression can be extended to handle responses that are polytomous, i.e. taking r>2 categories. (Note: The word

Modeling and inference for an ordinal effect size measure

STATISTICS IN MEDICINE Statist Med 2007; 00:1 15 Modeling and inference for an ordinal effect size measure Euijung Ryu, and Alan Agresti Department of Statistics, University of Florida, Gainesville, FL

Links Between Binary and Multi-Category Logit Item Response Models and Quasi-Symmetric Loglinear Models

Links Between Binary and Multi-Category Logit Item Response Models and Quasi-Symmetric Loglinear Models Alan Agresti Department of Statistics University of Florida Gainesville, Florida 32611-8545 July

8 Nominal and Ordinal Logistic Regression

8 Nominal and Ordinal Logistic Regression 8.1 Introduction If the response variable is categorical, with more then two categories, then there are two options for generalized linear models. One relies on

Generalized Linear Models for Non-Normal Data

Generalized Linear Models for Non-Normal Data Today s Class: 3 parts of a generalized model Models for binary outcomes Complications for generalized multivariate or multilevel models SPLH 861: Lecture

Generalized Linear Model under the Extended Negative Multinomial Model and Cancer Incidence

Generalized Linear Model under the Extended Negative Multinomial Model and Cancer Incidence Sunil Kumar Dhar Center for Applied Mathematics and Statistics, Department of Mathematical Sciences, New Jersey

Categorical Data Analysis Chapter 3

Categorical Data Analysis Chapter 3 The actual coverage probability is usually a bit higher than the nominal level. Confidence intervals for association parameteres Consider the odds ratio in the 2x2 table,

LOGISTICS REGRESSION FOR SAMPLE SURVEYS

4 LOGISTICS REGRESSION FOR SAMPLE SURVEYS Hukum Chandra Indian Agricultural Statistics Research Institute, New Delhi-002 4. INTRODUCTION Researchers use sample survey methodology to obtain information

LOGISTIC REGRESSION Joseph M. Hilbe

LOGISTIC REGRESSION Joseph M. Hilbe Arizona State University Logistic regression is the most common method used to model binary response data. When the response is binary, it typically takes the form of

Reports of the Institute of Biostatistics

Reports of the Institute of Biostatistics No 02 / 2008 Leibniz University of Hannover Natural Sciences Faculty Title: Properties of confidence intervals for the comparison of small binomial proportions

Confidence Intervals of the Simple Difference between the Proportions of a Primary Infection and a Secondary Infection, Given the Primary Infection

Biometrical Journal 42 (2000) 1, 59±69 Confidence Intervals of the Simple Difference between the Proportions of a Primary Infection and a Secondary Infection, Given the Primary Infection Kung-Jong Lui

PROD. TYPE: COM. Simple improved condence intervals for comparing matched proportions. Alan Agresti ; and Yongyi Min UNCORRECTED PROOF

pp: --2 (col.fig.: Nil) STATISTICS IN MEDICINE Statist. Med. 2004; 2:000 000 (DOI: 0.002/sim.8) PROD. TYPE: COM ED: Chandra PAGN: Vidya -- SCAN: Nil Simple improved condence intervals for comparing matched

Stat 5101 Lecture Notes

Stat 5101 Lecture Notes Charles J. Geyer Copyright 1998, 1999, 2000, 2001 by Charles J. Geyer May 7, 2001 ii Stat 5101 (Geyer) Course Notes Contents 1 Random Variables and Change of Variables 1 1.1 Random

Multivariate Extensions of McNemar s Test

Multivariate Extensions of McNemar s Test Bernhard Klingenberg Department of Mathematics and Statistics, Williams College Williamstown, MA 01267, U.S.A. e-mail: bklingen@williams.edu and Alan Agresti Department

ST3241 Categorical Data Analysis I Generalized Linear Models. Introduction and Some Examples

ST3241 Categorical Data Analysis I Generalized Linear Models Introduction and Some Examples 1 Introduction We have discussed methods for analyzing associations in two-way and three-way tables. Now we will

Testing Goodness Of Fit Of The Geometric Distribution: An Application To Human Fecundability Data

Journal of Modern Applied Statistical Methods Volume 4 Issue Article 8 --5 Testing Goodness Of Fit Of The Geometric Distribution: An Application To Human Fecundability Data Sudhir R. Paul University of

Model Estimation Example

Ronald H. Heck 1 EDEP 606: Multivariate Methods (S2013) April 7, 2013 Model Estimation Example As we have moved through the course this semester, we have encountered the concept of model estimation. Discussions

Repeated ordinal measurements: a generalised estimating equation approach

Repeated ordinal measurements: a generalised estimating equation approach David Clayton MRC Biostatistics Unit 5, Shaftesbury Road Cambridge CB2 2BW April 7, 1992 Abstract Cumulative logit and related

Testing Statistical Hypotheses

E.L. Lehmann Joseph P. Romano Testing Statistical Hypotheses Third Edition 4y Springer Preface vii I Small-Sample Theory 1 1 The General Decision Problem 3 1.1 Statistical Inference and Statistical Decisions

In Defence of Score Intervals for Proportions and their Differences

In Defence of Score Intervals for Proportions and their Differences Robert G. Newcombe a ; Markku M. Nurminen b a Department of Primary Care & Public Health, Cardiff University, Cardiff, United Kingdom

Investigating Models with Two or Three Categories

Ronald H. Heck and Lynn N. Tabata 1 Investigating Models with Two or Three Categories For the past few weeks we have been working with discriminant analysis. Let s now see what the same sort of model might

Multilevel Statistical Models: 3 rd edition, 2003 Contents

Multilevel Statistical Models: 3 rd edition, 2003 Contents Preface Acknowledgements Notation Two and three level models. A general classification notation and diagram Glossary Chapter 1 An introduction

A COMPARISON OF POISSON AND BINOMIAL EMPIRICAL LIKELIHOOD Mai Zhou and Hui Fang University of Kentucky

A COMPARISON OF POISSON AND BINOMIAL EMPIRICAL LIKELIHOOD Mai Zhou and Hui Fang University of Kentucky Empirical likelihood with right censored data were studied by Thomas and Grunkmier (1975), Li (1995),

Describing Contingency tables

Today s topics: Describing Contingency tables 1. Probability structure for contingency tables (distributions, sensitivity/specificity, sampling schemes). 2. Comparing two proportions (relative risk, odds

Negative Multinomial Model and Cancer. Incidence

Generalized Linear Model under the Extended Negative Multinomial Model and Cancer Incidence S. Lahiri & Sunil K. Dhar Department of Mathematical Sciences, CAMS New Jersey Institute of Technology, Newar,

Linear Regression Models P8111

Linear Regression Models P8111 Lecture 25 Jeff Goldsmith April 26, 2016 1 of 37 Today s Lecture Logistic regression / GLMs Model framework Interpretation Estimation 2 of 37 Linear regression Course started

LISA Short Course Series Generalized Linear Models (GLMs) & Categorical Data Analysis (CDA) in R. Liang (Sally) Shan Nov. 4, 2014

LISA Short Course Series Generalized Linear Models (GLMs) & Categorical Data Analysis (CDA) in R Liang (Sally) Shan Nov. 4, 2014 L Laboratory for Interdisciplinary Statistical Analysis LISA helps VT researchers

Testing Statistical Hypotheses

E.L. Lehmann Joseph P. Romano, 02LEu1 ttd ~Lt~S Testing Statistical Hypotheses Third Edition With 6 Illustrations ~Springer 2 The Probability Background 28 2.1 Probability and Measure 28 2.2 Integration.........

Testing for independence in J K contingency tables with complex sample. survey data

Biometrics 00, 1 23 DOI: 000 November 2014 Testing for independence in J K contingency tables with complex sample survey data Stuart R. Lipsitz 1,, Garrett M. Fitzmaurice 2, Debajyoti Sinha 3, Nathanael

Longitudinal Modeling with Logistic Regression

Newsom 1 Longitudinal Modeling with Logistic Regression Longitudinal designs involve repeated measurements of the same individuals over time There are two general classes of analyses that correspond to

Poisson Regression. Ryan Godwin. ECON University of Manitoba

Poisson Regression Ryan Godwin ECON 7010 - University of Manitoba Abstract. These lecture notes introduce Maximum Likelihood Estimation (MLE) of a Poisson regression model. 1 Motivating the Poisson Regression

Inverse Sampling for McNemar s Test

International Journal of Statistics and Probability; Vol. 6, No. 1; January 27 ISSN 1927-7032 E-ISSN 1927-7040 Published by Canadian Center of Science and Education Inverse Sampling for McNemar s Test

Chapter 1 Statistical Inference

Chapter 1 Statistical Inference causal inference To infer causality, you need a randomized experiment (or a huge observational study and lots of outside information). inference to populations Generalizations

Outline of GLMs. Definitions

Outline of GLMs Definitions This is a short outline of GLM details, adapted from the book Nonparametric Regression and Generalized Linear Models, by Green and Silverman. The responses Y i have density

Stat 642, Lecture notes for 04/12/05 96

Stat 642, Lecture notes for 04/12/05 96 Hosmer-Lemeshow Statistic The Hosmer-Lemeshow Statistic is another measure of lack of fit. Hosmer and Lemeshow recommend partitioning the observations into 10 equal

ij i j m ij n ij m ij n i j Suppose we denote the row variable by X and the column variable by Y ; We can then re-write the above expression as

page1 Loglinear Models Loglinear models are a way to describe association and interaction patterns among categorical variables. They are commonly used to model cell counts in contingency tables. These

" M A #M B. Standard deviation of the population (Greek lowercase letter sigma) σ 2

Notation and Equations for Final Exam Symbol Definition X The variable we measure in a scientific study n The size of the sample N The size of the population M The mean of the sample µ The mean of the

Generalized Linear Modeling - Logistic Regression

1 Generalized Linear Modeling - Logistic Regression Binary outcomes The logit and inverse logit interpreting coefficients and odds ratios Maximum likelihood estimation Problem of separation Evaluating

Generalized linear models

Generalized linear models Douglas Bates November 01, 2010 Contents 1 Definition 1 2 Links 2 3 Estimating parameters 5 4 Example 6 5 Model building 8 6 Conclusions 8 7 Summary 9 1 Generalized Linear Models

Generalized Linear Models (GLZ)

Generalized Linear Models (GLZ) Generalized Linear Models (GLZ) are an extension of the linear modeling process that allows models to be fit to data that follow probability distributions other than the

Hierarchical Generalized Linear Models. ERSH 8990 REMS Seminar on HLM Last Lecture!

Hierarchical Generalized Linear Models ERSH 8990 REMS Seminar on HLM Last Lecture! Hierarchical Generalized Linear Models Introduction to generalized models Models for binary outcomes Interpreting parameter

Bootstrap Procedures for Testing Homogeneity Hypotheses

Journal of Statistical Theory and Applications Volume 11, Number 2, 2012, pp. 183-195 ISSN 1538-7887 Bootstrap Procedures for Testing Homogeneity Hypotheses Bimal Sinha 1, Arvind Shah 2, Dihua Xu 1, Jianxin

Log-linear Models for Contingency Tables

Log-linear Models for Contingency Tables Statistics 149 Spring 2006 Copyright 2006 by Mark E. Irwin Log-linear Models for Two-way Contingency Tables Example: Business Administration Majors and Gender A

Basic Medical Statistics Course

Basic Medical Statistics Course S7 Logistic Regression November 2015 Wilma Heemsbergen w.heemsbergen@nki.nl Logistic Regression The concept of a relationship between the distribution of a dependent variable

For Bonnie and Jesse (again)

SECOND EDITION A P P L I E D R E G R E S S I O N A N A L Y S I S a n d G E N E R A L I Z E D L I N E A R M O D E L S For Bonnie and Jesse (again) SECOND EDITION A P P L I E D R E G R E S S I O N A N A

On Properties of QIC in Generalized. Estimating Equations. Shinpei Imori

On Properties of QIC in Generalized Estimating Equations Shinpei Imori Graduate School of Engineering Science, Osaka University 1-3 Machikaneyama-cho, Toyonaka, Osaka 560-8531, Japan E-mail: imori.stat@gmail.com

What s New in Econometrics. Lecture 13

What s New in Econometrics Lecture 13 Weak Instruments and Many Instruments Guido Imbens NBER Summer Institute, 2007 Outline 1. Introduction 2. Motivation 3. Weak Instruments 4. Many Weak) Instruments

Models for Binary Outcomes

Models for Binary Outcomes Introduction The simple or binary response (for example, success or failure) analysis models the relationship between a binary response variable and one or more explanatory variables.

arxiv: v2 [stat.me] 8 Jun 2016

Orthogonality of the Mean and Error Distribution in Generalized Linear Models 1 BY ALAN HUANG 2 and PAUL J. RATHOUZ 3 University of Technology Sydney and University of Wisconsin Madison 4th August, 2013

BIAS OF MAXIMUM-LIKELIHOOD ESTIMATES IN LOGISTIC AND COX REGRESSION MODELS: A COMPARATIVE SIMULATION STUDY

BIAS OF MAXIMUM-LIKELIHOOD ESTIMATES IN LOGISTIC AND COX REGRESSION MODELS: A COMPARATIVE SIMULATION STUDY Ingo Langner 1, Ralf Bender 2, Rebecca Lenz-Tönjes 1, Helmut Küchenhoff 2, Maria Blettner 2 1

GEE for Longitudinal Data - Chapter 8

GEE for Longitudinal Data - Chapter 8 GEE: generalized estimating equations (Liang & Zeger, 1986; Zeger & Liang, 1986) extension of GLM to longitudinal data analysis using quasi-likelihood estimation method

Latent Variable Models for Binary Data. Suppose that for a given vector of explanatory variables x, the latent

Latent Variable Models for Binary Data Suppose that for a given vector of explanatory variables x, the latent variable, U, has a continuous cumulative distribution function F (u; x) and that the binary

Ron Heck, Fall Week 8: Introducing Generalized Linear Models: Logistic Regression 1 (Replaces prior revision dated October 20, 2011)

Ron Heck, Fall 2011 1 EDEP 768E: Seminar in Multilevel Modeling rev. January 3, 2012 (see footnote) Week 8: Introducing Generalized Linear Models: Logistic Regression 1 (Replaces prior revision dated October

11. Generalized Linear Models: An Introduction

Sociology 740 John Fox Lecture Notes 11. Generalized Linear Models: An Introduction Copyright 2014 by John Fox Generalized Linear Models: An Introduction 1 1. Introduction I A synthesis due to Nelder and

Discussion of the paper Inference for Semiparametric Models: Some Questions and an Answer by Bickel and Kwon

Discussion of the paper Inference for Semiparametric Models: Some Questions and an Answer by Bickel and Kwon Jianqing Fan Department of Statistics Chinese University of Hong Kong AND Department of Statistics

12 Generalized linear models

12 Generalized linear models In this chapter, we combine regression models with other parametric probability models like the binomial and Poisson distributions. Binary responses In many situations, we

Correlation and regression

1 Correlation and regression Yongjua Laosiritaworn Introductory on Field Epidemiology 6 July 2015, Thailand Data 2 Illustrative data (Doll, 1955) 3 Scatter plot 4 Doll, 1955 5 6 Correlation coefficient,

Regression models for multivariate ordered responses via the Plackett distribution

Journal of Multivariate Analysis 99 (2008) 2472 2478 www.elsevier.com/locate/jmva Regression models for multivariate ordered responses via the Plackett distribution A. Forcina a,, V. Dardanoni b a Dipartimento

Introduction to Generalized Models

Introduction to Generalized Models Today s topics: The big picture of generalized models Review of maximum likelihood estimation Models for binary outcomes Models for proportion outcomes Models for categorical

Theory and Methods of Statistical Inference. PART I Frequentist likelihood methods

PhD School in Statistics XXV cycle, 2010 Theory and Methods of Statistical Inference PART I Frequentist likelihood methods (A. Salvan, N. Sartori, L. Pace) Syllabus Some prerequisites: Empirical distribution

Theory and Methods of Statistical Inference. PART I Frequentist theory and methods

PhD School in Statistics cycle XXVI, 2011 Theory and Methods of Statistical Inference PART I Frequentist theory and methods (A. Salvan, N. Sartori, L. Pace) Syllabus Some prerequisites: Empirical distribution

TOLERANCE INTERVALS FOR DISCRETE DISTRIBUTIONS IN EXPONENTIAL FAMILIES

Statistica Sinica 19 (2009), 905-923 TOLERANCE INTERVALS FOR DISCRETE DISTRIBUTIONS IN EXPONENTIAL FAMILIES Tianwen Tony Cai and Hsiuying Wang University of Pennsylvania and National Chiao Tung University

CHAPTER 17 CHI-SQUARE AND OTHER NONPARAMETRIC TESTS FROM: PAGANO, R. R. (2007)

FROM: PAGANO, R. R. (007) I. INTRODUCTION: DISTINCTION BETWEEN PARAMETRIC AND NON-PARAMETRIC TESTS Statistical inference tests are often classified as to whether they are parametric or nonparametric Parameter

Modern Likelihood-Frequentist Inference. Donald A Pierce, OHSU and Ruggero Bellio, Univ of Udine

Modern Likelihood-Frequentist Inference Donald A Pierce, OHSU and Ruggero Bellio, Univ of Udine Shortly before 1980, important developments in frequency theory of inference were in the air. Strictly, this

Unit 9: Inferences for Proportions and Count Data

Unit 9: Inferences for Proportions and Count Data Statistics 571: Statistical Methods Ramón V. León 12/15/2008 Unit 9 - Stat 571 - Ramón V. León 1 Large Sample Confidence Interval for Proportion ( pˆ p)

A MODIFICATION OF THE HARTUNG KNAPP CONFIDENCE INTERVAL ON THE VARIANCE COMPONENT IN TWO VARIANCE COMPONENT MODELS

K Y B E R N E T I K A V O L U M E 4 3 ( 2 0 0 7, N U M B E R 4, P A G E S 4 7 1 4 8 0 A MODIFICATION OF THE HARTUNG KNAPP CONFIDENCE INTERVAL ON THE VARIANCE COMPONENT IN TWO VARIANCE COMPONENT MODELS

Model comparison and selection

BS2 Statistical Inference, Lectures 9 and 10, Hilary Term 2008 March 2, 2008 Hypothesis testing Consider two alternative models M 1 = {f (x; θ), θ Θ 1 } and M 2 = {f (x; θ), θ Θ 2 } for a sample (X = x)

STAT 705: Analysis of Contingency Tables

STAT 705: Analysis of Contingency Tables Timothy Hanson Department of Statistics, University of South Carolina Stat 705: Analysis of Contingency Tables 1 / 45 Outline of Part I: models and parameters Basic

Glossary for the Triola Statistics Series

Glossary for the Triola Statistics Series Absolute deviation The measure of variation equal to the sum of the deviations of each value from the mean, divided by the number of values Acceptance sampling

Maximum Likelihood (ML) Estimation

Econometrics 2 Fall 2004 Maximum Likelihood (ML) Estimation Heino Bohn Nielsen 1of32 Outline of the Lecture (1) Introduction. (2) ML estimation defined. (3) ExampleI:Binomialtrials. (4) Example II: Linear

Theory and Methods of Statistical Inference

PhD School in Statistics cycle XXIX, 2014 Theory and Methods of Statistical Inference Instructors: B. Liseo, L. Pace, A. Salvan (course coordinator), N. Sartori, A. Tancredi, L. Ventura Syllabus Some prerequisites:

Comparing IRT with Other Models

Comparing IRT with Other Models Lecture #14 ICPSR Item Response Theory Workshop Lecture #14: 1of 45 Lecture Overview The final set of slides will describe a parallel between IRT and another commonly used

The purpose of this section is to derive the asymptotic distribution of the Pearson chi-square statistic. k (n j np j ) 2. np j.

Chapter 9 Pearson s chi-square test 9. Null hypothesis asymptotics Let X, X 2, be independent from a multinomial(, p) distribution, where p is a k-vector with nonnegative entries that sum to one. That

FULL LIKELIHOOD INFERENCES IN THE COX MODEL

October 20, 2007 FULL LIKELIHOOD INFERENCES IN THE COX MODEL BY JIAN-JIAN REN 1 AND MAI ZHOU 2 University of Central Florida and University of Kentucky Abstract We use the empirical likelihood approach

Chapter 19: Logistic regression

Chapter 19: Logistic regression Self-test answers SELF-TEST Rerun this analysis using a stepwise method (Forward: LR) entry method of analysis. The main analysis To open the main Logistic Regression dialog

Three-Way Contingency Tables

Newsom PSY 50/60 Categorical Data Analysis, Fall 06 Three-Way Contingency Tables Three-way contingency tables involve three binary or categorical variables. I will stick mostly to the binary case to keep

CHAPTER 1: BINARY LOGIT MODEL

CHAPTER 1: BINARY LOGIT MODEL Prof. Alan Wan 1 / 44 Table of contents 1. Introduction 1.1 Dichotomous dependent variables 1.2 Problems with OLS 3.3.1 SAS codes and basic outputs 3.3.2 Wald test for individual

The Logit Model: Estimation, Testing and Interpretation

The Logit Model: Estimation, Testing and Interpretation Herman J. Bierens October 25, 2008 1 Introduction to maximum likelihood estimation 1.1 The likelihood function Consider a random sample Y 1,...,

STAT5044: Regression and Anova

STAT5044: Regression and Anova Inyoung Kim 1 / 18 Outline 1 Logistic regression for Binary data 2 Poisson regression for Count data 2 / 18 GLM Let Y denote a binary response variable. Each observation

Bayesian Linear Models

Bayesian Linear Models Sudipto Banerjee 1 and Andrew O. Finley 2 1 Department of Forestry & Department of Geography, Michigan State University, Lansing Michigan, U.S.A. 2 Biostatistics, School of Public

A Bootstrap Test for Causality with Endogenous Lag Length Choice. - theory and application in finance

CESIS Electronic Working Paper Series Paper No. 223 A Bootstrap Test for Causality with Endogenous Lag Length Choice - theory and application in finance R. Scott Hacker and Abdulnasser Hatemi-J April 200

Estimated Precision for Predictions from Generalized Linear Models in Sociological Research

Quality & Quantity 34: 137 152, 2000. 2000 Kluwer Academic Publishers. Printed in the Netherlands. 137 Estimated Precision for Predictions from Generalized Linear Models in Sociological Research TIM FUTING

From Practical Data Analysis with JMP, Second Edition. Full book available for purchase here. Contents About This Book... xiii About The Author... xxiii Chapter 1 Getting Started: Data Analysis with JMP...

Using Estimating Equations for Spatially Correlated A

Using Estimating Equations for Spatially Correlated Areal Data December 8, 2009 Introduction GEEs Spatial Estimating Equations Implementation Simulation Conclusion Typical Problem Assess the relationship

UNIVERSITÄT POTSDAM Institut für Mathematik

UNIVERSITÄT POTSDAM Institut für Mathematik Testing the Acceleration Function in Life Time Models Hannelore Liero Matthias Liero Mathematische Statistik und Wahrscheinlichkeitstheorie Universität Potsdam

Mixed Models for Longitudinal Ordinal and Nominal Outcomes

Mixed Models for Longitudinal Ordinal and Nominal Outcomes Don Hedeker Department of Public Health Sciences Biological Sciences Division University of Chicago hedeker@uchicago.edu Hedeker, D. (2008). Multilevel

9 Generalized Linear Models

9 Generalized Linear Models The Generalized Linear Model (GLM) is a model which has been built to include a wide range of different models you already know, e.g. ANOVA and multiple linear regression models

Solutions for Examination Categorical Data Analysis, March 21, 2013

STOCKHOLMS UNIVERSITET MATEMATISKA INSTITUTIONEN Avd. Matematisk statistik, Frank Miller MT 5006 LÖSNINGAR 21 mars 2013 Solutions for Examination Categorical Data Analysis, March 21, 2013 Problem 1 a.

Maximum Likelihood, Logistic Regression, and Stochastic Gradient Training

Maximum Likelihood, Logistic Regression, and Stochastic Gradient Training Charles Elkan elkan@cs.ucsd.edu January 17, 2013 1 Principle of maximum likelihood Consider a family of probability distributions

Bayesian Linear Models

Bayesian Linear Models Sudipto Banerjee 1 and Andrew O. Finley 2 1 Biostatistics, School of Public Health, University of Minnesota, Minneapolis, Minnesota, U.S.A. 2 Department of Forestry & Department

EDF 7405 Advanced Quantitative Methods in Educational Research. Data are available on IQ of the child and seven potential predictors.

EDF 7405 Advanced Quantitative Methods in Educational Research Data are available on IQ of the child and seven potential predictors. Four are medical variables available at the birth of the child: Birthweight

Variable Selection for Generalized Additive Mixed Models by Likelihood-based Boosting

Variable Selection for Generalized Additive Mixed Models by Likelihood-based Boosting Andreas Groll 1 and Gerhard Tutz 2 1 Department of Statistics, University of Munich, Akademiestrasse 1, D-80799, Munich,

arxiv: v2 [math.st] 13 Apr 2012

The Asymptotic Covariance Matrix of the Odds Ratio Parameter Estimator in Semiparametric Log-bilinear Odds Ratio Models Angelika Franke, Gerhard Osius Faculty 3 / Mathematics / Computer Science University

Hypothesis testing:power, test statistic CMS:

Hypothesis testing:power, test statistic The more sensitive the test, the better it can discriminate between the null and the alternative hypothesis, quantitatively, maximal power In order to achieve this

Optimal rejection regions for testing multiple binary endpoints in small samples

Optimal rejection regions for testing multiple binary endpoints in small samples Robin Ristl and Martin Posch Section for Medical Statistics, Center of Medical Statistics, Informatics and Intelligent Systems,

Statistics 3858 : Contingency Tables

Statistics 3858 : Contingency Tables 1 Introduction Before proceeding with this topic the student should review generalized likelihood ratios ΛX) for multinomial distributions, its relation to Pearson

Machine Learning Linear Classification. Prof. Matteo Matteucci

Machine Learning Linear Classification Prof. Matteo Matteucci Recall from the first lecture 2 X R p Regression Y R Continuous Output X R p Y {Ω 0, Ω 1,, Ω K } Classification Discrete Output X R p Y (X)

Penalized likelihood logistic regression with rare events

Penalized likelihood logistic regression with rare events Georg Heinze 1, Angelika Geroldinger 1, Rainer Puhr 2, Mariana Nold 3, Lara Lusa 4 1 Medical University of Vienna, CeMSIIS, Section for Clinical