Regression Analysis by Example

Transcription

1 Regression Analysis by Example Fourth Edition SAMPRIT CHATTEFUEE Department of Health Policy Mount Sinai School of Medicine New York, NY ALI S. HAD1 Department of Mathematics The American University in Cairo Cairo, Egypt WILEY- INTERSCl ENCE A JOHN WILEY & SONS, INC., PUBLICATION

2 This Page Intentionally Left Blank

3 Regression Analysis by Example

4 WILEY SERIES IN PROBABILITY AND STATISTICS Established by WALTER A. SHEWHART and SAMUEL S. WILKS Editors: David J. Balding, Noel A. C. Cressie, Nicholas I. Fisher, Iain M. Johnstone, J. B. Kadane, Geert Molenberghs, Louise M. Ryan, David I$? Scott, Adrian F. M, Smith, Jozef L. Teugels Editors Emeriti: Vic Barnett, J. Stuart Hunter, David G. Kendall A complete list of the titles in this series appears at the end of this volume.

5 Regression Analysis by Example Fourth Edition SAMPRIT CHATTEFUEE Department of Health Policy Mount Sinai School of Medicine New York, NY ALI S. HAD1 Department of Mathematics The American University in Cairo Cairo, Egypt WILEY- INTERSCl ENCE A JOHN WILEY & SONS, INC., PUBLICATION

6 Copyright 26 by John Wiley & Sons, Inc. All rights reserved. Published by John Wiley & Sons, Inc., Hoboken, New Jersey. Published simultaneously in Canada. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 17 or 18 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 1923, (978) 75-84, fax (978) , or on the web at Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 11 1 River Street, Hoboken, NJ 73, (21) I, fax (21) , or online at Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (8) , outside the United States at (3 17) or fax (3 17) Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic format. For information about Wiley products, visit our web site at Library of Congress Cataloging-in-Publieation Data: Chatterjee, Samprit, Regression analysis by example. - 4th ed. / Samprit Chatterjee, Ah S. Hadi. p. cm. Includes bibliographical references and index. ISBN-I (cloth : acid-free paper) ISBN-I (cloth : acid-free paper) 1. Regression analysis. 1. Title. QA278.2.C dc Printed in the United States of America

7 Dedicated to: Allegra, Martha, and Rima - S. C. My mother and the memory of my father - A. S. H. It s a gift to be simple... Old Shaker hymn True knowledge is knowledge of why things are as they are, and not merely what they are. Isaiah Berlin

8 This Page Intentionally Left Blank

9 CONTENTS Preface 1 Introduction What Is Regression Analysis? Publicly Available Data Sets Selected Applications of Regression Analysis Agricultural Sciences Industrial and Labor Relations History Government Environmental Sciences Steps in Regression Analysis Statement of the Problem Selection of Potentially Relevant Variables Data Collection Model Specification Method of Fitting Model Fitting Model Criticism and Selection Objectives of Regression Analysis Scope and Organization of the Book Exercises xiii vii

10 Viii CONTENTS 2 Simple Linear Regression Introduction Covariance and Correlation Coefficient Example: Computer Repair Data The Simple Linear Regression Model Parameter Estimation Tests of Hypotheses Confidence Intervals Predictions Measuring the Quality of Fit Regression Line Through the Origin Trivial Regression Models Bibliographic Notes Exercises 3 Multiple Linear Regression Introduction Description of the Data and Model Example: Supervisor Performance Data Parameter Estimation Interpretations of Regression Coefficients Properties of the Least Squares Estimators Multiple Correlation Coefficient Inference for Individual Regression Coefficients Tests of Hypotheses in a Linear Model Testing All Regression Coefficients Equal to Zero Testing a Subset of Regression Coefficients Equal to Zero Testing the Equality of Regression Coefficients Estimating and Testing of Regression Parameters Under Constraints Predictions Summary Exercises Appendix: Multiple Regression in Matrix Notation 4 Regression Diagnostics: Detection of Model Violations 4.1 Introduction 4.2 The Standard Regression Assumptions 4.3 Various Types of Residuals 4.4 Graphical Methods 4.5 Graphs Before Fitting a Model

11 CONTENTS ix One-Dimensional Graphs Two-Dimensional Graphs Rotating Plots Dynamic Graphs Graphs After Fitting a Model Checking Linearity and Normality Assumptions Leverage, Influence, and Outliers Outliers in the Response Variable Outliers in the Predictors Masking and Swamping Problems Measures of Influence Cook s Distance Welsch and Kuh Measure Hadi s Influence Measure The Potential-Residual Plot What to Do with the Outliers? Role of Variables in a Regression Equation Added-Variable Plot Residual Plus Component Plot Effects of an Additional Predictor Robust Regression Exercises Qualitative Variables as Predictors Introduction Salary Survey Data Interaction Variables Systems of Regression Equations Models with Different Slopes and Different Intercepts Models with Same Slope and Different Intercepts Models with Same Intercept and Different Slopes Other Applications of Indicator Variables Seasonality Stability of Regression Parameters Over Time 141 Exercises 143 Transformation of Variables Introduction Transformations to Achieve Linearity Bacteria Deaths Due to X-Ray Radiation Inadequacy of a Linear Model Logarithmic Transformation for Achieving Linearity 158

12 X CONTENTS Transformations to Stabilize Variance 6.5 Detection of Heteroscedastic Errors 6.6 Removal of Heteroscedasticity 6.7 Weighted Least Squares 6.8 Logarithmic Transformation of Data 6.9 Power Transformation 6.1 Summary Exercises Weighted Least Squares 7.1 Introduction 7.2 Heteroscedastic Models Supervisors Data College Expense Data 7.3 Two-Stage Estimation 7.4 Education Expenditure Data 7.5 Fitting a Dose-Response Relationship Curve Exercises The Problem of Correlated Errors Introduction: Autocorrelation Consumer Expenditure and Money Stock Durbin-Watson Statistic Removal of Autocorrelation by Transformation Iterative Estimation With Autocorrelated Errors Autocorrelation and Missing Variables Analysis of Housing Starts Limitations of Durbin-Watson Statistic Indicator Variables to Remove Seasonality Regressing Two Time Series Exercises Analysis of Collinear Data 9.1 Introduction 9.2 Effects on Inference 9.3 Effects on Forecasting 9.4 Detection of Multicollinearity 9.5 Centering and Scaling Centering and Scaling in Intercept Models Scaling in No-Intercept Models 9.6 Principal Components Approach 9.7 Imposing Constraints I

13 CONTENTS xi 9.8 Searching for Linear Functions of the P s 9.9 Computations Using Principal Components 9.1 Bibliographic Notes Exercises Appendix: Principal Components 1 Biased Estimation of Regression Coefficients Introduction Principal Components Regression Removing Dependence Among the Predictors Constraints on the Regression Coefficients Principal Components Regression: A Caution Ridge Regression Estimation by the Ridge Method Ridge Regression: Some Remarks Summary Exercises Appendix: Ridge Regression 11 Variable Selection Procedures 11.1 Introduction 11.2 Formulation of the Problem 11.3 Consequences of Variables Deletion 11.4 Uses of Regression Equations Description and Model Building Estimation and Prediction Control 11.5 Criteria for Evaluating Equations Residual Mean Square Mallows C, 1 I S.3 Information Criteria: Akaike and Other Modified Forms 11.6 Multicollinearity and Variable Selection 11.7 Evaluating All Possible Equations 11.8 Variable Selection Procedures Forward Selection Procedure Backward Elimination Procedure Stepwise Method 11.9 General Remarks on Variable Selection Methods 11.1 A Study of Supervisor Performance Variable Selection With Collinear Data The Homicide Data

14 Xii CONTENTS Variable Selection Using Ridge Regression Selection of Variables in an Air Pollution Study A Possible Strategy for Fitting Regression Models Bibliographic Notes Exercises Appendix: Effects of Incorrect Model Specifications 12 Logistic Regression Introduction Modeling Qualitative Data The Logit Model Example: Estimating Probability of Bankruptcies Logistic Regression Diagnostics Determination of Variables to Retain Judging the Fit of a Logistic Regression The Multinomial Logit Model Multinomial Logistic Regression Example: Determining Chemical Diabetes Ordered Response Category: Ordinal Logistic Regression Example: Determining Chemical Diabetes Revisited Classification Problem: Another Approach Exercises 13 Further Topics 13.1 Introduction 13.2 Generalized Linear Model 13.3 Poisson Regression Model 13.4 Introduction of New Drugs 13.5 Robust Regression 13.6 Fitting a Quadratic Model 13.7 Distribution of PCB in U.S. Bays Exercises Appendix A: Statistical Tables References index 37 1

15 PREFACE Regression analysis has become one of the most widely used statistical tools for analyzing multifactor data. It is appealing because it provides a conceptually simple method for investigating functional relationships among variables. The standard approach in regression analysis is to take data, fit a model, and then evaluate the fit using statistics such as t, F,and R2. Our approach is much broader. We view regression analysis as a set of data analytic techniques that examine the interrelationships among a given set of variables. The emphasis is not on formal statistical tests and probability calculations. We argue for an informal analysis directed towards uncovering patterns in the data. We utilize most standard and some not so standard summary statistics on the basis of their intuitive appeal. We rely heavily on graphical representations of the data, and employ many variations of plots of regression residuals. We are not overly concerned with precise probability evaluations. Graphical methods for exploring residuals can suggest model deficiencies or point to troublesome observations. Upon further investigation into their origin, the troublesome observations often turn out to be more informative than the well-behaved observations. We notice often that more information is obtained from a quick examination of a plot of residuals than from a formal test of statistical significance of some limited nullhypothesis. In short, the presentation in the chapters of this book is guided by the principles and concepts of exploratory data analysis. Our presentation of the various concepts and techniques of regression analysis relies on carefully developed examples. In each example, we have isolated one xiii

16 xiv PREFACE or two techniques and discussed them in some detail. The data were chosen to highlight the techniques being presented. Although when analyzing a given set of data it is usually necessary to employ many techniques, we have tried to choose the various data sets so that it would not be necessary to discuss the same technique more than once. Our hope is that after working through the book, the reader will be ready and able to analyze hisker data methodically, thoroughly, and confidently. The emphasis in this book is on the analysis of data rather than on formulas, tests of hypotheses, or confidence intervals. Therefore no attempt has been made to derive the techniques. Techniques are described, the required assumptions are given, and finally, the success of the technique in the particular example is assessed. Although derivations of the techniques are not included, we have tried to refer the reader in each case to sources in which such discussion is available. Our hope is that some of these sources will be followed up by the reader who wants a more thorough grounding in theory. We have taken for granted the availability of a computer and a statistical package. Recently there has been a qualitative change in the analysis of linear models, from model fitting to model building, from overall tests to clinical examinations of data, from macroscopic to the microscopic analysis. To do this kind of analysis a computer is essential and we have assumed its availability. Almost all of the analyses we use are now available in software packages. We are particularly heartened by the arrival of the package R, available on the Internet under the General Public License (GPL). The package has excellent computing and graphical features. It is also free! The material presented is intended for anyone who is involved in analyzing data. The book should be helpful to those who have some knowledge of the basic concepts of statistics. In the university, it could be used as a text for a course on regression analysis for students whose specialization is not statistics, but, who nevertheless, use regression analysis quite extensively in their work. For students whose major emphasis is statistics, and who take a course on regression analysis from a book at the level of Rao (1973), Seber (1977), or Sen and Srivastava (199), this book can be used to balance and complement the theoretical aspects of the subject with practical applications. Outside the university, this book can be profitably used by those people whose present approach to analyzing multifactor data consists of looking at standard computer output (t, F, R2, standard errors, etc.), but who want to go beyond these summaries for a more thorough analysis. The book has a Web site: This Web site contains, among other things, all the data sets that are included in this book and more. Several new topics have been introduced in this edition. The discussion in Section 2.1 about the regression line through the origin has been considerably expanded. In the chapter on variable selection (Chapter 1 l), we introduce information measures and illustrate their use. The information criteria help in variable selection by

17 PREFACE XV balancing the conflicting requirements of accuracy and complexity. It is a useful tool for arriving at parsimonious models. The chapter on logistic regression (Chapter 12) has been considerably expanded. This reflects the increased use of the logit models in statistical analysis. In addition to binary logistic regression, we have now included a discussion of multinomial logistic regression. This extends the application of logistic regression to more diverse situations. The categories in some multinomial are ordered, for example in attitude surveys. We also discuss the application of the logistic model to ordered response variable. A new chapter titled Further Topics (Chapter 13) has been added to this edition. This chapter is intended to be an introduction to a more advanced study of regression analysis. The topics discussed are generalized linear models (GLM) and robust regression. We introduce the concept of GLM and discuss how the linear regression and logistic regression models can be regarded as special cases from a large family of linear models. This provides a unifying view of linear models. We discuss Poisson regression in the context of GLM, and its use for modeling count data. We have attempted to write a book for a group of readers with diverse backgrounds. We have also tried to put emphasis on the art of data analysis rather than on the development of statistical theory. We are fortunate to have had assistance and encouragement from several friends, colleagues, and associates. Some of our colleagues at New York University and Cornell University have used portions of the material in their courses and have shared with us their comments and comments of their students. Special thanks are due to our friend and former colleague Jeffrey Simonoff (New York University) for comments, suggestions, and general help. The students in our classes on regression analysis have all contributed by asking penetrating questions and demanding meaningful and understandable answers. Our special thanks go to Nedret Billor (Cukurova University, Turkey) and Sahar El-Sheneity (Cornell University) for their very careful reading of an earlier edition of this book. We also thank Amy Hendrickson for preparing the Latex style files and for responding to our Latex questions, and Dean Gonzalez for help with the production of some of the figures. Brooksville, Maine Cairo, Egypt SAMPRIT CHATTERJEE ALI S. HADI

18 This Page Intentionally Left Blank

19 CHAPTER 1 I NTROD UCTION 1.I WHAT IS REGRESSION ANALYSIS? Regression analysis is a conceptually simple method for investigating functional relationships among variables. A real estate appraiser may wish to relate the sale price of a home from selected physical characteristics of the building and taxes (local, school, county) paid on the building. We may wish to examine whether cigarette consumption is related to various socioeconomic and demographic variables such as age, education, income, and price of cigarettes. The relationship is expressed in the form of an equation or a model connecting the response or dependent variable and one or more explanatory or predictor variables. In the cigarette consumption example, the response variable is cigarette consumption (measured by the number of packs of cigarette sold in a given state on a per capita basis during a given year) and the explanatory or predictor variables are the various socioeconomic and demographic variables. In the real estate appraisal example, the response variable is the price of a home and the explanatory or predictor variables are the characteristics of the building and taxes paid on the building. We denote the response variable by Y and the set of predictor variables by XI, Xp,..., X,, where p denotes the number of predictor variables. The true relationship between Y and XI, Xp,..., X, can be approximated by the regression Regression Analysis by Example, Fourth Edition. By Samprit Chatterjee and Ali S. Hadi 26 John Wiley & Sons, Inc. 1

20 2 INTRODUCTION model y = f(xl1 x21..., X,) + 1 where E is assumed to be a random error representing the discrepancy in the approximation. It accounts for the failure of the model to fit the data exactly. The function f(x1, X2,..., X,) describes the relationship between Y and XI, X2,..., X,. An example is the linear regression model Y = Po + p1x1 + p2x ppx, + El (1.a where pol,&...,,lip, called the regression parameters or coefficients, are unknown constants to be determined (estimated) from the data. We follow the commonly used notational convention of denoting unknown parameters by Greek letters. The predictor or explanatory variables are also called by other names such as independent variables, covariates, regressors, factors, and carriers. The name independent variable, though commonly used, is the least preferred, because in practice the predictor variables are rarely independent of each other. 1.2 PUBLICLY AVAILABLE DATA SETS Regression analysis has numerous areas of applications. A partial list would include economics, finance, business, law, meteorology, medicine, biology, chemistry, engineering, physics, education, sports, history, sociology, and psychology. A few examples of such applications are given in Section 1.3. Regression analysis is learned most effectively by analyzing data that are of direct interest to the reader. We invite the readers to think about questions (in their own areas of work, research, or interest) that can be addressed using regression analysis. Readers should collect the relevant data and then apply the regression analysis techniques presented in this book to their own data. To help the reader locate real-life data, this section provides some sources and links to a wealth of data sets that are available for public use. A number of datasets are available in books and on the Internet. The book by Hand et al. (1994) contains data sets from many fields. These data sets are small in size and are suitable for use as exercises. The book by Chatterjee, Handcock, and Simonoff (1995) provides numerous data sets from diverse fields. The data are included in a diskette that comes with the book and can also be found in the World Wide Web site.' Data sets are also available on the Internet at many other sites. Some of the Web sites given below allow the direct copying and pasting into the statistical package of choice, while others require downloading the data file and then importing them into a statistical package. Some of these sites also contain further links to yet other data sets or statistics-related Web sites. The Data and Story Library (DASL, pronounced "dazzle") is one of the most interesting sites that contains a number of data sets accompanied by the "story" or ' jsirnonowcasebook

21 SELECTED APPLICATIONS OF REGRESSION ANALYSIS 3 background associated with each data set. DASL is an online library2 of data files and stories that illustrate the use of basic statistical methods. The data sets cover a wide variety of topics. DASL comes with a powerful search engine to locate the story or data file of interest. Another Web site, which also contains data sets arranged by the method used in the analysis, is the Electronic Dataset Ser~ice.~ The site also contains many links to other data sources on the Internet. Finally, this book has a Web site: This site contains, among other things, all the data sets that are included in this book and more. These and other data sets can be found in the book's Web site. 1.3 SELECTED APPLICATIONS OF REGRESSION ANALYSIS Regression analysis is one of the most widely used statistical tools because it provides simple methods for establishing a functional relationship among variables. It has extensive applications in many subject areas. The cigarette consumption and the real estate appraisal, mentioned above, are but two examples. In this section, we give a few additional examples demonstrating the wide applicability of regression analysis in real-life situations. Some of the data sets described here will be used later in the book to illustrate regression techniques or in the exercises at the end of various chapters Agricultural Sciences The Dairy Herd Improvement Cooperative (DHI) in Upstate New York collects and analyzes data on milk production. One question of interest here is how to develop a suitable model to predict current milk production from a set of measured variables. The response variable (current milk production in pounds) and the predictor variables are given in Table 1.1. Samples are taken once a month during milking. The period that a cow gives milk is called lactation. Number of lactations is the number of times a cow has calved or given milk. The recommended management practice is to have the cow produce milk for about 35 days and then allow a 6- day rest period before beginning the next lactation. The data set, consisting of 199 observations, was compiled from the DHI milk production records. The Milk Production data can be found in the book's Web site Industrial and Labor Relations In 1947, the United States Congress passed the Taft-Hartley Amendments to the Wagner Act. The original Wagner Act had permitted the unions to use a Closed *DASL'S Web site is: 3http://www-unix.oit.umass.edurstatdata/

22 4 INTRODUCTION Variable Current Previous Fat Protein Days Lactation I79 Table 1.1 Variables for the Milk Production Data Definition Current month milk production in pounds Previous month milk production in pounds Percent of fat in milk Percent of protein in milk Number of days since present lactation Number of lactations Indicator variable ( if Days 5 79 and 1 if Days > 79) Table 1.2 Variables for the Right-To-Work Laws Data Variable Definition COL Cost of living for a four-person family PD Population density (person per square mile) URate State unionization rate in 1978 POP Population in 1975 Taxes Property taxes in 1972 Income RTWL Per capita income in 1974 Indicator variable (1 if there is right-to-work laws in the state and otherwise) Shop Contract4 unless prohibited by state law. The Taft-Hartley Amendments made the use of Closed Shop Contract illegal and gave individual states the right to prohibit union shops5 as well. These right-to-work laws have caused a wave of concern throughout the labor movement. A question of interest here is: What are the effects of these laws on the cost of living for a four-person family living on an intermediate budget in the United States? To answer this question a data set consisting of 38 geographic locations has been assembled from various sources. The variables used are defined in Table 1.2. The Right-To-Work Laws data are given in Table 1.3 and can also be found in the book's Web site History A question of historical interest is how to estimate the age of historical objects based on some age-related characteristics of the objects. For example, the variables 4Under a Closed Shop Contract provision, all employees must be union members at the time of hire and must remain members as a condition of employment. 'Under a Union Shop clause, employees are not required to be union members at the time of hire, but must become a member within two months, thus allowing the employer complete discretion in hiring decisions.

23 SELECTED APPLICATIONS OF REGRESSION ANALYSIS 5 Table 1.3 The Right-To-Work Laws Data City COL PD URate Pop Taxes Income RTWL Atlanta Austin Bakersfield Baltimore Baton Rouge Boston Buffalo Champaign-Urbana Cedar Rapids Chicago Cincinnati Cleveland Dallas Dayton Denver Detriot Green Bay Hartford Houston Indianapolis Kansas City Lancaster, PA Los Angeles Mi 1 waukee Minneapolis, St. Paul Nashville New York Orlando Philadelphia Pittsburgh Portland St. Louis San Diego San Francisco Seattle Washington Wichita Raleigh-Durham

24 6 INTRODUCTION Table 1.4 Variable Year MB BH BL NH Variables for the Egyptian Skulls Data Definition Approximate Year of Skull Formation (negative = B.C.; positive = A.D.) Maximum Breadth of Skull Basibregmatic Height of Skull Basialveolar Length of Skull Nasal Height of Skull in Table 1.4 can be used to estimate the age of Egyptian skulls. Here the response variable is Year and the other four variables are possible predictors. The original source of the data is Thomson and Randall-Maciver (193, but they can be found in Hand et al. (1994), pp An analysis of the data can be found in Manly (1 986). The Egyptian Skulls data can be found in the book s Web site Government Information about domestic immigration (the movement of people from one state or area of a country to another) is important to state and local governments. It is of interest to build a model that predicts domestic immigration or to answer the question of why do people leave one place to go to another? There are many factors that influence domestic immigration, such as weather conditions, crime, tax, and unemployment rates. A data set for the 48 contiguous states has been created. Alaska and Hawaii are excluded from the analysis because the environments of these states are significantly different from the other 48, and their locations present certain barriers to immigration. The response variable here is net domestic immigration, which represents the net movement of people into and out of a state over the period divided by the population of the state. Eleven predictor variables thought to influence domestic immigration are defined in Table 1.5. The data are given in Tables 1.6 and 1.7, and can also be found in the book s Web site Environmental Sciences In a 1976 study exploring the relationship between water quality and land use, Haith (1 976) obtained the measurements (shown in Table 1.8) on 2 river basins in New York State. A question of interest here is how the land use around a river basin contributes to the water pollution as measured by the mean nitrogen concentration (mghter). The data are shown in Table 1.9 and can also be found in the book s Web site.

25 STEPS IN REGRESSION ANALYSIS 7 Table 1.5 Variables for the Study of Domestic Immigration Variable State NDIR Unemp Wage Crime Income Metrop Poor Taxes Educ BusFail Temp Region Definition State name Net domestic immigration rate over the period Unemployment rate in the civilian labor force in 1994 Average hourly earnings of production workers in manufacturing in 1994 Violent crime rate per 1, people in 1993 Median household income in 1994 Percentage of state population living in metropolitan areas in 1992 Percentage of population who fall below the poverty level in 1994 Total state and local taxes per capita in 1993 Percentage of population 25 years or older who have a high school degree or higher in 199 The number of business failures divided by the population of the state in 1993 Average of the 12 monthly average temperatures (in degrees Fahrenheit) for the state in 1993 Region in which the state is located (northeast, south, midwest, west) 1.4 STEPS IN REGRESSION ANALYSIS Regression analysis includes the following steps: Statement of the problem Selection of potentially relevant variables Data collection Model specification Choice of fitting method Model fitting Model validation and criticism Using the chosen model(s) for the solution of the posed problem. These steps are examined below.

26 8 INTRODUCTION Table 1.6 First Six Variables of the Domestic Immigration Data State NDIR Unemp Wage Crime Income Metrop Alabama Arizona Arkansas California Colorado Connecticut Delaware Florida Georgia Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Pennsylvania Rhode Island Carolina Dakota Tennessee Texas Utah Vermont Virginia Washington West Virginia Wisconsin Wyoming I I

27 STEPS tn REGRESSION ANALYSIS 9 Table 1.7 Last Six Variables of the Domestic Immigration Data State Poor Taxes Educ BusFail Temp Region Alabama Arizona Arkansas California Colorado Connecticut Delaware Florida Georgia Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Pennsylvania Rhode Island Carolina Dakota Tennessee Texas Utah Vermont Virginia Washington West Virginia Wisconsin Wyoming I I West West West Northeast West Midwest Midwest Midwest Midwest Northeast Northeast Midwest Midwest Midwest West Midwest West Northeast Northeast Midwest Northeast Midwest Midwest West Northeast Northeast Midwest West Northeast Midwest Midwest West

28 1 INTRODUCTION Table 1.8 Variables for Study of Water Pollution in New York Rivers Variable Y X1 x2 x3 x4 Definition Mean nitrogen concentration (mg/liter) based on samples taken at regular intervals during the spring, summer, and fall months Agriculture: percentage of land area currently in agricultural use Forest: percentage of forest land Residential: percentage of land area in residential use Commercial/Industrial: percentage of land area in either commercial or industrial use Table 1.9 The New York Rivers Data Row River Y X1 x2 x3 x Olean Cassadaga Oatka Neversink Hackensack Wappinger Fishkill Honeoye Susquehanna Chenango Tioughnioga West Canada East Canada Saranac Ausable Black Schoharie Raquette Oswegatchie Cohocton oo

29 STEPS IN REGRESSION ANALYSIS Statement of the Problem Regression analysis usually starts with a formulation of the problem. This includes the determination of the question(s) to be addressed by the analysis. The problem statement is the first and perhaps the most important step in regression analysis. It is important because an ill-defined problem or a misformulated question can lead to wasted effort. It can lead to the selection of irrelevant set of variables or to a wrong choice of the statistical method of analysis. A question that is not carefully formulated can also lead to the wrong choice of a model. Suppose we wish to determine whether or not an employer is discriminating against a given group of employees, say women. Data on salary, qualifications, and sex are available from the company s record to address the issue of discrimination. There are several definitions of employment discrimination in the literature. For example, discrimination occurs when on the average (a) women are paid less than equally qualified men, or (b) women are more qualified than equally paid men. To answer the question: On the average, are women paid less than equally qualified men? we choose salary as a response variable, and qualification and sex as predictor variables. But to answer the question: On the average, are women more qualified than equally paid men? we choose qualification as a response variable and salary and sex as predictor variables, that is, the roles of variables have been switched Selection of Potentially Relevant Variables The next step after the statement of the problem is to select a set of variables that are thought by the experts in the area of study to explain or predict the response variable. The response variable is denoted by Y and the explanatory or predictor variables are denoted by XI, XZ,..., X,, where p denotes the number of predictor variables. An example of a response variable is the price of a single family house in a given geographical area. A possible relevant set of predictor variables in this case is: area of the lot, area of the house, age of the house, number of bedrooms, number of bathrooms, type of neighborhood, style of the house, amount of real estate taxes, etc Data Collection The next step after the selection of potentially relevant variables is to collect the data from the environment under study to be used in the analysis. Sometimes the data are collected in a controlled setting so that factors that are not of primary interest can be held constant. More often the data are collected under nonexperimental conditions where very little can be controlled by the investigator. In either case, the collected data consist of observations on n subjects. Each of these n observations consists of measurements for each of the potentially relevant variables. The data are usually recorded as in Table 1.1. A column in Table 1.1 represents a variable, whereas a row represents an observation, which is a set of p + 1 values for a single subject (e.g., a house); one value for the response variable and one value for each

30 ~~~ ~ 12 INTRODUCTION Table 1.1 Notation for the Data Used in Regression Analysis Observation Response Predictors Number Y X1 x Y1 511 x12... Y Y XP XlP Q P 53P of the p predictors. The notation xij refers to the ith value of the jth variable. The first subscript refers to observation number and the second refers to variable number. Each of the variables in Table 1.1 can be classified as either quantitative or qualitative. Examples of quantitative variables are the house price, number of bedrooms, age, and taxes. Examples of qualitative variables are neighborhood type (e.g., good or bad neighborhood) and house style (e.g., ranch, colonial, etc.). In this book we deal mainly with the cases where the response variable is quantitative. A technique used in cases where the response variable is binary6 is called logistic regression. This is introduced in Chapter 12. In regression analysis, the predictor variables can be either quantitative andor qualitative. For the purpose of computations, however, the qualitative variables, if any, have to be coded into a set of indicator or dummy variables as discussed in Chapter 5. If all predictor variables are qualitative, the techniques used in the analysis of the data are called the analysis of variance techniques. Although the analysis of variance techniques can be introduced and explained as methods in their own right, it is shown in Chapter 5 that they are special cases of regression analysis. If some of the predictor variables are quantitative while others are qualitative, regression analysis in these cases is called the analysis of covariance Model Specification The form of the model that is thought to relate the response variable to the set of predictor variables can be specified initially by the experts in the area of study based on their knowledge or their objective andor subjective judgments. The hypothesized model can then be either confirmed or refuted by the analysis of the collected data. Note that the model need to be specified only in form, but it can still depend on unknown parameters. We need to select the form of the function A variable that can take only one of two possible values such as yes or no, 1 or, and success or failure, is called a binary variable See, for example, the books by Scheffk (1959). Iversen (1976). Wildt and Ahtola (1978). Krishnaiah (198), Iversen and Norpoth (1987), Lindman (1992). and Christensen (1996)