Introduction. Chapters 4 and 14. Bivariate Linear Regression Model: Part I

Introduction Chapters 4 and 14 Bivariate Linear Regression Model: Part I We have developed a number of tests to examine the statistical relationship between variables Covariance Correlation Difference in means These tests demonstrate a statistical association between two variables More educated have higher incomes Kids who take SAT prep course have greater gains in scores Kids who study more have higher grades Bankruptcy rates higher in counties with slot machines 1 2 In many cases, researchers are only interested ub identifying whether there is a statistically significant relationship between variables Have obesity rates changed over time? Are women paid less than men? Have test scores of MD high school seniors improved over time? However, in many instances, researchers want to know more than a statistical relationship they want to know something about the magnitude of the relationship How much will bankruptcy rates change if a state allows gambling? How much will lottery sales fall when slot machines are legalized? What is the best guess of what housing prices will be next quarter? 3 4

To answer these questions, we need a more detailed model of statistical relationships A model that allows us to examine how movements in one variable (e.g., prices, income, laws) impact the value of another Requires that we place more structure on the problem Problem: less chance our model is correct Advantage: get something in return predictive ability 5 Note to be able to predict the impact of one variable on another, you must be able to measure the casual impact of one variable on another However, most statistical relationships do not measure casual mechanisms. In most cases, isolating a causal relationship is more difficult than examining correlations or differences in means Just because there are difference in means or non-zero correlation coefficient does not mean that there is a causal relationship between variables 6 Example 1 Suppose you wanted to estimate the impact of more police on crime Collect data on crime and the size of the police force in 100 cities Find out that the relationship (correlation) is positive Is this casual do more police actually increase crime? What else could explain this relationship? 7 Washington Post, August 15, 1997 page A3 Lasting Effects Found From Spanking Children Antisocial Behavior Is Increased, Study Says Spanking children is apt to cause more long-term behavioral problems than most parents who use that approach to discipline may realize, a new study reports. Children who get spanked regularly are more likely over time to cheat or lie, to be disobedient at school and to bully others, and have less remorse for what they do wrong, according to the study by researchers at the University of New Hampshire. It is being published this month in the medical journal Archives of Pediatrics and Adolescent Medicine. "When parents use corporal punishment to reduce antisocial behavior, the long-term effect tends to be the opposite," the study concludes. 8

Reuters New Service, Oct 30, 2003 BOSTON, Oct. 30 Greater diversity on college campuses significantly lowers rates of binge drinking among high-risk students, according to results of a Harvard University study released Thursday. The research may enable college administrators to fine-tune their admissions and housing policies to cut rates of binge drinking, study authors said. If you have younger white males together to the exclusion of other groups, you re going to have fewer role models for lighter or nondrinking behavior, Henry Wechsler, the study s lead author, told Reuters. That may explain why fraternities have had such a high level of drinking problems. To come up with new ways of tackling the problem, Wechsler and his colleagues decided to look at demographics on college campuses 9 In both studies, the authors detect an association Children who are spanked have more behavioral problems Students who attend more diverse colleges drink less Can we assign a causal interpretation to these relationships? Are we sure that spanking kids less will reduce crime or having more diverse colleges reduces drinking? 10 In these two chapters Generate a model that helps measure the impact of x on y regression analysis The model allows us to measure the strength of the statistical association between two variables This is an accurate estimate ONLY IF we have measured the casual relationship Under what conditions do we have a casual relationship? Identify the statistical properties of the model In most cases, authors want to estimate the causal impact of x on y But in very few cases do they satisfy the necessary assumptions 11 12

Model building Before we build a statistical model of the relationship between two variables, we need an example of a model Model is input/output function Given observed characteristics, what is the likely outcome The key for these type of models is the distinction between independent and dependent variables Dependent variables outcomes of the system or model variables that are potentially altered by outside forces Independent variables factors that are determined outside the model states of nature or characteristics that can be manipulated to change dependent variables Dependent = f(independent) 13 14 Simplest example: Clinical Trial Dependent variable medical condition (mortality cholesterol level, blood pressure) Independent variable new medicine Blood pressure = f(dosage) We can manipulate the dosage to alter blood pressure In most cases the work is not so simple Crime and police example Want to know impact of more police on crime Dependent (Crime) Independent (police) Crime = f(police) Problem Police = g(crime) Hire more police in places with more crime 15 16

Theory of Demand Exogenous = Independent Endogenous = Dependent Workhorse model of economic theory Core of econ 306 incentives matter Model set up Consumers derive utility from consumption of 2 goods (x,y) U = U(x,y) Utility function has specific properties Pick utility maximizing bundle of (x,y) subject to constraints Fixed prices for goods: P x and P y Fixed income, I 17 18 Y I/P y Y 1 X 1 I/P x X Utility is maximized at the point where the marginal utility derived from spending $1 on x just equals the marginal utility of spending $1 on y Mu x /P x = Mu y /P y If Mu x /P x > Mu y /P y, take $1 out of y, and put it into x, utility increases 3 exogenous variables: P x, P y and I 2 endogenous variables: x and y 19 20

Exogenous variables are variables that are determined outside the system they are not influenced by the choices being made Called independent variables Prices of CDs do not change if you decide to buy more or less Your salary is based on your value to your employer, not your ability to spend money Endogenous variables are outcomes of a system Dependent variables A function of the independent variables of the system X = f x (P x, P y, I) Y = f y (P x,p y, I) 21 22 Why is the theory of demand so useful? Able to use the theory to predict how behavior will change when exogenous factors are altered What will happen to the demand for x when P x increases? What will happen to the demand for y when I increases? Comparative statics Note the distinction between endogenous and exogenous factors To build a statistical model that will allow us to predict the changes in outcomes, we need to assume a direction of causation Prices alter how much you will purchase Number of slot machines determine bankruptcy rates Years of education alter earnings ability Our model will only accurately measure the impact of x on y if this assumption is correct 23 24

Behavioral or causal model Y i is endogenous variable for observation i Represents some outcome of interest (e.g., consumption, test scores, income, etc.) Defined the same for all people FOR NOW assume that we can measure only 1 input to Y x i Exogenous variable determined outside the system There are many inputs that impact or cause the value of Y The only one we can measure or the only one of interest is x Hypothesize that x and y are related Changes in external values of x will alter value of y comparative statics Place some structure on the relationship between x and y Linear model y i = α + βx i + ε i 25 26 Linear model y i = α + βx i + ε i α and β are population values represent the true relationship between x and y Unfortunately these values are unknown The job of the researcher is to estimate these values Notice that if we differentiate y with respect to x, we obtain dy/dx = β β represents how much y will change for a fixed change in x Increase in income for more education Change in crime or bankruptcy when slots are legalized Increase in test score if you study more 27 Put some concreteness on problem State of Maryland is experiencing a significant budget shortfall Drop in revenues due to recession Increase in expenditures due to higher k-12 school spending initiatives Short-term solution raise tax on cigarettes by 34 cents/pack Problem a tax hike will reduce consumption (theory of demand) Question for state as taxes are raised, how much will cigarette consumption fall 28

Benefits and Costs of Model Suppose y is a state s per capita consumption of cigarettes x represents taxes on cigarettes Question how much will y fall if x is increased by 34 cents/pack? Problem many reasons why people smoke cost is but one of them Placed more structure on the model, therefore we can obtain precise statements about the relationship between x and y These statement will be true so long as the hypothesized relationship is true As you place more structure on any model, the chance that the assumptions of the model are correct declines. 29 30 Cigarette Consumption and Taxes Data (Y) State per capita cigarette consumption for the years 1980-1997 (X) tax (State + Federal) in real cents per pack Scatter plot of the data Negative covariance between variables When x>n, more likely that y<n When x<n, more likely that y>n Goal: pick values of α and β that best fit the data Define best fit in a moment Per capita packs/year 300 250 200 150 100 50 0 0 20 40 60 80 100 120 Tax per pack (cents) 31 32

Per capita packs/year Cigarette Consumption and Taxes 300 250 200 150 100 50 0 0 20 40 60 80 100 120 Tax per pack (cents) Quick review: Covariance and correlation Cov(x,y) = E[(x - µ x )(y - µ y )] = F xy Covariance between x and y If Cov(X,Y)>0 and Y>N, we expect X>n If Cov(X,Y)<0 and Y>N, we expect X<n Scale dependent 33 34 Correlation coefficient corr(x,y) = D xy = cov(x,y)/f x F y Scale independent -1 # D xy # 1 n observations of x and y (x i,y i ) n = sample mean of x N= sample mean of y s 2 x = [G i (x i - n)]/(n-1) s 2 y = [G i (y i - N)]/(n-1) J xy = G i (x i - n)(y i - N)/(n-1) K xy = J xy /s x s y 35 36

Height (in inches) 85 80 75 70 65 60 55 50 I IV Plot: Height vs. Weight 50 100 150 200 250 300 350 400 Weight (in pounds) II III What is ε i? There are many factors that determine a state s level of cigarette consumption Some of these factors we can measure, but for what ever reason, we do not have data Education, age, income, etc. Some of these factors we cannot measure Dislike of cigarettes, anti-smoking sentiment of population ε i measure what we do cannot measure in our model 37 38 What is ε i? Y Given linear model y i = α + βx i + ε i We can predict an level of consumption given parameter values y p i = α + βx i The predicted value will not always be accurate sometimes we will over or under predict the true value Because of the linear relationship between x and y, predictions will lie along a line Y 1 Y p 2 Y p 1 Y 2 ε 1 >0 X 1 X 2 ε 2 <0 y=α+βx X 39 40

What is ε i? Notation The difference between the actual and predicted value is the error ε i y i -y p i = y i - α + βx i = ε i We never actually observe ε i. This is the true error based on the population values of α and β. Because we do not know α and β, we never know ε i. We can however estimate values of ε I by estimating values of α and β. Our goal, is to choose values for α and β subject to some criteria True model y i = α + βx i + ε i We observe data points (y i,x i ) The parameters α and β are unknown The actual error (ε i ) is unknown Estimated model (a,b) are estimates for the parameters (α,β) e i is an estimate of ε i where e i =y i -a-bx i How do you estimate a and b? 41 42 Objective: Minimize sum of squared errors Min Σ i e i2 = Σ i (y i a bx i ) 2 Minimize the sum of squared errors (SSE) Treat positive and negative errors equally Over or under predict by 5 is the same magnitude of error Quadratic form The optimal value for a and b are those that make the 1 st derivative equal zero Functions reach min or max values when derivatives are zero 43 Y X 2 X 1 X f(x) 44

Properties of summations n=(1/n)σ i x i Therefore, Σ i x i = n n Consider sum of deviations from mean Σ i (x i - n) = Σ i x i - Σ i n = Σ i x i -nn = Σ i x i - Σ i x i = 0 Sum of deviations from means equal zero Sum of deviations Σ i (x i - n) 2 = Σ i (x i - n)(x i - n) = Σ i (x i - n)x i - Σ i (x i - n) n n does not vary across i, so Σ i (x i - n) n = n Σ i (x i - n) Recall from above that Σ i (x i - n) =0, so Σ i (x i - n) 2 = Σ i (x i - n)x i 45 46 Using the same type of algebra, we can show that Σ i (x i - n)(y i -N) = Σ i (x i - n)y i = Σ i x i (y i -N) SSE = Σ i (y i a bx i ) 2 To minimize a function, choose values of a and b that force the 1 st derivatives to zero d(sse)/da = - 2 Σ i (y i a bx i ) = 0-2 Σ i (y i a bx i ) = 0 Multiply by 1/2 Σ i (y i a bx i ) = 0 Divide by n (1/n) Σ i (y i a bx i ) = 0 47 48

Rewrite all terms (1/n)Σ i (y i ) (1/n)Σ i (a) (1/n)Σ i (bx i ) = 0 Note that» (1/n)Σ i (y i ) = N» (1/n)Σ i (a) = (1/n)(na) = a» (1/n)Σ i (bx i ) = (b/n) Σ i (x i ) = b n Therefore N -a -b n = 0 And a= N -b n What is derivative of SSE with respect to b? SSE = Σ i (y i a bx i ) 2 d(sse)/db = -2 Σ i x i (y i a bx i ) = 0 From previous slide, we know that a= N -b n Substitute this into d(sse)/db -2 Σ i x i [y i a bx i ] = -2 Σ i x i [y i (N -b n) bx i ] = 0 Collect like terms -2 Σ i x i [(y i N) b(x i - n)] = 0 49 50 Multiply both side by (1/2) Σ i x i [(y i N) b(x i - n)] = 0 Expand expression Σ i x i [(y i N)] b Σ i x i [(x i - n)] = 0 Solve for b b Σ i x i [(x i - n)] = Σ i x i [(y i N)] b=σ i x i [(y i N)] / Σ i x i [(x i - n)] and a= N -b n Recall summation identities Σ i (x i - n)(y i -N) = Σ i (x i - n)y i = Σ i x i (y i -N) Σ i (x i - n) 2 = Σ i (x i - n)x i Therefore b=[σ i (x i -n)(y i N)] / [Σ i (x i - n) 2 ] Divide numerator and denominator by (n-1) (Does not change value) b=[(1/(n-1))σ i (x i -n)(y i N)] / [(1/(n-1))Σ i (x i - n) 2 ] 51 52

Look at numerator [(1/(n-1))Σ i (x i -n)(y i N)] By definition, sample covariance between x and y J xy = [(1/(n-1))Σ i (x i -n)(y i N)] Look at denominator [(1/(n-1))Σ i (x i - n) 2 ] By definition, sample variance of x s 2 x = [(1/(n-1))Σ i (x i - n)2 ] Therefore b= J xy /s 2 x Can re-arrange even further Write s 2 x =s x s x Multiply by ratio s y /s y b= J xy /s 2 x = [J xy /(s x s y )][ s y /s x ] Notice that [J xy /(s x s y )] = K xy = sample correlation coefficient b= J xy /s 2 x = [K xy ][ s y /s x ] Knowing s x s y and = K xy, we can estimate b 53 54 In summary Example By minimizing the sum of squared errors, we pick the values of a and b that best fit the data The optimal values of a and b are: b= J xy /s 2 x = [K xy ][ s y /s x ] a= N -b n Look at results from SAS output for PROC CORR Estimates correlation coefficients proc corr data=one; title 'correlation coefficients'; title2 'cigarette consumption and excise taxes'; var cons tax; run; 55 56

Descriptive Statistics Using the results x=tax and y=cons J xy 0.58591 n =49.60816, s x =14.31725 N=111.21481, s y =27.83818 b= [K xy ][ s y /s x ] = -0.58591[27.83818/14.31725] = -1.139 a= N-bn = 111.21481 (-1.139)(49.60816) = 167.72 b=dy/dx = -1.139 For every penny increase in taxes, per capita consumption falls by 1.139 packs per year A 34 cent increase in taxes will reduce consumption by (34)(1.139) = 38.7 packs per person per year 57 58 Per capita packs/year Cigarette Consumption and Taxes 300 250 200 150 100 50 0 0 20 40 60 80 100 120 Tax per pack (cents) Example 2: Education and Earnings Stylized fact: log wages or earnings is linear in education (above a certain range) Interpreted as a return to education Theoretical models why this would be the case Linear model: y=ln(weekly wages) endogenous variable x=years of education exogenous factor y i = α + βx i + ε i 59 60

Average ln(weekly Wages) vs Educ Notice that β has a different interpretation β=dy/dx In this case, y=ln(wages) dln(wages)/dx = (1/wages)dWages/dX dwages/wages = % change in changes (change in wages over base wages) when the endogenous variable is a natural log, β=dy/dx is interpreted as % change in y for a unit change in x Average ln(weekly Wages) 6.80 6.60 6.40 6.20 6.00 5.80 5.60 5.40 5.20 5.00 0 2 4 6 8 10 12 14 16 18 20 Years of Education 61 62 ln(weekly Wages) ln(weekly Wages) vs. Education 7.5 7.0 6.5 6.0 5.5 5.0 4.5 4.0 0 2 4 6 8 10 12 14 16 18 20 Years of Education Descriptive Statistics x=education and y=ln(weekly wages) J xy = 0.4214 n =12.96, s x =3.00 N=6.03, s y =0.537 b= [K xy ][ s y /s x ] = 0.4214[0.537/3.00] = 0.075 a= N-bn = 6.03 (0.075)(12.96) = 5.05 63 64

A couple of mathematical notes ln(weekly Wages) vs. Education ln(weekly Wages) 7.5 7.0 6.5 6.0 5.5 5.0 4.5 4.0 0 2 4 6 8 10 12 14 16 18 20 Years of Education e i =y i -a-bx i Prediction error Consider the sum of the predicted errors Σ i e i = Σ I [y i -a-bx i ] = nn-na -nbn = n[n- a -bn] Recall the definition of a a= N- bn Substitute this into the equation above Σ i e i = n[n- a -bn]=n[n- (N- bn) - bn] = 0 The sum of the predicted errors equals zero 65 66 Consider a second summation Σ i e i (a+bx i ) = Σ i e i y p i Rewrite Σ i e i a + Σ i e i bx i = aσ i e i + bσ i e i x i Above we showed that Σ i e i =0 so the 1st term is 0 Look at the 2 nd term Σ i e i x i = Σ I (y i -a-bx i )x i This equals zero as well why well look at the first-order condition Recall that d(sse)/db = -2 Σ i x i (y i a bx i ) = 0 Multiply both sides by (1/2) Σ i x i (y i a bx i ) = Σ i x i e i = 0 Therefore, we have two results Σ i e i (a+bx i ) = Σ i e i y p i =0 Σ i e i =0 67 68

How well do we fit the data? R 2 For any value of x and given estimated values of a and b, we can predict values of y. This predicted value will (in most cases) differ from the actual value that exists for this particular value of x. We need some way of measuring how well our model fits the data R 2 How much variation in Y is there in our data? Σ i (y i -N) 2 sums of squares total numerator in sample variance Recall that» s 2 y =[Σ i (y i -N)2 ]/(n-1) For any x, we can produce a predicted value for y y p i =a+bx i 69 70 The error in our equation is by definition e i =y i -a-bx i y i =y p i + e i Calculate the average y N=(1/n)Σ i y i = (1/n)Σ i [y p i + e i ] = (1/n)Σ i [y p i ] + (1/n) Σ I [e i ] Previously, we showed that Σ I [e i ]=0, so N=(1/n)Σ i y i = (1/n)Σ i [y p i ] = Np Mean of predicted values for y equals the mean of y y i =y p i + e i Subtract mean of y from both sides (y i -N)=(y p i - Np )+ e i Square both sides (y i -N) 2 =[(y p i - Np )+ e i ] 2 = (y p i - Np ) 2 + (y p i - Np ) e i + e i 2 Sum over all I Σ i (y i -N) 2 = Σ i [(y p i - Np ) 2 + (y p i - Np )e i + e i2 ] =Σ i (y p i - Np ) 2 + Σ i [(y p i - Np )e i ]+ Σ i e i 2 It is easy to show that the middle term equals 0 71 72

Σ i [(y p i - Np )e i ] = Σ i (y p i e i ) - Σ i Np e i = Σ i (y p i e i ) - Np Σ i e i =0 Therefore Σ i (y i -N) 2 = Σ i (y p i - Np ) 2 + Σ i e i 2 Σ i (y i -N) 2 = sum of squared total = SST Σ i (y p i - Np ) 2 = sum of squared model = SSM Σ i e i2 = sum of squared error = SSE SST = SSM + SSE 1 = SSM/SST + SSE/SST SSM/SST = 1 SSE/SST = R 2 SST is how much variation there is in total in the endogenous variable of interest With our choice of a and b, we can predict a certain amount of this variation (SSM) The fraction of SST that we can predict with our model is defined as R 2 R 2 lies between 0 and 1 Perfect fit: R 2 =1 No fit: R 2 =0 73 74 How to run a regression in SAS R 2 for the two examples proc reg data=one; title 'ols regression -- consumption on taxes'; model cons=tax; run; Consumption and taxes: R 2 = 0.343 Ln(wages) and education: R 2 = 0.178 Work through first example: SSM = 243956 SSE = 466686 SST = 710642 R 2 = SSM/SST = 243956/710642 = 0.343 75 76