Multiple Regression Analysis. Basic Estimation Techniques. Multiple Regression Analysis. Multiple Regression Analysis

Transcription

1 Multiple Regression Analysis Basic Estimation Techniques Herbert Stocker University of Innsbruck & IIS, University of Ramkhamhaeng Regression Analysis: Statistical procedure for quantifying (economic) relationships and testing hypothesis about them. A regression model specifies a (usually linear) relationship between a dependent variable and a independent (or explanatory) variable an error term u = β +βx +u Multiple Regression Analysis: When there is more than one explanatory variable. Multiple Regression Analysis Estimates the relationship between a dependent variable (left-hand side) and independent variables (right-hand side). = β +βx +βx +u is the dependent variable, that should be explained by the independent variables X and X. Dependent and independent variables can be observed (there are data available for them). The parameters β, β and β cannot be observed, but they can be estimated by linear regression from the data Parameter estimation. Multiple Regression Analysis = β +βx +βx +u The parameters β, β and β describe the relation between dependent and independent variables: = β, X = β X Parameters describe marginal effects, e.g. if X increases by unit will change by β units! Can be done with spreadsheet software.

2 Multiple Regression Analysis For example, a demand function Q = β βp +βadv (Q...quantity sold, P...Price, Adv... Spending for Advertisement) can be estimated, if data for Q, P and Adv are available, e.g. Q = 6..P +.Adv describes the relationship between the dependent variable Quantity demanded (left-hand side) and the independent variables Price and Advertisement (right-hand side). Multiple Regression Analysis Q = 6..P +.Adv The coefficient of P tells us, that holding Advertisement constant an increase of price for one unit will decrease (negative sign!) the expected quantity sold by. units. The coefficient of Adv tells us, that holding Price constant an increase of the spending for Advertisement by one unit will increase (positive sign!) the expected quantity sold by. units. Regression Analysis: Introduction Descriptive Regression Analysis: The Population Regression Function (PRF) We have observations (data) on variables and X... Collected observations: 6 i X = (X) = β +βx 6 7 8X Regression analysis allows to calculate β and β!

3 Regression Analysis We decompose the observed variable (e.g. quantity sold) in a deterministic and a stochastic component. i = β +βxi + }{{}}{{} ui random Ŷi systematic The deterministic component is explained by the model (i.e. the part of that is explained by X) and usually denoted with a hat on the variable (e.g. Ŷ). The random component is not explained by the model an called the error term (often denoted with u or ε). 6 Decomposition of u Ŷ i = f(xi)? ui: i = Ŷi +ui Ŷ u u Ŷ Ŷ Ŷ X u u Ŷi: systematic stochastic Ŷi =.+.X Regression Analysis Regression analysis is based on the minimization of the sum of squared errors! We cannot use the simple sum of error terms, because positive and negative errors would compensate for each other. Therefore, we use the sum of squared errors. This procedure allows us to calculate the parameters β and β. Graphically... Calculation of β and β 6 i = β +βxi +ui Ŷi = β +βxi ui = i Ŷi Ŷ u u Ŷ Ŷ uu u Ŷ Ŷ X uu min i u i = i (i Ŷi)

4 Calculation of β and β Mathematically: minimizing the sum of the squared error terms min β,β N ui = ( ) i Ŷi = (i β βxi) i= Solution to this minimization problem: β = Cov(,X) Var(X) β = Ȳ β X where Ȳ /N i i and X /N i Xi. Multiple Regression These results can easily extended to more than variables, e.g. = β +βx +βx + +βkxk but we need matrix algebra for the formulas (Excel can do it!). For multiple regressions the coefficients give the marginal effect of a variable, keeping all other variables constant (ceteris paribus)! e.g. = β X if X changes by one unit we expect to change by β units, while X,...,Xk are held constant! Coefficient of Determination Coefficient of Determination R Coefficient of Determination (R ): A measure of how the overall estimating equation fits the data. It shows the fraction of the variation in the dependent variable, which is explained statistically by the variables included in the regression. The Coefficient of Determination R gives the share of the variation in which is explained by the regression (i.e., by all X variables). It can be shown that R is the square of (Pearson s) correlation coefficient between the observed values of and the fitted values Ŷ. If the regression has a constant term (intercept) R is always between (best possible fit) and (worst possible fit).

5 X 6 8 X X 6 8 X Regressions with different R : R =, R =, R =,8 R =,99 Population & Sample Inferential Regression Analysis: The Sample Regression Function (SRF) Population & Sample Population Up to now we did only descriptive analysis: Which β und β describes the relationship in the population best? But: only rarely we can observe the population, usually we observe only samples! Question: How can we infer from the sample on the population? Example: we assume a population of subjects and draw samples thereof. observations: = f(x)? N X Sample N X Sample..8 c.. c..9 b.. a.7.8 c..6 a.9.7 a.7.6 a.9. b..9 c 6.. a 6.. b 7..6 c 7.. b 8.. c 8..9 b b 9.. b.8. a.7. c

6 Population: observations relationship in the population X Ŷi =.7+.Xi True relation in the population and errors u of the population X Ŷi =.7+.Xi Random-Sample a (N a = 6) X Ŷi =.7+.Xi a i =.98+.9Xi +e a i i X S.9.7 a 6.. a.8. a.. a..6 a.7.6 a Random-Sample b (N b = 7) i b =.+.9Xi +ei b X Ŷi =.7+.Xi i X S..8 b.7.8 b 7..6 b 8.. b.. b..9 b.7. b

7 Random-Sample c (N c = 7) X Ŷi =.7+.Xi i c =.+.Xi +ei c i X S..9 c.9. c c 6.. c 7.. c 8..9 c 9.. c For each random sample we get different coefficients b and b! i b =.+.9Xi +ei b X i =.7+.Xi +ui i c =.+.Xi +ei c a i =.98+.9Xi +e a i b and b are random variables! (but β =.7 and β =. are parameters) i b =.+.9Xi +ei b X i =.7+.Xi +ui i c =.+.Xi +ei c a i =.98+.9Xi +e a i Estimated coefficients have a sampling distribution: This allows us to do statistics!

8 Regression Analysis & Statistics Notation: in what follows we use different symbols for values of the population and values estimated out of a sample. Coefficients: we use β for the population and b for the sample. Sometimes you also see β for sample-estimates. Residuals: we use u for the population errors and e (or û) for the sample residuals. Residuals are the sample analoge to the population errors! Regression Analysis & Tests of Hypothesis Single Hypothesis Single Hypothesis Remember, for each sample we draw we get different estimates b and b for the true but unknown β and β. Therefore, the regression coefficients b,b,...,bk are itself random variables. We have determined the sampling distribution of these random variables: If the population error term is normally distributed, or the sample is sufficiently large (say, N > ), the the estimated coefficients are also (approximately) normally distributed. We can calculate the standard errors of these coefficients und do hypothesis tests as usual. Regression Analysis & Statistics Standard Error of coefficient bk: sbk A measure of the precision of an estimated regression analysis coefficient. It shows how much the coefficient would vary in regressions from different samples. If the standard error is large relative to the coefficient, we should not trust the coefficient!

9 Single Hypothesis The t-statistic to test a single hypothesis is usually calculated as follows: (estimate hypothesized value) t = standard error of estimate In most cases: H : βk =, therefore t = bk sbk with N K degrees of freedom. with : standard error of bk, N: number of sbk observations, K: number of estimated coefficients This is the t-statistic given in the usual regression output. Regression Analysis & Statistics t-test: A test based on the ratio of the estimated regression coefficient to its standard error that is used to determine the statistical significance of the coefficient, i.e., to test whether the estimated coefficient is statistically significant different from zero: t = bk sbk If the absolute value of the calculated t-statistic is larger than the critical value of the theoretical t-statistic we can reject the null-hypothesis that there is no relationship between this explanatory variable and the dependent variable, i.e. we say the coefficient is significant. Regression Analysis & Statistics Regression Analysis & Statistics t-test: Rule of thumb: if in a normal sample (size > ) the absolute value of the calculated t-statistic is larger than the coefficient is often statistically significant different from zero. p-value: gives the probability to find a coefficient of the estimated magnitude in a random sample, when in the population is in fact no relationship between the variables (Type I error). p-value is the probability we would observe the t statistic we did, if the null hypothesis were true. The p-value is usually based in a t-test, it gives essentially the same information. For the usual percent significance level the p-value should be smaller than.!

10 F Statistic The F statistic given in the regression output of most programs is used to calculate the probability, that all right-hand (independent or X-) variables except the intercept are jointly different from zero, i.e H : β = β =... = βk = Again, the p-value of of this F statistic is the probability we would observe the F statistic we did, if the null were true. If this null hypothesis cannot be rejected the regression is of course rather worthless. Typical Regression Output Dependent Variable: Q Method: Least Squares Included observations: Coefficient Std. Error t-statistic Prob C P M R-squared.9668 Mean dependent var 7.7 Adjusted R-squared S.D. dependent var 8.7 S.E. of regression 8.68 Sum squared resid 79. F-statistic.86 Durbin-Watson stat.79 Prob(F-statistic). For publications usually written as: Q = P +.7 M (9.) (7.) (.7) R =.97, s = 8.6, F-Stat =.9, N = Excel Regression Output Functional Form

11 Functional Form Isoelastic Demand OLS can be used for relationships that are not strictly linear in X and by using nonlinear functions of X and will still be linear in the parameters. ou can take... the natural log of X, or both. quadratic forms of X. interactions of X variables (e.g. X X) However, the interpretation of the coefficient differs If the variables are in logarithms the coefficients are elasticities; i.e. log-linear curves have constant elasticities! When log() = β +βlog(x) then β is the constant elasticity between X and. Example: Log-linear demand curve: log(q) =..log(p) +.log(i) Price elasticity =., Income elasticity =. Interpretation of log-models Log-Linear Regression Model If the model is log-log ln() = b +bln(x)+e b is the elasticity of with respect to X. If the model is log-lin ln() = b +bx +e b is approximately the percentage change in given a unit change in X.

12 Why use log models? Some Rules of Thumb Log models are invariant to the scale of the variables since measuring percent changes. Log-Log models give a direct estimate of elasticity. The distribution of ln() is more narrow, limiting the effect of outliers and heteroskedasticity. Attention: Can only be used for positive (log of negative numbers is not defined!). What types of variables are often used in log form? Baht or dollar amounts that must be positive. Very large variables, such as population. Variables that have a natural growth rate. What types of variables are often used in level form? Variables that are a proportion or percent. Growth rates or changes. Quadratic Models Quadratic regression models are appropriate when the curve fitting the scatter plot is either -shaped or -shaped. A quadratic equation, = a+bx +cx, can be transformed into a linear form by computing a new variable Z = X, which is then substituted for X in the regression. Then, the regression equation to be estimated is. = a+bx +cz Quadratic Models For a model of the form = b +bx +bx +e we can t interpret b alone as measuring the change in with respect to X, we need to take into account b as well, since d = b +bx dx if b is positive and b is negative then is increasing in X at first, but will eventually turn around and be decreasing in X.

13 Quadratic Models Thank you!