CHAPTER 2 SIMPLE LINEAR REGRESSION

Size: px
Start display at page:

Download "CHAPTER 2 SIMPLE LINEAR REGRESSION"

Transcription

1 CHAPTER 2 SIMPLE LINEAR REGRESSION 1

2 Examples: 1. Amherst, MA, annual mean temperatures, Summer mean temperatures in Mount Airy (NC) and Charleston (SC), Scatterplots outliers? influential values? independent v. dependent variables 2

3 Mean Temperature Charleston A B Year Mount Airy Figure 2.1. (a) Plot of mean temperature against year for Amherst data. (b) Plot of mean summer temperature in Charleston against mean summer temperature in Mount Airy. In each case a least squares regression line is shown on the plot. 3

4 The basic model: y i = α + βx i + ϵ i, 1 i n. (a) ϵ 1,..., ϵ n uncorrelated with mean 0 and variance σ 2 (b) ϵ 1,..., ϵ n independent N[0, σ 2 ]. σ 2 unknown in practice 4

5 Method of Least Squares Rewrite basic equation as y i = β 0 + β 1 (x i x) + ϵ i, 1 i n. ( x = 1 n ni=1 x i ) Method: choose estimates ˆβ 0, ˆβ 1 to minimize S = n i=1 { yi ˆβ 0 ˆβ 1 (x i x) } 2. See picture. Solution is ˆβ 0 = ˆβ 1 = ni=1 y i = ȳ, n ni=1 y i (x i x) ni=1 (x i x) 2. 5

6 12 10 y x Figure 2.2. Illustration of the least squares criterion on a small artificial data set. The fitted straight line is chosen so as to minimize the sum of squares of vertical distances from the observations to the line. 6

7 Estimation of σ 2 First define residuals (see next figure) e i = y i ˆβ 0 ˆβ 1 (x i x) The e i s are estimates of the original ϵ i s. Therefore, an obvious estimate of ˆσ 2 would be ni=1 ˆσ 2 e 2 = i. n It turns out this is a biased estimator; an unbiased estimator is s 2 = ni=1 e 2 i n 2. The numerator is the residual sum of squares (RSS); s 2 is the mean sum of squares (MSS). 7

8 Residual x Figure 2.3. Residuals calculated for the same artificial data set as in Figure 2.2. The fitted straight line has been subtracted from the observations so that the residuals may be plotted directly. 8

9 Organizing the Computations Some useful formulae: (xi x) 2 = x 2 i n x2, (yi ȳ) 2 = y 2 i nȳ2, (xi x)(y i ȳ) = x i y i n xȳ, e 2 i = (y i ȳ) 2 { (x i x)y i } 2 (xi x) 2. 9

10 Amherst Example x i = i = 1,..., 162 ( x = 81.5) ˆβ 0 = 7.393, ˆβ 1 =.0116, so the fitted regression line is y i = (x i 81.5) + ϵ i, y i = x i + ϵ i. e 2 i = , s 2 =.4434 =

11 Statistical Properties of the Estimators Under assumptions (a) about ϵ i s: E{ˆβ 0 } = 1 n E { yi } = 1 n {β0 + β 1 (x i x)} = β 0, { } yi (x E{ˆβ 1 } = E i x) (xi x) 2 = {β0 + β 1 (x i x)}(x i x) (xi x) 2 = β 1, V ar{ˆβ 0 } = σ2 n, V ar{ˆβ 1 } = Cov{ˆβ 0, ˆβ 1 } = 0. σ 2 (xi x) 2, 11

12 The estimators of β 0 and β 1 are the best linear unbiased estimators (BLUEs), i.e. out of all estimators that are both linear and unbiased, they have the smallest variance (the Gauss- Markov theorem). We prove this property for ˆβ 1 : Consider another estimator β 1 = c i y i. The choice c i = (x i x)/ (x i x) 2 is the least squares estimator. Unbiasedness implies ci = 0, ci (x i x) = 1. If β 1 ˆβ 1 = c i y i then c i = 0, c i (x i x) = 0. 12

13 The covariance of β 1 ˆβ 1 and ˆβ 1 is (xi x)c i (xi x) 2 σ2 = 0. Therefore, V ar{ β 1 } = V ar{ β 1 ˆβ 1 + ˆβ 1 } = V ar{ β 1 ˆβ 1 } + V ar{ˆβ 1 } V ar{ˆβ 1 }. This proves the result. 13

14 Distribution of s 2 : If we make assumption (b) about the ϵ i s (independent normal) then (n 2)s 2 σ 2 χ 2 n 2. This may be used to construct confidence intervals for σ 2. 14

15 Hypothesis Tests, Confidence Intervals and Prediction Intervals Now make assumptions (b) throughout. Basic distributional results: σ known, ˆβ n 0 β 0 σ (xi x) 2 ˆβ 1 β 1 σ N[0, 1], N[0, 1]. σ unknown, ˆβ n 0 β 0 s (xi x) 2 ˆβ 1 β 1 s t n 2, t n 2. 15

16 Example of Application Test H 0 : β 1 = 0. Reject at level α if (xi x) 2 ˆβ 1 s > t n 2;1 α/2. A 100(1 α)% confidence interval for β 1 may be defined to be s ˆβ 1 ± t n 2;1 α/2 (xi x) 2. The expression for ˆβ 1. s (xi is the standard error x) 2 Tests and confidence intervals for σ 2 : use χ 2 n 2 distribution result given earlier. 16

17 For the Amherst data, we earlier found ˆβ 1 = By applying the above formulae, we find the standard error of ˆβ 1 is.6659 = The t ratio for β 1 is = This is very highly significant. However, for this example, the assumption of independent observations may not be true. We shall see later that they are in fact correlated and this considerably increases the standard error of ˆβ 1, though not to the extent that the data would pass a test that β 1 = 0. 17

18 The prediction problem Given a new experiment to be conducted at x = x, what can we say about the corresponding y = y? Obvious point estimator is ŷ = ˆβ 0 + ˆβ 1 (x x). Gauss-Markov theorem: this is the BLUE for this problem. The right hand side has variance { σ 2 1 n + (x x) 2 } (xi x) 2. The 100(1 α)% confidence interval is ŷ ± t n 2;1 α/2 s 1 n + (x x) 2 (xi x) 2. 18

19 The distinction between confidence intervals and prediction intervals What we have just calculated is a confidence interval for the quantity β 0 + β 1 (x x). It doesn t reflect the full variability that is involved in predicting a specific future observation. Suppose y = β 0 + β 1 (x x) + ϵ where ϵ is a random error (uncorrelated with ϵ 1,.., ϵ n ) with the same mean 0 and variance σ 2. As before, we have ŷ = ˆβ 0 + ˆβ 1 (x x). Therefore, y ŷ = (β 0 ˆβ 0 ) + (β 1 ˆβ 1 )(x x) + ϵ. 19

20 y ŷ = (β 0 ˆβ 0 ) + (β 1 ˆβ 1 )(x x) + ϵ. The three terms on the right hand side are uncorrelated and have total variance { σ 2 1 n + (x x) 2 } (xi x) If σ 2 is known, an exact 100(1 α)% prediction interval for y is given by ŷ ± z 1 α/2 σ 1 n + (x x) 2 (xi x) For unknown σ 2, the corresponding interval is ŷ ± t n 2;1 α/2 s 1 n + (x x) 2 (xi x) Note the extra +1 compared with the earlier confidence intervals. 20

21 Example. Predict the temperature for Amherst in the year 2010 (corresponding to x = 175). We earlier found ˆβ 0 = 7.393, ˆβ 1 =.0116, ( and x = 81.5), so the point prediction is ( )=8.478, and the estimated prediction variance.4434 { ( ) } =.4571 = (.676) 2. The point of the t 160 distribution is 1.975, so the 95% prediction interval is 8.478± = (7.14, 9.81). On the other hand,.4434 { ( ) } = = (.117) 2 so the corresponding 95% confidence interval is (8.25,8.71). 21

22 The amherst.dat file looks like this: year temp

23 R Program for LS regression amh<-read.table(file="amherst.dat",header=t) attach(amh) names(amh) [1] "year" "temp" amh.lm <- lm(temp~year) summary(amh.lm) Call: lm(formula = temp ~ year) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) <2e-16 *** year <2e-16 *** --- Signif. codes: 0 *** ** 0.01 * Residual standard error: on 160 degrees of freedom Multiple R-Squared: ,Adjusted R-squared: F-statistic: on 1 and 160 DF, p-value: < 2.2e-16 23

24 R Program for prediction pr1<-predict(amh.lm,data.frame(year=c(165, 170, 175)), se.fit=t) pr1 $fit $se.fit $df [1] 160 $residual.scale [1]

25 Residuals ith residual defined by e i = y i ˆβ 0 ˆβ 1 (x i x) Possible plots of residuals: (a) e i against x i (b) e i against any omitted covariate (c) plots to examine autocorrelation or normal distribution of residuals Things we are looking for: Outliers Evidence of nonlinearity Evidence of other departures from assumptions, e.g. residuals autocorrelated or having non-constant variance 25

26 (a) (b) Residual 1 0 Residual x x (c) 1.0 Residual x Figure 2.4. Some examples of residual plots illustrating different kinds of discrepancy from a linear model. 26

27 Mount Airy and Amherst examples plot residual against x i, and also a histogram of residuals A Residual Count B Mount Airy Mean Temp Residual Figure 2.5. Plots of residuals from a linear model fitted to Mount Airy Charleston data. (a) Residual versus x value. (b) Histogram of residuals. 27

28 Residual Count Year Residual Figure 2.6. Plots of residuals from a linear model fitted to Amherst data. (a) Residual versus x value. (b) Histogram of residuals. 28

29 A more sophisticated way of deciding whether the residuals are normally distributed is to use a probability plot. There are many variants of this idea, but here we describe one of the simplest versions: 1. Calculate residuals e 1,..., e n, arrange in order, so that e 1 e 2... e n. 2. Define d i = e i /s to standardize residuals to approximately common variance Plot d i against z (i 0.5)/n. (Here z α denotes the α-probability point of the standard normal distribution, i.e. if Z N[0, 1] then Pr{Z z α } = α.) 29

30 (a) (b) Observed value Observed value Expected value Expected value Figure 2.7. Normal probability plot of residuals from (a) Mount Airy Charleston data, (b) Amherst data. 30

31 Formal Tests of Normality (a) The Shapiro-Wilk test, and related tests (b) Tests based on the empirical distribution function (EDF) 31

32 The Shapiro-Wilk Test Simplest version considers only the case y i N[µ, σ 2 ], µ and σ 2 unknown (no regression component) Let e i = y i ȳ, order e 1,..., e n as e 1... e n Let m i be the mean of e i when the y i are i.i.d. N[0, 1]. The idea of the Shapiro-Wilk test is to measure the correlation between the e i s and the m i s. The precise formula used for this is of the form W = ( a i e i )2 (e i ) 2. Tables for the coefficients a i and for the percentage points of W were given by Shapiro and Wilk in their original paper (1965). They are coded into R as shapiro.test(), and SAS as part of PROC UNIVARIATE. 32

33 Variants on the Shapiro-Wilk Test Shapiro-Francia test (1972) used correlation coefficient between e i and m i rather than computing optimal coefficients a i (but the exact computation of m i is still necessary) Looney-Gulledge (1986) simplified Shapiro- Francia by using an approximation to m i : r = z i e i 2 z i e 2 i. Percentage points of r : Computed by Looney and Gulledge in case when original data are N[µ, σ 2 ] Compute directly by simulation in regression context 33

34 Tests based on the EDF Suppose y 1,..., y n are observations. Define the empirical distribution function or EDF: F n (y) = 1 n {Number of observations y}, y (, ). Suppose we believe that the true distribution function is some given F 0 (y). There are a number of test statistics to determine how close F n is to F 0 : D = max y (Kolmogorov-Smirnov), W 2 = n (Cramér von Mises), A 2 = n F n (y) F 0 (y) {F n(y) F 0 (y)} 2 df 0 (y) (Anderson-Darling). {F n (y) F 0 (y)} 2 F 0 (y) {1 F 0 (y)} df 0(y) 34

35 In practice, there are simplifying computational formulae for all three tests (see text for details). Next, we describe how to compute the percentage points of these tests. 35

36 1. If F 0 is completely specified (no unknown parameters), then the percentage points do not depend on F 0 and have been tabulated in many sources (e.g. Biometrika Tables for Statisticians). 2. A more practical situation is something like F 0 (y) = Φ ( ) y µ σ where µ and σ 2 are unknown and Φ is the standard normal d.f. In that case it is usual to estimate µ and σ 2 by the usual sample mean and variance, and to base the EDF tests on the estimated F 0. This is the situation tabulated in PROC UNIVARIATE and many other textbooks and packages. However, even this does not allow for the estimation of regression parameters. 3. A third option is to use simulation directly, including refitting the regression model to each simulated sample. 36

37 Implementing in R Approximate method: First fit regression by lm command, then extract residuals and apply shapiro.test() or ks.test() > shapiro.test(residuals(amh.lm)) Shapiro-Wilk normality test data: residuals(amh.lm) W = , p-value = > ks.test(residuals(amh.lm), pnorm, amh.lm$sig) One-sample Kolmogorov-Smirnov test data: residuals(amh.lm) D = 0.076, p-value = alternative hypothesis: two.sided 37

38 Implementing in R Alternatively, use the code normtest.r (available on the course website) to simulate the exact distribution of all four test statistics. norm.test <- function(y, x, nsim=1000) { ### y: response x: covariates ### Program to perform goodness of fit tests ### for regression data ## The next lines compute the Looney-Gulledge ## statistic c1, the Kolmogorov-Smirnov statistic ## d1, the Cramer-von Mises statistic w1 and the ## Anderson-Darling statistic a1 nreg <- lm(y~x) n <- length(y) q1<-qqnorm(nreg$resid,plot=f) c1<-cor(q1$x,q1$y) u1<-pnorm(sort(nreg$resid),mean=0, sd=summary(nreg)$sigma) d1<-max(c((1:n)/n-u1,u1-(0:(n-1))/n)) w1<-sum((u1-((1:n)-0.5)/n)^2)+1/(12*n) a1<--sum((2*(1:n)-1)*log(u1)+(2*n+1-2*(1:n)) *log(1-u1))/n-n count<-rep(0,4) names(count) <- c("l-g", "K-S", "C-V", "A-D") 38

39 for(j in 1:nsim){ temp1<-rnorm(n) nreg<-lm(temp1~x) q2<-qqnorm(nreg$resid,plot=f) c2<-cor(q2$x,q2$y) if(c2<c1)count[1]<-count[1]+1 u2<-pnorm(sort(nreg$resid),mean=0, sd=summary(nreg)$sigma) d2<-max(c((1:n)/n-u2,u2-(0:(n-1))/n)) w2<-sum((u2-((1:n)-0.5)/n)^2)+1/(12*n) a2<--sum((2*(1:n)-1)*log(u2)+(2*n+1-2*(1:n))*log(1-u2))/n-n if(d2>d1)count[2]<-count[2]+1 if(w2>w1)count[3]<-count[3]+1 if(a2>a1)count[4]<-count[4]+1 } ## return the percentage of times the simulation ## resulted in a more extreme value of the test ## statistic than the one computed from the real data. return(count/nsim) } > source("normtest.r") > norm.test(temp, year) L-G K-S C-V A-D

40 The ANOVA Table Start from the formula (yi a) 2 = (y i ȳ) 2 + n(ȳ a) 2. If y i N[µ, σ 2 ] (i.i.d.) then this leads to the formula ( ) y i µ 2 ( ) y = i ȳ 2 (ȳ ) µ 2 + n σ σ σ or in distributional terms, χ 2 n = χ2 n 1 + χ2 1. (The two terms on the right hand side are independent.) 39

41 The regression set-up (yi ȳ) 2 = {y i ˆβ 0 ˆβ 1 (x i x) + ˆβ 1 (x i x)} 2 = {y i ˆβ 0 ˆβ 1 (x i x)} 2 + ˆβ 1 2 (xi x) 2 since the cross-product vanishes: {yi ˆβ 0 ˆβ 1 (x i x)}(x i x) = 0. Standardize: (yi ȳ) 2 σ 2 = e 2 i (xi x) 2 σ 2. σ 2 + ˆβ 1 2 If β 1 = 0, this has the distributional interpretation χ 2 n 1 = χ2 n 2 + χ2 1 where, again, the two terms on the right hand side are independent. 40

42 The calculations lead us to the ANOVA Table, as follows: SOURCE SS D.F. MS Regression β 1 2 (xi x) 2 1 β 1 2 (xi x) 2 Residual e 2 i n 2 e 2 i /(n 2) Total (yi ȳ) 2 n 1 Table 2.3. Example of an ANOVA table. Testing for a significant regression coefficient: when β 1 = 0, β 2 1 (xi x) 2 e 2 i /(n 2) F 1,n 2. 41

43 SOURCE SS D.F. MS Regression Residual Total Table 2.4. ANOVA table for the Mount Airy Charleston data. F ratio is = 35.0, the 95% and 99.9% points of F 1,47 are and (alternatively: the p-value is ). Clear evidence that the null hypothesis (β 1 = 0) is false. 42

44 SOURCE SS D.F. MS Regression Residual Total Table 2.5. ANOVA table for the Amherst data. F ratio is = 107.6, the 95% and 99.9% points of F 1,160 are 3.90 and 11.24; the p-value is computed by R as 0. Again the test rejects β 1 = 0, but we emphasize again that this assumes all the model assumptions are correct (independent errors, common variance, normal distribution). 43

45 Implementation in R > anova(amh.lm) Analysis of Variance Table Response: temp Df Sum Sq Mean Sq F value Pr(>F) year < 2.2e-16 *** Residuals Signif. codes: 0 *** ** 0.01 *

46 Example Consider the following data: x y x y x y x y y x 45

47 Testing the fit of the linear regression model Is the relationship really linear? This can be tested if there are repeated observations for some of the x values. Suppose there are K < n distinct x values, labelled x 1,..., x K. Also let n k denote the number of observations taken at x k, so that n = K 1 n k, and let the individual observations be y kj, j = 1,..., n k. Define S 1 = β 2 1 S 2 = k S 3 = k S 4 = k S 5 = k k j n k (x k x) 2, {y kj β 0 β 1 (x k x)} 2, n k {ȳ k β 0 β 1 (x k x)} 2, j j (y kj ȳ k ) 2, (y kj ȳ) 2. 46

48 Source SS D.F. MS Regression S 1 1 S 1 Residual S 2 n 2 S 2 /(n 2) Between groups S 3 K 2 S 3 /(K 2) Within groups S 4 n K S 4 /(n K) Total S 5 n 1 Table 2.5. ANOVA table for test of fit Under the null hypothesis that the linear regression model is correct, the ratio of sums of squares F = S 3 K 2 n K S 4 has an F K 2,n K distribution. 47

49 k x k n k ȳ k ŷ k n k (ȳ k ŷ k ) 2 (y kj ȳ k ) Detailed calculations for data of Table 2.7 SOURCE SS D.F. MS Regression Residual Between groups Within groups Total ANOVA table for data of Table 2.7 F = 6.216/1.011 = From tables of the F 3,6 distribution, we find that the upper-α point is 4.76 at α =.05 and 6.60 at α =

50 Implementation in R > lfit <-read.table(file="lfit.dat",header=t) > attach(lfit) > lfit.lm1 <- lm(y~x) > lfit.lm2 <- lm(y~factor(x)) > aov(lfit.lm1, lfit.lm2) Analysis of Variance Table Model 1: y ~ x Model 2: y ~ factor(x) Res.Df RSS Df Sum of Sq F Pr(>F) * --- Signif. codes: 0 *** ** 0.01 *

51 Simultaneous confidence and prediction intervals The simultaneous confidence interval problem may be stated as follows: find G so that, for 1 k K,with probability at least 1 α, each of the inequalities { β 0 + β 1 (x k x)} {β 0 + β 1 (x k x)} G, s 1 n + (x k x)2 (xi x) 2 is satisfied simultaneously. (1) We outline the Bonferroni procedure and the Working-Hotelling procedure. The general Scheffé procedure and its proof left to Ch

52 The Bonferroni procedure Let A k be the event that the inequality (1) is satisfied with x = x k, and let Ā k denote the complementary event. Bonferroni s procedure is based on the elementary inequality and hence Pr{Ā 1 Ā 2... Ā K } K k=1 Pr{Ā k } Pr{A 1 A 2... A K } 1 K k=1 Pr{Ā k }. (2) Therefore, to ensure that the left-hand side of (2) is at least 1 α, it suffices to ensure that Pr{Āk } α. Usually, though not invariably, this is achieved by defining the confidence interval bounds so that Pr{Ā k } α/k for each k. Thus, we may write G = t n 2;1 α/(2k). (3) 51

53 Mount Airy-Charleston Example Find 95% simultaneous confidence intervals for the mean temperature in Charleston corresponding to temperatures x 1 = 20, x 2 = 22.5, x 3 = 25, x 4 = 27.5, x 5 = 30 in Mount Airy. In this case α =.05, K = 5, so G = t 47;.995 = The confidence interval standard errors, defined as s 1 n + (x k x)2 (xi x) 2, are calculated to be.3428,.1161,.1566,.3881,.6260, respectively for x 1,..., x 5. The point predictions, β 0 + β 1 (x x), are , , , , Therefore, the five simultaneous confidence intervals are given by ± = (24.075, ), ± = (26.103, ), ± = (27.414, ), ± = (28.213, ), ± = (28.995, ). 52

54 For prediction intervals, the calculations are the same except that the standard errors for prediction are given by s n + (x k x)2 (xi x) 2, As a point of comparison, for K = 1 we have G = instead of G = the simultaneous confidence or prediction intervals are 33% wider than the one-at-a-time intervals. 53

55 32 Charleston Mount Airy Figure 2.8. Five simultaneous confidence intervals (inner bands, solid lines) and five simultaneous prediction intervals (outer bands, dashed lines) for the Mount Airy Charleston data series. 54

56 The Working-Hotelling procedure In this method, the constant G in (1) is defined by G 2 = 2F 2;n 2;1 α (4) where F ν1 ;ν 2 ;1 α denotes the (1 α) lower probability point of the F distribution with ν 1 degrees of freedom in the numerator and ν 2 degrees of freedom in the denominator. This is a special case of Sheffé s procedure for confidence interval. For Mount Airy and Charleston, F 2,47;.95 = (use qf(0.95,2,47) in R), so G = Note that this G is smaller than the of Bonferroni, so it is preferable to use the Scheffé procedure in this instance. Because this calculation is independent of the number of comparison K, we can use it to calculate simultaneous confidence intervals for all x k at once. An example of this is shown in Fig

57 32 30 Charleston Mount Airy Figure 2.9. Point estimates (straight line, centre) and 95% simultaneous confidence bands (outer curves, dashed) for the mean response at all x values between 20 and 30, as calculated by the Working-Hotelling procedure. 56

58 Inverse Regression and Calibration Earlier we considered the prediction problem: given a new pair (x, y ) and knowing only x, predict y. Now we consider the opposite problem: given y, predict x. Most common context for this problem is calibration, in which x indicates some precise measurement of a quantity and y is a quicker but less precise measurement. The problem is to calibrate the measurement, i.e. to infer x from y, in the most precise way possible. 57

59 First Solution: Inverse Regression Obvious point estimator is ˆx = x + y ˆβ 0 ˆβ 1. Problem of how to get an interval estimate. First note y ˆβ 0 ˆβ 1 (x x) = s (xi x) n + (x x) 2 ˆβ 1 ( ˆx x ) t s 1 n + (x x) 2 n 2. (xi x) An exact 100(1 α)% prediction interval for x would be ˆx s ± t 1 n 2;1 α/2 ˆβ 1 n + (x x) 2 (xi x)

60 Unfortunately this is not realizable because the width of the interval depends on x, which is unknown. As a practical approximation, however, we can estimate this by ˆx, which leads to the interval ˆx s ± t 1 n 2;1 α/2 ˆβ 1 n + ( ˆx x) 2 (xi x) The method is usually called inverse regression 59

61 Second Solution: Direct Regression However there is another solution which seems even easier to adopt just interchange the role of x and y everywhere, i.e. treat the original regression problem as a regression of x i on y i, then the inverse regression problem becomes a direct prediction problem for which the earlier solution (Section 2.4) is applicable. What s wrong with that? 60

62 Third Solution: Exact Solution of the Inverse Regression Problem ˆβ 1 ( ˆx x ) t s 1 n + (x x) 2 n 2. (xi x) Define c = t n 2;1 α/2. With probability α, ˆβ 1 ( ˆx x ) s 1 n + (x x) 2 (xi x) c. The boundary points for x occur when { ˆβ 1 2 ( ˆx x ) 2 = c 2 s 2 1 n + (x x) 2 } (xi x) With luck, this equation will have two real roots (for x ) and the interval between them will be our desired interval. 61

63 Residual 1.5 B A Count Charleston Mean Temp Residual Figure Plots of residuals from a linear model fitted to direct regression of Charleston temperatures on Mount Airy temperatures. (a) Residual versus x value. (b) Histogram of residuals. 62

64 Residual Count Mean Temperature Residual Figure Plots of residuals from a linear model fitted to time vs. Amherst mean temperatures. (a) Residual versus x value. (b) Histogram of residuals. 63

65 (a) (b) Observed value Observed value Expected value Expected value Figure Normal probability plot of residuals from (a) Charleston Mount Airy data, (b) Amherst data, both fitted by the direct method whereby the original roles of x and y are reversed. 64

66 Amherst Example Predict x corresponding to y = 10. Inverse regression method leads to ˆx = and the approximate 95% prediction interval (185.6,426.8). Direct regression method leads to ˆx = and 95% prediction interval (98.3,245.4). Exact inverse method leads to (191.4,437.6). Conclusion: The exact and approximate inverse methods are not much different (despite huge standard error), but the direct method leads to a completely different solution. 65

67 Charleston II I Mount Airy Figure Plot of mean summer temperature in Charleston (y) against mean summer temperature in Mount Airy (x), together with the regression lines for y on x (I) and for x on y (II). 66

68 Formulate in terms of the bivariate normal distribution. Suppose X N(µ X, σ 2 X ), Y N(µ Y, σ 2 Y ), and that the correlation coefficient is ρ, in other words, Density: E{(X µ X )(Y µ Y )} = ρσ X σ Y. = f X,Y (x, y) ( 1 exp 1 x µx 2πσ X σ Y 1 ρ 2 2(1 ρ 2 ) σ X ( ) 2 ( ) ( ) y µy x µx y µy + 2ρ. σ Y σ X σ Y ) 2 Compute the conditional density of Y X = x using f Y X (y x) = f X,Y (x, y). f X (x) given 67

69 f Y X (y x) {( 1 y µy exp 2(1 ρ 2 ) 1 = exp 2σY 2 (1 ρ2 ) { σ Y ) ρ ( x µx σ X )} 2 y µ Y ρσ Y σ X (x µ X ) This is the density of a normal distribution with mean µ Y + ρσ Y σ X (x µ X ) and variance σ 2 Y (1 ρ2 ). In practice, do not know µ X, µ Y, etc., estimate by µ X = x, σ 2 X = s 2 (xi X = x) 2, n 1 µ Y = ȳ, σ 2 Y = s 2 (yi Y = ȳ) 2, n 1 (xi x)(y ρ = r = i ȳ) (xi x) 2 (y i ȳ) 2. } 2. 68

70 Now, suppose we have a new pair (x, y ), but we only observe x, and we want to predict the unknown value y. Based on the conditional distribution, we estimate the conditional mean of Y given X = x to be µ Y + ρ σ Y σ X (x µ X ) = β 0 + β 1 (x x). Estimated conditional variance turns out to be n 2 n 1 s2. Almost the same as the original regression formulae, which treated the X s as fixed constants, not random variables. 69

71 Moral of the story... If both variables are random, predicting one of them giving the other is (almost) the same as treating the conditioning variable as fixed constants. However, if one variable truly is non-random (as when it is time ), this analogy doesn t help. For predicting X given Y, direct regression bases the prediction on f X Y (x y) f X (x)f Y X (y x). while inverse regression uses only f Y X (y x). The difference is in the existence of a prior distribution f X (x). 70

72 Inference about the correlation coefficient How to obtain tests and confidence intervals for the correlation coefficient ρ? Fisher s z-transformation (Fisher 1921, Gayen 1951) Transform r into one which is approximately normal: where z = 1 2 log ( 1 + r 1 r ) N ( ζ, ζ = 1 ( ) 1 + ρ 2 log. 1 ρ 1 n 3 ), (5) Example. For the Mount Airy Charleston data set, we find r =.653 and so z =.781 by (5). An approximate 95% confidence interval for ζ is.781 ± 1.96 = (.492, 1.070). Transforming 46 back, we find that a 95% confidence interval for ρ is (.456,.789). 71

73 Autocorrelation Corr{ϵ i, ϵ i+k } = E(ϵ iϵ i+k ) σ 2 = ρ ϵ (k). Assume this does not depend on i (stationarity assumption) Estimate ρ ϵ (k) by r ϵ (k) = n k i=1 e ie i+k ni=1 e 2 i This is considered significant if r ϵ (k) > 2 n. 72

74 Example. Amherst: first 10 sample autocorrelations of the residuals are.185,.156,.065,.002,.001,.047,.023,.079,.023,.065. Here n = 162, 2/ n =.157, so it looks as though the first two autocorrelations are statistically significant. The remaining autocorrelations for k > 2 are very small, and in addition there is no evidence from Fig. 2.6 of changing variance, so we may assume the series to be stationary. Charleston/Mount Airy: first 10 sample autocorrelations.016,.009,.315,.041,.010,.010,.204,.064,.047,.065. In this case, with n = 49, 2/ n =.286. Only k = 3 is significant. 73

75 Corrleation Lag Figure First 10 serial correlations for Amherst data, with approximate 95% error bounds if the true series is independent. 74

76 Corrleation Lag Figure First 10 serial correlations for Mount Airy Charleston data, with approximate 95% error bounds if the true series is independent. 75

77 The effect of autocorrelation on the regression estimates Suppose we decide that a series is indeed autocorrelated, and assume that it is stationary with at least the first few autocorrelations ρ ϵ (k) estimated by the sample autocorrelations r ϵ (k). How should this affect our regression analysis? Suppose we apply standard ordinary least squares analysis and compute estimators β 0 and β 1. The main effect of correlation is that the standard error formulae for these estimators are wrong. 76

78 Variance of β 0 = ȳ: We have Var(ȳ) = 1 n 2 n = σ2 n 2 n i=1 j=1 n n i=1 j=1 Cov(y i, y j ) ρ ϵ ( i j ) = σ2 n 2{n + 2(n 1)ρ ϵ(1) + 2(n 2)ρ ϵ (2) ρ ϵ (n 1)} = σ2 n { ( 1 1 n ) n ρ ϵ(n 1) ρ ϵ (1) }. 77

79 Variance of β 1 : { } (xi x)(y Var i ȳ) (xi x) 2 = = = where σ 2 { (x i x) 2 } 2 σ 2 (xi x) σ 2 (xi x) 2 n 1 k=1 ρ X (k) = r X (k) = n n i=1 j=1 n k n (x i x)(x j x)ρ ϵ ( i j ) i=1 (x i x)(x i+k x) ni=1 (x i x) 2 k=1 ρ ϵ (k)ρ X (k) n k i=1 (x i x)(x i+k x) ni=1 (x i x) 2. ρ ϵ(k) 78

80 In practice, we will estimate r ϵ (k) for k up to some cutoff K, and estimate the variances of β 0 and β 1 by and s 2 n s 2 (xi x) K k=1 ( 1 k ) n K k=1 r ϵ (k) r ϵ (k)r X (k). 79

81 K Char. β 0 Char. β 1 Amh. β 0 Amh. β Table Corrected standard errors assuming autocorrelation for the Charleston and Amherst data for several different values of the largest lag K 80

Ch 2: Simple Linear Regression

Ch 2: Simple Linear Regression Ch 2: Simple Linear Regression 1. Simple Linear Regression Model A simple regression model with a single regressor x is y = β 0 + β 1 x + ɛ, where we assume that the error ɛ is independent random component

More information

13 Simple Linear Regression

13 Simple Linear Regression B.Sc./Cert./M.Sc. Qualif. - Statistics: Theory and Practice 3 Simple Linear Regression 3. An industrial example A study was undertaken to determine the effect of stirring rate on the amount of impurity

More information

Simple Linear Regression

Simple Linear Regression Simple Linear Regression ST 430/514 Recall: A regression model describes how a dependent variable (or response) Y is affected, on average, by one or more independent variables (or factors, or covariates)

More information

Simple Linear Regression

Simple Linear Regression Simple Linear Regression In simple linear regression we are concerned about the relationship between two variables, X and Y. There are two components to such a relationship. 1. The strength of the relationship.

More information

Stat 427/527: Advanced Data Analysis I

Stat 427/527: Advanced Data Analysis I Stat 427/527: Advanced Data Analysis I Review of Chapters 1-4 Sep, 2017 1 / 18 Concepts you need to know/interpret Numerical summaries: measures of center (mean, median, mode) measures of spread (sample

More information

SAS Procedures Inference about the Line ffl model statement in proc reg has many options ffl To construct confidence intervals use alpha=, clm, cli, c

SAS Procedures Inference about the Line ffl model statement in proc reg has many options ffl To construct confidence intervals use alpha=, clm, cli, c Inference About the Slope ffl As with all estimates, ^fi1 subject to sampling var ffl Because Y jx _ Normal, the estimate ^fi1 _ Normal A linear combination of indep Normals is Normal Simple Linear Regression

More information

Answer Keys to Homework#10

Answer Keys to Homework#10 Answer Keys to Homework#10 Problem 1 Use either restricted or unrestricted mixed models. Problem 2 (a) First, the respective means for the 8 level combinations are listed in the following table A B C Mean

More information

Assignment 9 Answer Keys

Assignment 9 Answer Keys Assignment 9 Answer Keys Problem 1 (a) First, the respective means for the 8 level combinations are listed in the following table A B C Mean 26.00 + 34.67 + 39.67 + + 49.33 + 42.33 + + 37.67 + + 54.67

More information

Inference for Regression

Inference for Regression Inference for Regression Section 9.4 Cathy Poliak, Ph.D. cathy@math.uh.edu Office in Fleming 11c Department of Mathematics University of Houston Lecture 13b - 3339 Cathy Poliak, Ph.D. cathy@math.uh.edu

More information

Master s Written Examination

Master s Written Examination Master s Written Examination Option: Statistics and Probability Spring 05 Full points may be obtained for correct answers to eight questions Each numbered question (which may have several parts) is worth

More information

Linear Regression. In this problem sheet, we consider the problem of linear regression with p predictors and one intercept,

Linear Regression. In this problem sheet, we consider the problem of linear regression with p predictors and one intercept, Linear Regression In this problem sheet, we consider the problem of linear regression with p predictors and one intercept, y = Xβ + ɛ, where y t = (y 1,..., y n ) is the column vector of target values,

More information

Section 4.6 Simple Linear Regression

Section 4.6 Simple Linear Regression Section 4.6 Simple Linear Regression Objectives ˆ Basic philosophy of SLR and the regression assumptions ˆ Point & interval estimation of the model parameters, and how to make predictions ˆ Point and interval

More information

MS&E 226: Small Data

MS&E 226: Small Data MS&E 226: Small Data Lecture 15: Examples of hypothesis tests (v5) Ramesh Johari ramesh.johari@stanford.edu 1 / 32 The recipe 2 / 32 The hypothesis testing recipe In this lecture we repeatedly apply the

More information

Multiple Linear Regression

Multiple Linear Regression Multiple Linear Regression ST 430/514 Recall: a regression model describes how a dependent variable (or response) Y is affected, on average, by one or more independent variables (or factors, or covariates).

More information

Ch 3: Multiple Linear Regression

Ch 3: Multiple Linear Regression Ch 3: Multiple Linear Regression 1. Multiple Linear Regression Model Multiple regression model has more than one regressor. For example, we have one response variable and two regressor variables: 1. delivery

More information

Lecture 1: Linear Models and Applications

Lecture 1: Linear Models and Applications Lecture 1: Linear Models and Applications Claudia Czado TU München c (Claudia Czado, TU Munich) ZFS/IMS Göttingen 2004 0 Overview Introduction to linear models Exploratory data analysis (EDA) Estimation

More information

Correlation and the Analysis of Variance Approach to Simple Linear Regression

Correlation and the Analysis of Variance Approach to Simple Linear Regression Correlation and the Analysis of Variance Approach to Simple Linear Regression Biometry 755 Spring 2009 Correlation and the Analysis of Variance Approach to Simple Linear Regression p. 1/35 Correlation

More information

Lecture 2: Basic Concepts and Simple Comparative Experiments Montgomery: Chapter 2

Lecture 2: Basic Concepts and Simple Comparative Experiments Montgomery: Chapter 2 Lecture 2: Basic Concepts and Simple Comparative Experiments Montgomery: Chapter 2 Fall, 2013 Page 1 Random Variable and Probability Distribution Discrete random variable Y : Finite possible values {y

More information

Introduction and Single Predictor Regression. Correlation

Introduction and Single Predictor Regression. Correlation Introduction and Single Predictor Regression Dr. J. Kyle Roberts Southern Methodist University Simmons School of Education and Human Development Department of Teaching and Learning Correlation A correlation

More information

Module 6: Model Diagnostics

Module 6: Model Diagnostics St@tmaster 02429/MIXED LINEAR MODELS PREPARED BY THE STATISTICS GROUPS AT IMM, DTU AND KU-LIFE Module 6: Model Diagnostics 6.1 Introduction............................... 1 6.2 Linear model diagnostics........................

More information

Circle the single best answer for each multiple choice question. Your choice should be made clearly.

Circle the single best answer for each multiple choice question. Your choice should be made clearly. TEST #1 STA 4853 March 6, 2017 Name: Please read the following directions. DO NOT TURN THE PAGE UNTIL INSTRUCTED TO DO SO Directions This exam is closed book and closed notes. There are 32 multiple choice

More information

Regression and the 2-Sample t

Regression and the 2-Sample t Regression and the 2-Sample t James H. Steiger Department of Psychology and Human Development Vanderbilt University James H. Steiger (Vanderbilt University) Regression and the 2-Sample t 1 / 44 Regression

More information

MODULE 4 SIMPLE LINEAR REGRESSION

MODULE 4 SIMPLE LINEAR REGRESSION MODULE 4 SIMPLE LINEAR REGRESSION Module Objectives: 1. Describe the equation of a line including the meanings of the two parameters. 2. Describe how the best-fit line to a set of bivariate data is derived.

More information

Lecture 15. Hypothesis testing in the linear model

Lecture 15. Hypothesis testing in the linear model 14. Lecture 15. Hypothesis testing in the linear model Lecture 15. Hypothesis testing in the linear model 1 (1 1) Preliminary lemma 15. Hypothesis testing in the linear model 15.1. Preliminary lemma Lemma

More information

Homework 2: Simple Linear Regression

Homework 2: Simple Linear Regression STAT 4385 Applied Regression Analysis Homework : Simple Linear Regression (Simple Linear Regression) Thirty (n = 30) College graduates who have recently entered the job market. For each student, the CGPA

More information

unadjusted model for baseline cholesterol 22:31 Monday, April 19,

unadjusted model for baseline cholesterol 22:31 Monday, April 19, unadjusted model for baseline cholesterol 22:31 Monday, April 19, 2004 1 Class Level Information Class Levels Values TRETGRP 3 3 4 5 SEX 2 0 1 Number of observations 916 unadjusted model for baseline cholesterol

More information

STAT5044: Regression and Anova

STAT5044: Regression and Anova STAT5044: Regression and Anova Inyoung Kim 1 / 49 Outline 1 How to check assumptions 2 / 49 Assumption Linearity: scatter plot, residual plot Randomness: Run test, Durbin-Watson test when the data can

More information

Regression Models - Introduction

Regression Models - Introduction Regression Models - Introduction In regression models there are two types of variables that are studied: A dependent variable, Y, also called response variable. It is modeled as random. An independent

More information

3. Linear Regression With a Single Regressor

3. Linear Regression With a Single Regressor 3. Linear Regression With a Single Regressor Econometrics: (I) Application of statistical methods in empirical research Testing economic theory with real-world data (data analysis) 56 Econometrics: (II)

More information

Chapter 8 (More on Assumptions for the Simple Linear Regression)

Chapter 8 (More on Assumptions for the Simple Linear Regression) EXST3201 Chapter 8b Geaghan Fall 2005: Page 1 Chapter 8 (More on Assumptions for the Simple Linear Regression) Your textbook considers the following assumptions: Linearity This is not something I usually

More information

IES 612/STA 4-573/STA Winter 2008 Week 1--IES 612-STA STA doc

IES 612/STA 4-573/STA Winter 2008 Week 1--IES 612-STA STA doc IES 612/STA 4-573/STA 4-576 Winter 2008 Week 1--IES 612-STA 4-573-STA 4-576.doc Review Notes: [OL] = Ott & Longnecker Statistical Methods and Data Analysis, 5 th edition. [Handouts based on notes prepared

More information

Math 423/533: The Main Theoretical Topics

Math 423/533: The Main Theoretical Topics Math 423/533: The Main Theoretical Topics Notation sample size n, data index i number of predictors, p (p = 2 for simple linear regression) y i : response for individual i x i = (x i1,..., x ip ) (1 p)

More information

Simple Linear Regression

Simple Linear Regression Simple Linear Regression MATH 282A Introduction to Computational Statistics University of California, San Diego Instructor: Ery Arias-Castro http://math.ucsd.edu/ eariasca/math282a.html MATH 282A University

More information

STAT420 Midterm Exam. University of Illinois Urbana-Champaign October 19 (Friday), :00 4:15p. SOLUTIONS (Yellow)

STAT420 Midterm Exam. University of Illinois Urbana-Champaign October 19 (Friday), :00 4:15p. SOLUTIONS (Yellow) STAT40 Midterm Exam University of Illinois Urbana-Champaign October 19 (Friday), 018 3:00 4:15p SOLUTIONS (Yellow) Question 1 (15 points) (10 points) 3 (50 points) extra ( points) Total (77 points) Points

More information

STAT 4385 Topic 03: Simple Linear Regression

STAT 4385 Topic 03: Simple Linear Regression STAT 4385 Topic 03: Simple Linear Regression Xiaogang Su, Ph.D. Department of Mathematical Science University of Texas at El Paso xsu@utep.edu Spring, 2017 Outline The Set-Up Exploratory Data Analysis

More information

STAT5044: Regression and Anova. Inyoung Kim

STAT5044: Regression and Anova. Inyoung Kim STAT5044: Regression and Anova Inyoung Kim 2 / 47 Outline 1 Regression 2 Simple Linear regression 3 Basic concepts in regression 4 How to estimate unknown parameters 5 Properties of Least Squares Estimators:

More information

Chapter 7: Simple linear regression

Chapter 7: Simple linear regression The absolute movement of the ground and buildings during an earthquake is small even in major earthquakes. The damage that a building suffers depends not upon its displacement, but upon the acceleration.

More information

Lectures on Simple Linear Regression Stat 431, Summer 2012

Lectures on Simple Linear Regression Stat 431, Summer 2012 Lectures on Simple Linear Regression Stat 43, Summer 0 Hyunseung Kang July 6-8, 0 Last Updated: July 8, 0 :59PM Introduction Previously, we have been investigating various properties of the population

More information

(a) The percentage of variation in the response is given by the Multiple R-squared, which is 52.67%.

(a) The percentage of variation in the response is given by the Multiple R-squared, which is 52.67%. STOR 664 Homework 2 Solution Part A Exercise (Faraway book) Ch2 Ex1 > data(teengamb) > attach(teengamb) > tgl summary(tgl) Coefficients: Estimate Std Error t value

More information

Chapter 3: Multiple Regression. August 14, 2018

Chapter 3: Multiple Regression. August 14, 2018 Chapter 3: Multiple Regression August 14, 2018 1 The multiple linear regression model The model y = β 0 +β 1 x 1 + +β k x k +ǫ (1) is called a multiple linear regression model with k regressors. The parametersβ

More information

Chapter 3: Regression Methods for Trends

Chapter 3: Regression Methods for Trends Chapter 3: Regression Methods for Trends Time series exhibiting trends over time have a mean function that is some simple function (not necessarily constant) of time. The example random walk graph from

More information

Regression Models - Introduction

Regression Models - Introduction Regression Models - Introduction In regression models, two types of variables that are studied: A dependent variable, Y, also called response variable. It is modeled as random. An independent variable,

More information

36-707: Regression Analysis Homework Solutions. Homework 3

36-707: Regression Analysis Homework Solutions. Homework 3 36-707: Regression Analysis Homework Solutions Homework 3 Fall 2012 Problem 1 Y i = βx i + ɛ i, i {1, 2,..., n}. (a) Find the LS estimator of β: RSS = Σ n i=1(y i βx i ) 2 RSS β = Σ n i=1( 2X i )(Y i βx

More information

Regression. Marc H. Mehlman University of New Haven

Regression. Marc H. Mehlman University of New Haven Regression Marc H. Mehlman marcmehlman@yahoo.com University of New Haven the statistician knows that in nature there never was a normal distribution, there never was a straight line, yet with normal and

More information

Lecture 3. Experiments with a Single Factor: ANOVA Montgomery 3.1 through 3.3

Lecture 3. Experiments with a Single Factor: ANOVA Montgomery 3.1 through 3.3 Lecture 3. Experiments with a Single Factor: ANOVA Montgomery 3.1 through 3.3 Fall, 2013 Page 1 Tensile Strength Experiment Investigate the tensile strength of a new synthetic fiber. The factor is the

More information

1) Answer the following questions as true (T) or false (F) by circling the appropriate letter.

1) Answer the following questions as true (T) or false (F) by circling the appropriate letter. 1) Answer the following questions as true (T) or false (F) by circling the appropriate letter. T F T F T F a) Variance estimates should always be positive, but covariance estimates can be either positive

More information

Statistics for Engineers Lecture 9 Linear Regression

Statistics for Engineers Lecture 9 Linear Regression Statistics for Engineers Lecture 9 Linear Regression Chong Ma Department of Statistics University of South Carolina chongm@email.sc.edu April 17, 2017 Chong Ma (Statistics, USC) STAT 509 Spring 2017 April

More information

Density Temp vs Ratio. temp

Density Temp vs Ratio. temp Temp Ratio Density 0.00 0.02 0.04 0.06 0.08 0.10 0.12 Density 0.0 0.2 0.4 0.6 0.8 1.0 1. (a) 170 175 180 185 temp 1.0 1.5 2.0 2.5 3.0 ratio The histogram shows that the temperature measures have two peaks,

More information

Overview Scatter Plot Example

Overview Scatter Plot Example Overview Topic 22 - Linear Regression and Correlation STAT 5 Professor Bruce Craig Consider one population but two variables For each sampling unit observe X and Y Assume linear relationship between variables

More information

1 Introduction 1. 2 The Multiple Regression Model 1

1 Introduction 1. 2 The Multiple Regression Model 1 Multiple Linear Regression Contents 1 Introduction 1 2 The Multiple Regression Model 1 3 Setting Up a Multiple Regression Model 2 3.1 Introduction.............................. 2 3.2 Significance Tests

More information

Linear Regression. In this lecture we will study a particular type of regression model: the linear regression model

Linear Regression. In this lecture we will study a particular type of regression model: the linear regression model 1 Linear Regression 2 Linear Regression In this lecture we will study a particular type of regression model: the linear regression model We will first consider the case of the model with one predictor

More information

COMPREHENSIVE WRITTEN EXAMINATION, PAPER III FRIDAY AUGUST 26, 2005, 9:00 A.M. 1:00 P.M. STATISTICS 174 QUESTION

COMPREHENSIVE WRITTEN EXAMINATION, PAPER III FRIDAY AUGUST 26, 2005, 9:00 A.M. 1:00 P.M. STATISTICS 174 QUESTION COMPREHENSIVE WRITTEN EXAMINATION, PAPER III FRIDAY AUGUST 26, 2005, 9:00 A.M. 1:00 P.M. STATISTICS 174 QUESTION Answer all parts. Closed book, calculators allowed. It is important to show all working,

More information

Inference in Regression Analysis

Inference in Regression Analysis Inference in Regression Analysis Dr. Frank Wood Frank Wood, fwood@stat.columbia.edu Linear Regression Models Lecture 4, Slide 1 Today: Normal Error Regression Model Y i = β 0 + β 1 X i + ǫ i Y i value

More information

Applied Econometrics (QEM)

Applied Econometrics (QEM) Applied Econometrics (QEM) based on Prinicples of Econometrics Jakub Mućk Department of Quantitative Economics Jakub Mućk Applied Econometrics (QEM) Meeting #3 1 / 42 Outline 1 2 3 t-test P-value Linear

More information

STATISTICS 174: APPLIED STATISTICS FINAL EXAM DECEMBER 10, 2002

STATISTICS 174: APPLIED STATISTICS FINAL EXAM DECEMBER 10, 2002 Time allowed: 3 HOURS. STATISTICS 174: APPLIED STATISTICS FINAL EXAM DECEMBER 10, 2002 This is an open book exam: all course notes and the text are allowed, and you are expected to use your own calculator.

More information

Statement: With my signature I confirm that the solutions are the product of my own work. Name: Signature:.

Statement: With my signature I confirm that the solutions are the product of my own work. Name: Signature:. MATHEMATICAL STATISTICS Homework assignment Instructions Please turn in the homework with this cover page. You do not need to edit the solutions. Just make sure the handwriting is legible. You may discuss

More information

Chapter 2 Inferences in Simple Linear Regression

Chapter 2 Inferences in Simple Linear Regression STAT 525 SPRING 2018 Chapter 2 Inferences in Simple Linear Regression Professor Min Zhang Testing for Linear Relationship Term β 1 X i defines linear relationship Will then test H 0 : β 1 = 0 Test requires

More information

Summer School in Statistics for Astronomers V June 1 - June 6, Regression. Mosuk Chow Statistics Department Penn State University.

Summer School in Statistics for Astronomers V June 1 - June 6, Regression. Mosuk Chow Statistics Department Penn State University. Summer School in Statistics for Astronomers V June 1 - June 6, 2009 Regression Mosuk Chow Statistics Department Penn State University. Adapted from notes prepared by RL Karandikar Mean and variance Recall

More information

9 Correlation and Regression

9 Correlation and Regression 9 Correlation and Regression SW, Chapter 12. Suppose we select n = 10 persons from the population of college seniors who plan to take the MCAT exam. Each takes the test, is coached, and then retakes the

More information

10. Alternative case influence statistics

10. Alternative case influence statistics 10. Alternative case influence statistics a. Alternative to D i : dffits i (and others) b. Alternative to studres i : externally-studentized residual c. Suggestion: use whatever is convenient with the

More information

BIOS 2083 Linear Models c Abdus S. Wahed

BIOS 2083 Linear Models c Abdus S. Wahed Chapter 5 206 Chapter 6 General Linear Model: Statistical Inference 6.1 Introduction So far we have discussed formulation of linear models (Chapter 1), estimability of parameters in a linear model (Chapter

More information

Lecture notes on Regression & SAS example demonstration

Lecture notes on Regression & SAS example demonstration Regression & Correlation (p. 215) When two variables are measured on a single experimental unit, the resulting data are called bivariate data. You can describe each variable individually, and you can also

More information

Unit 10: Simple Linear Regression and Correlation

Unit 10: Simple Linear Regression and Correlation Unit 10: Simple Linear Regression and Correlation Statistics 571: Statistical Methods Ramón V. León 6/28/2004 Unit 10 - Stat 571 - Ramón V. León 1 Introductory Remarks Regression analysis is a method for

More information

1 Multiple Regression

1 Multiple Regression 1 Multiple Regression In this section, we extend the linear model to the case of several quantitative explanatory variables. There are many issues involved in this problem and this section serves only

More information

Chapter 1: Linear Regression with One Predictor Variable also known as: Simple Linear Regression Bivariate Linear Regression

Chapter 1: Linear Regression with One Predictor Variable also known as: Simple Linear Regression Bivariate Linear Regression BSTT523: Kutner et al., Chapter 1 1 Chapter 1: Linear Regression with One Predictor Variable also known as: Simple Linear Regression Bivariate Linear Regression Introduction: Functional relation between

More information

Statistics - Lecture Three. Linear Models. Charlotte Wickham 1.

Statistics - Lecture Three. Linear Models. Charlotte Wickham   1. Statistics - Lecture Three Charlotte Wickham wickham@stat.berkeley.edu http://www.stat.berkeley.edu/~wickham/ Linear Models 1. The Theory 2. Practical Use 3. How to do it in R 4. An example 5. Extensions

More information

Fall 2017 STAT 532 Homework Peter Hoff. 1. Let P be a probability measure on a collection of sets A.

Fall 2017 STAT 532 Homework Peter Hoff. 1. Let P be a probability measure on a collection of sets A. 1. Let P be a probability measure on a collection of sets A. (a) For each n N, let H n be a set in A such that H n H n+1. Show that P (H n ) monotonically converges to P ( k=1 H k) as n. (b) For each n

More information

Estadística II Chapter 4: Simple linear regression

Estadística II Chapter 4: Simple linear regression Estadística II Chapter 4: Simple linear regression Chapter 4. Simple linear regression Contents Objectives of the analysis. Model specification. Least Square Estimators (LSE): construction and properties

More information

Econometrics I KS. Module 2: Multivariate Linear Regression. Alexander Ahammer. This version: April 16, 2018

Econometrics I KS. Module 2: Multivariate Linear Regression. Alexander Ahammer. This version: April 16, 2018 Econometrics I KS Module 2: Multivariate Linear Regression Alexander Ahammer Department of Economics Johannes Kepler University of Linz This version: April 16, 2018 Alexander Ahammer (JKU) Module 2: Multivariate

More information

Lectures 5 & 6: Hypothesis Testing

Lectures 5 & 6: Hypothesis Testing Lectures 5 & 6: Hypothesis Testing in which you learn to apply the concept of statistical significance to OLS estimates, learn the concept of t values, how to use them in regression work and come across

More information

R 2 and F -Tests and ANOVA

R 2 and F -Tests and ANOVA R 2 and F -Tests and ANOVA December 6, 2018 1 Partition of Sums of Squares The distance from any point y i in a collection of data, to the mean of the data ȳ, is the deviation, written as y i ȳ. Definition.

More information

Simple Regression Model Setup Estimation Inference Prediction. Model Diagnostic. Multiple Regression. Model Setup and Estimation.

Simple Regression Model Setup Estimation Inference Prediction. Model Diagnostic. Multiple Regression. Model Setup and Estimation. Statistical Computation Math 475 Jimin Ding Department of Mathematics Washington University in St. Louis www.math.wustl.edu/ jmding/math475/index.html October 10, 2013 Ridge Part IV October 10, 2013 1

More information

Chapter 1 Linear Regression with One Predictor

Chapter 1 Linear Regression with One Predictor STAT 525 FALL 2018 Chapter 1 Linear Regression with One Predictor Professor Min Zhang Goals of Regression Analysis Serve three purposes Describes an association between X and Y In some applications, the

More information

Matrices and vectors A matrix is a rectangular array of numbers. Here s an example: A =

Matrices and vectors A matrix is a rectangular array of numbers. Here s an example: A = Matrices and vectors A matrix is a rectangular array of numbers Here s an example: 23 14 17 A = 225 0 2 This matrix has dimensions 2 3 The number of rows is first, then the number of columns We can write

More information

F3: Classical normal linear rgression model distribution, interval estimation and hypothesis testing

F3: Classical normal linear rgression model distribution, interval estimation and hypothesis testing F3: Classical normal linear rgression model distribution, interval estimation and hypothesis testing Feng Li Department of Statistics, Stockholm University What we have learned last time... 1 Estimating

More information

Simple linear regression

Simple linear regression Simple linear regression Biometry 755 Spring 2008 Simple linear regression p. 1/40 Overview of regression analysis Evaluate relationship between one or more independent variables (X 1,...,X k ) and a single

More information

INTERVAL ESTIMATION AND HYPOTHESES TESTING

INTERVAL ESTIMATION AND HYPOTHESES TESTING INTERVAL ESTIMATION AND HYPOTHESES TESTING 1. IDEA An interval rather than a point estimate is often of interest. Confidence intervals are thus important in empirical work. To construct interval estimates,

More information

401 Review. 6. Power analysis for one/two-sample hypothesis tests and for correlation analysis.

401 Review. 6. Power analysis for one/two-sample hypothesis tests and for correlation analysis. 401 Review Major topics of the course 1. Univariate analysis 2. Bivariate analysis 3. Simple linear regression 4. Linear algebra 5. Multiple regression analysis Major analysis methods 1. Graphical analysis

More information

Simple Linear Regression

Simple Linear Regression Simple Linear Regression EdPsych 580 C.J. Anderson Fall 2005 Simple Linear Regression p. 1/80 Outline 1. What it is and why it s useful 2. How 3. Statistical Inference 4. Examining assumptions (diagnostics)

More information

Estimating σ 2. We can do simple prediction of Y and estimation of the mean of Y at any value of X.

Estimating σ 2. We can do simple prediction of Y and estimation of the mean of Y at any value of X. Estimating σ 2 We can do simple prediction of Y and estimation of the mean of Y at any value of X. To perform inferences about our regression line, we must estimate σ 2, the variance of the error term.

More information

Formal Statement of Simple Linear Regression Model

Formal Statement of Simple Linear Regression Model Formal Statement of Simple Linear Regression Model Y i = β 0 + β 1 X i + ɛ i Y i value of the response variable in the i th trial β 0 and β 1 are parameters X i is a known constant, the value of the predictor

More information

Nature vs. nurture? Lecture 18 - Regression: Inference, Outliers, and Intervals. Regression Output. Conditions for inference.

Nature vs. nurture? Lecture 18 - Regression: Inference, Outliers, and Intervals. Regression Output. Conditions for inference. Understanding regression output from software Nature vs. nurture? Lecture 18 - Regression: Inference, Outliers, and Intervals In 1966 Cyril Burt published a paper called The genetic determination of differences

More information

Chapter 1. Linear Regression with One Predictor Variable

Chapter 1. Linear Regression with One Predictor Variable Chapter 1. Linear Regression with One Predictor Variable 1.1 Statistical Relation Between Two Variables To motivate statistical relationships, let us consider a mathematical relation between two mathematical

More information

Coefficient of Determination

Coefficient of Determination Coefficient of Determination ST 430/514 The coefficient of determination, R 2, is defined as before: R 2 = 1 SS E (yi ŷ i ) = 1 2 SS yy (yi ȳ) 2 The interpretation of R 2 is still the fraction of variance

More information

Lab 3 A Quick Introduction to Multiple Linear Regression Psychology The Multiple Linear Regression Model

Lab 3 A Quick Introduction to Multiple Linear Regression Psychology The Multiple Linear Regression Model Lab 3 A Quick Introduction to Multiple Linear Regression Psychology 310 Instructions.Work through the lab, saving the output as you go. You will be submitting your assignment as an R Markdown document.

More information

Multivariate Linear Regression Models

Multivariate Linear Regression Models Multivariate Linear Regression Models Regression analysis is used to predict the value of one or more responses from a set of predictors. It can also be used to estimate the linear association between

More information

Inference for the Regression Coefficient

Inference for the Regression Coefficient Inference for the Regression Coefficient Recall, b 0 and b 1 are the estimates of the slope β 1 and intercept β 0 of population regression line. We can shows that b 0 and b 1 are the unbiased estimates

More information

General Linear Model: Statistical Inference

General Linear Model: Statistical Inference Chapter 6 General Linear Model: Statistical Inference 6.1 Introduction So far we have discussed formulation of linear models (Chapter 1), estimability of parameters in a linear model (Chapter 4), least

More information

Lecture 6 Multiple Linear Regression, cont.

Lecture 6 Multiple Linear Regression, cont. Lecture 6 Multiple Linear Regression, cont. BIOST 515 January 22, 2004 BIOST 515, Lecture 6 Testing general linear hypotheses Suppose we are interested in testing linear combinations of the regression

More information

Linear Regression for Air Pollution Data

Linear Regression for Air Pollution Data UNIVERSITY OF TEXAS AT SAN ANTONIO Linear Regression for Air Pollution Data Liang Jing April 2008 1 1 GOAL The increasing health problems caused by traffic-related air pollution have caught more and more

More information

Econometrics Summary Algebraic and Statistical Preliminaries

Econometrics Summary Algebraic and Statistical Preliminaries Econometrics Summary Algebraic and Statistical Preliminaries Elasticity: The point elasticity of Y with respect to L is given by α = ( Y/ L)/(Y/L). The arc elasticity is given by ( Y/ L)/(Y/L), when L

More information

Central Limit Theorem ( 5.3)

Central Limit Theorem ( 5.3) Central Limit Theorem ( 5.3) Let X 1, X 2,... be a sequence of independent random variables, each having n mean µ and variance σ 2. Then the distribution of the partial sum S n = X i i=1 becomes approximately

More information

MFin Econometrics I Session 4: t-distribution, Simple Linear Regression, OLS assumptions and properties of OLS estimators

MFin Econometrics I Session 4: t-distribution, Simple Linear Regression, OLS assumptions and properties of OLS estimators MFin Econometrics I Session 4: t-distribution, Simple Linear Regression, OLS assumptions and properties of OLS estimators Thilo Klein University of Cambridge Judge Business School Session 4: Linear regression,

More information

STATISTICS 479 Exam II (100 points)

STATISTICS 479 Exam II (100 points) Name STATISTICS 79 Exam II (1 points) 1. A SAS data set was created using the following input statement: Answer parts(a) to (e) below. input State $ City $ Pop199 Income Housing Electric; (a) () Give the

More information

Peter Hoff Linear and multilinear models April 3, GLS for multivariate regression 5. 3 Covariance estimation for the GLM 8

Peter Hoff Linear and multilinear models April 3, GLS for multivariate regression 5. 3 Covariance estimation for the GLM 8 Contents 1 Linear model 1 2 GLS for multivariate regression 5 3 Covariance estimation for the GLM 8 4 Testing the GLH 11 A reference for some of this material can be found somewhere. 1 Linear model Recall

More information

CAS MA575 Linear Models

CAS MA575 Linear Models CAS MA575 Linear Models Boston University, Fall 2013 Midterm Exam (Correction) Instructor: Cedric Ginestet Date: 22 Oct 2013. Maximal Score: 200pts. Please Note: You will only be graded on work and answers

More information

Chapter 1 Statistical Inference

Chapter 1 Statistical Inference Chapter 1 Statistical Inference causal inference To infer causality, you need a randomized experiment (or a huge observational study and lots of outside information). inference to populations Generalizations

More information

ST430 Exam 2 Solutions

ST430 Exam 2 Solutions ST430 Exam 2 Solutions Date: November 9, 2015 Name: Guideline: You may use one-page (front and back of a standard A4 paper) of notes. No laptop or textbook are permitted but you may use a calculator. Giving

More information

Section 3: Simple Linear Regression

Section 3: Simple Linear Regression Section 3: Simple Linear Regression Carlos M. Carvalho The University of Texas at Austin McCombs School of Business http://faculty.mccombs.utexas.edu/carlos.carvalho/teaching/ 1 Regression: General Introduction

More information

[y i α βx i ] 2 (2) Q = i=1

[y i α βx i ] 2 (2) Q = i=1 Least squares fits This section has no probability in it. There are no random variables. We are given n points (x i, y i ) and want to find the equation of the line that best fits them. We take the equation

More information