CHAPTER 2 SIMPLE LINEAR REGRESSION
|
|
- Ronald Peregrine Hood
- 6 years ago
- Views:
Transcription
1 CHAPTER 2 SIMPLE LINEAR REGRESSION 1
2 Examples: 1. Amherst, MA, annual mean temperatures, Summer mean temperatures in Mount Airy (NC) and Charleston (SC), Scatterplots outliers? influential values? independent v. dependent variables 2
3 Mean Temperature Charleston A B Year Mount Airy Figure 2.1. (a) Plot of mean temperature against year for Amherst data. (b) Plot of mean summer temperature in Charleston against mean summer temperature in Mount Airy. In each case a least squares regression line is shown on the plot. 3
4 The basic model: y i = α + βx i + ϵ i, 1 i n. (a) ϵ 1,..., ϵ n uncorrelated with mean 0 and variance σ 2 (b) ϵ 1,..., ϵ n independent N[0, σ 2 ]. σ 2 unknown in practice 4
5 Method of Least Squares Rewrite basic equation as y i = β 0 + β 1 (x i x) + ϵ i, 1 i n. ( x = 1 n ni=1 x i ) Method: choose estimates ˆβ 0, ˆβ 1 to minimize S = n i=1 { yi ˆβ 0 ˆβ 1 (x i x) } 2. See picture. Solution is ˆβ 0 = ˆβ 1 = ni=1 y i = ȳ, n ni=1 y i (x i x) ni=1 (x i x) 2. 5
6 12 10 y x Figure 2.2. Illustration of the least squares criterion on a small artificial data set. The fitted straight line is chosen so as to minimize the sum of squares of vertical distances from the observations to the line. 6
7 Estimation of σ 2 First define residuals (see next figure) e i = y i ˆβ 0 ˆβ 1 (x i x) The e i s are estimates of the original ϵ i s. Therefore, an obvious estimate of ˆσ 2 would be ni=1 ˆσ 2 e 2 = i. n It turns out this is a biased estimator; an unbiased estimator is s 2 = ni=1 e 2 i n 2. The numerator is the residual sum of squares (RSS); s 2 is the mean sum of squares (MSS). 7
8 Residual x Figure 2.3. Residuals calculated for the same artificial data set as in Figure 2.2. The fitted straight line has been subtracted from the observations so that the residuals may be plotted directly. 8
9 Organizing the Computations Some useful formulae: (xi x) 2 = x 2 i n x2, (yi ȳ) 2 = y 2 i nȳ2, (xi x)(y i ȳ) = x i y i n xȳ, e 2 i = (y i ȳ) 2 { (x i x)y i } 2 (xi x) 2. 9
10 Amherst Example x i = i = 1,..., 162 ( x = 81.5) ˆβ 0 = 7.393, ˆβ 1 =.0116, so the fitted regression line is y i = (x i 81.5) + ϵ i, y i = x i + ϵ i. e 2 i = , s 2 =.4434 =
11 Statistical Properties of the Estimators Under assumptions (a) about ϵ i s: E{ˆβ 0 } = 1 n E { yi } = 1 n {β0 + β 1 (x i x)} = β 0, { } yi (x E{ˆβ 1 } = E i x) (xi x) 2 = {β0 + β 1 (x i x)}(x i x) (xi x) 2 = β 1, V ar{ˆβ 0 } = σ2 n, V ar{ˆβ 1 } = Cov{ˆβ 0, ˆβ 1 } = 0. σ 2 (xi x) 2, 11
12 The estimators of β 0 and β 1 are the best linear unbiased estimators (BLUEs), i.e. out of all estimators that are both linear and unbiased, they have the smallest variance (the Gauss- Markov theorem). We prove this property for ˆβ 1 : Consider another estimator β 1 = c i y i. The choice c i = (x i x)/ (x i x) 2 is the least squares estimator. Unbiasedness implies ci = 0, ci (x i x) = 1. If β 1 ˆβ 1 = c i y i then c i = 0, c i (x i x) = 0. 12
13 The covariance of β 1 ˆβ 1 and ˆβ 1 is (xi x)c i (xi x) 2 σ2 = 0. Therefore, V ar{ β 1 } = V ar{ β 1 ˆβ 1 + ˆβ 1 } = V ar{ β 1 ˆβ 1 } + V ar{ˆβ 1 } V ar{ˆβ 1 }. This proves the result. 13
14 Distribution of s 2 : If we make assumption (b) about the ϵ i s (independent normal) then (n 2)s 2 σ 2 χ 2 n 2. This may be used to construct confidence intervals for σ 2. 14
15 Hypothesis Tests, Confidence Intervals and Prediction Intervals Now make assumptions (b) throughout. Basic distributional results: σ known, ˆβ n 0 β 0 σ (xi x) 2 ˆβ 1 β 1 σ N[0, 1], N[0, 1]. σ unknown, ˆβ n 0 β 0 s (xi x) 2 ˆβ 1 β 1 s t n 2, t n 2. 15
16 Example of Application Test H 0 : β 1 = 0. Reject at level α if (xi x) 2 ˆβ 1 s > t n 2;1 α/2. A 100(1 α)% confidence interval for β 1 may be defined to be s ˆβ 1 ± t n 2;1 α/2 (xi x) 2. The expression for ˆβ 1. s (xi is the standard error x) 2 Tests and confidence intervals for σ 2 : use χ 2 n 2 distribution result given earlier. 16
17 For the Amherst data, we earlier found ˆβ 1 = By applying the above formulae, we find the standard error of ˆβ 1 is.6659 = The t ratio for β 1 is = This is very highly significant. However, for this example, the assumption of independent observations may not be true. We shall see later that they are in fact correlated and this considerably increases the standard error of ˆβ 1, though not to the extent that the data would pass a test that β 1 = 0. 17
18 The prediction problem Given a new experiment to be conducted at x = x, what can we say about the corresponding y = y? Obvious point estimator is ŷ = ˆβ 0 + ˆβ 1 (x x). Gauss-Markov theorem: this is the BLUE for this problem. The right hand side has variance { σ 2 1 n + (x x) 2 } (xi x) 2. The 100(1 α)% confidence interval is ŷ ± t n 2;1 α/2 s 1 n + (x x) 2 (xi x) 2. 18
19 The distinction between confidence intervals and prediction intervals What we have just calculated is a confidence interval for the quantity β 0 + β 1 (x x). It doesn t reflect the full variability that is involved in predicting a specific future observation. Suppose y = β 0 + β 1 (x x) + ϵ where ϵ is a random error (uncorrelated with ϵ 1,.., ϵ n ) with the same mean 0 and variance σ 2. As before, we have ŷ = ˆβ 0 + ˆβ 1 (x x). Therefore, y ŷ = (β 0 ˆβ 0 ) + (β 1 ˆβ 1 )(x x) + ϵ. 19
20 y ŷ = (β 0 ˆβ 0 ) + (β 1 ˆβ 1 )(x x) + ϵ. The three terms on the right hand side are uncorrelated and have total variance { σ 2 1 n + (x x) 2 } (xi x) If σ 2 is known, an exact 100(1 α)% prediction interval for y is given by ŷ ± z 1 α/2 σ 1 n + (x x) 2 (xi x) For unknown σ 2, the corresponding interval is ŷ ± t n 2;1 α/2 s 1 n + (x x) 2 (xi x) Note the extra +1 compared with the earlier confidence intervals. 20
21 Example. Predict the temperature for Amherst in the year 2010 (corresponding to x = 175). We earlier found ˆβ 0 = 7.393, ˆβ 1 =.0116, ( and x = 81.5), so the point prediction is ( )=8.478, and the estimated prediction variance.4434 { ( ) } =.4571 = (.676) 2. The point of the t 160 distribution is 1.975, so the 95% prediction interval is 8.478± = (7.14, 9.81). On the other hand,.4434 { ( ) } = = (.117) 2 so the corresponding 95% confidence interval is (8.25,8.71). 21
22 The amherst.dat file looks like this: year temp
23 R Program for LS regression amh<-read.table(file="amherst.dat",header=t) attach(amh) names(amh) [1] "year" "temp" amh.lm <- lm(temp~year) summary(amh.lm) Call: lm(formula = temp ~ year) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) <2e-16 *** year <2e-16 *** --- Signif. codes: 0 *** ** 0.01 * Residual standard error: on 160 degrees of freedom Multiple R-Squared: ,Adjusted R-squared: F-statistic: on 1 and 160 DF, p-value: < 2.2e-16 23
24 R Program for prediction pr1<-predict(amh.lm,data.frame(year=c(165, 170, 175)), se.fit=t) pr1 $fit $se.fit $df [1] 160 $residual.scale [1]
25 Residuals ith residual defined by e i = y i ˆβ 0 ˆβ 1 (x i x) Possible plots of residuals: (a) e i against x i (b) e i against any omitted covariate (c) plots to examine autocorrelation or normal distribution of residuals Things we are looking for: Outliers Evidence of nonlinearity Evidence of other departures from assumptions, e.g. residuals autocorrelated or having non-constant variance 25
26 (a) (b) Residual 1 0 Residual x x (c) 1.0 Residual x Figure 2.4. Some examples of residual plots illustrating different kinds of discrepancy from a linear model. 26
27 Mount Airy and Amherst examples plot residual against x i, and also a histogram of residuals A Residual Count B Mount Airy Mean Temp Residual Figure 2.5. Plots of residuals from a linear model fitted to Mount Airy Charleston data. (a) Residual versus x value. (b) Histogram of residuals. 27
28 Residual Count Year Residual Figure 2.6. Plots of residuals from a linear model fitted to Amherst data. (a) Residual versus x value. (b) Histogram of residuals. 28
29 A more sophisticated way of deciding whether the residuals are normally distributed is to use a probability plot. There are many variants of this idea, but here we describe one of the simplest versions: 1. Calculate residuals e 1,..., e n, arrange in order, so that e 1 e 2... e n. 2. Define d i = e i /s to standardize residuals to approximately common variance Plot d i against z (i 0.5)/n. (Here z α denotes the α-probability point of the standard normal distribution, i.e. if Z N[0, 1] then Pr{Z z α } = α.) 29
30 (a) (b) Observed value Observed value Expected value Expected value Figure 2.7. Normal probability plot of residuals from (a) Mount Airy Charleston data, (b) Amherst data. 30
31 Formal Tests of Normality (a) The Shapiro-Wilk test, and related tests (b) Tests based on the empirical distribution function (EDF) 31
32 The Shapiro-Wilk Test Simplest version considers only the case y i N[µ, σ 2 ], µ and σ 2 unknown (no regression component) Let e i = y i ȳ, order e 1,..., e n as e 1... e n Let m i be the mean of e i when the y i are i.i.d. N[0, 1]. The idea of the Shapiro-Wilk test is to measure the correlation between the e i s and the m i s. The precise formula used for this is of the form W = ( a i e i )2 (e i ) 2. Tables for the coefficients a i and for the percentage points of W were given by Shapiro and Wilk in their original paper (1965). They are coded into R as shapiro.test(), and SAS as part of PROC UNIVARIATE. 32
33 Variants on the Shapiro-Wilk Test Shapiro-Francia test (1972) used correlation coefficient between e i and m i rather than computing optimal coefficients a i (but the exact computation of m i is still necessary) Looney-Gulledge (1986) simplified Shapiro- Francia by using an approximation to m i : r = z i e i 2 z i e 2 i. Percentage points of r : Computed by Looney and Gulledge in case when original data are N[µ, σ 2 ] Compute directly by simulation in regression context 33
34 Tests based on the EDF Suppose y 1,..., y n are observations. Define the empirical distribution function or EDF: F n (y) = 1 n {Number of observations y}, y (, ). Suppose we believe that the true distribution function is some given F 0 (y). There are a number of test statistics to determine how close F n is to F 0 : D = max y (Kolmogorov-Smirnov), W 2 = n (Cramér von Mises), A 2 = n F n (y) F 0 (y) {F n(y) F 0 (y)} 2 df 0 (y) (Anderson-Darling). {F n (y) F 0 (y)} 2 F 0 (y) {1 F 0 (y)} df 0(y) 34
35 In practice, there are simplifying computational formulae for all three tests (see text for details). Next, we describe how to compute the percentage points of these tests. 35
36 1. If F 0 is completely specified (no unknown parameters), then the percentage points do not depend on F 0 and have been tabulated in many sources (e.g. Biometrika Tables for Statisticians). 2. A more practical situation is something like F 0 (y) = Φ ( ) y µ σ where µ and σ 2 are unknown and Φ is the standard normal d.f. In that case it is usual to estimate µ and σ 2 by the usual sample mean and variance, and to base the EDF tests on the estimated F 0. This is the situation tabulated in PROC UNIVARIATE and many other textbooks and packages. However, even this does not allow for the estimation of regression parameters. 3. A third option is to use simulation directly, including refitting the regression model to each simulated sample. 36
37 Implementing in R Approximate method: First fit regression by lm command, then extract residuals and apply shapiro.test() or ks.test() > shapiro.test(residuals(amh.lm)) Shapiro-Wilk normality test data: residuals(amh.lm) W = , p-value = > ks.test(residuals(amh.lm), pnorm, amh.lm$sig) One-sample Kolmogorov-Smirnov test data: residuals(amh.lm) D = 0.076, p-value = alternative hypothesis: two.sided 37
38 Implementing in R Alternatively, use the code normtest.r (available on the course website) to simulate the exact distribution of all four test statistics. norm.test <- function(y, x, nsim=1000) { ### y: response x: covariates ### Program to perform goodness of fit tests ### for regression data ## The next lines compute the Looney-Gulledge ## statistic c1, the Kolmogorov-Smirnov statistic ## d1, the Cramer-von Mises statistic w1 and the ## Anderson-Darling statistic a1 nreg <- lm(y~x) n <- length(y) q1<-qqnorm(nreg$resid,plot=f) c1<-cor(q1$x,q1$y) u1<-pnorm(sort(nreg$resid),mean=0, sd=summary(nreg)$sigma) d1<-max(c((1:n)/n-u1,u1-(0:(n-1))/n)) w1<-sum((u1-((1:n)-0.5)/n)^2)+1/(12*n) a1<--sum((2*(1:n)-1)*log(u1)+(2*n+1-2*(1:n)) *log(1-u1))/n-n count<-rep(0,4) names(count) <- c("l-g", "K-S", "C-V", "A-D") 38
39 for(j in 1:nsim){ temp1<-rnorm(n) nreg<-lm(temp1~x) q2<-qqnorm(nreg$resid,plot=f) c2<-cor(q2$x,q2$y) if(c2<c1)count[1]<-count[1]+1 u2<-pnorm(sort(nreg$resid),mean=0, sd=summary(nreg)$sigma) d2<-max(c((1:n)/n-u2,u2-(0:(n-1))/n)) w2<-sum((u2-((1:n)-0.5)/n)^2)+1/(12*n) a2<--sum((2*(1:n)-1)*log(u2)+(2*n+1-2*(1:n))*log(1-u2))/n-n if(d2>d1)count[2]<-count[2]+1 if(w2>w1)count[3]<-count[3]+1 if(a2>a1)count[4]<-count[4]+1 } ## return the percentage of times the simulation ## resulted in a more extreme value of the test ## statistic than the one computed from the real data. return(count/nsim) } > source("normtest.r") > norm.test(temp, year) L-G K-S C-V A-D
40 The ANOVA Table Start from the formula (yi a) 2 = (y i ȳ) 2 + n(ȳ a) 2. If y i N[µ, σ 2 ] (i.i.d.) then this leads to the formula ( ) y i µ 2 ( ) y = i ȳ 2 (ȳ ) µ 2 + n σ σ σ or in distributional terms, χ 2 n = χ2 n 1 + χ2 1. (The two terms on the right hand side are independent.) 39
41 The regression set-up (yi ȳ) 2 = {y i ˆβ 0 ˆβ 1 (x i x) + ˆβ 1 (x i x)} 2 = {y i ˆβ 0 ˆβ 1 (x i x)} 2 + ˆβ 1 2 (xi x) 2 since the cross-product vanishes: {yi ˆβ 0 ˆβ 1 (x i x)}(x i x) = 0. Standardize: (yi ȳ) 2 σ 2 = e 2 i (xi x) 2 σ 2. σ 2 + ˆβ 1 2 If β 1 = 0, this has the distributional interpretation χ 2 n 1 = χ2 n 2 + χ2 1 where, again, the two terms on the right hand side are independent. 40
42 The calculations lead us to the ANOVA Table, as follows: SOURCE SS D.F. MS Regression β 1 2 (xi x) 2 1 β 1 2 (xi x) 2 Residual e 2 i n 2 e 2 i /(n 2) Total (yi ȳ) 2 n 1 Table 2.3. Example of an ANOVA table. Testing for a significant regression coefficient: when β 1 = 0, β 2 1 (xi x) 2 e 2 i /(n 2) F 1,n 2. 41
43 SOURCE SS D.F. MS Regression Residual Total Table 2.4. ANOVA table for the Mount Airy Charleston data. F ratio is = 35.0, the 95% and 99.9% points of F 1,47 are and (alternatively: the p-value is ). Clear evidence that the null hypothesis (β 1 = 0) is false. 42
44 SOURCE SS D.F. MS Regression Residual Total Table 2.5. ANOVA table for the Amherst data. F ratio is = 107.6, the 95% and 99.9% points of F 1,160 are 3.90 and 11.24; the p-value is computed by R as 0. Again the test rejects β 1 = 0, but we emphasize again that this assumes all the model assumptions are correct (independent errors, common variance, normal distribution). 43
45 Implementation in R > anova(amh.lm) Analysis of Variance Table Response: temp Df Sum Sq Mean Sq F value Pr(>F) year < 2.2e-16 *** Residuals Signif. codes: 0 *** ** 0.01 *
46 Example Consider the following data: x y x y x y x y y x 45
47 Testing the fit of the linear regression model Is the relationship really linear? This can be tested if there are repeated observations for some of the x values. Suppose there are K < n distinct x values, labelled x 1,..., x K. Also let n k denote the number of observations taken at x k, so that n = K 1 n k, and let the individual observations be y kj, j = 1,..., n k. Define S 1 = β 2 1 S 2 = k S 3 = k S 4 = k S 5 = k k j n k (x k x) 2, {y kj β 0 β 1 (x k x)} 2, n k {ȳ k β 0 β 1 (x k x)} 2, j j (y kj ȳ k ) 2, (y kj ȳ) 2. 46
48 Source SS D.F. MS Regression S 1 1 S 1 Residual S 2 n 2 S 2 /(n 2) Between groups S 3 K 2 S 3 /(K 2) Within groups S 4 n K S 4 /(n K) Total S 5 n 1 Table 2.5. ANOVA table for test of fit Under the null hypothesis that the linear regression model is correct, the ratio of sums of squares F = S 3 K 2 n K S 4 has an F K 2,n K distribution. 47
49 k x k n k ȳ k ŷ k n k (ȳ k ŷ k ) 2 (y kj ȳ k ) Detailed calculations for data of Table 2.7 SOURCE SS D.F. MS Regression Residual Between groups Within groups Total ANOVA table for data of Table 2.7 F = 6.216/1.011 = From tables of the F 3,6 distribution, we find that the upper-α point is 4.76 at α =.05 and 6.60 at α =
50 Implementation in R > lfit <-read.table(file="lfit.dat",header=t) > attach(lfit) > lfit.lm1 <- lm(y~x) > lfit.lm2 <- lm(y~factor(x)) > aov(lfit.lm1, lfit.lm2) Analysis of Variance Table Model 1: y ~ x Model 2: y ~ factor(x) Res.Df RSS Df Sum of Sq F Pr(>F) * --- Signif. codes: 0 *** ** 0.01 *
51 Simultaneous confidence and prediction intervals The simultaneous confidence interval problem may be stated as follows: find G so that, for 1 k K,with probability at least 1 α, each of the inequalities { β 0 + β 1 (x k x)} {β 0 + β 1 (x k x)} G, s 1 n + (x k x)2 (xi x) 2 is satisfied simultaneously. (1) We outline the Bonferroni procedure and the Working-Hotelling procedure. The general Scheffé procedure and its proof left to Ch
52 The Bonferroni procedure Let A k be the event that the inequality (1) is satisfied with x = x k, and let Ā k denote the complementary event. Bonferroni s procedure is based on the elementary inequality and hence Pr{Ā 1 Ā 2... Ā K } K k=1 Pr{Ā k } Pr{A 1 A 2... A K } 1 K k=1 Pr{Ā k }. (2) Therefore, to ensure that the left-hand side of (2) is at least 1 α, it suffices to ensure that Pr{Āk } α. Usually, though not invariably, this is achieved by defining the confidence interval bounds so that Pr{Ā k } α/k for each k. Thus, we may write G = t n 2;1 α/(2k). (3) 51
53 Mount Airy-Charleston Example Find 95% simultaneous confidence intervals for the mean temperature in Charleston corresponding to temperatures x 1 = 20, x 2 = 22.5, x 3 = 25, x 4 = 27.5, x 5 = 30 in Mount Airy. In this case α =.05, K = 5, so G = t 47;.995 = The confidence interval standard errors, defined as s 1 n + (x k x)2 (xi x) 2, are calculated to be.3428,.1161,.1566,.3881,.6260, respectively for x 1,..., x 5. The point predictions, β 0 + β 1 (x x), are , , , , Therefore, the five simultaneous confidence intervals are given by ± = (24.075, ), ± = (26.103, ), ± = (27.414, ), ± = (28.213, ), ± = (28.995, ). 52
54 For prediction intervals, the calculations are the same except that the standard errors for prediction are given by s n + (x k x)2 (xi x) 2, As a point of comparison, for K = 1 we have G = instead of G = the simultaneous confidence or prediction intervals are 33% wider than the one-at-a-time intervals. 53
55 32 Charleston Mount Airy Figure 2.8. Five simultaneous confidence intervals (inner bands, solid lines) and five simultaneous prediction intervals (outer bands, dashed lines) for the Mount Airy Charleston data series. 54
56 The Working-Hotelling procedure In this method, the constant G in (1) is defined by G 2 = 2F 2;n 2;1 α (4) where F ν1 ;ν 2 ;1 α denotes the (1 α) lower probability point of the F distribution with ν 1 degrees of freedom in the numerator and ν 2 degrees of freedom in the denominator. This is a special case of Sheffé s procedure for confidence interval. For Mount Airy and Charleston, F 2,47;.95 = (use qf(0.95,2,47) in R), so G = Note that this G is smaller than the of Bonferroni, so it is preferable to use the Scheffé procedure in this instance. Because this calculation is independent of the number of comparison K, we can use it to calculate simultaneous confidence intervals for all x k at once. An example of this is shown in Fig
57 32 30 Charleston Mount Airy Figure 2.9. Point estimates (straight line, centre) and 95% simultaneous confidence bands (outer curves, dashed) for the mean response at all x values between 20 and 30, as calculated by the Working-Hotelling procedure. 56
58 Inverse Regression and Calibration Earlier we considered the prediction problem: given a new pair (x, y ) and knowing only x, predict y. Now we consider the opposite problem: given y, predict x. Most common context for this problem is calibration, in which x indicates some precise measurement of a quantity and y is a quicker but less precise measurement. The problem is to calibrate the measurement, i.e. to infer x from y, in the most precise way possible. 57
59 First Solution: Inverse Regression Obvious point estimator is ˆx = x + y ˆβ 0 ˆβ 1. Problem of how to get an interval estimate. First note y ˆβ 0 ˆβ 1 (x x) = s (xi x) n + (x x) 2 ˆβ 1 ( ˆx x ) t s 1 n + (x x) 2 n 2. (xi x) An exact 100(1 α)% prediction interval for x would be ˆx s ± t 1 n 2;1 α/2 ˆβ 1 n + (x x) 2 (xi x)
60 Unfortunately this is not realizable because the width of the interval depends on x, which is unknown. As a practical approximation, however, we can estimate this by ˆx, which leads to the interval ˆx s ± t 1 n 2;1 α/2 ˆβ 1 n + ( ˆx x) 2 (xi x) The method is usually called inverse regression 59
61 Second Solution: Direct Regression However there is another solution which seems even easier to adopt just interchange the role of x and y everywhere, i.e. treat the original regression problem as a regression of x i on y i, then the inverse regression problem becomes a direct prediction problem for which the earlier solution (Section 2.4) is applicable. What s wrong with that? 60
62 Third Solution: Exact Solution of the Inverse Regression Problem ˆβ 1 ( ˆx x ) t s 1 n + (x x) 2 n 2. (xi x) Define c = t n 2;1 α/2. With probability α, ˆβ 1 ( ˆx x ) s 1 n + (x x) 2 (xi x) c. The boundary points for x occur when { ˆβ 1 2 ( ˆx x ) 2 = c 2 s 2 1 n + (x x) 2 } (xi x) With luck, this equation will have two real roots (for x ) and the interval between them will be our desired interval. 61
63 Residual 1.5 B A Count Charleston Mean Temp Residual Figure Plots of residuals from a linear model fitted to direct regression of Charleston temperatures on Mount Airy temperatures. (a) Residual versus x value. (b) Histogram of residuals. 62
64 Residual Count Mean Temperature Residual Figure Plots of residuals from a linear model fitted to time vs. Amherst mean temperatures. (a) Residual versus x value. (b) Histogram of residuals. 63
65 (a) (b) Observed value Observed value Expected value Expected value Figure Normal probability plot of residuals from (a) Charleston Mount Airy data, (b) Amherst data, both fitted by the direct method whereby the original roles of x and y are reversed. 64
66 Amherst Example Predict x corresponding to y = 10. Inverse regression method leads to ˆx = and the approximate 95% prediction interval (185.6,426.8). Direct regression method leads to ˆx = and 95% prediction interval (98.3,245.4). Exact inverse method leads to (191.4,437.6). Conclusion: The exact and approximate inverse methods are not much different (despite huge standard error), but the direct method leads to a completely different solution. 65
67 Charleston II I Mount Airy Figure Plot of mean summer temperature in Charleston (y) against mean summer temperature in Mount Airy (x), together with the regression lines for y on x (I) and for x on y (II). 66
68 Formulate in terms of the bivariate normal distribution. Suppose X N(µ X, σ 2 X ), Y N(µ Y, σ 2 Y ), and that the correlation coefficient is ρ, in other words, Density: E{(X µ X )(Y µ Y )} = ρσ X σ Y. = f X,Y (x, y) ( 1 exp 1 x µx 2πσ X σ Y 1 ρ 2 2(1 ρ 2 ) σ X ( ) 2 ( ) ( ) y µy x µx y µy + 2ρ. σ Y σ X σ Y ) 2 Compute the conditional density of Y X = x using f Y X (y x) = f X,Y (x, y). f X (x) given 67
69 f Y X (y x) {( 1 y µy exp 2(1 ρ 2 ) 1 = exp 2σY 2 (1 ρ2 ) { σ Y ) ρ ( x µx σ X )} 2 y µ Y ρσ Y σ X (x µ X ) This is the density of a normal distribution with mean µ Y + ρσ Y σ X (x µ X ) and variance σ 2 Y (1 ρ2 ). In practice, do not know µ X, µ Y, etc., estimate by µ X = x, σ 2 X = s 2 (xi X = x) 2, n 1 µ Y = ȳ, σ 2 Y = s 2 (yi Y = ȳ) 2, n 1 (xi x)(y ρ = r = i ȳ) (xi x) 2 (y i ȳ) 2. } 2. 68
70 Now, suppose we have a new pair (x, y ), but we only observe x, and we want to predict the unknown value y. Based on the conditional distribution, we estimate the conditional mean of Y given X = x to be µ Y + ρ σ Y σ X (x µ X ) = β 0 + β 1 (x x). Estimated conditional variance turns out to be n 2 n 1 s2. Almost the same as the original regression formulae, which treated the X s as fixed constants, not random variables. 69
71 Moral of the story... If both variables are random, predicting one of them giving the other is (almost) the same as treating the conditioning variable as fixed constants. However, if one variable truly is non-random (as when it is time ), this analogy doesn t help. For predicting X given Y, direct regression bases the prediction on f X Y (x y) f X (x)f Y X (y x). while inverse regression uses only f Y X (y x). The difference is in the existence of a prior distribution f X (x). 70
72 Inference about the correlation coefficient How to obtain tests and confidence intervals for the correlation coefficient ρ? Fisher s z-transformation (Fisher 1921, Gayen 1951) Transform r into one which is approximately normal: where z = 1 2 log ( 1 + r 1 r ) N ( ζ, ζ = 1 ( ) 1 + ρ 2 log. 1 ρ 1 n 3 ), (5) Example. For the Mount Airy Charleston data set, we find r =.653 and so z =.781 by (5). An approximate 95% confidence interval for ζ is.781 ± 1.96 = (.492, 1.070). Transforming 46 back, we find that a 95% confidence interval for ρ is (.456,.789). 71
73 Autocorrelation Corr{ϵ i, ϵ i+k } = E(ϵ iϵ i+k ) σ 2 = ρ ϵ (k). Assume this does not depend on i (stationarity assumption) Estimate ρ ϵ (k) by r ϵ (k) = n k i=1 e ie i+k ni=1 e 2 i This is considered significant if r ϵ (k) > 2 n. 72
74 Example. Amherst: first 10 sample autocorrelations of the residuals are.185,.156,.065,.002,.001,.047,.023,.079,.023,.065. Here n = 162, 2/ n =.157, so it looks as though the first two autocorrelations are statistically significant. The remaining autocorrelations for k > 2 are very small, and in addition there is no evidence from Fig. 2.6 of changing variance, so we may assume the series to be stationary. Charleston/Mount Airy: first 10 sample autocorrelations.016,.009,.315,.041,.010,.010,.204,.064,.047,.065. In this case, with n = 49, 2/ n =.286. Only k = 3 is significant. 73
75 Corrleation Lag Figure First 10 serial correlations for Amherst data, with approximate 95% error bounds if the true series is independent. 74
76 Corrleation Lag Figure First 10 serial correlations for Mount Airy Charleston data, with approximate 95% error bounds if the true series is independent. 75
77 The effect of autocorrelation on the regression estimates Suppose we decide that a series is indeed autocorrelated, and assume that it is stationary with at least the first few autocorrelations ρ ϵ (k) estimated by the sample autocorrelations r ϵ (k). How should this affect our regression analysis? Suppose we apply standard ordinary least squares analysis and compute estimators β 0 and β 1. The main effect of correlation is that the standard error formulae for these estimators are wrong. 76
78 Variance of β 0 = ȳ: We have Var(ȳ) = 1 n 2 n = σ2 n 2 n i=1 j=1 n n i=1 j=1 Cov(y i, y j ) ρ ϵ ( i j ) = σ2 n 2{n + 2(n 1)ρ ϵ(1) + 2(n 2)ρ ϵ (2) ρ ϵ (n 1)} = σ2 n { ( 1 1 n ) n ρ ϵ(n 1) ρ ϵ (1) }. 77
79 Variance of β 1 : { } (xi x)(y Var i ȳ) (xi x) 2 = = = where σ 2 { (x i x) 2 } 2 σ 2 (xi x) σ 2 (xi x) 2 n 1 k=1 ρ X (k) = r X (k) = n n i=1 j=1 n k n (x i x)(x j x)ρ ϵ ( i j ) i=1 (x i x)(x i+k x) ni=1 (x i x) 2 k=1 ρ ϵ (k)ρ X (k) n k i=1 (x i x)(x i+k x) ni=1 (x i x) 2. ρ ϵ(k) 78
80 In practice, we will estimate r ϵ (k) for k up to some cutoff K, and estimate the variances of β 0 and β 1 by and s 2 n s 2 (xi x) K k=1 ( 1 k ) n K k=1 r ϵ (k) r ϵ (k)r X (k). 79
81 K Char. β 0 Char. β 1 Amh. β 0 Amh. β Table Corrected standard errors assuming autocorrelation for the Charleston and Amherst data for several different values of the largest lag K 80
Ch 2: Simple Linear Regression
Ch 2: Simple Linear Regression 1. Simple Linear Regression Model A simple regression model with a single regressor x is y = β 0 + β 1 x + ɛ, where we assume that the error ɛ is independent random component
More information13 Simple Linear Regression
B.Sc./Cert./M.Sc. Qualif. - Statistics: Theory and Practice 3 Simple Linear Regression 3. An industrial example A study was undertaken to determine the effect of stirring rate on the amount of impurity
More informationSimple Linear Regression
Simple Linear Regression ST 430/514 Recall: A regression model describes how a dependent variable (or response) Y is affected, on average, by one or more independent variables (or factors, or covariates)
More informationSimple Linear Regression
Simple Linear Regression In simple linear regression we are concerned about the relationship between two variables, X and Y. There are two components to such a relationship. 1. The strength of the relationship.
More informationStat 427/527: Advanced Data Analysis I
Stat 427/527: Advanced Data Analysis I Review of Chapters 1-4 Sep, 2017 1 / 18 Concepts you need to know/interpret Numerical summaries: measures of center (mean, median, mode) measures of spread (sample
More informationSAS Procedures Inference about the Line ffl model statement in proc reg has many options ffl To construct confidence intervals use alpha=, clm, cli, c
Inference About the Slope ffl As with all estimates, ^fi1 subject to sampling var ffl Because Y jx _ Normal, the estimate ^fi1 _ Normal A linear combination of indep Normals is Normal Simple Linear Regression
More informationAnswer Keys to Homework#10
Answer Keys to Homework#10 Problem 1 Use either restricted or unrestricted mixed models. Problem 2 (a) First, the respective means for the 8 level combinations are listed in the following table A B C Mean
More informationAssignment 9 Answer Keys
Assignment 9 Answer Keys Problem 1 (a) First, the respective means for the 8 level combinations are listed in the following table A B C Mean 26.00 + 34.67 + 39.67 + + 49.33 + 42.33 + + 37.67 + + 54.67
More informationInference for Regression
Inference for Regression Section 9.4 Cathy Poliak, Ph.D. cathy@math.uh.edu Office in Fleming 11c Department of Mathematics University of Houston Lecture 13b - 3339 Cathy Poliak, Ph.D. cathy@math.uh.edu
More informationMaster s Written Examination
Master s Written Examination Option: Statistics and Probability Spring 05 Full points may be obtained for correct answers to eight questions Each numbered question (which may have several parts) is worth
More informationLinear Regression. In this problem sheet, we consider the problem of linear regression with p predictors and one intercept,
Linear Regression In this problem sheet, we consider the problem of linear regression with p predictors and one intercept, y = Xβ + ɛ, where y t = (y 1,..., y n ) is the column vector of target values,
More informationSection 4.6 Simple Linear Regression
Section 4.6 Simple Linear Regression Objectives ˆ Basic philosophy of SLR and the regression assumptions ˆ Point & interval estimation of the model parameters, and how to make predictions ˆ Point and interval
More informationMS&E 226: Small Data
MS&E 226: Small Data Lecture 15: Examples of hypothesis tests (v5) Ramesh Johari ramesh.johari@stanford.edu 1 / 32 The recipe 2 / 32 The hypothesis testing recipe In this lecture we repeatedly apply the
More informationMultiple Linear Regression
Multiple Linear Regression ST 430/514 Recall: a regression model describes how a dependent variable (or response) Y is affected, on average, by one or more independent variables (or factors, or covariates).
More informationCh 3: Multiple Linear Regression
Ch 3: Multiple Linear Regression 1. Multiple Linear Regression Model Multiple regression model has more than one regressor. For example, we have one response variable and two regressor variables: 1. delivery
More informationLecture 1: Linear Models and Applications
Lecture 1: Linear Models and Applications Claudia Czado TU München c (Claudia Czado, TU Munich) ZFS/IMS Göttingen 2004 0 Overview Introduction to linear models Exploratory data analysis (EDA) Estimation
More informationCorrelation and the Analysis of Variance Approach to Simple Linear Regression
Correlation and the Analysis of Variance Approach to Simple Linear Regression Biometry 755 Spring 2009 Correlation and the Analysis of Variance Approach to Simple Linear Regression p. 1/35 Correlation
More informationLecture 2: Basic Concepts and Simple Comparative Experiments Montgomery: Chapter 2
Lecture 2: Basic Concepts and Simple Comparative Experiments Montgomery: Chapter 2 Fall, 2013 Page 1 Random Variable and Probability Distribution Discrete random variable Y : Finite possible values {y
More informationIntroduction and Single Predictor Regression. Correlation
Introduction and Single Predictor Regression Dr. J. Kyle Roberts Southern Methodist University Simmons School of Education and Human Development Department of Teaching and Learning Correlation A correlation
More informationModule 6: Model Diagnostics
St@tmaster 02429/MIXED LINEAR MODELS PREPARED BY THE STATISTICS GROUPS AT IMM, DTU AND KU-LIFE Module 6: Model Diagnostics 6.1 Introduction............................... 1 6.2 Linear model diagnostics........................
More informationCircle the single best answer for each multiple choice question. Your choice should be made clearly.
TEST #1 STA 4853 March 6, 2017 Name: Please read the following directions. DO NOT TURN THE PAGE UNTIL INSTRUCTED TO DO SO Directions This exam is closed book and closed notes. There are 32 multiple choice
More informationRegression and the 2-Sample t
Regression and the 2-Sample t James H. Steiger Department of Psychology and Human Development Vanderbilt University James H. Steiger (Vanderbilt University) Regression and the 2-Sample t 1 / 44 Regression
More informationMODULE 4 SIMPLE LINEAR REGRESSION
MODULE 4 SIMPLE LINEAR REGRESSION Module Objectives: 1. Describe the equation of a line including the meanings of the two parameters. 2. Describe how the best-fit line to a set of bivariate data is derived.
More informationLecture 15. Hypothesis testing in the linear model
14. Lecture 15. Hypothesis testing in the linear model Lecture 15. Hypothesis testing in the linear model 1 (1 1) Preliminary lemma 15. Hypothesis testing in the linear model 15.1. Preliminary lemma Lemma
More informationHomework 2: Simple Linear Regression
STAT 4385 Applied Regression Analysis Homework : Simple Linear Regression (Simple Linear Regression) Thirty (n = 30) College graduates who have recently entered the job market. For each student, the CGPA
More informationunadjusted model for baseline cholesterol 22:31 Monday, April 19,
unadjusted model for baseline cholesterol 22:31 Monday, April 19, 2004 1 Class Level Information Class Levels Values TRETGRP 3 3 4 5 SEX 2 0 1 Number of observations 916 unadjusted model for baseline cholesterol
More informationSTAT5044: Regression and Anova
STAT5044: Regression and Anova Inyoung Kim 1 / 49 Outline 1 How to check assumptions 2 / 49 Assumption Linearity: scatter plot, residual plot Randomness: Run test, Durbin-Watson test when the data can
More informationRegression Models - Introduction
Regression Models - Introduction In regression models there are two types of variables that are studied: A dependent variable, Y, also called response variable. It is modeled as random. An independent
More information3. Linear Regression With a Single Regressor
3. Linear Regression With a Single Regressor Econometrics: (I) Application of statistical methods in empirical research Testing economic theory with real-world data (data analysis) 56 Econometrics: (II)
More informationChapter 8 (More on Assumptions for the Simple Linear Regression)
EXST3201 Chapter 8b Geaghan Fall 2005: Page 1 Chapter 8 (More on Assumptions for the Simple Linear Regression) Your textbook considers the following assumptions: Linearity This is not something I usually
More informationIES 612/STA 4-573/STA Winter 2008 Week 1--IES 612-STA STA doc
IES 612/STA 4-573/STA 4-576 Winter 2008 Week 1--IES 612-STA 4-573-STA 4-576.doc Review Notes: [OL] = Ott & Longnecker Statistical Methods and Data Analysis, 5 th edition. [Handouts based on notes prepared
More informationMath 423/533: The Main Theoretical Topics
Math 423/533: The Main Theoretical Topics Notation sample size n, data index i number of predictors, p (p = 2 for simple linear regression) y i : response for individual i x i = (x i1,..., x ip ) (1 p)
More informationSimple Linear Regression
Simple Linear Regression MATH 282A Introduction to Computational Statistics University of California, San Diego Instructor: Ery Arias-Castro http://math.ucsd.edu/ eariasca/math282a.html MATH 282A University
More informationSTAT420 Midterm Exam. University of Illinois Urbana-Champaign October 19 (Friday), :00 4:15p. SOLUTIONS (Yellow)
STAT40 Midterm Exam University of Illinois Urbana-Champaign October 19 (Friday), 018 3:00 4:15p SOLUTIONS (Yellow) Question 1 (15 points) (10 points) 3 (50 points) extra ( points) Total (77 points) Points
More informationSTAT 4385 Topic 03: Simple Linear Regression
STAT 4385 Topic 03: Simple Linear Regression Xiaogang Su, Ph.D. Department of Mathematical Science University of Texas at El Paso xsu@utep.edu Spring, 2017 Outline The Set-Up Exploratory Data Analysis
More informationSTAT5044: Regression and Anova. Inyoung Kim
STAT5044: Regression and Anova Inyoung Kim 2 / 47 Outline 1 Regression 2 Simple Linear regression 3 Basic concepts in regression 4 How to estimate unknown parameters 5 Properties of Least Squares Estimators:
More informationChapter 7: Simple linear regression
The absolute movement of the ground and buildings during an earthquake is small even in major earthquakes. The damage that a building suffers depends not upon its displacement, but upon the acceleration.
More informationLectures on Simple Linear Regression Stat 431, Summer 2012
Lectures on Simple Linear Regression Stat 43, Summer 0 Hyunseung Kang July 6-8, 0 Last Updated: July 8, 0 :59PM Introduction Previously, we have been investigating various properties of the population
More information(a) The percentage of variation in the response is given by the Multiple R-squared, which is 52.67%.
STOR 664 Homework 2 Solution Part A Exercise (Faraway book) Ch2 Ex1 > data(teengamb) > attach(teengamb) > tgl summary(tgl) Coefficients: Estimate Std Error t value
More informationChapter 3: Multiple Regression. August 14, 2018
Chapter 3: Multiple Regression August 14, 2018 1 The multiple linear regression model The model y = β 0 +β 1 x 1 + +β k x k +ǫ (1) is called a multiple linear regression model with k regressors. The parametersβ
More informationChapter 3: Regression Methods for Trends
Chapter 3: Regression Methods for Trends Time series exhibiting trends over time have a mean function that is some simple function (not necessarily constant) of time. The example random walk graph from
More informationRegression Models - Introduction
Regression Models - Introduction In regression models, two types of variables that are studied: A dependent variable, Y, also called response variable. It is modeled as random. An independent variable,
More information36-707: Regression Analysis Homework Solutions. Homework 3
36-707: Regression Analysis Homework Solutions Homework 3 Fall 2012 Problem 1 Y i = βx i + ɛ i, i {1, 2,..., n}. (a) Find the LS estimator of β: RSS = Σ n i=1(y i βx i ) 2 RSS β = Σ n i=1( 2X i )(Y i βx
More informationRegression. Marc H. Mehlman University of New Haven
Regression Marc H. Mehlman marcmehlman@yahoo.com University of New Haven the statistician knows that in nature there never was a normal distribution, there never was a straight line, yet with normal and
More informationLecture 3. Experiments with a Single Factor: ANOVA Montgomery 3.1 through 3.3
Lecture 3. Experiments with a Single Factor: ANOVA Montgomery 3.1 through 3.3 Fall, 2013 Page 1 Tensile Strength Experiment Investigate the tensile strength of a new synthetic fiber. The factor is the
More information1) Answer the following questions as true (T) or false (F) by circling the appropriate letter.
1) Answer the following questions as true (T) or false (F) by circling the appropriate letter. T F T F T F a) Variance estimates should always be positive, but covariance estimates can be either positive
More informationStatistics for Engineers Lecture 9 Linear Regression
Statistics for Engineers Lecture 9 Linear Regression Chong Ma Department of Statistics University of South Carolina chongm@email.sc.edu April 17, 2017 Chong Ma (Statistics, USC) STAT 509 Spring 2017 April
More informationDensity Temp vs Ratio. temp
Temp Ratio Density 0.00 0.02 0.04 0.06 0.08 0.10 0.12 Density 0.0 0.2 0.4 0.6 0.8 1.0 1. (a) 170 175 180 185 temp 1.0 1.5 2.0 2.5 3.0 ratio The histogram shows that the temperature measures have two peaks,
More informationOverview Scatter Plot Example
Overview Topic 22 - Linear Regression and Correlation STAT 5 Professor Bruce Craig Consider one population but two variables For each sampling unit observe X and Y Assume linear relationship between variables
More information1 Introduction 1. 2 The Multiple Regression Model 1
Multiple Linear Regression Contents 1 Introduction 1 2 The Multiple Regression Model 1 3 Setting Up a Multiple Regression Model 2 3.1 Introduction.............................. 2 3.2 Significance Tests
More informationLinear Regression. In this lecture we will study a particular type of regression model: the linear regression model
1 Linear Regression 2 Linear Regression In this lecture we will study a particular type of regression model: the linear regression model We will first consider the case of the model with one predictor
More informationCOMPREHENSIVE WRITTEN EXAMINATION, PAPER III FRIDAY AUGUST 26, 2005, 9:00 A.M. 1:00 P.M. STATISTICS 174 QUESTION
COMPREHENSIVE WRITTEN EXAMINATION, PAPER III FRIDAY AUGUST 26, 2005, 9:00 A.M. 1:00 P.M. STATISTICS 174 QUESTION Answer all parts. Closed book, calculators allowed. It is important to show all working,
More informationInference in Regression Analysis
Inference in Regression Analysis Dr. Frank Wood Frank Wood, fwood@stat.columbia.edu Linear Regression Models Lecture 4, Slide 1 Today: Normal Error Regression Model Y i = β 0 + β 1 X i + ǫ i Y i value
More informationApplied Econometrics (QEM)
Applied Econometrics (QEM) based on Prinicples of Econometrics Jakub Mućk Department of Quantitative Economics Jakub Mućk Applied Econometrics (QEM) Meeting #3 1 / 42 Outline 1 2 3 t-test P-value Linear
More informationSTATISTICS 174: APPLIED STATISTICS FINAL EXAM DECEMBER 10, 2002
Time allowed: 3 HOURS. STATISTICS 174: APPLIED STATISTICS FINAL EXAM DECEMBER 10, 2002 This is an open book exam: all course notes and the text are allowed, and you are expected to use your own calculator.
More informationStatement: With my signature I confirm that the solutions are the product of my own work. Name: Signature:.
MATHEMATICAL STATISTICS Homework assignment Instructions Please turn in the homework with this cover page. You do not need to edit the solutions. Just make sure the handwriting is legible. You may discuss
More informationChapter 2 Inferences in Simple Linear Regression
STAT 525 SPRING 2018 Chapter 2 Inferences in Simple Linear Regression Professor Min Zhang Testing for Linear Relationship Term β 1 X i defines linear relationship Will then test H 0 : β 1 = 0 Test requires
More informationSummer School in Statistics for Astronomers V June 1 - June 6, Regression. Mosuk Chow Statistics Department Penn State University.
Summer School in Statistics for Astronomers V June 1 - June 6, 2009 Regression Mosuk Chow Statistics Department Penn State University. Adapted from notes prepared by RL Karandikar Mean and variance Recall
More information9 Correlation and Regression
9 Correlation and Regression SW, Chapter 12. Suppose we select n = 10 persons from the population of college seniors who plan to take the MCAT exam. Each takes the test, is coached, and then retakes the
More information10. Alternative case influence statistics
10. Alternative case influence statistics a. Alternative to D i : dffits i (and others) b. Alternative to studres i : externally-studentized residual c. Suggestion: use whatever is convenient with the
More informationBIOS 2083 Linear Models c Abdus S. Wahed
Chapter 5 206 Chapter 6 General Linear Model: Statistical Inference 6.1 Introduction So far we have discussed formulation of linear models (Chapter 1), estimability of parameters in a linear model (Chapter
More informationLecture notes on Regression & SAS example demonstration
Regression & Correlation (p. 215) When two variables are measured on a single experimental unit, the resulting data are called bivariate data. You can describe each variable individually, and you can also
More informationUnit 10: Simple Linear Regression and Correlation
Unit 10: Simple Linear Regression and Correlation Statistics 571: Statistical Methods Ramón V. León 6/28/2004 Unit 10 - Stat 571 - Ramón V. León 1 Introductory Remarks Regression analysis is a method for
More information1 Multiple Regression
1 Multiple Regression In this section, we extend the linear model to the case of several quantitative explanatory variables. There are many issues involved in this problem and this section serves only
More informationChapter 1: Linear Regression with One Predictor Variable also known as: Simple Linear Regression Bivariate Linear Regression
BSTT523: Kutner et al., Chapter 1 1 Chapter 1: Linear Regression with One Predictor Variable also known as: Simple Linear Regression Bivariate Linear Regression Introduction: Functional relation between
More informationStatistics - Lecture Three. Linear Models. Charlotte Wickham 1.
Statistics - Lecture Three Charlotte Wickham wickham@stat.berkeley.edu http://www.stat.berkeley.edu/~wickham/ Linear Models 1. The Theory 2. Practical Use 3. How to do it in R 4. An example 5. Extensions
More informationFall 2017 STAT 532 Homework Peter Hoff. 1. Let P be a probability measure on a collection of sets A.
1. Let P be a probability measure on a collection of sets A. (a) For each n N, let H n be a set in A such that H n H n+1. Show that P (H n ) monotonically converges to P ( k=1 H k) as n. (b) For each n
More informationEstadística II Chapter 4: Simple linear regression
Estadística II Chapter 4: Simple linear regression Chapter 4. Simple linear regression Contents Objectives of the analysis. Model specification. Least Square Estimators (LSE): construction and properties
More informationEconometrics I KS. Module 2: Multivariate Linear Regression. Alexander Ahammer. This version: April 16, 2018
Econometrics I KS Module 2: Multivariate Linear Regression Alexander Ahammer Department of Economics Johannes Kepler University of Linz This version: April 16, 2018 Alexander Ahammer (JKU) Module 2: Multivariate
More informationLectures 5 & 6: Hypothesis Testing
Lectures 5 & 6: Hypothesis Testing in which you learn to apply the concept of statistical significance to OLS estimates, learn the concept of t values, how to use them in regression work and come across
More informationR 2 and F -Tests and ANOVA
R 2 and F -Tests and ANOVA December 6, 2018 1 Partition of Sums of Squares The distance from any point y i in a collection of data, to the mean of the data ȳ, is the deviation, written as y i ȳ. Definition.
More informationSimple Regression Model Setup Estimation Inference Prediction. Model Diagnostic. Multiple Regression. Model Setup and Estimation.
Statistical Computation Math 475 Jimin Ding Department of Mathematics Washington University in St. Louis www.math.wustl.edu/ jmding/math475/index.html October 10, 2013 Ridge Part IV October 10, 2013 1
More informationChapter 1 Linear Regression with One Predictor
STAT 525 FALL 2018 Chapter 1 Linear Regression with One Predictor Professor Min Zhang Goals of Regression Analysis Serve three purposes Describes an association between X and Y In some applications, the
More informationMatrices and vectors A matrix is a rectangular array of numbers. Here s an example: A =
Matrices and vectors A matrix is a rectangular array of numbers Here s an example: 23 14 17 A = 225 0 2 This matrix has dimensions 2 3 The number of rows is first, then the number of columns We can write
More informationF3: Classical normal linear rgression model distribution, interval estimation and hypothesis testing
F3: Classical normal linear rgression model distribution, interval estimation and hypothesis testing Feng Li Department of Statistics, Stockholm University What we have learned last time... 1 Estimating
More informationSimple linear regression
Simple linear regression Biometry 755 Spring 2008 Simple linear regression p. 1/40 Overview of regression analysis Evaluate relationship between one or more independent variables (X 1,...,X k ) and a single
More informationINTERVAL ESTIMATION AND HYPOTHESES TESTING
INTERVAL ESTIMATION AND HYPOTHESES TESTING 1. IDEA An interval rather than a point estimate is often of interest. Confidence intervals are thus important in empirical work. To construct interval estimates,
More information401 Review. 6. Power analysis for one/two-sample hypothesis tests and for correlation analysis.
401 Review Major topics of the course 1. Univariate analysis 2. Bivariate analysis 3. Simple linear regression 4. Linear algebra 5. Multiple regression analysis Major analysis methods 1. Graphical analysis
More informationSimple Linear Regression
Simple Linear Regression EdPsych 580 C.J. Anderson Fall 2005 Simple Linear Regression p. 1/80 Outline 1. What it is and why it s useful 2. How 3. Statistical Inference 4. Examining assumptions (diagnostics)
More informationEstimating σ 2. We can do simple prediction of Y and estimation of the mean of Y at any value of X.
Estimating σ 2 We can do simple prediction of Y and estimation of the mean of Y at any value of X. To perform inferences about our regression line, we must estimate σ 2, the variance of the error term.
More informationFormal Statement of Simple Linear Regression Model
Formal Statement of Simple Linear Regression Model Y i = β 0 + β 1 X i + ɛ i Y i value of the response variable in the i th trial β 0 and β 1 are parameters X i is a known constant, the value of the predictor
More informationNature vs. nurture? Lecture 18 - Regression: Inference, Outliers, and Intervals. Regression Output. Conditions for inference.
Understanding regression output from software Nature vs. nurture? Lecture 18 - Regression: Inference, Outliers, and Intervals In 1966 Cyril Burt published a paper called The genetic determination of differences
More informationChapter 1. Linear Regression with One Predictor Variable
Chapter 1. Linear Regression with One Predictor Variable 1.1 Statistical Relation Between Two Variables To motivate statistical relationships, let us consider a mathematical relation between two mathematical
More informationCoefficient of Determination
Coefficient of Determination ST 430/514 The coefficient of determination, R 2, is defined as before: R 2 = 1 SS E (yi ŷ i ) = 1 2 SS yy (yi ȳ) 2 The interpretation of R 2 is still the fraction of variance
More informationLab 3 A Quick Introduction to Multiple Linear Regression Psychology The Multiple Linear Regression Model
Lab 3 A Quick Introduction to Multiple Linear Regression Psychology 310 Instructions.Work through the lab, saving the output as you go. You will be submitting your assignment as an R Markdown document.
More informationMultivariate Linear Regression Models
Multivariate Linear Regression Models Regression analysis is used to predict the value of one or more responses from a set of predictors. It can also be used to estimate the linear association between
More informationInference for the Regression Coefficient
Inference for the Regression Coefficient Recall, b 0 and b 1 are the estimates of the slope β 1 and intercept β 0 of population regression line. We can shows that b 0 and b 1 are the unbiased estimates
More informationGeneral Linear Model: Statistical Inference
Chapter 6 General Linear Model: Statistical Inference 6.1 Introduction So far we have discussed formulation of linear models (Chapter 1), estimability of parameters in a linear model (Chapter 4), least
More informationLecture 6 Multiple Linear Regression, cont.
Lecture 6 Multiple Linear Regression, cont. BIOST 515 January 22, 2004 BIOST 515, Lecture 6 Testing general linear hypotheses Suppose we are interested in testing linear combinations of the regression
More informationLinear Regression for Air Pollution Data
UNIVERSITY OF TEXAS AT SAN ANTONIO Linear Regression for Air Pollution Data Liang Jing April 2008 1 1 GOAL The increasing health problems caused by traffic-related air pollution have caught more and more
More informationEconometrics Summary Algebraic and Statistical Preliminaries
Econometrics Summary Algebraic and Statistical Preliminaries Elasticity: The point elasticity of Y with respect to L is given by α = ( Y/ L)/(Y/L). The arc elasticity is given by ( Y/ L)/(Y/L), when L
More informationCentral Limit Theorem ( 5.3)
Central Limit Theorem ( 5.3) Let X 1, X 2,... be a sequence of independent random variables, each having n mean µ and variance σ 2. Then the distribution of the partial sum S n = X i i=1 becomes approximately
More informationMFin Econometrics I Session 4: t-distribution, Simple Linear Regression, OLS assumptions and properties of OLS estimators
MFin Econometrics I Session 4: t-distribution, Simple Linear Regression, OLS assumptions and properties of OLS estimators Thilo Klein University of Cambridge Judge Business School Session 4: Linear regression,
More informationSTATISTICS 479 Exam II (100 points)
Name STATISTICS 79 Exam II (1 points) 1. A SAS data set was created using the following input statement: Answer parts(a) to (e) below. input State $ City $ Pop199 Income Housing Electric; (a) () Give the
More informationPeter Hoff Linear and multilinear models April 3, GLS for multivariate regression 5. 3 Covariance estimation for the GLM 8
Contents 1 Linear model 1 2 GLS for multivariate regression 5 3 Covariance estimation for the GLM 8 4 Testing the GLH 11 A reference for some of this material can be found somewhere. 1 Linear model Recall
More informationCAS MA575 Linear Models
CAS MA575 Linear Models Boston University, Fall 2013 Midterm Exam (Correction) Instructor: Cedric Ginestet Date: 22 Oct 2013. Maximal Score: 200pts. Please Note: You will only be graded on work and answers
More informationChapter 1 Statistical Inference
Chapter 1 Statistical Inference causal inference To infer causality, you need a randomized experiment (or a huge observational study and lots of outside information). inference to populations Generalizations
More informationST430 Exam 2 Solutions
ST430 Exam 2 Solutions Date: November 9, 2015 Name: Guideline: You may use one-page (front and back of a standard A4 paper) of notes. No laptop or textbook are permitted but you may use a calculator. Giving
More informationSection 3: Simple Linear Regression
Section 3: Simple Linear Regression Carlos M. Carvalho The University of Texas at Austin McCombs School of Business http://faculty.mccombs.utexas.edu/carlos.carvalho/teaching/ 1 Regression: General Introduction
More information[y i α βx i ] 2 (2) Q = i=1
Least squares fits This section has no probability in it. There are no random variables. We are given n points (x i, y i ) and want to find the equation of the line that best fits them. We take the equation
More information