Topic 3: Sampling Distributions, Confidence Intervals & Hypothesis Testing ECO22Y5Y: Quantitative Methods in Economics Dr. Nick Zammit University of Toronto Department of Economics Room KN3272 n.zammit utoronto.ca November 22, 217 Dr. Nick Zammit (UofT) Topic 3 November 22, 217 1 / 35 Road Map Sampling Distributions, Confidence Intervals & Hypothesis Testing Key Concepts: 1 Definitions (Sample Proportions, Sample Means, Standard Error, Confidence Intervals, Hypothesis Testing, Margin of Error, p-values) 2 Tables/Plots (Normal Critical Values, Student-t Critical Values, χ 2 -Critical Values, Confidence Regions vs. Rejection Regions) 3 Test Statistics/Intervals (One Proportion Intervals, One Mean Intervals, Difference of Proportions/Means) 4 Ideas (Central Limit Theorem, Type I & Type II Errors, Level of Confidence/Significance, Power of Tests, Setting Sample Size) Dr. Nick Zammit (UofT) Topic 3 November 22, 217 2 / 35
What are Sampling Distributions? Sampling Distribution The distribution of proportions or means over many independent samples from the same population A distribution of sample distributions Sampling Error The sampling variability from one sample to another The larger the sample size the smaller the sampling error Dr. Nick Zammit (UofT) Topic 3 November 22, 217 3 / 35 Empirical Laws Central Limit Theorem Whatever the distribution of X, as the number of terms in the sum becomes large, the distribution of X tends to a normal distribution Applies regardless of whether the underlying distribution is continuous and symmetric like the uniform distribution, continuous and asymmetric like the chi- squared distribution, or even discrete such as the Binomial distribution. CLT implies random samples can be taken from a population and the distribution of sample statistics is normally distributed Dr. Nick Zammit (UofT) Topic 3 November 22, 217 4 / 35
The Central Limit Theorem Example Random Dice Rolls (5 Obs) Mean Dice Rolls (5 obs, 5 means) 2 1 12 23 34 45 56 6Dice Roll 5 51 15 Percent 3.3 3.4 3.5 3.6 3.7 3.8 E(X) Dice Rolls (5 Obs) Random Random Dice Rolls (5 Obs) 2 2 Random Dice Rolls (5 Obs) 5 Percent 1 15 1 2 3 4 Dice Roll 5 6 5 Percent 1 15 1 2 3 4 Dice Roll 5 6 2 Random Dice Rolls (5 Obs) Mean Dice Rolls (5 obs, 5 means) Percent 5 1 15 15 1 2 3 4 Dice Roll 5 6 5 Percent 1 3.3 3.4 3.5 E(X) 3.6 3.7 3.8 Dr. Nick Zammit (UofT) Topic 3 November 22, 217 5 / 35 Characteristics of Sampling Distributions Sampling Distributions for Proportions Given several assumptions/conditions are satisfied then the sampling distribution of ˆp is modelled by a Normal distribution with µ(ˆp) = p and SD(ˆp) = pq n Sampling Distributions for Means Given several assumptions/conditions are satisfied then the sampling distribution of x is modelled by a Normal distribution with µ( x) = x and σ( x) = SD( x) = σ n Dr. Nick Zammit (UofT) Topic 3 November 22, 217 6 / 35
Standard Error Standard Error of Proportions An estimate of standard deviation for a sampling distribution For a sample proportion ˆp the standard error is SE(ˆp) = ˆp ˆq n Standard Error of Mean An estimate of standard deviation for a sampling distribution For a sample mean x the standard error is SE( x) = s n Dr. Nick Zammit (UofT) Topic 3 November 22, 217 7 / 35 Assumptions and Conditions for Normality Assumptions 1 Independence Assumption The sampled values must be independent of each other Conditions 1 Randomization Condition The data values must be sampled randomly 2 The 1% Condition The sample n should be no more tha% of the population 3 Large-Sample Condition If the underlying population distribution is not unimodal and symmetric the sample size should be large (n 5) If the underlying population distribution is unimodal and symmetric the sample size can be small (n < 5) Dr. Nick Zammit (UofT) Topic 3 November 22, 217 8 / 35
Graphical Checks for Normality (Large-Sample Condition) The Normal Probability Plot Scatter plot of X values on the vertical axis against Normal Scores on the horizontal axis Deviations from a straight line indicate non-normality Normal Scores Solve for Order Statistic Medians (OSMs): U i = (i a) (n+1 2a) for i = 1, 2,..., n where a = 3/8 if n 1 and a =.5 if n > 1 Normal Scores are inverse normal function values of OSMs: N i = G(U i ) where G(X ) is the inverse of normal cdf Dr. Nick Zammit (UofT) Topic 3 November 22, 217 9 / 35 Graphical Checks for Normality 4 5 6 7 8 9 grade -2-1 1 12 2Normal Score Normal Probability Plot of Student Grades Normal Probability Plot of Student Grades 4 5 6 grade 7 8 9-2 -1 Normal Score 1 2 Dr. Nick Zammit (UofT) Topic 3 November 22, 217 1 / 35
What is a Confidence Interval? Confidence Interval Provides a range of likely values for the true but unknown population parameter (such as a proportion or mean) Given some margin of error based on the level of confidence determines what guesses of the true parameter are likely Confidence Intervals take the form: Estimate ± Margin of Error (ME) Dr. Nick Zammit (UofT) Topic 3 November 22, 217 11 / 35 How is Margin of Error Determined? Margin of Error Margin of Error (ME) = Standard Error (SE) Critical Value (CV) Critical Value Choose a level of confidence corresponding to a probability value from the sampling distribution Use the distribution (ex. Normal, Student-t, Chi-Squared, etc...) to look up the CV from the inverse CDF This can be done using a table or Stata Dr. Nick Zammit (UofT) Topic 3 November 22, 217 12 / 35
Critical Values for Common Confidence Levels Confidence Interval Critical Values Level of Confidence Critical Value (c = 1 α) Zc.75 1.15.8 1.28.85 1.44.9 1.645.95 1.96.98 2.33.99 2.58 Dr. Nick Zammit (UofT) Topic 3 November 22, 217 13 / 35 Estimating Confidence Intervals for One Parameter One Proportion z-interval ˆp ± Zc SE(ˆp) ˆp ˆq where SE(ˆp) = n and n ˆp > 1 nˆq > 1 One Mean z-interval x ± Z c SE( x) where SE( x) = s n and independence, randomization, 1% condition hold Dr. Nick Zammit (UofT) Topic 3 November 22, 217 14 / 35
How to set sample size? Deciding Sample Size For a one proportion z-interval: ( ) ˆp ˆq ME = Zc n = n ( ) 1 ME 2 (Zc ) 2 (ˆp ˆq) Setting Sample Size For a one mean z-interval: ( ) s ME = Zc n n = ( ) 1 ME 2 (Zc ) 2 ( s 2) Dr. Nick Zammit (UofT) Topic 3 November 22, 217 15 / 35 Estimating Confidence Intervals for Finite Samples One Mean t-interval x ± t c SE( x) where SE( x) = s n, tc = t1 α,n 1 and the independence, randomization, and 1% condition all hold When do you use t instead of z? With small (any finite?) sample confidence intervals for the mean use t-distribution not standard normal When s is estimated by SE( x) then use t-distribution Dr. Nick Zammit (UofT) Topic 3 November 22, 217 16 / 35
Confidence Interval for Difference in Two Parameters Difference in Two Proportion z-interval ˆp 1 ˆp 2 ± Z c SE(ˆp 1 ˆp 2 ) where SE(ˆp 1 ˆp 2 ) = SE(ˆp 1 ) 2 + SE(ˆp 2 ) 2 = ˆp1 ˆq 1 + ˆp 2 ˆq 2 Difference in Two Means t-interval x 1 x 2 ± t c SE( x 1 x 2 ) where SE( x 1 x 2 ) = SE( x 1 ) 2 + SE( x 2 ) 2 = s 2 1 + s2 1 Dr. Nick Zammit (UofT) Topic 3 November 22, 217 17 / 35 What is Hypothesis Testing? Hypothesis Testing Assesses the validity of a hypothesis (called the null hypothesis H ) about an unknown population parameter (θ) Requires the formulation of an alternative hypothesis (H A ) against which to test the null hypothesis Forces the investigator to choose a level of significance based on the trade-off between type I & type II errors Dr. Nick Zammit (UofT) Topic 3 November 22, 217 18 / 35
Approach to Hypothesis Testing The Five (Actual) Steps of Hypothesis Testing Plan Do 1 Hypothesis (formulate H and H A ) 2 Level of Significance (choose α) 3 Assumptions (check for independence, randomization, 1% condition) 4 Data (summarize it) 5 Statistical test (calculate test stat or associated p-value) Report 6 Statistical significance (report how significant) 7 Conclusion (reject the null or fail to reject the null) 8 Implications (interpret the conclusion) Dr. Nick Zammit (UofT) Topic 3 November 22, 217 19 / 35 How to formulate a Hypothesis (Step 1) H vs. H A The null hypothesis (H ) is the maintained hypothesis assumed to be true until proven otherwise The hypothesis (H A ) is a composite hypothesis (encompassing many outcomes) that must be true if the null is not Two Sided Test: One Sided Test: H : θ = θ H A : θ θ H : θ θ H : θ θ H A : θ > θ H A : θ < θ Dr. Nick Zammit (UofT) Topic 3 November 22, 217 2 / 35
How to choose α? (Step 2) Level of Significance? The tolerance you are willing to accept for an incorrect null hypothesis to be accepted The flip side to your level of confidence The probability of making a type I error Error Types Dr. Nick Zammit (UofT) Topic 3 November 22, 217 21 / 35 Data to Summarize? (Step 3) Data for Hypothesis Testing 1 Do you know the population distribution? Lets assume yes 2 Do you know the population variance? Lets assume yes 3 What is the Critical Value? Get this from appropriate distribution Standard Normal Critical Values Level of Significance One Sided CV Two Sided CV (α) Zα Zα/2.1 1.282 1.645.5 1.645 1.96.1 2.326 2.58 Dr. Nick Zammit (UofT) Topic 3 November 22, 217 22 / 35
Calculate Test Statistic (Step 4) What is the test statistic? Assume: H : µ µ and H A : µ > µ where X N(µ, σ 2 /n) Pr( X > x) = Pr ( Z > x µ ) σ/ Test Stat = Z = x µ n σ/ n Assume: H : µ = µ and H A : µ µ where X N(µ, σ 2 /n) Pr( X = x) = Pr ( Z = x µ ) σ/ Test Stat = Z = x µ n σ/ n Note: we are assuming we know σ 2 or we need to use s 2 and t-dist Dr. Nick Zammit (UofT) Topic 3 November 22, 217 23 / 35 How do we conclude? (Step 5) Critical Value Approach Take our test statistic calculated in step 4 (Z = x µ σ/ n ) Take the critical value found in step 3 (Z α or Z α/2 ) Reject H if Z > Z Fail to reject H if Z Z Dr. Nick Zammit (UofT) Topic 3 November 22, 217 24 / 35
How do we conclude? (Step 5 - Alt) P-Value Approach Compare probability value associated with test statistic with level of significance instead of critical value Reject H if p-value α Fail to reject H if p-value > α Flow diagram summarizing p-value method for t-test: Dr. Nick Zammit (UofT) Topic 3 November 22, 217 25 / 35 How effective is our test? Calculating power of a test If we knew the true population mean we could calculate the exact power of our test If we don t know the true population mean we can still discuss the relative power of tests Power of a test will increase if: 1 The mean of the test statistic is farther from the true mean 2 The significance level of the test increases 3 We correctly calculate one sided test instead of two sided test 4 We increase our sample size Dr. Nick Zammit (UofT) Topic 3 November 22, 217 26 / 35
Lets return to step 3 Alternative situations in Hypothesis Testing 1 Do you know the population distribution? If NO invoke CLT 2 Do you know the population variance? If NO calculate SE( x) and use t-dist 3 Do you know either of the above? If NO invoke CLT twice, calculate SE( x) and use t-dist Dr. Nick Zammit (UofT) Topic 3 November 22, 217 27 / 35 When do I need a t-dist? If you know σ 2 then you should use normal If you do not know σ 2 then you should use t If you have a large sample and don t know σ 2 you can approximate t-dist. with normal Normal compared with t-dist. a Pr(X > a) b Pr(X > a) t-dist(1 df) 2.326.21 1.645.65 t-dist(2 df) 2.326.16 1.645.58 t-dist(3 df) 2.326.14 1.645.55 t-dist(5 df) 2.326.12 1.645.53 t-dist(1 df) 2.326.11 1.645.52 Normal 2.326.1 1.645.5 Dr. Nick Zammit (UofT) Topic 3 November 22, 217 28 / 35
Test for the difference in proportions/means Formal Steps Confirm independence of samples If samples independent and known σ 2 proceed with appropriate difference test (equal or unequal variances) If samples independent but unknown σ 2 test σ 2 1 = σ2 2 If σ 2 1 = σ 2 2 use difference test with equal variances If σ 2 1 σ 2 2 use difference test with unequal variances If samples are not independent use difference test for matched pairs Dr. Nick Zammit (UofT) Topic 3 November 22, 217 29 / 35 Difference independent means (known unequal variances) What is the test statistic? Assume: H : µ 1 = µ 2 = and H A : µ 1 = µ 2 Pr( X 1 X 2 = x 1 x 2 ) = Pr where X 1 X 2 N(, σ2 1 + σ2 2 ) n 2 Z = ( x 1 x 2 ) σ 2 1 + σ2 2 n 2 Test Stat = Z = ( x 1 x 2 ) σ 2 1 + σ2 2 n 2 Reject H if Z > Z α/2 Dr. Nick Zammit (UofT) Topic 3 November 22, 217 3 / 35
Testing variance of a distribution What is the test statistic? Assume: H : σ 2 σ 2 and H A : σ 2 > σ 2 where X i N(µ, σ 2 ) and (n 1)s2 X σ 2 χ 2 n 1 Test Stat = χ 2 = (n 1)s2 X σ 2 Reject H if χ 2 > χ 2 α,(n 1) Dr. Nick Zammit (UofT) Topic 3 November 22, 217 31 / 35 Difference independent means (unknown equal variances) What is the test statistic? Assume: H : µ 1 = µ 2 = and H A : µ 1 = µ 2 where X 1 X 2 t(, s 2 p ( 1 + 1 n 2 DoF: v = ( + n 2 2) and s 2 p = ( 1)s 2 1 + (n 2 1)s 2 2 + n 2 2 ) ) Test Stat = t = ( x 1 x 2 ) s p 1 + 1 n 2 Reject H if t > t α/2,(v) Dr. Nick Zammit (UofT) Topic 3 November 22, 217 32 / 35
Difference independent means (unknown unequal variances) What is the test statistic? Assume: H : µ 1 = µ 2 = and H A : µ 1 = µ 2 where X 1 X 2 t(, s2 1 + s2 2 n 2 ) and DoF: v = [(S 1 2/) + (S2 2/n 2)] 2 (S1 2/) 2 1 + (S2 2 /n 2) 2 n 2 1 Test Stat = t = ( x 1 x 2 ) s 2 1 + s2 2 n 2 Reject H if t > t α/2,(v) Dr. Nick Zammit (UofT) Topic 3 November 22, 217 33 / 35 Testing the difference of matched pair sample means How does this test work? Consider n to be the number of pairs of d = X 1 X 2 Calculate the mean of the difference d = Calculate the standard deviation s d = n d i i=1 n n (d i d) 2 i=1 n 1 Choose null ex. H : d = and H A : d Calculate a test stat for the difference t = d s d / n Reject H if t > t α/2,(n 1) Dr. Nick Zammit (UofT) Topic 3 November 22, 217 34 / 35
Supplementary References Dr. Nick Zammit (UofT) Topic 3 November 22, 217 35 / 35