1 Descriptive statistics. 2 Scores and probability distributions. 3 Hypothesis testing and one-sample t-test. 4 More on t-tests

Transcription

1 Overall Overview INFOWO Statistics lecture S3: Hypothesis testing Peter de Waal Department of Information and Computing Sciences Faculty of Science, Universiteit Utrecht 1 Descriptive statistics 2 Scores and probability distributions 3 Hypothesis testing and one-sample t-test 4 More on t-tests 5 Homegeneity and reliability 6 Correlation and prediction 7 Analysis of variance 8 Chi 2 -test 9 Q&A lecture Lecture S3: 1 / 47 Lecture S3: 2 / 47 Today Today s overview Does the facebook diet really work???? Or is the Breezer diet better? Recap Hypotheses testing (Chapter 8) Test procedure Hypotheses H0 and H 1 t-distribution (Chapter 9) One-sample t-test??? Lecture S3: 3 / 47 Lecture S3: 4 / 47

2 Recap Hypothesis testing Normal distribution: Shape Parameters µ and σ Calculations Sample and Sampling distribution: Sample distribution = (theoretical) distribution of data when one person/item is measured Sampling distribution = distribution of data when sample of n items are measured and average is taken. Central Limit Theorem: Sampling distribution is approximately normal. Confidence intervals Recal from Lecture M1: Hypothesis Empirical formulation of proposition, stated as relationship between variables. Examples: There is relation between a students IQ score and his grade point average. Students who live at home with their parents spend more time on Facebook. There is a positive correlation between Facebook use and narcissism. Lecture S3: Recap 5 / 47 Lecture S3: Hypothesis testing Introduction 6 / 47 Hypothesis testing Hypothesis testing THE BRAND NEW FACEBOOK DIET (AS SEEN ON THE WEB)! Spend time on and lose Randomly select test group Put test group on Facebook regime (8 hours per day) for 4 weeks Weigh test persons Compare with known mean from reference population Results? People may gain weight due to inactivity People may lose weight due to lack of time to eat Weight may not change at all... without excessive exercises! Lecture S3: Hypothesis testing Introduction 7 / 47 Lecture S3: Hypothesis testing Introduction 8 / 47

3 Some data Basic experimental situation Mean weight of reference population (before diet) is µ = 80, σ = 20 After the test trial: Mean weight in test group is X = 76. Does this mean the diet works? Or is it random fluctuation? What if X = 86? What if X = 70? Lecture S3: Hypothesis testing Introduction 9 / 47 Lecture S3: Hypothesis testing Introduction 10 / 47 Hypothesis testing 1. Formulation of the hypotheses Pose two possible, exclusive, hypotheses about the world or about a population: A statistical method that uses sample data to evaluate a hypothesis about a population Basic steps of hypothesis testing 1 Formulate hypotheses 2 Set criteria for decision 3 Collect data and compute sample statistic 4 Make decision Hypotheses Null hypothesis (H 0 ): States that, in the general population, there is no change, no difference, or no relationship. Alternative hypothesis (H 1 ): States that there is a change, a difference, or a relationship in the general population. Diet example: H 0 H 1 : µ after = 80, (Facebook diet has no effect) : µ after 80, (Facebook diet has effect) Lecture S3: Hypothesis testing Procedure 11 / 47 Lecture S3: Hypothesis testing Procedure 12 / 47

4 2. Set criteria for decision 2. Set the criteria for the decision If H 0 is true, which values for sample means are likely? Significance level or alpha level Defines boundary between likely and unlikely Denoted by symbol α Value is determined beforehand (i.e. before you take a sample!) Typical values are α = 0.05 or α = Critical region The extreme sample values that are very unlikely Boundaries of critical region are determined by α. Lecture S3: Hypothesis testing Procedure 13 / 47 Lecture S3: Hypothesis testing Procedure 14 / 47 Critical region of Z = X 80 σ X for α = 0.05 Check! True or false? The critical region defines unlikely values if the null hypothesis is true. (True) If the alpha level is decreased, the critical region becomes smaller. (True) Lecture S3: Hypothesis testing Procedure 15 / 47 Lecture S3: Hypothesis testing Procedure 16 / 47

5 Critical region boundaries 3. Collect data and compute sample statistic Data is collected after hypotheses are formulated. Data is collected after criteria for decision are set. This sequence assures objectivity. Compute a sample statistic (in this case Z-score) to show the exact position of the sample. Lecture S3: Hypothesis testing Procedure 17 / 47 Lecture S3: Hypothesis testing Procedure 18 / Make decision Examples After calculation of the sample statistic: If sample data are in the critical region: null hypothesis is rejected. If the sample data are not in the critical region: the researcher fails to reject the null hypothesis. Example outcome experiment A: Sample size n = 16 Observed sample mean is X = 75 Z-score for observed sample mean is Z = X µ (σ/ n) Decision: Not in critical region, so retain H 0 Example outcome experiment B: = /4 = 1 Sample size n = 25 Observed sample mean is X = 88 Z-score for observed sample mean is Z = X µ σ/ n Decision: In critical region, so reject H 0 = /5 = +2 Lecture S3: Hypothesis testing Procedure 19 / 47 Lecture S3: Hypothesis testing Procedure 20 / 47

6 Check! Why the null hypothesis? True or False When the Z-score is quite large, it shows the null hypothesis is true. (False) A decision to retain the null hypothesis means you showed that the treatment has no effect. (False) Question: Seems odd to focus on null hypothesis, which we do not believe to be true? Answer: In logic, it is easier to demonstrate that a universal hypothesis is false than to prove that it is true. (Recall Popper s falsification criterion from Lecture M1!) So: Usually the alternative hypothesis corresponds to your experimental hypothesis Lecture S3: Hypothesis testing Procedure 21 / 47 Lecture S3: Hypothesis testing Procedure 22 / 47 What could possibly go wrong? Possible test outcomes Type I error H0 is true, but by chance outcome is such that H 0 is rejected. (Test has indicated a non-existant treatment effect) Probability that type I error occurs is equal to significance level α. Type II error H0 is not true, but outcome is such that H 0 is not rejected. (Test has failed to detect a real treatment effect) Probability of Type II error is sometimes denoted with symbol β. Summary H 0 true H 1 true (no effect) (effect exists) retain H 0 OK Type II error reject H 0 Type I error OK Lecture S3: Hypothesis testing Uncertainty and errors 23 / 47 Lecture S3: Hypothesis testing Uncertainty and errors 24 / 47

7 Some remarks Directional (one-tailed) test Terminology in literature: A result is called significant or statistically significant if it makes us reject the null hypothesis. Factors influencing hypothesis test: Size of difference between sample mean and original population mean: Appears in numerator of the Z-score Variability of the scores: Influences size of the standard error So far: Two-sided (two-tailed) hypothesis: Does not indicate a direction for the possible effect or relation What if you expect an effect in a certain direction? One-sided (one-tailed) hypothesis: Indicates a possible direction for the assumed effect or relation Number of scores in the sample: Influences size of the standard error Lecture S3: Hypothesis testing Uncertainty and errors 25 / 47 Lecture S3: Hypothesis testing Directional hypothesis 26 / 47 Example directional hypothesis Critical region THE BREEZER DIET (AS SEEN ON MTV)! How does drinking 4 Breezers a day affect your weight? We expect that Breezers make you gain weight. 1 Formulate the hypothesis: H0 : µ after 80 (null hypothesis) H1 : µ after > 80 (alternative hypothesis) 2 Set criteria for decision: Significance level α = 0.05 Critical region: Z 1.65 (From Column C in Table B.1) We take a sample of n = 25 test persons. Lecture S3: Hypothesis testing Directional hypothesis 27 / 47 Lecture S3: Hypothesis testing Directional hypothesis 28 / 47

8 Example directional hypothesis 3 Collect data and compute sample statistic Sample size n = 25 Population σ = 20 Sample mean X = 87 Standard error of the means is σm = 20 5 = Z = = Make decision: Z-score is in critical region, so we reject H0 So: We reject H 0 and conclude that Breezers makes you gain weight! Lecture S3: Hypothesis testing Directional hypothesis 29 / 47 Unknown variance More often than not the population variance σ is unknown So also standard error of the mean σ M is not known What to do? Use sample standard deviation s 2 = SS n 1 as estimate for σ2. σ 2 s Replace σ M = n with estimated standard error s 2 M = n If Variance σ known, use: If Variance σ unknown, use: σ M = σ s M = s n n Z = X µ σ M t = X µ s M Z has a standard normal t has a t-distribution with distribution under H 0 df = n 1 under H 0 Lecture S3: Hypothesis testing Directional hypothesis 30 / 47 t-distribution t-distribution: plots Is a family of distributions Resembles the standard normal distribition in shape and spread Has a bit more mass in the tails (flatter) Has one parameter: degrees of freedom (df) For df = the t-distribution equals the standard normal distribution Sometimes also called Student distribution. William Sealey Gosset ( ) Lecture S3: Hypothesis testing t-distribution 31 / 47 Lecture S3: Hypothesis testing t-distribution 32 / 47

9 Example Example (continued) We would like to test the following hypothesis Information science students on average spend 20 hours per week on INFOWO H 0 : µ = 20 H 1 : µ 20 We set criteria for decision (null hypothesis) Significance level α = 0.05 (alternative hypothesis Assume we have a sample of n = 10 Information Science students Observed sample mean X = 21.2 Observed standard deviation s = 3.4 Recall calculation of t: t = X µ s M = X µ (s/ n) Observed data: t = X µ (s/ = n) (3.4/ 10) = = 1.11 Lecture S3: Hypothesis testing t-distribution 33 / 47 Lecture S3: Hypothesis testing t-distribution 34 / 47 Example (continued) Two-sided test, significance level α = 0.05 Decision rule: Reject H0 if t < t crit or if t > t crit Do not reject H0 if t crit t < t crit How do we determine t crit? Look up value in Table B.2 t-test: critical value t Table cum. prob t.50 t.75 t.80 t.85 t.90 t.95 t.975 t.99 t.995 t.999 t.9995 one-tail two-tails df z % 50% 60% 70% 80% 90% 95% 98% 99% 99.8% 99.9% Confidence Level Lecture S3: Hypothesis testing t-distribution 35 / 47 Lecture S3: Hypothesis testing t-distribution 36 / 47

10 Example (continued) One-sample t-test Two-sided test, significance level α = t crit = (df = n 1 = 9) Decision rule: Reject H0 if t < or t > Do not reject H0 if t < So? t = 1.11, so do not reject H 0. Properties of the one-sample t-test Compare one sample mean with a reference value (a mean value that was determined earlier or beforehand) Population standard deviation σ unknown Sample size n < 120 Use t-distribution: t = X µ s M = X µ (s/ n) Determine the correct value for degrees of freedom (df = n 1) Use Table B.2 to determine critical value Lecture S3: Hypothesis testing t-distribution 37 / 47 Lecture S3: Hypothesis testing One-sample t-test 38 / 47 One-sample t-test: income example One-sample t-test: income example Student Income Question: Is your income different from what the average Dutch student (living away from home) needs? According to the National Institute for Family Finance Information (NIBUD) the average Dutch student needs per month: e 962. Hypothesis: Your income is different from what the average Dutch student, living away from home, needs? Formulation of hypotheses: H0 : µ = 962 (null hypothesis) H1 : µ 962 (alternative hypothesis) Significance level: α = 0.05 Lecture S3: Hypothesis testing One-sample t-test 39 / 47 Lecture S3: Hypothesis testing One-sample t-test 40 / 47

11 One-sample t-test: income example One-sample t-test: income example Your net income Your income: So: Income per month n Valid Missing 3.00 Mean Std. Deviation Your average income per month is e 833 This is e 129 less than the NIBUD average income for Dutch students Std.error Mean = Std.Deviation/ Question: Is this difference due to randomness or it is significant? Lecture S3: Hypothesis testing One-sample t-test 41 / 47 Lecture S3: Hypothesis testing One-sample t-test 42 / 47 One-sample t-test: income example SPSS: Menu Analyze > Compare Means > One-sample T Test... One-sample t-test: Formula s and output t = X µ s M df = n 1 s M = s n SPSS Output So, s = , s M = = t = = 1.95 t crit 2.01, (df = 48), so H 0 not rejected. Lecture S3: Hypothesis testing One-sample t-test 43 / 47 Lecture S3: Hypothesis testing One-sample t-test 44 / 47

12 One-sample t-test in SPSS: More output One-sample t-test: income example Conclusion: (Also: the proper way of reporting the result) Sig. column: p-value or significance value of the test result Rule Probability to get the measured data or smaller under the null hypothesis (or P( t > 1.950) under H 0 ). Indication of how extreme the measured data is. Your average income (according to the questionnaire) is e 833 per month This is e 129 less than the NIBUD average income for Dutch students, living away from home This difference is not significant: t = 1.95 (df = 48), p =.057 (two-sided) If p-value α, do not reject H 0, If p-value < α, reject H 0. So? Do not reject H 0. Lecture S3: Hypothesis testing One-sample t-test 45 / 47 Lecture S3: Hypothesis testing One-sample t-test 46 / 47 Lessons learnt What is inferential statistics? The proper procedure for hypothesis testing One-sample t-test: What it is And how to use it The role of the hypothesis in research Lecture S3: Hypothesis testing One-sample t-test 47 / 47