Monte Carlo Studies. Monte Carlo Studies. Sampling Distribution

Transcription

1 Monte Carlo Studies Do not let yourself be intimidated by the material in this lecture This lecture involves more theory but is meant to imrove your understanding of: Samling distributions and tests of significance Tye I error Power and Tye II error Monte Carlo Studies Samling Distribution What is it? Examles Emirical and theoretical samling distributions Monte Carlo Procedures Purose Procedure Studying the t Distribution Samling Distribution Imagine an emirical ulation of 000 observations (numbers) We can calculate: the mean (μ) = 100 the standard deviation (σ) = 8 describe the shae of the distribution Examles: normal, exonential, rectangular 1

2 Samling Distribution Take a samle of 10 observations Calculate samle M and SD Would they equal 100 and 8 resectively? What would haen if we took a number of samles of 10 observations and: Comuted the mean of the samle means Comuted the mean of the samle SDs What would these values look like? Samling Distribution Let s reeat with indeendent t-test Obtain two indeendent samles from their ulations Comute means, SDs, and t values Reeat 1000 times and obtain 1000 t values Plot the distribution of t values This is your emirical distribution Samling Distribution Comare to theoretical distribution of t Mathematicians give us the following density function y df + 1 t = C 1 + df This function is used to determine the critical values for t-tests

3 Monte Carlo Methods Puroses: To study samling distributions emirically To study the effects of violating assumtions We will focus on the t distribution Can study Tye I error rate when null is true and you have violated assumtions Can study Tye II error rate and ower when null is false Monte Carlo Methods General Procedure: Establish the ulation of interest Obtain random samles Comute statistic of interest Reeat several times to obtain stable estimates (e.g., 1000) Make a frequency distribution of your results This is your emirical samling distribution Monte Carlo Methods Things you can investigate: Proortion of the number of emirical values that exceed some critical value (based on the theoretical distributions) Study the effect of violating any assumtions How would you demonstrate the central limit theorem? 3

4 Studying the t Distribution Assumtions of the indeendent grous t-test 1. Observations come from normally distributed ulations. Poulation variances are equal 3. Observations reresent random samles from ulations 4. Null hyothesis is true Boneau Study (1960) Investigated the effects of: violating normality violating homogeneity of variance Unequal samle sizes when the null hyothesis is true (i.e., samles are drawn from ulations that are equal: μ1= μ= μ) Boneau Study (1960) To determine if violation of Normality influenced Tye I error: Run N(0,1)5 N(0,1)5 Run E(0,1)5 E(0,1)5 Comare roortions of Tye I errors: i.e., number of times that t observed was outside the range of t critical to + t critical Use tests of roortions (see next slide) 4

5 Tests of roortions: Tests of roortions: 1. Comaring a samle roortion with a ulation value Z = q q = 1 n. Comaring two samle roortions Z = q + n1 n n = number of tests n + n n + n 1 1 = q = 1 1 Results of Boneau Study (1960).05 level: Exected 50 significant t's out of 1000 t- tests.01 level: Exected 10 significant t's out of 1000 t- tests All assumtions satisfied: Samling from same normal distribution N(0,1)5 N(0,1)5 53 beyond critical value for.05 Exected 50 z = 0.44, ns (see next slide to calculate this value) 9 beyond critical value for.01 Exected 10 Z = -0.3, ns Conclusion: very similar to exected values 5

6 Examle of calculation: Z = n q q = 1 n = number of tests (.050)(.950) 1000 Z = 0.44, ns Critical value.05 = 1.96 Critical value.01 =.58 Both samles from same rectangular distribution R(0,1)5 R(0,1)5 51 beyond critical value for.05 Exected 50 z = 0.15, ns 10 beyond critical value for.01 Exected 10 Z = 0, ns Very similar results with samles sizes = 15 Conclusion: very little effect Other tests violating normality similar results As long as n is reasonably large (n=15) Heterogeneous variances N(0,1)5 N(0,4)5 64 z =.03, < z =.54, <.05 N(0,1)15 N(0,4)15 49 z = -.15, ns 11 z = 0.3, ns Conclusion: Tye I error rate a bit higher with small n 6

7 Heterogeneous variances in combination with different samle sizes N(0,1)5 N(0,4)15 10 z = -5.80, <.01 1 z = -.86, <.01 Denominator of t is larger when the larger grou has larger variance Conclusion: larger denominator, smaller t s, fewer rejections N(0,1)15 N(0,4)5 160 z = 15.96, < z = 15.89, <.01 Conclusion: oosite when the larger grou has the smallest variance What if the null is false? Examle: N(0,1)5 N(1,1)5 89/1000 rejections of the null (at.05) Power =.89 Tye II error = 1 ower = /1000 rejections of the null (at.01) Power =.103 Tye II error rate =.897 Not very good How could you increase ower? Demonstration of the effect of samle size on ower Analysis 1: samle size = 5 N(0,1)5 N(1,1)5 85/1000 rejections of the null (at.05) Power =.85 Tye II error = 1 ower = /1000 rejections of the null (at.01) Power =.090 Tye II error rate =.910 Not very good Note: these results not exactly as revious slide but very close 7

8 Demonstration of the effect of samle size on ower Analysis : samle size = 10 N(0,1)10 N(1,1)10 570/1000 rejections of the null (at.05) Power =.570 Tye II error = 1 ower = /1000 rejections of the null (at.01) Power =.93 Tye II error rate =.707 Better but still not good enough Demonstration of the effect of samle size on ower Analysis 3: samle size = 15 N(0,1)15 N(1,1)15 763/1000 rejections of the null (at.05) Power =.763 Tye II error = 1 ower = /1000 rejections of the null (at.01) Power =.506 Tye II error rate =.494 Better but still not good enough Demonstration of the effect of samle size on ower Analysis 4: samle size = 0 N(0,1)0 N(1,1)0 879/1000 rejections of the null (at.05) Power =.879 Tye II error = 1 ower = /1000 rejections of the null (at.01) Power =.665 Tye II error rate =.335 Much better 8

9 Demonstration of the effect of samle size on ower Let s test whether ower is imroved significantly when we increase samle size from 15 to 0 Recall our results: (879/1000 at.05 for n=0) and (763/1000 for n=15) 1 Z = Z = = 6.77, < q + (.81)(.179) + n1 n n1 + n = n + n 1 q = ( 1000) +.763( 1000) = 000 q = 1.81 Values required for significance for given df df t t