the logic of parametric tests

Transcription

1 the logic of parametric tests define the test statistic (e.g. mean) compare the observed test statistic to a distribution calculated for random samples that are drawn from a single (normal) distribution. the distribution is parametrized based on your sample ask what is the probability of the data under the model

2 the logic of parametric tests: example with a t-test t-distribution under H0: the distribution of the test statistic calculated for 2 random samples drawn from a single (normal) distribution Group 0 Group 1

3 the logic of parametric tests: example with a t-test step 1: extract sample data mean var diff group group Group 0 Group 1

4 the logic of parametric tests: example with a t-test step 2: calculate the test statistic - t mean var diff group group

5 the logic of parametric tests: example with a t-test step 2: calculate the test statistic - t compare test statistic to a value from a theoretical distribution mean var diff group group t = , df = , p-value =

6 the logic of parametric tests define the test statistic (e.g. mean) compare the observed test statistic to a distribution calculated for random samples that are drawn from a single (normal) distribution. ask what is the probability of the data under the model This is where all the assumptions (normality, homogeneity of avarice) come from!

7 Assumptions: t-test 1) normality of the data 2) samples are independent! its possible to test the 1st assumption using histograms, qqplot, and tests for normality (e.g. Shapiro-Wilk test) often, the problem is lack of power due to small n

8 Assumptions: ANOVA 1) normality of the data 2) samples are independent 3) homogeneity of variance (critical)! its possible to test the 1st assumption using histograms, qqplot, and tests for normality. power problem more extreme its critical to test for homogeneity of variance (levenetest in library car)

9 Assumptions: regression 1) normality of the residuals 2) samples are independent 3) homogeneity of variance! it is generally difficult to test regression assumptions. its possible to test the 1st assumption using histograms, qqplot, and tests for normality on residuals. remember to think about power

10 Assumptions: regression 1) normality of the residuals 2) samples are independent 3) homogeneity of variance! Response variable Covariate

11 Assumptions: regression 1) normality of the residuals 2) samples are independent 3) homogeneity of variance! Response variable model=lm(y~x) hist(model$residuals) plot(model$fitted.values,y) Covariate points in extreme x values have strong leverage

12 More assumptions: regression 1) normality of the residuals 2) samples are independent 3) homogeneity of variance 4) X is known with no error! fecundity population density library(lmodel2) lmodel2(density~ fecundity, data=data, nperm=99)

13 More assumptions: regression Call: lmodel2(formula = Predicted_by_model ~ Survival, data = mod2ex1, nperm = 99) n = 54 r = r-square = Parametric P-values: 2-tailed = e-15 1-tailed = e-15 Angle between the two OLS regression lines = degrees Permutation tests of OLS, MA, RMA slopes: 1-tailed, tail corresponding to sign A permutation test of r is equivalent to a permutation test of the OLS slope P-perm for SMA = NA because the SMA slope cannot be tested Regression results Method Intercept Slope Angle (degrees) P-perm (1-tailed) 1 OLS MA SMA NA Confidence intervals Method 2.5%-Intercept 97.5%-Intercept 2.5%-Slope 97.5%-Slope 1 OLS MA SMA Eigenvalues: H statistic used for computing C.I. of MA: The interesting aspect of the MA regression equation in this example is that library(lmodel2) lmodel2(density~ fecundity, data=data, nperm=99)

14 More assumptions: regression For species data, samples cannot be truly considered independent, because they share a common history its possible to account for this correlation if phylogenetic information is available If this is applicable for your data, learn more at species data wiki/phylogenetic_comparative_methods or here fecundity population density

15 What if my assumptions are invalid?

16 the logic of parametric tests define the test statistic (e.g. mean) compare the observed test statistic to a distribution calculated for random samples that are drawn from a single (normal) distribution. ask what is the probability of the data under the model! Can I compare my data to another distribution?

17 Permutation, Montecarlo, and bootstrap: what s the deal? Permutation & randomization tests: generating the probability of test statistics from the data, rather than a theoretical distribution Montecarlo: generating the probability of test statistics from the process, rather than a theoretical distribution Bootstrap, Jackknife: estimating bias and precision of estimates from the data, rather than a theoretical distribution

18 Permutation, Montecarlo, and bootstrap: what s the deal? Permutation & randomization tests: generating the probability of test statistics from the data, rather than a theoretical distribution Montecarlo: generating the probability of test statistics from the process, rather than a theoretical distribution Bootstrap, Jackknife: estimating bias and precision of estimates from the data, rather than a theoretical distribution

19 the logic of randomisation tests define the test statistic (e.g. mean) shuffle the data, extract test statistic repeat for all possible permutations (permutation test) or a sub-sample of them (randomization) ask what is the probability of the observed test statistics under the generated distribution

20 the logic of randomisation tests: example with a t-test step 1: extract sample data mean var diff group group Group 0 Group 1

21 the logic of randomisation tests: example with a t-test step 2: shuffle the data, extract the test statistic. repeat. group0 group1 diff iteration iteration iteration iteration iteration iteration iteration iteration iteration

22 the logic of parametric tests ask what is the probability of the observed test statistics under the generated distribution! Min. : st Qu.: Median : Mean : rd Qu.: Max. :

23 the logic of randomisation tests: example with a t-test define the test statistic (e.g. mean) shuffle the data, extract test statistic repeat for all possible permutations (permutation test) or a sub-sample of them (randomization) ask what is the probability of the observed test statistics under the generated distribution No assumptions regarding the distribution of population

24 the logic of randomisation tests: example with a t-test step 1: extract sample data real.diff=(data$dependent[group0]-data$dependent[group1])

25 the logic of randomisation tests: example with a t-test step 2: shuffle the data, extract the test statistic. randomvector=sample(n) mock.data=data$dependent[randomvector] mock.diff=(data$dependent[group0]-data$dependent[group0])

26 the logic of randomisation tests: example with a t-test step 2: shuffle the data, extract the test statistic. repeat all.diff=matrix(na,1000,1)! for (i in 1:1000){! randomvector=sample(n) mock.data=data$dependent[randomvector] mock.diff=(data$dependent[group0]-data$dependent[group0]) all.diff[i]=mock.diff }

27 the logic of randomisation tests: example with a t-test ask what is the probability of the observed test statistics under the generated distribution p=(length(which(all.diff > real.diff)) + length(which(all.diff < -real.diff)))/1000

28 the logic of randomisation tests: example with a t-test define the test statistic (e.g. mean) shuffle the data, extract test statistic repeat for all possible permutations (permutation test) or a sub-sample of them (randomization) ask what is the probability of the observed test statistics under the generated distribution possible to choose other statistics e.g. (t) or (f)

29 Permutation, Montecarlo, and bootstrap: what s the deal? Permutation & randomization tests: generating the probability of test statistics from the data, rather than a theoretical distribution Montecarlo: generating the probability of test statistics from the process, rather than a theoretical distribution Bootstrap, Jackknife: estimating bias and precision of estimates from the data, rather than a theoretical distribution

30 the logic of randomisation tests: example using Lloyd s index define the test statistic (e.g. Lloyd s index) model the process. for example, place organisms randomly on a grid, with parameters (density) matching your s calculate the test statistic (Lloyd s index) repeat multiple times ask what is the probability of the observed test statistics under the generated distribution

31 the logic of randomisation tests: example using Lloyd s index define the test statistic (e.g. Lloyd s index) סדור אקראי צבור

32 the logic of randomisation tests: example using Lloyd s index define the test statistic (e.g. Lloyd s index) place organisms randomly on a grid, with parameters (density) matching yours. calculate Lloyd s L= 1.06

33 the logic of randomisation tests: example using Lloyd s index define the test statistic (e.g. Lloyd s index) place organisms randomly on a grid, with parameters (density) matching your s L= 1.06 L= 1.12 L= 0.95 calculate Lloyd s index repeat multiple times L= 0.99 L= 1.02 L= 1.01

34 the logic of randomisation tests: example using Lloyd s index define the test statistic (e.g. Lloyd s index) model the process. for example, place organisms randomly on a grid, with parameters (density) matching your s calculate the test statistic (Lloyd s index) repeat multiple times ask what is the probability of the observed test statistics under the generated distribution

35 the logic of randomisation tests: example using Lloyd s index define the test statistic (e.g. Lloyd s index) model the process. for example, place organisms randomly on a grid, with parameters (density) matching your s calculate the test statistic (Lloyd s index) repeat multiple times ask what is the probability of the observed test statistics under the generated distribution one can make more complex models, i.e. place organisms that have an interaction between them

36 Permutation, Montecarlo, and bootstrap: what s the deal? Permutation & randomization tests: generating the probability of test statistics from the data, rather than a theoretical distribution Montecarlo: generating the probability of test statistics from the process, rather than a theoretical distribution Bootstrap, Jackknife: estimating bias and precision of parameters from the data, rather than a theoretical distribution

37 the logic of bootstrap compute the parameter of interest (e.g. number of species) from your n samples. the sample estimate is ŝ. sample (with replacement) n samples from your original dataset calculate the parameter of interest: ŝb repeat B times (For SE and bias estimation , For CI calculation 1000) Use the results to generate an empirical sampling distribution of ŝ.

38 the logic of bootstrap The bootstrap estimate of the parameter The bootstrap standard error (i.e. the standard deviation of the bootstrap estimate) The bootstrap estimate of the bias: The bias corrected estimate: Definition: Bias=S_hat S; where S is the true parameter. Hence, S=S_hat bias where the bias is estimated by (S_hat_bs - S_hat)