STATISTICS ASSIGNMENT 2

Transcription

1 STATISTICS ASSIGNMENT 2 Matteo Sostero June 10, 2010 Introduction The following document is a brief statistical report as part of the second assignment. It covers the issues raised on a dataset of 120 students of the degree course in Economics & Management. The full R code, including the commands producing tables and graphs, is given in the Sostero_Assignment_2.R script. 1 Remark First two Questions assume that observed units are the reference population, whereas the other two assume that observed units are a sample from some population whose properties are inferred from the sample. 1 Question What is the distribution of students according to province? The distribution of students according to province can be derived with a frequency table. Other Padova Treviso Venezia Unknown absolute frequency relative frequency Table 1: Frequency table of provinces in dataset A noteworthy feature of the dataset is that it includes three students for whom the province is unknown. 1.2 Suppose to draw at random (with or without replacement) a sample of students. Describe the probability distribution of the (random) number of students whose residence province is Venezia (VE). What is the corresponding expectation? Let us consider both sampling modalities: in the case of sampling with replacement, the probability distribution of observing k Venetian students in a sample of size is described by the binomial probability function. Since there are 42 Venetians in the population of 120, the probability of picking one of them when sampling with replacement is p = 42/120 = p(x = k) = ( k ) 0.35 k 0.65 k 1 This document was created with R version ( ), on a i386-pc-mingw32. 1

2 When sampling without replacement, however, the probability of picking k Venetian students in a sample of from a population of 120, of which 42 are Venetians and 78 are not is described by the hypergeometric distribution: ( ) ( ) k 120 k p(x = k) = ( ) 120 A graphical representation of these probability function is given by the following graph. Sampling with replacement Sampling without replacement Probability Function Probability Function k Venetians in sample k of Venetians in sample We will restrict our analysis to samples obtained with replacement. The corresponding expectation is: E(X) = µ = np = 0.35 = Consider a (real) sample of students. What is the observed number of students whose residence province is Venezia? What is the observed absolute error with respect to expectation? What is the probability of observing an higher error? We generate a sample of students called sam_ with a pseudorandom number generator: > set.seed( ) > rseq <- sort(sample(1:dim(ems)[1],, replace = T)) > sam_ <- ems[rseq, ] > length(sam_$prov[sam_$prov == "VE"]) [1] 11 The number of of Venetian students in the sample is 11, with an (absolute) error of 0.5 from the expectation. The random sample seems to depict quite accurately the population distribution of Venetians. In order to gauge whether this is due merely to sampling luck, we estimate the probability of obtaining a higher absolute error. Observing a (strictly) higher error, in this case, means computing the probability of observing [0, 9] [12, ] Venetians in the sample. > sum(dbinom(c(0:9, 12:),, 0.35)) [1] There is a probability of around 70.2% of observing a strictly larger error than the one of the abovementioned sample. Notice that, since the binomial distribution is discrete, the probability of observing bigger sampling errors with respect to the expectation is the same for any error ε < 1. It seems, therefore, that this sample is mercifully close to the expectation. 2

3 2 Question Suppose again to draw at random (with replacement) a sample of students. What is the joint distribution of the number of students of the four categories of residence province? What is the expectation vector? Let X ve, X pd, X tv, X ot, X uk be the total number of possible observed students from the provinces of Venice, Padua, Treviso, other provinces and unknown locations, respectively. Each has probability p ve, p pd, p tv, p ot, p uk of success. Let also x ve, x pd, x tv, x ot, x uk be the actual number of observed occurrences in the sample. The probability that X ve = x ve, X pd = x pd, X tv = x tv, X ot = x ot, X uk = x uk is described by the multinomial distribution: ( ) p(x ve = x ve, X pd x pd, X tv = x tv, X ot = x ot, X uk = x uk ) = p xve x ve x pd x tv x ot x ve p x pd pd pxtv p xot ot p x uk uk uk The expectation vector is E(X ve ) E(X tv ) E(X pd ) E(X ot ) = E(X uk ) n p ve n p tv n p pd n p ot n p uk = = = = = Consider a (real) sample of students and the corresponding classification of students according to the four categories. What is the probability of the observed result? We create another sample with replacement and call it sam 2. > set.seed(190793) > rseq_2 <- sort(sample(1:dim(ems)[1],, replace = T)) > sam 2 <- ems[rseq_2, ] The corresponding frequency table is: Other Padova Treviso Venezia Unknown absolute frequency relative frequency Table 2: Frequency table of provinces in sample The probability of observing this result, following the multinomial distribution, is derived from the joint frequency function: ( ) p(11, 6, 5, 11) The exact probability can be computed precisely with R by using the absolute frequencies in the sample (abs_f_s) as a vector of observed occurences and the corresponding relative frequencies in the population (rel_f) as vector of probabilities. 2 > dmultinom(x = abs_f_s, size =, prob = rel_f) [1] The probability of observing the above-mentioned result is about 0.23%. 2 Both vectors were created to build their corresponding frequency tables, cf. the script Assignment_2.R for details. 3

4 3 Question What is the distribution of E&M students according to gender? Once again, we use a frequency table: F M absolute frequency relative frequency Table 3: Frequency table of genders in dataset 3.2 Discuss the statistical hypothesis that in the reference population the frequency of male students is equal to 1/2. If we assume that the E&M dataset is a sample of a larger underlying population, we can describe the gender ratio with a Binomial distribution, where the modality male is counted as a success, with n = 120. We use a binomial test to check the null hypothesis H 0 : p s = 0.5, the alternative hypothesis is H 0 : p s 0.5. > binom.test(c(56, 64), p = 0.5, alternative = "two.sided", conf.level = 0.95) Exact binomial test data: c(56, 64) number of successes = 56, number of trials = 120, p-value = alternative hypothesis: true probability of success is not equal to percent confidence interval: sample estimates: probability of success The p-value of this test is relatively high, so we can be quite confident in keeping the null hypothesis. Furthermore, we notice that 0.5 is well within the confidence interval (0.37, 0.55), which is consistent with the result. 4 Question (Consider only second year students) Compare the performance of male and female students according to the number of (recorded) credits. Let s partition the sample according to gender. > ems2y <- ems[ems$year == "2", c(1, 5)] > Fcr <- ems2y$ncr[ems2y$gen == "F"] > Mcr <- ems2y$ncr[ems2y$gen == "M"] > length(fcr) [1] > length(mcr) [1] 33 4

5 A noteworthy feature of this sorting is that it excludes students for whom the number of credits has not been recorded. Only 63 of the 120 units provide data concerning credits. Although these are quite evenly split between genders, the small size of the sample can undermine the accuracy of our inferences. We compute relevant summary statistics for the two sub-samples and produce a boxplot describing the distribution of credits according to gender. Min. 1st Qu. Median Mean 3rd Qu. Max. Sd. Males Females Table 4: Summary statistics for 2 nd year students by gender > boxplot(ems2y$ncr ~ ems2y$gen, notch = T, horizontal = T) F M One thing that stands out from the comparison is that the medians, and their 95% confidence intervals coincide, while means and standard deviations don t. This sems to indicate that the two distributions are shaped differently. Looking at the actual distribution of units, with a stem-and-leaf plot unveils some other interesting features: > stem(fcr) The decimal point is 1 digit(s) to the right of the > stem(mcr) The decimal point is 1 digit(s) to the right of the 5

6 As far as the distribution of credits of female students is concerned, the data seem to be quite scattered. The pattern does not show a well-behaved unimodality and symmetry of the samples. 4.2 How do you evaluate (statistically)the hypothesis that females obtain better results? In order to check the hypothesis with a t-test, we need to make sure that both samples can be approximated by a normal distribution. > ks.test(mcr, "pnorm", mean(mcr), sd(mcr)) One-sample Kolmogorov-Smirnov test data: Mcr D = , p-value = alternative hypothesis: two-sided > ks.test(fcr, "pnorm", mean(fcr), sd(fcr)) One-sample Kolmogorov-Smirnov test data: Fcr D = , p-value = alternative hypothesis: two-sided The Kolmogorov-Smirnov test for both samples does not reject the hypothesis of normality and allows to use the t-test to check our hypothesis. Since we are only interested in assessing whether female students score more credits than males, we set the t-test accordingly. The null hypothesis is H 0 : µ F cr µ Mcr = 0, the alternative hypothesis is H 0 : µ F cr µ Mcr > 0. > t.test(fcr, Mcr, alternative = "greater", conf.level = 0.95) Welch Two Sample t-test data: Fcr and Mcr t = , df = , p-value = alternative hypothesis: true difference in means is greater than 0 95 percent confidence interval: Inf sample estimates: mean of x mean of y The p-value (6.5%) is relatively low and could allow to reject the null hypothesis. The data, therefore, seem to indicate that female students score more credits than male ones, with the caveat of the restricted sample size. 6