Statistics. Nicodème Paul Faculté de médecine, Université de Strasbourg. 9/5/2018 Statistics

Transcription

1 Statistics Nicodème Paul Faculté de médecine, Université de Strasbourg file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 1/62

2 Course logistics Statistics Course website: ( Lecture slides and lecture notes Lectures, quizzes and practical exercises Exam file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 2/62 2/62

3 Course structure Descriptive statistics and probability Estimation Hypothesis testing (one sample, two-sample tests) Independence test (Chi-square, Fisher's exact) One-way ANOVA file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 3/62 3/62

4 Statistics - De nition A statistic is a quantity or numerical value calculated from a set of data. - Average height of people living in Strasbourg Statistics refer to global caracteristics of population Number of people who smoke Number of people owning a car Relation between smoking and owing a car Statistics is the scienti c discipline that provides methods to make sense of data. - - Descriptive statistics : collecting, summarizing and presenting data Inferential statistics : making inferences, hypothesis testing, determining relationships and making predictions file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 4/62 4/62

5 Statistics applications Biology - Comparison betwen two population of mice: knockout versus wildtype Medecine - Perform clinical trials and data analysis Pharmacy - Knowing whether a new drug is better than the current one Finance - Pricing and portfolio management, risk modelling Agriculture - Plant breeding, the study of the in uence of particular factors on agricultural production, measuring of contribution of production factors, fertilizers and technical progress. file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 5/62 5/62

6 Terminology A population is collection of individuals or objects about which information is desired. A sample is a subset of the population selected for study. A random sample of size n is a sample that is selected in such a way that ensures that every di erent possible sample of the desired size has the same chance of being selected. A variable is any characteristic whose value may change from one individual or object to another. A variable can be categorical: - - Nominal (color : red, black, green, white) Ordinal (size : small, medium, big) A variable can be numerical: - - Discrete (number of s received per day) Continuous (height, weight) file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 6/62 6/62

7 Data Example. Consider di erent activity levels in sports such as weak (W), moderate (M) and intense (I). A sample of ten individuals results in di erent possible data sets. Univariate data Activity : W M M W I W I M W W Age : Bivariate data Activity, Age : (W, 35) (M, 33) (M, 50) (W, 21) (I, 40) (W, 39) (I, 51) (M, 47) (W, 36) (W, 30) file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 7/62 7/62

8 Multivariate Data Variables as columns and individuals as rows Multivariate sample data set ID SBP TOBACCO LDL ADIPOSITY FAMHIST OBESITY ALCOHOL AGE CHD Present Absent Present Present Present Present Absent Present Present Present file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 8/62 8/62

9 Categorical data representation Sample data for activity levels M M M I I M I M I I W M M I M I I M I M W M M M M M M M M W M I I M M M M M M I M I W M M M I M M M M I W M I I M W M M W M M M M M M W I M M M M M M M I M I W M I M M M M M M M M file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 9/62 9/62

10 Categorical data representation - barplot Frequency distribution ACTIVITY FREQUENCY RELATIVE FREQUENCY Weak Moderate Intense file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 10/62 10/62

11 Group comparison for categorical data file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 11/62 11/62

12 Continuous data representation - histogram B j ( x 0, h) = [ x 0 + (j 1)h, x 0 + jh[, j Z 1, I{ x i B j ( x 0, h)} = { 0, if (, h) x i B j x 0 otherwise n i=1 1 hist(x) = I{ x i B j ( x 0, h)}i{x B j ( x 0, h)} hn j Z file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 12/62 12/62

13 Histogram file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 13/62 13/62

14 Histogram The histogram is a method of displaying data. It displays the shape of the distribution of data values. The range of the data is divided into intervals proportion of the observations falling in each bin c i ], ] a i is plotted. b i or bins, and the number or A histogram is said to be unimodal if it has a single peak, bimodal if it has two peaks and multimodal if it has more than two peaks. A histogram is symmetric if there is a vertical line of symmetry such that the part of the histogram to the left of the line is a mirror image of the part to the right. A unimodal histogram that is not symmetric is said to be skewed. - - If the upper tail of the histogram stretches out much farther than the lower tail, then the distribution of values is positively skewed or right skewed. If the lower tail is much longer than the upper tail, the histogram is negatively skewed or left skewed. file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 14/62 14/62

15 Histogram file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 15/62 15/62

16 Comparison between groups for variable SBP file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 16/62 16/62

17 Comparison between groups for variable SBP file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 17/62 17/62

18 Comparison between groups for variable Age file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 18/62 18/62

19 Comparison between groups for variable Age file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 19/62 19/62

20 Measures of location x 1 x 2 x n by x, is: The sample mean of a sample consisting of numerical observations,,...,, denoted xˉ = 1 n n i=1 x i The population mean, denoted by μ, is the average of all x values in the entire population. The sample median or Q 2 is obtained by rst ordering the n observations from smallest to largest as. Then: x (1) x (2)... x (n) sample median = x (n+1)/2, if n is odd 1 ( + ), if n is even 2 x n x n 2 ( 2 +1) file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 20/62 20/62

21 Measures of location Data = 75, 69, 88, 93, 95, 54, 87, 88, 27 Ordered data = 27, 54, 69, 75, 87, 88, 88, 93, 95 Sample median = 87 If Data = 100, 75, 69, 88, 93, 95, 54, 87, 88, What is the median? Submit Show Hint Show Answer Clear file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 21/62 21/62

22 Measures of location For any particular number r between 0 and 100, the rth percentile is a value such that r percent of the observations in the data set fall at or below that value. The lower quartile or 25th percentile or Q 1 The upper quartile or 75th percentile or Q 3 is the median of the lower half of the sample. is the median of the upper half of the sample. The mode is the most observed value file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 22/62 22/62

23 Robust statistics file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 23/62 23/62

24 Check yourself The value can be used as a measure of skewness (either right or left). If this statistic is less than 1, the distribution is most likely left skewed. True False mean median Submit Show Hint Show Answer Clear file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 24/62 24/62

25 Measures of dispersion The sample variance, denoted by s 2, is the sum of squared deviations from the mean divided by n 1. That is, s 2 1 = ( x i xˉ) 2 n 1 i=1 n The sample standard deviation is the positive square root of the sample variance and is denoted by s. σ 2 n The variance, denoted by, is the sum of squared deviations from the mean divided by. That is, n σ 2 1 = ( x i μ) 2 n i=1 file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 25/62 25/62

26 Check yourself Which of the below data sets has the lowest standard deviation? You do not need to calculate the exact standard deviations to answer this question. 0,1,2,3,4,5,6 0,1,3,3,3,5,6 100, 100, 100, 100, 100, 100, 101 0, 25, 50, 100, 125, 150, 1000 Submit Show Hint Show Answer Clear file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 26/62 26/62

27 Measures of dispersion The standard deviation of the population is the positive square root of the variance and is σ denoted by. The interquartile range (IRQ), is a measure of variability de ned as: IRQ = upper quartile lower quartile An observation is an outlier if it is more than 1.5(IRQ) away from the nearest quartile. An outlier is extreme if it is more than 3(IRQ) from the nearest quartile and it is mild otherwise. The coe cient of variation (CV)is a normalized measure of variability de ned as: s CV = 100 xˉ file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 27/62 27/62

28 Boxplot: description file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 28/62 28/62

29 Example: boxplot comparison file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 29/62 29/62

30 Check yourself Which of the following statements is supported by the plot? The mean of the distribution is smaller than its median It is not possible to estimate the median without knowing the sample size The distribution is multimodal The IQR of the distribution is roughly 10 Submit Show Hint Show Answer Clear file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 30/62 30/62

31 Check yourself Which of the following statements is not supported by the plot? Both distributions are unimodal B is more variable than A Median of A is higher than median of B Both distributions are roughly symmetric Submit Show Hint Show Answer Clear file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 31/62 31/62

32 Check yourself Suppose we have a drug that we know, from long experience, cures a patient with some speci c illness in 70% of cases. A new drug is proposed as having a higher cure rate than the present one. To assess this claim, the new drug is given to 1000 people su ering from the illness, among these, 741 are cured. Do we have signi cant evidence that this new drug is better than the current one? Yes No Need more information to decide Submit Show Hint Show Answer Clear file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 32/62 32/62

33 Probability - motivation Suppose we have a drug that we know, from long experience, cures a patient with some speci c illness in 70% of cases. A new drug is proposed as having a higher cure rate than the present one. To assess this claim, the new drug is given to 1000 people su ering from the illness, among these, 741 are cured. Do we have signi cant evidence that this new drug is better than the current one? Consider the following hypotheses: - H 0 : the new drug is equally e ective than the the current one (hypothesis of no e ect or no di erence or not better) - H 1 : the new drug is better than the current one Probability calculation - If the new drug is equally e ective as the current one, how likely is it that, by chance, 741 or more people given the new drug will be cured? Statisical inference - Based on the above probability calculation, the data may provide convincing evidence that the new drug is better than the current one. file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 33/62 33/62

34 Check yourself Suppose that the probabiltiy to observe 741 or more cured patients under the assumption that the new medicine in no better that the old is Do the data provide convincing evidence that the new drug is better than the current one? Yes No Need more information Submit Show Hint Show Answer Clear file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 34/62 34/62

35 Probability - terminology A random experiment is any activity or situation in which there is uncertainty about which of two or more possible outcomes will result. A bernoulli trial is a random experiment with exactly two possible outcomes: success or failure. - Tossing a coin with Head or H and Tail or T as possible outcome - A patient can be cured by the new medicine or not The collection of all possible outcomes of a random experiment is the sample space Ω the experiment. An outcome from the sample is denoted as. ω for Examples of sample space: Ω = {H, T}, Ω = {HH, HT, T H, T T} An event E is any collection of outcomes from the sample space of a chance experiment. A simple event is an event consisting of exactly one outcome. Tossing a coin twice and obtain at least one head : E = {HH, HT, T H} file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 35/62 35/62

36 Probability - axioms A function that assigns a real number to each event is a probability distribution or a P P(A) A probability measure if it satis es the following three axioms: P(A) 0 1. for every A 2. P(Ω) = 1 A 1, A 2,... A i A j = i j 3. If are disjoint, meanning for, i P( ) = P( ) A i i=1 A i file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 36/62 36/62

37 Random variables - De nition A random variable X is a real-valued function de ned on a sample space. In other terms, a random variable associates a numerical value to each outcome of a random experiment. A random variable X is discrete if its set of possible values is discrete. Otherwise, it is continous. X(H) = 1 X(T) = 0 Tossing a coin:,. We noted X = {0, 1} Drug trial: number of patient cured by the new medecine in a sample of a 1000 patients. X = {0, 1, 2,..., 1000} file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 37/62 37/62

38 Discrete probability - distribution The probability distribution of a discrete random variable X taking values in {,,..., } can be represented by a table: Probability distribution of X X x 1 x 2... x n P p 1 p 2... p n 0 p i 1 n i=1 p i = 1 x 1 x 2 x n Drug trial with Cured = 1 and Not cured = 0 Probability distribution example X P file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 38/62 38/62

39 Couple of discrete random variables Given two discrete random variables X and Y, we can de ne a new random variable (X, Y) whose joint distribution is de ned by: n i=1 m j=1 p ij 0 p ij 1 = 1 p ij = P(X = x i, Y = y j ) with,. The distribution can be represented as a table: Y X x 1 x 2 y 1 p 11 p 21 y 2 p 12 p y m p 1m p 2m x n p n1 p n2... p nm x i p i. m j=1 p ij The marginal distribution of X : P(X = ) = = y j p.j n i=1 p ij The marginal distribution of Y : P(Y = ) = = file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 39/62 39/62

40 Example - Diagnosing Tuberculosis (TB) Before 1998, culturing was the existing gold standard for diagnosing TB This method took 10 to 15 days to yield a positive or negative result. In 1998, investigators evaluated a DNA technique that turned out to be much faster ("LCx: A Diagnostic Alternative for the Early Detection of Mycobacterium tuberculosis Complex," Diagnostic Microbiology and Infectious Diseases [1998]: ). T models the outcome of the gold standard method: 1 indicates TB, 0 not TB N models the outcome of the DNA test: 1 indicates positive test, 0 negative test The data is summarized in the following table: T N file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 40/62 40/62

41 Example - Joint distribution calculation T N P(N = 0, T = 0) =, P(N = 0, T = 1) = P(N = 1, T = 0) =, P(N = 1, T = 1) = T N file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 41/62 41/62

42 Check yourself Calculate P(T = 1). Choose the right answer Not de ned Submit Show Hint Show Answer Clear file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 42/62 42/62

43 Check yourself Calculate P(N = 0). Choose the right answer Not de ned Submit Show Hint Show Answer Clear file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 43/62 43/62

44 Parameters of a random variable X The expectation of a discrete random variable X taking the values probabily values p 1, p 2,..., p n is the number:,,..., x 1 x 2 x n, with n μ = E[X] = i=1 x i p i We call variance of X, the number if it exists: σ 2 = V(X) = E[(X E[X] ) 2 ] = E[ X 2 ] E[X ] 2 n = p i ( x i μ) 2 i=1 σ X is called the standard deviation of. X Y E[X] E[Y ] a b If and are two random variables with expected values and, and two real numbers, we have the following: E[X + Y ] = E[X] + E[Y ], E[aX + b] = ae[x] + b file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 44/62 44/62

45 Discrete distribution - Bernoulli p L(X) = B(p) A random variable X follows a Bernoulli distribution with parameter noted, if it takes only two values commonly noted 0 and 1 with probabilities: P(X = 1) = p P(X = 0) = 1 p - Example: drug trial where a patient is cured with a probability 0.7 The expected value of X is p as: The variance of X is p(1 p) as: E[X] = 1 p + 0 (1 p) = p V(X) = E[(X E[X] ) 2 ] = E[ X 2 ] E[X ] 2 = p p 2 = p(1 p) file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 45/62 45/62

46 Discrete distribution - Binomial distribution X 1, X 2,..., X n B(p) the random variable Y = X 1 + X X n distribution noted B(n; p) with parameters n, and p. Its distribution is de ned by: Given n independent random variables having the same distribution, with n taken the values 0, 1,..., n follows a binomial n P(Y = k) = ( ) (1 p k = 0, 1,..., n k pk ) n k ( ) = and x! = x (x 1) (x 2) k n! k!(n k)! As sum of independent Bernoulli random variables we have: E[Y ] = np V(Y ) = np(1 p) file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 46/62 46/62

47 Binomial distribution - Example Sickle cell anemia is a genetic blood disorder where red blood cells lose their exibility and assume an abnormal, rigid, "sickle" shape, which results in a risk of various complications. If both parents are carriers of the disease, then a child has a 25% chance of having the disease, 50% chance of being a carrier, and 25% chance of neither having the disease nor being a carrier. If two parents who are carriers of the disease have 3 children, what is the probability that: (a) two will have the disease? (b) none will have the disease? (c) at least one will neither have the disease nor be a carrier? file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 47/62 47/62

48 Binomial distribution - Example X Let be a random variable that represents the number of children with the disease and the number of children that have neither the disease nor be a carrier. We have: L(X) = B(3; 0.25) and L(Y ) = B(3; 0.25) Y Answers to the questions: - - (a) (b) 3 P(X = 2) = ( ) (1 0.25) = = P(X = 0) = ( ) ( = ( = ) 3 ) 3 - (c) P(Y = 1) + P(Y = 2) + P(Y = 3) = 1 P(Y = 0) = 0.58 file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 48/62 48/62

49 Normal distribution A random variable is said to follow a normal distribution de parameters and σ 2 > 0 if: X N (μ; ) σ 2 μ R 1 1 f X (t) = exp( (t μ ) 2 ), t R E(X) = μ and V ar(x) = σ 2π 2σ 2 σ 2 file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 49/62 49/62

50 Normal distribution rule file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 50/62 50/62

51 Check yourself A doctor collects a large set of heart rate measurements that approximately follow a normal distribution. He only reports 3 statistics, the mean = 110 beats per minute, the minimum = 65 beats per minute, and the maximum = 155 beats per minute. Which of the following is most likely to be the standard deviation of the distribution? Submit Show Hint Show Answer Clear file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 51/62 51/62

52 Calculate with the normal distribution L(X) = N (μ; ) N (0; 1) σ 2 Z = X μ If, then random variable has the standard normal distribution σ L(X) = N (μ; σ 2 ) [a, b[ If and given an interval: a μ X μ b μ P(a X < b) = P( < ) σ σ σ a μ b μ P(a X < b) = P( Z < ) σ σ b μ a μ P(a X < b) = P(Z < ) P(Z ) σ σ b μ a μ P(a X < b) = Φ( ) Φ( ) σ σ Φ is the cummulative distribution of the standard normal such that Φ(z) = P(Z z). file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 52/62 52/62

53 Standard normal distribution table P(Z 0.14) = P(Z 0.58) = P(0.14 Z 0.58) = = file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 53/62 53/62

54 Calculations P(Z > 0.23) = 1 P(Z 0.23) = = file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 54/62 54/62

55 Calculations P(Z 0.53) = P(Z 0.53) = 1 P(Z 0.53) = = file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 55/62 55/62

56 Calculations with L(X) = N (25; 16) P(X 26.4) = P((X 25)/4 ( )/4) = P(Z 0.35) = file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 56/62 56/62

57 Calculations If L(X) = N (100; 25), calculate P(90 X 105) X P( ) = P( 2 Z 1) P( 2 Z 1) = P(Z 1) P(Z < 2) P(Z 1) P(Z < 2) = P(Z 1) (1 P(Z 2)) P( 2 Z 1) = P(Z 1) + P(Z 2) 1 P( 2 Z 1) = P( 2 Z 1) = P(90 X 105) = file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 57/62 57/62

58 Properties X 1 X 2 N ( μ 1 ; σ1 2 ) α β If two random variables et are independant with distribution and N ( μ 2 ; σ2 2 ) respectively and, real numbers, then: L( + ) = N ( +, + ) X 1 X 2 μ 1 μ 2 σ 2 1 σ 2 2 L( ) = N (, + ) X 1 X 2 μ 1 μ 2 σ 2 1 σ 2 2 L(αX 1 β X 2 ) = N (αμ 1 β μ 2, α 2 σ1 2 + β 2 σ2 2 ) L( X 1 ) = N (15; 16) L( X 2 ) = N (10; 9) Y = If and, let X 1 X 2, we have: P( 3) = P(Y 3) X 1 X 2 Y 5 2 P( X 1 X2 3) = P( ) 5 5 P( 3) = P(Z 0.4) = P(Z > 0.4) X 1 X 2 P( 3) = 1 P(Z 0.4) = X 1 X 2 file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 58/62 58/62

59 Check yourself X 1 X 2 X 3 N (0, 1) Y = , and are independent and normally distributed with the same normal distribution. X 1 X 2 X 3 What is the distribution of Y? Binomial Normal Multivariate normal We cannot add random variables Submit Show Hint Show Answer Clear file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 59/62 59/62

60 Check yourself Y = X 1 X 2 X 3 What is the expected value of Y? Submit Show Hint Show Answer Clear file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 60/62 60/62

61 Check yourself Y = X 1 X 2 X 3 What is variance of Y? Submit Show Hint Show Answer Clear file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 61/62 61/62

62 See you next time file:///users/home/npaul/enseignement/esbs/ /cours/01/index.html#21 62/62 62/62