Statistics. Nicodème Paul Faculté de médecine, Université de Strasbourg

Size: px

Start display at page:

Download "Statistics. Nicodème Paul Faculté de médecine, Université de Strasbourg"

Claire Juliana Merritt
5 years ago
Views:

1 Statistics Nicodème Paul Faculté de médecine, Université de Strasbourg

2 Course logistics Statistics & Experimental plani cation Course website: ( Lecture slides and lecture notes Lectures, quizzes and practical exercises R statistical software Exam 2/59

3 Statistics - De nition A statistic is a quantity or numerical value calculated from a set of data. - Average height of people living in Strasbourg Statistics refer to global caracteristics of population Number of people who smoke Number of people owning a car Relation between smoking and owing a car Statistics is the scienti c discipline that provides methods to make sense of data. - - Descriptive statistics : collecting, summarizing and presenting data Inferential statistics : making inferences, hypothesis testing, determining relationships and making predictions 3/59

4 Statistics applications Biology - Comparison betwen two population of mice: knockout versus wildtype Medecine - Perform clinical trials and data analysis Pharmacy - Knowing whether a new drug is better than the current one Finance - Pricing and portfolio management, risk modelling Agriculture - Plant breeding, the study of the in uence of particular factors on agricultural production, measuring of contribution of production factors, fertilizers and technical progress. 4/59

5 Terminology A population is collection of individuals or objects about which information is desired. A sample is a subset of the population selected for study. A random sample of size n is a sample that is selected in such a way that ensures that every di erent possible sample of the desired size has the same chance of being selected. A variable is any characteristic whose value may change from one individual or object to another. A variable can be categorical: - - Nominal (color : red, black, green, white) Ordinal (size : small, medium, big) A variable can be numerical: - - Discrete (number of s received per day) Continuous (height, weight) 5/59

6 Data Variables as columns and individuals as rows Show 10 entries Search: ID AGE SEX CHESTPAIN RESTBP CHOL MAXHR HD Typical No Asymptomatic Yes Asymptomatic Yes Nonanginal No Nontypical No Nontypical No Asymptomatic Yes Asymptomatic No Asymptomatic Yes Asymptomatic Yes Showing 1 to 10 of 303 entries Previous Next 6/59

7 Graphical representation - barplot Asymptomatic Nonanginal Nontypical Typical 7/59

8 Graphical representation - histogram of Age FREQ (20,30] (30,40] (40,50] (50,60] (60,70] (70,80] BIN 8/59

9 Terminology Distribution of a variable over a sample is given by: ( c 1, n 1 ), ( c 2, n 2 ),..., ( c k, n k ) - n i represents the frequency associated with c i - For categorical variable c i is any value of the variable - For numerical variable c i is an interval ], ] a i b i 9/59

10 Categorical data distribution - Terminology The frequency for a particular category is the number of times the category appears in the data set. The relative frequency is the proportion of the observations that belong to that category, it is caculated as: p = c N Where c is the frequency and N is the number of observations in the data The distribution for a categorical variable is a table that displays the possible categories along with the associated frequencies or relative frequencies. A barchart or barplot is a graphical representation of the distribution of a categorical variable. 10/59

11 Histogram The histogram is a method of displaying data. It displays the shape of the distribution of data values. The range of the data is divided into intervals proportion of the observations falling in each bin c i a i ], ] is plotted. b i or bins, and the number or A histogram is said to be unimodal if it has a single peak, bimodal if it has two peaks and multimodal if it has more than two peaks. A histogram is symmetric if there is a vertical line of symmetry such that the part of the histogram to the left of the line is a mirror image of the part to the right. A unimodal histogram that is not symmetric is said to be skewed. - - If the upper tail of the histogram stretches out much farther than the lower tail, then the distribution of values is positively skewed or right skewed. If the lower tail is much longer than the upper tail, the histogram is negatively skewed or left skewed. 11/59

12 Example 12/59

13 Example 13/59

14 Barplot: group comparison Grouped Stacked No Yes Asymptomatic Nonanginal Nontypical Typical 14/59

15 Histogram: group comparison 77.0 Grouped Stacked No Yes CHOL 15/59

16 Measures of location The sample mean of a sample consisting of numerical observations x 1, x 2,..., x n, denoted by x, is: xˉ = 1 n x i n i=1 The population mean, denoted by μ, is the average of all x values in the entire population. The sample median or Q 2 is obtained by rst ordering the n observations from smallest to largest as x (1) x (2)... x (n). Then: sample median = x (n+1)/2, if n is odd 1 ( + ), if n is even 2 x n x n ( +1) /59

17 Measures of location Data = 75, 69, 88, 93, 95, 54, 87, 88, 27 Ordered data = 27, 54, 69, 75, 87, 88, 88, 93, 95 Sample median = 87 If Data = 100, 75, 69, 88, 93, 95, 54, 87, 88, What is the median? Submit Show Hint Show Answer Clear 17/59

18 Measures of location For any particular number r between 0 and 100, the rth percentile is a value such that r percent of the observations in the data set fall at or below that value. The lower quartile or 25th percentile or Q 1 The upper quartile or 75th percentile or Q 3 is the median of the lower half of the sample. is the median of the upper half of the sample. The mode is the most observed value 18/59

19 Example: Histogram FREQ (0,5] (5,10] (10,15] (15,20] (20,25] (25,30] (30,35] (35,40] (40,45] BIN What is the value of Q 1 given the samples size is 38? 10 5 Q 1 = 5 + ( ) 38 = /59

20 Empirical cummulative distribution The empirical cummulative distribution associated with a sample is the function any real number t, by the expression: card{ : t} F n (t) = n Son graphe est appelé le graphique des fréquences cumulées. x i x i F n de ned for 20/59

21 Empirical cumulative distribution 0.8 FREQ FAILURE 21/59

22 Robust statistics 22/59

23 Check yourself The statistic can be used as a measure of skewness (either right or left). If this statistic is less than 1, the distribution is most likely left skewed. True False mean median Submit Show Hint Show Answer Clear 23/59

24 Measures of dispersion The sample variance, denoted by s 2, is the sum of squared deviations from the mean divided by n 1. That is, s 2 1 = ( x i xˉ) 2 n 1 i=1 n The sample standard deviation is the positive square root of the sample variance and is denoted by s. The variance, denoted by σ 2, is the sum of squared deviations from the mean divided by n. That is, n σ 2 1 = ( x i μ) 2 n i=1 24/59

25 Check yourself Which of the below data sets has the lowest standard deviation? You do not need to calculate the exact standard deviations to answer this question. 0,1,2,3,4,5,6 0,1,3,3,3,5,6 100, 100, 100, 100, 100, 100, 101 0, 25, 50, 100, 125, 150, 1000 Submit Show Hint Show Answer Clear 25/59

26 Measures of dispersion The standard deviation of the population is the positive square root of the variance and is denoted by σ. The interquartile range (IRQ), is a measure of variability de ned as: IRQ = upper quartile lower quartile An observation is an outlier if it is more than 1.5(IRQ) away from the nearest quartile. An outlier is extreme if it is more than 3(IRQ) from the nearest quartile and it is mild otherwise. The coe cient of variation (CV)is a normalized measure of variability de ned as: s CV = 100 xˉ 26/59

27 Boxplot: description 27/59

28 Example:water quality comparison 28/59

29 Check yourself Which of the following statements is supported by the plot? The mean of the distribution is smaller than its median It is not possible to estimate the median without knowing the sample size The distribution is multimodal The IQR of the distribution is roughly 10 Submit Show Hint Show Answer Clear 29/59

30 Check yourself Which of the following statements is not supported by the plot? Both distributions are unimodal B is more variable than A Median of A is higher than median of B Both distributions are roughly symmetric Submit Show Hint Show Answer Clear 30/59

31 Probability - motivation Suppose we have a drug that we know, from long experience, cures a patient with some speci c illness in 70% of cases. A new drug is proposed as having a higher cure rate than the present one. To assess this claim, the new drug is given to 1000 people su ering from the illness, among these, 741 are cured. Do we have signi cant evidence that this new drug is better than the current one? H 0 : the new drug is equally e ective than the the current one H 1 : the new drug is better than the current one Probability calculation - If the new drug is equally e ective as the current one, how likely is it that, by chance, 741 or more people given the new drug will be cured? Statisical inference - Based on the above probability calculation, the data may provide convincing evidence that the new drug is better than the current one. 31/59

32 Check yourself Suppose that the probabiltiy to observe 741 or more cured patients under the assumption that the new medicine in no better that the old is Do the data provide convincing evidence that the new drug is better than the current one? Yes No Submit Show Hint Show Answer Clear 32/59

33 Probability - terminology A random experiment is any activity or situation in which there is uncertainty about which of two or more possible outcomes will result. A bernoulli trial is a random experiment with exactly two possible outcomes: success or failure. - - Tossing a coin with Head or H and Tail or T as possible outcome A patient can be cured by the new medicine or not The collection of all possible outcomes of a random experiment is the sample space Ω the experiment. An outcome from the sample is denoted as ω. for Examples of sample space: Ω = {H, T }, Ω = {HH, HT, T H, T T } An event E is any collection of outcomes from the sample space of a chance experiment. A simple event is an event consisting of exactly one outcome. Tossing a coin twice and obtain at least one head : E = {HH, HT, T H} 33/59

34 Random variables - De nition A random variable X is a real-valued function de ned on a sample space. In other terms, a random variable associates a numerical value to each outcome of a random experiment. A random variable X is discrete if its set of possible values is discrete. Otherwise, it is continous. Tossing a coin: X = {0, 1} Drug trial: number of patient cured by the new medecine in a sample of a 1000 patients. X = {0, 1, 2,..., 1000} 34/59

35 Discrete probability - distribution The probability distribution of a discrete random variable X taking values in { x 1, x 2,..., x n } can be represented by a table: Probability distribution of X X P x 1 p 1 x 2 p x n p n 0 p i 1 n i=1 p i = 1 Drug trial with Cured = 1 and Not cured = 0 Probability distribution example X P /59

36 Couple of discrete random variables Given two discrete random variables X and Y, we can de ne a new random variable (X, Y) whose joint distribution is de ned by: n i=1 m j=1 p ij = P (X =, Y = ) p ij x i y j with 0 p ij 1, = 1. The distribution can be represented as a table: Y X x 1 x 2 y 1 p 11 p 21 y 2 p 12 p y m p 1m p 2m x n p n1 p n2... p nm The marginal distribution of X : P(X = x i ) = p = m i. j=1 p ij The marginal distribution of Y : P(Y = y j ) = p.j = n i=1 p ij 36/59

37 Example - Diagnosing Tuberculosis (TB) Before 1998, culturing was the existing gold standard for diagnosing TB This method took 10 to 15 days to yield a positive or negative result. In 1998, investigators evaluated a DNA technique that turned out to be much faster ("LCx: A Diagnostic Alternative for the Early Detection of Mycobacterium tuberculosis Complex," Diagnostic Microbiology and Infectious Diseases [1998]: ). T models the outcome of the gold standard method: 1 indicates TB, 0 not TB N models the outcome of the DNA test: 1 indicates positive test, 0 negative test The data is summarized in the following table: T N /59

38 Example - Joint distribution calculation T N P(N = 0, T = 0) =, P(N = 0, T = 1) = P(N = 1, T = 0) =, P(N = 1, T = 1) = T N /59

39 Check yourself Calculate P(T = 1). Choose the right answer Not de ned Submit Show Hint Show Answer Clear 39/59

40 Check yourself Calculate P(N = 0). Choose the right answer Not de ned Submit Show Hint Show Answer Clear 40/59

41 Parameters of a random variable X The expectation of a discrete random variable X taking the values probabily values p 1, p 2,..., p n is the number: = n μ = E[X] i=1 x i p i x 1, x 2,..., x n, with We call variance of X, the number if it exists: n σ 2 = V ar(x) = E[(X E[X] ) 2 ] = E[ X 2 ] E[X ] 2 = p i ( x i μ) 2 i=1 σ is called the standard deviation of X. If X and Y are two random variables with expected values E[X] and E[Y], a and b two real numbers, we have the following: E[X + Y ] = E[X] + E[Y ], E[aX + b] = ae[x] + b 41/59

42 Discrete distribution - Bernoulli A random variable X follows a Bernoulli distribution with parameter p noted L(X) = B(p), if it takes only two values commonly noted 0 and 1 with probabilities: P(X = 1) = p P(X = 0) = 1 p - Example: drug trial where a patient is cured with a probability 0.7 The expected value of X is p as: The variance of X is p(1 p) as: E[X] = 1 p + 0 (1 p) = p V ar(x) = E[(X E[X] ) 2 ] = E[ X 2 ] E[X ] 2 = p p 2 = p(1 p) 42/59

43 Discrete distribution - Binomial distribution Given X 1, X 2,..., X n n independent random variables having the same distribution B(p), the random variable Y = X 1 + X X n taken the values 0, 1,..., n follows a binomial distribution noted B(n; p) with parameters n, and p. Its distribution is de ned by: with n n P(Y = k) = ( ) (1 p k = 0, 1,..., n k pk ) n k ( ) = and x! = x (x 1) (x 2) k n! k!(n k)! As sum of independent Bernoulli random variables we have: E[Y ] = np V ar(y ) = np(1 p) 43/59

44 Binomial distribution - Example Sickle cell anemia is a genetic blood disorder where red blood cells lose their exibility and assume an abnormal, rigid, "sickle" shape, which results in a risk of various complications. If both parents are carriers of the disease, then a child has a 25% chance of having the disease, 50% chance of being a carrier, and 25% chance of neither having the disease nor being a carrier. If two parents who are carriers of the disease have 3 children, what is the probability that: (a) two will have the disease? (b) none will have the disease? (c) at least one will neither have the disease nor be a carrier? 44/59

45 Binomial distribution - Example Let X be a random variable that represents the number of children with the disease and Y the number of children that have neither the disease nor be a carrier. We have: L(X) = B(3; 0.25) and L(Y ) = B(3; 0.25) Answers to the questions: - - (a) (b) 3 P(X = 2) = ( ) (1 0.25) = = P(X = 0) = ( ) ( = ( = ) 3 ) 3 - (c) P(Y = 1) + P(Y = 2) + P(Y = 3) = 1 P(Y = 0) = /59

46 Normal distribution A random variable X is said to follow a normal distribution N (μ; ) de parameters and σ 2 > 0 if: σ 2 μ R 1 1 f X (t) = exp( (t μ ) 2 ), t R E(X) = μ and V ar(x) = σ 2 σ 2π 2σ 2 46/59

47 Normal distribution rule 47/59

48 Check yourself A doctor collects a large set of heart rate measurements that approximately follow a normal distribution. He only reports 3 statistics, the mean = 110 beats per minute, the minimum = 65 beats per minute, and the maximum = 155 beats per minute. Which of the following is most likely to be the standard deviation of the distribution? Submit Show Hint Show Answer Clear 48/59

49 Calculate with the normal distribution If L(X) = N (μ; σ 2 ), then random variable Z = X μ has the standard normal distribution N (0; 1) σ If L(X) = N (μ; σ 2 ) and given [a, b[ an interval: a μ X μ b μ P(a X < b) = P( < ) σ σ σ a μ b μ P(a X < b) = P( Z < ) σ σ b μ a μ P(a X < b) = P(Z < ) P(Z ) σ σ b μ a μ P(a X < b) = F Z ( ) F Z ( ) σ σ F Z is the cummulative distribution of the standard normal. 49/59

50 Standard normal distribution table P(Z 0.14) = P(Z 0.58) = P(0.14 Z 0.58) = = /59

51 Calculations P(Z > 0.23) = 1 P(Z 0.23) = = /59

52 Calculations P(Z 0.53) = P(Z 0.53) = 1 P(Z 0.53) = = /59

53 Calculations with L(X) = N (25; 16) P(X 26.4) = P((X 25)/4 ( )/4) = P(Z 0.35) = /59

54 Calculations If L(X) = N (100; 25), calculate P(90 X 105) X P( ) = P( 2 Z 1) P( 2 Z 1) = P(Z 1) P(Z < 2) P(Z 1) P(Z < 2) = P(Z 1) (1 P(Z 2)) P( 2 Z 1) = P(Z 1) + P(Z 2) 1 P( 2 Z 1) = P( 2 Z 1) = P(90 X 105) = /59

55 Properties If two random variables X 1 et X 2 are independant with distribution N ( μ 1 ; σ 2 1 ) and N ( μ 2 ; σ 2 2 ) respectively and α, β real numbers, then: L( X 1 + X 2 ) = N ( μ 1 + μ 2, σ σ 2 2 ) L( X 1 X 2 ) = N ( μ 1 μ 2, σ σ 2 2 ) L(αX 1 β X 2 ) = N (αμ 1 β μ 2, α 2 σ β 2 σ 2 2 ) If L( X 1 ) = N (15; 16) and L( X 2 ) = N (10; 9), let Y = X 1 X 2, we have: P( X 1 X 2 3) = P(Y 3) Y 5 2 P( X 1 X 2 3) = P( ) 5 5 P( X 1 X 2 3) = P(Z 0.4) = P(Z > 0.4) P( X 1 X 2 3) = 1 P(Z 0.4) = /59

56 Check yourself X 1, X 2 and X 3 are independent and normally distributed with the same normal distribution N (0, 1). Y = 2X 1 2 X 2 + X 3 2 What is the distribution of Y? Poisson Uniform Normal Not de ned Submit Show Hint Show Answer Clear 56/59

57 Check yourself Y = 2X 1 2 X 2 + X 3 2 What is the expected value of Y? Submit Show Hint Show Answer Clear 57/59

58 Check yourself Y = 2X 1 2 X 2 + X 3 2 What is variance of Y? Submit Show Hint Show Answer Clear 58/59

59 See you next time 59/59

Statistics. Nicodème Paul Faculté de médecine, Université de Strasbourg. 9/5/2018 Statistics

Statistics. Nicodème Paul Faculté de médecine, Université de Strasbourg. 9/5/2018 Statistics Statistics Nicodème Paul Faculté de médecine, Université de Strasbourg file:///users/home/npaul/enseignement/esbs/2018-2019/cours/01/index.html#21 1/62 Course logistics Statistics Course website: http://statnipa.appspot.com/