Statistics, Data Analysis, and Simulation SS 2017

Transcription

1 Statistics, Data Analysis, and Simulation SS Statistik, Datenanalyse und Simulation Dr. Michael O. Distler Mainz, 27. April 2017 Dr. Michael O. Distler Statistics, Data Analysis, and Simulation SS / 55

2 What we ve learned so far Fundamental concepts random variable, probability frequentist vs. bayesian interpretation Dr. Michael O. Distler Statistics, Data Analysis, and Simulation SS / 55

3 1.2 Probability density function probability mass function (pmf) probability density function (pdf) of a measured value (=random variable) f(n) n f(x) f (n) discrete f (x) continuous Normalization: f (n) 0 f (n) = 1 f (x) 0 f (x) dx = 1 Probability: n p(n 1 n n 2 ) = n 2 x n 1 f (n) p(x 1 x x 2 ) = x2 x 1 f (x)dx Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

4 Definitions Cumulative distribution function (CDF): F(x) = x f (x )dx, F( ) = 0, F( ) = 1 Example: Decay time t of a radioactive nucleus with mean life time τ: f (t) = 1 τ e t/τ F(t) = 1 e t/τ f(t)*12s F(t) t/s Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

5 Expectation values and moments Mean: A random variable X takes on the values X 1, X 2,..., X n with probability p(x i ), then the expected value of X ( mean ) is X = X = n X i p(x i ) i=1 The expected value of an arbitrary function h(x) for a continuous random variable is: E[h(x)] = The mean ist the expected value of x: E[x] = x = h(x) f (x)dx x f (x)dx Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

6 Expectation values and moments standard deviation = {mean (deviation from x) 2 } 1/2 σ 2 = (x x) 2 = = (x x) 2 f (x)dx (x 2 2x x + x 2 ) f (x)dx = x 2 2 x x + x 2 = x 2 x 2 σ 2 = Variance, σ = Standard deviation Discrete distributions: ( x 2 ( x) 2 ) σ 2 = 1 N N Attention: This is the definition of the variance! To get a bias free estimation of the variance, 1 1 N will be replaced by N 1. Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

7 Expectation values and moments Moments are the expected value of x n and of (x x ) n. They are called nth algebraic moment µ n and nth central moment µ n, respectivly. Skewness v(x) is a measure of the asymmetry of the probability distribution of a random variable x: v = µ 3 σ 3 = E[(x E[x])3 ] σ 3 Kurtosis is a measure of the peakedness of the probability distribution of a random variable x. β 2 = µ 4 σ 4 = E[(x E[x])4 ] σ 4 γ 2 = β 2 3 (excess kurtosis) Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

8 Example distributions Negative Skew Positive Skew Kurtosis of well-known distributions D: Laplace distribution, γ 2 = 3 S: hyperbolic secant distribution, γ 2 = 2 L: logistic distribution, γ 2 = 1.2 N: normal distribution, γ 2 = 0 C: raised cosine distribution, γ 2 = W: Wigner semicircle distribution, γ 2 = 1 U: uniform distribution, γ 2 = 1.2 Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

9 Descriptive statistics Describing the distribution of a single variable, including its central tendency median, mode, mean dispersion range, quantiles, measures of spread: variance and standard deviation, rms, FWHM shape skewness, kurtosis,... Characteristics of a variable s distribution may also be depicted in graphical or tabular format, including histograms. Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

10 Binomial distribution The binomial distribution is the discrete probability distribution of the number of successes r in a sequence of n independent yes/no experiments, each of which yields success with probability p (Bernoulli experiment). P(r) = ( n r ) p r (1 p) n r P(r) is normalized. Proof: Binomial theorem with q = 1 p. The mean of r is: n r = E[r] = rp(r)= np The variance σ 2 is V [r] = E[(r r ) 2 ] = r=0 n (r r ) 2 P(r)= np(1 p) r=0 Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

11 Example: How big is the chance to get with n = 6 throws of a dice exactly zero times the 6, exactly twice the 6, and at least once the 6? For a correct dice is p = 1/6 and ( ) 1 0 ( ) 5 6 ( ) 6 P(0) = = 33.5% ( ) 1 2 ( ) 5 4 ( ) 6 P(2) = = 20.1% P( 1) = (1 P(0)) = 66.5% Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

12 Benford s law Benford s law, also called the first-digit law, refers to the frequency distribution of digits in many (but not all) real-life sources of data (atomic weights, baseball results, electricity bills,...): digit probability 30.1% 17.6% 12.5% 9.7% 7.9% digit probability 6.7% 5.8% 5.1% 4.6% A set of numbers is said to satisfy Benford s law if the leading digit d occurs with probability p(d) = log 10 (d + 1) log 10 (d) Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

13 Benford s law Example: The size of the files in my home directory: d n d p / % p(d) theor. /% ± ± ± ± ± ± ± ± ± Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

14 Benford s law find ~ -type f -printf "%s\n" awk { l=log($1)/log(10); a[int(exp(log(10)*(l-int(l))))]++; N++; }END{ print N; for (i=1;i<10;i++) { printf "%d %5d %5.1f +/- %3.1f %5.1f\n", i,a[i],100*a[i]/n,100*sqrt(a[i])/n, 100*(log(i+1)-log(i))/log(10); } } Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

15 1.3 Special discrete distributions (Poisson distribution) The Poisson distribution gives the probability of getting exactly r events when the number of trials is very large and the probability of the occurrence of an event in a single trial p is very small, with a finite mean r = µ = np. The Poisson distribution can be derived as a limit of the binomial distribution and has only one parameter, namely the mean µ. The Poisson distribution is given as: P(r) = µr e µ r! The Poisson distribution occurs in many cases where one counts things or events, such as the number of nuclear reactions or particle decays or the number of fish caught in a fishing competition. Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

16 Poisson distribution The Poisson distribution ist given by: The mean is: The variance is: The skewness is: P(r) = µr e µ r! r = µ V [r] = σ 2 = µ v = µ 3 σ 3 = 1 µ The excess kurtosis is: γ 2 = 1 µ µ = µ = µ = µ = Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

17 Death by horse kicks in the Prussian army Since 1898 the number of over a period of 20 years killed cavalrymen in the Prussian army is given in many textbooks. Deaths r Σ Years per corps with r deaths Expected # The total number of deaths is 122, and the mean number of deaths per corps and year is µ = 122/200 = The agreement between the expected and observed numbers is very good - actually too well. More examples: Radioactive decay Printing errors per page in books Simultaneously made scientific discoveries Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

18 1.4 Special probability densities Uniform distribution: This probability distribution is constant in between the limits x = a and x = b: f (x) = Mean and variance: { 1 b a a x < b 0 otherwise x = E[x] = a + b 2 V [x] = σ 2 = (b a)2 12 Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

19 Gaussian distribution The most important probability distribution - also called normal distribution: f (x) = 1 e (x µ)2 2σ 2 2πσ The Gaussian distribution has two parameters, the mean µ and the variance σ 2. The probability distribution with mean µ = 0 and variance σ 2 = 1 is named standard normal distribution or short N(0, 1). The Gaussian distribution can be derived from the binomial distribution for large values of n and r and similarly from the Poisson distribution for large values of µ. Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

20 Gaussian distribution dx N(0, 1) = = ( ) dx N(0, 1) = = ( ) dx N(0, 1) = = ( ) FWHM: useful to estimate the standard deviation: FWHM = 2σ 2ln2 = 2.355σ Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

21 Gaussian distribution Left side: The binomial distribution for n = 10 and p = 0.6 in comparison to the Gaussian distribution for µ = np = 6 and σ = np(1 p) = 2.4. Right side: The Poisson distribution for µ = 6 and σ = 6 in comparison to the Gaussian distribution. Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

22 Cumulative Gaussian distribution The cumulative Gaussian distribution Φ(x) = 1 2πσ x e (t µ) 2 2σ 2 dt. cannot be expressed analytically and must be evaluated numerically. F(x) = 1 z e x2 2. 2π However it can be expressed in terms of the Gaussian error function erf(x) which is available on many modern calculators or computer libraries erf(x) = 2 π x Φ(x) = e t2 dt. ( ( )) x µ 1 + erf. 2σ Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

23 Cumulative Gaussian distribution *(1+erf(x/sqrt(2))) 0.4*exp(-0.5*x*x) Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

24 Full moon and accidents Do more accidents happen on days with full moon? To discover such an effect the number of accidents in many German cities are compared. We find that in Hamburg, the average number of accidents on days with full moon 10.0 with a standard deviation of 1.0, and on the other days it is 7.0 with negligible error. This effect is significant? Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

25 Full moon and accidents Do more accidents happen on days with full moon? To discover such an effect the number of accidents in many German cities are compared. We find that in Hamburg, the average number of accidents on days with full moon 10.0 with a standard deviation of 1.0, and on the other days it is 7.0 with negligible error. This effect is significant? But this doesn t mean anything in reality. If one is conducting this investigation in 200 cities, then the probability that the accident rate differs more than 3 standard deviations from the mean in any one city is: And this probability is not small = 0.23 Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

26 press any key Dr. Michael O. Distler Statistics, Data Analysis, and Simulation SS / 55

27 Chi-square distribution If x 1, x 2,..., x n are independend random variables distributed according to the standard Gaussian distribution with mean 0 and variance 1, then the sum u = χ 2 = n i=1 x 2 i ist distributed according to a χ 2 distribution f n (u) = f n (χ 2 ) where n is called the number of degrees of freedom. f n (u) = ( 1 u ) n/ e u/2 Γ(n/2) The χ 2 distribution has a maximum at (n 2). The mean is found to be n and the variance is 2n. Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

28 Chi-square distribution pdf(2,x) pdf(3,x) pdf(4,x) pdf(5,x) pdf(6,x) pdf(7,x) pdf(8,x) pdf(9,x) Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

29 Chi-square cumulative distribution function The probability for χ 2 n to take on a value in the interval [0, x] cdf(2,x) cdf(3,x) cdf(4,x) cdf(5,x) cdf(6,x) cdf(7,x) cdf(8,x) cdf(9,x) Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

30 χ 2 vs. χ 2 red. Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55

31 Chi-square distribution with 5 d.o.f % c.l. [ ] Dr. Michael O. Distler <distler@uni-mainz.de> Statistics, Data Analysis, and Simulation SS / 55