Statistika pro informatiku

Transcription

1 Statistika pro informatiku prof. RNDr. Roman Kotecký DrSc., Dr. Rudolf Blažek, PhD Katedra teoretické informatiky FIT České vysoké učení technické v Praze MI-SPI, ZS 2011/12, Přednáška 5 Evropský sociální fond. Praha & EU: Investujeme do vaší budoucnosti Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 1 / 23

2 Recapitulation Recapitulation Bernoulli random variable: p X (0) = 1 p and p X (1) = p, E(X) = p, var(x) = pq. Binomial random variable: p X (k) = ( n k) pk (1 p) n k, k = 1,..., n, E(X) = n p, var(x) = n p q. Geometric random variable: p X (k) = (1 p) k 1 p, k = 1, 2,..., E(X) = 1 p, var(x) = q p 2. Poisson random variable: p X (k) = λk k! e λ, k = 0, 1, 2,..., E(X) = var(x) = λ. Independence of random variables X and Y : {X = x} and {Y = y} are independent for each x and y. Random variables X and Y are noncorrelated: E(XY) = E(X)E(Y). Independence of X and Y implies that they are noncorrelated. var(ax) = a 2 var(x). var(x + Y) = var(x) + var(y) if X and Y are noncorrelated. Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 2 / 23

3 Probability on uncountable spaces σ-algebra of events σ-algebra Minimal requirements for a reasonable set of events (in view the definition of a probability P): Definition A set F P(Ω) is a σ-algebra if (i) Ω F, (ii) A F implies that A c F, and (iii) A 1, A 2, F implies l 1 A l F. Examples If Ω is countable, we can take F = P(Ω). For Ω R, take the smallest σ-algebra containing all open intervals, F = B(Ω) (Borel σ-algebra). For Ω = {0, 1} N, take the smallest σ-algebra containing all sets of the form C(S, ω) = {ω Ω : ω S = ω}, where S N is finite and ω {0, 1} S. Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 3 / 23

4 Probability on uncountable spaces Definition of probability Definition of probability Definition Probability is a function P : F [0, 1] such that (N) P(Ω) = 1, (A) If A 1, A 2, F are pairwise disjoint (A i A j = for i j), then P( l 1 A l ) = P(A l ) l 1 (N)... normalisation, (A)... σ-additivity. Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 4 / 23

5 Probability on uncountable spaces Assigning probabilities to events Defining probabilities We need to assign a value P(A) [0, 1] for each A F. A standard way of introducing probabilities: Examples Discrete case: (countable Ω with F = P(Ω)). Consider a number p(ω) [0, 1] for each ω Ω such that ω Ω p(ω) = 1. For any A Ω, take P(A) = p(ω). ω A Continuous case: (Ω R n with F = B(Ω)). Consider a function ϱ : Ω [0, ] such that {x R n : ϱ(x) c} B(R n ) for all c > 0 and Ω ϱ(x)dx = 1. For any A B(Rn ), take P(A) = ϱ(x)dx. A Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 5 / 23

6 Probability on uncountable spaces Uniform distributions Uniform probability distributions Discrete case Assume that Ω is finite and the distribution is uniform (like for coin toss, rolling of a die, etc.). Then, p(ω) = 1 Ω and P(A) = A # of favourable outcomes = Ω # of possible outcomes for any A Ω. Continuous case Assume that Ω R n is Borel set of finite volume, 0 < λ n (Ω) = Ω dx < and take ϱ(x) = 1/λ n (Ω). Uniform distribution: P(A) = A 1/λ n (Ω)dx = λn (A) λ n (Ω) for any A B(Ω). Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 6 / 23

7 Probability on uncountable spaces Uniform distributions Example (Bertrand s paradox) Random chord χ in a circle of radius one. χ inscribed equilateral triangle? What is P(A), where A = { χ > l} and l is the side of an l χ 1 Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 7 / 23

8 Probability on uncountable spaces Uniform distributions Example Depends on what is meant by random : 1. Choose uniformly the centre point of χ: x χ Ω 1 = {x R 2 : x < 1}, A 1 = {x Ω 1 : x < 1/2}, P 1 (A 1 ) = π(1/2)2 π1 2 = Choose uniformly the angular size and the direction (irrelevant due to symmetry) of χ as seen from the centre: Ω 2 = (0, π], A 2 = (2π/3, π], P 2 (A 2 ) = π/3 π = 1 3. χ 3. Choose uniformly the distance and the direction (irrelevant) of χ from the centre: Ω 3 = [0, 1), A 3 = [0, 1/2), P 3 (A 3 ) = Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 8 / 23

9 Probability on uncountable spaces Properties of probabilities As before (subaditivity, monotonicity, inclusion-exclusion,... ). In addition: Theorem Consider a probability P on (Ω, F) and a sequence of events A 1, A 2, F. Then σ-continuity: If either A n A (i.e., A 1 A 2... and A = n 1 A n ) or A n A (A 1 A 2... and A = n 1 A n ), then lim n P(A n ) = P(A). Proof. A n A: P(A) = P( n 1 (A n \ A n 1 )) = n 1 P(A n \ A n 1 ) = lim n n k=1 P(A k \ A k 1 ) = lim n P(A n ). Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 9 / 23

10 Probability on uncountable spaces Properties of probabilities Application for a repeated coin toss Example (Usefulness of reasoning with sequences) A fair coin is tossed repeatedly. Show that, with probability one, a head turns up eventually. First, P(some head turns up) = 1 P(no head ever). Now, denoting A n = {no head in first n tosses} and A = {no head ever}, we have A n+1 A n and A = n 1 A n. Indeed, we have A A n for every n and thus A n 1 A n. On the other hand, n 1 A n A since A c n 1 A c n. Indeed, if ω / A, there is a head in the sequence ω and thus there exists n such that there is a head in first n tosses: ω / A n, i.e. ω A c n. Thus, P(no head ever) = lim P(no head in first n tosses) = lim n n 2 n = 0. Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 10 / 23

11 Continuous random variables Density of a random variable Definition A random variable X is called discrete if X(Ω) R is countable. Probability mass function p X : R [0, 1] is given by p X (x) = P(X = x). Cumulative distribution function is F X (x) = x:x x p X(x). A random variable X is called continuous if P(a X b) = b a f X (x)dx, a b R for some nonnegative (and integrable) function f X (x). The function f X (x) is called probability density of the random variable X. P(a X b) f X (x) a b Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 11 / 23

12 Continuous random variables Cumulative distribution function Remark Cumulative distribution function of a continuous random variable X is thus defined by F X (x) = P(X x) = x f X (u)du. F X (x) =P(X x) f X (x) x Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 12 / 23

13 Continuous random variables Properties of density Properties of the density of a random variable Remark For any continuous random variable X with density f X (x): P(X B) = f X (x)dx, B R. ({X B} has to be measurable.) B In particular, P(a X b) = b a f X (x)dx, a b. dfx (x) f X (x) = x R. (More precisely, for almost all x.) dx f X (x) 0 for almost all x. Hence also p X (x) = P(X = x) = 0 for any x. Overall area below the density f X of the random variable X equals 1: 1 = P( X ) = f X (x)dx. Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 13 / 23

14 Continuous random variables Expectation Expectation of a continuous random variable Definition Expectation of a continuous random variable X with density f X is defined by E(X) = x f X (x) dx. E(X) is conceived as the Expected value for the next random experiment, or the weighted mean, or the barycenter of all possible values. f X (x) EX Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 14 / 23

15 Continuous random variables Variance and general moments of the random variable Variance of a continuous random variable Definice Consider a continuous random variable X with density f X. Expectation of a function g(x) is defined by E(g(X)) = g(x) f X (x) dx. Variance of X is defined by Rozptyl var(x) = E[(X E(X)) 2 ] = (x E(X)) 2 f X (x) dx. We conceive var(x) as a mean quadratic deviation from the mean value (i.e. from the barycenter). Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 15 / 23

16 Continuous random variables Variance and general moments of the random variable General moments of a continuous random variable Definice Consider a continuous random variable X with density f X. the k-th moment of X is defined by m k = E(X k ) = x k f X (x) dx, k = 1, 2,... the k-th central moment of X is defined by σ k = E((X E(X)) k ) = E((X m 1 ) k ) = (x m 1 ) k f X (x) dx. The variance var(x) = σ 2 is often denoted as σ 2 or σx 2 with σ called the standard deviation. směrodatná odchylka Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 16 / 23

17 Examples of continuous random variables Uniform distribution Uniform distribution Example Random variable X has the uniform distribution on the interval [a, b], if its density is 1 b f X (x) = a, for a x b 0, otherwise. f X (x) Notation: X Unif(a, b) a b (Rovnoměrné = Uniform) Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 17 / 23

18 Examples of continuous random variables Uniform distribution Example (continuation) If X Unif(a, b), then E(X) = x f X (x) dx = = 1 b a 1 2 x 2 = a + b. 2 b a b a x 1 b a dx = 1 b a b 2 a 2 2 f X (x) a EX = a + b 2 b Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 18 / 23

19 Examples of continuous random variables Uniform distribution Example (continuation) var(x) = E(X 2 ) (E(X)) 2. For X Unif(a, b), we get E(X 2 ) = x 2 f X (x) dx = = 1 b a 1 3 x 3 = a2 + ab + b 2. 3 b a b a = b3 a 3 3(b a) x 2 1 b a dx Hence var(x) = E(X 2 ) (E(X)) 2 = a2 + ab + b 2 3 (b a)2 = 12 (a + b)2 4 Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 19 / 23

20 Examples of continuous random variables Uniform distribution Example (continuation) Summarizing, for a random variable with the uniform distribution: E(X) = a + b, var(x) = 2 (b a)2. 12 Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 20 / 23

21 Examples of continuous random variables Exponential distribution Exponential distribution Example A random variable X has an exponential distribution with parameter λ, if its density is (for some λ > 0) { λe λx, for x 0 f X (x) = 0, otherwise. Notation: X Exp(λ) (λ > 0 is the parameter of the distribution) 2 λ= λ=1 λ=1/ Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 21 / 23

22 Examples of continuous random variables Exponential distribution Example (continuation) The expectation of X Exp(λ) is computed by the integration by part u v = u v u v: per partes, po částech E(X) = x f X (x) dx = x λe λx dx 0 (u = x; u = 1); (v = λe λx ; v = e λx ) = ( x e λx ) + e λx dx 0 0 = 0 e λx λ = 1 λ 0 Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 22 / 23

23 Examples of continuous random variables Exponential distribution Example (continuation) Similarly, E(X 2 ) = x 2 f X (x) dx = x 2 λe λx dx 0 (u = x 2 ; u = 2x); (v = λe λx ; v = e λx ) = ( x 2 e λx ) + 2x e λx dx 0 0 = λ 0 x λe λx dx = 2 λ E(X) = 2 λ 2 ) 2 = 1 ( Hence var(x) = E(X 2 ) (E(X)) 2 = 2 1 λ 2 λ λ 2 Summarizing, the expectation and variance of a random variable with exponential distribution is E(X) = 1 λ, var(x) = 1 λ 2. Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 5 23 / 23