Algorithms for Uncertainty Quantification

Transcription

1 Algorithms for Uncertainty Quantification Tobias Neckel, Ionuț-Gabriel Farcaș Lehrstuhl Informatik V Summer Semester 2017

2 Lecture 2: Repetition of probability theory and statistics

3 Example: coin flip

4 Example Experiment 1 given a fair coin (heads and tails) flip it once Q: what is the probability of getting head/tail? Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

5 Example Experiment 1 given a fair coin (heads and tails) flip it once Q: what is the probability of getting head/tail? Answer possible outcomes: head, tail P(head) = P(tail) = 1/2 Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

6 Example (cont d) Experiment 2 given two fair coins flip them twice Q: what is the probability of getting {(head, head),(tail, head)}? What about {(head,head),(head,tail)}? Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

7 Example (cont d) Experiment 2 given two fair coins flip them twice Q: what is the probability of getting {(head, head),(tail, head)}? What about {(head,head),(head,tail)}? Answer possible outcomes: (head, head), (tail, head), (head, tail), (tail, tail) P({(head,head),(tail,head)}) = 1/2 P({(head,head),(head,tail)}) = 1/2 Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

8 Formal definition of probability

9 Two perspectives to probability Frequentist probability = frequency with which an event occurs if the experiment is repeated a large number of times ( ) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

10 Two perspectives to probability Frequentist probability = frequency with which an event occurs if the experiment is repeated a large number of times ( ) Bayesian probability = distribution of subjective values constructed or updated as data is observed Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

11 Probability space A probability space is a triple (Ω,F,P), where Ω: sample space; set of all possible outcomes F : σ algebra; set of events s.t. each event is a set containing zero or more outcomes P : F [0,1] probability measure that satisfies: 1. P( ) = 0 2. P(Ω) = 1 3. if A i F and A i A j =, then P( i=1 A i) = i=1 P(A i) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

12 Coins flip example revised Let h := head, t:= tail Experiment 1 sample space Ω = {h,t} σ algebra F = {,{h},{t},{h,t}} events of interest A = {h}, B = {t} P(A ) = P(B) = 1/2 Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

13 Coins flip example revised Let h := head, t:= tail Experiment 1 sample space Ω = {h,t} σ algebra F = {,{h},{t},{h,t}} events of interest A = {h}, B = {t} P(A ) = P(B) = 1/2 Experiment 2 sample space Ω = {(h,h),(t,h),(h,t),(t,t)} σ algebra F = {,(h,h),(h,t),(t,h),(t,t),{(h,h),(h,t),...},ω} events of interest A = {(h,h),(t,h)}, B = {(h,h),(h,t)} P(A ) = P(B) = 1/2 Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

14 Univariate concepts

15 Univariate concepts: random variables A random variable is a function X : Ω R s.t. {ω Ω X(ω) x} F Example If, in Experiment 2 (two coins flip), X(ω) counts the number of tails, 0, ω = (h,h) 1, ω = (h,t) X(ω) = 1, ω = (t,h) 2, ω = (t,t) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

16 Univariate concepts: random variables A random variable is a function X : Ω R s.t. {ω Ω X(ω) x} F Example If, in Experiment 2 (two coins flip), X(ω) counts the number of tails, 0, ω = (h,h) 1, ω = (h,t) X(ω) = 1, ω = (t,h) 2, ω = (t,t) X is said to be discrete if it takes values in a countable subset {x 1,x 2,...} R; otherwise, it is said to be continuous Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

17 Univariate concepts: CDF Every random variable X has an associated cumulative distribution function (CDF) F X : R [0,1] F X = P(ω Ω X(ω) x) Often, the CDF is expressed as F X (x) = P{X x} Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

18 Univariate concepts: CDF Every random variable X has an associated cumulative distribution function (CDF) F X : R [0,1] F X = P(ω Ω X(ω) x) Often, the CDF is expressed as F X (x) = P{X x} Example 0, ω = (h,h) 1, ω = (h,t) If X(ω) = 1, ω = (t,h) 2, ω = (t,t) 0, x < 0 1/4, 0 x < 1, then F X = 3/4, 1 x < 2 1, x 2 Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

19 Univariate concepts: PDF A random variable X is continuous if its CDF is absolutely continuous, i.e. F X (x) = x f X (s)ds, x R (1) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

20 Univariate concepts: PDF A random variable X is continuous if its CDF is absolutely continuous, i.e. F X (x) = x f X (s)ds, x R (1) From Equation 1, the derivative is called the probability density function (PDF) of X f X (x) = df X(x), f X : R [0, ) dx Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

21 Univariate concepts: PDF Properties of the PDF Let supp(f X ) = {x R : f X (x) 0} f X (x) 0, x supp(f X ) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

22 Univariate concepts: PDF Properties of the PDF Let supp(f X ) = {x R : f X (x) 0} f X (x) 0, x supp(f X ) supp(f X ) f X(x)dx = 1 Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

23 Univariate concepts: expectation, variance Let supp(f X ) = {x R : f X (x) 0} The expectation (mean value, first statistical moment) of a continuous random variable X with PDF f X is defined as µ := E[X] = xf X (x)dx, supp(f X ) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

24 Univariate concepts: expectation, variance Let supp(f X ) = {x R : f X (x) 0} The expectation (mean value, first statistical moment) of a continuous random variable X with PDF f X is defined as µ := E[X] = xf X (x)dx, supp(f X ) The variance (density s variability, second central statistical moment) of a continuous random variable X with PDF f X is defined as σ 2 := Var(X) = (x E[X]) 2 f X (x)dx = E[X 2 ] E[X] 2 supp(f X ) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

25 Univariate concepts: discrete random variables The probability mass function (PMF) of a discrete random variable X is given by f X (x) = P(X = x) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

26 Univariate concepts: discrete random variables The probability mass function (PMF) of a discrete random variable X is given by f X (x) = P(X = x) The (sample) mean ˆX and variance S 2 of a discrete random variable X with equiprobable realizations X 1,...,X n are ˆX = 1 n n X i, S 2 = 1 n 2 i=1 n 1 (X i ˆX) i=1 Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

27 Example distributions Discrete distributions Binomial Poisson Bernoulli geometric... Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

28 Example distributions Discrete distributions Binomial Poisson Bernoulli geometric... Continuous distributions Normal (Gaussian) Uniform Beta Gamma... Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

29 Example continuous distributions: normal distribution The PDF of the normal distribution is f X : R [0, ), f X (x) = 1 σ 2π exp( (x µ)2 /2σ 2 ) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

30 Example continuous distributions: normal distribution The PDF of the normal distribution is f X : R [0, ), f X (x) = 1 σ 2π exp( (x µ)2 /2σ 2 ) The notation X N (µ,σ 2 ) means that X is normally distributed with mean µ and variance σ Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

31 Example continuous distributions: normal distribution The PDF of the normal distribution is f X : R [0, ), f X (x) = 1 σ 2π exp( (x µ)2 /2σ 2 ) The notation X N (µ,σ 2 ) means that X is normally distributed with mean µ and variance σ When µ = 0, σ = 1, X N (0,1) is a standard normal random variable Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

32 One dimensional normal distribution Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

33 Example continuous distributions: uniform distribution The PDF of the uniform distribution is f X : [a,b] {0, 1 b a }, where I [a,b] (x) = { 1, x [a,b] 0, otherwise f X (x) = 1 b a I [a,b](x), Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

34 Example continuous distributions: uniform distribution The PDF of the uniform distribution is f X : [a,b] {0, 1 b a }, where I [a,b] (x) = { 1, x [a,b] 0, otherwise f X (x) = 1 b a I [a,b](x), The notation X U (a,b) means that X is uniformly distributed on the interval [a, b] Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

35 Example continuous distributions: uniform distribution The PDF of the uniform distribution is f X : [a,b] {0, 1 b a }, where I [a,b] (x) = { 1, x [a,b] 0, otherwise f X (x) = 1 b a I [a,b](x), The notation X U (a,b) means that X is uniformly distributed on the interval [a, b] Expectation and variance of the uniform distribution If X U (a,b), E[X] = a+b (b a)2 2, Var(X) = 12 Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

36 One dimensional uniform distribution Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

37 Multivariate concepts

38 Multivariate concepts: random vectors, covariance, correlation X : Ω R n, X = [X 1,X 2,...,X n ] is called an n-dimensional random vector, where X 1,X 2,...,X n are random variables Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

39 Multivariate concepts: random vectors, covariance, correlation X : Ω R n, X = [X 1,X 2,...,X n ] is called an n-dimensional random vector, where X 1,X 2,...,X n are random variables The covariance of two random variables X and Y is cov(x,y ) = E[(X E[X])(Y E[Y ])] = E[XY ] E[X]E[Y ] Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

40 Multivariate concepts: random vectors, covariance, correlation X : Ω R n, X = [X 1,X 2,...,X n ] is called an n-dimensional random vector, where X 1,X 2,...,X n are random variables The covariance of two random variables X and Y is cov(x,y ) = E[(X E[X])(Y E[Y ])] = E[XY ] E[X]E[Y ] The Pearson correlation coefficient of two random variables X and Y is ρ XY = cov(x, Y) Var(X)Var(Y ), ρ XY [ 1,1] Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

41 Multivariate concepts: expectation, variance of a sum Let a 1,a 2,...,a n R and X 1,X 2,...,X n be random variables E [ n ] n a i X i = a i E[X i ] i=1 i=1 Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

42 Multivariate concepts: expectation, variance of a sum Let a 1,a 2,...,a n R and X 1,X 2,...,X n be random variables E [ n ] n a i X i = a i E[X i ] i=1 i=1 Var ( n ) n a i X i = ai 2 Var(X i) + 2 a i a j cov(x i,x j ) i=1 i=1 i<j If X i,x i+1 are uncorrelated, i.e. ρ Xi X i+1 = 0, 1 i n, Var ( n ) n a i X i = ai 2 Var(X i) i=1 i=1 Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

43 Multivariate concepts: independence Let A, B denote two events. The joint probability of A and B, P(A,B), is defined as P(A,B) = P(A B) = P(A B)P(B), where P(A B) is the probability of A given that B already happened Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

44 Multivariate concepts: independence Let A, B denote two events. The joint probability of A and B, P(A,B), is defined as P(A,B) = P(A B) = P(A B)P(B), where P(A B) is the probability of A given that B already happened A and B are independent if P(A,B) = P(A)P(B), i. e. P(A B) = P(A) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

45 Multivariate concepts: independence Let A, B denote two events. The joint probability of A and B, P(A,B), is defined as P(A,B) = P(A B) = P(A B)P(B), where P(A B) is the probability of A given that B already happened A and B are independent if P(A,B) = P(A)P(B), i. e. P(A B) = P(A) The random variables X and Y are independent if their joint PDF (PMF) is f XY (x,y) = f X (x)f Y (y) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

46 Multivariate concepts: i.i.d. random variables Random variables X 1,X 2,...,X n are called independent and identically distributed (i.i.d.) with PDF f X if they are mutually independent and, if f Xi is the PDF of X i, 1 i n, f X1 = f X2 =... = f Xn := f X and f X (x 1,x 2,...x n ) = n i=1 f X (x i ) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

47 Example multivariate distribution: multivariate normal The n-dimensional random vector X is normally distributed with mean vector µ = [µ 1, µ 2,..., µ n ] T and covariance matrix V,V ij = cov(x i,x j ), written X N (µ,v ), if f X (x) = where V is the determinant of V 1 (2π)n V exp[ 1 2 (x µ)v 1 (x µ) T ] Standard multivariate normal: µ = [0,0,...,0] T, V = I n Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

48 Standard bivariate normal distribution Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester

49 Summary

50 Summary Coin flip experiment Probability space (Ω,F,P) Univariate concepts Random variables: discrete and continuous Cumulative distribution function (CDF) Continuous random variables Discrete random variables Examples: normal, uniform distributions Multivariate concepts Random vectors Covariance, correlation Independent and identically distributed random variables Example: bivariate normal distribution Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester