Algorithms for Uncertainty Quantification

Algorithms for Uncertainty Quantification Tobias Neckel, Ionuț-Gabriel Farcaș Lehrstuhl Informatik V Summer Semester 2017

Lecture 2: Repetition of probability theory and statistics

Example: coin flip

Example Experiment 1 given a fair coin (heads and tails) flip it once Q: what is the probability of getting head/tail? Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 4

Example Experiment 1 given a fair coin (heads and tails) flip it once Q: what is the probability of getting head/tail? Answer possible outcomes: head, tail P(head) = P(tail) = 1/2 Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 4

Example (cont d) Experiment 2 given two fair coins flip them twice Q: what is the probability of getting {(head, head),(tail, head)}? What about {(head,head),(head,tail)}? Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 5

Example (cont d) Experiment 2 given two fair coins flip them twice Q: what is the probability of getting {(head, head),(tail, head)}? What about {(head,head),(head,tail)}? Answer possible outcomes: (head, head), (tail, head), (head, tail), (tail, tail) P({(head,head),(tail,head)}) = 1/2 P({(head,head),(head,tail)}) = 1/2 Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 5

Formal definition of probability

Two perspectives to probability Frequentist probability = frequency with which an event occurs if the experiment is repeated a large number of times ( ) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 7

Two perspectives to probability Frequentist probability = frequency with which an event occurs if the experiment is repeated a large number of times ( ) Bayesian probability = distribution of subjective values constructed or updated as data is observed Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 7

Probability space A probability space is a triple (Ω,F,P), where Ω: sample space; set of all possible outcomes F : σ algebra; set of events s.t. each event is a set containing zero or more outcomes P : F [0,1] probability measure that satisfies: 1. P( ) = 0 2. P(Ω) = 1 3. if A i F and A i A j =, then P( i=1 A i) = i=1 P(A i) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 8

Coins flip example revised Let h := head, t:= tail Experiment 1 sample space Ω = {h,t} σ algebra F = {,{h},{t},{h,t}} events of interest A = {h}, B = {t} P(A ) = P(B) = 1/2 Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 9

Coins flip example revised Let h := head, t:= tail Experiment 1 sample space Ω = {h,t} σ algebra F = {,{h},{t},{h,t}} events of interest A = {h}, B = {t} P(A ) = P(B) = 1/2 Experiment 2 sample space Ω = {(h,h),(t,h),(h,t),(t,t)} σ algebra F = {,(h,h),(h,t),(t,h),(t,t),{(h,h),(h,t),...},ω} events of interest A = {(h,h),(t,h)}, B = {(h,h),(h,t)} P(A ) = P(B) = 1/2 Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 9

Univariate concepts

Univariate concepts: random variables A random variable is a function X : Ω R s.t. {ω Ω X(ω) x} F Example If, in Experiment 2 (two coins flip), X(ω) counts the number of tails, 0, ω = (h,h) 1, ω = (h,t) X(ω) = 1, ω = (t,h) 2, ω = (t,t) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 11

Univariate concepts: random variables A random variable is a function X : Ω R s.t. {ω Ω X(ω) x} F Example If, in Experiment 2 (two coins flip), X(ω) counts the number of tails, 0, ω = (h,h) 1, ω = (h,t) X(ω) = 1, ω = (t,h) 2, ω = (t,t) X is said to be discrete if it takes values in a countable subset {x 1,x 2,...} R; otherwise, it is said to be continuous Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 11

Univariate concepts: CDF Every random variable X has an associated cumulative distribution function (CDF) F X : R [0,1] F X = P(ω Ω X(ω) x) Often, the CDF is expressed as F X (x) = P{X x} Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 12

Univariate concepts: CDF Every random variable X has an associated cumulative distribution function (CDF) F X : R [0,1] F X = P(ω Ω X(ω) x) Often, the CDF is expressed as F X (x) = P{X x} Example 0, ω = (h,h) 1, ω = (h,t) If X(ω) = 1, ω = (t,h) 2, ω = (t,t) 0, x < 0 1/4, 0 x < 1, then F X = 3/4, 1 x < 2 1, x 2 Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 12

Univariate concepts: PDF A random variable X is continuous if its CDF is absolutely continuous, i.e. F X (x) = x f X (s)ds, x R (1) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 13

Univariate concepts: PDF A random variable X is continuous if its CDF is absolutely continuous, i.e. F X (x) = x f X (s)ds, x R (1) From Equation 1, the derivative is called the probability density function (PDF) of X f X (x) = df X(x), f X : R [0, ) dx Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 13

Univariate concepts: PDF Properties of the PDF Let supp(f X ) = {x R : f X (x) 0} f X (x) 0, x supp(f X ) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 14

Univariate concepts: PDF Properties of the PDF Let supp(f X ) = {x R : f X (x) 0} f X (x) 0, x supp(f X ) supp(f X ) f X(x)dx = 1 Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 14

Univariate concepts: expectation, variance Let supp(f X ) = {x R : f X (x) 0} The expectation (mean value, first statistical moment) of a continuous random variable X with PDF f X is defined as µ := E[X] = xf X (x)dx, supp(f X ) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 15

Univariate concepts: expectation, variance Let supp(f X ) = {x R : f X (x) 0} The expectation (mean value, first statistical moment) of a continuous random variable X with PDF f X is defined as µ := E[X] = xf X (x)dx, supp(f X ) The variance (density s variability, second central statistical moment) of a continuous random variable X with PDF f X is defined as σ 2 := Var(X) = (x E[X]) 2 f X (x)dx = E[X 2 ] E[X] 2 supp(f X ) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 15

Univariate concepts: discrete random variables The probability mass function (PMF) of a discrete random variable X is given by f X (x) = P(X = x) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 16

Univariate concepts: discrete random variables The probability mass function (PMF) of a discrete random variable X is given by f X (x) = P(X = x) The (sample) mean ˆX and variance S 2 of a discrete random variable X with equiprobable realizations X 1,...,X n are ˆX = 1 n n X i, S 2 = 1 n 2 i=1 n 1 (X i ˆX) i=1 Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 16

Example distributions Discrete distributions Binomial Poisson Bernoulli geometric... Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 17

Example distributions Discrete distributions Binomial Poisson Bernoulli geometric... Continuous distributions Normal (Gaussian) Uniform Beta Gamma... Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 17

Example continuous distributions: normal distribution The PDF of the normal distribution is f X : R [0, ), f X (x) = 1 σ 2π exp( (x µ)2 /2σ 2 ) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 18

Example continuous distributions: normal distribution The PDF of the normal distribution is f X : R [0, ), f X (x) = 1 σ 2π exp( (x µ)2 /2σ 2 ) The notation X N (µ,σ 2 ) means that X is normally distributed with mean µ and variance σ Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 18

Example continuous distributions: normal distribution The PDF of the normal distribution is f X : R [0, ), f X (x) = 1 σ 2π exp( (x µ)2 /2σ 2 ) The notation X N (µ,σ 2 ) means that X is normally distributed with mean µ and variance σ When µ = 0, σ = 1, X N (0,1) is a standard normal random variable Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 18

One dimensional normal distribution Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 19

Example continuous distributions: uniform distribution The PDF of the uniform distribution is f X : [a,b] {0, 1 b a }, where I [a,b] (x) = { 1, x [a,b] 0, otherwise f X (x) = 1 b a I [a,b](x), Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 20

Example continuous distributions: uniform distribution The PDF of the uniform distribution is f X : [a,b] {0, 1 b a }, where I [a,b] (x) = { 1, x [a,b] 0, otherwise f X (x) = 1 b a I [a,b](x), The notation X U (a,b) means that X is uniformly distributed on the interval [a, b] Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 20

Example continuous distributions: uniform distribution The PDF of the uniform distribution is f X : [a,b] {0, 1 b a }, where I [a,b] (x) = { 1, x [a,b] 0, otherwise f X (x) = 1 b a I [a,b](x), The notation X U (a,b) means that X is uniformly distributed on the interval [a, b] Expectation and variance of the uniform distribution If X U (a,b), E[X] = a+b (b a)2 2, Var(X) = 12 Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 20

One dimensional uniform distribution Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 21

Multivariate concepts

Multivariate concepts: random vectors, covariance, correlation X : Ω R n, X = [X 1,X 2,...,X n ] is called an n-dimensional random vector, where X 1,X 2,...,X n are random variables Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 23

Multivariate concepts: random vectors, covariance, correlation X : Ω R n, X = [X 1,X 2,...,X n ] is called an n-dimensional random vector, where X 1,X 2,...,X n are random variables The covariance of two random variables X and Y is cov(x,y ) = E[(X E[X])(Y E[Y ])] = E[XY ] E[X]E[Y ] Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 23

Multivariate concepts: random vectors, covariance, correlation X : Ω R n, X = [X 1,X 2,...,X n ] is called an n-dimensional random vector, where X 1,X 2,...,X n are random variables The covariance of two random variables X and Y is cov(x,y ) = E[(X E[X])(Y E[Y ])] = E[XY ] E[X]E[Y ] The Pearson correlation coefficient of two random variables X and Y is ρ XY = cov(x, Y) Var(X)Var(Y ), ρ XY [ 1,1] Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 23

Multivariate concepts: expectation, variance of a sum Let a 1,a 2,...,a n R and X 1,X 2,...,X n be random variables E [ n ] n a i X i = a i E[X i ] i=1 i=1 Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 24

Multivariate concepts: expectation, variance of a sum Let a 1,a 2,...,a n R and X 1,X 2,...,X n be random variables E [ n ] n a i X i = a i E[X i ] i=1 i=1 Var ( n ) n a i X i = ai 2 Var(X i) + 2 a i a j cov(x i,x j ) i=1 i=1 i<j If X i,x i+1 are uncorrelated, i.e. ρ Xi X i+1 = 0, 1 i n, Var ( n ) n a i X i = ai 2 Var(X i) i=1 i=1 Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 24

Multivariate concepts: independence Let A, B denote two events. The joint probability of A and B, P(A,B), is defined as P(A,B) = P(A B) = P(A B)P(B), where P(A B) is the probability of A given that B already happened Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 25

Multivariate concepts: independence Let A, B denote two events. The joint probability of A and B, P(A,B), is defined as P(A,B) = P(A B) = P(A B)P(B), where P(A B) is the probability of A given that B already happened A and B are independent if P(A,B) = P(A)P(B), i. e. P(A B) = P(A) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 25

Multivariate concepts: independence Let A, B denote two events. The joint probability of A and B, P(A,B), is defined as P(A,B) = P(A B) = P(A B)P(B), where P(A B) is the probability of A given that B already happened A and B are independent if P(A,B) = P(A)P(B), i. e. P(A B) = P(A) The random variables X and Y are independent if their joint PDF (PMF) is f XY (x,y) = f X (x)f Y (y) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 25

Multivariate concepts: i.i.d. random variables Random variables X 1,X 2,...,X n are called independent and identically distributed (i.i.d.) with PDF f X if they are mutually independent and, if f Xi is the PDF of X i, 1 i n, f X1 = f X2 =... = f Xn := f X and f X (x 1,x 2,...x n ) = n i=1 f X (x i ) Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 26

Example multivariate distribution: multivariate normal The n-dimensional random vector X is normally distributed with mean vector µ = [µ 1, µ 2,..., µ n ] T and covariance matrix V,V ij = cov(x i,x j ), written X N (µ,v ), if f X (x) = where V is the determinant of V 1 (2π)n V exp[ 1 2 (x µ)v 1 (x µ) T ] Standard multivariate normal: µ = [0,0,...,0] T, V = I n Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 27

Standard bivariate normal distribution Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 28

Summary

Summary Coin flip experiment Probability space (Ω,F,P) Univariate concepts Random variables: discrete and continuous Cumulative distribution function (CDF) Continuous random variables Discrete random variables Examples: normal, uniform distributions Multivariate concepts Random vectors Covariance, correlation Independent and identically distributed random variables Example: bivariate normal distribution Dr. rer. nat. Tobias Neckel Algorithms for Uncertainty Quantification Summer Semester 2017 30