Expectation. DS GA 1002 Probability and Statistics for Data Science. Carlos Fernandez-Granda

Transcription

1 Expectation DS GA 1002 Probability and Statistics for Data Science Carlos Fernandez-Granda

2 Aim Describe random variables with a few numbers: mean, variance, covariance

3 Expectation operator Mean and variance Covariance Conditional expectation

4 Discrete random variables Average of the values of a function weighted by the pmf E (g (X )) = x R g (x) p X (x) E (g (X, Y )) = x R X x R Y g (x, y) p X,Y (x, y) ( ( )) E g X = x1 g ( x) p X ( x) x 2 x n

5 Continuous random variables Average of the values of a function weighted by the pdf E (g (X )) = x= g (x) f X (x) dx E (g (X, Y )) = x= y= g (x, y) f X,Y (x, y) dx dy ( ( )) E g X = g ( x) f X ( x) dx 1 dx 2... dx n x 1 = x 2 = x n=

6 Discrete and continuous random variables E (g (C, D)) = = c= d R D d R D g (c, d) f C (c) p D C (d c) dc c= g (c, d) p D (d) f C D (c d) dc

7 St Petersburg paradox A casino offers you a game Flip an unbiased coin until it lands on heads You get 2 k dollars where k = number of flips Expected gain?

8 St Petersburg paradox E (Gain) = 2 k 1 2 k k=1

9 St Petersburg paradox E (Gain) = 2 k 1 2 k k=1 =

10 Linearity of expectation For any constants a and b and any functions g 1 and g 2 E (a g 1 (X, Y ) + b g 2 (X, Y )) = a E (g 1 (X, Y )) + b E (g 2 (X, Y )) Follows from linearity of sums and integrals (ag 1 (x, y) + bg 2 (x, y))p X,Y (x, y) x R X x R Y = a g 1 (x, y) p X,Y (x, y) + b g 2 (x, y) p X,Y (x, y) x R Y x R Y x R X x R X

11 Example: Coffee beans Company buys coffee beans from two local producers Beans from Colombia: C tons/year Beans from Vietnam: V tons/year Model: C uniform between 0 and 1 V uniform between 0 and 2 C and V independent What is the expected total amount of beans B?

12 Example: Coffee beans E (C + V )

13 Example: Coffee beans E (C + V ) = E (C) + E (V )

14 Example: Coffee beans E (C + V ) = E (C) + E (V ) = = 1.5 tons

15 Example: Coffee beans E (C + V ) = E (C) + E (V ) = = 1.5 tons Holds even if C and V are not independent

16 Independence If X, Y are independent then E (g (X ) h (Y )) = E (g (X )) E (h (Y ))

17 Independence E (g (X ) h (Y )) = x= y= g (x) h (y) f X,Y (x, y) dx dy

18 Independence E (g (X ) h (Y )) = = x= y= x= y= g (x) h (y) f X,Y (x, y) dx dy g (x) h (y) f X (x) f Y (y) dx dy

19 Independence E (g (X ) h (Y )) = = x= y= x= y= = E (g (X )) E (h (Y )) g (x) h (y) f X,Y (x, y) dx dy g (x) h (y) f X (x) f Y (y) dx dy

20 Expectation operator Mean and variance Covariance Conditional expectation

21 Mean The mean or first moment of X is E (X ) It s the center of mass of the distribution

22 Bernoulli E (X ) = 0 p X (0) + 1 p X (1) = p

23 Binomial A binomial is a sum of n Bernoulli random variables X = n i=1 B i

24 Binomial A binomial is a sum of n Bernoulli random variables X = n i=1 B i ( n ) E (X ) = E B i i=1

25 Binomial A binomial is a sum of n Bernoulli random variables X = n i=1 B i ( n ) E (X ) = E B i = i=1 n E (B i ) i=1

26 Binomial A binomial is a sum of n Bernoulli random variables X = n i=1 B i ( n ) E (X ) = E B i = i=1 n E (B i ) i=1 = np

27 Mean of important random variables Random variable Parameters Mean Bernoulli p p Geometric p 1 p Binomial n, p np Poisson λ λ Uniform a, b a+b 2 Exponential λ 1 λ Gaussian µ, σ µ

28 Cauchy random variable 0.3 fx (x) x f X (x) = 1 π(1 + x 2 ).

29 Cauchy random variable E(X ) = = 0 x π(1 + x 2 ) dx x π(1 + x 2 ) dx 0 x π(1 + x 2 ) dx

30 Cauchy random variable E(X ) = = 0 x π(1 + x 2 ) dx x π(1 + x 2 ) dx 0 x π(1 + x 2 ) dx 0 x π(1 + x 2 ) dx = 0 1 2π(1 + t) dt = lim t log(1 + t) 2π

31 Cauchy random variable E(X ) = = 0 x π(1 + x 2 ) dx x π(1 + x 2 ) dx 0 x π(1 + x 2 ) dx 0 x π(1 + x 2 ) dx = 1 0 2π(1 + t) dt log(1 + t) = lim t 2π =

32 Mean of a random vector Vector formed by the means of its components E (X 1 ) ( ) E X := E (X 2 ) E (X n ) By linearity of expectation, for any matrix A R m n and b R m ( E AX + ) ( ) b = A E X + b

33 The mean as a typical value The mean is a typical value of the random variable The probability that X equals E (X ) can be zero The mean can be severely distorted by a subset of extreme values

34 Density with subset of extreme values 0.1 fx (x) x Uniform random variable X with support [ 4.5, 4.5] [99.5, 100.5]

35 Density with subset of extreme values E (X ) = x f X (x) dx + x f X (x) dx x= 4.5 x=99.5 = = 10

36 Density with subset of extreme values 0.1 fx (x) x

37 Median Midpoint of the distribution: number m such that P (X m) 1 2 and P (X m) 1 2 For continuous random variables F X (m) = m f X (x) dx = 1 2

38 Density with subset of extreme values F X (m) = m 4.5 = m f X (x) dx

39 Density with subset of extreme values F X (m) = m 4.5 = m = 1 2 f X (x) dx m = 0.5

40 Density with subset of extreme values 0.1 Mean Median fx (x) x

41 Variance The mean square or second moment of X is E ( X 2) The variance of X is Var (X ) := E ((X E (X )) 2) = E ( X 2 2X E (X ) + E 2 (X ) ) = E ( X 2) E 2 (X ) The standard deviation of X is σ X := Var (X )

42 Bernoulli E ( X 2) = 0 p X (0) + 1 p X (1) = p Var (X ) = E ( X 2) E 2 (X ) = p p 2 = p (1 p)

43 Variance of common random variables Random variable Parameters Variance Bernoulli p p (1 p) Geometric p 1 p p 2 Binomial n, p np (1 p) Poisson λ λ Uniform a, b (b a) 2 12 Exponential λ 1 λ 2 Gaussian µ, σ σ 2

44 Geometric (p = 0.2) px (k) k

45 Binomial (n = 20, p = 0.5) k

46 Poisson (λ = 25) k

47 Uniform [0, 1] fx (x) x

48 Exponential (λ = 1) x

49 Gaussian (µ = 0, σ = 1) x

50 Variance The variance operator is not linear, but Var (a X + b) = E ((a X + b E (a X + b)) 2) = E ((a X + b ae (X ) b) 2) = a 2 E ((X E (X )) 2) = a 2 Var (X )

51 Bounding probabilities using expectations Aim: Characterize behavior of X to some extent using E (X ) and Var (X )

52 Markov s inequality For any nonnegative random variable X and any a > 0 P (X a) E (X ) a

53 Markov s inequality Consider the indicator variable 1 X a X a 1 X a 0

54 Markov s inequality Consider the indicator variable 1 X a X a 1 X a 0 E (X ) a E (1 X a )

55 Markov s inequality Consider the indicator variable 1 X a X a 1 X a 0 E (X ) a E (1 X a ) = a P (X a)

56 Age of students at NYU Mean: 20 years How many are younger than 30?

57 Age of students at NYU Mean: 20 years How many are younger than 30? P(A 30) E (A) 30

58 Age of students at NYU Mean: 20 years How many are younger than 30? At least 1/3 P(A 30) E (A) 30 = 2 3

59 Chebyshev s inequality For any positive constant a > 0, P ( X E (X ) a) Var (X ) a 2

60 Chebyshev s inequality For any positive constant a > 0, P ( X E (X ) a) Var (X ) a 2 Corollary: If Var (X ) = 0 then P (X E (X )) = 0

61 Chebyshev s inequality For any positive constant a > 0, P ( X E (X ) a) Var (X ) a 2 Corollary: If Var (X ) = 0 then P (X E (X )) = 0 For any ɛ > 0 P ( X E (X ) ɛ) Var (X ) ɛ 2 = 0

62 Chebyshev s inequality Define Y := (X E (X )) 2 By Markov s inequality P ( X E (X ) a) = P ( Y a 2)

63 Chebyshev s inequality Define Y := (X E (X )) 2 By Markov s inequality P ( X E (X ) a) = P ( Y a 2) E (Y ) a 2

64 Chebyshev s inequality Define Y := (X E (X )) 2 By Markov s inequality P ( X E (X ) a) = P ( Y a 2) E (Y ) = a 2 Var (X ) a 2

65 Age of students at NYU Mean: 20 years, standard deviation: 3 years How many are younger than 30?

66 Age of students at NYU Mean: 20 years, standard deviation: 3 years How many are younger than 30? P(A 30) P( A 20 10)

67 Age of students at NYU Mean: 20 years, standard deviation: 3 years How many are younger than 30? At least 91 % P(A 30) P( A 20 10) Var (A) 100 = 9 100

68 Expectation operator Mean and variance Covariance Conditional expectation

69 Covariance The covariance of X and Y is Cov (X, Y ) := E ((X E (X )) (Y E (Y ))) = E (XY Y E (X ) X E (Y ) + E (X ) E (Y )) = E (XY ) E (X ) E (Y ) If Cov (X, Y ) = 0, X and Y are uncorrelated

70 Covariance Cov (X, Y ) Cov (X, Y )

71 Variance of the sum Var (X + Y ) = E ((X + Y E (X + Y )) 2) ( = E (X E (X )) 2) + E ((Y E (Y )) 2) + 2E ((X E (X )) (Y E (Y ))) = Var (X ) + Var (Y ) + 2 Cov (X, Y )

72 Variance of the sum Var (X + Y ) = E ((X + Y E (X + Y )) 2) ( = E (X E (X )) 2) + E ((Y E (Y )) 2) + 2E ((X E (X )) (Y E (Y ))) = Var (X ) + Var (Y ) + 2 Cov (X, Y ) If X and Y are uncorrelated, then Var (X + Y ) = Var (X ) + Var (Y )

73 Independence implies uncorrelation Cov (X, Y ) = E (XY ) E (X ) E (Y ) = E (X ) E (Y ) E (X ) E (Y ) = 0

74 Uncorrelation does not imply independence X, Y are independent Bernoulli with parameter 1 2 Let U = X + Y and V = X Y Are U and V independent? Are they uncorrelated?

75 Uncorrelation does not imply independence p U (0) p V (0) p U,V (0, 0)

76 Uncorrelation does not imply independence p U (0) = P (X = 0, Y = 0) = 1 4 p V (0) p U,V (0, 0)

77 Uncorrelation does not imply independence p U (0) = P (X = 0, Y = 0) = 1 4 p V (0) = P (X = 1, Y = 1) + P (X = 0, Y = 0) = 1 2 p U,V (0, 0)

78 Uncorrelation does not imply independence p U (0) = P (X = 0, Y = 0) = 1 4 p V (0) = P (X = 1, Y = 1) + P (X = 0, Y = 0) = 1 2 p U,V (0, 0) = P (X = 0, Y = 0) = 1 4

79 Uncorrelation does not imply independence p U (0) = P (X = 0, Y = 0) = 1 4 p V (0) = P (X = 1, Y = 1) + P (X = 0, Y = 0) = 1 2 p U,V (0, 0) = P (X = 0, Y = 0) = 1 4 p U (0) p V (0) = 1 8

80 Uncorrelation does not imply independence Cov (U, V ) = E (UV ) E (U) E (V ) = E ((X + Y ) (X Y )) E (X + Y ) E (X Y ) = E ( X 2) E ( Y 2) E 2 (X ) + E 2 (Y )

81 Uncorrelation does not imply independence Cov (U, V ) = E (UV ) E (U) E (V ) = E ((X + Y ) (X Y )) E (X + Y ) E (X Y ) = E ( X 2) E ( Y 2) E 2 (X ) + E 2 (Y ) = 0

82 Correlation coefficient Pearson correlation coefficient of X and Y ρ X,Y := Cov (X, Y ) σ X σ Y. Covariance between X /σ X and Y /σ Y

83 Correlation coefficient σ Y = 1, Cov (X, Y ) = 0.9, ρ X,Y = 0.9 σ Y = 3, Cov (X, Y ) = 0.9, ρ X,Y = 0.3 σ Y = 3, Cov (X, Y ) = 2.7, ρ X,Y = 0.9

84 Cauchy-Schwarz inequality For any X and Y E (XY ) E (X 2 ) E (Y 2 ). and E (XY ) = E (X 2 ) E (Y 2 E (Y ) Y = 2 ) E (X 2 ) X E (XY ) = E (X 2 ) E (Y 2 E (Y ) Y = 2 ) E (X 2 ) X

85 Cauchy-Schwarz inequality We have Cov (X, Y ) σ X σ Y and equivalently ρ X,Y 1 In addition ρ X,Y = 1 Y = c X + d where c := { σy σ X if ρ X,Y = 1, σ Y σ X if ρ X,Y = 1, d := E (Y ) ce (X )

86 Covariance matrix of a random vector The covariance matrix of X is defined as Var (X 1 ) Cov (X 1, X 2 ) Cov (X 1, X n ) Cov (X 2, X 1 ) Var (X 2 ) Cov (X 2, X n ) Σ X = Cov (X n, X 2 ) Cov (X n, X 2 ) Var (X n ) ( = E X X ) ( ) ( ) T T E X E X

87 Covariance matrix after a linear transformation Σ A X + b

88 Covariance matrix after a linear transformation ( ( Σ AX + b = E AX + ) ( b AX + ) ) T ( b E AX + ) ( b E AX + ) T b

89 Covariance matrix after a linear transformation ( ( Σ AX + b = E AX + ) ( b AX + ) ) T ( b E AX + ) ( b E AX + ) T b ( = A E X X ) T A T + ( ) T ( ) b E X A T + A E X b T + b b T ( ) ( ) T ( ) A E X E X A T A E X b T ( ) T b E X A T b b T

90 Covariance matrix after a linear transformation ( ( Σ AX + b = E AX + ) ( b AX + ) ) T ( b E AX + ) ( b E AX + ) T b ( = A E X X ) T A T + ( ) T ( ) b E X A T + A E X b T + b b T ( ) ( ) T X X A T b b T = A A E ( E ( X X T ) E ( ) T ( ) E X A T A E X b T b E ( ) X E ( X ) T ) A T

91 Covariance matrix after a linear transformation ( ( Σ AX + b = E AX + ) ( b AX + ) ) T ( b E AX + ) ( b E AX + ) T b ( = A E X X ) T A T + ( ) T ( ) b E X A T + A E X b T + b b T ( ) ( ) T X X A T b b T = A A E ( E ( X X T ) E = AΣ X A T ( ) T ( ) E X A T A E X b T b E ( ) X E ( X ) T ) A T

92 Variance in a fixed direction For any unit vector u ) Var ( u T X = u T Σ X u

93 Direction of maximum variance To find direction of maximum variance we must solve arg max u 2 =1 ut Σ X u

94 Linear algebra Symmetric matrices have orthogonal eigenvectors Σ X = UΛU T λ = [ ] u 1 u 2 u n 0 λ 2 0 [ u1 u 2 ] T u n 0 0 λ n

95 Linear algebra λ 1 = max u 2 =1 ut Au u 1 = arg max u 2 =1 ut Au λ k = max u 2 =1,u u 1,...,u k 1 u T Au u k = arg max u T Au u 2 =1,u u 1,...,u k 1

96 Direction of maximum variance λ1 = 1.22, λ2 = 0.71 λ1 = 1, λ 2 = 1 λ1 = 1.38, λ2 = 0.32

97 Coloring Goal: Transform uncorrelated samples with unit variance so that they have a prescribed covariance matrix Σ 1. Compute the eigendecomposition Σ = UΛU T. 2. Set y := U Λ x where λ1 0 0 Λ := 0 λ λn

98 Coloring Σ Y

99 Coloring Σ Y = U ΛΣ X Λ T U T

100 Coloring Σ Y = U ΛΣ X Λ T U T = U ΛI Λ T U T

101 Coloring Σ Y = U ΛΣ X Λ T U T = U ΛI Λ T U T = Σ

102 Coloring X Λ X U Λ X

103 Generating Gaussian random vectors Goal: Sampling from an n-dimensional Gaussian random vector with mean µ and covariance matrix Σ 1. Generate n independent standard Gaussian samples x 2. Compute the eigendecomposition Σ = UΛU T 3. Set y := U Λ x + µ For non-gaussian random vectors, coloring does not necessarily preserve the distribution

104 For Gaussian rvs uncorrelation implies mutual independence Uncorrelation implies σ σ2 2 0 Σ X = σn 2 which in turn implies 1 f X ( x) = ( (2π) n Σ exp 1 ) 2 ( x µ)t Σ 1 ( x µ)

105 For Gaussian rvs uncorrelation implies mutual independence Uncorrelation implies σ σ2 2 0 Σ X = σn 2 which in turn implies 1 f X ( x) = ( (2π) n Σ exp 1 ) 2 ( x µ)t Σ 1 ( x µ) = n i=1 ( ) 1 exp (x i µ i ) 2 (2π)σi 2σi 2

106 For Gaussian rvs uncorrelation implies mutual independence Uncorrelation implies σ σ2 2 0 Σ X = σn 2 which in turn implies 1 f X ( x) = ( (2π) n Σ exp 1 ) 2 ( x µ)t Σ 1 ( x µ) = = n i=1 ( ) 1 exp (x i µ i ) 2 (2π)σi 2σi 2 n f Xi (x i ) i=1

107 Expectation operator Mean and variance Covariance Conditional expectation

108 Conditional expectation Expectation of g (X, Y ) given X = x? E (g (X, Y ) X = x) = Can be interpreted as a function y= h (x) := E (g (X, Y ) X = x) g(x, y) f Y X (y x) dy, The conditional expectation of g (X, Y ) given X is It s a random variable E (g (X, Y ) X ) := h (X )

109 Iterated expectation For any X and Y and any function g : R 2 R E (g (X, Y )) = E (E (g (X, Y ) X ))

110 Iterated expectation h (x) := E (g (X, Y ) X = x) = y= g (x, y) f Y X (y x) dy

111 Iterated expectation h (x) := E (g (X, Y ) X = x) = y= g (x, y) f Y X (y x) dy E (E (g (X, Y ) X )) = E (h (X ))

112 Iterated expectation h (x) := E (g (X, Y ) X = x) = y= g (x, y) f Y X (y x) dy E (E (g (X, Y ) X )) = E (h (X )) = x= h (x) f X (x) dx

113 Iterated expectation h (x) := E (g (X, Y ) X = x) = y= g (x, y) f Y X (y x) dy E (E (g (X, Y ) X )) = E (h (X )) = = x= x= h (x) f X (x) dx y= f X (x) f Y X (y x) g (x, y) dy dx

114 Iterated expectation h (x) := E (g (X, Y ) X = x) = y= g (x, y) f Y X (y x) dy E (E (g (X, Y ) X )) = E (h (X )) = = x= x= h (x) f X (x) dx y= = E (g (X, Y )) f X (x) f Y X (y x) g (x, y) dy dx

115 Example: Desert Car traveling through the desert Time until the car breaks down: T State of the motor: M State of the road: R Model: M uniform between 0 (no problem) and 1 (very bad) R uniform between 0 (no problem) and 1 (very bad) M and R independent T exponential with parameter M + R

116 Example: Desert E (T ) = E (E (T M, R))

117 Example: Desert E (T ) = E (E (T M, R)) ( ) 1 = E M + R

118 Example: Desert E (T ) = E (E (T M, R)) ( ) 1 = E M + R = dm dr m + r

119 Example: Desert E (T ) = E (E (T M, R)) ( ) 1 = E M + R = = dm dr m + r log (r + 1) log (r) dr

120 Example: Desert E (T ) = E (E (T M, R)) ( ) 1 = E M + R = = dm dr m + r log (r + 1) log (r) dr = log 4 = 1.39

121 Grizzlies in Yellowstone Model for the weight of grizzly bears in Yellowstone: Males: Gaussian with µ := 240 kg and σ := 40kg Females: Gaussian with µ := 140 kg and σ := 20kg There are about the same number of females and males

122 Grizzlies in Yellowstone E (W ) = E (E (W S))

123 Grizzlies in Yellowstone E (W ) = E (E (W S)) = E (W S = 0) + E (W S = 1) 2

124 Grizzlies in Yellowstone E (W ) = E (E (W S)) E (W S = 0) + E (W S = 1) = 2 = 180 kg

125 Bayesian coin flip Bayesian methods often endow parameters of discrete distributions with a continuous marginal distribution You suspect a coin is biased You are uncertain about the bias so you model it as a random variable with pdf f B (b) = 2t for t [0, 1] What is the expected value of the coin flip X?

126 Bayesian coin flip E (X ) = E (E (X B))

127 Bayesian coin flip E (X ) = E (E (X B)) = E (B)

128 Bayesian coin flip E (X ) = E (E (X B)) = E (B) = 1 0 2b 2 db

129 Bayesian coin flip E (X ) = E (E (X B)) = E (B) = 1 0 = 2 3 2b 2 db