Lecture 13: Covariance. Lisa Yan July 25, 2018

Transcription

1 Lecture 13: Covariance Lisa Yan July 25, 2018

2 Announcements Hooray midterm Grades (hopefully) by Monday Problem Set #3 Should be graded by Monday as well (instead of Friday) Quick note about Piazza 2

3 Goals for today As engineers, why are we doing all of this? Wrap-up of joint/independent RVs Covariance Definition, intuition, and properties Properties of independent RVs Correlation Bringing back the Poisson Paradigm 3

4 Where are we going with all of this? As engineers, we want to: 1. Model things that can be random. 2. Find average values of situations that let us make good decisions. So far we have learned: If possibilities are finite, we can count equally likely outcomes If we are dealing with a 1-D domain, we can use 1-D RVs with parameters But the real world is more complex: Multi-dimensional (e.g., 3-D world) We don t know parameters, much less what RV to use We perform multiple experiments to help us with engineering. (RVs) (expectation) 4

5 Where are we going with all of this? As engineers, we want to: 1. Model things that can be random. 2. Find average values of situations that let us make good decisions. We perform multiple experiments to help us with engineering. i.i.d.: We often want to aggregate the results of repeated experiments: Throwing darts or flipping coins Counts of successes or arrivals Clinical trials Or we just want to model more complex functions: Receiving a signal that had some added noise Receiving web server requests from different sources (RVs) (expectation) Averages/ Counts = sums Sums of things distributed similarly 5

6 Where are we going with all of this? As engineers, we want to: 1. Model things that can be random. 2. Find average values of situations that let us make good decisions. We perform multiple experiments to help us with engineering. Counts and averages are both sums of RVs. (RVs) (expectation) This week + last week: Defining joint PMF/PDF How to model sums of 2 RVs How to deal with uncertainty of model (coin flips only) Next week (week 6): Finding expectation of complex situations Defining sampling and using existing data Mathematical bounds w.r.t modeling the average Week 7: bringing it all together Sampling + average = Central Limit Theorem Samples + modeling = finding the best model parameters given data 6

7 Summary of last time Z = X + Y, where X and Y are independent RVs: $ % & = ( $ * (,)$. (&,) ) 0 % & = *, 0. &, 4, (discrete X and Y) (continuous X and Y) Convenient sums of independent RVs: Binomial: if same p of trial success, sum up trial counts n i Poisson: sum up rates " we proved that if X ~ Poi(lp) and Y ~ Poi(l(1-p)), X + Y ~ Poi(p). (Web Servers) Normal: sum up means, sum up variances On homework, you will find distribution of ax + by (linear comb. of normal) 7

8 Sum of independent uniform RVs Let X and Y be independent RVs, where X ~ Unif(0,1) and Y ~ Unif(0,1). What is the distribution of Z = X + Y? Solution: * " #$% & = ( " # &, " %, -, = ( " # &, " %, -, = ( )*.. Note: for different a, need bounds on y and a such that " # &, = 1 = ( / ( )/ 1 -, = & 0 & 1 1 -, = 2 & 1 & 2 0 otherwise / f X + Y (a) 1 1 / " # &, -, 2 a 8

9 Defects on a hard drive A single point defect is uniformly distributed over a disk of radius R. Are X and Y independent? Solution: ' " # = % ",) #, * +* = 1 -. &' / % +* 0: = & / % +* & 6 3 &2 3 " # = 2./ # / -. / If 0 #. If 0 #. ) * = 2./ y / -. / If 0 *., by symmetry 1 ",) #, * = 9-. / if # / + * /. / 0 otherwise " # > * ",) #, * No. X and Y are dependent 9

10 Defects on a hard drive A single point defect is uniformly distributed over a disk of radius R. Calculate the expected distance of the defect from the center, i.e., ", where D = % & + ( & ) Solution: = 7& 2 & : " = 8 ;(" > 7)?7 = = 8 9 : 1 7& 2 &?7 1 ) *,, -,. = / 12 & if - & +. & 2 & 0 otherwise : ( )?7 = 7 7A : 32 & CD9 =

11 Expectation as a summary For a single variable, we have a few summary values: Mean, " Variance (a measure of spread), Var " For two variables: In both distributions: " = $, Var " = Var $ Difference: how the two variables vary with each other. 11

12 Covariance The covariance of two variables X and Y is: Cov, # = % % # % # = % # % % # Proof of second part: Cov, # = % % # % # = % # % # % # + % % # = % # % % # % % # + % % % # = % # % # % % % # + % % # = % # % % # 12

13 Covariance of a die roll Roll our favorite 6-sided die. Define two indicator variables: X = 1 if roll in {1,2,3,4} Y = 1 if roll in {3,4,5,6} What is Cov(X,Y)? Solution: " = 2/3, ) = 2/3 Cov ", ) = ") " [)] ") = +, -./ 0,1 (-,.) = = 7 8 Cov ", ) = ") " ) = 7 8 ; < = 7 < What does negative covariance mean? Consider:? " = 1 = 2/3, whereas? " = 1 ) = 1 = 1/2 Observing Y = 1 (positive) makes X = 1 (positive) less likely 13

14 Covariance of humans Cov 0, 1 = [1] Weight Height W H = # = # = What is the covariance of weight (W) and height (H)? Solution: Cov, # = = What does positive covariance mean? Observing high W makes high H more likely 14

15 Covariance Q: Is the covariance positive, negative, or zero? A: positive 15

16 Properties of Covariance Let X and Y be arbitrary random variables Cov, # = % % # % # = % # % % # Symmetry: Cov, # = Cov #, Variance, redefined: Cov, = % % % = Var Non-linear: Cov ' + ), # = 'Cov, # Covariance and sums of X 1, X 2,, X n and Y 1, Y 2,, Y m : Cov + +,, #, = +, Cov +, #, (covariance of all pairs) 16

17 Variance of Sums of Variables Proof: % Var "#$ % & " = "#$ % Var & " + 2 "#$ % *#+,$ Cov & ", & * For 2 variables: Var & +. = Var & + Var. + 2Cov &,. Var % "#$ & " = Cov % "#$ & ", % "#$ & " (Cov &, & = Var & ) = % % "#$ *#$ Cov & ", & * = % "#$ Var & " + % % "#$ *#$,*0" = % "#$ Var & " + 2 % % "#$ *#+,$ Cov & ", & * Cov & ", & * (covariance of all pairs) (symmetry) 17

18 Break Attendance: tinyurl.com/cs109summer

19 Independent RVs Definition: Two random variables X and Y are independent if: ",$ %, & = " % $ & ( ",$ %, & = ( " % ( $ & ) ",$ %, & = ) " % ) $ & (discrete) (continuous) (any X and Y) By definition these are bidirectional Properties of independence (next 2 slides) Not bidirectional If X and Y are independent, then these properties hold. If these properties hold, X and Y are not necessarily independent. 19

20 Product of independent RVs If X and Y are independent, then: "# = " # % " h # = % " h # Proof: % " h # = () ) () ) % * h +,-,/ *, + 0* 0+ ) ) = () () % * h +,- *, / + 0* 0+ ) ) = () h +,/ + 0+ () % *,- * 0* ) ) = () % *,- * 0* () h +,/ + 0+ = % " h # 20

21 Independent RVs Definition: Two random variables X and Y are independent if: ) *,+,, - = ) *, ) + -. *,+,, - =. *,. + - / *,+,, - = / *, / + - (discrete) (continuous) (any X and Y) By definition these are bidirectional Properties of independence: "# = " # Cov ", # = 0 Var " + # = Var " + Var # + 2Cov ", # = Var " + Var # Not bidirectional 21

22 Covariance does not imply independence Let take on values 1,0,1 with equal probability (1/3). 1 if X=0 Define & = ( 0 otherwise Y X p Y (y) 0 1/3 0 1/3 2/ /3 0 1/3 1. What are )[] and )[&]? p X (x) 1/3 1/3 1/3 1 ) = 1, - + 0, - + 1, - = 0 ) & = 0 / - + 1, - =, - 2. What is the covariance of X and Y? & = 0 for all X and Y Cov, & = ) & ) ) & = 0 0, = 0-3. Are X and Y independent? 0 & = 0 = 1 = 0 2/3 = 0(& = 0) No Even though X and Y are clearly dependent 22

23 Statistics of functions of 2 RVs Sum: always can be separated, regardless of independence " + $ = " + $ Product: can only be separated if X, Y independent "$ = " $ Variance for any X and Y: Var " + $ = Var " + Var $ + 2Cov ", $ Variance for X, Y independent: Var " + $ = Var " + Var $ If all X 1, X 2,, X n are independent:, Var ( " ) = ( Var " ) )*+, )*+ 23

24 Variance of Binomial For a binomial RV X~Bin(n,p), " # = %& (expectation of sum = sum of expectation) Proof of variance: Var # = %&(1 &) Define: # + as an indicator variable for success on trial,. - # + = 1 = & 1 Var # = Var. +/0 # + 1 =. +/0 Variance of sum of independent RVs = sum of variance 1 Var # + =. +/0 &(1 &) Variance of Bernoulli = %&(1 &) 24

25 Correlation The correlation of two random variables X and Y is Cov ", $ ", $ = Var " Var $ Note: 1 ", $ 1 Measures the linearity of the relationship between X and Y: If $ = )" + +, then Cov ", $ = Cov ", )" + + = )Cov ", " = )Var " = sgn()) ) 1 (Var " ) 1 = sgn()) () 1 Var " )Var " = sgn()) Var $ Var " ", $ = sgn()) (sgn()): sign of )) 25

26 Correlation quantifies linearity ", $ = Cov ", $ Var " Var $ ", $ = 1 $ = )" + + ) =, - /, / ", $ = 0 ", $ = 1 uncorrelated $ = )" + + ) =, - /, / Var $ = Var )" + + =, -, / 0 Var " =, - 0 $ = " 0 No linear relationship == Cov(X,Y) = 0, but X and Y could be nonlinearly related 26

27 Spurious Correlations ", $ is used a lot in statistics to quantify the relationship b/t X and Y. Correlation: Correlation does not imply causation 27

28 Do indicator variables correlate? Define indicator variables " and # for events A and B. 1 if + occurs " = 2 0 otherwise What is Cov ", #? Solution: %[ " ] = )(+), %[ # ] = )(-), % " # = ) +- = ) + - )(-) Cov ", # = % " # % " % # = ) + - ) - ) + )(-) à Sign of Cov ", # determined by ) + - ) + = ) - ) + - ) + 1 if - occurs # = 2 0 otherwise (in terms of P(A), P(B), P(A B) ) + - > ) + Þ / ", # > 0 / 0, 1 = ) + - = ) + Þ / ", # = 0 (independence, Cov( ", # ) = 0 ) + - < ) + Þ / ", # < 0 Cov 0, 1 Var 0 Var 1 28

29 Multinomial Random Variable Consider n independent trials of an experiment. Each trial results in one of m outcomes, with respective probabilities ( ", ( ),, ( % where % 23" ( 2 = 1 2 is number of trials with outcome k for k = 1,..., m. ",, % Mul& ', ( ", ( ),, ( % * " =, ", ) =, ),, % =, % = where % 23", 2 = ' and ', ",, ),,, % ( " -. ( ) - / ( % - 0 ', ",, ),,, % is a multinomial coefficient. We expect that the 2 are negatively correlated. 29

30 Covariance of the multinomial Consider n independent trials of an experiment. Each trial results in one of m outcomes, w.p. ", $,, & where & ()" ( = 1, ( is number of trials with outcome k for k = 1,..., m. What is Cov, -,, (, where. 0? Say, rolls of 3 and rolls of 5 Solution: Define:Indicator 1 ( 2 = 1 if trial 2 has outcome 0, 0 otherwise 3 1 ( 2 = (,, ( = 5 4)" 1 ( = -,, - = 5 4)" 1-2 Cov, -,, ( = 5 6)" 5 =)" Cov 1-7, 1 ( < = 5 6)=)" Cov 1-7, 1 ( < = 5 6)" ( ( 7 = 5 6)" ( 7 = 5 6)" ( - ( ) = ; - ( sum of pairwise covariances a b: trial a and b independent, Cov 1-7, 1 ( < = 0 trial a cannot have outcome i and j, ( 7 =0, -,, ( are negatively correlated. 30

31 The Poisson Paradigm Multinomial distributions: Count of strings hashed to buckets in hash table Number of server requests across machines in cluster Distribution of words/tokens in an Etc. When (# outcomes) is large, " # is small, and for equally likely outcomes, where " # = % & : Cov ' (, ' # = +" ( " # = +, à 0 as m gets large For large, ' (, ' # are very mildly negatively correlated, and the Poisson paradigm is applicable. 31

32 Summary of this time Cov, # = % % # % # = % # % % # Covariance is a linear relationship. Independence: Implies uncorrelated (but uncorrelated does NOT imply independence) For n independent variables: Var + ()* ( = + ()* Var ( Correlation is a normalized version of Covariance: Cov, #,, # = Var Var # 32