2. Variance and Covariance: We will now derive some classic properties of variance and covariance. Assume real-valued random variables X and Y.

Transcription

1 CS450 Final Review Problems Fall 08 Solutions or worked answers provided Problems -6 are based on the midterm review Identical problems are marked recap] Please consult previous recitations and textbook problems for additional practice Independence: Suppose X, Y are independent (a) recap] What is P(X = x Y = y)? Solution: P(X = x) P(Y = y) (b) recap] What is P(X = x Y = y)? Solution: P(X x) P(Y y) (c) recap] Prove that P(X = x) + P(Y = y) P(X = x Y = y) = P(X x Y y) How else could we describe this quantity? Solution: P(X = x) + P(Y = y) P(X = x Y = y) = P(X = x Y = y) = P(X x Y y) This is the probability of X = x or Y = y (d) Now suppose X and Y are both uniformly distributed (independently) on {0, }, and take Z = X Y, where denotes exclusive or (in other words Z = 0 if X = Y, and Z = otherwise) Is each pair of X, Y, Z independent? Are X, Y, and Z mutually independent? Solution: Each pair is independent, which can easily be verified from the definition: P(X = x Z = y) = = P(X = x), and the same can be said for Y and Z The three however are not mutually independent; if they were, P(X = Y = Z = ) would be, but it is in fact 0, as = 0 8 Variance and Covariance: We will now derive some classic properties of variance and covariance Assume real-valued random variables X and Y (a) recap] If X and Y are independent, is it always the case that Cov(X, Y ) = 0 (under the independence assumption)? Either prove the statement or demonstrate that it is false with a counterexample Hint: recall that EX Y ] = EX ] EY ] X, Y independent Solution: The statement is true Cov(X, Y ) = E(X EX ])(Y EY ])] Definition of Covariance = EX Y X EY ] Y EX ] + EX ] EY ]] Unfactoring = EX Y ] EX EY ]] EY EX ]] + EEX ] EY ]] Linearity of Independence = EX Y ] EX ] EY ] EX ] EY ] + EX ] EY ] Linearity of Independence = EX ] EY ] EX ] EY ] EX ] EY ] + EX ] EY ] X, Y Independent EX ] EY ] = 0 Cancellation (b) Prove the inequality VαX + βy ] = α VX ] + β VY ] + αβcov(x, Y ), for any α, β R (c) Prove the inequality Cov(X + Y, Z ) = Cov(X, Z ) + Cov(Y, Z ) (d) Derive an equation for Cov(X, Y ) in terms of EX ], EY ], EX Y ] Hint: This result is similar to the well-known VX ] = E X ] (EX ]) 3 Problems with Dice: Suppose I roll a fair 4 sided die and a fair 8 sided die (independently), each marked with consecutively increasing integers, starting from Let X, Y be the random variables associated with the number rolled on the 4-sided and 8-sided die, respectively (a) recap] What are the expectations and variances of X and Y? Solution: EX ] = = i= i = 5 and VX ] = 4 4 i= i E X ] = 5, 4 EY ] = 9 and VY ] = 4

2 (b) recap] What is the expectation and variance of X + Y? Solution: By linearity of expectation, additivity of variance under independence, EX + Y ] = 7, VX + Y ] = 6 4 (c) What are the expectation and variance of min(x, Y )? (d) recap] Suppose I flip a fair coin, if heads I roll the 4 sided die, and if tails, I roll the 8 sided die What are the expectation and variance of the roll value? Solution: Expectation is 7, variance is 7 4 (e) recap] What is the probability that the 8-sided die is at least 7 and the 4-sided die is even? Solution: These events are independent, P(Y 7 X even) = P(Y 7) P(X even) = 4 = 8 (f) recap] What is the probability that the 8-sided die is at least 7 or the 4-sided die is even? Solution: The two simplest ways to compute this are P(Y 7 4-sided even) = P(Y < 7) P(X odd) (de Morgan s law) or P(Y 7 X even) = P(Y 7) + P(X even) P(Y 7 X even) Either way, the solution is 5 8 (g) recap] Suppose I observe that Y 7, and the X is even What is the probability that X + Y = 9? Solution: 4 4 Conditional Exponentials and Conditional Expectations: Suppose X is exponentially distributed with rate parameter λ (a) recap] Suppose we observe that X > α for some α > 0 Given this observation, how is X distributed? { 0 : x α Solution: X X > α has density f X X >α (x) = λe λ(x α), which is an exponential random : x > α variable, right-shifted by α (b) recap] What is EX X > α] for some α > 0 Try to use the previous result to solve this problem Solution: Using linearity of expectation, the solution to the previous part, and the expectation of the exponential distribution, we obtain α + λ (c) recap] Solve 4b again, this time using only integration and calculus (Hint: use integration by parts) Do your answers agree? Which strategy do you prefer? Solution: This is essentially a derivation of the exponential distribution: see this page for details (d) recap] Suppose I flip unbiased coins until achieving the first heads Then for each coin I flipped, I draw an exponential random variable with rate parameter λ and sum them What is the expectation of their sum? Solution: By linearity of expectation, this is equivalent to asking for EX Y ], where X is the exponential and Y the geometric random variable As X and Y are independent, EX Y ] = EX ] EY ] = λ 5 Gaussians, Expectations and Variances: Suppose X and Y are jointly normally distributed with means and variances µ X, σ X and µ Y, σ Y, respectively, and let σ X,Y = Cov(X, Y ) Hint: Recall that VαX + βy ] = α VX ] + β VY ] + αβcov(x, Y ) (a) recap] What are the expectation and variance of X + Y? Bonus: How is this quantity distributed? Solution: By linearity of expectation, µ X +µ Y, and by the above variance summation formula, VX + Y ] = σx + σy + σ X,Y Sums of Gaussians are themselves Gaussian, thus this sum is Gaussian (with the aforementioned expectation and variance) (b) recap] What is the variance of αy? How is this quantity distributed? Solution: V αy ] = α σ Y Scaled Gaussians are themselves Gaussian, thus this quantity is Gaussian distributed (c) recap] What are the expectation and standard deviation of αx βy? Bonus: How is this quantity distributed? Solution: EαX βy ] = αµ X βµ Y, and the standard deviation is VαX βy ] = α σ X + β σ Y αβσ X,Y Again this quantity is Gaussian distributed

3 (d) recap] Suppose X, Y are independent What is the joint density f X,Y (x, y)? Simplify your solution within reason Solution: f X,Y (x, y) = f X (x) f Y (y) = πσ X σ Y e (x µ X ) σ X (y µ Y ) σ Y 6 Fun with Probability Density Functions: Suppose X is distributed with PDF f X (x) = x : x 0, ) : x, 3), Z 0 : otherwise with some normalization constant Z such that R f X (x) dx = f X (x) = (a) recap] What is the value of Z? Try to compute this quantity using geometry and using calculus Make sure your answers agree! Solution: Z = f X (x) dx = 5 (b) recap] What is the CDF of X? 0 : x < 0 Solution: CDF X (x) = x : x 0, ) Z x : x, 3) Z : x 3 (c) recap] Suppose Y = X, where is the floor operator that takes any real number onto the largest integer not greater than its argument What are the PMF and CMF of Y? Hint: Recall that for Y = g(x ), it holds that P(Y = y) = P(X {x g(x) = y}) : y = 0 Solution: PMF Y (y) = P(X x, x + )) = : y {, }, Z 0 : otherwise 0 : y < 0 CMF Y (y) = : y 0, ) Z 3 : y, ) Z : y (d) recap] Without appealing to computing their values, how do EX ] and EY ] compare? Defend your answer Solution: EY ] < EX ]; informally, the mass in Y is shifted to the left, rigorously, the quantile function (inverse cumulative distribution function) of X dominates that of Y, which yields the conclusion (e) recap] Bonus: Suppose I draw (sample) a random value X with density f X For any i > 0, if X i >, I discard it and restart the process for a new X i+ drawn with density f X with probability max(0, min( x, )), otherwise I let Z equal X i Prove that f Z (x) = f Z (x) P(X is discarded) + f X (x) P(X is kept X ) Solution: The statement is true, as in case, Z = X, which occurs with probability P(X is is kept X ), which contributes f X (x) P(X is is kept X ) to the PDF, and in case, Z is distributed with PDF f Z, which happens with probability P(X is discarded), contributing the f Z (x) P(X is discarded) term (f) recap] Bonus: Use the solution to part (e) to derive f Z ( ) Solution: f Z (x) = f Z (x) P(X is discarded) + f X (x) P(X is not discarded X ) f Z (x)( P(X is discarded)) = f X (x) P(X is kept X ) f Z (x) = f X (x) P(X is kept X ) P(X is discarded) x : x 0, ) f Z (x) = x : x, ) 0 : Otherwise See Above Algebra Algebra Substitution 3

4 7 Linear Regression: Suppose data points x R m and labels y R m, where each x i is distributed uniformly on, ], and each y i N (x i, + x i,, σ ) Define the risk of linear regression over a sample (x, y) of size m as ˆR m (x, y) = min w R,w 0 R m (x i w + w 0 y i ) (a) As a function of m, what is m E m i= (x i w y i ) ] for w =,? (b) As a function of m, what is E m minw0 R m i= (x i w + w 0 y i ) ] for w =,? Hint: This formula resembles the sample variance formula Can you relate w 0 to the sample mean? (c) What is E ˆRm (x, y)], for m {,, 3}? Does this trend continue as m increases? What about for higher-dimensional data? (d) Interpret the meaning of and derive the value of lim argmin m w R,w 0 R m i= (x i w + w 0 y i ) i= (e) Interpret the meaning of and derive the value of lim E ˆRm (x, y)] m (f) Prove or disprove the following statement: E ˆRm (x, y)] E min w R m ] (x i w y i ) i= What does this tell you about including the bias term w 0 in linear regression? (g) Bonus: Prove or disprove the following statement: ] m > : E ˆRm (x, y) ] E ˆRm (x, y) 8 Asymptotic Average Theorems: The Pareto distribution with parameter α has density ρ α (x) = α x for α+ x, and 0 otherwise Take X to be Pareto distributed with α =, Y to be Pareto distributed with α =, and Z to be Pareto distributed with α = 3 Hint: Use 8a and 8b and asymptotic theorems to solve 8c and 8d Pay attention to the assumptions made by each theorem! We recommend against evaluating these limits directly (a) Derive the expectation of X, Y, and Z Solution: EX ] =, EY ] =, EZ ] = 3 (b) Derive the variance of X, Y, and Z Solution: VX ] =, VY ] =, EZ ] = 3 4 m (c) For x drawn iid from X, Y, and Z (separately), what is lim m m i= x i? Prove your answer Solution: Each average is equal to the expectations derived in 8a This is a consequence of the law of large numbers (d) For x drawn iid from X, Y, and Z (separately), how is lim m m m i= x i distributed? Prove your answer Solution: The central limit theorem doesn t apply to the first two cases, but we may conclude that lim m m m i= Zi is Gaussian with mean 3 n and variance 3 4 4

5 9 Markov Chains: Consider a Markov chain with states {,, 3} and transition matrix T = You are not required to fully simplify your answers for this problem (a) Find the stationary distribution of the Markov chain (b) Find the probability of being in state 3 after 3 steps if the chain begins at state 0 (c) Find the probability distribution over states after 4 steps if the chain begins at a state chosen uniformly at random 0 Hypothesis Testing: Suppose that under the null hypothesis, a server receives requests such that the time X between them is exponentially distributed (independently) with rate parameter λ (a) Develop a -tailed test based on a single request X that falsely rejects the null hypothesis with probability δ if an observed request interval is too small (b) Develop a -tailed test based on a single request that falsely rejects the null hypothesis with probability δ if an observed request interval is too small or too large Select your tails so they have equal probability under the null hypothesis (c) How do the -tailed tests for large and small time intervals and the -tailed test differ? When would you apply each type of test? (d) Now suppose you wait for n requests, each occuring X i units of time after the previous Take min n i= X i to be your new test statistic Construct a -tailed test based on this statistic Hint: Recall that the rate of the minimum of n independent rate λ exponential random variables is itself exponential with rate nλ (e) Again wait for n requests, and take n n i= X i to be your new test statistic Using the central limit theorem, how is this statistic distributed (asymptotically) under the null hypothesis? Now develop a -tailed asymptotic test using this statistic (f) Suppose that under H 0, λ =, and under H, λ = 3 As a function of n and δ, what is the probability of rejecting the null hypothesis given H for your minimum and Gaussian sum tests? 5