Probability Theory Refresher

Transcription

1 Machine Learning WS24 Module IN264 Sheet 2 Page Machine Learning Worksheet 2 Probabilit Theor Refresher Basic Probabilit Problem : A secret government agenc has developed a scanner which determines whether a person is a terrorist. The scanner is fairl reliable; 95% of all scanned terrorists are identified as terrorists, and 95% of all upstanding citizens are identified as such. An informant tells the agenc that eactl one passenger of aboard an aeroplane in which ou are seated is a terrorist. The agenc decide to scan each passenger and the shift looking man sitting net to ou is tested as TERRORIST. What are the chances that this man is a terrorist? Show our work! The random variable T indicates, if a person is a terrorist or not, i.e. with the given information above: p(t = ) =. and p(t = ) =.99. The random variable S indicates, if the scanner tests TERRORIST or not. So, p(s = T = ) =.95 (given in the tet) and therefore (laws of probabilit) p(s = T = ) =.5. Similaril, p(s = T = ) =.95 and p(s = T = ) =.5. We are interested in p(t= S=): p(t = S = ) = p(s = T = )p(t = ) p(s = T = )p(t = ) + p(s = T = )p(t = ) = Note that in the denominator, we compute p(s = ) using the law of total probabilit (also known as sum rule). Problem 2: Two balls are placed in a bo as follows: A fair coin is tossed and a white ball is placed in the bo if a head occurs, otherwise a red ball is placed in the bo. The coin is tossed again and a red ball is placed in the bo if a tail occurs, otherwise a white ball is placed in the bo. Balls are drawn from the bo three times in succession (alwas with replacing the drawn ball back in the bo). It is found that on all three occasions a red ball is drawn. What is the probabilit that both balls in the bo are red? Show our work! Denote b RRR the event that 3 red balls are drawn. Similaril, denote b rr the event that 2 red balls are placed in the bo, rw the event that first a white and then a red ball are placed in the bo, and wr and ww for the remaining two possibilities. We know that Furthermore Therefore p(rw) = p(rw) = p(wr) = p(ww) = 4 p(rrr rr) =, p(rrr rw) = p(rrr wr) = /8, p(rrr ww) = p(rr RRR) = p(rrr rr)p(rr) p(rrr) = /4 5/6 = 4 5 where p(rrr) = {rr,rw,wr,ww} p(rrr )p() is computed with the sum rule. Submit to homework@class.brml.org with subject line homework sheet 2 b 24//2, :59 CET

2 Machine Learning WS24 Module IN264 Sheet 2 Page 2 Problem 3: There are eleven urns labeled b u {,, 2,..., }, each containing ten balls. Urn u contains u black balls and u white balls. Alice selects an urn u at random and draws N times with replacement from that urn, obtaining n B black balls and N n B white balls. If after N = draws n B = 3 black balls have been drawn, what is the probabilit that the urn Alice is using is urn u? Problem 4: Now, let Alice draw another ball from the same urn. What is the probabilit that the net drawn ball is black (show our work)? Our goal is to compute P (u n B, N). Given are P (n B u, N) and P (u): Use the above rules: Assuming that P (N u) = P (N): Thus: P (n B u, N) = ( N )b n B n u ( b u ) N n B B P (u, n B, N) = P (u n B, N)P (n B, N) P (u, n B, N) = P (n B u, N)P (u, N) = P (n B u, N)P (N)P (u) P (u n B, N) = P (n B u, N)P (u)p (N) P (n B, N) Appling again the sum rule for the denominator: finall resulting in: P (n B, N) = P (n B u, N)P (u)p (N) u P (u n B, N) = We derive the following conditional distribution: P (n B u, N)P (u) u P (n B u, N)P (u) u P (u n B = 3, N = ) Now, let Alice draw another ball from the same urn. What is the probabilit that the net drawn ball is black? Submit to homework@class.brml.org with subject line homework sheet 2 b 24//2, :59 CET

3 Machine Learning WS24 Module IN264 Sheet 2 Page 3 Let B N+ denote net ball is black. Thus, using the sum rule: P (B N+ n B, N) = P (B N+ u, n B, N)P (u n B, N) u For a fied u, balls are drawn with replacement, therefore P (B N+ u, n B, N) = b u, thus P (B N+ n B, N) = b u P (u n B, N).333 u Problem 5: Calculate the mean and the variance of the uniform random variable X with PDF p() =, [, ], and elsewhere. Let X Uniform(, ), i.e. and E[X] = E[X 2 ] = + + d = 2 d = and thus V ar[x] = E[X 2 ] E[X] 2 = /3 /4 = /2. d = [ 2 2 ] =.5 2 d = [ 3 3 ] = 3 Problem 6: results: Consider two random variables X and Y with joint densit p(, ). Prove the following two E[X] = E Y [E X Y [X]] () V ar[x] = E Y [V ar X Y [X]] + V ar Y [E X Y [X]] (2) Here E X Y [X] denotes the epectation of X under the conditional densit p( ), with a similar notation for the conditional variance. Consider the discrete case onl, using the sum rule and term reordering as the onl necessar strategies: ( p( )) p() = p( )p() = p(, ) = p() = E[X] Using this result and carefull deal with the meanings of the various smbols, we can derive also the second Submit to homework@class.brml.org with subject line homework sheet 2 b 24//2, :59 CET

4 Machine Learning WS24 Module IN264 Sheet 2 Page 4 equation: E Y [V ar X Y [X]] = = = (( E X Y [X]) 2 p( )) p() 2 p(, ) 2 ( 2 p() 2 E 2 X Y [X]p() + = E X [X 2 ] E Y [E 2 X Y [X]] V ar Y [E X Y [X]] = (E X Y [X] E Y [E X Y [X]]) 2 p() = p( )) E X Y [X]p() + E 2 X Y [X]p() E 2 X Y [X]p() 2E X[X] E X Y [X]p() + EX[X] 2 = E Y [E 2 X Y [X]] 2E2 X[X] + E 2 X[X] = E Y [E 2 X Y [X]] E2 X[X] E 2 X Y [X]p(, ) Thus we get: E Y [V ar X Y [X]] + V ar Y [E X Y [X]] = E X [X 2 ] E 2 X[X] = var[x] There is an alternative, shorter wa: E Y [V ar X Y [X]] + V ar Y [E X Y [X]] = E Y [E X Y [X 2 ]] E Y [E X Y [X 2 ]] + E Y [E X Y [X 2 ]] E 2 Y [E X Y [X]] = E Y [E X Y [X 2 ]] E 2 X [X] Consider these two results with setting X as a parameter θ that we want to learn and Y as a random variable representing a possible observed data set D. Thus E θ [θ] = E D [E θ [θ D]] sas that the posterior mean of θ (this is the right hand side of the above equation), averaged over the distribution generating the data (out of which one particular realization of D is choosen) is equal to the prior mean of θ. This is wired! Here, one mies baesian analsis (posteriors) with frequentist view points (epectation over all possible data sets) and thus gets a rather strange result. With var[θ] = E D [var D [θ D]] + var D [E θ [θ D]] one gets an interesting statement regarding the epected posterior variance of θ (this is the first term on the right hand side). It is alwas smaller than the original prior variance (left hand side)! (but onl in average, as alread said), because the variance of the posterior mean (second term on the right side) is a positive quanitit. 2 Probabilit Inequalities Inequalities are useful for bounding quantities that might otherwise be hard to compute. We ll begin with a simple inequalit, called the Markov inequalit after Andrei A. Markov, a student of Pafnut Submit to homework@class.brml.org with subject line homework sheet 2 b 24//2, :59 CET

5 Machine Learning WS24 Module IN264 Sheet 2 Page 5 Chebshev. 2. Markov Inequalit Let X be a non-negative, discrete random variable, and let c > be a positive constant. Problem 7: Show that P (X > c) E[X]. c Now, consider flipping a fair coin n times. Using the Markov Inequalit, what is the probabilit of getting more than (3/4)n heads? E[X] = p() = c we get an upper bound of 2/3. p() + >c p() >c p() c p() = cp (X > c) >c 2.2 Chebshev Inequalit Appl the Markov Inequalit to the deviation of a random variable from its mean, i.e. for a general random variable X we wish to bound the probabilit of the event { X E[X] > a}, which is the same as the event {(X E[X]) 2 > a 2 }. Problem 8: Prove that P ( X E[X] > a) V ar(x) a 2 holds. Again, consider flipping a fair coin n times. Now use the Chebshev Inequalit to bound the probabilit of getting more than (3/4)n heads. P ( X E[X] > a) = P ((X E[X]) 2 > a 2 ) V ar(x) a 2 Then start from the statement and form it into something similar to the above: P (X > 3/4n) = P (X /2n > /4n) P ( X E[X] /4n) n/4 n/6 = 4 n Note that we are considering here X as the sum of independent coin flips and thus using the rule: variance of the sum of independent variables is the sum of the variances. Interpretation of the result: with more tosses, the epected result converges to the mean value, so we have a convergence to zero in this case. Submit to homework@class.brml.org with subject line homework sheet 2 b 24//2, :59 CET

6 Machine Learning WS24 Module IN264 Sheet 2 Page Jensen s Inequalit Let f be a conve function. If λ,..., λ n are positive numbers with λ + + λ n =, then for an,..., n I: f(λ + + λ n n ) λ f( ) + + λ n f( n ) Problem 9: Prove this inequalit b using induction on n. For n = 2 nothing is to prove, definition of conveit. Assume the statement is true for n: f( i= λ i i ) = f(λ + ( λ ) i=2 where we used the fact that i λ i i λ f( ) + ( λ )f( λ i=2 λ i i ) λ i= λ i λ = and thus could appl the induction assumption. λ i f( i ) Submit to homework@class.brml.org with subject line homework sheet 2 b 24//2, :59 CET