Def :Corresponding to an underlying experiment, we associate a probability with each ω Ω, Pr{ ω} Pr:Ω [0..1] such that Pr{ω} = 1

Transcription

1 Read : GKP 8.1,8.2 Quicksort is the sorting technique of choice although its worst-case behavior is inferior to that of many others. Why? What does average case mean? Seeking a phone number (which exists) in a phone book sees "on average" half the elements. Why? Def : Probability space (sample space) - Ω (nonempty) Ωdice={(1 1),...,(6 6)} Ωdice =36 (Assume dice distinguishable, i.e., Ωcoin={sequences of coin flips until 2 straight heads} ={(h h), (t h h), (h t h h), (t t h h),...} Ωcoin = (2,3) (3,2) ) Ω phone-book={ ,..., } Ω phone-book 134,400 A P.Zywien ΩUG3={unlabelled, undirected graphs of 3 vertices}= Def :Corresponding to an underlying experiment, we associate a probability with each ω Ω, Pr{ ω} Pr:Ω [0..1] such that Pr{ω} = 1 For fair distinguishable dice, Pr{ω}=1/36 for each ω Ω For Ωphone-book, can assume: -Pr{ω}=1/134,000 for each ω Ω -a closer approximation to usage, where Pr{ }>>Pr{ } Centrum Box Office Jas.Nkansah For ΩUG3, can assume: -Pr{ω}=1/4 for each ω Ω -if we start with labelled graph & add each edge with (independent) prob.=1/2, then Pr 1/8 3/8 3/8 1/8 Def : An event is a subset A Ω. Ωdice: A doubles ={(1 1),...,(6 6)} A 7 11 ={(1 6),...,(6 1), (5 6), (6 5)} Ωphone-book: A movie theater ΩUG3: A connected ={o-o-o,triangle} LN2-1

2 Def :Probability of an event: Pr{A}= Ωdice: Pr{A doubles }=6/36=1/6 Pr{A 7-11 }=8/36=2/9 Pr{ω} ω A Def : A random variable (r.v.) is a function from a sample space to the reals. X:Ω R+. (For us, R + because we measure resource consumption) For ω=(x y) Ωdice, X(ω)=x+y X( (1 3) )=4 For ω Ωcoin, X(ω)=# heads X( (h t t h t h h) )=4 Corresponding to each value x of each r.v. X is an event A={ω X(ω)=x}, and we extend Pr so that Pr{X =x}=pr{x}= Pr{ω}. ω A Mean We usually let r.v. be resource (time, memory) consumed by an algorithm, queue length of processes awaiting processor,... We usually want "typical" behavior of an r.v. Given values ( ), -mean = ( )/5=2.8 -median =3 (middle value after sorting) -mode =1 (most frequent value) In probability, we repeat an experiment many times & expect each value x of X to occur with frequency proportional to Pr{X=x}. -mean of X= x * Pr{X = x} x X(Ω) -median of X={x Pr{X x} 1/2 & Pr{X x} 1/2} -mode of X={x Pr{X=x} Pr{ X=y} for all y X(Ω)} Mean also called expected value of X, E[X] (In GKP, EX) For ω=(x y) Ωdice, X(ω)=x+y, E[X]=7 Note that if Pr{X=1} = Pr{X=2} = 0.5, then E[X] = 1.5, although X will never be 1.5 If X, Y are r.v.s on the same space Ω, then X+Y is also a r.v. on Ω. Def : X,Y are independent r.v.s if Pr{X =x Y =y}=pr{x =x}*pr{y =y } X,Y independent means they have no effect on each other. Because faces of 2 dice independent, Pr{ (1,3) }=Pr{1}*Pr{3}=(1/6)*(1/6). What is expected value of sum of faces if we throw 2 dice? E[X +Y]? =E[X]+E[Y]? Theorem : For r.v.s X 1,...,Xn defined on Ω, if E[X 1 ],...,E[Xn] exist, then E X k = E[X k ]. (Note: no assumption on independence, in fact, some Xi can be repeated) Proof : (For the case n=2) Let X=X 1,Y=X 2 E[X + Y] = ( X(ω) + Y(ω) ) * Pr{ω} = X(ω) * Pr{ω} + Y(ω) * Pr{ω} = E[X] + E[Y] x,y LN2-2

3 (In general) E X k = (ω) * Pr{ω} = X k X k (ω) * Pr{ω} = E[X k ] Ex : If 106 people throw their keys in a large hat, & each person retrieves a key "at random", what's the expected number of people who get their own key? Define a r.v. for each person such that we can combine the r.v.s to get the answer. Let Xk=if person i gets own key then 1 else 0. A less useful r.v. would be Xk=if person i gets own key then 42 else -6. For any experiment, {people who get their own key} = X k, & solution to problem is E X k. (Note: X k is an r.v.) By previous theorem, E X k = E[X k ]. For each Xk, E[Xk]=1/n. So E X k = 1 n =1. Ex : Simulation To estimate π, consider throwing darts at following target such that Pr{dart in area} area. Area of circle=π, area of square=4. Let r.v.x=if dart in circle then 1 else 0 E[X]=π/4. Throwing n darts, E X k = E[X k ] =n E[X]= nπ 4 π= 4 n * E X k = 4 n * E[X k ] (defun test () ; test if dart in circle (<= (+ (square (random 1.0)) (square (random 1.0))) 1.0)) (defun square (n) (* n n)) (defun count_successes (n) ; count darts in circle LN2-3

4 (cond ((= n 0) 0) ((test) (+ 1 (count_successes (- n 1)))) (t (count_successes (- n 1))))) (defun estimate-pi (n) (/ (* 4.0 (count_successes n)) n))? (estimate-pi 10) 3.6? (estimate-pi 100) 3.2? (estimate-pi 1000) 3.188? (estimate-pi 5000) Ex : Monte-Carlo Integration Suppose f:[a,b] -> [0,d]. ;; random x in [a,b], b>a (defun random_in_interval (a b) (+ a (random (float (- b a))))) (defun test (f) (>= (funcall f (random_in_interval a b)) (random_in_interval c d))) (defun count_successes (f n) (cond ((= n 0) 0) ((test f) (+ 1 (count_successes f (- n 1)))) (t (count_successes f (- n 1))))) (defun monte-carlo (f n) (* (- b a) (- d c) (/ (count_successes f n) (float n)))) (defun func (x) (/ 1.0 x)) (setf a 1) (setf b 500) (setf c 0) (setf d 1)? (log 500) ? (monte-carlo #'func 1000) ? (monte-carlo #'func 1000) 4.99 LN2-4

5 ? (monte-carlo #'func 1000) 6.986? (monte-carlo #'func 5000) Note that the estimate of π was Monte-Carlo. How quickly do we converge? For π, Pr{106 darts outside circle}>0. We bound the preceding probability; it never = 0. Given δ,ε (0,1), if I = integral, h = value returned by Monte-Carlo, Pr{ h -I < ε} 1- δ whenever # iterations n (1/ ε2 δ ). That is δ 1/ ε2. That is, for a fixed tolerance δ (say 0.05), 1 more decimal digit of precision requires 100 times the number of trials. In 1987, some Japanese computed π to 134*106 places. This is not the way to compute π. Monte-Carlo better than traditional numerical integration techniques for integrals of high dimension, since it's independent of dimension. From another angle, in estimating I= f(x)dx with estimator F n = 1 n ( ) f X k, the estimator is unbiased, i.e., E F n F n = E[ F n ] + ε, where ε = Vf Ω ( ) [ ] = f(x)dx, n = σ f, inverting this n yields that the number of samples to yield a desired error ε is n = Vf ε 2 More on Speed of Convergence : To estimate a parameter with n independent, identically distributed r.v.s Xi, drawn from a population with mean µ and standard deviation σ, Central Limit Theorem states that is normally distributed with mean nµ and variance nσ2. This can be X k transformed to a standard normally distributed r.v. (µ=0,σ=1) X k nµ Z = (dividing numerator & denominator by n) nσ = X µ where X = 1 σ X k, the sample mean. From another angle, choose n n some constant α <1, and let y α /2 be that value of Y such that Φ(Y)=1 - α /2. Pr{- y α /2 Z y α /2 } = Pr{ X + σy α / 2 µ X σy α / 2 } = 1- α. n n Interval X ± σy α / 2 usually called confidence interval and 1- α called n confidence level (usually as a %) For confidence level of 90%, y α/2 = 1.65, so 90% confident that µ covered by X ± σy α / 2 n Ex : Estimating the size of backtrack search trees. Consider the 4-queens problem, with the search tree for 1 queen/row. level # nodes Ω LN2-5

6 0 0 1 / / \ \ col: / apple shape =( ) size = 17 How big is the tree for the 25-queens problem? What is critical mass of fissionable material where, for a given neutron, Pr{i offspring}, i 0. What is probability of extinction of Selkow name given: -patrilineal naming, -there are no Selkows outside my house. For Americans in 1939, Pr{Xi = 0} Pr{Xi = k } (0.2126)*(0.5893)k -1, k 1 Consider sequence of r.v.s, X 0,X 1,..., where Xi estimates # level i. We want E[Xi]= # level i. size = Experiment : X 0 :=1 X i 0 i, so size is an r.v. i :=0 current_node :=root of tree neighbors :=children (current_node ) while neighbors > 0 do i :=i +1 Xi := neighbors *Xi -1 current_node :=random_select(neighbors ) neighbors :=children (current_node ) shape := (X 0,X 1,...) size := Do it for 8-queens. Theorem : E[size] is correct. Proof : (By induction on depth of tree) Basis : X 0 =1 I.H. : True for all trees of depth k. I.S. : X i 0 i LN2-6

7 E[size ]= n Pr{choosing i th child}* n * E[size(i th subtree)] 1 = n n * n * size( i th` subtree) = 1 + size( i th` subtree) 1 n Variance Suppose 100 lottery tickets are sold each week, and the prize is $100. Should we buy 2 tickets the same week, or 1 ticket in each of 2 lotteries? Let X 1 & X 2 be r.v.s denoting the amount we win with each ticket. E[X 1 ] = 0*Pr{we lose} + 100*Pr{we win} = = 1 & for X 2. E[X 1 +X 2 ]=E[X 1 ]+E[X 2 ]=2 (for both strategies). To distinguish between strategies, look at probability distributions: Ω (winnings) $0 $100 $200 1 drawing drawings To measure the "spread", we use Def : The variance of r.v. X is VX, alternatively V(X) or σ2(x), where VX=E[(X-E[X])2]. Note that X-E[X] and (X-E[X])2 are r.v.s For our example, let X=X 1 +X 2, E[X]=2, and Ω={0, 100, 200}. V X = 0*Pr{0} + 100*Pr{100} + 200*Pr{200} For 1 drawing V X = (0-2)2*.98 + (100-2)2*.02 + (200-2)2*0 = 196 For 2 drawings V X=(0-2)2*.9801+(100-2)2*.0198+(200-2)2*.0001=198 Computational simplification : Given a large Ω, computing E[(X-E[X])2] requires: -compute E[X] -for each x X(Ω), compute (x-e[x])*pr{x} To compute this in 1 step, note that V X=E[(X-E[X])2] = E[X2-2XE[X]+E[X]2] = E[X2]-2E[X]E[X]+E[X]2=E[X2]-E[X]2 (variance = the mean of the square - the square of the mean) LN2-7

8 For 1 drawing, VX = (02* * *0) - 4 = 196 For 2 drawings, VX = (02* * *.0001) - 4 = 198 If we could buy 100 tickets, betting it all in 1 lottery yields E[X]=100 and VX = 0, but betting it in 100 different lotteries yields E[X]=100 but Pr{X=0}=.99100= but a >0 probability of winning $10,000 Further Simplification : If r.v.s X 1,...,X n are independent, then V X k V ( X k ) Standard deviation = σ(x)= VX. Why do we care about variance? Chebyshev's inequality: Pr{(X-E[X])2 α ) VX/α for all α, or, replacing 1 α by c2*vx yields Pr{ X-E[X] cσ(x) ) 2. Probability X will be more than c 1 c standard deviations from the mean 2. Rolling fair dice n times, total c 35n value about 7n. Variance of n rolls. Choosing c=10, at least 99% of 6 the time the total of n =106 rolls between 6.976*106 and 7.024*106. LN2-8