Itroducto to Probablty Nader H Bshouty Departmet of Computer Scece Techo 32000 Israel e-mal: bshouty@cstechoacl 1 Combatorcs 11 Smple Rules I Combatorcs The rule of sum says that the umber of ways to choose a elemet from oe of two dsjots sets s the sum of the cardaltes of the sets That s, f A ad B are two fte sets wth o members commo, the A B = A + B The rule of product says that the umber of ways to choose a ordered par s the umber of ways to choose the frst elemet from A ad the secod elemet from B That s, f A ad B are two fte sets, the A B = A B 12 Smple Coutg Strgs A strg over a set S s a sequece of elemets of S For example, there are 8 bary strgs of legth 3: 000,001,010,011,100,101,110,111 A -strg over a set S ca be vewed as a elemet of the Cartesa product S of -tuples; thus, there are S strgs of legth Permutatos A permutato of a fte set S s a ordered sequece of all the elemets of S, wth each elemet appearg exactly oce For example, f S = {a, b, c}, there are 6 permutatos of S: abc,acb,bac,bca,cab,cba There are! permutatos of a set of elemets A -permutato of S s a ordered sequece of of S, wth o elemet appearg more tha oce the sequece The umber of -permutatos of a -set s: Combatos ( 1)( 2) ( + 1) =! ( )!
A -combato of a -set S s smply a -subset of S There are sx 2-combatos of the 4-set {a, b, c, d}: ab, ac, ad, bc, bd, cd The umber of -combatos of a -set 13 Equaltes!!( )! Bomal coeffcets We use the otato ( ) (read choose ) to deote the umber of -combatos of a -set We have! =!( ) Ths formula s symmetrc ad : = These umbers are also ow as bomal coeffcets, due to ther appearace the bomal expaso: (x + y) = x y A specal case: We also have 2 = =0 =0 = 1 1 = 1 1 1 = + 1 14 Iequaltes Bomal bouds For 1, we have the lower boud: 2
The upper boud: For all 0 (see 2), e ( ), where for coveece we assume that 0 0 = 1 For = λ, where 0 λ 1, ths boud ca be rewrtte as (see R3): 2 H(λ), λ where H(λ) = λlgλ (1 λ)lg(1 λ) s the etropy fucto ad where, for coveece, we assume that 0 lg 0 = 0, so that H(0) = H(1) = 0 For 0 λ 1 2 we have (see R4) λ For ay 0, j 0, 0, ad j +, The Strlg s approxmato s log! = 0 j + 2 H(λ) ( j ) j ( + 1 ) log 1 log(2π) + o(1), 2 2 2 = 22 (1 + O(1/)) π 2 Probablty 21 Itroducto We defe probablty terms of a sample space S, whch s a set whose elemets are called elemetary evets For flppg two dstgushable cos, we ca vew the sample space S = HH, HT, T H, T T A evet s a subset of the sample space S The evet S s called the certa evet, ad the evet Ø s called the ull evet We say that two evets A ad B are mutually exclusve f A B = Ø Axoms of probablty A probablty dstrbuto Pr{} o a sample space S s a mappg from evets of S to real umbers such that the followg probablty axoms are satsfed: 3
1 Pr[A 0 for ay evet A 2 Pr[S = 1 3 Pr[A B = Pr[A + Pr[B for ay two mutually exclusve evets A ad B We call Pr[A the probablty of the evet A Results: Pr[Ø = 0, f A B, the Pr[A Pr[B For A (the complemet of A), Pr[A = 1 Pr[A For ay two evets A ad B, Pr[A B = Pr[A + Pr[B Pr[A B Pr[A + Pr[B Dscrete probablty dstrbutos A probablty dstrbuto s dscrete f t s defed over a fte or or coutably fte sample spacethe for ay evet A, Pr[A = s A Pr[s If S s fte ad every elemetary evet s S has probablty Pr[s = 1/ S the we have the uform probablty dstrbuto o S A far co s oe whch the probablty of obtag a head s the same as the probablty of obtag a tal, that s, 1/2 22 Codtoal probablty ad Idepedece Codtoal probablty formalzes the oto of havg pror partal owledge of the outcome of a expermet The codtoal probablty of a evet A gve that aother evet B occurs s defed to be: Pr[A B Pr[A B = Pr[B Wheever Pr[B 0, (we read Pr[A B as the probablty of A gve B ) Two evets are depedet f, Pr[A B = Pr[A Pr[B, whch s equvalet, f Pr[B 0, to the codto Pr[A B = Pr[A A collecto A 1, A 2,, A of evets s sad to be parwse depedet f Pr[A A j = Pr[A Pr[A j For all 1 < j We say that they are (mutually) depedet f every -subset A 1, A 2,, A of collecto, where 2 ad 1 1 < 2 < <, satsfes: Pr[A 1 A 2 A = Pr[A 1 Pr[A 2 Pr[A 4
Bayes s Theorem From Bayes s theorem: Pr[A B = Pr[A B = Pr[A Pr[B A Pr[B Pr[A Pr[B A Pr[A Pr[B A + Pr[A Pr[B A Boole s equalty: For ay fte or coutably fte sequece of evets A 1, A 2,, Pr[A 1 A 2 Pr[A 1 + Pr[A 2 + For collecto of evets A 1, A 2,, A, Pr[A 1 A 2 A = Pr[A 1 Pr[A 2 A 1 Pr[A 3 A 1 A 2 Pr[A A 1 A 2 A 1 23 Dscrete Radom Varables ad Expectato A (dscrete) radom varable s a fucto from a fte or coutably fte sample space S to the real umbers For radom varable X ad a real umber x, we defe the evet X = x to be [s S : X(s) = x; thus, Pr[X = x = [s S:X(s)=x Pr[s The fucto: f(x) = Pr[X = x, s the probablty desty fucto of the radom varable X It s commo for several radom varables to be defed o the same sample space If X ad Y are radom varables, the fucto: f(x, y) = Pr[X = x ad Y = y Is the jot probablty desty fucto of X ad Y For a fxed value y, Pr[Y = y = x Pr[X = x ad Y = y We defe two radom varables X ad Y to be depedet f for all x ad y, the evets X = x ad Y = y are depedet or, equvaletly, f for all x ad y, we have Pr[X = x ad Y = y = Pr[X = x Pr[Y = y Expected value of a radom varable The expected value (or, syoymously, expectato or mea) of a dscrete radom varable s E[X = x Pr[X = x x Sometmes the expectato of s deoted by µ x or, whe the radom s apparet from cotext, smply by µ The expectatos of the sum of two radom varables s the sum of ther expectatos, that s, E[X + Y = E[X + [Y If s ay radom varable, ay fucto g(x) defes a ew radom varable g(x) If the expectato of g(x) s defed, the E[g(X) = x g(x) Pr[X = x 5
Lettg g(x) = ax, we have for ay costat a, E[aX = ae[x E[aX + Y = ae[x + E[Y Whe two radom varables X ad Y are depedet ad each has a defed expectato, E[XY = E[XE[Y I geeral, whe radom varable X 1, X 2,, X are mutually depedet, E[X 1 X 2 X = E[X 1 E[X 2 E[X Whe a radom varable X taes o values from the atural umbers N = {0, 1, 2, }, there s a ce formula for ts expectato: Varace ad stadard devato E[X = Pr[X The varace of a radom varable X wth mea E[s: V ar[x = E[(X E[X) 2 = E[X 2 E 2 [X We have E[X 2 = V ar[x + E 2 [X V ar[ax = a 2 V ar[x Whe X ad Y are depedet radom varables, V ar[x +Y = V ar[x+v ar[y I geeral, f radom varables X 1, X 2,, X are parwse depedet, the V ar[ = V ar[ The stadard devato of a radom varable s the postve square root of the varace of X Marov s equalty: (see Rule 1) For a oegatve radom varable X Pr[X t E[X t for all t > 0 ad Pr[X λe[x 1 λ See more results [G Appedx A Chebychev s equalty: 6
Pr[ X E[X V ar[x 2 Beayme-Chebyschev Let X 1,, X m be parwse depedet radom varables such that E[ = µ ad V ar[ = σ 2 The See the proof [G Appedx A [ m Pr m µ λ σ2 λ 2 m 24 Geometrc ad Bomal dstrbutos A co flp s a stace of a Beroull tral, whch s defed as a expermet wth oly two possble outcomes: success, whch occurs wth probablty p, ad falure, whch occurs wth probablty q = 1 p The trals are mutually depedet o each has the same probablty p for success The geometrc dstrbuto Suppose we have a sequece of Beroull trals, each wth a probablty p of success ad a probablty q = 1 p of falure Let the radom varable X be the umber of trals eeded to obta a success Pr[X = = q 1 p A probablty dstrbuto satsfyg ths s sad to be a geometrc dstrbuto The bomal dstrbuto E[X = 1/p V ar[x = q/p 2 Defe the radom varable X to be the umber of success trals Pr[X = x = p q A probablty dstrbuto satsfyg ths s sad to be a bomal dstrbuto Defe: b(;, p) = p (1 p) We have: E[X = p ad V ar[x = pq The bomal dstrbuto b(;, p) creases as rus from 0 to utl t reaches the mea p, ad the t decreases Let 0, let 0 < p < 1, let q = 1 p, ad let 0 The, b(;, p) p q 7
For 0, b(;, 1/2) 2 H(/) The tals of the bomal dstrbuto Let X 1, X 2,, X be depedet radom varables such that {0, 1} ad E[ = p The ths s equvalet to a sequece of Beroull trals, where success occurs wth probablty p Let X be the radom varable deotg the total umber of successes The for 0, the probablty of at least success s: ad [ Pr = b(;, p) = p For p < <, the probablty of more tha successes s: [ ( )p Pr > = b(;, p) < b(;, p), p =+1 [ Pr p λ [ Pr = b(;, p) pe λ λ For 0 < < p, the probablty of fewer tha success s: [ 1 Pr < = b(;, p) < =0 (1 p) q b(;, p) p Cheroff Let X 1,, X be depedet radom varables such that {0, 1} ad E[ = p The Multplcatve Form Addtve Form [ Pr [ Pr p [ Pr [ Pr p p λp e λ2p/3 λp e λ2p/2 p λ e 2λ2 8 λ e 2λ2
ad [ Pr See aother boud [G Appedx A p λ 2e 2λ2 Hoeffdg Let X 1,, X m be depedet radom varables such that [a, b ad E[ = p The [ Pr p λ 2e 2λ2 /(b a) 2 Other Results Cosder depedet radom varables X 1,, X (a sequece of Beroull trals) where E[ = p (where the th tral, for = 1, 2,,, success occurs wth probablty p ad falure occurs wth probablty q = 1 p ) Let µ = E [ The for r > µ, [ µe r Pr µ r r Pr[X µ r e r2 /2 3 Examples Example 1 Radom Halvg: A game where you radomly uformly choose a teger 1 1 ad the 1 2 1 utl you get 1 Show that the expected umber of steps s 1 + 1 1/ Soluto: Let X be the expected umber of steps the X 1 = 1 ad E[X = 1 + E 1 [E[ Now + 1 E[X +1 E[X = 1 + 1 E[X +1 ad therefore E[X +1 = E[X + (1/) Acowledgemet: To Vva Bshouty for helpg me wth ths survey Refereces [AS N Alo ad J H Specer The Probablstc Method [CLR T Corme, C E Leserso ad R L Rvest Itroducto to Algorthms [G O Goldrech Moder Cryptography, Probablstc Proofs ad Pseudo-radomess [MR R Motwa ad P Raghava Radomzed Algorthms 9
4 Proofs Rule 1 Marov s equalty: for all t > 0 Proof We have Pr[X t E[X t E[X = x Pr[X = x x x t Pr[X = xx Pr[X tt Rule 2 We have Proof We have = ( + ( )) ( ) ( ) Rule 3 For 0 λ 1 we have where H(λ) = λlgλ (1 λ)lg(1 λ) Proof Use Rule 2 2 H(λ), λ Rule 4 For 0 λ 1 2 we have λ 0 2 H(λ) Proof We have λ 1 = (λ + (1 λ)) λ (1 λ) = λ ( λ (1 λ) 1 λ λ ) λ λ = 2 H(λ) 0 (1 λ) ( λ 1 λ ) λ 10