Average Case Complexity

Size: px

Start display at page:

Download "Average Case Complexity"

Claud Preston
6 years ago
Views:

1 Probability

2 Applicatios Aalysis of Algorithms Average Case Complexity Mote Carlo Methods Spam Filters

3 Probability Distributio Basic Cocepts S = {a 1,, a } = fiite set of outcomes = sample space p :S [0,1] Evets -- p(a k ) 0 -- p(a k ) = 1 k=1 A evet E is a subset of the possible outcomes S -- Pr(E) = p(a i ) a i E For equally likely outcomes -- Pr(E) = E S

4 Example -- Dice Dice Ordered Pairs: 36 possible outcomes (m, ) Sums: 11 possible outcomes {2, 3,, 12} Uordered Pairs: 21 possible outcomes, 6 doubles ad 15 o-doubles Moral: Must describe both the experimet ad the possible outcomes. -- Ordered Pairs: Pr(m, ) = 1/ Sums: Pr(2) = 1/ 36, Pr(4) = 3 / 36,, Pr(7) = 6 / Uordered Pairs: Pr(double) = 6 / 36, Pr(each odouble) = 2 / 36

5 Example -- Poker Poker Hads # poker hads = C(52, 5) # 3 of a kid = C(13, 1) C(4, 3) Pr(3 of a kid) = C(13, 1) C(4, 3) C(52, 5).01 # flushes = C(4, 1)C(13, 5) -- Pr( flush) = C(4, 1) C(13, 5) C(52, 5).00396

6 Example -- Bridge Splits i Bridge # oppoet bridge hads = C(26, 13) 4 Missig Trumps split = C(4, 2) C(22, 11) C(26, 13) 40% split = 2 C(4, 3) C(22, 10) C(26, 13) 50% split = 2 C(4, 4) C(22, 9) C(26,13) 10%

7 More Examples Choose Up Sides 10 kids, 5 per team # possible teams with player x = C(9, 4) # possible teams with players x ad y = C(8, 3) Pr(two frieds o same team) = C(8, 3) C(9,4) = 4 9 < 1 2 Hatcheck Problem # hat permutatios: P() =! # hat deragemets: D() =! 1 1 1! + 1 2!! e Pr(deragemet) 1 =.3679 ( > 7) e

8 Complemetary Evets Setup S = {a 1,, a } = fiite set of outcomes S E = set of possible outcomes E c = S E = complemetary set of outcomes -- P(E) =1 P(E c ) c -- P(E) = p(e k ) = 1 p(e k j ) j

9 Coi Tosses Examples E = at least oe head i tosses E c = o heads i tosses -- p(e c ) = 1/ 2 -- p(e) = 1 p(e c ) = 1 1/ 2 = Birthday Problem E = at least 2 out of people havig same birthday (day ad moth) E c = o 2 people have the same birthday -- p(e c ) = (364 / 365)(363 / 365) ((365 ) / 365) -- p(e) = 1 p(e c ) p(e) >1 / 2 whe > 26

10 Biomial Distributio Beroulli Trials t = probability of evet (success) 1 t = probability of oevet (failure) B k (t) = probability of k evets (successes) i idetical, idepedet trials Formulas B k (t) = ( k )t k (1 t) k -- t k = probability of k successes -- (1 t) k = probability of k failures -- ( k ) = umber of ways exactly k successes ca occur i trials

11 Examples Coi Tossig t = probability of heads B k (t) = ( k )t k (1 t) k = probability of exactly k heads i tosses Ur Models -- Samplig with Replacemet w white balls, b black balls t = w / (w + b) = probability of selectig a white ball B k (t) = ( k )t k (1 t) k = probability of selectig exactly k white balls i trials Radom Walk i Pascal s Triagle t = probability of turig right B k (t) = ( k )t k (1 t) k = probability of ladig i kth bi at the bottom

12 Mote Carlo Methods Computatio of π Simulatio of Radom Walks

13 Coditioal Probability

14 Coditioal Probability Formula Pr(E F) = Pr(E F) Pr(F) provided that Pr(F) > 0 Proof (for equally likely evet) Pr(E F) = E F F = E F / S F / S = Pr(E F) Pr(F) Proof (for arbitrary probability distributios) F is the ew Sample Space Observatio Coditioal probability is ofte tricky -- see below.

15 Example -- Cards Playig Cards Draw oe card out of possible outcomes Pr(red) = = 1 2 Pr(diamod) = = Pr(diamod red) = 1 / 4 1/ 2 = Pr(red diamod) = 1 / 4 1 / 4 = Pr(diamod ace) = 1/ 52 4 / 52 = 1 4

16 Example -- Cois Cois Flip two peies (distiguished by dates) -- 4 possible outcomes -- Pr(2 heads first pey is a head) = 1/ 4 1/ 2 = 1/ 2 -- Pr(2 heads 1 head) =?

17 Example -- Cois Probabilities Flip two peies (distiguished by dates) -- 4 possible outcomes -- Pr(2 heads first pey is a head) = 1/ 4 1/ 2 = Pr(2 heads 1 head) = 1/ 4 3 / 4 =1 / 3 (??? see below) Possible Evets Possible Evets = {HH, HT, TH, TT} Possible Evets with First Pey Head = {HH, HT} Possible Evets with at Least Oe Head = {HH, HT, TH}

18 Example -- Childre Boy-Girl Boys ad Girls are equally likely. You ask: Do you have ay boys? Ma respods: Yes. Ma voluteers: I have two childre; oe is a boy. Ma says: I have two childre: the firstbor is a boy. Questio Pr(2 boys 1 boy) =?

19 Example -- Childre Aalysis You ask: Do you have ay boys? Ma respods: Yes. -- Pr(2 boys 1 boy) = Possible Evets = {BB, BG, GB} Ma voluteers: I have two childre; oe is a boy. -- Pr(2 boys 1 boy) = 1/ 4 2 / 4 = Possible Evets -- BB -- (1/ 4)1 -- BG -- (1/ 4)(1/ 2) -- GB -- (1/ 4)(1/ 2)

20 Aalysis (cotiued) Ma says: I have two childre: the firstbor is a boy. -- Pr(2 boys 1 boy) = Possible Evets -- BB -- 1/ 2 -- BG -- 1/ 2 Moral Protocol Matters See Teasers Paper -- Probability Depeds o the Protocol

21 Example -- Childre Boy-Girl Boys ad Girls are equally likely Woma says: I have a girl. Woma says: I have a girl amed Alice. Questios Pr(2 girls 1 girl) =? Pr(2 girls 1 girl amed Alice) =?

22 Example -- Childre Aalysis I have a girl. -- Pr(2 girls 1 girl) = 1/ 4 3/ 4 = Possible Evets = {GG, BG, GB} I have a girl amed Alice. -- Pr(2 girls 1 girl amed Alice) = Possible Evets = {AB, BA, AG, GA} Coclusio Protocol Matters

23 Iformatio Leaks Logical Puzzles Islad of Perfect Logicias A Daughter amed Alice Cryptography Frequecy of Letters i Alphabet Timig Chael Power Chael Sublimial Chaels -- Message i the Noise

24 Example -- Cards Colored Cards Give 2 Cards -- red/red ad red/white Pick a card at radom ad select a side at radom Questio Probability of (2 red 1 red) =?

25 Example -- Cards Colored Cards Give 2 Cards -- red/red ad red/white Pick a card at radom ad select a side at radom Probability of (2 red 1 red) = 1/ 2 3/ 4 = RR -- (1/ 2)1= 1/ 2 -- RW -- (1/ 2)(1 / 2) = 1/ 4 More likely to be red o back, sice red/red has two chaces to lad o red.

26 Example -- Mote Hall Problem Protocol Three Doors Oe Fabulous Prize You Pick a Door at Radom Host (Mote Hall) Opes a Differet Door -- No Prize Offers to Trade Your Door for His Remaiig Door Questio Should you Make the Deal? Does it Matter?

27 Example -- Mote Hall Problem Aalysis Pr(Prize Behid Your Oe Door) = 1 3 Pr(Prize Behid His Two Doors) = 2 3 Pr(Prize Behid Door He Opes) = 0 Pr(Prize Behid Door He Does Not Ope) = 2 3 Solutio: Make the Deal!

28 Mote Hall Problem -- Variatios Variatio 1 You choose your door AFTER Mote Hall opes his door. Pr(Prize Behid Your Oe Door) = 1 2. Switchig Doors does NOT Chage Your Odd of Wiig. Variatio 2 Mote Hall opes oe of his doors at RANDOM. -- If Mote fids the prize, you lose. -- If Mote does ot fid the prize, Pr(Prize Behid Your Oe Door) = 1/ 3 2 / 3 = 1 2. Switchig Doors does NOT Chage Your Odd of Wiig.

29 Example -- Prisoer Problem Protocol Judge seteces Tom or Dick or Harry to hag -- pr(tom will hag) = 1/ 3 Tom asks jailer to tell him a ame of oe of the other two who will NOT be haged. Jailer says: Dick will ot be haged. Questios Have the Odds Chaged for Tom? Does it matter to Tom that Dick will NOT be haged? What is the probability that Tom will be haged?

30 Example -- Prisoer Problem Protocol Judge seteces Tom or Dick or Harry to hag -- pr(tom haged) = 1/ 3 Tom asks jailer to tell him a ame of oe of the other two who will NOT be haged. Jailer says: Dick will ot be haged. Aalysis pr(tom ad ~Dick ~Dick) = pr(tom will hag) = 1/3 1/2 (1/ 3) (2 / 3) (2 / 3) = 1/ 3 Reaso Tom hags pr(dick selected) = 1/ 2 Tom does NOT hag pr(dick selected) = 1

31 Bayes Theorem

32 Motivatio Problem Fid p(f) give that E has occurred. Fid p(f E) if we kow p(e F).

33 Basic Relatios Lemma 1: p(e F) = p(e F) p(f) Proof: p(e F) = p(e F) p(f) p(e F) = p(e F) p(f) Lemma 2: p(f E) p(e) = P(E F)p(F) Proof: p(e F) = p(e F) p(f) p(f E) = p(f E) p(e) p(f E) p(e) = P(E F) p(f)

34 Bayes' Theorem Bayes' Formula p(f E) = p(e F) p(f) p(e F)p(F) + p(e F c )p(f c ) Proof: By Lemma 1 p(f E) = P(E F) p(e) = p(e F) p(f) p(e) But E = (E F) (E F c ) is a disjoit uio, so agai by Lemma 1 p(e) = p(e F) + p(e F c ) = p(e F) p(f)+ p(e F c ) p(f c )

35 Example: Tests for Rare Diseases Notatio F = the evet that a perso is sick with a very rare disease E = the evet that a perso tests positive for this rare disease Protocol p(f) = 1/100, 000 very rare disease p(e F) = 99 /100 test correct 99% of the time for sick people p(e c F c ) = 995 /1000 test correct 99.5% of time for healthy people Problems p(f E) = probability that a perso that tests positive is actually ill =? p(f c E c ) = probability that a perso that tests egative is actually healthy =?

36 Example: Tests for Rare Diseases (cotiued) Probabilities p(f) = 1/100, 000 = Probability of Illess p(f c ) = 1 1/100,000 = Probability of Health p(e F) = 99 / 100 =.99 Probability Sick Perso Tests Positive p(e c F) = 1 99 /100 =.01 Probability Sick Perso Tests Negative p(e c F c ) = 995 /100 =.995 Probability Healthy Perso Tests Negative p(e F c ) = /100 =.005 Probability Healthy Perso Tests Positive p(f E) = p(e F) p(f) p(e F)p(F) + p(e F c )p(f c ) = (.99)(.00001) (.99)(.00001)+ (.005)(.99999).002 p(f c E c ) = p(e c F c ) p(f c ) p(e c F c ) p(f c ) + p(e c F)p(F) = (.995)(.99999) (.995)(.99999)+ (.01)(.00001)

37 Tests for Rare Diseases (cotiued) Rare Diseases -- p(f) = 1/100, 000 Test Positive -- Do Not Worry -- p(f E).002 Test Negative Healthy -- p(f c E c ) Commo Diseases -- p(f) = 1/10 Test Positive Sick -- p(f E) = Test Negative Healthy p(e F) p(f) p(e F)p(F) + p(e F c )p(f c ) = (.99)(.1) (.99)(.1)+ (.005)(.9) p(f c E c ) = p(e c F c ) p(f c ) p(e c F c ) p(f c ) + p(e c F)p(F) = (.995)(.9) (.995)(.9)+ (.01)(.1).99888

38 Example: Spam Filters Notatio S = the evet that the message is spam E = the evet that the message cotais the word w Protocol p(s) = 9 /10 most of my messages are spam p(e S) = p(w) probability that w appears i spam p(e S c ) = q(w) probability that w appears i a real message Problems p(s E) = probability that the message is spam if w appears =? p(s c E c ) = probability that the message is ot spam if w does ot appear =?

39 Example: Spam Filters (cotiued) Probabilities p(s) = 9 /10 =.9 Probability of Spam p(s c ) = 1 9 /10 =.1 Probability of Real Message p(e S) = p(w) Probability w appears i Spam p(e c S) = 1 p(w) Probability w does NOT appears i Spam p(e S c ) = q(w) Probability w appears i Message p(e c S c ) = 1 q(w) Probability w does NOT appear i Message p(s E) = p(e S) p(s) p(e S)p(S)+ p(e S c ) p(s c ) = 9 p(w) 9 p(w) + q(w) p(s c E c ) = p(e c S c ) p(s c ) p(e c S c ) p(s c ) + p(e c S)p(S) = 1 q(w) 1 q(w) ( ) + 9( 1 p(w) )

40 Mote Hall Problem Notatio F = evet that a perso selects the door with the prize E = evet that the door opeed by Mote Hall does ot cotai the prize Protocol p(f) = 1/ 3 You have a 1/3 chace of selectig the prize p(e F) = 1 Mote CANNOT have prize if you do p(e F c ) = 1 Mote NEVER opes the door with the prize Problem p(f E) = probability that your door has the prize if Mote Hall s does ot =?

41 Mote Hall Problem (cotiued) Probabilities p(f) = 1/ 3 Probability your door has the prize p(f c ) = 2 / 3 Probability your door does ot have prize p(e F) = 1 Mote CANNOT have prize if you do p(e F c ) = 1 Mote NEVER opes the door with the prize Coditioal Probability p(f E) = p(e F) p(f) p(e F)p(F) + p(e F c )p(f c ) = (1)(1/ 3) (1)(1 / 3)+ (1)(2 / 3) = 1/ 3 Coclusio Switch doors with Mote Hall

42 Mote Hall Problem -- Variatio Probabilities p(f) = 1/ 3 Probability your door has the prize p(f c ) = 2 / 3 Probability your door does ot have prize p(e F) = 1 Mote CANNOT have prize if you do p(e F c ) = 1/ 2 Mote selects a door at radom Coditioal Probability p(f E) = p(e F) p(f) p(e F)p(F) + p(e F c )p(f c ) = (1)(1/ 3) (1)(1 / 3)+ (1/ 2)(2 / 3) = 1/ 2 Coclusio Switchig does NOT chage the odds.

43 Daughter Problem Notatio F = evet that a perso with two childre has two daughters E = evet that a perso with two childre has at least oe daughter Protocol p(f) = 1/ 4 oe of four possible case: BB, BG, GB. GG p(f c ) = 3/ 4 three of four possible case: BB, BG, GB. GG p(e F) = 1 if you have two daughters, you have at least oe p(e F c ) = 2 / 3 two out of three cases: BB, BG, GB Problem p(f E) = probability perso has two daughters if they have oe daughter =?

44 Daughter Problem (cotiued) Probabilities p(f) = 1/ 4 Probability of two daughters p(f c ) = 3/ 4 Probability of at most oe daughter p(e F) = 1 If you have two daughter, you have at least oe p(e F c ) = 2 / 3 Two out of three cases: BB, BG, GB Coditioal Probability p(f E) = p(e F) p(f) p(e F)p(F) + p(e F c )p(f c ) = (1)(1/ 4) (1)(1 / 4) + (2 / 3)(3/ 4) = 1/ 3

45 A Daughter Named Alice Notatio F = evet that a perso with two childre has two daughters E = evet that a perso with two childre has oe daughter amed Alice p = percetage of girls amed Alice Protocol p(f) = 1/ 4 oe of four possible case: BB, BG, GB. GG p(f c ) = 3/ 4 three of four possible case: BB, BG, GB. GG p(e F) = 2 p Alice is a girl s ame ad there are two girls p(e F c ) = (2 / 3) p two of three possible cases: BB, GB, BG Problem p(f E) = probability perso has two daughters if they daughter amed Alice =?

46 A Daughter Named Alice (cotiued) Probabilities p(f) = 1/ 4 Probability of two daughters p(f c ) = 3/ 4 Probability of at most oe daughter p(e F) = 2p Alice is a girl s ame ad there are two girls p(e F c ) = 2p / 3 two of three possible cases: BB, GB, BG Coditioal Probability p(f E) = p(e F) p(f) p(e F)p(F) + p(e F c )p(f c ) = (2 p)(1 / 4) (2 p)(1/ 4)+ (2p / 3)(3 / 4) = 1/ 2

47 Expectatio ad Average Case Complexity

48 Radom Variables Defiitio A Radom Variable X is a Fuctio from the sample space S of a experimet to the real umbers. X :S R Examples Coi Tossig Dice -- S = {HH, HT, TH, TT} -- X = Number of heads -- S = {(1,1), (1, 2),, (6, 6) } -- X = Sum of Dots

49 Expectatio Setup S = {a 1,, a } = sample space pr :S [0,1] = probability distributio X :S R = radom variable Expectatio E(X) = Pr(a i ) X(a i ) = Pr(X = r i ) r i i=1 Weighted Average i=1 Additivity E(X +Y ) = E(X)+ E(Y )

50 Additivity Formula E(X +Y ) = E(X)+ E(Y ) Proof ( ) E(X +Y ) = Pr(a i ) X(a i )+Y (a i ) i=1 = Pr(a i )X(a i ) + Pr(a i )Y (a i ) = E(X)+ E(Y ) l=1 Advice Try to use Additivity, NOT the Defiitio of Expectatio i=1

51 Direct Method Dice S = {(1,1), (1, 2),, (6, 6) } X = Sum of Dots E(X) = Pr(s i )X(s i ) = k= = = 7 Summatio Method X 1 = Number of Dots of First Die X 2 = Number of Dots of Secod Die X = X 1 + X 2 E(X k ) = = 21 6 = 3.5 E(X) = E(X 1 )+ E(X 2 ) = = 7

52 Beroulli Trials Biomial Distributio X(Experimet) = Number of Successes E(X) = kb k (t) = t k=0 Direct Method -- By Brute Force Algebra (see ext page) Summatio Method X k = 1 for success o the kth trial = 0 for failure o the kth trial X = X k k=1 E(X) = E(X k ) = t = t k=1 k=1

53 Expectatio: Biomial Distributio Algebra E(X) = k=0 kb k (t) = k( k )t k (1 t) k k =1 = k = t k =1 k =1! k!( k)! t k (1 t) k ( 1)! (k 1)! (( 1) (k 1) )! t k 1 ( 1) (k 1) (1 t) 1 = t ( k 1)t k 1 (1 t) ( 1) ( k 1) k =1 1 (t) = t B k 1 = t k=1

54 Aother Example Birthday Problem Fid umber of people,, i a room, so that expectatio is at least oe that two people have the same birthday. Setup X() = # people with the same birthday (day ad moth) X ij (a i, a j ) = 1 if a i ad a j bor o same day ad moth = 0 otherwise X() = X ij (a i, a j ) i< j

55 Solutio X() = # people with the same birthday X() = X ij (a i, a j ) i< j -- p(x ij =1) = 1/ E(X ij ) = 1/ # X ij = C(, 2) -- E(X) = (1/ 365) C(,2) C(, 2) 378 X() 1 NOT quite the same as the umber eeded to make the probability 1/2.

56 Average Case Complexity Setup S = {a 1,, a } = possible iputs to a algorithm X(a i ) = umber of operatios used by the algorithm for iput a i Formula E(X) = Pr(a i ) X(a i ) = Average Case Complexity i=1

57 Example -- Liear Search Assumptios elemets i UNORDERED list: a 1,, a Pr(x = a i ) = 1 (equal likelihood) X(a i ) = umber of comparisos used to locate a i is i Average Case Complexity E(x) = i = 1 i=1 i = 1 i=1 ( +1) 2 = Result makes good sese as a average -- half more, half less Worst Case Complexity =

58 Liear Search -- Revisited Assumptios elemets i UNORDERED list: a 1,, a Pr(x = a i ) = 1 (equal likelihood) Pr(x i list) = p X(a i ) = umber of comparisos used to locate a i is 2i compare to curret elemet -- check for ed of list -- iside ad outside the loop X(a ot i list) = umber of comparisos to determie a ot i list is oe additioal compariso o ( +1) st time through the loop

59 Liear Search -- Revisited (cotiued) Average Case Complexity E(x) = (2 + 2)(1 p)+ i=1 (2i +1)p = (2 + 2)(1 p)+ p i=1 (2i +1) = (2 + 2)(1 p)+ (( +1) 2 1) p = (2 + 2)(1 p)+ ( + 2)p = (2 + 2) p

60 Variace ad Stadard Deviatio

61 Variace ad Stadard Deviatio Setup S = {a 1,, a } = sample space p :S [0, 1] = probability distributio X = radom variable Variace V(X) = ( X(s i ) E(X) ) 2 p(s i ) i=1 Stadard Deviatio σ(x) = V(X) = ( X(s i ) E(X) ) 2 p(s i ) i=1 Objective To measure deviatio of a radom variable X from its average value E(X) (expectatio)

62 Variace ad Expectatio Theorem: V(X) = E(X 2 ) ( E(X) ) 2 Proof: V(X) = ( X(s i ) E(X) ) 2 p(s i ) i=1 = X(s i ) 2 p(s i ) 2E(X) X(s i ) p(s i )+ E(X) 2 p(s i ) i=1 i=1 = E(X 2 ) 2E(X)E(X) + E(X) 2 = E(X 2 ) ( E(X) ) 2 i=1

63 Example O Die E(X) = = 21 6 = 7 2 ( E(X) ) 2 = = = 49 4 E(X 2 ) = = 91 6 V(X) = E(X 2 ) E(X) ( ) 2 = =

64 Expectatio ad Idepedet Variables Defiitio X,Y are called idepedet radom variables if p(x = r ad Y = s) = P(X = r)p(x = s) for all r,s Theorem: X,Y idepedet E(XY ) = E(X) E(Y ) Proof: E(XY ) = Pr(a i )X(a i )Y (a i ) i=1 ( ) = r 1 r 2 Pr(X = r 1 ) ad Pr(Y = r 2 ) r 1, r 2 = r 1 Pr(X = r 1 ) r 1 = E(X)E(Y ) r 2 Pr(Y = r 2 ) r 2

65 Variace ad Idepedet Variables Theorem: X,Y idepedet V(X +Y ) = V(X) +V(Y ) Proof: V(X +Y ) = E ((X +Y ) 2 ) ( E(X + Y )) 2 = E(X 2 + 2XY +Y 2 ) ( E(X) ) 2 2E(X)E(Y ) ( E(Y )) 2 = E(X 2 )+ 2E(XY )+ E(Y 2 ) ( E(X) ) 2 2E(X)E(Y ) ( E(Y )) 2 = E(X 2 ) ( E(X) ) 2 + E(Y 2 ) ( E(Y )) 2 {E(XY ) = E(X)E(Y )} = V(X)+V(Y )

66 Two Dice Notatio X 1 = umber of dots o first die X 2 = umber of dots o secod die X = X 1 + X 2 = sum of dots o both dice Variace V(X 1 ) = V(X 2 ) = V(X) = V(X 1 + X 2 ) = =

67 Beroulli Trials 1 Beroulli Trial X(t) = 1 success = 0 failure E(X) = t V(X) = E(X 2 ) ( E(X) ) 2 = t t 2 = t(1 t) Beroulli Trials V(X) = V(X X ) = V(X 1 )+ +V(X ) V(X) = t(1 t)

f X (12) = Pr(X = 12) = Pr({(6, 6)}) = 1/36

f X (12) = Pr(X = 12) = Pr({(6, 6)}) = 1/36 Probability Distributios A Example With Dice If X is a radom variable o sample space S, the the probablity that X takes o the value c is Similarly, Pr(X = c) = Pr({s S X(s) = c} Pr(X c) = Pr({s S X(s)