Note: Hadouts DO NOT replace the book. I most cases, they oly provide a guidelie o topics ad a ituitive feel. The math details will be covered i class, so it is importat to atted class ad also you MUST read the book. 1 Motivatio ad Applicatios As stated by Laplace, Probability is commo sese reduced to calculatio. You eed to first lear the theory required to correctly do these calculatios. The examples that I solve ad those i the book ad the homeworks will provide a woderful practical motivatio as to why you eed to lear the theory. If you patietly grasp the basics, especially the first 4 chapters of BT, it will be the most useful thig you ve ever leart - whether you pursue a career i EE or CE or Ecoomics or Fiace or Maagemet ad also while you try to ivest i stocks or gamble i Las Vegas! Applicatios: commuicatios (telephoes, cell phoes, TV,...), sigal processig (image ad video deoisig, face recogitio, trackig movig objects from video,...), systems ad cotrol (operatig a airplae, fault detectio i a system,...), predictig reliability of a system (e.g. electrical system), resource allocatio, iteret protocols, o-egieerig applicatios (e.g. medicie: predictig how prevalet a disease is or well a drug works, weather forecastig, ecoomics). 2 Itroductio: Topics Covered (BT, 1.1-1.6) What is Probability Set Theory Basics Probability Models Coditioal Probability Total Probability ad Bayes Rule Idepedece Coutig 3 What is Probability? Measured relative frequecy of occurrece of a evet. Example: toss a coi 100 times, measure frequecy of heads or compute probability of raiig o a particular day ad moth (usig past years data) Or subjective belief about how likely a evet is (whe do ot have data to estimate frequecy). Example: ay oe-time evet i history or how likely is it that a ew experimetal drug will work? Two types of problems eed to be solved 1
1. Specify the probability model or lear it (statistics). 2. Use the model to compute probability of differet evets. We will assume the model is give ad will focus o 2. i this course. 4 Set Theory Basics Set: ay collectio of objects (elemets of a set). Discrete sets Fiite umber of elemets, e.g. umbers of a die Or ifiite but coutable umber of elemets, e.g. set of itegers Cotiuous sets Caot cout the umber of elemets, e.g. all real umbers betwee 0 ad 1. Uiverse (deoted Ω): cosists of all possible elemets that could be of iterest. I case of radom experimets, it is the set of all possible outcomes. Example: for coi tosses, Ω = {H, T }. Empty set (deoted φ): a set with o elemets Subset: A B: if every elemet of A also belogs to B. Strict subset: A B: if every elemet of A also belogs to B ad B has more elemets tha A. Belogs:, Does ot belog: / Complemet: A or A c, Uio: A B, Itersectio: A B A {x Ω x / A} A B {x x A, or x B}, x Ω is assumed. A B {x x A, ad x B} Visualize usig Ve diagrams (see book) Disjoit sets: A ad B are disjoit if A B = φ (empty), i.e. they have o commo elemets. DeMorga s Laws (A B) = A B (1) (A B) = A B (2) Proofs: Need to show that every elemet of LHS (left had side) is also a elemet of RHS (right had side) AND vice versa. Details i class. Read sectio o Algebra of Sets, pg 5 2
5 Probabilistic models There is a uderlyig process called experimet that produces exactly ONE outcome. A probabilistic model: cosists of a sample space ad a probability law Sample space (deoted Ω): set of all possible outcomes of a experimet Evet: ay subset of the sample space Probability Law: assigs a probability to every set A of possible outcomes (evet) Choice of sample space (or uiverse): every elemet should be distict ad mutually exclusive (disjoit); ad the space should be collectively exhaustive (every possible outcome of a experimet should be icluded). Probability Axioms: 1. Noegativity. P(A) 0 for every evet A. 2. Additivity. If A ad B are two disjoit evets, the P(A B) = P(A) + P(B) (also exteds to ay coutable umber of disjoit evets). 3. Normalizatio. Probability of the etire sample space, P(Ω) = 1. Probability of the empty set, P(φ) = 0 (follows from Axioms 2 & 3). Sequetial models, e.g. three coi tosses or two sequetial rolls of a die. Tree-based descriptio: see Fig. 1.3 Discrete probability law: sample space cosists of a fiite umber of possible outcomes, law specified by probability of sigle elemet evets. Example: for a fair coi toss, Ω = {H, T }, P(H) = P(T) = 1/2 Discrete uiform law for ay evet A: P(A) = umber of elemets i A Cotiuous probability law: e.g. Ω = [0, 1]: probability of ay sigle elemet evet is zero, eed to talk of probability of a subiterval, [a, b] of [0, 1]. Example 1.4, 1.5 Properties of probability laws 1. If A B, the P(A) P(B) 2. P(A B) = P(A) + P(B) P(A B) 3. P(A B) P(A) + P(B) 4. P(A B C) = P(A) + P(A B) + P(A B C) 5. Note: book uses A c for A (complemet of set A). 6. Proofs: i book. Some will be doe i class. 7. Visualize: see Ve diagrams i book. 3
6 Coditioal Probability Give that we kow that a evet B has occurred, what is the probability that evet A occurred? Deoted by P(A B). Example: Roll of a 6-sided die. Give that the outcome is eve, what is the probability of a 6? Aswer: 1/3 Whe umber of outcomes is fiite ad all are equally likely, P(A B) = umber of elemets of A B umber of elemets of B (3) I geeral, P(A B) P(A B) P(B) (4) P(A B) is a probability law (satisfies axioms) o the uiverse B. Exercise: show this. Examples/applicatios Example 1.7, 1.8, 1.11 Costruct sequetial models: P(A B) = P(B)P(A B). Example: Radar detectio (Example 1.9). What is the probability of the aircraft ot preset ad radar registers it (false alarm)? See Fig. 1.9: Tree based sequetial descriptio 7 Total Probability ad Bayes Rule Total Probability Theorem: Let A 1,...A be disjoit evets which form a partitio of the sample space ( i=1 A i = Ω). The for ay evet B, Visualizatio ad proof: see Fig. 1.13 Example 1.13, 1.15 P(B) = P(A 1 B) +... P(A B) = P(A 1 )P(B A 1 ) +... P(A )P(B A ) (5) Bayes rule: Let A 1,...A be disjoit evets which form a partitio of the sample space. The for ay evet B, s.t. P(B) > 0, we have P(A i B) = P(A i)p(b A i ) P(B) Iferece usig Bayes rule = P(A i )P(B A i ) P(A 1 )P(B A 1 ) +... P(A )P(B A ) (6) There are multiple causes A 1, A 2,..A that result i a certai effect B. Give that we observe the effect B, what is the probability that the cause was A i? Aswer: use Bayes rule. See Fig. 1.14 Radar detectio: what is the probability of the aircraft beig preset give that the radar registers it? Example 1.16 False positive puzzle, Example 1.18: very iterestig! 4
8 Idepedece P(A B) = P(A) ad so P(A B) = P(B)P(A): the fact that B has occurred gives o iformatio about the probability of occurrece of A. Example: A= head i first coi toss, B = head i secod coi toss. Idepedece : DIFFERENT from mutually exclusive (disjoit) Evets A ad B are disjoit if P(A B) = 0: caot be idepedet if P(A) > 0 ad P(B) > 0. Example: A = head i a coi toss, B = tail i a coi toss Idepedece: a cocept for evets i a sequece. Idepedet evets with P(A) > 0, P(B) > 0 caot be disjoit Coditioal idepedece ** Idepedece of a collectio of evets P( i S A i ) = Π i S P(A i ) for every subset S of {1, 2,..} Reliability aalysis of complex systems: idepedece assumptio ofte simplifies calculatios Aalyze Fig. 1.15: what is P(system fails) of the system A B? Let p i = probability of success of compoet i. m compoets i series: P(system fails) = 1 p 1 p 2... p m (succeeds if all compoets succeed). m compoets i parallel: P(system fails) = (1 p 1 )...(1 p m ) (fails if all the compoets fail). Idepedet Beroulli trials ad Biomial probabilities A Beroulli trial: a coi toss (or ay experimet with two possible outcomes, e.g. it rais or does ot rai, bit values) Idepedet Beroulli trials: sequece of idepedet coi tosses Biomial: Give idepedet coi tosses,what is the probability of k heads (deoted p(k))? probability of ay oe sequece with k heads is p k (1 p) k umber of such sequeces (from coutig argumets): ( ) k p(k) = ( ) k p k (1 p) k, where ( ) k! ( k)!k! Applicatio: what is the probability that more tha c customers eed a iteret coectio at a give time? We kow that at a give time, the probability that ay oe customer eeds coectio is p. Aswer: p(k) k=c+1 5
9 Coutig Needed i may situatios. Two examples are: 1. Sample space has a fiite umber of equally likely outcomes (discrete uiform), compute probability of ay evet A. 2. Or compute the probability of a evet A which cosists of a fiite umber of equally likely outcomes each with probability p, e.g. probability of k heads i coi tosses. Coutig priciple (See Fig. 1.17): Cosider a process cosistig of r stages. If at stage 1, there are 1 possibilities, at stage 2, 2 possibilities ad so o, the the total umber of possibilities = 1 2... r. Example 1.26 (umber of possible telephoe umbers) Coutig priciple applies eve whe secod stage depeds o the first stage ad so o, Ex. 1.28 (o. of words with 4 distict letters) Applicatios: k-permutatios. distict objects, how may differet ways ca we pick k objects ad arrage them i a sequece? Use coutig priciple: choose first object i possible ways, secod oe i 1 ways ad so o. Total o. of ways: ( 1)...( k + 1) =! ( k)! If k =, the total o. of ways =! Example 1.28, 1.29 Applicatios: k-combiatios. Choice of k elemets out of a -elemet set without regard to order. Most commo example: There are people, how may differet ways ca we form a committee of k people? Here order of choosig the k members is ot importat. Deote aswer by ( ) k Note that selectig a k-permutatio is the same as first selectig a k-combiatio ad! the orderig the elemets (i k!) differet ways, i.e. ( k)! = ( ) k k! Thus ( ) k =! k!( k)!. How will you relate this to the biomial coefficiet (umber of ways to get k heads out of tosses)? Toss umber j = perso j, a head i a toss = the perso (toss umber) is i committee Applicatios: k-partitios. ** A combiatio is a partitio of a set ito two parts Partitio: give a -elemet set, cosider its partitio ito r subsets of size 1, 2,..., r where 1 + 2 +... r =. Use coutig priciple ad k-combiatios result. Form the first subset. Choose 1 elemets out of : ( ) 1 ways. Form secod subset. Choose 2 elemets out of 1 available elemets: ( 1 ) 2 ad so o. Total umber of ways to form the partitio: ( )( 1 ) ( 1 2... ( 1 2... r 1 ) ) r =! 1! 2!... r! 6