Department of Electrical Engineering University of Arkansas ELEG 3143 Probability & Stochastic Process Ch. 1 Experiments, Models, and Probabilities Dr. Jing Yang jingyang@uark.edu
OUTLINE 2 Applications (Elementary) Set Theory Probability Conditional Probability Law of Total Probability and ayes Theorem Independence
APPLICATIONS 3 Why probability? Most real world events have uncertain (or random) outcomes Flipping a coin Football games Lifetime of IPAD/computer The actual resistance of 1k Ohm resistor How do we characterize these random events? How do we predict the outcome of a random event? Probability theory Provides a complete set of mathematical tools and theories that can accurately and precisely describe the statistical behaviors of uncertain phenomena. It is a branch of mathematics It has a wide range of applications to Engineers.
APPLICATIONS 4 Random input signals The input for a system might be random Random system characteristics The system itself has random characteristics E.g. The components inside a system has random values A 1k Ohm resistor might have an actual value of 1.01k Ohm E.g. Noise: random electrical disturbance caused by the random movement (thermal motion) of electrons It is present at all electrical systems Electrical System input output
APPLICATIONS 5 System reliability What is the expected life cycle of a given system? What is the probability of failure of a system? Warranty duration, insurance policy, etc. Random sampling It might be too costly to inspect every single elements Only sample a small population at random, then deduce the general behavior from the sample results E.g. product inspection for quality control How to design the random sampling process? Computer simulation A low-cost and efficient way to test the performance of a system Random inputs, random system characteristics (e.g. random component value, random disturbance, )
OUTLINE 6 Applications (Elementary) Set Theory Probability Conditional Probability Law of Total Probability and ayes Theorem Independence
SET THEORY 7 Set A collection of things (elements) Example: Capital letter A denotes a set Small letter x is an element of set A: x A c is not an element of set A: c A How to define a set? Name the elements A { x, y, z} Give a rule C { x 2 A { x, y, z} x 1,2,3,4,5} D { x x 1,2,3 } A set can have an infinite number of elements. 2
SET THEORY 8 Subset A is a subset of if every member of A is also a member of Mathematical notation: C is a subset of D: C D Any set is a subset of itself Null (Empty) set A set that has no element Null set is a subset of any set: A, for any A Universal set S This set includes all things of interest for a given application E.g. square of positive integer numbers E.g. flip a coin, S ={head (h), tail (t)} E.g. throw a die, S = {1, 2, 3, 4, 5, 6}
SET THEORY: VENN DIAGRAM 9 Venn diagram A geometric representation to display the relationship among sets, where the universal set S is represented by a large rectangle and the sets are represented by closed surfaces inside the rectangle. E.g. C A S C A
SET THEORY: OPERATIONS 10 Union The union of two sets A and is a set consisting of all the elements from either A or or both Denoted as A ( A union ) Corresponds to the logic OR operation A S S A A
SET THEORY: OPERATIONS 11 Intersection The intersection of two sets A and is the set consisting of all elements that are contained in both A and. Denoted as A ( A intersect ) Corresponds to the logic AND operation
SET THEORY: OPERATIONS 12 Complement The complement of set A is a set containing all the elements of S that are not in A. Denoted as Ā or A c ( A complement / not A ) Properties S A A S (A) A A A A
SET THEORY: OPERATIONS 13 Collectively exhaustive sets A collection of sets A 1,, A n is collectively exhaustive if A 1 A 2 A n = S A shorthand for the unions for n sets A i = A 1 A 2 A n n i=1
SET THEORY: OPERATIONS 14 Mutually exclusive sets Two sets A and are mutually exclusive (disjoint) if A A collection of sets A 1,, A n is mutually exclusive if A i A j = φ, i j A S A shorthand for the intersections for n sets A i = A 1 A 2 A n n i=1 Partition A collection of sets is a partition if it is both mutually exclusive and collectively exhaustive.
SET THEORY: OPERATIONS 15 DeMorgan s Laws c c c A A S S S A A A
SET THEORY: OPERATIONS 18
OUTLINE 19 Applications (Elementary) Set Theory Probability Conditional Probability Law of Total Probability and ayes Theorem Independence
PROAILITY 20 Experiment: Consists of the procedure, observation and model E.g. Procedure: flip a coin and let it land on a table Observation: observe head or tail faces you after the coin lands Model: Head and tail are equally likely. The result of each flip is unrelated to previous flips. Two experiments with the same procedure but with different observations are different E.g. Flip a coin three times and observe the sequence of heads and tails Outcome Flip a coin three times and observe the number of heads An outcome of an experiment is any possible observation of that experiment. A random experiment has uncertain outcomes before the experiment is performed.
PROAILITY 21 Sample space S The sample space of a random experiment is the finest-grain, mutually exclusive, collectively exhaustive set of all possible outcomes. Finest-grain: all possible distinguishable outcomes are identified separately. Mutually exclusive: if one outcome occurs, no other outcome also occurs. Collectively exhaustive: every possible outcome should be included. It is the universal set of all possible outcomes An outcome is an element in the sample space
PROAILITY 22 Sample space S Examples Flip a coin S = {h, t} Flip a coin three times and observe the sequence of heads and tails S = {hhh, hht, hth, htt, thh, tht, tth, ttt} Flip a coin three times and observe the number of heads S = {0, 1, 2, 3} Toss a die S = {1, 2, 3, 4, 5, 6} Test an integrated circuit to determine if it meets quality objectives S={accepted, rejected} Lifetime of a car S = [0, )
PROAILITY 23 Event An event is a set of outcomes of an experiment An event is a subset of the sample space S E.g. throwing a six-sided die E1 = {3}: the event that 3 appears E2 = {2, 4, 6}: the event that an even number appears. E.g. Testing a short circuit. Red light to indicate there is a short circuit and green light to indicate there is no short circuit. Consider an experiment of a sequence of three tests. An outcome of the experiment is a sequence of red and green lights. Denote each outcome by a three-letter word such as rgr. the event that light 2 is red : R2 ={grg, grr, rrg, rrr} the event that light 2 is green: G2 ={rgr, rgg, ggr, ggg}
PROAILITY 24 Event space A collectively exhaustive, mutually exclusive set of events Different from sample space in finest-grain property E.g. Flip a coin four times and observe the sequence of heads and tails. What is the sample space? How many elements are in the sample space? Let i = {outcomes with i heads}. Each i is an event containing one or more outcomes. The event space ={0, 1, 2, 3, 4} E.g.. Testing a short circuit three times. event space ={R2, G2}
PROAILITY 25 Theorem for event space For an event space = { 1, 2, }, and for any event A in the sample space, let C i = A i. For i j, the event C i and C j are mutually exclusive and A = C 1 C 2 Example: Test a circuit three times. Event space ={R2, G2} R2 ={grg, grr, rrg, rrr}, G2 ={rgr, rgg, ggr, ggg} A = rgg, grg, ggr : event that only one red light in the sequence C 1 = A R 2 = C 2 = A G 2 = A = C 1 C 2 grg rgg, ggr
PROAILITY 26 Theorem for event space Example: Flip a coin four times. The event space ={0, 1, 2, 3, 4}. A is the set of outcomes with less than three heads. A = tttt, httt, thtt, ttht, ttth, hhtt, htht, htth, thht, thth, tthh C 0 = A 0 = C 1 = A 1 =
PROAILITY 27 Probability Assign a number to each event in the sample space, such that the number is a measure of how likely the event is. E.g. flipping a coin. Assign 0.5 to head (H), 0.5 to tail (T) 0.5 is the probability of H, 0.5 is the probability of T E.g. tossing a die with event {1, 2} (2 or less) 1/3 is the probability of 2 or less Roughly speaking, the probability of an event is the proportion of the time that event is observed in a large number of runs of the experiment Relative-frequency view Mathematically expressed in axioms
AXIOMS OF PROAILITY 30 Definition: Probability Consider a (random) experiment with sample space S. For each event A of the sample space S, we assign it a real number P[A], which satisfies the following properties 1. 0 P[ A] 1 2. P[ S] 1 3. For any countable collection A 1, A2, of mutually exclusive events ( A A, i j) i j P n1 A n n1 P[ A A probability measure P[.] is a function that maps events in the sample space to real numbers. Then P[A] is called as the probability of event A. n ] A set is called "countably infinite" if it has one-to-one correspondence with the natural number set, N.
PROAILITY 31 Probability of the union of finite mutually exclusive events m i=1 If A = A 1 A m and A i A j = φ for i j, then P A = P[A i ] Probability of complement events Events A and A are always mutually exclusive. Then P[φ]=0 P[ A] P[ A] 1 Probability of the union of two events Consider a sample space S with two events A and. The probability that either A or happens (the probability of all outcomes either in A or ) P[ A ] P[ A] P[ ] P[ A] P[ A ] P[ A] P[ ] Notation: The probability that both A and happen (the probability of all outcomes in both A and ) is denoted as P[A ]=P A = P[A, ] Probabilities of A and if A : Monotonicity P[A] P[]
PROAILITY 32 Probability of an event = {s 1, s 2,, s m } s 1, s 2,, s m are the outcomes. m P = P[{s i }] i=1 Probability of an event A with an event space { 1, 2,, m } m P A = P[A i ] i=1
PROAILITY 33 Equally likely outcomes For an experiment with a sample space S = {s 1, s 2,, s n }, no one outcome is any more likely than any other, that is, n outcomes are equally likely. E.g. Tossing a fair coin For each equally likely outcome s i in an experiment, P s i = 1 n For an event = {s 1,, s m } in such an experiment, P = m n 1 i n 1 m n Example (Probability calculation ) Toss 2 fair coins Sample space S = { h, h, h, t, (t, h) (t, t)} E = {(h, h), (h, t)}, F = {(h, h), (t, h)}, G={(t, t)} P[ E G] P[ E F] P[ E F G]
PROAILITY 34 Example Monitor a call at Skype. Classify the call as an audio call (A), or a video call (V) depending on whether people are seeing each other during the call. Classify the call as long (L) if the call lasts more than 3 minutes; otherwise, the call as brief (). ased on data collected by Skype, we use the following probability model: P[A]=0.7, P[L]=0.6, P[AL]=0.35. What is the sample space? What is the probability of the sample space? Is {A, V} an event space? Is {L, } an event space? What is the probability of the union of all events in an event space? Find the following probabilities: 1. P V 2. P[] 3. P A, P V, P VL (draw a table to show the probability of each outcome) 4. P V L 5. P A L 6. P A V 7. P[L]
OUTLINE 35 Applications (Elementary) Set Theory Probability Conditional Probability Law of Total Probability and ayes Theorem Independence
CONDITIONAL PROAILITY 36 Example Consider an urn contains 100 resistors of different resistance and power ratings. The number of the different types are listed as follows Pick 1 resistor at random 1 Ohm 10 Ohm 100 Ohm Totals 1 W 10 20 30 60 10 W 10 20 10 40 Totals 20 40 40 100 What is the probability that the resistor has a power rating of 1 W? What is the probability that the resistor has a resistance of 10 Ohm? What is the probability that the resistor is 10 Ohm with 1 W rating?
CONDITIONAL PROAILITY 37 Conditional probability P[ A ] P[ A] P[ ] P[ A] P[ A] P[ A] Properties of conditional probability Conditional probability is still a probability 0 P[A ] 1 P = 1 P[ A] P[ A] P[ A] P[ ] P[ A ] If A = A 1 A 2 with A i A j = φ for i j, then P A = P A 1 + P A 2 +
CONDITIONAL PROAILITY 38 Conditional probability P[A ]: Probability of A given, i.e. given the condition that the event occurred, the probability that A occurs Example: (Cont d from the previous example) Pick one resistor. If the picked resistor has a 1W rating, what is the probability that the resistor is 10 Ohm? 1 Ohm 10 Ohm 100 Ohm Totals 1 W 10 20 30 60 10 W 10 20 10 40 Totals 20 40 40 100 P[10 Ohm 1W] = P[10Ohm, 1W] =P[10 Ohm 1W]P[1W]
CONDITIONAL PROAILITY 39 Example Roll two fair four-sided dice. Let X1 and X2 denote the number of dots that appear on die 1 and die 2. Let A be the event X1 2. What is P[A]? Let denote the event X2 > X1. What is P[]? What is P[A ]?
OUTLINE 41 Applications (Elementary) Set Theory Probability Conditional Probability Law of Total Probability and ayes Theorem Independence
LAW OF TOTAL PROAILITY & AYES THEOREM Law of total probability If we know P(A i) (i=1,2,,n), how do we find out P(A)? Conditional probability Unconditional probability? Suppose are mutually exclusive events and That is, we have an event space with for all i. Then, 42 ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ 2 2 1 1 2 1 n n n P A P P A P P A P A P A P A P A P n,, 2, 1. 2 1 S n n i i i P A P A P 1 ] [ ] [ ] [ },,, { 2 1 n 0 ] [ i P
LAW OF TOTAL PROAILITY & AYES THEOREM 43 Example A company has three machines 1, 2 and 3 for making 1kΩ resistors. It has been observed that 80% of resistors produced by 1 are within 50Ω of the nominal value. Machine 2 produces 90% of resistors within 50Ω of the nominal value. The percentage of machine 3 is 60%. Each hour, machine 1 produces 3000 resistors, 2 produces 4000 resistors, and 3 produces 3000 resistors. All of the resistors are mixed together at random in one bin and packed for shipment. What is the probability that the company ships a resistor that is within 50Ω of the nominal value.
LAW OF TOTAL PROAILITY & AYES THEOREM 45 ayes theorem P[A ] P[ A]? for P[A]>0, P[ P[ A ] P[ ] A] P[ A] Proof
LAW OF TOTAL PROAILITY & AYES THEOREM ayes theorem P[A i], P[i] P[i A]? Suppose are mutually exclusive events and That is, we have an event space. Then, Proof 46 n k k k i i i i i P A P P A P A P P A P A P 1 ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ n,, 2, 1. 2 1 S n },,, { 2 1 n
LAW OF TOTAL PROAILITY & AYES THEOREM 47 Example For the previous example about the resistors from a factory, we can learn that The probability that a resistor is from machine 3 is P[3]=0.3; The probability that a resistor is acceptable is P[A]=0.78; Given that a resistor is from machine 3, the conditional probability that it is acceptable is P[A 3]=0.6. What is the probability that an acceptable resistor comes from machine 3?
LAW OF TOTAL PROAILITY & AYES THEOREM 49 Example A lab blood test to detect a certain disease. If the disease is present, the test can detect it 95% of the time. However, the test also gives a false positive result for 1% of the healthy person being tested (that is, if a healthy person is tested, then, with probability 0.01, the result will imply the person has the disease). Assume 0.5% of the population actually has the disease. If the test result of a person is positive, what is the probability that the person actually has the disease?
OUTLINE 50 Applications (Elementary) Set Theory Probability Conditional Probability Law of Total Probability and ayes Theorem Independence
INDEPENDENCE 51 Independence Two events, A and, are independent, if and only if P[ A] P[ A] P[ ] If A and are independent P[ A ] P[ A] P[ A] P[ A]
INDEPENDENCE 52 Independence Independent mutually exclusive (disjoint) A and are independent P[ A ] P[ A] P[ ] A and are mutually exclusive P[ A ] P[ A] P[ ] P[ A ] In most situations, independent events are not mutually exclusive. Exceptions occur only when P[A] = 0 or P[] = 0.
INDEPENDENCE 53 Example A short-circuit tester has a red light to indicate there is a short circuit and a green light to indicate there is no short circuit. Consider an experiment consisting of a sequence of three tests. Assume the results for those three tests are independent with each other. Denote the event that the second light is red by R2 and denote the event that the second light is green by G2. Are R2 and G2 independent? Are R2 and G2 mutually exclusive?
INDEPENDENCE 54 Example continued Denote the event that the first light is red by R1. Are R1 and R2 independent? Are R1 and R2 mutually exclusive?
INDEPENDENCE 56 Independence Three events E, E E are (mutually) independent if and only if 1 2, Any two events are independent, i.e. E1 and E2 are independent, E1 and E3 are independent, E2 and E3 are independent; P[ E1E2 E3] P[ E1] P[ E2] P[ E3] Pairwise independence A sequence of events E, E, are called pairwise independent if any 1 2, E n pair of events are independent P[ EiE j ] P[ Ei ] P[ E j ] i j Example Pairwise independent <=> Independent? Let a ball be drawn from an urn containing 4 balls, numbered 1, 2, 3, 4. Let E = {1, 2}, F = {1, 3}, G = {1, 4}. Are E, F, G pairwise independent? Are E, F, G independent? Let E = {1, 3, 4}, F = {2, 3, 4}, G =φ. Are E, F, G pairwise independent? Are E, F, G independent? 3
INDEPENDENCE 57 Independence If n 3, E, E, 1 2, E n are independent if and only if Any n-1 events in the sequence are independent P E E E ] P[ E ] P[ E ] P[ E ] [ 1 2 n 1 2 n Example: E, E 1 E2, E3, 4 When n>2, it is complex to determine whether n events are mutually independent. However, if n events are mutually independent, the probability of the intersection of any subset of the n events is simply the product of the probabilities of these events.