Chapter 1: PROBABILITY BASICS

Transcription

1 Charles Boncelet, obability, Statistics, and Random Signals," Oxford University ess, 0. ISBN: Chater : PROBABILITY BASICS Sections. What Is obability?. Exeriments, Outcomes, and Events. Venn Diagrams.4 Random Variables.5 Basic obability Rules. obability Formalized.7 Simle Theorems.8 Comound Exeriments.9 Indeendence.0 Examle: Can S Communicate With D?.0. List All Outcomes.0. obability of a Union.0. obability of the Comlement. Examle: Now Can S Communicate With D?.. A Big Table.. Break Into Pieces.. obability of the Comlement. Comutational ocedures Summary oblems Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 of ECE 800

2 An understanding of obability and Statistics is necessary in most if not all work related to science and engineering. Statistics: the study of and the dealing with data. obability: the study of the likeliness of result, action or event occurring. Often based on rior knowledge or the statistics of similar or ast events! Terms: Random Variables, Random ocesses or Stochastic ocesses For any measured henomenon there will be Uncertainty, Exected Variations, Randomness, or even Exected Errors included. when an outcome is non-deterministic where an exact value is subject to errors e.g. noise, measurement Easy examles of such henomenon include all games of chance Fliing coins, rolling dice, dealing cards, etc. Engineering Alications include Realistic signals with noise or characteristic unknown arts Signal-to-noise Ratios, Noise-Power Measurements, Background Noise Exected Values, Variances, Distributions Thermal Motion, Electron Movement Reliability, Quality, Failure Rates, etc. Thermal Motion, Electron Movement obability theory is necessary for engineering system modeling and simulations. unknown initial conditions (random) noisy measurements, exected inaccuracies, etc. during oeration Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 of ECE 800

3 Different kinds of obability Suggested that there are essentially 4 tyes obability by Intuition o Lucky Numbers obability as the Ratio of Favorable to Total Outcomes (Classical Theory) o Measured Statistical Exectations obability as a Measure of the Frequency of Outcomes obability Based on Axiomatic Theory Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 of ECE 800

4 Definitions of used in obability Exeriment An exeriment is some action that results in an outcome. A random exeriment is one in which the outcome is uncertain before the exeriment is erformed. Possible Outcomes A descrition of all ossible exerimental outcomes. The set of ossible outcomes may be discrete or form a continuum. Trials Event The single erformance of a well-defined exeriment. An elementary event is one for which there is only one outcome. A comosite event is one for which the desired result can be achieved in multile ways. Multile outcomes result in the event described. Equally Likely Events/Outcomes When the set of events or each of the ossible outcomes is equally likely to occur. A term that is used synonymously to equally likely outcomes is random. Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 4 of ECE 800

5 obability as the Ratio of Favorable to Total Outcomes (Classical Theory) Dice examle obability, Statistics and Random ocesses for Engineers, 4th ed., Henry Stark and John W. Woods, Pearson Education, Inc., 0. There are ossible outcomes An elemental event can be defined as the total of the two die The total number of outcome resulting in each unique event is known. The robability of each event can be comuted and described if the die are fair. So the true odds can be comuted and a gambling game with skewed odds in the houses favor can be created from: htts://en.wikiedia.org/wiki/cras Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 5 of ECE 800

6 obability as the Ratio of Favorable to Total Outcomes (Classical Theory) fliing two coins examle Fli two coins: What are the ossible outcomes {HH, HT, TH, TT} Define an event as the getting of getting at least one Tail. obability is the favorable outcomes/total outcomes, = /4 Possible Outcomes with obabilities: HH robability /4 HT robability /4 TH robability /4 TT robability ¼ Possible events: one head, one tail, at least one head, at least one tail, at most one head, at most one tail, two heads, two tails no heads, or no tails. Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 of ECE 800

7 obability as a Measure of the Frequency of Outcomes Exeriment: Selecting a sequence of random numbers. The random numbers are between and 00. Determining the relative frequency of a single number as from 0 to 0,000 numbers are selected. The statistics of observed events is aroaching /00. (infinite trials?) Figure.- Event = {occurrence of number 5} (Numbers derived from website RANDOM.ORG). Figure.- Event = {occurrence of number } obability, Statistics and Random ocesses for Engineers, 4th ed., Henry Stark and John W. Woods, Pearson Education, Inc., 0. Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 7 of ECE 800

8 obability Based on an Axiomatic Theory Develo the coherent mathematical theory: Statistics collected data on random exeriments o Possible outcomes, samle sace, events, etc. From the statistics, robability structure can be observed and defined o Random rocesses follow defined robabilistic models of erformance. Mathematical roerties alied to robability derives derive new/alternate exectations o obabilistic exectations can be verified by statistical measurement. This can be considered as modeling a system rior to or instead of erforming an exeriment. Note that the results are only as good as the model or theory match the actual exeriment. Misuses, Miscalculations, and Paradoxes in obability Old time quotation There are three kinds of lies: lies, damned lies, and statistics! htts://en.wikiedia.org/wiki/lies,_damned_lies,_and_statistics From the CNN headlines Math is racist: How data is driving inequality, by Aimee Rawlins, Setember, 0 htt://money.cnn.com/0/09/0/technology/weaons-of-math-destruction/index.html Another examle As a erson, you are a unique individual and not a statistical robability but future chances may be based on others like you that have come before. The class as a whole may exhibit statistical exectations although it is made u of unique individuals. For Sci-Fi readers. Issac Asimov s Foundation Trilogy sychohistory used to redict the future. Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 8 of ECE 800

9 Sets, Fields and Events Concetually Defining a oblem Relative Frequency Aroach (statistics) Set Theory Aroach (formal math) Venn Diagrams (ictures based on set theory) If you like ictures try to use Venn Diagrams to hel understand the concets. Figure.4- Venn diagrams for set oerations. obability, Statistics and Random ocesses for Engineers, 4th ed., Henry Stark and John W. Woods, Pearson Education, Inc., 0. Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 9 of ECE 800

10 Set Theory Definitions (A review?!) Set Subset Sace A collection of objects known as elements A a, a,, a n The set whose elements are all members of another set (usually larger but ossible the same size). B a, a,, a n k therefore B A The set containing the largest number of elements or all elements from all the subsets of interest. For robability, the set containing the event descrition of all ossible exerimental outcomes. A i S, for all i subsets Null Set or Emty Set The set containing no elements A Venn Diagrams can hel when considering set theory A grahical (geometric) reresentation of sets that can rovide a way to visualize set theory and robability concets and can lead to an understanding of the related mathematical concets. from: Robert M. Gray and Lee D. Davisson, An Introduction to Statistical Signal ocessing, Cambridge University ess, 004. A df file version can be found at htt://www-ee.stanford.edu/~gray/s.html Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 0 of ECE 800

11 Equality More Set Theory Definitions Set A equals set B if and only if (iff) every element of A is an element of B AND every element of B is an element of A. A B iff A B and B A Sum or Union (logic OR function) The sum or union of sets results in a set that contains all of the elements that are elements of every set being summed. Laws for Unions S A A A A B B A A A A A A A S S A B A, if B A oducts or Intersection (logic AND function) The roduct or intersection of sets results in a set that contains all of the elements that are resent in every one of the sets. Laws for Intersections S A B B A A A A A A S A A B B, if B A Mutually Exclusive or Disjoint Sets Mutually exclusive or disjoint sets of no elements in common. A B NOTE: The intersection of two disjoint sets is a set the null set! A N Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 of ECE 800

12 Comlement The comlement of a set is the set containing all elements in the sace that are not elements of the set. Laws for Comlement A A and A A S S S A A A B, if B A A B, if B A DeMorgan s Law A B A B A B A B Differences The difference of two sets, A-B, is the set containing the elements of A that are not elements of B. Laws for Differences A B A B A A B B B A A A A A A A A A A S S A A A B Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 of ECE 800

13 Venn Diagram set theory concets. (D ictures that can hel you understand set theory) from: Robert M. Gray and Lee D. Davisson, An Introduction to Statistical Signal ocessing, Cambridge University ess, 004. Pdf file version found at htt://www-ee.stanford.edu/~gray/s.html (a) The sace (b) Subset G (c) Subset F (d) The Comlement of F (e) Intersection of F and G (f) Union of F and G Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 of ECE 800

14 More Venn Diagrams from: Robert M. Gray and Lee D. Davisson, An Introduction to Statistical Signal ocessing, Cambridge University ess, 004. Pdf file version found at htt://www-ee.stanford.edu/~gray/s.html (a) Difference F-G (b) Difference F-G Union with Difference G-F F G G F If events can be describe in set theory or Venn Diagrams, then robability can directly use the concets and results of set theory! What can be said about F G? [ read as the robability of event F union event G ] F G F G G F G F F G F G Therefore, F G F G F G Set algebra is often used to hel define robabilities Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 4 of ECE 800

15 Equalities in Set Algebra from: Robert M. Gray and Lee D. Davisson, An Introduction to Statistical Signal ocessing, Cambridge University ess, 004. Aendix A, Set Theory. Pdf file version found at htt://www-ee.stanford.edu/~gray/s.html Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 5 of ECE 800

16 Axiomatic Definitions Using Sets for obability For event A 0 A (Axiom &, Theorem.) S (Axiom ) 0 (Theroem.) Disjoint Sets If A B A B A B, then Comlement (comlementary sets) (defining the comlement may be easier sometimes) If A A A A S A A, then A A (Theorem.) Not a Disjoint Sets (solution) : (Theorem.5) If A B, then A B??? Maniulation () B A A B, the union of disjoint sets A B A A B A A B A Maniulation () Maniulation () Substitution for () B A B A B, the union of disjoint sets B A B A B A B A B A B B A B, rearranging from () A B A A B A B A B Note that we can generally define a bound where A B A B A B A B equality holds for A and B being disjoint sets! Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 of ECE 800

17 Examle: -sided die A: The robability of rolling a or a, event A, i A B: The robability of rolling a or 5, event B,5 5 B 5 C: The robability of event A or event B, event C A B C A B A B A B Note: C A B,,5 C C 5 When in doubt, write it out to double check your results! A Venn diagram may also hel. Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 7 of ECE 800

18 obability of A union of events (Theorem.: Inclusion-Exclusion Formula) An extension of the set theory for unions. obability, Statistics and Random ocesses for Engineers, 4th ed., Henry Stark and John W. Woods, Pearson Education, Inc., 0. Figure.5- Partitioning into seven disjoint regions Δ,...,Δ7.) 7 E i i j j If A B A B A B, what about A B C??? A B C A B C A B A C B C A B C E E E E i i i ji i ji E i k j E j E i E j E k Can you recognize a attern + singles - doubles + triles - quads etc A B C D E F What about??? Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 8 of ECE 800

19 obability, the relative frequency method: More Definitions The number of trials and the number of times an event occurs can be described as N N A N B N C the relative frequency is then r A N N A note that N N N A N B N N C r A rb rc When exerimental results aear with statistical regularity, the relative frequency tends to aroach the robability of the event. and Where A lim r N A A B C A is defined as the robability of event A. Mathematical definition of robability:. 0 A. A B C, for mutually exclusive events. An imossible event, A, can be reresented as 0 4. A certain event, A, can be reresented as A. A. Odds or robabilities can be assigned to every ossible outcome of a future trial, exeriment, contests, game that has some rior historical basis of events or outcomes. Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 9 of ECE 800

20 obability Formalized Axiom : obability defined between 0 and such that A 0 Axiom : obability for all events in the samle sace is.0 Axiom : obability of disjoint sets S S A A A A n and A i A, for i j j and total robability would say Where.0 A is defined as the robability of event A. Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 0 of ECE 800

21 Joint obability Comound Exeriments Defining robability based on multile events two classes for considerations. Indeendent exeriments: The outcome of one exeriment is not affected by ast or future exeriments. o fliing coins o reeating an exeriment after initial conditions have been restored o Note: these roblems are tyically easier to solve Deendent exeriments: The result of each subsequent exeriment is affected by the results of revious exeriments. o drawing cards from a deck of cards o drawing straws o selecting names from a hat o for each subsequent exeriment, the revious results change the ossible outcomes for the next event. o Note: these roblems can be very difficult to solve (the next exeriment changes based on revious outcomes!) Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 of ECE 800

22 Indeendence Two events, A and B, are indeendent if and only if A B A B Indeendence is tyically assumed when there is no aarent hysical mechanism by which the two events could deend on each other. For events derived from indeendent elemental events, their indeendence may not be obvious but may be able to be derived. Indeendence can be extended to more than two events, for examle three, A, B, and C. The conditions for indeendence of three events is A B A B B C B C A C A C A B C A B C Note that it is not sufficient to establish air-wise indeendence; the entire set of equations is required. For multile events, every set of events from n down must be verified. This imlies that n n equations must be verified for n indeendent events. Imortant oerties of Indeendence Unions hel in simlifying the intersection term if events are indeendent! Indeendent intersection with a Union A B A B A B A B A B A B A B C A B C There will be some examle roblems where you must determine if events are indeendent in order to solve the roblem. switch roblems in homework and skills Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 of ECE 800

23 Total obability For a sace, S, that consists of multile mutually exclusive events, the robability of a random event, B, occurring in sace S, can be described based on the conditional robabilities associated with each of the ossible events. oof: S A A A A n and A i A, for i j j B B S B A A A A B A B A B A B n A n B B A B A B A B A n But B A B A A, for A 0 i i i i Therefore B B A A B A A B A n A n Remember your math roerties: distributive, associative, commutative etc. alied to set theory. Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 of ECE 800

24 .0 Examle: Can S Communicate with D? (Switch roblems) Three links. Figure.4: Two-ath Network If Link is functional, communications occurs. If both Link and Link are functional communication occurs. If there are robabilities or likelihoods that Link,, and are oeration, what is the robability you can communicate? Note: Comm L L L, for all links indeendent! Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 4 of ECE 800

25 Consider three indeendent links and there robabilities Basic robability 000 L 0 L 0 L L 0 L 0 L L 0 L L, 0 L L 0 L 0 Comm S D L L Comm L, for all links indeendent! S D 0,00,0,0, S D 0,00,0,0, S D S D Alternate Derivation Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 5 of ECE 800

26 Comm S D L L L, for all links indeendent! S D L L L L L L L L L L L L L L L S D L L L L L L So do you refer using union and intersection concets? Figure.4: Two-ath Network Alying negative logic and S D s D S D L L L S D L L L L L L L L S D L L L S D L L L S D L L L s D S D Both negative and ositive logic aroaches should work! Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 of ECE 800

27 A Big Table Breaking it down into smaller ieces. Comm S D A B C S D A B C A B C A B A C B C A B C A B C A B A C B C A B C Alternately consider S D A BC A B C Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 7 of ECE 800

28 Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 8 of ECE 800 Then C B A D S Comm A B A Alternately consider 5 4 C B A C B A D S And D S D S Sorry, I like doing things the easiest way

29 Now consider unequal robabilities Remember the initial derivation? Comm S D L L L, for all links indeendent! S D L L L L L L L L L L L L L L L And finally, S D L L L L L L Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 9 of ECE 800

30 Exeriment : A bag of marbles, draw A bag of marbles: -blue, -red, one-yellow Objects: Marbles Attributes: Color (Blue, Red, Yellow) Exeriment: Draw one marble, with relacement Samle Sace: {B, R, Y} obability (relative frequency method) The robability for each ossible event in the samle sace is. Event obability Blue / Red / Yellow / Total / This exeriment would be easy to run and verify after lots of trials. see Matlab Sec_Marble.m ntrials = vs. 00 vs. 000 (reeat execution a few times) (Another roblem: if we ran trials, what is the robability that we get events that exactly match the robability? -Blue, -Red, Yellow - a much harder roblem) Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 0 of ECE 800

31 Exeriment : A bag of marbles, draw Exeriment: Draw one marble, relace, draw a second marble. with relacement Samle Sace: {BB, BR, BY, RR, RB, RY, YB, YR, YY} Define the robability of each event in the samle sace. Joint obability When a desired outcome consists of multile events. (Read the joint robability of events A and B). A, B Statistically Indeendent Events When the robability of an event does not deend uon any other rior events. If trials are erformed with relacement and/or the initial conditions are restored, you exect trial outcomes to be indeendent. A, B B, A A B Therefore The marginal robability of each event is not affected by rior/other events. The robability of event A given event B occurred is the same as the robability of event A and vice versa. A B A B A B and Alicable for multile objects with single attributes and with relacement. st-rows\ nd -col Blue Red Yellow Blue 9 Red 4 Yellow Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 of ECE 800

32 Next Concet Conditional obability When the robability of an event deends uon rior events. If trials are erformed without relacement and/or the initial conditions are not restored, you exect trial outcomes to be deendent on rior results or conditions. A B A when A follows B The joint robability is. A, B B, A A B B B A A Alicable for objects that have multile attributes and/or for trials erformed without relacement. Exeriment : A bag of marbles, draw without relacement Exeriment: Draw two marbles, without relacement Samle Sace: {BB, BR, BY, RR, RB, RY, YB, YR} Therefore st-rows \ nd -col Blue Red Yellow nd Marble Blue Red Yellow st Marble Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 of ECE 800

33 Matlab Marble Simulation Examles: Sec_Marble.m examle to show small versus large number of samle statistics vs. robability Sec_Marble.m examle to validate robability and/or small versus large number of trials Sec_Marble.m examle to validate robability and/or small versus large number of trials Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,, obability, Statistics, and Random Signals, Oxford University ess, February 0. B.J. Bazuin, Sring 08 of ECE 800