Some Basic Concepts of Probability and Information Theory: Pt. 1

Transcription

1 Some Basic Concepts of Probability and Information Theory: Pt. 1 PHYS 476Q - Southern Illinois University January 18, 2018 PHYS 476Q - Southern Illinois University Some Basic Concepts of Probability and Information Theory: January Pt. 1 18, / 14

2 Sample Spaces Any physical situation with an uncertain future can be thought of as an experiment, and the possible outcomes of a given experiment form a set Ω called sample space. Example Experiment: Consider flipping a two-sided circular coin whose mass is uniformly distributed. One side his heads (H) and the other is tails (T ). Sample space: Ω = {H, T }. Example Experiment: Consider flipping three such coins. Sample space: Ω = {(H, H, H), (H, H, T ), (H, T, H), (H, T, T ), (T, H, H), (T, H, T ), (T, T, H), (T, T, T )}. PHYS 476Q - Southern Illinois University Some Basic Concepts of Probability and Information Theory: January Pt. 1 18, / 14

3 Example: Seats at a Movie Theater Four friends go to the movies, two boys and two girls. They arrive late, and there are only four open seats left in the theater. (a) Consider a seating assignment experiment where the different outcomes correspond to different seating arrangements of the four people. How many outcomes are there to this experiment? PHYS 476Q - Southern Illinois University Some Basic Concepts of Probability and Information Theory: January Pt. 1 18, / 14

4 Example: Seats at a Movie Theater (b) Suppose the friends demand on sitting so that neither the two boys nor the two girls are sitting next to each other. How many different seating arrangements are there? (c) Two seats open up in the front of the theater and two of the friends will move to take them. How many different ways can the friends break into two groups of two? PHYS 476Q - Southern Illinois University Some Basic Concepts of Probability and Information Theory: January Pt. 1 18, / 14

5 Events In an experiment with sample space Ω, any subset of Ω is called an event. Every event corresponds to a collection of outcomes in the experiment. Example Experiment: Consider flipping two. Sample space: Ω = {(H, H), (H, T ), (T, H), (T, T )} The event E that the coins land the same way: E = {(H, H), (T, T )} Ω. For two events E 1 and E 2, their union E 1 E 2 consists of all outcomes that belong to either E 1 or E 2. Their intersection E 1 E 2 consists of all outcomes that belong to both E 1 and E 2. PHYS 476Q - Southern Illinois University Some Basic Concepts of Probability and Information Theory: January Pt. 1 18, / 14

6 Events Venn Diagram Example The power set 2 Ω is the collection of all subsets of Ω. Example If Ω = {H, T } then 2 Ω = {{H}, {T }, {H, T }, }. ( is the empty set) PHYS 476Q - Southern Illinois University Some Basic Concepts of Probability and Information Theory: January Pt. 1 18, / 14

7 Probability Distributions Definition A probability measure or probability distribution on some discrete sample space Ω is a function p : 2 Ω [0, 1] such that (i) Normalization: p(ω) = 1; (ii) Additivity: p(e 1 E 2 ) = p(e 1 ) + p(e 2 ) if E 1 and E 2 are two disjoint events. For an event E, we say that p(e) is the probability of event E occurring in an experiment with sample space Ω. When E = {ω} consists of just a single outcome, we write its probability simply as p(ω). The combination (Ω, p) of a sample space and a probability measure is called a probability space. In general, any non-negative additive function with domain 2 Ω and satisfying p( ) = 0 is called a measure on sample space Ω. PHYS 476Q - Southern Illinois University Some Basic Concepts of Probability and Information Theory: January Pt. 1 18, / 14

8 Example: The Monty Hall Problem Suppose you are a contestant on a game show that involves the following game of chance. There are three closed doors; one has a car behind it and the other two have goats behind them. You guess one of the doors, and the game show host opens up another door, one that has a goat behind it. Your are then given the chance to change your selection. Which of the following is true: (a) It is better to keep your door (b) It is better to choose the other door (c) It does not matter either way. PHYS 476Q - Southern Illinois University Some Basic Concepts of Probability and Information Theory: January Pt. 1 18, / 14

9 Example: The Monty Hall Problem Solution PHYS 476Q - Southern Illinois University Some Basic Concepts of Probability and Information Theory: January Pt. 1 18, / 14

10 Example: The Birthday Paradox How many people need to be attending a party so that it is more likely than not that at least two people have the same birthday? Solution PHYS 476Q - Southern Illinois University Some Basic Concepts of Probability and Information Theory: January Pt. 118, / 14

11 Random Variables Definition For a discrete probability space (Ω, p), a real-valued discrete random variable is a function X : Ω X R. Associated with every random variable is a probability space (X, p X ), where p X is the probability distribution given by p X (x) = p(x 1 (x)). We say that X = x with probability p X (x). Recall that for a general function f : X Y, its inverse f 1 is defined as f 1 (y) = {x X f (x) = y}. Example: If f (x) = x 2, then f 1 (4) = { 2, 2}. PHYS 476Q - Southern Illinois University Some Basic Concepts of Probability and Information Theory: January Pt. 118, / 14

12 Random Variables Example Let Ω be the sample space corresponding to three coin flips: Ω = {(H, H, H), (H, H, T ), (H, T, H), (H, T, T ), (T, H, H), (T, H, T ), (T, T, H), (T, T, T )}. Let X : Ω {0, 1, 2, 3} such that X (ω) is the number of Heads in ω. X ((H, H, H)) = 3, X ((H, H, T )) = 2,... X ((T, T, T )) = 0. Assuming uniform probability on the coin flips, the random variable X has distribution p X (0) = 1/8 p X (1) = 3/8 p X (2) = 3/8 p X (3) = 1/8. PHYS 476Q - Southern Illinois University Some Basic Concepts of Probability and Information Theory: January Pt. 118, / 14

13 Random Variables In practice, we do not need to have a specific experiment in time when working with random variables. Rather, we can just think of a random variable X as specifying some set X R and a distribution p X over X. Definition The expectation (or average) value of a real-valued discrete random variable X is given by E[X ] = xp X (x). x X The variance of a real-valued discrete random variable X is given by σ 2 (x) = E[X E[X ]] 2. PHYS 476Q - Southern Illinois University Some Basic Concepts of Probability and Information Theory: January Pt. 118, / 14

14 Random Variables Example Consider the previous random variable X on set X = {0, 1, 2, 3} with distribution p X (0) = 1/8 p X (1) = 3/8, p X (2) = 3/8 p X (3) = 1/8. PHYS 476Q - Southern Illinois University Some Basic Concepts of Probability and Information Theory: January Pt. 118, / 14