Lecture 10: Probability distributions TUESDAY, FEBRUARY 19, 2019

Transcription

1 Lecture 10: Probability distributions DANIEL WELLER TUESDAY, FEBRUARY 19, 2019

2 Agenda What is probability? (again) Describing probabilities (distributions) Understanding probabilities (expectation) Partial information (conditional probability) Probability distributions are useful for describing and predicting how likely random events are to occur. Histogram plots like the red, green, and blue color plots for the image above provide empirical probability distributions for real world signals such as digital images. Image credit: Daniel Schwen/Wikipedia. 2

3 What is probability? Recall: Probability describes the likelihood of a random event occurring. Likelihood: chance or relative frequency of occurrence Random event: some set of possible outcomes, of a trial or experiment, determined by chance. Outcomes are fairly arbitrary objects (e.g., heads or tails, dice roll, playing card) drawn from a set called the sample space. Unfortunately, such generality is also inconvenient mathematics works best with actual numbers. How do we transform coin flips, die rolls, card draws, etc., into numbers? Solution: define a random variable. 3

4 Random variables A random variable assigns a real number to any possible outcome of a trial or experiment. E.g.: heads = 0, tails = 1 Random variables allow us to translate general ideas about probability to numeric concepts: Finite or discrete sample space finite or discrete value range of random variable Continuous sample space continuous value range of random variable Events as sets of outcomes events as intervals, or collections of intervals, on real number line Probability function on events probability function on intervals, or ranges, on real line 4

5 Random variables Example: Define a random variable for a birthdate (discrete). Example: Define a random variable for a temperature (continuous). Image credits: (top) iambaker.net, (bottom) homesciencetools.com. 5

6 Probability functions We already have described probabilities as assigned to events, satisfying important axioms: 1) A probability of an event is a nonnegative real number between 0 and 1. (higher = more likely) 2) The probability of the sample space P(Ω) = 1. 3) The probability of any collection of mutually exclusive events occurring is the sum of the probabilities of the individual events: P ራ E n n = P E n n This sum may be countable or uncountable. This U operator is called the union of sets. Today: Let us translate these probability axioms from outcomes to random variables, from events to intervals. Instead of talking about P(heads) or P(two coin flips the same), we will talk about P(X > 1) or P(X 3). This leads to the definition of a cumulative distribution function, or cdf, defined on the real numbers. P X x = F X x 6

7 Probability functions Recall: the probability function is defined on a collection of events. This collection has certain requirements: 1) 2) 3) It turns out this is not enough to guarantee we can talk about probabilities for random variables. To do that, we would need what s called a Borel σ-algebra and a way of constructing set theoretic topologies containing minimal closed collections of open sets. Instead, what we will do is work backwards and assume that any interval of random variable X we want to measure has a well-defined event. It turns out topological theory would lead to this anyway. 7

8 Cumulative distribution functions Cumulative distribution functions F X (x) have three important properties: 1) The function is non-decreasing: if a > b, then F X (a) F X (b). 2) The function is right-continuous: lim x a + F X(x) = F X (a). 3) The function increases from 0 to 1: lim x F X(x) = 0, lim x + F X(x) = 1. Three cases of the cdf, for discrete, continuous, and hybrid (mixed) random variables: Discrete: jumps at discrete values, constant between them Continuous: continuous function, no jumps Hybrid: jumps at discrete values, continuous (but not necessarily constant) between them Image credit: Incnis Mrsi/Wikipedia. 8

9 Cumulative distribution functions Example: temperature range in the classroom Example: sum of values from a roll of a pair of fair 6-sided dice 9

10 Cumulative distribution function Why cumulative distribution function versus probability function? We said that random variables simplify analysis by describing probability outcomes with numbers. The cdf allows us to determine probabilities for these random variable representations. What kinds of intervals can we describe? P(X x) interval closed at (contains) maximum P(X > x) interval open at (does not contain) minimum Empty interval (empty set) and full range (sample space) Unions, intersections of these: P(a < X b), P(X a or X > b), For a discrete value, we can also define P(X = a), as the value of that jump: P X = a = F X a lim F x a For continuous values, this P(X = a) equals zero! Does this mean X x that continuous-valued random variables never occur? No, it just means we would need a different way to interpret them. We will focus on discrete-valued random variables in this course. 10

11 Discrete probability distributions For random variables with discrete values, we can use P(X = x) to describe how relatively likely different random variable values, and their associated outcomes, are to occur. This description is called a probability mass function (pmf). Note: pmf s do not apply to continuous random variables. Discrete probability distributions provide common language for describing these pmf s: Bernoulli(p) distribution: two values (usually 0 and 1), with P(X = 1) = p. Binomial(N,p) distribution: integer values from 0 to N, formed by a sum of N Bernoulli(p) variables. Many other discrete distributions exist and are covered in probability courses. We will focus on these two given their relation to information theory and digital communications (later). 11

12 Prediction Suppose we want to use probability to predict the outcome of an experiment. Given a random variable X for the outcome, and a probability distribution for that random variable, how do we come up with the best prediction of X? Some candidates: The most likely outcome (mode) The outcome with the smallest bias (mean) The middle outcome (median) The outcome with the smallest total error The issue is how do we define best, and how we use the information provided by the probability distribution, to achieve it. 12

13 Prediction Why not the most likely (mode)? Here s a counterexample: P(X=x) x Better alternative: mean (easy) Even better alternative: median (harder) 13

14 Prediction and expected value So, what about the mean? The mean, or expected value, includes the relative frequencies of all the values, not just the single largest. We compute the expected value E{X} as: E X = x P X = x x This expected value function yields a fixed number, since it sums over all the possibilities of X. Some useful properties of expectations: Linearity: if X, Y are random variables, and a, b are constants, then E{aX+bY} = ae{x}+be{y} Unbiased: we define bias as the difference of the prediction and the expected value E{X}, so = 0 If X, Y are uncorrelated, with means μ X, μ Y, then E{(X-μ X )(Y-μ Y )} = 0. More on this later. 14

15 Expected value Example: Let s imagine a fair coin flip: Example: How about rolling a fair six-sided die? 15

16 Expected value and variance The mean does not capture the variability, or how much the outcome is expected to change from trial to trial, of the random variable. It is possible for random outcomes to vary widely from the expected value (or any other prediction). How do we quantify this? Variance, or second central moment, measures squared-variability of X: σ X 2 = Var X = E X μ X 2 = x Standard deviation, σ X, is the square root of the variance. x μ X 2 P X = x This is one common measure of variability, or spread, of possible outcomes of a random trial. 16

17 Variance examples Example: variance of a fair coin flip (what if it s not fair?) Example: variance of a fair six-sided die roll 17

18 Properties of variance Just like expected value, the variance has certain properties: If we scale random variable X by a constant a, the Var(aX) = a 2 Var(X). If we add a constant to a random variable, the variance does not change. The variance of the sum of uncorrelated random variables is the sum of the variances: Var X + Y = E 2 X + Y μ X μ Y = E X μ 2 X + Y μ 2 Y + X μ X Y μ Y = Var X + Var Y + 0 Example: Variance of sum of uncorrelated die rolls 18

19 Conditional probability So far, we have mainly ignored relationships that can exist among random events. However, real events frequently are interdependent. For instance, the event of UVA men s basketball losing its second game against Duke probably had something to do with the first loss (e.g., most of the same players). Depending on the outcome of one experiment or trial (A), the likelihood of a subsequent trial (B) may change! We ask how likely B is to occur if A occurs (or does not occur). We write: P(B A), or the probability of B given A. Its definition is the ratio of two probabilities: P A B P B A = P A It is undefined if P(A) = 0 The numerator indicates the joint probability of A occurring and B occurring. 19

20 Conditional probability Example: Roll a six-sided die and flip that many fair coins. What is probability distribution of # of heads given a die-roll of 3? Of 6? 20

21 Independence Events A and B are independent if A tells us nothing about B, and vice versa. So P(A B) = P(A), and P(B A) = P(B). This also means that P(A B) = P(A)P(B). Otherwise, they are dependent. A similar definition holds for probabilities of random variables. We say X and Y are independent if P(X Y) = P(X) and P(Y X) = P(Y), where P(X,Y) is the joint distribution of X and Y, and P X Y = P X, Y P Y P Y X = P X, Y P X Independent random variables are also uncorrelated, but the converse frequently is not true. Why? 21

22 Independence What about more than two random variables? We can talk about independence of any pair of random variables. We can talk about independence among any group of random variables: P X, Y, Z, = P X P Y P Z If an entire group of random variables are independent, we say they are mutually independent. Mutual independence implies pairwise independence, but the converse is not true. In general, random variables X, Y, Z, are mutually independent if and only if E{f(X)g(Y)h(Z) } = E{f(X)}E{g(Y)}E{h(Z)} for any* function f, g, h,. This is an equivalent definition and also a powerful tool (proves uncorrelatedness among other things). * Technically, only those functions that have valid expected values 22

23 Bernoulli distribution Let s give our first probability distribution example: Bernoulli(p) is a two-valued discrete distribution (0 and 1) that takes probability P(X=1) = p. This means that P(X=0) = (1-p). Let s plot the cdf: What is the expected value E{X} in terms of p? What is the variance Var(X) in terms of p? 23

24 Towards the binomial distribution A Bernoulli distribution is one way to describe a coin flip, but the coin does not have to be fair. What if we flip a large number of coins? What is the fraction that comes up tails (1)? What can we say about the distribution of coin flips that come up heads or tails? What are we assuming? Independence of course 24

25 Towards the binomial distribution Let s imagine N coins are fair: P(0) = P(1) = 0.5. If we flip them, we can get 2 N possible outcomes. That s a lot to count: How many ways are there to choose n things from N items? How does this relate to counting heads or tails? Let s see, what is the probability of flipping 10 fair coins and getting exactly 5 heads?

26 Binomial distribution Now, let s imagine the flips are not necessary fair; P(1) = p. What is the probability of flipping N coins and getting n tails (1 s)? What does the cdf look like for binomial(5,0.5)? 26

27 Binomial distribution What is the expected value for X ~ binomial(n,p)? We use ~ to signify X s distribution. What is the variance for X ~ binomial(n,p)? Hint: use properties we ve discussed of expected value, variance. 27

28 Why Bernoulli and binomial? Why these distributions? It turns out coin flips and bits have a lot in common: Two values (heads/tails, 0 and 1) Fair or biased coin random bits (e.g., communication errors) We can describe large numbers of bits easily using a binomial distribution. Many other discrete (and continuous) probability distributions exist. Curious? Take APMA

29 Announcements Next time: Probability and information theory Homework #4 due today. Lab #4 will happen today. Homework #5 will go out next week. 29