Basics of probability

Transcription

1 Chapter 4 Basics of probability 4.1 The probability axioms Probabilities are defined over collections of sets. These collections have to have some structure to them. Definition 4.1 ( -field). A collection A of subsets of S is called a -field (or -algebra) if it satisfies the following 3 properties: 1. S 2A, 2. if A 2A, then A c 2A,and 3. if A 1,A 2,...2A, then S 1 i=1 A i 2A. Of course using some properties of sets, one can show that other sets must be in A if it is a -field. By the definition, (1) and (2) imply that ø 2A. We also have that if A and B 2A, then A \ B 2A, and we can show that T 1 i=1 A i 2 A. A few examples of -fields: For any S, {ø,s} is the smallest possible -field. The smallest non-trivial -field is {ø, A, A c,s} for some A in S. For any finite S, 2 S is a -field. For S = R, the Borel -field, B, is the smallest -field containing all the open sets in R. [This is a fairly special -field so it usually gets it s own letter. This is almost always the -field in the background when we talk about continuous probability measures. Every set you can think of is in here. There do exist subsets of the real line that are not in here, but they are very strange. This is a digression into esoterica.] Definition 4.2. Given a sample space S and an associated -field A, aprobability function (or probability measure) is a function P with domain A that satisfies the following 3 properties: 1. P (A) 08A 2A, 2. For a sequence A 1,A 2,... of pairwise disjoint sets in A, P ( S i A i)= P i P (A i), and 11

2 12 CHAPTER 4. BASICS OF PROBABILITY 3. P (S) =1. These are often called the Kolmogorov axioms. Whenever we talk about probabilities, we are talking about three entities simultaneously: the sample space S, an associated -field A, and a probability measure P. Often theoretical papers start with some sentence Consider the probability space (S, A,P),.... For this course, we will generally play fast and loose with these elements, but sometimes it is very important to be careful. Example 4.3. Consider a coin. The sample space is S = {H, T}. The associated -field is A =2 S. Since {H} and {T } form a partition of the sample space, then (for any probability measure) and 1=P (S) =P ({H}[{T }) =P ({H})+P ({T }), (4.1) P ({H}) =1 P ({T }). (4.2) Thus to completely specify a probability measure, it is enough to choose P ({H}). If I think the coin is fair, then I can take P ({H}) =1/2. Theorem 4.4. Let S be a finite sample space with n elements {s 1,...,s n }. Let A =2 S. Let p 1,...,p n be nonnegative numbers which sum to 1. For any A 2A, define P (A) = X p i. Then P is a probability measure on A. Verify the 3 axioms to prove this (easy). i:s i2a 4.2 Properties derived from the axioms Beware of measure theoretic butchery. Theorem 4.5. If P is a probability measure defined on the -field A and A 2A, then 1. P (ø) = 0; 2. P (A c )=1 P (A); 3. P (A) apple 1. Try to do these. Theorem 4.6. If P is a probability measure defined on the -field A and A, B 2A, then 1. P (B \ A c )=P (B) P (A \ B); 2. P (A [ B) =P (A)+P (B) P (A \ B); 3. If A B, then P (A) apple P (B).

3 4.3. CONDITIONAL PROBABILITY 13 These two results imply that, for a probability measure to be determined on all the elements in A it is enough to determine it s value on carefully chosen sub-collections. Example 4.7. Let S be a sample space and A a subset of S. Let A be the smallest -field that includes A, i.e. A = {ø, A, A c,s}. Then, any probability measure over A is completely defined by specifying P (A) (or equivalently P (A c )). Corollary 4.8 (Bonferroni). P (A \ B) P (A)+P (B) 1. Theorem 4.9. If P is a probability measure, then 1. P (A) = P 1 i=1 P (A \ C i) for any partition C 1,C 2,...; 2. P ([ 1 i=1 A i) apple P 1 I=1 P (A i) for any sets A 1,A 2,... (Boole s inequality). Proving part (1) is not so hard using only the set properties covered earlier. To prove (2), the trick is to define a disjoint collection A 0 1,A 0 2,... such that [ 1 i=1 A i = [ 1 i=1 A0 i.todothistake A 0 1 = A 1, 0 i=1 [ A 0 i = A i 1 A, i =2, 3,... Then use set properties on this partition. I m not going to cover the rest of section 1.2 in C&B, but you should read it. 4.3 Conditional probability Definition If A and B are events in S, andp (B) > 0, then the conditional probability of A given B, written P (A B) is P (A B) = j=1 A j P (A \ B) P (B) Essentially, this definition redefines the sample space to be B S. From this point of view, we can see that P ( B) is a probability measure in its own right, in that it satisfies the three axioms. Theorem 4.11 (Bayes Rule). Let A 1,A 2,... be a partition of the sample space S and let B be any set. Then for each A i, P (A i B) = P (B A i )P (A i ) P 1 j=1 P (B A j)p (A j ). Three characterizations of independence: P (A B) =P (A) P (B A) =P (B) P (A \ B) =P (A)P (B).

4 14 CHAPTER 4. BASICS OF PROBABILITY Exercise: show these are equivalent. Definition A collection of events A 1,...,A n are mutually independent if for any subcollection A k, k 2 K, we have! \ P A k = Y P (A k ). k2k 4.4 Random Variables k2k Definition A random variable is a function from a sample space S into the real numbers. Some examples Toss two dice. S =(s 1,s 2 ),s i 2{1, 2,...,6}. Interestedinthesum.X(s) =X((s 1,s 2 )) = s 1 + s 2. Toss a coin 25 times. S =(b 1,...,b 25 ), b i 2{H, T}. Interested in the number of heads. X(s) = P i b i. Random variables induce probability measures on the range space: P X (X 2 A) =P ({s 2 S : X(s) 2 A}). Can check that P X is a probability measure. Often suppress the dependence on X, i.e. write P (X 2 A) for P X (X 2 A). Sometimes it matters. Use X for random variables x for a value that it attains. 4.5 Distribution functions and densities Definition The cummulative distribution function (cdf) of a random variable X, denoted F X (x) is defined by F X (x) =P X (X apple x), 8x. Theorem A function F (x) is a cdf if and only if the following three conditions hold: 1. lim x! 1 F (x) =0and lim x!1 F (x) =1. 2. F (x) is a nondecreasing function of x. 3. F (x) is right-continuous; that is, 8x 0, lim x#x0 F (x) =F (x 0 ). Example Toss three coins. X counts the number of heads apple x apple 0 >< 1/8 0 apple x<1 F X (x) = 1/2 1 apple x<2 7/8 2 apple x<3 >: 1 3 apple x<1. Definition A random variable X is continuous if F X (x) is a continuous function of x. A random variable is discrete if F X (x) is a step function of x.

5 4.5. DISTRIBUTION FUNCTIONS AND DENSITIES 15 Definition The random variables X and Y are identically distributed if, for all A, P (X 2 A) =P (Y 2 A). In both of the above definitions, watch out for measure theoretic pathologies. Theorem The following are equivalent: X and Y are identically distributed. F X (x) =F Y (x) for all x. Definition The probability density function (pdf) is defined to be the function f X (x) that satisfies F X (x) = Z x 1 f X (t)dt. The probability mass function (pmf) is the discrete analogue: f X (x) =P (X = x). Really these are the same, you just have to interpret integration di erently. We will ignore these issues in this course. The following are all equivalent. X is has distribution P. X P. X F X where F X is as appropriate. X f X. X Y where Y P (X and Y are identically distributed) Theorem A function f X (x) is a pdf (pmf) of a random variable X if and only if f X (x) 0 for all x. R 11 f X(x)dx =1( P x f X(x) =1).