Random experiments may consist of stages that are performed. Example: Roll a die two times. Consider the events E 1 = 1 or 2 on first roll

Transcription

1 Econ 514: Probability and Statistics Lecture 4: Independence Stochastic independence Random experiments may consist of stages that are performed independently. Example: Roll a die two times. Consider the events E 1 = 1 or 2 on first roll E 2 = 2,3,5 on second roll 1

2 Because outcome on first roll does not affect the outcome on the second, the outcomes on the first roll in E 1 are randomly paired with the outcomes on the second roll in E 2. Hence P (E 1 E 2 = = 1 6 = = P (E 1P (E 2 where P (E 1, P (E 2 refer to probabilities associated with a single roll of a die. If P (E 1 E 2 = P (E 1 P (E 2 then the events E 1, E 2 need not be for independent stages of the experiment! 2

3 Example: Roll a die two times. Consider the events E 1 = sum of the outcomes is odd E 2 = outcome is 1 on first die Hence E 1 = {(1, 2, (1, 4, (1, 6, (2, 1, (2, 3, (2, 5, (3, 2, (3, 4, (3, 6, (4, 1, (4, 3, (4, 5, (5, 2, (5, 4, (5, 6, (6, 1, (6, 3, (6, 5} E 2 = {(1, 1, (1, 2, (1, 3, (1, 4, (1, 5, (1, 6} E 1 E 2 = {(1, 2, (1, 4, (1, 6} Hence P (E 1 E 2 = 3 36 = P (E 1P (E 2 =

4 Definition of stochastic independence General definition of independence starts from probability space (Ω, F, P. A sub-σ-field G is a σ-field such that all events E G are also in F, i.e. G F. Consider a collection of sub-σ-fields G i, i = 1,..., I where I need not be finite. 4

5 We consider two cases: I finite. The sub-σ-fields G i, i = 1,..., I are called independent if P (E 1... E I = P (E P (E I for all E i G i, i = 1,..., I. I infinite. The sub-σ-fields G i, i = 1,..., are called independent if for each finite set of indices S {1, 2,...} P ( i S E i = i S P (E i for all E i G i, i S. Note that the definition for I = requires independence for any finite sub collection of sub-σ-fields. 5

6 Example: Consider the three events E 1, E 2, E 3 Ω. Define the sub-σ-fields G i = {Ω,, E i, E c i }. Verify that the sub-σ-fields are independent if and only if (i P (E i E j = P (E i P (E j for i j = 1, 2, 3 (ii P (E 1 E 2 E 3 = P (E 1 P (E 2 P (E 3 This definition of independence of events is given in C&B. It is a special case of the general definition for sub-σ-fields. 6

7 If the sub-σ-fields are generated by collections of events we need not check the definition for all sets in the subσ-fields. Theorem 1 Let E 1,... E I be collections of events that are stable under finite intersections and with Ω E i for i = 1,..., I. If for all E i E i P ( I i=1e i = I P (E i then the σ-fields σ(e i generated by these collections are independent. i=1 Proof Fix E 2,... E I and define D 1 = {D P (D E 2... E I = P (DP (E 2... P (E I } Now E 1 D 1 and D 1 is a λ-system, because Ω D 1 7

8 If D 2 D 1 both in D 1, then because D 1 = D 2 (D 1 \ D 2 and D 2 (D 1 \ D 2 = P (D 1 P (E 2... P (E I = P (D 1 E 2... E I = = P ((D 2 (D 1 \ D 2 E 2... E I = = P ((D 2 E 2... E I ((D 1 \D 2 E 2... E I = P (D 2 E 2... E I + P ((D 1 \ D 2 E 2... E I Hence P ((D 1 \D 2 E 2... E I = (P (D 1 P (D 2 P (E 2... P (E I = = P (D 1 \ D 2 P (E 2... P (E I 8

9 If D i, i = 1, 2,... is an increasing sequence of sets, then D = i=1 D i = i=1 (D i \ D i 1 with D 0 =. This is a countable union of disjoint sets. Define C i = D i \ D i 1. Then P (D E 2... E I = P (( i=1c i E 2... E I = ( = P (C i E 2... E I = P (C i P ( E 2... E I = i=1 i=1 = P (DP ( E 2... E I By an earlier theorem σ(e 1 D 1, because E 1 is closed under finite intersections. Hence for all E 1 σ(e 1 and E 2 E 2,..., E I E I, P (E 1... E I = P (E 1... P (E I Repeat the argument with E 1 σ(e 1, E 3 E 3, E I E I fixed, to replace E 2 by σ(e 2 etc. 9

10 Application: Let G i, i = 1,..., I be a collection of independent sub-σ-fields. Let I j be disjoint subsets of {1,..., I}, then the σ-fields σ( i Ij G i are independent. Proof: Define E j as the set of all finite intersections of sets in i Ij G i. Because every set in i Ij G i is the intersection with itself, E j clearly generates σ( i Ij G i. It is also closed under finite intersections and it contains Ω. Apply the theorem. 10

11 Independence of random variables If i are random variables defined in (Ω, F, P, then we can define the sub-σ-fields G i = { 1 i (B B B} This is the σ-field generated by i. The random variables i, i = 1,..., I are called independent if the σ-fields that they generate are independent, i.e. if for all B i B. Pr( I i=1 i B i = I Pr( i B i i=1 This definition applies if I is finite. If I = we look at all finite subsets. 11

12 Application: By the theorem above i are independent if I Pr( i x i, i = 1,... I = Pr( i x i i=1 for all x i R. Take E i = {i 1 ((, x] x R}. 12

13 Theorem 2 Let 1,..., I be independent random variables on (Ω, F, P. Let f : R k R and g : R I k R be measurable functions and let 1 = f( 1,..., k and 2 = g( k+1,..., I be integrable random variables, then 1 and 2 are independent random variables and E( 1 2 = E( 1 E( 2 (1 Proof Denote the σ-field generated by i by G i. By definition the sub-σ-fields G i, i = 1,..., I are independent. This implies (see above that the σ-fields F 1 = σ( k i=1 G i and F 2 = σ( I i=1k+1 G i are independent. We need to show that 1 k (B F k for k = 1, 2 and all B B. I show this for 1. Define the function : Ω R k by 1 (ω (ω =. k (ω We have 1 1 (B = 1 (f 1 (B with by the measurability of f, f 1 (B B k the k-dimensional Borel σ- field generated by the sets B k = B 1... B k with B i B. Hence we need to show that 1 ( B F 1 for all B B k. Now because F 1 is a σ-field, so is D k = {C 1 (C F 1 } (check this!. Moreover for B k = B 1... B k, 1 (B k = k i=1 1 i (B k F 1. Because these sets generate B k, we have B k D k. Hence for all B B, 1 1 (B F 1. 13

14 To show (1 we first take 1, 2 as non-negative. By an earlier theorem these functions can be approximated (from below by the increasing sequence of simple functions kn = 1 4n 2 n Hence ( 1 = 2 n E( 1n 2n = 1 4 n = 1 4n 4 n 4 n i=1 Pr i=1 4 n j=1 4n i=1 i=1 4 n j=1 E I k i 2 n ( I 1 I i 2 n 2 = j 2 n ( Pr 1 i 2 n, 2 j = 2 n ( 1 i ( 1 2 n 2 n 4 n j=1 ( Pr 1 j = E( 2 n 1n E( 2n Apply monotone convergence to the left- and right-hand side to obtain E( 1 2 = E( 1 E( 2 For general integrable random variables we have Hence k = k+ k E( 1 2 = E( E( 1+ 2 E( E( 1 2 = = E( 1+ E( 2+ E( 1+ E( 2 E( 1 E( 2+ +E( 1 E( 2 = = (E( 1+ E( 1 (E( 2+ E( 2 = E( 1 E( 2 14

15 Product spaces and measures Let, be sets with σ-fields F, G. The set = {(x, y x, y } is called the product (set of and. A set F G with F F and G G is called a measurable rectangle. The product σ-field F G is the σ-field generated by the measurable rectangles. Let P be a (probability measure on F G. With P we can define marginal measures P 1, P 2 on F and G: P (F = P (F and P (G = P ( G. 15

16 If we define the random variables (x = x, x and (y = y, y (, then P is the distribution of the random vector P, P are the marginal distributions of and. If and are independent then P (F G = P (F P (G. If and are not independent, then the measure on the product σ-field can not be obtained from the marginal measures. 16

17 Instead of the marginal measure on we define a collection of measures (we use the same notation P = {P,x x } with the property that P,x (G is a measurable function [0, 1] for all G G. Such a collection of measures is called a kernel from, F to, G and a probability kernel if in addition P,x ( = 1 for all x. Our goal is to compute integrals for measurable functions f : R by repeated integration f(x, ydp,x (ydp (x 17

18 For this we need to establish that f(x, y is a measurable function of y for fixed x, if f is a measurable function f : R. f(x, ydp,x(y is a measurable function of x, if f is a measurable function f : R. The proofs require us to consider generating classes of functions and are beyond our scope (see e.g. Pollard s book referenced on the web site. In the sequel I will use these results. 18

19 Theorem 3 Let P = {P,x x } be a kernel from (, F to (, G with P,x ( < for all x, and let P be a measure on F. Let f : R be a measurable non-negative function. Then (i For each fixed x, f(x,. : R is a measurable function. (ii f(x, ydp,x(y which is a function R is a measurable function. (iii The iterated integral f(x, ydp,x (ydp (x = ( f(x, ydp,x (y dp (x (2 satisfies all requirements of an integral (linearity, monotonicity and monotone convergence. 19

20 Proof We do not prover (i and (ii. If (i and (ii hold, the integral on the right-hand side of (2 is welldefined, because all integrands are measurable (and nonnegative. In particular, the integral defines a measure P on the product space with σ-field F G. For E F G ( P (E = I E (x, ydp,x (y dp (x 20

21 We check the properties Linearity: Let f = c 1 f 1 + c 2 f 2. Then (c 1 f 1 (x, y + c 2 f 2 (x, ydp,x (ydp (x = = = = c 1 ( (c 1 f 1 (x, y + c 2 f 2 (x, ydp,x (y dp (x = (c 1 f 1 (x, ydp,x (y + c 2 f 2 (x, ydp,x (y dp (x = = c 1 +c 2 ( ( f 1 (x, ydp,x (y dp (x+ f 2 (x, ydp,x (y dp (x = f 1 (x, ydp,x (ydp (x+c 2 f 2 (x, ydp,x (ydp (x 21

22 Monotonicity: If f(x, y g(x, y for all (x, y, except possibly on a set A with P (A = 0. Note that if P (A = 0, then P,x (A x = 0 with A x = {y (x, y A} for all x, except possibly on a set with P measure 0. Hence, by the monotonicity of the (single integral we have for fixed x f(x, ydp,x (y g(x, ydp,x (y except for a set of P measure 0. A second application of the monotonicity of the single integral gives ( f(x, ydp,x (ydp (x = f(x, ydp,x (y dp (x ( g(x, ydp,x (y dp (x = g(x, ydp,x (ydp (x 22

23 Monotone convergence: Let f n f, then f(x, ydp,x (ydp (x = = = ( lim n lim f n(x, ydp,x (y dp (x = n ( f n (x, ydp,x (y dp (x = by monotone convergence of the single integral. 23

24 The sequence of functions lim f n(x, ydp,x (y n is itself a monotone sequence and hence we can again apply monotone convergence for the single integral to interchange the limit and integral to obtain f(x, ydp,x (ydp (x = = lim n ( = lim n f n (x, ydp,x (y dp (x = f n (x, ydp,x (ydp (x 24

25 Remarks The theorem also applies if P is a σ-finite kernel, i.e. if there is a countable partition of, F i G i with F i F, G i G such that P,x (B i < for all x A i. If the probability kernel P, x has a density f,x (y for all x with respect to measure µ x and P has a density f (x with respect to a measure µ, then we have g(x, ydp,x (ydp (x = = ( = g(x, yf,x (ydµ x (y f (xdµ(x = g(x, yf,x (yf (xdµ x (ydµ(x 25

26 Example: For µ x and µ both the Lebesgue measure and f,x (y = xe xy y 0 = 0 otherwise we have e.g. Pr((, A = f (x = 1 0 x 1 = 0 otherwise I A (x, yxe xy dydx 26

27 Product measures A special type of (probability kernel is the case that P,x = P. This is the case that P and P are the distributions of independent random variables. Consider the product space and product σ- field F G. For rectangles F G we can define a probability measure by P P (F G = P (F P (G. This is called the product measure. Theorem 4 There is only one measure P P defined on F G such that P P (F G = P (F P (G for rectangles F G with F F, G G. 27

28 Proof The set of all rectangles E = {F G F F, G G} is closed under finite intersections (see figure. Consider another probability measure P on F G such that P (F G = P P (F G = P (F P (G for rectangles F G E. Define D = {D F G P (D = P P (D}. Obviously E D. Now D is a λ-system Ω Ω D Let D 1 D 2 and both in D. Hence P (D 1 = P P (D 1 and P (D 2 = P P (D 2. Subtract to obtain P (D 2 \ D 1 = P (D 2 P (D 1 = = P P (D 2 P P (D 1 = P P (D 2 \ D 1 28

29 Let D 1 D 2... be an increasing sequence of sets in D. Then n=1d n = n=1(d n \D n 1 with D 0 =, i.e. the union of disjoint sets that are all in D. By the above result P (D n \ D n 1 = P P (D n \ D n 1 and hence P ( n=1d n = P (D n \ D n 1 = = n=1 P P (D n \ D n 1 = P P ( n=1d n n=1 Hence n=1d n D. We conclude that σ(e = F G D. 29

30 Note that on the rectangles P P (F G = P (F P (G = P (GP (F. If we consider the integral of f(x, y with respect to P P we can write it using Theorem 3 as ( f(x, ydp (y dp (x = f(x, ydp P (x, y = = ( f(x, ydp (x dp (y 30

31 Hence application of Theorem 3 to a product measure gives the Tonelli Theorem Theorem 5 For the probability space (, F G, P P we have if f is a non-negative measurable function f : R (i For fixed x f(x, y is a measurable function of y and for fixed y f(x, y is a measurable function of x. (ii f(x, ydp (y is a measurable function of x and f(x, ydp (x is a measurable function of y. (iii ( f(x, ydp (y dp (x = = ( f(x, ydp (x dp (y f(x, ydp P (x, y = 31

32 The Tonelli theorem can be extended to general integrable f. Because f = max{f +, f } with f + = max{f, 0} and f = max{ f, 0}, the Tonelli theorem implies that if f(x, y dp P (x, y < then f + (x, ydp P (x, y < f (x, ydp P (x, y < 32

33 This implies that, except for a set N of P measure 0, f + (x, ydp (y < and f (x, ydp (y < and, except for a set N of P measure 0, f + (x, ydp (x < To compute the integral of f with respect to the product measure as a repeated integral we need that f(x, ydp (y = f + (x, ydp (y f (x, ydp (y or f(x, ydp (x = f + (x, ydp (x f (x, ydp (x exists (if it exists it is also finite and integrable. 33

34 This is the case, except if (x, y N N, a set of P P measure 0. Note that changing the value of f on this set does not change the value of an integral with respect to P P. Hence if, e.g. f(x, ydp (x is not defined for y N we can set it equal to 0, without changing the integral of f with respect to P P. Theorem 6 Let f be an integrable function with respect to P P. Then (i For fixed x f(x, y is a measurable function of y and for fixed y f(x, y is a measurable function of x. (ii f(x, ydp (y exists and is finite and integrable, except for a set of P measure 0 and f(x, ydp (x exists and is finite and integrable, except for a set of P measure 0. (iii ( f(x, ydp (y dp (x = = ( f(x, ydp (x dp (y f(x, ydp P (x, y = 34