PROBABILITY THEORY LECTURE 3

Transcription

1 PROBABILITY THEORY LECTURE 3 Per Sidén Division of Statistics Dept. of Computer and Information Science Linköping University PER SIDÉN (STATISTICS, LIU) PROBABILITY THEORY - L3 1 / 15

2 OVERVIEW LECTURE 3 Transforms Probability generating function Moment generating function Characteristic function Transforms and distributions with random parameters PER SIDÉN (STATISTICS, LIU) PROBABILITY THEORY - L3 2 / 15

3 TRANSFORMS Finding the distribution of sum of random variables is hard. Convolution is messy. Transforms are functions that uniquely describe probability distributions. If you know the transform, you know the distribution, and vice versa. X d = Y g X (t) = g Y (t) Summation of independent variables corresponds to multiplication of transforms. Nice! PER SIDÉN (STATISTICS, LIU) PROBABILITY THEORY - L3 3 / 15

4 PROBABILITY GENERATING FUNCTION Applies to non-negative, integer-valued random variables. DEF The probability generating function of X is g X (t) = Et X = t n P(X = n) n=0 g X (t) is defined at least for t 1. TH If g X = g Y then p X = p Y. TH Let X 1, X 2,..., X n be independent. Then g X1 +X X n (t) = n g Xk (t) k=1 PER SIDÉN (STATISTICS, LIU) PROBABILITY THEORY - L3 4 / 15

5 PROBABILITY GENERATING FUNCTION, CONT. COR Let X 1, X 2,..., X n be independent and identically distributed. Then g X1 +X X n (t) = (g X (t)) n The name probability generating function comes from: P(X = n) = g (n) X (0) n! where g (n) X (t) is the nth derivative of g X (t) wrt to t. TH Factorial moments (if E X k < ) Moments can be computed E (X (X 1) (X k + 1)) = g (k) X (1) EX = g X (1) VarX = g X (1) + g X (1) ( g X (1) ) 2 PER SIDÉN (STATISTICS, LIU) PROBABILITY THEORY - L3 5 / 15

6 PROBABILITY GENERATING FUNCTION - EXAMPLES! binomial theorem: (x + y) n = n k=0 ( n k )x k y n k. Bernoulli, X Be(p) g X (t) = Binomial, X Bin(n, p) g X (t) = n k=0 t k( n k t n P(X = n) = t 0 q + t 1 p = q + pt n=0 ) p k q n k = n k=0 ( ) n (pt) k q n k = (q + pt) n k Let X 1,..., X n iid Be(p), then what is X = X X n? g X (t) = so X Bin(n, p). n i=1 g Xi (t) = n i=1 (q + pt) = (q + pt) n PER SIDÉN (STATISTICS, LIU) PROBABILITY THEORY - L3 6 / 15

7 PROBABILITY GENERATING FUNCTION - EXAMPLES! Poisson prob func: p(x = k) = e m m k /k!! e x = k=0 xk k! Poisson, X Po(m) g X (t) = t k e m m k k! k=0 = e m (mt) k k! k=0 = e m(t 1) If X 1 Po(m 1 ) independently of X 2 Po(m 2 ), what is X 1 + X 2? g X1 +X 2 (t) = e m 1(t 1) e m 2(t 1) = e (m 1+m 2 )(t 1) so X 1 + X 2 Po(m 1 + m 2 ). PER SIDÉN (STATISTICS, LIU) PROBABILITY THEORY - L3 7 / 15

8 MOMENT GENERATING FUNCTION g X (t) limited to non-negative integer-valued variables. DEF Moment generating function of a variable X ψ X (t) = Ee tx if the expectation exist and is finite for t < h, for some h > 0. TH If ψ X (t) exists for t < h for some h > 0, then All moments exist E X r < for all r > 0 EX n = ψ (n) X (0) for n = 1, 2,... Taylor expansion around t = 0 [note k e tx = X k e tx ] t k so e tx = 1 + Ee tx = 1 + n=1 n=1 t n X n n! t n n! EX n PER SIDÉN (STATISTICS, LIU) PROBABILITY THEORY - L3 8 / 15

9 MOMENT GENERATING FUNCTION - EXAMPLES X Be(p) ψ X (t) = Ee tx = qe t 0 + pe t 1 = q + pe t ψ X (t) = pet so E (X ) = ψ X (0) = p. ψ X (t) = pet so E (X 2 ) = ψ X (0) = p. Var(X ) = E (X 2 ) [E (X )] 2 = p p 2 = pq X Γ(p, a) ψ X (t) = 1 (1 at) p ψ X (t) = ap so E (X ) = ψ (1 at) p+1 X (0) = ap. ψ X (t) = a2 p(p+1) (1 at) p+2 so E (X 2 ) = ψ X (0) = a2 p(p + 1). Var(X ) = E (X 2 ) [E (X )] 2 = a 2 p(p + 1) a 2 p 2 = a 2 p. PER SIDÉN (STATISTICS, LIU) PROBABILITY THEORY - L3 9 / 15

10 MOMENT GENERATING FUNCTION, CONT. TH If h > 0 such that ψ X (t) = ψ Y (t) for t < h, then X d = Y. TH If X 1, X 2,..., X n are independent with moment generating functions that exist for t < h for some h > 0, then ψ X1 +...X n (t) = n i=1 ψ Xi (t), t < h TH Moment generating function of a linear combination a X + b ψ ax +b (t) = e tb ψ X (at) If X Γ(d, p), what is the distribution of Y = σ X? ψ X (t) = ψ Y (t) = 1 (1 dt) p 1 (1 dσt) p, which is the mgf of Γ(dσ, p). Gamma family is closed under scaling. PER SIDÉN (STATISTICS, LIU) PROBABILITY THEORY - L3 10 / 15

11 THE CHARACTERISTIC FUNCTION Moment generating function is not defined for all random variable. No mgf for Cauchy or LogNormal. The characteristic function is more general and exists for any variable, but complex valued. DEF The characteristic function of a random variable X is ϕ X (t) = Ee itx = E (cos tx + i sin tx ) where i is the imaginary number (i 2 = 1). X U(a, b), then ϕ X (t) = eitb e ita it (b a) PER SIDÉN (STATISTICS, LIU) PROBABILITY THEORY - L3 11 / 15

12 COMPLEX NUMBERS Complex number z = a + b i Re(z) = a is the real part of z Im(z) = b is the imaginary part of z Complex conjugate z = a b i Addition: z 1 + z 2 = a 1 + a 2 + (b 1 + b 2 ) i Multiplication: z 1 z 2 = a 1 a 2 b 1 b 2 + (a 1 b 2 + a 2 b 1 )i Modulus: z = a 2 + b 2. Length of vector. Complex exponentials: e ix = cos x + i sin x PER SIDÉN (STATISTICS, LIU) PROBABILITY THEORY - L3 12 / 15

13 THE CHARACTERISTIC FUNCTION, CONT. TH If ϕ X = ϕ Y then X = d Y. TH Let F be the distribution function of X. If F is continuous at a and b, and ϕ(t) dt < then f (x) = 1 e itx ϕ(t)dt 2π TH Characteristic function of a sums of independent variables ϕ X X n (t) = n ϕ Xi (t) i=1 TH Moments TH Linear combinations ϕ (k) X (0) = i k EX k ϕ ax +b (t) = e ibt ϕ X (at) PER SIDÉN (STATISTICS, LIU) PROBABILITY THEORY - L3 13 / 15

14 TRANSFORMS - DISTRIBUTIONS WITH RANDOM PARAMETERS Transforms are expected values (or t X, e tx or e itx ), so the law of iterated expectation is useful. Let X (N = n) Bin(n, p) and N Po(λ). What is the marginal distribution of X? X is non-negative and integer-valued, so g X (t) is defined. ( ) g X (t) = E E (t X N) = Eh(N) where h(n) = E (t X N = n) = (q + pt) n. We then have ( g X (t) = E (q + pt) N) = g N (q + pt) = e λ[(q+pt) 1] = e λp(t 1). X y N(0, y) and y Exp(1), then X L(1/ 2). Prove using characteristic functions. PER SIDÉN (STATISTICS, LIU) PROBABILITY THEORY - L3 14 / 15

15 TRANSFORMS - SUMS OF RANDOM NUMBER OF RANDOM VARIABLES TH Let S n = X 1 + X X n be a sum of i.i.d variables and N be a non-negative integer valued random variable. Then g SN (t) = g N (g X (t)) ψ SN (t) = g N (ψ X (t)). ϕ SN (t) = g N (ϕ X (t)) X 1, X 2,... Exp(1) (i.i.d) and N Fs (p). S N? ψ SN (t) = g N (ψ X (t)) = 1 1 t p S N Exp (1/p) PER SIDÉN (STATISTICS, LIU) PROBABILITY THEORY - L3 15 / 15