CHAPTER 1 DISTRIBUTION THEORY 1 CHAPTER 1: DISTRIBUTION THEORY

Transcription

1 CHAPTER 1 DISTRIBUTION THEORY 1 CHAPTER 1: DISTRIBUTION THEORY

2 CHAPTER 1 DISTRIBUTION THEORY 2 Basic Concepts

3 CHAPTER 1 DISTRIBUTION THEORY 3 Random Variables (R.V.) discrete random variable: probability mass function continuous random variable: probability density function

4 CHAPTER 1 DISTRIBUTION THEORY 4 Formal but Scary Definition of Random Variables R.V. are some measurable functions from a probability measure space to real space; probability is some non-negative value assigned to sets of a σ-field; probability mass the Radon-Nykodym derivative of the random variable-induced measure w.r.t. to a counting measure; probability density function the derivative w.r.t. Lebesgue measure.

5 CHAPTER 1 DISTRIBUTION THEORY 5 Descriptive quantities of univariate distribution cumulative distribution function: P (X x) moments (mean): E[X k ] quantiles mode centralized moments (variance): E[(X µ) k ] the skewness: E[(X µ) 3 ]/V ar(x) 3/2 the kurtosis: E[(X µ) 4 ]/V ar(x) 2

6 CHAPTER 1 DISTRIBUTION THEORY 6 Characteristic function (c.f.) ϕ X (t) = E[exp{itX}] c.f. uniquely determines the distribution F X (b) F X (a) = lim T f X (x) = 1 2π 1 2π T T e ita e itb it e itx ϕ(t)dt. ϕ X (t)dt

7 CHAPTER 1 DISTRIBUTION THEORY 7 Descriptive quantities of multivariate R.V.s Cov(X, Y ), corr(x, Y ) f X Y (x y) = f X,Y (x, y)/f Y (y) E[X Y ] = xf X Y (x y)dx Independence: P (X x, Y y) = P (X x)p (Y y), f X,Y (x, y) = f X (x)f Y (y)

8 CHAPTER 1 DISTRIBUTION THEORY 8 Equalities of double expectations E[X] = E[E[X Y ]] V ar(x) = E[V ar(x Y )] + V ar(e[x Y ]) Useful when the distribution of X given Y is simple!

9 CHAPTER 1 DISTRIBUTION THEORY 9 Examples of Discrete Distributions

10 CHAPTER 1 DISTRIBUTION THEORY 10 Binomial distribution Binomial(n, p): P (S n = k) = ( ) n k p k (1 p) n k, k = 0,..., n E[S n ] = np, V ar(s n ) = np(1 p) ϕ X (t) = (1 p + pe it ) n Binomial(n 1, p) + Binomial(n 2, p) Binomial(n 1 + n 2, p)

11 CHAPTER 1 DISTRIBUTION THEORY 11 Negative Binomial distribution P (W m = k) = ( ) k 1 m 1 p m (1 p) k m k = m, m + 1,... E[W m ] = m/p, V ar(w m ) = m/p 2 m/p Neg-Binomial(m 1, p) + Neg-Binomial(m 2, p) Neg-Binomial(m 1 + m 2, p)

12 CHAPTER 1 DISTRIBUTION THEORY 12 Hypergeometric distribution P (S n = k) = ( M k )( N M n k E[S n ] = Mn/(M + N) ) / ( N n V ar(s n ) = nmn(m+n n) (M+N) 2 (M+N 1) ), k = 0, 1,.., n

13 CHAPTER 1 DISTRIBUTION THEORY 13 Poisson distribution P (X = k) = λ k e λ /k!, k = 0, 1, 2,... E[X] = V ar(x) = λ, ϕ X (t) = exp{ λ(1 e it )} P oisson(λ 1 ) + P oisson(λ 2 ) P oisson(λ 1 + λ 2 )

14 CHAPTER 1 DISTRIBUTION THEORY 14 Poisson vs Binomial X 1 P oisson(λ 1 ), X 2 P oisson(λ 2 ) X 1 X 1 + X 2 = n Binomial(n, λ 1 λ 1 +λ 2 ) if X n1,..., X nn are i.i.d Bernoulli(p n ) and np n λ, then P (S n = k) = n! k!(n k)! pk n(1 p n ) n k λk k! e λ.

15 CHAPTER 1 DISTRIBUTION THEORY 15 Multinomial Distribution N l, 1 l k counts the number of times that {Y 1,..., Y n } fall into B l P (N 1 = n 1,..., N k = n k ) = ( ) n n 1,...,n k p n 1 1 p n k k n n k = n the covariance matrix for (N 1,..., N k ) n p 1 (1 p 1 )... p 1 p k..... p 1 p k... p k (1 p k )

16 CHAPTER 1 DISTRIBUTION THEORY 16 Examples of Continuous Distribution

17 CHAPTER 1 DISTRIBUTION THEORY 17 Uniform distribution Uniform(a, b): f X (x) = I [a,b] (x)/(b a) E[X] = (a + b)/2 and V ar(x) = (b a) 2 /12

18 CHAPTER 1 DISTRIBUTION THEORY 18 Normal distribution N(µ, σ 2 ): f X (x) = 1 2πσ 2 E[X] = µ and V ar(x) = σ 2 ϕ X (t) = exp{itµ σ 2 t 2 /2} exp{ (x µ)2 2σ 2 }

19 CHAPTER 1 DISTRIBUTION THEORY 19 Gamma distribution Gamma(θ, β): f X (x) = 1 β θ Γ(θ) xθ 1 exp{ x β }, x > 0 E[X] = θβ and V ar(x) = θβ 2 θ = 1 equivalent to Exp(β) θ = n/2, β = 2 equivalent to χ 2 n

20 CHAPTER 1 DISTRIBUTION THEORY 20 Cauchy distribution Cauchy(a, b): f X (x) = 1 bπ{1+(x a) 2 /b 2 } E[ X ] =, ϕ X (t) = exp{iat bt } often used as counter example

21 CHAPTER 1 DISTRIBUTION THEORY 21 Algebra of Random Variables

22 CHAPTER 1 DISTRIBUTION THEORY 22 Assumption X and Y are independent and Y > 0 we are interested in d.f. of X + Y, XY, X/Y

23 CHAPTER 1 DISTRIBUTION THEORY 23 Summation of X and Y Derivation: F X+Y (z) = E[I(X + Y z)] = E Y [E X [I(X z Y ) Y ]] = E Y [F X (z Y )] = F X (z y)df Y (y) Convolution formula: F X+Y (z) = f X f Y (z) F Y (z x)df X (x) F X F Y (z) f X (z y)f Y (y)dy = f Y (z x)f X (x)dx

24 CHAPTER 1 DISTRIBUTION THEORY 24 Product and quotient of X and Y (Y > 0) F XY (z) = E[E[I(XY z) Y ]] = F X (z/y)df Y (y) f XY (z) = f X (z/y)/yf Y (y)dy F X/Y (z) = E[E[I(X/Y z) Y ]] = F X (yz)df Y (y) f X/Y (z) = f X (yz)yf Y (y)dy

25 CHAPTER 1 DISTRIBUTION THEORY 25 Application of formulae N(µ 1, σ1) 2 + N(µ 2, σ2) 2 N(µ 1 + µ 2, σ1 2 + σ2) 2 Gamma(r 1, θ) + Gamma(r 2, θ) Gamma(r 1 + r 2, θ) P oisson(λ 1 ) + P oisson(λ 2 ) P oisson(λ 1 + λ 2 ) Negative Binomial(m 1, p) + Negative Binomial(m 2, p) Negative Binomial(m 1 + m 2, p)

26 CHAPTER 1 DISTRIBUTION THEORY 26 Summation of R.V.s using c.f. Result: if X and Y are independent, then ϕ X+Y (t) = ϕ X (t)ϕ Y (t) Example: X and Y are normal with ϕ X (t) = exp{iµ 1 t σ 2 1t 2 /2}, ϕ Y (t) = exp{iµ 2 t σ 2 2t 2 /2} ϕ X+Y (t) = exp{i(µ 1 + µ 2 )t (σ σ 2 2)t 2 /2}

27 CHAPTER 1 DISTRIBUTION THEORY 27 Further examples of special distribution assume X N(0, 1), Y χ 2 m and Z χ 2 n are independent; X Y/m Student s t(m), Y/m Z/n Snedecor s F m,n, Y Y + Z Beta(m/2, n/2).

28 CHAPTER 1 DISTRIBUTION THEORY 28 Densities of t, F and Beta distributions f t(m) (x) = Γ((m+1)/2) πmγ(m/2) 1 (1+x 2 /m) (m+1)/2 I (, ) (x) f Fm,n (x) = Γ(m+n)/2 Γ(m/2)Γ(n/2) (m/n) m/2 x m/2 1 (1+mx/n) (m+n)/2 I (0, ) (x) f Beta(a,b) (x) = Γ(a+b) Γ(a)Γ(b) xa 1 (1 x) b 1 I(x (0, 1))

29 CHAPTER 1 DISTRIBUTION THEORY 29 Exponential vs Beta- distributions assume Y 1,..., Y n+1 are i.i.d Exp(θ); Z i = Y Y i Y Y n+1 Beta(i, n i + 1); (Z 1,..., Z n ) has the same joint distribution as that of the order statistics (ξ n:1,..., ξ n:n ) of n Uniform(0,1) r.v.s.

30 CHAPTER 1 DISTRIBUTION THEORY 30 Transformation of Random Vector

31 CHAPTER 1 DISTRIBUTION THEORY 31 Theorem 1.3 Suppose that X is k-dimension random vector with density function f X (x 1,..., x k ). Let g be a one-to-one and continuously differentiable map from R k to R k. Then Y = g(x) is a random vector with density function f X (g 1 (y 1,..., y k )) J g 1(y 1,..., y k ), where g 1 is the inverse of g and J g 1 g 1. is the Jacobian of

32 CHAPTER 1 DISTRIBUTION THEORY 32 Example let R 2 Exp{2}, R > 0 and Θ Uniform(0, 2π) be independent; X = R cos Θ and Y = R sin Θ are two independent standard normal random variables; it can be applied to simulate normally distributed data.

33 CHAPTER 1 DISTRIBUTION THEORY 33 Multivariate Normal Distribution

34 CHAPTER 1 DISTRIBUTION THEORY 34 Definition Y = (Y 1,..., Y n ) is said to have a multivariate normal distribution with mean vector µ = (µ 1,..., µ n ) and non-degenerate covariance matrix Σ n n if f Y (y 1,..., y n ) = 1 (2π) n/2 Σ 1/2 exp{ 1 2 (y µ) Σ 1 (y µ)}

35 CHAPTER 1 DISTRIBUTION THEORY 35 Characteristic function ϕ Y (t) = E[e it Y ] = (2π) n 2 Σ 1 2 = ( 2π) n 2 Σ 1 2 = exp{ µ Σ 1 µ/2} ( 2π) n/2 Σ 1/2 exp{ y Σ 1 y 2 exp exp{it y 1 2 (y µ) Σ 1 (y µ)}dy + (it + Σ 1 µ) y µ Σ 1 µ }dy 2 { 1 2 (y Σit µ) Σ 1 (y Σit µ) +(Σit + µ) Σ 1 (Σit + µ)/2 } dy { = exp µ Σ 1 µ/2 + 1 } 2 (Σit + µ) Σ 1 (Σit + µ) = exp{it µ 1 2 t Σt}.

36 CHAPTER 1 DISTRIBUTION THEORY 36 Conclusions multivariate normal distribution is uniquely determined by µ and Σ for standard multivariate normal distribution, ϕ X (t) = exp{ t t/2} the moment generating function for Y is exp{t µ + t Σt/2}

37 CHAPTER 1 DISTRIBUTION THEORY 37 Linear transformation of normal r.v. Theorem 1.4 If Y = A n k X k 1 where X N(0, I) (standard multivariate normal distribution), then Y s characteristic function is given by ϕ Y (t) = exp { t Σt/2}, t = (t 1,..., t n ) R k and rank(σ) = rank(a). Conversely, if ϕ Y (t) = exp{ t Σt/2} with Σ n n 0 of rank k, then Y = A n k X k 1 with rank(a) = k and X N(0, I).

38 CHAPTER 1 DISTRIBUTION THEORY 38 Proof Y = AX and X N(0, I) ϕ Y (t) = E[exp{it (AX)}] = E[exp{i(A t) X}] = exp{ (A t) (A t)/2} = exp{ t AA t/2} Y N(0, AA ). Note that rank(aa ) = rank(a).

39 CHAPTER 1 DISTRIBUTION THEORY 39 Conversely, suppose ϕ Y (t) = exp{ t Σt/2}. There exists O, an orthogonal matrix, Σ = O DO, D = diag((d 1,..., d k, 0,..., 0) ), d 1,..., d k > 0. Define Z = OY ϕ Z (t) = E[exp{it (OY )}] = E[exp{i(O t) Y }] = exp{ (O t) Σ(O t) } = exp{ d 1 t 2 2 1/2... d k t 2 k/2}. Z 1,..., Z k are independent N(0, d 1 ),..., N(0, d k ) and Z k+1,..., Z n are zeros. Let X i = Z i / d i, i = 1,..., k and O = (B n k, C n (n k) ). Clearly, rank(a) = k. Y = O Z = Bdiag{( d 1,..., d k )}X AX.

40 CHAPTER 1 DISTRIBUTION THEORY 40 Conditional normal distributions Theorem 1.5 Suppose that Y = (Y 1,..., Y k, Y k+1,..., Y n ) has a multivariate normal distribution with mean µ = (µ (1), µ (2) ) and a non-degenerate covariance matrix Σ = ( Σ11 Σ 12 Σ 21 Σ 22 Then (i) (Y 1,..., Y k ) N k (µ (1), Σ 11 ). (ii) (Y 1,..., Y k ) and (Y k+1,..., Y n ) are independent if and only if Σ 12 = Σ 21 = 0. (iii) For any matrix A m n, AY has a multivariate normal distribution with mean Aµ and covariance AΣA. ).

41 CHAPTER 1 DISTRIBUTION THEORY 41 (iv) The conditional distribution of Y (1) = (Y 1,..., Y k ) given Y (2) = (Y k+1,..., Y n ) is a multivariate normal distribution given as Y (1) Y (2) N k (µ (1) +Σ 12 Σ 1 22 (Y (2) µ (2) ), Σ 11 Σ 12 Σ 1 22 Σ 21 ).

42 CHAPTER 1 DISTRIBUTION THEORY 42 Proof Prove (iii) first. ϕ Y (t) = exp{it µ t Σt/2}. ϕ AY (t) = E[e it AY ] = E[e i(a t) Y ] = exp{i(a t) µ (A t) Σ(A t)/2} = exp{it (Aµ) t (AΣA )t/2} AY N(Aµ, AΣA ).

43 CHAPTER 1 DISTRIBUTION THEORY 43 Prove (i). Y 1.. = ( I k k 0 k (n k) ) Y. Y 1. Y k N (( I k k 0 k (n k) ) µ, Y k ( I k k 0 k (n k) ) Σ ( I k k 0 k (n k) ) ).

44 CHAPTER 1 DISTRIBUTION THEORY 44 Prove (ii). t = (t (1), t (2) ) ϕ Y (t) = exp 1 2 [ it (1) µ (1) + it (2) µ (2) { t (1) Σ 11 t (1) + 2t (1) Σ 12 t (2) + t (2) Σ 22 t (2)}]. t (1) and t (2) are separable iff Σ 12 = 0. (ii) implies that two normal random variables are independent if and only if their covariance is zero.

45 CHAPTER 1 DISTRIBUTION THEORY 45 Prove (iv). Consider Z (1) N(0, Σ Z ) Z (1) = Y (1) µ (1) Σ 12 Σ 1 22 (Y (2) µ (2) ). Σ Z = Cov(Y (1), Y (1) ) 2Σ 12 Σ 1 22 Cov(Y (2), Y (1) ) +Σ 12 Σ 1 22 Cov(Y (2), Y (2) )Σ 1 22 Σ 21 = Σ 11 Σ 12 Σ 1 22 Σ 21. Cov(Z (1), Y (2) ) = Cov(Y (1), Y (2) ) Σ 12 Σ 1 22 Cov(Y (2), Y (2) ) = 0. Z (1) is independent of Y (2). The conditional distribution Z (1) given Y (2) is the same as the unconditional distribution of Z (1) Z (1) Y (2) Z (1) N(0, Σ 11 Σ 12 Σ 1 22 Σ 21).

46 CHAPTER 1 DISTRIBUTION THEORY 46 Remark on Theorem 1.5 The constructed Z in (iv) has a geometric interpretation using the projection of (Y (1) µ (1) ) on the normalized variable Σ 1/2 22 (Y (2) µ (2) ).

47 CHAPTER 1 DISTRIBUTION THEORY 47 Example X and U are independent, X N(0, σ 2 x) and U N(0, σ 2 u) Y = X + U (measurement error) ( σ 2 the covariance of (X, Y ): x σx 2 σx 2 σx 2 + σu 2 ) X Y N( σ 2 x σ 2 x+σ 2 u Y, σ 2 x σ4 x σ 2 x+σ 2 u ) N(λY, σ 2 x(1 λ)) λ = σ 2 x/σ 2 y (reliability coefficient)

48 CHAPTER 1 DISTRIBUTION THEORY 48 Quadratic form of normal random variables N(0, 1) N(0, 1) 2 χ 2 n Gamma(n/2, 2). If Y N n (0, Σ) with Σ > 0, then Y Σ 1 Y χ 2 n. Proof: Y = AX where X N(0, I) and Σ = AA. Y Σ 1 Y = X A (AA ) 1 AX = X X χ 2 n.

49 CHAPTER 1 DISTRIBUTION THEORY 49 Noncentral Chi-square distribution assume X N(µ, I) Y = X X χ 2 n(δ) and δ = µ µ. Y s density: f Y (y) = k=0 exp{ δ/2}(δ/2) k g(y; (2k + n)/2, 1/2). k!

50 CHAPTER 1 DISTRIBUTION THEORY 50 Additional notes on normal distributions noncentral t-distribution: N(δ, 1)/ χ 2 n/n noncentral F -distribution: χ 2 n(δ)/n/(χ 2 m/m) how to calculate E[X AX] where X N(µ, Σ): E[X AX] = E[tr(X AX)] = E[tr(AXX )] = tr(ae[xx ]) = tr(a(µµ + Σ)) = µ Aµ + tr(aσ)

51 CHAPTER 1 DISTRIBUTION THEORY 51 Families of Distributions

52 CHAPTER 1 DISTRIBUTION THEORY 52 location-scale family ax + b: f X ((x b)/a)/a, a > 0, b R mean ae[x] + b and variance a 2 V ar(x) examples: normal distribution, uniform distribution, gamma distributions etc.

53 CHAPTER 1 DISTRIBUTION THEORY 53 Exponential family examples: binomial, poisson distributions for discrete variables and normal distribution, gamma distribution, Beta distribution for continuous variables has a general expression of densities possesses nice statistical properties

54 CHAPTER 1 DISTRIBUTION THEORY 54 Form of densities {P θ }, is said to form an s-parameter exponential family: p θ (x) = exp exp{b(θ)} = { s k=1 exp{ η k (θ)t k (x) B(θ) } h(x) s η k (θ)t k (x)}h(x)dµ(x) < k=1 if {η k (θ)} = θ, the above form is called the canonical form of the exponential family.

55 CHAPTER 1 DISTRIBUTION THEORY 55 Examples X 1,..., X n N(µ, σ 2 ): exp { µ σ 2 n i=1 x i 1 2σ 2 n i=1 x 2 i n } 2σ 2 µ2 1 ( 2πσ) n X Binomial(n, p): exp{x log p 1 p + n log(1 p)} ( ) n x X P oisson(λ): P (X = x) = exp{x log λ λ}/x!

56 CHAPTER 1 DISTRIBUTION THEORY 56 Moment generating function (MGF) MGF for (T 1,..., T s ): M T (t 1,..., t s ) = E [exp{t 1 T t s T s }] the coefficients in the Taylor expansion of M T correspond to the moments of (T 1,..., T s )

57 CHAPTER 1 DISTRIBUTION THEORY 57 Calculate MGF in the exponential family Theorem 1.6 Suppose the densities of an exponential family can be written as the canonical form exp{ s k=1 η k T k (x) A(η)}h(x), where η = (η 1,..., η s ). Then for t = (t 1,..., t s ), M T (t) = exp{a(η + t) A(η)}.

58 CHAPTER 1 DISTRIBUTION THEORY 58 Proof exp{a(η)} = exp{ s k=1 η it i (x)}h(x)dµ(x). = M T (t) = E [exp{t 1 T t s T s }] s exp{ (η i + t i )T i (x) A(η)}h(x)dµ(x) k=1 = exp{a(η + t) A(η)}.

59 CHAPTER 1 DISTRIBUTION THEORY 59 Cumulant generating function (CGF) K T (t 1,..., t s ) = log M T (t 1,..., t s ) = A(η + t) A(η) the coefficients in the Taylor expansion are called the cumulants for (T 1,..., T s ) the first two cumulants are the mean and variance

60 CHAPTER 1 DISTRIBUTION THEORY 60 Examples revisited Normal distribution: η = µ/σ 2 and A(η) = 1 2σ 2 µ2 = η 2 σ 2 /2 M T (t) = exp{ σ2 2 ((η + t)2 η 2 )} = exp{µt + t 2 σ 2 /2}. From the Taylor expansion, for X N(0, σ 2 ), E[X 2r+1 ] = 0, E[X 2r ] = 1 2 (2r 1)σ 2r, r = 1, 2,...

61 CHAPTER 1 DISTRIBUTION THEORY 61 gamma distribution has a canonical form exp{ x/b + (a 1) log x log(γ(a)b a )}I(x > 0). η = 1/b, T = X A(η) = log(γ(a)b a ) = a log( 1/η) + log Γ(a). η M X (t) = exp{a log η + t } = (1 bt) a. E[X] = ab, E[X 2 ] = ab 2 + (ab) 2,...

62 CHAPTER 1 DISTRIBUTION THEORY 62 READING MATERIALS: Lehmann and Casella, Sections 1.4 and 1.5