U( θ, θ), U(θ 1/2, θ + 1/2) and Cauchy (θ) are not exponential families. (The proofs are not easy and require measure theory. See the references.

Transcription

1 Lecure 5 Exponenial Families Exponenial families, also called Koopman-Darmois families, include a quie number of well known disribuions. Many nice properies enjoyed by exponenial families allow us o provide rigorous proofs of some major resuls in saisical inference. We will only give an inroducion and refer o Barndorff-Nielsen (1978), Brown (1986), Shao (1999), Bickel and Doksum (2001) for more deailed discussion. 5.1 Original forms f(x θ) = exp[ C(θ), T (x) + D(θ)] h(x), x, (5.1) where h(x) 0, C(θ) = (C 1 (θ),..., C k (θ)), T (x) = (T 1 (x),..., T k (x)), and, is he inner produc defined in IR k. Example 5.1 For k = 1, express each of he following disribuions as (5.1) and idenify he funcions C, T, D and h: Bernoulli (θ), Poisson(θ), Exponenial (θ), N(θ, 1). For k = 2, wrie he densiy for N(µ, σ 2 ) wih θ = (µ, σ 2 ) as (5.1). Example 5.2 U( θ, θ), U(θ 1/2, θ + 1/2) and Cauchy (θ) are no exponenial families. (The proofs are no easy and require measure heory. See he references.) Example 5.3 For m > 1, a m-variae normal disribuion N m (µ, Σ) is an exponenial family, where µ is a m-vecor and Σ a m m posiive definie marix. he ranspose of marix A. Denoe by A In paricular, v is he column vecor as he ranspose of a row vecor v. We also le r(a) = a a mm be he race of a m m square marix A = (a ij ) i,j=1,...,m. The parameer θ = (µ, Σ) conains k = m+m+ m2 m 2 free componens. (why?) To wrie he densiy f(x θ) = 1 (2π) m Σ [ ] 1 exp 2 x µ, Σ 1 (x µ) in a form of (5.1), i suffices o noe in he exponen: µ Σ 1 x = x Σ 1 µ = x, Σ 1 µ, 1

2 and x Σ 1 x = r(x x Σ 1 ) = m i,j=1 (x x) ij (Σ 1 ) ji, where (A) ij = a ij in a marix A = (a ij ) i,j=1,...,m. Fac: Here is a useful Le 1,..., n be iid random variables wih 1 exponenial family. Show ha n exponenial family by idenifying C, T, D and h. Example 5.4 (mulinomial disribuions) Le 1,..., n be iid random variables wih a common disribuion P ( 1 = j) = p j, j = 1,..., r where r 2 is a posiive ineger, represening he number of caegories in he daa. For each j, denoe he coun (frequency) of caegory j among he n observaions by S n (j) = n l=1 I {l =j}, hen r j=1 S n (j) = n. To wrie he densiy as an exponenial family, i is imporan o noe he consrain p p r = 1. Hence we se θ = (p 1,..., p r 1 ) wih k = r 1 free componens and wrie f(x n θ) in a form of (5.1) where C(θ) = (log p 1 b(θ),..., log p r 1 b(θ) ) wih b(θ) = 1 r 1 j=1 p j, T (x n ) = (s (1) n,..., s (r 1) n ), D(θ) = n log b(θ) and h(x n ) = Naural paramerizaion and canonical forms To sudy he general mahemaical heory for exponenial families convenienly, we need o rewrie he original form (5.1) in a canonical form via he following naural paramerizaion. Assume he funcion C is a 1-1 mapping o avoid he idenifiabiliy problem. More specifically, if C(θ) = C(θ ) = η for θ θ, hen D(θ) = log e C(θ), T (x) h(x) ν (dx) = log e C(θ ), T (x) h(x) ν (dx) = D(θ ), which implies f(x θ) = f(x θ ), i.e. he parameer θ becomes non-idenifiable based on an observaion x. Call η = C(θ) a naural parameer and le ψ(η) = D(C 1 (η)) where C 1 denoes he inverse funcion of C. We rewrie (5.1) as f(x η) = exp[ η, T (x) ψ(η)] h(x). In wha follows, we abuse he noaion and simply denoe a naural parameer by θ insead of η unless a disincion is necessary. Definiion 1 Consider he parameer space Θ = { θ IR k : θ Θ is called a naural parameer for he canonical form } e θ, T (x) ν (dx) <. f(x θ) = exp[ θ, T (x) ψ(θ)] h(x), x ; (5.2) 2

3 where ψ(θ) = log e θ, T (x) ν (dx) is called he cumulan generaing funcion. F = {f( θ) : θ Θ} is called a minimal exponenial family of rank k if T = (T 1,..., T k ) and for every θ Θ, P θ ( k i=1 c i T i () = c 0 ) < 1 unless c i 0 i = 0, 1,..., k. 5.3 Properies We prove a number of useful properies for minimal exponenial families. Lemma 1 (convexiy) (i) ψ(θ) is a convex funcion on Θ. (ii) Θ is a convex se. Proof: (i) follows from he Hölder inequaliy and (ii) follows from (i). QED. Lemma 2 (exchangeabiliy beween θ j and, j = 1,..., k) Wihou loss of generaliy, assume Θ IR and le Θ o denoe he inerior of Θ. For every θ Θ o, d dθ e θt (x) ν (dx) = d dθ eθt (x) ν (dx). Proof: For every θ Θ o, δ > 0 such ha [θ δ, θ + δ] Θ o. Consider G() = e (θ+)t (x) eθt (x) ν (dx) where < δ/2. To show lim G() = lim 0 0 e (θ+)t (x) eθt (x) we need o have an upper bound for 0. Denoe T (x) by T and noe ha ν (dx), e(θ+)t (x) e θt (x) = e θt (x) e T (x) 1 uniformly near e T 1 e T 1 T e T T (e T + e T ) 2 (e δt + e δt ), 3

4 where he las inequaliy holds for < δ/2 and T > M wih some large M > 0. Since [ e (θ δ)t (x) + e (θ+δ)t (x)] ν (dx) <, he Lebesgue Dominaed Convergence Theorem applies. QED. The nex heorem shows ha he firs and second momens of T () in an exponenial family can be obained by differeniaing he cumulan generaing funcion ψ(θ). Theorem 1 Le f(x θ) = exp [ θ, T (x) ψ(θ)], θ Θ o be a minimal exponenial family of full rank k. Then (i) ψ(θ) = E θ T () ( gradien vecor = mean vecor ); (ii) 2 ψ(θ) = Cov θ T () ( Hessian marix = covariance marix ). Proof: Wih appropriae changes of he reference measure ν if needed, i follows from Lemma 2 ha θ j ψ(θ) = E θ T j (), j = 1,..., k; and 2 θ i θ j ψ(θ) = Cov θ (T i (), T j ()), i, j = 1,..., k. QED. Noe: The following exension is valid: for any posiive ineger l and non-negaive inegers l 1,..., l k wih l l k = l, we have l ki=1 θ l i i e ψ(θ) = k i=1 T l i i (x) e θ, T (x) ν (dx). Lemma 3 (Likelihood raio ideniy) For θ, θ Θ and any saisic V (), E θ V () = E θ [ f( θ ) f( θ) V () ], assuming all expecaions involved are finie. Proof: Change he measure from P θ o P θ. QED. Proposiion 1 (momen generaing funcions for exponenial families) For θ Θ o and any wih θ + Θ o, E θ e θ, T () = exp[ψ(θ + ) ψ(θ)]. 4

5 Proof: Noe ha f(x θ + ) f(x θ) = exp{, T (x) [ψ(θ + ) ψ(θ)]}. Seing V () = 1 and θ = θ + in Lemma 3 implies 1 = E θ+ e, T () exp{ [ψ(θ + ) ψ(θ)]}. QED 5