Maximum likelihood in log-linear models

Transcription

1 Graphical Models, Lecture 4, Michaelmas Term 2010 October 22, 2010

2 Generating class Dependence graph of log-linear model Conformal graphical models Factor graphs Let A denote an arbitrary set of subsets of V. A density f (or function) factorizes w.r.t. A if there exist functions ψ a (x) which depend on x a only and f (x) = a A ψ a (x). Similar to factorization w.r.t. graph, but A are not necessarily complete subsets of a graph. The set of distributions P A which factorize w.r.t. A is the hierarchical log linear model generated by A. To avoid redundancy, it is common to assume the sets in A to be incomparable in the sense that no subset in A is contained in any other member of A. A is the generating class of the log linear model.

3 Generating class Dependence graph of log-linear model Conformal graphical models Factor graphs For any generating class A we construct the dependence graph G(A) = G(P A ) of the log linear model P A. Since the pairwise Markov property has to hold for all members of P A, it has at least to hold for all positive members. The dependence graph is determined by the relation α β a A : α, β a. For sets in A are clearly complete in G(A) and therefore distributions in P A do factorize according to G(A). On the other hand, any graph with fewer edges would not suffice. They are thus also global, local, and pairwise Markov w.r.t. G(A).

4 Generating class Dependence graph of log-linear model Conformal graphical models Factor graphs As a generating class defines a dependence graph G(A), the reverse is also true. The set C(G) of cliques (maximal complete subsets) of G is a generating class for the log linear model of distributions which factorize w.r.t. G. If the dependence graph completely summarizes the restrictions imposed by A, i.e. if A = C(G(A)), A is conformal.

5 Generating class Dependence graph of log-linear model Conformal graphical models Factor graphs φ IJ φ IK I K J The factor graph of A is the bipartite graph with vertices V A and edges define by φ JK α a α a. Using this graph even non-conformal log linear models admit a simple visual representation.

6 Data in list form Log linear models Consider a sample X 1 = x 1,..., X n = x n from a distribution with probability mass function p. We refer to such data as being in list form, e.g. as case Admitted Sex 1 Yes Male 2 Yes Female 3 No Male 4 Yes Male...

7 Contingency Table Log linear models Data often presented in the form of a contingency table or cross-classification, obtained from the list by sorting according to category: Sex Admitted Male Female Yes No The numerical entries are cell counts n(x) = {ν : x ν = x} and the total number of observations is n = x X n (x).

8 Assume now p P A but otherwise unknown. The likelihood function can be expressed as L(p) = n ν=1 p(x ν ) = x X p(x) n(x). In contingency table form the data follow a multinomial distribution n! P{N(x) = n(x), x X } = x X n(x)! x X p(x) n(x) but this only affects the likelihood function by a constant factor.

9 The likelihood function L(p) = x X p(x) n(x), is continuous as a function of the ( X -dimensional vector) unknown probability distribution p. Since the closure P A is compact (bounded and closed), L attains its maximum on P A. Unfortunately, P A is not closed by itself so limits of factorizing distributions do not necessarily factorize. The maximum of the likelihood function may not necessarily on P A itself, so it is necessary in general to include the boundary points.

10 Indeed, it is also true that L has a unique maximum over P A, which we shall now show. For simplicity, we only establish uniqueness within P A. The proof is indirect, but quite simple. Assume p 1, p 2 P A with p 1 p 2 and L(p 1 ) = L(p 2 ) = sup p P A L(p). (1) Define p 12 (x) = c p 1 (x)p 2 (x), where c 1 = { x p1 (x)p 2 (x)} is a normalizing constant.

11 Then p 12 P A because p 12 (x) = c p 1 (x)p 2 (x) = c ψa(x)ψ 1 a(x) 2 = ψa 12 (x), a A a A where e.g. ψa 12 = c 1/ A ψa(x)ψ 1 a(x). 2 The Cauchy Schwarz inequality yields c 1 = x i.e. we have c > 1. p1 (x)p 2 (x) < x p 1 (x) p 2 (x) = 1 x

12 Hence L(p 12 ) = x = x p 12 (x) n(x) { c{ } n(x) p 1 (x)p 2 (x) = c n x p1 (x) n(x) p2 (x) n(x) = c n L(p 1 )L(p 2 ) > L(p 1 )L(p 2 ) = L(p 1 ) = L(p 2 ), which contradicts (1). Hence we conclude p 1 = p 2. The extension to P A is almost identical. It just needs a limit argument to establish p 1, p 2 P A p 12 P A. x

13 The maximum likelihood estimate ˆp of p is the unique element of P A which satisfies the system of equations nˆp(x a ) = n(x a ), a A, x a X a. (2) Here g(x a ) = y:y a=x a g(y) is the a-marginal of the function g. The system of equations (2) expresses the fitting of the marginals in A. It can be seen as an instance of the fact that in an exponential family (log-linear exponential), the MLE is found by equating the sufficient statistics (marginal counts) to their expectation.

14 Proof: Assume p P A is a solution to the equations (2). That p maximizes the likelihood function follows from the calculation below, where p P A is arbitrary and φ a = log ψ a : log L(p) = n(x) log p(x) = n(x) φ a (x) x X x X a A = n(x)φ a (x) a A x X = n(y)φ a (y) a A x a X a y:y a=x a = n(x a )φ a (x), a A x a X a as n(x a ) = y:y a=x a n(y).

15 Further we get log L(p) = n(x a )φ a (x) a A x a X a = np (x a )φ a (x) a A x a X a = np (x)φ a (x) a A x X = np (x) log p(x). x X Thus, for any p P A we have established that log L(p) = x X np (x) log p(x).

16 This is in particular also true for p. The information inequality now yields log L(p) = x X np (x) log p(x) x X np (x) log p (x) = log L(p ). The case of p P A needs an additional limit argument.

17 To show that the equations (2) indeed have a solution, we simply describe a convergent algorithm which solves it. This cycles (repeatedly) through all the a-marginals in A and fit them one by one. For a A define the following scaling operation on p: (T a p)(x) p(x) n(x a) np(x a ), x X where 0/0 = 0 and b/0 is undefined if b 0.

18 Fitting the marginals Log linear models The operation T a fits the a-marginal if p(x a ) > 0 when n(x a ) > 0: n(t a p)(x a ) = n p(y) n(y a) np(y y:y a ) a=x a = n n(x a) p(y) np(x a ) y:y a=x a = n n(x a) np(x a ) p(x a) = n(x a ). Consequently, we have T 2 a = T a. No reason to do it twice.

19 Make an ordering of the generators A = {a 1,..., a k }. Define S by a full cycle of scalings Define the iteration It then holds that Sp = T ak T a2 T a1. p 0 (x) 1/ X, p n = Sp n 1, n = 1,.... lim p n = ˆp n where ˆp is the unique maximum likelihood estimate of p P A, i.e. the solution of the equation system (2).

20 Known as the IPS-algorithm or IPF-algorithm, or as a variety of other names. Implemented e.g. (inefficiently) in R in loglin with front end loglm in MASS. Key elements in proof: 1. If p P A, so is T a p; 2. T a is continuous at any point p of P A with p(x a ) 0 whenever n(x a ) = 0; 3. L(T a p) L(p) so likelihood always increases; 4. ˆp is the unique fixpoint for T (and S); 5. P A is compact.

21 A simple example Log linear models Admitted Sex Yes No S-marginal Male Female A-marginal Admissions data from Berkeley. Consider A S, corresponding to A = {{A}, {S}}. We should fit A-marginal and S-marginal iteratively.

22 Initial values Log linear models Admitted Sex Yes No S-marginal Male Female A-marginal Entries all equal to 4526/4. Gives initial values of np 0.

23 Fitting S-marginal Log linear models Admitted Sex Yes No S-marginal Male Female A-marginal For example and so on =

24 Fitting A-marginal Log linear models For example Admitted Sex Yes No S-marginal Male Female A-marginal = and so on. Algorithm has converged, as both marginals now fit!

25 Normalised to probabilities Admitted Sex Yes No S-marginal Male Female A-marginal Dividing everything by 4526 yields ˆp. It is overkill to use the IPS algorithm as there is an explicit formula, as we shall see next time.

26 L attains its maximum on P A.

27 L attains its maximum on P A. The maximizer ˆp P A of L is unique.

28 L attains its maximum on P A. The maximizer ˆp P A of L is unique. The maximizer ˆp is determined as the unique solution within P A of the equations nˆp(x a ) = n(x a ), a A, x a X a.

29 L attains its maximum on P A. The maximizer ˆp P A of L is unique. The maximizer ˆp is determined as the unique solution within P A of the equations nˆp(x a ) = n(x a ), a A, x a X a. The maximizer is the limit of the convergent repeated fitting of marginals T a : p(x) p(x)n(x a )/{np(x a )}, x X, a A.