THE MAXIMUM LIKELIHOOD DEGREE OF MIXTURES OF INDEPENDENCE MODELS

Transcription

1 THE MAXIMUM LIKELIHOOD DEGREE OF MIXTURES OF INDEPENDENCE MODELS JOSE ISRAEL RODRIGUEZ AND BOTONG WANG Abstract The maximum likelihood degree (ML degree) measures the algebraic complexity of a fundamental optimization problem in statistics: maximum likelihood estimation In this problem, one maximizes the likelihood function over a statistical model The ML degree of a model is an upper bound to the number of local extrema of the likelihood function and can be expressed as a weighted sum of Euler characteristics The independence model (ie rank one matrices over the probability simplex) is well known to have an ML degree of one, meaning their is a unique local maxima of the likelihood function However, for mixtures of independence models (ie rank two matrices over the probability simplex), it was an open question as to how the ML degree behaved In this paper, we use Euler characteristics to prove an outstanding conjecture by Hauenstein, the first author, and Sturmfels; we give recursions and closed form expressions for the ML degree of mixtures of independence models 1 Introduction Maximum likelihood estimation is a fundamental computational task in statistics A typical problem encountered in its applications is the occurrence of multiple local maxima To be certain that a global maximum of the likelihood function has been achieved, one locates all solutions to a system of polynomial equations called likelihood equations; every local maxima is a solution to these equations The number of solutions to these equations is called the maximum likelihood degree (ML degree) of the model This degree was introduced in [3, 13] and gives a measure of complexity to the global optimization problem, as it bounds the number of local maxima The maximum likelihood degree has been studied in many contexts Some of these contexts include Gaussian graphical models [22], variance component models [10], and in missing data [14] In this manuscript, we work in the context of discrete random variables (for a recent survey in this context, see [17]) In our main results, we provide closed form expressions for ML degrees mixtures of independence models, which are sets of joint probability distributions for two random variables This answers an outstanding conjecture of [12] 11 Algebraic statistics preliminaries We consider a model for two discrete random variables, having m and n states respectively A joint probability distribution for two such random variables is written as an m n-matrix: p 11 p 12 p 1n (1) p 21 p 22 p 2n P = p m1 p m2 p mn 1

2 2 RODRIGUEZ AND WANG The (i, j)th entry p ij represents the probability that the first variable is in state i and the second variable is in state j By a statistical model, we mean a subset M of the probability simplex mn 1 of all such matrices P If iid samples are drawn from some distribution P, then the data is summarized by the following matrix (2) The entries of u are non-negative integers where u ij is the number of samples drawn with state (i, j): u 11 u 12 u 1n u 21 u 22 u 2n (2) u = u m1 u m2 u mn The likelihood function corresponding to the data matrix u is given by (3) l u (p) := p u pu pu mn mn Maximum likelihood estimation is an optimization problem for the likelihood function This problem consists of determining, for fixed u, the argmax of l u (p) on a statistical model M The optimal solution is called the maximum likelihood estimate (mle) and is used to measure the true probability distribution For the models we consider, the mle is a solution to the likelihood equations In other words, by solving the likelihood equations, we solve the maximum likelihood estimation problem Since the ML degree is the number of solutions to the likelihood equations, it gives a measure on the difficulty of the problem The model M mn in mn 1 is said to be the mixture of independence models and is defined to be the image of the following map: (4) ( m 1 n 1 ) ( m 1 n 1 ) 1 M mn (R 1, B 1, R 2, B 2, C) c 1 R 1 B1 T + c 2R 2 B2 T where R i, B j, C := [c 1, c 2 ] T are m 1 matrices, n 1, and 2 1 matrices respectively with positive entries that sum to one The Zariski closure of M mn yields the variety of rank at most 2 matrices over the complex numbers We will determine the ML degree of the models M mn by studying the topology of the Zariski closure Prior to our work, the ML degrees of these models were only known for small values of m and n In [12], the following table of ML degrees of M mn were computed: n = m = 3 : m = 4 : m = 5 : Mixtures of independence models appear in in many places in science, statistics, and mathematics In computational biology, the case where (m, n) equals (4, 4) is discussed in Example 13 of [20], and the data u consists of a pair of DNA sequences of length (u 11 + u u mn ) The m and n equal four because DNA has four nucleobases Another interesting case for computational biology is when (m, n) = (20, 20) because there are 20 essential amino acids [Remark 43],

3 THE MAXIMUM LIKELIHOOD DEGREE OF MIXTURES OF INDEPENDENCE MODELS 3 Our first main result [Theorem 312] proves a formula for the first row of the table: MLdegreeM 3n = 2 n+1 6 for n 3 Our second main result, is a recursive expression for ML degree of mixtures of independence models As a consequence, we are able to calculate a closed form expressions for each row of the above table of ML degrees [Corollary 42] Our techniques relate ML degrees to Euler characteristics Prior, Huh has shown that the ML degree of smooth algebraic statistical models M with Zariski closure X equals the signed topological characteristic of an open subvariety X o where X o is the set of points of X with nonzero coordinates and coordinate sums [16] More recent work of Budur and the second author show that the ML degree of a singular model is a stratified topological invariant In [1], they show that the Euler characteristic of X o is a sum of ML degrees weighted by Euler obstructions These Euler obstructions, in a sense, measure the multiplicity of the singular locus We conclude this introduction with illustrating examples to set notation and definitions 12 Defining the maximum likelihood degree We will use two notions of maximum likelihood degree The first notion is from a computational algebraic geometry perspective, where we define the maximum likelihood degree for a projective variety When this projective variety is contained in a hyperplane, the maximum likelihood degree has an interpretation related to statistics The second notion is from a topological perspective, where we define the maximum likelihood degree for a very affine variety, a subvariety of an algebraic torus (C ) n To P n+1 we associate the coordinates p 0, p 1,, p n, and p s (were s stands for sum) Consider the distinguished hyperplane in P n+1 defined by p p n p s (p s is the sum of the other coordinates) Let X be a generically reduced variety contained in the distinguished hyperplane of P n+1 not contained in any coordinate hyperplane We will be interested in the critical points of the likelihood function l u (p) := p u 0 0 pu 1 1 pu n n p u s s where u s := u 0 u n and u 0,, u n C The likelihood function has the nice property that, up to scaling, its gradient is a rational function l u (p) := [ u 0 p 0 : u 1 p 1 : : u n p n : u s p s ] Definition 11 Let u be fixed A point p X is said to be a critical point of the likelihood function on X if p is a regular point of X, each coordinate of p is nonzero, and the gradient l u (p) at p is orthogonal to the tangent space of X at p Example 12 Let X of P 4 be defined by p 0 + p 1 + p 2 + p 3 p s and p 0 p 3 p 1 p 2 For [u 0 : u 1 : u 2 : u 3 : u s ] = [2 : 8 : 5 : 10 : 25] there is a unique critical point for l u (p) on X This point is [p 0 : p 1 : p 2 : p 3 : p s ] = [70 : 180 : 105 : 270 : 625] Whenever each u i is not equal to 0, there is a unique critical point [(u 0 + u 1 )(u 0 + u 2 ) : (u 0 + u 1 )(u 1 + u 3 ) : (u 2 + u 3 )(u 0 + u 2 ) : (u 2 + u 3 )(u 1 + u 3 ) : (u 0 + u 1 + u 2 + u 3 ) 2 ] Definition 13 The maximum likelihood degree of X is defined to be the number of critical points of the likelihood function on X for general u 0,, u n

4 4 RODRIGUEZ AND WANG We say u in C n+1 is general, whenever there exists a dense Zariski open set U for which the number of critical points of l u (p) is constant and u U In Example 12, the Zariski open set U is the complement of the variety defined by u 1 u 2 u 3 u 4 u s = 0 Determining this Zariski open set explicitly is often quite difficult and commonly, probability one algorithms are used to compute ML degrees instead However, with our results, we compute ML degrees using Euler characteristics and topological arguments 13 Using Euler characteristics In the definition of maximum likelihood degree of a projective variety X, a critical point p X must have nonzero coordinates This means all critical points of the likelihood function are contained in the underlying very affine variety of X o := X\coordinate hyperplanes} In fact, the ML degree is directly related to the Euler characteristic of smooth X o Theorem 14 ([16]) Suppose X is a smooth projective variety of P n+1 Then, (5) χ(x o ) = ( 1) dim Xo MLdeg X The next example will show how to determine the signed Euler characteristic of a very affine variety Y Recall that the Euler characteristic is a homotopy invariant and satisfies the following properties The Euler characteristic is additive for algebraic varieties More precisely, χ(x) = χ(z) + χ(x \ Z), where Z is a closed subvariety of X The product property says χ(m N) = χ(m) χ(n) More generally, the fibration property says that if E B is a fibration with fiber F then χ(e) = χ(f) χ(b) Example 15 Consider X from Example 12 The variety X has the parameterization shown below P 1 P 1 X [x 0 : x 1 ] [y 0 : y 1 ] [x 0 y 0 : x 0 y 1 : x 1 y 0 : x 1 y 1 : x 0 y 0 + x 0 y 1 + x 1 y 0 + x 1 y 1 )] Let X o be the underlying very affine variety of X and consider O := P 1 \ [0 : 1], [1 : 0], [1 : 1]} the projective space with 3 points removed The very affine variety X o has a parameterization: O O X o [x 0 : x 1 ] [y 0 : y 1 ] [x 0 y 0 : x 0 y 1 : x 1 y 0 : x 1 y 1, (x 0 + x 1 )(y 0 + y 1 )] Since χ(p 1 ) = 2, after removing 3 points, we have χ(o) = 1 By the product property χ(o O) = 1, and hence χ(x o ) = 1 Because X o is smooth, by Huh s result, we conclude the ML degree of X o is 1 as well We call X the variety of 2 2 matrices with rank 1 and restricting to the probability simplex yields the independence model This example generalizes by considering the map P m 1 P n 1 X given by ([x 0 : : x m 1 ], [y 0 : : y n ]) [x 0 y 0 : : x m 1 y n 1 : i,j x i y j ] In this case, X is the variety of m n rank 1 matrices, and a similar computations shows that the ML degree is 1 in these cases as well (see Example 1 of [12])

5 THE MAXIMUM LIKELIHOOD DEGREE OF MIXTURES OF INDEPENDENCE MODELS 5 2 Whitney stratification, Gaussian degree, and Euler obstruction As we have just seen, for a smooth very affine variety, the ML degree is equal to the Euler characteristic up to a sign This is not always the case, when the very affine variety is not smooth For singular varieties, the ML degree is still related to the topology of the variety, but in a more subtle way from Whitney stratifications (Corollary 14) Here, we give a brief introduction to the topological notions of Whitney stratification and Euler obstruction 21 Whitney stratification Many differential geometric notions do not behave well when a variety has singularities, for instance, tangent bundles and Poincare duality This situation is addressed by stratifying the singular variety into finitely many pieces, such that along each piece the variety is close to a smooth variety A naive way to stratify a variety X is taking the regular locus X reg as the first stratum, and then take the regular part of the singular locus of X, ie, (X sing ) reg, and repeat this procedure This naive stratification does not always reflect the singular behavior of a variety as seen in the Whitney umbrella Whitney introduced conditions (now called Whitney regular) on a stratification, where many differential geometric results can be generalized to singular varieties A Whitney stratification satisfies the quite technical condition of Whitney regular, see [9, Page 37], [15, E37] for details Suppose X r is the subvariety of P mn contained in the distinguished hyperplane p p mn p s = 0 parametrizing rank r matrices of size m n Then, the naive stratification is indeed a Whitney stratification (see Proposition 31) Moreover, the stratification is given by the rank of the corresponding matrix We use Whitney stratifications in Corollary 14 We will see, up to a sign, the ML degree of a singular variety is equal to the Euler characteristic with some correction terms The correction terms are linear combinations of the ML degree of smaller dimensional strata of the Whitney stratification, whose coefficients turn out to be the Euler obstructions 22 Gaussian degrees We have defined the notion of maximum likelihood degree of a projective variety Sometimes, it is more convenient to restrict the projective variety to some affine torus and consider the notion of maximum likelihood degree of subvarieties of affine torus Let Y be a closed irreducible subvariety of (C ) n Such a variety is called a very affine variety Denote the coordinates of (C ) n by z 1, z 2,, z n The likelihood functions in the affine torus (C ) n are of the forms l u = z u 1 1 zu 2 2 zu n n Definition 21 Let Y (C ) n be a very affine variety Define the maximum likelihood degree of Y, denoted by MLdeg o (Y), to be the number of critical points of a likelihood function l u for general u 1, u 2,, u n Fix an embedding of P n P n+1 by [p 0 : p 1 : : p n ] [p 0 : p 1 : : p n : p 0 + p p n ] Given a projective variety X P n, we can consider it as a subvariety of P n+1 by the embedding we defined above Then as a subvariety of P n+1, X is contained in the hyperplane p 0 + p p n p s = 0 Consider (C ) n+1 as an open subvariety of P n+1, given by the open embedding (z 0, z 1,, z n ) [z 0 : z 1 : : z n : 1]

6 6 RODRIGUEZ AND WANG Now, for the projective variety X P n, we can embed X into P n+1 as described above, and then take the intersection with (C ) n+1 Thus, we obtain a very affine variety, which we donate by X o Lemma 22 The ML degree of X as a projective variety is equal to the ML degree of X o as a very affine variety, ie (6) MLdeg(X) = MLdeg o (X o ) Proof Fix general u 0, u 1,, u n C The ML degree of X is defined to be the number of critical points of the likelihood function (p 0 /p s ) u 0(p 1 /p s ) u 1 (pn /p s ) u n The ML degree of X o is defined to be the number of critical points of z u 0 0 zu 1 1 zu n n The two functions are equal on X o Therefore, they have the same number of critical points For the rest of this section, by maximum likelihood degree we always mean maximum likelihood degree of very affine varieties As observed in [2], the maximum likelihood degree is equal to Gaussian degree defined by Franecki and Kapranov [7] The main theorem of [7] relates the Gaussian degree with Euler characteristics In this section, we will review their main result together with the explicit formula from [6] to compute characteristic cycles First, we follow the notation in [2] Fix a positive integer n Denote the affine torus (C ) n by G and denote its Lie algebra by g Let T G be the cotangent bundle of G T G has a canonical symplectic structure For any γ g, let Ω γ T G be the graph of the corresponding left invariant 1-form on G Suppose T G is a Lagrangian subvariety of T G For a generic γ g, the intersection Ω γ is transverse and consists of finitely many points The number of points in Ω γ is constant when γ is contained in a nonempty Zariski open subset of g This number is called the Gaussian degree of, and denoted by gdeg( ) Let Y G be an irreducible closed subvariety of dimension d Denote the conormal bundle of Y reg in G by TY reg G, and denote its closure in T G by TY G Then T YG is an irreducible conic Lagrangian subvariety of T G Given any γ g, the left invariant 1-form corresponding to γ degenerates at a point P Y if and only if TY G Ω γ contains a point in TP G Thus, we have the following Lemma Lemma 23 ([2]) (7) MLdeg o (Y) = gdeg(t Y G) Let F be a bounded constructible complex on G and let CC(F) be its characteristic cycle Then CC(F) = j n j ( j ) is a Z-linear combination of irreducible conic Lagrangian subvarieties in the cotangent bundle T G The Gaussian degree and Euler characteristic are related by the following theorem Theorem 24 ([7]) (8) χ(g, F) = n j gdeg( j ) j

7 THE MAXIMUM LIKELIHOOD DEGREE OF MIXTURES OF INDEPENDENCE MODELS 7 23 Euler obstructions The Euler obstructions are defined to be the coefficients of some characteristic cycle decomposition (see equation (9)), and it is a theorem of Kashiwara that they can be computed as the Euler characteristic of some complex link (see Theorem 26) Let (S 1, S 2,, S k ) be a Whitney stratification of Y such that S 1 = Y reg Let e j1 be the Euler obstruction of the pair S j, S 1, which measures the singular behavior of Y along S j More precisely, e j1 are defined such that the following equality holds (See [6, 11] for more details) (9) CC(C S1 ) = e j1 [TS j G] 0 j k For example, e 11 = ( 1) dim Y Since χ(s 1 ) = χ(g, C S1 ), combining eqs (7) to (9), we have the following corollary that expresses χ(s 1 ) as a weighted sum of ML degrees Corollary 25 Let (S 1, S 2,, S k ) be a Whitney stratification of Y such that S 1 = Y reg Then, χ(s 1 ) = e j1 MLdeg o ( S j ) 1 j k where S j denotes the closure of S j in Y and e j1 be the Euler obstruction of the pair S j, S 1 Even though the abstract definition of Euler obstruction uses characteristic cycles, there is a concrete topological formula computing Euler obstructions due to Kashiwara Theorem 26 (Kashiwara 1 ) Fix a point z Z Then ) (10) e (mn) = ( 1) dim Z+1 χ c (B (X \ Z) φ 1 (ɛ) where B is a ball of radius δ in P mn 1 centered at z, φ is a general linear function defined on a normal slice N of Z at z and 0 < ɛ δ 1 By the formula above, every Euler obstruction we consider is computable Example 27 Consider the variety rank at most 2 matrices X in P 9 defined by det[p ij ] 3 3 = p 11 + p p 33 p s = 0 Denote the very affine subvariety of X by X o, that is, X o := p X p ij = 0 for all i, j and p s = 0} The Whitney stratification of X o consists of S 1, the regular points of X o, and S 2, the singular point of X o, which are the rank 1 matrices By Corollary 25, we have (11) χ(s 1 ) = e 11 MLdeg o ( S 1 ) + e 21 MLdeg o ( S 2 ) With this equation, we determine the ML degree of X The rank 1 matrices are known to have ML Degree one, so MLdeg o ( S 2 ) = 1 The Euler obstruction e 21 is equal to the Euler characteristic of some complex link, up to a sign (see Theorem 26 for the precise formula) In this case, the complex link turns out to be homeomorphic to a vector bundle over P 1 (see 1 Here the formula is in the form of [6, Theorem 11], see also [4, Page 100] and [8, 81]

8 8 RODRIGUEZ AND WANG Lemma 32 for more details) The sign in front of the Euler characteristic is negative, and hence e 21 = 2 The Euler obstruction e 11 is much easier to determine because this always equals ( 1) dim X So here e 11 = 1 In Subsection 32, we will calculate the Euler characteristic of χ(s 1 ) In fact, χ(s 1 ) = 12 Therefore, (11) implies that MLdeg(X) = 10 concluding the example In Example 27, we used Corollary 25 and topological computations to determine the ML degree of a singular variety In the next section we will again use Corollary 25 and topological computations to determine ML degrees 3 The ML degree for rank 2 matrices To P mn we associate the coordinates p 11,, p 1n,, p m1,, p mn, p s Let X mn denote the variety defined by p p mn p s = 0 and the vanishing of the 3 3 minors of the matrix p 11 p 12 p 1n (12) p (m 1)1 p (m 1)2 p (m 1)n p m1 p m2 p mn We think of X mn as the Zariski closure of the set of rank 2 matrices in the distinguished hyperplane of P mn Let Z mn be the subvariety of X mn defined by vanishing of the 2 2 minors of the matrix (12) Then Z mn is the singular locus of X mn for m, n 3 We will make topological computations to determine the ML degree of X mn Summarizing, Proposition 31 gives a Whitney decomposition of X mn and determines the Euler obstructions for this stratification Using these computations, we reduce our problem to determining a single Euler characteristic χ(x mn \ Z mn ) by Corollary 33 Next, Theorem 34 provides a closed form expression of χ(x mn \ Z mn ), for fixed m, in terms of the elements of the finite sequence Λ m of integers We conclude this section by computing Λ m for m = 3, thereby proving Theorem Calculating Euler obstructions We will start by proving general results for m, n 2 At the end of this section, we will specialize to the case where m = 3 With some topological computations, we determine λ 1, λ 2 of Λ 3, thereby giving a closed form expression of the ML degree of X 3n [Theorem 312] To ease notation, we let e (mn) denote the Euler obstruction e 21 for X mn and we have Y mn := X mn \ Z mn Proposition 31 The stratification X mn = Y mn Z mn is a Whitney stratification of X mn Moreover, if the Euler obstruction of the pair of strata (Z mn, Y mn ) is denoted by e (mn), then (13) e (mn) = ( 1) m+n 1 (minm, n} 1) Proof When m = 2 or n = 2, X mn = P mn 1 and Z mn X mn is a smooth subvariety Thus, the first part of the proposition follows Moreover, it follows from definition that e (mn) = ( 1) dim Z mn+1 Since dim Z mn = m + n 2, the second part of the proposition follows Therefore, we can assume m, n 3 and m n without loss of generality

9 THE MAXIMUM LIKELIHOOD DEGREE OF MIXTURES OF INDEPENDENCE MODELS 9 Notice that there is a left Gl(m, C) action and a right Gl(n, C) action on X mn that both preserve Z mn The total action by Gl(m, C) Gl(n, C) is transitive on Z mn Since Z mn is the singular locus of X mn, near a general point of Z mn, Z mn X mn has to be Whitney regular Now by the presence of the transitive action, Z mn X mn is Whitney regular at every point of Z mn Since Y mn = X mn \ Z mn, it follows (Y mn, Z mn ) is a Whitney stratification of X mn Next, we will compute the Euler obstructions using Theorem 26 Consider X mn as a subvariety of P mn 1, the projective space of all m n matrices In the next theorem and its proof, we simply write X and Z instead of X mn and Z mn and use the notion normal slice A normal slice of a variety Z at the point z is a general linear space with complementary dimension to Z containing the point z One can easily compute that dim Z = m + n 2 Since B (X \ Z) φ 1 (ɛ) is an evendimensional manifold, its Euler characteristics with or without compact support are equal by Poincare duality Since the Euler characteristic is homotopy invariant and since χ(p m 2 ) = m 1, the second part of the proposition follows from the following lemma where we show the link is homotopy equivalent to P m 2 Lemma 32 With the above notations and the assumption m n, we have B (X \ Z) φ 1 (ɛ) is homotopy equivalent to P m 2 Proof First, we give a concrete description of the normal slice N Notice that X P mn is contained in the distinguished hyperplane p p mn p s = 0 In this proof, we will consider X as a subvariety of P mn 1 with homogeneous coordinates p 11,, p mn Denote the affine chart p 11 = 0 of P mn 1 by U 11 Let a ij = p ij p 11 ((i, j) = (1, 1)) be the affine coordinates of U 11 and let a 11 = 1 Denote the origin of U 11 by O Now, we define a projection π : U 11 Z U 11 by (a ij ) (b ij ), where b ij = a i1 a 1j and a 11 = 1 Then U 11 becomes a vector bundle over Z U 11 via iπ : U 11 Z U 11 The preimage of O is the vector space parametrized by a ij with 2 i m, 2 j n In terms of matrices, we can think of π as the following map 1 a 12 a 1n a 21 a 22 a 2n π a m1 a m2 a mn 1 a 21 a m1 1 a 12 a 1n T 1 a 12 a 1n a 21 a 21 a 12 a 21 a 1n =, a m1 a m1 a 12 a m1 a n and we think of the preimage of O as a 22 a 2n = π a m2 a mn By the above construction, we can take the normal slice N at O to be the fiber π 1 (O) The intersection N X is clearly isomorphic to the affine variety (a ij ) 2 i m,2 j n rank 1} Thus, we can define a map ρ : N (X \ Z) P m 2 which maps the matrix (a ij ) 2 i m,2 j n } to one of its nonzero column vectors, as an element in P m 2 Since the rank of (a ij ) 2 i m,2 j n } is

10 10 RODRIGUEZ AND WANG 1, the map does not depend on which nonzero column vector we choose Using basic linear algebra, it is straightforward to check the following two statements about ρ The restriction of ρ to B (X \ Z) φ 1 (ɛ) is surjective The restriction of ρ to B (X \ Z) φ 1 (ɛ) has convex fibers Therefore, ρ induces a homotopy equivalence between B (X \ Z) φ 1 (ɛ) and P m 2 Corollary 33 Let X mn and Y o mn be defined as in the beginning of this section Then, (14) χ(y o mn) = MLdeg(X mn ) + ( 1) m+n 1 (minm, n} 1) Proof By Example 15, we know that MLdeg(Z mn ) = 1 One can easily compute that dim Xmn o = 2m + 2n 3 Now, the corollary follows from (13) and Corollary Calculating Euler characteristics and ML degrees In this subsection, an expression for the Euler characteristic χ(y o mn) is given to determine formulas for ML degrees Recall that X o mn is the complement of all the coordinate hyperplanes in X mn, and that Z mn X mn is the subvariety corresponding to rank 1 matrices of size m n Theorem 34 Fix m to be an integer greater than two Then, there exists a sequence, denoted Λ m, of integers λ 1, λ 2,, λ m 1 such that (15) χ(ymn) o = ( 1) n 1 1 i m 1 λ i i i m 1 λ i i + 1 in 1, for n 2 Before proving Theorem 15, we quote a hyperplane arrangement result, which follows immediately from the theorem of Orlik-Solomon (see eg [19] Theorem 590) Lemma 35 Let L 1,, L r be distinct hyperplanes in C s Suppose they are in general position, that is the intersection of any t hyperplanes from L 1,, L r } has codimension t, for any 1 t s Denote the complement of L 1 L r in C s by M Then if r = s + 1, then χ(m) = ( 1) s ; if r = s + 2, then χ(m) = ( 1) s (s + 1) We will use the proceeding lemma to compute Euler characteristics of stratum of the following stratifications Throughout, we assume that m is fixed Let U n denote the rank 2 matrices with nonzero coordinates whose entries sum to 1, ie, } U n := (a ij ) 1 i m,1 j n a ij C, a ij = 1, rank(a ij ) = 2 1 i m,1 j n A stratification of U n is given by the number of columns summing to zero a matrix has Defining U (l) n below, U (l) n := yields the stratification: (a ij ) 1 i m,1 j n U n l = # U n = U (0) n U (n) n, }} j 1 i m a ij = 0,

11 THE MAXIMUM LIKELIHOOD DEGREE OF MIXTURES OF INDEPENDENCE MODELS 11 where each U (l) n is a locally closed subvariety of U n Let U n denote the set of rank 2 matrices with nonzero column sums, ie, U n := Note by definition, we have U n = U (0) n (a ij ) 1 i m,1 j n U n a ij = 0 for each 1 j n 1 i m and (16) U n = Y o mn The following lemmas will show χ(u n) = χ(u n ) by proving χ(u (l) n ) = 0 for l 1 Lemma 36 (17) χ(u n ) = χ(u n) Proof of Lemma We define a C action on U n by putting t (a ij ) = a ij, where a ij = a ij if a 1j + + a mj = 0 t a ij if a 1j + + a mj = 0 It is straightforward to check the action preserves each U (l) n continuous on U (l) n and the action is transitive and for any l 1 Therefore, χ(u (l) n ) = 0 for any l 1, and hence χ(u n ) = χ(u (0) n ) = χ(u n) A column sum of U n can be any element of C Now we consider a subset of U n where the column sums are all exactly 1 Let V n denote the set of rank 2 matrices with column sums equal to 1, ie, } V n := (b ij ) 1 i m,1 j n b ij C, b ij = 1 for each 1 j n, rank(b ij ) = 2 1 i m In Lemma 37, we express the Euler characteristic of U n in terms of χ(v n ) Lemma 37 (18) χ(u n) = ( 1) n 1 χ(v n ) Proof of Lemma Let T n = (t j ) 1 j n (C ) n j t j = 1} Define a map F : T n V n U n by putting a ij = t j b ij ; we think of the jth element of T n as scaling the jth column of V n Clearly, F is an isomorphism Therefore, χ(u n) = χ(t n ) χ(v n ) T n can be considered as C n 1 removing n hyperplanes in general position By Lemma 35, χ(t n ) = ( 1) n 1, and hence χ(u n) = ( 1) n 1 χ(v n ) A stratification for V n can be given by the minimal j 0 such that the column vector (b ij0 ) is linearly independent from (b i1 ) Since i b ij = 1 for all j, if two column vectors are linearly dependent, they must be equal Defining V (l) below, V (l) n := n (b ij ) V n the column vectors satisfy (b i1 ) = (b i2 ) = = (b il ) = (b i(l+1) ) } },

12 12 RODRIGUEZ AND WANG yields the stratification of V n by locally closed subvarieties: Hence ( χ(v n ) = χ V n = V (1) n V (2) n V (n 1) n V (1) n We use W i defined below to stratify V n, ) ( + χ V (2) n ) ( + + χ V (n 1) n W n := (b ij ) V n the first two column vectors (b i1 ) and (b i2 ) are linearly independent} Then, we have the isomorphism V (l) n Therefore, [ b i1 b i1 b i1 }} l copies of b i1 = W n l+1 by the following map, ) b i(l+1) b in ] [b i1 b i(l+1) b in ] (19) χ(v n ) = χ(w 2 ) + χ(w 3 ) + + χ(w n ) For any l 2, we can define a map π l : W l W 2 by taking the first two column Thus, we can consider all W l as varieties over W 2 Lemma 38 For any l 2, W l = W3 W2 W 3 W2 W2 W 3 where there are l 2 copies of W 3 on the right hand side and the product is the topological fiber product In other words, take any point x W 2 the fiber of π l : W l W 2 over x is equal to the (l 2)-th power of the fiber of π 3 : W 3 W 2 over x Proof of Lemma Given l 2 elements in W 3 Suppose they all belong to the same fiber of π 3 : W 3 W 2 This means that we have (l 2) size m 3 matrices of rank 2, which all have the same first two columns Then we can collect the third column of each matrix, and put them after the same first two columns Thus we obtain a m l matrix, whose rank is still 2 In this way, we obtain a map W 3 W2 W 3 W2 W2 W 3 W l, [ ] [ ] [ ] bi1 b i2 b i3, bi1 b i2 b i4,, bi1 b i2 b il }} [ b i1 b i2 b i3 ] b il, (l 2) m 3 matrices which is clearly an isomorphism Given any point x W 2, we denote the fiber of π 3 : W 3 W 2 by F x Since Euler characteristic satisfies the product formula for fiber bundles, to compute the Euler characteristic of W l, it suffices to study the stratification of W 2 by the Euler characteristic of F x, and compute the Euler characteristic of each stratum More precisely, let W k 2 := x W 2 χ(f x ) = k} Then to compute χ(w l ), it suffices to compute χ(w k 2 ) for all k Let Λ m denote the sequence λ 0, λ 1,, λ m 1, where λ k are defined by λ k := χ(w k 2 ) for 0 k m 1

13 THE MAXIMUM LIKELIHOOD DEGREE OF MIXTURES OF INDEPENDENCE MODELS 13 As we will soon see in the proofs, we can consider F x as the complement of a point arrangement in C The arrangement is parametrized by the point x W 2 In other words, W 2 is naturally a parameter space of point arrangement in C According to the Euler characteristic of the corresponding point arrangement, W 2 is canonically stratified Our problem is to compute the Euler characteristic of each stratum The main difficulty to generalize our method to compute the ML degree of higher rank matrices is to solve the corresponding problem for higher dimensional hyperplane arrangements Lemma 39 For any x W 2, 0 χ(f x ) (m 1) Moreover, the map W 2 Z defined by x χ(f x ) is a semi-continuous function In other words, for any integer k, x W 2 χ(f x ) k} is a closed algebraic subset of W 2 In particular, the subsets x W 2 χ(f x ) = k}, 0 k (m 1), give a stratification of W 2 into locally closed subsets Proof of Lemma By definition, W 2 = (b ij ) 1 i m,j=1,2 b ij C, b i1 = 1 i m } 1 i m b i2 = 1, rank(b ij ) = 2 Fix an element x = (b ij ) W 2 By definition, the fiber F x of π 3 : W 3 W 2 is equal to the following: } F x = (b i3 ) 1 i m b i3 C, b i3 = 1, (b i3 ) is contained in the linear span of (b i1 ) and (b i2 ) 1 i m Since 1 i m b ij = 1, j = 1, 2, 3, for any (b i3 ) F x there exists β C such that (b i3 ) = β (b i1 ) + (1 β) (b i2 ) The condition that b i3 = 0 is equivalent to b i2 + β (b i1 b i2 ) = 0 Therefore, (20) F x = C \ b } i2 1 i m such that b b i1 i1 b i2 = 0 b i2 Notice that (b i1 ) = (b i2 ) Therefore, there has to be some i such that b i1 = b i2 Thus F x is isomorphic to C minus some points of cardinality between 1 and m, and hence the first part of the lemma follows The condition that χ(f x ) r is equivalent to the condition of some number of equalities b b i1 = b i2 and some number of overlaps among i2 b i1 b i2 Those conditions can be expressed by algebraic equations Thus, the locus of x such that χ(f x ) r is a closed algebraic subset in W 2 Example 310 When m = 3, the Euler characteristic χ(f x ) is in 0, 1, 2} More precisely, we can think of each F x to be the complement of 3 points in C, where we allow the points to overlap or go to infinity, but we do not allow all three points collapsing to one point The closed algebraic subset x W 2 χ(f x ) = 0} consists of 3 irreducible components of codimension 2

14 14 RODRIGUEZ AND WANG One of them is equal to (21) 1 1 α β α β α, β C, α = β and the other two are obtained from the first one by column permutations When x W 2 is in contained the the subset (21), the first point of the three-point configuration goes to infinity, and the other two points overlap Thus, the corresponding fiber F x is isomorphic to C minus a point The closed algebraic subset x W 2 χ(f x ) 1} consists of 6 irreducible components of codimension one Two of them are equal to t t (22) α β W 2 α, β, t C 1 t α 1 t β and (23) 1 α sα α sα 1 β sβ β sβ W 2 α, β, s C The other 4 components are obtained from the above two components by column permutations Notice that the listed two components are invariant under the permutation of the second and third columns When x W 2 belongs to the set (22), the first point of the three-point configuration goes to infinity When x W 2 belongs to the set (23), the second and the third point overlap Therefore, in any of these cases, F x is isomorphic to C minus two points Proof of Theorem 34 Thus far, we have shown with eqs (16) to (19), the following relations: (24) χ(ymn) o = χ(u n ) = χ(u n) = ( 1) n 1 χ(v n ) = ( 1) n 1 χ(w l ) 2 l n Recall that W k 2 = x W 2 χ(f x ) = k} and by Lemma 39 we have the following stratification, (25) W 2 = W 0 2 W 1 2 W m 1 2 Moreover, W2 k are locally closed algebraic subsets of W 2 The projection π 3 : W 3 W 2 induces a fiber bundle over each W2 k, and by definition of Wk 2, the fiber has Euler characteristic k With Lemma 38, we showed W l is isomorphic to an (l 2) fiber product of W 3 over W2 k Thus, restricting π l : W l W 2 to W2 k, the induced bundle s fiber has Euler characteristic ( k)l 2 Then, (26) χ(π 1 l (W k 2 )) = λ k ( k) l 2

15 THE MAXIMUM LIKELIHOOD DEGREE OF MIXTURES OF INDEPENDENCE MODELS 15 Since W2 k are locally closed algebraic subsets of W 2, π 1 l (W2 k ) are locally closed algebraic subsets of W l Therefore, the additivity of Euler characteristic implies the following, (27) χ(w l ) = χ(π 1 l (W2 k )) = λ k ( k) l 2 0 k m 1 0 k m 1 Here our convention is 0 0 = 1 By equations (24), (27) and Proposition 311, we have the following equalities ( χ(y o mn) = ( 1) n 1 2 l n 0 k m 1 λ k ( k) l 2 ) = ( 1) n 1 0 k m 1 λ k 1 ( k)n 1 1 ( k) The last line becomes the same as (15) by replacing k by i and showing λ 0 = 0 in Proposition 311 Proposition 311 Let λ 0 of Λ m be defined as above, then λ 0 = 0 We divert the proof of Proposition 311 to the end of the section Now, we specify to the case m = 3 for our first main result Theorem 312 [Main Result] The maximum likelihood degree of X 3n is given by the following formula (28) MLdeg(X 3n ) = 2 n+1 6 Proof In [13], the ML degree of X 32 and ML degree of X 33 are determined to be 1 and 10 respectively With this information it follows λ 1 + λ 2 = 0 and λ 2 = 12 by Theorem 34 The take away is that finitely many computations can determine infinitely many ML degrees Using these techniques we may be able to determine ML degrees of other varieties, such as symmetric matrices and Grassmanians, with a combination of applied algebraic geometry and topological arguments Proof of Proposition 311 Recall that λ 0 = χ(w 0 2 ) By definition, W0 2 consists of those (b ij) 1 i m,j=1,2 in W 2 such that the cardinality of the set b i1 /b i2 1 i m, b i1 = b i2 } is equal to 1 Notice that for (b ij ) 1 i m,j=1,2 W 0 2, (29) b i1 = b i1 = 0 1 i m 1 i m b i1 =b i2 b i2 =b i2 Therefore, we can define a C action on the set of m by 2 matrices (b ij ) 1 i m,j=1,2 } by setting t (b ij ) = (b ij ), where (30) b ij = b ij if b i1 = b i2 t b ij otherwise Now, it is straightforward to check this C action preserves W2 0 W2 0 Therefore, χ(w0 2 ) = 0 and the action is transitive on

16 16 RODRIGUEZ AND WANG 4 Recursions and closed form expressions In this section, we use Theorem 34 to give recursions for the Euler characteristic χ(y o mn) and thus the ML degree of X mn by (15) We break the recursions and give closed form expressions in Corollary The recurrence Recall that we have Y o mn := X o mn \ Z o mn By Theorem 34, giving a recursion for χ(y o mn) is equivalent to giving a recursion for MLdeg(X mn ) + ( 1) m+n 1 (minm, n} 1) The next theorem gives the recursion for χ(y o mn) Theorem 41 Fix m and let c i be the coefficient of t m i in (t + 1) m 1 r=1 (t r) For n > m, we have χ(ymn) o = c 1 χ(ym(n 1) o ) + c 2χ(Ym(n 2) o ) + + c mχ(ym(n m) o ) Proof By Theorem 34, we have χ(y o mn) = ( 1) n 1 1 i m 1 λ i i i m 1 λ i i + 1 in 1 for n 2 Therefore χ(ymn) o is an order m linear homogeneous recurrence relation with constant coefficients The coefficients of such a recurrence are described by the characteristic polynomial with roots 1, 1,, m 1, ie t m c 1 t m 1 c m = (t + 1) m 1 r=1 (t r) With these recurrences we determine the following table of ML degrees: n = m = 2 m = 3 m = 4 m = 5 m = 6 m = 7 m : m + 1 : m + 2 : m + 3 : m + 4 : m + 5 : m + 6 : Table 1 ML degrees of X mn Note that our methods are not limited to Table 41 We give closed form formulas in the next section for infinite families of mixture models 42 Closed form expressions In this subsection we provide additional closed form expressions Using an inductive procedure (described in the proof of Corollary 42), we determine Λ m for small m that can be extended to arbitrary m 2 2 Macaulay2 implementation at joisro/quicklinks/ecml16/mldegreemixturesm2

17 THE MAXIMUM LIKELIHOOD DEGREE OF MIXTURES OF INDEPENDENCE MODELS 17 Corollary 42 For fixed m = 2, 3, 7, the closed form expressions for MLdeg(X mn ) with m n are below: MLdeg(X 2n ) = 1 ( MLdeg(X 3n ) = n ( 3 2n 1) MLdeg(X 4n ) = n n ( 4 3n 1) MLdeg(X 5n ) = n n n ( 5 4n 1) MLdeg(X 6n ) = n n n n n 1) ( MLdeg(X 7n ) = n n n n n n 1) Proof We find these formulas using an inductive procedure to determine Λ m from Λ m 1 By Theorem 34, we have (15) gives us the following m 1 relations with n = 2, 3,, m: MLdeg (X m2) MLdeg (X m3) MLdeg (X mm) + ( 1)m 1 2 ( 1) m (m 1) = (m 1) (m 1) 2 1 m 1 2 m 1 (m 1) m 1 ( 1) 1 ( 1) 1 ( 1) 1 ( 1) 2 ( 1) 2 ( 1) 2 ( 1) m 1 ( 1) m 1 ( 1) m 1 For fixed m, this system of linear equations has 2m 2 unknowns: MLdeg(X mj ) for j = 2,, m and λ 1,, λ m 1 of Λ m By induction, we may assume we know Λ m 1 The Λ m 1 gives us a closed form expression for the ML degrees of X (m 1)j with j 2 Since MLdeg(X (m 1)j ) = MLdeg(X j(m 1) ), we have reduced our system of linear equations to m unknowns by substitution By Proposition 44, we have λ m 1 of Λ m equals (m 1) m! Substituting this value as well, we have a linear system of m 1 equations in m 1 unknowns: MLdeg(X mm ), λ 1, λ 2,, λ m 2 A simple linear algebra argument shows that there exists a unique solution of the system yielding each λ j of Λ m as well as MLdeg(X mm ) Using the inductive procedure described above, we determined the following table of Λ m to yield the closed form expressions we desired λ 1 λ 2 λ 3 λ 4 λ 5 λ 6 Λ 2 : 2 Λ 3 : Λ 4 : Λ 5 : Λ 6 : Λ 7 : Table 2 The λ i of Λ m λ 1 /2 λ 2 /3 λ m 1 /m Remark 43 With our recursive methods we determined the ML degree of X 20,20 to be While it is currently infeasible to track this large number of paths via numerical homotopy continuation as in [12], it may be possible to track much fewer paths using the topology of the variety Preliminary results by the 2016 Mathematics Research Community in Algebraic Statistics Likelihood Geometry Group for weighted independence models deform a variety with ML degree

18 18 RODRIGUEZ AND WANG one to a variety with large ML degree This deformation deforms the maximum likelihood estimate of one model to the estimate of another The stratifications presented here may lead to similar results in the case of mixtures Proposition 44 Let λ l of Λ m be defined as above Then λ m 1 of Λ m equals (m 1) m! Proof Recall that for λ m 1 of Λ m equals χ(w2 m 1 ) By definition, W2 m 1 consists of all (b ij ) 1 i m,j=1,2 W 2 such that b i1 = b i2 for all 1 i m and b i1 /b i2 are distinct for 1 i m Denote by B m the subset of (C \ 1}) m corresponding to m distinct numbers Then there is a natural map π : W2 m 1 B m, defined by (b ij ) (b 11 /b 12,, b m1 /b m2 ) The map is surjective Moreover, one can easily check that under the map π, W2 m 1 is a fiber bundle over B, whose fiber is isomorphic to the complement of m hyperplanes in C m 2 in general position By Lemma 35, the fiber has Euler characteristic ( 1) m 2 (m 1) The Euler characteristic of B m is equal to ( 1) m m! by induction In fact, B m is a fiber bundle over B m 1 with fiber homeomorphic to C \ m distinct points} Therefore, χ(w m 1 2 ) = ( 1) m 2 (m 1) ( 1) m m! = (m 1) m! 5 Conclusion and additional questions We have developed the topological tools to determine the ML degree of singular models We proved a closed form expression for 3 n matrices with rank 2 conjectured by [12] In addition, our results provide a recursion to determine the ML degree of a mixture of independence models where the first random variable has m states and the second random variable has n states Furthermore, we have shown how a combination of computational algebra calculations and topological arguments can determine an infinite family of ML degrees The next natural question is to determine the ML degree for higher order mixtures (rank r matrices for r > 2) Our results give closed form expressions in the corank 1 case by maximum likelihood duality [5] Maximum likelihood duality is quite surprising here because our methods would initially suggest that the corank 1 matrices have a much more complicated ML degree it would be very nice to give a topological proof in terms of Euler characteristics of maximum likelihood duality One possible approach is by applying these techniques to the dual maximum likelihood estimation problem described in [21] Another question concerns the boundary components of statistical models as described in [18] for higher order mixturescan we also use these topological methods to give closed form expressions of the ML degrees of the boundary components of the statistical model? Finally, one should notice that the formulas in Corollary 42 for the ML degrees involve alternating signs It would be great to give a canonical transformation of our alternating sum formula into a positive sum formula One reason why this might be possible is motivated by the work of [11] Here, entries of the data u are degenerated to zero and some of the the critical points go to the boundary of the algebraic torus (ie coordinates go to zero) Based on how the critical points go to the boundary, one partitions the ML degree into a sum of positive integers

19 THE MAXIMUM LIKELIHOOD DEGREE OF MIXTURES OF INDEPENDENCE MODELS 19 6 Acknowledgements We would like to thank Nero Budur and Jonathan Hauenstein for introducing the authors to one another We would also like to especially thank Jan Draisma for his comments to improve an earlier version of this manuscript We thank Sam Evens for the explanation of Theorem 26 References [1] N Budur and B Wang The signed Euler characteristic of very affine varieties arxiv: , 2014 [2] N Budur and B Wang Bounding the maximum likelihood degree To appear in Math Res Lett, 2015 [3] F Catanese, S Hoşten, A Khetan, and B Sturmfels The maximum likelihood degree Amer J Math, 128(3): , 2006 [4] A Dimca Sheaves in topology Universitext Springer-Verlag, Berlin, 2004 [5] J Draisma and J Rodriguez Maximum likelihood duality for determinantal varieties Int Math Res Not IMRN, (20): , 2014 [6] S Evens and I Mirković Characteristic cycles for the loop Grassmannian and nilpotent orbits Duke Math J, 97(1): , 1999 [7] J Franecki and M Kapranov The Gauss map and a noncompact Riemann-Roch formula for constructible sheaves on semiabelian varieties Duke Math J, 104(1): , 2000 [8] V Ginsburg Characteristic varieties and vanishing cycles Invent Math, 84(2): , 1986 [9] M Goresky and R MacPherson Stratified Morse theory In Singularities, Part 1 (Arcata, Calif, 1981), volume 40 of Proc Sympos Pure Math, pages Amer Math Soc, Providence, RI, 1983 [10] E Gross, M Drton, and S Petrović Maximum likelihood degree of variance component models Electron J Stat, 6: , 2012 [11] E Gross and J I Rodriguez Maximum likelihood geometry in the presence of data zeros In Proceedings of the 39th International Symposium on Symbolic and Algebraic Computation, ISSAC 14, pages , New York, NY, USA, 2014 ACM [12] J D Hauenstein, J I Rodriguez, and B Sturmfels Maximum likelihood for matrices with rank constraints J Algebr Stat, 5(1):18 38, 2014 [13] S Hoşten, A Khetan, and B Sturmfels Solving the likelihood equations Found Comput Math, 5(4): , 2005 [14] S Hosten and S Sullivant The algebraic complexity of maximum likelihood estimation for bivariate missing data In P Gibilisco, E Riccomagno, M P Rogantin, and H P Wynn, editors, Algebraic and Geometric Methods in Statistics, pages Cambridge University Press, 2009 Cambridge Books Online [15] R Hotta, K Takeuchi, and T Tanisaki D-modules, perverse sheaves, and representation theory, volume 236 of Progress in Mathematics Birkhäuser Boston, Inc, Boston, MA, 2008 Translated from the 1995 Japanese edition by Takeuchi [16] J Huh The maximum likelihood degree of a very affine variety Compos Math, 149(8): , 2013 [17] J Huh and B Sturmfels Likelihood geometry In Combinatorial algebraic geometry, volume 2108 of Lecture Notes in Math, pages Springer, Cham, 2014 [18] K Kubjas, E Robeva, and B Sturmfels Fixed points of the EM algorithm and nonnegative rank boundaries Ann Statist, 43(1): , 2015 [19] P Orlik and H Terao Arrangements of hyperplanes, volume 300 of Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences] Springer-Verlag, Berlin, 1992 [20] L Pachter and B Sturmfels Statistics In Algebraic statistics for computational biology, pages 3 42 Cambridge Univ Press, New York, 2005 [21] J I Rodriguez Maximum likelihood for dual varieties In Proceedings of the 2014 Symposium on Symbolic- Numeric Computation, SNC 14, pages 43 49, New York, NY, USA, 2014 ACM [22] C Uhler Geometry of maximum likelihood estimation in Gaussian graphical models Ann Statist, 40(1): , 2012

20 20 RODRIGUEZ AND WANG Jose Israel Rodriguez, The University of Chicago, Dept of Statistics, 5734 S University Ave Chicago, IL address: joisro@uchicagoedu URL: Botong Wang, University of Wisconsin-Madison, Department of Mathematics, Van Vleck Hall, 480 Lincoln Drive, Madison, WI address: wang@mathwiscedu URL: