A Derivation of the 3-entity-type model

Transcription

1 A Derivation of the 3-entity-type model The 3-entity-type model consists of two relationship matrices, a m n matrix X and a n r matrix Y In the spirit of generalized linear models we define the low rank representations of the relationship matrices to be X f 1 UV T ) and Y f V Z T ) where f 1 : R m n R m n and f : R n r R n r are the prediction links, and U R m k, V R n k, and Z R r k are the parameters of the model for k Z We further define parameter transformations G : R m k R m k, H : R n k R n k, I : R r k R r k, to model prior knowledge about our parameters, eg, regularizers We additionally required the convex conjugate, G U) = sup [U A GA)], U doma) where U A = tru T A) = ij U ija ij is the matrix dot product The overall loss function for our model is 1) LU, V, Z W, W) = αl 1 U, V W) + 1 α)l V, Z W) where we introduce fixed weight matrices for the observations, W R m n and W R n r The individual objectives on the reconstruction of X and Y are, respectively, ) 3) L 1 U, V W) = W F 1 UV T ) X UV T) + G U) + H V ), L V, Z W) = W F V Z T ) Y V Z T) + H V ) + I Z) The objective LU, V, Z W, W) is convex in any one of its arguments, but is in general non-convex in all its arguments As such, we use an alternating minimization scheme that optimizes one factor U, V, or Z at a time This appendix describes the derivation of both gradient and Newton update rules for U, V, and Z For completeness, Section A1 reviews useful definitions from matrix calculus The gradient of the objective, with respect to each argument is derived in Section A Finally, by assuming the loss is decomposable, we derive the Newton update in Section A3, whose additional cost over its gradient analogue is essentially a factor of k times more expensive A1 Matrix Calculus For the sake of completeness we define matrix derivatives, which generalizes both scalar and vector derivatives Using this definition of matrix derivatives, we also generalize the scalar chain and product rules to matrices The discussion herein is based on Magnus et al [1, ] Let M be an n q matrix of variables, where m j denotes the j-th column of M The vec-operator vecm yields an nq 1 matrix that stacks the columns of M: vecm = While there are several common and incompatible) definitions of matrix derivatives, the derivative of a n 1 vector f with respect to a m 1 vector x is almost universally defined as m 1 m m q f 1 Dfx) f x 1 x T = f n x 1 The matrix derivative of an m p matrix function Q of an n q matrix of variables M contains mnpq partial derivatives, and the matrix derivative arranges these partial derivatives into a matrix We define the matrix f 1 x m f n x m 1

2 derivative by coercing matrices into vectors, and using the above definition of the vector derivative, [QM)] 11 DQM) vecqm) vecm) T = [QM)] mp [QM)] 11 [QM)] mp which is an mp nq matrix of partial derivatives This definition encompasses vector and scalar derivatives as special cases The advantages of this formulation include i) unambiguous definitions for the product and chain rules, and ii) we can easily convert DQM) to the more common definition of the matrix derivative, M = m n1 m 1q via the first identification theorem [, ch 9] We additionally require the matrix chain [, pg 11] and matrix product [1] rules: Definition 1 Matrix Chain Rule) Given functions ϕ : R n q R m p, ϕ 1 : R l r R m p, and ϕ : R n q R l r the derivative of ϕm) = ϕ 1 Y ), where Y = ϕ M), is where A B is the matrix product DϕZ) = Dϕ 1 Y ) Dϕ Z) Definition Matrix Product Rule) Given an m p matrix ϕ 1 M) and a p r matrix ϕ M) the derivative of ϕ 1 M)ϕ M) with respect to M, Dϕ 1 ϕ )M) is where A B is the Kronecker product,, Dϕ 1 ϕ )Z) = ϕ M) T I m ) Dϕ 1 M) + I r ϕ 1 M)) Dϕ M) A Computing the Gradient To compute the derivative of Equation 1 with respect to U, V, and Z we require the following three lemmas: Lemma 1 For any differentiable function F : R m n R F 1 UV T ) = f 1 UV T )V, F V Z T ) = f V Z T )Z, F 1 UV T ) = f 1 UV T ) T U F V Z T ) = f V Z T ) T V Proof We only derive the result for ϕu) = F 1 UV T ) The proof is similar for the other three cases ϕu) can be expressed as the composition of functions: ϕu) = F 1 Y ), Y = ϕ U), ϕ U) = UV T Using the matrix chain rule DϕU) = DF 1 Y ) Dϕ U), we note that 4) Using the matrix product rule 5) DF 1 Y ) = vecf 1UV T ) vecuv T ) T = f 1 UV T ) Dϕ U) = V I m ) vecu vecu) T + I n U) vecv T vec U) T = V I m ) Combining equations 4 and 5 using the matrix chain rule yields DϕU) = vecf 1 UV T )V ) T The result follows immediately from the first identification theorem

3 Lemma Given that the entries of U and V are distinct, X UV T ) Y V Z T ) = XV, = Y Z, X UV T ) = X T U Y V Z T ) = Y T V Proof We derive the result for X UV T )/, the other three derivations are similar To avoid a long digression into matrix differentials, we prove the result by element-wise differentiation Noting that X UV T = x ji u jk v ik, i j we compute the derivative with respect to v pq k X UV T ) v pq = i = j j x ji u jk v ik v pq k x jp u jq = X T U) pq Since the result holds for all p {1,,n} and q {1,,k} it follows that X UV T )/ = X T U Lemma 3 For the l regularizers where a, b, c > 0 controls the strength of regularizers larger values = weaker regularization): GU) = a U Fro the derivatives for the convex conjugates are G U), HV ) = b V Fro = U a = A, H V ), IZ) = c Z Fro, = V b = B, I Z) = Z c = C Proof The result is easily proven by finding the convex conjugate and differentiating it Combining Lemmas 1,, and 3, and denoting the Hadamard element-wise) product of matrices A B, we have that 6) 7) 8) LU, V, Z) = α W f 1 UV T ) X )) V + A, LU, V, Z) = α W f 1 UV T ) X )) T U + 1 α) W f V Z T ) Y )) Z + B, LU, V, Z) = 1 α) W f V Z T ) Y )) T V + C Setting the gradient equal to zero yields update equations either for A, B, C or for U, V, Z An advantage of using a gradient update is that we can relax the requirement that the links are differentiable, replacing gradients with subgradients in the prequel A3 Computing the Newton Update One may be satisfied with a gradient step However, the assumption of a decomposable loss means that most second derivatives are set to zero, and a Newton update can be done efficiently, reducing to row-wise 3

4 optimization of U, V, and Z For the subclass of models where Equations 6-8 are differentiable and the loss is decomposable define, qu i ) = α W i f 1 U i V T ) X i )) V + Ai qv i ) = α W i f 1 UV T i ) X i )) T U + 1 α) Wi f V i Z T ) Y i ) ) Z + B i qz i ) = 1 α) W i f V Z T i ) Y i) ) T V + Ci Any local optimum of the loss corresponds to roots of the equations {qu i )} m i=1, {qv i )} n i=1, and {qz i )} r i=1 Using a Newton step, the update for U i is 9) U new i = U i η [qu i )] [q U i )] 1, where η [0, 1] is the step length, chosen using line search with the Armijo criterion [3, ch 3] The Newton steps for V i and Z i are analogous To describe the Hessians of the loss, q, we introduce the following notation for the Hessians of G, H and I : G i diag G U i )), H i diag H V i )), I i diag I Z i )) For case of l -regularization G i = diaga 1 1) For conciseness we also introduce the following terms: D 1,i diagw i f 1U i V T )), D,i diagw i f 1UV T i )), D 3,i diag W i f V T i Z)), D 4,i diag W i f V ZT i )) This allows us to describe the Hessians of Equation 1 with respect to each row of the parameter matrices Lemma 4 The Hessians of Equation 1 with respect to U i, V i, and Z i are q U i ) qu i ) i q Z i ) qz i ) i q V i ) qv i ) i = αv T D 1,i V + G i = 1 α)v T D 4,i V + I i = αu T D,i U + 1 α)z T D 3,i Z + H i Proof We prove the result for q U i ), noting that the other derivations are similar Since q ) and its argument U i are both vectors, DqU i ) is identical to qu i )/ i Ignoring the terms that do not vary with U i, DqU i ) = D [ αw i fu i V T ))V + A i ] = αd [ W i fu i V T ))V ] + DA i { = α V T I 1 ) vecw i fu i V T )) + I k V ) vecv } + vec G U i ) vecu i vecu i vecu i = α { V T I 1 ) D 1,i V + I k V ) 0 } + G i = αv T D 1,i V + G i Plugging the gradient qu i ) and the Hessian q U i ) into Equation 9 yields 10) U new i q U i ) = U i αv T D 1,i V + G i ) + αη W i X i fu i V T ) )) V ηa i = αu i V T D 1,i V + αη W i X i fu i V T ) )) D 1 1,i D 1,iV + U i G i ηa i = α U i V T + η W i X i fu i V T)) D 1 1,i ) D1,i V + U i G i ηa i 4

5 Likewise for Z i, 11) Z new i q Z i ) = 1 α) Z i V T + η W i Y i fv Zi T ) )) ) T D 1 4,i D 4,i V + Z i I i ηc i The derivation of the update for V i is similar, since LU, V, Z W, W) is a linear combination and the differential operator is linear: { Vi new q V i ) = α V i U T + η W i ) } X i fuvi T ))) T D 1 1) i D,i U + { 1 α) V i Z T + η Wi Y i fv i Z T ) )) ) } D 1 3,i D 3,i Z + V i H i ηb i While D {D j,i } 4 j=1 may not be invertible, ie, a diagonal entry is zero when the corresponding weight is zero, the form of the update equations shows that this does not matter If a diagonal entry in D is zero, replacing its corresponding entry in D 1 by any nonzero value does not change the result of Equations 10, 11, and 1, as the zero weight cancels it out References [1] J R Magnus and K M Abadir On some definitions in matrix algebra Working Paper CIRJE-F-46, University of Tokyo, Feb 007 [] J R Magnus and H Neudecker Matrix Differential Calculus with Applications in Statistics and Econometrics John Wiley, 007 [3] J Nocedal and S J Wright Numerical Optimization Springer Series in Operations Research Springer,