Interpretable Matrix Factorization with Stochasticity Constrained Nonnegative DEDICOM

Transcription

1 Interpretable Matri Factoriation ith Stochasticit Constrained Nonnegatie DEDICOM Rafet Sifa 1,2, Cesar Ojeda 1, Kostadin Cejoski 1, and Christian Bauckhage 1,2 1 Fraunhofer IAIS, Sankt Augustin, German 2 Uniersit of Bonn, Bonn, German.multimedia-pattern-recognition.info {name.surname}@iais.fraunhofer.de Abstract. Decomposition into Directed Components (DEDICOM) is a special matri factoriation technique to factorie a gien asmmetric similarit matri into a combination of a loading matri describing the latent structures in the data and an asmmetric affinit matri encoding the relationships beteen the found latent structures. Finding DEDI- COM factors can be cast as a matri norm minimiation problem that requires alternating least square updates to find appropriate factors. Yet, due to the a DEDICOM reconstructs the data, unconstrained factors might ield results that are difficult to interpret. In this paper e derie a projection-free gradient descent based alternating least squares algorithm to calculate constrained DEDICOM factors. Our algorithm constrains the loading matri to be column-stochastic and the affinit matri to be nonnegatie for more interpretable lo rank representations. Additionall, unlike most of the aailable approimate solutions for finding the loading matri, our approach takes the entire occurrences of the loading matri into account to assure conergence. We ealuate our algorithm on a behaioral dataset containing pairise asmmetric associations beteen ariet of game titles from an online platform. Keords: Unsuperised Learning, Matri Factoriation, Constrained Optimiation, Behaior Analsis 1 Introduction Matri and tensor factoriation methods hae been idel used in data science applications for mostl understanding hidden patterns in the datasets and learning representations for ariet of prediction tasks and recommender sstems [1, 5, 10, 11, 13, 16, 17]. Decomposition into Directed Components (DEDI- COM) [8] is a matri and tensor factoriation technique and a compact a of finding lo rank representations from asmmetric similarit data. The method has found applications in arious data science problems including analsis of temporal graphs [1], first order customer migration [15], natural language processing [4], spatio-temporal representation learning [2, 16] and link prediction in Copright 2017 b the paper s authors. Coping permitted onl for priate and academic purposes. In: M. Leer (Ed.): Proceedings of the LWDA 2017 Workshops: KDML, FGWM, IR, and FGDB. Rostock, German, September 2017, published at

2 Fig. 1: Illustration of to-a DEDICOM to partition a gien similarit matri S R n n encoding pair-ise asmmetric relations among n objects into a combination of a loading matri A R n k and a square affinit matri R R k k. In this representation A contains the latent factors here the columns describe the hidden structures in S and the relations among these structures are encoded b the asmmetric affinit matri R. Variet of constrained ersions of DEDICOM has been preiousl studied so as to improe the interpretabilit and the performance of finding appropriate factors. relational and knoledge graphs [13], here the source of the addressed problems ere primaril about analing asmmetric relationships beteen predefined entities. Formall, gien an asmmetric similarit matri S R n n encoding pairise asmmetric relations (i.e. s ij s ji ) among n objects, to-a DEDICOM finds a loading matri A R n k and a square affinit matri R R k k for representing S as S ARA T. (1) More precisel, DEDICOM approimation of the asmmetric association beteen elements i and j (i.e. s ij ) can be formulated in column ector notation as s ij a T i:ra j: = n ( ) T aib r b: aj: = b=1 n b=1 c=1 n a ib r bc a jc, (2) here a i: and a j: represent the ith and jth ro of A respectiel and r b: represents the bth ro of R. In this representation A contains the latent factors here the columns describe the hidden structures in S and the relations among these structures are encoded b the asmmetric affinit matri R. A pictorial illustration of to-a DEDICOM factoriation is shon in Fig. 1. Considering the three factor approimations of the similarit alues as in (1) and (2), the interpretation (especiall ith respect to scale) of the hidden structures from gien factor matrices A and R might ield misleading results ith the presence of negatie alues [15]. That is, the interpretation of the results is limited to onl consider nonnegatie affinities in R or the positie or negatie loadings of the corresponding points in (2). Additionall, hen analing nonnegatie data matrices (such as ones containing probabilities, counts and etc.), it is usuall beneficial to consider nonnegatie factor matrices that can be considered as condensed (or compressed) representations that can be used for informed decision making and representation learning [1, 8, 15, 16].

3 In this ork e address the issue of interpretabilit of the DEDICOM factors b introducing a conerging algorithm as an alternatie to the preiousl proposed methods in [1, 13, 15, 16]. Our method constrains the loading matri A to contain column stochastic ectors and (similar to [15, 16]) the affinit matri R to be nonnegatie. Formall this amounts to factorie a gien asmmetric similarit matri S as in (1) b enforcing r ij 0 {i, j} (3) and n a cb 0 a qb = 1 {c, b}. (4) q=1 Compared to the additie-scaling based representation introduced in [15, 16], constraining the columns of A to contain probabilities has the direct adantage of interpreting each element of A as the representatieness alue of the particular loading it represents. That is, the alue of a ib represents ho much the ith element in the dataset contributes to the bth latent factor. This is indeed parallel to the idea of Archetpal Analsis [3, 6, 17], here the miture matrices respectiel determine ho much a data point and archetpes contribute to respectiel constructing the archetpes and reconstructing each data point using the archetpes. In the folloing e ill first describe the general alternating least squares optimiation frameork and gie an oerie of the algorithms to find appropriate constrained and unconstrained DEDICOM factors. After that e ill introduce our algorithm b first studing our problem setting and deriing the algorithm step-b-step. This ill be folloed b a case stud coering a real orld application here e ill anale game-onership patterns from data containing asmmetric associations beteen games using the factors e etract b running our algorithm. Finall, e ill conclude our paper ith a summar and some directions for future ork. 2 Algorithms for Finding DEDICOM Factors Finding appropriate DEDICOM factors for matri decomposition can be cast as a matri norm minimiation problem ith the objectie E(A, R) = S ARA T 2, (5) hich is minimied ith respect to A and R. Non-cone minimiation problems of this kind usuall follo an iteratie alternating least squares (ALS) procedure here at each iteration e optimie oer a selected factor treating the other factors fied. It is important to note that, the minimied objectie function in (5) is cone in R for fied A hereas not cone in A for fied R, hich leads to optimal and sub-optimal approimate solutions hen respectiel updating R and A.

4 Starting ith defining update rules for the affinit matri R, e note that ith fied A, minimiation of (5) is a matri regression problem [1,13,15] ith global optimum solution R A SA T, (6) here A indicates Moore-Penrose pseudoinerse of A. Furthermore, if A is constrained to be column orthogonal (i.e. A T A = I k, here I k is the k k identit matri), the update for R simplifies to R A T SA [15]. Here the updates are not constrained to fall into a particular class of domain and might result in ith affinit matrices containing negatie elements. Constraining R hen minimiing (5) to contain onl nonnegatie alues can be cast as a nonnegatie least squares problem b transforming the arguments of (5) as E(R) = ec(s) ec(ara T ) 2 (7) here the ec operator ectories (or flattens) a matri B R m n as ec(b) = [b 11,..., b m1,..., b 1n,..., b mn ] T. (8) It is orth mentioning that since ectoriing (5) as shon in (7) does not affect the alue of the norm [15], e can make use of the propert of ectoriation to represent a ectoriation of multiplications of matrices as matri ector multiplication to rerite (7) as E(R) = ( ) ec(s) A A ec(r)) 2, (9) here represents the Kronecker product. As noted in [15], the epression in (9) can indeed be mapped to constrained linear regression problem [12] ith a conerging result (through an actie set algorithm) to define an update as R argmin R ( ) ec(s) A A ec(r)) 2 r ij 0 i, j. (10) Haing identified alternating least squares updates for constrained and unconstrained affinit matri R, e no turn our attention to define updates for the loading matri A. To this end, there hae been to different directions folloed preiousl: approimating the minimiation of (5) b treating a particular factor ith A constant [1, 13] and directl minimiing (5) b means of gradient based approaches. To start ith, former method considers stacking matri S and matri S T as [ S S T ] [ ] [ ] = ARA T AR T A T = A RA T R T A T, (11) and soles for A keeping A T fied [1, 13] to come up ith an update for A as A ( )( 1. SAR T + S T AR RA T AR T + R T A AR) T (12)

5 The latter method s ALS update for the loading matri A is based on directl minimiing (5) b considering first order gradient based optimiation [15, 16], hich in each update moes the current solution for A in the opposite direction of gradient of (5). To derie the gradient matri e start b defining (5) in terms of the trace operator as E(A, R) = tr [S ] T S S T ARA T AR T A T S + AR T A T ARA T [S ] T S 2S T ARA T + AR T A T ARA T = tr = tr [ ] [ ] [ ] S T S 2 tr S T ARA T + tr AR T A T ARA T. (13) [ ] Since traces are linear mappings and the term tr S T S does not depend on A, minimiing E(A, R) in (13) ith respect to A is equialent to minimiing E(A) hich e define as E(A) = E 1 (A) + E 2 (A) (14) gien that [ ] E 1 (A) = 2 tr S T ARA T (15) and [ ] E 2 (A) = tr AR T A T ARA T. (16) Considering the orthogonalit constraint on A, the term in (16) becomes independent of A. That is, since traces are inariant under cclic permutation e can reformulate (16) as [ ] [ ] [ ] E 2 (A) = tr AR T A T ARA T = tr R T A T ARA T A = tr R T R, (17) hich allos for onl considering the minimiation of the error term in (15) [15, 16]. Consequentl, e can define a gradient descent based update b considering the gradient matri E 1 (A) A = 2 ( S T AR + SAR T ), (18) hich ith a learning rate η A allos us to define an update for A as A A + 2η A (S T AR + SAR T ). (19) As a final step, in oder to assure the orthogonalit constraint on A e project the updated A b considering its QR decomposition [15, 16] A = QT, here Q R n k is orthogonal and T R k k is inertible upper triangular, and folloing the update A Q. This concludes our oerie of the methods to come up ith appropriate factors. For more detailed information about the alternating least squares solutions for DEDICOM factoriation for matrices and tensors e refer the reader to [1, 13, 15, 16].

6 3 Stochasticit Constrained Nonnegatie DEDICOM In this section e ill present a ne ALS algorithm to particularl consider DEDICOM factoriation ith column stochastic loadings and nonnegatie affinities for interpretabilit of the resulting factors. To begin ith, as noted in [15,16], e can assure nonnegatie affinities R b soling the nonnegatie least squares problem introduced in [12] as in (10). Net considering the theoretical properties of constraining the columns of matri A to (4), e note that each column resides in the standard simple n 1, hich is the cone hull of the standard basis ectors of R n. Formall, gien that δ ij represents the Kronocker delta and V = { 1, 2,..., n i = [δ i1,..., δ in ] T } is the set of the standard basis ectors of R n, the standard simple n 1 is defined as the compact set { n n } n 1 = α i i α i = 1 α i 0 i [1,..., n]. (20) i=1 i=1 This indicates that, finding an appropriate column stochastic matri A minimiing the cone objectie function in (5) can be reduced don to a constrained optimiation problem in the cone compact simple n 1. Constrained optimiation problems of this kind can be easil tackled using the Frank-Wolfe algorithm [7, 9], hich is also knon as the conditional gradient method [9]. The main idea behind the Frank-Wolfe algorithm is to minimie differentiable cone functions oer their domains forming compact cone sets using iteratie first-order Talor approimations until achieing conergence. Formall gien a differentiable cone function f : S R oer a compact cone set S, the Frank-Wolfe algorithm aims to sole for S, b iteratiel soling for min f() (21) s t = min s S st f( t ), (22) here f( t ) is the gradient of the optimied function f ealuated at the current solution t. Folloing that, at each iteration t, the algorithm estimates the ne solution b ealuating t+1 = t + α t (s t t ), (23) here α t (0,..., 1] is the learning rate that is usuall selected b performing a line search on (23) or set to be monotonicall decreasing as a function of t [7,9]. Additionall, one particular adantage of this optimiation method is that it allos for ɛ-approimation for a gien maimum number of iterations t ma [7,9]. Considering our special case, since the set of ertices of a standard simple is equialent to the set of standard basis V in R n [3], at each iteration Frank-Wolfe

7 algorithm moes the current solution into the direction of one of the standard basis until achieing conergence. Folloing that, since the stochasticit constraint does not allo for a simplification as in (17), e require the full deriation of the gradient matri E(A) A from (13) so as to adapt the Frank-Wolfe algorithm to find appropriate stochastic columns for A. To this end e start b considering E 2 (A) from (16) in terms of a scalar summation as tr [ AR T A T ARA T ] = u a u r a a r a u (24) and consider its partial deriatie ith respect to an arbitrar element a ij of A that can be ritten due to linearit of differentiation as E 2 (A) = u (a u r a a r a u ). (25) The epression in (25) can be further simplified b the product rule epansion E 2 (A) = u u u (a u r )(a a r a u ) + (a )(a u r a r a u ) + (a )(a u r a r a u ) + (a u r )(a u r a a ) (26) u and ealuating the deriaties results in E 2 (A) = r a a r a u + a u r a r a u + u a u r a r a u + u r a u r a a. (27) Finall, reformulating the four summations in (27) in terms of matri multiplications as E 2 (A) [ [ = 2 AR T A T AR ]ij ] + 2 ARA T AR T (28) ij

8 Randoml initialie alid A and R Let V = { 1, 2,..., n i = [δ i1,..., δ in ] T } hile Stopping condition is not satisfied do //Update A ith Frank-Wolfe procedure hile t t ma and updates of A are not small do //Define the gradient ( matri ) G = E(A) A = 2 AR T A T AR + ARA T AR T S T AR SAR T //Define the monotonicall decreasing learning rate α 2/(t + 2) for b {1,..., k} do //Find the minimier simple erte i i = argmin g bl l //Update columns a b of A a b a b + α( i a b ) //Update the iterator t t + 1 //Update R b soling the nonnegatie least squares problem ( ) R argmin ec(s) A A ec(r)) 2.r.t r ij 0 i, j R Alg. 1: A Frank-Wolfe based projection-free ALS algorithm to find Seminonnogatie DEDICOM factors ith column stochastic loading matri A and nonnegatie affinit matri R. At each iteration of the algorithm, e alternate beteen finding an optimal column stochastic A keeping R fied and finding a nonnegatie matri R keeping A fied. Note that the learning rate here is set to decrease monotonicall. allos us to define the gradient matri E2(A) A as E 2 (A) A = 2 ( AR T A T AR + ARA T AR T ). (29) Combining the results from (18) and (29) the full gradient matri of the minimied error norm for A is calculated as follos E(A) ( ) A = 2 AR T A T AR + ARA T AR T S T AR SAR T, (30) hich allos us to define a column-ise Frank-Wolfe algorithm to come up ith optimal column stochastic loading matri A minimiing (5). We list the essential steps of our adaptation of the projection free Frank-Wolfe algorithm in Alg. 1. In a nutshell, our algorithm starts b randoml initialiing (alid) matrices A and R, continues ith alternating beteen the Frank-Wolfe optimiation updates to come up ith optimal column stochastic A and alternating least squares

9 RSS ALS Iterations Fig. 2: Illustration of the eolution of the Residual Sum of Squares (RSS) for factoriing our empirical conditional probabilit matri indicating asmmetric relations beteen games using DEDICOM ith k = 3. Our proabl conerging algorithm monotonicall decreases the RSS at each iteration. updates for nonnegatie R until a predefined stopping condition is met. The stopping conditions are usuall selected based on reaching a maimum number of alternating least squares updates (not to be mied ith the maimum iteration count t ma of the Frank-Wolfe updates for columns of A in Alg. 1), haing minor subsequent changes in the minimied objectie (for our case the matri norm in (5)) or combining both of these conditions. 4 A Business Intelligence Application: Analing Asmmetric Game Onership Associations in an Online Gaming Platform In order to ealuate our algorithm, e consider a use-case from game analtics [14, 16], here e anale the asmmetric relationships among arious tpes of games. To this end e consider a dataset from an online gaming platform called Steam, hich hosts thousands of games of arious genres to a large plaer-base ith its sie ranging from 9 to 15 million 3 (dail actie) plaers. We used the dataset from [14] hich contains game onership information of more than si million users about 3007 titles. Representing the onership information as a bipartite matri Y {0, 1} m n indicating onership of information of m plaers for n games, e construct the asmmetric association among games in terms of their pairise empirical conditional probabilities. To this end, e define the directional similarit from game i to game j as s ij = {c ci 0 c} {b bj 0 b}, (31) {c ci 0 c} 3

10 Loadings Identifier Dominant Games a 1 TF 2/FPS TF 2, CS: Source and Red Orchestra: Ostfront a 2 Indie/Platformer WSS, Ethan and Ooi: Earth Adenture a 3 Flagships/AAA Left 4 Dead 2, Dota 2, Skrim, HL 2, Garr s Mod Table 1: Selection of games ith high loading alues for the analed dataset. here indicates set cardinalit and pq represents if the pth plaer ons the qth game. It is important to note that (31) is inherentl asmmetric and a special case of the so-called Tersk inde [18] ith prototpe and ariant eights of respectiel 1 and 0 (or ice ersa). Since the number of computations required to obtain the similarit matri S scales to O(mn 2 ), e parallelied the procedure of computing the similarit alues in (31) b utiliing a hbrid paralleliation technique that simultaneousl eploits both distributed and shared memor architectures in order to speedup the computation time. Haing obtained the asmmetric similarit matri ith the empirical conditional probabilit alues in S, e factorie it using our algorithm and present the results ith k = 3. To eplicitl ignore modeling the self-loops, at each alternating least squares iteration, e replaced the diagonal of S ith the diagonal of its current reconstruction [1]. In Fig. 2 e sho the residual sum of squares, hich decreases monotonicall and conerges to a minimum after 30 iterations. Analing the resulting factors, e obsere that the resulting to columns (or modes) a 1 and a 2 of A are sparse hereas the last column as dense in terms of non-ero probabilit alues, here the most dominant games ar from column to column. In Tbl. 1 e present the games ith high loadings in their corresponding column and obsere that mostl the free-to-pla flagship game Team Fortress 2 (TF 2), folloed b the FPS games CS: Source and Red Orchestra: Ostfront 41-45, contribute to a 1.On the other hand, the indie-platformer games Wooden Sen SeY (WSS), Ethan (Meteor Hunter) and Ooi: Earth Adenture contribute mostl to a 2. Finall, the mostl dense a 3 has er high loadings for all of the flagship games of the analed platform and other AAA games including Left 4 Dead 2, Dota 2, Skrim, Half-life 2 (HL 2) and Garr s Mod. Analing the resulting ro-normalied affinit matri that encodes the asmmetric relations beteen the modes from Fig. 3, the first ros indicates tendencies of TF 2 plaers more to the indie-platformer mode than to the AAA games because of the fact that TF 2 is mostl a singular game that people primaril prefer in the Steam platform [14]. On the other hand, inline ith the results from [14] the opposite directional associations, namel, from the flagships to the TF 2 are high hich is related to the fact that majorit of the plaers plaing one or more of the flagship games also prefer TF 2 [14]. Finall analing the second mode, similar to the results in [15] the self association is the highest association e obsere (indicating plaers preferring mostl to sta in the same genre), this is folloed b high associations to the TF 2 and the mode related to the flagship and AAA games.

11 TF2/FPS Indie/Platform Flagships/AAA TF2/FPS Indie/Platform Flagships/AAA Fig. 3: Results of factoriing the empirical conditional probabilit graph created for the pair-ise game onership information of more than si million plaers of an online gaming platform. The normalied nonnegatie affinit matri R shos relationships beteen groups automaticall determined b the DEDICOM algorithm. It is orth mentioning that, parallel to the results in [1], b increasing the number of modes k for the loading matri A, e obsered more detailed partitions regarding the modeled patterns. Running our algorithm ith k = 2, e obsered a higher reconstruction error and that the indie-platformer mode a 2 remains to be one of the factors hereas the games contributing to a 3 hae merged ith ones contributing to mode a 1. On the other hand, considering the resulting factors hen e ran our algorithm ith k = 4, e obtained a slight improement regarding the reconstruction error and obsered that the modes a 1 and a 2 remained the same, hoeer, mode a 3 has split into to different modes here both contain the same games as in a 3 and the ne mode contains additional indie and plaformer games (reducing the probabilit on the games of a 3 ) such as Finding Tedd, Gentlemen, Grai and Face Noir. 5 Conclusion and Future Work In this ork e studied the DEDICOM model to factorie asmmetric similarit matrices into a combination of a lo rank loading matri and an affinit matri. We gae an oerall oerie about the theoretical details and algorithms to come up ith appropriate factors. In essence, the affinit matri matri R indicates different directional structures that are determined b the columns of the loading matri A. Haing defined our objectie function to be the matri norm of the difference beteen the factoried asmmetric similarit matri and its DEDICOM reconstruction, e deried the gradient matri for this function ith respect to the loading matri A and presented a ariant of the Frank-Wolfe algorithm to obtain interpretable column stochastic loadings and nonnegatie affinities.running our algorithm on a dataset containing asmmetric pairise

12 relationships beteen more than 3000 games, e found interesting patterns indicating directional preferences among games. Our future ork inoles analing the performance of our model for different tasks including link prediction and representation learning [13, 16]. Considering the tensor etensions in [13, 16], our future ork also inoles etending the proposed model to tensors factoriations hich can allo us to anale asmmetric similarit matrices from different sources. In the contet of business intelligence and game analtics, this might allo us to anale and compare, for instance, preferences of different countries or plaer groups. References 1. Bader, B., Harshman, R., Kolda, T.: Temporal Analsis of Semantic Graphs using ASALSAN. In: Proc. of IEEE ICDM (2007) 2. Bauckhage, C., Sifa, R., Drachen, A., Thurau, C., Hadiji, F.: Beond Heatmaps: Spatio-Temporal Clustering using Behaior-Based Partitioning of Game Leels. In: Proc. of IEEE CIG (2014) 3. Bauckhage, C.: k-means Clustering ia the Frank-Wolfe Algorithm. In: Proc. of KDML-LWDA (2016) 4. Che, P.A., Bader, B.W., Rooskaa, A.: Using DEDICOM for Completel Unsuperised Part-Of-Speech Tagging. In: Proc. Workshop on Unsuperised and Minimall Superised Learning of Leical Semantics (2009) 5. Cremonesi, P., Koren, Y., Turrin, R.: Performance of Recommender Algorithms on Top-N Recommendation Tasks. In: Proc. ACM Recss (2010) 6. Cutler, A., Breiman, L.: Archetpal Analsis. Technometrics 36(4), (1994) 7. Frank, M., Wolfe, P.: An Algorithm for Quadratic Programming. Naal Research Logistics 3(1-2) (1956) 8. Harshman, R.A.: Models for Analsis of Asmmetrical Relationships among N Objects or Stimuli. In: Proc. Joint Meeting of the Pschometric Societ and the Societ for Mathematical Pscholog (1978) 9. Jaggi, M.: Reisiting Frank-Wolfe: Projection-Free Sparse Cone Optimiation. In: Proc. of ICML (2013) 10. Kantor, P., Rokach, L., Ricci, F., Shapira, B.: Recommender Sstems Handbook. Springer (2011) 11. Koren, Y., Bell, R., Volinsk, C.: Matri Factoriation Techniques for Recommender Sstems. Computer 42(8) (2009) 12. Lason, C.L., Hanson, R.J.: Soling Least Squares Problems. SIAM (1974) 13. Nickel, M., Tresp, V., Kriegel, H.: A Three-a Model for Collectie Learning on Multi-relational Data. In: Proc. of ICML (2011) 14. Sifa, R., Drachen, A., Bauckhage, C.: Large-Scale Cross-Game Plaer Behaior Analsis on Steam. In: Proc. of AAAI AIIDE (2015) 15. Sifa, R., Ojeda, C., Bauckhage, C.: User Churn Migration Analsis ith DEDI- COM. In: Proc. of ACM RecSs (2015) 16. Sifa, R., Srikanth, S., Drachen, A., Ojeda, C., Bauckhage, C.: Predicting Retention in Sandbo Games ith Tensor Factoriation-based Representation Learning. In: Proc. of IEEE CIG (2016) 17. Sifa, R., Bauckhage, C., Drachen, A.: Archetpal Game Recommender Sstems. In: Proc. of KDML-LWA (2014) 18. Tersk, A.: Features of Similarit. Pschological Reie 84(4) (1977)