Multi-relational Weighted Tensor Decomposition

Transcription

1 Multi-elational Weighted Tenso Decoposition Ben London, Theodoos Rekatsinas, Bet Huang, and Lise Getoo Depatent of Copute Science Univesity of Mayland, College Pak College Pak, MD Intoduction In eal-wold netwok data, thee often exist ultiple types of elationships (edges) that we would like to odel. Fo instance, in social netwoks, elationships between individuals ay be pesonal, failial, o pofessional. In this pape, we exaine a ulti-elational leaning scenaio in which the leane is given a sall set of taining exaples, sapled fo the coplete set of potential paiwise elationships; the goal is then to pefo tansductive infeence on the issing elationships. To do so, we pesent a tenso-based leaning odel, in which the elations ay be eithe binay- o ealvalued functions of the object pais. In ecent yeas, thee has been a gowing inteest in tenso ethods within the achine leaning counity, patially due to thei natual epesentation of ulti-elational data [, 2, 3, 4, 5, 6, 7, 9]. We popose a decoposition, based on [7], which is oe appopiate fo ulti-elational data. Unlike pevious ethods, we do not attept to decopose the input tenso diectly; instead, we explicitly odel a apping fo the low-ank epesentation to the obseved tenso, which is often bette suited to pediction. Fo exaple, a binay elationship can be odeled as the sign of a latent epesentation; this gives the latent epesentation oe feedo to incease the pediction agin, athe than epoduce {±} exactly. Ou poposed ethod, ulti-elational weighted tenso decoposition (MWTD), assues that the elationships ae deteined by a linea cobination of latent factos associated with each object. Leaning these factos, and thei inteactions in each elation, thus becoes analogous to a tenso decoposition (see Figue ). We foulate the decoposition as a nonlinea optiization poble, using a cobination of task-specific, weighted loss functions to enable siultaneous leaning of both discete- (binay) and continuously-quantified elations. By weighting the loss function, we ae able to leveage liited obseved (taining) elationships without fitting the unobseved (testing) ones. Futhe, due to ou decoposition, we ae able to tansfe infoation acoss the vaious types of elations duing leaning, thus bette exploiting the stuctue of ulti-elational data. We deonstate the efficacy of this appoach using synthetic data expeients. The esults show significant ipoveents in accuacy ove copeting factoization odels. 2 Multi-elational Weighted Tenso Decoposition Fix a set of objects {o,..., o } and a set of elations {R,..., R n }. We ae given a patially obseved tenso Y R n, in which each obseved enty is a (possibly noisy) easueent of a elationship y i,j,k R k (o i, o j ) and each unobseved enty is set to a null value. We ae additionally given a nonnegative weighting tenso W R + n, whee each enty w i,j,k [0, ] coesponds to a use-defined confidence, o cetainty, in the value of y i,j,k ; if y i,j,k is unobseved, Hee, we use the te elation loosely to include not only stict elations, fo which elationships ae eithe pesent o not, but also eal-valued functions. To siplify ou analysis, we assue that all elations ae syetic, though one can obtain an analogous deivation fo asyetic elations with slightly oe wok.

2 Input data ( Y k k A R k Inteactions A > Latent factos + b k Bias ( Figue : Each slice Y k of the input tenso is appoxiated by a function Φ k of a low-ank decoposition AR k A + b k. The latent factos A ae coon to all slices. Each R k deteines the inteactions of A in the k th elation, while b k accounts fo distibutional bias. then w i,j,k is necessaily zeo. The goal of ulti-elational tansduction in this tenso foulation is to infe the unobseved enties in Y. Ou fundaental assuption is that each elationship R k (o i, o j ) is equal to a apping Φ k applied to an eleent x i,j,k in an undelying low-ank tenso X R n. Each Φ k depends on the natue of the elation R k, and ay diffe acoss elations. Fo exaple, fo binay elations in {±}, Φ k is the sign function. We futhe assue that each fontal slice X k can be factoed as a ank- decoposition: X k = AR k A + b k, () whee A R, R k R and b k R (see Figue ). Note that we place no constaints on A o R k ; the coluns of A need not be linealy independent, and R k need not be positiveseidefinite. To infe the values of the issing (o uncetain) enties, we pedict each y i,j,k by coputing x i,j,k = a i R ka j + b k and applying the appopiate apping. The coluns of A can be intepeted as the global latent factos of the objects, whee the i th ow a i coesponds to the latent factos of object o i. Each R k deteines the inteactions of A in the k th elation. Thus, each pedicted elationship coes fo a linea cobination of the objects latent factos. Because the latent factos ae global, infoation popagates between elations duing the decoposition, thus enabling collective leaning. The addition of b k accounts fo distibutional bias within each elation. The key distinction between ou appoach and pevious tenso decopositions [, 5, 7], fo ultielational leaning is that, instead of diectly decoposing the input tenso, we odel the apping fo X to Y. Moeove, we explicitly odel the potential spasity and uncetainty in the obsevations, poducing oe accuate pedictions even when obseved (taining) data is liited (see Section 3). To copute the decoposition in Equation, we iniize the following egulaized objective: f(a, R, b) λ 2 A 2 F + n λ 2 R k 2 F + t ( W k (l k (Y k, X k )) ), (2) whee λ 0 is a egulaization paaete, X k is coputed by Equation, and l k is a loss function that is applied eleent-wise to the k th slice. This ability to cobine ultiple loss functions is cental to ou appoach, as the appopiate penalty depends on the apping fo each X k to Y k. Though ost atix and tenso decopositions focus on iniizing the quadatic loss, this citeion ay not be optial fo cetain pediction tasks, such as binay pediction; by explicitly aking the loss function fo each slice task-specific, ou faewok offes oe flexibility than elated techniques. To iniize Equation 2, we use quasi-newton optiization; in paticula, we ipleent ou faewok using the liited-eoy Boyden-Fletche-Goldfab-Shanno (L-BFGS) algoith. This equies the gadients of f(a, R, b) w..t. A, R k and b k. Leveaging the syety of R k, we deive these as A f(a, R, b) = λa + 2 (W k Xk l k (Y k, X k )) AR k (3) Rk f(a, R, b) = λr k + A (W k Xk l k (Y k, X k )) A, (4) 2

3 bk f(a, R, b) = t ( W k ( Xk l k (Y k, X k )) ), (5) whee denotes the Hadaad (i.e., eleent-wise) poduct, and Xk l k (Y k, X k ) is the gadient of l k w..t. X k. Though this accoodates any diffeentiable loss function, we now pesent two that ae applicable to any elational pobles, and deive thei coesponding gadients. Quadatic Loss: The ost coon loss function used in atix and tenso factoization is the quadatic loss, which we define as l q (y, x) 2 (y x)2. Miniizing the quadatic loss coesponds to the setting in which each elationship is diectly appoxiated by a linea cobination of latent factos, i.e., Φ k is the identity and Y k X k. Fo this loss function, the loss gadient is siply Xk l k (Y k, X k ) (X k Y k ). Sooth Hinge Loss: While the quadatic loss ay be appopiate fo leaning eal-valued functions, it is soeties ill-suited fo leaning binay elations. Fo binay classification, the goal is to coplete a patially obseved slice Y k {±}. Recall that the apping Φ k is the sign function, and so y i,j,k sgn(x i,j,k ). Appoxiating {±} with a quadatic penalty ay yield a sall-agin solution, since high-confidence pedictions will push low-confidence pedictions close to the decision bounday. To get a lage-agin solution, we use the sooth hinge loss [8], /2 z if z 0, l h (y, x) h(y x), whee h(z) ( z) 2 /2 if 0 < z <, 0 if z. Unlike the standad hinge loss, the sooth hinge is diffeentiable eveywhee. To obtain closed-fo gadients, we define tensos P, Q R n, whee { { yi,j,k if y p i,j,k i,j,k x i,j,k <, if 0 < yi,j,k x and q i,j,k i,j,k <, 0 othewise, 0 othewise. This siplifies the sooth hinge to { ( 2pi,j,k x i,j,k + q i,j,k x 2 i,j,k h(y i,j,k x i,j,k ) = )/2 if y i,j,k x i,j,k <, 0 othewise, which we can diffeentiate w..t. X k to obtain Xk l k (Y k, X k ) (Q k X k P k ). 3 Expeients In this section, we copae MWTD with elated tenso and atix decopositions [7, 8] in seveal expeients using synthetic data. While we have poising esults fo eal-wold data we do not epot the hee due to space estictions. The fist ethod we copae against MWTD is the RESCAL odel [7]. In RESCAL, unobseved elationships ae teated as negative exaples. To distinguish between (un)obseved elationships and negative exaples, we odified the epesentation of binay elationships fo {0, } to {±} fo obseved data and zeos elsewhee. 2 The second technique is axiu-agin atix factoization (MMMF) [8], a odel fo atix econstuction. As such, it is not designed fo ultielational data; howeve, we can use it to econstuct each slice of the tenso individually. Since the decoposition of MMMF diffes significantly fo MWTD, to ensue a fai copaison, we un a vaiant of MWTD that decoposes each slice sepaately instead of jointly, using a sepaate A k fo k =,..., n. This is eant to equalize the discepancy in decopositions, while isolating the deficiencies of non-collective leaning. We efe to this odel as MMMF+. To geneate the synthetic data, we stat by coputing a low-ank tenso ˆX R n as ˆX k  ˆR k  + E k, fo k =,..., n, whee  R and ˆR k R ae sapled fo a noal distibution, and E k R is low-level, noally-distibuted noise. Fo the fist expeient, we constuct n = 3 binay elations (i.e., slices), ove = 500 objects, using ank = 0. We efe 2 We find that this odification ipoves RESCAL s pefoance ove the oiginal ethod. 3

4 AUC PR (avg 0 uns) MWTF MMMF+ RESCAL MMMF AUC PR (avg 0 uns) MSE (avg 0 uns) % tain (a) BinaySynthetic % tain (b) MixedSynthetic (AUC-PR) % tain (c) MixedSynthetic (MSE) Figue 2: Results of the synthetic expeients, with standad deviations. We plot the AUC-PR fo BinaySynthetic (a) and MixedSynthetic (b) datasets. We also plot the MSE fo MixedSynthetic (c); note that the MSE is an eo easue, and thus lowe nubes ae bette. to this dataset as BinaySynthetic. To geneate a binay Y {±} n, we ound the values of ˆX k using the 90 th pecentile of ˆX as a theshold. This poduces a heavy skew towads the negative class, as is typical in eal ulti-elational data. Fo the second expeient, we constuct one binay elation and one eal-valued elation, again ove 500 objects, with ank 0. We efe to this dataset as MixedSynthetic. We evaluate ove taining sizes t {3, 5, 0, 5, 20, 25} pecent, using the eaining enties fo testing. Fo each taining set, we withold a ando 25% of enties as a validation set fo finding a good egulaization paaete λ. Once identified, we then etain on the full taining set using λ and evaluate on the test set. The esults we epot ae aveaged ove 0 uns pe taining size. On BinaySynthetic, MWTD achieves a statistically significant 3 lift ove the copeting ethods fo taining sizes 5% and up. We attibute these esults to two piay advantages: the weighted objective function, with its ixtue of task-specific loss functions, and the global latent factos. The weighted objective allows exploiting sall aounts of obseved data, without fitting the unobseved enties. Since RESCAL teats all enties as obseved, it tends to fit the unobseved enties in spasely populated tensos. Futheoe, we obseved ipoved pefoance ove MMMF, since the latent factos ae specific to each slice, while in MWTD, infoation fo one slice is popagated to the othes via the global latent factos. On MixedSynthetic, MWTD s ipoveent ove RESCAL and MMMF, in both AUC-PR and MSE, is statistically significantly fo all taining sizes. Copaed to MMMF+, MWTD s eduction in MSE is significant fo sizes 0% and above. Yet though MMMF+ is copetitive with MWTD in MSE fo sizes 3-5%, since MMMF+ is unable to tansfe infoation between slices, it still lags significantly behind MWTD in AUC-PR fo all taining sizes. We consideed exaining the benefit of lage-agin leaning in isolation (without weighting); howeve, this is not especially eaningful fo tansduction, as the hinge loss does not wok with unobseved enties. In expeients not shown, we found that weighting alone (i.e., with quadatic loss) showed ipoveent ove RESCAL in BinaySynthetic, but not as uch as with the (lage-agin) sooth hinge loss. We also expeiented with vaying the ank of the decoposition fo 5 to 40, but because the ethods we test use L2 egulaization on the latent factos, the esults ae nealy identical acoss anks. Even with ank coss-validation pe ethod, MWTD still doinated the othe ethods. We believe this is because the L2 egulaize is the piay coplexity contol, so vaying the ank has little effect on the pedictions. Finally, the unning tie fo MWTD is soewhat slowe than that of RESCAL, in pat because of the oe coplex objective function; yet it eains efficient because the spasity of the obseved tenso. On BinaySynthetic, using the optial λ on 25% taining data, leaning executes in appoxiately ten seconds using ou MATLAB ipleentation; this is copaable to RESCAL, which takes appoxiately five seconds. 3 We easue statistical significance in all expeients using a 2-saple t-test with ejection theshold

5 Refeences [] B. Bade, R. Hashan, and T. Kolda. Tepoal analysis of seantic gaphs using ASALSAN. In Poc. of the 7th IEEE Intenational Conf. on Data Mining (ICDM), [2] D. Dunlavy, T. Kolda, and W. Kegeleye. Multilinea algeba fo analyzing data with ultiple linkages. Technical Repot, [3] D. Dunlavy, T. Kolda, and E. Aca. Tepoal link pediction using atix and tenso factoizations. ACM Tans. on Knowledge Discovey fo Data, 5(2), 20. [4] S. Gao, L. Denoye, and P. Gallinai. Link patten pediction with tenso decoposition in ulti-elational netwoks. In IEEE Syposiu on Cop. Intell. and Data Mining, 20. [5] R. Hashan. Models fo analysis of asyetical elationships. In Fist Joint Meeting of the Psychoetic Society and the Society fo Matheatical Psychology, 978. [6] H. Kashia, T. Kato, Y. Yaanishi, M. Sugiyaa, and K. Tsuda. Link popagation: A fast sei-supevised leaning algoith fo link pediction. In SIAM Intenational Confeence on Data Mining (SDM), [7] M. Nickel, V. Tesp, and H. Kiegel. A thee-way odel fo collective leaning on ultielational data. In Poc. of the 28th Intenational Conf. on Machine Leaning (ICML), 20. [8] J. Rennie and N. Sebo. Fast axiu agin atix factoization fo collaboative pediction. In In Poc. of the 22nd Intenational Conf. on Machine Leaning (ICML), [9] L. Xiong, X. Chen, T. Huang, J. Schneide, and J. Cabonell. Tepoal collaboative filteing with Bayesian pobabilistic tenso factoization. In SIAM Intenational Confeence on Data Mining (SDM),