Kernel based collaborative filtering for very large scale top-n item recommendation

Transcription

1 ESANN 06 proceedings, Eropean Syposi on Artificial Neral Networks, Coptational Intelligence and Machine Learning. Brges (Belgi), 7-9 April 06, i6doc.co pbl., ISBN Available fro Kernel based collaborative filtering for very large scale top-n ite recoendation Mirko olato and Fabio Aiolli University of adova - Departent of Matheatics Via Trieste, 63, 35 adova - Italy Abstract. The increasing availability of iplicit feedback datasets has raised the interest in developing effective collaborative filtering techniqes able to deal asyetrically with nabigos positive feedback and abigos negative feedback. In this paper, we propose a principled kernelbased collaborative filtering ethod for top-n ite recoendation with iplicit feedback. We present an efficient ipleentation sing the linear kernel, and how to generalize it to other kernels preserving efficiency. We copare or ethod with the state-of-the-art algorith on the Million Songs Dataset achieving an exection abot 5 tie faster, while having coparable effectiveness. Introdction Collaborative filtering (CF) techniqes can ake recoendation to a ser exploiting inforation provided by siilar sers. The typical CF setting consists of a set U of n sers, a set I of ites, and the so-called rating atrix R = {ri } Rn. In this paper we focs on iplicit feedback, and so we asse binary ratings, ri {0, }, where ri = eans that ser interacted with ite i (nabigos feedback) and ri = 0 eans there is not evidence that ser interacted with ite i (abigos feedback). Unlike traditional CF algoriths for explicit feedback, where one wants to accrately predict ratings for each nseen ser-ite pair, the goal in the iplicit feedback doain is to generate a top-n ranking of ites. Top-N recoendation with iplicit feedback was the sbject of one recent rearkable challenge organized by Kaggle, the Million Songs Dataset challenge [], that was defined on a very large dataset with roghly.m sers and 380K ites (i.e., songs) for a total of abot 50M ratings. The winning soltion described in [] (here called MSDW) is an extension of the well known ite-based nearest-neighbors (NN) algorith [3] that ses an asyetric siilarity easre, called asyetric cosine. Besides its otstanding perforance in ters of A@500, the MSD winning soltion is also easily scalable to very large datasets. However, one drawback of this soltion is that it is not theoretically well fonded. More recently, a new principled algorith for CF (CF-OMD) which explicitly optiizes the AUC has been proposed with very nice perforances on the MovieLens dataset [4]. Unfortnately, this last algorith cannot be proptly applied to large datasets as it reqires the optiization of n qadratic probles each one defined on variables. This work was spported by the University of adova nder the strategic project BIOINFOGEN.

2 ESANN 06 proceedings, Eropean Syposi on Artificial Neral Networks, Coptational Intelligence and Machine Learning. Brges (Belgi), 7-9 April 06, i6doc.co pbl., ISBN Available fro Here, we propose a variant of the CF-OMD algorith that akes it applicable to very large datasets achieving an exection tie abot 5 ties faster than the MSDW algorith on the MSD dataset. Secondly, we present strategies that allow the sae algorith to be applied with qite general kernels withot loss in efficiency. CF-OMD (Optiization of the Margin Distribtion) In this section we present a CF algorith, called CF-OMD [4], for top-n recoendation inspired by preference learning, and designed to explicitly axiize the AUC (Area Under the ROC Crve). Consider the noralized rating atrix Rn, with colns xi = ri kri k and let I be the set of ites rated by the ser. Let also define the probability distribtionover the positive and negative ites for as Γ = {α R i I αi =, i I αi = }. Then, for each test ser, the following convex optiization proble has to be solved: () α = argin α Y Y Λ α, α Γ where Y is a diagonal atrix, Y = diag(y ), sch that yi = if i I, otherwise, and Λ is a diagonal atrix sch that Λii = λp if i I, otherwise Λii = λn, where λp and λn are reglarization paraeters. These paraeters balance the contribtion of the nabigos ratings (λp ) and the abigos ones (λn ). Once solved the optiization proble the scores of the ser is calclated by r = Y α, and the recoendation is ade accordingly. Althogh this algorith has shown state-of-the-art reslts in ters of AUC, it is not sitable to deal with large dataset. In fact, let asse that each optiization proble can be solved by an algorith with a coplexity qadratic on the nber of paraeters. Then the global coplexity wold be O(nts ), where nts is the nber of sers in the test set, and for the MSD it wold be O(09 ). 3 Efficient CF-OMD Analyzing the reslts reported in [4], the athors noticed that high vales of λn did not particlarly affect the reslts, becase it tends to flatten the contribtion of the abigos negative feedbacks toward the average, itigating the relevance of noisy inforation. In CF contexts the data sparsity is particlarly high, this eans, on average, that the nber of abigos negative feedbacks is orders of agnitde greater than the nber of positive feedbacks. Forally, given a ser, let = I and = I \ I then =, where, and generally O() = O( ). On the basis of this observation, we can siplify the optiization proble (), by fixing λn =, which eans that i I, αi = : α = argin α α λp kα k α µ, α Γ ()

3 ESANN 06 proceedings, Eropean Syposi on Artificial Neral Networks, Coptational Intelligence and Machine Learning. Brges (Belgi), 7-9 April 06, i6doc.co pbl., ISBN Available fro where α are the probabilities associated with the positive ites, is the sb-atrix of containing only the colns corresponding to the positive ites and µ = ( i I xi ) is the centroid of the convex hll spanned by the negative ites. The nber of paraeters in () is and hence the coplexity fro O(nts ) is dropped to O(nts ), where = E[ I ]. In MSD which leads to a coplexity O(08 ). 3. Ipleentation trick Notwithstanding the hge iproveent in ters of coplexity, a naı ve ipleentation wold have an additional cost de to the calclation of µ. For all sers in the test set the cost wold be O(nts n ), where = E[ I \ I ], and it can be approxiated with O(nts n). To overcoe this bottleneck, we propose an efficient increental way of cal µ= clating µ. Consider the ean over all ites i I xi, then, for a given µ i I xi. Fro a coptational ser, we can express µ = point of view, it is sfficient to copte the s i I xi once (i.e., µ) and then, for every µ, sbtract the s of the positive ites. Using this siple trick, the overall coplexity drops to O(n) O(nts ). In the experiental section we sccessflly applied this algorith to the MSD achieving copetitive reslts against the state-of-the-art ethod bt with higher efficiency. 4 Kernelized CF-OMD The ethod proposed in Section 3, can be seen as a particlar case of a kernel ethod. In fact, is a kernel atrix, let call it K with the corresponding (linear) kernel fnction K : Rn Rn R. Given K we can reforlate () as: (3) α = argin α K α λp kα k α q, α Γ where q : qi = j I K(xi, xj ). Actally, inside the optiization proble (3) we can plg any kernel fnction. We will refer to this ethod as CF-KOMD. Generally speaking, the application of kernel ethods on hge dataset have an intractable coptational coplexity. Withot any shrewdness the proposed ethod wold not be applicable becase of the coptational cost of the kernel atrix and q. An iportant observation is that the coplexity is strictly connected with the sparsity of the kernel atrix which is, nfortnately, coonly dense. However, we can leverage on an iportant reslt to keep the kernel as sparse as possible withot changing the soltion of CF-KOMD. In [5] Karnick et al. observed that: if a fnction f : R R adits a Maclarin expansion with only nonnegative coefficients i.e., n=0 an xn, an 0, then it defines a positive definite kernel as K : (x, y) 7 f (hx, yi). As ephasized in [5], any kernels sed in practice [6] satisfy the above-entioned condition. 3

4 ESANN 06 proceedings, Eropean Syposi on Artificial Neral Networks, Coptational Intelligence and Machine Learning. Brges (Belgi), 7-9 April 06, i6doc.co pbl., ISBN Available fro Consider the application of this reslt on the polynoial kernel Kp : (x, y) 7 (hx, yi c)d where c R and d N. Kp can be defined as: d d (di) i Kp (x, y) = c hx, yi. (4) i i=0 When the polynoial is not hoogeneos (i.e., c 6= 0) the kernel atrix indced by Kp is dense de to the zero degree ter (i.e., cd ) which is added to all entries. Since adding a constant to a whole atrix eans a space translation, it can be deonstrated that this operation does not affect the argin in CF-KOMD. For this reason we can sparsify the kernel by reoving the factor cd obtaining a kernel atrix whose sparsity depends on the distribtion of the inpt data. Let K = be a kernel atrix and let (Kij 6= 0) be the probability that the entry Kij is not zero. Given the a-priori probabilities (xih 6= 0) and (xjh 6= 0), we can say that (Kij 6= 0) = ( (xih 6= 0) (xjh 6= 0))n. Anytie both xi and xj are poplar ites, i.e., (xih 6= 0) and (xjh 6= 0) are high, then (Kij 6= 0) tends to be high as well. On the contrary, when one of the two vectors represents an npoplar ite then the probability (Kij 6= 0) goes to zero. In CF contexts this sitation is pshed towards the liit since the poplarity distribtion generally follows a power low, and this often garantees the sparsity of the reslting kernel. Using the sparsified kernel, we can frther optiize the coplexity by providing a good approxiation of q that can be copted only once, instead of nts ties. The idea consists in replacing every qi with an estiate of E[K(xi, x)]. Forally, consider, withot any loss of generality, a noralized kernel fnction K(x K and let the approxiation of q be q s.t. q i = i, xj ). At each j I coponent of q, the approxiation error is bonded by which is linear on the sparsity of the dataset. 5 (see Appendix A), Experients and Reslts Experients have been perfored coparing the proposed ethods against the state-of-the-art ethod on MSD (MSDW) with respect to the ranking qality and coptational perforance. We sed two datasets: MSD, described in Section, and Movielens, which consists of 3850 sers and 73 ites for a total of 35K ratings. Methods have been copared sing the A [] and AUC easres. All ethods have been ipleented in ython. In this section we will refer to the Efficient CF-OMD with ECF-OMD and to the Kernelized CF-OMD with CF-K. 5. Movielens dataset The Movielens dataset has been randoly divided into a training set of roghly 50K ratings and a test set of 60K ratings. Since this dataset contains ratings in We sed CVOT package to solve the optiization proble MSDW ipleentation is available at aiollicodemsd The 4

5 ESANN 06 proceedings, Eropean Syposi on Artificial Neral Networks, Coptational Intelligence and Machine Learning. Brges (Belgi), 7-9 April 06, i6doc.co pbl., ISBN Available fro the for of a 5 stars preference, we had to convert the into binary ones where all vales greater than 0 are treated as. This test ais to show the accracy and the coptational perforance of the proposed ethods on a edi size dataset. Table sarizes the reslts. A@00 AUC MSDW (α) ECF-OMD (λp ) CF-K (λp ) Table : Ranking accracy on Movielens dataset sing AUC and A@00. We tested MSDW fixing the locality paraeter [] q = and varying the asyetric cosine weight α. For ECF-OMD we tried different λp bt its effect is inial on the final ranking, and for this reason we fixed it dring the CFK experient. In this experient we sed the polynoial kernel of degree with c =. Reslts show that both proposed ethods have higher AUC and A@00 with a slightly better perforance for CF-K. With this dataset all ethods terinate in few seconds. 5. MSD We sed MSD as described in the Kaggle challenge3 : the training set is coposed by M sers (pls 0K sers as validation set) with all their listening history and for the rest (i.e., 00K sers) only the first half of the history is provided, while the other half constittes the test set. In these experients we fixed the λp paraeter to the best perforing one on the Movielens dataset. Reslts are presented in Table. In this case MSDW aintains its record perforance in ters of A@500, while for the AUC all ethods have very good reslts. This nderline the fact that both ECF-OMD and CF-K try to optiize the AUC rather than the A. A@500 AUC MSDW (α, q) 5, ECF-OMD (λp ) CF-K (λp ) Table : Ranking accracy on MSD sing AUC and A@500. The coptational costs on this dataset are reported in Figre. The reslts are the average copting tie over K test sers. All ethods rn on a achine with 50Gb of RAM and x Eight-Core Intel(R) eon(r) CU E5-680 Actally the ties in Figre have a constant overhead de to read operations. Reslts show that ECF-OMD and CF-K are alost 5 tie faster than MSDW even thogh they reqire ore RAM to store the kernel atrix. It is worth to notice that CF-K has a coptational tie 3 5

6 ESANN 06 proceedings, Eropean Syposi on Artificial Neral Networks, Coptational Intelligence and Machine Learning. Brges (Belgi), 7-9 April 06, i6doc.co pbl., ISBN Available fro MSDW 7.5hrs ECF-OMD.55hrs CF-K 0.75hrs Fig. : Average coptational tie in hors for K sers. very close to ECF-OMD, and this highlights the positive effects of the coplexity optiization presented in this paper. A Appendix A. Optiization proble siplification Let µ defined as in Sec.3 and let, be the sb-atrices of containing only the colns corresponding, respectively, to the positive and negative ites for. Then, by fixing λn =, we can siplify () as: µ α = argin kα k λp kα k α = argin kα k kµ k α µ λp kα k = () α A. Approxiation error Let Kij = K(xi, xj ), then: q i qi = Kij Kij Kij Kij = Kij j I j I j I j I j I = Kij Kij Kij Kij j I j I j I j I =. References [] Brian McFee, Thierry Bertin-Mahiex, Daniel.W. Ellis, and Gert R.G. Lanckriet. The illion song dataset challenge. In roceedings of the st international conference copanion on World Wide Web, WWW Copanion, pages , New York, NY, USA, 0. ACM. [] Fabio Aiolli. Efficient top-n recoendation for very large scale binary rated datasets. In ACM Recoender Systes Conference, pages 73 80, Hong Kong, China, 03. [3] Mknd Deshpande and George Karypis. Ite-based top-n recoendation algoriths. ACM Trans. Inf. Syst., ():43 77, 004. [4] Fabio Aiolli. Convex AUC optiization for top-n recoendation with iplicit feedback. In ACM Recoender Systes Conference, pages 93 96, New York, USA, 04. [5] rshotta Kar and Harish Karnick. Rando featre aps for dot prodct kernels. In Neil D. Lawrence and Mark A. Girolai, editors, roceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics (AISTATS-), vole, pages , 0. [6] Bernhard Scholkopf and Alexander J. Sola. Learning with Kernels: Spport Vector Machines, Reglarization, Optiization, and Beyond. MIT ress, Cabridge, MA, USA, 00. 6