Incorporating Diversity in a Learning to Rank Recommender System

Transcription

1 Proceedings of the Twenty-Ninth International Florida Artificial Intelligence Research Society Conference Incorporating Diversity in a Learning to Rank Recommender System Jacek Wasilewski and Neil Hrley Insight Centre for Data Analytics University College Dblin Dblin, Ireland {acek.wasilewski, neil.hrley}@insight-centre.org Abstract Reglarisation is typically applied to the optimisation obective of matrix factorisation methods in order to avoid overfitting. In this paper, we explore the se of reglarisation to enhance the diversity of the recommendations prodced by these methods. Given a matrix of pairwise item distances, we add reglarisation terms dependent on the item distances to the accracy obective of a learning to rank matrix factorisation formlation. We examine the impact of these reglarisers on the latent factors prodced by the algorithm and show that sch reglarisation does indeed promote diversity. The reglarisation comes at a cost of performance in terms of accracy and ltimately the approach cannot greatly enhance diversity withot a conseqent fall-off in accracy. Introdction Recommender systems have become biqitos in online systems and services. Their goal is to help address the choice overload problem, by filtering a large set of possible selections into a mch smaller set of recommended items that a ser is likely to be interested in. Recommendations are generally based on a set of implicit or explicit item ratings gathered from sers in past interactions. Matrix factorisation has proven an effective means of prodcing accrate recommendations. In sch methods, the rating matrix is factored into two low-rank matrices, representing ser- and item-latent factors and predicted ratings are obtained by mltiplying the corresponding ser and item factors. Interest in promoting the diversity of recommendations has increased in recent years. In general, the promotion of diversity is in opposition to the reqirement of high accracy. Many offline stdies have fond that the more diverse a recommendation is, the less likely it is to match the ser s preference and conversely, a highly accrate set is likely to consist of many similar recommendations. Some work, e.g. (Ekstrand et al. 204), has fond positive correlations between diversity and accracy as sbectively perceived by sers in ser trials. To date, many of the approaches to diversity enhancement have been developed in the context of memory-based algorithms, or alternatively, diversity enhancement has been considered as a separate post-processing step carried ot after the initial rating predictions have been obtained. In this paper, we Copyright c 206, Association for the Advancement of Artificial Intelligence ( All rights reserved. explore whether it is possible to tackle the accracy and diversity problems together in a single training phase. Reglarisation has been sed in matrix factorisation algorithms, principally to control for overfitting, by constraining the size of the latent factors. However, in the literatre on recommender systems, different types of reglarisers have been proposed in order to incorporate other side information into the obective fnction optimisation to spport the recommendation. For example, in (Jamali and Ester 200) a social reglarisation is proposed to incorporate social network information into the optimisation, by encoraging sers who are close in the social network to have similar ser latent factors. We are motivated by sch work to consider whether appropriate reglarisation can be sed to enhance diversity. Given an item distance matrix, in this paper, we propose a nmber of reglarisations that se the distance matrix to encorage the optimisation to prodce factors that reslt in a diverse set of items in the recommendation list. We evalate the reglarisation methods on two datasets. Ultimately, we observe that, while some of the proposed reglarisers are effective in promoting diversity, the diversity cannot be largely improved withot a conseqent fall-off in accracy. The paper is organised as follows: after reviewing the state-of-the-art and smmarising the learning-to-rank method pon which or reglarisers are applied, we propose a nmber of different reglarisation terms and discss their likely effectiveness. We then describe how to incorporate sch reglarisation in an alternating least sqares optimisation framework. In the evalation section, we test the reglarisers and compare their performance. Related Work The generation of personalised rankings from implicit feedback data has received some attention in recent work in recommender systems (H, Koren, and Volinsky 2008; Pilászy, Zibriczky, and Tikk 200; Jahrer and Töscher 202; Takács and Tikk 202). In this paper, we focs on incorporating diversity into the learning to rank algorithm for implicit feedback, proposed in (Takács and Tikk 202), althogh or method can be applied to any matrix factorisation formlation. Work on diversity has largely focsed on variants of the Maximm Marginal Relevance (MMR) re-ranking principle introdced originally in (Carbonell and Goldstein 998) and sed the diversification of recommen- 572

2 dations in work sch as (Ziegler et al. 2005; Zhang and Hrley 2008). In this approach, the final recommendation is prodced in two steps: first a list of recommendation candidates is prodced for each ser and then the top-n items are selected one by one in a way that an item and a list of already selected items has the highest diversity vale. Re-ranking based on the intent-aware framework has also been proposed (Vargas, Castells, and Vallet 20). In contrast to this work, we focs on tackling accracy and diversity ointly dring model training. A comprehensive framework for evalating novelty and diversity is given in (Vargas and Castells 20). We se this framework in the evalation of or proposed method. Measring Diversity Given a set, U, of sers with n = U and a set, I, of items with m = I and an n m matrix R containing ratings given by the each ser for some of the m items, the top-n recommendation problem is, for a given ser, to recommend a list L of N items that the ser is likely to enoy. The accracy of the recommendation algorithm can be measred sing varios different metrics, by comparing L with hold-ot data. We assme that there also exists an m m matrix D, with elements d(i, ) giving a distance between items i and and that, as well as being accrate, we wold like the recommendation to be novel or diverse. Within the framework for evalating novelty and diversity in recommender systems proposed in (Vargas and Castells 20), the novelty of items is measred with the respect to a particlar context. We concentrate on the expected intra-list diversity (EILD) which measres the novelty of recommended items with respect to the other items in the recommended list. A recommended list with a high EILD vale contains items that are very different to each other, according to the distance measre, d(.,.). The fll expression incorporates rank discont and relevance-awareness, sch that, given a recommended list L = {i,...,i N } of size N = L for a ser, EILD(L )= N C k disc(k)disc(l k)p(rel i k,)p(rel i l,)d(i k,i l ), k=,l=;l k where disc(l k) = disc(max(,l k)) reflects a relative rank discont for an item at position l knowing that position k has been reached, rel is the relevance of an item to a ser, and C k is a normalising constant. Ignoring rank and relevance, the metric redces to the intra-list distance (ILD) (Zhang and Hrley 2008; Ziegler et al. 2005): ILD(L )= N(N ) i, L d(i, ) Learning to Rank for Recommendation We focs on matrix factorisation approaches to recommendation in which the training phase involves learning a lowrank n k latent ser matrix P and a low-rank m k latent item matrix Q, sch that the estimated rating ˆr i can be expressed as: ˆr i = p T q i, where p T is the th row of P, q T i is the i th row of Q and k is the chosen dimension of the latent space. P and Q are learned throgh the minimisation of an accracy-based obective. A nmber of sch obectives have been proposed in the literatre and the reglarisation methods we propose here cold be incorporated with any sch obective. Since we are interested in ranking rather than rating prediction, we focs on the learning to rank obective fnction proposed originally in (Jahrer and Töscher 202) and frther developed in (Takács and Tikk 202) i.e. we take acc(p, Q) to be: c i s [(ˆr i ˆr ) (r i r )] 2 + β( Q 2 + P 2 ) U i I I () where c i and s are parameters of the obective fnction and β is the standard reglarisation parameter of normbased reglarisation to avoid over-fitting. We consider the implicit feedback case in which c i =0if r i =0, and otherwise. The role of c i is to select ser-item pairs corresponding to positive feedbacks from all possible pairs. s is an importance weighting for item. Reglarisation to Enhance Diversity We explore the se of reglarisation to enhance the diversity of the recommendation, by choosing an optimisation obective of the form min acc(p, Q) + λreg(p, Q) P,Q where acc(.) is the accracy obective and reg(.) is a reglarisation term. To choose an appropriate reglarisation, it is sefl to initially consider the diversity obective alone. Representing a recommendation as an m-dimensional vector x sch that x(i) = when item i L and x(i) = 0 otherwise, the ILD of the recommendation may be written as the qadratic form N(N ) xt Dx = N(N ) m i m d(i, )x(i)x(). (2) Expanding (2) sing the real eigenvectors v,...,v m of the symmetric matrix D, with corresponding eigenvales α α m we have that m x T Dx = α (v T x) 2. (3) = This sggests that a high diversity set can be selected by choosing x to maximise (v T x) 2, that is, to choose x(i) = when i is among the N largest components (v () 2,...,v (m) 2 ). Applying this rationale to the selection of a reglariser, we note that fixing P and solving max Q k Q(,l) T DQ(,l) = tr(q T DQ), (4) l= 573

3 for Q fixed, reslts in Q(., l) =v for l =,...k. The reslting ratings, ˆr i, are then proportional to v (i), since p T q i = ( l p l)v (i). A potential drawback of this approach is that, as we cannot control for the sign of the eigenvector, the largest magnitde components of v may correspond to the smallest ratings rather than the largest. An alternative reglariser is given by: d(i, ) q i q 2 =tr(q T L D Q) (5) i where L D = E D is the Laplacian matrix of D and E is the diagonal matrix with i th diagonal entry eqal to d(i, ). Minimising a qadratic form of the Laplacian is a well-known strategy for minimising the edge-ct of a bipartitioning of the rows of the matrix D which in or context amonts to minimising d(i, ). i R / R The critical points again occr at the eigenvectors, v,ofl D. The Laplacian has some nice properties with respect to optimisation. It is positive semi-definite with a minimm eigenvale of 0 obtained for the eigenvector v =(,...,) T.It follows that for all other eigenvectors, v() =0. Moreover, a vector in the span of the eigenvectors corresponding to the largest few eigenvales tends to have high magnitde vales in components corresponding to high diversity sets. Since v() =0, these components occr on either side of the sorted vector i.e. they are either large positive or negative vales, so that the sign problem of the previos reglariser is no longer a problem. The above reglarisers can be natrally extended to P sing the following expressions: d(i, )(p T (q i q )) 2 =tr(pq T L D QP T ), i and d(i, )(p T q i )(p T q ) = tr(pq T DQP T ). i We explore these reglarisers in the case of the Netflix genre distance (see Evalation section for a description of the dataset). In this case, the eigenvales of D lie in the range [ 42.4, ] and the eigenvales of L D lie in the range [0, 939.5]. We perform a gradient descent pdate Q (l+) =Q (l) α(λq (l) H ± reg(q (l) ) to optimise the above reglarisers sing the the Netflix genre distance (see later section) and a randomly chosen P matrix. Here H=P T P when sing one of the P-dependent reglarisers and H=Iotherwise. λ is chosen to ensre a positive definite Hessian and the sign in front of the gradient term is chosen as negative when maximisation of the reglarisation term is reqired and positive when minimisation is reqired. To ensre global convergence, α =.9/(λ + α ) for maximisation and α =.9/(λ α m ) for minimisation. Ten pdate iterations are carried ot. The reslting Q is sed to generate ratings and the top N =50ratings are selected to form Table : Netflix dataset, N =50items, Genre Distance Baseline ILD RankALS 0.66 Random Set 0.77 Max Diversity Set 0.95 Table 2: Netflix dataset, Genre Distance, N = 50, ILD achieved by different reglarisers Reglariser ILD λ max tr(q T L D Q) LapDQ-max ,320 min tr(q T L D Q) LapDQ-min max tr(pq T L D QP T ) PLapDQ-max ,320 max tr(pq T L D QP T ) PLapDQ-min max tr(q T DQ) DQ-max ,500 min tr(q T DQ) DQ-min 0.00,500 the recommender set. The average diversity of the reslting set over 00 randomly selected sers is shown in Table 2, which can be compared in Table with the baseline mean ILD vales obtained for random sets, sets prodced by the non-diversified learning-to-rank algorithm (RankALS) and maximm diversity sets obtained throgh a greedy maximisation from a random initial item. From this analysis, maximisation of the Laplacian reglarisers wold appear to be the best strategy. We will evalate if this holds tre when the reglariser is combined with the accracy obective. ALS Algorithm Naive minimization of () is expensive as the nmber of terms is T I, where T is the nmber of transactions in the rating matrix R. The original algorithm employed in (Jahrer and Töscher 202) sed the stochastic gradient descent (SGD) algorithm and this was improved to an alternating least sqares (ALS) approach in (Takács and Tikk 202). The ALS consists of two steps the P-step and the Q-step, in which the obective fnction is initially minimised w.r.t. P, keeping Q fixed and then w.r.t. Q keeping P fixed. This reqires a calclation of the gradients with respect to P and Q. The P-step may be rearranged into the following linear system to solve for each row of P at step l: βi+ c i i I I c i i I I s (q (l ) i q (l ) )(q (l ) i s (r i r )(q (l ) i q (l ) ) q (l ) ) T p (l) = Similarly, for each row of Q, at step l, the Q- step may be rearranged as the linear system: βi+ s c i p (l) p (l) T q (l) i = I ( c i p (l) p (l) T )( s q (l ) )+ s c i (r i r )p (l). I I 574

4 Table 3: Gradients of the Reglarisers for ALS algorithm Reglariser p q PLapDQ Q T L D QP T P T PQ T L D LapDQ 0 Q T L D DQ 0 Q T D Note that the Q-step reqires old vales of Q from the previos iteration in the RHS vector. We refer to this algorithm as RankALS. We add λreg(p, Q) to the obective, where we choose λ<0to promote soltions that maximise the reglariser and λ > 0 to promote soltions that minimise the reglariser. The gradients of the reglarisers are smmarised in Table 3. For example, in the case of the PLapDQ reglariser, the derivative w.r.t. P is given by Q T L D QP T = i, l iq i q T p. Hence, one possibility to incorporate this reglarisation into the ALS pdate eqations is to add T λ i, l iq (l ) i q (l ) to the LHS matrix. However, we find that more stable soltions are obtained if instead the T p (l ) RHS is pdated with λ i, l iq (l ) i q (l ). In general, we incorporate the diversity reglarisation by modifying the RHS of the pdate eqations sing vales of P and Q from the previos step. Diversity Distribtion Testing Diversity We have discssed some approaches to enhancing the diversity as measred by the ILD of a recommendation list. To test or models, we generate recommendations for a set of U test U of randomly selected test sers and estimate the expected ILD of recommendation lists generated by the model sing the sample mean ILD observed over these sers: ˆμ = d(i, ) (6) U test N(N ) U test i R To test the significance of any observed differences in ILD, we need the standard error of this sample estimator. However, the pairwise differences are not independent, as each item index i appears N times in the set of distances that are averaged to obtain the ILD of each ser. In (Giorgi and Bhattacharya 202), assming only independence across individals, an nbiased estimator of the sampling variance is obtained in the context of comparing intra-individal genetic diversity between poplations. Adopting this to or context, we find an nbiased estimator for Var{ˆμ} as a weighted sm of the sample covariances: ˆσ 0 = (d(i, ) ˆμ)(d(k, l) ˆμ) i<<k<l R ˆσ = (d(i, ) ˆμ)(d(i, k) ˆμ) i<<k R ˆσ 2 = (d(i, ) ˆμ)(d(i, ) ˆμ). i<< R Choosing the weights to ensre that the expected vale of the estimator is Var{ˆμ}, it is possible to show that the reslting estimator is: ˆσ 2 8ˆσ 0 +8ˆσ +4ˆσ 2 = U test ( U test )(N(N )) 2 which, if we assme that the covariance is zero when all indices differ, redces to ˆσ 2 =(8ˆσ +4ˆσ 2) / [ U test N(N ) ( U test (N 2)(N 3) + ( U test )(4N 6))] a formla which agrees with that obtained in (Giorgi and Bhattacharya 202) when U test =. Note that U test ˆσ 2 is an nbiased estimator of the standard deviation of the ILD. We can se this estimator in a paired test for difference in ILD between two models, sing the t-statistic, t paired = ˆμ m ˆμ m2 ˆσ 2 m +ˆσ 2 m 2 which is approximately normally distribted, when U test is large. The mean and spread of ILD vales that are possible among a set of items, depends on the method sed to calclated the item distance. It is therefore sefl to se standardised measres to compare the impact of or diversification methods. We se sch standard measres in the following evalation section. Evalation In this section we present the reslts of or experiments. First we briefly describe the data sets we have sed, followed by the evalation methodology and finally the reslts. Datasets Two datasets are evalated as follows: Netflix: The fll Netflix data set (Bennett and Lanning 2007) consists of 00,480,507 ratings from to 5 from 480,89 sers on 7,770 items. Using IMDb, 28 movie genres have been identified and associated with the movies in the dataset, sch that 9,320 movies have at least one associated genre. Ratings for movies withot genres have been removed. Following (Takács and Tikk 202), ratings are implicitized by assigning if the rating vale is 5, and 0 otherwise, leaving 7,678,86 positive implicit ratings for 9,35 items and 457,07 sers. This final set has been split an 80/20 ratio into train and test sets, containing, respectively,4,43,088 and 3,535,773 ratings. MovieLens 20m: The biggest MovieLens data set released in 205 consists of 20,000,263 ratings from 0.5 to 5 with a step-size of 0.5, from 38,493 sers on 27,278 items, enriched by 8 genres. Items withot genre information have been removed, implicit ratings have been created from ratings eqal to 5, giving a data set consisting of 2,898,660 ratings from 3,839 sers and 4,474 items. This has been split into a training set containing 2,38,928 items and test set with 579,732 items

5 random random Reglariser LapDQ max LapDQ min DQ max DQ min PLapDQ max PLapDQ min Reglariser LapDQ max LapDQ min DQ max DQ min PLapDQ max PLapDQ min Figre : Diversity - accracy trade-off on the Netflix data set. Figre 2: Diversity - accracy trade-off on the MovieLens 20m data set. Evalation Protocol In all experiments with RankALS, we set the nmber of factors k =20and we rn the training phase for 0 iterations. We set β =0, following (Takács and Tikk 202) in which it is reported that no accracy improvement is obtained when standard reglarisation is sed. We set the item importance weighing s = U, the nmber of sers who rated. Accracy and diversity has been checked for different λ vales which control the level of diversity. A set of different metrics has been sed to measre accracy and diversity. For accracy we report reslts of Precision, Recall and ndcg, the diversity is measred by EILD; all metrics are evalated at N =20; As the ratings are binary, we do not employ a relevance model in the calclation of EILD, bt we do se the logarithmic rank discont, which is the same as that employed in the ndcg accracy metric: disc(k) = log 2 (k+). We also report the expected profile distance (EPD) metric (Vargas and Castells 20). High vale indicates high diversity. As a baseline, we have sed a diversity-enhancing MMR re-ranker. The re-ranker has a λ parameter that controls the accracy-diversity trade-off. In or experiment this parameter has been set to λ =0.5which means that we eqally weight diversity and accracy. In order to benefit from the re-ranker, a larger candidate set of items has to be picked before generating the final recommendations. We set the size to be twice bigger than the N, which in or case is 40. The RankSys 2 framework has been sed to rn and evalate the experiments sing bilt-in metrics. Reslts Figres and 2 show the diversity/accracy trade-off plots of different reglarisation methods, for different λ vales on, respectively, the Netflix and MovieLens 20m datasets. For both data sets, the LapDQ reglarisers prodce the best 2 reslts. Varying λ allows the level of diversity to be controlled, by deciding how mch accracy to be sacrificed in order to gain diversity. LapDQ-max shows higher tning possibility. For example, on the Netflix data set, we can increase the diversity from p to with a drop in ndcg from 0.00 to Using the LapDQ-min reglariser, diversity of can be achieved with drop only to Similar behavior can be observed on the Movie- Lens data set: the LapDQ-min reglariser increases diversity from to with a small decrease in the accracy, from to The DQ reglariser does not perform well on either data sets, confirming the observations of or reglariser analysis. With this reglariser, on the Netflix data set some increase in diversity can be observed, althogh the decrease in the accracy is significant, while on MovieLens, it was hard to find any setting that wold allow for a sefl accracy/diversity trade-off. The PLapDQ-max reglariser performs reasonable well, while not reaching the performance of the LapDQ reglariser. Table 4 shows the overall reslts of or experiments of reglariser approaches. We may observe that all accracy measres decrease with the increase of diversity. Reslts are similar whether or not the discont model is sed, except for the LapDQ reglarisers where the logarithmic discont reslts in slightly better diversity performance, at least for the Netflix data set. Increase in EILD leads to an increase in the EPD metric as well, with approximately the same tradeoff. Comparison of the re-ranker approach and reglarisers approach shows that even thogh reglarisers can beat the re-ranker in terms of higher diversity in some settings, they sffer more in terms of accracy. Right now, the re-rankers offer better accracy-diversity trade-off in the off-line evalation. We have standardised the EILD reslts of the best performing algorithms sing the mean and standard deviation of a random recommendation. The mean EILD score of RankALS + LapDQ-min algorithm on Netflix data set is 576

6 Netflix MovieLens 20m Prec Recall ndcg EILD EPD no disc log disc no disc log disc Random RankALS MMR LapDQ-min LapDQ-max DQ-min DQ-max PLapDQ-min PLapDQ-max Random RankALS MMR LapDQ-min LapDQ-max DQ-min DQ-max PLapDQ-min PLapDQ-max Table 4: Reslts on Precision@20 (Prec), Recall@20, ndcg@20, EILD@20 and EPD@20 on different data sets and different reglarisers. For EILD and EPD metrics reslts withot discont are present and with logarithmic discont model. ILD vales have been calclated over all sers and all differences are significant according to the paired t test. The best reslts for each metric across all of tested diversification methods are highlighted in bold standard deviations smaller than the random algorithm. The RankALS is standard deviations smaller than then the random algorithm. The diversity reglariser is a global reglariser, in so far as it seeks to maximise the average performance over the entire poplation. This means it does not necessarily improve the diversity of each individal ser and it is possible that some sers experience a decrease in diversity, in comparison to the non-diversified algorithm. It is therefore interesting to look at the impact of the method across sers. For the RankALS + LapDQ-min on the Netflix data set we observe that 87% of the sers have increased diversity over RankALS and 2% have decreased diversity. For the remaining %, there is no change. For the same algorithm bt a smaller λ vale i.e. less diversification we have 46% of the sers experiencing an increase in diversity bt 52% with decreased diversity compared to RankALS. This illstrates an isse with the approach of global diversification that improved performance for some sers can come at a cost of a redction in performance for others. Conclsions and Frther Work The research presented here aimed to contine and explore the work pblished in (Hrley 203). A nmber of diversity reglarisers have been proposed and evalated, showing that it is possible to incorporate diversity into the training phase of a learning to rank algorithm for recommender systems. Of the proposed reglarisers, the LapDQ reglariser showed the best performance among those compared. A nmber of short-comings of this approach can be identified. In particlar, optimising for an global average improvement in diversity means that boosting the diversity for some sers can mean a redction in diversity for others. Ultimately, when the diversity term is strong enogh, all sers experience a diversity boost, bt the accracy generally deteriorates. Moreover, it may be better if diversification focsed on the top candidate items, rather than across all items. Ways to address these short-comings may be directions for ftre work. Acknowledgments This proect has been fnded by Science Fondation Ireland nder Grant No. SFI/2/RC/2289. References Bennett, J., and Lanning, S The Netflix Prize. KDD Cp and Workshop 3 6. Carbonell, J., and Goldstein, J The Use of MMR, Diversity-Based Reranking for Reordering Docments and Prodcing Smmaries. Proceedings of the 2st annal international ACM SIGIR conference on Research and development in information retrieval Ekstrand, M. D.; Harper, F. M.; Willemsen, M. C.; and Konstan, J. a User perception of differences in recommender algorithms. Proceedings of the 8th ACM Conference on Recommender systems - RecSys Giorgi, E. E., and Bhattacharya, T A Note on Two-Sample Tests for Comparing Intra-Individal Genetic Seqence Diversity between Poplations. Biometrics 68(4): H, Y.; Koren, Y.; and Volinsky, C Collaborative filtering for implicit feedback datasets. In Proceedings of the 2008 Eighth IEEE International Conference on Data Mining, ICDM 08, Washington, DC, USA: IEEE Compter Society. Hrley, N. J Personalised Ranking with Diversity. Proceedings of the 7th ACM conference on Recommender systems - RecSys 3 2(): Jahrer, M., and Töscher, A Collaborative filtering ensemble for ranking. Jornal of Machine Learning Research - Proceedings Track 8: Jamali, M., and Ester, M A Matrix Factorization Techniqe with Trst Propagation for Recommendation in Social Networks. Proceedings of the forth ACM conference on Recommender systems - RecSys Pilászy, I.; Zibriczky, D.; and Tikk, D Fast als-based matrix factorization for explicit and implicit feedback datasets. In Proceedings of the forth ACM conference on Recommender systems, RecSys 0, New York, NY, USA: ACM. Takács, G., and Tikk, D Alternating Least Sqares for Personalized Ranking. Proceedings of the 6th ACM conference on Recommender systems - RecSys Vargas, S., and Castells, P. 20. Rank and Relevance in Novelty and Diversity Metrics for Recommender Systems. Proceedings of the 5th ACM conference on Recommender systems - RecSys 09. Vargas, S.; Castells, P.; and Vallet, D. 20. Intent-oriented Diversity in Recommender Systems. Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval Zhang, M., and Hrley, N Avoiding monotony: Improving the diversity of recommendation lists. In Proceedings of 2nd ACM International Conference on Recommender Systems. Ziegler, C.; McNee, S. M.; Konstan, J. A.; and Lasen, G Improving recommendation lists throgh topic diversification. In Proceedings of the 4th International Conference on World Wide Web,