Advanced Click Models & their Applications to IR

Size: px

Start display at page:

Download "Advanced Click Models & their Applications to IR"

Lizbeth Hunt
5 years ago
Views:

1 Advanced Click Models & their Applications to IR (Afternoon block 2) Aleksandr Chuklin, Ilya Markov Maarten de Rijke University of Amsterdam Google Switzerland SIGIR 2015 Tutorial AC IM MdR Advanced Click Models & their Applications to IR 50

2 Afternoon block 2 Outline 1 Demo 2 Guest Talk 3 Applications 4 Future Research 5 Recap AC IM MdR Advanced Click Models & their Applications to IR 51

3 Live demo: Building an advanced click model (cont d) Demo AC IM MdR Advanced Click Models & their Applications to IR 52

4 Implementing TCM AC IM MdR Advanced Click Models & their Applications to IR 53

5 Implementing TCM AC IM MdR Advanced Click Models & their Applications to IR 53

6 Implementing TCM 1 Adding TCM-specific parameters (a) Create TCMMatch, TCMNew, TCMFresh as subclasses of ParamEM (b) Add these parameters to TCM, re-run training (c) Still get the same output as in PBM 2 Update the EM formulas for attractiveness and examination (a) Add matching and freshness to the EM formulas, re-run training (b) Get the updated output 3 Implement EM formulas for other parameters (a) Plug the EM updating formulas into Param.update(), re-run training (b) Output the trained TCM parameters AC IM MdR Advanced Click Models & their Applications to IR 54

7 Session parameters Rank Document 1 u 1 2 u 2 3 u 3 4 u 4 5 u 5 AC IM MdR Advanced Click Models & their Applications to IR 55

8 Session parameters Rank Document 1 u 1 2 u 2 3 u 3 4 u 4 5 u 5 Session Parameters {α u1 q, ɛ 1, µ, ν, φ} {α u2 q, ɛ 2, µ, ν, φ} {α u3 q, ɛ 3, µ, ν, φ} {α u4 q, ɛ 4, µ, ν, φ} {α u5 q, ɛ 5, µ, ν, φ} AC IM MdR Advanced Click Models & their Applications to IR 55

9 EM update rules for PBM parameters C r = 1 E r = 1, A r = 1 AC IM MdR Advanced Click Models & their Applications to IR 56

10 EM update rules for PBM parameters Attractiveness α (t+1) uq = 1 S uq Examination C r = 1 E r = 1, A r = 1 s S uq γ r (t+1) = 1 S r s S r c (s) u + c (s) u + ( ( 1 c (s) u 1 c (s) u ( ) ( ) 1 γ (t) r ) α (t) uq 1 γ r (t) α uq (t) 1 α (t) uq ) γ (t) r 1 γ r (t) α uq (t) AC IM MdR Advanced Click Models & their Applications to IR 56

11 EM update rules for TCM parameters Attractiveness C r = 1 E r = 1, A r = 1, M = 1, F r = 1 Examination AC IM MdR Advanced Click Models & their Applications to IR 57

12 EM update rules for TCM parameters Attractiveness α (t+1) uq = 1 S uq C r = 1 E r = 1, A r = 1, M = 1, F r = 1 s S uq c (s) u + ( 1 c (s) u ) ( 1 ɛ (t) r µ (t) φ (t) ) α (t) uq 1 ɛ (t) r α uq (t) µ (t) φ (t) Examination ɛ (t+1) r = 1 S r s S r c (s) u + ( 1 c (s) u ) ( 1 α (t) uq µ (t) φ (t) ) ɛ (t) r 1 ɛ r (t) α uq (t) µ (t) φ (t) AC IM MdR Advanced Click Models & their Applications to IR 57

13 EM update formulas (recap) Each node X depends only on its parents P(X ) A D C E θ (t+1) c = s S c i s (X P c (s) i s S B = 1, P(X (s) c i ) ) = p C, Ψ ) c i s (P(X P c (s) i ) = p C, Ψ AC IM MdR Advanced Click Models & their Applications to IR 58

14 EM update formulas (recap) Each node X depends only on its parents P(X ) P(C) = {A, B} A D C E θ (t+1) c = s S c i s (X P c (s) i s S B = 1, P(X (s) c i ) ) = p C, Ψ ) c i s (P(X P c (s) i ) = p C, Ψ AC IM MdR Advanced Click Models & their Applications to IR 58

15 EM update formulas (recap) Each node X depends only on its parents P(X ) P(C) = {A, B} A D C E θ (t+1) c = P(A) = s S c i s (X P c (s) i s S B = 1, P(X (s) c i ) ) = p C, Ψ ) c i s (P(X P c (s) i ) = p C, Ψ AC IM MdR Advanced Click Models & their Applications to IR 58

16 TCM formulas P(M = 1) = µ P(N = 1 M = 0) = 1 P(N = 1 M = 1) = ν P(F r = 1 H r = 0) = 1 P(F r = 1 H r = 1) = φ P(E r = 1 E <r, S <r, C <r ) = ɛ r H r = 1 previous occurence of u r P(A r = 1) = α ur q C r = 1 M = 1, F r = 1, E r = 1, A r = 1 AC IM MdR Advanced Click Models & their Applications to IR 59

17 EM updating rules for TCM parameters (cont d) Query matching the user s information need µ (t+1) = 1 S P s S ( ) M (s) = 1 C AC IM MdR Advanced Click Models & their Applications to IR 60

18 EM updating rules for TCM parameters (cont d) Query matching the user s information need P(M (s) = 1 C) =? if C (s) 0 where (l) is the last session in the task µ (t+1) = 1 S P s S ( ) M (s) = 1 C AC IM MdR Advanced Click Models & their Applications to IR 60

19 EM updating rules for TCM parameters (cont d) Query matching the user s information need P(M (s) = 1 C) = 1 if C (s) 0 where (l) is the last session in the task µ (t+1) = 1 S P s S ( ) M (s) = 1 C AC IM MdR Advanced Click Models & their Applications to IR 60

20 EM updating rules for TCM parameters (cont d) Query matching the user s information need P(M (s) = 1 C) =? if (s) = (l) where (l) is the last session in the task µ (t+1) = 1 S P s S ( ) M (s) = 1 C AC IM MdR Advanced Click Models & their Applications to IR 60

21 EM updating rules for TCM parameters (cont d) Query matching the user s information need P(M (s) = 1 C) = 1 if C (s) 0 or (s) = (l) where (l) is the last session in the task µ (t+1) = 1 S P s S ( ) M (s) = 1 C AC IM MdR Advanced Click Models & their Applications to IR 60

22 EM updating rules for TCM parameters (cont d) Query matching the user s information need P(M (s) = 1 C) = 1 if C (s) 0 or (s) = (l) otherwise where (l) is the last session in the task µ (t+1) = 1 S P s S ( ) M (s) = 1 C AC IM MdR Advanced Click Models & their Applications to IR 60

23 EM updating rules for TCM parameters (cont d) Query matching the user s information need = P(M (s) = 1 C) = 1 if C (s) 0 or (s) = (l) otherwise P(M (s) = 1 C) = P(M (s) = 1 C (s), (s) (l)) P(C (s), (s) (l) M (s) = 1)P(M (s) = 1) P(C (s), (s) (l) M (s) = 1)P(M (s) = 1) + P(C (s), (s) (l) M (s) = 0)P(M (s) = 0) = P(C (s) M = 1)νµ P(C (s) M = 1)νµ + 1 µ where (l) is the last session in the task µ (t+1) = 1 S P s S ( ) M (s) = 1 C AC IM MdR Advanced Click Models & their Applications to IR 60

24 EM updating rules for TCM parameters (cont d) Submitting a new query after a matching query ν (t+1) = s S P ( N (s) = 1, M (s) = 1 C ) P ( M (s) = 1 C ) AC IM MdR Advanced Click Models & their Applications to IR 61

25 EM updating rules for TCM parameters (cont d) Submitting a new query after a matching query P(N (s) = 1, M (s) = 1 C) =? if (s) = (l) otherwise P(N (s) = 1, M (s) = 1 C) =? ν (t+1) = P ( N (s) = 1, M (s) = 1 C ) P ( M (s) = 1 C ) s S AC IM MdR Advanced Click Models & their Applications to IR 61

26 EM updating rules for TCM parameters (cont d) Submitting a new query after a matching query P(N (s) = 1, M (s) = 1 C) = 0 if (s) = (l) otherwise P(N (s) = 1, M (s) = 1 C) =? ν (t+1) = P ( N (s) = 1, M (s) = 1 C ) P ( M (s) = 1 C ) s S AC IM MdR Advanced Click Models & their Applications to IR 61

27 EM updating rules for TCM parameters (cont d) Submitting a new query after a matching query P(N (s) = 1, M (s) = 1 C) = 0 if (s) = (l) otherwise P(N (s) = 1, M (s) = 1 C) = P(M (s) = 1 C) ν (t+1) = P ( N (s) = 1, M (s) = 1 C ) P ( M (s) = 1 C ) s S AC IM MdR Advanced Click Models & their Applications to IR 61

28 EM updating rules for TCM parameters (cont d) Submitting a new query after a matching query P(N (s) = 1, M (s) = 1 C) = 0 if (s) = (l) otherwise P(N (s) = 1, M (s) = 1 C) = P(M (s) = 1 C) ν (t+1) = P ( N (s) = 1, M (s) = 1 C ) P ( M (s) = 1 C ) s S Freshness parameter φ (t+1) = 1 S s S 1 s u s:h (s) u ( c (s) u ( ( ) + 1 c u (s) 1 ɛ (t) r α (t) uq µ (t) ) φ (t) 1 ɛ (t) r α uq (t) µ (t) φ (t) AC IM MdR Advanced Click Models & their Applications to IR 61 )

29 Next steps: Testing AC IM MdR Advanced Click Models & their Applications to IR 62

30 Next steps: Testing Extend LogLikelihood to work with search tasks AC IM MdR Advanced Click Models & their Applications to IR 62

31 Next steps: Testing Extend LogLikelihood to work with search tasks Derive formulas for TCM click probabilities (full and conditional) AC IM MdR Advanced Click Models & their Applications to IR 62

32 Next steps: Testing Extend LogLikelihood to work with search tasks Derive formulas for TCM click probabilities (full and conditional) Plug the formulas into TCM.get full click probs() and TCM.get conditional click probs() AC IM MdR Advanced Click Models & their Applications to IR 62

33 Afternoon block 2 Outline 1 Demo 2 Guest Talk 3 Applications 4 Future Research 5 Recap AC IM MdR Advanced Click Models & their Applications to IR 63

34 Guest talk Building a Click Model: From Idea to Implementation Yiqun Liu Tsinghua University

35 Afternoon block 2 Outline 1 Demo 2 Guest Talk 3 Applications 4 Future Research 5 Recap AC IM MdR Advanced Click Models & their Applications to IR 65

36 Applications Simulating users User interaction analysis Inference of relevance Model-based metrics AC IM MdR Advanced Click Models & their Applications to IR 66

37 Simulating users Simulating users in the same sessions (data expansion) AC IM MdR Advanced Click Models & their Applications to IR 67

38 Simulating users Simulating users in the same sessions (data expansion) Simulating users in new sessions AC IM MdR Advanced Click Models & their Applications to IR 67

39 Simulating users Simulating users in the same sessions (data expansion) Simulating users in new sessions Simulating users on learning-to-rank datasets AC IM MdR Advanced Click Models & their Applications to IR 67

40 Simulating users in new sessions Algorithm 1 Simulating user clicks for a new query session. Input: click model M, query session s Output: vector of simulated clicks (c 1,..., c n ) 1: for r 1 to s do 2: p r P M (C r = 1 C 1 = c 1,..., C r 1 = c r 1 ) 3: Generate c r from Bernoulli(p r ) 4: end for AC IM MdR Advanced Click Models & their Applications to IR 68

41 User interaction analysis distance distance rank rank Figure 2: Left: Mean over 21 training sets of the probabilities of examination for all the ra combinations. Right: Standard deviation of the probabilities on the left pane estimated empi 21 replications. The darker the color, the higher the probability or the standard deviation. left pane signals the probability of examinations of a query session with one click at position Figure: UBM examination parameters. It is interesting to compare the diagonals on Fig. 2. A user ses stating that users tend to browse docu Picture taken from G.E. Dupret whoand never B. clicks Piwowarski. a document A user stays browsing on the highest model diagonal to predict search bottom engine mostclick of the data time. fromthey observe past observations. In SIGIR, where probabilities ACM Press. are comparatively large, suggesting that snippets receive substantially more user a he searches more actively than if he had already clicked once. others. We reach the same conclusion on Assuming that AC IM MdR none of the documents Advanced is attractive, Click themodels prob- & their two first Applications snippets aretoalmost IR always 69exam

42 Inference of relevance R uq = P(S u = 1 E ru = 1) = P(S u = 1 C u = 1) P(C u = 1 E ru = 1) AC IM MdR Advanced Click Models & their Applications to IR 70

43 Inference of relevance R uq = P(S u = 1 E ru = 1) = P(S u = 1 C u = 1) P(C u = 1 E ru = 1) PBM, CM, UBM, DCM R uq = P(C u = 1 E ru = 1) = α uq AC IM MdR Advanced Click Models & their Applications to IR 70

44 Inference of relevance R uq = P(S u = 1 E ru = 1) = P(S u = 1 C u = 1) P(C u = 1 E ru = 1) PBM, CM, UBM, DCM R uq = P(C u = 1 E ru = 1) = α uq DBN R uq = P(S u = 1 C u = 1) P(C u = 1 E ru = 1) = σ uq α uq AC IM MdR Advanced Click Models & their Applications to IR 70

45 Model-based metrics Utility-based metrics n umetric = P(C r = 1) U r r=1 AC IM MdR Advanced Click Models & their Applications to IR 71

46 Model-based metrics Utility-based metrics umetric = Effort-based metrics emetric = n P(C r = 1) U r r=1 n P(S r = 1) F r r=1 AC IM MdR Advanced Click Models & their Applications to IR 71

47 Expected reciprocal rank AC IM MdR Advanced Click Models & their Applications to IR 72

48 Expected reciprocal rank RR = 1 r, where S r = 1 AC IM MdR Advanced Click Models & their Applications to IR 72

49 Expected reciprocal rank RR = 1 r, where S r = 1 ERR = r 1 r P(S r = 1) AC IM MdR Advanced Click Models & their Applications to IR 72

50 DBN P(A r = 1) = α ur q P(E 1 = 1) = 1 P(E r = 1 S r 1 = 1) = 0 P(E r = 1 S r 1 = 0) = γ P(S r = 1 C r = 0) = 0 P(S r = 1 C r = 1) = σ ur q AC IM MdR Advanced Click Models & their Applications to IR 73

51 DBN P(A r = 1) = α ur q P(E 1 = 1) = 1 P(E r = 1 S r 1 = 1) = 0 P(E r = 1 S r 1 = 0) = γ P(S r = 1 C r = 0) = 0 P(S r = 1 C r = 1) = σ ur q P(S r = 1) =? AC IM MdR Advanced Click Models & their Applications to IR 73

52 DBN: Satisfaction P(S r = 1) = P(S r = 1 C r = 1) P(C r = 1) AC IM MdR Advanced Click Models & their Applications to IR 74

53 DBN: Satisfaction P(S r = 1) = P(S r = 1 C r = 1) P(C r = 1) = P(S r = 1 C r = 1) P(A r = 1) P(E r = 1) AC IM MdR Advanced Click Models & their Applications to IR 74

54 DBN: Satisfaction P(S r = 1) = P(S r = 1 C r = 1) P(C r = 1) = P(S r = 1 C r = 1) P(A r = 1) P(E r = 1) = σ ur q α ur q P(E r = 1) AC IM MdR Advanced Click Models & their Applications to IR 74

55 DBN: Satisfaction P(S r = 1) = P(S r = 1 C r = 1) P(C r = 1) = P(S r = 1 C r = 1) P(A r = 1) P(E r = 1) = σ ur q α ur q P(E r = 1) = R ur q P(E r = 1) AC IM MdR Advanced Click Models & their Applications to IR 74

56 DBN: Satisfaction P(S r = 1) = P(S r = 1 C r = 1) P(C r = 1) = P(S r = 1 C r = 1) P(A r = 1) P(E r = 1) = σ ur q α ur q P(E r = 1) = R ur q P(E r = 1) P(E r = 1) =? AC IM MdR Advanced Click Models & their Applications to IR 74

57 DBN: Examination P(E r = 1) = P(E r = 1 E r 1 = 1) P(E r 1 = 1) AC IM MdR Advanced Click Models & their Applications to IR 75

58 DBN: Examination P(E r = 1) = P(E r = 1 E r 1 = 1) P(E r 1 = 1) = P(E r = 1 S r 1 = 0, E r 1 = 1) P(S r 1 = 0 E r 1 = 1) P(E r 1 = 1) AC IM MdR Advanced Click Models & their Applications to IR 75

59 DBN: Examination P(E r = 1) = P(E r = 1 E r 1 = 1) P(E r 1 = 1) = P(E r = 1 S r 1 = 0, E r 1 = 1) P(S r 1 = 0 E r 1 = 1) P(E r 1 = 1) = γ (1 α ur 1 q σ ur 1 q) P(E r 1 = 1) AC IM MdR Advanced Click Models & their Applications to IR 75

60 DBN: Examination P(E r = 1) = P(E r = 1 E r 1 = 1) P(E r 1 = 1) = P(E r = 1 S r 1 = 0, E r 1 = 1) P(S r 1 = 0 E r 1 = 1) P(E r 1 = 1) = γ (1 α ur 1 q σ ur 1 q) P(E r 1 = 1) = γ (1 R ur 1 q) P(E r 1 = 1) AC IM MdR Advanced Click Models & their Applications to IR 75

61 DBN: Examination P(E r = 1) = P(E r = 1 E r 1 = 1) P(E r 1 = 1) = P(E r = 1 S r 1 = 0, E r 1 = 1) P(S r 1 = 0 E r 1 = 1) P(E r 1 = 1) = γ (1 α ur 1 q σ ur 1 q) P(E r 1 = 1) = γ (1 R ur 1 q) P(E r 1 = 1) r 1 ( = γ (1 Rui q) ) i=1 AC IM MdR Advanced Click Models & their Applications to IR 75

62 Expected reciprocal rank ERR = r 1 r P(S r = 1) AC IM MdR Advanced Click Models & their Applications to IR 76

63 Expected reciprocal rank ERR = r = r 1 r P(S r = 1) 1 r R u r q P(E r = 1) AC IM MdR Advanced Click Models & their Applications to IR 76

64 Expected reciprocal rank ERR = r = r = r 1 r P(S r = 1) 1 r R u r q P(E r = 1) 1 r R u r q r 1 ( γ (1 Rui q) ) i=1 AC IM MdR Advanced Click Models & their Applications to IR 76

65 Model-based metrics Model-based metric Click model Utility-based Effort-based DBN usdbn ERR DBN EBU rrdbn DCM udcm rrdcm UBM uubm AC IM MdR Advanced Click Models & their Applications to IR 77

66 Model-based metrics Table 2: Pearson correlation between o ine and absolute online metrics. Superscripts represent statistically significant di erence from ERR and EBU. -RR Max- Min- Mean- UCTR PLC Precision Precision DCG ERR EBU rrdbn úú úú úú ùù ú rrdcm úú úú úú ùù ú usdbn ùù ùù ùù úú ùù udcm ùù ùù ùù ùù uubm ú ùú ùú ú power do low value tion of a ings. As criminativ tistical te test as it distributi Track dat in Table 4 (rrdbn, inative po Anothe metrics E tive powe 5 Initially, scription evaluation distinguis 6 In total A. Chuklin, P. Serdyukov, and M. de Rijke. Click model-based information retrieval metrics. In SIGIR, ACM Press. 500 AC IM MdR Advanced Click Models & their Applications to IR 78

67 Afternoon block 2 Outline 1 Demo 2 Guest Talk 3 Applications 4 Future Research 5 Recap AC IM MdR Advanced Click Models & their Applications to IR 79

68 Future research Foundations of user modeling Rich interactions and alternative search environments Fusion of ideas AC IM MdR Advanced Click Models & their Applications to IR 80

69 Foundations of user modeling Current click models are mostly based on probabilistic graphical models ru Eu uq Au Cu document u AC IM MdR Advanced Click Models & their Applications to IR 81

70 Foundations of user modeling Current click models are mostly based on probabilistic graphical models Alternatives are: Undirected Markov networks Partially directed conditional random fields ru Eu Cu document u uq Au AC IM MdR Advanced Click Models & their Applications to IR 81

Foundations of user modeling Current click models are mostly based on probabilistic graphical models Alternatives are: Undirected Markov networks

71 Foundations of user modeling Current click models are mostly based on probabilistic graphical models Alternatives are: Undirected Markov networks Partially directed conditional random fields Structure learning ru Eu Cu document u uq Au AC IM MdR Advanced Click Models & their Applications to IR 81

72 Rich interactions and alternative search environments Interactions beyond clicks Devices beyond desktop computers AC IM MdR Advanced Click Models & their Applications to IR 82

73 Fusion of ideas User behavior analysis for sponsored search AC IM MdR Advanced Click Models & their Applications to IR 83

74 Fusion of ideas User behavior analysis for sponsored search User experience studies AC IM MdR Advanced Click Models & their Applications to IR 83

75 Fusion of ideas User behavior analysis for sponsored search User experience studies Network and game theory AC IM MdR Advanced Click Models & their Applications to IR 83

76 Fusion of ideas User behavior analysis for sponsored search User experience studies Network and game theory Learning to rank AC IM MdR Advanced Click Models & their Applications to IR 83

77 Afternoon block 2 Outline 1 Demo 2 Guest Talk 3 Applications 4 Future Research 5 Recap AC IM MdR Advanced Click Models & their Applications to IR 84

78 Afternoon block 2 Recap Demo: building an advanced click model Guest talk by Yiqun Liu on going from idea to implementation when building a click model Various applications of click models Directions for future research on user modeling AC IM MdR Advanced Click Models & their Applications to IR 85

79 We are done Aim of the tutorials today has been to Offer a unified view of click models, both basic ones and more advanced ones Compare click models Point you to ready-made formulas Point you to available packages and datasets AC IM MdR Advanced Click Models & their Applications to IR 86

80 We are done Aim of the tutorials today has been to Offer a unified view of click models, both basic ones and more advanced ones Compare click models Point you to ready-made formulas Point you to available packages and datasets Two things to end: Thank you! Go and create your own click models AC IM MdR Advanced Click Models & their Applications to IR 86

81 We are done Aim of the tutorials today has been to Offer a unified view of click models, both basic ones and more advanced ones Compare click models Point you to ready-made formulas Point you to available packages and datasets Two things to end: Thank you! Go and create your own click models AC IM MdR Advanced Click Models & their Applications to IR 86

82 Acknowledgments All content represents the opinion of the authors which is not necessarily shared or endorsed by their respective employers and/or sponsors. AC IM MdR Advanced Click Models & their Applications to IR 87

Click Models for Web Search

Click Models for Web Search Lecture 1 Aleksandr Chuklin, Ilya Markov Maarten de Rijke a.chuklin@uva.nl i.markov@uva.nl derijke@uva.nl University of Amsterdam Google Research Europe AC IM MdR Click Models