Click Models for Web Search

Size: px

Start display at page:

Download "Click Models for Web Search"

Bruce Maxwell
6 years ago
Views:

1 Click Models for Web Search Lecture 1 Aleksandr Chuklin, Ilya Markov Maarten de Rijke a.chuklin@uva.nl i.markov@uva.nl derijke@uva.nl University of Amsterdam Google Research Europe AC IM MdR Click Models for Web Search 1

2 Aleksandr Chuklin Currently at Google Zürich Previously at Yandex Moscow Research interests: user experience evaluation and modelling Participated at RuSSIR 2009 in Petrozavodsk and RuSSIR 2011 in Saint Petersburg AC IM MdR Click Models for Web Search 2

3 Ilya Markov Postdoctoral researcher at the University of Amsterdam PhD at the University of Lugano Research interests: heterogeneous search environments Distributed IR, federated search, aggregated search User behavior, user-oriented evaluation Teach MSc courses on IR and Web Search AC IM MdR Click Models for Web Search 3

4 Long-term relations with RuSSIR RuSSIR 2007, student RuSSIR 2010, lecturer on Distributed IR (with Fabio Crestani) RuSSIR 2011, member of organizing committee RuSSIR 2015, chair of program committee RuSSIR 2016, lecturer AC IM MdR Click Models for Web Search 4

5 Course on Information Retrieval in St. Petersburg information-retrieval/2016-autumn/ AC IM MdR Click Models for Web Search 5

6 Maarten de Rijke Currently at the University of Amsterdam Ongoing collaborations with Bloomberg Labs, Google, Microsoft Research, Yandex Moscow Research interests: semantic search, online learning to rank Always looking for strong new PhD students AC IM MdR Click Models for Web Search 6

7 The book AC IM MdR Click Models for Web Search 7

8 Other course materials clickmodels.weebly.com/russir-2016-course.html Demos and practical sessions: clickmodels.weebly.com/russir-2016-setup.html github.com/markovi/pyclick AC IM MdR Click Models for Web Search 8

9 Course content Basic Click Models Parameter Estimation Evaluation Applications Results Data and Tools Advanced Models Recent Studies Future Research AC IM MdR Click Models for Web Search 9

10 Lectures Basic Click Models Parameter Estimation Evaluation Lecture 1 Lecture 2 Applications Lecture 2 Results Lecture 3 Practical 1 Data and Tools Advanced Models Recent Studies Lecture 4 Practical 2 Lecture 5 Future Research AC IM MdR Click Models for Web Search 10

11 Course overview Basic Click Models Parameter Estimation Evaluation Applications Results Data and Tools Advanced Models Recent Studies Future Research AC IM MdR Click Models for Web Search 11

12 This lecture Basic Click Models Parameter Estimation Evaluation Applications Results Data and Tools Advanced Models Recent Studies Future Research AC IM MdR Click Models for Web Search 12

13 Lecture outline 1 Introduction 2 Basic click models 3 Click probabilities AC IM MdR Click Models for Web Search 13

14 Web search AC IM MdR Click Models for Web Search 14

15 Why clicks? AC IM MdR Click Models for Web Search 15

16 Why clicks? Reflect user interests Help to improve search Help to evaluate search Ongoing and future research: other user search interactions mouse movements scrolling touch gestures AC IM MdR Click Models for Web Search 16

17 What can we do with clicks? AC IM MdR Click Models for Web Search 17

18 What can we do with clicks? count click-through rate (CTR) Global CTR = Rank-based CTR = Query-document CTR = # clicks # shown docs # clicks at rank r # shown docs at rank r # u is clicked for q # u is shown for q Some notation: u URL (or document), q query AC IM MdR Click Models for Web Search 18

19 Why click models? AC IM MdR Click Models for Web Search 19

20 Why click models? Scientific modelling is a scientific activity, the aim of which is to make a particular part or feature of the world easier to understand, define, quantify, visualize, or simulate by referencing it to existing and usually commonly accepted knowledge. Wikipedia, Scientific modelling AC IM MdR Click Models for Web Search 20

21 Why click models? Click models make user clicks in web search easier to understand, define, quantify, visualize, or simulate using (mostly) probabilistic graphical models. AC IM MdR Click Models for Web Search 21

22 Click log Yandex Relevance Prediction Challenge AC IM MdR Click Models for Web Search 22

23 Why do we need click models? Understand users Simulate users Approximate document relevance Evaluate search AC IM MdR Click Models for Web Search 23

24 Lecture outline 1 Introduction 2 Basic click models Random click model CTR models Position-based model Cascade model Dynamic Bayesian network model User browsing model 3 Click probabilities AC IM MdR Click Models for Web Search 24

25 Lecture outline 2 Basic click models Random click model CTR models Position-based model Cascade model Dynamic Bayesian network model User browsing model AC IM MdR Click Models for Web Search 25

26 Random click model P click P click P click P click P click AC IM MdR Click Models for Web Search 26

27 Random click model Terminology C u binary random variable denoting a click on document u Document u is clicked: C u = 1 Document u is not clicked: C u = 0 P(C u = 1) probability of click on document u P(C u = 0) = 1 P(C u = 1) Random click model (RCM) Any document can be clicked with the same (fixed) probability P(C u = 1) = const = ρ AC IM MdR Click Models for Web Search 27

28 Random click model P(C u1 = 1) = ρ P(C u2 = 1) = ρ P(C u3 = 1) = ρ P(C u4 = 1) = ρ ρ = # clicks = Global CTR # shown docs P(C u5 = 1) = ρ AC IM MdR Click Models for Web Search 28

29 Lecture outline 2 Basic click models Random click model CTR models Position-based model Cascade model Dynamic Bayesian network model User browsing model AC IM MdR Click Models for Web Search 29

30 Rank-based CTR model P(C u1 = 1) = ρ 1 P(C u2 = 1) = ρ 2 P(C u3 = 1) = ρ 3 P(C u4 = 1) = ρ 4 P(C ur = 1) = ρ r = P(C u5 = 1) = ρ 5 # clicks at rank r # shown docs at rank r AC IM MdR Click Models for Web Search 30

31 Query-document CTR model P(C u1 = 1) = ρ u1 q P(C u2 = 1) = ρ u2 q P(C u3 = 1) = ρ u3 q P(C u4 = 1) = ρ u4 q P(C u = 1) = ρ uq = # u is clicked for q # u is shown for q P(C u5 = 1) = ρ u5 q AC IM MdR Click Models for Web Search 31

32 CTR models: summary Random click model (global CTR): Rank-based CTR: P(C u = 1) = ρ P(C ur = 1) = ρ r Query-document CTR: P(C u = 1) = ρ uq AC IM MdR Click Models for Web Search 32

33 CTR models: demo Demo AC IM MdR Click Models for Web Search 33

34 Lecture outline 2 Basic click models Random click model CTR models Position-based model Cascade model Dynamic Bayesian network model User browsing model AC IM MdR Click Models for Web Search 34

35 Position-based model P read (1), P click (u 1 q) P read (2), P click (u 2 q) P read (3), P click (u 3 q) P read (4), P click (u 4 q) P read (5), P click (u 5 q) AC IM MdR Click Models for Web Search 35

36 Position-based model: examination Terminology Examination = reading a snippet E r binary random variable denoting examination of a snippet at rank r Snippet at rank r is examined: E r = 1 Snippet at rank r is not examined: E r = 0 P(E r = 1) probability of examination of rank r P(E r = 0) = 1 P(E r = 1) Position-based model (PBM) Examination depends on rank P(E r = 1) = γ r AC IM MdR Click Models for Web Search 36

37 Position-based model γ 1, P click (u 1 q) γ 2, P click (u 2 q) γ 3, P click (u 3 q) γ 4, P click (u 4 q) γ 5, P click (u 5 q) AC IM MdR Click Models for Web Search 37

38 Position-based model: attractiveness Terminology Attractiveness = a user wants to click on a document after examining (reading) its snippet A u binary random variable showing whether document u is attractive to a user, given query q Document u is attractive: A u = 1 Document u is not attractive: A u = 0 P(A u = 1) probability of attractiveness of document u P(A u = 0) = 1 P(A u = 1) Position-based model (PBM) Attractiveness depends on a query-document pair P(A uq = 1) = α uq AC IM MdR Click Models for Web Search 38

39 Position-based model γ 1, α u1 q γ 2, α u2 q γ 3, α u3 q γ 4, α u4 q γ 5, α u5 q AC IM MdR Click Models for Web Search 39

40 Position-based model: summary P(E ru = 1) = γ ru P(A u = 1) = α uq P(C u = 1) = P(E ru = 1) P(A u = 1) AC IM MdR Click Models for Web Search 40

41 Position-based model: probabilistic graphical model ru uq E u A u C u document u AC IM MdR Click Models for Web Search 41

42 Position-based model: exercises P(E ru = 1) = γ ru P(A u = 1) = α uq P(C u = 1) = P(E ru = 1) P(A u = 1) E ru = 0 C u = 0 A u = 0 C u = 0 E ru = 1 (C u = 1 A u = 1) A u = 1 (C u = 1 E ru = 1) AC IM MdR Click Models for Web Search 42

43 Position-based model: demo Demo AC IM MdR Click Models for Web Search 43

44 Lecture outline 2 Basic click models Random click model CTR models Position-based model Cascade model Dynamic Bayesian network model User browsing model AC IM MdR Click Models for Web Search 44

45 Position-based model P(E ru = 1) = γ ru P(A u = 1) = α uq P(C u = 1) = P(E ru = 1) P(A u = 1) AC IM MdR Click Models for Web Search 45

46 Cascade model 1 Start from the first document 2 Examine documents one by one 3 If click, then stop 4 Otherwise, continue AC IM MdR Click Models for Web Search 46

47 Cascade model E r = 1 and A ur = 1 C r = 1 P(A ur = 1) = α ur q P(E 1 = 1) = 1 }{{} start from first P(E r = 1 E r 1 = 0) = 0 }{{} examine one by one P(E r = 1 C r 1 = 1) = 0 }{{} if click, then stop P(E r = 1 E r 1 = 1, C r 1 = 0) = 1 }{{} otherwise, continue AC IM MdR Click Models for Web Search 47

48 Cascade model: probabilistic graphical model ur 1q urq document u r 1 document u r Ar 1 Ar Cr 1 Cr... Er 1 Er... AC IM MdR Click Models for Web Search 48

49 Click models so far CTR models + count clicks (simple and fast) do not distinguish examination and attractiveness Position-based model (PBM) User browsing model + examination and attractiveness examination of a document at rank r does not depend on examinations and clicks above r Cascade model (CM) Dynamic Bayesian network + cascade dependency of examination at r on examinations and clicks above r only one click is allowed AC IM MdR Click Models for Web Search 49

50 Lecture outline 2 Basic click models Random click model CTR models Position-based model Cascade model Dynamic Bayesian network model User browsing model AC IM MdR Click Models for Web Search 50

51 Dynamic Bayesian network model Cascade model Dynamic Bayesian network AC IM MdR Click Models for Web Search 51

52 Dynamic Bayesian network model 1 Start from the first document 2 Examine documents one by one 3 If click, read actual document and can be satisfied 4 If satisfied, stop 5 Otherwise, continue with fixed probability AC IM MdR Click Models for Web Search 52

53 Dynamic Bayesian network model: satisfaction Terminology Satisfaction = a user reads the clicked document and satisfies his/her information need S u binary random variable showing whether document u is satisfactory for query q P(S u = 1) probability of satisfactoriness of document u, P(S u = 0) = 1 P(S u = 1) Dynamic Bayesian network model (DBN) If a user is satisfied, he/she stops Otherwise, continues with fixed probability P(E r = 1 S r 1 = 1) = 0 }{{} if satisfied, stop P(E r = 1 E r 1 = 1, S r 1 = 0) = γ }{{} otherwise, continue AC IM MdR Click Models for Web Search 53

54 Dynamic Bayesian network model: summary E r = 1 and A ur = 1 C r = 1 P(A ur = 1) = α ur q P(E 1 = 1) = 1 P(E r = 1 E r 1 = 0) = 0 P(S r = 1 C r = 1) = σ }{{} ur q if click, can be satisfied P(E r = 1 S r 1 = 1) = 0 }{{} if satisfied, stop P(E r = 1 E r 1 = 1, S r 1 = 0) = γ }{{} otherwise, continue AC IM MdR Click Models for Web Search 54

55 Dynamic Bayesian network: probabilistic graphical model ur 1q ur 1q urq urq A r 1 S r 1 A r S r C r 1 C r... E r 1 document u r 1 E r document u r... AC IM MdR Click Models for Web Search 55

56 Dynamic Bayesian network model: demo Demo AC IM MdR Click Models for Web Search 56

57 Lecture outline 2 Basic click models Random click model CTR models Position-based model Cascade model Dynamic Bayesian network model User browsing model AC IM MdR Click Models for Web Search 57

58 User browsing model Position-based model User browsing model AC IM MdR Click Models for Web Search 58

59 Position-based model γ 1, α u1 q γ 2, α u2 q γ 3, α u3 q γ 4, α u4 q γ 5, α u5 q P(E r = 1 C r = 1, C r +1 = 0,..., C r 1 = 0) = γ rr AC IM MdR Click Models for Web Search 59

60 User browsing model γ 10, α u1 q γ 21, α u2 q γ 31, α u3 q γ 41, α u4 q γ 54, α u5 q P(E r = 1 C r = 1, C r +1 = 0,..., C r 1 = 0) = γ rr AC IM MdR Click Models for Web Search 60

61 User browsing model: summary P(C u = 1) = P(E ru = 1) P(A u = 1) P(A u = 1) = α uq P(E r = 1 C r = 1, C r +1 = 0,..., C r 1 = 0) = γ rr AC IM MdR Click Models for Web Search 61

62 User browsing model: probabilistic graphical model urq document u r A r C r rr 0 E r... AC IM MdR Click Models for Web Search 62

63 Basic click models summary CTR models: counting clicks Position-based model (PBM): examination and attractiveness Cascade model (CM): previous examinations and clicks matter Dynamic Bayesian network model (DBN): satisfactoriness User browsing model (UBM): rank of previous click AC IM MdR Click Models for Web Search 63

64 Probability theory Partitioned probability: A = A 1 A 2, A 1 A 2 = P(A) = P(A 1, A 2 ) = P(A 1 ) + P(A 2 ) Bayes rule P(A B) P(B) = P(B A) P(A) B causes A: B A P(B) = P(B A) P(A) AC IM MdR Click Models for Web Search 64

65 Probability theory (cont d) B A, A = A 1 A 2, A 1 A 2 = P(B) = P(B A) P(A) = P(B A 1, A 2 ) P(A 1, A 2 ) = P(B A 1, A 2 ) (P(A 1 ) + P(A 2 )) = P(B A 1, A 2 ) P(A 1 ) + P(B A 1, A 2 ) P(A 2 ) = P(B A 1 ) P(A 1 ) + P(B A 2 ) P(A 2 ) P(B) = P(B A 1 ) P(A 1 ) + P(B A 2 ) P(A 2 ) AC IM MdR Click Models for Web Search 65

66 Lecture outline 1 Introduction 2 Basic click models 3 Click probabilities AC IM MdR Click Models for Web Search 66

67 Click probabilities Full probability probability that a user clicks on a document at rank r P(C r = 1) Conditional probability probability that a user clicks on a document at rank r given previous clicks P(C r = 1 C 1,..., C r 1 ) AC IM MdR Click Models for Web Search 67

68 Dependency between examination and clicks ru uq E u A u C u document u AC IM MdR Click Models for Web Search 68

69 Full click probability P(C r = 1) = + P(C r = 1 E r = 1) P(E r = 1) P(C r = 1 E r = 0) P(E r = 0) = P(A ur = 1) P(E r = 1) + 0 = α ur qɛ r AC IM MdR Click Models for Web Search 69

70 Cascade models: dependency between examinations ur 1q urq document ur 1 document ur Ar 1 Ar Cr 1 Cr... Er 1 Er... AC IM MdR Click Models for Web Search 70

71 Full click probability P(C r = 1) = P(A ur = 1) P(E r = 1) = α ur qɛ r ɛ r+1 = P(E r+1 = 1) = + P(E r = 1) P(E r+1 = 1 E r = 1) P(E r = 0) P(E r+1 = 1 E r = 0) = ɛ r P(E r+1 = 1 E r = 1) + 0 ( = ɛ r + P(E ) r+1 = 1 E r = 1, C r = 1) P(C r = 1 E r = 1) P(E r+1 = 1 E r = 1, C r = 0) P(C r = 0 E r = 1) AC IM MdR Click Models for Web Search 71

72 Full click probability: Dynamic Bayesian network model Dynamic Bayesian network model: satisfactoriness ur 1q ur 1q urq urq Ar 1 Sr 1 Ar Sr Cr 1 Cr... Er 1 document ur 1 Er document ur... ( P(C r+1 = 1) = α ur+1qɛ r + P(E ) r+1 = 1 E r = 1, C r = 1) P(C r = 1 E r = 1) P(E r+1 = 1 E r = 1, C r = 0) P(C r = 0 E r = 1) P(C r+1 = 1) = α ur+1qɛ r ( + (1 σ u r q)γ α ur q γ (1 α ur q) ) AC IM MdR Click Models for Web Search 72

73 Conditional click probability P(C r = 1 C 1,..., C r 1 ) = P(C r = 1 C <r ) = + P(C r = 1 E r = 1, C <r ) P(E r = 1 C <r ) P(C r = 1 E r = 0, C <r ) P(E r = 0 C <r ) = P(A ur = 1) P(E r = 1 C <r ) + 0 = α ur qɛ r P(E r+1 = 1 E r = 1, C r = 1) c r (s) ɛ r+1 = + P(E r+1 = 1 E r = 1, C r = 0) ɛr (1 α ur q) (1 c r (s) ) 1 α ur qɛ r c (s) r a click on rank r in query session s AC IM MdR Click Models for Web Search 73

74 Click probabilities summary Full probability P(C r+1 = 1) = ( α ur+1qɛ r Conditional probability + P(E r+1 = 1 E r = 1, C r = 1) P(C r = 1 E r = 1) P(E r+1 = 1 E r = 1, C r = 0) P(C r = 0 E r = 1) P(C r+1 = 1 C 1,..., C r ) P(E r+1 = 1 E r = 1, C r = 1) c r (s) = α ur+1q + P(E r+1 = 1 E r = 1, C r = 0) ɛr (1 α ur q) (1 c (s) r ) 1 α ur qɛ r ) AC IM MdR Click Models for Web Search 74

75 Lecture 1 summary CTR models: counting clicks Position-based model (PBM): examination and attractiveness Cascade model (CM): previous examinations and clicks matter Dynamic Bayesian network model (DBN): satisfactoriness User browsing model (UBM): rank of previous click AC IM MdR Click Models for Web Search 75

76 Lecture 1 summary What do click models give us? General Understanding of user behavior Specific Conditional click probabilities Full click probabilities Attractiveness and satisfactoriness for query-document pairs AC IM MdR Click Models for Web Search 76

77 Course overview Basic Click Models Parameter Estimation Evaluation Applications Results Data and Tools Advanced Models Recent Studies Future Research AC IM MdR Click Models for Web Search 77

78 Next lecture Basic Click Models Parameter Estimation Evaluation Applications Results Data and Tools Advanced Models Recent Studies Future Research AC IM MdR Click Models for Web Search 78

79 Acknowledgments All content represents the opinion of the authors which is not necessarily shared or endorsed by their respective employers and/or sponsors. AC IM MdR Click Models for Web Search 79

Advanced Click Models & their Applications to IR

Advanced Click Models & their Applications to IR (Afternoon block 2) Aleksandr Chuklin, Ilya Markov Maarten de Rijke a.chuklin@uva.nl i.markov@uva.nl derijke@uva.nl University of Amsterdam Google Switzerland