CS246: Mining Massive Datasets Jure Leskovec, Stanford University

Transcription

1 CS246: Mnng Massve Datasets Jure Leskovec, Stanford Unversty

2 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 2 Hgh dm. data Graph data Infnte data Machne learnng Apps Localty senstve hashng PageRank, SmRank Flterng data streams SVM Recommen der systems Clusterng Communty Detecton Web advertsng Decson Trees Assocaton Rules Dmensonal ty reducton Spam Detecton Queres on streams Perceptron, knn Duplcate document detecton

3 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 3 Gven some data: Learn a functon to map from the nput to the output Gven: Tranng examples! ", $ = &! " unknown functon & Fnd: A good approxmaton to & for some

4 2/19/18 Jure Leskovec, Pnterest Machne Learnng Class, Wnter Supervsed: Gven labeled data {x,y}, learn f(x)=y Unsupervsed: Gven only unlabeled data {x}, learn f(x) Semsupervsed: Gven some labeled and some unlabeled data Actve learnng: Whenever we predct f(x)=y, we then receve true y * Transfer learnng: Learn f(x) so that t works well on new doman f(z)

5 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 5 Would lke to do predcton: estmate a functon f(x) so that y = f(x) Where y can be: Real number: Regresson Categorcal: Classfcaton Complex object: Rankng of tems, Parse tree, etc. Data s labeled: Have many pars {(x, y)} x vector of bnary, categorcal, real valued features y class: {1, 1}, or a real number

6 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 6 Task: Gven data (X,Y) buld a model f() to predct Y based on X Strategy: Estmate! = $ % on (', )). Hope that the same $(%) also works to predct unknown ) The hope s called generalzaton Tranng data Test data Overfttng: If f(x) predcts well Y but s unable to predct Y We want to buld a model that generalzes well to unseen data X X Y Y

7 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 7 tranng ponts 1) Tranng data s drawn ndependently at random accordng to unknown probablty dstrbuton!(#, %) 2) The learnng algorthm analyzes the examples and produces a classfer ' Gven new data #, % drawn from (, the classfer s gven # and predcts * = '(#) The loss (*, *) s then measured Goal of the learnng algorthm: Fnd ' that mnmzes expected loss. ( []

8 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 8 test data!(#, %) (#, %) tranng data % # Tranng set ' Learnng algorthm ( Why s t hard? We estmate on tranng data but want the to work well on unseen future (.e., test) data %) % loss functon L(%), %)

9 Goal: Mnmze the expected loss mn & ' [)] $ But, we don t have access to but only to tranng sample,: mn & [)] $ So, we mnmze the average loss on the tranng data: 2/19/18 mn. / = 1 $ 2 3 L /(6 7), : 7 ; 7<= Problem: Just memorzng the tranng data gves us a perfect model (wth zero loss) Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 9

10 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 10 Gven: A set of N tranng examples {(# $, & $ ), (# (, & ( ),, (# *, & * )} A loss functon, Choose:. / =. / 3 Fnd: The weght vector 4 that mnmzes the expected loss on the tranng data = 5 6 = L 4 # ; <, & ; ;>$

11 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 11 Problem: Stepwse Constant Loss functon Loss f w (x) Dervatve s ether 0 or

12 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 12 Approxmatng the expected loss by a smooth functon Replace the orgnal objectve functon by a surrogate loss functon. E.g., hnge loss: 5 %& ' = 1 ( ) max 0, 1! 0 1(3 0 ) 067 When! = 1:

13 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 13 Example: Spam flterng Instance space x Î X ( X = n data ponts) Bnary or realvalued feature vector x of word occurrences d features (words other thngs, d~100,000) Class y Î Y y: Spam (1), Ham (1)

14 !(#, %): dstrbuton of emal messages # and ther true labels % ( spam, ham ) Tranng sample: a set of emal messages that have been labeled by the user Learnng algorthm: What we study! ': The classfer output by the learnng alg. Test pont: A new emal # (wth ts true, but hdden, label %) Loss functon (()*, )): 2/19/18 predcted label % true label % spam ham spam 0 10 not spam 1 0 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 14

15 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 15 We wll talk about the followng methods: Support Vector Machnes Decson trees Man queston: How to effcently tran (buld a model/fnd model parameters)?

16

17 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 17 Want to separate from usng a lne Data: Tranng examples: (x 1, y 1 ) (x n, y n ) Each example : x = ( x (1),, x (d) ) x (j) s real valued y Î { 1, 1 } Inner product:,! # = & (() (() (. Whch s best lnear separator (defned by w,b)?

18 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 18 A B C Dstance from the separatng hyperplane corresponds to the confdence of predcton Example: We are more sure about the class of A and B than of C

19 Margn!: Dstance of closest example from the decson lne/hyperplane The reason we defne margn ths way s due to theoretcal convenence and exstence of generalzaton error bounds that depend on the value of margn. 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 19

20 Remember: The Dot product! # =! # %&' (! 234(! = *! (,). 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 20 /,01

21 Dot product! # =! # %&' ( What s ) *, ) *,? x 2 x 1 ) w x b = 0 x 2 x 1 x 2 x 1 ) ) In ths case ), In ths case,, ), So, roughly corresponds to the margn Bottom lne: Bgger bgger the separaton 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 21

22 Dstance from a pont to a lne A (x A (1), x A (2) ) d(a, L) H w L w x b = 0 Let: Lne L: w xb = w (1) x (1) w (2) x (2) b=0 w = (w (1), w (2) ) Pont A = (x A (1), x A (2) ) Note we assume! " = $ Pont M on a lne = (x M (1), x M (2) ) (0,0) M (x M (1), x M (2) ) d(a, L) = AH = (AM) w = (x A (1) x M (1) ) w (1) (x A (2) x M (2) ) w (2) = x A (1) w (1) x A (2) w (2) b = w A b Remember x M (1) w (1) x M (2) w (2) = b snce M belongs to lne L 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 22

23 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 23 % w x b = 0 Predcton = sgn(w x b) Confdence = (w x b) y For thdatapont:! " = % & " ( ) " Want to solve: *, *./! " %,( " Can rewrte as maxg w, g s. t. ", y ( w x b) ³ g

24 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 24 Maxmze the margn: Good accordng to ntuton, theory (c.f. VC dmenson ) and practce max g w, g s. t. ", y ( w x b) ³ g g g g w xb=0! s margn dstance from the separatng hyperplane Maxmzng the margn

25

26 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 26 Separatng hyperplane s defned by the support vectors Ponts on / planes from the soluton If you knew these ponts, you could gnore the rest Generally, d1 support vectors (for d dm. data)

27 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 27 Problem: Let! " $ % = ' then (! " ($ % = (' Scalng w ncreases margn! Soluton: x 2 w xb=1 x 1 w xb=0 w xb=1 Work wth normalzed w: ' =!! " $ % w w Let s also requre support vectors " ) to be on the plane defned by:! " ) $ = ±, 5 w = / 0 (2) 4 267

28 Want to maxmze margn! What s the relaton between x 1 and x 2?! " =! $ $& ' ' We also know: '! " * = " '! $ * = " So: '! " * = " '! $ $& ' ' * = " '! $ * $& ' ' = " ' 1 Þ g = w xb=1 w xb=0 w xb=1 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 28 x 2 w w 2g x 1 w 1 = w w w Note: w w = w 2

29 We started wth g max, w g s. t. ", y ( w x b) arg maxg = arg max mn w 1 w 1 2 w s. t. ", y ( w x b) 2 = arg mn w ³ arg mn 1 2 g But w can be arbtrarly large! We normalzed and... Then: ³ 1 Ths s called SVM wth hard constrants = w 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, x 2 w w 2g w xb=1 x 1 w xb=0 w xb=1

30 If data s not separable ntroduce penalty: mn w 1 2 s. t. ", y w 2 ( w x C (#number of b) ³ 1 Mnmze ǁwǁ 2 plus the number of tranng mstakes Set C usng cross valdaton How to penalze mstakes? All mstakes are not equally bad! mstakes) 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 30 w xb=0

31 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 31 Introduce slack varables x mn w, b, x ³ 0 s. t. ", 1 2 y w 2 ( w x If pont x s on the wrong sde of the margn then get penalty x n C åx b) = 1 ³ 1x x x j w xb=0 For each data pont: If margn ³ 1, don t care If margn < 1, pay lnear penalty

32 mn w s. t. ", 1 2 y w 2 ( w x C (#number of b) ³ 1 What s the role of slack penalty C: C= : Only want to w, b that separate the data C=0: Can set x to anythng, then w=0 (bascally gnores the data) (0,0) mstakes) 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 32 small C ḇg C good C

33 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 33 SVM n the natural form arg mn w, b 1 2 w w C å SVM uses Hnge Loss : 0/1 loss n max { 0,1 y ( w x b) } Margn = 1 Emprcal loss L (how well we ft tranng data) Regularzaton parameter penalty mn w, b 1 2 s. t. ", y w 2 ( w x n Cåx = 1 b) ³ 1x Hnge loss: max{0, 1z} z = y ( x w b)

34

35 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 35 mn w, b 1 2 s. t. ", y ( x w Want to estmate! and "! Standard way: Use a solver! Solver: software for fndng solutons to common optmzaton problems Use a quadratc solver: Mnmze quadratc functon Subject to lnear constrants Problem: Solvers are neffcent for bg data! n w w C å = 1 b) x ³ 1 x

36 Want to mnmze J(w,b): Compute the gradent Ñ(j) w.r.t. w (j) 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 36 å = = = Ñ n j j j j w y x L C w w w b L J 1 ) ( ) ( ) ( ) ( ), ( ), ( else 1 ) (w f 0 ), ( ) ( ) ( j j x y b x y w y x L = ³ = å( ) å å = = = þ ýü î í ì = n d j j j d j j b x w y C w w b J 1 1 ) ( ) ( 1 2 ) ( 2 1 ) ( 0,1 max ), ( Emprcal loss!(# $ & $ )

37 2/20/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 37 Gradent descent: Iterate untl convergence: For j = 1 d Evaluate: j) f ( w, b) ÑJ = ( j) Update: w w (j) w (j) hñj (j) w w Problem: Computng ÑJ (j) takes O(n) tme! n sze of the tranng dataset n ( ( j) L( x = å, y ) w C ( j) = 1 w h learnng rate parameter C regularzaton parameter

38 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 38 Stochastc Gradent Descent Instead of evaluatng gradent over all examples evaluate t for each ndvdual tranng example j) ( j) L( x, y ÑJ ( x ) = w C ( j) w Stochastc gradent descent: ( ) Iterate untl convergence: For = 1 n For j = 1 d Compute: ÑJ (j) (x ) Update: w (j) w (j) h ÑJ (j) (x ) ÑJ We just had: n ( j) ( j) L( x, y ) = w Cå ( j) = 1 w Notce: no summaton over anymore

39

40 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 40 Example by Leon Bottou: Reuters RCV1 document corpus Predct a category of a document One vs. the rest classfcaton n = 781,000 tranng examples (documents) 23,000 test examples d = 50,000 features One feature per word Remove stopwords Remove low frequency words

41 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 41 Questons: (1) Is SGD successful at mnmzng J(w,b)? (2) How quckly does SGD fnd the mn of J(w,b)? (3) What s the error on a test set? Standard SVM Fast SVM SGDSVM Tranng tme Value of J(w,b) Test error (1) SGDSVM s successful at mnmzng the value of J(w,b) (2) SGDSVM s super fast (3) SGDSVM test set error s comparable

42 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 42 SGD SVM Conventonal SVM Optmzaton qualty: J(w,b) J (w opt,b opt ) For optmzng J(w,b) wthn reasonable qualty SGDSVM s super fast

43 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 43 Need to choose learnng rate h and t 0 ht æ L( x, y ) ö wt 1 wt ç wt C t t0 è w ø Leon suggests: Choose t 0 so that the expected ntal updates are comparable wth the expected sze of the weghts Choose h: Select a small subsample Try varous rates h (e.g., 10, 1, 0.1, 0.01, ) Pck the one that most reduces the cost Use h for next 100k teratons on the full dataset

44 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 44 Sparse Lnear SVM: Feature vector x s sparse (contans many zeros) Do not do: x = [0,0,0,1,0,0,0,0,5,0,0,0,0,0,0, ] But represent x as a sparse vector x =[(4,1), (9,5), ] Can we do the SGD update more effcently? æ w w hç w C è Approxmated n 2 steps: w L( x, y ) w hc w w w( 1h) L( x, y w ) ö ø cheap: x s sparse and so few coordnates j of w wll be updated expensve: w s not sparse, all coordnates need to be updated

45 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 45 Soluton 1:! = # % Represent vector w as the product of scalar s and vector v Then the update procedure s: Two step update procedure: (1) (2) L( x, y ) w w hc w w w( 1h) (1) % = % () *,,/ *! (2) # = #(1 () Soluton 2: Perform only step (1) for each tranng example Perform step (2) wth lower frequency and hgher h

46 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 46 Stoppng crtera: How many teratons of SGD? Early stoppng wth cross valdaton Create a valdaton set Montor cost functon on the valdaton set Stop when loss stops decreasng Early stoppng Extract two (very) small sets of tranng data A and B Tran on A, stop by valdatng on B Number of tranng epochs on A s an estmate of k Tran for k epochs on the full dataset

47 Idea 1: One aganst all Learn 3 classfers vs. {o, } vs. {o, } o vs. {, } Obtan: w b, w b, w o b o How to classfy? Return class c arg max c w c x b c 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 47

48 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 48 Idea 2: Learn 3 sets of weghts smoultaneously! For each class c estmate w c, b c Want the correct class y to have hghest margn: w y x b y ³ 1 w c x b c "c ¹ y, " (x, y )

49 Optmzaton problem: To obtan parameters w c, b c (for each class c) we can use smlar technques as for 2 class SVM SVM s wdely perceved a very powerful learnng algorthm 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 49 c c y y n c w b b x w b x w C w x x ³ å å = 1 mn 1 c 2 2 1, y c " ³ " ¹ " 0,, x

50

51 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 51 New settng: Onlne Learnng Allows for modelng problems where we have a contnuous stream of data We want an algorthm to learn from t and slowly adapt to the changes n data Idea: Do slow updates to the model SGDSVM makes updates f msclassfyng a datapont So: Frst tran the classfer on tranng data. Then for every example from the stream, f we msclassfy, update the model (usng a small learnng rate)

52 Protocol: User comes and tell us orgn and destnaton We offer to shp the package for some money ($10 $50) Based on the prce we offer, sometmes the user uses our servce (y = 1), sometmes they don't (y = 1) Task: Buld an algorthm to optmze what prce we offer to the users Features x capture: Informaton about user Orgn and destnaton Problem: Wll user accept the prce? 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 52

53 Model whether user wll accept our prce:! = $(&; () Accept: y =1, Not accept: y=1 Buld ths model wth say Perceptron or SVM The webste that runs contnuously Onlne learnng algorthm would do somethng lke User comes User s represented as an (x,y) par where x: Feature vector ncludng prce we offer, orgn, destnaton y: If they chose to use our servce or not The algorthm updates w usng just the (x,y) par Bascally, we update the w parameters every tme we get some new data 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 53

54 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 54 We dscard ths dea of a data set Instead we have a contnuous stream of data Further comments: For a major webste where you have a massve stream of data then ths knd of algorthm s pretty reasonable Don t need to deal wth all the tranng data If you had a small number of users you could save ther data and then run a normal algorthm on the full dataset Dong multple passes over the data

55 2/19/18 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, 55 An onlne algorthm can adapt to changng user preferences For example, over tme users may become more prce senstve The algorthm adapts and learns ths So the system s dynamc