Intrusion Detection and Malware Analysis

Transcription

1 Intrusion Detection and Malware Analysis IDS feature extraction Pavel Laskov Wilhelm Schickard Institute for Computer Science

2 Metric embedding of byte sequences Sequences 1. blabla blubla blablabu aa 2. bla blablaa bulab bb abla 3. a blabla blabla ablub bla 4. blab blab abba blabla blu Subsequences Histograms of subsequences Geometry 2 3 Features blablabu blablaa blablu blabla bulab ablub blab abla abba blu bla bb aa b a 1 4

3 Formalization of embedding A sequence x from an alphabet Σ of cardinality N: x Σ A language L of pre-defined words: Example languages: n-grams bag-of-words L Σ = {w w Σ } all subsequences bag-of-delimiters Embedding function defined over the language: frequency φ w (x) : count for w in x binary flag

4 Similarity measures for embeddings Metric embedding enables application of various vectorial similarity measures over sequences, e.g. Kernels k(x, y) Distances d(x, y) Linear RBF φ w (x)φ w (y) w L exp(d(x, y) 2 /σ) Manhattan Minkowski φ w (x) φ w (y) w L k w L φ w (x) φ w (y) k Similarity coefficients Jaccard, Kulczynski,... Hamming Chebyshev sgn φ w (x) φ w (y) w L max φ w(x) φ w (y) w L

5 Abstract similarity measure Outer loop: s(x, y) = w L m(x, y, w) Inner function: m + (φ w (x), φ w (y)), m(x, y, w) = mx (φ w (x)), my (φ w (y)), if w matches x,y if w mismatches x if w mismatches y

6 Inner function computation m + (p, q) mx (p) my (q) Kernel functions Linear p q 0 0 Distances Manhattan p q p q Minkowski (p q) k p k q k Chebyshev max p q p q

7 Data structures: an overview How should we store subsequences to ensure linear-time extraction and matching?

8 Data structures: an overview How should we store subsequences to ensure linear-time extraction and matching? Hash tables: simple and relatively efficient; limited embeddings, hash table size difficult to choose. Sorted arrays: simple and highly efficient (contiguous storage!); limited embeddings Tries: moderately complex and efficient; limited embeddings. Suffix trees: unlimited embeddings; very complex, high constants and memory consumption.

9 Sorted array representation Rieck and Laskov Extract subsequences and store them in an array Sort the array value phi[x]. The length of an array X is denoted by X. In order to support effi comparison, For theany fields pair of of Xsequences, are sorted byfind contained matching words, and e.g. mismatching using the lexicogra order of thentries alphabet by A. looping Figureover 1 illustrates sorted arrays. the sorted arrays of 3-grams extracted the two example Example: sequences x = abbaa, x and y. y = baaaab X word[x] phi[x] abb 1 baa 1 bba 1 Y aaa 2 aab 1 baa 1 Figure 1: Sorted arrays of 3-grams for x = abbaa and y = baaaab. The number in field indicates the number of occurrences.

10 How to sort (sub-)sequences?

11 Radix sort at byte level Simple, linear running time How to sort (sub-)sequences?

12 How to sort (sub-)sequences? Radix sort at byte level Simple, linear running time Store subsequences in machine words, use numeric sorting Simple, superlinear running time, extremely low constants

13 How to sort (sub-)sequences? Radix sort at byte level Simple, linear running time Store subsequences in machine words, use numeric sorting Simple, superlinear running time, extremely low constants Ditto, use radix sorting at bit-level

14 Suffix tree: a definition A suffix tree for an m-character string S stores all suffixes of S. S = ababc$ ab b c$ 5 abc$ c$ abc$ c$

15 Properties of suffix trees A suffix tree has exactly m leaves numbered 1 to m. Each internal node has at least two children. Each edge is labeled by a non-empty substring of S. All edges of the same node begin with different symbols. For any leaf i, the concatenation of the labels on the path from root to i is the suffix of S starting at position i, i.e. S[i..m].

16 What are suffix trees good for? Problem: Given a string S of length n and a pattern p of length m, m n, find positions of all occurrences of P in S. Classical solution: O(m + n) (e.g. Knuth-Morris-Pratt) Suffix tree solution: O(m) S = ababc$, P = ab ab b c$ 5 abc$ c$ abc$ c$

17 Compact suffix tree storage Labels are replaced by index ranges. Internal nodes contain depth and leaf counts. Suffix links point to subtrees corresponding to the next suffix. S = ababc$ [1, 2] [2, 2] [5, e] [3, e] [5, e] [3, e] [5, e]

18 Chang & Lawler Algorithm: an example Given a suffix tree for S, we can count matching substrings in S and P by walking along P and S: S = ababc$, P = baaaba ab b c$ abc$ c$ abc$ c$

19 Chang & Lawler Algorithm: an example Given a suffix tree for S, we can count matching substrings in S and P by walking along P and S: scan b S = ababc$, P = baaaba ab c$ b abc$ c$ abc$ c$

20 Chang & Lawler Algorithm: an example Given a suffix tree for S, we can count matching substrings in S and P by walking along P and S: scan a : MATCH, count 1 S = ababc$, P = baaaba ab b c$ abc$ c$ abc$ c$

21 Chang & Lawler Algorithm: an example Given a suffix tree for S, we can count matching substrings in S and P by walking along P and S: scan a : MISMATCH S = ababc$, P = baaaba ab b c$ abc$ c$ abc$ c$

22 Chang & Lawler Algorithm: an example Given a suffix tree for S, we can count matching substrings in S and P by walking along P and S: scan a : MISMATCH S = ababc$, P = baaaba ab b c$ abc$ c$ abc$ c$

23 Chang & Lawler Algorithm: an example Given a suffix tree for S, we can count matching substrings in S and P by walking along P and S: scan b : MATCH, count 2 S = ababc$, P = baaaba ab b c$ abc$ c$ abc$ c$

24 Chang & Lawler Algorithm: an example Given a suffix tree for S, we can count matching substrings in S and P by walking along P and S: scan a : MATCH, count 1 S = ababc$, P = baaaba ab b c$ abc$ c$ abc$ c$

25 Generalized suffix tree (GST) A suffix tree for more than one string. Creation: concatenate two strings with different delimiters and build a single suffix tree Example: GST for x = abbaa and y = baaaa : 6 6 a # $ b a # $ bbaa# aa baa# a # $ aa$ # a$ $

26 Similarity computation using GST 2-grams abbaa baaaa abbaa baaaa = 0 aa ab ba bb 6 6 a # $ b a # $ bbaa# aa baa# a # $ aa$ # a$ $

27 Similarity computation using GST 2-grams abbaa baaaa abbaa baaaa = 3 aa 1 3 ab ba bb 6 6 a # $ b a # $ bbaa# aa baa# a # $ aa$ # a$ $

28 Similarity computation using GST 2-grams abbaa baaaa abbaa baaaa = 3 aa 1 3 ab 1 0 ba bb 6 6 a # $ b a # $ bbaa# aa baa# a # $ aa$ # a$ $

29 Similarity computation using GST 2-grams abbaa baaaa abbaa baaaa = 4 aa 1 3 ab 1 0 ba 1 1 bb 6 6 a # $ b a # $ bbaa# aa baa# a # $ aa$ # a$ $

30 Similarity computation using GST 2-grams abbaa baaaa abbaa baaaa = 4 aa 1 3 ab 1 0 ba 1 1 bb a # $ b a # $ bbaa# aa baa# a # $ aa$ # a$ $

31 Lessons learned Extraction of features from packet payloads is tricky but can be efficiently done with specialized data structures. In practice, sorted arrays work best for computation of similarity measures Suffix trees are the most powerful data structure for feature extraction: will be used for other problems.

32 Recommended reading D. Gusfield. Algorithms on strings, trees, and sequences. Cambridge University Press, K. Rieck and P. Laskov. Linear-time computation of similarity measures for sequential data. Journal of Machine Learning Research, 9(Jan):23 48, E. Ukkonen. Online construction of suffix trees. Algorithmica, 14(3): , 1995.