Anomaly Detection. Davide Mottin, Konstantina Lazaridou. HassoPlattner Institute. Graph Mining course Winter Semester 2016

Transcription

1 Anomaly Detection Davide Mottin, Konstantina Lazaridou HassoPlattner Institute Graph Mining course Winter Semester 2016

2 and Next week February 7, third round presentations Slides are due by February 6, You are invited to the KDD Open Day: February 15, starting at 4pm in building E, first floor. Please fill the course/lecturers evaluation form online We are looking for enthusiasts and motivated students for theses in data mining, database, graph mining Current graph mining theses include (but are not limited to) Graph database reusability Personalized graph summarization Supervised Graph Reconstruction If you are interested send an to 2

3 Acknowledgements Some part of this lecture is taken from: 3

4 Lecture road Feature based approaches Matrix factorization Matrix factorization 4

5 Anomaly detection An outlier is an observation that differs so much from the other observations as to arouse suspicion that it was generated by a different mechanism (Hawkins Definition of Outlier, 1980) No unique definition Context dependent 5

6 Anomalies Rare (e.g., rare combination of categorical attribute values) Isolated points in n-d spaces Surprising (don't fit well in our mental/statistical model == need too many bits under MDL) 6

7 The study of anomalies in graphs Unlabeled/Labeled (Attributed) Graphs Static/Dynamic Graphs Un-/Semi-/- Supervised Graph Techniques 7

8 Anomalies in Weighted Graphs Can we detect nodes that are different from the others? Can we explain why? Anomalies 8

9 Problem Sketch Embed into multidimensional space and analyze the points 9

10 OddBall: Approach 1. For each node, 1. Extract ego-net (=1-step neighborhood) 2. Extract features (#edges, total weight, etc.) features that could yield laws features fast to compute and interpret 2. Detect patterns: regularities 3. Detect anomalies: distance to patterns Akoglu, L., McGlohon, M. and Faloutsos, C.. Oddball: Spotting anomalies in weighted graphs. PAKDD,

11 Which Features? Ego-net features: N " : Number of neighbors of ego-net i E " : Number of edges in ego-net W " : Total weight of ego-net They follow power laws!!! E " N " ', 1 < α < 2 W " E "., β 1 λ " W " 1, 0.5 γ 1 λ " : Principal eigenvalue of the weighted adjacency matrix of the ego-net 11

12 Plotting features Fitting line (power law family) 12

13 Anomaly detection score out-line(i)= ;<= B ;CD B log y " Cx " Distance from fitting line Cx 9 : power law fitting line for a feature pair (x,y) The fitting line is experimentally found plotting pairs of features in a log-log plot Quantify the distance of x " from the fitting line Cx " 9 13

14 OddBall: anomaly detection 14

15 Lecture road Feature based approaches Matrix factorization Matrix factorization 15

16 Finding patterns with matrix factorization A Typical Procedure: Low-rank matrices Residual matrix Graph Adj. Matrix A A = F x G + R community anomalies An Illustrative Example Tong, H. and Lin, C.Y. Non-Negative Residual Matrix Factorization with Application to Graph Anomaly Detection. In SDM,

17 Improve Interpretation by Non-negativity A Typical Procedure: Graph Adjacency Matrix A community A = F x G + R anomalies Interpretation by Non-negativity Non-negative Matrix Factorization F >= 0; G >= 0 (for community detection) An Example Non-negative Residual Matrix Factorization R(i,j) >= 0; for A(i,j) > 0 (for anomaly detection) This Paper 17

18 Optimization Formulation Non-negative residuals => Construct a residual graph Weighted Frobenius Form Common in Any Matrix Factorization Weight Unique for this technique Non-negative residual 0/1 weight Common in Any Matrix Factorization Unique for this technique Non-negative residual 18

19 In the next episode Student Presentation Survey of other graph algorithms And not much more 19

20 Questions? 20

21 References Akoglu, L., McGlohon, M. and Faloutsos, C.. Oddball: Spotting anomalies in weighted graphs. PAKDD, Tong, H. and Lin, C.Y. Non-Negative Residual Matrix Factorization with Application to Graph Anomaly Detection. In SDM, Xing, E.P., Ng, A.Y., Jordan, M.I. and Russell, S. Distance metric learning with application to clustering with side-information. In NIPS,