Adding rigor to the comparison of anomaly detector outputs

Size: px

Start display at page:

Download "Adding rigor to the comparison of anomaly detector outputs"

Roxanne Flynn
5 years ago
Views:

1 Adding rigor to the comparison of anomaly detector outputs Romain Fontugne, National Institute of Informatics / SOKENDAI, Tokyo Pierre Borgnat, Physics Lab, CNRS, ENS Lyon Patrice Abry, Physics Lab, CNRS, ENS Lyon Kensuke Fukuda, National Institute of Informatics / PRESTO JST, Tokyo April 25, 2010 Adding rigor to the comparison of anomaly detector outputs, Fontugne, Borgnat, Abry, Fukuda 1

2 Motivation Anomaly detection in backbone traffic Active research domain Wavelet [IMC 02], PCA [SIGCOMM 05, SIGMETRICS 07], gamma law [LSAD 07], association rule [IMC 09]... Tricky evaluation, lack of common ground truth: Manual inspection Synthetic traffic Comparison with other methods Similar problems arise in traffic classification Adding rigor to the comparison of anomaly detector outputs, Fontugne, Borgnat, Abry, Fukuda 2

3 Goal Long term goal: Provide common ground truth data Labeling MAWI archive Combining several anomaly detector results Ground truth relative to the state of the art Goal of this work: Find relations between outputs of different classifiers Adding rigor to the comparison of anomaly detector outputs, Fontugne, Borgnat, Abry, Fukuda 3

Problem statement: Eventx=Eventy?? Event (= anomaly detector s alarm) Set of traffic feature containing at least 2 timestamps and one traffic feature. i.e. one flow, one IP address, a set of flows, a set of packets.

4 Problem statement: Eventx=Eventy?? Event (= anomaly detector s alarm) Set of traffic feature containing at least 2 timestamps and one traffic feature. i.e. one flow, one IP address, a set of flows, a set of packets... Main difficulties Different granularities: Event1=Event2?=Event3? Overlapping: Event4=Event5? Different points of view: Event1=Event6? Adding rigor to the comparison of anomaly detector outputs, Fontugne, Borgnat, Abry, Fukuda 4

Proposed method Approach Identify similar events by using community mining on graph Overview Oracle: Uncover relations between traffic and events Graph gen.

5 Proposed method Approach Identify similar events by using community mining on graph Overview Oracle: Uncover relations between traffic and events Graph gen.: Represent events and their relations in a graph Community Mining: Find similar events by looking at dense components Adding rigor to the comparison of anomaly detector outputs, Fontugne, Borgnat, Abry, Fukuda 5

6 Oracle Uncover relations between original traffic and events List the events that match each packet of the original traffic i.e. pkt1:{ip1 : 80 IP2 : 12345} = Event1:{srcIP = IP1} Adding rigor to the comparison of anomaly detector outputs, Fontugne, Borgnat, Abry, Fukuda 6

Graph generator Build a non-directed weighted graph from the Oracle output Nodes are events and edges are shared packets Weight on each edge: similarity measure,

7 Graph generator Build a non-directed weighted graph from the Oracle output Nodes are events and edges are shared packets Weight on each edge: similarity measure, Simpson index, E 1 E 2 / min( E 1, E 2 ), E i : packets matching event i Adding rigor to the comparison of anomaly detector outputs, Fontugne, Borgnat, Abry, Fukuda 7

8 Community mining Identify community (= dense component) in the graph Louvain algorithm 1 : based on Modularity 2 Take into account node connectivity and edge weight 1 Blondel et al.: Fast unfolding of communities in large networks. J.STAT.MECH. (2008) 2 Newman, Girvan: Finding and evaluating community structure in networks. Phys. Rev.E (Feb 2004) Adding rigor to the comparison of anomaly detector outputs, Fontugne, Borgnat, Abry, Fukuda 8

9 Data and anomaly detectors Data set MAWI archive (trans-pacific link) During the outbreak of the Sasser worm (08/2004) Anomaly detectors Sketches and multiresolution gamma modeling 3 Report source or destination IP Image processing: Hough transform 4 Report set of packets 3 Dewaele, G., Fukuda, K., Borgnat, P., Abry, P., Cho, K.: Extracting hidden anomalies using sketch and non gaussian multiresolution statistical detection procedures. SIGCOMM LSAD 07 4 Fontugne, R., Himura, Y., Fukuda, K.: Evaluation of anomaly detection method based on pattern recognition. IEICE Trans. on Commun. E93-B(2) (February 2010) Adding rigor to the comparison of anomaly detector outputs, Fontugne, Borgnat, Abry, Fukuda 9

10 Results Graph Reported events; Gamma-based: 332, Hough-based: 873 Intersection 235 and 247 events: 124 connected components Biggest component: 47 events (G.34, H.13), 8 communities d;142055pkt s;142054pkt 1;142054pkt s;149836pkt ;142054pkt d;87904pkt 1;87904pkt s;32795pkt e-05;1pkt s;32794pkt e-05;1pkt 1;142054pkt 1;87904pkt 1;87904pkt d;67971pkt s;10616pkt 1;10616pkt s;71331pkt ;67945pkt ;10610pkt s;5098pkt d;80692pkt 1;5098pkt s;64299pkt 1;5098pkt d;102053pkt 1;5098pkt ;50961pkt s;37450pkt ;29126pkt ;80391pkt 1;64299pkt 1;37450pkt d;885pkt s;843pkt ;1pkt s;502pkt ;71pkt s;504pkt ;115pkt s;2963pkt ;1pkt 1;843pkt d;1830pkt s;940pkt ;356pkt s;7985pkt ;1751pkt s;1016pkt ;2pkt s;3307pkt ;2pkt d;635pkt ;8pkt ;706pkt d;595pkt ;1pkt ;132pkt s;860pkt d;2857pkt ;4pkt s;8083pkt ;856pkt s;384pkt ;83pkt s;6575pkt ;877pkt d;171pkt 1;171pkt d;247pkt s;386pkt 1;247pkt s;159pkt ;55pkt ;55pkt s;2777pkt d;10965pkt ;1287pkt s;12282pkt ;1912pkt s;11029pkt ;2391pkt s;2611pkt ;507pkt s;6569pkt ;1681pkt d;10835pkt ;2197pkt 1;10835pkt s;261435pkt d;3276pkt 1;3276pkt d;9815pkt 1;9815pkt d;2783pkt 1;2783pkt d;11798pkt 1;11798pkt d;4047pkt 1;4047pkt d;1442pkt 1;1442pkt s;42747pkt ;40566pkt d;20749pkt ;13040pkt ;7924pkt s;348pkt d;3769pkt 1;348pkt s;1093pkt d;1639pkt 1;1093pkt s;247pkt ;1pkt d;5933pkt s;33509pkt ;2153pkt s;21160pkt ;5910pkt d;99pkt 1;99pkt d;792pkt ;374pkt d;2380pkt ;780pkt d;994pkt ;374pkt d;297pkt 1;297pkt d;253pkt ;216pkt d;6644pkt ;1680pkt d;264pkt ;187pkt d;612pkt ;188pkt d;406pkt ;157pkt d;304pkt ;156pkt d;559pkt ;198pkt d;380pkt ;212pkt d;260pkt ;113pkt d;273pkt ;96pkt d;399pkt ;109pkt d;319pkt ;88pkt d;632pkt ;250pkt d;347pkt ;146pkt d;288pkt ;142pkt d;277pkt ;98pkt d;583pkt ;186pkt d;470pkt ;147pkt d;410pkt ;119pkt d;288pkt ;79pkt d;265pkt ;80pkt ;5544pkt 1;612pkt 1;406pkt 1;260pkt 1;277pkt 1;99pkt s;956pkt d;917pkt ;13pkt s;5210pkt ;870pkt d;356pkt ;21pkt ;44pkt s;5038pkt ;53pkt s;67012pkt ;136pkt d;185pkt ;71pkt s;450pkt d;653pkt ;154pkt d;93713pkt s;93713pkt 1;93713pkt s;98741pkt 1;93713pkt 1;93713pkt s;658pkt d;442pkt ;158pkt ;1pkt s;1287pkt ;48pkt s;273pkt d;4559pkt 1;273pkt s;465pkt ;26pkt ;363pkt ;156pkt ;169pkt s;42184pkt ;11pkt ;114pkt ;13pkt d;573pkt ;6pkt d;727pkt ;4pkt ;2pkt ;105pkt s;405pkt ;3pkt ;2pkt ;16pkt s;554pkt ;5pkt s;2515pkt ;9pkt ;2pkt ;13pkt s;1099pkt d;352pkt ;12pkt s;62pkt ;42pkt s;1079pkt d;3683pkt ;429pkt d;475pkt ;5pkt s;10212pkt ;867pkt ;3583pkt s;17701pkt ;1pkt ;27pkt ;12pkt d;1058pkt s;15083pkt ;3pkt s;30818pkt ;3pkt s;30461pkt ;3pkt s;15063pkt ;3pkt 1;15063pkt d;482pkt 1;482pkt d;1218pkt 1;1218pkt d;2305pkt 1;2305pkt d;2555pkt 1;2555pkt d;4687pkt 1;4687pkt d;286pkt ;136pkt s;5036pkt 1;5036pkt d;341pkt ;171pkt 1;4687pkt d;544pkt s;244pkt ;16pkt d;392pkt ;9pkt s;695pkt d;1185pkt ;1pkt s;551pkt ;1pkt d;1pkt 1;1pkt d;718pkt ;1pkt ;1pkt 1;32794pkt d;63363pkt s;63418pkt 1;63363pkt s;66456pkt ;62944pkt ;62944pkt d;349pkt s;8813pkt ;220pkt d;258pkt ;252pkt d;3099pkt ;78pkt d;1117pkt ;454pkt s;1522pkt 1;1522pkt d;2551pkt 1;2551pkt ;5pkt d;259pkt 1;259pkt d;1074pkt s;707pkt ;78pkt s;811pkt ;9pkt d;8097pkt ;16pkt d;4805pkt ;23pkt d;2345pkt s;374pkt ;369pkt s;301pkt 1;301pkt s;75pkt 1;75pkt s;1175pkt ;567pkt d;1760pkt s;900pkt ;442pkt s;430pkt ;8pkt s;1089pkt d;424pkt ;58pkt d;347pkt ;109pkt d;4862pkt ;64pkt d;843pkt 1;843pkt d;1097pkt ;1081pkt ;16pkt d;250pkt 1;250pkt d;11896pkt 1;11896pkt s;23207pkt ;10079pkt d;13312pkt ;647pkt ;143pkt ;22177pkt 1;13312pkt ;10488pkt d;1950pkt 0.3;129pkt s;216pkt ;155pkt s;124pkt ;41pkt s;320pkt ;51pkt s;40pkt 1;40pkt s;527pkt ;103pkt s;83pkt ;41pkt s;171pkt ;121pkt s;73pkt ;40pkt d;39pkt 1;39pkt 1;39pkt d;668pkt s;313pkt 1;313pkt d;388pkt 1;388pkt 1;313pkt s;334pkt s;173pkt 1;173pkt d;2687pkt s;2687pkt 1;2687pkt s;253pkt 1;253pkt ;70pkt ;44pkt s;2294pkt 1;2294pkt s;933pkt 1;933pkt s;3339pkt ;3319pkt d;5930pkt ;5910pkt ;1937pkt 1;3339pkt s;396pkt ;5pkt s;208pkt 1;208pkt ;5pkt d;1905pkt s;1444pkt ;1249pkt d;1962pkt ;1249pkt s;884pkt ;712pkt ;1249pkt s;259pkt d;28494pkt 1;259pkt s;73pkt 1;73pkt s;539pkt ;370pkt 1;73pkt s;43pkt ;13pkt s;552pkt 1;552pkt s;1636pkt 1;1636pkt s;975pkt ;536pkt s;793pkt ;336pkt s;1pkt 1;1pkt ;460pkt s;7290pkt s;6209pkt 1;6209pkt s;61pkt 1;61pkt d;18248pkt ;11305pkt s;460pkt ;240pkt s;239pkt 1;239pkt s;670pkt ;40pkt 1;7290pkt 1;6209pkt d;2386pkt ;191pkt s;19250pkt ;2132pkt s;70pkt 1;70pkt s;7636pkt ;1pkt d;2385pkt ;470pkt 1;70pkt ;1993pkt d;300pkt s;260pkt 1;260pkt d;7973pkt s;502pkt 1;502pkt s;210pkt 1;210pkt s;2033pkt 1;2033pkt s;1432pkt 1;1432pkt d;1548pkt 1;1548pkt ;1358pkt s;103242pkt s;107977pkt 1;103242pkt d;109463pkt s;4729pkt 1;4729pkt s;3203pkt 1;3203pkt s;3328pkt 1;3328pkt s;15073pkt ;14304pkt s;25388pkt ;24466pkt s;12554pkt 1;12554pkt s;16676pkt ;15807pkt s;5950pkt 1;5950pkt s;5050pkt ;4842pkt s;8016pkt ;7692pkt s;8103pkt ;7796pkt d;71474pkt ;67913pkt s;233pkt ;2pkt s;91pkt ;1pkt ;4635pkt 1;4729pkt 1;3203pkt 1;3203pkt 1;3328pkt 1;15073pkt 1;25388pkt 1;12554pkt 1;16676pkt d;438pkt s;3728pkt ;400pkt d;5966pkt s;30466pkt ;5pkt s;15080pkt ;5pkt s;276pkt s;4186pkt ;203pkt ;3839pkt d;29845pkt s;3831pkt ;1578pkt s;1867pkt 1;1867pkt d;27291pkt ;24723pkt 1;3831pkt 1;1867pkt d;11595pkt s;37091pkt 1;11595pkt d;1407pkt s;1399pkt ;1397pkt d;9593pkt s;8485pkt ;7453pkt s;1774pkt ;458pkt s;1143pkt 1;1143pkt s;1211pkt 1;1211pkt s;2978pkt s;58367pkt 1;2978pkt s;563pkt s;325pkt 0.84;273pkt d;4564pkt ;3198pkt s;69pkt 1;69pkt 1;1522pkt 1;2551pkt 1;1522pkt d;1481pkt ;489pkt s;1147pkt ;457pkt s;371pkt ;184pkt s;1554pkt ;484pkt ;21pkt ;1507pkt d;456pkt s;580pkt ;410pkt s;528pkt s;250pkt 0.372;93pkt d;2667pkt s;19380pkt 1;2667pkt s;603pkt s;1056pkt ;597pkt d;3661pkt s;3373pkt ;1958pkt d;4997pkt s;5005pkt 1;4997pkt d;820pkt s;1477pkt ;815pkt d;2728pkt s;675pkt ;472pkt s;46382pkt ;144pkt d;2522pkt ;269pkt d;4356pkt ;428pkt d;2105pkt 1;2105pkt d;6613pkt 1;6613pkt s;1258pkt s;5067pkt 1;1258pkt d;11009pkt s;18359pkt ;10955pkt d;322pkt s;173pkt ;172pkt s;434pkt s;121pkt 1;121pkt s;288pkt s;817pkt ;282pkt d;433pkt s;644pkt 1;433pkt d;10918pkt s;1421pkt 1;1421pkt s;10612pkt ;7409pkt s;40pkt 0.55;22pkt s;468pkt s;4216pkt ;400pkt ;130pkt ;3841pkt d;39791pkt s;2677pkt 1;2677pkt s;337pkt 1;337pkt s;6523pkt ;2332pkt s;38973pkt ;11393pkt s;19753pkt ;6728pkt s;49145pkt ;14418pkt d;80286pkt ;25666pkt 1;2677pkt ;38828pkt ;41154pkt d;762pkt 1;762pkt d;301pkt 1;301pkt d;233pkt 1;233pkt d;615pkt 1;615pkt d;440pkt ;296pkt ;144pkt d;576pkt 1;576pkt d;432pkt 1;432pkt d;729pkt 1;729pkt d;443pkt 1;443pkt s;1921pkt s;2080pkt ;1625pkt s;1595pkt s;6481pkt ;1106pkt d;625pkt s;994pkt ;607pkt d;290pkt s;38pkt 1;38pkt s;15426pkt s;15426pkt 1;15426pkt s;15004pkt s;15124pkt 1;15004pkt s;14461pkt s;14475pkt 1;14461pkt s;7972pkt s;15527pkt ;7411pkt s;14951pkt s;15112pkt 1;14951pkt s;15062pkt s;15062pkt 1;15062pkt s;15347pkt s;15347pkt 1;15347pkt s;24792pkt s;25248pkt ;24783pkt s;15174pkt s;15174pkt 1;15174pkt d;34145pkt s;33034pkt ;26820pkt s;30pkt1;30pkt d;32751pkt ;26537pkt s;3940pkt ;3507pkt 1;32751pkt d;866pkt s;15559pkt ;7pkt s;12003pkt ;3pkt s;1034pkt ;137pkt s;15204pkt s;14654pkt 1;14654pkt s;7014pkt s;7014pkt 1;7014pkt s;15553pkt s;15553pkt 1;15553pkt s;15795pkt s;15554pkt ;15448pkt s;14806pkt s;14806pkt 1;14806pkt s;9378pkt s;9378pkt 1;9378pkt s;15408pkt s;15216pkt 1;15216pkt s;7686pkt s;7811pkt 1;7686pkt d;3318pkt s;182pkt 1;182pkt d;2465pkt s;2310pkt 1;2310pkt d;1750pkt ;1585pkt ;1585pkt d;444pkt s;761pkt ;284pkt s;580pkt ;2pkt d;384pkt s;7074pkt ;3pkt s;7912pkt ;3pkt s;4558pkt s;4816pkt ;4511pkt s;15324pkt s;15324pkt 1;15324pkt s;11556pkt d;4943pkt 1;4943pkt s;1735pkt d;1037pkt ;723pkt d;1374pkt ;908pkt s;8531pkt d;8684pkt 1;8531pkt s;1980pkt d;1792pkt 1;1792pkt d;59719pkt s;59670pkt 1;59670pkt s;61947pkt ;59343pkt d;655pkt ;215pkt d;60507pkt ;58741pkt ;165pkt ;59343pkt ;215pkt ;58741pkt ;165pkt ;215pkt 1;60507pkt ;165pkt ;215pkt ;165pkt ;165pkt s;352pkt ;16pkt s;273pkt 1;273pkt s;13451pkt d;6522pkt 1;6522pkt s;4198pkt d;1901pkt ;1864pkt s;6852pkt d;138pkt ;67pkt s;497pkt d;453pkt 1;453pkt s;4126pkt d;1599pkt 1;1599pkt s;3830pkt d;1498pkt ;1470pkt s;217pkt d;237pkt 1;217pkt s;461pkt d;184pkt ;7pkt s;364pkt d;233pkt ;182pkt s;842pkt d;2324pkt ;766pkt s;14707pkt d;14593pkt 1;14593pkt s;302pkt d;152pkt 1;152pkt s;946pkt d;1351pkt 1;946pkt s;293pkt d;96pkt 1;96pkt s;361pkt s;361pkt 1;361pkt s;264pkt s;264pkt 1;264pkt d;367pkt s;755pkt ;352pkt s;1152pkt s;1281pkt ;1054pkt s;2016pkt s;24pkt 1;24pkt s;12957pkt s;12916pkt 1;12916pkt s;15335pkt s;14899pkt 1;14899pkt s;823pkt s;880pkt 1;823pkt s;235pkt s;215pkt 1;215pkt s;10521pkt s;15429pkt 1;10521pkt s;105pkt s;71pkt 1;71pkt s;898pkt s;796pkt 1;796pkt s;505pkt s;505pkt 1;505pkt s;226pkt s;461pkt 1;226pkt s;245pkt s;229pkt 1;229pkt s;489pkt s;720pkt 1;489pkt s;408pkt s;853pkt 1;408pkt d;431pkt s;278pkt ;2pkt d;769pkt ;8pkt s;158pkt ;1pkt d;375pkt s;5289pkt ;1pkt Adding rigor to the comparison of anomaly detector outputs, Fontugne, Borgnat, Abry, Fukuda 10

11 Simple connected components Two event component 86 small components, mainly Sasser Gamma-based = red; Hough-based = green (1) Sasser infected host. (2) Different src.ip and dest.ip. Adding rigor to the comparison of anomaly detector outputs, Fontugne, Borgnat, Abry, Fukuda 11

12 Large connected components I Large component with one community 38 components having more than two events RSync traffic identified by 5 events Adding rigor to the comparison of anomaly detector outputs, Fontugne, Borgnat, Abry, Fukuda 12

13 Large connected components II DNS traffic 29 events in which 27 are from the gamma-based detector Adding rigor to the comparison of anomaly detector outputs, Fontugne, Borgnat, Abry, Fukuda 13

14 Communities in components Distinct traffics Network scan on port 3128 and nntp traffic Adding rigor to the comparison of anomaly detector outputs, Fontugne, Borgnat, Abry, Fukuda 14

15 Communities in components Same kind of traffic 14 events reporting HTTP traffic s;273pkt s;352pkt s;396pkt 1;273pkt ;16pkt ;5pkt 1;208pkt d;4805pkt s;208pkt ;5pkt ;44pkt ;9pkt s;811pkt ;23pkt d;1074pkt 1;253pkt ;78pkt s;253pkt ;70pkt s;707pkt ;16pkt d;8097pkt 1;2294pkt 1;933pkt ;3319pkt s;2294pkt s;933pkt s;3339pkt ;5910pkt ;1937pkt 1;3339pkt d;5930pkt Adding rigor to the comparison of anomaly detector outputs, Fontugne, Borgnat, Abry, Fukuda 15

16 Discussion Advantages Uncover relations between classifier outputs Able to compare outputs of different kinds of classifiers Applications Comparing/combining anomaly detectors Clarifying output of a single detector Understanding detector sensitivity to parameter tuning Adding rigor to the comparison of anomaly detector outputs, Fontugne, Borgnat, Abry, Fukuda 16

17 Conclusion and future work Conclusion Uncover relations between classifiers outputs Graph theory General and rigorous method Future work Deeper analysis of the method Combining anomaly detectors Labelling MAWI Adding rigor to the comparison of anomaly detector outputs, Fontugne, Borgnat, Abry, Fukuda 17

18 Thank you! Questions? Adding rigor to the comparison of anomaly detector outputs, Fontugne, Borgnat, Abry, Fukuda 18

19 [1] Fontugne, R., Borgnat, P., Abry, P., Fukuda, K.: Uncovering relations between traffic classifiers and anomaly detectors via graph theory. TMA (2010) Adding rigor to the comparison of anomaly detector outputs, Fontugne, Borgnat, Abry, Fukuda 19

An Automatic and Dynamic Parameter Tuning of a Statistic-based Anomaly Detection Algorithm

An Automatic and Dynamic Parameter Tuning of a Statistic-based Anomaly Detection Algorithm Yosuke Himura The University of Tokyo him@hongo.wide.ad.jp Kensuke Fukuda National Institute of Informatics /