Fast Botnet Detection From Streaming Logs Using Online Lanczos Method

Transcription

1 Fast Btet Detecti Frm Streamig Lgs Usig Olie Laczs Methd Zheg Che, Xili Yu 3, Chi Zhag 4, Ji Zhag 1, Cui Li 1, B Sg, Jialiag Ga, Xiahua Hu, Wei-Shih Yag 3, Erjia Ya 1 CA echlgies, Ic. Cllege f Cmputig & Ifrmatics, Drexel Uiversity 3 Departmet f Mathematics, emple Uiversity 4 Departmet f Cmputer Sciece, Marylad Uiversity at Baltimre Cuty Abstract Btet, a grup f crdiated bts, is becmig the mai platfrm f malicius Iteret activities like DDOS, click fraud, web scrapig, spam/rumr distributi, etc. his paper fcuses desig ad experimet f a ew apprach fr btet detecti frm streamig web server lgs, mtivated by its wide applicability, real-time prtecti capability, ease f use ad better security f sesitive data. Our algrithm is ispired by a Pricipal Cmpet Aalysis (PCA) t capture crrelati i data, ad we are first t recgize ad adapt Laczs methd t imprve the time cmplexity f PCA-based btet detecti frm cubic t sub-cubic, which eables us t mre accurately ad sesitively detect btets with slidig time widws rather tha fixed time widws. We ctribute a geeralized lie crrelati matrix update frmula, ad a ew termiati cditi fr Laczs iterati fr ur purpse based errr bud ad -decreasig eigevalues f symmetric matrices. O ur dataset f a ecmmerce website lgs, experimets shw the time cst f Laczs methd with differet time widws are csistetly ly 0% t 5% f PCA. Keywrds-bt detecti; btet; streamig lgs; crrelati matrix; Laczs iterati; lie algrithm. I. INRODUCION A bt is a sftware applicati that rus autmated scripts ver the Iteret [1 t perfrm malicius tasks like DOS attack, website statistics skew, click fraud, price/ifrmati scrapig, spam/rumr distributi, etc. raditial sigle bts usually execute their tasks at a rate much higher tha average huma users t achieve their gal withi a time limit. Recet sigle-bt detecti methds, like peak-fidig [, utlier detecti [3, threat prpagati [4, mre r less use this prperty f sigle bts. A btet, as the ame suggests, is a grup f bts that wrk i a crdiated fashi. I ctrast t sigle bts, a btet, especially thse large-scale btets, they might request resurces at a huma-like speed, but altgether they place a heavy burde the servers ad cllect large amut f ifrmati. Because bts i a btet behave huma-like, they are much harder t detect, ad have becme a key platfrm fr may Iteret attacks. Lg data have bee cmmly leveraged fr bth bt ad btet detecti. Geerally, we ca say a lg is a sequece f data etries rder by timestamp, where each data etry carries several fields that recrd the prperties f a activity at a specific time. he mai bjective f this paper is t develp a efficiet btet detecti frm large-scale streamig web server lgs with hst idetifier, request idetifier, ad time stamp; fr example, i this paper we will experimet Apache HP access lgs. A hst idetifier culd be a IP addresses r a MAC addresses, r aythig similar; a request idetifier culd be a URL r a IP address, r web API ame, r aythig similar; the ccrete frms f thse idetifiers deped the type f stream-i lgs. Fr Apache HP access lg, a hst idetifier culd be a IP address, r a hst ame, ad a request idetifier is a URL pitig t sme resurce the server. Althugh we fcus sever lgs, hwever, the apprach ca be used t mitr ay streamig lg data with similar ifrmati fr crdiated behavir. he methd develped i this paper is the key part f a larger bt/btet detecti system prttype verviewed i secti II. he abve bjective is mtivated by ur research ccers, busiess gal ad system requiremets [5. First is wider applicability. As far as we kw, there still lacks researches a geerally applicable btet detecti methd fr web servers. Mst recet btet detecti methds ivlve a particular type f -lg data. Fr example, [6 uses captcha test results t discver search egie bts, [7 takes advatage f a emulatr t iteract with the btet, [8 is based kwledge f prtcl ad DNS traffic, [9 eeds t cstruct a user-user graph based lgi activities. hese methds typically ited fr a special purpse ad eed additial effrts t cllect ad pre-prcess ifrmati t readily available the server ad smetimes eed the mdeler t uderstad advaced Iteret structure. I ctrast, ur apprach is quite lightweight, which ly relies abve-metied basic ifrmati preset i varius access/activity lgs f cmputer systems/sftware that hst etwrk services. Secdly, the time cmplexity. Users stream i their lg data t ur system; the system mitrs the streamig lg data ad i tur prvide real-time warig ptetial malicius hsts. Such real-time feedback has requiremet methd cmplexity as well as sesitivity t crdiated attack frm btet. Lgs usually cme i a large vlume. Users f a medium-scale ecmmerce website ca geerate 100,000 lg etries i 30 miutes. Mre ppular website culd have a much bigger umber. As far as we kw, all previus papers metied here d t meet ur requiremet. Fr example, methd like i [9 eeds hudreds f cmputers t ru hurs t figure ut the btet. hirdly, we believe ur apprach will be mre secure i the sese that users are t required t prvide sesitive lw-level hardware ifrmati. Our methd als des t eed t itercept ad ispect Iteret packets like [10 [11 [1, which culd brig additial security ccers. Lg is the ly data we eed, ad ur methd eve allws users t aymize the idetifiers

2 befre they stream i. Last t the least, it is t pssible fr e methd t detect all srts f bts, ad therefre mder idustrial bt detecti system is fte a esemble [13-15 itegratig hetergeus methds t ehace detecti capability. he simplicity ad ease f use f ur apprach as discussed abve: less data preprcessig, less requiremet specialized kwledge, less ivlvemet with sesitive data, makes fr better itegrati with ther methds. he methd we prpse t adapt is Laczs iterati [16, 17. It is a methd i umerical liear algebra t estimate eigevalues ad eigevectrs, with rigrus thery its errr ad cvergece. Detailed discussi is i secti III. his idea is ispired by Pricipal Cmpet Aalysis (PCA) that captures data crrelati by cmputig eigevalues eigevectrs f crrelati matrix. he mai ctributi f this paper ca be summarized as the fllwig, t the extet f ur kwledge: 1) we are the first t ivestigate PCA-based btet detecti frm streamig lgs; ) we are the first t ctribute a algrithm that updates crrelati matrix fr the mst geeral case f slidig widw i secti III.B; 3) we are first t first t recgize ad adapt Laczs methd fr fast btet detecti, ad we ivate the termiati cditi i ur setup usig herem 1 ad herem that leads t early termiati f each iterati; ur experimets i secti IV further shws its effectiveess. B. Bt Detecti System Prttype Our algrithm plays a key rle i a bt detecti prttype system, which has bee filed fr patet [5, 3, 4. he geeral wrk flw f this system is illustrated i Figure 1, wrkig side by side with a Markv chai-based behavir mdel [4. he latter views bts frm a differet perspective ad detects them by their strage activities the sever. Fr example, a huma visitig a ecmmerce website usually first brwse prducts, make several searches, lg i, add prduct t carts, ad check ut. Hwever, it is hard fr bts t fllw this rutie. Such rutie ca be mdelled by Markv chai trasitis, ad bt visit sequece will have lw prbability i this Markv chai. Hwever, this behavir mdel ca ly detect sigle bts r btet-bts idividually, lackig the ability t discver a btet as a whle. As metied earlier, we favr a lightweight btet detecti algrithm, s it ca be mre easily itegrated with the wrkflw. II. BACKGROUND A. Pricipal Cmpet Aalysis Pricipal Cmpet Aalysis (PCA) is best kw as a dimesi reducti techique, is als a ppular methd i amaly detecti t detect utliers, fr example, fidig cyber amalies [18, 19; thse data pits t wellrepreseted by the pricipal cmpets are csidered utliers r amalies. It is als applied fr the cverse purpse t check if there is high-level crdiati i the data, which has bee successfully applied t the mitrig f idustrial prcesses [0, 1. PCA als has bee cmbied with KL-divergece t detect btet frm search egie lgs [; i that paper KL-divergece is used t filter users with usual click distributi, but still full PCA with cubic cmplexity is applied t detect crrelati is the user is deemed uusual. A similar basic idea ca als be fud i [5. Mathematically, PCA fids the eigevalues ad their eigevectrs f the cvariace r crrelati matrix s.t. the eigevectrs are rthrmal. hse rthrmal eigevectrs are the srted by their eigevalues ad frm rtated crdiate f the space ad are called (first, secd, ) pricipal cmpets. I future discussi, we call the largest eigevalue assciated with the first pricipal cmpet as the pricipal weight. he majr prblem f PCA, r i particular the calculati f the pricipal weight frm the crrelati matrix, is its cubic time cmplexity. he techical purpse f this paper is t use Laczs methd t reduce this time cmplexity. We discuss mre abut this i secti III.C. Figure 1 he wrkflw f ur bt/btet detecti prttype system. III. A. Prblem Frmulati PROBLEM & APPROACH Fr btet detecti f streamig lg data, PCA prvides a gd start pit, ad a straightfrward sluti culd be as the fllwig: 1) split the streamig lgs it time widws accrdig t a specified iterval; ) fr each time widw, cvert the lg etries withi a specified time iterval t a hst-request matrix X with the iteger value at ith rw ad jth clum, deted by X(i, j), represets the umber f times hst j makes request i; 3) ru PCA each time widw, ad PCA ca be applied X t check if the pricipal weight exceeds certai threshld. Abve prcedure ca detect either

3 sigle bts with a large vlume f traffic, r a btet where each bt might t make may requests but they crrelate with each ther ad altgether still prduce a high vlume f traffic. A example is illustrated i Figure. ad clums might be appeded, ad ther rws ad clums might have value chage. Let X t R t be the request-hst matrix befre the tth widw slide, t N. he the mea f each clum is give i the vectr b t = 1 t (X t ) 1 t (1) (a) Figure (a) w sigle bts, each f which has clearly higher visit rates that -bt hsts. Althugh their visit cuts are t crrelated, they dmiate -bt visits, leadig t a high pricipal weight f (b) A btet with crrelated visit cuts, with a high pricipal weight f Netheless, questis arise i the adaptati f PCA t ur applicati. First, the time cmplexity f full PCA used i [ is cubic, which easily breaks dw whe a large umber f lgs cme i. I ur applicati, we are prbably iterested i ly the largest eigevalue ad its eigevectr. It is bviusly t ecessary fr a full PCA. Secdly, divisi it time widws seems t suitable fr mitr f streamig data. We are facig a accuracy-sesitivity dilemma: a small widw will weake the algrithm s ability t fid bts, sice fewer data might t prvide sufficiet evidece fr PCA; a larger widw will slw dw the algrithm s sesitivity, e.g. i a massive attack the sever might already be brught dw befre the algrithm starts t aalyze the last 30-mi widw. here is uiversal criteri fr gd widw legth, ad it is difficult t cduct experimets fr differet server lgs. A mre prfessial practice is slidig widw, as i idustrial prcess mitr [1, 5, 6, but this makes cmputati eve mre itese. I prcess ctrl, the features as clums f data matrix is usually at mst hudreds f clums, while i ur applicati, the features are tes f thusads f hst idetifiers. Frm abve discussi, reducig time cmplexity is the tp pririty if we desire t use slidig widw. Our research prblem ca thus be summarized as cstructig a fast algrithm suitable fr usig slidig widw t mitr largescale streamig lgs fr ptetial bts/btets. B. Crrelati Matrix Update Csider the request-hst matrix itrduced i previus secti. We evaluate the pricipal weight frm the crrelati matrix betwee hsts. After each widw slide, this matrix will chage, s des the crrelati matrix. Our first prblem is hw t update the crrelati matrix after every time widw slide befre updatig the pricipal weight. he time cmplexity fr cmputig crrelati matrix f a m request-hst matrix is O(m ), s it is highly uecmic ad udesirable t re-cmpute etire crrelati matrix. [1 calculated the updates fr addig ew rws ad deletig ld rws. Our case is mre cmplicated: after every widw slide, sme rws ad clums might be remved, sme ew rws (b) where 1 t = [1,,1 R t is a clum vectr f legth t with all cmpets beig 1. he we call the fllwig as the cetralize request-hst matrix, because each f its clum has zer mea X t = X t 1 t b t () where Σ t = diag(σ t,1,, σ t,mt ) is a diagal matrix with the jth diagal elemet σ t,j beig the stadard deviati f the jth clum f X t. he crrelati matrix fr X t is thus R t = 1 t 1 Σ t 1 X t X t Σ t 1 (3) Our gal is t fid the update frmula fr R usig ifrmati frm X t, b t, X t ad R t. Fr simplicity, we ca assume that the hsts are the same befre ad after the widw slide, withut lss f geerality, i.e. clum chages ca be disregarded whe calculatig the updates. his is because, if there are ay added clums r remved clums caused by widw slide, we ca simply add crrespdig clums (ad rws if ecessary) it b t, X t, R t, ad use the mdified es fr ur future ifereces. Als te crrelati is t depedet rw rder, therefre rws f X t ca be arbitrarily arraged fr ur cveiece. Suppse after the tth slide, 1) ew rws X are appeded t the bttm f X t, each rw represetig a ew request-hst vectr t curretly i X t ; ) rws X tp f X t are remved, meaig thse requests disappear after the widw slide; 3) ther rws have value chage deted by matrix X c where ly the tp c f X c are -zer., Sice every widw slide is small ad f fixed distace, thus c is usually small ad ca be treated as a cstat. I future discussi, we als write X t = [ X X t where X t likewise, X t = [ X X t cetralized data f X is the last t rws f X t ; where X ad X t are ad X t (). We have the fllwig relati fr X X c X, where 0 is equivalet t X replaced by zers. [ 0 X respectively like i, X, ad m t zer matrix, ad mt detes a except fr the remve rws are = [ X t [ X X c X X (4)

4 he update f mea-value is w give by ( t )b = t b t (X ) 1 (X ) 1 (X c ) 1 C (5) Let Δb = b b t, ad te X = X 1 b is data cetralizati like i (), we the have X = [ 0 X [ 0 1 b = [ X t 1 t b t 1 t b t [ X X c [ 0 1 b X = [ X t 1 t b t 1 t b t [ X [ X c X 1 b = [ X t (1 t b t [ X [ = [ X t [ = [ [ X c X t X X c X 1 1 t X 0 X c t b b t b t ) (b O(m t ( t )) O(m t ) X 1 t Δb 1 t b ) where we te the tp rws f X must all be zer, ad thus the time cmplexity f X is O(m t ( t )) = O(m t ). We w start calculatig the update fr the stadard deviatis i Σ. σ,j = = [ X t [ X X c 0 [ b (j)1 X 1 X t 1 t b t (j) 1 t b t (j) [ X X c [ 0 b (j)1 t b (j)1 X 1 We ca simplify (7) piece by piece. First expad, X t b t (j)1 t b t (j)1 t [ X X c [ 0 b (j)1 t = X t b t (j)1 t b t (j)1 t [ X X c [ 0 b (j)1 t [b t (j)1 t [ X X c [ 0 b (j)1 [X t t b t (j)1 t (8) where X t b t (j)1 t = ( t 1)σ t,j (9) (6) (7) b t (j)1 t [ X X c [ 0 b (j)1 t (10) b t (j)1 t X = [ Δb (j)1 t X c 1 t X t 1 t b t (j)1 t = 0 [b t (j)1 t [X t b t (j)1 t = 0 (11) Usig (11), we have the fllwig imprtat simplificati. 0 [ b (j)1 t = (b (j)1 t [X t b t (j)1 t [ b (j)1 0 t = [ b (j)1 X 0 t t ) [X t b t (j)1 t Plug (9)~(1) back t (8) ad the (7), we have ( 1)σ,j = ( t 1)σ t,j [ X O( ) b t (j)1 t X O(c ) Δb (j)1 t O( c ) b (j)1 X c X c X t X O( ) (1) b (j)1 (13) Recall the tp c rws f X c are -zer, thus frm (13) the time cmplexity fr updatig each stadard deviati is O( c ), liear t the umber f rws that are affected by the widw slide. he ttal cmplexity fr updatig all stadard deviatis is O(m t ( c )). At last we update the crrelati matrix. Usig the furth idetity f (6) ad a techique like (1), we have ( 1)R = (X = ( X tσ 1 ( X tσ 1 = Σ 1 X t X t Σ 1 Σ (1 t b t [ X X c 1 ) (X [ 1 X Σ (1 t b t [ X X c O(m t ) [ 1 X Σ Σ 1 1 Y Σ Σ 1 ) 0 1 t b 0 1 t b O(m t ) = ( t 1)Σ 1 1 Σ t R t Σ t Σ Σ 1 1 Y Σ where 1 ) Σ 1 ) Σ Y = X t (1 t b t [ X 0 [ X c 1 t b ) (1 t b t [ X 0 [ X c 1 t b ) (1 t b t [ X 0 [ X c 1 t b ) X X We the use the fllwig idetities multiple times ) ) (14) (15)

5 X t 1 t = 0 X t X t = X t (X t 1 t b t 1 t b t ) = X t X t X t 1 t b t = X t X t 0 1 t b [ t arrive at ad = 1 t b [ 1 0 (t b (16) ) (17) Δb 1 t 1 t Δb = t Δb Δb (18) X t ([ X [ X c = X t (1 t b = X t [ X = (X = (1 t Δb X (1 t b t [ X [ X c [ X = t Δb Δb 0 1 t b [ X X c 1 b X c X t 0 mt 1 t b 1 b X c Δb 1 t ) 0 (t [ 1 ) ) [ X [ X X c ) b ) ) (1 t b t [ X [ 1 (1 t Δb 1 X c b X c b X c [ X 1 Σ [ X 1 (19) 0 mt 1 t b 1 b X c ) ) b X c (0) Ctiue the simplificati (sketch calculati, sme steps are lg ad hece mitted, but they are t hard t verify). Δb 1 t [ X = Δb (1 = Δb (1 [ X = (X = X 1 b X c 1 X 1 b X c X X b (X c t [ X ) (X ) (X c 1 b X c b X c 1 b 1 t b ) ) (X c ) Δb Δb X c ) ) (X c ) (1) () Fially, plug (1) ad () back t (0), ad the plug (19) ad (0) back t (15), ad after certai rearragemet, we will have O(m t ) O(m t ) Y = ( t )Δb Δb X Δb 1 O(m t ) X X X O(m t c ) X (X t X c 1 t Δb ) X c (3) Fr the last added f (3), we ca further use f the last idetity f (6) t reuse the result f X. Δb X t X c 1 t = X (1:, : ) (4) X c he time cmplexity fr update f crrelati matrix R is clearly O(m t ( c )) by (14) ad (3). All updates are summarized i Algrithm 1. Cmbiig (6) ad (13) the ttal cmplexity fr crrelati matrix update is O(mi{m t, m t ( c )}), which ca be csidered as quadratic if c is treated as a small cstat. his cmplexity is theretically dw frm straightfrward cmplete re-evaluati by pwer f 1. I practice, we smetimes culd expect eve better accelerati, as Δb, X c etc. are very fte sparse. C. Laczs Methd I future discussi, fr cveiece f mathematical aalysis, we assume the crrelati matrix is rmalized t uit ttal variace by dividig every elemet by the umber f clums, s that the su f all eigevalues is 1, the largest eigevalue, i.e. the pricipal weight is betwee rage [0,1, ad a eigevalue larger tha 0.5 must be the largest eigevalue. I implemetati, we shuld istead rmalize the cmputed eigevalue t the rage f [0,1 fr better umerical stability. With updated crrelati matrix R k1, ur task is t determie if R k1 has a large eigevalue, i.e. the pricipal weight, is larger tha a threshld. If s, we csider there exists ptetial bt visits i the curret time widw, ad ctiue t fid thse hsts that have high crrelati with the pricipal cmpet. Several methds suit fr this task, icludig sigular value decmpsiti, which cmputes the full PCA i cubic time, r Rayleigh qutiet iterati, which fids the exact eigevalue ad eigevectr i cubic time [7. Fr ur purpse, we discussed i III.A that a accurate evaluati f eigevalue is t ecessary. We will ivestigate Laczs methd [17, 8, a umerical methd t apprximate the eigevalues, fr its use i ur applicati. Give ay symmetric matrix R R m m (the crrelati matrix is symmetric) ad ay -zer iitial vectr x R m, the Laczs iterati is expressed as the fllwig prcess, v 0 = 0, v 1 = x x v j = Rv j1 Rv j1, v j1 v j1 v j1 v j { v j = v j, j =,3, v j (5) he iterati is guarateed t termiate at j = k 0 whe it fuds v k0 = 0, where k 0 is the smallest psitive iteger s.t. R k 01 x spa{x, Rx,, R k 0x}, ad is guarateed t be well-defied, i.e. v j 0, fr j = 1,, k 0. It ca be prved that V k0 1 = [v 1,, v k0 1 is a rthrmal basis f

6 spa{x, Rx,, R k 0x}. Let dete the stadard ier prduct, ad let V k = [v 1,, v k, 1 k k 0 1, the the Laczs decmpsiti gives RV k = V k k v k1 e k, k = 1,, k 0 1 (6) where e k R k is a vectr with ly the kth cmpet beig 1 ad all ther cmpets beig 0, ad Rv 1, v 1 v v Rv k =, v (7) v k ( v k Rv k, v k ) is a symmetric tridiagal matrix. Sice v k1 is rthgal t every clum f V k, the (6) is equivalet t V k RV k = k, k = 1,, k 0 1 (8) Oe ca chse t use either (6) r (8) fr their w cveiece. he mre crucial pit here fr ur purpse is that the eigevalues ad eigevectrs f k ca estimate thse f R. he eigevalues ad eigevectrs f a symmetric tridiagal matrix is a basic task i umerical liear algebra, [16, 1, 9-3. hey take advatage f the special frm f symmetric tridiagal matrices t ru faster ad mre accurate tha algrithms fr geeral matrices. he eigevalue apprximati is rbust because the errr ca be shw buded by the fllwig imprtat therem, herem 1. Fr ay eigevalue λ f k, there exists a eigevalue λ f R s.t λ λ v k1 ; r equivaletly if R has m eigevalues {λ 1,, λ m }, the we have mi λ i λ v k1. Further, if μ is a eigevectr f 1 i m eigevalue λ f k, the there exists a eigevalue λ f R s.t λ λ v k1 e k,μ. μ he therem is stated with a shrt prf sketch i [17. Sice k R k k, μ R k, thus e k, μ is the abslute value f the last cmpet f eigevectr μ, ad e k,μ 1, meaig the secd errr bud is equal r tighter. Sice cmputati f μ ly takes liear time fr a tridiagal matrix, we will use the secd bud. e k,μ is usually a small μ value ad it decreases with k [17, 33. he apprximati ca als cveietly ru i a lie style due t ather useful therem, which implies the maximum eigevalue f k is -decreasig with k. hus, if we are t satisfied with the errr bud d, we ca icrease k fr better result. μ herem. Fr ay symmetric matrix, the maximum eigevalues f its leadig pricipal submatrices are always -decreasig. Based abve, we prpse the fllwig t termiate the iterati early fr ur applicati. Suppse we use a threshld ω > 0.5 fr pricipal weight, ad let d = v k1 e k,μ, the the μ practical meaig f herem 1 is: if we fid a large eigevalue λ f k, the if λ d ω, the the largest eigevalue f the rmalized crrelati matrix R must exceed ω, ad a warig ca be immediately raised; i ctrast, if λ d < ω ad λ d 0.5, the the largest eigevalue f R must t exceed ω, ad we ca simply wait fr ext widw slide. Fr λ d < 0.5, we setup as threshld c, ad if λ d stays belw 0.5 fr c ruds, the the largest eigevalue f R is t likely t exceed ω, ad we ca ctiue t ext widw slide. his is because the largest eigevalue f k appraches the largest eigevalue f R frm belw, therefre whe the estimated eigevalue plus the errr bud stays belw 0.5 fr may iteratis, it becmes icreasigly ulikely fr the eigevalue t exceed a threshld ω > 0.5. Oce the largest eigevalue λ f k is estimated, its eigevectr μ ca be fud trivially, because we ca set μ (1) t ay psitive umber if v λ, ad μ (1) = 0 therwise, the ther cmpets f μ ca be slved iteratively by k μ = λ μ, ad μ = V kμ is the estimated V k μ pricipal cmpet f R. he crrelati betwee hst j ad μ ca be cmputed by ρ j = μ (j) λ. We recmmed fidig the first kee pit f the descedigly srted list f all ρ j, e.g. the pit befre the first sharp slpe, which is ispired by the scree plt f PCA; meawhile we ca disregard hsts with ρ j < ω i case the kee pit is t lw. Hsts satisfyig bth criteria are csidered as ptetial bts. Fially, we give sme mre facts that will be used i the algrithm. First a lse bud f eigevalues f k are give by λ l λ λ u where λ l = max{mi{ Rv 1, v 1 v, Rv i, v i v i1 v i }, 0} λ u = mi{max{ Rv 1, v 1 v, Rv i, v i v i1 v i }, trace k } i =,, k (9) Secdly, the ith leadig pricipal mir p i (λ) f k λi ca be cmputed as 1 i = 0 p i (λ) = { Rv 1, v 1 λ i = 1 ( Rv i, v i λ)p i1 (λ) v i p i (λ) i =,, k (30) ad we use s k (λ) t dete the umber f sig chages i p 0,, p k. A cmplete Laczs methd based bt detecti algrithm is w as belw, based tridiagal-matrix eigevalue estimati algrithm i [16, 1, where we ivate the termiati cditi fr ur particular applicati usig herem 1 ad herem.

7 he ttal time cmplexity f Algrithm fr e widw slide is O(k m ) where k is the value f k whe the cmputati fr the curret widw slide eds, ad m is the umber f distict hsts. I practice, the average cmplexity scales ear quadratic as the average umber f k grws slwly with m. Fr detecti f multiple btets, the part i marked by i Algrithm ca be mdified t cmpute mre eigevalues f k by estig a lp the same as the d-while lp except fr the ttal variace is 1 mius the pricipal weight, ad termiati cditis eed crrespdig adjustmet. We mit the detail fr space. IV. EXPERIMEN & EVALUAION We simulated bth sigle bts ad btet a ecmmerce web server ad cllected abut fur mths lg data, ttalig 315,688,764 Apache access lg etries fr ur experimets, with 3,075,108 distict request idetifiers (URL t website resurces) ad distict,519,0 hst idetifiers (IP addresses r hstame). he huma-like visit rate is estimated by average iterval betwee tw requests f HML web pages, which is 39 secs fr this dataset. Sme f the hsts i the dataset mark themselves as bts. hey are mstly search egies, like ggle-bts, big-bts, yah-bts. Hwever, they cat be used as gld stadard because they are very well-behaved eve thugh they are bts ad hard t detect. hus, we have t ru simulati. he simulati is real the website server s the geerated lgs are realistic, ad it is guarateed that it is abslutely harmless fr the target server. he simulati is de i fur mdes t mimic differet types f bts. 1) Sigle request: the simulatr radmly chse a lik frm a fixed list ad keeps repeatedly visitig the lik fr tw hurs, the the simulatr picks the ext lik. ) Radm list: fr every visit, the simulatr radmly chses a lik frm a list t visit. 3) Fixed list: the simulatr visits liks f a fixed list i its rigial rder. 4) Fcused radm walk: the simulatr wrks like a crawler it dwlads a webpage, extracts liks f particular patter frm it, ad the radmly picks the ext lik t visit. he fur mdes ca mimic differet types f bts. Fr example, DOS/DDOS might take the frm f ay f the first three types; click fraud r statistic skewer are usually the secd ad the third frm, visitig a list f desired target liks; price scraper r web crawler culd be ay f the last three types. We have several further cmmets: 1) multiple lg etries culd be geerated fr a sigle visit; ) fr all mdes, the simulatr requests at a Gaussia distributed huma-like radm rate estimated by the true huma visits; 3) ly e cmputer with a distict IP is used t simulate each mde, therefre the simulati is harmless fr a website that hadles tes f thusads f custmers each day. he simulated lgs fr each visit are idetified, ad the duplicated with ew hst ids ad mixed as desired fr varius experimet purpses. Fr example, we ca mimic sigle-bts f high visit rate f each type by duplicatig them several times i a time widw withut chagig their hst id; we ca mimic a massive bt et f huma-like visit rate by chsig lgs f several differet visits, duplicate them may times with distict hst ids, ad radmly mixed them with existig lgs i several csecutive widws. he simulated lgs are used as gld stadard fr perfrmace aalysis. A. Perfrmace Aalysis his secti presets cmparis f accuracy, rutime ad sesitivity betwee PCA ad ur Laczs-based algrithm. We ru PCA with fixed widw ad Laczs with bth fixed widw (deted by Laczs-B) ad slidig widw (deted by Laczs-S). All experimets are restricted t e thread fr fairess. Bth methds have researches parallelism as metied i secti II.A. Fr this paper, we fcus experimets with e thread. Fr ur purpse, we defie the accuracy as the percetage f kw bt visits that ca be crrectly marked by the algrithm. We cmpare the Laczs-methd based algrithm with the full PCA ur dataset ad measure bth ruig time, accuracy ad sesitivity. We experimet bth fixed widws ad slidig widws f legth frm 10 mis t 50 mis. Lgs fr 100 sigle bt visits are placed at radm time pits with 30 times t 50 times faster visit rate tha humas; 100 btets f 10 t 100 hsts are placed at radmly chse time pits with 1 t 5 times huma-like visit rate, ad fr simplicity, we let bts i a btet start wrkig simultaeusly at the chse time pit, ad their visits d t verlap. Each bt/btet visit lasts frm 10 mis t hrs. Widw slide step is 10% f the widw legth. Fr parameters f Algrithm, we let ω = 0.65, ε 1 = 10 10, ε = 0.01, k l, k u, k s are set t 10%, 80% ad 1% f the umber f hsts i the widw, ad

8 c = 5. he value f ε 1 is give by [16, 1. he chice f ω, ε, c will be experimeted i ext secti. he results fr time ad accuracy are shw i able 1, ad it prvides clear evidece fr the advatage f ur algrithm agaist PCA. 10m 0m 30m 40m 50m Sigle Bts Btets (3) Ru ime (3) PCA 70.5% 5.4% 1s (1) /13s () Laczs-B 70.1% 50.3% 3.6s/0s Laczs-S 73.5% 63.% 5.5s/44s PCA 59.0% 65.6% 44s/6s Laczs-B 58.% 64.0% 1s/81s Laczs-S 63.4% 78.9% 15s/11s PCA 53.7% 79.6% 89s/1473s Laczs-B 5.6% 78.1% 1s/134s Laczs-S 59.8% 87.9% 3s/184s PCA 48.1% 80.9% 131s/1959s Laczs-B 47.5% 79.1% 7s/177s Laczs-S 54.1% 88.% 30s/14s PCA 4.1% 77.8% 163s/607s Laczs-B 41.6% 76.1% 3s/18s Laczs-S 50.3% 85.5% 35s/8s able 1 Accuracy & rutime cmparis f PCA with fixed time widw, Laczs with fixed time widw (Laczs-B) ad Laczs with slidig time widw (Laczs-S). Results i bld are better. (1) Average rutime f the algrithm all time widws; () maximum rutime; (3) detecti f btets with better accuracy ad less ruig time is the mai techical purpse f this research. Overall, we have several cclusis frm able 1: 1) It is t pssible t ru PCA with slidig widw f legth lger tha 0m, as its cmputati time will geerally exceed the slidig step; ) Laczs-B has cmpetitive perfrmace i cmparis t PCA fr bt detecti; 3) Laczs has much better ruig time, ad average it grws almst liearly with the widw legth; 4) Laczs with slidig widw csistetly has higher accuracy tha fixed widw. A subtler implicati f abve results is abut chice f widw legth. Fr sigle bts, lger widw legth damages perfrmace, which is the weakess f all crrelati-based btet detecti algrithms. his is because if there exists mre tha e t-s-crrelated sigle bts i a time widw, the pricipal weight will pluge, ad lger time widw legth icreases the prbability f this situati. Fr btet, withi a rage, accuracy icreases with widw legth, because mre data prvide mre statistical evidece f crrelati if the btet exists, especially whe btets iclude radmess i their visit patter, like the radm walk ad radm list i ur simulati. Hwever, lger time widw usually makes the algrithm less capable f detectig bts with shrter visit durati. Fr example, fr 50-mi widw legth, btet visits f less tha 0 miutes becme less discverable. Give the assumpti that btets fte have t maitai at least a request per 30 t 40 secs (the huma-like visit rate f ur dataset) t achieve its gal at a reasable cst, 40 miutes data geerally are sufficiet fr expsig them. I practice, we wuld recmmed ruig at least tw threads t mitr a lg stream, e with shrt legth (like 10-mi, 15-mi) fr discvery f sigle bts ad shrtdurati visits f btets, ad the ther with medium legth (like 30-mi, 40-mi) fr detecti f ther btets. If detecti f slwer btet is desired, we ca add e mre thread with lger widw legth. We w preset the sesitivity f bth PCA ad Laczs- S which is defied as hw much time des the algrithm eeds t detect the bts sice their iitial requests. he experimets are de fr 10-mi widw legth ad 40-mi widw legth t test hw what percetage f bts are detected withi a specified sesitivity. he results are shw i Figure 3 with a pit (x, y) a curve fr algrithm z meas x% f the bts detected by z are detected withi y miutes f their iitial request. A higher curve implies better sesitivity. he verall average sesitivity results are shw i able. We ca bserve a clear advatage f Laczs-S frm bth Figure 3 ad able. he advatage cmes frm tw surces: less cmputati time f Laczs, ad the slidig widw. Ultimately, it is all due t the less time cmplexity f Laczs, which eables us t use slidig widw rather tha fixed widws. (a) 10-mi widw (b) 40-mi widw Figure 3 Experimet results fr the sesitivity f PCA ad Laczs-S. -Bt suffix meas results fr sigle bts, -Btet suffix meas results fr btet-bts. x-axis marks the time i miutes sice iitial request, y-axis idicates amg bts detected by the algrithm, what percetage is detected withi crrespdig miutes sice iitial request. Fr example, a pit at (x, y) the curve fr PCA-Btet meas x% f all detected btet-bts are detected withi y miutes sice their iitial request. 10m-Bt 10m-Btet 40m-Bt 40m-Btet PCA 9.85m 10.69m 38.0m 4.58m Laczs-S 4.94m 7.34m 5.5m 33.35m able Results f average sesitivity (average time the algrithm eeds t detect the bts sice their iitial requests) fr PCA ad Laczs-S with 10-mi ad 40-mi widw legths. B. Experimet Parameters his secti presets experimets parameters ω, c ad ε ad prvides sme isight it chice f their values. he experimets is the whle dataset with Laczs-B, because Laczs-S takes t much time, ad the results f Laczs-B shuld be gd eugh fr ur discussi. I the case f real btet detecti, we ca ru the fllwig experimets histrical lgs ad decide gd values fr the parameters. Fr chice f ω, we first ru Laczs-B the rigial data (withut simulati) ver all fixed 40-mi widws, ad the use the Markv chai behavir mdel i secti II.B t recgize ad remve stragely-behaved suspicius hsts, ad the ru the Laczs-B ver all widws agai. he pricipal weight distributi chage is shw i Figure 4. he histgram shws mst suspicius hsts are i widw with pricipal weights higher tha 0.5 quite pssibly ctai bts. Hwever, 1) ur algrithm requires that ω > 0.5 ; ) we emphasize btet detecti, ad mile pricipal weight is less likely t imply existece f btet; 3) the rate f false alarm

9 shuld be ctaied at a reasable level. Csiderig all these factrs, we chse ω = 0.65 as ur threshld. Figure 4 Pricipal weight distributi chage after usig the Markv chai behavir mdel t remve stragely-behaved hsts frm the lgs f all 40- mi fixed time widws. Fr chice f c, it affects the early termiati fr a widw where bts are t likely t exit, ad a smaller c will decease rutime, but pssibly itrduce certai errr. Fr chice f ε, it affects the early termiati fr a widw where bts are detected, ad smaller ε idicates a mre accurate pricipal cmpet is desired. he results usig Laczs-B with bth 10-mi widw ad 40-mi widw are shw i Figure 5, ad fr the experimets i previus secti we chse parameter value at the elbw pits i Figure 5 (b),(d). We i additi remark that: 1) fr shrter time widw, larger c may take larger value withut hurtig much accuracy, thus if we ru a secd thread with shrt time widw as suggested i chice f widw legth i previus secti, we culd specify a larger c ; ) ε has strger effect perfrmace, thus we recmmed settig it t a small value eve fr shrt time widw. Figure 5 Relati betwee c, ε with average rutime ad accuracy fr Laczs methd. Smaller c, ε imprves accuracy at cst f mre rutime. Fr experimets i previus secti, we chse c, ε at the elbw pit c = 5 ad ε = f (c) ad (d). C. Btet Examples We apply ur algrithm the rigial data withut simulated lgs, ad 4,557 distict hsts are marked as ptetial bts, with 3,417 f them recgized as btet-bts i 3 ptetial btets. Btets discvered i differet time widws are merged if they share tw r mre hsts. We use a autmatic prgram t cmpare the IPs with the Barracuda IP reputati database ad 88% f them have pr reputati. Sme f the btets are search egie like crawl xx.xxx.gglebt.cm that ca be actually cfirmed. Besides that, we maually cfirmed e f the tp-rated btet that d t mark themselves as bts i the access lg, which cmes frm a website mitr cmpay Aturis, as shw i Figure 6. Sme ther btets are demstrated i Figure 7, which clearly shws that e bt i the btet might play the rle f a ctrller. Figure 6 A tp-rated btet is maually cfirmed frm a cmpay prvidig website mitrig service. At the time ur experimet, they d t mark themselves as bts i the Apache access lg. (a) c, 10-mi widw (b) c, 40-mi widw Figure 7 Sme ther examples f recgized btets, where we ca see usually e bt might play the rle f a ctrller. (c) ε, 10-mi widw (d) ε, 40-mi widw V. CONCLUSION & FUURE WORK he mai bjective f this paper is t recgize btets frm streamig web server lgs. We recgize ad adapt Laczs methd t the applicati f btet detecti. Fr

10 this purpse, we first develp the lie crrelati matrix updates, ad feed them t the Laczs iteratis. Makig use f Laczs errr bud the -decreasig eigevalues f symmetric matrices, ad the special prperties f ur applicati, a methd is prpsed t termiate the iteratis early. Our apprach imprves time use f eigevalue-based btet detecti frm cubic t sub-cubic, which eables us t mitr the lg stream by slidig widws, rather tha batchbased detecti. Experimets shw the time cst f Laczs methd with differet time widws are csistetly ly 0% t 5% f PCA. I the future, we culd further the research i tw directis: 1) fidig its gd use i ther amaly detecti applicatis; ) cmpesate its weakess i sigle bt detecti by the Markv-chai behavir mdel as metied i secti II.B. REFERENCES 1. Duham, K. ad J. Melick, Malicius bts: a iside lk it the cyber-crimial udergrud f the iteret. 008: CrC Press.. egeler, F., et al. Btfider: Fidig bts i etwrk traffic withut deep packet ispecti. i Prceedigs f the 8th iteratial cferece Emergig etwrkig experimets ad techlgies. 01. ACM. 3. Kag, A.R., et al., Olie game bt detecti based partyplay lg aalysis. Cmputers & Mathematics with Applicatis, (9): p Carter, K.M., N. Idika, ad W.W. Streilei. Prbabilistic threat prpagati fr malicius activity detecti. i Acustics, Speech ad Sigal Prcessig (ICASSP), 013 IEEE Iteratial Cferece IEEE. 5. Ji Zhag, Z.C., Chi Zhag, Bt Detecti Based Divergece ad Variace, I. CA echlgies. Patet Applicati US017010US1. 016: Uited States. 6. Kag, H., et al. Large-scale bt detecti fr search egies. i Prceedigs f the 19th iteratial cferece Wrld wide web ACM. 7. hmas, K. ad D.M. Nicl. he Kbface btet ad the rise f scial malware. i Malicius ad Uwated Sftware (MALWARE), 010 5th Iteratial Cferece IEEE. 8. Maasrah, A.M., et al., Detectig btet activities based abrmal DNS traffic. arxiv preprit arxiv: , Zha, Y., et al. BtGraph: Large Scale Spammig Btet Detecti. i NSDI Gebel, J. ad. Hlz, Rishi: Idetify Bt Ctamiated Hsts by IRC Nickame Evaluati. HtBts, : p Gu, G., et al. BtHuter: Detectig Malware Ifecti hrugh IDS-Drive Dialg Crrelati. i USENIX Security Sympsium Wurziger, P., et al. Autmatically geeratig mdels fr btet detecti. i Eurpea sympsium research i cmputer security Spriger. 13. Bhatia, S., D. Schmidt, ad G. Mhay. Esemble-based dds detecti ad mitigati mdel. i Prceedigs f the Fifth Iteratial Cferece Security f Ifrmati ad Netwrks. 01. ACM. 14. Rayaa, S. ad L. Akglu, Less is mre: Buildig selective amaly esembles. ACM rasactis Kwledge Discvery frm Data (KDD), (4): p Bt Detecti ad Mitigati. Available frm: Glub, G.H. ad C.F. Va La, Matrix cmputatis. Vl : JHU Press. 17. Allaire, G. ad S.M. Kaber, Numerical liear algebra. Vl : Spriger. 18. Xie, M., S. Ha, ad B. ia. Highly efficiet distace-based amaly detecti thrugh uivariate with PCA i wireless sesr etwrks. i rust, Security ad Privacy i Cmputig ad Cmmuicatis (rustcm), 011 IEEE 10th Iteratial Cferece IEEE. 19. Pevy,., M. Rehák, ad M. Grill. Detectig amalus etwrk hsts by meas f pca. i Ifrmati Fresics ad Security (WIFS), 01 IEEE Iteratial Wrkshp. 01. IEEE. 0. Wise, B. ad N. Ricker. Recet advaces i multivariate statistical prcess ctrl: imprvig rbustess ad sesitivity. i Prceedigs f the IFAC. ADCHEM Sympsium Li, W., et al., Recursive PCA fr adaptive prcess mitrig. Jural f prcess ctrl, (5): p Yu, F., Y. Xie, ad Q. Ke. Sbtmier: large scale search bt detecti. i Prceedigs f the third ACM iteratial cferece Web search ad data miig ACM. 3. Chi Zhag, Z.C., Ji Zhag, Bt Detecti System Based O Deep Learig, I. CA echlgies, Patet Applicati US1. 016: Uited States. 4. Zheg Che, J.Z., Chi Zhag, Bt Detecti Based O Behavir Aalytics, I. CA echlgies, Patet Applicati US US1. 016: Uited States. 5. Jeg, J.-C., Adaptive prcess mitrig usig efficiet recursive PCA ad mvig widw PCA algrithms. Jural f the aiwa Istitute f Chemical Egieers, (4): p Wag, X., U. Kruger, ad G.W. Irwi, Prcess mitrig apprach usig fast mvig widw PCA. Idustrial & Egieerig Chemistry Research, (15): p Rayleigh qutiet iterati. Available frm: 8. Cullum, J.K. ad R.A. Willughby, Laczs algrithms fr large symmetric eigevalue cmputatis: Vl. I: hery. 00: SIAM. 9. Da Fseca, C., O the eigevalues f sme tridiagal matrices. Jural f Cmputatial ad Applied Mathematics, (1): p Osipv, A., Evaluati f small elemets f the eigevectrs f certai symmetric tridiagal matrices with high relative accuracy. Applied ad Cmputatial Harmic Aalysis, Kaha, W., Accurate eigevalues f a symmetric tri-diagal matrix. 1966, SANFORD UNIV CA DEP OF COMPUER SCIENCE. 3. Cuppe, J., A divide ad cquer methd fr the symmetric tridiagal eigeprblem. Numerische Mathematik, (): p Xu, G. ad. Kailath, Fast estimati f pricipal eigespace usig Laczs algrithm. SIAM Jural Matrix Aalysis ad Applicatis, (3): p