Using Fuzzy Pattern Recognition to Detect Unknown Malicious Executables Code

Transcription

1 Usng Fuzzy Paern Recognon o Deec Unknown Malcous Execuables Code Boyun Zhang,, Janpng Yn, and Jngbo Hao School of Compuer Scence, Naonal Unversy of Defense Technology, Changsha 40073, Chna hnxzby@yahoo.com.cn Deparmen of Compuer Scence, Hunan Publc Secury College, Changsha 4038, Chna Absrac. An nellgen deec sysem o recognon unknown compuer vrus s proposed. Usng he mehod based on fuzzy paern recognon algorhm, a malcous execuable code deecon nework model s desgned also. Ths model arge a Wn3 bnary vruses on Inel IA3 archecures. I could deec known and unknown malcous code by analyzng her behavor. We gahered 43 bengn and 09 malcous execuable programs ha are n he Wndows Porable Execuable (PE) forma as daase for expermen. Afer exracng he mos relevan API calls as feaure, he fuzzy paern recognon algorhm o deec compuer vrus was evaluaed. Inroducon Malcous code s any code added, changed, or removed from a sofware sysem o nenonally cause harm or subver he sysem s nended funcon []. Such sofware has been used o compromse compuer sysems, o desroy her nformaon, and o render hem useless.excellen echnology exss for deecng known malcous execuables. Sofware for vrus deecon has been que successful, and programs such as McAfee rus Scan and Noron Anrus are ubquous. These programs search execuable code for known paerns. One shorcomng of hs mehod s ha we mus oban a copy of a malcous program before exracng he paern necessary for s deecon. Our effors o address hs problem have resuled n a felded applcaon, bul usng echnques from fuzzy paern recognon and machne learnng. The Malcous Execuable Classfcaon Sysem currenly deecs unknown malcous execuables code whou removng any obfuscaon. As far as know, our expermens s he frs me o esablshed mehods based on fuzzy paern recognon applyng o deec malcous execuables. In he followng secons, we descrbe relaed research n he area of malcous code deecon. Then we llusrae he archecure of our deec model n secon 3. Secon 4 deals he mehod of exracon feaure from program, and sang he deec engne work procedure. Secon 5 deals he mplemenaon and expermen resuls. We sae our plans for fuure work n Secon 6. L. Wang and Y. Jn (Eds.): FSKD 005, LNAI 363, pp , 005. Sprnger-erlag Berln Hedelberg 005

2 Bay New or ks 630 B. Zhang, J. Yn, and J. Hao Relaed Work There have been few aemps o use machne learnng and daa mnng for he purpose of denfyng new or unknown malcous code. In an early aemp, Lo e al. [] conduced an analyss of several programs evdenly by hand and denfed ell-ale sgns, whch hey subsequenly used o fler new programs. Researchers a IBM's T.J.Wason Research Cener have nvesgaed neural neworks for vrus deecon and have ncorporaed a smlar approach for deecng boo-secor vruses no IBM's An-rus sofware [3]. More recenly, nsead of focusng on boo-secor vruses, Schulz e al. [4] used daa mnng mehods, such as naïve Bayes, o deec malcous code. There are oher mehods of guardng agans malcous code, such as obec reconclaon, whch nvolves comparng curren fles and drecores o pas copes. One can also compare crypographc hashes. One can also aud runnng programs and sacally analyze execuables usng pre-defned malcous paerns. These approaches are no based on daa mnng. 3 Model Srucure We frs descrbe a general framework for deecng malcous execuable code. Fgure llusraes he proposed archecure. The framework s dvded no 3 par: Applcaon Server, Deec Server, and rus Deec Frewall based on characer code scannng. Before a fle save o he applcaon server, wll be scanned by he vrus deec frewall. If he fle s nfeced wh vrus hen quaranne. Oherwse f here s no malcous nformaon abou he fle, wll be replcaed copes. Then, one copy wll be sen o he applcaon server, anoher one wll be sen o he deec server based on Fuzzy Paern Recognon (FPR) deec engne. A he followng sage, he fle s feaures s exraced n he deec server. The deec server drves deec engne based on FPR check he copy agan. Accordng o he resul from deec server, f he fle s nfeced wh unknown malcous code, he applcaon server wll be remnd o remove he copy from s applcaon daabase. And hen quaranne n a specal daabase or sen o an exper o analyze by hand. Fg.. Archecure

3 Usng Fuzzy Paern Recognon o Deec Unknown Malcous Execuables Code 63 4 Malcous Code Deec Engne 4. Feaure Exracon Our frs nuon no he problem was o exrac nformaon from he PE execuables ha would dcae s behavor. We choose he Wndows API funcon calls as he man feaure. Los of API funcon calls by racng he programs n he ranng se could be obaned. I s surely ha each API calls play dfferen role on deecng malcous code. When an API call ofen appears n he malcous codes bu seldom n he bengn codes, so plays more mporan role n deecon. Here we use mean square devaon as he man parameer o selec API funcon calls as program s feaure. The mean square devaon beween classes compued as follow: () Tracng each sample program n he ranng se o oban API calls sequence A= {, A A,..., AP },( p), coun each API funcon( A ) frequency A n every malcous execuables. And coun s frequency N A n every bengn execuables N ; N () Compue average frequency EA ( ) and EA ( ) of each API funcon( A ) n malcous execuables se and bengn execuables se as: s EA ( ) = A, n N N EA ( ) = s = A () n = (3) Compue oal mean frequency ( ) where s s he number of malcous execuables,n s he number of bengn execuables. EA of each API funcon A as: N E( A ) + E( A ) E( A ) = () (4) Compue mean square devaon DA ( ) of each API funcon N A as: DA ( ) = ( EA ( ) EA ( )) + ( EA ( ) EA ( )) (3) A he las sage, we sored he API funcon call sequence on DA ( ),and choose he frs -h API funcon as he fuzzy feaure vecor. An example of feaure vecor shows n able. Table. Feaure ecor Ls Sample Program s behavor Search Fle API Funcon Calls FndClose ;FndFrsFleA; FndNexFleA ;FndResourceA DLLS reference KERNEL3.dll

4 63 B. Zhang, J. Yn, and J. Hao 4. Deecon Algorhm The resul of deec a compuer program s only bengn or malcous. We could ge a se of feaure from each sample fle x, gven C s he class se {bengn, malcous}, C denoes bengn, C denoes malcous. Our goal s o deermne wha class s afer he fle s feaure F was obaned. In our mehod, a program fle could be descrbed by fuzzy se. Gven Q= { q, q,..., q n } s he doman of a fuzzy se M = { µ / q, µ / q,..., µ / q } Where n s he number of feaures, µ s a real number whch value s beween[0,], µ / q s he degree of membershp of he es fle whch has feaure q. So he bengn code and malcous code could be descrbe by fuzzy se on doman Q. Gven EA ( ) s he mean frequency of a API funcon call n he malcous fles, EA ( N ) s he mean frequency of a API funcon call n he bengn fles, The malcous fle se s membershp funcon creae from he normal dsrbuon of F dsrbuon as: n n (4) µ 0, EA ( ) < 0 =, ( ) 0 ( EA ( )) ( E( A )) / σ e E A (5) where σ = max{ EA ( ), EA ( ),..., EA ( )}/ 3, s he number of feaures. And he bengn fle se N s membershp funcon s: µ 0, EA ( ) < 0 =, ( ) 0 N N N( EA ( )) ( EA ( N )) / σ N e E A (6) In he same way, he es fle s membershp funcon express as: 0, A < 0 µ M( A ) = ( ) / A σ e, A 0 Durng he ranng sep, we compue he frequency of all feaures over malcous code se. Accordng o membershp funcon µ ( EA ( )),we ge fuzzy se as: = { µ / A, µ / A,..., µ / A} In he same way, we compue he frequency of all feaure over bengn code se. So we ge fuzzy se N as: N = { µ / A, µ / A,..., µ / A} N N N For a es fle M, by racng s API funcon calls frs, we could ge he API call sequence. Then he frequency of all feaure was compued oo. Then { A, A,..., A } was ge, where =88 n our expermen. (7) (8) (9)

5 Usng Fuzzy Paern Recognon o Deec Unknown Malcous Execuables Code 633 Accordng o membershp funcon µ ( A ),we ge fuzzy se M : M M = { µ / A, µ / A,..., µ / A}. (0) On he Second sep, he degree of smlary ψ ( M, ) beween M and, ψ ( M, N ) beween M and N were compued as follow: A B ψ ( A, B ) = ( ( µ µ ) ) = () Where ψ ( AB, ) s Eucld degree of smlary. A he las sep, we can deermne whch class he es fle s by Theorem. Theorem : f sasfy he follow equaon: ψ( A, B ) = ψ( A, B ), hen classfy A n and B n he same class. Where A, B ( =,,..., n) s fuzzy ses, ψ ( AB, ) s he Eucld degree of smlary beween A and B. 5 Expermen Resuls We esmae our resuls over daa se n able. The daa se conssng of PE forma execuables was composed of 43 bengn programs and 09 malcous execuables.the malcous execuables were downloaded from hp://vx.nelux.org and hp:// The clean programs were gahered from a freshly nsalled Wndows 000 server machne. Each sample was labeled by a commercal vrus scanner wh he correc class label(malcous or bengn) for our mehod. Afer verfcaon of he daa se he nex sep of our mehod was o exrac feaures from he programs usng API racng ool-apispy.exe ha we desgned. To evaluae our sysem we were neresed n several quanes: (). False Negave, he number of malcous execuable examples classfed as bengn;(). False Posves, he number of bengn programs classfed as malcous execuables. We were neresed n he deecon rae of he classfer. In our case hs was he percenage of he oal malcous programs labeled malcous. We were also neresed n he false posve rae. Ths was he percenage of bengn programs whch were labeled as malcous, also called false alarms. For he algorhms we ploed he deecon rae usng Recever Operang Characersc(ROC) curves. The ROC curves n Fg. show ha our mehod had he lowes False Negave rae, 4.45%. Noce ha he curve s down slowly when he number of samples ncreases. Ths s very f o deec malcous code when he malcous sample obaned s dffcul. In anoher expermens[5], we had used a algorhm based on K Neares Neghbor(KNN) o classfy he daa se n able. The resul s shown n Fg.3.Tha algorhm had he lowes false posve rae, 4.8%. The Fuzzy Paern Recognon algorhm(fpr) has beer deecon raes han he algorhm based on KNN. Bu he KNN algorhm occupes less compue resources han FPR. The rade-off beween deec rae and sysem overhead mus be hnk over n praccal applcaon.

6 634 B. Zhang, J. Yn, and J. Hao Table. Daase n expermen Sample space Tranng se Tes Se Bengn Code Malcous Code sum Fg.. Fuzzy Paern Recognon ROC Fg. 3. K Neares Neghbor ROC 6 Concluson We presened a mehod for deecng prevously undeecable malcous execuables. As our knowledge, hs s he frs me ha usng fuzzy paern recognon algorhm o deec compuer vrus. However, he rae of error aler seems hgh n our expermen. So fuure work nvolves exendng our learnng algorhms o beer ulze API call sequences and oher feaure of vrus. We are plannng o use Neural nework o gan hgher accuracy and deecon raes. We also would lke o mplemen he sysem on a nework of compuers o evaluae s performance n erms of me and space n real world envronmens. Fnally, we are plannng on esng hs mehod over a larger se of malcous and bengn execuables. References. McGraw,G., Morse,G.: Aackng malcous code: A repor o he Infosec Research Councl. IEEE Sofware. 5(000) Lo,R., Lev,K., Olsson,R.: MCF: A malcous code fler. Compuers & Secury.4 (995) Tesauro,G., Kephar,J., Sorkn,G.: Neural neworks for compuer vrus recognon. IEEE Exper. (996) Schulz,M., Eskn,E., Zadok,E., Solfo,S.: Daa mnng mehods for deecon of new malcous execuables. In: Proceedngs of he IEEE Symposum on Secury and Prvacy. IEEE Press, Los Alamos, CA, (00) ZHANG Boyun,YIN Janpng,ZHANG Dngxng,HAO Jngbo.:Unkown compuer vrus deecon based on K-neares neghbor algorhm. Compuer Engneerng and Applcaons. 6(005)7-0