An Effective Feature Selection Scheme via Genetic Algorithm Using Mutual Information 1

Transcription

1 An Effectve Feature Selecton Scheme va Genetc Algorthm Ung Mutual Informaton Chunka K. Zhang and Hong Hu Member IEEE, Department of Mechancal Engneerng and Automaton, Harbn Inttute of Technology, Shenzhen Graduate School, Shenzhen, Chna, Abtract. In the artfcal neural network (A), feature electon a wellreearched problem, whch can mprove the network performance and peed up the tranng of the network. The tattcal-baed method and the artfcal ntellgence-baed method have been wdely ued to feature electon, and the latter are more attractve. In th paper, ung genetc algorthm (GA) combnng wth mutual nformaton (MI) to evolve a nearoptmal nput feature ubet for A propoed, n whch mutual nformaton between each nput and each output of the data et employed n mutaton n evolutonary proce to purpoefully gude earch drecton baed on ome crteron. By examnng the forecatng at the Autralan Bureau of Meteorology, the mulaton of three dfferent method of feature electon how that the propoed method can reduce the dmenonalty of nput, peed up the tranng of the network and get better performance. Introducton In the artfcal neural network (A), feature electon a well-reearched problem, amed at reducng the dmenonalty and noe n nput et to mprove the network performance and peed up the tranng of the network []. Many algorthm for feature electon have been propoed. Conventonal method are baed on the tattcal tool, uch a the partal F-tet, correlaton coeffcent, redual mean quare [,3,4,5] and mutual nformaton (MI) [6,7,8,9]. Although the tattcal-baed feature electon technque are wdely ued, they uffer from many lmtaton [0]. Frtly, mot of them are computatonally expenve, becaue the comparon of all feature ubet equvalent to a combnatoral problem whoe ze exponentally ncreae wth the growng number of feature. Secondly, the elected feature ubet cannot be guaranteed optmal. For example, n the mutual nformaton method, electng a fxed number of nput from a ranked lt contng of combnaton along wth ngle entre omewhat problematcal, and once a feature added at an early tep, t cannot be removed although t may not conttute the bet ubet of feature n conjuncton wth the later elected feature. Fnally, there are a number of parameter that need to be et a pror. For example, the number of feature added or removed, the gnfcance level for electng feature and the fnal feature ze. Th work upported by Hgh-tech Indutralzaton Specal Reearch Project of Chna L. Wang and Y. Jn (Ed.): FSKD 005, LAI 364, pp , 005. Sprnger-Verlag Berln Hedelberg 005

2 74 C.K. Zhang and H. Hu Becaue the problem of feature electon can be formulated a a earch problem to fnd a nearoptmal nput ubet, o the artfcal ntellgence technque, uch a genetc algorthm (GA), ued to elect the optmal ubet of feature [,, 3]. In contrat wth the tattcal-baed method, the artfcal ntellgence-baed method are more attractve, a they can fnd nearoptmal feature ubet n lower computatonal cot and the earch proce nvolve no uer electable parameter, uch a the fnal feature ze and the gnfcaton level etc.. In addton, they have the potental to multaneou network evoluton and feature electon. In mot GA-baed method, only the correct recognton rate of a certan neural network utlzed to gude the earch drecton. In [4], Il-Seok Oh propoed the hybrd GA for feature electon, whch embed local earch operaton nto the mple GA, but ueful nformaton uch a tattcal nformaton between nput and output n data et don t be added n earch proce. In th paper, we propoed a new feature electon cheme for A va genetc algorthm ung mutual nformaton. In th method, mutual nformaton (MI) between nput and output employed n mutaton n GA to purpoefully gude the evolutonary earch drecton baed on ome crteron, whch can peed up the earch proce and get better performance. By examnng the forecatng at the Autralan Bureau of Meteorology [5], the mulaton of three dfferent method of feature electon how that the propoed method can reduce the dmenonalty of nput, peed up the tranng of the network and get better performance. The ret of th paper organzed a follow. Secton decrbe mutual nformaton (MI) and GA. Secton 3 the hybrd of GA and MI ued to evolve an optmum nput ubet for an A. Secton 4 preent expermental reult n a real forecatng problem. The paper concluded n Secton 5. Background. Defnton of Mutual Informaton In the nformaton theory founded by Shannon [6], the uncertanty of a random varable C meaured by entropy H (C). For two varable X and C, the condtonal entropy H ( C X ) meaure the uncertanty about C when X known, and MI, I ( X ; C), meaure the certanty about C that reolved by X. Apparently, the relaton of H (C), H ( C X ) and I ( X ; C) : or, equvalently, H ( C) H ( C X ) + I( X ; C) = () I ( X ; C) H ( C) H ( C X ) =, A we know, the goal of tranng clafcaton model to reduce the uncertanty about predcton on cla label C for the known obervaton X a much a poble. In term of the mutual nformaton, the purpoe jut to ncreae MI I ( X ; C)

3 An Effectve Feature Selecton Scheme va Genetc Algorthm Ung Mutual Informaton 75 a much a poble, and the goal of feature electon naturally to acheve the hgher I X wth the fewer feature. ( ; C) Wth the entropy defned by Shannon, the pror entropy of cla varable C expreed a where (c) H C H C) = P( c)log P( c) ( () c C P repreent the probablty of C, whle the condtonal entropy ( X ) H C X ) = x)( c x)log c x) ) dx The MI between X and C ( (3) I x X ; C) = x c C c C c, x) c, x)log dx P( c) x) ( (4) Mutual nformaton can, n prncple, be calculated exactly f the probablty denty functon of the data known. Exact calculaton have been made for the Gauan probablty denty functon. However, n mot cae the data not dtrbuted n a fxed pattern and the mutual nformaton ha to be etmated. In th tudy, the mutual nformaton between each nput and each output of the data et etmated ung Fraer & Swnney method [9]. The mutual nformaton of ndependent varable zero, but large between two trongly dependent varable wth the maxmum poble value dependng on the ze of the data et. And th aume that all the nput are ndependent and that no output n fact a complex functon of two or more of the nput varable.. Genetc Algorthm GA an effcent earch method due to t nherent parallelm and powerful capablty of earchng complex pace baed on the mechanc of natural electon and populaton genetc. The method of ung GA to elect nput feature n the neural network traghtforward. In GA, every canddate feature mapped nto ndvdual (bnary chromoome) where a bt (gene) denote the correpondng feature elected and a bt of 0 (gene) denote the feature elmnated. Succeve populaton are generated ung a breedng proce that favor ftter ndvdual. The ftne of an ndvdual condered a meaure of the ucce of the nput vector. Indvdual wth hgher ftne wll have a hgher probablty of contrbutng to the offprng n the next generaton ( Survval of the Fttet ). There are three man operator that can nteract to produce the next generaton. In replcaton ndvdual trng are coped drectly nto the next generaton. The hgher the ftne value of an ndvdual, the hgher the probablty that that ndvdual wll be coped. ew ndvdual are produced by matng extng ndvdual. The probablty that a trng wll be choen a a parent ftne dependent. A number of croover

4 76 C.K. Zhang and H. Hu pont are randomly choen along the trng. A chld produced by copyng from one parent untl a croover pont reached, copyng then wtchng to the other parent and repeatng th proce a often a requred. An bt trng can have anythng from to - croover pont. Strng produced by ether reproducton or croover may then be mutated. Th nvolve randomly flppng the tate of one or more bt. Mutaton needed o new generaton are more than jut a reorganzaton of extng genetc materal. After a new generaton produced, each ndvdual evaluated and the proce repeated untl a atfactory oluton reached. The procedure of GA for feature electon expreed a follow: Procedure of genetc algorthm for feature electon Intalzaton Populaton ze P Intal populaton wth ubet of Y P c Croover probablty P m Mutaton probablty T Maxmum number of generaton k 0 Evoluton Evaluaton of ftne of P whle ( k < T and P doe not converge) do Breeder Selecton Croover wth P c Mutaton wth P m Evaluaton of ftne of P Replcaton Dperal + k k 3 The Propoed Method for Feature Selecton In order to reduce tme of calculatng MI between ngle nput and output n the whole data et, we randomly elect ome data from data et wth probablty 0.5 to contruct a data et named MI et. Ung Fraer & Swnney method, the mutual nformaton x between each canddate nput and each output n MI et etmated, whch contruct a data et D = { x, =,..., }, x repreent the mutual nformaton of th canddate nput, and mean there are canddate nput. Then calculate the mathematcal tattc of x : the mean x and tandard devaton

5 An Effectve Feature Selecton Scheme va Genetc Algorthm Ung Mutual Informaton 77 x = x = (5) = = ( x x ) (6) And defne the three et whch atfy D = D U D U D3 : D = { x x x > }, D = { x x x } D3 = { x x x < } In GA, we ue mutual nformaton between each canddate nput and each output to gude the mutaton baed on ome crteron, a follow: where g = 0 rand x D x D 3 (7) x D g repreent th gene n a bnary chromoome, t mean th canddate nput. x of th canddate nput belong to D, t mean t a If the mutual nformaton hghly correlated nput for each output, o nclude t nto nput feature ubet; f the mutual nformaton x of th canddate nput belong to D, t mean t a general correlated nput for each output, o randomly nclude t nto nput feature ubet; If the mutual nformaton x of th canddate nput belong to D 3, t mean t lttle correlated nput for each output, o exclude t from nput feature ubet. The procedure of the propoed method for feature electon ame a the procedure of GA for feature electon except the tep of mutaton wth P. Mutaton wth P m x of th canddate nput belong to If feature ubet; If nto nput feature ubet; If feature ubet. x of th canddate nput belong to x of th canddate nput belong to 3 m D, nclude t nto nput D, randomly nclude t D, exclude t from nput

6 78 C.K. Zhang and H. Hu 4 Expermental Stude The temperature data for Autrala wa taken from the TOVS ntrument equpped OAA atellte n 995 [3]. Infrared oundng of 30km horzontal reoluton wa upplemented wth mcrowave oundng of 50 km horzontal reoluton. Th data et wa ued to evaluate the technque for electng the nput ubet. A number of ngle output network were developed, each etmatng the actual temperature at one of 4 preure level (000, 700, 300 & 50 hpa) gven the radance meaured by atellte. Thee are four of the tandard preure level (level, 3, 6 and 9) meaured by atellte and radoonde ounder. The nput et of TOVS readng to be ued by thee network wa extracted ung each of the three technque: GA, MI [6] and the propoed method. The approprate target output temperature wa provded by collocated radoonde meaurement. In MI method, a common nput vector length of 8 wa ued a ntal expermentaton had proved th to be a utable value. In GA and the propoed method, =50, T =60, P c =0.6 and P m =0.0. And the m-- network ue a learnng rate of 0. and momentum of 0.8 for 0,000 teraton, where m repreent the number of nput. And the ftne functon defned to be / RMSE, and the root mean quare error (RMSE) calculated by where ( ) / RMSE = ( Y Yr ) (8) Y r the dered target value, and Y the output of network. After electng an optmal nput ubet ung one of the above technque, thee nput were aeed by mean of an evaluaton neural network whoe archtecture wa choen baed on ntal experment. The network ued hdden neuron and wa traned ung fxed parameter to facltate comparon between the varou technque. It wa traned for 000 pae through the data et ung a learnng rate of 0. and a momentum of 0.8. The network wa teted after each pa though the tranng data wth the bet reult beng recorded. The overall performance of th tetng network wa aumed to reflect the appropratene of th partcular electon of nput. The reult reported are the mean RMSE value obtaned from tranng the ten evaluaton network at each level and hould be a reaonable reflecton of the nherent worth of the nput electon. The reult ung the full nput et (all avalable nput) are ncluded n the table for comparon. Mean of RMSE (K) derved from all 3 technque and ung all nput for level,3, 6 & 9 ndcated n Table, and elected nput ubet ndcated n Table. Table. Mean of RMSE (K) derved from all 3 technque and ung all nput Full GA MI the propoed method Level Level Level Level

7 An Effectve Feature Selecton Scheme va Genetc Algorthm Ung Mutual Informaton 79 Table. Selected nput ubet for varou level GA MI the propoed method Level, 3, 7, 5, 7, 8,, 0, 4,,, 4, 3, 8, 4, 0,, 4, 9, 0, 3, Level 3 0, 3, 6, 8, 0,, 4, 5, 7, 8, 9, 4,, 7, 5, 0, 3,, 9, 3, 4, 7, 0,, 4, 5, 7, 8, 9, 0, Level 6 3, 6, 8, 4, 8, 4, 0, 4, 3, 5, 3, 4, 0, 3, 4, 8, 3, Level 9 0, 4, 6, 8, 4, 6, 7, 8, 3, 4, 5, 4, 8, 6,, 0 4, 5, 6, 8,, 4, 0 Table ndcate that the propoed method exhbted better performance than the other technque at all level. GA wa margnally better than MI, outperformng t n level and 3. Level 3 nteretng n that all three technque produced network wth wore performance, epecally MI. Th eem to ndcate that the predctve capablty at th level pread more acro the nput there le redundancy of nformaton. It hould be noted that at th dffcult level GA and the propoed method outperformed MI. Table ndcate that although there conderable mlarty between GA and the propoed method there are ubtantal dfference between the nput elected, and GA and the propoed method elected the dfferent number of nput for all level, epecally n level 3, the number of nput large than MI, whch explan the reaon why the performance of them better than MI. In contrat wth GA, the propoed method can get more lttle number of nput wthout lo of performance, and the content of nput ubet the hybrd of that GA and MI. In addton, t wa found that there wa very lttle ncreae n performance after 43 generaton for the propoed method, but 56 generaton for GA. 5 Concluon We propoed an effectve feature electon cheme ung genetc algorthm (GA) combnng wth mutual nformaton (MI), n whch mutual nformaton between each nput and each output of the data et employed n mutaton n evolutonary proce to purpoefully gude earch drecton baed on ome crteron. By examnng the forecatng at the Autralan Bureau of Meteorology, the mulaton of three dfferent method of feature electon how that the propoed method can reduce the dmenonalty of nput, peed up the tranng of the network and get better performance. Reference. Dah, M., Lu, H.: Feature electon for clafcaton. Intellgent Data Analy, vol. (997) D.C. Montgomery and E.A. Peck: Introducton to Lnear Regreon Analy. John Wley & Son, ew York (98). 3. A. Sen and M. Servatava: Regreon Analy: Theory, Method, and Applcaton. Sprnger-Verlag, ew York (990).

8 80 C.K. Zhang and H. Hu 4. Holz, H. J. and Loew, M. H.: Relatve feature mportance: A clafer-ndependent approach to feature electon. In: Gelema, E. S. and Kanal, L.. (ed.), Pattern Recognton n Practce IV. Amterdam: Elever (994) H. Wang, D. Bell, F. Murtagh: Automatc approach to feature ubet electon baed on relevance. IEEE Tran. PAMI (3). (999) Belnda Cho, Tm Hendtla, and Kevn Bluff: A Comparon of eural etwork Input Vector Selecton Technque. LAI 309. Sprnger-Verlag Berln Hedelberg (004) Kwak, C-H. Cho: Input feature electon by mutual nformaton baed on parzen wndow. IEEE Tran. PAMI 4 (). (00) D. Huang,Tommy W.S. Chow: Effectve feature electon cheme ung mutual nformaton. eurocomputng vol. 63. (005) A.M. Fraer & H.L. Swnney: Independent Coordnate for Strange Attractor from Mutual Informaton. Phycal Revew A, Vol. 33/. (986) T. Cba, F.F. Soule, P. Gallnar and S. Raudy: Varable electon wth neural network. eurocomputng. (996) W. Sedlech and J. Sklanky: A note on genetc algorthm for large-cale feature electon. Pattern Recognton Letter, 0. (989) C. Emmanould, A. Hunter, J. Macntyre and C. Cox: Selectng feature n neurofuzzy modelng by multobjectve genetc algorthm. Artfcal eural etwork, (999) J.H. Yang and V. Honavar: Feature Subet Selecton Ung a Genetc Algorthm. IEEE Intellgent Sytem, vol. 3, no.. (998) Oh, I.-S., Lee, J.-S., Moon, B.-R.: Hybrd genetc algorthm for feature electon. IEEE Tran. Pattern Analy and Machne Intellgence 6. (004) J. LeMarhall: An Intercomparon of Temperature and Moture Feld Derved from TIROS Operatonal Vertcal Sounder Data by Dfferent Retreval Technque. Part I: Bac Stattc. Journal of Appled Meteorology, Vol 7. (988) T.M. Cover, J.A.: Thoma: Element of Informaton Theory. Wley, ew York, (99).