Jifeng Zuo School of Science, Agricultural University of Hebei, Baoding , Hebei,China

Transcription

1 Rev. Téc. Ing. Unv. Zula. Vol. 39, Nº 7, 76 80, 016 do: / Proof on Decson Tree Algorthm Jfeng Zuo School of Scence, Agrcultural Unversty of Hebe, Baodng , Hebe,Chna Pepe Ja Basc Course Department, Hebe Fnance Unversty, Baodng , Hebe,Chna Abstract The relaton between weghted entropy of the unon of several attrbute values and the sum of the weghted entropy of the sngle attrbute value s studed. The proof s gven for the concluson that the weghted entropy of the unon of several attrbute values s not less than the sum of the weghted entropy of the sngle attrbute value, and the theoretcal foundaton for ID3 algorthm s presented. The results of the experment conform the concluson. Key words: Entropy, ID3 Algorthm, Decson tree, Condtonal Attrbutes, Decson Attrbutes 1. INTRODUCTION Decson tree (Breman and Fredman, 1984 method s applcaton wde of nduce and reason cally one of the algorthm. Decson tree, be at arthms accordng to carry on decson classfcaton make use of tree of structure data record carry on classfcaton, among them a leaf's crunode of tree representatve match a certan condton of the attrbute set, accordng to dfferent value of the attrbute establshment decson tree of each branch; The structure each subtree of statures node passed to return later on. ID3 algorthm s a knd of decson tree of typcal model to nduce calculate way and t be what Qunlan (J.R. Qunlan, 1986 put forward frst. Its core thought s to make use of entropy prncple (Deng, and Tan, 014, choce the entropy be mnmum of attrbute be classfcaton attrbute, pass to return a ground of expand the brunch of decson tree, completon decson tree of structure, creaton be a set of classfcaton rule. Among them the entropy of the attrbute defnton s the attrbute lst attrbute value of weghted entropy t wth, n the process of born tree, each crunode only have an attrbute value (weghted entropy homoy of attrbute value see make an attrbute value, ts advantage s adopton from crest get down don't return to trace of the strategy search all of attrbute space, the algorthm of establshment decson tree s smple, the depth s small, classfcaton speed quck. Wth Table 1 s example, Table 1 medum have 14 examples, each example mply 4 classfcaton attrbute, Outlook, Temperature, Humdty, Wndy and a decson attrbute PlayTenns. The bass ID3 algorthm, can born dagram 1 medum of decson tree. Table 1. Opposte at the target conceptran of the PlayTenns example Day Outlook Temperature Humdty Wndy PlayTenns D1 Sunny Hot Hgh Weak No D Sunny Hot Hgh Strong No D3 Overcast Hot Hgh Weak D4 Ran Mld Hgh Weak D5 Ran Cool Normal Weak D6 Ran Cool Normal Strong No D7 Overcast Cool Normal Strong D8 Sunny Mld Hgh Weak No D9 Sunny Cool Normal Weak D10 Ran Mld Normal Weak D11 Sunny Mld Normal Strong D1 Overcast Mld Hgh Strong D13 Overcast Hot Normal Weak D14 Ran Mld Hgh Strong No Creaton as follows of classfcaton rule: Rule 1: If Outlook=Sunny, and Humdty=Normal, be n keepng wth to play ball; Rule : If Outlook=Sunny, and Humdty=Hgh, then sn't sutable for to play ball; Rule 3: If Outlook=Overcast, be n keepng wth to play ball; Rule 4: If Outlook=Ran, and Wnd=Weak, be n keepng wth to play ball; 76

2 Rev. Téc. Ing. Unv. Zula. Vol. 39, Nº 7, 76 80, 016 Rule 5: If, Outlook=Ran, and Wnd=Strong, then sn't sutable for to play ball; Outlook Sunny Overcas Ran Humdt Wnd Hgh Normal Strong Weak No No Fgure 1. From Fgure 1 born decson tree Ths text frst research a few some attrbute value of an attrbute combne of weghted entropy t wth the attrbute sngle attrbute value of weghted entropy t wth of relaton, get such as draw a concluson: a few some attrbute value of an attrbute combne of weghted entropy t and be no smaller than that attrbute lst attrbute value of weghted entropy t wth. Then dd an experment n the Pma database, from the experment also certfcate conclude theory of accuracy, provded theores for the ratonalty of ID3 algorthm and experment up of certfcate.. SOME BASIC CONCEPT AND BASIC METHOD Defnton entropy s the generous character of a knd of ndetermnaton degree of nformaton, suppose a system S have all a rate to dstrbute p { p}(0 p 1, 1,,, n the entropy (Shannon entropy of the system S defnton s n 1 ES ( pln p In - types only the classfcaton the problem make a set S, assumpton S from target value s postve sample or target value for negatve sample consttute and then the entropy of the set S s, ES ( p p p p, among them, p s n the S of postve the example s n the S share of comparson, p s n the S of negatve the example s n the S share of comparson. The ID3 algorthm prncple: Establsh the subset PN to mply p postve example and n ant- example, the possblty of an example belong p to postve example set PE s, p n,the possblty belong to an ant- example set NE s n p n A decson tree can be thnk the nformaton source for postve example set and the ant- example set, as a result creaton these nformaton of expectaton nformaton s I( p, n p ( p n ( n p n pn pn pn Establsh the attrbute A take a value for { A1, A,, A n }, they PN the classfcaton gather for r subset. Establsh PN mply p postve example, n ant- example, son tree PN demand of expectaton nformaton s I ( p, n.and the tree that regards the A as the root demand of expectaton nformaton for each son tree demand of expectaton add of nformaton power average value, namely ( r p n EA Ip (, n p n But take A as a root carry on classfcaton ncome of the nformaton ncrease a beneft for 1 gan( A I ( p, n E( A 77

3 Rev. Téc. Ing. Unv. Zula. Vol. 39, Nº 7, 76 80, 016 A good classfcaton attrbute wll make the nformaton ncrease a beneft to enlarge The ID3 choce make the Gan(A have the bggest attrbute A*Be a node, to the A*of dssmlarty take value rghtness should of the r subset PN gather to pass to return to adjust to use above-mentoned born process born sub- node..1. Man Concluson and Certfcate Underneath gve two attrbute value combne of entropy wth ths two attrbute value weghted entropy of wth t of relaton axoms, then from mathematcs nductve method can get general concluson. Axoms establsh S 1, S s two attrbute set of some one attrbute, S 1 ( m 1, n 1, S ( m, n, among them m ( 1, mean S ( 1, have m ( 1, postve example,, n ( 1, mean S ( 1, have n ( 1, negatve example. Suppose ES ( ( 1, mean the entropy of S ( 1,, S mean the number of example ncluded n the set S,So S ( ( ( S ES ES ES S S S S S Certfcate: We frst certfcate x ( xc y ( yd ( x y ( x ycd x x y y( x y ( x y (1 ( Among them. x 0, y 0, c 0, d 0. Then, agan the certfcate(1 type establsh. Make ucd (, x( xc y( yd ( x y( x ycd (3 1 x x y From u c (, c d ( ln x ycd But at that tme So, be x =0 can get c, 1 x x y 1 xy u c ( 0 ln ( xc ( x ycd ln ( xc ( x y, the type (3 attan the bggest value,, take to type ( 3, sortng can get : umax (, c d x x y y( x y( x y 1 y x y The same reason, u d (, c d ( 0,can get,but at that tme ln yd x ycd, 3 1 y x y 1 x u d ( 0 ln ( yd ( x ycd ln y( xc ( x y So, be, the type (3 attan the bggest value,, take to type ( 3, sortng can get : u (, c d x x y y( x y( x y max umax (, c d x x y y( x y( x y So, at that tme u (, c d x x y y( x y( x y. Thus the type ( establsh. max Make 1, 1,, x m y n c m d n,take to type(can get m m n n ( m n ( m n m ( m m n ( n n ( m n ( m n m n (4 78

4 Rev. Téc. Ing. Unv. Zula. Vol. 39, Nº 7, 76 80, 016 Together reason, make x m, y n, c m1, d n 1,take to type(can get m m n n ( m n ( m n m ( m m n ( n n ( m n ( m n m n (5 add(4to(5, sortng can get m n m n m n m n m1n1 m1n1 m n m n m m n n ( m m ( n n m1m n1n m1m n1n (6 So, can get m1 n1 m1 m1 n1 n1 ( m m n n m n m n m n m n m n m m n n ( m m n n m n m n m n m n 1 1 m m m m m m n n m m n n n n n n m m n n m m n n (7 S1 Establsh ( 1 ( ( 1 S ES ES ES S S S S S 1 1 If S X but S S1S,from(1 can get thus. S1 S S1 S ES ( 1 ES ( ES ( 1S X X X (8 Thus, f X S1S S 3,so S1 S S3 S1 S S3 ES ( 1 ES ( ES ( 3 ES ( 1S ES ( 3 X X X X X (9 From mathematcs nducton can get, a few some attrbute value of an attrbute combne of weghted entropy t and be no smaller than that attrbute lst attrbute value of weghted entropy t wth. Ths concluson enuncaton, the ID3 algorthm be partal to the attrbute value more attrbute. From certfcate process can get, only have be two set of postve example wth negatve example pece become comparson, ths s two set of entropy equal and equal t unon set of entropy thus for cent merger the algorthm provded theores bass... Experment Analyss We start experment analyss wth the Pma database, should database n total have 768 sample, each sample contan 8 condton attrbute(each condton attrbute have several attrbute value, a decson attrbute, decson the attrbute have two attrbute value.(belong to two types of classfcaton problem We usage t cross verfcaton method, every tme from the data concentraton random take out 400 sample, agan wll ths 400 sample average be dvded n to not and mutually hand over of 0 set, start experment, total carred on 10 tmes. Experment step: 1 Adopton the mnmum entropy be long-lost to turn a method[8] arthms accordng to carry on prepare processng; Calculaton the sum of weghted entropy of the frst attrbute. 79

5 Rev. Téc. Ing. Unv. Zula. Vol. 39, Nº 7, 76 80, In frst value, unted the two attrbute value whch value s 1,the other s 0. 4 Unted the two attrbute value whch value s 1,the other s 0, then, calculaton the sum of weghted entropy for the frst value. 5 Compared the two sum of the weghted entropy. Experment result such as Table show, among the frst experment, 0 sets of n the data have 18 sets of data attrbute value combne empress of weghted entropy t wth bg at the attrbute value combne front of weghted entropy t wth, sets of data attrbute value combne empress of weghted entropy t wth equal an attrbute value combne front of weghted entropy t wth. Experment result enuncaton, majorty crcumstance, attrbute value combne empress of weghted entropy t wth bg at the attrbute value combne front of weghted entropy t wth, attrbute value combne empress of weghted entropy t wth bggest equal combne prevous of. Ths mutually ft together wth the concluson of our certfcate. Table. Experment comparson Experment number Attrbute value D after Attrbute value the D after of tmes combne bg n combne front combne equal combne front 1 18/0 /0 19/0 1/0 3 18/0 /0 4 17/0 3/0 5 19/0 1/0 6 0/0 0/0 7 18/0 /0 8 15/0 5/ / /0 3/0 3. CONCLUSIONS Ths text pass contrast an attrbute of a few some attrbute value combne of weghted entropy t wth should attrbute lst attrbute value of weghted entropy t wth of sze, provde theores foundaton for the ratonalty of ID3 algorthm, and make use of experment analyss verfcaton our concluson. Thus elucdaton, use the ID3 algorthm born decson tree, choose to use lst attrbute value better than several attrbute value of combne. Acknowledgments Ths work s supported by the Project of Baodng scence and Technoy Bureau (No.13zs00 and Fund project of Agrcultural Unversty of Hebe Performance optmzaton of sngle class classfer (No.LG01613, Scence research youth fund project of Hebe Provnce (No.QN REFERENCES Breman L, Fredman J (1984 Classfcaton and Regresson Trees. Chapman & Hall :New York. Fayyad U.M., Iran K.B.(1993 Mult-nterval dscretzaton of contnuous-valued attrbutes for classfcaton learnng, Proceedngs of the 13th Internatonal Jont Conference on Artfcal Intellgence, pp J.R. Qunlan(1979 Dscoverng Rule by Inducton from large Collecton of Examples. Expert Systems n the Mcro Electroncs Age: Ednburgh Unversty Press. J.R. Qunlan(1986 Inducton of Decson Tree. Machne learnng, pp N. Y. Deng, Y. J. Tan, C. H. Zhang (014 Support Vector Machnes.Optmzaton Based Theory, Algorthms, and Extensons, CRC Press. Song We (005 Informaton theory foundaton and applcaton, Chn Hua Unversty publsher: Pekng Y. Tan, Y. Sh, X. Lu (014 Recent advances on support vector machnes research, Technocal and Economc Development of Economy,18 (1,pp Zengchang Qn, Jonathan Lawry (013 Decson tree learnng wth fuzzy labels, Informaton Scence, 17(3, pp