Research on the Massive Data Classification Method in Large Scale Computer Information Management huangyun

Transcription

1 Itratoa Crc o Automato, Mchaca Cotro ad Computatoa Egrg (AMCCE 05) Rsarch o th Massv Data Cassfcato Mthod Larg Sca Computr Iformato Maagmt huagyu Chogqg ctroc grg Carr Acadmy, Chogqg 4733, Cha Kywords: arg sca computr formato maagmt; massv data; Baysa bf twor; th tact udrstadg Abstract. I th procss th massv data cassfcato arg-sca computr formato maagmt systm, du to th arg amout data, ad cudd arg umbr fatur data, th corrato data s rducd, rsutg ow ffccy computr oprato. A mod for massv data dpth cassfcato mg basd o bf twor s put forward. Accordg to th rato btw th probabts data a th data doma, th corrato btw owdg ad data doma ca b frrd. Through th trag samp st fd th most sutab Baysa bf twor for th samp data, th accordg to th possb maagmt structur ad th tact udrstadg dgr btw data samps, th optma souto wth data maagmt structur arg sca computr. Th xprmta rsuts show that, usg th mprovd agorthm for massv data cassfcato procssg, ca mprov th accuracy cassfcato, ad achv satsfactory rsuts. Itroducto Data cassfcato s th cor part computr data xtracto []. By usg th mthod data cassfcato, t ca ma rasoab cassfcato th massv data accordg to th usr s ds, so as to raz th accurat xtracto th ffctv twor data, mprov th oprato ffccy twor [], th mthod has a hgh vau appcato th fd databas fds, bcomg th hot topc rqurd rsarch fd data maagmt arg computr [3]. Currty, data cassfcato agorthm may cuds th mthods basd o th support vctor mach, wavt trasform ad th data ga agorthm [4-6]. Amog thm, th most commoy usd s th wavt trasform agorthm. I ordr to avod th dfcts tradtoa mthods, a mod for massv data dpth cassfcato mg basd o bf twor s put forward. Accordg to th rato btw th probabty data a th data doma ad possb data maagmt structur ad th tact udrstadg btw samps, th optma cassfcato data wth th data maagmt structur th arg computr s obtad. Th xprmta rsuts show that, usg th mprovd agorthm for massv data cassfcato procssg, ca mprov th accuracy cassfcato, ad achv satsfactory rsuts. Th prcp aayss massv twor data cassfcato Massv twor data cassfcato s a powrfu guarat twor commucato. Usuay, th dstrbuto bg data twor data s th orma dstrbuto, whch ca b dscrbd by ~ N(0, δ ). Usg th foowg formua cacuat th trasformato paramtrs arg data twor: Z = Y + N(0, δ ) () Th twor data ar tratd by usg wavt trasform, th data s st subjct to th orma dstrbuto, whch ca b dscrbd usg th foowg formua: Z = z = y + N(0, σ ) = Y + N(0, σ ) () It ca b s that th possbty twor data subjctd to orma dstrbuto th trva ( ν 3 σν, + 3 σ) s so, th data formato maagmt th arg sca computr 05. Th authors - Pubshd by Atats Prss 093

2 ca b dvdd to two sctos accordg to wavt trasform paramtrs. Assumg that 3 > σ, th dgr mportac ths data s hghr, othrws, t s owr. Amog thm, s th wavt trasform paramtr cudd data. U U If 3σ U, 3σ U ad U 3σ U U, U ca b obtad, ad fay, th thrshod wavt trasform procssg s acqurd. I th abov mtod status, usg th foowg formua ca ma wavt trasform procssg: wh 3 σ < U, 3σ ad σ σ 3 3, U U sg( ), 3s = sg( ) ( U), U < 3s 0, < U (3) (3 σ ) σ ca b obtad, ad fay, th thrshod (3 σ ) U σ wavt trasform procssg s acqurd.i th abov mtod status, usg th foowg formua ca ma wavt trasform procssg: (3 s ) sg( ), 3s = 0, < 3s (4) Th mod for dpth cassfcato mg massv data basd o Baysa bf twor Baysa bf twor. Dfto :sttg a arbtrary varab st x = { x, x, x}, f a combato cotgt probabty o x s dsprs, th t ca b dfd usg Baysa bf twor as foows: B = G, θ (5) Amog ths, s a m dmsoa vctor quatty; G s a drctd acycc graph, th arcs rprst a fuctoa dpdc; θ rprsts a group paramtr to ma quatzato twor. Dfto : If thr s a arc from th varab Y to, th Y s 's parts or th drct prcursor, wh s th succssor Y. Oc gv ts parts, a varabs acycc graph th graph dpdt th o-succssor ods, a varabs parts G ar as a st P ( ) a. Th massv data mg procss basd o Baysa bf twor. Sttg a group data trag samp D = { x, x,, x}, x s s xamp, through stmato fucto S( BD ) fd a Baysa bf twor whch s sutab for ths samp. Usg fucto S( BD ) ca stmat th tact udrstadg dgr btw a th possb maagmt structurs ad data samps, to th optma cassfcato data. Baysa bf twor agorthms ar dscrbd as foows: Iput: massv data trag samp st D = { x, x,, x },taz twor B 0, ad vauat th fucto S( B D) = S( Pa( ), D), paramtr Output:th optma twor from,,, to () comprss: accordg to D ad B, th us caddat comprsso, from,,,, sct a caddat farthr st C ( C ) for. hr, a dgraph H = ( xe, ) s dfd, whr, { j,, j } E= j C. () maxmzato: fd a Baysa bf twor B = G, Θ whch ca maxmzd vauat th 094

3 fucto SB ( D, ) whr,,, C G H Pa ( ) C ad rtur B. Through aayzg vauato fucto, ad appyg caddat comprss agorthm, t ca compt th ftr that data w bcom th data st Pa( ). Sct th most probabty varabs ca bcom, to gt th corratos dsty btw varabs, t ds to tgrat to dpdc fucto IY (, ): IY (, ) = D ( PY (, ) PPY ( ) ( ) (6) Amog thm, ( ( ) ( ) ( )og P ( ) DKL P Q P PY ( ) =. KL Th caddat comprss agorthm s as foowg: Iput: th wght cacuato a Baysa bf twor B S( BD) data st D = { x, x,, x }, paramtr Output:for a data, rtur a caddat C st for a, =,,,. () for a j,cacuat I(, j), j ad, Pa( ) ()sct th mt whch has th hghst wght, = Pa( ) caddat st, C = Pa( ) Y { x,, x } bac to { C }. Exprmta rsuts ad aayss By usg smuato stwar matab 7. costruct th xprmt vromt. Th umbr a th data a databas arg computr s st up as 000. Th umbr fatur data typs dd for cassfcato ar 0 ds. From th abov databas, 0 dffrt charactrstcs th data ar radomy sctd, th spcfc stuato ca b dscrbd wth th fgur. I th fgur, ach coor rprsts o d th charactrstcs data. Fgur th dstrbuto dagram dffrt attrbut data Fgur th cassfcato rsuts wh th typs fatur data ar fw Accordg to th xprmta rsuts fgur, t ca b ard, f th typs fatur data s ratvy fw, th ffccy data cassfcato utzg th mprovd agorthm s smar to ad th tradtoa agorthm. Wh th typ th fatur data th databas ar mor, usg dffrt agorthms for fatur data cassfcato rspctvy, th fatur data cassfcato rsuts obtad ca b usd to dscrb by fgur 3: 095

4 Fgur 3 th cassfcato rsuts wh th typs fatur data ar mor Accordg to th xprmta rsuts fgur 3, t ca b ard, f th typs fatur data s ratvy mor, th ffccy data cassfcato utzg th mprovd agorthm s hghr tha th tradtoa agorthm. Wh th typ th fatur data th databas ar fwr, usg dffrt agorthms ma 5 tms rdudat data cassfcato, th fatur data cassfcato rsuts obtad ca b usd to dscrb by tab : Tab Dffrt agorthms data tab data wh th typs fatur data ar fwr Th umbr xprmts cassfcato usg support vctor mach agorthm cassfcato usg wavt trasform agorthm cassfcato usg data ga agorthm cassfcato usg fuzzy support vctor mach agorthm I crcumstacs mor typs fatur data, th us dffrt agorthms to ma 5 fatur data cassfcato, th data th xprmta procss ar aayzd, to gt th xprmta rsuts tab : Tab Dffrt agorthms data tab data wh th typs fatur data ar mor cassfcato cassfcato cassfcato Th umbr cassfcato usg support usg wavt usg fuzzy usg data ga vctor mach trasform support vctor xprmts agorthm agorthm agorthm mach agorthm Through th abov xprmts t ca b ard, usg th mprovd agorthm for massv data cassfcato arg-sca computr formato maagmt, ca avod th dfct th poor corrato btw th mass data th databas, so as to mprov th ffccy fatur data cassfcato. 096

5 Cocusos For th probm th procss th massv data cassfcato arg-sca computr formato maagmt systm, du to th vtab ow corrato data causd by arg amout data, rsutg ow ffccy computr oprato, a mod for massv data dpth cassfcato mg basd o bf twor s put forward. Accordg to th rato btw th probabts data a th data doma, th corrato btw owdg ad data doma ca b frrd. Th, through th trag samp st fd th most sutab Baysa bf twor for th samp data, th accordg to th possb maagmt structur ad th tact udrstadg dgr btw data samps, th optma cassfcato data wth data maagmt structur arg sca computr. Th xprmta rsuts show that, usg th mprovd agorthm for massv data cassfcato procssg, ca mprov th accuracy cassfcato, ad achv satsfactory rsuts. Rfrcs [] Jag Hg, Chag Japg. Study SAR dyamc targt dtcto basd o mutp sga cassfcato mthod [J]. Computr smuato 0.9: [] Wa Chagxua, Lu pg. ML databas tchoogy. Bjg: Tsghua Uvrsty prss, 008:37-00 [4] Su Yzhog. Th thory ad bass appcato ML. Bjg: Bjg Uvrsty Posts ad Tcommucatos prss.000:30-34 [3] Zhou ushg, L Shuag. Modg ad Smuato automatc wb pag cassfcato [J]. Computr smuato, 0.0:-4. [4] Ha Tog. Trac ad maag th umbr cocurrt usrs to mprov th ffccy databas systm [J]. Iformato tchoogy ad formatzato, 0.5: [5] Logqa, Guo Jch. Th vstgato ad aayss 985 Uvrsty s Lbrary sf-but databas. [J]. Rsarchs brary scc, 00.8:7-3. [6] L Wj. Optma dsg schm arg databas ORACLE databas [J]. Tchoogy wd, 0.9: