A METHOD OF PHASED INTEGRATED SEMANTIC SIMILARITY COMPUTATION

Transcription

1 3 s March 03. Vol. 49 No JATIT & LLS. All rghs reserved. ISSN: ja.org E-ISSN: A METHOD OF PHASED INTEGRATED SEMANTIC SIMILARITY COMPUTATION MA JUNHONG Lecurer, X a Ieraoal Uversy, Prvae Educao Research Ceer, Shaax, Cha E-mal: maxaofe93@63.com ABSTRACT Ths paper proposes a mehod of phased egraed smlary o mprove he exsg Chese ex algorhms. I compues ex smlary secos from he seeces,paragraphs,o he hole ex. Combg h he characerscs of each seco,he ex facors s also bleded each seco,srvg o he bes accuracy. A las, e have esablshed a smlary compug sysems, compared he e mehod h oher commo mehods Chese formao processg. The expermes dcae ha he mproved algorhm has acheved beer resuls. Keyords: Texs smlary, Vecor Space Model, Semac smlary, Term egh,tf-idf. INTRODUCTION A prese, he eork of formao resources s grog, may of hem s useless, reduda duplcao of formao,o oly akes up a lo of sysem space, bu also makes he processg effcecy of he sysem o reduce. Accurae ad effecve ex smlary mehod s oe of he mehods o solve ex processg problems.oe fudameal ad mpora ork formao processg s ex smlary, hch s he key echology he exual daa mg ha relaed o may mpora applcao researches, for example, he area of docume copy ex caegorzao, ex cluserg, formao rereval, queso aserg sysem, ad ec. I s orh furher research ad dscusso because of s de applcaos. The exsg ex smlary compug model has eakesses such as defcecy raoale ad compleeess docume properes fg. Chese ex udersadg ad processg s more challegg relave o Eglsh couerpar[]. Currely he mos dely used mehod s based o he sascs of he radoal ex smlary calculao[]. Such as VSM used Chese ex rereval sysem, he cose of he agle of he vecor of he Eucldea space used ex classfcao smlary calculao[3]. Hoever, such mehods are complcaed, ally esve, o ally ad srucural relaoshps. Udersadg Chese laguage from he ve of s more approprae ha from he sascal mehod. Therefore, he mehod based o he smlary s grog cocer. The basc dea of he smlary calculao s deparg from a smple algebrac sascs, excavaos ex deeper represeao, ad makes relaed applcaos calculao more accurae[4]. Ths approach reduces he ork of he corpus ad he rag ses, parcular, s more approprae for he udersadg of he Chese ex. The am of hs paper s o mprove he exsg algorhms. For hs am, e compared deal varees of ex smlary mehod Chese formao processg ad aalyze her characerscs ad defecs. Furhermore, a e mehod of phased egraed smlary, hch s a mproved mehod, has bee pu forard. Ths mehod bleds ex facors h each seco, mproves he Chese ex smlary compug effcecy ad accuracy.i has cera applcably ad feasbly. Ths gude provdes deals o asss auhors preparg a paper for publcao JATIT so ha here s a cossecy amog papers. These srucos gve gudace o layou, syle, llusraos ad refereces ad serve as a model for auhors o emulae. Please follo hese specfcaos closely as papers hch do o mee he sadards lad do, ll o be publshed. 85

2 3 s March 03. Vol. 49 No JATIT & LLS. All rghs reserved. ISSN: ja.org E-ISSN: CHARACTERISTICS OF THE TEXT SIMILARITY COMPUTING ANALYSIS. Defo of ex smlary hs paper: The core of he ex smlary comparso s comparg he dfferece beee o of he gve ex (or bye sream, ec.), usually uses a umber of [0, ] as measure [5]. I dffere felds, he meag of smlary degree s dffere, ad s ellecual propery also have cera effec, so Ths paper argues ha he should o be judged solely by he smlar ords or smlar seeces ad paragraphs, should also coa udersadg. So e make he follog defo: Tex smlary s o po o for a gve o or more of he ex, hrough he layers of of seece paragraph o ge he overall smlary beee hem, ad a he same me coas a cera smlary.. TF-IDF(Term Frequecy-Ivered Docume Frequecy) I VSM (Vecor Space Model), he ex s represeed o he vecors hch are composed of he feaure. Ad he egh of feaure s he ex Vecor Space coordaes[6]. Feaure eghs for ex represeao plays a very mpora role; also affec he ex smlary resuls. A prese he commoly used mehod s TF-IDF, IG (Iformao Ga), MI (Muual Iformao), DF (Docume Frequecy) χ(chi),ad ec. Amog hese mehods, IG,CHI,ad MI belog o supervsed feaure seleco algorhm, DF belog o usupervsed algorhm hch ca be drecly used cluserg. TF-IDF ca effecvely dsgush he hgh frequecy ords ad he lo dffereae ords. I compug, he o aspecs are he major cosderaos: () Ths feaure s affecs a ex (Usually measured by frequecy, amely he TF) () Ths feaure s affecs he hole ex se (ca dsgush beee dffere ex, amely IDF) If L s ex oal of ex se, represes he egh, he[7]: IDF = log L DF j () j = TFj IDF j = TFj log L DF j () Accordg o dffere ormalzed processg, he use of TF - IDF formula s dffere.ths mehod s he mos commoly used eghg mehod, ca effecvely dsgush beee hgh frequecy ords ad lo of he degrees of dsco ord. 3. METHOD OF PHASED INTEGRATED SEMANTIC SIMILARITY COMPUTATION 3. Semac sregheed egh mehod TF IDF s a mehod based o ord frequecy. If a ord ex preprocessg sage s seleced as he feaure ords, he he of he ord frequecy he res of seleced ords ll be dscrmae reaed equally[8]. Ths mehod cosdered s oo oesded, does o ake o he facors of coex or applcao felds, ec. Therefore, e propose a mproved mehod: I ex preprocessg sage ca be combed h he ex heme, scope, applcaos ad so o, o gve he erm egh as f. A frs, usg he radoal TF-IDF formula o calculae a erm egh j, he calculaed usg he formula 3: ' j = β * j + β * f (3) Here roduces o parameers: βad β, β+β=. The eghs of j calculaed h he radoal formula sll holds larger proporo, bu eed o be adjused h he facors. The former accoued for by β, usually be 0.8 or 0.9; he laer accoued for by β, ypcally be 0. or 0.referg o β. Ad f ll be se accordg o he specfc eeds, such as combg h he deeco of ex classfcao, subjec or keyords, erm eghs he feld of f ca be se o, he res of he o - erms ca be o 0.8, geeral ord ca be se o smlary s seps The algorhm based o he udersadg ofe eeds o buld a e ex represeao model, hrough he judgme beee ex dsace, relevace o acheve. A prese, Hoe ords eork has more maure, h he ad of Hoe koledge srucure ad he koledge represeao, hs paper preses a e fuso Mehod of phased egraed smlary. 86

3 3 s March 03. Vol. 49 No.3 S(L JATIT & LLS. All rghs reserved,r ),. S (L,R ),, S (L,R m ) S (L,R ), S (L,R ),, S (L,R m ) ISSN: ja.org E-ISSN: The Frs ork s he ex level dvso, S (L Seece,R ), S (L smlary,r ),, S (L,R m ) formula amely he ex s dvded o paragraphs, of he L ad R: paragraphs are dvded o seeces, seeces o ords; secodly, phased calculag he max( S( L, R ), S( L, R ),, S( L, Rm )) smlary of ords, seeces, paragraphs o oba S( L, R) = he ex smlary. A each sage he (6) smlary has bee bleded, I order o accuraely ge he smlary of compleg h he combao of local o global. hese o seeces, e calculae Sm (L, R)ad The frs sep: ord smlary compug I Hoe koledge srucure, because Sm (R,L) by he formula 5,he ake he average he cocep s composed of prmarly, he he of he o, esurg he uqueess of he smlary calculao of he ords ca be see as calculaed resuls. he meag of he prmarly smlary calculao. The hrd sep: paragraph smlary Ad prmarly hypoymy form a ree herarchy, compug ca calculae by he ay of calculag he Suppose here are o paragraphs X ad Y, dsace. X s dvded o seeces, Y s dvded o d Geeral uses he follog formula : seeces, he formula s as follos: a Sm( p, p) = d + a (4) max( (, ), (, ),, (, )) S X Y S X Y S X Yd Amog hem, he pl ad p sad o S( X, Y ) = *0.5 prmarly, her dsace s d, o behalf of he p ad p pah legh. a s a adjusable parameers, s value s 0.5 hch s o behalf of he smlary of pah legh. max( (, ), (, ),, (, )) he compug he prmary smlary e S Y X S Y X S Y X d + cosder her relave poso o he prmary *0.5 classfcao ree[9], ge he follog formula: a h h Sm( p, p) = (7) d + a (5) I algorhm desg, hese seeces have Tha meags cosder he explaaory bee calculaed smlary paragraphs, hese relaos of he o prmarly o amed he prmary ords have bee calculaed smlary he dsace, he compug he prmary dsace. seeces, are o loger calculae aga, drecly se Secod seps: Seece Smlary he prevous value o mprove he effcecy. Compug The fourh sep: Tex Smlary Usg he mproved TF - IDF feaure Compug exraco algorhm, Seece L (cludg The compug mehod meoed above feaure ords) ad L (cludg m feaure ords) meas akg he maxmums o add up, ad he s represeed as a vecor form of he follog: akes a average, hch s equal reame o he L=,,, ords, seeces he paragraph[0]. Cosderg ( ) ha accordg o he paragraphs poso of he ex, R=,,, m s mporace, amely he fluece of he ex s H ( L, R ) s he o seece smlary also dffere, should be gve dffere egh. If marx: he key paragraphs are smlar, he hole ex smlary creases. Suppose D, D are o exs o be comparg, D has m paragraphs; D has paragraphs. I order o coveece, ufed use X expressed a paragraph D; use Y expressed a paragraph D. Accordg o he formula 6 he calculaed paragraphs smlary recorded as: S (X, Y), S (X, Y)... Sm (X, Y), SS (X, Y) sad he e paragraph smlary, here s he formula: 87

4 3 s March 03. Vol. 49 No JATIT & LLS. All rghs reserved. ISSN: ja.org E-ISSN: SS = * (8) ( X, Y ) β * S ( X, Y ) + β I hs formula, β +β =, s a paragraph egh hose specfc value ca be se accordg o he requreme. Ths paper akes 0.8 β ad 0. β, he key paragraph egh s., he ormal paragraph egh s. Accordg o he formula 8, D ad D smlary formula s as follos: Sm( D, D ) m = SS ( X, Y ) m (9) I addo, he acual applcao process, cosderg he log ex paragraphs usually s more ad a lo of lo smlary s almos zero mpac o he hole ex, ca se a smlary hreshold advace hch paragraphs loer ha he hreshold value are o loger be joed he egh so as o reduce he overhead of he sysem, furher mprove he effcecy. 4. COMPOSITION AND IMPLEMENTATION OF SYSTEMS 4. modules ad flo char Tex ses The sysem cludes a ex preprocessg (Chese auomac ord Word segmeao), he ex feaure vecor represeao, segmeao feaure vecor module Exrac he ex o compare Tex feaure represeao seleco, ad smlary, maly composed of he follog modules: () Tex lbrary module: for sorage he Chese exs o compug smlary; () Word segmeao dcoary module: maageme ad maeace he dcoary for Chese ord segmeao; (3) Chese ord segmeao module: for a gve Chese ex ord segmeao processg ad ambguy correco[]; (4) Feaure exraco module: accordg o he ord segmeao resuls ad sascs of ord frequecy aalyss, exrac delegae o compare ex feaure vecor ad each ery coas he correspodg egh; (5) Smlary module: Compuao of he ex feaure vecor smlary, he accordg o he gve Smm hreshold o deerme he smlary beee exs, ad obaed he resuls. The orkflo of he sysem s four seps as follos: () ake ou o ex from he prepared ex se; () Use he Chese ord segmeao ool,ik Aalyzer 3. for ord processg; (3) Exrac ex feaure vecors ad deerme he eghs, (4) Depedg o he dffere smlary algorhm seps o calculae he smlary beee he exs se. Feaure The composo ad Smlary orkflo char as follos exraco fg : Tex calculao module feaure module vecors Calculao resuls Tradoal VSM Improved TF-IDF egraed ompuao Compare Sm m Fgure : Tex smlary compug sysem ad flo char 4. Expermes ad resuls aalyss 4.. Daa ses of he experme Ths experme uses he es daa se of a research u hch s seleced from 40 exs abou compuer scece, Chese laguage ad leraure, ecoomc maageme felds, ad accordg o he dffere legh dvded o 3 groups, respecvely 88

5 3 s March 03. Vol. 49 No JATIT & LLS. All rghs reserved. ISSN: ja.org E-ISSN: usg hree kds of smlary mehod o es: he radoal mehod based o VSM, he sregheed TF-IDF ad he phased egraed. The Tradoal VSM hou smlary processg s calculaed by he cose coeffce. I he radoal mehod of VSM, We use (T, T,..., T ) ad (T, T,..., T ) as he vecor of he ex D ad he ex D[], here s a formula as follos: Sm( D, D) = cosθ = T T * T * T (0) The seleced ex legh dsrbuo s as follos: Groupg Numbers Table : Tex legh dsrbuo Tex legh rage Tex cous 000~ ~ ~ The Expermeal Resuls I ve of he precedg dscusso, e use precso ad recall as evaluao crera, smlary hreshold s se o 0., greaer ha hs hreshold ll be regarded as smlar ex. umber of correcly deeced smlar exs precso = umber of all he deeced smlar exs () umber of correcly deeced smlar exs recall = umber of acual exsece of smlar exs () The frs se of expermes he resuls are as follos: Table :Comparso of smlary mehod assessme mehod precso recall Fgure.: Comparso Of Smlary Compuao Mehod The secod expermeal resuls are as follos: Table 3:Comparso of smlary mehod precso radoal VSM recall sregheed TF-IDF radoal VSM sregheed TF-IDF egraed egraed precso recall assessme mehod radoal VSM sregheed TF-IDF radoal VSM sregheed TF- IDF egraed egraed precso recall Fgure 3.: Comparso Of Smlary Compuao Mehod 89

6 3 s March 03. Vol. 49 No JATIT & LLS. All rghs reserved. ISSN: ja.org E-ISSN: The hrd groups of experme resuls are as follos: Table 4:Comparso of smlary mehod 3 Assessme mehod precso Trado al VSM precso recall radoal VSM sregheed TF- IDF egraed Fgure 4.: Comparso of smlary mehod 3 5. CONCLUSION Semac sreghee d TF-IDF egraed recall Accordg o he o formulas of precso ad recall rae o aalyss,whe precso rae s very hgh bu he recall rae s very lo, ha meas he sysem fd smlar ex perceage s hgh, he umber of mscarrage of jusce s less, bu o comprehesve fd ha here are may smlar ex dd o fd ou; O he corary, f he recall rae s hgh ad he accuracy s lo, ha meas looked May o-smlary of he ex as smlar, hch caused a lo of mscarrage of jusce, bu he acual exsece smlar exs mosly have bee foud ou. Boh cases are o deal,oly boh cosderable o sho ha he exs bee foud ou s really smlar,ad have he acual exsg smlar ex bascally all fd ou. As ca be see from he hree groups of expermeal daa, he mehod of hs paper hose precso rae ad recall rae s farly basc, hle he oher o algorhms are slghly some devao. Compared h he radoal VSM based o cose algorhm, hs mehod has beer effec, ad he sreghe TF - IDF algorhm also rased some al effcecy. Bu h he crease of ex legh, he effcecy of our algorhm s reduced someha, as ca be see for smaller legh of ex smlary compug, he algorhm orks beer. Chese ex smlary calculao process s very complcaed, ad here are may uceraes he specfc applcao. Ho o esablsh a beer model, ad apply o more specfc felds, such as cluserg, auomac absrac, s he ex sep of our research focus. REFERENCES: [] Salo G, The sae of rereval sysem evaluao, Iformao Processg ad Maageme, , 99 [] Yu Peg, Hogx Wa, Web Tex Cluserg ad Evaluao Algorhm Based o Fuzzy Se,JDCTA: Ieraoal Joural of Dgal Coe Techology ad s Applcaos, Vol. 7, No., pp. ~ 8, 03. [3] Og Sou Ch, Narayaa Kulahuramayer&Alv W.Yeo. Auomac Dscovery of Coceps from Tex. I Proceedgs of IEEE Ieraoal Coferece o Web Iellgece, Washgo.DC, USA,pp : 046~049,006 [4] Jula Seddg, Dmar Kazakov. WordNebased Tex Docume Cluserg,I Proceedgs of he 3rd Workshop o Robus Mehods Aalyss of Naural Laguage Daa, USA,04-3,00. [5] Hu-l Ta, Xa-feg Lu. The Research of The Seleco of Koledge Reasog Mehod Based o Geec Algorhm. Compuer Koledge ad Techology,Vol. 34, No., pp. 55 ~ 59, 0. [6] X-Qa J, Chese Tex Smlary Algorhm Research Based O Semac Smlary, Zhejag Uversy of Techology, 0. [7] YAN We,Zhag Jazhog,We Dase,Wag Xagcheg,Zheg Webo, "A Novel Selfadapve Case-based Reasog Techque o Predc Busess Falure",IJACT: Ieraoal Joural of Advacemes Compug Techology, Vol. 4, No. 3, pp. 376 ~ 384, 0. [8] ILAMPIRAY.P, Effce Resource Ulzao of Web Usg daa cluserg ad assocao rule mg, JATIT:Joural of Theorecal ad Appled Iformao Techology,Vol. 37 No.,pp.~6,0. [9] X Xu, Research o Parameers Correlao ad Opmzao Tex Smlary 830

7 3 s March 03. Vol. 49 No JATIT & LLS. All rghs reserved. ISSN: ja.org E-ISSN: Measureme Specaly, Ceral Souh Uversy,Chag Sha Hua P.R.C, 00. [0] Jg-Fa Wag, To-Sep Job Iformao Rereval based o Docume Smlary,Tsghua Uversy,Bejg,