Estimating Aspects in Online Reviews Using Topic Model with 2-Level Learning

Transcription

1 Esimaing Aspcs in Onlin Rviws Using Topic Modl wih 2-Lvl Larning Takuya Konishi, Taro Tzuka, Fuminori Kimura, and Akira Mada Absrac In his papr, w propos a mhod for simaing lan aspcs in onlin rviw documns. Rviw aspcs rprsn faurs of ims or srvics valuad by usrs. W can xpc o acquir usful faurs for usrs by discovring hir rviw aspcs. W apply opic modls o his problm. Exising work proposd mhods for simaing h opics of h whol documn or ss of snncs wih various window sizs. In his papr, w propos wo-lvl larning approach ha conncs adjacn snncs whn hir opics ar similar, and r-simas opics onc again using h drmind procssing unis. In h xprimns of prcision using prplxiy, w confirm our proposd mhod improvs on h xising mhod. Indx Trms onlin rviw, opic modl, snimn analysis, x mining. I. INTRODUCTION Th quaniy of various onlin documns publishd on h Wb coninus o incras. Thrfor, w nd labora and ffciv mhods o acquir usful informaion from hs documns on h Wb. Howvr, i is somims difficul o analyz hs documns by gnral framworks, bcaus usr gnrad informaion is no wll organizd; for xampl, sparsnss of informaion, noiss, or biass. For insanc, usr gnrad documns ar no ncssarily long nough (.g. blogs, wir, or BBS). Ths shornss of dscripions in documns is a major problm whn hs documns ar analyzd by gnral daa mining chniqus, naural languag procssing, or sophisicad saisical modls. In hs cass, propris of h arg documn ss mus b accoun for. W focus on onlin rviw documns. Rviw documns dscrib h propris of ims or srvics and ar providd by rviwrs hrough blogs or consumr gnrad mdia. Thy provid radrs usful informaion for dcisionmaking whn buying h rviwd im or srvic. Sinc w can obain hs rviw documns in bulk nowadays, many rsarchrs hav bn sudying hm by mans of saisical approachs ovr h las n yars. On of h main opics in his domain is snimn analysis. Pang al. hough of his problm as suprvisd documn classificaion [1]. Thy applid som classificaion modls (Naiv Bays, Maximum nropy modl, and SVM) o rviw documns givn snimn aspc (posiiv or ngaiv), and valuad ffcivnss of hs prcision. T. Konishi is wih h Gradua School of Scinc and Enginring, Risumikan Univrsiy, Noji-Higashi, Kusasu, Shiga, Japan, -mail: (cm005069@d.risumi.ac.jp). F. Kimura, and A. Mada ar wih h Collag of Informaion Scinc and Enginring, Risumikan Univrsiy, Noji-Higashi, Kusasu, Shiga, Japan, -mail: ({amada@mdia, fkimura@is}.risumi.ac.jp). T. Tzuka is wih h Gradua School of Library, Informaion and Mdia Sudis, Univrsiy of Tsukuba, 1-2 Kasuga, Tsukuba Ciy, Ibaraki, , Japan, mail: (zuka@slis.sukuba.ac.jp). In his papr, w considr xisnc of rviw aspcs. Each subjc coms ino som cagoris; for insanc, consumr lcronics has digial camras, lvisions, dskops, or lapops. W assum ha a rviw documn s in which ach valuad subjc blongs o h sam cagory has shard aspcs. Th discovry of hs aspcs is bnficial for usrs o undrsand subjcs and rviw documns. In his ask, w apply opic modls usd for documn modling [2]. This mhod assums ach documn has mulipl opics. Topics ar rprsnd by a probabilisic gnraiv modl and ar simad from h documn s. W assum ha hs opics corrspond o h abovmniond rviw aspcs. Topic modls hrby rprsn ach rviw aspc as word disribuion. Whil ordinary opic modls can ra som gnral documns, hy canno xrac maningful opics from rviw documns bcaus of h similariy across ach documn. In prvious work, Tiov and McDonald dvlopd a mor nhancd opic modl for his problm calld Muli-grain LDA [3]. I is possibl o xrac rviw aspcs by incorporaing h opic modl wih a nonparamric framwork. I is a sophisicad modl using a variy of lan variabls. In his papr, w propos a wo-lvl larning opic modl algorihm for rviw documns. I consiss of wo sps. Firs, i simas opics for ach snnc in h firs larning, and conncs snncs using h rsul of firs larning. Scond, i larns all groupd snncs daa in h scond larning. Whil his mhod is somwha simpl, w show ha i can rduc complxiy in a quaniaiv xprimn. Th rmaindr of his papr is srucurd as follows. In Scion 2, w xplain h problm whn w ra rviw documns in opic modls. In Scion 3, w show Muligrain LDA (MG-LDA), which cachs h propry of rviw documn, and also show simplifid Muli-grain LDA (smg- LDA), which focuss on only local opics and provids our baslin modl. Addiionally, w mnion h problm of using window lmns in rms of complxiy. In Scion 4, w dscrib our proposd algorihm. Scion 5 provids an mpirical xprimn of bwn h baslin modl and our mhod. Finally, w summariz our papr and discuss fuur work. II. THE CHARACTERISTICS OF REVIEW DOCUMENTS In his scion, w dscrib h characrisics of rviw documns ha ar h problms in xracing opics. Topic modls basically rquir documn ss o b rprsnd as bag-of-words. Bag-of-words is a simpl assumpion ha ignors h ordr of words whn procssing words in ach group (.g. a documn) and only accouns for h frquncy of ach word. This assumpion is convnin o mploy o h probabilisic modls for xracing global propris of

2 h arg documn s lik opics. Gnral opic modls apply his assumpion o documn lvl; ha is o say, h daa has only informaion abou how many ims ach word appars in ach documn. Howvr, his assumpion causs h problm for opic xracion in rviw documns. All rviws ha dscrib h sam cagory ar vry similar o ach ohr. For insanc, considr rviws of digial camras. W can suppos imag qualiy of a digial camra is a rviw aspc. This opic is on of h imporan aspcs characrizing digial camra rviws. Consqunly, i will appar in almos all h documns. In ohr words, words rprsning imag qualiy will appar in any documn. This problm maks xracing hs opics difficul bcaus words rlad o his opic ha appar ar vry similar o ach ohr. In fac, hrough h xprimn of applying lan Dirichl allocaion, which is h mos gnral opic modl [2], o rviw documns, w found ha w canno xrac h dsird maningful opics w wan. Thrfor, w hav o accoun for mor local disinguishing faurs in ach documn. Th simpl ida is o assum snnc lvl bag-of-words. Snncs rprsn mor local faurs han documns. In fac his assumpion works wll o som xn. On problm in his ida is h lack of faurs in ach snnc. If w rmov h non-imporan faur words (.g. sop words) in all snncs, hy hardly hav any faur words lf. This is a problm whn h modl composs snnc-spcific opic disribuion bcaus ach disribuion is somwha spars. In conclusion, w hop ha opic modls capur local opics rprsning faurs of smallr unis han a whol documn. In addiion, h disribuion composd by h modls is mor informaiv. Th ky poin for solving h problm is how o drmin h procssing unis. W nd o dcid h appropria siz of unis, which is largr han a snnc bu smallr han a documn. Fig. 1. α θ jm M j γ ψ js S j u ji z ji A graphical rprsnaion in simplifid Muli-grain LDA. Algorihm 1 Gnraiv procss of smg-lda 1: for all opic (=1...T ) do 2: Draw ϕ Dir(β) 3: nd for 4: for all documn j (=1...D) do 5: for all snnc s of documn j (s=1... S j ) do 6: Draw ψ js Dir(γ) 7: nd for 8: for all sliding window m of documn j (m=1...m j ) do 9: Draw θ jm Dir(α) 10: nd for 11: for all okn i in snnc s of doucmn j(i=1...n j ) do 12: Draw u ji Muli(ψ js ) 13: Draw z ji Muli(θ juji ) 14: Draw w ji Muli(ϕ zji ) 15: nd for 16: nd for w ji N j D β φ T III. RELATED WORK In his scion w brifly inroduc h spcific opic modl for rviw documns calld Muli-grain LDA [3]. Th modl is rprsnd as a probabilisic gnraiv modl. W ovrviw h modl and discuss h rol of windows in his modl. A. MG-LDA Tiov and McDonald dvlopd h Muli-grain LDA, which capurs h global and local opics [3]. Thy assumd ha rviw documns had global opics ha rprsn global propris apparing in documn lvl (.g. im spcific opics) and local opics ha rprsn local propris apparing in som snnc lvls (i.. raabl aspc). W sd h ffcivnss of his modl. Howvr i did no ncssarily xrac maningful global opics. In conras i works wll for xracing local opics. In addiion, sinc our mhod only focuss on local opics, w ignor h xisnc of global opics in h modl. W call h simplifid Muli-grain LDA modl smg-lda in his papr. smg-lda is a probabilisic gnraiv modl. In gnral, probabilisic gnraiv modls dfin h gnraiv procss of daa, and all daa ar gnrad in accordanc wih his procss. smg-lda follows h gnraiv procss of Algorihm 1. Bfor w sar o xplain his procss, w dno som noaion. Firs T is h numbr of opics, D is h numbr of documns and S j is h numbr of snncs in h documn j. M j is h numbr of windows in documn j. This is dcidd by window siz K and S j, i.. M j = K + S j 1. Also Dir() dnos h Dirichl disribuion, and M uli() dnos h mulinomial disribuion in h abov gnraiv procss. Addiionally, α, β, and γ ar hyprparamr vcors for ach Dirichl disribuion. Firs, his modl sampls word probabiliy vcors ϕ from h Dirichl prior Dir(β) for ach opic. For ach documn, his modl sampls window probabiliy vcors ψ js from Dirichl prior Dir(γ) for ach snnc, and sampls opic probabiliy vcors θ jm from Dirichl prior Dir(α) for ach window. Finally, i sampls hr variabls for ach word in all documns. u ji dnos a window assignmn of okn i in documn j, z ji dnos a opic assignmn of okn i in documn j, and w ji dnos a word of okn i in documn j. Ths hr kinds of variabls ar sampld from corrsponding mulinomial disribuion. Undr hs gnraiv procsss, w nd o sima opic assignmn z and window assignmn u for ach okn.

3 Documn Snnc i-1 Snnc i Snnc i+1 Snnc i+2 Fig. 2. An illusraion of window lmns in Muli-grain opic modl. This insanc dnos wo window sizs. W mploy h Gibbs sampling o infr his modl. Th condiional probabiliis for Gibbs sampling ar givn by : p(z ji = w ji = v, u ji = m, w ji, z ji, u ji ) = nv, ji + β n j,m, ji + α n, ji + V β n j,m, ji + α 0 (1) p(u ji = m z ji =, w, z ji, u ji ) = nj,m, ji + α n j,s m, ji + γ n j,m, ji + α 0 n j,s, ji + Kγ (2) whr n v, ji is h numbr of ims word v appars in opic xcp okn ji (i. okn i in documn j), n j,m, ji is h numbr of ims opic appars in window m of documn j xcp okn ji, and n j,s m, ji is h numbr of ims window m appars in snnc s of documn j xcp okn ji. Each do wih coun noaions in dnominaors rprsns marginal couns. In addiion, all valus of hyprparamr vcor α affc simaion rsul snsiivly, so w sima hm using Minka s fixd poin iraion [4]. Rmaining hyprparamr vcors s h sam valu for h lmns of ach vcor (spcifically, β = 0.1 and γ = 1.0). W show h graphical modl for his modl in Figur 1. B. Complxiy of Prvious modl W considr h window lmn of Muli-grain opic modls in his subscion. Th window siz affcs how many snncs h modl accouns for (s Figur 2). Sinc h modl can ak accoun of no only on snnc bu also nighboring snncs using a window, w may nd o considr h rol of h window. smg-lda (also MG-LDA) rquirs a dcision on h window siz in modl slcion as wll as h numbr of opics. smg-lda pu a mixur modl of window variabls ino pracic, ha is o say, his modl assums ha ach snnc has mulipl windows. This is a lil complx for h daa, and w considr only on window is nough for ach snnc. Morovr, his modl nds o fix h window siz across h arg corpus. This rsuls in a lack of flxibiliy. For an xrm xampl, a documn composd of on snnc is assignd o wo or hr windows. This is obviously rdundan. In h nx scion, w propos h modl ha accouns for nighboring snncs wihou mulipl windows. W will solv h problm using wo-lvl larning. IV. PROPOSED ALGORITHM W propos a nw larning algorihm oward h abov, mniond problm. This mainly consiss of h firs larning, conncing nighboring snncs, and h scond larning. Firsly, i conducs h firs larning for vry on snnc. Th rsul givs inrim opic assignmns for ach okn. Ths assignmns giv opic disribuion for ach snnc indircly. Spcifically, h opic disribuion of snnc s in documn j is givn as follows: T p(z θ js ) = (θ js ) z (3) =1 whr p(z θ js ) dnos h opic disribuion of snnc s in documn j, and is rprsnd as mulinomial disribuion (also calld cagorical disribuion). z is h 1-of-K rprsnaion (and z is h h lmn of h z), and θ js is h h lmn of opic probabiliy vcor θ js (i.. h opic probabiliy valu p(z = θ js ) = θ js ). Thn our mhod conncs ach snnc using h simad opic disribuion. W considr boundaris of snncs in ach documn. Th numbr of boundaris in documn j, B j is S j 1. Th mhod valuas h conncion of nighboring snncs for ach boundary. Whil w nd h cririon o dcid whhr h snncs ar conncing or no, w apply h Jnsn-Shannon divrgnc (JS-divrgnc) o valua i. JS-divrgnc is applid whn masuring h similariy of probabiliy disribuion. Unlik h Kullback-Liblr divrgnc, which is h common masur for similariy of probabiliy disribuion, JS-divrgnc saisfis h symmric rlaion. W mploy his masur o obain h similariy of nighboring snncs. Thrfor, w can calcula h similariy bwn h snnc immdialy bfor h boundary s and h snnc immdialy bhind h boundary s + in boundary b of documn j using JS-divrgnc as follows: whr sim(s, s+ ) = JS(p p + ) ( = 1 T p 2p log 2 p =1 + p + ) T + p + 2p + log p + p + =1 p = p(z θ js ), p + = p(z θ js + ) p = p(z = θ j ), p + = p(z = θ j + ) According o h propry of JS-divrgnc, h similariy has non-ngaiviy and bcoms zro if and only if ach disribuion is qual o ach ohr. If h valu is lowr, h mhod inrprs h similariy o b highr. Our mhod valuas nighboring snncs using his masur, and if h valu of similariy is blow h hrshold, hs snncs ar conncd. Th dcision of h hrshold will b dscribd in h nx scion. As h rsul of h abov procss, w can obain h nw daas in which smanically similar snncs ar conncd. Finally, h mhod conducs h scond larning using h rnwd daas. Our proposd algorihm is shown in Algorihm 2 and illusrad in Figur 3. W dno ha (4)

4 Boundary Topic disribuions assumd vry groups Firs larning for ach snnc Calcula similariy for ach nighboring snnc Connc snncs in accordanc wih similariy Scond larning for ach groupd snnc On snnc sim(s - 1, s + 1) sim(s - 2, s + 2) sim(s - 3, s + 3) sim(s - 4, s + 4) sim(s - 5, s + 5) Groupd snncs Fig. 3. An illusraion of our proposd algorihm. Th ky ida is o connc nighboring snncs according o h firs larning. W can obain h opic disribuion for ach group of snncs in h scond larning. Algorihm 2 Our proposd algorihm 1: Excu firs larning for procssing uni ha is in ach snnc 2: for all documn j (=1...D) do 3: for all snnc boundary b (=1...B j ) do 4: Calcula h similariy sim(s, s+ ) of nighboring snncs s and s+ according o (4) 5: nd for 6: for all snnc boundary b (=1...B j ) do 7: if sim(s, s+ ) < hrshold hn 8: Connc h nighboring snncs 9: nd if 10: nd for 11: nd for 12: Excu scond larning for rnwd procssing uni ha groups smanically similar snncs by h abov procsss scond larning assums opic disribuion in vry groupd snnc in Figur 3. Th disribuion has mor informaion han h disribuion for ach snnc. Th ida of h proposd algorihm is considrably simpl and dos no nd addiional variabls o conrol h window unlik h smg-lda. Whil h algorihm assums i is possibl ha firs larning simas opics for ach snnc o som xn, i can drmin h adapiv procssing unis for ach documn. I dos no ovrfi oward a documn ha has only a fw snncs. V. EXPERIMENTAL RESULTS W dscrib h mpirical rsuls of our proposd algorihm in his scion. Firs, w xplain h sup for his xprimn. W prpar wo Japans rviw documn ss and on English rviw documn s. For Japans rviws, w us wo ral onlin rviw documn ss abou digial camras and lapops. W obaind hs documns from h Japans onlin pric comparison si, kakaku.com [5]. Kakaku.com provids h informaion of various ims or srvics, and publishs rviw documns conribud by usrs. W collcd 13,638 rviws abou digial camras and 12,211 rviws abou lapops from his si. W usd nouns as faur words in hs daa ss. For English rviws, w us on rviw s abou digial camra in Amazon.com [6]. W collcd 11,279 digial camra rviws. W rmovd som sop words and usd smming for his daa. W uilizd hs hr daa ss as xprimnal daa. W usd prplxiy o valua modls. Prplxiy indicas h prformanc of prdicion for nw words in ach modl. W usd 90% words as raining daa, and 10 % words as s daa. Prplxiy is dfind as follows: xp( 1 N s log p(w s w raining )) (5) whr N s is h numbr of words in h s daa, w s is h s of s words, and w raining is h s of raining words. W compar h proposd algorihm wih h prvious modl. Th modl o compar is smg-lda inroducd in scion 3. Our proposd mhod nds o drmin a hrshold of h similariy in ach boundary. In his xprimn, w drmind in accordanc wih rankings of similariis. A ranking of similariis can b obaind as a squnc of similariis in dscnding ordr (i.. ascnding ordr in h valu). In his way, h hrshold changs in vry modl slcion (.g. numbr of opics). W mploy highr-ranking valu in h ranking, and dmonsrad h prliminary xprimn of prplxiy in shifing h proporion of h conncd snncs on h basis of similariis. For Japans rviws, h op 20% of highr similariis ar h bs rsul in digial camra rviws, and h op 25% ar h sam in lapop rviws (i.. hrsholds ar qual o h valu a ach prcnag). For English rviws, h op 30% ar h sam in digial camras. W mak us of valus gaind in his way o compar our rsuls o hos in prvious work. Firs w show h xprimnal rsuls of Japans rviws in Figur 4 and 5. Figur 4 shows h rsul of digial camra rviws, and Figur 5 shows h rsuls of lapop rviws. In boh figurs, our proposd algorihm surpasss window siz 2 and 3 modls, and slighly dos so for h window siz 1. Gnrally, whil all modls hardly diffr from ach ohr in small numbrs of opics, larg diffrncs aris in largr numbrs of opics. Whil h improvmn is no larg whn compard o h window siz 1 modl, our mhod succdd in h improving in h siuaion accouning for surrounding snncs diffrnly from h ohr wo modls.

5 Window siz 1 Window siz window siz 1 window siz Window siz window siz lvl (h = 0.2) lvl (h=0.3) y i x l p r P y i x l p r P Numbr of Topics Numbr of Topics Fig. 4. A comparison bwn our proposd algorihm and smg-lda modls in Japans digial camra rviws. Fig. 6. A comparison bwn our proposd algorihm and smg-lda modls in English digial camra rviws. y i x l p r P Numbr of Topics window siz 1 window siz 2 window siz 3 2 lvl (h=0.25) Fig. 5. A comparison bwn our proposd algorihm and smg-lda modls in Japans lapop rviws. Figur 6 shows h rsuls of English rviws abou digial camras. Whil h rnd is mainly h sam as h rsuls of Japans rviws, i shows h grar improvmn han Japans ons for our mhod compard o smg-lda modls. W hink hs diffrncs ar causd by h diffrnc of documn srucur bwn Japans and English rviws. Japans rviws ar comparaivly succinc, and ar rlaivly shor on avrag. English rviws ar mor dscripiv han Japans ons, and ar rlaivly long on avrag. For hos rasons, our algorihm works wll in English han Japans. In hs xprimns, w slcd fixd hrsholds o connc smanically similar snncs as dscribd abov. W dcidd hm on basis of mpirical rsuls. Howvr, w will considr ways ha ar mor suiabl o dcid h hrshold. If h proposd mhod dcids conncions on h basis of mor subsaniad cririon, w can xpc h mhod o b improvd. VI. CONCLUSION W proposd h nw algorihm for simaing h opics in onlin rviws. W focusd on h complxiy of h prvious work calld MG-LDA, and w considrd a mor plain framwork ha uss naural assumpion. Th rsuls of his modl wr br han hos of h prvious work in rms of prplxiy, which is usd h gnral indx in valuaing opic modls. Hr w considr ohr rlad works and h fuur work. W hink our proposd ida is closly rlad o asks abou x sgmnaion or paragraph dcion [7][8]. Thy nd h appropria x sgmn and characrizing documns. W xpc ha our mhod is applicabl o hs asks. Also w xcud h our algorihm by using wo-sp larning. I is no ncssarily lgan o us simply sparad larning from h poin of viw of a probabilisic modl. Thus, w ar going o considr h adapiv modl ha can xcu larning onc. ACKNOWLEDGMENT This work was suppord in par by MEXT Gran-in- Aid for Sragic Formaion of Rsarch Infrasrucur for Priva Univrsiy Sharing of Rsarch Rsourcs by Digiizaion and Uilizaion of Ar and Culural Marials (Gran Numbr: S ) and MEXT Gran-in-Aid for Young Sciniss (B) Rsarch on Informaion Accss across Languags, Priods, and Culurs (Ladr: Akira Mada, Gran Numbr: ). REFERENCES [1] B. Pang, L. L, and S. Vaihyanahan, Thumbs up? Snimn Classificaion using Machin Larning Tchniqus, in Proc of Confrnc on Empirical Mhods in Naural Languag Procssing, pp.79-86, [2] D. M. Bli, A. Ng, and M. I. Jordan, Lan Dirichl allocaion, in Journal of Machin Larning Rsarch, Vol.3, No.5, pp , [3] I. Tiov and R. McDonald, Modling Onlin Rviws wih Muligrain Topic Modls, in Proc of h 17h Inrnaional World Wid Wb Confrnc, pp , [4] T. P. Minka, Esimaing a Dirichl disribuion, Tchnical rpor, hp: //rsarch.microsof.com/n-us/um/popl/minka/paprs/dirichl/,2003 [5] kakaku.com, hp://kakaku.com/

6 [6] Amazon.com, hp:// [7] J. Eisnsin and R. Barzilay, Baysian unsuprvisd opic sgmnaion, in Proc of Confrnc on Empirical Mhods in Naural Languag Procssing, pp , [8] M. Jong and I. Tiov, Muli-documn opic sgmnaion, in Proc of h 19h ACM inrnaional confrnc on Informaion and knowldg managmn, pp , 2010.