Combining Subword and State-level Dissimilarity Measures for Improved Spoken Term Detection in NTCIR-11 SpokenQuery&Doc Task

Transcription

1 Combining Subwod and Stat-lvl Dissimilaity Masus fo Impovd Spokn Tm Dtction in NTCIR-11 SpoknQuy&Doc Task ABSTRACT Mitsuaki Makino Shizuoka Univsity Johoku, Hamamatsu-shi, Shizuoka , Japan In cnt yas, dmands fo distibuting o saching multimdia contnts a apidly incasing and mo ffctiv mthod fo multimdia infomation tival is dsiabl. In th studis on spokn documnt tival systms, much sach has bn psntd focusing on th task of spokn tm dtction (STD), which locats a givn sach tm in a lag st of spokn documnts. Rcntly, in such spokn documnt tival task, th has bn incasing intst in using a spokn quy not only fo impoving usability but also fo low-souc languags which may hav much os by LVCSR systms. In this pap, w popos spokn tm dtction mthod using multipl scoing and dissimilaity masus fo spokn quy. Ou poposd mthod is intndd to convt th spokn quy into a syllabl squnc by LVCSR and do sach that taks into account th acoustic dissimilaity on spokn documnts LVCSR tanscipts. Th xpimntal sults showd that ou poposd systm impov th pfomanc compad to baslin systm. Tam Nam SHZU Subtasks SQ-STD Kywods spokn tm dtction, acoustic dissimilaity masu, distanc btwn two distibutions, spokn quy 1. INTRODUCTION Spokn tm dtction (STD) is a task which locats a givn sach tm in a lag st of spokn documnts. A simpl appoach fo STD is a txtual sach on Lag Vocabulay Continuous Spch Rcogniz (LVCSR) tanscipts. Howv, th pfomanc of STD is lagly affctd if th spokn documnts includ out-of-vocabulay (OOV) wods o th LVCSR tanscipts includ cognition os fo invocabulay (IV) wods. Thfo, many appoachs using a subwod-unit basd spch cognition systm hav bn poposd[1, 2, 3, 4. Th kywod spotting mthods fo subwod squncs basd on dynamic tim waping(dtw)- basd matching o n-gam indxing appoachs hav shown th obustnss fo cognition os and OOV poblms. Atsuhiko Kai Shizuoka Univsity Johoku, Hamamatsu-shi, Shizuoka , Japan kai@sys.ng.shizuoka.ac.jp Also, hybid appoachs with multipl spch cognition systms of wod-basd LVCSR and subwod-unit basd spch cogniz hav shown th futh pfomanc impovmnt fo both IV and OOV quy tms[5, 6, 7. In STD of txt quy, w hav poposd an appoach basd on th two-pass spokn tm dtction mthod with statlvl acoustic dissimilaity masus [8 and it is also usd in combination with n-gam confidnc-basd scoing fo impovd STD accuacy[9,. Th mthod fo txt quy basd on nw acoustic fatu psntation, which w call distibution-distanc vcto (DDV), has shown a significant impovmnt compad with simpl DTW-basd matching appoachs[8, 9,. In this pap, w popos a STD appoach in which a spokn quy is convtd into a syllabl squnc by automatic spch cognition and apply ou STD mthod fo txt quy by tating th syllabl squnc as sam as txt quy. 2. BASELINE SPOKEN TERM DETECTION SYSTEM Th baslin systm adopts a DTW-basd spotting mthod which pfoms matching btwn subwod squncs of quy tm and spokn documnts and outputs matchd sgmnts. In NTCIR-9 SpoknDoc STD baslin systm[11, a simila systm with th local distanc masu basd on phonmunit dit distanc is usd. In ou baslin systm, th local distanc masu is dfind by a syllabl-unit acoustic dissimilaity as usd in [6. Th distanc btwn subwods x and y, D sub (x, y), is calculatd by th DTW-basd matching of two subwod HMMs with th local distanc dfind by th distanc btwn two stat s output distibutions. W dfin th distanc btwn two Gaussian mixtu modls P and Q as D BD (P, Q) = min u,v BD(P {u}, Q {v} ) (1) wh BD(P {u}, Q {v} ) dnots th Bhattachayya distanc btwn th u-th Gaussian componnt of P and th v-th Gaussian componnt of Q. At th ppocssing stag, N-bst cognition sults fo a spokn documnt achiv a obtaind by wod-basd and syllabl-basd spch cognition systms with N-gam languag modls of cosponding unit. Thn, th wod-basd cognition sults a convtd into subwod squncs. At th stag of STD fo spokn quy input, th quy 413

2 is convtd into a syllabl squnc by wod-basd and syllabl-basd cognition, and th DTW-basd wod spotting with an asymmtic path constaint is pfomd. Thn, th systm chcks if th quy is composd only of wods in LVCSR systm s lxicon. If th quy is judgd as IV wods, wod-basd cognition sults (convtd into syllabl squnc) a usd. Othwis syllabl-basd cognition sults a usd. Finally, a st of sgmnts with a dissimilaity masu lss than a thshold is obtaind as th tival sult. 3. PROPOSED SPOKEN TERM DETECTION METHOD 3.1 Poposd systm ovviw Ovviw of ou poposd STD systm is shown in Fig. 1. Th systm adopts two-pass statgy fo both fficint pocssing and impovd STD pfomanc against cognition os. On of th fist pass mthods simply pfoms th DTW-basd quy tm spotting as dscibd in Sction 2. Th scond pass is a quy tm vifi which pfoms two kinds of dtaild scoing (scoing) fo ach candidat sgmnt found in th fist pass. Th diffnt appoachs to scoing sqmnts at th fist and scond passs and thi combinations a dscibd in th following sctions. 3.2 IV/OOV discimination of spokn quy tm In poposd STD systm, whn spokn quy is inputtd, th quy is convtd into a syllabl squnc by wodbasd and syllabl-basd automatic spch cognizs. Som ou sults(uns) potd in Sction 4 intoduc an automatic discimination of IV and OOV quis so that wodbasd and syllabl-basd tanscipts a slctivly usd. W stimat if th quy is composd only of wods in LVCSR systm s lxicon o not by using both of wod-basd and syllabl-basd cognition scos. If th quy is composd only of vocabulay wods (IV), wod-basd cognition sults (convtd into syllabl squnc) a usd. If th quy is stimatd to hav OOV wods, syllabl-basd cognition sults a usd. W usd th log-liklihood atio which is dfind in Eq.(2) o Eq.(3) fo IV/OOV discimination. If th log-liklihood atio is lss than a thshold, thn th quy is disciminatd as OOV. LR L = log P L(w) P L (ŵ) wh, P L(w) and P L(ŵ) a languag liklihood scos of th bst candidat w fom wod-basd LVCSR systm and th bst candidat ŵ fom syllabl-basd LVCSR systm, spctivly. LR AL = log P A(X q w)p L (w) P A (X q ŵ)p L (ŵ) wh, P A(X q w) and P A(X q ŵ) a acoustic liklihood scos of th bst candidat w fom wod-basd LVCSR systm and th bst candidat ŵ fom syllabl-basd LVCSR systm, spctivly. 3.3 N-gam confidnc-basd scoing Fo finding th occunc of ctain subwod squnc fom th lattic, n-gam confidnc-basd lvanc scoing (2) (3) has bn ffctivly usd to dal with th cognition o poblm[12. W adopts th n-gam confidnc-basd scoing mthod as an additional filting pocss which pcds o follows th two-pass spotting and scoing passs mntiond in th pvious sction. Th lvanc sco is compad with a thshold paamt to filt out unlikly spch sgmnts bfo th two-pass match is pfomd. Lt th ˆQ = {w 1,, w M } b th stimatd subwod squnc of a spokn quy tm and {w i,, w i+n 1} (i = 1,, M n + 1) dnot patial n-gams of th quy tm. W dfin th lvanc sco R n gam of spch sgmnt X s and quy tm ˆQ fo ach od of n as M n+1 R n gam = i=1 Ẃ W (X s) CM (Ẃ )C(Ẃ, {w i,, w i+n 1}) (4) wh C(Ẃ, {w i,, w i+n 1 }) is th occunc count of n-gam {w i,, w i+n 1} in sntnc hypothsis Ẃ which is includd in subwod lattic W (X s), and CM (Ẃ ) dnots th confidnc sco of sntnc Ẃ as th postioi pobability in lattic W (X s ). Th final lvanc sco is obtaind by Sco CM (X s, ˆQ) N = a n R n gam (5) n=1 wh a n is a wight paamt. In pactic, Eq. (4) is quivalntly calculatd by fficint fowad-backwad algoithm fom a subwod lattic[ Rscoing with stat-lvl psntation (2nd pass) As dscibd in Sction 2, th fist-pass quy tm spotting pfoms DTW-basd matching by using th subwodlvl local distanc mtic D sub (x, y). Th output is a st of alignd subwod squncs which hav th dissimilaity sco blow a thshold. Th scond pass fist xpands th alignd subwod squncs into th cosponding HMM stat squncs and calculats dissimilaity sco basd on a stat-lvl local distanc mtic. A simpl appoach to calculat dissimilaity sco btwn HMM stat squncs is th DTW matching basd on th local distanc masu dfind in (1). Th dissimilaity scos obtaind fo ach candidat sgmnts a compad with a thshold. W f to this dissimilaity sco as Sco BD. Ou pvious study intoducd nw acoustic dissimilaity sco basd on a distanc-vcto psntation which is dfind fo ach HMM stat. Lik a stuctual fatu psntation poposd in [14 and a slf similaity matix in [15, w can consid a fatu psntation fo ach HMM stat basd on th distancs btwn a tagt stat and all stats in a st of subwod-unit HMMs. It is xpctd that such stuctual fatu can stimat mo obust acoustic dissimilaity masu fo compaing th subwod squncs including cognition os. Lt th P = {P s }(s = 1, 2,, S) b a st of all distibutions in subwod-unit HMMs. W dfin a distanc vcto fo th HMM stat s as ϕ(s) = (D BD(P s, P 1), D BD(P s, P 2),, D BD(P s, P S)) T (6) W f to this vcto psntation as distibution-distanc vcto (DDV). 414

3 Spokn Documnt Achiv {X s } Automatic spch cogniz (ASR) Acoustic Modl (subwod unit HMMs) Final Rsults spokn quy X q Us Automatic spch cogniz (ASR) liklihood sco subwod stings IV/OOV IV/OOV discimination subwod stings & confidnc scos Wod / Subwod Lattics subwod-lvl local distanc mtic Spotting with subwod-lvl psntation (1st pass) n-gam confidnc basd scoing(cm) Subwod-lvl DTW (baslin) Spokn tm dtcto Candidat sgmnts P-calculation of acoustic similaity stat-lvl local distanc mtics Rscoing with stat-lvl psntation (2nd pass) Stat-lvl DTW (DP-statBD) DDV-fatu basd scoing (BD-DDV) Figu 1: Ovviw of poposd STD systm To simplify th calculation of dissimilaity sco using th DDV psntation, w can utiliz th alignmnt btwn two stat squncs obtaind as a sult of calculating Sco BD. Lt th F = c 1, c 2,, c k,, c K b th statlvl alignmnt and th c k = (a i, b j ) psnts th cospondnc btwn th i-th stat in HMM stat squnc A = a 1, a 2,, a I and th j-th stat in HMM stat squnc B = b 1, b 2,, b J. W invstigat th following dfinition basd on th DDV psntation. { S } 1/2 max 1 k K s=1 ψ s(c k ) 2 Sco DDV L2max = K S wh ψ s(c k ) is th s-th lmnt of th vcto ϕ(a i) ϕ(b j). Sco DDV L2Max uss th maximum valu of all L2 noms in DDV fatu vcto squncs and thus it mphasizs th most dissimila pat in a subwod squnc. Finally, th abov mntiond dissimilaity scos basd on th stat-lvl psntations can b combind as (7) Sco fusion = α Sco BD + (1 α) τ Sco DDV (8) wh α(0 α 1) is a wight cofficint and τ is a constant fo adjusting th sco ang. To duc th computational cost, th local distanc valus btwn stats can b ppad bfohand by using a st of subwod-unit HMM paamts. 4. EVALUATION 4.1 Expimntal stup W compad two baslin mthods dscibd in Sction 2 (baslin1,baslin2) and svn two-pass mthods dscibd in Sction 3 (SHZU1-7). SHZU1,2 and 3 hav bn submittd to th NTCIR-11 SpoknQuy&Doc SQ-STD task fomal un and thi labls cospond to th un IDs SHZU- SPK1,2 and SHZU-TXT3 psntd in th ovviw pap[16. Additional fou uns (SHZU4-7) a pfomd aft th fomal un submission. In all conditions, w usd fnc automatic tansciption cognizd by matchd modls (REF-WORD-MATCH, REF-SYLLABLE-MATCH) fo tagt documnts. Th diffncs of ths conditions a th cognition mthod of th quy and IV/OOV discimination mthod of th quy. Tabl 1 shows th diffncs of ach mthods. Tabl 1: Conditions of LVCSR systm usd fo tanscibing spokn quy and documnts Languag modl (LM) LM. unit/ IV/OOV dsci. baslin1 nounlm wod baslin2 nounlm syllabl baslin3 manual wod SHZU1 nounlm (Eq.(2)) SHZU2 flm (Eq.(2)) SHZU3 manual wod SHZU4 nounlm flm (Eq.(3)) (Eq.(3)) SHZU6 nounlm wod SHZU7 nounlm syllabl In Tabl 1, Languag modl (LM) column shows th languag modl to b usd in LVCSR systm fo tanscibing th quy. In Languag modl (LM) column, nounlm dnots th wod-unit and syllabl-unit n-gam languag modls taind by noun phass in CSJ copus, whil flm dnots th matchd wod-unit and syllabl-unit languag modls povidd by task oganizs, and manual shows that w usd manual (coct) tansciption. LM unit column shows th cognition unit usd in LVCSR systm fo tanscibing tagt documnt and quy. LM unit column also shows th dfinition usd in IV/OOV discimination of th quy. In LM unit column, wod shows that w tatd all quy as IV wod and syllabl shows that w tatd all quy as OOV wod. In od to adjust th paamts such as thsholds, w usd dy un quy st of th NTCIR-11 SpoknQuy&Doc SQ-STD task as a dvlopmnt st. All paamts a adjustd without distinction btwn IV and OOV quis. Adjusting th wight α in Eq.(8), α = 1 xhibitd bst pfomanc fo th dy un quy st. Thfo, w usd th Sco BD only as Sco fusion. 415

4 Tabl 2: IV/OOV discimination accuacy[ % coct SHZU SHZU SHZU Tabl 4: STD pfomanc of txt quy task (SQ-STD) [% Quy st Rcall Pcision F-masu IV baslin SHZU OOV baslin SHZU All baslin SHZU Tabl 3: STD pfomanc of spokn quy task (SQ-STD) [% Quy st Rcall Pcision F-masu baslin baslin SHZU IV SHZU SHZU SHZU SHZU baslin baslin SHZU OOV SHZU SHZU SHZU SHZU baslin baslin SHZU All SHZU SHZU SHZU SHZU NTCIR-11 SQ-STD task sults Tabl 2 shows IV/OOV discimination accuacy fo ach mthod. Sinc th IV at of quy st fo NTCIR-11 SpoknQuy&Doc SQ-STD task fomal un is 97.54%, and th quy tms a dominatd by IV cass, th intoduction of IV/OOV discimination dos not poduc significant sult. nt NTCIR-11 task, it sms that th sult is consistnt Tabl 3 shows th sults (call, pcision, and F-masu(max)) with th fact that ou DDV-basd appoach xhibitd mo fo spokn quy STD mthods (baslin1,2 and SHZU1-7 significant ffct fo OOV quy tms as shown in Fig. 5. xcluding SHZU3). Tabl 4 shows th sults fo txt quy STD mthods (baslin3 and SHZU3). F-masu(max) is th maximum valu of F-masu whn th thshold is adjustd. Fig. 2,3,4 shows th call-pcision cuvs of ach mthod. Th sult shows that th two-pass mthods (SHZU1-7) outpfoms th baslin mthods which us only th fist pass. Th sult shows that th uns with nounlm (SHZU1,4) xhibit btt pfomanc in compad with th uns with flm (SHZU2,5). As fo th ffct of IV/OOV discimination, th uns with Eq.(3) (SHZU4,5) achivd btt pfomanc in compad with th uns with Eq.(2) (SHZU1,2) dspit th loss of IV/OOV discimination accuacy. It should b notd that th un (SHZU6) which didn t apply IV/OOV discimination (which cospond to IV/OOV discimina- 70 [ % n io is c P Rcall [% txt quy SHZU1 SHZU2 SHZU3 SHZU4 SHZU6 SHZU7 baslin1 baslin2 baslin3 Figu 2: Rcall-Pcision cuvs(iv quy tms) tion accuacy of 97.54%) dosn t impov th pfomanc in compad with SHZU4. Th diffnc of pfomanc btwn SHZU3 and th oth two-pass mthods is quit lag. This sms to b du to th poo accuacy of LVCSR systm fo spokn quis. As dscibd abov, w adjustd paamt without distinction btwn IV and OOV quis. Thfo, by adjusting th paamts spaatly fo ach of IV and OOV quy typs, it would b possibl to impov th pfomanc. W also adjustd th wight α to 1 and didn t us Sco DDV in Eq.(8). Howv, in STD by txt quy of NTCIR- SpoknDoc-2 modat-siz task [17 in which a subst of spokn documnts fo NTCIR-11 is usd, it is shown that it is ffctiv to us th Sco DDV. Fo you infomation, w show th ffct of th sco wight paamt α fo NTCIR- SpoknDoc-2 modat-siz task in Fig. 5 [, 18. Although dtaild analysis is quid in tms of diffnt consqunc btwn pio NTCIR- and cu- 5. CONCLUSIONS In this pap, w intoducd spokn tm dtction mthod using multipl scoing and dissimilaity masus fo spokn quy. Expimntal sult shows that two-pass spokn tm dtction mthod in which th stat-lvl acoustic dissimilaity and using th languag modl taind by noun phass fo cognition of vaious typs of spokn quy is ffctiv in impoving STD pfomanc. Sinc ou mthod is a simpl xtnsion of th convntional DTW-basd mthod, it is staightfowad to combin with indxing tchniqus (.g. [6) fo spding up ou STD systm. Also, an automatic stimation of optimal paamts, such as a sco thshold and wight, o sco nomalization 416

5 70 SHZU1 SHZU2 55 [ % n io is c P txt quy SHZU3 SHZU4 SHZU6 45 [% s u 35 a -m F 25 Baslin(IV) Baslin(OOV) BD-DDV(IV) SHZU7 baslin1 15 BD-DDV(OOV) Rcall [% baslin2 baslin wight(α) Figu 3: Rcall-Pcision cuvs(oov quy tms) 70 [ % n io is c P Rcall [% txt quy Figu 4: Rcall-Pcision cuvs(all) SHZU1 SHZU2 SHZU3 SHZU4 SHZU6 SHZU7 baslin1 baslin2 baslin3 mthods[19 a ncssay to achiv futh impovmnt and obustnss fo spokn documnts in th al wold. 6. REFERENCES [1 Y. Itoh, t al.: Constucting Japans Tst Collctions fo Spokn Tm Dtction, Poc. of INTERSPEECH, pp (). [2 K. Iwami, t al.: Out-of-vocabulay tm dtction by n-gam aay with distanc fomcontinuous syllabl cognition sults, Poc. of Spokn Languag Tchnology Wokshop, pp (). [3 N. Aiwadhani, t al.: Phonm Rcognition Basd on AF-HMMs with an Optimal Paamt St, Poc. of NCSP, pp (12). [4 N. Kanda, t al.: Opn-vocabulay kywod dtction fom sup-lag scal spch databas, Poc. of MMSP, pp (08). [5 K.Iwami, t al.: Efficint out-of-vocabulay tm dtction by N-gam aay in dics with distanc fom a syllabl lattics, Poc. of ICASSP, pp (11). [6 S.Nakagawa. t al.: A obust/fast spokn tm dtction mthod basd on a syllabl n-gam indx with a distanc mtic, Spch Communication, Vol.55, pp (13). [7 H. Nishizaki, t al. : Spokn Tm Dtction Using Multipl Spch Rcognizs Outputs at NTCIR-9 SpoknDoc STD subtask, Poc. of NTCIR-9 Wokshop Mting, pp (11). Figu 5: Effct of th sco wight paamt α (STD systm with Sco DDV L2Max ). Th cuv IV and OOV show th bakdown of STD pfomanc fo invocabulay quis and out-of-vocabulay quis, spctivly. (NTCIR- SpoknDoc-2 modat-siz task) [8 N. Yamamoto and A. Kai : Spokn Tm Dtction Using Distanc-Vcto basd Dissimilaity Masus and Its Evaluation on th NTCIR- SpoknDoc-2 Task, Poc. of th th NTCIR Wokshop Mting, pp , (13). [9 N. Yamamoto and A. Kai : Using acoustic dissimilaity masus basd on stat-lvl distanc vcto psntation fo impovd spokn tm dtction, Poc. of APSIPA ASC 13 (13). [ M. Makino, N. Yamamoto and A. Kai : Utilizing Stat-lvl Distanc Vcto Rpsntation fo Impovd Spokn Tm Dtction by Txt and Spokn Quis, Poc. of INTERSPEECH (14). [11 T. Akiba, t al.: Ovviw of th IR fo Spokn Documnts Task in NTCIR-9 Wokshop, Poc. of NTCIR-9 Wokshop Mting, pp (11). [12 H. L, P. Chou and L. L : Opn-vocabulay tival of spokn contnt with shot/long quis considing wod/subwod-basd acoustic fatu similaity, Poc. of INTERSPEECH (12). [13 F. Wssl, R. Schlut, K. Machy and H. Ny : Confidnc masus fo lag vocabulay continuous spch cognition, IEEE Tans. on Spch and Audio Pocssing, Vol.9, No.3, pp (01). [14 N. Minmatsu t al.: Stuctual psntation of th ponunciation and its us fo CALL, Poc. of Spokn Languag Tchnology Wokshop, pp (06). [15 A. Muscaillo, t al.: Zo-souc audio-only spokn tm dtction basd on a combination of tmplat matching tchniqus, Poc. of INTERSPEECH, pp (11). [16 Tomoyosi Akiba, Hiomitsu Nishizaki, Hioaki Nanjo and Gath J. F. Jons : Ovviw of th NTCIR-11 SpoknQuy&Doc task, In Pocdings of th NTCIR-11 Confnc, Tokyo, Japan, (14). [17 Tomoyosi Akiba, Hiomitsu Nishizaki, Kiyoaki Aikawa, Xinhui Hu, Yoshiaki Itoh, Tatsuya Kawahaa, Siichi Nakagawa, Hioaki Nanjo and Yoichi Yamashita : Ovviw of th NTCIR- SpoknDoc-2 Task, Poc. of th th NTCIR Wokshop Mting, (13). 417

6 [18 M. Makino, N. Yamamoto and A. Kai : Using Acoustic Dissimilaity Masus Basd on Distanc Vcto Rpsntation fo Impovd Spokn Tm Dtction by Txt and Spokn Quis, Poc. of th ighth annual Spokn Documnt Pocssing Wokshop(SDPWS) (14). (in Japans) [19 B. Zhang, t al.: Whit Listing and Sco Nomalization fo Kywod Spotting of Noisy Spch, Poc. of INTERSPEECH (12). 418