Empirical Estimates of Adaptation: The chance of Two Noriegas is closer to p /2 than p 2

Transcription

1 Emirical Estimates f Adatati: Te cace f Tw Nriegas is clser t / ta Keet W. Curc AT&T Labs-Researc, 80 Park Ave., Flram Park, NJ., USA kwc@researc.att.cm Abstract Reetiti is very cmm. Adative laguage mdels, wic allw rbabilities t cage r adat after seeig just a few wrds f a text, were itrduced i seec recgiti t accut fr text cesi. Suse a dcumet metis Nriega ce. Wat is te cace tat e will be metied agai? If te first istace as rbability, te uder stadard (bag-f-wrds) ideedece assumtis, tw istaces ugt t ave rbability, but we fid te rbability is actually clser t /. Te first meti f a wrd bviusly deeds frequecy, but surrisigly, te secd des t. Adatati deeds mre lexical ctet ta frequecy; tere is mre adatati fr ctet wrds (rer us, tecical termilgy ad gd keywrds fr ifrmati retrieval), ad less adatati fr fucti wrds, clices ad rdiary first ames.. Itrducti Adative laguage mdels were itrduced i te Seec Recgiti literature t mdel reetiti. Jeliek (997,. 54) describes cace-based mdels wic cmbie tw estimates f wrd (gram) rbabilities, Pr L, a lcal estimate based a relatively small cace f recetly see wrds, ad Pr G, a glbal estimate based a large traiig crus.. Additive: Pr A (w) = λpr L (w) + ( λ) Pr G (w). Case-based: Pr C (w) = λ Pr L (w) if w cace λ Pr G (w) terwise Ituitively, if a wrd as bee metied recetly, te (a) te rbability f tat wrd (ad related wrds) suld g way u, ad (b) may ter wrds suld g dw a little. We will refer t (a) as sitive adatati ad (b) as egative adatati. Our emirical exerimets cfirm te ituiti tat sitive adatati, Pr( + adat), is tyically muc larger ta egative adatati, Pr( adat). Tat is, Pr( + adat) >> Pr(rir) > Pr( adat). Tw metds, Pr( + adat ) ad Pr( + adat ), will be itrduced fr estimatig sitive adatati.. Pr( + adat ) = Pr(w test w istry). Pr( + adat ) = Pr(k k ) df / df Te tw metds rduce similar results, usually well witi a factr f tw f e ater. Te first metd slits eac dcumet it tw equal ieces, a istry rti ad a test rti. Te adated rbabilities are mdeled as te cace tat a wrd will aear i te test rti, give tat it aeared i te istry. Te secd metd, suggested by Curc ad Gale (995), mdels adatati as te cace f a secd meti (rbability tat a wrd will aear tw r mre times, give tat it aeared e r mre times). Pr( + adat ) is arximated by df / df, were df k is te umber f dcumets tat ctai te wrd/gram k r mre times. (df k is a geeralizati f dcumet frequecy, df, a stadard term i Ifrmati Retrieval.) Bt metds are -arametric (ulike cace mdels). Parametric assumtis, we arriate, ca be very werful (better estimates frm less traiig data), but errrs resultig frm iarriate assumtis ca utweig te beefits. I tis emirical ivestigati f te magitude ad sae f adatati we decided t use cservative -arametric metds t edge agaist te risk f iarriate arametric assumtis. Te tw lts (belw) illustrate sme f te reass fr beig ccered abut stadard arametric assumtis. Te first lt sws te umber f times tat te wrd said aears i eac f te 500 dcumets i te Brw Crus (Fracis & Kucera, 98). Nte tat tere are quite a few dcumets wit mre ta 5 istaces f said, esecially i Press ad Ficti. Tere are als quite a few dcumets wit ardly ay istaces f said, esecially i te Leared gere. We ave fud a similar atter i ter cllectis; said is mre cmm i ewswire (Assciated Press ad Wall Street Jural) ta tecical writig (Deartmet f Eergy abstracts).

2 Te secd lt (belw) cmares tese Brw Crus bservatis t a Piss. Te circles idicate te umber f dcumets tat ave x istaces f said. As metied abve, Press ad Ficti dcumets ca meti said 5 times r mre, wile dcumets i te Leared gere migt t meti te wrd at all. Te lie sws wat wuld be exected uder a Piss. Clearly te lie des t fit te circles very well. Te rbability f said deeds may factrs (e.g, gere, tic, style, autr) tat make te distributis brader ta cace (Piss). We fid esecially brad distributis fr wrds tat adat a lt. frequecy umber f dcs said i Brw Crus Press Religi Hbbies Lre Belle-Lettres Gv t Leared dcumet umber Piss Des t Fit freq Ficti Humr We will sw tat adatati is uge. Pr( + adat) is fte several rders f magitude larger ta Pr(rir). I additi, we fid tat Pr( + adat) as a very differet sae frm Pr(rir). By cstructi, Pr(rir) varies ver may rders f magitude deedig te frequecy f te wrd. Iterestigly, tug, we fid tat Pr( + adat) as almst deedece wrd frequecy, altug tere is a strg lexical deedece. Sme wrds adat mre ta ters. Te result is quite rbust. Wrds tat adat mre i e crus als ted t adat mre i ater crus f similar material. Bt te magitude ad esecially te sae (lack f deedece frequecy as well as deedece ctet) are ard t cature i a additive-based cace mdel. Later i te aer, we will study eigbrs, wrds tat d t aear i te istry but d aear i dcumets ear te istry usig a ifrmati retrieval ti f ear. We fid tat eigbrs adat mre ta -eigbrs, but t as muc as te istry. Te sae is i betwee as well. Neigbrs ave a mdest deedecy frequecy, mre ta te istry, but t as muc as te rir. Neigbrs are a extesi f Flria & Yarwsky (999), w used tic clusterig t build a laguage mdel fr ctexts suc as: It is at least te Serb side a real setback t te x. Teir wrk was mtivated by seec recgiti alicatis were it wuld be desirable fr te laguage mdel t favr x = eace ver x = iece. Obviusly, acustic evidece is t very elful i tis case. Trigrams are als t very elful because te strgest clues (e.g., Serb, side ad setback ) are beyd te widw f tree wrds. Flria & Yarwsky cluster dcumets it abut 0 tics, ad cmute a searate trigram laguage mdel fr eac tic. Neigbrs are similar i sirit, but surt mre tics.. Estimates f Adatati: Metd Metd slits eac dcumet it tw equal ieces. Te first alf f eac dcumet is referred t as te istry rti f te dcumet ad te secd alf f eac dcumet is referred t as te test rti f te dcumet. Te task is t redict te test rti f te dcumet give te istry. We start by cmutig a ctigecy table fr eac wrd, as illustrated belw: Dcumets ctaiig stages i 990 AP test test istry a =68 b =505 istry c =557 d =76787 Tis table idicates tat tere are (a) 68 dcumets wit stages i bt te first alf (istry) ad te secd alf (test), (b) 505 dcumets wit stages i just te first alf, (c) 557 dcumets wit stages i just te secd alf, ad (d) 76,787 dcumets wit stages i eiter alf. Psitive ad egative adatati are defied i terms a, b, c ad d.

3 Pr( + adat ) = Pr(w test w istry) a a + b Pr( adat ) = Pr(w test w istry) c c + d Adated rbabilities will be cmared t: Pr(rir) = Pr(w test) (a + c)/ D were D = a + b + c + d. Psitive adatati teds t be muc larger ta te rir, wic is just a little larger ta egative adatati, as illustrated i te table belw fr te wrd stages i fur years f te Assciated Press (AP) ewswire. We fid remarkably csistet results we we cmare e year f te AP ews t ater (tug tics d cme ad g ver time). Geerally, te differeces f iterest are uge (rders f magitude) cmared t te differeces amg varius ctrl cditis (at mst factrs f tw r tree). Nte tat values are mre similar witi clums ta acrss clums. Pr(+adat) >> Pr(rir) > Pr( adat) rir +adat adat surce w AP87 stages AP AP AP9. Adatati is Lexical We fid tat sme wrds adat mre ta ters, ad tat wrds tat adat mre i e year f te AP als ted t adat mre i ater year f te AP. I geeral, wrds tat adat a lt ted t ave mre ctet (e.g., gd keywrds fr ifrmati retrieval (IR)) ad wrds tat adat less ave less ctet (e.g., fucti wrds). It is fte assumed tat wrd frequecy is a gd (iverse) crrelate f ctet. I te sycliguistic literature, te term ig frequecy is fte used syymusly wit fucti wrds, ad lw frequecy wit ctet wrds. I IR, iverse dcumet frequecy (IDF) is cmmly used fr weigtig keywrds. Te table belw is iterestig because it questis tis very basic assumti. We cmare tw wrds, Keedy ad excet, tat are abut equally frequet (similar rirs). Ituitively, Keedy is a ctet wrd ad excet is t. Tis ituiti is surted by te adatati statistics: te adatati rati, Pr( + adat)/ Pr(rir), is muc larger fr Keedy ta fr excet. A similar atter lds fr egative adatati, but i te reverse directi. Tat is, Pr( adat)/ Pr(rir) is muc smaller fr Keedy ta fr excet. Keedy adats mre ta excet _ rir +adat adat surce w AP90 Keedy AP AP AP90 excet AP AP9 I geeral, we exect mre adatati fr better keywrds (e.g., Keedy ) ad less adatati fr less gd keywrds (e.g., fucti wrds suc as excet ). Tis bservati rus cuter t te stadard ractice f weigtig keywrds slely te basis f frequecy, witut csiderig adatati. I a related aer, Umemura ad Curc (submitted), we describe a term weigtig metd tat makes use f adatati (smetimes referred t as burstiess). Distictive surames adat mre ta rdiary first ames rir +adat adat surce w AP90 Nriega AP AP90 Aristide AP AP90 Escbar AP AP90 J AP AP90 Gerge AP AP90 Paul AP9 Te table abve cmares surames wit first ames. Tese surames are excellet keywrds ulike te first ames, wic are early as useless fr IR as fucti wrds. Te adatati rati, Pr( + adat)/ Pr(rir), is muc larger fr te surames ta fr te first ames. Wat is te rbability f seeig tw Nriegas i a dcumet? Te cace f te first e is Accrdig t te table abve, te cace f tw is abut 0. 75, clser t / ta. Fidig a rare wrd like Nriega i a dcumet is like ligtig. We migt t exect

4 ligtig t strike twice, but it aes all te time, esecially fr gd keywrds. 4. Smtig (fr lw frequecy wrds) Tus far, we ave see tat adatati ca be large, but t demstrate te sae rerty (lack f deedece frequecy), te cuts i te ctigecy table eed t be smted. Te rblem is tat te estimates f a, b, c, d, ad esecially estimates f te ratis f tese quatities, becme ustable we te cuts are small. Te stadard metds f smtig i te seec recgiti literature are Gd-Turig (GT) ad Held-Out (HO), described i sectis 5. & 5.4 f Jeliek (997). I bt cases, we let r be a bserved cut f a bject (e.g., te frequecy f a wrd ad/r gram), ad r * be ur best estimate f r i ater crus f te same size (all ter tigs beig equal). 4. Stadard Held-Out (HO) HO slits te traiig crus it tw alves. Te first alf is used t cut r fr all bjects f iterest (e.g., te frequecy f all wrds i vcabulary). Tese cuts are te used t gru bjects it bis. Te r t bi ctais all (ad ly) te wrds wit cut r. Fr eac bi, we cmute N r, te umber f wrds i te r t bi. Te secd alf f te traiig crus is te used t cmute C r, te aggregate frequecy f all te wrds i te r t bi. Te fial result is simly: r * = C r / N r If te tw alves f te traiig crra r te test crra ave differet sizes, te r * suld be scaled arriately. We cse HO i tis wrk because it makes few assumtis. Tere is arametric mdel. All tat is assumed is tat te tw alves f te traiig crus are similar, ad tat bt are similar t te testig crus. Eve tis assumti is a matter f sme ccer, sice majr stries cme ad g ver time. 4. Alicati f HO t Ctigecy Tables As abve, te traiig crus is slit it tw alves. We used tw differet years f AP ews. Te first alf is used t cut dcumet frequecy df. (Dcumet frequecy will be used istead f stadard (term) frequecy.) Wrds are bied by df ad by teir cell i te ctigecy table. Te first alf f te crus is used t cmute te umber f wrds i eac bi: N df,a, N df,b, N df,c ad N df,d ; te secd alf f te crus is used t cmute te aggregate dcumet frequecy fr te wrds i eac bi: C df,a, C df,b, C df,c ad C df,d. Te fial result is simly: a * df = C df,a / N df,a, b * df = C df,b / N df,b, c * df = C df,c / N df,c ad d * df = C df,d / N df,d. We cmute te rbabilities as befre, but relace a, b, c, d wit a *, b *, c *, d *, resectively. Prbability Histry () >> Neigbrd () >> Prir () Dcumet Frequecy (df) Wit tese smted estimates, we are able t sw tat Pr( + adat), labeled i te lt abve, is larger ad less deedet frequecy ta Pr(rir), labeled. Te lt sws a tird gru, labeled fr eigbrs, wic will be described later. Nte tat te s fall betwee te s ad te s. Tus far, we ave see tat adatati ca be uge: Pr( + adat) >> Pr(rir), fte by tw r tree rders f magitude. Peras eve mre surrisigly, altug te first meti deeds strgly frequecy (df), te secd des t. Sme wrds adat mre (e.g., Nriega, Aristide, Escbar) ad sme wrds adat less (e.g., J, Gerge, Paul). Te results are rbust. Wrds tat adat mre i e year f AP ews ted t adat mre i ater year, ad vice versa. 5. Metd : Pr( + adat ) S far, we ave limited ur atteti t te relatively simle case were te istry ad te test are te same size. I ractice, tis w t be te case. We were ccered tat te bservatis abve migt be artifacts smew caused by tis limitati. We exerimeted wit tw araces fr uderstadig te effect f tis limitati ad fud tat te size f te istry des t cage Pr( + adat) very muc. Te first arac slit te istry ad te test at varius its ragig frm 5% t 95%. Geerally, Pr( + adat ) icreases as te size f te test rti grws relative t te size f te istry, but te effect is

5 relatively small (mre like a factr f tw ta a rder f magitude). We were eve mre cviced by te secd arac, wic uses Pr( + adat ), a cmletely differet argumet fr estimatig adatati ad des t deed te relative size f te istry ad te test. Te tw metds rduce remarkably similar results, usually well witi a factr f tw f e ater (eve we adated rbabilities are rders f magitude larger ta te rir). Pr( + adat ) makes use f df j (w), a geeralizati f dcumet frequecy. df j (w) is te umber f dcumets wit j r mre istaces f w; (df is te stadard ti f df). Pr( + adat ) = Pr(k k ) = df / df Metd as sme advatages ad sme disadvatages i cmaris wit metd. O te sitive side, metd ca be geeralized t cmute te cace f a tird istace: Pr(k k ). But ufrtuately, we d t kw w t use metd t estimate egative adatati; we leave tat as a e questi. Adatati is uge (ad ardly deedet frequecy) Dcumet Frequecy (df) Prbability Pr(k >= ) = df* / D Pr(k >= k >= ) = df* / df* Pr(k >= k >= ) = df* / df* Te lt (abve) is similar t te lt i secti 4. wic swed tat adated rbabilities (labeled ) are larger ad less deedet frequecy ta te rir (labeled ). S t, te lt (abve) sws tat te secd ad tird metis f a wrd (labeled ad, resectively) are larger ad less deedet frequecy ta te first meti (labeled ). Te lt i secti 4. used metd wereas te lt (abve) uses metd. Bt lts use te HO smtig, s tere is ly e it er bi (df value), rater ta e er wrd. 6. Neigbrds (Near) Flria ad Yarwsky s examle, It is at least te Serb side a real setback t te x, rvides a ice mtivati fr eigbrds. Suse te ctext (istry) metis a umber f wrds related t a eace rcess, but des t meti te wrd eace. Ituitively, tere suld still be sme adatati. Tat is, te rbability f eace suld g u quite a bit (sitive adatati), ad te rbability f may ter wrds suc as iece suld g dw a little (egative adatati). We start by artitiig te vcabulary it tree exaustive ad mutually exclusive sets: ist, ear ad ter (abbreviatis fr istry, eigbrd ad terwise, resectively). Te first set, ist, ctais te wrds tat aear i te first alf f te dcumet, as befre. Oter is a catcall fr te wrds tat are i eiter f te first tw sets. Te iterestig set is ear. It is geerated by query exasi. Te istry is treated as a query i a ifrmati retrieval dcumetrakig egie. (We imlemeted ur w rakig egie usig simle IDF weigtig.) Te eigbrd is te set f wrds tat aear i te k 0 r k 00 t dcumets retured by te retrieval egie. T esure tat te tree sets artiti te vcabulary, we exclude te istry frm te eigbrd: ear = wrds i query exasi f ist ist Te adatati rbabilities are estimated usig a ctigecy table like befre, but we w ave a tree-way artiti (ist, ear ad ter) f te vcabulary istead f te tw-way artiti, as illustrated belw. Dcumets ctaiig eace i 99 AP test istry a =5 b =60 c =96 d =7457 test ist a =5 b =60 ear e =479 f =56 ter g =484 =5057 I estimatig adatati rbabilities, we ctiue t use a, b, c ad d as befre, but fur ew variables are itrduced: e, f, g ad, were c =e + g ad d = f +.

6 Pr(w test) (a + c)/ D Pr(w test w ist) a /(a + b) Pr(w test w ear) e /(e + f ) Pr(w test w ter) g /(g + ) rir ist ear ter Te table belw sws tat Keedy adats mre ta excet ad tat eace adats mre ta iece. Tat is, Keedy as a larger sread ta excet betwee te istry ad te terwise case. _ rir ist ear ter src w AP9 Keedy AP AP9 excet AP AP9 eace AP AP9 iece AP9 We df is small (df < 00), HO smtig is used t gru wrds it bis by df. Adatati rbabilities are cmuted fr eac bi, rater ta fr eac wrd. Sice tese rbabilities are imlicitly cditial df, tey ave already bee weigted by df i sme sese, ad terefre, it is uecessary t itrduce a additial exlicit weigtig sceme based df r a simle trasfrm teref suc as IDF. Te exerimets belw slit te eigbrd it fur classes, ragig frm better eigbrs t wrse eigbrs, deedig exasi frequecy, ef. ef (t) is a umber betwee ad k, idicatig w may f te k t scrig dcumets ctai t. (Better eigbrs aear i mre f te t scrig dcumets, ad wrse eigbrs aear i fewer.) All te eigbrd classes fall betwee ist ad ter, wit better eigbrs adatig mre ta wrse eigbrs. 7. Exerimetal Results Recall tat te task is t redict te test rti (te secd alf) f a dcumet give te istry (te first alf). Te fllwig table sws a selecti f wrds (srted by te tird clum) frm te test rti f e f te test dcumets. Te table is searated it tirds by riztal lies. Te wrds i te t tird receive muc iger scres by te rsed metd (S) ta by a baselie (B). Tese wrds are suc gd keywrds tat e ca fairly cfidetly guess wat te stry is abut. Mst f tese wrds receive a ig scre because tey were metied i te istry rti f te dcumet, but laid-ff receives a ig scre by te eigbrd mecaism. Altug laid-ff is t metied exlicitly i te istry, it is bviusly clsely related t a umber f wrds tat were, esecially layffs, but als tices ad cuts. It is reassurig t see te eigbrd mecaism dig wat it was desiged t d. Te middle tird sws wrds wse scres are abut te same as te baselie. Tese wrds ted t be fucti wrds ad ter lw ctet wrds tat give us little sese f wat te dcumet is abut. Te bttm tird ctais wrds wse scres are muc lwer ta te baselie. Tese wrds ted t be ig i ctet, but misleadig. Te wrd arms, fr examle, migt suggest tat stry is abut a military cflict. S B lg(s/b) Set Term ist Bider ist layff ist tices ist Beig ear laid-ff ist cuts ear rjects ist said ear4 auced ear4 As ear emlyed ter ter maaged ear additial ter wave ter arms Te rsed scre, S, sw i clum, is: Pr(w ist) if w ist Pr(w ear ) if w ear Pr(w ear Pr S (w) = ) if w ear Pr(w ear ) if w ear Pr(w ear 4 ) if w ear 4 Pr(w ter) terwise were ear trug ear 4 are fur eigbrds (k = 00). Wrds i ear 4 are te best eigbrs (ef 0) ad wrds i ear are te wrst eigbrs (ef = ). Te baselie, B, sw i clum, is: Pr B (w) = df / D. Clum cmares te first tw clums. We alied tis rcedure t a year f te AP ews ad fud a sizable gai i ifrmati

7 average: 0.75 bits er wrd tye er dcumet. I additi, tere were may mre big wiers (0% f te dcumets gaied bit/tye) ta big lsers (0% lst bit/tye). Te largest wiers iclude lists f majr cities ad teir temeratures, lists f majr currecies ad teir rices, ad lists f cmmdities ad teir rices. Neigbrds are quite successful i guessig te secd alf f suc lists. O te ter ad, tere were a few big lsers, e.g., articles tat summarize te majr stries f te day, week ad year. Te secd alf f a summary article is almst ever abut te same subject as te first alf. Tere were als a few ed-f-dcumet delimiters tat were garbled i trasmissi causig tw differet dcumets t be treated as if tey were e. Tese garbled dcumets teded t cause truble fr te rsed metd; i suc cases, te istry cmes frm e dcumet ad te test cmes frm ater. I geeral, te rsed adatati metd erfrmed well we te istry is elful fr redictig te test rti f te dcumet, ad it erfrmed rly we te istry is misleadig. Tis suggests tat we ugt t measure tic sifts usig metds suggested by Hearst (994) ad Flria & Yarwsky (999). We suld t use te istry we we believe tat tere as bee a majr tic sift. 8. Cclusis Adative laguage mdels were itrduced t accut fr reetiti. It is well kw tat te secd istace f a wrd (r gram) is muc mre likely ta te first. But wat we fid surrisig is just w large te effect is. Te cace f tw Nriegas is clser t / ta. I additi t te magitude f adatati, we were als surrised by te sae: wile te first istace f a wrd deeds very strgly frequecy, te secd des t. Adatati deeds mre ctet ta frequecy; adatati is strger fr ctet wrds suc as rer us, tecical termilgy ad gd keywrds fr ifrmati retrieval, ad weaker fr fucti wrds, clices ad first ames. Te sae ad magitude f adatati as imlicatis fr sycliguistics, ifrmati retrieval ad laguage mdelig. Psycliguistics as teded t equate wrd frequecy wit ctet, but ur results suggest tat tw wrds wit similar frequecy (e.g., Keedy ad excet ) ca be distiguised te basis f teir adatati. Ifrmati retrieval as teded t use frequecy i a similar way, weigtig terms by IDF (iverse dcumet frequecy), wit little atteti aid t adatati. We rse a term weigtig metd tat makes use f adatati (burstiess) ad exasi frequecy i a related aer (Umemura ad Curc, submitted). Tw estimati metds were itrduced t demstrate te magitude ad sae f adatati. Bt metds rduce similar results. Pr( + adat ) = Pr(test ist) Pr( + adat ) = Pr(k k ) Neigbrds were te itrduced fr wrds suc as laid-ff tat were t i te istry but were clse ( laid-ff is related t layff, wic was i te istry). Neigbrds were defied i terms f query exasi. Te istry is treated as a query i a ifrmati retrieval dcumet-rakig system. Wrds i te k trakig dcumets (but t i te istry) are called eigbrs. Neigbrs adat mre ta ter terms, but t as muc as wrds tat actually aeared i te istry. Better eigbrs (larger ef) adat mre ta wrse eigbrs (smaller ef). Refereces Curc, K. ad Gale, W. (995) Piss Mixtures, Jural f Natural Laguage Egieerig, :, Flria, R. ad Yarwsky, D. (999) Dyamic Nlcal Laguage Mdelig via Hierarcical Tic-Based Adatati, ACL, Fracis, W., ad Kucera, H. (98) Frequecy Aalysis f Eglis Usage, Hugt Miffli Cmay, Bst, MA. Hearst, M. (994) Ctext ad Structure i Autmated Full-Text Ifrmati Access, PD Tesis, Berkeley, available via earst. Jeliek, F. (997) Statistical Metds fr Seec Recgiti, MIT Press, Cambridge, MA, USA. Umemura, K. ad Curc, K. (submitted) Emirical Term Weigtig: A Framewrk fr Studyig Limits, St Lists, Burstiess ad Query Exasi.