A Text-based HMM of Foreign Affair Sentiment

Transcription

1 A Text-baed HMM of Foreign Affair Sentiment Sean M. Gerrih epartment of Computer Science Princeton Univerity Princeton, NJ avid M. Blei epartment of Computer Science Princeton Univerity Priceton, NJ Abtract We preent a time-erie model for foreign relation, in hich the pairie entiment beteen nation i inferred from ne article. We decribe a model of dyadic interaction and illutrate our proce of etimating entiment uing Amazon Mechanical Turk label. Acro article from tenty year of the of the Ne York Time, e predict ith modet error on heldout country pair. The ritten hitory of foreign relation i peppered ith the bia of politic and hindight. Becaue hitorian currently have limited reource to revie thouand of relevant ne ource, reearch may be biaed by popular knoledge or even politic and culture. The goal of thi ork i to create a hitory of the relationhip beteen the orld nation uing the text of nepaper article and political commentary. An aumption of our ork i that the tenion beteen to nation or a arm and robut relationhip beteen them i reflected by the language e ue to dicu them. Uing thi aumption, e outline a model that i able to infer the relationhip beteen pair of countrie hoe relationhip ha not been oberved in training. An advantage of a text-baed approach to hitory i that e can incorporate information from all article of a given collection ith modet computational cot. Thi mean that hitorian and political cientit can then earch and revie thouand of hitorical document to identify forgotten or overlooked blip in hitory. 1 Model Reflecting the intuition above, e aume that each nation lie in a pace of latent foreign entiment. A patial a model ha to benefit. Firt, it provide interpretability: nation ith imilar poition in thi latent pace tend to interact more poitively, hile nation further apart tend to have more tenion in their relationhip. A patial model alo allo u to dra on idea from multidimenional caling, hich ha been ued uccefully in both political cience [1, 2] and ocial netork modeling [3, 4]. In the model e outline belo, e aume that each country c take a poition x c IR P in the pace of latent political entiment. International relation are determined by the interaction of countrie poition x c. A temporal model of interaction. Foreign relation are not tatic, hoever; nation alliance and preference change over time ith the evolution of economie, tech- β xc,t-1 xc,t xc,t+1 xc,t-1 xc,t xc,t t-1 t t+1 Figure 1: A time-erie model of countrie interaction. Peudo-obervation of are added for regularization. Amazon Mechanical Turk label are ued to fit β, hich i ued to infer unoberved entiment. C 1

2 nology, and culture. We make an aumption that each piece of ne d about to nation c 1 and c 2 reflect ome underlying entiment d beteen them. We make thi a fully temporal model by alloing each country mean poition x c,t to drift over time ith the Markov tranition x ct x c,t 1 N( x c,t 1,σ 2 K), (1) a hon in Figure 1. At any time t, tate c 1 may interact ith tate c 2 in the folloing ay: x c1,d N( x c1,t,σ 2 ) x c2,d N( x c2,t,σ 2 ) d := x T c 1,d x c 2,d, here e interpret d a the entiment beteen c 1 and c 2 a reflected by article d. When c 1 and c 2 are imilar (a meaured by their inner product), their entiment d ill be poitive; if they are diimilar, their entiment ill be negative. More extreme value indicate tronger entiment. When a ne ource dicue the relationhip beteen thee nation, the author choice of ord d reflect the relationhip beteen the countrie. We model thi entiment ith the text of the article d. Uing text regreion [5], the entiment i modeled on ordcount d of the article: d d,β N ( T d β,σ2 W ). We decribe ho to fit β ith Amazon Mechanical Turk orker in Section Related ork Spatial model uch a Item Repone Theory (IRT) have been developed over the pat century by quantitative ocial cientit for analyzing behavior. While much of thi ork ha been ued to model parliamentary voting behavior, thee technique have alo been ued to model voting in the UN General Aembly. Gartzke et al., for example, ue thee vote and alliance model to tudy the nation affinitie [6]. Thee model have been developed for dyadic data more fully in netork model uch a the latent pace model [3, 7], in hich the probability of a link beteen to node i a function of their latentpace ditance. The qualitative relationhip of entitie dyadic relationhip ha been more fully developed ith text by the relational topic model, hich ue free text to model the relationhip beteen actor in an unupervied etting [4]. 2 Inference We fit the MAP objective of thi probabilitic model. Thi ha the benefit of both clean expoition and imple implementation, and it can be interpreted a a form of unregularized variational inference. We optimize the MAP objective in thi model uing traditional EM. M Step. In the M tep, the mean of each country poition i updated uing a modified Kalman filter. Thi tep differ from a tandard Kalman filter in that e may have no or multiple obervation on any given date. We alo add peudo-obervation for each country at each day t ith mean 0 and variance 10. Thee obervation are a form of time-erie regularization and reflect the ene that a lack of ne i effectively neutral ne. The prior over the end of the chain are tandard normal. E Step. In the E-tep, our goal i to infer nation poition x c,d x c,td, d ith each interaction d given their mean and the entiment d for thi interaction. We find thee poition by gradient acent on x c,d. 3 Etimation of entiment To infer the entiment d beteen to countrie, e treat the correponding ne article a a bagof-ord and ue text regreion [5]. In each of thee cae, the entiment d i firt etimated on (2) 2

3 Figure 2: A creenhot of a Mechanical Turk labeling tak. Sometime relationhip may be complicated; both rater gave thi example a core of lightly poitive. a training et uing Amazon Mechanical Turk (AMT). AMT i a crodourcing platform hich provide acce to thouand of orker ho perform imple tak over the Internet. Thee label are then attached to all article by predicting their entiment. 4 Experiment We fit and evaluate thi model over ne article dicuing 245 nation from tenty year of the Ne York Time (NYT). Thi collection panned the year 1987 to 2007, a period hich included both Gulf ar; the collape of the Soviet Union; the reunification of Germany; September 11th, 2001; and countle other orld event. 4.1 ataet and tokenization We made an important aumption that the cope of foreign entiment dicuion i at the level of a paragraph. We therefore ued the ubet of paragraph hich dicu exactly to nation a document d, a et of 257,472 ditinct paragraph hich came from the foreign, buine, financial, and magazine dek of the nepaper during thi period. We ued tandard topord filter and removed the geographic tag for labeling nation from the entiment vocabulary. 4.2 Mechanical Turk Label To fit the model, e aked Amazon Mechanical Turk rater to rate the entiment beteen to nation mentioned in a paragraph in the range -5 = mortal enemie,...,5 = very good relationhip. With thee training example, e fit the coefficient β of the text regreion dicued in Section 1. Thi coefficient a then fixed in the joint model in Figure 1 to allo u to learn entiment from paragraph ord. We trained and fit the model ith the folloing procedure: 1. Randomly elect 3607 paragraph dicuing pair of 245 countrie. 2. Label each of thee paragraph entiment ith to rating from Amazon Mechanical Turk. 3. Hold out 42 random country pair (244 paragraph) for teting. 4. Etimate entiment model parameter β uing the training paragraph. 5. Infer the patial entiment model ith thee parameter on all 257,472 paragraph. 6. Evaluate model entiment prediction on the heldout 244 tet paragraph. 4.3 Reult Heldout entiment For to nation c 1 and c 2 mentioned together at time t, e predict their entiment to be c1,c 2 = x c1,t x c2,t. Baed on thi etimate, the average quared error for heldout nation i 2.32 (R 2 = 0.51), coniderably better than a baeline of text regreion, hich had mean-quarederror 5.53; for comparion, average inter-rater quared error the minimum theoretically poible 3

4 Ip1 afghanitan Ip2 0.6 north_korea afghanitan outh_korea japan pakitan ruia/ur irael france itzerland a outh_korea a itzerland united_ta a united_tate india a afghanitan a a a france a india a a irael a japan a north_korea a pakitan a ruia/ur Ip irael japan indiafrance th_korea outh_korea pakitan itzerland afghanitan ruia/ur Ip1 a afghanitan a a a france a india a a irael a japan a north_korea united_ta a pakitan a ruia/ur a outh_korea a itzerland a united_tate Mutual entiment ith the United State irael ruia/ur itzerland ate (a) (b) (c) japan yugolavia afghanitan irael japan ruia/ur itzerland yugolavia Figure 3: (a) Example poition of elected countrie in the latent pace of national entiment, in Sentiment i given by the inner product beteen to vector. (b) Example poition of elected countrie in (c) Mutual entiment = x T c, x u, ith the United State over time. The to Iraq ar and September 11th, 2001 are marked. Table 1: Average error predicting entiment d beteen heldout nation pair. Model Mean Squared Error Mean Abolute Error Inter-rater agreement 1.77 (7.11) (2.07) Text regreion Prior variance Prior variance Prior variance Prior variance Prior variance a 1.77, and the quare of the difference beteen rater a We alo found that uing peudo obervation ith even modet obervation variance improve predictive performance; thee reult are ummarized in Table 1. Nation poition are illutrated in Figure 3. Figure 3(a,b) ho nation relative poition and interaction for a to- dimenional model; heldout error lightly increaed for higher dimenion. The United State relationhip ith Iraq (ee Figure 3(c)) erve a an excellent example of thi model in action; the relationhip beteen thee nation degrade during both the Gulf War and the invaion of Iraq folloing September 9th, Irael relationhip ith the United State demontrate one of the model donfall: hile Irael i conidered a cloe ally of the United State, the rater 95 rating of thee nation mention had mean 0.12 and tandard deviation Becaue the model i only a good a it rating, the nation appear to have a rocky relationhip. Future ork There are many ay in hich e hope to expand thi ork. Creating a joint model of text, entiment, and vote in bodie uch a the UN General aembly i one poible direction. Another improvement on thi model ould aign a entiment intercept term for each nation to Equation 2. Finally, e hope to explore more unupervied model, in hich the e characterize the relationhip beteen nation, uing an approach related to relational topic model [8]. Acknoledgment avid M. Blei i upported by ONR , NSF CAREER , AFOSR 09NL202, the Alfred P. Sloan foundation, and a grant from Google. Yahoo! provided many of the machine on hich thee experiment ere performed. 4

5 Reference [1] Martin, A.., K. M. Quinn. ynamic ideal point etimation via markov chain monte carlo for the u.. upreme court, Political Analyi, 10: , [2] Jackman, S. Multidimenional analyi of roll call data via bayeian imulation: Identification, etimation, inference, and model checking. Political Analyi, 9(3): , [3] Hoff, P., A. E. Raftery, M. S. Handcock. Latent pace approache to ocial netork analyi. Journal of the American Statitical Aociation, 97: , [4] Chang, J.,. M. Blei. Relational topic model for document netork. Proceeding of the 12th International Conference on Artificial Intelligence and Statitic (AIStat) 2009, 5, [5] Kogan, S.,. Levin, B. Routledge, J. Sagi, N. Smith. Predicting rik from financial report ith regreion. In ACL Human Language Technologie, Aociation for Computational Linguitic, [6] Gartzke, E. Kant e all jut get along? opportunity, illingne, and the origin of the democratic peace. American Journal of Political Science, 42(1):1 27, [7] Sarkar, P., A. W. Moore. ynamic ocial netork analyi uing latent pace model. ACM SIGK Exploration Neletter, 7(2):31 40, [8] Chang, J., J. Boyd-Graber, S. Gerrih, C. Wang,. Blei. Reading tea leave: Ho human interpret topic model. In Neural Information Proceing Sytem (NIPS),