IPSJ SIG Technical Report Vol.2014-MPS-100 No /9/25 1,a) 1 1 SNS / / / / / / Time Series Topic Model Considering Dependence to Multiple Topics S

Transcription

1 1,a) 1 1 SNS /// / // Time Series Topic Model Considering Dependence to Multiple Topics Sasaki Kentaro 1,a) Yoshikawa Tomohiro 1 Furuhashi Takeshi 1 Abstract: This pater proposes a topic model that considers dependence to multiple topics. In time series documents such as news, blog articles, and SNS user posts, topics evolve with depending on one another, and they can die, be born, merge, or split at any time. The conventional models cannot model the evolution of all of the above aspects because they assume that each topic depends on only one previous topic. In this paper, we propose a new topic model which assumes that a topic can depend on previous multiple topics. This paper shows that the proposed topic model can capture the topic evolution of death, birth, merger, and split and can model time series documents more adequately than the conventional models. Keywords: Topic Model, Time Evolution, Time Series Document 1. Web SNS [1], [2], [4], [10], [14], [15] 1 a) sasaki@cmplx.cse.nagoya-u.ac.jp bag-of-words Latent Dirichlet Allocation (LDA) [5]LDA 1

2 k k [1], [2], [4], [15] Temporal Dirichlet Process Mixture (TDPM) Multi-dependent Temporal Dirichlet Process Mixture (MdTDPM) /// Dirichlet Process Mixture Model Dirichlet Process Mixture Model (DPM) DPM Dirichlet Process (DP) DP [6] G 0 γ G DP G DP(γ,G 0 ) γ G G 0 DP Stick-Breaking Process[9] Chinese Resutaurant Process (CRP)[3] [8] CRP CRP 1 i-1 x 1:i 1 z 1:i 1 i x i z i m k P (z i = k z 1:i 1,γ)= (1) i + γ 1 γ P (z i = k new z 1:i 1,γ)= (2) i + γ 1 m k k k new z 1:i 1 CRP m k m k /(i + γ 1) γ/(i + γ 1) CRP DPM M z = z 1:M (i) z CRP(γ) (ii) φ k G 0 G 0 (iii) for i =1,..., Mx i p(x φ zi ) 2.2 Temporal Dirichlet Process Mixture DPM Ahmed DPM Temporal Dirichlet Process Mixture (TDPM) [1]TDPM TDPM CRP Recurrent Chinese Restaurant Process (RCRP) RCRP CRP t i x t,i z t,i P (z t,i = k I t I t 1 z t 1,z t,1:i 1,γ) = m t 1,k + m t,k M t 1 + i + γ 1 P (z t,i = k I t I t 1 z t 1,z t,1:i 1,γ) γ = M t 1 + i + γ 1 (3) (4) m t,k t k M t t z t t k I t m t,k =0k I t 1 m t 1,k =0I t t RCRP x t,i k k m t,k 2

3 γ Φ t-1,1 Φ t,1 z 1,i z 2,i z T,i t,k,1 Φ t-1,k t, k, k Φ t,k w 1,i,j w 2,i,j w T,i,j t, k, K N 1,i N 2,i N 1,i M 1 M 2 M 1 Φ t-1,k Φ 1,k Φ 2,k Φ T,k 2 MdTDPM β 2,k β T,k 1 TDPM m t 1,k RCRP TDPM t (i) z t RCRP(γ,z t 1 ) (ii) for k I t, φ t,k φ t 1,k P (φ t,k φ t 1,k )ifk I t 1 or φ t,k G 0 G 0 (φ t,k )ifk I t 1 (iii) for i =1,..., N t x t,i p(x φ t,zi ) TDPM k I t 1 φ t,k φ t 1,k k I t k I t 1 t k k I t k I t 1 t k 3. TDPM MdTDPM β P (φ t,k It 1 ˆφ t 1,k,β t,k ) v φ β t,k ˆφ t 1,k,v 1 t,k,v (5) φ t,k,v φ t,k v β t,k φ t 1,k z t,i = k I t 1 β φ t,k Dirichlet(β) (6) TDPM 1 t (i) z t RCRP(γ,z t 1 ) (ii) (iii) for each topic k I t φ t,k Dirichlet(β t,k ˆφt 1,k )ifk I t 1 or φ t,k Dirichlet(β) ifk I t 1 for each document i =1,..., M t for each word j =1,..., N t,i w t,i,j Multinomial(φ zt,i ) 3.2 MdTDPM MdTDPM k I t 1 φ t,k {φ t 1,k } K k =1 3.1 TDPM TDPM t i d t,i N t,i ) w t,i = {w t,i,j } Nt,i j=1 z t,i φ t,zt,i w t,i,j [16] φ t,zt,i z t,i = k I t 1 β t,k ˆφ t 1,k φ t,k Dirichlet( β t,k,k ˆφt 1,k ) (7) k β t,k,k t k k 3.3 MdTDPM MdTDPM EM [13], [15], [16] β t,k,k ˆφ t,k EM 3

4 t=1 t=2 t= t=1 t=2 t= t=1 t=2 t= (a) cdpm (b) TDPM (c) MdTDPM 3 EM [8] [7] t i d t,i z t,i P (z t,i z t\i, z t 1, W t, ˆΦ t 1, β t ) (8) P (z t,i = k z t\i )P (w t,i z t,i = k, W t\i, ˆΦ t 1, β t ) K t 1 t 1 { } Kt 1 ˆΦ t 1 = ˆφt 1,k β t = {β t,k } Kt 1 k=1 W t = {w t,i } Mt i=1 k=1 \i i β t P (W t, z t...) 4. TDPM DPM Evolutionary Net-Cluster Generative Model (Evo-NetClus)[11] continuous Dirichlet Process Mixture (cdpm)[10] Evo-NetClus cdpm Evo-NetClus Twitter Twitter cdpm t k φ t,k φ t 1,k β V Vβ φ t,k φ t,k Dirichlet(Vβˆφ t 1,k ) (9) 3 cdpmtdpmmdtdpm 1 () DPM cdpm TDPM MdTDPM 2636(30.5) 2194(33.0) (51.6) (39.0) (29.6) (22.7) (42.4) (17.5) cdpm // TDPM MdTDPM // TDPM MdTDPM 5. YOMIURI ONLINE * ,373 * ,880 5 stop words 6, ,863 7, ,760 *1 *2 4

5 DPM cdpm TDPM DPM cdpm TDPM 1500 MdTDPM 1000 MdTDPM t t (a) (b) MdTDPM W test ( perplexity =exp log P (W ) test) N test (10) (10) N test DPMcDPMTDPM DPM TDPM 3.1 γ 1β 0.1 TDPM β t,k 100 MdTDPM β t,k,k k = k 100 β 1 9:1 90% 10% s s 1 MdTDPM DPM 5

6 cdpm TDPM MdTDPM cdpmtdpm 4 MdTDPM TDPM MdTDPM TDPM 5 MdTDPM *3 MdTDPM /// * Hierarchical Dirichlet Processes (HDP) [12] [1] Ahmed, A. and Xing, E. P.: Dynamic Non-Parametric Mixture Models and the Recurrent Chinese Restaurant Process: with Applications to Evolutionary Clustering, Proc. SDM 08 (2008). [2] Ahmed, A. and Xing, E. P.: Timeline: A dynamic hierarchical Dirichlet process model for recovering birth/death and evolution of topics in text stream, Proc. UAI 10 (2010). [3] Aldous, D.: Exchangeability and Related Topics, In École d été deprobabilités de Saint-Flour,XIII, Berlin, Springer, pp (1985). [4] Blei, D. M. and Lafferty, J. D.: Dynamic topic models, Proc. ICML 06, ACM, pp (2006). [5] Blei, D. M., Ng, A. Y. and Jordan, M. I.: Latent dirichlet allocation, the Journal of machine Learning research, Vol. 3, pp (2003). [6] Ferguson, T. S.: A Bayesian analysis of some nonparametric problems, The annals of statistics, Vol. 1, No. 2, pp (1973). [7] Minka, T.: Estimating a Dirichlet distribution, Technical report, MIT (2000). [8] Neal, R. M.: Markov chain sampling methods for Dirichlet process mixture models, Journal of computational and graphical statistics, Vol. 9, No. 2, pp (2000). [9] Sethuraman, J.: A constructive definition of Dirichlet priors, Statistica Sinica, Vol. 4, pp (1994). [10] Si, J., Mukherjee, A., Liu, B., Li, Q., Li, H. and Deng, X.: Exploiting Topic based Twitter Sentiment for Stock Prediction, Proc. ACL 13, pp (2013). [11] Sun, Y., Tang, J., Han, J., Gupta, M. and Zhao, B.: Community evolution detection in dynamic heterogeneous information networks, Proc. MLG 10, ACM, pp (2010). [12] Teh, Y. W., Jordan, M. I., Blei, M. J. and Blei, D. M.: Hierarchical dirichlet processes, Journal of the american statistical association, Vol. 101, No. 476 (2006). [13] Wallach, H. M.: Topic modeling: beyond bag-of-words, Proc. ICML 06, ACM, pp (2006). [14] Wei, X., Sun, J. and Wang, X.: Dynamic Mixture Models for Multiple Time-Series, Proc. IJCAI 07, pp (2007). [15] 2009 (2009). [16] Vol. 93, No. 6, pp (2010). 6