Evaluation of Japanese Text Information Features Based on the Readability

Transcription

1 DEIM Forum 2018 C2-1 Evaluation of Japanese Text Information Features Based on the Readability WEB ,071 NMF NMF LSI LDA 1. [3] WEB 1 WEB [1] SIPS Sympathize Identify Participate Share & Spread [2] 2 SIPS D.K. SMCR [4] SMCR (Source) (Message) (Channel) (Receiver) C.S. (sign) (interpretant) (object) [5] (icon: ) (index: ) (symbol: ) [6] ( ) ( ) 1 Oracle Social Relationship Management Cloud : ( ) ( ) ( gs/index.html ) ( ) 2 Klout : 3 Twitter : 4 Instagram : 5 Youtube : 6 Apple : 7 Cisco - Support Community : 8 Intel - Support Communities : 9 Slack : 10 Yammer : : 1.html : 13 JNN :

2 [7] ( ) , SDGs Sustainable Development Goals (2017 ) 47% 15 Environment Social Governance ESG : : 16 : sokusin.html : 18 : [8] (2017 P.33) (2016 P.3) 2. 2 [9] 6 [10] 1 TF-IDF Bag-of-words Bag-of-words (2)(3)(5)(6) 1 [10] [11] 4 [12] [13] [14]

3 C.S. [15] [16] [17] ( ) , [18] [19] [20] 19 [21] [22] ( ) 3. 2 [23] [24] [24] [25] [26] [27] [28] LSI Latent Semantic Indexing [29] LDA Latent Dirichlet Allocation [30] LSI [31] LDA [32] 8 19 Flesch-Kincaid readability tests : readability tests [33] [34] [35] [36] [37] [38] [39] [40] [41] [42] [43] MVR [44] MeCab [45] 20 IPADIC MUC 21 IREX 22 [46] [47] [48] [49] [50] [51] [52] [53] [54] [55] [56] [57] [58] [59] SVM [60] NMF:Nonnegative Matrix Factorization [61] NMF Y H U 20 MeCab : 21 MUC-6 : 22 IREX : 23 :

4 NMF NMF ,071 NMF / 1 (1 [ ][ ][ ] ) 1 (1 [ ] ) 1 (1 [ ]) 1 (1 ) ( ) 8 ( ) ( ) 8 ( ) 1 ( [24]) 1 ( [24]) 2 [28] Latent Semantic Indexing(LSI) [29] Latent Dirichlet Allocation(LDA) [30] LSI N U T H N D V LSI (1) D V ( ) N U T H 2 F RO = Ndv u T 2 d h v (1) d=1 v=1 U = (u 1,, u D) K D H = (h 1,, h V ) K V K LSI LSI 24 Gensim [62] 25 [63] 26 LDA w d ϕ θ W d = (1,, D) k = (1,, K) w d (2) N d p(w d θ d, Φ) = Σ K k=1p(k θ d )p(w dn ϕ k ) (2) n=1 θ d Φ N d d w dn d n ϕ k k LDA LSI Gensim [62] θ d [64] LSI LDA K LSI K [65] K=300 LDA K= (LSI) [29] 300 u D (LDA) [30] 300 θ d LSI U D K LSI K LDA θ d LDA θ d / 1,449 Pandas 27 Numpy gensim : 26 Blei Lab : 27 Pandas : 28 NumPy :

5 3 / (1) (2) (3) (1) (2) ( ) (3) ( ) O B IPA ( [44]) MVR 1 - ( [44]) 4 - ( [45]) ( [48]) 1 - ( [48]) (NMF) NMF [61] NMF Y H U NMF Y ( 0) Y 0 1 L1 H U NMF Y (Principal Component Analysis) (Factor Analysis) Y Y 4. 2 Y ( i = 1,..., N ) ( j = 1,..., K ) N K K 2,071 NMF Y K M H ( m = 1,..., M ) NMF (3) ( yi,j ) NK ΣM m=1h j,mu m,i (3) (3) (y i,j) NK Y h i,m H u m,i U 2 NMF Y HU β (4) D β ( y x ) = y y β 1 x β 1 β 1 y β x β β (4) β (β 0) Itakura-Saito (β 1) Kullback-Leibler (β = 2) [61] Y Poisson Kullback-Leibler 4. 4 (4) NMF NMF M < min(k, N) Y Y H U (5) h j,1 u 1,i u 2,i. u m,i + h j,2 u 1,i u 2,i. u m,i h j,m u 1,i u 2,i. u m,i (5) (5) U u m,i i m Y ( i = 1,..., N ) ( j = 1,..., K ) N K 2 (5) N 2 Y U 2 U u m,i m 2 m NMF U 4. 5 H Y NMF (6) m j Y ( j = 1,..., K ) K 2,071 Y HU (H = h j,1,..., h j,m) (6)

6 (6) h j,m m j ( 4. 2) NMF ( 4. 3) U H ( 4. 4) H ( 4. 5) PDF PDF WEB Python 33 MeCab [45] LSI LDA Gensim [62] Y Pandas NumPy NMF scikit-learn UTF-8 NMF N=51689 Normalization Form KC (NFKC) U 4 NMF M M= m ( ) 3 ( ) 29 ISO :2017 : 30 RFC8118 : 31 PDF Acrobat DC 32 PDF TXT PDF Text : 33 Welcome to Python.org : 34 scikit-learn : 35 scikit-learn : 36 Unicode Technical Reports : ( ) ( ) (I- ) (I- ) ( - ) (B- ) (I- ) (1.133) (I- ) ( ) ( ) ( - ) 3 6. NMF U H , NMF β [66], [67] 7. JSPS JP16H WordNet:

7 [1] Gantz, John, and David Reinsel. The Digital Universe in 2020: Big Data, Bigger Digital Shadows, and Biggest Growth in the Far East. IDC, [2] SIPS : [3] [4] : [5] I [6] [7] (1) [8] [9]?; ( 9 ) [10] 2005 [11] Barzilay, R.; Elhadad, N., Inferring Strategies for Sentence Ordering in Multidocument News Summarization, Journal Of Artificial Intelligence Research, Volume 17, pages 35-55, [12] [13] : [14] (DDC)( ) [15] [16] (I) [17] : [18] [19] Kincaid, J.P., Fishburne, R.P., Rogers, R.L., and Chissom, B.S. (1975). Derivation of new readability formulas (automated readability index, fog count, and flesch reading ease formula) for Navy enlisted personnel. Research Branch Report Chief of Naval Technical Training: Naval Air Station Memphis. [20] [21] NL [22] [23] ( ) [24] [25] LINE [26] [27] [28] ( ) 2015 [29] Landauer, T. K. and Dumais, S. T., A Solution to Plato s Problem: The Latent Semantic Analysis Theory of Acquisition, Induction and Representation of Knowledge, Psychological Review, 104: 2, pp , [30] David M. Blei, Andrew Y. Ng, and Michael I. Jordan Latent dirichlet allocation. J. Mach. Learn. Res. 3, [31] SVM TOD [32]. NLC, [33] D5-1 [34] (1) [35] : [36] 1984 [37] [38]. D [39] SLP [40] ( 1) [41] Q&A [42] [43] : [44] : [45] Taku Kudo, Kaoru Yamamoto, Yuji Matsumoto, Applying Conditional Random Fields to Japanese Morphological Analysis, In Proceedings of the Conference on Empirical

8 Methods in Natural Language Processing (EMNLP 04) [46] 13 pp [47] Wikipedia 16 pp [48] [49] [50] [51] Hiroya Takamura, Takashi Inui, and Manabu Okumura Extracting semantic orientations of words using spin model. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics (ACL 05). Association for Computational Linguistics, Stroudsburg, PA, USA, [52] NL [53] Web [54] [55] Web [56] Stijn De Saeger Web [57] : [58] ( : ) [59] GA [60] Support Vector Machine [61] [62] Rehruvrek, R. and Sojka, P. (2010). Software Framework for Topic Modelling with Large Corpora. Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks (p./pp ), May, Valletta, Malta: ELRA. [63] N. Halko, P. G. Martinsson, and J. A. Tropp Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions. SIAM Rev. 53, 2 (May 2011), [64] Matthew D. Hoffman, David M. Blei, and Francis Bach Online learning for Latent Dirichlet Allocation. In Proceedings of the 23rd International Conference on Neural Information Processing Systems - Volume 1 (NIPS 10), J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta (Eds.), Vol. 1. Curran Associates Inc., USA, [65] Roger B. Bradford An empirical study of required dimensionality for large-scale latent semantic indexing applications. In Proceedings of the 17th ACM conference on Information and knowledge management (CIKM 08). ACM, New York, NY, USA, [66] (1) [67] 2008 A 1 NAIST-JENE Wikipedia / ( A 1 / Unicode 10.0 a b (1) b (2) b (3) b (1) b (2) b (3) b [38] c [40] [41] IPA ipadic version d ipadic version d NAIST-JENE e [49] c [50] c [51] f [52] g a Unicode 10.0 Character Code Charts b c d ipadic version e NAIST Japanese ENE Dictionary on Wikipedia f takamura/pndic ja.html g expressions.html