Paragraph Topic Classification

Size: px

Start display at page:

Download "Paragraph Topic Classification"

Domenic McCarthy
6 years ago
Views:

1 Paragraph Topic Cassification Eugene Nho Graduate Schoo of Business Stanford University Stanford, CA Edward Ng Department of Eectrica Engineering Stanford University Stanford, CA Introduction The project attempts to predict the topics of paragraph-ength texts. We posit that machine earning agorithms can extend a person s capabiity to consume information through the automated categorization of text and through an increased capabiity to ocate reevant information. We formay define the probem as constructing a mode that, given an input of text, outputs a vector of reevant topics. The project uses approximatey Wikipedia artice summaries categorized by Wikipedia users a priori. A number of approaches were taken incuding Naive Bayes, One-vs-Rest Cassifier (OvR with GoVe Vectors, Latent Dirichet Aocation (LDA/OvR, GoVe Vectors/LDA/OvR, Convoution Neura Networks (CNN, and Long Short Term Memory networks (LSTM. 2 Reated Work We found a number of papers associated with topic cassification, which we wi segment into traditiona, topic modeing, and finay the use of neura networks approaches to topic cassification. 2.1 Traditiona Approaches The use of Naive Bayes in topic cassification has been preeminent given its dua properties of being accurate and quick to train (Ting et. a (2011. However, there are variations of Naive Bayes that woud have ikey improved our performance, specificay an agorithm deveoped by Wang et. a (2012 which used Naive Bayes og-count ratios as features into an SVM mode (known as NB-SVM, and using bigrams rather than tf-idf. This combined mode (with a sma modification, known as NB-LM is considered to be one of the state-of-the-art supervised earning agorithms for topic cassification. Further research indicated the use of bigrams (and n-grams in genera does not revea any improved performance (Tan et. a, 2002, eaving the impementation of NB-LM as the primary focus on any future impementation using traditiona earning agorithms. 2.2 Topic Modeing Another approach to topic cassification is to presume that for a given dataset of corpora, there are n number of topics distributed, and that for each given document there is some probabiity that they beong to a given topic. We impement Latent Dirichet Aocation

2 2.3 Neura Networks The recent insurgent interest of neura networks have aso pushed the appication of neura networks to incude topic cassification. Convoution neura networks, initiay targeted for image cassification appications, have been modified to aso perform natura anguage processing with a high rate of success and considered to be state-of-the-art for topic cassification. We impement a rudimentary convoution neura network based on Kim et. a (2014 (with the sight modification of using GoVe vectors rather than word2vec which provided the highest rate of accuracy out of a our agorithms. However, there are a coupe of variations that may have improved our performance on this task, incuding the use of individua characters as inputs (Zhang et. a (2015 and training embeddings through the use parae convoution ayers with differing region sizes (Johnson et. a (2014. The increased compexity of neura network topoogy is often associated with the need for arger training data sets, with Zhang et. a (2015 expicity mentioning that a dataset of severa miions needs to be provided for character-eve convoutiona neura nets to outperform more traditiona approaches. In order for these other topoogies to function we, our dataset of Wikipedia artices woud have been required to be much arger. However, using parae convoution ayers may have been an area of future exporation. 3 Dataset We used Wikipedia artices summaries as our data. Each artice s first summary paragraph served as our input, and its topic assignments as our abes. We scraped Wikipedia for tota 400, 000 inputs, a beonging to one or more of the foowing five topics: mathematics, poitics, computer science, fim, and music. We imited our scope of topics to five to test the efficacy of our modes without requiring an unwiedy amount of data, and thus computationa resources. Our training and test sets were and entries respectivey, with an average word count of 93 for each summary. We used Wikipedia because it is one of the few avaiabe sources of topicay abeed, mutiabe dataset. We insisted on soving a mutiabe probem because, in rea-word appications, a piece of text is rarey about one topic ony. An exampe entry is: In mathematics, the minimum k-cut, is a combinatoria optimization probem that requires finding a set of edges whose remova woud partition the graph to k connected components. These edges are referred to as k-cut. The goa is to find the minimum-weight k-cut. This partitioning can have appications in VLSI design, data-mining, finite eements and communication in parae computing. 4 Features We used term frequency-inverse document frequency (tf-idf and GoVe as our main features. 4.1 Term Frequency-Inverse Document Frequency We used tf-idf because it captures how important a word is to a document in a corpus by cacuating the frequency of a word in a given document offset by its frequency in the overa corpus, de-emphasizing common words ike is and the. It is aso a common baseine feature for information retrieva tasks. The weighting scheme used for tf-idf is the defaut provided by skearn. 4.2 Goba Vectors for Word Representation (GoVe Later in the project, we used pre-trained GoVe to feed richer representation of input text into our modes. GoVe represents words as vectors, with the vector representing the co-occurrence of the word with other words over a goba set. The cost function which GoVe minimizes is J(θ = 1 W f(p ij(u T i v j og P ij 2 2 i,j=1 where P ij is the probabiity of word i occurring in the context of j, W represents the entire vocabuary, f provides is a given weighting function, and u and v are the vector representations of i and j respectivey. 2

3 Our hypothesis for using GoVe vectors was that they woud more effectivey capture the underying reationships between words characterizing the same topic because reated words are ikey represented by simiar vectors. We used pre-trained GoVe vectors from Stanford s NLP group 1 (trained on Wikipedia 2014 corpus, 100-dimensions. 5 Methods Our overa approach to text cassification was starting with a simpe baseine mode and graduay adding more sophisticated features or mode architecture to see if and how the resuts improve. Tabe 1 summarizes our approaches and rationae. Tabe 1: Summary of approaches and rationae Mode Approach [features] Naive Bayes [tf-idf] OvR [GoVe] LDA + OvR [tf] LDA + GoVE + OvR [tf] CNN [GoVe] Rationae Common baseine mode for text cassification One-vs-Rest supports mutiabe earning; richer feature (GoVe To capture atent topics more effectivey Compementary behavior (word embedding + topic distribution Fast; generaizes we; oca word order is not important 5.1 Baseine: Naive Bayes We used Naive Bayes for our baseine impementation. Naive Bayes is often a go-to strategy for baseine given its speed and ease of its training. We trained Naive Bayes from skearn for muticass topic cassification, where paragraphs that fit into mutipe topics were cassified to be under a singe compound topic. Since we imit ourseves to 5 topics, there are 2 5 = 32 possibe abes to choose from. Input feature was a tf-idf matrix. 5.2 GoVe + One-vs-Rest SVM cassifier To test our hypothesis that feeding richer representation of our input text woud improve the resuts, we used GoVe as our feature and fed it into a inear Support Vector Machine (SVM cassifier using One-vs-Rest strategy (OvR. We chose OvR because it enabes us to do mutiabe cassification by constructing k SVMs, where k is the number of casses. Given m training data (x (1, y (1,..., (x (m, y (m, where x (i R n and y (i R k indicates casses x (i beongs to, the th SVM soves the foowing probem: 1 min w, b,ξ 2 wt w + c m i=1 ξ (i s.t. w T x (i + b 1 ξ (i, if y (i = 1 w T x (i + b 1 + ξ (i, if y (i = 0 where ξ i 0 for a i and c is a penaty parameter. In our impementation, x (i was constructed by first vectorizing each word of the ith exampe into a R 100 vector using the pre-trained GoVe weights, and then coapsing the matrix into a singe vector using tf-idf as the weighting factor. For a new exampe x, OvR conducts the foowing for each : y = sgn( w T x + b 5.3 LDA + One-vs-Rest SVM cassifier LDA is a 3-eve hierarchica Bayesian mode often used for topic modeing in natura anguage processing. Fuy expaining the mode is outside the scope of this paper, but one can get an intuition by observing how a singe document is modeed: ( N p(w α, β = p(θ α p(z n θp(w n z n, β dθ n=1 z n where α and β are parameters, θ~dirichet(α is a random variabe representing topic mixture, w is the document, z n is a topic for n th word of the document, and w n is n t h word. Intuition is that the term inside parenthesis, representing each document, is modeed as p(z zn p(w dn z dn. In other words, each document is modeed as random mixtures over atent topics, where each topic is characterized by a distribution over words. Our hypothesis for choosing LDA was that modeing 1 3

4 documents this way woud hep us addresses head-on a core chaenge we were facing: How can we cassify documents based on underying themes, rather than presence or absence of certain key words?" Since LDA is an unsupervised earning mode, we used its output, document topic matrix (number of inputs number of topics, as our design matrix X that gets fed into the OvR cassifier. 5.4 Convoutiona Neura Network (CNN Convoutiona Neura Networks (CNN appy ayers of convoving fiters to features. Traditionay CNNs have been to computer vision, but recenty they are increasingy appied to natura anguage processing probems. Instead of appying fiters to image pixes, CNNs in NLP appy the fiters to the matrices representing paragraphs, typicay consisting of rows of word embeddings such as word2vec or GoVe. We wi not define CNNs in mathematica formaism in this paper, but the essence of CNNs can be seen at the convoution ayers. Given the document matrix x 1 : n R n k, where x i s (word embedding vectors ike GoVe for a n words are stacked together, convoution occurs when a fiter w R hk is appied to a window of h words to produce a new feature. A feature c i is generated from window of words x i : i + h 1 by c i = f(w x i:i+h 1 + b where b R is a bias term and f is a non-inear activation function. This fiter is appied to each possibe window of words in the paragraph, producing n h + 1 of c i s above. We chose CNN for two main reasons. First, we beieve CNNs do what LDA does capturing atent topics but in a much more powerfu way. By appying convoving fiters and abstracting away from the raw word embeddings, CNNs can capture the "shapes of topics" the same way CNNs appied to image recognition capture the shapes of objects. Second, CNNs are fast and reativey easy to train with imited GPU resources, which enabed us to experiment with various hyperparameter settings without extensive infrastructure setup. The mode architecture is a muti-abe variant of the CNN architecture proposed in Kim et. a (2014, using a static embedding ayer, and an extra fuy-connected ayer. In order to do muti-abe cassification we used a binary cross entropy oss function and sigmoid activation. Figure 1: Graph of CNN Mode Architecture 5.5 Long Short Term Memory (LSTM LSTM is a Recurrent Neura Network (RNN architecture that is we-suited to earn from sequentia experience when ong time ags of unknown size exist between important events. It accompishes this by having gates: computing the foowing i t = σ (W (i x t + U (i h t 1, f t = σ (W (f x t + U (f h t 1, o t = σ (W (o x t + U (o h t 1 c t = tanh (W (c x t + U (c h t 1, c t = f t c t 1 + i t c+ t, h t = o t tanh(c t Intuition is that at each time t, given input x t and hidden state from t 1, LSTM ce has the power to decide whether to input, forget, or output the data based on function σ, thereby enabing the overa network to earn when "important events" may be cose or far apart in a given sequence of data. We tried LSTM mainy as a benchmark against our CNN resuts. RNN architecture is widey viewed as the go-to strategy for natura anguage processing (NLP tasks because anguage is a sequentia type of data. Our mode architecture is defined by an embedding ayer, a singe ayer of 200 LSTM ces, and a feed-forward ayer using sigmoid activation. As with CNN, it is muti-abe variant using a binary cross entropy oss function. 4

6 Resuts and Discussion The overa resuts for our various approaches at paragraph topic cassification are summarized in Figure 2 beow. There are a few points worth noting about these resuts.

5 6 Resuts and Discussion The overa resuts for our various approaches at paragraph topic cassification are summarized in Figure 2 beow. There are a few points worth noting about these resuts. Figure 2: Metrics for a Approaches First, Naive Bayes was very robust despite its simpicity, and given its preeminence in topic cassification, this is not at a surprising. Second, the GoVe and One-vs-Rest mode performed very poory on accuracy. We beieve this has to do with the fact that we coapsed the GoVe document matrix into a vector to feed into the cassifier. We suspect they began to converge toward simiar, overapping pattern the more data we used. This coud be because GoVe synthesizes the reationship between words with vector distances, and with more data, different types of differences a get refected with the simiar distances. As a data point, GoVe + OvR achieved 40% accuracy when trained over ony sampes. Third, contrary to the initia hypothesis, LDA did not work we in topic prediction. We beieve the principa reason is the difficuty to contro the way an unsupervised agorithm earns. The output of LDA we used for cassification document topic matrix is LDA s best guess about the ikeihood of a given sampe beonging to each of the five custers it earned to differentiate between. A corpus with m documents coud be custered into five groups in 5 m different ways, and it is neary impossibe to exacty aign the way an unsupervised earning agorithm custers them with the way humans did a priori. Fourth, both neura networks far outperformed other approaches with 97% accuracy. It is noteworthy that CNN and LSTM both ended up with very simiar metrics, but CNN was much faster and robust. CNN s training time for sampes was on average 30 minutes on a aptop CPU, whereas LSTM took 3 hours for haf the number of sampes. Lasty, as the next section on hyperparameter tuning wi discuss, CNN was very robust, showing consistenty effective resuts across various hyperparameter settings. In contrast, LSTM s resuts varied widey depending on hyperparameters Hyperparameter Tuning Our CNN mode proved reativey stabe over various hyperparameter settings. For exampe, deeper networks performed ony marginay better, but at much higher train and test time cost. Going from 1 ayer to 3, we saw an increase of 0.5 percentage points for both reca and precision. Changing dropout ayers, number of fuy-connected ayers, number and size of fiters, and activation function between ReLU and tanh did not yied more than 5% difference in accuracy. On the other hand, LSTM was very sensitive to the number of epochs, going from 40% to 86% as we increased the epochs from 1 to 5. The resuts aso varied depending on whether we aowed embeddings to be continuousy updated ( 10% higher when aowed, as we as the number of LSTM ces per ayer (80% for 100 ces vs 86% 200 ces. In the end, however, LSTM performed marginay better than CNN. 7 Future Work We woud ike to further perform hyperparamter tuning of the modes impemented, perhaps with grid search. In addition, given additiona time, we woud have ideay expored the weights assigned to our CNN and LSTM mode to make sense of how a given corpus is weighted. We woud aso re-estabish our baseine using the NB-SVM mode proposed by Wang et. a (

6 References [1] Ting, S. L., Ip, W. H., Tsang, A. H. (2011. Is Naive Bayes a good cassifier for document cassification?. Internationa Journa of Software Engineering and Its Appications, 5(3, [2] Yoo, J. Y., Yang, D. (2015. Cassification Scheme of Unstructured Text Document using TF-IDF and Naive Bayes Cassifier. [3] Wang, S., Manning, C. D. (2012, Juy. Baseines and bigrams: Simpe, good sentiment and topic cassification. In Proceedings of the 50th Annua Meeting of the Association for Computationa Linguistics: Short Papers-Voume 2 (pp Association for Computationa Linguistics. [4] Bei, D. M., Ng, A. Y., Jordan, M. I. (2003. Latent dirichet aocation. Journa of machine Learning research, 3(Jan, [5] Ramage, D., Ha, D., Naapati, R., Manning, C. D. (2009, August. Labeed LDA: A supervised topic mode for credit attribution in muti-abeed corpora. In Proceedings of the 2009 Conference on Empirica Methods in Natura Language Processing: Voume 1-Voume 1 (pp Association for Computationa Linguistics. [6] Tan, C. M., Wang, Y. F., Lee, C. D. (2002. The use of bigrams to enhance text categorization. Information processing management, 38(4, [7] Kim, Y. (2014. Convoutiona neura networks for sentence cassification. arxiv preprint arxiv: [8] Zhang, X., Zhao, J., LeCun, Y. (2015. Character-eve convoutiona networks for text cassification. In Advances in Neura Information Processing Systems (pp [9] Johnson, R., Zhang, T. (2014. Effective use of word order for text categorization with convoutiona neura networks. arxiv preprint arxiv:

Convolutional Networks 2: Training, deep convolutional networks

Convolutional Networks 2: Training, deep convolutional networks Convoutiona Networks 2: Training, deep convoutiona networks Hakan Bien Machine Learning Practica MLP Lecture 8 30 October / 6 November 2018 MLP Lecture 8 / 30 October / 6 November 2018 Convoutiona Networks