CUSTOMER REVIEW FEATURE EXTRACTION Heng Ren, Jingye Wang, and Tony Wu

Transcription

1 CUSTOMER REVIEW FEATURE EXTRACTION Heng Ren, Jingye Wang, an Tony Wu Abstract Popular proucts often have thousans of reviews that contain far too much information for customers to igest. Our goal for the project is to implement a system that extracts opinions from these reviews an summarizes them in a concise form. This allows customers to quickly get an overview of a prouct an manufactures to efficiently process prouct feebacks. In the past, we focuse on the feature extraction irectly on the wor level. First, we woul use association rule to extract all the <noun => aj.> rules an then use Pointwise Mutual Information to juge the polarization of the ajective. Processing irectly on wor level neglects the information containe on the sentence level. In this paper, we iscuss how to extract features by first investigate on sentence level an then ive into the wor level. Introuction Extracting features irectly on wor level neglects the information containe in the sentence level. For example, there might be a sentence such as Toay is a goo ay appearing in the reviews. If we only perform the algorithm on the wor level, we woul consier <ay, goo> as a feature of the prouct, but obviously it is not. Here we propose a metho that first etermines which sentences in a review might contain prouct features. If we are confient that a sentence oes have feature, we woul further process the sentence using the association rule an PMI to extract the features. To juge whether a sentence has any features, we use several Machine Learning methos. We will compare the performance of each metho. Algorithms an Implementation Naïve Bayesian Naïve Bayesian technique is a powerful metho for classification. In our problem, the two classes are Review Sentences with prouct feature (C 1 ) an Review Sentences without prouct feature (C 0 ). Pr C 1 = 1{C i = C 1 } training ze Pr(C 0 ) = 1 Pr(C 1 ) The features we use for training are iniviual wors within each sentence. We enote each sentence as S i an each wor as W j. The probability of the wor S i appearing in a sentence of class C k is given by: P W j C k = Wj 1 C i = C k #{W j in S i } 1 C i = C k #{W j in S i } nce some wor might never appear in sentences of a particular class, we o not want to have a sentence with 0 probability of being in a particular class. We eploy a slightly moifie version of Laplace Smoothing where λ = 0.1 instea of 1. P W j C k = Wj 1 C i = C k #{W j in S i } C i = C k #{W j in S i } W Finally, we have the following Naïve Bayesian classifier to etermine the probability of a sentence being in a particular class, using the wors in that sentence as features: P C 1 S i = P(C 1 ) P(W j in S i C 1 ) Ck P(C k ) P(W j in S i C k )

2 P(C 0 S i ) = 1 P(C 1 S i ) To avoi unerflow, we can alternatively use log for calculation: log P C 1 S i log(p C 0 S i ) = logp C 1 logp(c 0 ) + logp(w j in S i C 1 ) logp(w j in S i C 0 ) If the value obtaine above is greater than 0, we claim that the sentence is more likely to have prouct feature(s) than not. The wors that appear more often in feature sentences are inicative of a sentence likely to have features. The following list contains the top feature inicator wors from our stuies: fits, convenient, strong, clips, packe, variable, s, exceptional, freezes, glitches Conversely, the following list of wors inicates that the sentence is likely to not have any prouct features mentione: orere, olympus, sell, consumers, ran, shopping, compete, register, replacing, assume Overall, we achieve an accuracy of 78% by training on 8600 labele sentences an testing on 600 sentences. fake classifier pretening that the entire unlabele set is negatively labele. The spies will enable us to evelop a threshol for being negative. Any sentence that is more unlikely to be positive than all the spies are consiere to be Reliably Negative (RN). Spy Step in S-EM: 1. RN = NULL; 2. S = Sample(P, 15%); 3. Us = U S; 4. Ps = P - S; 5. Assign each sentence in Ps class label 1; 6. Assign each sentence in Us class label -1; 7. NB(Us, Ps); 8. Classify each sentence in Us using the NB classifier; 9. Determine a probability threshol th using S; 10. for each sentence Us 11. if its probability Pr(1 ) < th then 12. RN = RN {}; After obtaining the RN set, we will run an EM algorithm, which is essentially an iterative version of the Naïve Bayesian algorithm. The only ifference here is that instea of having class labels be {0, 1}, we use a continuous range [0, 1] for the probability of each sentence being in a particular class. The inicator functions in Naïve Bayesian are now probabilities instea. Spy EM Sometimes, labele ata may be har to come by. This is where spies become useful. Given a small set of positively labele ata an a large set of unlabele ata, we esignate a subset of the positive ata as spies an unleash them into the unlabele set. Then we use the same Naïve Bayesian algorithm that we just iscusse to train a EM Step in S-EM: 1. Assign each sentence in P the class label 1; 2. Assign each sentence in RN class label -1; 3. Each sentence in Q (= U - RN) is assigne a probabilistic label, Pr(1 ). E Step: Revise P(class sentence) M Step: Calculate the new P(class) an P(wor class) 4. Run the EM algorithm until it converges.

3 The following plot shows test set accuracy as a function of EM iterations. The EM algorithm seems to converge after approximately 100 steps. Rocchio Metho coul be escribe as follows. 1. Let c + = α 1 P 2. Let c = α 1 N P β 1 N N β 1 P 3. if <c +, > <c, > classify as negative else classify as positive N ; P ; Overall, S-EM achieves an accuracy of 67%. This is lower than Naïve Bayesian because we are pretening that the negatively labele set is unlabele. In a more realistic situation where we o have a large amount of unlabele ata, only using Naïve Bayesian will not enable us to solve the problem. Rocchio Metho Rocchio Metho trains the classifier by the following way. First, each ocument is represente as a vector = (q 1, q 2, q 3,, q n ). Each element q i represents a wor w i in the ocument an is calculate as q i = tf i if i where tf i is the term frequency, the number of times that wor w i occurs in ; an if i is the inverse ocument frequency, if i = log( D f(w i ) ) Here D is the total number of ocuments an f(w i ) is the number of ocuments where wor w i occurs at least once. Here, accoring to [Buckley et al., 1994], we use α = 16 an β = 4. Moreover, we use the inner prouct to measure the similarity of two vectors. For each test ocument, if it is more similar to positive c +, we assign it as positive. SVM base on Rocchio nce in Rocchio, we only use the inner prouct to measure the similarity of two passages, the accuracy rate was not so goo. Hence now we switch to using SVM, base on the Rocchio formulation. Let the training sample set be { x 1, y 1, x 2, y 2,, (x n, y n )}, where x i is the representation of a passage in Rocchio s format, an y i is the label for the passages. To eal with noisy labels, we use the soft margin SVM, which is min 1 2 ωt ω + C s. t. y i ω T x i + b 1 ξ i, i = 1, 2, n Here C is the parameter that controls the amount of training errors allowe. We employe ifferent types of kernels in SVM metho linear kernel, polynomial kernel an raial basis function kernel. Base on our training set, we compare the performance of each type of kernel as follows: i ξ i

4 Accuracy Kernel Precision Recall F Linear polynom Gamma Here we use Precision, Recall, an F-score as the measure of performance. F score takes into account of both recall an precision, F = 2pr/(p+r). The numbers above are all base on the best performance for each metho after tuning the parameters. Biase-SVM Up until now we assume that all samples are correctly tagge. However, tagging the ata is tough an teious an if we cannot get enough tagge ata, the training algorithm woul face potential over-fitting problems. To solve this problem, Bing et al propose the LPU algorithm. It uses both the tagge ata an untagge ata. First, it assumes that all untagge ata are negative. Iteratively, the algorithm woul train a SVM to classify the positive an negative samples until the ratio of positive an negative samples converges. The problem is that, since SVM is sensitive to noises, if in any iteration the traine SVM is largely affecte by noises, later ones woul be worse. We can vary C + an C separately. Intuitively, we give a large value for C + an a small value for C because the untagge set, which is assume to be negative, also contains positive ata. We use a separate valiation set to verify the performance of the resulting classifier with the selecte values for C+ an C-. We use F score as the performance measure. Kernel Precision Recall F Linear polynom Gamma Evaluation Now that we have presente the five algorithms we use to etermine if a Review Sentence contains a prouct feature, we woul like to compare their performances: Performance Comparison Bing et. al. propose Biase -SVM in [Bing Liu, et.al 2003]. Biase-SVM formulation uses two parameters C + an C separately to weight positive errors an negative errors. Thus the problem changes to min 1 2 ωt ω + C + s. t. y i ω T x i + b 1 ξ i, i = 1, 2, n i ξ i Algorithm It seems that for this particular task, SVM with no untagge samples outperforms all other algorithms by achieving an accuracy rate of over 80%. When using spy samples, the performance of SVM ecreases largely. However, since we also use iterative algorithm in SVM, we can see that the SVM s performance is still better than S-EM.

5 ba terrible stupi great entertaining ifficult goo funny SO Combination To close the loop, we woul like to present some results outsie the scope of the project focus, just for completeness. After obtaining prouct features, we woul like to know if customers like the particular feature or not. This can be one easily with an algorithm calle PMI. Point-wise Mutual Information (PMI) algorithm uses mutual information as a measure of the strength of semantic association between two wors. The Point-wise Mutual Information between two wors, W 1 an W 2, is efine as follows: P(W 1, W 2 ) PMI W 1, W 2 = log( P W 1 P(W 2 ) ) Where P(W 1,W 2 ) is the probability that W 1 an W 2 both occur in the same phrase. Semantic Orientation of a phrase is calculate as follows: SO(phrase) = PMI(phrase,"excellent") PMI(phrase,"poor") We can use Semantic Orientation to Determine the polarity of a particular wor. If the wor appear more often with excellent, it is more likely to have a positive connotation. The following plot shows some examples: Opinion Polarity Finally, once we have obtaine the prouct features as well as their relative polarity from the customer reviews, we can sort them an present them in a summary form as emonstrate below: PRODUCT FILE NAME : D-Link DWL-G120 There are 40 review(s) total. PROS : setup (5.0) connection (4.0) price (4.0) car (2.0) connections (2.0) CONS: performance (-5.0) port (-2.0) sensitivity (-2.0) Conclusion In this paper, we showe a two-step metho for extracting features from customer reviews. As the first step, we woul juge whether each sentence has a feature or not; if it oes, we woul further process the sentence an extract the features. The first step is mainly base on Machine Learning methos an the secon step is mainly base on Data Mining an Natural Language Processing Methos. We have emonstrate in this project that it is tractable to etermine customer opinions on iniviual prouct features. When all training ata are correctly tagge, we showe that SVM is the best an when not all training ata is tagge, S-EM an B-SVM almost have the same performance Opinions