Learning Naive Bayes Classifier from Noisy Data

Transcription

1 UCLA Compuer Science Deparmen Technical Repor CSD-TR No Learning Naive Bayes Classifier from Noisy Daa Yirong Yang, Yi Xia, Yun Chi, and Richard R Munz Universiy of California, Los Angeles, CA 90095, USA {yyr,xiayi,ychi,munz}@csuclaedu Absrac Classificaion is one of he major asks in knowledge discovery and daa mining Naive Bayes classifier, in spie of is simpliciy, has proven surprisingly effecive in many pracical applicaions In real daases, noise is ineviable, because of he imprecision of measuremen or privacy preserving mechanisms In his paper, we develop a new approach, LinEar-Equaion-based noise-aware bayes classifier (LEE- WAY ), for learning he underlying naive Bayes classifier from noisy observaions Using linear sysem of equaions and opimizaion mehods, LEEWAY reconsrucs he underlying probabiliy disribuions of he noise-free daase based on he given noisy observaions By incorporaing he noise model ino he learning process, we improve he classificaion accuracy Furhermore, as an esimae of he underlying naive Bayes classifier for he noise-free daase, he reconsruced model can be easily combined wih new observaions ha are corruped a differen noise levels o obain a good predicive accuracy Several experimens are presened o evaluae he performance of LEEWAY The experimenal resuls show ha LEEWAY is an effecive echnique o handle noisy daa and i provides higher classificaion accuracy han oher radiional approaches keywords: naive Bayes classifier, noisy daa, classificaion, Bayesian nework 1 Inroducion Classificaion is one of he major asks in knowledge discovery and daa mining Naive Bayes classifier, in spie of is simpliciy, has proven surprisingly effecive in many pracical applicaions, including naural language processing, paern classificaion, medical diagnosis and informaion rerieval [12] The inpu daase for naive Bayes classifier is a se of srucured uples comprised of <feaure vecor, class value> pairs The fundamenal assumpion of naive Bayes classifier is ha he feaure variables are condiionally independen given he class value This classifier learns from he raining daase he condiional probabiliy disribuion of each feaure variable X i given he class value c Given a new insance < x 1, x 2,, x n > of he feaure vecor < X 1, X 2,, X n >, he goal of he classificaion hen is o predic is class value c wih he highes poserior probabiliy P (C = c x 1, x 2,, x n ) The classificaion accuracy depends no only on he learning algorihm, bu also on he qualiy of he inpu daase In a real daase, noise is ineviable,

2 2 because of he imprecision of measuremen or privacy preserving mechanisms [1] In his paper, we develop a new approach, LinEar-Equaion-based noise-aware bayes classifier (LEEWAY ), for learning he underlying naive Bayes classifier from noisy observaions Using linear sysem of equaions and opimizaion mehods, LEEWAY reconsrucs he probabiliy disribuions as an esimae of he underlying real naive Bayes classifier By incorporaing he noise model ino he learning process, we improve he classificaion accuracy Furhermore, wih new observaions ha are corruped a differen noise levels, we obain an exended naive Bayes srucure ha combines he reconsruced probabiliy disribuions wih he noise informaion From his exended naive Bayes srucure, we obain a beer classifier of he new observaions Since he noise model can be added eiher on he class variable or on feaure variables or boh, we will sudy each of he scenarios respecively in secion 3 11 Relaed Work Noise idenificaion and daa cleaning have been sudied in several differen daa mining asks, including classificaion ([10], [5], [17]), paern discovery ([8], [20]), privacy preserving ([1]), speech recogniion ([19]), and Bayesian nework learning ([16], [2]) Differen noise models are inroduced for differen purposes In privacy preserving daa mining, [1] sudied he disorion mehod, in which daa are randomly perurbed, and developed a way o reconsruc probabiliy disribuion from disored daa [2] defined he noise model as he condiional probabiliy disribuion of he observed feaure variable given he noise-free feaure variable, and analyzed he effecs of noise on learning Bayesian neworks [20] inroduced compaibiliy marix as a way o provide a probabilisic connecion from he observed daa o he underlying rue daa, and developed mehods o discover long sequenial paerns in a noisy environmen The noion of an uncerain daabase was firs inroduced in [13] for associaion rule mining In an uncerain daase, insead of giving explici values, each uple conains a ag value for each feaure variable ha gives he probabiliy of he feaure values appearing in he underlying noise-free daase given his observaion There are several ways o represen and compensae for noise in he observed daase One approach is o develop robus algorihms ha allow for noise by avoiding over-fiing he model o he daa ([15], [6]) Anoher approach is o pre-processing he inpu daa before learning [5] applied a se of learning algorihms o creae classifiers as filers o idenify and eliminae mislabelled raining insances [10] examined and exended C45 decision ree algorihm by pruning he ouliers There are wo weaknesses in eliminaing corruped uples Firs, by eliminaing he whole uple, i also eliminaes poenially useful informaion such as he uncorruped feaure values Secondly, when here is a large amoun of noise in he daase, he amoun of informaion in he remaining clean daase may no be sufficien for building he classifier Therefore, new pre-processing echniques have been developed o correc he corruped daa [11] presened an approach for idenifying corruped fields and using he remaining non-corruped

3 fields for subsequen modelling and analysis [16] used Bayesian mehods o clean corruped daa ha have dependencies among feaures However, his echnique requires an exper o provide a model in he form of a Bayesian nework for he basic srucure of he daa, and a small amoun of cleaned daa [17] presened a mehod for correcing misclassified daa o improve classificaion accuracy based on he oher prediced feaure values and he remaining feaure values However, in a noisy daase where each value is corruped, o correc he corruped daa is neiher easy nor effecive Thus, furher echniques are developed Insead of correcing each value, [1] reconsruced he probabiliy disribuions of he underlying noise-free daase wih coninuous feaure variables 3 12 Our Conribuions The main conribuions of his paper are: (1) We focus on learning naive Bayes classifier from noisy daase, where each value is corruped and noise is added on eiher feaure values or class values or boh For such noisy daase, neiher eliminaing nor correcion echnique will be effecive or easy for pre-processing (2) We inroduce exended naive Bayes srucures, which combine he naive Bayes classifier for he noise-free daase and he noisy observaions (3) Insead of pre-processing he noisy observaions, we incorporae he noise model ino he learning process and develop a new approach, LinEar-Equaion-based noiseaware bayes classifier (LEEWAY ), o reconsruc he condiional probabiliy disribuion of he feaure variables given he class value for he underlying noisefree daase LEEWAY is a general mehod in he sense ha i is suiable for any noise model (4) We have performed several experimens o evaluae he performance of LEEWAY on differen noisy daases The experimenal resuls show ha LEEWAY is an effecive echnique o handle noisy daa and i provides higher classificaion accuracy han oher radiional approaches The res of he paper is organized as follows In secion 2, we give a brief descripion of he problem and inroduce exended naive Bayes srucures In secion 3, we develop LEEWAY approach for learning he naive Bayes classifier from noisy raining daa in hree scenarios according o wheher noise is added on feaure variables or class variable or boh In secion 4, we presen experimenal resuls Finally, secion 5 gives conclusions and fuure research work direcions 2 Problem Descripion Le D be he noise-free daase ha conains feaure vecor < X 1, X 2,, X n > and he class variable C Le X i be he domain of feaure variable X i (i = 1,, n) and S be he domain of C Then each uple in D has he srucure < x 1,j1, x 2,j2,, x n,jn, c i >, where x i,ji ( X i ) is a value of X i and c i ( S) is a class value Le X i be he size of domain X i and S be he size of domain S Afer adding noise on D, he observed noisy daase D conains a se of uples in he form of < x 1,j 1, x 2,j 2,, x n,j n, c i >, where x i,j i ( X i ) is a value of he observed feaure variable X i and c i ( S) is a value of observed class variable

4 4 C We assume ha a noisy variable akes is values from he same domain as he corresponding noise-free variable Given D, our goal is o consiue he naive Bayes classifier for D, and apply he learned classifier o new observaions for classificaion Based on he fundamenal assumpion of naive Bayes classifier ha he feaure variables are independen given he class value, P (X 1, X 2,, X n C = c) = n P (X i C = c) (1) he condiional probabiliies P (X i C = c) for each feaure variable X i can be esimaed separaely and independenly Therefore, in he following analysis, we will focus on one feaure variable and all he analysis is applicable o oher feaure variables Le X be a feaure variable ranging over he feaure vecor in D and X be he domain of feaure variable X In his paper, we consider only discree feaure variables and discree class variable In a noisy environmen, he underlying naive Bayes classifier srucure can be exended o include he observed noisy variables We assume ha noise is added on each value of each uple independenly, and each uple in he noisy daase is observed independenly Under his assumpion, he exended srucures for hree scenarios, according o wheher noise is added o feaure variables or he class variable or boh, are shown in Fig 1 As we can see from hese exended srucures, he observed feaure variables preserve he condiional independencies given he class value i=1 C X1 C Xn add noise on feaure variables X1 Xn X1 Xn X1 C Xn add noise on class variable C X1 Xn C (a) (b) C X1 C Xn add noise on boh feaure variables and class variable X1 Xn X1 Xn C (c) Fig 1 Exended naive Bayes classifier srucures: (a) case of noisy feaure variables wih noise-free class variable; (b) case of noise-free feaure variables wih noisy class variable; (c) case of noisy feaure variables and noisy class variable

5 5 3 Learning Naive Bayes Classifier from Noisy Daa In he following subsecions, we sudy each of he hree scenarios presened in Fig 1 and describe our LEEWAY mehod for learning naive Bayes classifiers from noisy daa 31 Noisy Feaure Variables wih Noise-free Class Variable In his scenario, noise is added on each feaure value in each uple independenly, while he observed class value is noise free A good example would be a daase perurbed for privacy preserving purpose As shown in Figure 1 (a), he observed feaure variables preserve he condiional independencies given he class value In his case, he major ask of learning he naive Bayes classifier is o esimae he condiional probabiliy P (X C) based on he noisy observaion < X, C > in D According o probabiliy heory, we have P (X C) = P (X, X = x i C) x i X = P (X X = x i, C) P (X = x i C) x i X = P (X X = x i ) P (X = x i C) (2) x i X The las equaion holds due o he assumpion ha X is independen of C given X = x i Specifically, P (X = x j C) = P (X = x j X = x i ) P (X = x i C), x j X (3) x i X For each fixed class value, here are X equaions, one for each P (X = x j C) In marix form, hese equaions are: P (X =x 1 X=x 1 ) P (X =x 1 X=x 2 ) P (X =x 1 X=x X ) P (X =x 2 X=x 1 ) P (X =x 2 X=x 2 ) P (X =x 2 X=x X ) P (X =x X X=x 1 ) P (X =x X X=x 2 ) P (X =x X X=x X ) P (X=x 1 C) P (X=x 2 C) P (X=x X C) = P (X =x 1 C) P (X =x 2 C) P (X =x X C) (4) Le P (X = x j X = x i ) = p ji, x i, x j X Then he marix (p ji ) X X gives a clear represenaion of he likelihood of value disorion Then Eq 4 can be rewrien as p 11 p 12 p 1, X p 21 p 22 p 2, X p X,1 p X,2 p X, X P (X = x 1 C) P (X = x 2 C) P (X = x X C) = P (X = x 1 C) P (X = x 2 C) P (X = x X C) (5)

6 6 Since P (X = x j C) (x j X ) can be esimaed from he observed daase and p ji can be obained from he noise model, he soluion o he above se of linear equaions will give he values of P (X = x i C), (x i X ) If 0 P (X = x i C) 1, hen his se of values is a feasible soluion As an illusraion, we assume a simple noise model on he feaure variable X as follows 1 : { 1, if i = j P (X = x j X = x i ) =, if i j for all x i, x j X (6) Tha is, for noise level, he observed value of a feaure variable is he same as is real value wih probabiliy 1 and oher values wih equal probabiliy Then, Eq 4 can be rewrien as: (1 ) (1 ) (1 ) P (X = x 1 C) P (X = x 2 C) P (X = x X C) = P (X = x 1 C) P (X = x 2 C) P (X = x X C) (7) Afer a series of linear ransformaion, we have (1 ) (1 ) 0 Therefore, 0 0 (1 ) for all x i X P (X = x i C) = P (X = x i C) (1 ) = P (X=x 1 C) P (X=x 2 C) P (X=x X C) = P (X =x 1 C) P (X =x 2 C) P (X =x X C) (8) 1 1 X P (X = x i C) 1 X (9) Firs, i is obvious ha Eq 9 saisfies he normalizaion condiion ha x P (X = x i X i C) = 1 1 The following analysis can be applied o oher noise models

7 Secondly, in order o be a feasible soluion, he values in Eq 9 mus saisfy he condiion ha 0 P (X = x i C) 1, for any x i X Tha is, P (X = x i C) (a) P (X = x i C) 1 (b) 0 < 1 1 X (c) or P (X = x i C) (d) P (X = x i C) 1 (e) (10) 1 1 X < 1 (f) For condiion (c) and (f): If = 1 1 X, hen P (X = x j X = x i ) = 1 X, for any x i, x j X I means, given he real value of X, is observaion X akes any value in X wih he same probabiliy 1 X Thus, X is independen of X, and he marix of coefficiens of Eq 7 is singular Since (1 1 X 7 ) increases as X increases, < 1 1 X holds as long as < 05 (05 is he hreshold when X = 2) To have noise level < 05 is also reasonable in pracice Acually, when < 05, he marix of coefficiens of Eq 7 is a sricly diagonally dominan marix, and Eq 7 has a unique soluion, no necessary a feasible one hough When condiion (f) is saisfied for higher noise level, similarly, Eq 7 has a unique soluion, no necessary a feasible one hough Condiion (a) is saisfied if Condiion (d) is saisfied if Condiion (b) is saisfied if Condiion (e) is saisfied if ( X 1) min xi X P (X = x i C) (11) ( X 1) max xi X P (X = x i C) (12) 1 max xi X P (X = x i C) (13) 1 min xi X P (X = x i C) (14) From he above analysis, we can see ha he unique soluion given in Eq 9 is a feasible soluion if saisfies he following bounds: 0 < min {( X 1) min xi X P (X = x i C), 1 max xi X P (X = x i C)} or max {( X 1) max xi X P (X = x i C), 1 min xi X P (X = x i C)} < 1 (15) However, in pracical applicaions, only he frequencies of (X = x i C) can be couned from he sampled daase as an esimae of he probabiliies P (X = x i C) Thus, he equaions in Eq 4 do no always hold exacly In his case and

8 8 he siuaion when noise level is beyond he bound in Eq 15, we can ge a feasible approximaion by minimizing he deviaion beween boh sides of Eq 4 We use square-error o measure he disance, and ake he consrains defined by probabiliy heory as he consrains of he objecive funcion Puing in a mahemaical way, his becomes an opimizaion problem described as follows: min P (X=xi C) x i X x j X ( xi X P (X = x j X = x i ) P (X = x i C) P (X = x j C)) 2 subj o 0 P (X = x i C) 1, for any x i X x i X P (X = x i C) = 1 Given a new observaion < X 1 = x 1, X 2 = x 2,, X n = x n >, he classificaion ask is o decide which class his new insance belongs o If he new observaion has noise differen from he raining daase, which is also common in pracice such as a series of real-ime noisy observaions, simply applying he naive Bayes classifier of he noisy raining daase can no fi in all he noise levels and will decrease he predicive accuracy However, if we consider he feaure values in he new observaion as insances of some noisy variables, we can combine he naive Bayes classifier srucure of he noise-free daase wih he newly-observed variables The exended srucure wih parameers is shown in Fig 2 P(C) P(X1 C) C P(Xn C) X1 Xn P(X1 X1) P(Xn Xn) X1 Xn Fig 2 Naive Bayes classifier srucure combined wih new noisy observaions The class value wih he highes poserior probabiliy P (C = c X 1 = x 1, X 2 = x 2,, X n = x n ) gives he classificaion of he new observaion Based on he exended srucure shown in Fig 2, he maximal poserior probabiliy P (C = c X 1 = x 1, X 2 = x 2,, X n = x n ) can be esimaed as follows: max ci SP (C = c i X 1 = x 1, X 2 = x 2,, X n = x n ) = max ci S P (X 1 =x 1,X 2 =x 2,,X n =x n C=c i ) P (C=c i ) P (X 1 =x1,x 2 =x2,,x n =xn ) = max ci SP (X 1 = x 1, X 2 = x 2,, X n = x n C = c i ) P (C = c i ) n = max ci S j=1 P (X j = x j C = c i ) P (C = c i ) ( n X ) = max ci S j=1 k=1 P (X j = x j X j = x k )P (X j = x k C = c i ) P (C = c i ) (16) Therefore, from he learned naive Bayes classifier for he noise-free daase, we can ge a good esimae of he poserior probabiliy P (C = c X 1 = x 1, X 2 = x 2,, X n = x n ), and hus give a good predicion

9 32 Noise-free Feaure Variables wih Noisy Class Variable In his scenario, noise is added on he class variable, while he feaure variables are noise free A good example is he daase generaed from one or more machine learning algorihms The uples in such daase are also known as mislabelled raining insances [5] An empirical daase is he medical records which are classified by he paien s sympom Some cases are likely o be confused, because hey have similar sympoms The exended naive Bayes classifier srucure in his case is shown in Figure 1 (b) As we can see from he figure, he observed class variable C depends direcly on is real value c i and he noise model In his case, he major ask of learning he naive Bayes classifier is o esimae he condiional probabiliy P (X C) based on he noisy observaion < X, C > in D Similar o he analysis in secion 31, we have P (X C ) = P (X C = c j ) P (C = c j C ) (17) Specifically, c j S P (X C = c i ) = P (X C = c j ) P (C = c j C = c i ), c i S (18) c j S For each fixed value of a feaure variable, here are S equaions, one for each P (X C = c i ), i = 1, 2,, S In marix form, hese equaions are P (C=c 1 C =c 1 ) P (C=c 2 C =c 1 ) P (C=c S C =c 1 ) P (C=c 1 C =c 2 ) P (C=c 2 C =c 2 ) P (C=c S C =c 2 ) P (C=c 1 C =c S ) P (C=c 2 C =c S ) P (C=c S C =c S ) P (X C=c 1 ) P (X C=c 2 ) P (X C=c S ) = 9 P (X C =c 1 ) P (X C =c 2 ) P (X C =c S ) (19) Le P (C = c j C = c i ) = p ij, c i, c j S Then he marix (p ij ) S S, also known as he compaibiliy marix in [20], gives a clear represenaion of he likelihood of value subsiuions Then, Eq 19 can be rewrien as p 11 p 12 p 1, S p 21 p 22 p 2, S p S,1 p S,2 p S, S P (X C = c 1 ) P (X C = c 2 ) P (X C = c S ) = P (X C = c 1 ) P (X C = c 2 ) P (X C = c S ) (20) Since P (X C = c i ), (c i S) can be esimaed from he observed daase and p ij can be obained from he noise model, he soluion o he above se of linear equaions will give he values of P (X C = c j ), (c j S) If 0 P (X C = c j ) 1, hen his se of values is a feasible soluion As an illusraion, we assume a simple noise model on he class variable C as follows 2 : { 1, if i = j P (C = c j C = c i ) =, if i j for all c i, c j S (21) 2 The following analysis can be applied o oher noise models

10 10 According o his compaibiliy marix, he probabiliy of C aking he same value as C based on he observaions is 1, and oher values differen from C wih he same probabiliy Then, Eq 19 can be rewrien as: (1 ) (1 ) (1 ) P (X C = c 1 ) P (X C = c 2 ) P (X C = c S ) = P (X C = c 1 ) P (X C = c 2 ) P (X C = c S ) (22) Afer a series of linear ransformaion, we have (1 ) (1 ) (1 ) = P (X C =c 1 ) c i S P (X C =c i ) P (X C =c 2 ) c i S P (X C =c i ) P (X C =c S ) c i S P (X C =c i) P (X C=c 1 ) P (X C=c 2) P (X C=c S ) (23) Therefore, P (X C = c j ) = P (X C = c j ) c i S P (X C = c i ) (1 ) 1 = 1 S P (X C = c j ) 1 S P (X C = c i ) c i S (24) for all c j S Firs, i is obvious ha soluions in Eq 24 saisfy he normalizaion condiion ha x X P (X = x C = c j) = 1, for any c j S Secondly, in order o be a feasible soluion, he values in Eq 24 mus saisfy he condiion ha 0 P (X C = c j ) 1, for any c j S Tha is, P (X C = c i ) c k S P (X C = c k ) (a) P (X C c = c i ) 1 + k S P (X C =c k ) S (b) 0 < 1 1 S (c) or P (X C = c i ) c k S P (X C = c k ) (d) c k S P (X C =c k ) S P (X C = c i ) 1 + (e) (25) 1 1 S < 1 (f)

11 For condiion (c) and (f): If = 1 1 S, hen P (C = c j C = c i ) = 1 S, for any c i, c j S I means, given he observed value of C, is noise-free variable C akes any value in S wih he same probabiliy 1 S Thus, C is independen of C, and he marix of coefficiens of Eq 22 is singular Since (1 1 1 S ) increases as S increases, < 1 S holds as long as < 05 (05 is he hreshold when S = 2) To have noise level < 05 is also reasonable in pracice Acually, when < 05, he marix of coefficiens of Eq 22 is a sricly diagonally dominan marix, and Eq 22 has a unique soluion, no necessary a feasible one hough When condiion (f) is saisfied for higher noise level, similarly, Eq 22 has a unique soluion, no necessary a feasible one hough Condiion (a) is saisfied if 11 ( S 1) min c i SP (X C = c i ) c k S P (X C = c k ) (26) Similarly, condiion (d) is saisfied if ( S 1) max c i SP (X C = c i ) c k S P (X C = c k ) (27) For condiion (b), since and c k S P (X C = c k ) S > 1 + S 1 c k S c k S P (X C = c k ) S 0 S (28) c 1 + k S P (X C = c k ) S S 1 = S 1 P (X C = c k ) + 1 S S 1 > S 1 c k S P (X C = c k ) (29) hus, condiion (b) is saisfied if ( ) ( S 1) 1 max ci SP (X C = c i ) S c k S P (X C = c k ) Similarly, condiion (e) is saisfied if ( ) ( S 1) 1 min ci SP (X C = c i ) S c k S P (X C = c k ) (30) (31)

12 12 Because S min ci SP (X C = c i ) P (X C = c k ) S max ci SP (X C = c i ) c k S (32) from he above analysis, we can see ha he unique soluion given in Eq 24 is a feasible soluion if saisfies he following bounds: ( ) 0 < min or max () min ci S P (X C =c i ) c k S P (X C =c k ) () max ci S P (X C =c i ) c k S P (X C =c k ), () 1 max ci S P (X C =c i ) S c k S P (X C =c k ) (, () 1 min ci S P (X C =c i ) S c k S P (X C =c k ) ) < 1 (33) However, in pracical applicaions, only he frequencies of (X C = c i ) can be couned from he sampled daase as an esimae of he probabiliies P (X C = c i ) Thus, he equaions in Eq 19 do no always hold exacly In his case and he siuaion when noise level is beyond he bound in Eq 33, we can ge a feasible approximaion wih he opimizaion mehod in he similar way as in secion 31: ( 2 cj S P (X=x k C=c j) P (C=c j C =c i) P (X=x k C =c i)) min P (X=xk C=c j ) c j S,x k X c i S x k X subj o 0 P (X C=c j ) 1, for any c j S x k X P (X=x k C=c j )=1, for any c j S 33 Noisy Feaure Variables wih Noisy Class Variable In his scenario, noise is added on each feaure value and class value in each uple simulaneously and independenly As shown in Figure 1 (c), he observed feaure variables preserve he condiional independencies given he class value Besides, he observed class variable C depends direcly on is real value c i and he noise model In his case, he major ask of learning he naive Bayes classifier is o esimae he condiional probabiliy P (X C) based on he noisy observaion < X, C > in D This can be done in wo seps Firs, we correc he noise in he feaure values Given C = c k (c k S), we have P (X = x j C = c k ) = P (X = x j X = x i, C = x k ) P (X = x i C = c k ) x i X = P (X = x j X = x i ) P (X = x i C = c k ) (34) x i X If P (X X) is known, we can ge an esimae of he probabiliy disribuion P (X C = c k ) for each c k S wih he same mehod in secion 31

13 Secondly, we correc he noise in he class values P (X = x i C = c k ) = P (X = x i C = c j, C = x k ) P (C = c j C = c k ) c j S 13 = P (X = x i C = c j ) P (C = c j C = c k ) (35) c j S If P (C C ) is known, we can ge an esimae of he probabiliy disribuion P (X C) wih he same mehod in secion 32 4 Experimens In his secion, we describe he experimens o evaluae LEEWAY in erms of is classificaion accuracy according o differen noise levels We focus on he scenario of noisy feaure variables wih he noise-free class variable The oher wo scenarios can be sudied in he similar way Firs we describe he daase used in he experimens and noise inroducion mechanism Then we explain how he experimens are carried ou and discuss he experimenal resuls 41 Daase Descripion We choose he Nursery daase from UCI machine learning reposiory [4] as he underlying noise-free daase The Nursery daase was derived from a hierarchical decision model originally developed o rank applicaions for nursery schools I has complee insances, 5 classes, and 8 nominal aribues each wih 3 5 possible values We ake 2/3 of he daase as he raining daase and he remaining 1/3 as he esing daase A probabiliy is inroduced o conrol he noise level We apply noise levels of 0 1 in incremens of 005 In he following experimens, we use wo differen noise models: [Model U] This noise model is defined as follows: { 1, if i = j P (X = x j X = x i ) = p ji =, if i j for all x i, x j X (36) Similar o ha defined in [2], he noisy daase D a noise level is generaed as follows: for each value x i of he noise-free feaure variable X in he noise-free daase D, he observed feaure value in D will remain as x i wih probabiliy 1, and will be replaced by oher values in X, each wih probabiliy [Model R] The noise model is defined as follows: { P (X 1, if i = j = x j X = x i ) = p ji = r ji, if i j for all x i, x j X (37) where r ji is a random number and j,j i r ji =

14 14 As a random version of Model U, he noisy daase D a noise level is generaed as follows: for each value x i of he noise-free feaure variable X in he noise-free daase D, he observed feaure value in D will remain as x i wih probabiliy 1, and will be replaced by anoher value in X wih probabiliy r ji 42 Experimenal Resuls For each noise level, we generae 10 random samples of he noisy raining daases or noisy esing daases, and he resuls are averaged over all 10 samples We perform he following hree ses of experimens Learning naive Bayes classifier from noisy raining daases In his case, he noisy raining daases are generaed from he wo noise models described above In boh models, differen levels of noise are arificially inroduced ino he feaure values of he raining daase Afer learning he underlying naive Bayes classifier, we apply i o he clean esing daase and evaluae is classificaion accuracy We compare he performance of LEEWAY wih ha of he radiional naive Bayes classifier which akes he noisy observaions as noise-free daases The experimenal resuls for noise Model U and Model R are shown in Fig 3 and Fig 4, respecively 1 1 Classificaion Correc Rae Tradiional Mehod LEEWAY Feaure Noise Level Classificaion Correc Rae Tradiional Mehod LEEWAY Feaure Noise Level Fig 3 Classificaion accuracy of naive Bayes classifier learned from noisy raining daase wih noise Model U vs noise level Fig 4 Classificaion accuracy of naive Bayes classifier learned from noisy raining daase wih noise Model R vs noise level Boh Fig 3 and 4 demonsrae he following common properies: 1 The radiional naive Bayes classifier is quie oleran o noise a lower levels For noise level 05, he chance of he he observed variable X keeping is correc value x i remains high Thus, he condiional probabiliy P (X = x i C) is close o P (X = x i C) Furhermore, under he condiional independence assumpion, he poserior probabiliy disribuion of he class

15 15 variable given a new observaion is: P (C = c i X 1 = x 1, X 2 = x 2,, X n = x n ) P (X1=x1,X2=x2,,Xn=xn C=ci) P (C=ci) = P (X 1 =x 1,X 2 =x 2,,X n =x n ) n j=1 = P (X j=x j C=c i ) P (C=c i ) P (X 1=x 1,X 2=x 2,,X n=x n), c i S (38) Thus, he order of he poserior probabiliy P (C = c i X 1 = x 1, X 2 = x 2,, X n = x n ) for class variable does no change much So, when esed on a clean daase, i will give a correc classificaion wih high probabiliy However, LEEWAY performs a leas as well as he radiional mehod for lower noise levels 2 For noise level = 06 08, LEEWAY reaches is wors performance and he resuls have a larger variance over all he random daase samples In he Nursery daase, each feaure variable has 3 5 possible values When = 06 08, 1, and he marix of coefficiens in Eq 4 is close o singular Thus, neiher equaion sysem nor opimizaion mehod will give a good esimae of P (X C) 3 For noise level 06, he classificaion accuracy of he radiional naive Bayes classifier drops quickly o 0, because he chance of he observed variable X keeping is real value x i is no longer dominan Thus, he probabiliy disribuion of he noisy daase is far from ha of he noise-free daase and he classificaion accuracy grealy decreases However, LEEWAY combines he noise informaion wih he learning procedure and recovers he underlying probabiliy disribuion Thus, is classificaion accuracy sill says high Tesing he learned naive Bayes classifier on noisy esing daases In his case, differen levels of noise are arificially inroduced ino he feaure values of he esing daase We use noise Model U o generae he noisy esing daases, and perform wo experimens In one experimen, we firs learn he underlying naive Bayes classifier from differen noisy raining daases wih noise level varying from 0 o 10, and hen apply hese classifiers o a esing daase wih noise level 02 o evaluae heir classificaion accuracy In he oher experimen, we firs learn he underlying naive Bayes classifier from a noisy raining daase wih noise level 02 and hen apply i o differen noisy esing daases wih noise level varying from 0 o 10 o evaluae heir classificaion accuracy We compare he performance of LEEWAY wih ha of he radiional naive Bayes classifier which akes he noisy observaions as noise-free daases The experimenal resuls of esing he naive Bayes classifiers learned from differen noisy raining daases on a fixed noisy esing daase is shown in Fig 5 The resuls of esing he naive Bayes classifier learned from a fixed noisy raining daase on differen noisy esing daases is shown in Fig 6 Boh Fig 5 and 6 indicae ha LEEWAY is applicable o he scenario when he esing daase is corruped wih noise level differen from he raining daase Fig 5 has similar properies as Fig 3 and 4, excep ha is opimal classificaion accuracy which occurs a = 0 is degraded from 09 o 07 since

16 16 Classificaion Correc Rae Tradiional Mehod LEEWAY Feaure Noise Level on Training Daase Classificaion Correc Rae Tradiional Mehod 01 LEEWAY Feaure Noise Level on Tesing Daase Fig 5 Tesing he naive Bayes classifiers learned from differen noisy raining daases on esing daase wih 02 noise level Fig 6 Tesing he naive Bayes classifier learned from a raining daase wih 02 noise level on differen noisy esing daases he esing daase has 02 noise Fig 6 indicaes ha LEEWAY has similar performance as he radiional naive Bayes classifier mehod for noise a lower levels, and beer performance for noise a higher levels Bayesian Mehod We also compare our performance wih ha of he Bayesian mehod If we view he exended naive Bayes srucures in Fig 1 as special Bayesian neworks and ake he unknown noise-free variables as hidden variables, he ask here is o learn he poserior condiional probabiliy disribuion wih his known nework srucure and he hidden variables Anoher echnique is o exend he raining daase by adding he unknown noise-free variables and ake he exended daase as an incomplee daase ha does no have values for he added unknown variables Learning parameers of Bayesian neworks from incomplee daa can be done by gradien mehods discussed in [3] and [18], or EM inroduced in [7], or Gibbs sampling discussed in [9] We use he Bayesian nework oolbox developed by Kevin Murphy a MIT [14] The resul of he Bayesian mehod is shown in Fig 7 As shown in Fig 7, he performance of Bayesian mehod for his problem urns ou o be worse The wo echniques described above are no suiable o handle he noise daa in his case The exended naive Bayes srucures in Fig 1 are no exacly Bayesian neworks wih hidden variables, because he srucure and he space of he unobserved variables are known bu he learning procedure does no ake his prior informaion The exended raining daase is no exacly incomplee daase, eiher, because here is no a single insance of hese unobserved variables So, we expec is performance o be no beer han our approach Besides, he complexiy of learning Bayesian nework parameers wih hidden variables or incomplee daa is much higher

17 17 1 Classificaion Correc Rae Feaure Noise Level Fig 7 Learning naive Bayes classifier from noisy daa wih Bayesian mehod 5 Conclusion and Fuure Direcions In his paper, we addressed he problem of learning naive Bayes classifiers from daa where noise is added eiher on he class variable or on feaure variables or boh We proposed a new approach of reconsrucing he underlying condiional probabiliy disribuions from he observed noisy daase We experimenally evaluaed our approach on he Nursery daase whose feaure variables were arificially corruped wih differen levels of noise, and compared is performance wih he radiional naive Bayes classifier mehod and he Bayesian mehod The experimenal resuls demonsrae ha our approach improves he classificaion accuracy for feaure-corruped daa Several issues remain o be sudied One is o sudy he sample complexiy of learning naive Bayes classifier from noisy daa The probabiliy disribuion of he observed daase is approximaed by couning he frequencies Thus, is esimaion accuracy depends on he size of he raining daase The more daa are sampled, he more accurae he esimaes are However, in real applicaions, i is hard o ge a large size daase which also covers he whole feaure variables So, a problem of ineres is he sample size for a relaively reliable and sable classifier Anoher possible direcion is o sudy an uncerain daase inroduced in [13] and develop an approach of learning a naive Bayes classifier from he uncerain daa Naive Bayes classifier is a simple Bayesian classifier In classificaion problems, he raining daase conains a disinguished class variable and condiional independencies among feaure variables Making use of he independency relaionship can simplify he learning process However, dependencies among feaure variables do exis in real applicaion daases As he issue of learning Bayesian nework from daa becomes more and more popular, anoher exension of our work is o learn general Bayesian nework from noisy daa which embeds he dependencies among variables

18 18 Acknowledgemens This maerial is based upon work suppored by he Naional Science Foundaion under Gran Nos and Any opinions, findings, and conclusions or recommendaions expressed in his maerial are hose of he auhors and do no necessarily reflec he views of he Naional Science Foundaion References 1 R Agrawal and R Srikan Privacy-preserving daa mining In Proc of he ACM SIGMOD Conference on Managemen of Daa 2000 (SIGMOD 00), pages ACM Press, May M Bendou and P Muneanu Learning bayesian neworks from noisy daa In Proc of he 5h Inernaional Conference on Enerprise Informaion Sysems (ICEIS 2003), pages 26 33, April J Binder, D Koller, S Russell, and K Kanazawa Adapive probabilisic neworks wih hidden variables Machine Learning, 29: , C L Blake and C J Merz UCI reposiory of machine learning daabases hp://wwwicsuciedu/ mlearn/mlreposioryhml, C E Brodley and M A Friedl Idenifying and eliminaing mislabeled raining insances AAAI/IAAI, 1, P Clark and T Nible The cn2 inducion algorihm Machine Learning, 3(4): , A P Dempser, N M Laird, and D B Rubin Maximum likelihood from incomplee daa via he em algorihm Journal of he Royal Saisical Sociey, B 39:1 39, A Evfimievski, R Srikan, R Agrawal, and J Gehrke Privacy preserving mining of associaion rules In Proc of 8h ACM SIGKDD Inl Conf on Knowledge Discovery and Daa Mining (KDD), July W Gilks, S Richardson, and D Spiegelhaler Markov Chain Mone Carlo Mehods in Pracice CRC Press, G H John Robus decision rees: Removing ouliers from daabases In Proc of he Firs Inernaional Conference on Knowledge Discovery and Daa Mining (KDD 95), pages AIII Press, J Kubica and A Moore Probabilisic noise idenificaion and daa cleaning Technical Repor CMU-RI-TR-02-26, CMU, T M Michell Machine Learning McGraw-Hill, R R Munz and Y Xia Mining frequen iemses in uncerain daases In ICDM 03 Workshop on Fundaions and New Direcions in Daa Mining, November K P Murphy The bayes ne oolbox for malab Compuing Science and Saisics, 2001, hp://wwwaimiedu/ murphyk/sofware/bnt/bnhml 15 J R Quinlan Simplifying decision rees Inernaional Journal of Man-Machine Sudies, 27(3): , S Schwarm and S Wolfman Cleaning daa wih bayesian mehods, C M Teng Correcing Noisy Daa Machine Learning, B Thiesson Acceleraed quanificaion of bayesian neworks wih incomplee daa In Proc of he Firs Inernaional Conference on Knowledge Discovery and Daa Mining (KDD 95), pages AIII Press, 1995

19 19 A Wendemuh Modeling uncerainy of daa observaion In Proc of In Conf on Acousics, Speech and Signal Processing (ICASSP 2001), pages , J Yang, W Wang, P S Yu, and J Han Mining long sequenial paerns in a noisy environmen In Proc of he ACM SIGMOD Conference on Managemen of Daa 2002 (SIGMOD 02), June