MMI-based Training for a Probabilistic Neural Network

Transcription

1 MMI-based Training for a Probabilisti Neural Network Nan Bu and Tosio Tsuji Department of te Artifiial Complex Systems Engineering Hirosima University Higasi-Hirosima, JAPAN bu@bsys.irosima-u.a.jp Osamu Fukuda National Institute of Advaned Industrial Siene and Tenology Tsukuba, JAPAN fukuda.o@aist.go.jp Abstrat Probabilisti neural networks (PNNs) tat inorporate te Bayesian deision rule and statistial models ave been widely used for pattern lassifiation. Effiient estimation of te PNN s weigts, owever, is still a major problem. In tis paper, we propose a new training seme based on a disriminative riterion, maximum mutual information (MMI), and apply tis metod to te log-linearized Gaussian mixture network (LLGMN) wi is one of te PNNs. Te MMI training aieves a onsistent estimator of network weigts, and inludes te onventional maximum likeliood (ML) algoritm as a speial ase. Also, te dynamis of terminal attrator (TA) is introdued for iteration ontrol of te MMI training. Finally, te lassifiation ability of te proposed metod is examined wit a pattern lassifiation problem of te eletromyogram (EMG) signals, and found tat te MMI training results in better lassifiation tan te onventional ML algoritm. I. INTRODUCTION Reently, neural networks ave been inreasingly popular in a pattern lassifiation field beause of teir outstanding performane in approximating te desired funtional mapping between input and output patterns. However, several fators ave indered te development of NN lassifiers su as te slow learning onvergene, need for a large amount of training data, and loal minima. To takle tese problems, numerous attempts ave been made to integrate speifi knowledge into te NN ariteture. In one approa, te NN alled te probabilisti neural network (PNN), is trained to estimate te probability density funtion (pdf) of te pattern in order to improve te lassifiation ability [1]-[5]. In [4], Tsuji et al. ave proposed a feedforward probabilisti NN, a log-linearized Gaussian mixture network (LLGMN), wi is based on a log-linear model and a Gaussian mixture model (GMM). LLGMN is a tree-layer NN wit te semiparametri model of te pdf inorporated, te outputs of wi are expeted to approximate te posterior probabilities of te patterns to be disriminated. Te units in te idden layer an be interpreted as mixture omponents in GMM, and te weigt oeffiients orrespond to te statistial parameters in GMM, su as te mixture oeffiients, mean values, and standard deviation. As an appliation example, LLGMN as been suessfully used for te eletromyogram (EMG) pattern lassifiation problem [5]. A maximum likeliood (ML) riterion is employed to estimate te weigts of LLGMN wit a bakpropagation-based training algoritm. Te ML riterion as been extensively used to estimate unknown parameters of ypotesized pdf models. Unbiased estimators an be obtained, if 1) te true distribution of te sampled data is inluded in te spae of te distribution of te ypotesized models, and 2) suffiient training data is available [6]. Yet if te assumptions are not satisfied, te ML riterion may no longer provide a reliable estimator, tus ausing mislassifiation, and ig lassifiation performane may not be expeted. Altoug LLGMN as a reasonable struture as a NN lassifier, it suffers te drawbaks of te ML riterion as well. Terefore, a better disriminative training riterion is needed in order to rea te full potential of LLGMN. In te field of spee reognition, a maximum mutual information (MMI) riterion as been used to train lassifier based on idden Markov model (HMM), and it as been sown tat a training algoritm based on te MMI riterion an provide more onsistent lassifiation tan ML [6]-[1]. Te MMI estimation was popularized by L.R. Bal [7] wo applied it to HMM parameter estimation in Sine ten, many researers ave reported te promising disriminative training ability of MMI [6][8]-[1]. Valtev et al. implemented te MMI estimation for optimizing te struture and parameters of a ontinuous density HMM-based voabulary reognition system [8], and Woodland and Povey obtained better reognition auray tan te orresponding ML estimation [9]. Also, in [1], Slüter et al. used te MMI riterion for ontinuous spee reognition, and a better word reognition rate was aieved tan te ML riterion. Te prinipal idea of MMI, wen applied to estimate HMMs, is maximizing te ratio of te orret model to all oter models, wi an be simply expressed in te form: θ m arg max θ m P (O m ), P (m)p (O m) were O is a time sequene of aousti data, and m is a model or a lass tat ategorizes te data; θ is te parameter of HMM model; m is te orret model for te given data, and θ m is te estimated parameter for model m. P (m) and P (O m) are te prior probability for model m and te posterior probability for O. In ontrast, te ML estimation just maximizes te likeliood, namely, P (O m ). Tis is an important benefit of MMI, sine not only te orret model, but all possible models of te training data are onsidered. Furtermore, te MMI riterion attempts to maximize te disrimination between te orret model and te inorret models, and to maximize te model separability rater tan a likeliood funtion [6]. From /3/$ IEEE 2661

2 O1,1 (3) (3) (3) O1 O OC 1,1 (1,1) w 1 (1) O 1 x1 O1,M 1 1,M 1,m x2 (,m) w O,m X1 X2 X X H Nonlinear transformation Fig. 1. xd Te struture of LLGMN. (1) O H OC,M C C,MC te viewpoint of disrimination ability, te MMI riterion is suit for training probabilisti NN, su as LLGMN. Inspired by te above, te present paper proposes a MMIbased training seme for LLGMN instead of te onventional ML metod to improve pattern disrimination. A gradient optimization metod is used aording to te bakpropagation rule, and te onept of a terminal attrator (TA) is introdued into te training seme for iteration ontrol, onsidering tat standard gradient metods are often slow to onverge. MMI is a onsistent estimator, and it is expeted tat better disrimination an be realized wit te proposed training seme. Tis paper is organized as follows. Setion II briefly introdues LLGMN and te ML training algoritm. Te MMIbased training algoritm and te omparison between MMI and ML riterion are desribed in Setion III. Setion IV proposes te extended training seme inluding te dynamis of TA. Te results of pattern lassifiation experiments of te eletromyogram (EMG) are presented in Setion V. Te final setion onludes te paper. II. A PROBABILISTIC NEURAL NETWORK - LLGMN Te LLGMN is based on te Gaussian mixture model (GMM) and te log-linear model of pdf. By applying te loglinear model to a produt of te mixture oeffiient and te mixture omponent of GMM, te semiparametri model of pdf is inorporated into a tree-layer feedforward NN as sown in Fig. 1. For training, a simple algoritm based on te ML riterion and te bakpropagation rule is employed [4]. A. Struture of LLGMN First, in te pre-proess, te input vetor x R d is onverted into te modified vetor X R H as follows: X (1, x T,x 2 1,x 1 x 2,,x 1 x d,x 2 2,x 2 x 3,, x 2 x d,,x 2 d ) T (1) were x i,i 1, 2,,d, are te elements of x and H 1+d(d+3)/2. Te first layer onsists of H units orresponding to te dimension of X and te identity funtion is used for ativation of ea unit. (1) O ( 1,,H) in Fig. 1 denotes te output of te t unit in te first layer. In te seond layer, ea unit reeives te output of te first layer weigted by te weigt w (,m) and outputs te posterior probability of ea omponent. Te relationsips between te input of unit {, m} in te seond layer ( I,m ) and te output ( O,m ) are defined as H I,m (1) O w (,m) 1 exp[ I,m ] O,m C MC 1 m 1 exp[ I,m ] (3) were w (C,MC ) ( 1,,H). Te tird layer onsists of C units orresponding to te number of lasses. Te unit ( 1,,C) sums up te outputs of M units {, m} (m 1,,M ) in te seond layer. Te funtion between te input and te output is desribed as (3) O (3) I M m1 O,m (4) were te output (3) O orresponds to te posterior probability of lass. By optimizing te weigt w (,m), LLGMN is expeted to approximate te posterior probability P ( x) ( 1,,C), wen an input vetor x is provided to it. Te next subsetion briefly desribes te supervised training algoritm based on ML riterion. B. Training Seme Based on an ML Criterion In te training proedure, a set of vetor X (x (1),, x (N) ) and te orresponding teaer vetor ( 1,,T,, C )(n 1,,N) are used. Te teaer vetor provides perfet lassifiation, tat is, ĉ 1for te partiular lass ĉ and T for te oter lasses. Te network is assumed to aquire te probability distribution of te training data if for all x te output (3) O is lose enoug to te teaer vetor. Te ML-training estimates te weigt W as N C W ML arg max P ( x ). (5) W n1 1 Taking te logaritm of te rigt side of (5), te following log-likeliood funtion an be derived: L n1 1 log P ( x ) n1 1 T log (3) O (6) were (3) O orresponds to P ( x ). Te objetive funtion J is defined as J ML J n T log (3) O (7) n1 n

3 and training proess is to minimize J ML, tat is, to maximize te likeliood. Te weigt modifiation w (,m) is given as follows: n w (,m) w (,m) w (,m) 1 M m 1 ( (3) O η ( n1 1 n w (,m) T log (3) O ) log (3) O (3) O (3) O O,m were η> is te learning rate. A. MMI Training Criterion O,m I,m ) O,m (3) O III. MMI-BASED TRAINING I,m w (,m) (8) X (9) Te MMI riterion used in te present paper is based on Sannon s information teory [11]. Consider two random variables C and X, te mutual information (MI), I(C; X), is defined as I(C; X) H(C) H(C X) (1) were H(C) is te entropy of C and H(C X) is te onditional entropy of C given X. Tey an be expressed in te form: H(C) P () log P () (11) H(C X) P (, x) log P ( x) x (12) were P (), P (, x), and P ( x) stand for prior probability of, joint probability of and x, and posterior probability of, respetively. Beause entropy is also interpreted as a measure of unertainty of a random variable, it is reasonable to take te MI between te lass (C) and te input data (X) as te redution of unertainty of lasses given te observed data generated from tese lasses. Suppose tat te distribution of prior probability of C is known, i.e. H(C) is onstant, maximizing I(C; X) means minimizing te onditional entropy H(C X), intuitively, te unertainty of te distribution of P ( x) is redued. Te parameters in PNN, e.g. LLGMN, are tus expeted to be estimated aording to te MMI riterion. Extending (1) wit (11) and (12), I(C; X) an be rewritten as follows I(C; X) P ( x)p (x) log P ( x) P (). (13) x Tus, te MI an be expressed as a funtion of te weigts in PNN, tat is, I(C; X) f(p ( x)) f(g(w )), and by te MMI riterion, te weigt W is estimated as W MMI arg max I(C; X) W arg max P ( x)p (x) log P ( x) W P (). (14) x Aording to te definition in II, te objetive funtion for LLGMN training based on MMI, J MMI, is now defined as J MMI I(C; X) n1 1 n1 1 P ( x )P (x ) log P ( x ) P () (3) O P (x (3) O ) log P (). (15) Te MI an be maximized by minimizing te objetive funtion J MMI. B. Comparing MMI wit ML Rewriting te objetive funtion (15), we get J MMI P ( x )P (x )(log P () log P ( x )) n1 1 P (, x ) log P () P (, x ) log P ( x ). n1 1 (16) Assuming tat te prior probability P () and te joint probability P (, x ) are or 1, P (, x ) log P () in (16) equals beause P (, x ) log P () P (x )P () log P (). Furtermore, P (, x ) an be interpreted as te teaing vetor, beause P (, x ) equals 1 wen x is generated from lass and for te oter ases. Consequently, (16) an be simplified as J MMI P (, x ) log P ( x ) n1 1 n1 1 n1 1 log P ( x ) T log (3) O J ML. (17) Tis means tat J ML is inluded in J MMI, and is a speial ase of J MMI. On te oter and, by symmetry, I(C; X) also follows tat I(C; X) H(X) H(X C). (18) As te distribution of P (x) is not affeted by te weigt of LLGMN, H(X) an be onsidered to be onstant. Similar to te explanation for (1)-(12), maximizing te MI would redue te unertainty of te distribution of P (x ). As a 2663

4 result, bot te posterior probability distribution, p( x), and te emission probability distribution, p(x ), an be estimated simultaneously wen maximizing te MI. Unlike MMI, te ML riterion an estimate just one probability distribution. If bot te ML estimation and te MMI estimation rea global minima in te training proess of LLGMN, te resultant weigts sould be te same. However, wen te global minima are not reaable or te training iteration is stopped at some loal minima, training wit te MMI riterion would provide better disrimination. IV. MMI TRAINING ALGORITHM WITH TERMINAL ATTRACTOR Tis setion introdues te details of te proposed training seme. Te modified objetive funtion J M is defined on te basis of te MMI objetive funtion (15), J M I I(C; X) I (3) O P (x (3) O ) log (19) P () n1 1 were I is onstant and equals te maximum of I(C; X), so tat J M as te minimum as. However, it is diffiult to alulate I wit no desired NN outputs value, e.g. te teaer vetor. Furtermore, witout teaer vetor, te NN would be trained like an self-organizing maps (SOM), and training of te SOM wit a gradient metod must be toug. In tis paper, for simplifiation, we applied te teaer vetor defined in II-B to (19), and defined te following objetive funtion: J I n1 1 (3) O P (x ) log (3) O P () (2) were P () N n1 P ( x )P (x )( 1,,C), and it is assumed P (x )1/N (n 1,,N). It is onsidered tat I(C; X) beomes maximum wen (3) O T, wi suggests tat, given te input x, te desired probabilities are obtained; tus we get I log C. For standard gradient metod, te weigt modifiation w (,m) is derived as w (,m) η w (,m) wit a fixed η>as te learning rate. as follows: w (,m) w (,m) ( n1 1 n1 1 (3) O ( (3) O (3) O w (,m) P (x ) log P (x ) log (21) an be derived (3) O P () ) (3) O P ( ) ) M m 1 n1 1 (3) O O,m O,m I,m I,m w (,m) (3) P O (x )[log (3) P ( ) O P (x ) P ( ) +1] (δ, (3) O ) O,mX (22) were δ, equals 1 wen, and oterwise. Beause te standard gradient metod is always ritiized for its slow onvergene, tis paper inorporates te dynamis of te terminal attrator (TA) [12] into te training seme in order to regulate te onvergene time. Te differential equation of TA is defined as u u β. (23) Wen te parameter β is determined as <β<1, u is a monotonially non-inreasing funtion, and always onverges stably to te equilibrium point in a finite time, sine te Lipsitz onditions are violated at u : d u du βu β 1 u. (24) u Te onvergene time is fixed depending on te initial ondition u u : tf u du t f dt u u u1 β (1 β) < (25) were β determines ow te dynamis onverges, su as smootly or sarply. Inorporating TA into te objetive funtion (2) of LL- GMN an regulate te onvergene time of te training, and te network training onverges to te global minimum or one of te loal minima in a finite speified time [13]. Te weigts of LLGMN are onsidered as te time dependent ontinuous variables, and time derivative of w,m is defined as: ẇ,m η ta γ w,m (26) γ M J β H 1 m1 1 ( w,m ) 2 (27) were η ta > is positive, and γ is alulated using te onstant β. Te time derivative of te energy funtion J an be alulated as: M H ( ) J w,m ẇ,m 1 m1 1 η ta J β. (28) Aording to te dynamis of TA (23)-(25), te onvergene time an be given as t f tf dt J 1 β η ta (1 β) Jf J dj J J 1 β J 1 β f η ta (1 β) (29) 2664

5 Power level C. 1 C. 2 C. 3 C. 4 C. 5 C [mv] Classifiation ratio (%) Number of training data MMI ML Entropy, H(t) Classifiation result 1.. /n/ /o/ /e/ /u/ /i/ /a/ Classified Mislassified Time [se] Fig. 2. An example of te lassifiation result based on te proposed MMI training algoritm. were J is an initial value of te energy funtion J alulated using initial weigts, and J f is te final value of J at te equilibrium point. For J f, te equal sign of (29) is eld. Tus, te onvergene time an be speified by learning rate η ta. In ontrast, for J f, te onvergene time is always less tan te upper limit of (29). V. EXPERIMENTS Pattern lassifiation experiments for te EMG signals were onduted using te training metods based on MMI and ML riteria. Te EMG signal used were six-annel data (d 6,H 28) orresponding to six lasses (C 6), wi were measured from six mimeti and ervial musles of a patient wit ervial spine injury, and six lasses are orresponding to six voable sounds, i.e. /a/, /i/, /u/, /e/, /o/, and /n/. In te experiments, te patient was asked to utter te six vowels in te order. Te EMG signals were measured wit a sampling frequeny f d 1 [Hz], ten retified and filtered by a Butterwort filter (utoff frequeny: 1 [Hz]). Ea sampled data was normalized to make te sum of six annels equal 1, and te feature vetors x [x 1,,x 6 ] were inputted into te LLGMN. A power level was estimated from te EMG signals, and was ompared wit a prefixed tresold to determine weter te patient uttered or not. Te LLGMN inludes 28 units in te first layer, 18 units in te seond layer orresponding to te total number of omponents (M 3, ( 1,...,6)), and six units in te tird layer. In training, te objetive funtions J (2) and J ML (7) were used. Te dynamis of TA was used in bot te MMI and te Fig. 3. rule 1. Fig. 4. rule 2. Classifiation ratio (%) Classifiation ratios for various training data number wit deision Number of training data MMI ML Classifiation ratios for various training data number wit deision ML training algoritm, and set wit β.8, t f 1se and t.25 se, tat results 4 iterations. An example of te lassifiation result based on te proposed MMI training algoritm is sown in Fig. 2. In tis figure, six annels of EMG signals, te power level, te entropy H(t) alulated from te output probability of LLGMN, and te lassifiation results are plotted. Te gray areas indiate no utterane. Te lassifiation rate was about 9.9% in tis experiment. Altoug mislassifiations were made for /o/, it sould be notied tat EMG patterns for utterane of /u/ and /o/ are similar, furtermore, te power level of /o/ is relatively lower tan te oters. To verify te disrimination performane of te proposed training algoritm, omparison experiments wit ML were onduted. Five sets of training data were used for bot metods to train LLGMN, and in ea set te numbers of training data for ea lass are anged from 1 to 4. After training, 1 data patterns for ea lass tat were not used in training were prepared for lassifiation. Te lass for te input pattern is deided wit two deision rules: 1) Te lass wit te largest posterior probability. 2) Te lass wose posterior probability is larger tan.8. If none is larger tan.8, te determination is suspended. Te experimental results were given in terms of te ratio of orret lassifiation, wi is te average of te lassifiation ratios of six lasses. Fig. 3 plots te mean values and te standard deviations of te ratios of orret lassifiation for various numbers of training data, wit te deision rule 1). 2665

6 Entropy (bit) Fig. 5. outputs Number of training data MMI ML Effet of te training data number on te entropy of LLGMN s From tis figure, we an see tat te MMI metod gives better lassifiation tan ML for all training data numbers. Te lassifiation ratios aording to deision rule 2) are sown in Fig. 4. Wen a strit deision rule is used, te result of ML metod degrades mu more tan MMI. Given te input data, LLGMN trained wit MMI riterion generally outputted iger posterior probabilities for te orresponding orret lasses, and tis demonstrates te superior estimation ability and te training effiieny of te MMI metod. Te MMI training metod realizes a better estimation of te parameters in LLGMN in te same training period, so tat LLGMN approximates more aurate posterior probability tan te ML metod. On te oter and, wen te number of training data dereases, te results based on te MMI metod keep in a ig level in Figs. 3 and 4, wile lassifiation rate of te MLtrained LLGMN tends to derease in Fig. 4. Te ML metod annot aieve a onsistent estimation wen te training data is not suffiient. Tis an also be illustrated by examining te entropy of te NN s outputs, H (see Fig. 5), wi is defined as H (3) O log (3) 2 O. (3) 1 Higer entropy means more unertainty of te lassifiation result. In ontrast to te ML training metod, we find te MMI training provides a reliable estimate of ea weigt in LLGMN even wit a small training data size. VI. CONCLUSIONS In te present paper, an MMI based training algoritm as been proposed for a probabilisti NN, LLGMN. Te algoritm is derived from te formulation of mutual information. Te unertainty of te probability distribution tat is te funtion of NN s weigts is redued wen maximizing te mutual information between te input and output of LLGMN, and tus, te network weigts an be optimized aording to te MMI riterion. Comparing wit te onventional ML training riterion, te ML riterion an be interpreted as a speial ase of MMI; furtermore, MMI training an teoretially optimize te posterior probability and te emission probability simultaneously. For simplifiation, a supervised training metod as been proposed in tis paper, and te dynamis of TA as been introdued into te training algoritm to regulate te onvergene time. To examine te performane of te proposed training algoritm, pattern lassifiation experiments were onduted on te EMG signal. Te results sowed tat te MMI training metod aieves more effiient estimate of te NN weigts, and performs better lassifiation tan ML. Wit respet to te entropy of te NN s outputs, it is lear tat te MMI metod provides more reliable estimates of te NN s weigts. Furter resear is needed to make a detailed investigation into te probabilisti interpretation on te MMI and te relationsip between MMI and ML. Also, it will be interesting to apply te proposed metod to oter probabilisti NNs [14]. ACKNOWLEDGMENT Te autors would like to tank New Energy and Industrial Tenology Development Organization (NEDO) of Japan for te finanial support. REFERENCES [1] D.F. Spet, Probabilisti neural networks, Neural Networks, Vol. 3, No. 1, pp , 199. [2] J.S. Bridle, Probabilisti interpretation of feedforward lassifiation network outputs, wit relationsips to statistial pattern reognition, in Neuroomputing: Algoritms, Aritetures, and Appliations, S.F. Fogelman and J. Hault, Eds. New York: Springer-Verlag, 1989, pp [3] H.G.C. Tråvén, A neural network approa to statistial pattern lassifiation by semiparametri estimation of probability density funtions, IEEE Trans. Neural Networks, Vol. 2, No. 3, pp , [4] T. Tsuji, O. Fukuda, H. Iinobu, and M. Kaneko, A Log-Linearized Gaussian Mixture Netwrok and Its Appliation to EEG Pattern Classifiation, IEEE Trans. on Systems, Man, and Cybernetis-Part C: Appliations and Reviews, Vol. 29, No. 1, pp. 6-72, [5] O. Fukuda, T. Tsuji, A. Otsuka, and M. Kaneko, EMG-based umanrobot interfae for reabilitation aid, Pro. IEEE Int. Conf. on Robotis and Automation, pp , Leuven, [6] L.R. Bal, M. Padmanaban, D. Naamoo, and P.S. Gopalakrisnan, Disriminative Training Of Gaussian Mixture Models For Large Voabulary Spee Reognition Systems, Proeeding of International Conf. on Aoustis, Spee and Signal Proessing, Vol. 2, pp , Atlanta, [7] L.R. Bal, P.F. Brown, P.V. de Souza, and R.L. Merer, Maximum Mutual Information Estimation of Hidden Markov Model Parameters for Spee Reognition, Proeeding of International Conf. on Aoustis, Spee and Signal Proessing, pp , Tokyo, [8] V. Valtev, J.J. Odell, P.C. Woodland, S.J. Young, MMIE Training of Large Voabulary Reognition Systems, Spee Communiation, Vol. 22, No. 4, pp , [9] P.C. Woodland and D. Povey, Large Sale Disriminative Training for Spee Reognition, Pro. of te Worksop on Automati Spee Reognition, Paris, Frane, 2. [1] R. Slüter, W. Maerey, B. Müller, H. Ney, Comparison of Disriminative Training Criteria and Optimization Metods for Spee Reognition, Spee Communiation, Vol. 34, No. 3, pp , 21. [11] T.M. Cover, J.A. Tomas, Elements of Information Teory, Wiley, New York, [12] M. Zak, Terminal attrators for addressable memory in neural networks, Pysis Letters A, Vol. 133, No. 1,2, pp , [13] O. Fukuda, T. Tsuji, and M. Kaneko, Pattern lassifiation of EEG signals using a log-linearized Gaussian mixture neural networks, Pro. IEEE Int. Conf. Neural Networks, Vol. V, pp , [14] T. Tsuji, N. Bu, O. Fukuda, and M. Kaneko, A reurrent log-linearized Gaussian mixture network, IEEE Trans. Neural Networks, Vol. 14, No. 2, pp ,