Neural network based Boolean factor analysis of parliament voting

Transcription

1 Neural network based Boolean factor analyss of parlament votng Frolov A.A. 1, Polyakov P.Y. 2, Husek D. 3, and Rezankova H. 4 1 Insttute of Hgher Nervous Actvty and Neurophysology of the Russan Academy of Scences, Butlerova 5a, Moscow, Russa aafrolov@mal.ru 2 Insttute of Optcal Neural Technologes of the Russan Academy of Scences, Vavlova 44, Moscow, Russa labsteclo@mal.ru 3 Insttute of Computer Scence, Academy of Scences of the Czech Republc, Pod Vodárenskou věží 2, Prague, Czech Republc dusan@cs.cas.cz 4 Unversty of Economcs, nám. Churchlla, Prague 3, Czech Republc rezanka@vse.cz Summary. The sparse encoded Hopfeld lke neural network s modfed to provde the Boolean factor analyss. New, more effcent method of sequental factor extracton, based on the characterstcs behavor of the Lyapunov functon s ntroduced. Effcency of ths attempt s shown not only on smulated data but on real data from Russan parlament but as well. Key words: Boolean factor analyss, neural networks, socal networks 1 Introducton Our theoretcal analyss and computer smulatons [FSH04] revealed that Hopfeldlke neural networks are capable of the performng Boolean factor analyss (BFA) of sgnals of hgh dmenson and complexty. Factor analyss s a procedure whch maps orgnal sgnals nto the space of factors. The prncpal component analyss (PCA) s a classcal example of such mappng n the lnear case. Lnear factor analyss mples that each orgnal N-dmensonal case can be presented as X = FS + ε (1) where F s a matrx N L of factor loadngs, S s a L-dmensonal vector of factor scores and ε an error. Each component of S gves contrbuton of a correspondng factor n the orgnal sgnal. Columns of loadng matrx F gve vectors presentng correspondng factors n the sgnal space. These vectors are termed factors n followng. The mappng of the orgnal space to the factor space means that sgnals are represented by vectors S nstead of orgnal vectors X. Dmensonalty of vectors S s lover than the dmensonalty of sgnals X. Thereby the factor analyss provdes hgh compresson of orgnal sgnals.

2 862 Frolov A.A., Polyakov P.Y., Husek D., and Rezankova H. The BFA mples that a complex vector sgnal has a form of the Boolean sum of weghted bnary factors: X = S l F l. (2) In ths case, orgnal sgnals, factor scores and factor loadngs are bnary and mappng of the orgnal sgnal to the factor space means dentfcaton of factors that were mxed n the sgnal. The mean number of factors mxed n the sgnals we term sgnal complexty C. For the case of large dmensonalty and complexty of sgnals t was a challenge [FSH04] to utlze for the BFA the Hopfeld-lke neural network wth parallel dynamcs. Bnary patterns X of the sgnal space are treated as actvtes of N bnary neurons (1 - actve, 0 - nonactve) wth gradually rangng synaptc connectons between them. Durng the learnng stage patterns X (m) are stored n the matrx of synaptc connectons J accordng to the correlatonal Hebban rule: J j = M (X (m) m=1 q (m) (X (m) j q (m) ), j, J = 0, (3) where M s the number of patterns n the learnng set and bas q (m) = N È =1 X (m) /N s the total actvty of the m-th pattern. Ths form of bas corresponds to the bologcally plausble global nhbton beng proportonal to an overall neuronal actvty. Addtonally to N prncpal neurons of the Hopfeld network descrbed above we ntroduced one specal nhbtory neuron actvated durng the presentaton of every pattern of the learnng set and connected wth all prncpal neurons by bdrectonal connectons. Patterns of the learnng set are stored n the vector J of the connectons accordng to Hebban rule: J = M m=1 where q = M È (X (m) m=1 q (m) ) = M(q q), (4) X (m) /M s a mean actvty of the -th neuron n the learnng set and q s a mean actvty of all neurons n the learnng set. It s also supposed that the exctablty of the ntroduced nhbtory neuron decreases nversely proportonal to the sze of the learnng set beng 1/M after storng of all ts patterns. Due to the Hebban learnng rule (3), neurons whch represent one factor and therefore tend to fre together, become more tghtly connected than neurons belongng to dfferent factors, consttutng attractor of network dynamcs. Ths property of factors s a base of the proposed two-run procedure of factor search. Its ntalzaton starts by presentaton of random ntal pattern X (n) wth k (n) = r (n) N actve neurons. The actvty k (n) s supposed to be much smaller than the actvty of all factors. On presentaton of X (n), network actvty X evolves to some attractor. The evoluton s determned by the parallel dynamcs equaton n dscrete tme. At each tme step: X (t + 1) = Θ(h (t) T(t)), = 1,, N, X (0) = X (n) (5)

3 Neural network based Boolean factor analyss of parlament votng 863 where h are components of the vector of synaptc exctatons h (t) = N j=1 J jx j(t) (1/M)J N j=1 J j X j(t), (6) Θ s step functon, and T(t) s actvaton threshold. The frst term n (6) gves synaptc exctatons provded by the prncpal neurons of the Hopfeld network and the second one by the addtonal nhbtory neuron. The use of the nhbtory neuron s equvalent to the substracton of (1/M)J J j = M(q q)(q j q) from J j. Thus È (6) can be rewrtten as h (t) = N J jx j(t) where J = J Mqq T, q s a vector wth j=1 components q q and q T s a transposed q. As shown n [FSH04] the replacement of common connecton matrx J by J, frst, completely suppressed two global attractors whch domnate n network dynamcs for large sgnal complexty C, and second, made the sze of attractor basns around factors to be ndependent of C. At each tme step of the recall process the threshold T(t) was chosen n such a way that the level of the network actvty was kept constant and equal to k n. Thus, on each tme step k n wnners (neurons wth the greatest synaptc exctaton) were chosen and only they were actve on the next tme step. To avod uncertanty n the choce of wnners when several neurons had synaptc exctatons at the level of the actvaton threshold, small random nose was added to the actvaton threshold of each ndvdual neuron. The ampltude of the nose was put to be less than the smallest ncrement of the synaptc exctaton gven by formula (6). Ths ensured that neurons wth the hghest exctatons were kept to be wnners n spte of the random nose be added to the neurons thresholds. Nose to ndvdual neurons was fxed durng the whole recall process to provde ts convergence. As shown n [PFH06], ths choce of actvaton thresholds allows for stablzaton of the network actvty n pont or cyclc attractor of length two. When the actvty stablzes at the ntal level of actvty k (n), k (n) +1 neurons wth maxmal synaptc exctaton are chosen for the next teraton step, and the network actvty evolves to some attractor at the new level of actvty k (n) + 1. Then the level of actvty ncreases to k (n) + 2, and so on, untl the number of actve neurons reaches the fnal level r f N wth r f > p. Thus, one tral of the recall procedure contans (r f r (n) )N external steps and several steps nsde each external step to reach some attractor for a fxed level of actvty. At the end of each external step the relatve Lyapunov functon was calculated by formula Λ = X T (t + 1)JX(t)/(rN), (7) where X T (t + 1) and X(t) are two network states n the cyclc attractor (for pont attractor X T (t + 1) = X(t) ). The relatve Lyapunov functon s a mean synaptc exctaton of neurons belongng to some attractor at the end of the external step wth k = rn neurons. Attractors wth the hghest Lyapunov functon would be obvously wnners n the most trals of the recall process. Thus, more and more trals are requred to obtan new attractor wth relatvely small value of Lyapunov functon. To overcome ths problem the domnant attractors should be deleted from the network memory. The deleton was performed accordng to Hebban unlearnng rule by substracton J j, j from synaptc connectons J j where

4 864 Frolov A.A., Polyakov P.Y., Husek D., and Rezankova H. J j = η J(X)[(X(t 1) r)(xj(t) r) + (Xj(t 1) r)(x(t) r), (8) 2 J(X) s the average synaptc connecton between actve neurons of the attractor, X(t 1) and X(t) are patterns of network actvty at last tme steps of teraton process, r s the level of actvty, and η s an unlearnng rate. For pont attractor X(t) = X(t 1) and for cyclc attractor X(t 1) and X(t) are two states of attractor. If unlearn rule (8) s appled the tme to reveal all factors s proportonal to the number of elements n connecton matrx,.e. N 2 There are three mportant smlartes between the descrbed procedure of the BFA and lnear PCA. Frst, PCA s based on the smlar covarance matrx as s the connecton matrx n Hopfeld network. Second, factor search n PCA can be performed by the teraton procedure smlar to that descrbed by (5) and (6) but bnarzaton of synaptc exctatons by the step functon must be replaced by ther normalzaton: X (t+1)h (t)/ h(t). Then the teraton procedure startng from any random state converges to egenvector f 1 of covaraton matrx wth the largest egenvalue Λ 1. Just ths egenvector s treated as the frst factor n PCA. Thrd, to obtan the next factor the frst factor must be deleted from the covaraton matrx by the substracton of Λ 1f 1f1, T and so on. The substracton s smlar to Hebban unlearnng (8). However, the BFA by Hopfeld-lke network has one prncpal dfference from lnear PCA. Attractors of teraton procedure n PCA are always factors whle n Hopfeld-lke networks the teraton procedure can converge to factors (true attractors) and to spurous attractors whch are far from all factors. Thus, two man questons arse n vew of the BFA by the Hopfeld-lke network. Frst, how often would network actvty converge to one of the factors startng from the random state? Second, s t possble to dstngush true and spurous attractors when network actvty converges to some stable state? Both these questons are answered n the next Secton. 2 Artfcal sgnals To reveal peculartes of true and spurous attractors we performed computer experments wth artfcal sgnals. Each pattern of the learnng set s supposed to be a Boolean superposton of exactly C factors and each factor s supposed to contan exactly n = pn 1-s and (1 p)n 0-s. Thus, each factor f (l) Bn N and for each pattern of the learnng set, vector of factor scores S BC L where È Bn N = X X {0, 1}, N X = n. We supposed factor loadngs and factor scores =1 to be statstcally ndependent. As an example, Fg. 1 demonstrates changes of relatve Lyapunov functon for N = 3000, L = 5300, p = 0.02 and C = 10. A recall process started at r (n) = Trajectores of network dynamcs form two separated groups. As shown n Fg. 2, the trajectores wth hgher values of the Lyapunov functon are true and wth lower ones are spurous. Ths Fgure relates values of the Lyapunov functon for patterns of network actvty at ponts r = p to maxmal overlaps of these patterns wth factors. The overlap between two patterns X (1) and X (2) wth p actve neurons was calculated by formula m(x (1),X (2) 1 ) Np(1 p) N =1 (X (1) p)(x (2) p)

5 Neural network based Boolean factor analyss of parlament votng 865 1,0 1,0 0,8 0,6 0,8 0,4 0,2 0,01 0,02 0,03 r 0,6 0,0 0,2 0,4 0,6 0,8 1,0 Ov Fg. 1 Relatve Lyapunov functon λ n dependence on the relatve network actvty r. Fg. 2 Values of normalzed Lyapunov functon n relaton to overlaps wth the closest factors. Accordng ths formula the overlap between equal patterns s equal to 1 and the mean overlap between ndependent patterns s equal to 0. Patterns wth a hgh Lyapunov functon have hgh overlap wth one of the factors, whle the patterns wth a low Lyapunov functon are far from all the factors. It s shown that true and spurous trajectores are separated by the values of ther Lyapunov functons. In Fgs 1 and 2 the values of the Lyapunov functon are normalzed by a mean value of ths functon over true attractors at the pont r = p. The second characterstc feature of true trajectores s the exstence of a knk at a pont r = p where the level of network actvty concdes wth that n factors (see Fg. 1). When r < p, the ncrease of r results n almost lnear ncrease of the relatve Lyapunov functon. The ncrease of r occurs n ths case due to the jonng of neurons belongng to the factors that are strongly connected wth other neurons of the factor. Then the jonng of new neurons results n proportonal ncrease of mean synaptc exctaton to the actve neurons of factor that s just equal to ther relatve Lyapunov functon. When r > p, the ncrease of r occurs due to jonng of some random neurons that are connected wth the factor by week connectons. Thus, the ncrease of the relatve Lyapunov functon for true trajectory sharply slows and t tends to the values of the Lyapunov functon for spurous trajectores. The use of these two features of true trajectores provdes relable tool for recognton of factors and f unlearn rule s appled the tme to reveal all factors s proportonal to the number of elements n connecton matrx,.e. N 2. Ths phenomenon was clearly confrmed n our prevous papers [FHP04, HFR05, PFH06] where our method was used for textual data analyss. 3 Analyss of parlament votng There are many real problems [DLW03] when the BFA s helpful. We proved ts contrbuton n the nformaton retreval [FHP04,HFR05,PFH06], analyzng textual data. For the followng analyss we used as a data source the record of deputes votng n the Russan parlament n 2004 [PAT05]. Each poolng s encoded as a

6 866 Frolov A.A., Polyakov P.Y., Husek D., and Rezankova H. bnary vector wth component 1 f the correspondent deputy voted affrmatvely and 0 negatvely. The number of votng cases durng the year was The number of deputes (consequently the dmensonalty of sgnal space and network sze) amounts 430 (20 deputes voted less than 10 tmes were excluded from the analyss). Fg. 3 shows the Lyapunov functon along trajectores startng from 1500 random ntal states. All these states converge to four trajectores. Two of them have obvous knks and therefore were dentfed as two factors. The factor wth the hghest Lyapunov functon contans 50 deputes and completely concdes wth the fracton of the Communst Party (CPRF). Another factor contans 36 deputes. All of them belong to the fracton of Lberal-Democratc Party (LDPR) whch contans totally 37 deputes. Thus one of the members of ths fracton fell out of the correspondng factor. The ponted knks at the correspondng trajectores gve evdence that these fractons are most dscplne and ther members vote coherently. Fg. 4 demonstrates number of actve neurons Fg. 3 Relatve Lyapunov functon λ n dependence on the number of actve neurons. Thck ponts are knks of the frst and second factors number of actve neurons Fg. 4 The same as n Fg. 3 after deletng two frst factors. Thck pont s knk of the thrd factor. trajectores after deletng of the two factors. Startng from 1500 ntal states they converge to only two trajectores. One of them has a knk but t s not so strct as for CPRF and LDPR factors. We supposed that the pont where the second dervatve of the Lyapunov functon by k has mnmum s the thrd factor. The factor contans 37 deputes. All of them belong to the fracton Motherland (ML) whch contans totally 41 deputes. Thus 4 of ts members fell out of the factor. The fuzzness of knk at the trajectory gves evdence that ths fracton s not as homogeneous as the two frst ones and actually the fracton splt at two fractons n Matchng of neurons along the second trajectory n Fg. 4 wth the lst of deputes has shown that t corresponds to the fracton Unted Russa (UR). Ths fracton s the largest and contans totally 285 deputes but s less homogeneous. Therefore the Lyapunov functon along the trajectory s low and t has no knk at all. Fg. 5 shows trajectores of neurodynamcs after addtonal deletng the thrd factor from the network. Two remanng trajectores contan members of UR and ndependent deputes (ID). The upper trajectory contans only members of UR and lower one - manly ID but also members of UR. Ths s addtonal evdence of heterogenety of UR. Factors

7 Neural network based Boolean factor analyss of parlament votng 867 Table 1. Relaton between parlament fractons and factors. fractons/factors UR 283 / / 0 0 / 0 0 / 0 2 / 0 CPRF 0 / 0 51 / 49 0 / 0 0 / 2 0 / 0 LDPR 1 / 2 0 / 0 36 / 35 0 / 0 0 / 0 ML 3 / 3 0 / 0 0 / 0 37 / 38 1 / 0 Independent 1 / 14 0 / 0 0 / 1 0 / 1 15 / 0 UR and ID were dentfed by mnmums of the second dervatves along the correspondng trajectores. The general relaton between the parlament fractons and obtaned factors s shown n Table I. The ft between the fractons and the factors was estmated by F-measure. Averaged over all fractons t amounted to As number of actve neurons Fg. 5 The same as n Fgs. 3 and 4 after deletng three frst factors. Fg. 6 2D map of electoral college. Thn lnes - borders of clusters. - UR, - CPRF, - LDPR, - ML, - ID. obtaned factors do not overlap so we may nterpret them as clusters and compare our results wth those obtaned by some tradtonal methods of clusterng [MPP05]. Frst, we tred clusterng methods based on smlarty matrx. Smlarty between two deputes was calculated by comparson of vectors of ther votng. We used dfferent measures of smlarty: Eucldan dstance, cosne, Jaccard and Dce. Both herarchal and K-means clusterng gave clusters far from parlament fractons: all fractons ntersected n clusters and fracton LDPR could not be separated from ER at all. Second, we performed mappng of parlament members by the method of multdmensonal scalng. The results are shown n Fg.6. Obtaned map was clustered. The border of clusters are depcted by thn lnes. Generally, as factors obtaned before, clusters concde wth parlament fractons except for ndependent deputes. The results of clusterng and factorzaton are compared n the Table 1. The mean F-measure amounted to 0.95, that s slghtly smaller than that obtaned for factors.

8 868 Frolov A.A., Polyakov P.Y., Husek D., and Rezankova H. 4 Concluson We have shown that the modfed recurrent neural network s capable of performng the BFA of the sgnals of hgh dmenson and complexty. The new, more effcent method of sequental factor extracton, based on the characterstcs behavor of the Lyapunov functon was descrbed and used for analytcal tasks. The effcency of ths attempt s shown not only on the smulated data but on real data as well. In our prevous papers we showed ts hgh effcency n applcaton for textual data. Here ts ablty s demonstrated n the feld of poltcs. The resultng factors were not overlappng because of the data nature. Ths allowed us to compare the results obtaned by the BFA wth those obtaned by some tradtonal methods of clusterng. None of them gave approprate results. Only results obtaned by the multdmensonal scalng method were partally successful, but stll worse then the BFA. Acknowledgement Ths work was partally supported by grant RFBR No , by the projects No.1ET , 201/05/0079 and by the Insttutonal Research Plan AVOZ awarded by the Grant Agency of the Czech Republc. References [DLW03] De Leeuw, J.: Prncpal component analyss of bnary data. Applcaton to roll-call analyss. (2003) [FHM03] Frolov, A.A., Husek, D., Muravev, I.P.: Informatonal effcency of sparsely encoded Hopfeld-lke autoassocatve memory. Optcal Memory & Neural Networks, 12, (2003) [FSH04] Frolov, A.A., Srota, A.M., Husek, D., Muravev, I.P. Polyakov, P.Y.: Bnary factorzaton n Hopfeld-lke neural networks: sngle-step approxmaton and computer smulatons. Neural Networks World, 14, (2004) [FHP04] Frolov, A.A., Husek, D., Polyakov, P.A., Rezankova, H., Snasel, V.: Bnary Factorzaton of Textual Data by Hopfeld-Lke Neural Network. In: Antoch J.(ed) Computatonal Statstcs. Physca Verlag, Hedelberg, (2004) [HFR05] Husek, D., Frolov, A.A., Rezankova, H., Snasel, V., Polyakov, P.Y.: Neural Network Nonlnear Factor Analyss of Hgh Dmensonal Bnary Sgnals. In: SITIS Unversty of Bourgogne, Djon, (2005) [PFH06] Polyakov, P.Y., Frolov, A.A., Husek, D.: Bnary factor analyss by Hopfled network and ts applcaton to automatc text classfcaton. In: Neuronformatcs MIFI, Moscow, Russa (2006) [PAT05] [MPP05]