SIMULTANEOUS DEMPSTER-SHAFER CLUSTERING AND GRADUAL DETERMINATION OF NUMBER OF CLUSTERS USING A NEURAL NETWORK STRUCTURE.

Transcription

1 SIMULTANEOUS DEMPSTER-SHAFER CLUSTERING AND GRADUAL DETERMINATION OF NUMBER OF CLUSTERS USING A NEURAL NETWORK STRUCTURE Johan Schubert Department of Informaton System Technology Dvson of Command and Control Warfare Technology Defence Research Establshment SE 7 9 Stockholm, Sweden schubert@sto.foa.se ABSTRACT In ths paper we extend an earler result wthn Dempster- Shafer theory [ Fast Dempster-Shafer Clusterng Usng a Neural Network Structure, n Proc. Seventh Int. Conf. Informaton Processng and Management of Uncertanty n Knowledge-Based Systems (IPMU 98)] where several peces of evdence were clustered nto a fxed number of clusters usng a neural structure. Ths was done by mnmzng a metaconflct functon. We now develop a method for smultaneous clusterng and determnaton of number of clusters durng teraton n the neural structure. We let the output sgnals of neurons represent the degree to whch a peces of evdence belong to a correspondng cluster. From these we derve a probablty dstrbuton regardng the number of clusters, whch gradually durng the teraton s transformed nto a determnaton of number of clusters. Ths gradual determnaton s fed back nto the neural structure at each teraton to nfluence the clusterng process.. INTRODUCTION In ths paper we develop a neural network structure for smultaneous clusterng of evdence wthn Dempster- Shafer theory [] and gradual determnaton of number of clusters. The clusterng s done by mnmzng a metaconflct functon. The studed problem concerns the stuaton when we are reasonng wth multple events whch should be handled ndependently. We use the clusterng process to separate the evdence nto clusters that wll be handled separately. In an earler paper [] we developed a method based on clusterng wth a neural network structure nto a fxed number of clusters. We used the structure of the neural network, but no learnng was done to set the weghts of the network. Instead, all the weghts were set drectly by a method usng the conflct n Dempster s rule as nput. Ths clusterng approach was a great mprovement on computatonal complexty compared to a prevous method based on teratve optmzaton [ 7], although ts clusterng performance was not equally good. In order to mprove on clusterng performance a hybrd of the two methods has also been developed [8]. Here, the dea of gradual determnaton of number of clusters s developed and ntegrated wth the neural structure to run smultaneously wth the clusterng process. In [] a methodology was developed for fndng a posteror probablty dstrbuton concerned wth the number of clusters. Ths was based on the fnal clusterng result of an teratve optmzaton of the metaconflct functon and partal specfcatons of nonspecfc peces of evdence, uncertan wth respect to whch events they were referrng to []. Here, ths approach s expanded to nclude a gradual determnaton of number of clusters. Instead of usng the fnal clusterng result and partal specfcatons, we use ncremental clusterng states durng the teraton n the neural network, where we let an output sgnal of a neuron represent the degree to whch a pece of evdence belongs to the correspondng cluster. Ths yelds a probablty dstrbuton for the number of clusters. By usng a change n entropy of the output sgnals n the neural structure durng teraton, we can gradually transform the probablty dstrbuton nto a determnaton of number of clusters. The result of ths gradual determnaton s fed back nto the neural network durng the teraton to nfluence the clusterng process. In the early stages of the teraton the number of clusters vary slowly wth the change n the gradual determnaton of number of clusters and any changes n conflct n the clusters. In the latter stages, the teraton converges on a fxed number of clusters. The dea to use a neural network for optmzaton was nspred by an approxmate soluton to the travelng salesman problem by Hopfeld and Tank [9]. The clusterng methodology developed over several papers was ntally ntended for preprocessng of ntellgence nformaton for stuaton analyss n antsubmarne warfare [ ]. In secton we descrbe the problem at hand and n secton we contnue wth the neural structure for smultaneous clusterng and gradual determnaton of number of clusters. Frst, we descrbe the approach to clusterng wth a neural structure. Secondly, we focus on the gradual determnaton of number of clusters and how ths s ntegrated nto the neural structure to run smultaneously wth the clusterng process. The computatonal and clusterng performance s nvestgated n secton. Fnally, n secton, we draw conclusons.. THE PROBLEM If we receve several peces of evdence about dfferent and separate events and the peces of evdence are mxed up, we want to arrange them accordng to whch event they are referrng to. Thus, we partton the set of all peces of evdence χ nto subsets where each subset refers to a n Proc. 999 Informaton, Decson and Control (IDC 99), pp., Adelade, February 999, IEEE, Pscataway, NJ, 999.

2 Metalevel Metaconflct c c c c c multple events. The metaconflct s derved as the plausblty that the parttonng s correct when the conflct n each subset s vewed as a pece of metalevel evdence aganst the parttonng of the set of evdence, χ, nto the subsets, χ. DEFINITION. Let the metaconflct functon, e 9 e e e e e e 8 e χ χ Partton e e e e 7 e χ χ Four subsets OK? Fg. The conflct n each subset of the partton becomes a pece of evdence at the metalevel. partcular event. In fgure these subsets are denoted by χ and the conflct when all peces of evdence n χ are combned by Dempster s rule s denoted by c. Here, thrteen peces of evdence are parttoned nto four subsets. When the number of subsets s uncertan there wll also be a doman conflct c whch s a conflct between the current hypothess about the number of subsets and our pror belef. The partton s then smply an allocaton of all peces of evdence to the dfferent events. Snce these events do not have anythng to do wth each other, we wll analyze them separately. Now, f t s uncertan to whch event some peces of evdence s referrng we have a problem. It could then be mpossble to know drectly f two dfferent peces of evdence are referrng to the same event. We do not know f we should put them nto the same subset or not. Ths problem s then a problem of organzaton. Evdence from dfferent events that we want to analyze are unfortunately mxed up and we are facng a problem n separatng them. To solve ths problem, we can use the conflct n Dempster s rule when all peces of evdence wthn a subset are combned, as an ndcaton of whether these peces of evdence belong together. The hgher ths conflct s, the less credble that they do. Let us create an addtonal pece of evdence for each subset wth the proposton that ths s not an adequate partton. We have a smple frame of dscernment on the metalevel Θ = {AdP, AdP}, where AdP s short for adequate partton. Let the proposton take a value equal to the conflct of the combnaton wthn the subset, m χ ( AdP) = Conf( { e j e j χ }). These new peces of evdence, one regardng each subset, reason about the partton of the orgnal evdence. Just so we do not confuse them wth the orgnal evdence, let us call ths evdence metalevel evdence and let us say that ts combnaton and the analyss of that combnaton take place on the metalevel, fgure. We establsh [] a crteron functon of overall conflct called the metaconflct functon for reasonng wth r Mcf( r, e, e,, e n ) = ( c ) ( c ), = be the conflct aganst a parttonng of n peces of evdence of the set χ nto r dsjont subsets χ. Here, c s the conflct n subset and c s the conflct between r subsets and propostons about possble dfferent number of subsets. We wll use the mnmzng of the metaconflct functon as the method of parttonng the evdence nto subsets representng the events. Ths method wll also handle the stuaton when the number of events are uncertan.. NEURAL STRUCTURE We wll study a test problem where peces of evdence, all smple support functons wth elements from Θ, are clustered nto an unknown number of clusters, where Θ = {,,,..., }. Snce the elements are all the dfferent subsets of the frame there s always a partton nto fve clusters wth a global mnmum to the metaconflct functon equal to zero. If you put all evdence wth the element nto χ, of the remanng elements all those wth the element nto χ, etc., you get zero conflct n every cluster. The reason we choose a problem where the mnmum metaconflct s zero s that t makes a good test example for evaluatng performance. When mnmzng the metaconflct functon usng a neural structure we wll choose an archtecture that mnmzes a sum. Thus, we have to make some change to the functon that we want to mnmze. If we take the logarthm of one mnus the metaconflct functon, we can change from mnmzng Mcf to mnmzng a sum. Let us change the mnmzaton as follows mn Mcf = mn ( c ) max Mcf = max ( c ) max log( Mcf ) = max log ( c ) = max log( c ) = mn log( c ) where log( c ) [, ] s a weght. Snce the mnmum of Mcf s obtaned when the fnal sum s mnmal, the mnmzaton of the fnal sum yelds the same result as a mnmzaton of Mcf would have. Thus, n the neural network we wll not let the weghts be drectly dependent on the conflcts between dfferent peces of evdence but rather on log( c jk ), where c jk s the conflct between the jth and kth pece of evdence;

3 n n n Cluster st nd rd th th th n st nd rd th th Evdence... n 7 Fg. Neural network. Each column corresponds to a cluster and each row corresponds to a pece of evdence. st st th 9th th 7th st teraton teraton teraton teraton teraton teraton th teraton 9th teraton m j m k, conflct c jk =, no conflct. Ths s a slght smplfcaton snce the neural structure wll now mnmze a sum of log( c jk ). Let us study the calculatons takng place n the neural network durng an teraton. We use the same termnology as Hopfeld and Tank [] wth nput voltages as the weghted sum of nput sgnals to a neuron, output voltages as the output sgnal from a neuron, and nhbton terms as negatve weghts. For each neuron n mn we calculate an nput voltage u as the weghted sum of all sgnals from row m and column n and from a doman term, fgure. Ths sum s the prevous nput voltage for the prevous teraton of n mn plus a gan factor η tmes a sum of fve terms. The frst term s the sum of output voltages V n of all neurons of the same column as n mn, weghted by a data-term nhbton dt tmes the weght of conflct log( c n ) plus a global nhbton g. The second term s the sum of output voltages V mj of all neurons of the same row as n mn, weghted by a row-nhbton r and the global nhbton. The thrd term s the gradual determnaton of number of clusters. For a neuron n mn n column r we have the gradual determnaton factor gd t ( χ = r) weghted by a doman-nhbton Dt plus the global nhbton. The two last terms are an exctaton bas eb mnus the prevous nput voltage of n mn. Thus, the new nput voltage to n mn at teraton t + s u t mn + = u t mn + η [ dt log( c n ) + g] V n + [ r + g] V mj + [ Dt + g] gd t ( χ = n) + eb u t mn j n where gd t ( χ = n) [, ], see secton.. We have used the followng parameter settngs: η =, dt =, r =, Dt =, g =, and the exctaton bas eb = 8. From the new nput voltage to n mn we can calculate a new output voltage of n mn V t mn + -- tanh ut mn + = + ( ) u where tanh s the hyperbolc tangent, u =., and V t mn + [,]. Intally, before the teraton begns, each neuron s rd 7th st th 9th rd teraton teraton teraton teraton teraton teraton ntated wth an nput voltage of u + nose where u = u atanh n -- 7th teraton st teraton Fg. Sxteen dfferent states (teratons) of a neural network wth 8 neurons. From left to rght: The convergence of clusterng peces of evdence nto an unknown number of clusters at every fourth teraton. In each snapshot of an teraton each of the sx columns represent one possble cluster and each of the rows represent one pece of evdence. The lnear dmenson of each square s proportonal to the output voltage of the neuron and represent the degree to whch a pece of evdence belongs to a cluster. In the fnal state each row has one output voltage of. and fve output voltages of.. A pece of evdence, represented by a row, s now clustered nto the cluster where the output voltage s.. and atanh s the hyperbolc arc tangent. The ntal nput voltage s set at u + δu where δu, the nose, s a random number chosen unformly n the nterval. u δu. u. In each teraton all new voltages are calculated from the results of the prevous teraton. Ths contnues untl convergence s reached. As long as the weghts of the neural network s symmetrc convergence s always guaranteed. In fgure the convergence of a neural network wth rows and sx columns for clusterng peces of evdence nto a unknown number of clusters s shown. After convergence s acheved the conflct wthn each cluster s calculated by combnng those peces of evdence for whch the output voltage for the column s.. We now have a conflct for each subset and can calculate the overall metaconflct, Mcf, by the prevous formula.

4 .8... Fg. m χ ( χ χ) for all χ over teratons.. Gradual determnaton of number of clusters Makng a gradual determnaton of the number of clusters s a process wth several steps. Frst, we use the dea that each pece of evdence n a subset supports the exstence of that subset to the degree that that pece of evdence supports anythng at all []. Durng an teraton t s uncertan to whch cluster a peces of evdence belongs. Therefore, we wll use every pece of evdence n each cluster but weghted by ts output voltage for the cluster. All these weghted peces of evdence n each cluster χ are then combned. The degree to whch the result from ths combnaton supports anythng at all other than the entre frame s the degree to whch these peces of evdence taken together supports the exstence of χ. Thus, we have t + t + m χ ( χ χ) = k ( V mn + V mn m m ( Θ) ), m t + t + m χ ( Θ) = k ( V mn + V mn m m ( Θ) ) m where k s the conflct n Dempster s rule of the combnaton. Should nothng be supported by the evdence n some cluster that evdence s meanngless and can be thrown away and ths partcular cluster s not needed. In fgure m χ ( χ χ) s calculated for all sx possble clusters over teratons. Ths s the same problem as n fgure. We notce how the basc probablty of the th clusters drops off very fast durng the frst few teratons and remans low for the remander of the teratons. Also the basc probablty of the th cluster drops off ntally but comes back agan durng the teraton process to fnsh hgh. These sx peces of evdence m χ, one regardng each subset, are then combned. We have m χ (( χ ) χ) = m χ ( χ χ) m ( χ Θ ), j ( χ χ ) j χ j χ n m χ ( Θ) = m χ ( Θ). = where χ χ and χ = { χ, χ, χ, }. From ths we can create a new type of evdence by exchangng all propostons n the prevous ones that are conjunctons of r terms, χ, for one proposton n the new type of evdence that s on the form χ r. Here, r. The sum of probablty of all conjunctons of length r n the prevous peces of evdence s then awarded to the focal element n ths new pece of evdence whch supports the proposton that χ r. We get,.8... s the conflct n that fnal combnaton. Thus, by vewng each pece of evdence n a subset as support for the exstence of that subset we are able to derve a posteror probablty dstrbuton concerned wth the queston of how many subsets there are. Ths dstrbuton s plotted n fgure. It s nterestng to observe that the basc probabltes for propostons such as m χ ( χ r) has now been moved upwards when we derve the probabltes m ( χ = r). In fgure 7 we see the same plot projected on the teraton probablty axes. We notce the ntal drop off n the alternatves wth fve and especally wth sx clusters, as well as the early rse of the three and four cluster alternatves. In the frst few teratons the sx cluster alternatve s stll the preferable choce, but very quckly the preferable choce becomes fve and then four clusters. After tryng to cluster the evdence nto four Fg. m χ ( χ r) for all χ over teratons. m χ ( χ r) = m χ (( χ ) χ), χ * χ * = r m χ ( Θ) = m χ ( Θ). Taken as a whole ths gves us an opnon about the probablty of dfferent numbers of subsets. In fgure we see the basc probablty m χ ( χ r) for dfferent mnmal number of clusters over the entre teraton. Note that we are no longer talkng about ndvdual clusters but of the derved support regardng the actual number of clusters. These newly created peces of evdence can now be combned wth a pror probablty dstrbuton, m ()., from the problem specfcaton. Ths s a doman dependent dstrbuton. Wthout a pror probablty dstrbuton there s nothng to hold the clusterng together and we could fnd as many clusters as peces of evdence f that was allowed by the neural structure. In these trals we have chosen a dstrbuton where m( χ = r) = p ( r ), where p s a constant. We get m ( χ = r) = m( χ = r) m k χ ( Θ) + m χ ( χ j) j = where n n k = m( χ = r) m χ ( χ j).8... = j = + Fg. m ( χ = r) for all χ over teratons.

5 ... χ = χ = χ = χ = m ( χ = r) Fg 7.. clusters the nternal conflcts become too large and t s necessary to reclam the ffth cluster. Ths s observed here n the dramatc shft around the teraton. After that shft, the fve cluster alternatve remans preferable throughout the remander of the teraton. Sooner or later t becomes necessary to determne how many clusters there are. The bass for makng such a determnaton s the probablty dstrbuton n fgure 7. The obvous choce s that of the hghest probablty, and that s how we wll decde n the fnal teraton, but we must avod makng an early determnaton. As seen n fgure 7, an early choce based on maxmum probablty s lkely to be false. However, nether can we wat untl the fnal teraton to determne the number of clusters. We must gve the neural teraton a good opportunty to converge on the rght problem. The dea s to accept the probablty dstrbuton n fgure 7 but make a gradual determnaton of number of clusters throughout the teratve process that s not fnal untl the last teraton. We do ths by measurng how far we have traveled from the ntal state untl the fnal state when convergence s reached. Intally all output voltages of the neurons are scattered maxmally among the dfferent choces whle n the fnal state there s no scatterng at all. A normalzed Shannon entropy s thus a good measure of how far we have traveled towards the fnal state, the entropy beng zero at the fnal state. The entropy at teraton t s calculated over all peces of evdence and all clusters as Entropy t = Vmn t logvmn t m n where the normalzed entropy α t s Entropy t dvded by Entropy. The normalzed entropy s plotted n fgure 8 as a sold lne, t goes from to n teratons. The gradual determnaton s made by substtutng the probablty dstrbuton n fgures and 7 wth gd t ( χ = r) = α t + α t m ( χ = r), ff s. m ( χ = r) m ( χ = s), gd t ( χ = r) = α t m ( χ = s), otherwse In fgure 9 the gradual determnaton of the posteror probablty dstrbuton n fgures s plotted. In ts frst teraton they are dentcal, gradually the determnaton takes place and n the fnal teraton an exact determnaton of fve clusters s made. From fgure 9 we observe that even though the exact determnaton of fve clusters does not take place untl the fnal teraton, the stuaton s pretty clear long before. At the nd and th teratons gd ( χ = ) =.9 and gd ( χ = ) =.9, respectvely, gvng the clusterng process ample tme to converge on the rght number of clusters. Also, n fgure we notced a good clusterng takng place n the ffth cluster from the 7th teraton, gvng the process some addtonal teratons to converge.. PERFORMANCE In ths secton we wll compare the clusterng performance and computatonal complexty of the smultaneous clusterng and gradual determnaton of numbers of clusters presented n ths paper wth clusterng nto a known number of fve clusters whch was done n []. The comparson s done usng the prevous descrbed peces of evdence wth dfferent random basc probablty assgnments over ten runs. In fgure 8 the metaconflct over teratons of one of the runs resultng n fve clusters s shown, dashed lne. In fgure the conflct per cluster s shown. We notce the tendency that the conflct drops off n one cluster at a tme. Quckly the conflct drops off n cluster sx and then n fve. Intally, there s a very hgh conflct n cluster two but t also, drops off wthn the frst ten teratons. The remander of the teraton concerns manly clusters one, three and four. Cluster three holds out untl the mddle of the teraton but towards the end most of the remanng conflct s stuated n cluster one. In table we fnd a somewhat hgher computaton tme for the new approach, as would be expected, snce we have two dfferent convergng processes runnng smultaneously. Table : Computaton tme and teratons (mean). # Clusters unknown Neural structure tme (s) teratons Fg 9. The gradual determnaton of number of clusters..... Fg 8. Normalzed entropy α t, sold lne, and metaconflct, dashed lne, over teratons. Fg. Conflct per cluster durng the teraton.

6 If we consder that the clusterng nto a known number of clusters has to be performed for many possble dfferent numbers of clusters one after the other, ths can be a sgnfcant savng of tme. In the ten runs of clusterng peces of evdence nto an unknown number of clusters four of these runs ended up usng fve clusters, fve runs used sx clusters and one run used only four clusters. The use of sx clusters n some of the runs s the result of a below average clusterng performance. If the number of clusters had been fxed to fve, we would have seen a hgher than average conflct n these runs. Here, ths s nterpreted as a need for an addtonal cluster. Snce the propostons of the evdence are the dfferent subsets of the frame Θ = {,,,..., }, t wll never be possble to fnd a conflct free parttonng of the evdence nto four clusters. The reason why such a partton occurred once, s that the basc probablty assgnments attached to the propostons are random number between and. If two propostons are n conflct and ther assgnments are close to zero the numerc conflct s stll a small number and mght be found acceptable compared to the use of an addtonal cluster. If all assgnments had been. we could never have had four clusters n ths test. The clusterng performance s somewhat dffcult to measure because of the dfferent number of subsets. A poor clusterng mght yeld more clusters than a good one wth a lower conflct per cluster. For ths reason we wll compare the performance of those runs that resulted n fve clusters when clusterng nto a unknown number of clusters wth clusterng nto a known fve clusters. We have four runs resultng n fve clusters. They are consdered the four best runs out of ten. We compare them to the four best runs out of ten when clusterng nto a known fve clusters. We also compare the four runs wth the same set of random assgnments when the number of clusters s known to be fve, table. Table : Metaconflct. # Clusters unknown best runs of best of. mean of.8. runs wth same assgnments best of. mean of.9 As expected, the conflct of the new approach s hgher, but the actual numbers are qute small. The best crtera of a good clusterng performance s the conflct per cluster and pece of evdence. These are tabulated n table. For example, we fnd a mean conflct per cluster and pece of evdence of only n the case of clusterng nto an unknown number of clusters. Ths can be compared wth the average numerc conflct of % between two conflctng peces of evdence. Table : Mean metaconflct per cluster and evdence. # Clusters unknown best of runs / cluster.. / evdence.. runs wth same assgnments / cluster. / evdence.. CONCLUSIONS We have demonstrated that t s possble to use a neural network structure to perform smultaneous clusterng of Dempster-Shafer evdence wth a gradual determnaton of the number of clusters when ths s unknown. We found the computatonal and clusterng performance to be almost as good as when clusterng nto a fxed number of clusters. Ths approach s advantageous, snce now only one clusterng process has to be performed. REFERENCES [] G. Shafer, A Mathematcal Theory of Evdence, Prnceton Unversty Press, Prnceton, 97. [] J. Schubert, Fast Dempster-Shafer Clusterng Usng a Neural Network Structure, n Proc. Seventh Int. Conf. Informaton Processng and Management of Uncertanty n Knowledge-Based Systems (IPMU 98), pp. 8, Unversté de La Sorbonne, Pars, France, July 998, Edtons EDK, Pars, 998. [] J. Schubert, On Nonspecfc Evdence, Int. J. Intell. Syst., Vol 8, pp [] J. Schubert, Specfyng Nonspecfc Evdence, Int. J. Intell. Syst., Vol, pp.. [] J. Schubert, Fndng a Posteror Doman Probablty Dstrbuton by Specfyng Nonspecfc Evdence, Int. J. Uncertanty, Fuzzness and Knowledge- Based Syst., Vol, pp. 8. [] J. Schubert, Cluster-based Specfcaton Technques n Dempster-Shafer Theory, n Symbolc and Quanttatve Approaches to Reasonng and Uncertanty, Proc. European Conf. (ECSQARU 9), pp. 9, Unversty of Frbourg, Swtzerland, July 99, Sprnger-Verlag (LNAI 9), Berln, 99. [7] J. Schubert, Creatng Prototypes for Fast Classfcaton n Dempster-Shafer Clusterng, n Qualtatve and Quanttatve Practcal Reasonng, Proc. Frst Int. Jont Conf. (ECSQARU-FAPR 97), pp., Bad Honnef, Germany, 9 June 997, Sprnger-Verlag (LNAI ), Berln, 997. [8] J. Schubert, A Neural Network and Iteratve Optmzaton Hybrd for Dempster-Shafer Clusterng, n Proc. EuroFuson98 Int. Conf. Data Fuson, pp. 9, Great Malvern, UK, 7 October 998. [9] J.J. Hopfeld, and D.W. Tank, Neural Computaton of Decsons n Optmzaton Problems, Bol. Cybern., Vol, pp.. [] U. Bergsten, J. Schubert, and P. Svensson, Applyng Data Mnng and Machne Learnng Technques to Submarne Intellgence Analyss, n Proc. Thrd Int. Conf. on Knowledge Dscovery and Data Mnng (KDD 97), pp. 7, Newport Beach, USA, 7 August 997, The AAAI Press, Menlo Park, 997. [] J. Schubert, Cluster-based Specfcaton Technques n Dempster-Shafer Theory for an Evdental Intellgence Analyss of Multple Target Tracks, Ph.D. Thess, TRITA NA 9, Royal Insttute of Technology, Sweden, 99, ISRN KTH/NA/R 9/ SE, ISSN 8 9, ISBN [] J. Schubert, Cluster-based Specfcaton Technques n Dempster-Shafer Theory for an Evdental Intellgence Analyss of Multple Target Tracks (Thess Abstract), AI Communcatons, Vol 8, pp. 7.