An Algorithm for Pruning Redundant Modules in Min-Max Modular Network

Transcription

1 An Agorithm for Pruning Redundant Modues in Min-Max Moduar Network Hui-Cheng Lian and Bao-Liang Lu Department of Computer Science and Engineering, Shanghai Jiao Tong University 1954 Hua Shan Rd., Shanghai , China Emai: Abstract The min-max moduar (M 3 ) network is a framework that is capabe of soving arge-scae pattern cassification probems in a parae way. The M 3 network has been successfuy appied to severa arge-scae rea-word prbems. When a compex probem is decomposed into a number of separabe probems, however, the M 3 network sufferes from its high redundancy of individua modues. This paper proposes an agorithm, caed back-searching (BS) agorithm, to prune these redundant modues. The main idea behind the BS agorithm is to use the actua outputs of the trained M 3 network associated with training data to find out the redundant modues by means of back searching. In order to ensure the correctness of the agorithm, we prove two propositions theoreticay, namey the sufficient proposition and the necessary proposition, and perform simuations on severa benchmark and rea-word probems. The simuation resuts indicate that most of a redundant modues can be pruned by our proposed agorithm and the pruned network has the same generaization performance as the origina network. I. INTRODUCTION The min-max moduar (M 3 ) network [1][2] is a framework that is capabe of soving arge-scae pattern cassification probems in parae. In comparison with other moduar neura network modes and machine earning approaches, the M 3 network has the foowing two important advantages. a) The task decomposition scheme is more genera than one-vs-one scheme, which wi be referred to as the part-vs-part or PVP scheme throughout this paper. By using the PVP scheme, any user can decompose a compex muticass probem into many two-cass subprobems as sam as needed. Neither domain speciaists nor prior knowedge of the probem is required. Since each of the two-cass subprobems can be treated as a competey separabe cassification probem in the earning phase, a of the two-cass subprobems can be earned in a parae way. b) After earning each of the two-cass subprobems with a network modue, a trained network modues can be integreted into a M 3 network automaticay according to two modue combination rues, namey the minimization pricinpe and the maximization principe [2]. In the ast few years, the M 3 network has been successfuy appied to many arge-scae rea-word probems such as partof-speech tagging [5], singe-tria EEG signa cassification [6], and text categorization [8]. In M 3 network, a K-cass cassification probem is decomposed into a series of K(K 1)/2 two-cass probems. These two-cass probems are to discriminate cass C i from C j for i = 1,..., K 1 and j = i + 1,..., K, whie the existence of the training data beonging to the other K-2 casses is ignored. If these two-cass probems are sti hard to be earned, they can be divided into a set of two-cass subprobems as sma as needed. Consequenty, a arge-scae and compex K-cass cassification probem can be soved effortessy and efficienty by earning a series of smaer and simper two-cass probems in a parae way. In order to make the framework more cear, we show the M 3 network in Fig. 1. This M 3 network incude three parts: individua network modues, MIN units and MAX units (see Fig. 1). Here, the basic function of an MIN unit is to find a minimum vaue from its mutipe inputs and the basic function of an MAX unit is to find a maximum vaue from its mutipe inputs, and the individua network modue can be any cassifier such as MLP and SVM. According to the PVP scheme, a compex muticass probem can be decomposed into many ineary separabe probems, each of which has ony two different training sampes. The merit of this task decomposition is that the earning convergence can be guaranteed by using inear discriminant function or GZC discriminant function, and incrementa earning can be easiy impemented. However, the demerit is that there are a arge number of redundant modues in the M 3 network. So how to prune these redundant modues is one of the most important issues for further appying M 3 network to soving arge-scae rea-word probems. In this paper, we propose a pruning agorithm caed back searching (BS) agorithm to reduce the redundancy of the M 3 network. The motivation of the BS agorithm comes from the fact that the actua output of a redundant modue must have the same vaue as the output of the MIN unit when a training sampe is presented to the network as an input. Therefore, by using a back-searching technic, the redundant modues can be found out. In the next section we firsty define the redundant probem of the M 3 network. Then we describe our BS agorithm in section III. At ast, we present the experimenta resuts and concusions in section IV and section V, respectivey. II. REDUNDANT PROBLEM OF THE M 3 NETWORK A. Task Decomposition Let T be the training set of a K-cass cassification probem and the K casses are represented by C 1, C 2,..., C K, respectivey.

2 T = {(X, Y )} L =1 (1) where X R d is the input vector, Y R K is the desired output, and L is the number of training data. Suppose the K training input sets, X 1, X 2,...,X K, are expressed as { X i = (X (i) } Li =1 for i = 1,..., K (2) where L i is the number of training inputs in cass C i, X (i) is the th sampe beonging to cass C i and a of X (i) X i have the same desired outputs, and K i=1 L i = L. It is known that a K-cass probem defined by (1) can be divided into K(K 1)/2 two-cass subprobems, each of which is given by { } T ij = (X (i) Li { }, +1) (X (j) Lj, 1) =1 =1 for i = 1,..., K 1 and j = i + 1,..., K (3) Even though a K-cass probem is broken into K(K 1)/2 reativey smaer two-cass probems, some of the two-cass probem may be sti hard to be earned. We have suggested that T ij defined by (3) can be further decomposed into a number of two-cass subprobems as sma as needed according to the cass reations among training data [2]. Assume that the input set X i defined by (2) is further partitioned into N i (1 N i L i ) subsets in the form of { X ij = (X (ij) } L (j) i =1 for j = 1,..., N i (4) where L (j) i is the number of training inputs incuded in X ij, and Ni j=1 X ij = X i After dividing the training input set X i into N i subsets X ij (4), the training set for each of the smaer and simper twocass probems can be given by T (u,v) ij = where X (iu) { } (u) L { } (v) (X (iu) i L, +1) (X (jv) j, 1) =1 =1 for u = 1,..., N i, v = 1,..., N j i = 1,..., K 1 and j = i + 1,..., K (5) X iu and X (jv) X jv are the input vectors beonging to cass C i and cass C j, respectivey, N i u=1 L(u) i = L i, and N j v=1 L(v) j = L j. From (5), we see that if N i = L i, i.e., each of the twocass subproems contains ony two different training data. Obviousy, these two-cass subprobems are ineary separabe probems. B. Modue combination From (4) and (5), we can see that a K-cass probem is divided into K 1 K N i N j (6) two-cass subprobems. i=1 j=i+1 If N i = L i, a K-cass cassification probem can be divided into many ineary separabe subprobems. The number of these ineary separabe subprobems can be expressed as K 1 K i=1 j=i+1 L i L j (7) Obviousy, equation (7) shows an upper bound on the number of subprobems that can be obtained by dividing a K-cass cassification probem into many ineary separabe subprobems. After constructing each individua inear discriminant function as a base cassifier, a the individua modues are integrated into a min-max moduar network with MIN, MAX, or/and INV units according to two modue combination rues [2]. This kind of M 3 network wi be referred to as inear- M 3 network throughout this paper. A inear-m 3 network is iustrated in Fig. 1. x 1,1 M i, j 1,2 M i, j 1,Lj i,j Li,1 M i,j Li,2 M i,j Li,Lj M i,j MIN 1 MIN Li MAX Fig. 1. The M 3 network consists of L i L j individua network modues, L i MIN units, and one MAX unit. C. Reduntant probem of M 3 network Athough we can decompose a compex probem into a series of independent inear separabe probems, and then combine them to get the soutions to the origina probem, there are a arge number of redundant modues in the network when finay integrating a of the trained modues into a M 3 network. For exampe, see Fig. 2 (a), there are tota six ines and each ine stands for a inear discriminant function that has been generated by one positive training sampe and one

3 negative training sampe. We can see that, in this figure, the boundary of the two different areas (gray and white) is decided ony by four of the six ines, whie the other two ines are of no use and they are so caed redundant modues in the M 3 network. This redundant probem may be a serious probem when the number of ineary separabe probems becomes very arge, especiay for rea-word probems. In this paper, we focus on the probem of how to prune the redundant modues in the inear-m 3 network. D. Terminoogy definition Suppose X is the input space of a M 3 network, U i is the ith individua network modue of a MIN unit, {MIN/U i } is the residua part of the MIN unit after pruning U i from this MIN unit, and T is the training set of the origina MIN unit. For the purpose of convenience, we present the definitions of severa terminoogies as foows. Definition 1 : For any x X, if U i (x) {MIN/U i }(x), then we ca U i a redundant modue of the MIN unit. Definition 2 : If there exists an x X, that U i (x) < {MIN/U i }(x), then we ca U i a non-redundant modue of the MIN unit. Definition 3 : For any x T, if U i (x) {MIN/U i }(x), then we ca U i an aowed redundant modue of the MIN unit. Definition 4 : The ratio between the number of the pruned modues and the number of the tota origina modues is defined as the degree of pruning: d p = number of pruned modues number of origina modues 100% Here we have made the definition of the aowed redundant modue from the view of training set, but not from the view of input space, considering that it is aowed for this kind of modues in the phase of training a cassifier. Furthermore, since the MIN and MAX units are simiar to the ogica AND and OR gates respectivey [3], a the redundant modues of each MIN unit wi compose of the redundant modues of one MAX modue, i.e. the redundant modues of the M 3 network. III. BACK-SEARCHING PRUNING ALGORITHM In this section we describe our back-searching pruning agorithm for inear-m 3 network. The reason why inear-m 3 network was chosen as the object is that the inear-m 3 network is the reativey simpe and basic network among our various kinds of M 3 networks. We can further modify the agorithm presented here for other kind of M 3 networks in a simiar way. We ony focus on pruning redundant modues of inear- M 3 network throughout this paper. A. Agorithm In the BS pruning agorithm, we assume that a of the individua modues in a MIN unit are redundant units firsty, so we tag them as fase, then by using a back-searching method, we find out a the non-redundant modues in the MIN unit, and at ast we deete a the residua modues that are not found out in the end of the agorithm. For a trained inear- M 3 network, we use the foowing BS agorithm to prune its redundant modues. 1) Suppose a inear-m 3 network is M 3 and has L i and L j training sampes of cass C i and cass C j respectivey. According to two modue combination rues, M 3 wi incude L i MIN units and one MAX unit, and each MIN units incude L j individua network modues. 2) For each unit MIN k (k = 1, 2,..., L i ) do the foowing steps: a) Tag fase to each individua network modue U r (r = 1, 2,..., L j ) of unit MIN k. b) Suppose the training set of unit MIN k is T k = {a k, b 1, b 2,..., b Lj }, here a k C i and b 1, b 2,..., b Lj C j. c) For each training sampe x T k of unit MIN k, do the foowing steps: i) Cacuate the output vaue of unit MIN k, y, ii) Search back for a individua network modues whose outputs have the same vaue with y, iii) If U r (x) < {MIN k /U r }(x), then U r modue be tagged as true. d) Deete a individua network modues that are marked with fase. 3) End. B. Propositions In this subsection, we prove the sufficient and necessary propositions for the BS agorithm theoreticay. Proposition 1: (A sufficient proposition) A the redundant modues of a trained M 3 network are pruned by the BS agorithm. Proof: Suppose U r is one of the redundant modues of M 3, T is the training set of U r, and X is the input space of M 3, then we have T X. From the definition of redundant modues (see Definition 1) and T X, we have U r (x) {MIN/U r }(x) for any training sampe x T, so after running the BS agorithm, U r wi be tagged as fase and then be pruned. Proposition 2: (A necessary proposition) A modues that have been pruned by BS agorithm must be aowed redundant modues. Proof: Suppose U r is one of the individua modues that have been pruned by the BS agorithm, and T is the training set of U r. From the BS agorithm, we can concude that for any training sampe x T, there is U r (x) {MIN/U r }(x), so from the definition of aowed redundant modue (see Definition 3), we can see that U r is an aowed redundant modue. Suppose a M 3 network has N MIN units and each MIN unit has M training sampes, then the time compexity of the BS agorithm is O(NM 2 ). For a arge scae probem, this pruning agorithm may be a time consumed process.

4 IV. EXPERIMENTS To evauate the effectiveness of the BS pruning agorithm, we perform six experiments on both benchamark [3] and reaword probems [7]. A the experiments were performed on a 2.8 GHz P4 PC/Win2000. (a) (c) Fig. 2. Iustrations of decision boundaries formed by origina inear-m 3 network and pruned inear-m 3 network. (a) The decision boundary formed by a MIN unit and it s individua modues (ines) before pruning; (b) The decision boundary formed by a MIN unit and its individua modues (ines) after pruning; (c) The decison boundary formed by a M 3 network and it s individua modues (ines) before pruning; (d) The decision boundary formed by a M 3 network and it s individua modues (ines) after pruning. Here symbo means a positive data from cass C i, and symbo * means a negative data from cass C j ). Fortunatey, the whoe pruning process can be done before empoying the pruned network to rea appications. We show some experimenta resuts about time of pruning in Tabe I. C. Iustration To iustrate the BS pruning agorithm, we present a simpe exampe depicted in Fig. 2. In this figure, symbo means a positive sampe from cass C i, symbo * means a negative sampe from cass C j, and each ine means a inear discriminant function as we as an individua modue of the M 3 network. Each of the individua modues is determined by one positive sampe from cass C i and one negative sampe frome the cass C j. From Fig. 2(a), we see that two redundant modues in a MIN unit are pruned by the BS agorithm and the pruning resut is depicted in Fig. 2 (b). The boundaries formed by the origina M 3 network and the pruned M 3 network are showed in Figs. 2 (c) and (d), respectivey. Here 9 redundant modues have been pruned, and the degree of pruning is 37.5%. From Figs. 2 (c) and (d), we can see that the decision boundaries (dark area) before and after pruning are identica. (b) (d) A. The Two-spiras Probem In the first experiment, we evauate the BS pruning agorithm on a toy two-spiras benchmark probem. The data incude 194 training sampes and 1,700 test sampes, respectivey. Fig. 3 (a) shows the origina two-spiras probem. The bue symbos stand for positive training data and the white * symbos stand for negative training data. We use these data to traine a inear-m 3 network and the decision boundary formed by the trained inear-m 3 network is depicted in Fig. 3 (b). The origina network consists of 18,432 individua modues. By using the BS pruning agorithm, we prune 15,763 redundant modues from the trained network, and a 85% of degree of pruning is obtained. The decision boundary formed by the pruned network is depicted in Fig. 3 (c). From Figs. 3 (b) and (c), we can see that the decision boundaries before and after pruning are identica. This indicates that the pruned network maintains its generaization performance, whie the individua modues and response time are greaty reduced. There are ony 2,769 individua modues remained in the pruned network, and the response time is speeded up to 7.33 times in comparison with the origina network (see Tabe I). B. Iris Pants Database This data set contains three casses of 50 instances each, where each cass refers to a type of iris pant with four data attributes. One cass is ineary separabe from the other two; the atter are not ineary separabe from each other. The first 25 data are used as training data and the remaining data are used as test data in our experiment. There are 3,750 individua modues in the origina M 3 network, comparing to 614 individua modues in the pruned M 3 network, a 83% of degree of pruning is obtained and the response time is speeded up to 7.5 times in comparison with the origina network. (see Tabe I). C. Image Segmentation Data The image segmentation probem [3] is a rea-word probem. The instances were drawn randomy from a database of seven outdoor images. The images were hand-segmented to create a cassification for every pixe as one of brick-face, sky, foiage, cement, window, path, and grass. The probem consists of 210 training data and 2,100 test data. The number of attributes is 18 and the number of casses is seven. There are 37,800 individua modues in the origina M 3 network, comparing to 8,812 individua modues remained in the pruned M 3 network, a 76% of degree of pruning is obtained and the response time is speeded up to 4.60 times in comparison with the origina network (see Tabe I).

5 TABLE I PERFORMANCE COMPARISON OF THE ORIGINAL NETWORK AND THE PRUNED NETWORK ON BENCHMARK AND REAL-WORLD PROBLEMS. HERE TIME UNIT IS MS Name Before pruning After pruning Time Degree Speed up # Modue Response time Correct rate # Modue Response time Correct rate of Pruning of pruning two-spiras 18, % 2, % % 7.33 iris 3, % % 10 83% 7.50 image 37, % 8, % % 4.60 optdigits 19,691,545 1,637, % 16,679,524 14,57, % 8,765,985 15% 1.12 Fauty Image1 3,220 2, % 2,352 1, % 144,748 26% 1.82 Fauty Image2 48,000 42, % 20,060 22, % 5,228,344 58% 1.87 (b) (a) Fig. 3. Two-spiras probem and decision boundary comparison. (a) The training sampes of the two-spiras probem; (b) The decision boundary formed by the origina M 3 network; (c) The decision boundary formed by the pruned M 3 network. D. Optica Recognition of Handwritten Digits The optica recognition of handwritten digits probem [3] is a ten-cass cassification probem. The instances were drawn from a tota of 43 peope, 30 contributed to the training set and different 13 to the test set. The probem consists of 3,823 training data and 1,797 test data. The number of attributes is 64 and a the input attributes are integers in the range 0 to16. There are 19,691,545 individua modues in the origina M 3 network, comparing to the 16,679,524 individua modues remained in the pruned M 3 network, a 15% of degree of pruning is obtained and the response time of the pruned network is speeded up to 1.12 times in comparison with the origina network (see Tabe I). (c) E. Fauty image diagnosis We compare the performance of the origina M 3 network with that of the pruned network on an industria image database acquired from a product ine at a eading manufacturing company [7]. The object is to detect the faut image data from arge scae correct image database. We firsty extract the origina image into 64 64=4,096 dimension vectors using waveet transform, then train the origina M 3 network using the extracted training data and examine the performance of the trained origina M 3 network. We prune the redundant modues in the origina network and then do the same testing processes using the same data set. The number of training and test sampes in the first simuation are 70/23 and 26/23, and in the second simuation they are 400/60 and 100/23 (here a/b means a correct sampes and b faut sampes). In the first simuation, there are 3,220 individua modues in the origina M 3 network, comparing to 2,352 individua modues remained in the pruned M 3 network, a 26% of degree of pruning is obtained and the response time is speeded up to 1.82 times in comparison with the origina network (see Tabe I). In the second simuation, there are 48,000 individua modues in the origina M 3 network, comparing to 20,060 individua modues remained in the pruned M 3 network, a 58% of degree of pruning is obtained and the response time is speeded up to 1.87 times in comparison with the origina network (see Tabe I). The resuts from Tabe I show that the pruned M 3 networks have amost the same correct recognition rates as the origina M 3 networks, whie the highest degree of pruning is up to 85% and the test time is speeded up to 7.33 times. From the simuation resuts, we can see that the degree of pruning is not ony reative to the number of training data, but aso reative to the number of attributes and the number of casses in the probem. A probem with more training sampes, ess number of attributes and ess number of casses wi have more redundant individua modues in M 3 network, and consequenty the BS pruning resuts wi be more efficient than others on data sets. The consistence of the degrees of pruning and the response time shows the effectiveness of the BS agorithm.

6 V. CONCLUSIONS An pruning agorithm has been presented in this paper for pruning the redundant modues in the inear-m 3 network. The centra idea of this agorithm is to buid up a inear- M 3 network firsty, then use the training data to find out the redundant modues by using a back-searching technic. We have proved the necessary and sufficient propositions to ensure the correctness of the proposed pruning agorithm. The simuation resuts show that a highest degree of pruning 85% can be obtained and the response time is speeded up to 7.33 times at most, meanwhie the pruned network maintains the same generaization performance as that of the origina network. The simuation resuts indicate our pruning agorithm is efficient and effective. The pruned network s generaization abiity and the various versions of BS agorithm shoud be further expoited. ACKNOWLEDGMENT This work was supported in part by the Nationa Natura Science Foundation of China via the grants NSFC and NSFC REFERENCES [1] B. L. Lu and M. Ito, Task decomposition based on cass reations: a moduar neura network architecture for pattern cassification, In: Mira, J., Moreno-Diaz, R., Cabestany, J.(eds.), Bioogica and Artificia Computation: From Neuroscience to Technoogy, Lecture Notes in Computer Science, Springer Vo.1240 (1997) [2] B. L. Lu and M. Ito, Task Decomposition and Modue Combination Based on Cass Reations: A Moduar Neura Network for Pattern Cassification. IEEE Trans. Neura Networks, vo. 10, no. 5, pp , [3] C. J. Merz and P. M. Murphy, UCI Repository of mechine earning databases, Univ. Caifornia, Dept. Inform. Comput. Sci. Irvine, CA, Avaiabe mearn/mlrepository.htm. [4] B. L. Lu and M. Ichikawa, Emergent On-ine Learning in Min-Max Moduar Neura Networks, Proc. of IEEE/INNS Int. Conf. on Neura Networks, Washington DC, USA, pp , [5] B. L. Lu, Q. Ma, M. Ichikawa, and H. Isahara, Efficient part-of-speech tagging with a min-max moduar neura network, Appied Inteigence, vo. 19 pp , [6] B. L. Lu, J. Shin, and M. Ichikawa, Massivey parae cassification of singe-tria EEG signas using a min-max moduar neura network, IEEE Trans. Biomedica Engineering, vo. 51, no. 3, pp , [7] B. Huang and B. L. Lu, Faut diagnosis for industria images using a min-max moduar neura network, Lecture Notes in Computer Science, vo. 3316, pp , [8] K. Wu, F. Y. Liu, H. Zhao, and B. L. Lu, Fast text categorization with a min-max moduar support vector machine, to appear in Proc. of 2005 IEEE/INNS Int. Joint Conf. on Neura Networks, Montrea, Canada. [9] B. L. Lu and M. Ichikawa, Emergent on-ine earning with a Gaussian zero-crossing discriminant function, Proc. of IEEE/INNS Int. Joint Conf. on Neura Networks, Honouu, USA, pp , 2002