ANALYSIS OF DECISION BOUNDARIES IN LINEARLY COMBINED NEURAL CLASSIFIERS

Transcription

1 ANALYSIS OF DECISION BOUNDARIES IN LINEARLY COMBINED NEURAL CLASSIFIERS Kagan Tumer and Joydeep Ghoh Department of Electrical and Computer Engineering, Univerity of Texa, Autin, TX Abtract: Combining or integrating the output of everal pattern claifier ha led to improved performance in a multitude of application. Thi paper provide an analytical framework to quantify the improvement in claification reult due to combining. We how that combining network linearly in output pace reduce the variance of the actual deciion region boundarie around the optimum boundary. Thi reult i valid under the aumption that the a poteriori probability ditribution for each cla are locally monotonic around the Baye optimum boundary. In the abence of claifier bia, the error i hown to be proportional to the boundary variance, reulting in a imple expreion for error rate improvement. In the preence of bia, the error reduction, expreed in term of a bia reduction factor, i hown to be le than or equal to the reduction obtained in the abence of bia. The analyi preented here facilitate the undertanding of the relationhip among error rate, claifier boundary ditribution, and combining in output pace. Keyword: combining, deciion boundary, neural network, pattern claification, hybrid network, variance reduction.

2 1 Introduction Training a parametric claifier involve the ue of a training et of data with known claification to etimate or learn the parameter of the choen model. A tet et, coniting of pattern previouly uneen by the claifier, i then ued to determine the claification performance. Thi ability to meaningfully repond to novel pattern, or generalize, i an important apect of a claifier ytem and in eence, the true gauge of performance [1, ]. Given infinite training data, conitent claifier approximate the Bayeian deciion boundarie to arbitrary preciion, therefore providing imilar generalization [3]. However, often only a limited portion of the pattern pace i available or obervable [4, 5]. Given a finite and noiy data et, different claifier typically provide different generalization (or different deciion boundarie) [6]. For example, when claification i performed uing a multilayered, feed-forward artificial neural network, different weight initialization, or different architecture (number of hidden unit, hidden layer, node activation function etc.) reult in difference in performance. It i therefore neceary to train a multitude of network when approaching a claification problem to enure that a good model/parameter et i found. However, electing uch a claifier i not necearily the ideal choice, ince potentially valuable information may be wated by dicarding the reult of le-ucceful claifier [7]. In order to avoid the potential lo of information through electing only one claifier, the output of all the available claifier can be pooled before a deciion i made. Thi approach i particularly ueful for difficult problem, uch a thoe that involve a large amount of noie, limited number of training data, or unuually high dimenional pattern. The overall architecture of a combiner i hown in Figure 1. The output of an individual claifier uing a ingle feature et i given by f ind. Multiple claifier, poibly trained on different feature et, provide the combined output f comb. There are everal method of combining that have proved effective in improving the claifier performance. Simple averaging of the output of individual claifier ha been

3 f comb Combiner f ind Claifier 1 Claifier m Claifier N Feature Set 1 Feature Set Feature Set M Raw Data from Oberved Phenomenon Figure 1: Combining Strategy. The olid line leading to f ind repreent the deciion of a pecific claifier, while the dahed line lead to f comb, the output of the combiner. uggeted by different reearcher a an alternative to electing the bet network [8, 9, 10]. Method that elect the cla with the highet activation value, ue the geometric mean or entropy baed criteria, or perform a majority vote have been analyzed [11, 1, 13]. Method baed on confidence factor obtained through the theory evidence have alo been tudied [14]. Weighted averaging ha been propoed, along with different method of computing the proper claifier weight [8, 9]. A urvey of leading combining technique, along with experimental reult i given in [11, 1]. Combining technique uch a majority voting can generally be applied to any type of claifier, while other rely on pecific output, or pecific interpretation of the output. For example, the confidence factor method relie on the interpretation of the output 3

4 a the belief that the pattern belong to a given cla [15, 10]. Averaging, on the other hand, ue the reult that the output of parametric claifier that are trained to minimize a cro-entropy or mean quare error (MSE) function, given one-of-n deired output, approximate the a poteriori probability denitie of each cla [16]. In particular, the MSE i hown to be equivalent to MSE = K 1 + i x D i (x)(p(c i x) f i (x)) dx where K 1 and D i (x) depend on the cla ditribution only, f i (x) i the output of the node repreenting cla i given an output x, p(c i x) denote the poterior probability and the ummation i over all clae. Thu minimizing the (expected) MSE correpond to a weighted leat quare fit of the network output to the poterior probabilitie [16, 17]. For regreion (or function approximation) problem, recent work analyzing the effect of linear combining i available [18, 19]. However, depite the increaing body of experimental reult howing claification improvement due to combining, there ha been no analytical tudy that can quantify the achievable gain. In thi paper we analytically tudy the effect of combining in output pace with a focu on the relationhip between deciion boundary ditribution and error rate. Our objective i to provide an analyi encapulating the mot commonly ued combining trategy, namely, averaging in output pace. The analyi focue on boundary ditribution, and how the parameter of that ditribution influence the error rate. Ultimately, our goal i to both quantify and predict the error reduction due to combining. Cla Boundary Analyi in Abence of Bia A mentioned above, the output of certain claifier are expected to approximate the correponding a poteriori cla probabilitie if they are reaonably well trained. Thu the deciion boundarie obtained by uch claifier are expected to be cloe to Bayeian deciion boundarie. Moreover, thee boundarie will occur in region where the number of training 4

5 ample belonging to the two mot locally dominant clae are comparable. We will focu our analyi to network performance around the deciion boundarie. Conider the boundary between cla i and j. Firt, let u expre the output repone of the i th unit of a one-of-n claifier network to a given input x a 1 : f i (x) = p i (x) + ǫ i (x), (1) where p i (x) i the a poteriori probability ditribution of the i th cla given input x, and ǫ i (x) i the error aociated with the i th output. The following analyi i for calar x, for implicity. However, the analyi can be readily extended for multi-dimenional input. p (x) i f (x) i Optimum Boundary Obtained Boundary p (x) j f (x) j Cla i Cla j x * x b x b Figure : Error region aociated with approximating the a poteriori probablitie. For the Baye optimum deciion, a vector x i aigned to cla i if p i (x) > p k (x), k i, o the Baye optimal boundary i the loci of all point x : p i (x ) = p j (x ) for a two-cla problem. Since our claifier provide f i ( ) intead of p i ( ), the deciion boundary obtained, x b, may vary from the ideal boundary (ee Figure ). Let b denote the amount by which 1 If two or more claifier need to be ditinguihed, a upercript i added to f i(x) and ǫ i(x) to indicate the claifier number. Here, p i(x) i ued for implicity to denote p(c i x). 5

6 the boundary of the claifier differ from the ideal boundary (b = x b x ). We have: f i (x + b) = f j (x + b), by definition of the boundary. Thi implie: p i (x + b) + ǫ i (x b ) = p j (x + b) + ǫ j (x b ). () Now, let u aume that the poterior are locally monotonic function around the deciion boundarie. Thi hypothei i well founded ince typically the boundarie are located in tranition region where the poterior are not in a local extrema. Then a linear approximation of p k (x) around x provide: p k (x + b) p k (x ) + b p k (x ) k, where p k ( ) denote the derivative of p k( ). With thi ubtitution, Equation become: p i (x ) + b p i(x ) + ǫ i (x b ) = p j (x ) + b p j(x ) + ǫ j (x b ). (3) Now, ince p i (x ) = p j (x ), we get: b (p j (x ) p i (x )) = ǫ i (x b ) ǫ j (x b ). Finally we obtain: where: b = ǫ i(x b ) ǫ j (x b ), (4) = p j (x ) p i (x ). (5) Equation 4 can be ued to obtain the ditribution of b. Let the error ǫ i (x b ) be broken into a bia and a zero-mean noie term (ǫ i (x b ) = β i + η i (x b )). For the time being, the bia i aumed to be zero (i.e. β k = 0 k), and the error i entirely due to noie. The cae with non-zero bia will be dicued in the next ection. Let the noie η k (x) be independent k, 6

7 and have Gauian ditribution with zero-mean and σ η k variance 3. Then, b i a Gauian random variable with zero-mean and variance σ b where: σ b = σ η i + σ η j. Figure how the a poteriori probabilitie obtained by a non-ideal claifier, and the added error region aociated with it. The lightly haded area provide the Bayeian error region. The darkly haded area i the added error region aociated with electing a deciion boundary that i offet by b, ince pattern correponding to the darkly haded region are erroneouly aigned to cla i by the claifier, although ideally they hould be aigned to cla j. Let u now divert our attention to the effect of combining multiple claifier. In what follow, the combiner denoted by ave perform an arithmetic average in output pace. If N claifier are available, by uing the ave combiner, we obtain an approximation to p i (x) given by: fi ave (x) = 1 N N fi m (x), m=1 which can be written a: fi ave (x) = p i (x) + η i (x), where: η i (x) = 1 N N ηi m (x). m=1 If the error of different claifier are independent, the variance of η i i given by: σ η i = 1 N N ση i m m=1. 3 Each output of each network doe approximate a mooth function, and therefore the noie for two nearby pattern on the ame cla (i.e. η k (x) and η k (x + x)) i correlated. The independence aumption applie to inter-cla noie (i.e. η i(x) and η j(x)), not intra-cla noie. 7

8 The boundary x ave then ha an offet b ave, where: fi ave (x + b ave ) = fj ave (x + b ave ), and the variance σb ave can be computed in a manner imilar to σ b, reulting in: σ b ave = σ η i + σ η j. In particular, if σ η m i = σ η l i, m,l, we get: σ η i = 1 N σ η i, (6) which lead to: σ b ave = σ η i + σ η j N, or: σ b ave = σ b N. (7) Thi reduction in variance can be readily tranlated into a reduction in error rate, ince a narrower boundary ditribution mean the likelihood that a boundary will be near the ideal one i increaed. In effect, uing the evidence of more than one claifier reduce the variance of the cla boundary from the ideal one, thereby providing a tighter error-prone area. In order to etablih the exact improvement in the claification rate, the expected added error region will be computed, and the relationhip between claifier boundary variance and error rate will be explored further in Section Cla Boundary Analyi in Preence of Bia In general, the etimate of the poterior probabilitie obtained by a network will be biaed, i.e. β k 0. A dicued, in the previou ection, the error i expreed a the um of bia and noie, reulting in: f i (x) = p i (x) + β i + η i (x). 8

9 Here, β i i the bia introduced by the claifier 4, and η i (x) i the zero-mean noie term of Section. Proceeding in a manner imilar to that of Section, one readily obtain: b = η i(x b ) η j (x b ) + β, (8) where i a in Equation 5 and: β = β i β j. Again taking the noie to be independent between clae and Gauian with zero-mean and variance σ η i, we conclude that b i a Gauian random variable with mean β and variance σb, which i given by Equation 6. Combining multiple claifier through ave provide: f ave i (x) = p i (x) + β i + η i (x), where: β i = 1 N N βi m, m=1 and η i (x) = 1 N N ηi m (x). m=1 The variance of η i (x) i given in Section. The boundary x ave ha an offet b ave given by: b ave = η i(x) η j (x) + β. The ditribution of b ave can be obtained from thoe of η i (x) and η j (x), and yield a Gauian ditribution with mean β and variance σb ave. 4 The bia i expected to be different for ditinct clae. If the bia term i a imple additive contant, independent of the cla (that i β i = β j), then in the difference f i(x) f j(x), the biae cancel out, reducing the deciion boundary to the one of the previou ection. 9

10 The effect of combining i le clear in thi cae, ince the average bia ( β) i not necearily le than each of the individual biae. The effect of the bia on the error region will be tudied in detail in Section 4.. However, from an inpection of the ditribution of b ave certain obervation can be made. If the bia i extremely mall, and the error i mainly due to the variance, combining can be an effective tool. If however error are mainly due to high bia, thi type of combining become effective only if the biae are not highly correlated. Thee limiting cae for the error (motly bia or motly variance) alo how a new approach to tackling the well known bia/variance problem [3]. By keeping the bia very mall for each claifier, achieved by uing larger network than neceary, combining reduce the error, mainly due to variance, ignificantly. Thee reult highlight the baic trength of combining, which not only provide improved error rate, but i alo a method of controlling the bia and variance component of the error eparately. The election of network ize and training regime can then directly reflect thi reult. 4 Added Error Region Analyi 4.1 Added Error in the Abence of Bia In the previou ection, we howed that combining i an effective way of reducing the variance of the deciion boundarie. The quetion of how thi reult tranlate into improved claification reult i dicued in thi ection. The added error region aociated with a claifier, denoted by A(b), i given by: A(b) = x +b x (p j (x) p i (x)) dx, which i the darkly haded region in Figure. Baed on thi area, the expected added error, E add, i given by: E add = A(b)f b (b)db, (9) 10

11 where f b i the denity function for b, a dicued in Section. The expected error become: E add = x +b x (p j (x) p i (x)) f b (b) dxdb. p (x) i p j(x) h(b) x * x b b Figure 3: Linear added error region analyi. Now, recall that the boundary i located in the region where the poterior are locally monotonic. Thi allow the linear approximation dicued in Section, making the region A(b) triangular in hape (ee Figure 3). The accuracy of thi approximation depend on the proximity of the boundary to the ideal boundary. However, ince the boundary denity decreae exponentially with increaing ditance from the ideal boundary, we can expect the triangular region to reaonably repreent the added error area for mot likely (i.e. mall) value of b. The bae, h(b), of the triangular region i imply bp j (x) bp i (x). Thu we obtain: A(b) = 1 ) (b b p j(x ) b p i(x ) = 1 b, (10) where i given by Equation 5. Furthermore, due to the ymmetry of the problem, the integration of Equation 9 can be performed only for b 0, and multiplied by two. 11

12 By uing the value given by Equation 10 for A(b), and performing the integration for b 0, the expected error become: E add = 0 1 b πσb e b σ b db, (11) leading to: E add = πσb 0 b e b σ b db. Integrating by part we obtain: E add = σ ([ b b e b σ b π ] e b ) σ b db. The firt term give a value of 0 at both limit, and the econd term give expected error then yield: πσ b. The E add = σ b. (1) The importance of Equation 1 i that it provide a direct relationhip between the expected added error region and the variance of b, the amount by which the elected boundary differ from the ideal boundary. Any reduction in the variance of b i directly tranlated into a reduction in expected error rate. Let the error region aociated with claifier ave be denoted by E ave add : E ave add = σ b ave = σ b N = E add N. (13) Equation 13 quantifie the improvement due to combining N claifier. Under the aumption of Section, combining in output pace reduce added error region by a factor of N. Of coure, the total error, which i the um of Baye error and the added error, will be reduced by a maller amount, ince Bayeian error will be non-zero for problem with overlapping clae. The value provided by Equation 1 will generally tend to be a conervative bound on the added error. The total height h(b) i bounded by 1, ince it i the difference of two a 1

13 poteriori probability ditribution. A more accurate method for computing A(b) i to ue the following height equation: h(b) = b if 0 b 1 1 otherwie Uing thi height in computing the added error area lead Equation 9 to: ( 1 E add = A(b)f b (b)db A(b)f b (b)db ), which lead to: E add = πσb ( 1 0 b e b σ b db + 1 [A( 1 ) + (b 1 ) ] e b ) σ b db, or: E add = πσb (A(1 ) 1 ) πσb + πσb 1 b e b σ b db 1 e b σ b db b e b σ b db. (14) Equation 14 provide a more accurate added error term than Equation 1. However, due to it coniderable complexity, it i generally not preferable to compute the added error in thi form. If Equation 14 need to be explicitly computed, the following procedure can be followed: The firt term can be computed uing integration by part, the econd term can be expreed in term of Gauian ditribution function (F( )), and the third term can be integrated, leading to: E add = 3σ b e 1 σ b π + σ b ( F( 1 ) 1 ) 1 (1 F( 1 ) ). However, it i important to note that in a majority of cae, Equation 1 will be ufficient. Only when the boundary provided by claifier m fall in the region where the a poteriori probabilitie are at their limiting value i a more accurate expreion needed. A claifier 13

14 that repeatedly put the boundary in uch a region i of little ue in general. Therefore, it i reaonable to expect that mot claifier will provide boundarie that fall in the region where the linear approximation i adequate. 4. Added Error in the Preence of Bia In thi ection we compute the expected error in the preence of a bia β. The actual error area a computed in the previou ection i not affected by the bia. However the ditribution of b i affected and the expected value take a different form. Equation 9 lead to: E add (β) = 1 b πσb e (b β) σ b db, (15) which i imilar to Equation 11, but for the hift in the center of the ditribution of b. A change of variable i needed to enable u to compute thi expreion. Let y = b β (y ha the ame variance a b but ha zero mean). With thi change of variable we obtain: E add (β) = πσ (y + β) e y σ b dy, which lead to: E add (β) = + + πσ β πσ β πσ y e y σ b dy e y σ b dy y e y σ b dy. Simplifying each component in term of firt and econd moment provide: E add (β) = E[y ] + β + βe[y] or: E add (β) = (σ b + β ) (16) 14

15 Equation 16 reduce to Equation 1 if the bia i et to zero. The effect of combining i not a obviou a it previouly wa. Let u tudy the error aociated with E ave add (βave ): E ave add (βave ) = (σ b ave + (βave ) ) which i: E ave add(β ave ) = ( ) σ b N + β z (17) where β ave = β z, and z 1. Now let u limit the tudy to the cae where z N. Then 5 : leading to: E ave add (βave ) ( ) σ b + β z E ave add(β ave ) 1 z E add(β). (18) Equation 18 quantifie the error reduction in the preence of network bia. The improvement are more modet than thoe of the previou ection, ince both the bia and the variance of the noie need to be reduced. The actual reduction i given by min(z,n), demontrating that the maller reduction i the limiting factor. Thi reult underline why method aimed at reducing only the variance or only the bia generally do not lead to ignificant improvement in overall claification performance. 5 Dicuion Combining claifier in output pace ha led to improved performance in many application [1, 13, 0]. Thi paper concentrate on explaining the reaon for expecting uch improvement and to quantify the gain achieved. Under the aumption that the a poteriori 5 If z N, then the reduction of the variance become the limiting factor, and the reduction etablihed in the previou ection hold. 15

16 probability ditribution for each cla are locally monotonic function about the deciion boundarie, we howed that combining network in output pace reduce the variance in boundary location. Furthermore, the error region are directly computed and given in term of the boundary ditribution parameter. In the abence of network bia, the reduction in the error i directly proportional to the reduction in the variance. Moreover, if the network error are zero-mean i.i.d. Gauian, then the reduction in variance boundary location i by a factor of N, the number of claifier that are combined. In the preence of network bia, the reduction are le than or equal to N, depending on the correlation among the network biae. Although our analyi focued on only two clae, it i readily applicable to a multicla problem. Since the larget network output determine cla memberhip, only a handful of clae are likely at any given point in input pace. Therefore, even in a multi-cla problem, one only need to conider the two clae with the highet activation value in a given localized region. The ditribution of the boundary i hown to be Gauian through it relationhip with the noie term. If the noie prove to have a ditribution other than Gauian, the analyi can be modified to accommodate the new ditribution. The expected error given in Equation 9 i in general form, and any denity function can be ued from there on to reflect change in the ditribution function. For problem with higher dimenionality, the analyi become ignificantly more complicated, but retain the ame conceptual tructure. The added error area of Figure 3 become a volume for -dimenional ignal, and in general i an (n+1)-dimenional hypervolume for n-dimenional problem. In the mot general cae we have: E(m) = R n A h(x 1,x,,x n )dx 1 dx dx n f B (B) db. where h( ) delineate a multidimenional difference between a poteriori probabilitie defined over an n-dimenional region A, and f B (B) i the multidimenional denity function of the n-dimenional boundary B defined over R n. 16

17 Another important feature of combining that aroe from thi tudy relate to the claic bia/variance dilemma. Combining provide a method for decoupling the two component of the error to a degree, allowing a reduction in the overall error. Bia in the individual network can be reduced by uing larger network than required, and the increaed variance due to the larger network can be reduced during the combining tage. Studying the effect of thi coupling between different error and ditinguihing ituation that lead to the highet error reduction rate are the driving motivation behind thi work. That goal i attained by clarifying the relationhip between output pace combining and claification performance. The analyi preented here provide an undertanding of the interaction between the error rate and claifier boundary ditribution, and ultimately between error rate and output pace combining. Several practical iue that relate to thi analyi can now be addreed. Firt, let u note that ince in general each individual network will have ome amount of bia, the actual improvement will be le radical than thoe obtained in Section 4.1. It i therefore important to determine how the biae of individual network can be kept uncorrelated (or have only minimal correlation). One method i to ue network with architecture baed on different principle. For example, uing multi-layered perceptron and radial bai function network provide both global and local information proceing, enuring that the biae are not highly correlated. Another method i to train imilar network on different feature extracted from the ame underlying data. Although the ame network type i ued, the biae will le correlated, ince they are a function of the training data a well a the network. Experimental reult obtained by u on an oceanic data et with four ditinct clae upport the above concluion [1]. One final note that need to be conidered i the behavior of combiner for a large number of claifier (N). Clearly, the error cannot be arbitrarily reduced by increaing N indefinitely. Thi obervation however, doe not contradict the reult preented in thi analyi. For large N, the aumption that the error were i.i.d. break down, reducing the improvement due to each extra claifier. The number of claifier that yield the bet 17

18 reult depend on a number of factor, including the number of feature et extracted from the data, their dimenionality, and the election of the network architecture. The focu of thi paper i on combining in output pace through averaging. Although the implicity of averaging provide a pleaing framework, it i not the only method that yield encouraging reult. A mentioned previouly, there are many other poibilitie in combining network that require cloer invetigation. The ue of order tatitic, for example, promie to provide improvement that can be analytically tudied, and we are currently puruing that line of reearch. Acknowledgement: Thi reearch wa upported in part by AFOSR contract F , ONR contract N C-03, and NSF grant ECS Reference [1] E. Levin, N. Tihby, and S. A. Solla. A tatitical approach to learning and generalization in layered neural network. Proc. IEEE, 78(10): , Oct [] D. H. Wolpert. A mathematical theory of generalization. Complex Sytem, 4:151 00, [3] S. Geman, E. Bienentock, and R. Dourat. Neural network and the bia/variance dilemma. Neural Computation, 4(1):1 58, 199. [4] R. O. Duda and P. E. Hart. Pattern Claification and Scene Analyi. Wiley, New York, [5] K. Fukunaga. Introduction to Statitical Pattern Recognition. (nd Ed.), Academic Pre, [6] J. Ghoh and K. Tumer. Structural adaptation and generalization in upervied feedforward network. Journal of Artificial Neural Network, 1(4): ,

19 [7] D. H. Wolpert. Stacked generalization. Neural Network, 5:41 59, 199. [8] W.P. Lincoln and J. Skrzypek. Synergy of clutering multiple back propagation network. In D. Touretzky, editor, Advance in Neural Information Proceing Sytem-, page Morgan Kaufmann, [9] M.P. Perrone and L. N. Cooper. When network diagree: Enemble method for hybrid neural network. In R. J. Mammone, editor, Neural Network for Speech and Image Proceing, chapter 10. Chapmann-Hall, [10] K. Tumer and J. Ghoh. A framework for etimating performance improvement in hybrid pattern claifier. In Proceeding of the World Congre on Neural Network, page III:0 5, San Diego, INNS Pre. [11] J. Ghoh, S. Beck, and C.C. Chu. Evidence combination technique for robut claification of hort-duration oceanic ignal. In SPIE Conf. on Adaptive and Learning Sytem, SPIE Proc. Vol. 1706, page 66 76, Orlando, Fl., April 199. [1] J. Ghoh, K. Tumer, S. Beck, and L. Deuer. Integration of neural claifier for paive onar ignal. In C.T. Leonde, editor, Control and Dynamic Sytem Advance in Theory and Application, volume 77, page Academic Pre, [13] L. Xu, A. Krzyzak, and C. Y. Suen. Method of combining multiple claifier and their application to handwriting recognition. IEEE Tranaction on Sytem, Man and Cybernetic, (3): , May 199. [14] G. Rogova. Combining the reult of everal neural network claifier. Neural Network, 7(5): , [15] D. Heckerman. Probabilitic interpretation for MYCIN uncertainty factor. In L.N Kanal and J.F. Lemmer, editor, Uncertainty in Artificial Intelligence, page North-Holland,

20 [16] M.D. Richard and R.P. Lippmann. Neural network claifier etimate Bayeian a poteriori probabilitie. Neural Computation, 3(4): , [17] P.A. Shoemaker, M.J. Carlin, R.L. Shimabukuro, and C.E. Priebe. Leat quare learning and approximation of poterior probabilitie on claification problem by neural network model. In Proc. nd Workhop on Neural Network, WNN-AIND91,Auburn, page , February [18] S. Hahem. Optimal Linear Combination of Neural Network. PhD thei, Purdue Univerity, December [19] M. P. Perrone. Improving Regreion Etimation: Averaging Method for Variance Reduction with Extenion to General Convex Meaure Optimization. PhD thei, Brown Univerity, May [0] X. Zhang, J.P. Meirov, and D.L. Waltz. Hybrid ytem for protein econdary tructure prediction. J. Molecular Biology, 5: , 199. [1] K. Tumer and J. Ghoh. Boundary variance reduction for improved claification through hybrid network (Invited paper). In Application and Science of Artificial Neural Network, Proceeding of the SPIE, volume 49, page , April