Collective organization in an adaptative mixture of experts

Transcription

1 Coective organization in an adaptative mixture of experts Vincent Vigneron, Christine Fuchen, Jean-Marc Martinez To cite this version: Vincent Vigneron, Christine Fuchen, Jean-Marc Martinez. Coective organization in an adaptative mixture of experts. IEEE/SMC 18th internationa Conference Systems engineering (ICSE), Mar 1996, Las-Vegas, United States. pp., <ha-22137> HAL Id: ha Submitted on 29 Jan 28 HAL is a muti-discipinary open access archive for the deposit and dissemination of scientific research documents, whether they are pubished or not. The documents may come from teaching and research institutions in France or abroad, or from pubic or private research centers. L archive ouverte puridiscipinaire HAL, est destinée au dépôt et à a diffusion de documents scientifiques de niveau recherche, pubiés ou non, émanant des étabissements d enseignement et de recherche français ou étrangers, des aboratoires pubics ou privés.

2 COLLECTIVE ORGANIZATION IN AN ADAPTIVE MIXTURE OF EXPERTS Vincent Vigneron 1,2 and Christine Fuchen 2 and Jean-Marc Martinez 1 1 CEA de Sacay, DRN/ DMT/SERMA/LETR, Gif-sur-Yvette, France 2 CREL, 161, rue de versaies, 7818 Le Chesnay, France Abstract. The use of mutipe neura modes have attracted much interest recenty as predictive modes in system identification and contro in the statistic and in the connectionist communities. In this paper, we sha restrict our attention to one particuar form of density estimation caed a mixture mode. As we as providing powerfu techniques for density estimation, mixture modes find important appications in techniques for conditiona density estimation, in the technique of soft weight sharing and in mixture-of-experts mode. By exampe, the hierarchica mixture-of-experts of Jacobs & Jordan have a powerfu representationa capacity and abiity to hande with any muti-input muti-output mapping probem, yeding significanty faster training through the use of Expectation Maximisation agorithm. The genera approach consists to divide the probem into a series of sub-probem and assign a set of experts to each sub-probem. In this paper we design a new approach couping EM agorithm (non-parametric) and Backpropagation rue (parametric) for a supervised/unsupervised segmentation of data originating from different unknown sources. We present this method as a feasibe approach for earning any inverse mapping of causa systems, since Least-squares approach often eads to extremey poor performance if the image of an input is a non-convex region in the output. But in contrast to mixture-of-experts architecture, the competitions depend on the reative performance of their experts-networks, not on the input. This reasonning eads to a somewhat Bayesian version of the decision by postuating a refinement of the data. the Bayesian context is presented as an aternative to the currenty used frequency approach, which does not offer cear, compeing criterion for the design of statistica methods. This approach is non-destructive in the sense that it does not force the data into a possiby inappropriate representation. As a simpe iustration of this probem, we consider a probem of contro of uranium enrichment by spectrometry-γ. 1 Introduction Especiay in the fieds of cassification and time series prediction, ANN have made substantia contributions. Of particuar interest for modeing is their abiity to describe the behaviour of any non-inear systems, proved by [?]. From a modeing point of view, a NN is a particuar (often noninear) parametrisation of a regression mode for the system ike y(x) = F (x, W). The device is capabe to extract the underying structure in the exampes it was trained on and to generaise from them. But, since we have imited resources we woud ike to be abe to use a our data to fit the mode. Furthermore if the data originate from different sources, ike many kinds of physica phenomena, direct approaches are ikey to fai to represent the canceaed input-output reations. [?] cited the non-causa approach as particuary reevant to map one-to-many reations, where the image of an input is a non-convex region in the output. There are three key ideas in our treatment. First, we demonstrate that reconstruction methods can be used to form the joint density P (x, y), to estimate, given a particuar input x, the mixture representation for the vector function y = f(x). We show that these methods are preferabe on grounds of convenience and accuracy and we beieve they are more fexibe than other recenty proposed mixture of experts techniques. Secondy, we show that the mixture modes approach aow much more subte extraction of information from posterior distributions. Finay, we base our experiments and discussion on a mode for mixtures that aim to provide a simpe and generaisabe way of being strongy informative about the mixing proportion parameters, even as an aternative to query earning.

3 2 Non-causa mixture of modes 2.1 Background We consider the case where the different input sampes y (i) can be generated by a number n of known functions f, = 1,..., n, i.e. y (i) = f (x (i), θ ). The task then is to determine the functions f together with their respective contribution (i) from a given data set X = {x (i), y (i) } N i=1, where x R o and y R i. Since the functions are considered to be unknown, they have to be determined simutaneousy (see Fig. 1), i.e. the correct attribution has to be found in an unsupervised manner. x 1 x2 x3 RECONSTRUCTION x 4 x g 1 g g 2 3 g 4 superviseur Fig. 1. Competition-Cooperation in an inverse mode. The moduar approach presents three main advantages on the singe BP modes: it is abe to mode behavior, it earns faster than a goba mode and representation is easier to interpret : the moduar architecture takes advantage of task decomposition, but the statistician must decide which variabes to aocate to the networks. In this strategy, the input data y (i) are physicay segmented without prior knowedge about the sources. Through these hard spatia constraints, we attribute at each input-output pair (y (i), x (i) ) a parcimonious mode y (i) = f (x (i), θ ). This segmentation assigns an expert-network to a singe geographica region or dimension of y. It is not necessary to rank the y, to know which eements are neighbors and which one are not. We show recenty it is possibe to take into account spacia contiguity constraints. We therefore adapt the set of predictors f, = 1,..., n, by training the netwoks on the trainingdata set : we train the weights θ of network by performing a gradient descent θ C (i) θ on the squared costs C (i) = (f (x (i) ) y (i) ) 2. Now, we can rewrite the output as the basic mixture mode for independant observations y (i) as a mixture of inverse modes : x (i) n =1 p (i) g (y (i) θ ), independanty for i = 1,..., N. The objective of the anaysis is inference about the unknown mixing proportions p. We postuate a heterogeneous popuation from which our random sampe is drawn (the objective of this modeing step is not density estimation but fusion of data). The weighting coefficient p (i) corresponds to the reative probabiity for a contribution of network and they are constrained to be n =1 p = 1 and < p < 1 for a = 1,..., n. The g (y (i) θ ) is the contribution (see section 2.2)of each expert-network to the estimation of x.

4 2.2 The inverse mode This section outines the basic earning agorithm for finding the maximum ikeihood parameters of the mixture mode ([?,?]). [?] has shown that the EM agorithm is an aternate optimisation agorithm which considers the foowing decomposition of the (compete) ikeihood : (Θ X) = n =1 i=1 N u i og {p g (y (i), Σ )} n =1 i=1 N u i og u i, (1) where Θ (u 1,..., u n ; 1,..., n, Σ 1,..., Σ n ) is the parameters vector. The EM agorithm is an iterative method for maximising the former criterion. The intuition behind it is that if one had access to a hidden random variabe (here u) that indicated which data point was generated by which component, then the maximisation probem woud decoupe into a set of simpe maximisations. (Θ X) can be maximised by iterating the foowing two steps : E-step : u i = P p ĝ (x (i),σ ) M-step : ˆ (i) = j pj ĝj(x(i) j,σ j) Pi P u ix (i) i u, Σ i = i u i(x (i) ˆ )(x (i) P i ˆ ), ˆp = u i n using the Lagrange s equation L = (Θ X) + n =1 λ i( N i=1 u i 1), where the λ i are the Lagrange coefficients corresponding to the constraints N i=1 u i 1 =, i. The E (Expectation) step computes the expected compete data og ikeihood and the M (Maximisation) step finds the parameters that maximise this ikeihood 3. Convenient for cacuation and interpretation, the p s incorporate memory that is present in the process. This soution enabes the simutaneous determination of the numerica estimates of the a posteriori probabiity distribution functions (pdfs) g and their uncertainties Σ. An inverse probem entais the reiabiity of the inverse soution given the data set and the cass of modes f (. θ ) used to simuate the data. It arises in two rather distinct contexts ; the first expore the cassica frame of the bias-variance trade-off in the sense of [?]. In the second context, the g (. θ ) s is thought of as a convenient parcimonious representation of a non-standard density function, which is particuary we suited for any appropriate cass of mode, when a priori f information is avaiabe, i.e. by exampe the jacobian of each expert mode (see [?]). In this case, et both dx and be a finite shift in any mode input towards the soution x, the expected soution < x > and the variance σ 2 may be derived from : f < x >= p x (i) x dx, and σ2 = p 2 ( f x (i) x )2 dx < x > 2. (2) The margina pdf f x may be interpreted as the bayesian prior of the parameter p. For genera non-inear probems, its distribution cannot be predicted from a priori pdf or from the data pdf. In the foowing, we propose a compete procedure to estimate the output x of our inverse mode : 3 maximising the og-ikeihood is equivaent to minimising the non-equiibrium Hemhotz free-energy from a statistica Mechanics perspective.

5 Agorithm 1. for a given y (i), compute x (i) = arg max P x( (f (x (i) ) y (i) ) 2 ) x being chosen in the training data set. 2. compute the mixing parameters p foowing EM procedure 3. repeat x (i) = x (i) + α P p f δx (i) σ 2 = σ 2 + α P {p2 ( f ) 2 δx (i) < x > 2 } unti convergence. Tabe 1. Inverse earning agorithm. The derivatives propagation starting from the output units. f for each mode are cacuated by backward One of the advantages of this mode against the cassica stochastic inverse techniques is that gaussian priors are rarey adequat to describe our a priori beiefs about the mode parameters. A modified version of the agorithm 2.2 can be used for maximising aso the criterion in Eq. 1. We start by assuming that the outputs f (x i ) are distributed according to gaussians, i.e. p e βc. It differs from previous work in the sense the competition between the expert-networks depends ony on their reative performance and not on the input, in constrast to the mixture of experts architecture that uses an input-dependent gating-network ([?]). Bayes rue in this case gives for the weighting coefficients : ˆp = e βc j e βcj (3) where the inverse temperature parameter β guarantees unambiguous competition. Eq. 3 is derived from Gas Theory of statistica Mechanics, caed Maximum Entropy (see [?]). For β =, the predictors equay share the same probabiity 1 n. Increasing β enforces the competition, thereby driving the predictors to a speciaisation (see [?]) when using agorithm 1. In the next section, we iustrate through case studies the genera phenomenon of non-convex inverses in earning. 3 Appications and discussion We start in this section by gathering exempes that tease out aternative expanations of the data, which woud be difficut to discover by other approaches. Uranium enrichment measurements As a first exampe of a non-convex probem, we present prediction of Uranium enrichment from γ-spectra. These spectra are characterized by a vast number of highy correated inputs. [?,?] showed that Caibration procedure and matrix effects can be avoided by focusing the spectra anaysis on a imited region, caed K α X region. It is possibe to earn such Decared enrichment BP net Inverse Mode,711%.7-.72% % 1,416% % % 2,78% % %,111% % % 6,122% % % 9,48% % % Tabe 2. Min-Max of cacuated Enrichments with BP forward predictor and Inverse Mode. patterns, but with sti an excessive number of weights and exhibiting a high non-convexity (see Fig. 2.c). This singuar scheme referes extremey i-posed earning probem [?] where an exempe consists of a very arge input vector but where it is nevertheess the aim to earn and generaise froma reative number of exampes.

6 Approaches to earning the inverse γ-spectrometric map by samping the Kα region and directy estimating a function Ũ = ˆf(x) where x R 21 have met some success. Here, we propose an aternative method where we use our density estimation technique to form a mode of the enrichment predictor. In Tabe 2, we compare the errors obtained on a backpropagation forward network using crossentropy error criterion to earn the direct mapping. It can be seen that itte reaching error (about 1 4 ) can be obtained. Fig. 2.b shows the mixing proportions pdf. when a (,78%) spectrum is presented to the inverse mode. The credit assignement procedure on these 21 contributions is supervised (see ago. 1) to produce the fina estimation. Santa-Fe competition As an exampe for a high-dimensiona chaotic system, we use the aser data of the Santa-Fe Time series Prediction. The data are intensity measurements of a NH 3 aser beieved to be in a chaotic state, exhibiting Lorentz-ike dynamics..6. Absoute error MLP 3--1 with normaized inputs MNEX mode 6.4 Estimation of Uranium Enrichment Number of spectrum Number of output-node Fig. 2. (a) Absoute error in the enrichment estimation,(b) Exampe of enrichment vaue (at,78 %) predicted by the Mixture of s, c) 3-dimensions curves of K αx region on the whoe data set 1 Energy (MeV) Patterns counts counts rank of the expert Patterns rank of the expert Patterns Fig. 3. (a) pot of the Santa-Fe Time series Prediction, (b) mixing proportions for a test set with Maximum Entropy agorithm, (c) mixing proportions for a test set with EM agorithm We imit our anaysis to the prediction of a sma numbers of step. 12, data points were earned by BP earning to map the inverse reation with experts (we coud have imited ourseves to a

7 dozen) using 2 iterations and iterating 3, times of the EM agorithmm, enough for approximate convergence. The density was then used to estimate the foowing 2 steps of the series. These outputs were then compared to the rea vaues to obtain an error measure. This mean eucidean errors were ess than 1 1 for both inverse modes. The ocaisation of mutipe attribution of the experts (see Fig. 3.b and 3.c) very informative of nature of the anaysed phenomena. It is an interpretabe, ow dimensiona projection of the data and often can ead to improved prediction performance. 4 concusion We have studied the feasabiity of appying ANNs to uranium enrichment measurement. The network is shown to be abe to cacuate 23 U concentrations. Our resuts appear to be at the state of art in automated quantifying methods for isotopes in a mixture of components. Here, we have proposed an indirect approach to the non-convexity probem based on forming an interna mode of the physica mode. This method has been demonstrated as a reiabe too for deaing with few data under adverse conditions. The reconstruction part of the inverse agorithm can be viewed as inking adjacent predictions. It permits ready incorporation of resuts from the statistica iteratures on missing data to yied fexibe supervised and unsupervised earning architectures. Acknowegments The authors wish to thank C. Fuche from CREL for her support in this research.