Neural Networks Trained with the EEM Algorithm: Tuning the Smoothing Parameter

Transcription

1 eural etworks Trained wit te EEM Algorit: Tuning te Sooting Paraeter JORGE M. SATOS,2, JOAQUIM MARQUES DE SÁ AD LUÍS A. ALEXADRE 3 Intituto de Engenaria Bioédica, Porto, Portugal 2 Instituto Superior de Engenaria do Porto, Porto, Portugal 3 IT etworks and Multiedia Group, Covilã, Portugal js@isep.ipp.pt, jsa@fe.up.pt and lfbaa@di.ubi.pt Abstract: - Te training of eural etworks and particularly Multi-Layer Perceptrons (MLP's) is ade by iniizing an error function usually known as "cost function". In our previous works, we apply te Error Entropy Miniization (EEM) algorit in classification and its optiized version using, as cost function, te entropy of te errors between te outputs and te desired targets of te neural network. One of te difficulties in ipleenting te EEM algorit is te coice of te sooting paraeter, also known as window size, in te Parzen Window probability density function estiation for te coputation of te entropy and its gradient. We present ere a forula yielding te value of te sooting paraeter, depending on te nuber of data saples and on te neural network output diension. Several experients wit real data sets were ade in order to sow te validity of te proposed forula. Key-Words: - Entropy, Parzen, Sooting Paraeter, Cost Function Introduction Te use of entropy and related concepts in learning systes is well known. Te Error Entropy Miniization concept was introduced in [] and we use te sae approac in [2] wit te goal of perforing classification. Te EEM Algorit for classification use, as cost function, te Renyi's Quadratic Entropy of te error, between te output of te neural network and te desired targets. We ave ade a furter optiization of te EEM algorit in [3] introducing a variable learning rate to acieve bot a faster convergence of te algorit and an autoatic adaptation of te value of te learning rate to te specific proble. In order to copute te Renyi's Quadratic Entropy of te error one ust coose a value for te sooting paraeter in te Parzen Window etod, tat is best suited for a specific data set. In all known works using tis etod, giving as exaple ours in [2,3] and oter autors in [,4], te value of te sooting paraeter is always experientally selected. Knowing tat tis approac is not te ost appropriate, in tis paper, we study te beavior of te algorit wit different data sets and propose a forula to copute te sooting paraeter for a specific data set and neural network. In section 2 te EEM algorit and furter optiizations are presented. In section 3 te sooting paraeter for entropy and gradient coputation is discussed and a specific forula is presented. In section 4 te results of several experients are presented sowing te validity of te forula. In te final section, we present te conclusions. 2 Te EEM Algorit Let y = { y i } R, i =,..,, be a set of saples fro te output vector Y R of a apping n R R : Y = g( w, X ), were w is a set of eural etwork () weigts, X is te input vector and is te diensionality of te output vector. Let d = d i {, } be te desired targets and ei = d i yi te error for eac data saple. In order to copute te Renyi's Quadratic Entropy of e we use te Parzen Window probability density function (pdf) estiation. Tis etod estiates te pdf as e ei f ( e) = K( ) () i= were K is a kernel function and te bandwidt or sooting paraeter. We use te Gaussian kernel wit zero ean and unitary covariance atrix: = T G( e, I) exp ( e e ) 2 (2π ) 2 (2) Te Reniy's Quadratic Entropy of te error e can be estiated, applying te integration of gaussian kernels [5], by Hˆ ( e) log 2 = logv ( e) = 2 ei e j G(,2I) i= j= (3)

2 Te gradient of V (e) is: ei e j Fi = G(,2I )( e ) 2 i e + j (4) 2 j= Tis gradient is back-propagated into te MLP te sae way as in te MSE algorit. Te update of te neural network weigts is perfored using w = ± η V. Te ± eans tat we can iniize w ( ) or axiize (+) te entropy. We also proved in [2] tat we can use tis cost function for classification if te output of te neural network is on [ a, a] and te targets are in { a, a}, a R. Since one of te probles of te EEM algorit is its slow convergence, we introduced in [3] an optiization of te algorit based on te use of a variable learning rate (VLR), during te training pase, wic applies te rule: ( n ) ( n) ( n ) η u if H < ( ) H n η = (5) ( n ) ( n) ( n ) η d if H H being u and d te increasing (up) and decreasing (down) factors of te learning rate η in consecutive iterations. Tis optiization iproved te perforance of te original algorit and we called it te EEM-VLR algorit. 3 Sooting Paraeter In te EEM algorit, te error entropy estiation is different according to te vector e diension. Tis diension depends on te nuber of MLP output neurons tat depends on te nuber of classes, C, and teir coding. If we use binary coding, te diension of vector e will be log 2 C wereas if we use te -out-of- C coding te nuber of output neurons as well as te diension of vector e will be equal to te nuber of classes. In te experients tat we perfored, we obtained good results for bot codings but we suggest te use of binary coding only wen te nuber of classes is a power of 2; oterwise, we get an excessive nuber of outputs copared wit te nuber of classes. One of te probles of pdf estiation using te Parzen Window etod, besides te coice of te kernel, is te coice of te sooting paraeter. In te EEM algorit te value of depends on: te different coding of te nuber of classes; te nuber of data saples; te diension of te vector e. For continuous f (x) te estiated density function will converge to te true density as, wen: and (6) Silveran proposed in [6], for unidiensional cases assuing Gaussian distributed saples, and using a Gaussian kernel, te following forula for te sooting paraeter:.2 op =.6σ (7) were σ is te saple standard deviation and is te nuber of saples. For ultidiensional cases, also assuing noral distributions and using te noral kernel, Bowan and Azzalini [7] proposed te forula: op = σ ( 2) + (8) were is te diension of vector x. An iportant fact tat ipedes, in our case, te use of forulas 7 and 8 is tat our algorit uses te entropy of e as a control variable, i.e., te algorit progresses only if te entropy at a given iteration is saller tan at te previous one. Since te entropy value is proportional to te sooting paraeter value, used to copute it, if one uses a value for proportional to te variance of e, one igt be increasing, by tis siple fact, te entropy value and te algorit fails to reac te iniu. Tis eans tat we are liited to use a value for tat does not depend on σ. Considering tat variable e takes values, in te unidiensional case, in te interval [ 2,2], and tat te axiu standard deviation in tis case is 2, we considered to use tis value to replace te standard deviation in forulas 7 and 8. So, tese forulas becae: and.2 op = 2.2 (9) op = 2 ( 2) + () ote tat, in te EEM algorit, we only need to copute te entropy and its gradient; we do not

3 need to estiate te probability density function of e. Tis is a relevant fact because, in te gradient descent etod, ore iportant tan coputing wit extree precision te gradient is to get wit relative precision its direction. In addition, te coputation wit extree accuracy of te probability density function causes te entropy to ave ig variability. Tis fact could lead to te occurrence of local inia. Given tese considerations, we do not esitate in using values iger tan te ones usually proposed for pdf estiation. Taking into account te experiental results wit several data sets we tried to forulate a rule tat yields iger values of for saller data sets tan tose obtained wit forula and still inducing te sae beavior. We ten arrived at te following proposed new forula wit siilar beavior as () op = 25 () otice te decreasing beavior wit and increasing beavior wit as sould be desired. A coparison between te values of obtained wit forulas and for different values of is sown in Fig.. In te several experients tat we perfored using forula te results were very satisfactory, as we will see in te next section. 4 Experients We perfored several experients wit different real data sets, wit different nuber of saples and different nuber of classes, suarized in Table Table Real data sets used for te experients. Data sets Saples Features Classes Ionospere Sonar wdbc Iris Wine Pb (Te Pb2 data set can be found in [8] and all te oters can be found in te UCI repository of acine learning database). We also produced an artificial data set and used it as a 2-class and as a 4- class classification proble to try to establis te influence of te nuber of classes in te value of te sooting paraeter. Te two versions of te artificial data set (tat we call XOR-n), siilar to an XOR proble, but wit soe noise added, are sown in Fig = = = Fig. - Value of for forulas (dased) and (solid) for =2, 3 and 4. (Marked points refer to experients wit data sets entioned in section 4).

4 In all experients we used (I,n,) MLP's, were I is te nuber of input neurons, n is te nuber of neurons in te idden layer and is te nuber of output neurons. We applied te cross-validation etod using alf of te data for training and alf for testing. Te experients for eac data set were perfored varying te nuber of neurons in te idden layer, te value of te sooting paraeter and using different nuber of epocs XOR n 2 classes XOR n 4 classes.5.5 Fig. 2 - Artificial data set for te 2 first experients. Te results are sown in Table 2 and Table 4. Eac result is te ean error of 2 repetitions. For eac experient we igligted te best results perfored wit te sae nuber of epocs in order to get te needed guidance about te optiu value for te sooting paraeter. In Table 2 we sow te results of te classification errors for te XOR-n data sets. Table 2 Errors (%) for XOR-n data sets, 2 and 4 classes (Resp. 8, 2 and 6 epocs fro top to botto) XOR-n 2 classes uber of Hiden-Layer eurons , 36,5 28,25 24,4 23, 23,38 27,3 2,4 35,43 3,38 26,43 23,93 27, 28,5 2,8 37,9 27,8 24,45 26,65 26,38 26,48 3,2 36,28 26,33 26,78 23,88 25,65 23,68 3,6 35,93 26,3 27, 22,43 27,8 27,2 4, 35,4 25,75 26,83 24,93 2, 24,8 4,4 35,33 27,75 27,25 22,3 26,65 22,2 4,8 35,63 24,7 24,98 23,73 25,33 22,65 5,2 34,45 28,6 26,75 23,5 2,2 24,68 5,6 36,35 25,68 23,75 24,93 25,5 24, 6, 34,2 27,53 23,7 22,38 25,55 23,6 2, 35, 24,3 24,43 22,75 24,48 23,55 2,4 35,78 29,68 25,8 23,88 25,75 26,38 2,8 34, 26,25 25, 25,45 26,75 27,3 3,2 33,68 25,5 24,8 28,55 26,63 25,63 3,6 33,83 24,65 24,53 25,3 24,7 26,43 4, 34,95 25,8 2,2 22,88 26,23 22,5 4,4 33,2 23,38 24, 25,8 23,3 24,5 4,8 33,98 24, 23,68 23,88 23,43 23, 5,2 32,93 22,6 23,63 24,8 24,2 25,8 5,6 33,78 23,4 22,75 23,8 24,93 23,25 6, 33,98 25,8 23,33 22,75 22,63 24,5 2, 36,28 27,3 22,78 23,43 23,7 22,55 2,4 34,68 26,28 24,93 24,3 26,6 27,5 2,8 35, 27, 24,4 27,43 25,5 26,55 3,2 33,65 26,58 24,75 25,25 26,6 25,83 3,6 34,9 22, 26,48 24,8 24,98 25, 4, 33,78 25,63 2,28 24,33 25,8 23,48 4,4 34,3 24,4 23,58 25,53 23,73 26,73 4,8 33,68 24, 23,38 22,55 22,78 23,7 5,2 35, 2,9 24,45 23,25 23,38 23, 5,6 33,3 2,33 23,8 24,33 23,6 23,6 6, 34,7 22,3 22,8 24,33 22,95 23,33 XOR-n 4 classes uber of Hidden-Layer eurons , 9, 9,8 8,3 9,3 8,88 9, 2,4 7,65 8,3 9, 8,73 7,4 8,83 2,8 8,9 7,98 7,9 7,85 9,5 8,23 3,2 7,75 8,8 7,73 8,3 7,75 8,58 3,6 9, 8,5 7,95 8,3 8,53 8,3 4, 7,38 7,73 8,65 8,2 9,3 8,38 4,4 8,8 8,78 8, 8,3 8,8 8,7 4,8 8,68 7,98 8,48 9,8 8,5 8,25 5,2 8,83 7,7 7,3 9,53 8,7 8,7 5,6 8,68 8,33 7,65 8,4 9,5 9,3 6, 7,4 9, 8, 9,53 7,73 8,45 2, 8,63 9,43 8,63 2,8 9,7 2,63 2,4 2,38 9,28 8,55 8,95 9,73 2,45 2,8 8,33 8,98 9,3 8,48 8,88 8,38 3,2 8,28 8,23 7,75 8,5 8,53 8,65 3,6 8,2 8,5 8,95 8,73 7,75 8,8 4, 8,48 8,23 8,8 8,7 8,5 7,78 4,4 7,68 8,8 8,98 8,48 8,5 9,5 4,8 7,6 8,6 8,93 8,35 8,8 8,5 5,2 7,38 8,8 7,8 7,95 8,63 8,78 5,6 7,5 8,23 7,68 8,65 8,78 9,5 6, 7,85 8,8 7,85 8,8 7,68 8,55 2, 8,35 9,45 9,8 9,2 8,83 9,48 2,4 8,6 8,63 9, 9,3 9,85 9,8 2,8 7,93 9,5 9,75 9,6 9,35 9,38 3,2 8,88 9,38 9,68 9,35 9,38 9,58 3,6 8,8 7,95 8,9 8,9 9,9 9,5 4, 7,33 8,85 9,3 8,28 9,65 9,28 4,4 8,43 2,8 9,28 9,6 9,45 9,53 4,8 8,38 9,3 9,3 9,5 8,98 9,8 5,2 7,4 9,68 8,85 8,78 9,98 9,33 5,6 7,6 8,45 8,33 9,3 9,58 9,8 6, 8,8 8,7 9,45 9,3 9,38 9,8

5 Coparing te two tables, we can see tat, an increased nuber of classes deand an increased value of te sooting paraeter. In te first case, (2-class proble), te optiu value for is about 4. and in te second case, (4-class proble), te optiu value for is about 4.8. Te classification errors for te real data sets are sown in Table 4. For eac data set, we perfored experients wit different nuber of epocs. In Table 3 we present te values of for te iniu classification errors ( Best colun) and te value of tat represents, in our opinion, tose sallest classification errors ( Suggested colun). We also present te values proposed for eac data set by forulas and. Data sets Table 3 Values of for eac data set driven by te experients and defined by forulas and. r. of Classes r. of Saples Best Suggested ( in. Errors) For. For. Ionospere ,6 4,2 2,67,85 Sonar ,2 3,9 3,47,92 wdbc 2 569,4,6 2,,78 Xor , 4,6 3,54,93 Iris 3 5 4, 3,8 5,,5 Wine ,6 4,2 4,59,2 Pb2 4 68,8 2, 2,87,93 Xor ,2 4,8 5,,7 Te experients clearly sow tat, for sall values of, te classification errors are significantly large and, terefore, te use of forulas like (), used for density estiation, are not appropriate for te EEM algorit. Figure sows (crosses and open circles), te values obtained in te experients wit all data sets were we can easily see teir relation wit te values of te proposed forula. For eac data set te crosses are te suggested values given by te sallest classification errors and te circles are te values for te iniu classification error. 5 Conclusions In tis paper, we ave presented a forula for te value of te bandwidt paraeter,, of te entropy estiation, to use in te EEM-VLR algorit. Tis forula, contrary to te ones proposed by Silveran and Bowan, does not depend on σ due to te iterative nature of te EEM-VLR algorit. We sowed tat it produces very good results using a set of experients were te values proposed by our forula are uc closer to te best ones found epirically, tan te values obtained using te Silveran and Bowan forulas. We expect tat tis contribution ay save tie and effort to any practitioners using tis type of iterative entropic algorits involving Parzen Window density estiation. Acknowledgent Tis work was supported by te Portuguese Fundação para a Ciência e Tecnologia (project POSI/EIA/5698/24). References: [] D. Erdogus and J. Principe, An Error-Entropy Miniization Algorit for Supervised Training of onlinear Adaptive Systes, Trans. On Signal Processing, Vol. 5, o. 7, 22, pp [2] J. M. Santos, L. A. Alexandre, and J. Marques de Sá, Te error entropy iniization algorit for neural network classification, in Proceedings of te 5t International Conference on Recent Advances in Soft Coputing, A. Lofti, Ed. ottinga Trent University, 24, pp [3] J. M. Santos, J. Marques de Sá, L. A. Alexandre, and F. Sereno, Optiization of te error entropy iniization algorit for neural network classification, in Intelligent Engineering Systes Troug Artificial eural etworks, C.H.Dagli, A. L. Buczak, D. L. Enke, M. J. Ebrects, and O. Ersoy, Eds., vol. 4. ASME Press, 24, pp [4] D. Erdogus and J. Principe, "Entropy Miniization Algorit for eural etworks," Proceedings of te Intl. Joint Conf. on eural etworks, 2, pp [5] D. Xu and J. Princpe, Training lps layer-bylayer wit te inforation potential, in Intl. Joint Conf. on eural etworks, 999, pp [6] B. W. Silveran, Density Estiation for Statistics and Data Analisys. Capan & Hall, 986, vol. 26. [7] A. W. Bowan and A. Azzalini, Applied Sooting Tecniques for Data Analysis. London: Oxford University Press, 997. [8] R. Jacobs, M. Jordan, S. owlan, and G. Hinton, Adaptive ixtures of local experts, eural Coputation, pp. pp.79 87, 99.

6 Table 4 - Errors (%) for te data sets Ionospere (resp. 4, 6 and 8 epocs), Sonar (resp. 5, and 5 epocs),wdbc (resp. 4, 6 and 8 Epocs), Iris (resp. 4, 6 and 8 epocs),wine (resp. 4, 6 and 8 epocs) and Pb2 (resp. 2, 25 and 3 epocs) for different nuber of idden-layer neurons and several values of te sooting paraeter. Ionospere - 4, 6 and 8 epocs ,4 33,84 9,54 2,4 5,29 6,26 7,3 2,7 2,76 2,97 3,24 3,54 3,27 3,7 3,2 3, 3,9 3,7 3,49,8 2,47 2,7 3,2 3,3 2,69 2,6 3,3 3,9 2,9 3, 2,8 3,4 3,47 3,28 3,4 3,8 2,87 3,2 2,2 2,7 2,86 2,8 3,2 3,3 3, 3,4 3,6 2,88 2,9 2,8 2,83 4,2 3,4 3,4 3, 2,8 3,5 2,6 3,7 2,4 2,37 2,82 2,93 3,7 3,7 2,83 3,6 2,94 3,6 2,7 3,33 2,83 2,94 2,53 3,6 2,69 3 2,7 2,97 2,74 2,77 2,7 3,23 3,7 3,3 3,23 2,86 2,84 3,9 3, 2,7 2,7 2,33 2,9 2,9 3,4 3, 2,8 2,8 2,67 2,64 2,84 2,9 2,5 2,77 2,66 2,94 2,6 3, 3, 3, 2,73 2,83 2,59 3,8 2,73 2,79 2,7 2,26 2,46 2,26 3,7 3, 3,36 2,89 2,24 2,87 2,8 2,62 2,88 2,24 2,69 2,63 4,2 3,9 2,39 2,67 2,73 2,59 2,97 3,6 3,29 2,82 2,46 2,4 2,4 3,4 2,4 2,64 2,34 2,94 2,99 4,6 2,96 2,8 2,6 2,8 2,43 2,63 2,77 2,7 2,6 2,44 2,29 2,96 2,6 2,4 3,29 2,67 2,73 2,72 5 2,7 2,6 2,9 2,7 2,69 2,87 3, 2,44 2,44 2,29 3,6 2,87 3,6 2,56 2,6 2,6 2,63 3,6 Sonar - 5, and 5 epocs , 49,47 48,63 49,8 48,97 48,56 5,65 48,36 49,9 49,3 49,4 48,4 49,8 5,82 5,48 5, 49,86 5,82 49,95,3 48,49 45,77 45,67 43,22 4,37 43,73 36,7 35,36 37,74 35,53 34,52 3,42 3,63 28, 27,8 29,88 26,9 25,5,6 24,9 23,65 23,6 22,96 22,88 23,7 24,64 23,44 23,4 24,6 23,94 23,8 25,53 24,52 23,92 24,8 25, 23,6,9 24,78 23,27 22,96 22,4 22,65 23,3 24,2 24,52 23,5 22,86 23,29 23,8 24,23 24,52 23,8 22,33 24,3 24,52 2,2 23,7 23,68 23,53 24,28 23,77 23,66 24,6 24,33 23,49 22,84 24,42 23,8 25,3 24,45 23,27 22,67 22,74 23,5 2,5 24,37 23,5 22,28 23,8 23,34 24,8 23,99 22,4 24,52 24,35 22,8 22,74 24,6 24,6 22,26 22,67 23,4 23,25 2,8 23,82 23,7 23,6 22,72 22,76 22,36 23,5 24,47 23, 22,8 23, 23,87 24,33 23,85 23,99 23, 22,86 22,67 3, 24,4 24,6 22,98 22,98 22,98 22,8 24,64 24,4 22,45 23,82 23,29 22,74 24,98 23,7 23,29 23,5 22,26 22,88 3,4 24, 23,82 24,8 24,4 23,8 23,3 24,6 24,35 23,5 22,2 23,27 2,32 24,42 23,56 22,4 23,32 23,7 22, 3,7 24,5 23,82 2,73 22,8 22,2 22,8 23,56 23,58 23,37 23,44 23,44 2,6 25,4 23,58 23,5 22,84 23,27 22,7 4, 24,26 23,5 23,42 23,56 22,2 22,96 23,58 23,77 23,3 22,4 22,67 22,4 23,73 23,89 23,8 2,95 22,7 22,6 Wisconsin breast cancer - 4, 6 and 8 epocs , 5,34 52,3 5,22 49,44 46,88 5,54 5,6 49,32 47,58 45,97 48,8 46,52 49,48 48,6 48,58,2 8,83,37 9,4 9,74 2,8,7,46,48 9,74 9,47 4,4 2,89 5,49 5,83 6,,4 2,53 2,33 2,65 2,57 3, 2,79 2,8 3,7 2,95 2,82 2,77 2,8 3,8 3, 2,92,6 2,77 2,7 2,6 2,7 2,77 2,97 2,83 2,9 2,83 2,96 2,95 3,23 3,75 3,6 2,9,8 2,83 2,75 2,9 2,84 2,78 3,5 3,6 3,6 2,97 3,5 2,96 3,4 3, 3,8 3,8 2, 2,84 2,87 2,64 2,58 2,97 2,97 3, 3,6 3,7 3,8 3,26 2,87 3,2 2,99 3,6 2,2 2,9 2,77 2,9 3,7 2,9 2,72 3,6 2,8 2,97 3,23 3,3 3, 3,7 2,95 3,5 2,4 2,67 2,76 2,92 2,84 2,59 2,94 3,4 3,4 2,88 3,24 3,6 3,26 3,22 2,99 3,36 2,6 3,28 2,88 3,3 2,97 3,28 3,7 2,93 3,2 3,6 3,49 3,3 3,8 3,6 3,2 3,34 2,8 2,89 2,85 2,79 2,94 2,95 3,6 3,2 3,25 3,7 3,28 3,5 3,5 3,28 3,2 3,4 3, 2,69 2,9 3,9 3,9 2,98 2,98 3,4 3,6 3,7 3,37 3,3 3,49 3,49 3,35 3,35 Iris - 4, 6 and 8 epocs , 7,97 7,2 6,33 9,2 7,27 5,7 5,23 5,7 4,4 4,9 5,3 4,57 4, 4,23 4,3 2,4 7,7 6,63 4,6 4,63 4,24 4,8 3,67 4,5 3,87 4,3 7,83 3,73 4,8 4,23 4,23 2,8 6,77 4,43 4,77 3,83 4,7 5,87 4,67 4,27 4,7 4,3 4,6 4,5 3,77 3,57 4, 3,2 7,37 4,87 4, 4,4 4,4 8,4 4, 3,93 4,27 3,8 5,27 4,4 3,73 3,83 3,93 3,6 4,7 6,67 4,67 4, 4,3 6,37 5, 3,97 3,97 4,3 4,57 4,7 4,63 4, 4, 4, 5,5 4,3 4,83 5,5 4,53 6,6 3,97 3,97 3,87 4,53 5,57 5,73 3,83 3,5 3,8 4,4 4,8 4,2 4,37 4,73 4,23 6,6 4,47 3,8 4,2 4,37 7,3 4,23 3,67 4,3 4,57 4,8 5,5 4, 4,33 4,57 4,37 6,37 3,8 4,4 4,5 4,7 5,7 4,53 4,3 4,2 4, 5,2 7,4 4,67 4,63 4,97 4,5 4,6 4,23 4,7 4,3 4, 6, 4,53 4,53 3,9 3,87 5,6 5,54 3,9 4,2 4,47 4,27 5,4 4,53 4,2 4,4 4, 4,7 4,3 4,57 4, 3,73 6, 5, 4,37 4,7 4, 4,53 5,7 4,3 4,43 4,3 4, 5, 5, 4,3 3,83 3,8 Wine - 4, 6 and 8 epocs ,4 46,4 34,92 29,3 22,89 7,89 8,6 7,75 6,77 2,25 3,79 2,3 2,8 2,6 2,56 2,3 2,92 2,67 2,56 2,39 2,67 2,39,8 8,54 2,6 2,28 2,28 2,28 2,53 2,45 2,6 2,64 2,33 2,6 2,39 2,5 2,36 2,47 2,64 2,42 2,75 2,64 2,95 2,59 2,2 2,28 2,44 2,9 2,56 2,5 2,67 2,36 2,95 2,28 2,42 2,56 2,6 2,59 2,3 2,53 2,64 2,6 2,73 2,59 2,28 2,64 2,6 2,3 2,84 2,25,94 2,36 2,42 2,33 2,28 2,39 2,9 2,4 2,36 2,33 2,28 2,47 2,8 2,64 2,78 2,47 2,64 2,89 3, 2,47 2,42 2,36,97 2,39 2,3 2,22 2,42 2,39 2,44 2,8 2,56 2,28 2,22 2,8 2,56 2,42 2,84 2,6 2,87 2,3 3,4 2,33 2,59 2,9 2,5 2,78 2,8,94 2,5 2,22 2,89 2,22 2,3 2,6 2,45 2,7 2,33 2,44 2,42 2,25 2,47 2,42 3,8 2,64 2,59 2,4 2, 2,3 2,3 2,33 2,5 2,39 2,64,94 2,42 2,64 2,22 2,8 2,59 2,84 2,7 2,78 2,36 2,75 4,2 2,4 2,28 2,22 2, 2,64,94 2,42 2,45 2,36,88 2,75 2,45 2,9 2,44 2,78,88 2,67 2,36 2,87 2,3 2,53 4,6 2,56 2,28 2,6 2,39 2,,83,9 2,47 2,3 2,78 2,39 2,6 2,28 2,6 3, 2,36 2,5 2,95 2,39 2,89 2,87 5, 2,5,86 2,42 2,6 2,22,97 2,2 2,89 2,53 2, 2,36 2,7 2,39 2,28 3,6 2,84 2,6 2,44 2,2 2,42 2,7 Pb2-2, 25 and 3 epocs ,6 24,9 6,7 3,26 4,46 4,7 4,62 22,29 5,7 2,76,2,65,23 8,73 2,,63,78,68 9,3,8 2,25,53 8,9 9,29 8,88 8,2 22,89 7,8 8,5 8,29 7,76 8,36 25,8 8, 7,5 7,89 8,39 8,25 2, 25,49 9, 7,82 8, 8,5 8,92 24,94 7,65 7,82 8,4 7,64 9,8 25,7 8,36 8,43 7,55 8,8 8,7 2,2 28,72 8,79 8,66 8,49 8,85 9,3 28,46,35 8,24 7,8 7,58 8,92 27,2 9,4 8,7 7,94 8,5 7,6 2,4 28,64,65 8,77 9,8 8,35 9,88 3,83 2,38 9,6 8,53 8,2 8,8 29,65,77 8,4 8,7 9,76 7,76 2,6 28,94 9,73 9,9 9,34 9,7 8,73 3,3 3,22 8,77,58 9,8 9,26 29,3,78 9,43 7,72 8,3 8,9 2,8 28,77,99,28 9,36 9,6 9,36 28,75,97 8,9 8,22 9,89 9,38 28,63,2 8,77 9,83 9,37 8,82 3, 29,35,5,37 9,5 8,3 9,89 28,29,8 9,6,43,5 8,63 28,3,55 7,55 8,26 8,5 8,89