Multilayer Perceptron Neural Network (MLPs) For Analyzing the Properties of Jordan Oil Shale

Size: px

Start display at page:

Download "Multilayer Perceptron Neural Network (MLPs) For Analyzing the Properties of Jordan Oil Shale"

Darcy Robbins
6 years ago
Views:

1 World Alied Sciences Journal 5 (5): , 2008 ISSN IDOSI Publications, 2008 Multilayer Percetron Neural Network (MLPs) For Analyzing the Proerties of Jordan Oil Shale 1 Jamal M. Nazzal, 2 Ibrahim M. El-Emary and 3 Salam A. Najim 1,3 Al Ahliyya Amman University, P.O. Box 19328, Amman, Jordan 2 King Abdulaziz University, P.O. Box 18388, Jeddah, King Saudi Arabia Abstract: In this aer, we introduce the multilayer recetron neural network and describe how it can be used for function aroximation. The back roagation algorithm (including its variants) is the rincile rocedure for training multilayer ercetrons. Car must be taken when training ercetron network to ensure that they do not over fit the training data and then fail to generalize well in new situations. So the main urose of this aer lies in studying the effect of changing both the No of hidden layers of MLPs and the No of rocessing elements that exist in the hidden layers on the analyzed roerties of Jordan Oil Shale. After constructing such a MLP and changing the number of hidden layers, we found that with increasing the number of rocessing elements in the hidden layers. We reach to an otional outut results w.r.t the exerimental one or the analytical formulated one. This obtained outut is comletely matched with the main concets of theoretical visions of ANN. Key words: MLP El-lojjin oil sigmoid ercetron DM-1 Dm-2 INTRODUCTION A geochemical analysis of El-Lajjun oil shall in Jordan was carried out [1]. It was found that El-Lajjun oil shale consists of the following grou: Organic matter, biogenic calcite and aatite, detrital clay minerals and quartz. The calorific values of 100 samles were determined. The effect of bore deth, calcium carbonate, organic carbon and sulfur content on the calorific values was studied [1]. The results were well correlated by some emirical formula given by [1]: Calorific value= (CaCo 3 ) (S) (C org ) (1) With correlation coefficient of and with an average standard error of 2.63%. The current resurgence of interest in ANNs is largely due to the emergence of owerful new methods as well as to the availability of comutational ower suitable fir simulation. The field is articularly exciting today because ANN algorithms and architectures can be imlemented in VLSI technology for real time alications [2]. Currently two rincial streams can be identified in ANN research. Researches in the first stream are concerned with modeling the brain and thereby exlain its cognitive behavior. On the other hand, the rimary aim of researches in the second streams is to construct useful comuters for real world roblems of attern recognition by drawing these rinciles. In this aer we want to give a comlete descrition to neural networks and their alication in analyzing and studying the roerties of Jordan oil shale as a recent technique. At the same time, we would like to investigate the effect of changing the number of rocessing elements that exist in the hidden layer of the ANN on the forecasted arameters. To check the effectiveness of the described ANN, a comarative study was done between the estimated arameters by ANN and that one that is obtained through exeriment or calculated using the formula given in Eg.1. In this aer, we concentrate on the most common neural network architecture called the multilayer ercetron MLPs. For the urose of this aer, we will look at neural network as function aroximators [3]. As shown in Fig. 1, we have some unknown function that we wish to aroximate. We want to adjust the arameters of the network so that it will roduce the same resonse as the unknown function, if the same inut is alied to both systems. For our alication, the unknown function may corresond to a system (oil shale) we are trying to control as a result of analysis, in which case the neural network will be the identified lant model. The unknowing function could also reresent the inverse of a system (oil shale) we are trying to control and analyze, in which case the neural network can be used to imlement the controller. This aer is organized from six sections. In section two, we describe the Corresonding Author: Dr. Jamal M. Nazzal, Al Ahliyya Amman University, P.O. Box 19328, Amman, Jordan 546

2 World Al. Sci. J., 5 (5): , 2008 Inut Unknown Function Neural Network Predicted Outut Error Adatation Outut Fig. 1: Neural network as function aroximator neuron model. Section three was devoted to Multilayer Percetion Architecture. Section five discusses a lot of obtained results. Finally section six resents the results, conclusion and future works done by others. Section four was dedicated to describe the oeration of Multilayer Percetion Neural Networks. NEURON MODEL The multilayer ercetion neural network is built u of simle comonents. In the beginning, we will describe a single inut neuron which will then be extended to multile inuts. Next, we will stack these neurons together to roduce layers [4]. Finally, the layers are cascaded together to form the network. Single-inut neuron: A single-inut neuron is shown in Fig. 2. The scalar inut is multilied by the scalar weight W to form W, one of the terms that is sent to the summer. The other inut, 1, is multilied by a bias b and then assed to the summer. The summer outut n often referred to as the net inut, goes into a transfer function f which roduces the scalar neuron outut a (sometimes "activation function" is used rather than transfer function and offset rather than bias). From Fig. 2, both w and b are both adjustable scalar arameters of the neuron. Tyically the transfer function is chosen by the designer and then the arameters w and b will be adjusted by some learning rule so that the neuron inut/outut relationshi meet some secific goal. The transfer function in Fig. 2 may be a linear or nonlinear function of n. A articular transfer function is chosen to satisfy some secification of the roblem that the neuron is attemting to solve. One of the most commonly used functions is the log-sigmoid transfer function, which is shown in Fig. 3 [4]. This transfer function takes the inut (which may have any value between lus and minus infinity) and Fig. 2: Single inut neuron [4] Fig. 3: Log-sigmiod transfer function squashes the outut into the range 0 to 1 according to the exression: 1 a = (2) n 1 + e The log-sigmoid transfer function is commonly used in multi-layer networks that are trained using the back roagation algorithm. Multile-inut neuron: Tyically, a neuron has more than one inut. A neuron with R inuts is shown in Fig. 4. The individual inuts 1, 2,, g are each 547

3 World Al. Sci. J., 5 (5): , 2008 dimensions of are dislayed below the variable as Rx1, indicating that the inut is a single vector of R elements. These inuts go to the weight matrix W, which has R columns but only one row in this single neuron case. A constant 1 enters the neuron as an inut and is multilied by a scalar bias b. The net inut to the transfer function f is n, which is the sum of the bias b and the roduct W. The neuron's outut is a scalar in this case. If there exit more than one neuron, the network outut would be a vector. Fig. 4: Multile-inut neuron MULTILAYER PERCEPTRON NETWORK ARCHITECTURES Commonly one neuron, even with many inuts, may not be sufficient. We might need five or ten, oerating arallel, in what we will call a layer". This concet of a layer is discussed below. Fig. 5: Neuron with R inuts, abbreviated notations weighted by corresonding elements W 1,1 W 1,2,.W 1,R of the weight matrix W. The neuron has a bias b, which is summed with the weight inuts to form the net inut n: n = W1,11 + W1, W1,R R + b (3) This exression can be written in matrix form as: A layer of neurons: A single-layer network of S is shown in Fig. 6. Note that each of the R inuts is connected to each of the neurons and that the weight matrix now has s rows. The layer includes the weight matrix, the summers, the bias vector b, the transfer function boxes and the outut vector a. Each element of the inut vector is connected to each neuron through the weight matrix W. Each neuron has a bias b i, a summer, a transfer function f and an outut a i. Taken together, the oututs form the outut vector a. It is common for the number of inuts to a layer to be different from the number of neurons (i.e. R S). The inut vector elements enter the network through the weight matrix W: n = W + b (4) Where the matrix W for the single neuron case has only one row. Now the neuron outut can be written as: a = f(w+ b) (5) A articular convention in assigning the indices of the elements of the weight matrix has been adoted [4]. The first index indicates the articular neuron destination for the weight. The second index indicates the source of the signal fed to the neuron. Thus, the indices in W 1,2 say that this weight reresents the connection to the first (and only) neuron from the second source [4]. A multile-inut neuron using abbreviated notation is shown in Fig. 5. As shown in Fig. 5, the inut vector is reresented by the solid vertical bar at left. The Fig. 6: Layer of S neurons 548

4 World Al. Sci. J., 5 (5): , 2008 Fig. 7: Layer of S neurons, abbreviated notation W... W 1,1 1,R W = (6) W... W s,1 S,R The row indices of the elements of matrix W indicate the destination neuron associated with that weight, while the column indices indicate the source of the inut for that weight. Thus, the indices in W 3,2 say that this weight reresents the connection to the third neuron from the second source. The S-neuron, R-inut, one-layer network also can be drawn in abbreviated notation as shown in Fig. 7. Here again, the symbols below the variables tell that for this layer, is a vector of length R, W is an S*R matrix and a and b are vectors of length S. As defined reviously, the layer includes the weight matrix, the summation and multilication oerations, the bias vector b, the transfer function boxes and the outut vector. Multile layers of neurons: Now consider a network with several layers which has been imlemented in this aer for the urose of analyzing Jordan oil shale roerties. In this network each layer has its own weight matrix W, its own bias vector b, a net inut vector n and an outut vector a. Some additional notation should be introduced to distinguish between these layers. Suerscrits are used to identify these layers. The number of the layer as a suerscrit is aended to the names for each of these variables. Thus, the weight matrix for the second layer is written as W 2. This notation is used in the three-layer network shown in Fig. 8. As shown in Fig. 8, there are R inuts, S 1 neurons in the first layer, S 2 neurons in the second layer, etc. The outut of layers one and two are the inuts for layers two and three. Thus layer 2 can be viewed as a one-layer network with R=S 1 inuts, S= S 2 neurons and an S 1 *S 2 weight matrix W 2. The inut to layer 2 is a 1 and the outut is a 2. A layer whose outut is the network outut is called an outut layer. The other layers are called hidden layers. The network shown in Fig. 8 has an outut layer (layer3) and two hidden layers (layers 1 and 2) STRUCTURE AND OPERATION OF MULTILAYER PERCEPTRON NEURAL NETWORK (MLP) MLP neural networks consist if units arranged in layers [5]. Each layer is comosed of nodes and in the fully connected network considered here each node connects to every node in subsequent layers. Each MLP is comosed of a minimum of three layers consisting of an inut layer, one or more hidden layers and an outut layer. The inut layer distributes the inuts to subsequent layers. Inut nodes have liner activation functions and no thresholds. Each hidden unit node and each outut node have thresholds associated with them in addition to the weights. The hidden unit nodes have nonlinear activation functions and the oututs have linear activation functions. Hence, each signal feeding into anode in a subsequent layer has the original inut multilied by a weight with a threshold added and then Fig. 8: Three layer network 549

5 World Al. Sci. J., 5 (5): , 2008 Fig. 9: Tyical three layer multilayer ercetron neural network is assed through an activation function that may be linear or nonlinear (hidden units). A tyical three layer network is shown in Fig. 9. Only three layer MLPs will be considered in this work since these networks have been shown to aroximate any continuous function [6-8]. For the actual three layers MLP, all of the inuts are also connected directly to all of the oututs [5]. The training data consist of a set N V training atterns (x,t ) where P reresents the attern number. In Fig. 9, X P corresonds to the N-dimensional inut vector of the Pth training attern and Y P corresonds to the M-dimensional outut vector from the trained network for the P th attern. For ease of notation and analysis, threshold on hidden units and outut units are handled by assigning the value of one to an augmented vector comonent denoted by X (N+1). The outut and inut units have linear activations. The inut to the J th hidden unit, net P (j) is exressed [5] as: net (j) N 1 = + W (j,k)x (k) 1 j N k= 1 hi h (7) With the outut activation for the P th training attern, O (j), being exressed by: The overall erformance of the MLP is measured by the mean square error (MSE) exressed by : Where: 1 1 E = E = [t (i) y (i)] N N Nv Nv M 2 = 1 = 1 i= 1 (10) M 2 i= 0 (11) E = [t (i) y (i)] E Corresonds to the error for the P th attern and t is the desired outut for the P th attern. This is also allows the calculation of the naing error for the i th outut unit to be exressed by: 1 E [t (i) y (i)] M 2 i = 1 = (12) N v with the i th outut for the P th training attern exressed by: (13) N 1 N h = + k 1 oi + = j= 1 oi Y(i) W (i,k)x (k) W (i,j).o (j) O ( j) = f(net ( j)) (8) The nonlinear activation is tyically chosen to be the sigmoid function 1 f(net (j)) = (9) net(j) 1 + e In (7) and (8), the N inut units are reresented by the index K and W hi (J,K) denotes the weight connecting the K th inut unit to the J th hidden unit. In (13), W oi (i,k) reresents the weight from the inut nodes to the outut nodes and W oh (i,j) reresents the weight from the hidden nodes to the outut nodes. CONCLUSION When investigators design neural networks for the alication resented in this aer, there are many ways used to investigate the effects of network structure which refers to the secification of network size (i.e. number of hidden units) when the number of inuts and 550

6 World Al. Sci. J., 5 (5): , 2008 Fig. 10: CaCo3 versus gross calorific value Fig. 11: Corg versus gross calorific value Fig. 12: S versus gross calorific value 551

7 World Al. Sci. J., 5 (5): , 2008 Table 1: Results summary Performance MLP ANN 3:10:1 MLP ANN 3:20:1 MLP ANN 3:10:10:1 MLP ANN 3:20:10:1 MSE NMSE MAE Min Abs Error Max Abs Error r oututs are fixed (5) which in turn affects on the o/ redicted arameters. A well organized one maybe based on designing a set of different size networks in an ordinary fashion, each with one or more hidden units than the revious one. This aroach can be designated as design methodology one (DM-1). Alternatively, the second one is based on designing different size networks in no articular order. This aroach can be designated as design methodology two (DM-2). These two design aroaches are significantly different and the aroach chosen often is based on the goals set for the network design and the associated erformance. In general, the more thorough aroach often used for DM-l may take more time to develo the selected network since we may be interested in trying to achieve a trade-off between network erformance and network size. However, the DM-1 aroach used in this work may roduce a suerior design since the design can be ursued until we will be satisfied that future increases in network size roduces diminishing returns in terms of decreasing training time or testing errors. When we design MLP network to analyze the roerties of Jordan oil shale, we reach to the following facts: MLP with a lot of hidden layers (3) erforms more better than with one or two hidden layer secially regarding the outut erformance arameters. Also, when we comare between the o/ of MLP and the mathematical formula, we found that our o/ erformance arameter are best suited with the exerimental. As a future work, we recommend to use hybrid aroaches, which are a combination of ANN with other techniques like exert systems, Fuzzy logic and Genetic Algorithm (GA) to make such analysis of Jordan oil Shale roerties. REFERENCES 1. Anabtawi, M.Z. and jamal M. Nazzal, Effect of comosition of el-lajjun oil shale on its calorific value. Journal of Testing and Evaluation, : Nitin Malik, Artificial Neural Networks and their Alications. National conference on Unear thing Technological Develoments & their transfer for serving Masses, GLA ITM, Mathura, Mathura, India. 3. Martin, T. Hagan and Howard B. Demuth, Neural Networks for control, 4. Haykin, S., Neural Networks: A Comrehensive Foundation, 2nd Edn., New Jersey: Prentice-Hall. 5. Walter, H. Delashmit and Michael T. Manry, Recent Develoments in Multilayer Percetron Neural Networks. Proceedings of the 7 th Annual Memhis Area Engineering and Science Conference, MAESC Hornik, K., M. Stinchcombe and H., White, Multilayer feed forward Networks are Universal Aroximators, Neural Networks, 2 (5): 35g, Hornik, K., M. Stinchombe and H. White, Universal Aroximation of an unknown Maing and its Derivatives Using Multilayer Feed forward Networks. Neural Networks, 3 (5): Cybenko, G., Aroximation by suerosition of a sigmoidal function. Mathematics of control, Signal and Systems, 2 (4):

Recent Developments in Multilayer Perceptron Neural Networks

Recent Developments in Multilayer Perceptron Neural Networks Recent Develoments in Multilayer Percetron eural etworks Walter H. Delashmit Lockheed Martin Missiles and Fire Control Dallas, Texas 75265 walter.delashmit@lmco.com walter.delashmit@verizon.net Michael