Nonlinear Regression using Particle Swarm Optimization and Genetic Algorithm

Transcription

1 International Journal Computer Applications (5 ) Volume 153 No, November 1 Nonlinear Regression using Particle Swarm Optimization and Genetic Pakize Erdoğmuş Düzce University, Computer Engineering Department, DUZCE Simge Ekiz Düzce University, Computer Engineering Department, DUZCE ABSTRACT Nonlinear regression is a type regression which is used for modeling a relation between the independent variables and dependent variables. Finding the proper regression model and coefficients is important for all disciplines. In this study, it is aimed at finding the nonlinear model coefficients with two wellknown populationbased optimization algorithms. Genetic s (GA) and Particle Swarm Optimization (PSO) were used for finding some nonlinear regression model coefficients. It is shown that both algorithms can be used as an alternative way for coefficients estimation nonlinear regression models. Keywords Nonlinear regression, Genetic, Particle Swarm Optimization 1. INTRODUCTION Regression is predicting the coefficients for a function. So it is known as function estimation or approximation. Depending on the relationship between dependent variables and independent variables, a regression problem can be two types. There can be linear regression function as well as nonlinear regression function (Kumar, 15). Nonlinear Regression based prediction has already been successfully implemented in various areas scientific research and technology. It is mostly used for function estimation or solutions that enable modeling a dependence between two variables (Akoa, Simeu,& Lebowsky, 13). Lu, Sugano, Okabe, and Sato (11) used Adaptive Linear Regression in order to estimate gaze coordinates and sub pixel eye alignment. Martinez, Carbone, and Pissaloux (1) realized gaze estimation using local features and nonlinear regression. In another study, nonlinear regression analysis technique is used to develop a model for prediction the flashover voltage ceramic insulators (porcelain, flashover voltage nonceramic insulators (NCIs) (Venkataraman & Gorur, ). Nonlinear regression is applied to Internet traffic monitoring by Frecon et al. (1). A nonlinear regression procedure, based on an original branch and bound resolution procedure, is devised for the full identification bivariate OfBm. The Nonlinear regression model is predicted in some ways. One the solution methods is the linear model approximation. But converting nonlinear regression model to linear is not possible for all the models. So there are some methods for the nonlinear regression model. One the solution methods for nonlinear regression is an iterative estimation the parameters like GaussNewton (Gray, Docherty, Fisk, & Murray, 1). The main idea is the minimization the nonlinear least squares. So, nonlinear regression is an optimization problem. Not only some classic optimization methods but also some heuristic algorithm can be used to find the optimum parameters the nonlinear regression model. The disadvantage the classic optimization method like GaussNewton can be trapped local minimal (Lu, Yang, Qin, Luo, Momayez, 1). So; heuristic methods can be an alternative for the nonlinear regression parameter estimation. In this study, two wellknown populationbased algorithms are used to find the optimum parameters the nonlinear regression model. The rest the paper is organized as follows; in the following section, Nonlinear Regression is explained. In the third section, Nonlinear Regression with GA and PSO for some test problems are applied. In the fourth section, experimental results with comparisons are presented. Finally, conclusions about the results are given.. NONLİNEAR REGRESSION For a scientific study, the first step is modeling the variation between the dependent values and independent values. After decided which model describes the variation best, parameter estimation the model is made. With the development computer technology, it has been developed some algorithms for the estimation the models' parameters. Regression model can be linear or nonlinear. Unlike linear regression, there are a few limitations a nonlinear regression model. The way in which the unknown parameters in the function are estimated, however, is conceptually the same as it is in linear least squares regression ( Nonlinear Least s Regression, n.d.). A nonlinear model can be a basic form, given by Equation (1). Y=f(x,A)+e; (1) The function in Equation (1) is nonlinear and has some parameters given with A=(A1,A,..). The aim is to estimate these parameters given with A. Some the methods for the nonlinear regression are GaussNewton and Levenberg Marquardt methods. Both methods are based on the minimization the nonlinear least squares. The GaussNewton method is a simple iterative gradient descent optimization method. The iterative gradient method is used to modify a mathematical model by minimizing the leastsquares between the modeled response and observed values (Minot, Lu, & Li, 15). But this method requires starting values for the unknown parameters. So the performance the method is related to the initial values the parameters. Bad starting values can cause the convergence to the local minimum as it is seen most the classical optimization methods. In comparison to gradientbased methods, Newtontype methods are advantageous with respect to convergence rate, which is usually quadratic. The

2 difficulty is that Newtontype methods require solving a matrix inversion at each iteration (Minot et al., 15). The LevenbergMarquardt algorithm is an iterative technique that locates a local minimum a multivariate function that is expressed as the sum squares several nonlinear, realvalued functions. It has become a standard technique for nonlinear leastsquares problems, widely adopted in various disciplines for dealing with datafitting applications. LevenbergMarquardt algorithm solves nonlinear least squares problems by combining the steepest descent and GaussNewton method. It introduces a damping parameter ʎ into the classical GaussNewton algorithm (Polykarpou, & Kyriakides, 1). When the parameters are far from the optimal, the damping factor has larger values and the method acts more like a steepest descent method, but guaranteed to converge. The damping factor has small values when the current solution is close to the optimal and acts like a Gauss Newton method (Lourakis, & Argyros, 5). The algorithms GausNewton and LevenbergMarquardt Methods are given in Figure 1(Basics on Continuous Optimization, 11). International Journal Computer Applications (5 ) Volume 153 No, November 1 crossover, and mutation are applied to the population. The pseudo code GA is given Figure. The advantage GA is that it is popular and applied successfully nearly in every area. Because GA operators, sometimes it can t be practical to use GA for the solution an optimization problem. Create initial population and calculate fitness values do Select best individuals for the next generation Apply elitism Apply crossover Apply mutation until terminating condition met Figure. The pseudo code GA 3. Particle Swarm Optimization PSO is one the successfully studied populationbased heuristic search algorithm inspired by the social behaviors flocks (Bamakan, Wang, & Rayasan, 1). In PSO, each solution called particle. Particles consist the swarm. Figure 1. GaussNewton and Levenberg Marquardt algorithms for nonlinear regression METHODS GaussNewton LevenbergMarquardt f: R n R, f x = m i=1 (f i (x)) R n to R x () an initial solution x, a local minimum the cost function f. begin k ; while STOP CRIT and k < k max do x k+1 x k + δ k ; wit δ (k) =arg min δ F(x (k) ) + J F x k δ ; k k + 1; return x k end f: R n R, f x = m i=1 (f i (x)) R n to R x () an initial solution x, a local minimum the cost function f. begin k ; λ max diag(j T J) ; x x () ; while STOP CRIT and k < k max do Find δ suc tat J T J + λdiag J T J δ = J T f; x x + δ; if f x < f x then x x ; λ λ v ; else λ vλ; k k + 1; return x 3. NONLİNEAR REGRESSION WITH GA AND PSO 3.1 Genetic GA is one the most studied evolutionary computation technique since David Goldberg was firstly published (Holland, 15; Ijjina, and Chalavadi, 1). GA is a populationbased heuristic search algorithm inspired by the theory evolution. In the algorithm, best properties are transferred from the generation to generation with crossover and elitism. starts some initial random solutions the optimization problem called individuals. Each individual consists the variables the optimization problem called chromosome. GA uses some genetic operators such as crossover, mutation, and elitism in order to find the optimum solution. In each generation, fitness function values for each individual are calculated. The best individuals are selected for the next generation by some methods such as tournt selection or roulette wheel. After the selection, elitism, In PSO, the algorithm starts with initial solutions. Each initial solution represents the coordinate a particle. Particles starts fly to hyperspace. Particles adapt their velocities according to social and individual information. After the iterations, particles coordinates converge to the best particle coordinate which is the global optimum solution (Liu,& Zhou, 15). PSO has quite simple and fast converging algorithm. There is no operator. There are two important formulas in PSO. Particles move according to this formula given in () and (3) (Cavuslu, Karakuzu, & Karakaya, 1). v i k+1 = K v i k + φ 1 rand k p best x i k k + φ rand g best x i k x i k+1 = x i k + v i k+1 (3) PSO has little parameters. The constriction factor(k) is a damping effect on the amplitude an individual particle s ()

3 International Journal Computer Applications (5 ) Volume 153 No, November 1 oscillations. 1 and represent the cognitive and social parameters, respectively. Rand is random number uniformly distributed. P best i k is the best position for the i.th particle at the k.th iteration, g best is the global best position, x i k, v i k are the position and velocity the i.th particle at the k.th iteration respectively. Although PSO is a populationbased algorithm, it has many advantages such as simplicity, little parameters to be adjusted and rapid convergence. The pseudo code PSO is given Figure 3. P is the number particles. Figure 3. The pseudo code PSO Generate initial swarm(p) do for i=1:p update local best update global bet update velocity and location end until stopping criteria met problems. The average and standard deviation the estimated parameter is given in the tables. So the given results are accepted the real solution. The results found with GA and PSO are compared with these results. Table 1. Nonlinear Regression Test Problems Problem Number Name Model 1 Chwirut1 y = f(x i β)+ = e β 1x β +β 3 x Chwirut y = f(x i β)+ = e β 1x β +β 3 x + 3 Gauss1 y = f(x i β)+ = β 1 e β x +β 3 e (x β 4) /β 5 + β e (x β ) /β + 4 Nelson log y = f(x i β)+ = β 1 β x 1 e β 3x + 5 Kirby y = f(x i β)+ = β 1+ β x+ β 3 x 1+ β 4 x+ β 5 x + Gauss3 y = f(x i β)+ = β 1 e β x + β 3 e (x β 4) /β 5 + β e (x β ) /β + ENSO y = f x i β + = β 1 + β cos πx/1 + β 3 sin πx/1 β 5 cos πx/β 4 + β sin πx/β 4 + β cos πx/β + β sin πx/β + Thurber y = f x i β + = β 1 + β x + β 3 x + β 4 x β 5 x + β x + β x 3 + Rat43 y = f x i β + = β 1 (1 + e β β 3 x ) 1/β Bennett5 y = f x i β + = β 1 + (β + x) 1/β Optimization the parameters Nonlinear Regression model with GA and PSO In this study, some nonlinear regression problems are selected for the testing the performance GA and PSO with classical nonlinear methods ( Nonlinear Least s Regression, n.d.). The nonlinear models the problems are given in Table 1. Some initial starting values for the nonlinear regression models are also given. The difficulty levels the problems, the number parameters and models classification are given in Table. The estimation the parameters found classical nonlinear regression analysis are given with the 3

4 International Journal Computer Applications (5 ) Volume 153 No, November 1 Table. Properties Nonlinear Regression Test Problems Problem Number Name Difficulty Level Model Classification Number Parameter /Number Observation 1 Chwirut1 Lover Exponential 3/14 Chwirut Lover Exponential 3/54 3 Gauss1 Lover Exponential /5 4 Nelson Average Exponential 3/1 5 Kirby Average Rational 5/151 Gauss3 Average Exponential /5 ENSO Average Miscellaneous /1 Thurber Higher Rational /3 Rat43 Higher Exponential 4/15 1 Bennett5 Higher Miscellaneous 3/154 The object function in this study is the difference between the real values and the calculated values with the estimated models. There is no constrained. So the problem is unconstrained optimization problem. The aim is to minimize the squares the object function given in Equation (4). Fobj= m i=1 (Y i f i (x)) (4) GA and PSO parameters used in this study are given Table 3. GA PopulationSize:1 Generations:1 CrossoverFraction:. Table 3. GA and PSO parameters PSO SwarmSize:1 SelfAdjustment:1.4 SocialAdjustment:1.4 EliteCount:1 Max iteration:*number variable SelectionFcn:Roulette 4. EXPERIMENTAL RESULTS AND ANALYSIS The selected nonlinear regression problems are solved with GA and PSO. In this study, codes are written by Matlab with Intel Core Duo 3.GHz processor, 4bit Windows version operating system. Results are given in Table 4 Table 13. As it has seen in the tables, the parameters found GA are mostly nearer to nonlinear regression parameters. PSO solution absolute errors have been compared with GA solution absolute errors. As it has been seen in Table 4(Chwirut1), Table 5(Chwirut), Table 1(ENSO), Table 11(Thurber), Table 1(Rat43) and Table 13(Benett5), results found with GA is nearer to real values found Nonlinear Regression. GA outperforms PSO in view the solution time also. 5. CONCLUSIONS As it is known, nonlinear regression with classical methods like GaussNewton and LevenbergMarquardt has some disadvantages. The first disadvantage the classical methods is that they require a lot mathematical operations. Matrix operations, gradient operation and Jacobean matrix calculation and some other mathematic operators have been required both GaussNewton and LevenbergMarquardt methods. Another disadvantage is that classic methods like GaussNewton can be trapped local minima. And the convergence to the local minima can be too slow. So the number iteration for the minimization the nonlinear least squares can be timeconsuming. Both classical methods require starting values for the unknown parameters. So the performances the methods are related to the initial values the parameters. Bad starting values can cause the convergence to the local minimum as it is seen most the classical optimization methods. So in order to overcome these difficulties, heuristic search algorithms are suggested as an alternative. In this study, the nonlinear least squares problems were solved with the same starting values given in the reference. According to the reference web site, reported results for nonlinear regression were confirmed by at least two different algorithms and stware packages using analytic derivatives. Results prove that GA and PSO are good alternatives for the classical nonlinear least squares regression. But GA is more successful in view parameters estimation. For future studies, it is aimed to test the classical methods like GaussNewton with some heuristic optimization algorithms and show the performances the methods in view both solution time and optimal values.. REFERENCES [1] Kumar, T. (15, February). Linear and Non Linear Regression Problem by K Nearest Neighbor Approach: By Using Three Sigma Rule. Computational Intelligence & Communication Technology (CICT), pp. 11. doi: 1.11/CICT [] Akoa, B. E., Simeu, E., and Lebowsky, F. (13, July). Video decoder monitoring using nonlinear regression. 13 IEEE 1th International OnLine Testing Symposium (IOLTS), pp doi: 1.11/IOLTS.13.43G. [3] Lu, F., Sugano, Y., Okabe, T., and Sato, Y. (11, November). Inferring human gaze from appearance via adaptive linear regression. 11 International Conference on Computer Vision, pp doi: 1.11/ICCV [4] Martinez, F., Carbone, A., and Pissaloux, E. (1, September). Gaze estimation using local features and nonlinear regression. 1th IEEE International 31

5 [5] Conference on Image Processing, pp doi:.11/icip.1.41 [] Venkataraman, S., and Gorur, R. S. (). Non linear regression model to predict flashover nonceramic insulators. 3th Annual North American Power Symposium, NAPS, pp. 3.doi: 1.11/NAPS [] Frecon, J., Fontugne, R., Didier, G., Pustelnik, N., Fukuda, K., and Abry, P. (1, March). Nonlinear regression for bivariate selfsimilarity identification application to anomaly detection in Internet traffic based on a joint scaling analysis packet and byte counts. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp [] Gray, R. A., Docherty, P. D., Fisk, L. M., and Murray, R. (1). A modified approach to objective surface generation within the GaussNewton parameter identification to ignore outlier data points. Biomedical Signal Processing and Control, 3, pp [] Lu, Z., Yang, C., Qin, D., Luo, Y., and Momayez, M. (1). Estimating ultrasonic timeflight through echo signal envelope and modified Gauss Newton method. Measurement, 4, pp [1] Nonlinear Least s Regression. (n.d.). Engineering Statistics Handbook. Retrieved from md14.htm [11] Minot, A., Lu, Y. M., and Li, N. (15). A distributed GaussNewton method for power system state estimation. IEEE Transactions on Power Systems, 31(5), pp doi: 1.11/TPWRS [1] Polykarpou, E., and Kyriakides, E. (1, April). Parameter estimation for measurementbased load modeling using the LevenbergMarquardt. APPENDIX International Journal Computer Applications (5 ) Volume 153 No, November 1 algorithm. Electrotechnical Conference (MELECON), 1 1th Mediterranean, pp. 1. doi: 1.11/MELCON [13] Lourakis, M. L. A., and Argyros, A. A. (5, October). Is LevenbergMarquardt the most efficient optimization algorithm for implementing bundle adjustment?. Tenth IEEE International Conference on Computer Vision (ICCV'5, 1, pp ). doi: 1.11/ICCV.5.1. [14] Basics on Continuous Optimization, (11, July). Retrieved from [15] Holland, J. H. (15). Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence. U Michigan Press, Retrieved from AJ&redir_esc=y [1] Ijjina, E. P., and Chalavadi, K. M. (1). Human action recognition using genetic algorithms and convolutional neural networks. Pattern Recognition,5, pp [1] Bamakan, S. M. H., Wang, H., and Ravasan, A. Z. (1). Parameters Optimization for Nonparallel Support Vector Machine by Particle Swarm Optimization. Procedia Computer Science, 1, pp [1] Liu, F., and Zhou, Z. (15). A new data classification method based on chaotic particle swarm optimization and least squaresupport vector machine. Chemometrics and Intelligent Laboratory Systems, 14, pp [1] Cavuslu, M. A., Karakuzu, C., and Karakaya, F. (1). Neural identification dynamic systems on FPGA with improved PSO learning. Applied St Computing, 1(), pp.1. Table 4. The Chwirut1 parameters estimation with GA, PSO and classical methods Chwirut1 Avg.,1,13,1,1,11,14,13 5,4E5 1.13E E,5,3,,3E,1,1,1,13E E E4,1,113,1 1,3E 5,15,15,15,E 1.533E.144E4 34,4 34,1 3,1 34,4 34,4 34,4, ,15,1,5 4,4,3,3,44,343,54 3

6 International Journal Computer Applications (5 ) Volume 153 No, November 1 Table 5. The Chwirut parameters estimation with GA, PSO and classical methods Chwirut Avg.,15,1,155,11,15,1,1,E E E,5,55,5,,5,5,5 4,1E E3.151E4,11,1,1,5,11,1,11 1,1E 1.15E E3 513,5 51, 513,55, 513,4 513, ,4 1,4E5 1,33 1,,51,433,4,431,354,53 Table. The Gauss1 parameters estimation with GA, PSO and classical methods, 1, Gauss1,,4 1, 1, 1,, Avg..11E E1,,1,,5,,,, 1.451E 1.141E4,,3,,1,,,, 1.433E E1 1, 1,1 1, 4,5 1, 1, 1,,.41111E E1 Parameter5, 5, 3,53,, 5, 4,,5.3133E E1 Parameter,,,,,,,,.14534E+1.313E1 Parameter 14, 15, 14,, 15, 15, 15,, 1.51E E1 Parameter 15, 15,1 15,3,5 15, 15, 15,, 1.335E E1 43, , , 31,44 43, 44, 43, 44 5,53 1,41 4,33,541,3,1,43,34,5 33

7 International Journal Computer Applications (5 ) Volume 153 No, November 1 Table. The Nelson parameters estimation with GA, PSO and classical methods Nelson Avg.,,1,454,13,,5,31,141,E,13,5E 14,1 1,1E 5 4,34E,3,33,,1,4,44,,34,5,,1,143 3,E,E 3,35E 5 1,15E,14E 3,35E,4 1,11,45,55,13,35,13,1.531E+ 5.11E E E.11454E E3 Table. The Kirby parameters estimation with GA, PSO and classical methods. Kirby 1, 1,51 1,14,3 1, 1, 1, Avg.,5,43,,,,1,5,41 1,445E Parameter5,, 5,45E 5,5E,,,,,5,51,5,4,54,51,54 5,4 1,34E 5 1, 1,11E 5, 1,E 4445, 1,,, 11,35 34,3 5 5,4 11,14E E+.3433E 1.33E E E E5 5,53E E E5 3,44E1.145E5.111E 1, 1,3 14,5 1,3 3,3,153,43,,53 Table. The Gauss3 parameters estimation with GA, PSO and classical methods Gauss3 Avg.,11,5,1,131,1,45,,11,15,11,1,,1,1,1, 1,,,1 1, 1,.43E E1,5E E E4 1, 3,E 1.553E E1 111,14 111,4 111,45,14 111,43 111, ,441 4, E E1 34

8 International Journal Computer Applications (5 ) Volume 153 No, November 1 Parameter5 3,4 3,545 3,14,543 3,1535 3,14 3,15,1.335E E1 Parameter 3,444 4, 3,,414 4,1 4,14 4,13, E E+ Parameter 14,33 14, 14,54,1 14,51 14,54 14,5, E E1 Parameter 1,4,55 1,131,5 1,3 1,3 1,, E E1 14, ,5 3 1, 4 1, 14,4 5 14,4 3 14,4,3,53 11,33,55,315,1 1,,,1 Table 1. The ENSO parameters estimation with GA, PSO and classical methods ENSO Avg. 1,55 1,5154 1,515,1 1,513 1,511 1,51, E E1 3,4 3, 3,,11 3,5 3,5 3,, 3.15E E1,5314,534,53,,53,5334,53, 5.313E E1 44,55 44,41 44,3,4 44,31 44,311 44,31, E+1.445E1 Parameter5 1,31 1, 1,3,5 1,3 1, 1,, E+.311E1 Parameter,4,553,5341,41,53,5,5, E E1 Parameter,4,1,4,,4,,,.1444E E1 Parameter,13,5,15,11,1,141,15,.134E E1 Parameter 1,4 1,4 1,41,4 1,4 1,4 1,4, 1.441E E1,53,3,551,1,53,54 5,53,1,3511,14 3,41,4 1,31, 1,,314 Table 11. The Thurber parameters estimation with GA, PSO and classical methods 1,5 5 13, Thurber 1, 3 3,543 Avg. 1,4 4 13, 1,,5 1.13E E+ 113, , 3 14,1,345 1,31 14, , 3 1, E E+1 4,4 551,14 4,4 41, 4, 5,15 541,1 3 5, 5.333E+.1E+1 35

9 International Journal Computer Applications (5 ) Volume 153 No, November 1 45,1 4,3 1,411,311 4,,54,143 14, E E+ Parameter5,4,33,45,3,45,5,3,4.54E E Parameter,3,431,33,1,31,41,31,3 3.5E E Parameter,,41,14,11 1,5E 5,5 3433, 14 13, 55,4 4 55,5 5,5,4,14 114, 414 4, ,115 13,11,5 14,553,,51 1,51 4,55, E E3 Table 1. The Rat43 parameters estimation with GA, PSO and classical methods,1, 1 Rat43,5,3 Avg.,,,43,5.4151E+ 1.31E+1 5,15 5,55 5,,14 5,31 5,43 5,5, E+.35E+,543,43,,1,5,11,5,.533E E1 1, 1,5 1,443,41 1, 1,311 1,4, 1.435E+.1335E1,41 3, 53 3, 4 4,15,4 5,1 55,1 1,53 1,54 1,41,13 3,5,1 1,131,51,53 Table 13. The Benett5 parameters estimation with GA, PSO and classical methods 5,1 44 1,4 4 Bennett5 35, 4 44,1 Avg. 3, 15,3 3, 3 4, E E+ 3,5 4,15 43,51 4,3 4,4 4,55 45,1 1, E E+, 1,,3,34,35 1,,51, E1.3E, 4,4,35 1,1,5,3,5,4,44 14,5,134 5,55, 1,3,44,13 IJCA TM : 3