THE PROBLEM of solving systems of linear inequalities

Transcription

1 452 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 46, NO. 4, APRIL 1999 Recurrent Neural Networks for Solving Linear Inequalities Equations Youshen Xia, Jun Wang, Senior Member, IEEE, Donald L. Hung, Member, IEEE Abstract This paper presents two types of recurrent neural networks, continuous-time discrete-time ones, for solving linear inequality equality systems. In addition to the basic continuous-time discrete-time neural-network models, two improved discrete-time neural networks with faster convergence rate are proposed by use of scaling techniques. The proposed neural networks can solve a linear inequality equality system, can solve a linear program its dual simultaneously, thus extend modify existing neural networks for solving linear equations or inequalities. Rigorous proofs on the global convergence of the proposed neural networks are given. Digital realization of the proposed recurrent neural networks are also discussed. Index Terms Linear equalities equations, recurrent neural networks. I. INTRODUCTION THE PROBLEM of solving systems of linear inequalities equations arises in numerous fields in science, engineering, business. It is usually an initial part of many solution processes, e.g., as a preliminary step for solving optimization problems subject to linear constraints using interior-point methods [1]. Furthermore, numerous applications, such as image restoration, computer tomography, system identification, control system synthesis, lead to a very large system of linear equations inequalities which needs to be solved within a reasonable time window. There are two classes of well-developed approaches for solving linear inequalities. One of the classes transforms this problem into a phase I linear-programming problem, which is then solved by using well-established methods such as the simplex method or the penalty method. These methods employ many of the matrix operations or have to deal with the difficulty in setting penalty parameters. The second class of approaches is based on iterative methods. Most of them do not need matrix manipulations the basic computational step in iterative methods is extremely simple easy to program. One type of iterative methods is derived from the relaxation method for linear inequalities [2] [6]. These methods are called relaxation methods because they consider one constraint at a time, so that in each iteration, all but one constraint is identified Manuscript received July 9, 1997; revised May 3, This work was supported in part by the Hong Kong Research Grants Council under Grant CUHK 381/96E. This paper was recommended by Associate Editor J. Zurada. Y. Xia J. Wang are with the Department of Mechanical Automation Engineering, Chinese University of Hong Kong, Shatin, NT, Hong Kong. D. L. Hung is with the School of Electrical Engineering Computer Science, Washington State University, Richl, WA USA. Publisher Item Identifier S (99) orthogonal projection is made onto the hyperplane corresponding to it from the current point. As a result, they are also called the successive orthogonal projection methods. Making an orthogonal projection onto a single linear constraint is computationally inexpensive. However, when solving a huge system which may have thouss of constraints, considering only one constraint at a time leads to slow convergence. Therefore, effective parallel solution methods which can process a group or all of constraints at a time are desirable. With the advances in new technologies [especially very large scale integration (VLSI) technology], the dynamicalsystems approach to solving optimization problems with artificial neural networks has been proposed [6] [16]. The neuralnetwork approach enables us to solve many optimization problems in real time due to the massively parallel operations of the computing units faster convergence properties. In particular, neural networks for solving linear equations inequalities have been presented in recent literature in separate settings. For solving linear equations, Cichocki Unbehauen [20], [21] first developed various recurrent neural networks. In parallel, Wang [18] Wang Li [19] presented similar continuous-time neural networks for solving linear equations. For solving linear inequalities, Cichocki Bargiela [20] developed three continuous-time neural networks using the aforementioned first approach. These neural networks have penalty parameters which decreases to zero as time increases to infinity in order to get better accuracy of solution. Labonte [22] presented a class of discrete-time neural networks for solving linear inequalities which implement each of the different versions of the aforementioned relaxation-projection method. He showed that the neural network that implemented the simultaneous projection algorithm developed by Pierro Iusem [4] had fewer neural processing units better computational performance. However, as Pierro Iusem pointed out, their method is only a special case of Censor Elfving s method [3]. Moreover, we feel that Censor Elfving s method can be more straightforward to be realized by a hardware implementation neural network than by the simultaneous projection method. In this paper, we generalize Censor Elfving s method propose two recurrent neural networks, continuous time discrete time, for solving linear inequality equality systems. Furthermore, two modified discrete-time neural networks with good values for step-size parameters are given by use of scaling techniques. The proposed neural networks retain the same merit as the simultaneous projection network are guaranteed to globally converge to a solution of the /99$ IEEE

2 XIA et al.: RECURRENT NEURAL NETWORKS FOR SOLVING LINEAR INEQUALITIES AND EQUATIONS 453 linear inequalities equations. In addition, the proof given in this paper is different from the ones presented before. This paper is organized as follows. In Section II, the formulations of the linear inequalities equations are introduced some related properties are discussed. In Section III, basic network models network architectures are proposed their global convergence is proved. In Section IV, two modified discrete-time neural networks are given their architectures global convergence is shown. In Section V, digital realization of the proposed discrete-time recurrent neural networks are discussed. In Section VI, operating characteristics of the proposed neural networks are demonstrated via some illustrative examples. Finally, Section VII concludes this paper. II. PROBLEM FORMULATION This section summarizes some fundamental properties of linear inequalities equations their basic application. Let, be arbitrary real matrixes let, be given vectors, respectively. No relation is assumed among, the matrix or can be rank deficient or even a zero matrix. We want to find a vector solving the systems The problem of (1) has a solution which satisfies all the inequalities equations if only if the intersect set between the solution of the one of is nonempty. It contains two special important cases. One is the system of inequalities (1) Thus, they can be formulated in the form of (1) where To study the problem of (1) we first define the following energy function: where. Let solve (1). From [14], we have the following proposition which shows the equivalence of (1) being zero. Proposition 1: The function is convex continuously differentiable (but not necessarily twice differentiable) piecewise quadratic if only if if only if. Although the Hessian of fails to exist at points where for any where is the row of the gradient of is globally Lipschitz. Proposition 2: where is globally Lipschitz with constant. Proof: Note that then Another is the system of equations with a nonnegative constraint So As an important application we consider the following linear program (LP): thus, for all, Minimize subject to (2) where. Its dual LP is the following Maximize subject to (3) Finally, the function property. Proposition 3: For any, has an important inequality By Kuhn Tucker conditions we know that is an optimal solution to (1) (2), respectively, if only if satisfies Proof: Using the lemmas in the Appendix the second-order Taylor formula, we can complete the proof of Proposition 3. (4)

3 454 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 46, NO. 4, APRIL 1999 Fig. 1. Architecture of the continuous-time recurrent neural networks. III. BASIC MODELS In this section, we propose two neural-network models, a continuous-time one a discrete-time one, for solving linear inequalities equations (1), discuss their network architectures. Then we prove global convergence of the proposed networks. A. Model Descriptions Using the stard gradient descent method for the minimization of the function we can derive the dynamic equation of the proposed continuous-time neural-network model as follows: (5) the corresponding discrete-time neural-network model where is a fixed-step parameter is the learning rate. On the basis of the set of differential equations (5) difference equations (6), the design of the neural networks implementing these equations is very easy. Figs. 1 2 illustrate the architectures of the proposed continuoustime discrete-time neural networks, respectively, where,,. It shows that each one has two layers of processing units consists of adders, simple limiters, integrators or time delays only. Compared with existing neural networks [20], [21] for solving (1), the proposed neural network in (5) contains no (6)

4 XIA et al.: RECURRENT NEURAL NETWORKS FOR SOLVING LINEAR INEQUALITIES AND EQUATIONS 455 Fig. 2. Architecture of the discrete-time recurrent neural networks. time-varying design parameter. Compared with existing neural networks [22] for solving (1), the proposed neural network in (6) can solve linear inequality /or equality systems, thus can linear program its dual simultaneously. Moreover, it is straightforward to realize in hardware implementation. B. Global Convergence We first give the result of global convergence for the continuous-time network in (5). Theorem 1: Let. The neural network in (5) is asymptotically stable in the large at a solution of (1). Proof: First, from Proposition 2 we obtain that for any an fixed initial point there exists only solution of the initial value problem associated with (5). Let then Since the function is continuously differentiable convex on it follows [23] that Note that then

5 456 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 46, NO. 4, APRIL 1999 thus so Hence, the solution for (5) is bounded. Furthermore, if then. So except at the equilibrium points. Therefore, the neural network in (5) is globally Lyapunov stable. Finally, the proof of the global convergence for (5) is similar to the one of Theorem 2 in our paper [13]. It should be pointed out that there are different advantages between continuous-time discrete-time neural networks. For example, the convergence properties of the continuoustime systems can be much better since certain controlling parameters (the learning rate) can be set arbitrarily large without affecting the stability of the system, in contrast to the discrete-time systems where the corresponding controlling parameters (the step parameter) must be bounded in a small range. Otherwise, the network will diverge. On the other h, in many operations discrete-time networks are preferable to their continuous-time counterparts because of the availability of design tools the compatibility with computers other digital devices. Generally speaking, a discrete-time neural-network model can be obtained from a continuoustime one by converting differential equations into appropriate difference equations though the Euler method. However, the resulting discrete-time mode is usually not guaranteed to be globally convergent since the controlling parameters may not be bounded in a small range. Therefore, we need to prove global convergence of the discrete-time neural network. Theorem 2: Let. Then the sequence generated by (6) is globally convergent to a solution of (1). Proof: First, from Proposition 3 we have On the other h, for any we have Note that then Thus, substituting (7) we get Since, thus is bounded. Then there exists a subsequence such that Then Substituting we get since is continuous, thus,. Finally, because is symmetric positive semidefi- is monotonically decreasing bounded. (7) then since the matrix nite. Thus, Moreover, where hence then the sequence thus has only one accumulation point Corollary 1: Assume that is bounded. If, then the sequence generated by (6) is globally convergent to a solution of (1). Proof: Note that is bounded any level set of the function is also so [18], then the sequence generated by (6) is bounded since is monotonically decreasing. Similar to the proof of Theorem 2 we can complete the rest of the proof. Remark 1: The above analytical results for discrete-time neural network in (6) provide only sufficient conditions for global convergence. Because they are not necessary conditions the network in (6) could still converge when. This point will be shown in illustrative examples.

6 XIA et al.: RECURRENT NEURAL NETWORKS FOR SOLVING LINEAR INEQUALITIES AND EQUATIONS 457 Remark 2: From the Courant Fischer minmax Theorem [24], it follows that, but this inequality is not strict. For example, let, then Proof: The case of Condition 1). Note that are symmetric positive definite. Let where, then similar to the proof of Proposition 3 we have Thus,. This shows that the step-size parameter of the network in (6) does not decrease necessarily as the number of the constraint in (1) increases. IV. SCALED MODELS From the preceding section we see that the convergence rate of the discrete-time neural network in (6) depends upon the step-size parameter thus upon the size of the maximum eigenvalue of the matrix. In the present section, by scaling techniques, we give two improved models which have good values for step-size parameters which do not depend upon the size of the maximum eigenvalue of the matrix. A. Model Descriptions Using scaling techniques, we introduce two modifications of (6). They are the following: (8) where Since with, thus where,, are symmetric semipositive definite matrixes. From the view of circuit implementation, the modified network models almost resemble the network model (6). There is only one difference among the three network models, that is, the difference lives in the connection weights of the second layer since the matrix,,, can be prescaled, respectively. However, these modified models shall have better values for step-size parameters than the basic model (6) when thus increase the convergence rate. B. Global Convergence Theorem 3: Assume that. If one of the following conditions is satisfied then the sequence generated by (8) is globally convergent to a solution of (1). 1) Let where is the th column vector of the matrix let where is the th column vector of the matrix. 2) Let rank rank let. (9) Thus the rest of proof is similar to the Proof of Theorem 2. The case of Condition 2). Note that thus So by the above mentioned proof for Condition 1) we can complete the rest of the proof. Theorem 4: Let. If one of the following conditions is satisfied, then the sequence generated by (9) is globally convergent to a solution of (1). 1) Let where is the th row vector of the matrix. 2) Let rank.

7 458 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 46, NO. 4, APRIL 1999 Proof: The case Condition of 1). Since is a symmetric positive definite matrix is as well. By Proposition 3 we have where we get. Substituting Corollary 2: Let let.ifone of the following conditions is satisfied, then the sequence generated by (8) is globally convergent to a solution of (2). 1) Let where is the th column vector of the matrix. 2) Let rank. Proof: Note that Because thus is monotonically decreasing bounded. Furthermore, we have then of Theorem 2 we can obtain. Similar to the proof The case of Condition 2). It is similar to the proof of Theorem 3 with the Condition 2). Remark 3: In order to solve (1), we see by the conditions of Theorems 3 4 that if rank rank, then we may use (8) If rank, then we may use (9). On the other h, for the computational simplicity of,,,, we should select (8) for the case, we should select (9) for the case. Remark 4: In general, rank rank. But rank often occurs. For example, let Then rank, rank, then by Theorem 3 we have the results of Corollary 2. Corollary 3: Let,.If one of the following conditions is satisfied, then the sequence generated by (9) is globally convergent to a solution of (3). 1) Let where is the th row vector of the matrix. 2) Let. Proof: Similar to the proof of Corollary 2. Remark 5: The result of Corollary 2 under Condition 1) has been given by Censor Elfving [6]. But our proof differs from theirs. Moreover, their method can not prove the results of Theorems 2 through 4. Remark 6: Although all the above mentioned theorems assume that there exists a solution of (1), that is, the systems (1) are consistent, the proposed models can identify this case. When the system is not consistent, the proposed models can give a solution of (1) in a least squares sense (a least squares solution to (1) is any vector that minimizes [6]). This point will be illustrated via Example 2 in Section VI. Remark 7: When the coefficient matrix of the system (1) is ill conditioned, the system of equations leads to stiff differential equations, thus, affect convergence rate. On the other h, if the row vector norms of the matrix is very large, the step size in model (9) is smaller, thus its convergence rate will decrease. To alleviate the stiffness of differential equations simultaneously to improve the convergence properties, one may use preconditioning techniques for the matrix [24]. or design a linear transformation for the vector V. DIGITAL REALIZATION The proposed discrete-time neural networks are suitable for digital realization. In this section, we discuss the implementation issues. The discrete-time neural networks represented by (6), (8), (9) can be put in the generalized form (10) rank rank For (6) From Theorems 3 4 we obtain easily the following two corollaries.

8 XIA et al.: RECURRENT NEURAL NETWORKS FOR SOLVING LINEAR INEQUALITIES AND EQUATIONS 459 Fig. 3. Block diagram of the one-dimensional systolic array. For (8) For (9) Note that. Let we can augment (10) by adding appropriate zeros into the matrixes vectors in (10) such that Then (10) can be rewritten as (11) In (11),,,,, can be precalculated will converge as the number of iteration increases. The converged includes the solution to the problem (1). To ease our later discussion let us define,,. For digital realization (11) can be realized by a onedimensional systolic array consisting of processing elements (PE s) as shown in Fig. 3. Where each PE is responsible for updating one element of the vector contains elements of the th row of the matrix rearranged with the following data structure: for is stored in the registers R1 R2 of the th PE. The element is then passed into the next PE s R1 R2 through the multiplexer MUX3 while at the same time processed by the current PE s multiplier-accumulator (MAC) units, MAC1 MAC2. After MAC executions, the values the th element of are produced by MAC1 MAC2, respectively. The th element of is stored in register R1 through the multiplexer MUX1, the th element of is stored in register R2 through the multiplexer MUX2. The systolic array then circulates the elements of in the ring. After another MAC executions the value of the th element of is generated by the MAC2 in PE stored in the PE s R2 register. Finally, the updated th element of is obtained by sum up (in R1) (in R2), stored back into R1 R2 for the next iteration cycle. Since operations in all PE s are strictly identical concurrent, the systolic array requires 2 -MAC execution time to complete an iteration cycle based on (11). Compared with a single processor that has the same MAC-execution speed as the PE for the computation based on (6), (8), (9), assuming all constant matrix or vector multiplications are precalculated, it will take -MAC execution time per iteration cycle. VI. ILLUSTRATIVE EXAMPLES In this section, we demonstrate the performance of the proposed neural networks for solving linear inequalities equations using four numerical examples. Example 1: Consider the following linear equality inequality system: Then where is an element of at the th row the th column. The functionality of an individual PE is shown in Fig. 4. At the beginning of an iteration, the th element of the vector It is easy to see that. Using the discrete-time neural network in (6) with the solution is.

9 460 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 46, NO. 4, APRIL 1999 Fig. 4. Data flow diagram of a processing element. Then Fig. 5. Transient behavior of the energy function in Example 1. First, we use the continuous-time neural network in (5) to solve the inequality systems. In this case, for any an initial point, the neural network always converges globally. Next, we use the discrete-time neural network in (6) to solve the above inequalities. Let, then the neural-network solution in a least squares sense is with the residue vector of. Fig. 6 shows the transient behavior of the recurrent neural network in this example. Example 3: Consider the following linear equations with nonnegative constraint: Fig. 5 depicts the convergence characteristics of the discretetime recurrent neural network with three different values of the design parameter. It shows that the states of the neural network converge to the solution to the problem within ten iterations. Example 2: Consider an inconsistent linear inequality system [19] Then, where is the identity matrix. Taking, we use the discretetime neural network in (8) to solve the above inequalities equations. When the initial point is the neuralnetwork solution is. Example 4: Consider the following linear program [16]: Minimize subject to

10 XIA et al.: RECURRENT NEURAL NETWORKS FOR SOLVING LINEAR INEQUALITIES AND EQUATIONS 461 Fig. 6. Transient behavior of the energy function in Example 2. whose optimal solution the following: Maximize subject to. Its dual program is The linear program its dual can be formulated the form of (1) where we use the discrete-time neural network in (6) to solve the above linear program its dual. Let let the initial point be, then the neural-network solution is. Since the actual value of the duality gap enables us to estimate directly the quality of the solution. Fig. 7 illustrates the values of energy function squared duality gap over iterations along the trajectory of the recurrent neural network in this example. It shows that the squared duality gap decreases zig zag while the energy function decreases monotonically. VII. CONCLUDING REMARKS Systems of linear inequalities equations are very important in engineering design, planning, optimization. In this paper we have proposed two types of globally convergent recurrent neural networks, a continuous-time a discretetime one, for solving linear inequality equality systems in Fig. 7. Transient behavior of the energy function duality gap in Example 4. real-time. In addition to the basic models, we have discussed two scaled discrete-time neural networks in order to improve the convergence rate ease the design task in selecting step-size parameters. For the proposed networks (continuous time discrete time) we have given detailed architectures of implementation which are composed of simple elements only, such as adders, limiters, integrators or time delays. Furthermore, each of the networks has the number of neurons increasing only linearly with the problem size. Compared with the existing neural networks for solving linear inequalities equations, the proposed ones have no need for setting a time-varying design parameter. The present neural networks can solve linear inequalities /or equations a linear program its dual simultaneously, thus, extend a class of discrete-time simultaneous projection networks described in [20] in computational capability. Moreover, our proof on the global convergence differs from any other published. The proposed neural networks are more straightforward to realize in hardware than the simultaneous projection networks. Further investigation has been aimed at the digital implementation verification of the proposed discrete-time neural networks on field-programmable gate arrays (FPGA). Lemma 1: For any APPENDIX then (12) Proof: Consider four cases as follows. 1) For both. So (36) holds. 2) For both. So (36) holds equally. 3) For both So (36) holds. (13)

11 462 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 46, NO. 4, APRIL ) For both. So (36) holds. From Lemma 1 we easily get the following similar result in [25]. Lemma 2: Let. Then for any Proof: Let Lemma 1 we have (14). Then.By [20] J. Wang H. Li, Solving simultaneous linear equations using recurrent neural networks, Inform. Sci., vol. 76, pp , [21] A. Cichocki A. Bargiela, Neural networks for solving linear inequalities, Parallel Comput., vol. 22, pp , [22] G. Labonte, On solving systems of linear inequalities with artificial neural networks, IEEE Trans. Neural Networks, vol. 8, pp , [23] J. M. Ortega W. G. Rheinboldt, Iterative Solution of Nonlinear Equations in Several Variables. New York: Academic, [24] G. H. Golub C. F. Loan, Matrix Computations, 3rd ed. Baltimore, MD: The Johns Hopkins Press, [25] X. Lu, An approximate Newton method for linear programming, J. Numer. Comput. Appl., vol. 15, no. 1, [26] T. H. Corman, C. E. Leiserson, R. L. Rivest, Introduction to Algorithms. New York: McGraw Hill, 1990, ch. 2. Furthermore, we can generalize the following result. Lemma 3: Let where. For any Proof: From Lemmas 1 2, the conclusion follows. REFERENCES (15) [1] D. Hertog, Interior Point Approach to Linear, Quadratic Convex Programming: Algorithms Complexity. Boston, MA: Kluwer, [2] S. Agmon, The relaxation method for linear inequalities, Canadian J. Math., vol. 6, pp , [3] Y. Censor T. Elfving, New method for linear inequalities, Linear Alg. Appl., vol. 42, pp , [4] A. R. De Pierro A. N. Iusem, A simultaneous projections method for linear inequalities, Linear Alg. Appl., vol. 64, pp , [5] Y. Censor, Row action techniques for huge sparse systems their applications, SIAM Rev., vol. 23, pp , [6] R. Bramley B. Winnicka, Solving linear inequalities in a least squares sense, SIAM J. Sci. Comput., vol. 17, no. 1, pp , [7] D. W. Tank J. J. Hopfield, Simple neural optimization networks: An A/D converter, signal decision circuit, a linear programming circuit, IEEE Trans. Circuits Syst., vol. 33, pp , [8] M. P. Kennedy L. O. Chua, Neural networks for nonlinear programming, IEEE Trans. Circuits Syst., vol. 35, pp , [9] A. Rodríguez-Vázquez, R. Domínguez-Castro, A. Rueda, J. L. Huertas, E. Sánchez-Sinencio, Nonlinear switched-capacitor neural networks for optimization problems, IEEE Trans. Circuits Syst., vol. 37, pp , [10] C. Y. Maa M. A. Shanblatt, Linear quadratic programming neural network analysis, IEEE Trans. Neural Networks, vol. 3, pp , [11] J. Wang, Analysis design of a recurrent neural network for linear programming, IEEE Trans. Circuits Cyst. I, vol. 40, pp , [12] J. Wang, A deterministic annealing neural network for convex programming, Neural Networks, vol. 7, no. 4, pp , [13] Y. Xia J. Wang, Neural network for solving linear programming problems with bounded variables, IEEE Trans. Neural Networks, vol. 6, pp , [14] Y. Xia, A new neural network for solving linear programming problems its applications, IEEE Trans. Neural Networks, vol. 7, pp , [15], A new neural network for solving linear quadratic programming problems, IEEE Trans. Neural Networks, vol. 7, pp , [16] A. Cichocki R. Unbehauen, Neural Networks for Optimization Signal Processing. London, U.K.: Wiley, [17], Neural networks for solving systems of linear equations related problems, IEEE Trans. Circuits Syst., vol. 39, pp , [18], Neural networks for solving systems of linear equations; Part II: Minimax least absolute value problems, IEEE Trans. Circuits Syst., vol. 39, pp , [19] J. Wang, Electronic realization of recurrent neural network for solving simultaneous linear equations, Electron. Lett., vol. 28, no. 5, pp , Youshen Xia received the B.S. M.S. degrees in computational mathematics from Nanjing University, China, in , respectively. Since 1995, he has been an Associate Professor with Department of Mathematics, Nanjing University of Posts Telecommunication in China. He is now working towards the Ph.D. degree in the Department of Mechanical Automation Engineering, Chinese University of Hong Kong, Shatin, NT, Hong Kong. His research interests include computational mathematics, neural networks, signal processing, control theory. Jun Wang (S 89 M 90 SM 93) received the B.S. degree in electrical engineering the M.S. degree in systems engineering from Dalian Institute of Technology, Dalian, China, the Ph.D. degree in systems engineering from Case Western Reserve University, Clevel, OH. He is now an Associate Professor of Mechanical Automation Engineering at the Chinese University of Hong Kong, Shatin, NT, Hong Kong. He was an Associate Professor at the University of North Dakota, Gr Forks. He has also held various positions at Dalian University of Technology, Case Western Reserve University, Zagar, Incorporated. His current research interests include theory methodology of neural networks, their applications to decision systems, control systems, manufacturing systems. He is the author or coauthor of more than 40 journal papers, several book chapters, two edited books, numerous conference papers. Dr. Wang is an Associate Editor of the IEEE TRANSACTIONS ON NEURAL NETWORKS. Donald L. Hung (M 90) received the B.S.E.E. degree from Tongji University, Shanghai, China, the M.S. degree in systems engineering the Ph.D. degree in electrical engineering from Case Western Reserve University, Clevel, OH. From August 1990 to July 1995 he was an Assistant Professor later an Associate Professor in the Department of Electrical Engineering, Gannon University, Erie, PA. Since August 1995 he has been on the faculty of the School of Electrical Engineering Computer Science, Washington State University, Richl, WA. He is currently visiting the Department of Computer Science Engineering, the Chinese University of Hong Kong, Shatin, NT, Hong Kong. His primary research interests are in applicationdriven algorithms architectures, reconfigurable computing, design of high-performance digital/computing systems for applications in areas such as image/signal processing, pattern classification, real-time control, optimization, computational intelligence. Dr. Hung is a member of Eta Kappa Nu.