Discrete time Generalized Cellular. Johan A.K. Suykens and Joos Vandewalle. Katholieke Universiteit Leuven

Transcription

1 Discrete time Generalized Cellular Neural Networks within NL q Theory Johan A.K. Suykens and Joos Vandewalle Katholieke Universiteit Leuven Department of Electrical Engineering, ESAT-SISTA Kardinaal Mercierlaan 94 B-001 Leuven (Heverlee), Belgium tel: /1/ fax: /1/ Johan.Suykens@esat.kuleuven.ac.be ESAT-SISTA report I submitted to Int. J. Circ. Theor. & Appl. Special issue on CNNs This research work was carried out at the ESAT laboratory and the Interdisciplinary Center of Neural Networks ICNN of the Katholieke Universiteit Leuven, in the framework of the Belgian Programme on Interuniversity Poles of Attraction, initiated by the Belgian State, Prime Minister's Oce for Science, Technology and Culture (IUAP-1). The scientic responsibility rests with its authors. 1

2 Abstract Generalized Cellular Neural Networks (GCCNs) were recently introduced by Guzelis & Chua. Instead of one single CNN, a set of CNNs was considered, interconnected in a feedforward, cascade or feedback way. The framework was in continuous time with sucient conditions for global asymptotic and I/O stability and the relation with classical nonlinear control theory such as the Lur'e problem was revealed. In this paper GCNNs are considered in a discrete time setting. The original system description is brought into a so-called NL q system form using state augmentation. NL q 's are a general class of nonlinear systems in state space form with a typical feature of having q 'layers' with alternating linear and nonlinear operators that satisfy a sector condition. Within NL q theory sucient conditions for global asymptotic and asymptotic stability are available. The results are closely related to modern control theory (H 1 theory and theory). Stability criteria are formulated as Linear Matrix Inequalities (LMIs). Checking stability involves the solution to a convex optimization problem. Furthermore it is shown by examples that existing GCCN congurations result into q = 1 values. Hence if one considers the q value of the NL q as a measure of complexity of the overall system, GCNNs are still have a low complexity from the analysis point of view. In addition more complex neural network architectures with q > 1 are discussed. Keywords. Generalized CNNs, recurrent neural networks, NL q systems, global asymptotic stability, dissipativity, L -gain, Linear Matrix Inequalities (LMIs).

3 1 Introduction After the introduction of the Cellular Neural Network (CNN) paradigm by Chua & Yang it became clear that CNNs are successful in solving image processing tasks and partial dierential equations (see e.g. Chua & Roska, Roska & Vandewalle 4 ). CNNs consist of simple nonlinear dynamical processing elements (called cells or neurons) that are only locally interconnected to their nearest neighbours and are usually arranged in a two dimensional array, which makes them attractive from the viewpoint of implementation. Besides continuous time also discrete time versions, algorithms and technology are available (Harrer & Nossek 9, Harrer 10 ). Nevertheless there are also limitations on the use and applications of one single CNN. Therefore there is increasing interest in creating more complex systems, that are built up of existing less complex ones (CNNs in this case). Recently Guzelis & Chua 8 introduced Generalized CNNs (GCNNs) where a set of CNNs is interconnected in a feedforward, cascade or feedback way. Also at the cell level generalizations were made by considering 'Chua's circuits' instead of 'neurons', leading to cellular 'nonlinear' networks instead of cellular 'neural' networks (Roska & Chua ). Because even simple nonlinear dynamical systems such as Chua's circuit can already exhibit complex behaviour, letting such circuits interact within an array, results in new rich phenomena (see Chua 4 for an overview). On the other hand stability analysis of interconnected systems becomes more complicated. This motivates the need for general frameworks for the analysis and design of those systems. The aim of this paper is precisely to show how NL q theory may contribute towards this purpose in a discrete time setting. NL q theory originated from the stability analysis of neural control systems (that consist of recurrent neural network models and controllers) and was introduced by Suykens et al. 5. Examples of NL q s include e.g. neural control systems, the Lur'e problem, Linear Fractional Transformations (LFTs), several kinds of recurrent neural networks (including multilayer) and digital lters with overow characteristic. Using state augmentation the original system descriptions are put into some standard nonlinear state space form, that is called the NL q system. The value of q is a measure for the overall complexity of the system. Whereas up to date the Lur'e problem (which consist of a linear dynamical system, feedback interconnected to a static nonlinear-

4 ity that satises a sector condition) was extensively studied in nonlinear control theory, in NL q theory higher complex systems can be analyzed and synthesized. The Lur'e problem corresponds to an NL 1 system. In Guzelis & Chua 8 it was shown that stability analysis of GCNNs can be made in the context of the Lur'e problem. Hence it is not very surprising that the discrete time GCNNs that are considered in this paper will reduce to NL 1 s. Sucient conditions for global asymptotic stability and I/O stability (dissipativity with 5 nite L -gain) of NL q s were derived in Suykens et al.. The conditions are very much related to conditions arising in modern control theory (H 1 and theory, see e.g. Packard & Doyle 0 ) and can be expressed as Linear Matrix Inequalities (LMIs) (see Boyd et al. 1 for LMIs in systems and control). LMI problems lead to nondierentiable but convex optimization problems, which means there is a unique minimum and moreover this minimum can be found in polynomial time (Nesterov & Nemirovskii 18 ). This paper is organized as follows. In Section we discuss discrete time GCCNs. Then NL q s are dened in Section and GCNNs are written as NL q s in Section 4. In Section 5 sucient conditions for global asymptotic and I/O stability of NL q s are presented, which are formulated as LMI problems in Section. Discrete time GCCNs Generalized CNNs are build up of CNNs, which are interconnected e.g. in a feedforward, feedback or cascade way (see Guzelis & Chua 8 ). They have a certain inter-layer (between the CNNs) and intra-layer (within the CNN) connectivity. Let us consider a GCNN that consists of L CNNs with layer index l = 1; :::; L. assumed to take the discrete time form 8 >< x l k+1 = A l x l k + Bl u l k z l k = C l x l k y l k = f(z l k) u l k = P i E l iy i k + P j F l jv j k + l ; The dynamics of the l-th CNN are with internal state x k R n l, input to the linear dynamical subcircuit uk R m l, output y k R p l and external input vk R q l. The upper index of the matrices and vectors is the 4 (1)

5 layer index. The notation E l i, F l j means that the l-th CNN is connected to the output of the i-th CNN and the j-th external input respectively. The discrete time index is k. The nonlinearity f(:) is taken elementwise when applied to a vector. If f(:) is a static nonlinearity belonging to sector [0; 1] we will use the notation (:) instead of f(:) in the sequel. Alternatively the dynamics (1) are written as 8 >< x l k+1 = A l x l k + B l E l lf(c l x l k) + B l P i=l E l if(c i x i k) + B l P j F l jv j k + Bl l y l k = f(c l x l k ): () The dynamics (1) are formulated at the CNN level and not at the cell level as was done in Guzelis & Chua 8. In this paper we make abstraction of the intra-layer connectivity. The local interconnection between the cells is then revealed by the sparseness of certain matrices in (1). NL q systems NL q systems are nonlinear discrete time systems of the following state space form (see Suykens et al. 5 ) 8 >< p k+1 =? 1 ( V 1? ( V :::? q ( V q p k + B q w k ) + B q?1 w k ) + :::) + B 1 w k ) e k = 1 (W 1 (W ::: q (W q p k + D q w k ) + D q?1 w k ) + :::) + D 1 w k ) with state vector p k R np, input vector w k R nw () and output vector e k R ne. Here? i, i (i = 1; :::; q) are diagonal matrices with diagonal elements j (p k ; w k ), j (p k ; w k ) [0; 1] for all values of p k, w k and depending continuously on the variables p k ; w k. The matrices V i, W i, B i, D i for i = 1; ::; q are constant with compatible dimensions. An I/O equivalent representation to () is 4 p k+1 e ext k 5 = qy i=1 i (p k ; w k )R i 4 p k w k 5 (4) with i = diagf? i;e ; i;e ; 0g (i = 1; :::; q) and R i = blockdiagfm i ; N i ; 0g, R q = [M q ; N q ; 0] (i = 1; :::; q? 1) where? 1;e =? 1,? i;e = diagf? i ; Ig, M 1 = [V 1 B 1 ], M i = [V i B i ; 0 I], 5

6 related to the NL 1 x k+1 = (W x k ): () 1;e = 1, i;e = diagf i ; Ig, N 1 = [W 1 D 1 ], N i = [W i D i ; 0 I] (i = ; :::; q). Furthermore e ext k corresponds to e k augmented with a number of zero elements in order to make Q q i=1 R i square. Note that k i k 1 because k? i k 1 and k i k 1 (1, or 1 norm). A typical aspect of NL q s are the q 'layers' in the state equation and the output equation (Fig.1). The NL q system is related to systems of the form 8 >< p k+1 = ( V 1 ( V :::( V q p k + B q w k ) + B q?1 w k ) + :::) + B 1 w k ) e k = (W 1 (W :::(W q p k + D q w k ) + D q?1 w k ) + :::) + D 1 w k ) where (:) is static nonlinear operator that satises the sector condition [0; 1]. Special cases for (:) are e.g. tanh(:) and sat(:). The fact that this NL q is related to nonlinear operators that satisfy a sector condition can be understood as follows. Consider a system (5) This can be written as with? = diagf i g and i x k+1 =?(x k )W x k = (w T i x k )=(w T i x k ). This is easily obtained by using an elementwise notation and based on the fact that tanh(:) is a diagonal nonlinearity. Eqn.() becomes x i := ( P j w P jx i j ) := ( P j wi j xj ) := i i P j w i jx j : j wi j xj P j w i j xj The time index is omitted here because of the assignment operator 0 := 0. The notation i i means that this corresponds to the diagonal matrix?(x k ). In case w T i x k = 0 de l' Hospital's rule or a Taylor expansion of (:) leads to i = 1. For an additional layer one obtains x k+1 = (V (W x k )); () x k+1 =? 1 (x k )V? (x k )W x k

7 where? 1 = diagf 1i g with 1i = (v T i (W x k))=(v T i (W x k)) and? = diagf i g with i = (w T i x k )=(w T i x k ). Indeed from an elementwise notation one has 4 GCNNs as NL q s x i := ( P j vj( P i l w j l xl ) P := ( P j vj i j j := 1 i i Pj v i j j j l w j l xl ) P l w j l x l : We will show now by examples how CNNs and Generalized CNNs can be written as NL q systems. This is done by applying state augmentation. Example 1. Consider a CNN of the form x k+1 = Ax k + B(Cx k ) + F v k + (8) Using the state augmentation k = (Cx k ) one obtains the state space description 8 >< x k+1 = Ax k + B k + F v k + k+1 = (CAx k + CB k + CF v k + C) Dening p k = [x k ; k ] and w k = [v k ; 1] one has the NL 1 system p k+1 =? 1 ( V 1 p k + B 1 w k ) (9) with matrices V 1 = 4 A B CA CB 5 ; B 1 = 4 F CF C 5 and? 1 = diagfi;?(x k ; k ; v k )g with k? 1 k 1. Example. Consider a Generalized CNN according to Fig. which consists of CNNs, containing feedforward, feedback and cascade interconnections. The system is described

8 as 8>< x 1 k+1 = A 1 x 1 k + B 1 (C 1 x 1 k) + E 1 5(C 5 x 5 k) + F 1 v k + 1 x k+1 = A x k + B (C x k) + E 1(C 1 x 1 k) + E (C x k) + x k+1 = A x k + B (C x k) + E (C x k) + x 4 k+1 = A 4 x 4 k + B4 (C 4 x 4 k ) + E4 (C x k ) + E4 (C x k ) + 4 x 5 k+1 = A 5 x 5 k + B 5 (C 5 x 5 k) + E 5 4(C 4 x 4 k) + 5 x k+1 = A x k + B (C x k) + E 4(C 4 x 4 k) + with external input v k. Applying the state augmentation (10) i k = (C i x i k); i = 1; ; :::; and dening p k = [x 1 k ; x k ; x k ; x4 k ; x5 k ; x k ; 1 k ; k ; k ; 4 k ; 5 k ; k ] V 1 = and w k = [v k ; 1], one obtains again an NL q system with q = 1 and matrices 4 A 1 B 1 E 5 1 A E 1 B E A E B A 4 E 4 E 4 B 4 A 5 E 4 5 B 5 A E 4 B C 1 A 1 C 1 B 1 C 1 E 5 1 C A C E 1 C B C E C A C E C B C 4 A 4 C 4 E 4 C 4 E 4 C 4 B 4 C 5 A 5 C 5 E 4 5 C 5 B 5 C A C E 4 C B 5 ; B 1 = 4 F C 1 F 1 C C 0 C 0 C C C (11) 5 Hence although the interconnectivity of this GCCN is rather complex, the overall system description is still low complex (q = 1) in the sense of NL q theory. This is due to the fact that the CNNs contain a linear dynamical subcircuit. In Suykens et al. 5 systems with higher q values were obtained in the context of neural control systems. So-called neural state space models and controllers were considered e.g. models of the form 8 >< x k+1 = W AB tanh(v A x k + V B u k + AB ) y k = W CD tanh(v C x k + V D u k + CD ) (1) 8

9 and controllers of the form 8 >< z k+1 = W EF tanh(v E z k + V F1 y k + V F d k + EF ) u k = W GH tanh(v G z k + V G1 y k + V G d k + GH ) (1) with x k the state of plant, u k the control signal, y k the output of the plant, z k the state of the controller and d k the reference input. W and V are interconnection matrices and are bias terms. These type of recurrent neural networks are general in the sense that they correspond to nonlinear models and controllers of the form 8 >< x k+1 = f(x k ; u k ) y k = g(x k ; u k ) and 8 > < z k+1 = h(z k ; y k ; d k ) u k = s(z k ; y k ; d k ) where the functions f(:), g(:), h(:) and s(:) are parametrized by multilayer feedforward neural networks architectures. It is well-known that these are universal approximators because any continuous nonlinear function can be approximated arbitrarily well on a compact interval by means of a multilayer feedforward neural networks with one or more hidden layers (Hornik et al., Cybenko, Funahashi, Leshno et al. 5;;1;15 ). This leads then to NL q s with q > 1 because of the multilayer feature of the recurrent neural networks. Based on this insight Generalized CNNs that correspond to q > 1 can be obtained as follows. For CNNs like (1) the nonlinearity f(:) satises usually a sector condition. However it has been shown in Guzelis & Chua 8 that Chua's circuit can be written in a form like (1) (but in continuous time and on the cell level). The nonlinearity of the nonlinear resistor plays then the role of the nonlinearity f(:) in the continuous time version to (1). Hence instead of taking yk l = (C l zk) l in (1) a useful extension is to take y l k = (W l (V l x l k + l )) where W l (V l x l k + l ) is a multilayer feedforward neural network with interconnection matrices W l, V l and bias vector l, which is able to represent any continuous nonlinear 9

10 mapping (including the nonlinearity used in Chua's circuit), provided there are 'enough' hidden neurons. This leads to the type of recurrent neural network proposed in the following example. Example. The use of a multilayer perceptron for f(:) in (1) gives Applying the state augmentation one obtains the state space description 8 >< x k+1 = Ax k + B(W (V x k + )) + F v k + (14) k = (W (V x k + )) x k+1 = Ax k + B k + F v k + k+1 = (W (V Ax k + V B k + V F v k + V + )) Dening p k = [x k ; k ], w k = [v k ; 1] this yields an NL q system with q = and V 1 = 4 I 0 0 W 5 ; V = 4 A B V A V B 5 ; B = 4 F V F V 5 ; B 1 = 0 (15) A similar GCCN like in Example could be considered, which consists of CNNs of the form (14). This would also lead to an NL. Hence from the viewpoint of NL q theory one has the following ordering in terms of the network complexity q Linear dynamical system CNN or GCNN with sector [0; 1] nonlinearity CNN or GCNN with multilayer perceptron nonlinearity Neural state space models and neural control systems NL q systems The fact that GCNNs are reducible to NL 1 s implies that the powerful GCCN architecture is still low complex with respect to analysis (at least from the viewpoint of NL q s). 10

11 5 Stability criteria for NL q s In this Section we review some sucient conditions for global asymptotic stability and I/O stability of NL q s, that were derived in Suykens et al Autonomous NL q s The following Theorem holds for autonomous NL q s. of Theorem 1 [Diagonal scaling]. A sucient condition for global asymptotic stability p k+1 = ( is to nd diagonal matrices D i such that qy i=1? i (p k )V i ) p k kd tot V tot D?1 totk q = D < 1 (1) where the matrices V tot and D tot are given by { V tot = 4 0 V 0 0 V... 0 V q V { V i R n h i n hi+1 ; nh1 = n hq+1 = n p { D tot = diagfd ; D ; :::; D q ; D 1 g, D i R n h i n hi diagonal matrices with nonzero diagonal elements. Proof: see Suykens et al. 5. Remarks. 11

12 The proof is based on the Lyapunov function (p) = kd 1 pk which is positive and radially unbounded (hence the equilibrium point p = 0 is unique if (1) holds) and makes use of the properties of induced norms. For q = 1 the proof works as follows. Given the system p k+1 =? 1 V 1 p k consider the Lyapunov function k = kd 1 p k k. Then k+1 = kd 1? 1 V 1 p k k = k? 1 D 1 V 1 D?1 1 D 1 p k k kd 1 V 1 D?1 1 k k because the diagonal matrices D 1 and? 1 commute and k? 1 k 1. So if kd 1 V 1 D?1 1 k < 1 then k = k+1? k < 0. For q = 1 the Theorem reduces to the results in Kaszkurewicz & Bhaya 14 theory (Packard & Doyle 0 ). and are closely related to diagonal scaling in control Less conservative criteria were also derived in 5. Instead of diagonal matrices D i, full matrices P i are allowed, provided some condition of diagonal dominance on the matrices P T i P i holds. 5. Non-autonomous NL q s: I/O properties The conditions of the following Theorem are sucient to guarantee I/O stability of the NL q. Theorem [l exist matrices D i such that then the following holds 1. For a nite time horizon N: with r 0 = kd 1 p 0 k. theory - Diagonal scaling]. Given the representation (4), if there kd tot R tot D?1 totk q = D < 1; (1) N?1 X k=0 ke k k D 1 N?1 X k=0 kw k k + r 0 (18)

13 . For N! 1: There exist constants c 1 ; c such that provided that fw k g 1 k=0 l. c (1? D)kpk + kek Dkwk + c 1 kp 0 k (19) where l n are given by { denotes the set of square summable sequences in C n. The matrices R tot and V tot R tot = 4 0 R 0 0 R... 0 R q R { R i R nr i nr i+1 ; nr1 = n rq+1 = n p + n w { D tot = diagfd ; D ; :::; D q ; D S1 g, D S1 = diagfd 1 ; I nw g, D 1 R npnp, D i R nr i nr i Proof: see Suykens et al. 5. Remarks. diagonal matrices with nonzero diagonal elements. For q = 1 the Theorem and the proof reduce to the one given in Packard & Doyle 0 for the state space upper bound test in control theory. Indeed LFTs (Linear Fractional Transformations) with real diagonal uncertainty block are special cases of NL q s. Like for the autonomous case sharper criteria based on diagonal dominance instead of diagonal scaling were derived in 5. There exists a close relationship between the internal stability of the autonomous case and the property of nite L -gain of Theorem. This was already stated in the work of Hill & Moylan 11;1, Willems 0;1; and van der Schaft 8 and becomes clear 1

14 through the concept of dissipativity. A dynamic system with input w k and output e k and state vector p k is called dissipative if there exists a nonnegative function (p) : R np! R with (0) = 0, called the storage function, such that 8w R nw and 8k 0: (p k+1 )? (p k ) W (e k ; w k ) where W (e k ; w k ) is called the supply rate. The NL q system is dissipative under the condition of Theorem, with storage function (p) = kd 1 pk, supply rate W (e k ; w k ) = kw D kk? ke kk and nite L -gain D < 1. Similar results hold for the case of diagonal dominance. Checking stability is an LMI problem The conditions of Theorem 1 & are of the form with and D tot = diagfd ; D ; :::; D q ; D S1 g with kd tot Z tot D?1 totk q < 1 (0) Z tot = V tot : global asymptotic stability = R tot : nite L -gain < 1 D S1 = D 1 : global asymptotic stability = diagfd 1 ; Ig : nite L -gain < 1 This condition (0) can be written as the Linear Matrix Inequality (LMI) Z T totd totz tot < D tot (1) The notation A < 0 means A negative denite. A < B means A? B negative denite. For an LMI M(x) < 0 with M = M T with unknown x R n, the matrix M depends anely on x: M(x) = M 0 + P m i=1 x i M i with M 0 = M T 0, M i = M T i. Finding a feasible x to such an LMI corresponds to solving a convex problem, which has a unique minimum. The book by Boyd 14

15 et al. 1 gives an overview of LMIs arising in systems and control. Moreover the convex problem can be solved in polynomial time (Nesterov & Nemirovskii 18 ). Other related work is done by Nemirovskii & Gahinet, Vandenberghe & Boyd, Overton 1;19;. Matlab software for solving (1) for the case Z tot = V tot is the function psv of Matlab's Robust Control Toolbox. Software for solving general type of LMIs is available in Matlab's LMI lab (Gahinet & Nemirovskii ). Conclusion In this paper we investigated Generalized CNNs in a discrete time setting. Some sucient conditions for global asymptotic stability and I/O stability for GCNNs were presented. This was done by transforming GCNNs to NL q systems. GCNNs that consist of CNNs with sector [0; 1] nonlinearity were reduced to NL 1 s, which still have a low complexity from the viewpoint of NL q theory. Existing sucient stability criteria within NL q theory reduce for q = 1 to well-known results in modern control theory. Checking stability according to these criteria involves solving an LMI problem, which is convex and computationally feasible. Furthermore it has been illustrated with examples how state augmentation can be applied in order to transform given systems into an NL q form. This gives insight into the overall structure and complexity of the interconnected system. Hence when novel multilayer highly complex and interconnected systems are created, NL q theory may contribute towards a unifying framework for the analysis and design of such dynamical systems. 15

16 References 1. Boyd S., El Ghaoui L., Feron E., Balakrishnan V., Linear matrix inequalities in system and control theory, SIAM (Studies in Applied Mathematics), Vol.15., (1994).. Chua L.O., Yang L., Cellular neural networks: Theory & Applications, IEEE Transactions on Circuits and Systems, Vol. 5, No.10, pp , (1988).. Chua L.O., Roska T., The CNN Paradigm, IEEE Transactions on Circuits and Systems-I, Vol.40, No., pp.14-15, (199). 4. Chua L.O., Chua's circuit 10 years later, International Journal of Circuit Theory and Applications, Vol., pp.9-05, (1994). 5. Cybenko G., Approximations by superpositions of a sigmoidal function, Mathematics of Control, Signals and Systems,, pp.18-19, (1989).. Funahashi K-I., On the Approximate Realization of continuous Mappings by Neural Networks, Neural Networks, Vol., pp.18-19, (1989).. Gahinet, P., Nemirovskii A., General-Purpose LMI Solvers with Benchmarks, Proceedings Conference on Decision and Control, pp. 1-15, (199). 8. Guzelis C., Chua L.O., Stability analysis of generalized cellular neural networks, International Journal of Circuit Theory and Applications, Vol.1, pp.1-, (199). 9. Harrer H., Nossek J.A., Discrete time cellular neural networks, International Journal of Circuit Theory and Applications, Vol.0, pp.45-4, (199). 10. Harrer H., Multiple layer discrete-time cellular neural networks using time-variant templates, IEEE Transactions on Circuits and Systems-II, Vol. 40, No., pp , (199). 11. Hill D.J., Moylan P.J., Connections between nite-gain and asymptotic stability, IEEE Transactions on Automatic Control, Vol.AC-5, No.5, pp.91-9, (1980). 1. Hill D.J., Moylan P.J., The stability of nonlinear dissipative systems, IEEE Transactions on Automatic Control, Vol.AC-1, pp.08-11, (19). 1

17 1. Hornik K., Stinchcombe M., White H., Multilayer feedforward networks are universal approximators, Neural Networks, Vol., pp.59-, (1989). 14. Kaszkurewicz E., Bhaya A., Robust stability and diagonal Liapunov functions, SIAM Journal on Matrix Analysis and Applications, Vol.14, No., pp , (199). 15. Leshno M., Lin V.Y., Pinkus A., Schocken S., Multilayer feedforward networks with a nonpolynomial activation function can approximate any function, Neural Networks, Vol., pp.81-8, (199). 1. Liu D., Michel A.N., Asymptotic stability of discrete time systems with saturation nonlinearities with applications to digital lters, IEEE Transactions on Circuits and Systems I, Vol.9, No.10, pp , (199). 1. Nemirovskii, A., Gahinet P., The Projective Method for Solving Linear Matrix Inequalities, Proceedings American Control Conference, pp , (1994). 18. Nesterov Y., Nemirovskii A., Interior point polynomial algorithms in convex programming, SIAM (Studies in Applied Mathematics), Vol.1., (1994). 19. Overton M.L., On minimizing the maximum eigenvalue of a symmetric matrix, SIAM Journal on Matrix Analysis and Applications, Vol.9, No., pp.5-8, (1988). 0. Packard A., Doyle J., The complex structured singular value, Automatica, Vol.9, No.1, pp.1-109, (199). 1. Perez-Munuzuri A.P., Perez-Munuzuri V., Perez-Villar V., Chua L.O., Spiral waves on a -D array of nonlinear circuits, IEEE Transactions on Circuits and Systems I, Vol.40, No.10, (199).. Polak E., Wardi Y., Nondierentiable optimization algorithm for designing control systems having singular value inequalities, Automatica, Vol.18, No., pp.- 8, (198).. Roska T., Chua L.O., The CNN Universal Machine: an analogic array computer, IEEE Transactions on Circuits and Systems-II, Vol.40, No., pp.1-1, (199). 1

18 4. Roska T., Vandewalle J., Cellular Neural Networks, John Wiley and Sons, (199). 5. Suykens J.A.K., De Moor B., Vandewalle J., NL q theory: a neural control framework with global asymptotic stability criteria, K.U. Leuven, Department of Electrical Engineering, ESAT-SISTA, Technical report 94-8I, submitted for publication, (1994).. The MathWorks Inc., Matlab, Robust Control Toolbox (Version 4.1), User's guide, (1994).. Vandenberghe L., Boyd S., A primal-dual potential reduction method for problems involving matrix inequalities, Mathematical Programming Series B, special issue on nondierentiable and large scale optimization, to appear, (199). 8. van der Schaft A.J., L -gain analysis of nonlinear systems and nonlinear state feedback H 1 control, IEEE Transactions on Automatic Control, Vol.AC-, pp.0-84, (199). 9. Vidyasagar M., Nonlinear systems analysis, Prentice-Hall, (199). 0. Willems J.C., The generation of Lyapunov functions for input-output stable systems, SIAM Journal Control, 9(1):105-14, (191a). 1. Willems J.C., Least squares stationary optimal control and the algebraic Riccati equation, IEEE Transactions on Automatic Control, AC-1():1-4, (191b).. Willems J.C., Dissipative dynamical systems I: General theory. II: Linear systems with quadratic supply rates. Archive for Rational Mechanics and Analysis, 45:1-4, (19). 18

19 List of Captions Figure 1. I/O equivalent representation for NL q system with the typical q 'layer' feature. The matrices R i are constant and the matrices i (p k ; w k ) are diagonal and satisfy the condition k i k 1 for all values of p k, w k. Figure. Generalized CNN that consists of CNNs, interconnected in a feedforward, cascade and feedback way. The GCNN can be written as an NL q system with q = 1 (see Example ). 19

20 Figures p k w k R Ω... 1 R Ω... 1 i i R q Ω q p k+1 e k Figure 1: CNN CNN CNN CNN CNN CNN Figure : 0