A Cross-Associative Neural Network for SVD of Nonsquared Data Matrix in Signal Processing

Transcription

1 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 5, SEPTEMBER A Cross-Associative Neural Network for SVD of Nonsquared Data Matrix in Signal Processing Da-Zheng Feng, Zheng Bao, Xian-Da Zhang Abstract This brief paper proposes a cross-associative neural network (CANN) for singular value decomposition (SVD) of a nonsquared data matrix in signal processing, in order to improve the convergence speed avoid the potential instability of the deterministic networks associated with the cross-correlation neural-network models. We study the global asymptotic stability of the network for tracking all the singular components, show that the selection of its learning rate in iterative algorithm is independent of the singular value distribution of a nonsquared matrix. The performances of CANN are shown via simulations. Index Terms Cross-associative neural network (CANN), global asymptotic stability, learning rate, signal processing, singular value decomposition (SVD). I. INTRODUCTION SIGNAL processing approaches based on singular value decomposition (SVD) of a data matrix or correlation matrix are usually robust [1]. Many signal-processing tasks can efficiently be achieved by SVD of a nonsquared matrix. Due to the importance of SVD in signal processing, a variety of iterative methods have been proposed by researchers who are experts in matrix algebra [2] [5]. These algorithms of updating SVD for tracking subspace can get the exact or approximate SVD of a nonsquared data matrix in a low complexity per update. On the other h, neural networks have provided effective parallel processing methods for algebraic computations such as the principal component analysis [6] [9], [15]. Neural networks can also provide an alternative approach for SVD of nonsquared matrix [10] [14]. By continuation of Oja s algorithm, Yuile et al. [10] Samardzija et al. [11] developed several recurrent neural networks, which extract the principal components of autocorrelation matrix of rom data streams. These neural networks can also obtain SVD of a nonsquared matrix, if only if their weight matrix is taken as or. However, if the data matrix is ill conditioned, then the operation or usually is numerically unstable should be avoided [1]. The gradient flows based on the least squares measure of differential equations for SVD [13], [14], [16] [19] are proved to be asymptotically convergent if all the singular values of are distinct. It is worth mentioning that Diamantaras Kung [20] proposed the cross-correlation neural-network models that can be directly used for extracting the cross-correlation features be- Manuscript received February 8, 2000; revised September 27, This work was supported in part by the National Science Foundation of China ( ). The authors are with Key Laboratory of Radar Signal Processing, Xidian University, Xi an , P.R. China ( dzfeng@rsp.xidian.edu.cn). Publisher Item Identifier S (01)05531-X. tween two high-dimensional data streams. The networks can efficiently extract the principal cross correlation features between two multidimensional time series in real time, whereas their deterministic form can directly be used for performing SVD of a nonsquared matrix. However, the cross-correlation neural network models are sometimes divergent for some initial state [21]. Moreover, both analytical experimental studies show that convergence of the above neural networks depends on the appropriate selection of the learning rate, but it is difficult to be determined in advance, since the learning rate are directly related to the underlying matrix. Hence, it is important to find a neural-network model with the fixed learning rate that can be chosen in advance. In order to improve the convergence speed eliminate the potential instability of the deterministic form of the cross-correlation neural-network models (DFCNNs) [20], we propose a cross-associative neural network (CANN) in which the learning rate is independent of the singular value distribution of a nonsquared data or cross-correlation matrix. The performances of the CANN are evaluated via computer simulations. Compared with the DFCNN, the CANN has two remarkable advantages: 1) its learning rate can be a fixed constant independent of the singular value distribution of the underlying matrix, which evidently increases the convergent speed of the CANN; 2) its state vectors have the unit-norm conservation. II. A NOVEL RECURRENT NEURAL NETWORK Consider a -dimensional sequence with sampling number large enough. If is stationary, its subspace can be extracted from its sampling data that can form the nonsquared data matrix ; when is nonstationary, the rank-2 modification of the data matrix is written as. It is worth mentioning that when sampling number is small, it is more suitable to extract directly the signal subspace from the data matrix. The rank-2 update of the autocorrelation matrix often makes the smallest eigenvalue tend to zero or a small negative value [22]. In this brief paper, we consider a neural network for performing SVD of a data matrix. Given a nonsquared matrix, without loss of generality, let. Let, be the th singular value, left singular vector right singular vector, respectively,. All the, is called the th singular component, then the stard SVD of is its equivalent form in which all the singular values are positive, where both are unitary, de /01$ IEEE

2 1216 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 5, SEPTEMBER 2001 notes the diagonal matrix made up of all the singular values [1]. Noticeably, in algebra, the SVD of has some nonstard forms that are described by or their equivalent form in which any nonzero singular value may be positive or negative. Our objective is to perform the (stard or nonstard) SVD of. The most direct methods are the matrix-algebra-based methods [2] [5], while the SVD can also be obtained by using the recurrent neural network such as the DFCNN [20]. We propose the following recurrent network for SVD of : (1a) (1b) for, where,, the superscript denotes transposition, (2a) (2b) The design objective of (1) is to make as. in (2b) is called the deflation transformation [20]. The design objective of (2) is to make in order to let as. In fact, if as, we directly verify that (2) can achieve its design objective. Since the second terms in right side of [20, (19) (20)] are indefinite, the DFCNN has the potential instability. Fortunately, the second terms in right side of (1) are the higher order decay terms compared with the first terms they can govern the convergence of (1), we expect that the above neural network is globally convergent. More importantly, the learning rate or time step length of the iteration algorithm for solving (1) can be taken a constant independent of the singular value distribution of (also see Remark 1). That is to say, compared with the DFCNN in [20], the recurrent network speeds up the convergence avoids the potential instability of the DFCNN [20]. The neural network for finding the first singular component is shown in Fig. 1. Its adaptability is indicated by change of connection weights with data matrix. Thus, the entire neural network has complex topologies. The iterative algorithm corresponding to (1) is (3a) (3b) Fig. 1. Block diagram of the neural network for tracking the first singular component. for, where is called the time-step length or learning rate. It is easily known that convergence of (3) greatly replies on the learning rate. An important problem is how to choose a good learning rate. It is worth noting that in the neural-network literature, many algorithms can be extended to solve the SVD problem, but their learning rate are difficult to be determined in advance. Hence, the main reason to adopt (1) is that the learning rate in (3) can be selected in advance. Moreover, in order to avoid dividing by zero, the initial state vectors should appropriately been selected so that. Once obtaining all the singular components associated with all the nonzero singular values, we can also obtain those associated with the zero singular values by Gram Schmidt orthogonalization. At this point, we establish an essential result. Lemma 1: Given (1) arbitrary initial values, then converge exponentially to 1as, their convergence is independent of the matrix. The proof of Lemma 1 is given in Appendix A. Remark 1: We refer to if as the unit-norm conservation. As seen from (A.2a) (A.2b), if, then there are always for any finite positive ; similarly, if, then there are always for any finite positive. Moreover, since the analytical solution (A.2a) (A.2b) is independent of the data matrix, the convergence of is uniform with respect to the data matrix. Importantly, since the decay rate of linear equation (A.1a) (A.1b) is 2, the suitable learning rate in (3) can be taken as. In order to guarantee that iteration algorithm (3) has the reliable, fast global convergence, we take the fixed learning rate that is confirmed to be suitable through many simulation tests. From Lemma 1, we directly deduce the following Corollary. Corollary 1 (Boundedness): For any bounded initial values, the state vectors of non- linear system (1) are bounded.

3 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 5, SEPTEMBER III. STABILITY THEORY First, we consider stability of the first component in (1) assume that is stationary. We will prove that the stable equilibrium point set of the first component case in (1) is given by or (4a) (4b) where has the distinct nonzero singular values with multiplicity ( ). It is easily shown that (4b) is equivalent to (4a). It is easily shown that for, is a set made up of the finite points, the direction of the first singular vectors can uniquely be determined, while for, is continuous, the choice of the first singular vectors is not unique. Any point in can be regarded as the first singular component. Hence, once such a point in is obtained, we can get the first singular component. Two distinct subsets in are defined as (5a) Theorem 1 (Globally Asymptotic Stability): Let nonzero singular values of be, with the corresponding normalized left singular vectors right singular vectors, respectively. Assume or for. Then in (1) globally asymptotically converges to as. Proof: From (1) (2) it is known that the state vectors are governed by through, while are not affected by. This feature provides the convenience for analysis. By Lemma 2, it is known that, hence we have,as. Once approaches, the repeated use of lemma 2gives as, which is the stability of (1) when. Mimicking this process, we can prove the stability of (1) when. This completes the proof of Theorem 1. Remark 3: We may expect that (1) perform the stard SVD, which is achieved by the following method: if the stable state vectors of (1) satisfy condition, then is replaced by. IV. SIMULATIONS Two simulations are presented. In Simulation 1, SVD of an ill-conditioned matrix is used to evaluate the efficiency of this parallel neural network. In Simulation 2, the solution of a TLS problem is obtained by neural-network-based SVD for nonsquared data matrix. Simulation 1: Consider the following matrix: (6) (5b) There is obviously, corresponds to the stard SVD of. Lemma 2 (Globally Asymptotic Stability of the First Component): Let nonzero singular values of be, with the corresponding normalized left singular vectors right singular vectors, respectively. Furthermore, let have the distinct singular values with multiplicity. Assume or for any. Then in (1) globally asymptotically converges to a point in as. Proof: See Appendix B. Remark 2: From Lemma 2 Result 3 in Appendix B, we conclude that when, in (1) globally asymptotically converges to a point in, while, in (1) globally asymptotically converges to a point in,as. where ( ) are the th components of 11- nine-dimensional orthogonal discrete cosine basis functions, respectively, i.e.,. (7a) (7b) The matrix given by (6) has nine nonzero singular values in which the entire three distinct singular values 10, have multiplicity 3. The data matrix is ill conditioned, since its condition number is [1]. We used three methods to compute the SVD of in (6). The estimation errors are defined as follows.

4 1218 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 5, SEPTEMBER ) The CANN: (8) 2) The DFCNN [20]: 3) The Moore s neural network [17]: (9) (a) (10) In the DFCNN Moore s neural networks, the suitable fixed time step length takes, the variables parameters in (9) (10) can be found in [20] [17], respectively. Romly generating the initial state vectors with unit-norm, the MATLAB software obtains the simulation results. The evolution curves of all the errors against iteration number shown in Fig. 2. The CANN has emerged as a useful neural-networkbased technique, when the underlying matrix is close to a singular matrix. Simulation 2: Give the following problem of identification for multiple input single output static system: (11) where is system parameter vector; is a five-dimensional white input vector; is the output. The input output samples are corrupted by additive white noise. The unbiased estimate of parameter can be gotten by the total least squares approach [1]. The estimation error is defined as, where represents the estimate of. Define the augmented output vector. The data matrix consists of 30 samples of the augmented output vector, where any row vector of the data matrix represent a sample of the augmented output vector. The evolution curves of against iteration number under cases (SNR 0.1, 0.25, 0.5) are shown in Fig. 3 from which we see that the proposed neural network can find the total least squares solution, while the DFCNN cannot. (b) V. CONCLUSION A new recurrent neural network for SVD of a nonsquared data matrix or cross-correlation matrix is proposed. The global convergence of this nonlinear system is studied. The norm of its state vectors is governed by stable ordinary differential equations independent of the data matrix globally exponentially converges to 1 in the fixed decay rate 2. Both theoretical analysis simulation results show that the time-step length or leaning rate in the iterative algorithm associated with this neural network is independent of the data matrix. The CANN has also emerged as a useful neural-network-based technique when the (c) Fig. 2. The convergent curves of all the estimation errors against iteration number, where integer 1 9 in figure (a) (b) shows the estimation errors associated with singular component 1 9. (a) ANN. (b) DFCNN. (c) Moore s approach. data matrix is close to a singular matrix. Simulation results show that there are not the significant difference of the estimation errors, computational complexity, convergent speed between the neural-network-based method the matrix-algebra-based method.

5 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 5, SEPTEMBER (A.1b) The analytical solutions of (A.1a) (A.1b), respectively, are given by (A.2a) (A.2b) Clearly, if, then. This completes the proof of Lemma 1. (a) APPENDIX B THE PROOF OF LEMMA 2 For convenience of analysis, let (B.1) Substituting (B.1) into (1), we can get (B.2a) (B.2b), shown at the bottom of the next page. In theory, (B.2a) (B.2b) are equivalent to (1). For the sake of notational simplicity, the single index is involved in all the below equations. For the first component, (B.2a) (B.2b) can be rewritten as (b) (B.3a) (B.3b) Result 1: in (B.3b) exponentially converges to zero at decay rate 1 as. The addition the subtraction of the two equations in (B.3a) yield (c) Fig. 3. The estimation errors against iteration number. APPENDIX A THE PROOF OF LEMMA 1 Differentiating with respect to considering (1), we can obtain (B.4) From the above equation, we directly deduce the following equations: (A.1a)

6 1220 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 5, SEPTEMBER 2001 (B.5a) namely (B.8b) (B.9a) (B.5b) Without loss of generality,,, are assumed. From (B.5a) (B.5b), we can obtain (B.9b) where are bounded by Corollary 1. Result 2: All the ( ) ( ) in (B.9a) (B.9b) exponentially converge to zero. With Result 1 Result 2, without the loss of generality, we assume that (B.6a) Considering the above assumption ( ), (B.3a) (B.3b) can be rewritten as (B.10) (B.6b) whose analytical solution is described by (B.7a) (B.7b) where ( ). From (B.7a) (B.7b), we can directly deduce the two inequalities: (B.8a) (B.11) Now, we only require further to study the globally asymptotic convergence of (B.11). Applying Lemma 1 or (A.2a) (A.2b) to (B.11) immediately yields (B.12a)

7 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 12, NO. 5, SEPTEMBER