CHAPTER 3 ARTIFICIAL NEURAL NETWORKS AND LEARNING ALGORITHM

Transcription

1 46 CHAPTER 3 ARTIFICIAL NEURAL NETWORKS AND LEARNING ALGORITHM 3.1 ARTIFICIAL NEURAL NETWORKS Introducton The noton of computng takes many forms. Hstorcally, the term computng has been domnated by the concept of programmed computng than neural computng. In programmed computng, algorthms are desgned and subsequently mplemented usng the currently domnant archtecture, whereas n neural computng, learnng replaces a pror program development. The neural computng offers a potental soluton to many currently unsolved problems n conventonal computng. Artfcal neural networks provde new tools and new foundatons for solvng practcal problems n predcton, decson and control, sgnal separaton, state estmaton, pattern recognton, data mnng, etc. Tradtonal statstcs tres to collect a huge lbrary of dfferent methods for dfferent tasks, but the bran s a lvng proof that one system can do t all, f there s data. It proves that a system can manage mllons of varables wthout beng confused. Nowadays, engneers and scentsts are tryng to develop ntellgent machnes. Artfcal Neural Systems (ANS) are examples of such machnes that have greater potental to further mprove the qualty of human lfe. Artfcal neural networks are collectons of mathematcal models that emulate some of the observed propertes of bologcal nervous systems and

2 47 draw on the analoges of adaptve bologcal learnng. They are capable of developng ther behavor through learnng. They learn through experences lke human bran. They are dynamcal systems n the learnng/tranng phase of ther operaton and convergence s an essental feature. There are dfferent types of ANNs. Some of the popular models nclude the BPN whch s generally traned wth the Generalzed Delta Rule (GDR), Learnng Vector Quantzaton (LVQ), Radal Bass Functon (RBF), Hopfeld, Adaptve Resonance Theory (ART) and Kohonen s Self-Organzng Feature Map (SOM) networks. Some ANNs are classfed as feedforward whle others are recurrent (.e., mplement feedback) dependng on how data s processed through the network. The synaptc weght update of ANNs can be carred out by supervsed methods, or by unsupervsed methods, or by fxed weght assocaton network methods. In the case of the supervsed methods, nputs and outputs are used, whereas, n the unsupervsed methods, only the nputs are used. In the fxed weght assocaton network methods, nputs and outputs are used along wth pre-computed and pre-stored weghts. Artfcal Neural Networks are used where a conventonal process s not sutable. the conventonal method cannot be easly delvered. the conventonal method cannot fully capture the complexty n the data and the stochastc behavor s mportant. an explanaton of the network s decson s not requred. The functon of a sngle neuron n an ANN s shown n Fgure 3.1. Each neuron receves nputs from other neurons and/or from external sources.

3 48 Lke a real neuron, the processng element has many nputs but has only a sngle output, whch s connected to many other processng elements n the network. Each processng element s numbered. It apples lnear/nonlnear functon on ts net nput to compute the output. Fgure 3.1 Functon of a sngle neuron In the above Fgure, the nput pattern s represented by [ x, x,...,x ] T X = (3.1) 1 2 n Each artfcal neuron s nput has an assocated weght whch ndcates the fracton (or amount) of the transfer of one neuron s output to another neuron s nput. The nputs for an artfcal neuron are from external sources or from other neurons. The notaton w j s used to represent the weght on the nterconnecton lnk from neuron j to neuron. The weght vector W s represented as [ w, w,...,w ] T W = (3.2) 1 2 n

4 49 The net nput value of a sngle neuron s determned by the weghted sum of nputs as gven by net() = net = w1x1 + w2x wnx n (3.3) = n wkxk (3.4) k= 1 where n s the number of nputs. A neuron (or unt) fres, f the sum of ts nputs exceeds some threshold value. If t fres, t produces an output, whch has been sent to the next layer neurons. In vector notaton, the net nput value s gven by net T = W X (3.5) The output value of a sngle neuron s obtaned as o() = f (net ) (3.6) The objectve of neural network desgn s to determne an optmal set of weghts w*. Therefore, the artfcal neurons nvolve two mportant processes. () Determne the net nput value by combnng the nputs. () Mappng the net nput value nto the neuron s output. Ths mappng may be as smple as usng the dentty functon or as complex as usng a nonlnear functon Backpropagaton Neural Network The area of speech sgnal separaton and recovery of orgnal sgnals from a mxed sgnal s a challengng doman for nformaton and system processng. Contnued attempt of usng artfcal neural network n speech

5 50 sgnal separaton s takng place (Cchock and Unbehauen 1996; Amar and Cchock 1998; Meyer et al 2006). They have ensured the separaton of extremely weak or badly scaled statonary sgnals, as well as a successful separaton even f the mxng matrx s very ll-condtoned. The Backpropagaton (BPN) neural network s a multlayer supervsed neural network whch uses steepest descent method to update weghts. Its archtecture as gven n Fgure 3.2 has an nput layer wth I nodes and an output layer wth K nodes and one hdden layer wth J nodes. Each neuron n the hdden layer has ts own nput weghts and the output of a neuron n a layer goes to all neurons n the next layer. Fgure 3.2 Archtecture of backpropagaton neural network The ntal weght values are randomly generated by the Matlab functon rand(). The nput layer does not process the mxture nput. It just dstrbutes the nput samples to all the neurons n the hdden layer. The output of each hdden layer neuron s obtaned by applyng the sgmod functon to ts net

6 51 nput value. Each hdden neuron output s fed to all the neurons n the output layer. Each neuron n the output layer, frst calculates the net nput value and then t apples nonlnear functon on the net nput to produce an output value m. In the earler stage of ths work, an attempt has been made to extract the ndvdual speech sgnals from an artfcally mxed speech sgnal usng Backpropagaton neural network and ts performance s compared wth RBF neural network. To compare ther performance, the waveforms of speeches of only two persons are recorded n a closed envronment for two seconds at the rate of 8KHz. The schematc dagram of speech sgnal recovery by BPN neural network s shown n Fgure 3.3. Tranng ` Testng Fgure 3.3 Schematc dagram of speech sgnal recovery 4000 samples of each speech sgnal shown n Fgure 3.4 are mxed and are used for tranng the BPN network.

7 52 (a) Speech sgnals of two persons (b) Mxed speech sgnal Fgure 3.4 Two speech sgnals and mxed sgnal (a) Recovery of speech 1 (b) Recovery of speech 2 Fgure 3.5 Recovery of speech sgnals by BPN neural network (a) Recovery of speech 1 (b) Recovery of speech 2 Fgure 3.6 Recovery of speech sgnals by RBF neural network

8 53 From the smulaton result shown n Fgure 3.5, t has been observed that the recovered sgnals by BPN are dstorted wth poor qualty of speech for lmted number of teratons (at whch RBF neural network s converged). The BPN network whch has 15 hdden neurons, takes 65 mnutes to reduce the error to The computatonal load s more snce the weghts between nput and hdden layer and weghts between hdden and output layer are updated. The major lmtaton of the backpropagaton neural network s ts slow convergence tme (.e. more tranng tme) and t has ended up wth local mnma. Moreover, there s no proof of convergence, although t seems to perform well n practce. Due to stochastc gradent descent on a nonlnear error surface, t s lkely that most of the tme the result may converge to some local mnmum on the error surface. Another major lmtaton s the problem of scalng and when the complexty of the problem s ncreased, there s no guarantee that good generalzaton would result. So, the network sze, such as the number of hdden layers has to be ncreased. Ths results n heavy burden on the computaton and network complexty. So the RBF neural network s traned wth the same data and from Fgure 3.6, t s observed that recovered sgnals by RBF network are obtaned wthout dstorton. The RBF network whch has two hdden neurons, takes 52 sec to reduce the error to The computatonal load s less snce only the weghts between hdden and output layer are updated Tranng Strategy of Backpropagaton Neural Network For the network to learn patterns, the weght updatng algorthm, Unsupervsed Stochastc Gradent Descent Algorthm (USGDA) has been used. The present work nvolves modfcaton of weghts to extract the

9 54 ndependent sgnals from mxed sgnals. The functon of the network s based on an unsupervsed learnng strategy. The nputs of a pattern are presented and the output of the network n the output layer s computed and the weghts n the output layer and hdden layer are updated by the weght update equaton and compared wth the prevous weght value. The total error.e., the dfference between the prevous weght value and the current weght value s determned. The total error for all patterns presented s calculated and f ths total error s greater than zero, the learnng rate parameter s vared by the Equaton (4.1) and the weghts are updated by the weght update Equatons (3.7) and (3.8). At each teraton, ths process decreases the total error of the network for all the patterns presented. To mnmze the total error to zero, the network s presented wth all the tranng patterns many tmes. Ths procedure s repeated untl the error becomes As gven n Fgure 3.2, R s the weght matrx between nput and hdden layer. Z s the weght matrx between the hdden and output layer. The weght value z kj on the nterconnecton from neuron j to neuron k s updated by the weght update equaton ( ) T ( ) ( ) ( ) ( ) z t + 1 = z t + lrp d t d t O + ε (3.7) kj kj 1 2 p where t - tme step, lrp - learnng rate parameter, = 0.99, ε - constant parameter, = 0.05 d 1 (t) = nv(det(z(t)) ( ) = + + d t [3m 4m 2.92m 5m 3.417m 2 k k k k k 0.78m m ] k k

10 55 T O p - Transpose of mxture sgnal p Smlarly, all the remanng weghts between hdden and output layer are updated by the Equaton (3.7). The weght value r j on the nterconnecton from neuron to neuron j s updated by the weght update equaton T ( ( ) ) ( ) ( ) 1 ( ) ( ) ( ) ( ) r t + 1 = z t + lrp d t d t O + ε z t 1+ exp( net j j 1 2 p kj pj (3.8) Smlarly, all the remanng weghts between nput and hdden layer are updated by the Equaton (3.8). To mplement ths algorthm, the speeches of two persons are recorded n a closed envronment for two seconds at the rate of 4 KHz. Lke ths, 25 speeches of males and 25 speeches of females are recorded and stored as.wav fles. Ffty combnatons (one male and one female; two males; two females) of dfferent speech waveforms (S 1 and S 2 ) are mxed artfcally by multplyng the speech sgnals wth varous coeffcents as gven by O 1 = 0.3 S S 2 (3.9) 500 samples of each mxture sgnal are preprocessed by the technque, normalzaton so that the nputs to the nodes of the nput layer are between zero and one. 40 mxture sgnals are used for tranng both BPN and RBF neural networks by the unsupervsed stochastc gradent descent algorthm and 10 mxture sgnals are used for testng the networks. From Fgure 3.7, t s found that after certan number of teratons (no. of teratons=2223, at whch the RBF network s converged), the BPN network s able to converge to only some extent. The network s found to convergence

11 56 slowly due to local mnma and slow tranng. When the number of nodes n hdden layer s 15, BPN network takes 13 hours and 5 mnutes for reducng the error to So, t has been observed that the recovered sgnals by BPN are dstorted for lmted number of teratons (at whch ASN-RBF neural network s converged) and the contents of speeches are retaned wth dstortons n the qualty of speech. Error No. of Itratons BPN ASN-RBF Fgure 3.7 Graph of error versus teratons for sample sze=500 (50 mxture sgnal), MSE= 0.01 and η=0.99 When the number of nodes n hdden layer s 2, RBF neural network takes 4 hours and 33 mnutes for reducng the error to 0.01 n 2223 teratons. So, the tranng tme of RBF network s 8 hours and 32 mnutes less than that of BPN network. Thus, the performance of the RBF network s found to be much superor to BPN n terms of recoverng the orgnal sgnals wth less tranng tme.

12 Adaptve Self-Normalzed Radal Bass Functon (ASN-RBF) Neural Network In recent years, there has been an ncreasng nterest n usng Radal Bass Functon neural network for many problems. Lke Backpropagaton and Counter propagaton neural networks, t s a feedforward neural network that s capable of performng nonlnear relatonshp between the nput and output vector spaces. RBF and BPN are both unversal approxmators..e., when they are desgned wth enough hdden layer neurons, they approxmate any contnuous functon wth arbtrary accuracy (Gros and Poggo 1989; Hartman et al 1990). Ths s a property they share wth other feedforward networks havng one hdden layer of nonlnear neurons. Hornk et al (1989) have shown that the nonlnearty need not be sgmod and t can be any of a wde range of functons. It s therefore not surprsng to fnd that there always exsts an RBF network capable of accurately mmckng a specfed BPN or vce-versa. The RBF network s found to be sutable for BSS problem snce t has the followng characterstcs. 1. It has faster learnng capablty and t s good at handlng nonlnear data. 2. It fnds nput to output mappng usng local approxmators and they requre fewer tranng samples. 3. It provdes smaller nterpolaton errors, hgher relablty and a more well-developed theoretcal analyss than BPN

13 58 As shown n Fgure 3.8, the ASN-RBF neural network conssts of three layers: an nput layer wth 500 neurons, a sngle layer of nonlnear processng neurons and an output layer wth 2, 3 or 4 neurons dependng on the number of sources. In Backpropagaton neural network, the weghts between hdden layer and output layer and also the weghts between hdden and nput layer are updated durng tranng. But, n RBF neural network, only the weghts between hdden and output layer are updated. The RBF network does not end up wth local mnma and the outputs of the hdden layer neurons are calculated by m m ( ) ( ) k ( ) p = f o = u ϕ o,c = u ϕ o c k k k k k k = 1 k = 1 for = 1,2,..., N (3.10) and the outputs of output layer neurons are calculated by N w j.p = β = 1 m j where 1 β = α (3.11) where p s the output of the hdden neuron, α s the convergence parameter used n the network, n 1 o R + s an nput vector and ( ) bass functon whch s gven by exp( D 2 ( 2 ) 2 ) T ( ) ( ) 2 j j ϕ s a radal k. λ, where D = O W O W and λ s the spread factor whch controls the wdth of the radal bass functon, U k s the weght matrx between nput and hdden layer, W s the weght matrx between hdden and output layer, N s j N 1 the number of neurons n the hdden layer and c R are the RBF centers n the nput vector space. For each neuron n the hdden layer, the Eucldean k

14 59 dstance between ts assocated center and the nput to the network s computed. The convergence parameter α s used n the network for faster convergence of the proposed learnng algorthm. Durng tranng, f t s very low, the total error becomes NaN (Not a Number) and the network s not converged. So, the convergence parameter s gradually ncreased from a lower value such that the network does not encounter wth NaN and the network s converged for a partcular value. Therefore, the total error s reduced to the tolerance value after a fnte number of teratons. m=500; N=20-2; n=2,3,4 Fgure 3.8 Topology of ASN-RBF neural network The convergence parameters used for dfferent expermental set ups are gven n Table 3.1. The ASN-RBF neural network archtecture s capable of performng nonlnear relatonshp between nput and output vector spaces. The scalng parameter β s used for post-processng to obtan the correct

15 60 output data. The centers c k are assumed to perform an adequate samplng of the nput vector space. They are usually chosen as subset of the nput data. The weght vector W k determnes the value of O whch produces the maxmum output from the neuron. The response at other values of O drops quckly as O devates from W becomng neglgble n value when O s far from W. Table 3.1 Convergence parameters used for expermental set ups Source sgnals No. of samples Convergence parameter (α) Max. Iteratons Crow2.wav,song1.wav crow2.wav, ssong1.wav wsong2.wav male1.wav,female1.wav, male2.wav male1.wav,female1.wav, male2.wav, female2.wav Tranng Strategy of ASN-RBF Neural Network There are two sets of parameters governng the mappng propertes of the RBF neural network: the weghts W k n the output layer and the centers c k of the radal bass functons. The ASN-RBF neural network s traned wth fxed centers. They are chosen n a random manner as a subset of the nput data set. After the network has been traned, some of the centers are removed

16 61 n a systematc manner wthout sgnfcant degradaton of the system performance. The locaton of the centers of the receptve felds s a crtcal ssue and there are many alternatves for ther determnaton. In the learnng algorthm, the center and correspondng hdden layer neuron are located at each nput vector n the tranng set. The dameter of the receptve regon, determned by the value of the spread factor λ s set at 0.01, has a profound effect upon the accuracy of the system. The objectve s to cover the nput space wth receptve felds as unformly as possble. If the spacng between centers s not unform, t s necessary for each hdden neuron to have ts own value of λ. 100 nputs (brds voces) are used for ths proposed network. Out of these, 80 nputs are used for tranng the ASN-RBF neural network and 20 nputs are used for testng the network. Each nput corresponds to dfferent combnatons of brds voces downloaded from the webste Dfferent combnatons of the brds voces are artfcally mxed and preprocessed by the technque, Normalzaton so that the nputs to the nodes of nput layer are between zero and one. The preprocessed nput s fed to the network n the form of 500 samples, correspondng to 500 neurons n the nput layer. The number of source sgnals s vared from two to four. The ASN-RBF neural network s also traned and tested for separaton of speech sgnals whch are recorded for about 2 sec n a closed envronment at the rate of 8 KHz. The proposed ASN- RBF neural network and learnng algorthm perform well under nonstatonary envronments and when the number of source sgnals s unknown and dynamcally changed.

17 UNSUPERVISED STOCHASTIC GRADIENT DESCENT LEARNING ALGORITHM To separate ndependent components from the observed sgnal, an objectve functon s requred. The objectve functon s chosen such that t gves orgnal sgnals when t s mnmzed. Durng tranng of the ASN-RBF neural network, one of the free parameters.e., the nterconnecton weghts between hdden layer and output layer are adjusted to mnmze the objectve functon. In sgnal processng (Taleb and Jutten 1998), when the components of the output vector become ndependent, ts jont probablty densty functon factorzes to margnal pdfs, gven by ( ) ( ) M M = 1 k f M,W = f m,w (3.12) where fm (m, W) s the margnal pdf of M, m s the th component of the output sgnal M and k s the number of source sgnals. The Equaton (3.12) s a constrant mposed on the learnng algorthm. The jont pdf of M parameterzed by W s wrtten as f M ( M,W) ( O) f = o (3.13) B where B s the determnant of the Jacoban matrx B. It s defned as B = m m m... o o o k m m m... o o o k m m m... o o o k k k 1 2 k (3.14)

18 63 Referrng to Equaton (1.4), each element n Equaton (3.14) s represented nterms of w as m o j = w j (3.15) Therefore, Equaton (3.14) s wrtten as w w... w k w w... w W k = (3.16) w w... w k1 k2 kk Now, Equaton (3.13) s wrtten as f M ( M,W) ( O) f = o (3.17) W To extract ndependent components from the observed sgnal, the dfference between the jont pdf and the product of margnal pdfs s determned. When the components become ndependent, the dfference becomes zero (.e. the jont pdf becomes equal to the product of margnal pdfs of the separated sgnals). It s expressed as M M = 1 k f ( M,W) f ( m,w) = 0 (3.18) Snce the logarthm provdes computatonal smplcty, t s taken on both sdes of Equaton (3.18) and t becomes, ( ) k M ( ) = M ( ) (3.19) = 1 log(f M, W) log f m, W

19 Equaton (3.19), Substtutng the value of f ( ) M 64 m,w from Equaton (3.17) n ( O) f log = log f m,w ( M ( )) k o W (3.20) = 1 Because the pdf of the nput vector s ndependent of the parameter vector W, the objectve functon for optmzaton becomes k ( ) ( M ) Φ (W) = log W log f (m, W) (3.21) = 1 Now, the Edgeworth seres has been used to expand the second term n Equaton (3.21). The edgeworth seres expanson of the random varable M about the Gaussan approxmate α(m) s gven by ( ) ( ) 2 fm m k3 k 4 10k3 = 1+ H3 ( m) + H4 ( m) + H6 ( m ) +... α m 3! 4! 6! (3.22) where α(m) denotes the probablty densty functon of a random varable M, normalzed to zero mean and unt varance, k denotes the cumulant of order of the standardzed scalar random varable M and H denotes the Hermte polynomal of order. The thrd and fourth order cumulants and Hermte polynomals are gven by k 3 =c 3 (3.23) H 3 (m) = m 3-3m (3.24) H 4 (m)=m 4-6m 2 +3 (3.25) 2 k 4 =c 4-3c 2 (3.26)

20 65 The cumulants are expressed nterms of moments. The r th order moment of m s gven by c,r = E m (3.27) r = E w.o n r r (3.28) r= 1 Substtutng the values of the cumulants and Hermte polynomals from Equatons (3.23), (3.24), (3.25) and (3.26) n Equaton (3.22) and takng logarthm on both sdes, t becomes f ( m ) ( m) M log = 0.75m m 0.365m + 0.5m 0.285m α m m (3.29) Dfferentatng Equaton (3.29) wth respect to w k, fm log α w ( m ) ( m) k = 3m + 4m 2.92m + 5m 3.417m m 0.056m O k Therefore, the optmzaton functon becomes ( m (t)) 3m 4m 2.92m 5m 3.417m 0.78m 0.056m (3.30) Ψ = (3.31) After smplfcaton, the gradent descent of Equaton (3.21) now becomes ( ) Φ W = W T Ψ ( m) O T W (3.32) The stochastc gradent descent algorthm for weght update s wrtten as

21 66 ( W) Φ W( t + 1) = W(t) η (3.33) W Substtutng the gradent of the cost functon from Equaton (3.32) n Equaton (3.33), the weght update rule s wrtten as T ( + ) = ( ) + η( ) Ψ ( ) T ( ) ( ) W t 1 W t t W m t O t (3.34) The Edgeworth seres s used for the approxmaton of probablty densty functons snce ts coeffcents decrease unformly and the error s controlled, so that t s a true asymptotc expanson. On the other hand, the terms n the Gram-Charler expanson do not tend unformly to zero from the vew pont of numercal errors;.e., n general, no term s neglgble compared to a precedng term Algorthm Descrpton Once the centers and spread factors have been chosen, the output layer weght matrx W s optmzed by unsupervsed learnng usng stochastc gradent descent technque. The tranng process conssts of the followng sequences as gven n Fgure 3.9. Step 1 : Step 2 : Intalze the parameters. a) Assgn weghts between nput and hdden layer. b) Assgn weghts between hdden and output layer. c) Set η=0.99, λ=0.09, ε = 0.05 and set M(t)=O(t). Read nput sgnals. Step 3 : Generate mxng matrx A. Step 4 : Step 5: Obtan the observed mxture sgnal O. Preprocess (.e., normalze) the mxture sgnals.

22 67 Step 6 : Recover source sgnals. () Apply the observed sgnal to the nput layer neurons. % Forward operaton % For each pattern n the tranng set a) Fnd the Hdden layer output. b) Fnd nputs to nodes n the output layer. c) Compute the actual output of output layer neurons. d) Determne delta W. e) Update the weghts between hdden and output layer. f) If the dfference between prevous weght value and current weght value s not equal to zero, then go to step 5 else stop tranng Step 7: Postprocess (.e., Denormalzaton) the output data. Step 8: Store and dsplay the separated sgnals. Thus, the development of ths algorthm nvolves maxmzaton of the statstcal ndependence between the output vectors M. It s equvalent to mnmzng the dvergence between the two dstrbutons: () Jont probablty densty functon f M (m,w) parameterzed by W.

23 68 n fm ( m,w). = 1 () Product of margnal densty functon of M, Fgure 3.9 Block dagram of source sgnal recovery Implementaton of USGDA Step 1: Get Source 1. [s1,srate,no_bts]=wavread('nukeanthem'); % Returns the sample rate n Hertz and the number of bts per sample (NBITS) used to encode the data n the fle. s1=wavread('nukeanthem.wav',num_samples);

24 69 sources(1,:)=s1; Step 2: Get Source 2. s2=wavread('dspafxf.wav',num_samples); % Returns only the frst N samples from each channel n the fle. sources(2,:)=s2; Step 3: Get Source 3. s3=wavread('utopia.wav',num_samples); sources(3,:)=s3; Step 4 : Intalze the parameters. a) Assgn weghts between nput and hdden layer. b) Assgn weghts between hdden and output layer. c) Set η=0.99, λ=0.09, ε = 0.05 and set M(t)=O(t). Step 5: Generate mxng matrx A. A=rand(num_mxtures,num_sources); Step 6: Step 7: Step 8: Obtan the observed mxture sgnal O by O = sources * A; Preprocess the mxture sgnal. Recover source sgnals. %Forward operaton % For each pattern n the tranng set () Fnd h (Hdden layer output). for j =1:p ds=0; for k=1:l x=o(j,k); c=center(,k); dff = x-c; ds = ds + dff.^2;

25 70 () end end ph(j)=exp((-ds)/(2*sg^2)); output_of_hddenn(j)=ph(j); Fnd nputs to nodes n the output layer. nput_to_outputn=output_of_hddenn*hou; () Compute the actual output (for output layer neurons) for b = 1:ol end output_of_outputn(b)=(nput_to_outputn(b)/α); (v) Fnd the dfference between prevous weght value and current weght value. (v) If the dfference s equal to zero, stop tranng, else fnd the weght update value.e., delta 1 ( ( )) d = nv det hou _ old ; for k=1:output_neurons ( ) = + + d k [3m 4m 2.92m 5m 3.417m 2 k k k k k end deloutput (d d ) O T = 1 2 ; 0.78m m ] k k (v) Update the weghts between hdden and output layer by the equaton W(t + 1) = W(t) + lrp deloutput (t) + ε ; Step 9: Step 10: Evaluate M(t) and postprocess t to obtan the orgnal sgnals. Repeat steps 8 and 9 untl the dfference between the prevous weght value and the current weght value s not equal to zero.

26 71 The radal bass functon ph = exp(-d 2 )/(2σ)^2 was evaluated over the nterval -1< x <1 and -1<y<1 as shown n the graph n Fgure Fgure 3.10 Evaluaton of radal bass functon over the nterval -1< x <1 and -1<y<1 Thus, the algorthm whch s mplemented, nvolves the condton of ndependency whch nturn related to varyng the weghts of RBF neural network, successfully separates the sgnals from the mxed nput sgnal. The ASN-RBF neural network and the proposed learnng algorthm perform well under non-statonary envronments and when the number of source sgnals s unknown and dynamcally changed.