Using Kernel Density Estimator in Nonlinear Mixture

Transcription

1 9 ECTI TRANSACTIONS ON ELECTRICAL ENG, ELECTRONICS, AND COMMUNICATIONS VOL3, NO AUGUST 5 Using Kernel Density Estimator in Nonlinear Mixture W Y Leong and J Homer, Non-members ABSTRACT Generally, blind separation of sources from their nonlinear mixtures is rather difficult This nonlinear mapping, constituted by unsupervised linear mixing followed by unknown and invertible nonlinear distortion, is found in many signal processing cases We propose using a kernel density estimator incorporated within an equivariant gradient algorithm to separate the nonlinear mixed sources The proposed algorithm is a generalization of natural gradient algorithm and Gram-Charlier series, which is able to adapt to the actual statistical distributions of the sources by estimating the kernel density distribution at the output signals Experiments are presented to illustrate the performance of our proposed nonlinear blind source separation method Keywords: Nonlinear signal Processing, blind source separation 1 INTRODUCTION Independent Component Analysis (ICA) [1-4] and the closely related Blind Source Separation (BSS) [5, 6] methodology have been well studied in the case of linear mixtures This analysis has received attention in signal processing, telecommunications [7] and speech enhancement systems The idea of ICA methodology is to find the independent components from mixtures of the independent sources based on a generative approach such as negentropy maximization, [8], maximum likelihood estimation [9] and cumulant-based [1] methods; where the goal is to explain how the observations are generated from the independent component sources It is assumed that there exist certain source signals which have generated the observed data through an unknown mapping The goal of the generative learning is to identify both the source signals and the unknown generative mapping via a recursive blind algorithm In general, most of the proposed blind source separation algorithms based on the theory of independent 4PSI7: Manuscript received on December 13, 4 ; revised on May 13, 5 This work is supported by The University of Queensland Graduate School Award Parts of this paper appeared at ISCIT4, Japan The authors are with School of Information Technology and Electrical Engineering The University of Queensland, Australia leong@iteeuqeduau, homerj@iteeuqeduau component analysis (ICA) assume the mixture mapping to be linear [6] Little work has been dedicated to the problem of blind source separation in nonlinear mixtures However, when the mapping is nonlinear, we naturally need to use a nonlinear adaptive algorithm It is evident that nonlinear mixing is far more difficult than the linear case This nonlinear mapping problem has already attracted some attention recently [11-13] In the nonlinear mapping model, the linear ICA theory does not have the pruning capability to learn and detect the nonlinear observations, as will be shown later The blind separation algorithms for the linear mixture model generally fail to extract the independent sources from nonlinear mixtures To tackle the nonlinear mapping problem, the nonlinear adaptive algorithms involve the use of one or more layers of hidden computational units in addition to the output layer [14] Among the contributions to the nonlinear mapping problem, Valpola, Giannakopoulos, Honkela and Karhunen [1] proposed using Bayesian ensemble learning in nonlinear ICA In ensemble learning, a simpler approximative distribution is fitted to the true posterior distribution by minimizing their Kullback-Leibler divergence Here the ensemble learning provides the necessary regularisation for nonlinear ICA by choosing the model and sources that have most probably generated the observed data Also, Taleb and Jutten [11] proved that it is not possible to separate the general nonlinearly mixed sources; the source independence assumption is not strong enough in the general nonlinear case They proposed a direct estimation of the score functions by minimizing the mean square error of the parameter vector in post-nonlinear mixtures The results show that the least mean square estimation of the score functions performs well with hard nonlinearities To justify the theoretical analysis, Taleb [15] investigated a structured nonlinear model framework, and proposed a stochastic algorithm to deal with the parametric nonlinear mixtures The main objective of this paper is to investigate a class of nonlinear transformations for which an iterative and equivariant algorithm is presented This proposed algorithm is related to the Equivariant Adaptive Source Separation via ICA () [16] algorithm, which is based on the idea of serial multiplicative updating It was proposed by Cardoso and Laheld [16] and extended by Karhunen etal in [17] The serial multiplicative update algorithm consists

2 Using Kernel Density Estimator in Nonlinear Mixture 93 s 1 s s n Fig1: A R 1 R R n Nonlinear function e 1 e e n g 1 g g n Inverse function W Nonlinear mixing and unmixing model of an nxn matrix valued nonlinear function, which is used for updating the linear unmixing matrix W Here n is the number of source signals and the number of observed mixtures The algorithm however is typically unsuitable for nonlinear mixtures The paper is organized as follows: In section II, the nonlinear model is presented and a set of learning rules is derived in section III based on an equivariant gradient criterion The learning rules are verified via simulation in section IV The concluding remarks are discussed in section V NONLINEAR MIXTURES The problem of nonlinear mapping from the unknown sources s(t)) = [s 1 (t),s (t),,s n (t)] T to the observations e(t) is modelled with the multi-layer perceptron (MLP) network as illustrated in Fig1 Here A represents a nxn linear mixing matrix and R = {R i } n i=1 represents an (n 1) vector of unknown one-to-one nonlinear mappings The unmixing system involves application of an (n 1) vector of nonlinear transfer function {g i } n i=1 followed by a linear unmixing nxn matrix, W This model is appropriate for multiple inputs and multiple outputs (MIMO) channels; where an n-sensor array provides the observed nonlinear mixture signals e(t) = [e 1 (t),e (t),,e n (t)] T We assume the received observation e(t) is a vector of nonlinear memoryless mixtures of n unknown statistically independent sources s(t): e(t) =R(As(t)) (1) where A R nxn is an unknown non-singular mixing matrix, which represents the hidden layer of the network To adjust and pre-process the received signals, normalization will be applied by centering the observable variables e The normalization method is chosen because it has gained good results in initializing the model vectors of mixtures Good initial values are important for the method since the network can effectively prune away unused components The normalization operation is y 1 y y n e nom = e E{e} () where E denotes the expectation operator Our goal is to recover the sources without any particular knowledge of their distributions and the nonlinear mixing mechanism The principle is to derive a separation structure (g, W) which involves the learning rule for the estimation of the matrix W and the nonlinear transfer parameters of g The unmixing output y is given by: y(t) =W(g e (t)) = W g (R (As(t))) (3) Our mission is to have the unmixed output components of y =(y 1,y,,y n ) as statistically independent as possible; this can be achieved by minimizing the mutual information of the output y We say that signal components are independent if their joint probability distribution function factorises as the product of the marginal densities [3, 6] p y (y) =p y (y 1,y,,y n )= n p yi (y i ) (4) i=1 where p yi (y i ) is the probability density function of the ith output component y i Maximizing the statistical independence corresponds to minimizing the mutual information of y The mutual information of the observed vector y can be measured by the Kullback-Leibler (KL) divergence of the multivariate density from the product of the marginal (univariate) densities [3] p y (y) I(y) = p y (y)log n i=1 p dy (5) y i (y i ) Note independence is achieved if and only if the KL divergence is equal to zero The KL divergence of eq(5) is equal to Shannon s mutual information I(y) between the components of output vector y [] This can be rewritten as I(y) = n H(y i ) H(y) (6) i=1 where H denotes the differential entropy: H(y i )= Elogp yi (y i ),where E denotes the expectation operator, and H(y) = H(y 1,,y n ) Thus estimation of the separation structure corresponds to the minimization of I(y); this requires the computation of its gradient with respect to the separation structure parameters W and g Rewriting eq(6), gives I(y 1,,y n ) = H(y 1 )++ H(y n ) H(y 1,,y n ) { = p y (y 1,y,,y n ) log p y(y 1,y,,y n ) } n i=1 p dy 1,dy,,dy n y i (y i ) p y (y) = p y (y)log n i=1 p dy (7) y i (y i )

3 94 ECTI TRANSACTIONS ON ELECTRICAL ENG, ELECTRONICS, AND COMMUNICATIONS VOL3, NO AUGUST 5 Fig: The Flow of Nonlinear Separation 3 EQUIVARIANT NONLINEAR SEPARA- TION To stabilize and equivariant the nonlinear mapping signals, we propose an equivariant gradient algorithm consistingofthekerneldensityf(y) estimation within the nonlinear system (W, ge(t)) See Fig Blind source separation of the nonlinear mixtures e(t) requires first to estimate the nonlinear estimation block g This procedure is done by applying an iterative procedure to isolate each component of the observations of the initial observation vector, e(t) = e (t): e k+l (t) =g(e k (t)) = e k (t) η F(e k (t)) (31) where η is a positive adaptation step size and F( ) is the probability density function vector of the initial nonlinear observation vector e(t) = e (t),k denotes iteration index The second stage involves iterative estimation of the linear unmixing matrix W via the following equivariant gradient algorithm: W k+l = W+η [I diag(f(y)) k y T k +diag(f(e(t)))]w k (3) where η is a positive adaptation step size, I is the identity matrix, f = logf(e(t)) is the kernel density score function of the pdf vector F(e(t)) diag( ) denotes diagonal matrix, the diagonal elements corresponding to the elements of n vector e The proposed equivariant gradient algorithm involves the probability density function F( ) and the corresponding kernel density f( ) We employ the Gram-Charlier and Edgeworth series [18, 19] to approximate the probability distribution function F The summary of the Gram-Charlier and Edgeworth series is as follow 3 1 Kernel Density Estimation Through the mixing operation, the observation vector e has a pdf F described approximately by a normal pdf Accordingly, it is quite natural to approximate F with a normal pdf This is the basis of the Gram-Charlier and Edgeworth series [18, 19] method which optimizes the determination of the probability density function (pdf) by data moments In particular, the pdf F(e(t)) can be represented via Hermite s polynomials H n (e) truncated expansion: F(e) = 1 [ (e m) ] k σ π exp σ C n H n (e) = αm σ(e) n=1 k C n H n (e) (33) n=1 where C n are coefficients to be determined and αm σ( ) is the Gaussian distribution vector having vector m as mean and vector σ as standard deviation The values of m and σ are calculated directly from the observation Note, each of the vector multiplication, division operations above are carried out on an element-by-element basis 3 Hermite s Polynomials Determination Referring to eq(33), the Hermite s polynomials are evaluated by the following iteration rules: H n (e) = ( 1)n αm σ(e) Dn e αm σ(g(e)) (34)

4 Using Kernel Density Estimator in Nonlinear Mixture 95 The Gaussian function is αm σ defined as αm σ(e) = 1 σ π exp{[e m] /σ } (35) The operator D n e is defined as D n e = dn d(e) n (36) The first six Hermite s polynomials terms are obtained as follow: H (e) = 1, H 1 (e) = (e-m) σ, H (e) = (e-m) σ 1 4 σ, H 3 (e) = (e-m)3 σ 3 (e-m) 8 H 4 (e) H 5 (e) = (e-m)4 σ 8 ( = (e-m) (e-m) 4 σ σ 8 σ 4, 6 (e-m) σ σ 4, 1 (e-m) σ σ 4 ), (37) 33 Coefficient C n Determination The coefficients C n can be evaluated by the orthogonalization process [1] F(e)H(e)de C n = H(e)σ(e)de = σn F(e)H(e)de (38) n! It can also be shown that C n+δd = n a= σ α α α! n+δ d α β= { ( 1) n+δ d α β (n + δ d α β)!β! m n+δ d α β u β}, (39) 4 SIMULATION RESULTS In this section, we describe several experiments that were made to evaluate the validity and performance of the proposed nonlinear algorithm, in comparison with the conventional ICA based [16] and [, 8] algorithms These experiments are used to assess the proposed nonlinear algorithm s ability to perform blind source separation of several nonlinear mixtures The complete proposed learning scheme for all the experiments is the same and corresponds to that shown in Fig First, normalization is used to prepare the observed mixtures to enable more robust and fast adaptation The normalization stage is also used to find sensible initial values for the posterior means of the mixtures Also, the initial weight W of the iterative algorithm is set as a matrix of random values The mixtures are then adjusted via the nonlinear function g i ( ) and the weight matrix, W In each simulation, a specified number of iterations is adopted, where one iteration means going through the full time sequence of the observations, e(t) As shown by the results, the proposed nonlinear algorithm is able to recover the sources from nonlinear mixtures that involved relatively smooth nonlinearities The experimental results of the above algorithms are presented for, 6 and 1 mixtures of sources This section investigates the performance by presenting four illustrative examples 4 1 Performance Analysis, Noise Absent [ We assume n =, A = distortion operations: ] and the nonlinear where n =, 1,, 3, and δ d is defined as { +1 if (n + δd )is odd δ d = if (n + δ d )is even (31) and the terms u k are k-order input moments defined as + u k = (e) k F(e)de (311) m = u 1,σ = u (u 1 ) The first six coefficients are C = u, C 1 = u 1 mu, C = 1 (u u 1 m + u m σ u ), C 3 = 1 3! (u3 3u m +3u 1 m u m 3 ) u σ 1 + u σ m, C 4 = 1 4! [u4 4u 3 m +6u (m σ ) 4u 1 (m 3 3σ m)+u (m 4 +3σ 4 ) 6σ m ], C 5 = 1 5! [u5 5u 4 m +1u 3 (m σ ) 1u (m 3 3σ m)+5u 1 (m 4 6σ m +3σ 4 ) u (m 5 1σ m 3 +15σ 4 m)], (31) e 1 (t) = A 1 s(t)+tanh(5 A 1 s(t)), e (t) = tanh(a s(t)) (41) where the data length is 1 t and A i = ith row of A These nonlinear mixtures are separated via the procedure of Fig The learning step size η is and 35 iterations were employed We present in Fig3-4 the behaviour of the proposed algorithm The joint distribution of the original sources, nonlinear mixtures and unmixed output are each displayed in the third row of Fig3 The original sources are both independent and the joint distribution is contained within a simple parallelogram The linear mixing matrix A and nonlinear functions of eq(41) are applied, producing nonlinear mixtures The corresponding joint distribution is no longer contained in a simple parallelogram At the final stage, the proposed nonlinear unmixing method has successfully compensated the effects of the nonlinear mixing operation Figure 4 shows the estimated nonlinear separating functions g 1, g and the corresponding compensation functions g i R 1 i

5 96 ECTI TRANSACTIONS ON ELECTRICAL ENG, ELECTRONICS, AND COMMUNICATIONS VOL3, NO AUGUST 5-1 Source Source 1 Nonlinear Mixture Nonlinear Mixture 1-1 Output Output 1 BER 1-1 noiseless -5dbW -1dBw -dbw Joint Distribution -1 1 Joint Distribution 1-1 Joint Distribution 1-4 Source Source Mixture Mixture 5 Output Output Fig3: The joint distributions of original sources, nonlinear mixtures and unmixed output signals 4 - nonlinear function, g nonlinear function, g Iteration 1 3 Fig5: BER of the output signal as a function of the 8 iterations over various noise power levels 4 3 Performance Analysis, when n> Consider now, 4, 6 sets of independent sources The components of s(t) are independent random variables identically and uniformly distributed in [ 3, 3] The proposed algorithm is applied to these mixtures with η =1 The data length is t = 5 and 1 iterations are employed 4 nonlinear function,g 1 compensation - nonlinear function, g compensation n=6 sources n=4 sources n= sources Fig4: The nonlinear transfer function has compensated the effect of the nonlinear distortion BER Performance Analysis, Noise Present Figure 5 displays the performance analysis (Bit Error Rate) with various level of Additive White Gaussian Noise (AWGN) present in the mixed signals The SNR in all cases is 1dB, the learning [ rate η] is 1, 1 6 the linear mixing matrix, A = and the 7 1 post nonlinear functions are same as in eq(41) As displayed in the previous experimental results, the proposed equivariant based algorithm is able to compensate nonlinear mapping in noiseless conditions This is further demonstrated by the result of Fig5 Notice that the noiseless system achieves the BER of 1 48 after iterations The results with noise present are achieved without any noise filtering Better results are achievable by employing conventional noise filtering Iteration 1 3 Fig6: BER over 1 iterations for, 4 and 6 different sources 4 4 Comparison with and Algorithm We present in Fig7 the behaviour of the proposed algorithm in comparison with the [16] and [, 8] algorithms We consider n = 6 and 1 nonlinear mixtures The output displays for the proposed algorithm, and are compared in

6 Using Kernel Density Estimator in Nonlinear Mixture 97 MSE Sources Sources MSE Sources Sources Per I ndex Mutual-Info Sources Per I ndex Mutual-Info Sources Fig7: Performance analysis between the proposed nonlinear algorithm, the and the method for n =6and 1 nonlinear mixtures: Top row: Mean-squared-error, Middle row: Mutual information, Bottom row: Performance index versus 35 iterations terms of mean-squared-error, mutual information and performance index respectively The performance index PI: PI = 1 ( n ( n p ij ) n max n p in 1 + j=1 i=1 i=1 j=1 n ( n p ij )) max n p nj 1 (4) where p ij is the i, j element of p = WgRA Note p is close to a permutation of the scaled identity matrix when the sources are separated This corresponds to PI = It can be seen that the proposed nonlinear algorithm is able to better recover the sources from the nonlinear distorted mixtures compared with the and algorithm Note, the algorithm because of its equivariant characteristic provides relatively good signal unmixing, but is still noticeably inferior to the proposed algorithm In our observation, the which is a linear ICA method is not generally suitable for separation of nonlinear mixtures; the output signals are distorted, unstable and failed to be unmixed properly A measure of separation performance is given by the similarity in kurtosis of the source and the corresponding unmixed signals Table 1 shows the kurtosis

7 98 ECTI TRANSACTIONS ON ELECTRICAL ENG, ELECTRONICS, AND COMMUNICATIONS VOL3, NO AUGUST 5 Table 1: The zero-mean white sources with sub-gaussian distribution tested with 4 and 6 nonlinear mixing Original Mixes Kurtosis (6 sources) Kurtosis (4 sources) Sources Kurtosis Kurtosis Recovered Recovered INFOMAX INFOMAX results for 4 and 6-mixing It can be seen that the proposed algorithm provides, in general, better kurtosis matching of source and output signals Note, the source signals are sub-gaussian and hence have negative kurtosis 5 CONCLUSION In this paper, we considered the problem of blind separation of nonlinearly mixed source signals We proposed a multilayer perceptron method which incorporates Gram-Charlier kernel density estimation and equivariant gradient learning to compensate the nonlinear mixing Simulation results demonstrate convergence and robustness of the proposed nonlinear algorithm Also, we compared and investigated the proposed nonlinear algorithm with the and algorithms Experimental results show that the proposed algorithm outperformed the and algorithm with lower BER and better independency References [1] PComon, Independent Component Analysis, A New Concept?, Special Issue on Higher-Order Statistics, vol 36, pp , April 1994 [] AHyvarinen and EOja, Independent Component Analysis: A Tutorial, Helsinki University of Technology, Finland April 1999 [3] TWLee, Independent Component Analysis: Theory and Applications Netherlands: Kluwer Academic Publishers, 1998 [4] Jutten and Herault, Independent Component Analysis versus Principal Components Analysis, presented at ProcEuropean Signal Processing Conf EUSIPCO, Grenoble, France, 1988 [5] ACichocki and SIAmari, Adaptive Blind Signal and Image Processing England: John Wiley & Sons,Ltd, [6] AHyvarinen, JKarhunen, EOja, Independent Component Analysis Canada: John Wiley & Sons, Inc, 1 [7] TRistaniemi, JJoutsensalo, Advanced ICAbased Receivers for Blocking Fading DS-CDMA Channels, Signal Processing, vol 8, pp , [8] Bell and Sejnowksi, An Information Maximization Approach to Blind Separation and Blind Deconvolution, Neural Computation, vol7,pp , 1995 [9] HPark, S-IAmari, KFukumizu, Adaptive Natural Gradient Learning Algorithms for Various Stochastic Models, Neural Networks, vol 13, pp , [1] Tobias Blaschke, Laurenz Wiskott, CuBICA: Independent Component Analysis by Simultaneous Third and Fourth Order Cumulant Diagonalization, Computer Science Preprint Server (CSPS), April 9, 3 [11] ATaleb, CJutten, Source Separation in Post- Nonlinear Mixtures, IEEE Transaction on Signal Processing, vol 47, pp 87-8, 1999 [1] HValpola, XGiannakopoulos, AHonkela, JKarhunen, Nonlinear Independent Component Analysis Using Ensemble Learning: Experiments and Discussion, Helsinki University of Technology, Neural Networks Research Centre, Finland [13] ATaleb, CJutten,SOlympieff, Source Separation in Post Nonlinear Mixtures: An Entropy- Based Algorithm, presented at International Conference on Acoustics, Speech, and Signal Processing, Washington, USA, May 1998 [14] SHaykin, Neural Networks: A Comprehensive Foundation, nd ed Englewood Cliffs, NJ: Prentice Hall, 1999 [15] ATaleb, A generic framework for blind source separation in structured nonlinear models, IEEE Trans on Signal Processing, vol 5, pp , 1 [16] JFCardoso and BHLaheld, Equivariant Adaptive Source Separation, IEEE Transactions on Signal Processing, vol 44, pp , Dec 1996 [17] JKarhunen and PPajunen, Hierarchic Nonlinear PCA Algorithms for Neural Blind Source Separation, presented at In NORSIG 96 Proceesings IEEE Nordic Signal Processing Symposium, Espoo, Finland, Sept 1996 [18] APelliccioni, The Gram-Charlier Method to Evaluate the Probability Density Function in

8 Using Kernel Density Estimator in Nonlinear Mixture 99 Monodimensional Case, Il Nuovo Cimento, vol, 1997 [19] MO brien, Using the Gram-Charlier expansion to produce Vibronic Band Shapes in Strong Coupling, Journal Physics: Condensed Matter, vol 4, pp , 199 WYLeong received the Bdegree in Electrical Engineering from The University of Queensland, Australia in She is currently working towards the PhD degree in the School of Information Technology and Electrical Engineering in the same university Her research interests include signal processing and telecommunications John Homer received the BSc degree in physics from the University of Newcastle, Australia in 1985 and the PhD degree in systems engineering from the Australian National University, Canberra, Australia, in 1995 Between his BSc and PhD studies he held a position of Research Engineer at Comalco Research Centre in Melbourne, Australia Following his PhD studies he has held research positions with the University of Queensland, Veritas DGC Pty Ltd and Katholieke Universiteit Leuven, Belgium He is currently a senior lecturer at the University of Queensland within the School of Information Technology and Electrical Engineering His research interests include signal and image processing, particularly in the application areas of telecommunications, audio and radar He is currently an Associate Editor of the Journal of Applied Signal Processing