KNOWN approaches for improving the performance of

Transcription

1 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS II: EXPRESS BRIEFS, VOL. 58, NO. 8, AUGUST Robust Quasi-Newton Adaptive Filtering Algorithms Md. Zulfiquar Ali Bhotto, Student Member, IEEE, and Andreas Antoniou, Life Fellow, IEEE Abstract Two robust quasi-newton (QN) adaptive filtering algorithms that perform well in impulsive-noise environments are proposed. The new algorithms use an improved estimate of the inverse of the autocorrelation matrix and an improved weight-vector update equation, which lead to improved speed of convergence and steady-state misalignment relative to those achieved in the nown QN algorithms. A stability analysis shows that the proposed algorithms are asymptotically stable. The proposed algorithms perform data-selective adaptation, which significantly reduces the amount of computation required. Simulation results presented demonstrate the attractive features of the proposed algorithms. Index Terms Adaptive filters, impulsive noise in adaptive filters, quasi-newton algorithms, robust adaptation algorithms. I. INTRODUCTION KNOWN approaches for improving the performance of adaptive filters in impulsive-noise environments involve the use of nonlinear clipping [1], [2], robust statistics [3] [5], or order statistics. The common step in the adaptive filters reported in [1] [5] is to detect the presence of impulsive noise by comparing the magnitude of the error signal with the value of a threshold parameter, which is a scalar multiple of the variance of the error signal, and then either stop or reduce the learning rate of the adaptive filter. In [1], the variance of the error signal is estimated by averaging the square of its instantaneous values, but this approach is not robust with respect to impulsive noise. In [2] [5], improved robustness is achieved by estimating the variance of the error signal using the median absolute deviation [6]. The adaptation algorithms in [1] and [3] use the Huber mixed-norm M-estimate objective function [6], and the algorithms in [4] and [5] use the Hampel three-part redescending M-estimate objective function [6]. The Huber function uses the L 1 norm to measure the amplitude of the error signal when the absolute error is larger than the threshold. The Hampel function, on the other hand, assigns a constant value to the error signal when the absolute error becomes larger than the threshold. Algorithms based on the Huber and Hampel functions offer similar performance. The nonlinear recursive least-squares (NRLS) algorithm in [2] uses nonlinear clipping to control the learning rate and offers better performance in impulsive-noise Manuscript received October 1, 2010; revised December 31, 2010 and March 16, 2011; accepted May 14, Date of publication July 22, 2011; date of current version August 17, This wor was supported by the Natural Sciences and Engineering Research Council of Canada. This paper was recommended by Associate Editor Z. Lin. The authors are with the Department of Electrical and Computer Engineering, University of Victoria, Victoria, BC V8W 3P6, Canada ( zbhotto@ece.uvic.ca; aantoniou@ieee.org). Color versions of one or more of the figures in this paper are available online at Digital Object Identifier /TCSII environments than the conventional RLS algorithm. The recursive least-mean (RLM) algorithm reported in [4] offers faster convergence and better robustness than the NRLS algorithm in impulsive-noise environments. The quasi-newton algorithm (QN) in [7] is not robust against impulsive noise. Simulation results in [5] show that the recursive QN (RQN) algorithm offers faster convergence and improved robustness in impulsive-noise environments relative to the QN algorithm in [7]. Algorithms of the Newton family such as those in [2], [4], [5], and [7] converge much faster than algorithms of the steepest-descent family [8]. The RLS, RLM, and RQN algorithms exponentially forget past input signal vectors in the estimate of the autocorrelation matrix, and therefore, the positive definiteness of the inverse of the autocorrelation matrix can be lost [9], [10]. As a result, this type of adaptive filter can become unstable in finiteprecision implementations. This form of explosive divergence in RLS adaptive filters is well documented in the literature, and ways and conditions to stabilize these filters are addressed in [9] and [10]. The QN algorithm reported in [7] is shown to be robust in terms of explosive divergence, as compared with RLS adaptive filters. In this brief, we propose two new robust QN algorithms that perform data-selective adaptation in updating the inverse of the autocorrelation matrix and the weight vector. The new algorithms are essentially enhancements of the algorithms we reported in [16] for applications that entail impulsive noise. A stability analysis shows that the proposed algorithms are asymptotically stable. Furthermore, simulation results show that the proposed algorithms offer improved performance relative to two nown QN (KQN) algorithms with respect to robustness, steady-state misalignment, computational efficiency, and tracing capability. This brief is organized as follows. In Section II, the proposed robust QN algorithms are described. In Section III, stability issues of the algorithms are discussed, and in Section IV, some practical issues concerning the implementation of the proposed algorithms are examined. Simulation results are presented in Section V, and conclusions are drawn in Section VI. II. PROPOSED ROBUST QN ALGORITHMS Two slightly different robust QN algorithms are possible, one using a fixed threshold and the other using a variable threshold, as will now be demonstrated. A. RQN Algorithm with a Fixed Threshold The objective of the proposed adaptation algorithm is to generate a series of weight vectors that would eventually solve the optimization problem minimize w E [ (d w T x ) 2] (1) /$ IEEE

2 538 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS II: EXPRESS BRIEFS, VOL. 58, NO. 8, AUGUST 2011 recursively, where E[ ] is the expected value of [ ], x is a vector of dimension M representing the input signal, d is the desired signal, and w is the weight vector, which is also of dimension M. An approximate solution of the problem in (1) can be obtained by using the weight-vector update equation w = w 1 +2μ S 1 x e (2) where μ is the step size, S 1 is a positive definite matrix of dimension M M, and e = d y (3) is the apriorierror for the output signal y = w T 1 x.ifs 1 in (2) is chosen as the M M identity matrix, then the update equation in (2) would minimize the objective function J w 1 = ( d w T 1x ) 2 (4) with respect to the steepest-descent direction, and a series of updates would eventually yield an approximate solution of the problem in (1). Other choices of S 1 would entail different search directions but would serve the same purpose as long as S 1 is positive definite. In order to use an approximation of the Newton direction S 1 in (2), we obtain S 1 by using the gradient of J w 1 in (4) in the ran-one update formula of the classical QN optimization algorithm [11], which is given by S = S 1 (δ S 1 ρ )(δ S 1 ρ ) T (δ S 1 ρ ) T (5) ρ where δ = w w 1 (6) ρ = e2 +1 w e2 w 1. (7) This formula satisfies the Fletcher QN condition S ρ = δ [12]. From (3), we note that e +1 would require future data x +1 and d +1. To circumvent this problem, we use the a posteriori error ɛ = d x T w (8) in place of e +1 in (7). As a first step in the proposed algorithm, we obtain a value of step size μ in (2) by solving the optimization problem minimize μ { d x T w γ, if e >γ 0, otherwise where γ is a prespecified error bound. The solution of this problem can be obtained as 1 μ = α (10) 2τ where τ = x T S 1x, and { 1 γ α = e, if e >γ 0, otherwise. (9) (11) In effect, step size μ is chosen to force equality ɛ = γ whenever e >γ. Since μ in (10) forces ɛ to assume the value of the prespecified error bound, i.e., ɛ = γ sign(e ) (12) we obtain ɛ 2 = 0, and hence from (7), we have ρ =2e x. (13) Vector δ, which is linearly dependent on S 1 x, can be obtained by using (2) and (6). Since the equality in (12) is satisfied for each update, we can use the a posteriori error to obtain δ =2γ sign(e )S 1 x (14) instead of the apriorierror used in the KQN algorithm reported in [8]. Now, substituting ρ and δ given by (13) and (14) in (5), we obtain an update equation for matrix S for the proposed robust QN algorithm as S 1 x x T S = S 1 α S 1 x T S. (15) 1x Substituting μ given by (10) in (2), we obtain the corresponding weight-vector update equation as S 1 x w = w 1 + α x T S e. (16) 1x In order to achieve robust performance against impulsive noise, we choose error bound γ as { e υθ γ =, if e >θ (17) γ c, otherwise where γ c is a prespecified error bound chosen as 5σ v, where σ 2 v is the variance of the measurement noise, 0 υ 1 is a scalar, and θ is a threshold parameter, which is estimated as where θ =1.98σ (18) σ 2 = λσ 2 1 +(1 λ) min(g ) (19) with 0 λ<1, g =[e 2 + ɛ e2 P +1 + ɛ ] is a vector of dimension P, where ɛ is a small scalar. Whenever e <γ, no update is applied. Consequently, the amount of computation and the required storage are significantly reduced since S is not evaluated in every iteration. A similar weight-vector update strategy has been used in set-membership adaptation algorithms [13] [15], but the mathematical framewor of those algorithms is very different from that of the proposed robust QN algorithm. The estimator in (19) is robust to outliers. A large σ0 2 would cause e to be less than θ during the transient state, and therefore, the algorithm would wor with error bound γ c, which would increase the initial rate of convergence. For a sudden system disturbance, θ would also be very large, in which case we obtain γ c < e <θ, and thus, the algorithm would again use error bound γ c, and therefore, the tracing capability of the algorithm would be retained. For an impulsive noise-corrupted error signal, θ would not increase, in which case the error bound would be θ = e υθ, and this would suppress the impulsive noise.

3 BHOTTO AND ANTONIOU: ROBUST QUASI-NEWTON ADAPTIVE FILTERING ALGORITHMS 539 B. RQN Algorithm With a Variable Threshold The RQN algorithm with a variable threshold is essentially the same as before except that the error bound γ c in (17) is estimated as ( ) d 2 ξ = βξ 1 +(1 β) min ξ 1, y 2 d 2 (20) ˆσ 2 = βˆσ 1 2 +(1 β) min (ˆσ 1,σ 2 2 ) (21) γ c, = ξ γ c, [1 + sign(1 ξ )] ˆσ (22) where γ c,0 is a rough estimate of σ v, and ξ 0 1. During steady state, we obtain ξ 0 and ˆσ 2 σ2 v, and hence, γ c, would be 2.24σ v, and this would yield reduced steady-state misalignment. In the transient state, ξ ξ 0, and therefore, the algorithm would wor with γ c, = ξ γ c,0, and this would yield faster convergence. The parameters in (20) and (21) are robust with respect to outliers as each is based on the minimum of its two most recent values. The variance is estimated as σ 2 = λσ 2 1 +(1 λ)median(g ). (23) For Gaussian signals, the median of the squares of the signal samples in the variance estimator usually gives a more accurate estimate of the variance of the signal than the instantaneous values of the squares of the signal samples. The use of the median operation in adaptive filters was introduced by Zou et al. in [4] and was later used by other researchers, e.g., in [18]. A variable threshold γ c is useful in applications where the noise variance σ 2 v is unnown. C. Discussion In the proposed algorithms, we obtain 0 α < 1 for both values of γ in (17), and hence, the estimated S in (15) would remain positive definite indefinitely if S is initialized with a positive definite matrix [16]. The RLS-type robust algorithms in [17] and [18] are implemented using a fast transversal filter implementation [8] to reduce their computational complexity from order M 2, which is denoted by O(M 2 ),too(m), and therefore, both algorithms would inherit the problems associated with the RLS algorithms of order M 2. Moreover, the Huber function used in [17] does not have a closed-form solution, and hence, the solution obtained can be suboptimal, and in addition, its tracing capability can be compromised [18]. The solution obtained in [18] can also be suboptimal, as shown in [18]. The adaptation algorithm in [18] is robust in the sense that it returns to the true solution without losing its initial convergence speed after being subjected to impulsive noise. Similarly, the proposed robust QN algorithms are robust with respect to impulsive noise in the sense that they return to the true solution faster than the initial convergence. The KQN and nown RQN (KRQN) algorithms do not employ a variable step size. However, the KRQN algorithm requires an additional amount of computation of O[M log 2 (M)] per iteration, as compared with the KQN algorithm and the proposed QN algorithm with a fixed threshold. III. STABILITY ANALYSIS Here, we address stability issues associated with the proposed RQN algorithms. For purposes of analysis, we assume that the desired response for the adaptive filter is generated as d = x T w opt (24) where w opt is the weight vector of an FIR filter. We establish the convergence behavior of the proposed RQN algorithm by examining the behavior of the weight-error vector, which is defined as w = w opt w. (25) Using (24) and (25), the error signal in (3) can be expressed as e = x T w 1. (26) Subtracting w opt from both sides of (16) and using the above relations, we obtain ( S 1 x x T ) w = I α x T S w 1 1x = P w 1. (27) The global asymptotic convergence of the weight-error vector can be assured if and only if E[P ] is time invariant and its eigenvalues are strictly inside the unit circle. However, certain strong independence assumptions have to be made to obtain a time-invariant description of a system such as that represented by (27). As an alternative, conditions for convergence with probability 1 can be achieved by using a system based on w rather than E[ w ]. A similar approach has been used to demonstrate the stability of other algorithms, such as, for example, the KQN algorithm in [7] and the constrained affine projection algorithm in [13]. Before considering the stability of the proposed RQN algorithm, we apply the matrix inversion lemma [8] to (15) to obtain R = S 1 = R α 1 + x x T (28) (1 α )τ where τ = x T S 1x. Since 0 <α < 1 in each update, matrices S and R remain positive definite and bounded indefinitely [16], and therefore, the following theorem can be established. Theorem 1: If the input signal is persistently exciting, then the system in (27) is asymptotically stable, and consequently, the proposed RQN algorithms are also asymptotically stable. Proof: Since matrices S and R are bounded and invertible according to Lemma 1 in [7], an equivalent system of equations (in the Lyapunov sense) for w in (27) can be obtained as w = R 1/2 w (29) where R = R T/2 R 1/2. If the system represented by (29) is stable or unstable, then the system represented by (27) is also stable or unstable, as appropriate. Taing the Euclidean norm of both sides of (29), we obtain w 2 = w T R w. (30)

4 540 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS II: EXPRESS BRIEFS, VOL. 58, NO. 8, AUGUST 2011 Substituting w given by (27) in (30), we get w 2 = w T 1P T R P w 1. (31) On the other hand, substituting R given by (28) in (31), we have w 2 = w T 1P T R 1 P w 1 α + w T (1 α )τ 1P T x x T P w 1. (32) It is easy to verify that P T R 1 P = R 1 (2 α ) α x x T τ (33) P T x x T P =(1 α ) 2 x x T. (34) Now, if we use (33) and (34) in (32) and then use (30), we have w 2 = w 1 2 α w T τ 1x x T w 1. (35) Since α (0, 1) and τ > 0 for all, we obtain w 2 w 1 2. (36) The left-hand side in (36) would be equal to the right-hand side in an interval [ 1, 2 ] if and only if x remains orthogonal to w 1 for all [ 1, 2 ]. However, in such a situation, we would obtain w = w 1 for all [ 1, 2 ] in (27). Therefore, w 2 = w 1 2 would hold true if and only if w 1 = w 1 +1 = = w 2 = w. However, if the input signal is persistently exciting, then there is an infinite number of sets S i = {x 1,i,...x 2,i} with M 2,i 1,i < M such that each set S i completely spans R M for some finite value of M > 0 [7], [14]. Thus, it would be impossible for w 1 to be orthogonal to x for all x S i, and as a result, w 2 < w 1 2, which proves that the system in (27) and, in turn, the proposed RQN algorithms are stable. As the number of sets is infinite, it follows that w 2 0 for, and therefore, the system in (27) and the proposed RQN algorithms are both stable and asymptotically stable. IV. PRACTICAL CONSIDERATIONS In low-cost fixed-point hardware implementations, the accumulation of roundoff errors can cause S to lose its positive definiteness. This problem can be eliminated by periodically reinitializing S using the identity matrix. This is also done in the QN algorithm in [7] whenever x T S 1x < 10 3 for the case of a fixed-point implementation and also in the classical optimization context. However, very frequent reinitializations could slow the convergence of the algorithm. A compromise number of reinitializations can be achieved by reinitializing S whenever e > 2.576σ, in which case the probability of reinitialization in a given iteration assumes the value of Reinitialization of other parameters is not required. In floatingpoint arithmetic, reinitialization is unnecessary. The variance estimator of the error signal in (19) should be initialized with a large value to ensure that the probability that γ = γ c in the transient state is increased. Although a rough choice of σ 2 0 would wor, we have used σ 2 0 = c 1 M/σ 2 v, where 0 <c 1 < 1 is a positive constant. This choice yields good results. The forgetting factors in (19) and (21) are chosen as λ =1 1/(c 2 M) and β =1 1/(c 3 M), respectively, where c 2 and c 3 are positive scalars [18]. The length P of vector g should be greater than the duration of the impulsive noise. A reduced υ in (17) would yield reduced steady-state misalignment and improved robustness with respect to impulsive noise. On the other hand, an increased γ c and θ would reduce the number of updates and yield reduced steady-state misalignment. However, the convergence speed would be compromised in such a situation. V. S IMULATION RESULTS In order to evaluate the performance of the proposed RQN algorithms with fixed and variable γ c, which are designated as the PRQN-I and PRQN-II algorithms, respectively, a system identification application was considered, as detailed below. For the sae of comparison, simulations were carried out with the KQN and KRQN algorithms in [5] and [7], respectively. The adaptive filter was used to identify a 36th-order low-pass FIR filter with a normalized cutoff frequency of 0.3. The input signal was generated by filtering a white Gaussian noise signal with zero mean and unity variance through a recursive filter with a single pole at The measurement noise added to the desired signal was a white Gaussian noise signal with zero mean and variances of 10 3 and 10 6 (signal-to-noise ratios (SNRs) of 30 and 60 db, respectively). The impulse response of the FIR filter was multiplied by 1 at iteration 2000 to chec the tracing capability of the algorithm. Impulsive noise with a maximum duration of 3T s, where T s is the sampling duration, was added to the desired signal at iterations 1000, 1300, and 3300 using a Bernoulli trial with probability [19]. All algorithms were initialized with S 0 = I, and w 0 = 0. The parameters for the PRQN-I and PRQN-II algorithms were set to υ =0.30, P =5, c 1 =1/(2M), c 2 =0.5and υ =0.30, P =15, ξ 0 =10, ˆσ 0 2 =10, c 3 =2, respectively. We have explored two options for the evaluation of S in the implementation of the KRQN algorithm, first using {[ ] } 1 λq(e )τ S = λs 1 + q(e )τ 2 S 1 x x T S 1 (37) and then using [ ] 1 λq(e )τ S = λs 1 + {S 1x q(e )τ 2 x T } S 1. (38) Although (37) and (38) are equivalent, the implementation of (37) was subject to numerical ill-conditioning, which caused instability. On the other hand, the implementation of (38) was found to be more robust, although it requires increased computational effort. We have used (38) in our implementation of the KRON algorithm. The parameters for the KRQN algorithm were set to the values suggested in [5]. The learning curves obtained from 1000 independent trials by using the KQN, KRQN, PRQN-I, and PRQN-II algorithms are illustrated in Fig. 1(a) and (b). As shown, the PRQN-I and PRQN-II algorithms offer robust performance with respect to impulsive noise and yield significantly reduced steady-state misalignment relative to those in the other algorithms. On the other hand, the KQN algorithm is seriously compromised

5 BHOTTO AND ANTONIOU: ROBUST QUASI-NEWTON ADAPTIVE FILTERING ALGORITHMS 541 Fig. 1. Learning curves for system-identification application. (a) SNR =30dB. (b) SNR =60dB. by impulsive noise in terms of both robustness and tracing capability. The KRQN algorithm also offers robust performance with respect to impulsive noise, but its tracing performance is not as good as those of the PRQN-I and PRQN-II algorithms. The total numbers of updates required by the PRQN-I and PRQN-II algorithms were 2230 and 1356 for an SNR of 30 db, and 2070 and 1207 for an SNR of 60 db, respectively, as compared with 4000 in the other algorithms. Note that two systems were identified in this experiment, one before iteration 2000 and the other after iteration Otherwise, the number of updates would be reduced to half. VI. CONCLUSION Two new robust QN adaptation algorithms have been developed on the basis of the mathematical framewor of the classical QN optimization algorithm, which lead to an improved estimate of the inverse of the Hessian. Lie the dataselective QN algorithm we described in [16], the proposed RQN algorithms incorporate data-selective adaptation, which significantly reduces the number of adaptations required. A stability analysis shows that the proposed RQN algorithms are asymptotically stable. Simulation results obtained in the case of a system identification application demonstrate that the proposed RQN algorithms converge faster than the KQN algorithms in [7] for medium to high SNRs. In addition, they offer improved robustness against impulsive noise, as well as improved computational efficiency and tracing relative the KRQN algorithm reported in [5]. REFERENCES [1] S. Koie, Adaptive threshold nonlinear algorithm for adaptive filters with robustness against impulsive noise, IEEE Trans. Signal Process., vol.45, no. 9, pp , Sep [2] J. F. Weng and S. H. Leung, Adaptive nonlinear RLS algorithm for robust filtering in impulse noise, in Proc. IEEE Inter. Symp. Circuits Syst., Jun. 1997, pp [3] P. Petrus, Robust Huber adaptive filter, IEEE Trans. Signal Process., vol. 47, no. 4, pp , Apr [4] Y. Zou, S. C. Chan, and T. S. Ng, A recursive least M-estimate (RLM) adaptive filter for robust filtering in impulse noise, IEEE Signal Process. Lett., vol. 7, no. 11, pp , Nov [5] Y. Zou and S. C. Chan, A robust quasi-newton adaptive filtering algorithm for impulse noise suppression, in Proc. IEEE Inter. Symp. Circuits Syst., May 2001, pp [6] P.J.RousseeuwandA.M.Leroy,Robust Regression and Outlier Detection. NewYor: Wiley, [7] M. L. R. de Campos and A. Antoniou, A new quasi-newton adaptive filtering algorithm, IEEE Trans. Circuits Syst. II, Analog Digit. Signal Process., vol. 44, no. 11, pp , Nov [8] P. S. R. Diniz, Adaptive Filtering: Algorithms and Practical Implementation, 3rd ed. NewYor: Springer-Verlag, [9] A. P. Liavas and P. A. Regalia, On the numerical stability and accuracy of the conventional recursive least squares algorithm, IEEE Trans. Signal Process., vol. 47, no. 1, pp , Jan [10] G. E. Bottomley and S. T. Alexander, A novel approach for stabilizing recursive least squares filters, IEEE Trans. Signal Process., vol.39,no.8, pp , Aug [11] A. Antoniou and W. S. Lu, Practical Optimization. New Yor: Springer- Verlag, [12] R. Fletcher, Practical Methods of Optimization. Hoboen, NJ: Wiley, [13] S. Werner, J. A. Apolinario, M. L. R. de Campos, and P. S. R. Diniz, Low complexity constrained affine-projection algorithms, IEEE Trans. Signal Process., vol. 53, no. 12, pp , Dec [14] P. S. R. Diniz and S. Werner, Set-membership binormalized data-reusing LMS algorithms, IEEE Trans. Signal Process., vol. 51, no. 1, pp , Jan [15] S. Werner and P. S. R. Diniz, Set-membership affine projection algorithm, IEEE Signal Process. Lett., vol. 8,no.8,pp ,Aug [16] M. Z. A. Bhotto and A. Antoniou, Improved quasi-newton adaptive filtering algorithm, IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 57, no. 8, pp , Aug [17] S. C. Chan and Y. Zou, A recursive least M-estimate algorithm for robust adaptive filtering in impulsive noise: Fast algorithm and convergence analysis, IEEE Trans. Signal Process., vol.52,no.4,pp , Apr [18] L. R. Vega, H. Rey, J. Benesty, and S. Tressens, A fast robust recursive least-squares algorithm, IEEE Trans. Signal Process., vol. 57, no. 3, pp , Mar [19] P. Z. Peebles, Probability, Random Variables and Random Signal Principles. NewYor: McGraw-Hill, 2000.