A Derivation of the Steady-State MSE of RLS: Stationary and Nonstationary Cases

Transcription

1 A Derivation of the Steady-State MSE of RLS: Stationary and Nonstationary Cases Phil Schniter Nov. 0, 001 Abstract In this report we combine the approach of Yousef and Sayed [1] with that of Rupp and Sayed [] to derive the steady-state mean-squared error MSE) of the recursive least squares RLS) algorithm in both stationary and non-stationary environments. Comparisons with the steady-state MSE of LMS are included. 1 Introduction The recursive least squares RLS) parameter update equation can be written as wn) wn 1) + Pn)un) dn) w H n 1)un) ) kn) ξ n) where Pn) is the inverse of the deterministic autocorrelation matrix Φn) Pn) Φ 1 n) n Φn) λ n i ui)u H i) + δλ n I i1 with 0 < λ 1. We assume a nonstationary environment with time-varying Wiener filter w n) that evolves as a random walk: dn) w H n 1)un) + ξ n) w n) w n 1) + qn) 1) ξ n) denotes the MMSE version of ξn) which, due to the MMSE orthogonality principle, is uncorrelated with un): Revised November 14, 003. Eun)ξ n) 0 1

2 Other statistical assumptions will be stated later. We will make extensive use of the parameter error vector wn), the a posteriori error e p n), and the a priori error e a n). wn) : w n) wn) e p n) : dn) w H n)un) ) ξ n) w H n) qn) ) H un) e a n) : dn) w H n 1)un) ) ξ n) ξn) ξ n) ) w H n 1)un) Energy Relation In this section we will derive a fundamental deterministic energy relationship. wn) qn)) H un) e p n) Note the use of the weighted Euclidean norm wn) wn 1) + Pn)un)ξ n) wn) wn 1) Pn)un)ξ n) + qn) 3) w H n 1)un) u H n)pn)un) ξn) 4) e a n) un) Pn) a B : a H Ba where B is a Hermitian positive-semi-definite matrix. Equation 4) implies that ξn) un) Pn) ea n) e p n) ) when un) 0 wn 1) un) Pn) wn) qn) Pn)un) e a n) e p n) ) un) 0 wn 1) un) 0 wn 1) µn)pn)un) e a n) e p n) ) where we use the psuedo-inverse to define µn) : ) un) Pn) Taking the P 1 n)-weighted norm of both sides of 5), we get wn) qn) for P 1 n) the left side and the following quantity for the right. ) w H n 1) µn)e a n)u H n)p n) + µn)e p n)u H n)pn) P 1 n) ) wn 1) µn)e an)pn)un) + µn)e pn)pn)un) wn 1) P 1 n) µn) e an) + µn)e pn)e a n) µn) e a n) + µn) e a n) µn)e a n)e p n) + µn)e p n)e a n) µn)e pn)e a n) + µn) e pn) wn 1) P 1 n) µn) e an) + µn) e p n) 5)

3 The energy relation is then summarized as wn) qn) P 1 n) + µn) e an) wn 1) P 1 n) + µn) e pn) 6) 3 Random Walk Assumptions We assume the following about the random-walk driving-noise qn) in 1). A.1) Eqn) 0 Eqn)q H n k) Qδ k where δ k denotes the Kronecker delta qn) un) qn) ξn) Examining the expectation of the leftmost term of the energy relation 6), we have E wn) qn) P 1 n) E wn) P 1 n) R EqH n)p 1 n) wn) + E qn) P 1 n) 7) From 3) we have wn) wn 1) + qn) Pn)un)ξ n) allowing us to simplify the second term on the right side of 7): E q H n)p 1 n) wn) E q H n)p 1 n) wn 1) + E qn) P 1 n) E qn) P 1 n) E q H n)un)ξ n) where the first and third terms vanished as a result of A.1). Plugging the previous realvalued) expression into 7) yields E wn) qn) P 1 n) E wn) P 1 n) E qn) P 1 n) 8) 4 Steady-State Analysis We claim that when the adaptation algorithm has reached steady state, E wn) P 1 n) E wn 1) P 1 n) Taking the expectation of 6) and incorporating 8), we find the following steady state relationship: E µn) e a n) E µn) e p n) + E qn) P 1 n) 9) 3

4 To proceed further, we make a few assumptions: A.) E un)u H n) R and R > 0 A.3) ξ n) i.i.d. ξ n) un) ξ n) wn 1) ξ n) e a n) A.4) E Pn)un)u H n) E Pn) E un)u H n) e.g., from λ 1) A.5) E un) Pn) e an) E un) Pn) E ea n) e.g., from large M) A.6) lim E Φ 1 n) lim E Φn) ) 1 e.g., Wishart theory [3, p. 45]) First, using the implications of A.3), we have Jn) : E ξn) E e a n) + ξ n) E e a n) + E ξ n) E e a n) + J min J emse : lim Jn) J min lim E e a n) Then, from ), 4), and A.3), we see that the first term on the right side of 9) can be written as E µn) e p n) E µn) e a n) un) Pn)ξn) E µn) 1 un) Pn)) ea n) + un) Pn)ξ n) E µn) ea n) E e a n) + E un) Pn) e an) + E un) Pn) ξ n) So the steady-state relationship 9) becomes after cancellation of E µn) e a n) terms) E e a n) E un) Pn) e an) + E un) Pn) ξ n) + E qn) P n) 10) 1 Looking at the first term on the right side of 10), we have, from A.4) and A.5), E un) Pn) e an) E un) Pn) E ea n) tr E Pn)un)u H n) ) E e a n) tr E Pn) E un)u H n) ) E e a n) tr E Pn) ) R E e a n) 4

5 where E Pn) requires further investigation. Using the fact that E Φn) n λ n i Eun)u H n) + δλ n I we find, via A.6), implying that i1 lim E Pn) 1 λ n 1 λ δλn I λ < 1 nr + δi λ 1 lim E un) Pn) e an) lim E Φn) ) 1 1 λ)r 1 λ < 1 0 λ 1 M1 λ)j emse λ < 1 0 λ 1 Looking next at the second term on the right side of 10), we get from A.3) and A.6) lim E un) Pn) ξ n) M1 λ)j min λ < 1 0 λ 1 For the last term on the right side of 10), assumptions A.1) and A.) yield E qn) P 1 n) E q H n)φn)qn) tr E qn)q H n)φn) ) tr E qn)q H n) n λ n i E un)u H n) + δλ n E qn)q H n) ) so that i1 tr QR ) n λ n i + δλ n tr Q ) i1 1 λ n 1 λ tr QR ) + δλ n tr Q ) λ < 1 n tr QR ) + δ tr Q ) λ 1 1 λ) lim E 1 tr QR ) λ < 1 qn) P 1 n) λ 1 and Q 0 0 λ 1 and Q 0 Collecting our findings on the asymptotic behavior of 10), we have M1 λ)j emse + M1 λ)j min + 1 λ) 1 tr QR ) λ < 1 J emse λ 1 and Q 0 0 λ 1 and Q 0 5

6 which, for the λ < 1 case, gives J emse, RLS M1 λ)j min + 1 λ) 1 tr QR ) 11) M1 λ) 1 M1 λ)j min + 1 λ) 1 tr QR )) when M1 λ) 1) Note the EMSE component due to J min and due to time variations i.e., Q). 5 Optimum Forgetting Factor The value of λ that minimizes J emse can be obtained by zeroing the partial derivative of 1) with respect to 1 λ). This yields tr QR ) 6 Comparison to LMS Recall the counterparts of 11) 14) for LMS: λ opt 1 13) MJ min J emse, RLSopt MJ min tr QR ) 14) µ trr)j min + µ 1 trq) µ trr) 1 ) µ trr)j min + µ 1 trq) trq) µ opt J min trr) 15) when µ trr) 16) 17) opt J min trr) trq) 18) Comparing the optimal values of EMSE for LMS and RLS we find opt J emse, RLS opt trr) trq) M tr QR ) 19) Thus the relationship between R and Q will determine which of the two algorithms gives lower steady-state MSE. A few instructive cases are presented below [3]. It is interesting to note that the simple LMS algorithm can have exhibit tracking performance superior to the computationally-intensive RLS algorithm in certain environments. However, it should be noted that the convergence of RLS is typically much faster than that of LMS. 6

7 Q σqi: In this case we find that the two algorithms provide essentially the same level of steady-state MSE. J emse MJ min σq trr) Q cr: In this case LMS yields lower steady-state MSE: opt J emse, RLS opt trr) M trr ) < 1 Q cr 1 : In this case RLS yields lower steady-state MSE: opt J emse, RLS opt 1 M trr) trr 1 ) > 1 The inequalities above can be proven using the Cauchy-Schwarz inequality for vectors: x H y x y. References [1] N. B. Yousef and A. H. Sayed, A unified approach to the steady-state and tracking analyses of adaptive filters, IEEE Trans. on Signal Processing, vol. 49, pp , Feb [] M. Rupp and A. H. Sayed, Robustness of Gauss-Newton recursive methods: A deterministic feedback analysis, Signal Processing, vol. 50, pp , [3] S. Haykin, Adaptive Filter Theory. Englewood Cliffs, NJ: Prentice-Hall, 4th ed.,