EEL 6502: Adaptive Signal Processing Homework #4 (LMS)

Transcription

1 EEL 6502: Adaptive Signal Processing Homework #4 (LMS) Name: Jo, Youngho WID: The purpose of this homework is to compare the performance between Prediction Error Filter and LMS (Least Mean Square Emor) Algorithm in terms of complexity of calculation, accuracy of solution (stirnation maladjustment), speed of adaptation and stationary versus non-stat ionary. vector In forward linear pmhctor, the tap inputs u(n - I), u(n - 21,.,,, u(n - M) define the M-by- 1 tapinput Hence, the correlation matrix of the tap input equals wherer(k) is the autocorrelation function ofinput process for lag k, where k = O,l,..., M -1. The cross-correlation vector between the tap inputs u(n - I), u (n - 21,..., u(n - M) and the desired response u(n) The variance of u En) equals r (0), since u(n) has zero mean. Thus, the optimum tapweight vector of the predictor is w = [w,, w,,..., w, p and on be solved by Wiener-Hopf Equation and forward predictor error power is: RW=PCSW=R-[P

2 At linear predictor, weight of prediction error filter is computed by autocomeation of input data and cross-correlation of input data and desired response using Wiener-Hopf Equation. As a result, performpnce of linear predictor depends on filter order and window size of input data in stationary and non-stationary input data. are as follow: In LMS algorithm which is one of special form of steepest descent method, procedures of computation Table 1 Summary of LMS Algorithm Parameters: M = number of taps (i.e. filter order or length) p = step-size O<p<- A- where 2- is maximum ualue of eigenvalue of autocwrelation of tap input data u(n). Initlallzation: if prior knowledge of the tap-weight vector qn) is available, use it to select an appropriate value for %O). Ohenvise, set q O) = 0. Data - Given u(n) = M - by - 1 tap - input vector at time n = [u(n),u(n - l),u(n - 21,..., u(n - M + 111' d(n) = desired response at time n - To be computed +(n + I) =estimate of tap -input vector at time n + 1 Computation: for n=0,1,2,3... compute - Estimation Error or Error signal 4n) = d(n) - iirt (n)u(n) = estimate of tap - input vector at time n + 1 +(n + 1) = +(n) + pu(n)e(n) 1 1. Complexity of Calculation Given number of taps (i.e. filter order) M - Linear Predictor: Wiener-Hopf Equation, w* = R-'P, uses matrix Inversion of autocornlation of input data and multiplication by cross-correlation matrix. Therefore, wmprexity of calculation is O(M3). - LMS: To update filter tap weight as shown using i(n + 1) = *(n)+ pu(n)e(tn) in Table 1, LMS algorithm uses matrix multiplication and addition. Therefore, complexity of cdculation is O(M). 2. Accuracy of the solution (Estimation Muadjustment) To compare accuracy of solution of linear predictor and LMS, normalized error power (MSE emr

3 divided by input power) is used. Figure 1 Comparison of Normalized Error Power of MG data Figure 2 Comparison of Normalized Error Power of Speech Signal

4 We can see that normalized error power of linear predictor (filter order 15 and window size 100) has better performance than LMS algorithm (filter order 1 5 and stepsize 0.08) in both stationary and non-stationary data as shown in Figure 1 and Figure 2. To see eewt of stepsize on eomalized error power of LMS for stationary and non-stationary data, averaged normalized error power of MG data and speech data is summarized in following Table 2 and Table 3. As you can see in Table 2 and 3, higher order filter for the same stepsize has better performance in both MG and speech data. However, smaller stepsize for the same filter order has much averaged normalized error power than bigger stepsize which satisfies 0 <,u < 1 / A-. This means that bigger stepsize has lugher speed of adaptation than smaller stepsize, has less averaged normalizsd m r power. Thmfore the more computation and data are required to reach optimal weights for smaller stepsize. Tbe effect of data size and stepsize on the speed of adaptation will be discussed in the later section. Table 2 Normalized Error Powers of LMS for MG Data Table 3 Normatized Error Powers of LMS for Speech Data In LMS algorithm, excess MSE and misadjustment M are: M excess MSE = A,, &-I) = km, u=o exccss MSE M= F: = ~ [ R I tr[~] where p is stepsize, tr[~] is trace matrix of autocorrelation matrix, and Am is eigenvaluc of autocorrclation of input data. Therefore, misadjusbnent in the LMS algorithm is proportional to stepsize p. For fdter order of 15, misadjustment of given stepsize is summarized in following Table 4. We can see that misadjustment of LMS algorithm is proportion to stepsize from Table 4.

5 Table 4 Misadjustment of LMS for different stepsize and fiiter order Stepsi w h) ,W 0.08 M= M=6 M= M= , Speed of adaptation Time constant of LMS is: The ration of largest eigenvalue and smailest eigenvalue also called eigenspread, and this varies from I to infinite. Therefore, smallest eigenvalue control adaptation time in given steps& and larger stepsize decrease time constant of LMS algorithm. Figure 3 Comparison of speed of adaptation For convergence of the LMS algorithm, we have to choose the stepsize as mentioned in Table 1. For stationary MG data, the maximum eigenvalue for filter of order 15 is Therefore, maximum stepsize

6 is and eigenspread is In top of Figure 3 wiib fixed filter order IS, we can see that learning curve of stepsize 0.08 converges faster than any other stepsize. In bottom of Figure 3 with fwd stepsize 0.04, nodzed ems power is affected by fii ter order. The cost function which is expressed by t = ~[e'(n)] = ~ [d' (n)]- 2pTw + W'RW is quadratic function of weights. In this case. MSE space is called performance surface. Figure 4 shows the 3- dimemoan1 prforrnance surface with 2 weights for MG data.

7 reaches certain area of optimal weights with increasing in raffling of weights. Stepsize 0.2, which greater than 11 A,, has much raffling and may have ovenhoots in adaptation of weight. Figure 5 Weight track of fllter order 2 with step size 0.01,0.04,0.057 and 0.2 To study the effect of data size in small stepsix, we use the same MG data several times and campared weigbt track in Figure 6. As we can that smaller stepsize incmues the number of computation to reach optimal weight In Left topmost of Figure 6, overlapped weight track for stepsize of 0.01 and is given for comparison. The weight tracks with small stepsize curves are smoother than large stepsize. This mans that finslr weight vectur values are more close to the o p W so the excess MSE is small, but need more computation. or mare data to rcach optimal weight vector vahes. Therefore, small stepsize decrease the excess MSE error and ruffling of weight but slower speed of adaptation in LMS algorithm. The * mark in data size 10,000 and 15,000 in Figure 6 is the last weight vector value hm data size 25,000 for the comparison. As you can see that small stepsize has little raffling but has slow speed of adaptation. In Tablc I, the initial condition for weights is set to zeros for the iteration. In following Figure 7, we have shown the effect of initial condition on the weight track for h e comparison of Figure 5 and 6. We can see that changing in initial condition makes different path for weight track, however updated weights of LSM track the direction of larger eigenvalue at first, and then follow tho direction of fast mode as mentioned before.

8 Figure 6 Comparison weight track for data she fmm 5,000 to 25,900 Figure 7 Effect of initial condition on weight tmck

9 4. Stationary versus Non-stattonary : Figures of merit of the LMS have been derived assuming stationarity conditions. Non-stationary may a&e at Iast in 2 important cases. - The desired signal is time varying (system identification). Here the autocorrelation function is time invariant, but the crass~t~elation function is time varying. - The input is non-stationary (equalization, noise cancellation). Here both autmorreiation and cross- correlation arc non-stat ionary. In non-stationary environments, the optimum set of weights change with time. Therefore, the LMS must not only find the minimum but also track the ~hange of minimum of the performance surface. In this cast, we have two error sources for the actual weights. One is related to the ems of the estimate of the &mt. other is related to the lag in the adaptive process. The first contribution is direct1 y proportional to stepsize and the second contribution is inversely proportional to stepsize. We can assume them additive so the total misadjustment has s single minimum. For comparison between stationary and non-stationary data, weight track of non-stationary speech signal is given in following Figure 8. As we can see in figure with respect to weight track of stationary MG data, weight track of non-stationary data has much rahe thzln stationary data due to nw-s tationarity of speech signal. Moreover path of weight track of non-stationary data is less straight than stationary data and has more overshoots and meandering to search optimum weight. The Figure 8 Weight tracks of speech dab

10 Red PM R d Put Rgure 9 Polezero plots of Isverse filter LMS and hear predictor Real Part Figure 10 Pole-zero plots of inverse filter for speech signal.

11 To consult the property of minimum-phase system of LMS algorithm for MG data and speech data, the pole-zen, plot are given in following Figwe 9 and Figure 10, respectively. As shown in Figure 9, we can see that some poles of LMS are outside of unit circle for both filtm of order 15 and 30, then unstable and non-minimum phase. However, all poles of inverse fr l ter of linear predictor all inside unit circle. Therefore, linear predictor has stable and casual inverse filter and minimum phase system. Xn Fj gure 10, effect of stepsize on the stability of inverse filter for order 15 studied using speech signal, As shown in figure, some poles of stepsize 0.8 are outside of unit circle. However all poles of small step art inside unit circle. This means that raffling of weight with larger stepsize contributes stability of inverse filter. This is dso true for stationary MG data. Therefore, we need to choose adequate stepsize to guarantee propay of minimum phase of inverse filter and stability of prediction error filter. With statistically stationary inputs, the quadratic performance surface is fixed (deterministic) and Wiener solution is fixed. With non-stationary input, the performance surface is varying in time and Wiener Solution is not fixed but time varying. In linear prediction problem, we assume the laally stationarity of speech data by taking window. However longer window degraded performance of non-stationary data by Wiener-Hopf Equation. The performance of LMS algorithms is equivalent when we use the same size for both stationary and non-stationary data. This means that the LMS have no assumptions or knowledge about the statistics of input or desired response. The algorithm does not use any averaging, squaring, perturbation, etc. Each components of dent is obtained from a single data input. However, Due to noisy estimation of LMS algorithm, excess MSE increase with stepsize and greater than the minimum value sf MSE error power by the linear predictor.