Finite-Sample Bias Propagation in the Yule-Walker Method of Autoregressive Estimation

Transcription

1 Proceedings of the 7th World Congress The International Federation of Automatic Control Seoul, Korea, July 6-, 008 Finite-Samle Bias Proagation in the Yule-Walker Method of Autoregressie Estimation Piet MT Broersen Deartment of Multi Scale Physics Delft Uniersity of Technology, The Netherlands Tel: , mtbroersen@tudelftnl Abstract: Lagged-roduct autocorrelation estimates hae a small triangular bias Howeer, using them to comute an autoregressie model with the Yule-Walker method can gie a strongly distorted sectral model in finite samles The distortion is shown for examles where the reflection coefficients are not ery close to one in absolute alue It will disaear asymtotically An objectie measure is resented to determine the smallest samle size for which the unbiased asymtotic theory is a fair aroximation INTRODUCTION The Yule-Walker (YW) equations describe the relation between the autoregressie (AR) arameters and the first lags of the autocorrelation function The YW estimation method soles the AR arameters from the YW equations, with the lagged-roduct estimates substituted for the autocorrelations (Kay and Marle, 98) The Leinson- Durbin recursion is an efficient solution of the YW equations For increasing AR orders, it comutes reflection coefficients, which are defined as the negaties of the artial correlation coefficients (eg Kay and Marle, 98; Stoica and Moses, 997; Broersen, 006) The arameters of an AR model of an arbitrary order q can be comuted from the first q reflection coefficients Those in turn are comletely determined by the first q lags of the autocorrelation function Shaman and Stine (988) showed that the bias of the YW method of AR estimation is gien by the general terms for AR models, with an additional contribution caused by the triangular bias of the lagged-roduct autocorrelation estimate They hae gien exlicit exressions for the first two estimated AR arameters The bias of the second arameter is a rational function of the arameters, which can become large if oles of the AR olynomial are close to one in absolute alue De Hoon et al (996) showed that the last arameter of order neer causes this extra bias for an AR( ) rocess They related the magnitude of the bias to the condition number of the Toelitz autocorrelation matrix, which contains only the estimated autocorrelations until order Also the ariance of AR estimates has been studied The asymtotical theory gies identical results for all AR estimation methods Well known AR estimation methods are the YW method, the method of Burg (967), seeral least squares solutions and the maximum likelihood method (eg Kay and Marle, 98; Stoica and Moses, 997; Broersen, 006) Kay and Makhoul (983) deried asymtotic exressions for the ariance of estimated reflection coefficients, for orders u to the true AR order Exlicit exressions were gien for AR rocesses until order 3 and an efficient recursie rocedure can comute that ariance for higher order AR rocesses Broersen (006) showed that emirical finite-samle aroximations for the ariance of estimated arameters deend on the method of AR estimation He used the emirical ariances in a finite-samle order selection criterion that is adated to the estimation method It turned out that the finite-samle selection criterion for the YW estimation method was almost identical with the asymtotic AIC criterion of Akaike (974) A detailed theoretical analysis of Wensink and Dijkhof (003) deried the exectations of the squared reflection coefficients in small samles of a white noise signal They determine the ariance of the reflection coefficients of general AR rocesses for all orders higher than the true AR order, which is imortant for a finite-samle order selection criterion The triangular bias turned out to be imortant if a reflection coefficient of any order lower than the true rocess order is close to + or If the true reflection coefficient of order m, less than the true order, is gien by m/n, where N is the samle size, the bias in the estimated reflection coefficient of order m + will be 50 % (Erkelens and Broersen, 997) The condition number of de Hoon et al (996) for the true autocorrelation function is indeendent of the number of obserations N On the other hand, the triangular bias of the lagged-roduct estimates diminishes with N and anishes asymtotically Therefore, no critical alue for that condition number can be gien AR models estimated with the YW method are asymtotically unbiased Hence, rocesses with a high condition number may gie a large bias in finite samles, but the bias will disaear for growing samle sizes This aer resents examles with heay YW bias, although the oles are not ery close to the unit circle or the reflection coefficients are not close to unity The bias can be comuted by alying the triangular autocorrelation bias to the true autocorrelation function, and soling the YW equations with that bias An objectie measure is gien that determines the minimum samle size for which the influence of the bias in the Yule-Walker method is not greater than the inaccuracy due to the estimation ariance of the AR arameters /08/$ IFAC / KR

2 Seoul, Korea, July 6-, 008 AR ESTIMATES WITH THE YULE-WALKER METHOD A discrete-time AR( ) rocess is a time series x n that can be written as (eg Kay and Marle, 98; Priestley, 98; Stoica and Moses, 997; Broersen, 006) x ax a x () n n n n, where n is a urely random rocess with mean zero and ariance In theory, any stationary stochastic rocess with a continuous sectral density can be written as an exact AR() rocess In ractice, finite order models are sufficient because high order arameters tend to be small The oles of the AR( ) rocess are the roots of the AR olynomial A(z): ( ) A z a z a z () Processes are called stationary if the oles are strictly within the unit circle The arametric ower sectrum h( ) of the AR( ) model is for < comuted with h ( ) j Ae j i ai e i The normalized sectrum ( ) is defined as the sectrum diided by the ariance of the signal: h / (4) x The autocoariance r(q) and the normalized autocorrelation (q) at lag q are defined for a signal with mean alue zero by xx n nq xn rq () E () q r(0) E All lags of the infinitely long true autocorrelation function of an AR( ) rocess are determined by true AR arameters with the YW equations, gien by (Kay and Marle, 98): ( q) a( q) a( q ) 0, q (6) ( q) ( q) Different efficient numerical methods hae been deeloed to sole the YW equations Recursie algorithms comute the reflection coefficients for increasing orders The first m reflection coefficients are determined by the first m lags of the true AR autocorrelation function, for all m Afterward, arameters for arbitrary AR orders q can be found from the first q reflection coefficients; see Stoica and Moses (997) Equation (6) can also be used for indexes q greater than, to extraolate the autocorrelation function The same relations (6) can be used to comute arameters from correlations and ice ersa They relate true or estimated AR arameters with true or estimated autocorrelations, resectiely Broersen (007) showed that efficiently estimated AR arameters sere as an efficient estimator for the autocoariance function To obtain a ositie semi-definite lagged-roduct (LP) estimator for the autocorrelation function of N obserations x n, the estimator for lag q should be biased and is gien by N q N () q LP r ()/ q r LP LP(0) x x n n q/ x n N n N n ˆ ˆ ˆ (3) (5) (7) Likewise, the ositie semi-definite LP estimator for the autocoariance at lag q uses the diisor N for N q contributions This gies the triangular bias q / N: Nq Nq q E r ˆ ( ) E E ( ) (8) LP q xx n nq xx n nq rq N n N N The bias of the normalized autocorrelation in (7) has some additional terms gien by Broersen (007) Those terms arise from the exectation of quotients, but they are not imortant for the sequel The theoretical exectation of the influence of the triangular bias on the biased alues of the AR() model arameters a i,b can be ealuated by substituting the biased exectations of the autocoariances in the YW relations (6): rq ( ) q/ Na, brq ( ) q / N (9) ab, r( q ) q / N 0, q,, The YW method of AR estimation has this exectation a i,b because it substitutes LP estimates (7) in (6) to comute the AR arameters â i of an AR( ) model with the equations ˆ ( q) aˆ ˆ ( q ) aˆ ˆ ( q ) 0, q (0) LP LP LP Extended YW equations for q > describe the arametric estimator for the continuation of the autocorrelation function The usual accuracy measure for time series models is the squared error of rediction PE, which can be comuted by using estimated arameters to redict a fresh realization of x n : xˆax ˆ aˆ x () n n n The PE is defined as the long-term aerage of x x Fresh or new data x n hae been generated with the same rocess equation (), but they hae not been used to estimate the arameters Both the true and the estimated arameters alues are known in Monte Carlo simulation exeriments For these articular situations where the true arameters are known, Broersen (006) showed that the exectation of the PE can be comuted without generating new data by using j Âe j h PE ĥ d A e ˆn d, () where Â(z) denotes an estimated AR olynomial He deried an efficient algorithm to comute the PE of () The model error ME for has been defined for AR models by scaling the PE with and by multilying with the samle size as ˆ PE ME ME AA, N (3) The argument of the ME is left if no confusion is ossible The exectation of the ME for unbiased efficiently estimated AR models of orders K, greater than or equal to the true order, equals K The ariance of each estimated arameter has a minimal contribution to the ME exectation The Cramér- Rao bound for the ME of an AR( ) rocess is gien by The condition number of the Toelitz correlation matrix R with the elements r (0) until r ( - ) has been used by de Hoon et al (996) to classify the sensitiity of an AR rocess to the YW bias It is defined as n 745

3 Seoul, Korea, July 6-, 008 R R, (4) where is some matrix norm, like the largest singular alue The norm is comuted for the exectation of the autocorrelation matrix and is indeendent of the samle size N An objectie measure for the quality of AR models estimated with the YW method for a gien samle size N is found by calculating the biased exectations of the autocoariances with (8), the arameters a i,b of the biased AR() model Â(z) with (9) and finally the ME(Â,A) with (3) The asymtotical exectation of this ME(Â,A) is zero A critical samle size for which the bias is no longer considered significant is defined to be ME(Â,A) = That is the samle size where the bias error becomes of the same magnitude as the ariance error that is caused by efficiently estimating the arameters of the AR model This limit for ME(Â,A) seems to be a little bit arbitrary Howeer, equal bias and ariance contributions are known from the deriation of the order selection criteria of Akaike (974) as an estimator for the Kullback-Leibler index Therefore, this choice has a firm statistical background 3 FINITE-SAMPLE AR THEORY Kay and Marle (98) describe many existing methods for AR estimation The method of YW was the first; the method of Burg (967) has nice roerties and is often used for AR estimation in ractice The asymtotical theory for the residual ariance s q and the rediction error PE(q) do not deend on the estimation method The residual ariance is most easily exressed as a function of the reflection coefficients For an AR model of order 0, the exectation of s 0 is gien by r(0), the ariance of the rocess x n For any arbitrary order q, the residual ariance s q is gien by s q q s0 ki i (5) where k i is the true or the estimated reflection coefficient of order i If true reflection coefficients are substituted in (5), their alue is zero for orders greater than the true and the residual ariance gies the final alue s = For estimated alues of k i, the residual ariance will kee decreasing aboe the true order because of the estimation ariance of k i A simle exlanation for the triangular bias of the YW AR method has been gien by Erkelens and Broersen (997) The residual ariance s q aears in the denominator of the Leinson-Durbin recursion to comute the reflection coefficient for the order q + (Kay and Marle, 98; Stoica and Moses, 997) Suose that a true k q has the unbiased alue /N A small bias of the magnitude /N would gie it the biased alue /N Substitution of the unbiased and of the biased alues for k q in (- k q ) gies the aroximations /N and 4/N, resectiely A small bias of magnitude /N in a reflection coefficient k q makes a difference of a factor in the residual ariance s q The error in s q roagates to all higher AR orders because (5) is in the denominator of Leinson- Durbin recursion for the comutation of successie reflection coefficients The triangular bias roagation is always influential if one of the true reflection coefficients of an order lower than the true order is close to unity, with a distance less than /N Howeer, for any alue k q = -, with a ery small, the finite-samle influence of the bias disaears asymtotically if the samle size N is much greater than / If the order K is at least the true order, the asymtotical AR exectations for unbiased models are determined by K s K (6) N E K, K EPE K, K N (7) This result can be alied in ractice with a reasonable accuracy if K is less then 0N For still higher AR orders, Broersen (006) showed that finite-samle theory gies an imroed accuracy That theory is based on aroximations for the ariance of reflection coefficients estimated from finite samles of a white noise rocess The asymtotical ariance /N is relaced by emirical finite-samle ariance coefficients Broersen (006) gies the aroximations iyw, iburg, N i N N N i (8) for YW and Burg estimates, resectiely YW ariances are smaller than the asymtotical alue /N because of the triangular bias in the autocorrelation function In contrast, Burg ariances are greater because each new Burg reflection coefficient is estimated from a shorter filtered signal The finite-samle exressions for s K and PE(K) become K FS K i, i E s, K K FS K i, i (9) E PE( ), K (0) By substituting the YW or the Burg ariance coefficients for i,, different results are obtained for the estimation methods The residual ariance s q is known for ractical data for all orders q, because it has been minimized to comute the AR arameters Howeer, PE(q) is not known A finite-samle aroximation uses the known s q to estimate the rediction accuracy in ractice with the finite-samle criterion FSC(q) i, i FSC( q) sq q q i i, () This aroximation can be comuted for N gien data x n It has a strong relation with the final rediction error FPE of Akaike (974) that has been used in his first order selection criterion It may be used for all model orders q, indeendent of the true AR order The alues of s q and FSC(q) are mainly determined by the true alues of the reflection coefficients for model orders less than the true order and by (9) and (0) for higher orders The exectation of FSC(q) for higher model orders is gien by E FS [PE(q)] For finite samles, it deends on the estimation method if q is greater than about N/0 746

4 Seoul, Korea, July 6-, YULE-WALKER METHOD APPLIED TO AR(4) To demonstrate the bias roagation, an AR(4) examle with four equal reflection coefficients has been studied This examle has the adantage that all unbiased true reflection coefficients are the same and the bias for different orders can be comared easily The triangular bias of (8) is alied to the true autocorrelation function for N = 00 and the biased alues for the reflection coefficients are found by substituting those biased correlations in (9) As it is hardly illustratie to derie the analytical exressions for this examle, numerical alues are resented as a function of the true reflection coefficients Table gies the true alue k, the four biased reflection coefficients and the two measures: ME(Â,A) and Table True reflection coefficient k and exectations for the biased ones for an AR(4) rocess generated with four equal true reflection coefficients k The ME of the biased AR(4) model as comuted with (9) and the condition number are gien for N = 00 and for different alues of the true k true biased exectations k k k k 3 k 4 ME(Â,A) The first reflection coefficient has a bias of the magnitude /N for all true alues of the reflection coefficient k Higher orders q hae a bias that is only close to q/n for k = 0; the strongest bias is found for the highest orders Finally, the true alue k = 099 = /N gies already a theoretical bias of 50 % for the second reflection coefficient k It is remarkable how the bias roagates to higher orders for alues of k much smaller than /N, which are rather far from the unit circle Moderate alues of the condition number lead already to strong biases and low condition numbers do not always guarantee a small bias The examle with k = 07 demonstrates that the moderate condition number 353 can gie a significant model error ME The demand that the ME(Â,A) is less than 4 is only met for k < 0578 in this AR(4) examle, with = 35 as the condition number The samle size N should be greater than to obtain the reduced bias influence ME(Â,A) < 4 for k = 09; the theoretical alue for the biased k 4 would be then The ariance influence will become really small for ery high N only Howeer, the bias influence is still anishing asymtotically This is also demonstrated in Table The alues of k 4 (biased) and ME(Â,A) of the AR(4) model are gien for increasing N Due to the multilication in the ME with the samle size in (3), the influence of the bias grows for N less than 000 It decreases roortional to /N for N greater than about in this examle with 4 equal true reflection coefficients ME becomes equal to the model order 4 for N is about Table The exectation of the biased reflection coefficient of order 4 and the corresonding model error ME(Â,A) of the bias in (9) for an AR(4) rocess with four equal true reflection coefficients 09, as a function of samle size N normalized rediction error N k 4 (biased) ME(Â,A) Estimated model accuracy with finite samle theory Yule Walker Burg Y W theory Burg theory AR model order Fig The estimated accuracy FSC(q) of () and the theoretical accuracy E FS [PE(q)] of (0) for AR models estimated from a single realization of N = 00 obserations of an AR(4) rocess with 4 equal true reflection coefficients 04 The model accuracy can be determined for estimated models of increasing orders Software is aailable to estimate many AR models and to select the best model order automatically (Broersen, 00) Fig gies results for an AR(4) examle with equal true reflection coefficients 04 where the bias is not influential Both YW and Burg AR estimates are shown The estimated accuracies FSC(q) are comuted with () and further the theoretical finite-samle exectations for unbiased models comuted with (0) are resented for the two methods Both methods follow their exectations The YW lines become flattened at higher orders due to the triangular autocorrelation bias of (8) at high correlation lags, but that has no further negatie influence on the accuracy because all true arameters are zero for orders greater than 4 The results for Burg and YW are ery close for model orders less than N/0, but differences become noticeable for higher orders 5 YULE-WALKER METHOD APPLIED TO AN AR(7) EXAMPLE The YW bias has two imortant roerties The first is that the bias is only strong after a reious significant reflection coefficient No bias is found if all reflection coefficients are small The second is that it cannot be comensated by using 747

5 Seoul, Korea, July 6-, 008 Table 3 The true arameters a i of an AR(7) rocess, the reflection coefficients k i and the biased exectations; N = 00 Poles of true and biased AR(7) rocess with N = 00 true biased order i a i k i k i (biased) higher model orders with additional AR arameters The influence of the bias is demonstrated with 00 obserations of an AR(7) examle The true arameters, reflection coefficients and their biased exectations are gien in Table 3 The bias is small for the first two reflection coefficients and increases for higher orders due to the rather high alue of k It is shown that the bias also has influence on the arameters of orders higher than the true order 7 Due to the triangular bias, all reflection coefficients belonging to the true biased autocorrelation function are non-zero Therefore, the triangularly biased YW model gets the theoretical order for all true Howeer, the alues are small for high orders and all true biased reflection coefficients aboe order 9 are less than 0 for N = 00; aboe order 6 they are less than 00 The best AR model order to be selected with the AIC criterion of Akaike (974) for estimates with the YW method can be comuted theoretically The reduction of the residual ariance is gien by (5) and that can be used in AIC The best order that can be selected for biased YW models would become 9 in this true AR(7) examle The AR(8) model of one order higher would be the best order for selection with AIC if the first extra reflection coefficient would be greater than /N The AR(9) model of two orders higher is the best if the last reflection coefficient is greater than /N and the exected sum of squares of the last two biased reflection coefficients is greater than /N In this examle with N = 00, their sum of squares is 0039 and fulfils this requirement The theoretical otimal order of the biased model becomes higher than the true order due to the triangular bias Howeer, the differences are small and the comarison between the true and the true biased model are made for the AR(7) model The model error ME(Â,A) is 74 for this examle, with Â(z) and A(z) as the triangularly biased olynomial of (9) and the true AR(7) olynomial, resectiely The condition number is 4547 This AR(7) examle would require 5000 obserations before the condition ME(Â,A) = 7 is met The AR(7) examle can be characterized by its oles, which are gien in Fig The true comlex conjugated oles are at the radii 0953, 0940 and 0936 All biased comlex oles are much further away from the unit circle, at radii 0886, 0859 and 0858 Also the angles of the oles are shifted, which will cause a shift of the frequencies of sectral eaks imaginary art real art Fig Poles of true and biased AR(7) rocess, where the theoretical triangular AR(7) bias is comuted for N = 00 The large dislacement of the ole on the real axis will gie a biased sectrum, which is much too large at high frequencies The accuracy of AR models of different orders is gien in Fig 3 The estimated model accuracy of the Burg method is close to the theoretical white noise exectation for orders 7 and higher Lower order models hae a larger ME alue because not all truly non-zero arameters are included The Burg models of orders higher than 6 are unbiased The influence of the triangular bias on the YW estimates is ery great It is seen that the accuracy may still become somewhat better for orders higher than 7; theoretically, the best order should be 9 for the estimates with triangular bias Howeer, normalized rediction error Estimated model accuracy with finite samle theory Yule Walker Burg Y W theory Burg theory AR model order Fig 3 The estimated accuracy FSC(q) of () and the theoretical accuracy E FS [PE(q)] of (0) for AR models estimated from N = 00 AR(7) obserations, as a function of the model order The strong bias of the Yule-walker estimates gies a ery large difference with the theoretical finitesamle accuracy that does not take the bias into account 748

6 Seoul, Korea, July 6-, 008 normalized logarithm of sectra Sectral densities of Burg and Yule Walker estimates, N = 00 true true bias Burg AR(7) YW AR(7) frequency [Hz] Fig 4 The estimated and the theoretical sectra of AR models estimated from N = 00 AR(7) obserations with the Burg and the YW method The order 7 was selected both for Burg and for YW estimates in this simulation run the general shae of the YW model accuracy looks like the theoretical shae of unbiased models, with the same tendency to become flattened at ery high orders Order selection between the YW candidates cannot select an accurate model, because all YW models are oor The bias roagates to all higher order models AR models with Burg are much better The estimated sectra of the Burg and YW method are comared with the true and with the exectation of the biased sectrum in Fig 4 The shift of the real ole to the left causes the distortion of the estimates at higher frequencies The shift of the comlex oles away from the unit circle makes the three sectral eaks less strong for the YW estimates The Burg estimates are a close aroximation to the true sectrum; the YW estimates aroximate the biased true sectrum The YW estimate of AR order N aroximately gies the eriodogram as sectral estimate The eriodogram suffers from the same triangular bias Hence, it will globally follow the biased sectrum in Fig 4 Howeer, the ariations around the true cure are much greater As indicated in Fig 3, YW models of all higher orders hae the bias error that was already resent in the YW estimate of the true order 7 The aerage model qualities ME of YW and Burg estimates, both with selected AR orders, hae been determined in 000 simulation runs The aerage ME for YW was 03 and for Burg the ME alue was 07 The quality of estimated Burg models with the automatically selected model orders of the ARMAsel rogram of Broersen (006) is rather close to the Cramér-Rao lower bound 7 for the ME of an AR(7) rocess This aerage quality includes the uncertainty of order selection; the aerage quality of AR(7) Burg models was 88 Not all rocesses with oles close to the unit circle are ery sensitie for the triangular bias of the Yule-Walker method As an examle, take the AR(7) rocess with all true arameters zero, excet a 7 that is equal to 07 All 7 oles would be at the equal radius 095 then The triangular bias moes them all to 094 for N = 00, at almost the same angles as the true AR(7) rocess The oles of this second examle are closer to the unit circle than the oles of Fig Howeer, the influence of the bias is ery much smaller, with ME(Â,A) equal to 047 and with the condition number = 57 6 CONCLUDING REMARKS Biased lagged-roduct autocorrelation estimates are used for the Yule-Walker method of AR arameter estimation The small triangular autocorrelation bias can cause a significant bias of the AR arameters and its sectrum in finite samles The model error is introduced as an objectie measure to quantify the significance of the triangular bias It determines the smallest samle size where roerties are asymtotic The triangular bias modifies the true order to for all biased AR() rocesses The best aroximating model order may become higher than the true order and deend on the samle size The locations and the height of sectral eaks may be changed considerably by the Yule-Walker bias Burg estimates are referred and do not hae those roblems REFERENCES Akaike, H (974) A new look at the statistical model identification IEEE Trans Autom Control, AC-9, Broersen, PMT (00) ARMASA Matlab toolbox, 00, [Online], Aailable at wwwmathworkscom/matlabcentral/fileexchange Broersen, PMT (006) Automatic Autocorrelation and Sectral Analysis London, UK: Sringer-Verlag Broersen, PMT (007) Historical misconcetions in autocorrelation estimation IEEE Trans on Instrumentation and Measurement, 56, Burg, JP (967) Maximum entroy sectral analysis Proc 37 th Meeting Soc of Exloration Geohysicists, 6 Erkelens, JS and PMT Broersen (997) Bias roagation in the autocorrelation method of linear rediction IEEE Trans on Seech and Audio Processing, 5, 6-9 Hoon, MJL de, THJJ an der Hagen, H Schoonewelle and H an Dam (996) Why Yule-Walker should not be used for autoregressie modelling AnnNucl Energy, 3, 9-8 Kay, S and J Makhoul (983) On the statistics of the estimated reflection coefficients of an autoregressie rocess IEEE TransAcoust, Seech, Signal Process, ASSP-3, Kay, SM and SL Marle (98) Sectrum analysis-a modern ersectie, Proc IEEE, 69, Priestley, MB (98) Sectral Analysis and Time Series London, UK: Academic Press Shaman, P and RA Stine (988) The bias of autoregressie coefficient estimators Journal of the American Statistical Association, 83, Stoica, P and R Moses (997) Introduction to Sectral Analysis Prentice Hall, Uer Saddle Rier, NJ Wensink, H E and WJ Dijkhof (003) On finite samle statistics for Yule-Walker estimates IEEE Trans on Information Theory, 49,