PARAMETER ESTIMATION OF NEARLY NON- STATIONARY AUTOREGRESSIVE PROCESSES

Transcription

1 RF - 86 PARAMETER ESTIMATION OF NEARLY NON- STATIONARY AUTOREGRESSIVE PROCESSES Final reor of suden work by Michiel J. L. de Hoon January - June 995 Delf Universiy of Technology, Faculy of Alied Physics, Dearmen of Reacor Physics. Suervisors: Ir Hielke SCHOONEWELLE Dr ir Tim VAN DER HAGEN Prof. dr ir Hugo VAN DAM

2 --

3 -3- ABSTRACT Auoregressive modeling of noise daa can be used for he deecion of malfuncioning in a nuclear reacor a an early sage by soing increases in he redicion error a he occurrence of an anomaly. An auoregressive model deends on a limied number of arameers, which are esimaed from measured noise daa. Several mehods are available o esimae hese arameers. Usually hese mehods aroximaely yield he same arameer esimaes. If he characerisic oles of he auoregressive rocess are locaed closely o he uni circle, i will exhibi a seudo-eriodic, almos non-saionary behaviour. In ha case he Yule-Walker esimaion echnique should no be used o deermine he auoregressive model, as i will lead o a large residual and redicion error variance as well as o incorrec resuls for he auoregressive arameers. In his reor he underlying mahemaical circumsances are shown ha cause he Yule-Walker echnique o be unreliable. Alying auoregressive analysis as described above o he 994 IAEA Benchmark es, i was ossible o deec leak noise in a seam generaor of a liquid meal fas breeder reacor. However, i is shown ha anomaly deecion is no ossible if he Yule-Walker aroach is emloyed due o he almos non-saionary behaviour of he benchmark noise daa.

4 -4- CONTENTS ABSTRACT 3 CONTENTS 4 LIST OF SYMBOLS 6. INTRODUCTION 9. THEORY OF AUTOREGRESSIVE MODELLING.. APPLICATION OF DATA WINDOWS 3.. RESIDUE AND PREDICTION ERROR 5.3. POLE LOCATION AND STABILITY 7.4. PARTIAL AUTOCORRELATION COEFFICIENTS AND REFLECTION COEFFICIENTS 9.5. SOME OTHER METHODS OF AUTOREGRESSIVE PARAMETER ESTIMATION 0 Yule-Walker aroach using he Boxcar-weighed auocovariance funcion Leas squares forward-backward aroach Burg s mehod.6. DETERMINATION OF THE AUTOREGRESSIVE MODEL ORDER USING AKAIKE S CRITERION 4 3. POLE LOCATION AND CONVERGENCE RATE AUTOREGRESSIVE PROCESS OF ORDER ONE HAVING A POLE NEAR THE UNIT CIRCLE 6 Unbiased esimaion echniques: leas squares and leas squares forwardbackward 8 Yule-Walker aroach 30

5 AUTOREGRESSIVE PROCESS OF ORDER TWO HAVING POLES NEAR THE UNIT CIRCLE 3 Esimaed ole locaions AUTOREGRESSIVE PROCESSES OF A GENERAL ORDER HAVING POLES NEAR THE UNIT CIRCLE 40 Magniude of he arial correlaion coefficiens 40 Condiion of he auocovariance marix COEFFICIENT ESTIMATION AND STABILITY AUTOREGRESSIVE ANALYSIS OF THE 994 IAEA BENCHMARK LEAST SQUARES APPROACH YULE-WALKER APPROACH BURG S METHOD PSEUDO-PERIODIC BEHAVIOUR AND ITS CONSEQUENCES 60 Magniude of he arial correlaion coefficiens 60 Condiion of he auocovariance marix 6 Pseudo-eriodic behaviour 6 5. CONCLUSION 65 LITERATURE 66 APPENDIX: AUTOREGRESSIVE PARAMETER ESTIMATES 68

6 -6- LIST OF SYMBOLS All of he following variables are dimensionless, unless saed oherwise. Symbol Descriion Uni a â a i â i â ki AIC b i C i c ij auoregressive arameer of a firs order auoregressive rocess esimaed auoregressive arameer of a firs order auoregressive rocess (forward) auoregressive arameer for delay i auoregressive arameer esimae for delay i auoregressive arameer esimae for delay i in ieraion k of Burg s mehod Akaike s informaion crierion backward auoregressive arameer esimae amliude of he harmonic comonen a angular frequency w i in a harmonic rocess coefficiens in he auocovariance marix (leas squares aroach) e i deviaion from a rocess ole o he oin z = E execancy oeraor oal squared error f samling frequency [Hz] f 0 fundamenal frequency [Hz] i j k running variable running variable reflecion coefficien for order k K esimaed reflecion coefficien for order number of harmonic comonens in a harmonic rocess

7 -7- N N i i R oal number of samles in realisaion y, which is used for arameer esimaion oal number of samles in realisaion x, which is used for daa redicion auoregressive model order ole of he auoregressive oeraor esimaed ole of he auoregressive oeraor magniude of he oles of a second order auoregressive rocess R auocovariance funcion of he sochasic rocess x R biased esimaes of he auocovariance funcion; coefficiens in he auocovariance marix (Yule-Walker aroach) R * var( e ) var( e ) vâr( e ) var( h ) var( h ) vâr( h ) x x x y unbiased esimaes of he auocovariance funcion discree ime variable variance of he urely random rocess e redicion error variance esimaed redicion error variance variance of he urely random rocess h residual variance esimaed residual variance realisaion of he auoregressive rocess; x and y are indeenden realisaions of he same auoregressive rocess samle of realisaion x, measured a ime rediced value of he samle of realisaion x measured a ime realisaion of he auoregressive rocess, from which he auoregressive arameers are esimaed; x and y are indeenden realisaions of he same auoregressive rocess

8 -8- y ŷ z - a da r e e h h samle of he realisaion y, measured a ime rediced value of he samle of realisaion y measured a ime backward shif oeraor argumen of he ole above he real axis of a second order auoregressive rocess deviaion of he auoregressive arameer esimae â r from he rocess arameer a r innovaion of he urely random rocess a ime in realisaion x; e and h are saisically idenical esimaed innovaion of he urely random rocess e a ime ; redicion error a ime innovaion of he urely random rocess a ime in realisaion y; e and h are saisically idenical esimaed innovaion of he urely random rocess h a ime in realisaion y; residue a ime [rad] h b k ( ) esimaed backward residue in ieraion k of Burg s mehod f h k ( ) k x k r r r j i w i esimaed forward residue in ieraion k of Burg s mehod condiion number esimaed residual variance in ieraion k of Burg s mehod uni delay auocorrelaion coefficien auocorrelaion coefficien for delay wo arial auocorrelaion coefficien for delay wo delay random variable, consiuing he hase of he harmonic comonen a angular frequency w i in a harmonic rocess angular frequency of he i-h harmonic comonen of a harmonic rocess

9 -9-. INTRODUCTION Reacor noise analysis can lay an imoran role in deecing malfuncioning of a reacor in an early sage, before he seady-sae arameers are influenced. Generally noise signals from nuclear reacors are no comleely random funcions, bu hey conain informaion abou he rocess from which hey originaed. Analysis of noise daa on heir saisical characerisics can be used for deecing anomalies during reacor oeraion. If he saisical roeries of he noise daa change dramaically, one migh exec ha he sysem generaing hese noise signals, in our case a nuclear reacor, is no funcioning in is normal fashion. Provided ha he saisical roeries of he noise signals change insananeously afer he occurrence of an anomaly, i may be deeced before he saionary hysical quaniies are severely influenced. Flucuaing quaniies such as noise daa can be analysed eiher in he frequency domain (e. g. using ower secral densiy analysis) or in he ime domain (analysis of covariance funcions). A widely emloyed aroach in ime domain noise analysis is he so-called auoregressive analysis. Using auoregressive analysis, a model of he measured noise daa is made in which each daa oin is assumed o be linearly relaed o a limied number of reviously measured daa oins. Once an auoregressive model has been fied o a record of measured noise daa, i can be used o redic fuure noise daa from is redecessors. Noneheless, hese redicions will never be erfecly exac, which means ha a redicion error is inroduced in he forecasing. Each redicion error can be calculaed once he real value of he daa oin is measured. Usually he redicion error will be relaively small. However, if an anomaly occurs, he reviously deermined auoregressive model will no longer be valid. Consequenly i may be execed ha he noise daa canno be rediced correcly any more. This will lead o a dramaic increase of he redicion error. Therefore an anomaly during reacor oeraion can be deeced by soing sudden increases of he redicion error variance. As he correcness of an auoregressive model can easily be validaed by assering ha he redicion error has a urely random disribuion, changes in he redicion error are usually easier o be recognised han changes in he measured signal iself. The auoregressive model used for noise daa redicion deends on a limied number of arameers, which are esimaed from measured noise daa. Several

10 -0- mehods exis o esimae he auoregressive arameers. I can be shown ha for large daa samles hese esimaion echniques should lead o aroximaely he same arameer esimaes. However, in some secial cases he so-called Yule-Walker esimaion mehod will lead o oor arameer esimaes, even for moderaely sized daa samles. The 994 IAEA benchmark noise daa (Journeau, 994), which were he inducemen owards his invesigaion, are an examle in which he Yule-Walker esimaion erforms oorly. As a resul of his, using a Yule-Walker esimae of he auoregressive rocess, i was no ossible o locae an anomaly in he resened noise daa, while he oher esimaion mehods could. Unforunaely, a rigorous mahemaical reamen of his henomenon is no ye available. In chaer, an overview of he various esimaion echniques for auoregressive arameers and heir meris is resened. The condiions leading o oor arameer esimaes in case of he Yule-Walker echnique are described in chaer 3, while some of he mahemaical circumsances causing hese resuls are exlained. Also some simulaions of auoregressive rocesses have been made o suor hese hyoheses. Finally in chaer 4, he auoregressive analysis of he 994 IAEA benchmark is resened. These sudies were made as a final reor of suden research work a he Dearmen of Reacor Physics, Inerfaculy Reacor Insiue, Delf Universiy of Technology. Secial hanks are due o ir Hielke Schoonewelle, who assised me during his research, and dr ir Pie Broersen (Delf Universiy of Technology, Faculy of Alied Physics), who inroduced me o he oic of auoregressive modelling and is ifalls.

11 --. THEORY OF AUTOREGRESSIVE MODELLING When alking abou ime series analysis, we refer o he rinciles and echniques dealing wih he analysis of observed daa from random rocesses in which he arameer denoes ime (Priesley, 994). A record of a ime-varying random rocess is called a ime series. A wide range of lieraure is available on hese subjecs, e. g. he exbooks by Box and Jenkins (970), Ljung (987) and Priesley (994). The heory of auoregressive modelling is given here insofar as resenly needed and is largely based on hese exbooks, while he discussion of daa windowing is based on he aer by Morf e al. (977). The saisical behaviour of a ime series can be analysed by describing i in erms of a saisical model. Numerous models are available as a ool of saisical analysis, one of which is based on he so-called auoregressive rocess, in which he successive samles y of a sochasic rocess are linearly deenden on heir redecessors: y + ay + a y + + a y = h (.) in which a i are he auoregressive arameers and he innovaions h are a saionary urely random rocess wih zero mean. An esimaed auoregressive model of he same order can be wrien as y + a y + a y + + a - - y- = h (.) in which â i are he auoregressive arameer esimaes and h are he esimaed innovaions. A clear disincion should be made beween he auoregressive rocess (.) and he corresonding auoregressive model (.) (Broersen and Wensink, 993). Using equaion (.), each daa samle can be rediced from is redecessors: y  = - a y - i i i= (.3) I is assumed in hese equaions ha he auoregressive model order is known. In racice, he model order has o be esimaed as well, which is usually done using Akaike s crierion, which is o be discussed in chaer.6.

12 -- As he samles y canno be rediced exacly, a residue is inroduced, which is defined as he difference beween he measured value and he esimaed value:  residue y - y = y + a i y-i = h (.4) i= which means ha he residue is equal o he esimaed innovaions, as inroduced in equaion (.). The mehods of esimaing he auoregressive arameers a i all fundamenally deend on he rincile of leas squares (Durbin, 960). In his resec, he oal squared error h ÂÁ  i -i Ë i= Ê ˆ =  = y + a y (.5) over a number of daa samles can be minimised by seing he arial derivaives wih resec o â i equal o zero: a i {,..., } = 0 " i Œ (.6) leading o he normal equaions  i=  { } a y y = - y y " j Œ,..., (.7) i -i - j  - j which form a se of equaions wih unknown variables. These normal equaions can be rewrien using he auocovariance marix c ij :  a icij = - c0 j " j Œ {,..., } (.8) i= in which  c y y ij -i - j (.9) The normal equaions can hen be rewrien in marix form:

13 -3- Êc c c Á c c c Á Á Á Ëc c c ˆÊ a ˆ Êc0 ˆ Á a Á c 0 Á = -Á Á Á Á Ëa Á Ëc 0 (.0).. APPLICATION OF DATA WINDOWS The range of summaion for has been omied in equaion (.5), (.7) and (.9) o be able o describe four yes of daa windowing. I is assumed ha he auoregressive arameers are esimaed using N daa oins from a realisaion y of he random rocess o be modelled. For convenience, he daa oins ouside his inerval are u equal o zero: y = 0 0 or > N (.) This means ha he resuling signal is a finiely windowed version of some oher infinie signal. Now four yes of windowing can be considered:. No windowing: The range of summaion in (.9) is < N. This means ha for each auocovariance value a sum of N - roducs is used, alhough for shifs smaller han more roducs are available. Therefore c π c3 π c34... and c3 π c4 π c35... e ceera. The resuling marix in equaion (.0) will be symmerical. The auoregressive arameers calculaed using no windowing consiue he exac leas squares soluion. Usually he sum in equaion (.9) is divided by he number of roducs, as a resul of which c = N - y y ij -i - j = + N  (.) forms an unbiased esimae of he auocovariance funcion a delay i - j. According o equaion (.0), dividing c ij by N - will no affec he arameer esimaes.. Prewindowing: The range of summaion in (.9) is 0 < N. All of he firs roducs are used as well, some of which are equal o zero, o calculae he auocovariance values. The las roducs are no being used. Therefore he inequaliies c π c3 π c34... and c3 π c4 π c35... e ceera remain, and he resuling marix in equaion (.0) will be symmerical.

14 -4-3. Poswindowing: The range of summaion in (.9) is < N +. The las roducs are being used as well o calculae he auocovariance values, while he firs roducs are negleced. Also in his case he auocovariance values for equal ime shifs are unequal (c π c3 π c34... and c3 π c4 π c35... e ceera), and he resuling marix in equaion (.0) will be symmerical. 4. Pre- and oswindowing: The range of summaion in equaion (.9) is - < <, or equivalenly 0 < N +. All of he available roducs are used for calculaing he auocovariance values, alhough some of hese roducs equal zero due o he added zeros. The auocovariance values will deend on he ime shif only (c = c3 = c34... and c3 = c4 = c35... e ceera), as a resul of which he resuling marix in (.0) will have he so-called Toeliz form: Ê R R R R R Á Á R R R R R Á R R R R R Á R R R R R Á Á Á ËR R R R R ˆ (.3) in which he marix coefficiens are given by R N =  y y- = N N y y - = = N  (.4) which can be derived from equaion (.9) using he range of summaion saed above. As a resul of summaing an unequal number of roducs for differen delays, R will be a biased esimae of he auocovariance funcion (Parzen, 96). Dividing by N will no affec he arameer esimaes, as can be seen from he normal equaions (.0) (and (.5)). The normal equaions are reduced o he so-called Yule-Walker equaions: Ê R R R Á R R R Á Á Á ËR R R ˆÊ a ˆ Ê R ˆ Á a Á R Á = -Á Á Á Á Ëa Á ËR 0 (.5)

15 -5- A general Toeliz sysem Ê R R R Á R R R Á Á Á ËR - R - R ˆÊ A ˆ Ê B ˆ Á A Á B Á = Á Á Á Á Ë A Á ËB 0 (.6) can be solved using he algorihm resened by Levinson (947), which was subsequenly sreamlined by Durbin (960) o solve equaion (.5) only. Consequenly his algorihm is resenly known as he Levinson-Durbin algorihm (Cybenko, 980). The re- and oswindowed case is alernaively called he Yule- Walker aroach. As in he windowed cases he number of summaed roducs is differen for each marix enry, he auocovariance esimaes will be biased. Clearly, for large daa samles he difference beween he esimaes obained by he various mehods will be small (Priesley, 994). However, as in racice all daa samles are finie, in some cases subsanial differences may arise beween hese aroaches (Broersen and Wensink, 993). In he resen reor a comarison will be made beween he various arameer esimaion echniques... RESIDUE AND PREDICTION ERROR Once he auoregressive arameers have been esimaed from he ime series y, he auoregressive model can be alied o an indeenden realisaion x of he same sochasic rocess. In erms of x, he auoregressive rocess (.) can be wrien as x + ax - + ax- + + a x- = e (.7) in which he innovaion rocess e is saisically idenical o he innovaion rocess h. The corresonding auoregressive model can be wrien as in equaion (.): x + a x + a x + + a - - x- = e (.8) in which â i are he auoregressive arameers esimaed from realisaion y and e are he esimaed innovaions.

16 -6- As in equaion (.3), each daa samle can be esimaed from is redecessors: x  = - a x - i i i= (.9) The difference beween he measured value and he esimaed value is now defined as he redicion error:  redicion error x - x = x + a ix-i = e (.0) i= which means ha he redicion error is equal o he esimaed innovaions, as inroduced in equaion (.8). A clear disincion should be made beween he residue and he redicion error and heir variance (Broersen and Wensink, 993). The residual variance var( h ) is a measure for he fi of he auoregressive model o hose daa ha have been used for he esimaion of he auoregressive arameers. I is equal o he execed value of he squared residue: ( h ) ( h ) var = E (.) since he execed value of he residue equals zero. In his equaion, E denoes he execancy oeraor. The residual variance can be esimaed from he daa oins used for he arameer esimaion: N ( h ) = Â( y - y ) vâr N - = + (.) which is equal o he oal squared error (.5) divided by he number of daa oins N - ha are rediced. Clearly he higher he order of he auoregressive model, he lower he residual variance will be. For he redicion of fuure daa, insead of he residual variance, he variance of he redicion error is essenial, which is equal o he execed value of he squared redicion error: var ( e ) ( e ) = E (.3)

17 -7- since he execed value of he redicion error equals zero. If he indeenden realisaion x conains N daa samles, he redicion error variance can be esimaed from he samle variance: vâr N ( e ) = Â( x - x ) N '- = + (.4) The esimaion of he auoregressive arameers is based on he minimisaion of he residual variance. However, a minimisaion of he residual variance does no mean ha he variance of he redicion error is minimised as well. This disincion is analogous o fiing a olynomial of order o + daa oins. The residual variance will be zero, bu he resuling model will no be reliable ouside he inerolaion domain, as a resul of which he variance of he redicion error will be very large. Therefore using a large model order migh lead o a small residual variance bu a large redicion error variance. Furhermore, one migh wonder wheher he leas squares esimaion of he auoregressive arameers will yield a beer fi o fuure values han he Yule-Walker aroach..3. POLE LOCATION AND STABILITY An auoregressive rocess as in equaion (.) can be rewrien by exressing i in erms of he backward shif oeraor z -, which is defined as yielding - z x = x - (.5) i i x + Ê aiz x x z aiz x Ë Á ˆ Ê ˆ Â = + ÁÂ = e (.6) Ë i= i= If we define a 0, equaion (.6) can be rewrien as e Ê ˆ - -i = z ÁÂaiz x Ë i= 0 (.7) Ignoring he so-called comlemenary funcion, which is he soluion of he homogeneous equaion

18 -8- Ê ˆ - -i z ÁÂaiz x = 0 (.8) Ë i= 0 (Priesley, 994), he soluion of equaion (.7) is formally given by Ê x = z ÁÂaiz Ë i= 0 -i ˆ - e (.9) The oeraor z Ê Á Ë Â i= 0 a z i -i ˆ - obviously conains a -fold zero a z = 0, as well as oles deermined by he characerisic equaion of he auoregressive rocess -i  a i z = 0 (.30) i= 0 I can be shown (Box and Jenkins, 960) ha he sabiliy of he auoregressive rocess is closely linked o he locaion of hese oles in he z-lane. Generally, hree siuaions can occur:. All of he roos of he characerisic equaion lie inside he uni circle in he z-lane. In his case he auoregressive rocess (.) will reresen a saionary rocess. Furhermore i can be shown (Priesley, 994) ha he auocovariance funcion exonenially decays o zero in a smooh form.. The characerisic equaion has roos ha lie ouside he uni circle in he z-lane. In his case he auoregressive rocess (.) will no be saionary. If all of he roos of he characerisic equaion lie ouside he uni circle, i is sill ossible o obain a saionary soluion of (.), bu now we have o use reverse ime and exress x as a linear funcion of resen and fuure values of e (Priesley, 994). 3. The characerisic equaion has roos ha lie on he uni circle. This is he mos difficul case, in which here is no convergen exansion of x in erms of eiher as or fuure values of e. This means ha he auoregressive rocess will only be saionary in case of e being idenical o zero. The resuling sochasic rocess is mos suiably described as a harmonic rocess, defined as: K ( j ) x = ÂC cos w + i i i i= (.3)

19 -9- in which K, { C i } and { w i } are consans, while he { j i } are indeenden random variables, each having a recangular disribuion on he inerval (-, ]. Noe ha he { j i } do no deend on, which means ha he { j i } are consan for each realisaion of x. I can be shown (Priesley, 994) ha he auocovariance funcion is given by K Â i= ( ) ( w ) R = C i cos i (.3) The auocovariance funcion is a sum of cosine erms, and herefore will never die ou. This roery conrass sharly wih he behaviour of he auocovariance funcion for a saionary auoregressive rocess. In ha case he auocovariance funcion may oscillae, bu i evenually decays o zero. This feaure forms he basis of esing for he exisence of a harmonic rocess suerimosed on, for examle, an auoregressive rocess (Priesley, 994). The sabiliy of an esimaed auoregressive model can be verified by subsiuing he esimaed auoregressive arameers ino equaion (.30). If here urns ou o be a roo lying ouside he uni circle, he esimaed auoregressive model becomes invalid. In ha case we are dealing wih a non-saionary sochasic rocess, for which he heory resened so far is no alicable. Forunaely some of he esimaion mehods are guaraneed o yield a sable auoregressive model. Among hese is he leas squares re- and oswindowed (Yule-Walker) aroach, in conrary o he ordinary leas squares aroach..4. PARTIAL AUTOCORRELATION COEFFICIENTS AND REFLECTION COEFFICIENTS From equaion (.), i is obvious ha he daa samles in an auoregressive rocess are correlaed for delays smaller han +. Also for larger delays, he daa samles will be correlaed, due o he effec of he inermediae daa oins. For insance, in case of a firs order auoregressive rocess, x is relaed o x -, which on is urn is relaed o x -. Therefore x is correlaed o x - due o he inermediae daa oin

20 -0- x -. Generally, his means ha he correlaion coefficiens do no equal zero for delays larger han he rocess order. The arial correlaion coefficien is a measure for he correlaion beween wo daa oins in which he effecs of he inermediae daa oins are eliminaed (Priesley, 994). All arial correlaion coefficiens are guaraneed o be less han uniy in absolue value, in accordance wih he definiion of a correlaion coefficien. In case of an auoregressive rocess of order, all arial correlaion coefficiens wih a delay greaer han are equal o zero. This rovides a way of assessing he auoregressive model order from he arial correlaion coefficiens, esimaed from he measured noise daa. The arial correlaion coefficiens can be esimaed by calculaing he so-called reflecion coefficiens. The reflecion coefficien k for delay is defined as he auoregressive arameer for delay in an auoregressive model of order (Kay, 988). The reflecion coefficien esimae k can be calculaed by esimaing he las auoregressive arameer in an auoregressive model of order. The arial correlaion coefficien for delay is equal o he oosie of he -h reflecion coefficien, which means ha he arial correlaion coefficiens can be esimaed by calculaing he reflecion coefficien esimaes. Nex o esimaing he auoregressive model order, he reflecion coefficiens can be used as a basis for he esimaion of he auoregressive arameers. This is acually done by Burg s mehod, which is o be discussed in he nex aragrah..5. SOME OTHER METHODS OF AUTOREGRESSIVE PARAMETER ESTIMATION Nex o he leas squares windowed and non-windowed esimaion mehods, some oher aroaches o he roblem of auoregressive arameer esimaion are available. These mehods usually ry o overcome he disadvanages of he reviously menioned echniques. A shor descriion of some of hese mehods and heir meris is lised below.

21 -- Yule-Walker aroach using he Boxcar-weighed auocovariance funcion As was menioned in chaer., he auoregressive arameer esimaion echniques using windowed daa emloy biased auocovariance esimaes, as for each esimae a differen number of roducs is used in he summaion in equaion (.9). In he reand oswindowed (Yule-Walker) case, he resuling bias can be adjused for by using he Boxcar-weighed auocovariance funcion: R * = N  y y N - = - (.33) The auocovariance values are divided by he acual number of roducs used in he summaion. Therefore hese auocovariance esimaes will be unbiased. As he auocovariance marix will remain in is Toeliz-form, he Levinson-Durbin algorihm is sill alicable. Several disadvanages exis o he use of he Boxcar-weighed auocovariance funcion. Firs of all, he correlaion esimaes will have a much larger variance han in case of he regular Yule-Walker aroach (Parzen, 96). Secondly, he esimaed auocovariance funcion migh no fulfil he requiremen of a auocovariance funcion o be a osiive semi-definie funcion. Consequenly, he Boxcar-weighed auocovariance funcion may lead o a Fourier secrum ha is negaive for some frequencies. Finally, using he Boxcar-weighed auocovariance funcion migh lead o an unsable auoregressive model, in conrary o he regular Yule-Walker aroach, which is guaraneed o yield a sable auoregressive model. Leas squares forward-backward aroach In his aroach, no only he forward redicion error is minimised, bu also he error made in he backward redicion of samles. Using backward redicion, he samles are esimaed using succeeding samles: y  = - b y + i i i= (.34) in which he backward auoregressive arameers esimaes b i are linked o he forward auoregressive arameers by

22 b a i = + -i -- (.35) The oal squared error now conains boh he forward and he backward squared error: = + i -i i= N N - = Ê Á Ë + ˆ + Ê Á Ë + ˆ  y Âa y  y Âb y (.36) = i + i i= Subsiuing equaion (.35) ino (.36) and uing he arial derivaives wih resec o â i equal o zero, he following normal equaions are obained: Ê c + c c + c c + c Á c + c c + c c + c Á Á Á Ëc0 + c c + c c + c , 0 0 -, ˆÊ a Á a Á Á Á Ëa ˆ Ê c Á c = -Á Á Á Ëc + c c 0 + c, -, - -, - 0, - ˆ (.37) The leas squares forward-backward aroach does no guaranee he auoregressive model o be sable. Burg s mehod The arameer esimaion aroach ha is nowadays regarded as he mos aroriae, is known as Burg s mehod. The reason for is oulariy is based on he combinaion of squared bias and variance being oimal. Burg s mehod is guaraneed o yield oles inside or on he uni circle and resembles he leas squares forwardbackward aroach. The following discussion is based on Kay (988). In conras o he reviously reaed esimaion rocedures, which esimaed he auoregressive arameers direcly, Burg s mehod firs esimaes he reflecion coefficiens and hen uses he Levinson-Durbin algorihm o deermine he arameers. The reflecion coefficien esimaes are obained by recursively minimising he esimaed residual variance for auoregressive models of differen orders. Firs he variance of he sochasic rocess is esimaed from he daa samles: R 0 = N N  y = (.38)

23 -3- which is equal o he zero-h order esimae x 0 of he variance of he innovaion rocess h: = R x 0 0 (.39) The zero-h order forward residue is hen defined as ( ) y { N} f h 0 = Œ, 3,..., (.40) while he zero-h order backward residue is given by ( ) y { N } b h 0 = Œ,,..., - (.4) Then he esimaed reflecion coefficiens k k, he variance esimaes x k and he arameer esimaes â ki are calculaed for increasing model order k: k k = N  f b - h ( ) h ( -) h = k+ N  = k+ f k- k- k- N  b ( ) + h ( -) = k+ k- (.4) ( k ) x x k = - k k- (.43) a ki { } ÏÔ a k -, i + kka k -, k -i i Œ,,..., k - = Ì ÓÔ k i = k k (.44) f in which he forward and backward residues h k ( ) and h b k ( ) are esimaed recursively: f f ( ) ( ) b h h h ( ) k = k- + kk k - - (.45) b b ( ) ( ) f h = h - + k h ( ) (.46) k k- k k- The resuling auoregressive arameer esimaes are given as { a,,..., a a } x is he final esimae for he variance of he urely random rocess. while

24 DETERMINATION OF THE AUTOREGRESSIVE MODEL ORDER USING AKAIKE S CRITERION Before he auoregressive arameers can be esimaed, a choice has o be made concerning he auoregressive model order, which is he number of foregoing daa samles being used for esimaing he nex one. Nowadays he model order is usually deermined using he informaion crierion develoed by Akaike (97, 974). This crierion can be alied in saisical model idenificaion in a wide range of siuaions and is no resriced o he auoregressive model or even o ime-series analysis (Priesley, 994). Akaike s informaion crierion is described in a general form in numerous exbooks on robabiliy and mahemaical saisics (e. g. Kay, 988). In case of a ime series model, Akaike s informaion crierion is a funcion of he auoregressive model order : ( ) ( ) ln vâr( ) AIC = N h + (.47) in which N is he number of daa samles and vâr( h ) is he residual variance esimae ( ) (.) for model order. The residual variance and herefore N ln vâr( ) h will decrease a increasing model order, while he second erm on he righ hand side of equaion (.47) will obviously increase. The aroriae auoregressive model order will be ha number of for which AIC aains is minimum value (Priesley, 994). In racice his means ha he auoregressive model is esimaed for increasing numbers of. The residual variance is esimaed for each model order using equaion (.), and alying equaion (.47) yields he values of AIC as a funcion of. Then he AIC is loed agains and is minimum is deermined. The corresonding value of will be aken as he aroriae auoregressive model order.

25 -5-3. POLE LOCATION AND CONVERGENCE RATE For large daa samles, he differences beween he arameer esimaes using he various mehods described in he revious chaer will be small. In some secial cases however, subsanial differences may arise beween hese aroaches even for daa samles of moderae size. The convergence rae of he various auoregressive arameer esimaion echniques is essenial in his resec. In chaer he relaionshi was exlained beween he locaion of he oles of he auoregressive oeraor and he behaviour of he corresonding auoregressive rocess. I was shown ha all oles have o lie inside he uni circle in order of he auoregressive rocess o be sable. In case of he oles lying on he uni circle, a saionary rocess can only exis if he noise erms e are equal o zero. One migh wonder wha will haen if he auoregressive rocess has oles in he neighbourhood of he uni circle. As oles on he uni circle reresen a harmonic rocess, an auoregressive rocess wih oles near he uni circle can be execed o demonsrae some kind of seudo-eriodic behaviour. In his case he auocovariance funcion can be described as a sum of weakly damed eriodic funcions. This means ha x dislays some kind of disored eriodiciy (Priesley, 994). Furhermore, as he noise erms e are sill resen, he auoregressive rocess may exhibi a kind of almos non-saionary behaviour. The ole locaions will also affec he convergence rae of he various arameer esimaion echniques and herefore heir reliabiliy. I is claimed by Priesley (994) ha areciable differences may arise beween he various esimaes, even for moderaely large daa samles, if he AR oeraor has a ole near he uni circle. In his case he Yule-Walker equaions may lead o oor arameer esimaes. Unforunaely, no horough heoreical reamen exiss o exlain his henomenon. In his chaer i is demonsraed wha haens if a ole is locaed closely o he uni circle and why some of he arameer esimaion echniques will erform oorly in ha case. As a full mahemaical reamen of a general order auoregressive model wih oles near he uni circle is no ye available, we will firs consider a firs and a second order model. Thereuon i is shown which mahemaical circumsances lead o oor arameer esimaes in case of a general order auoregressive model.

26 AUTOREGRESSIVE PROCESS OF ORDER ONE HAVING A POLE NEAR THE UNIT CIRCLE In case of a firs order auoregressive rocess, each daa samle deends linearly on is redecessor: x + a x - = e (3.) in which a is he auoregressive arameer and e is a saionary discree ime urely random rocess. The characerisic equaion (.30) for a firs order auoregressive rocess can be wrien as z + a = 0 (3.) which means ha he ole is locaed a z = - a. For a ª, he ole will be close o he uni circle. In order o redic fuure values of x, he auoregressive arameer should be esimaed using one of he mehods described in he revious chaer. I should be noed ha he urely random comonen e of a daa samle causes a lower bound for he redicion error variance. The various arameer esimaion echniques reviously resened will be comared here regarding heir redicion error variance. Burg s mehod shall no be considered here, because in case of a firs order model i is equivalen o he leas squares forward-backward aroach. For he following derivaion i is assumed ha he auoregressive arameer is esimaed using N daa oins. The redicion error variance is hen calculaed for an indeenden realisaion of he same sochasic rocess. A general exression for he redicion error variance can be derived from equaions (.0) and (.3) in case of a firs order model: ( e ) = ( x - x ) = var( e - a x + a x ) var var - - (3.3) According o his equaion, he redicion error variance will be affeced by he variance of he auoregressive arameer esimae, which can be wrien as

27 -7- ( ) var â a = - N - (3.4) for a firs order auoregressive rocess (Priesley, 994). This means ha for large daa samles as well as for a ª, he arameer esimae variance can be negleced. Consequenly, he arameer esimae in equaion (3.3) can be relaced by is execed value ( ) E â : ( e ) = ( e - a x + ( a) x ) var var E (3.5) - - in which E is he execancy oeraor. Because e is a urely random rocess, i is indeenden from receding values of x. Therefore equaion (3.5) can be rewrien as: ( e ) ( e ) ( ( a) a) ( x ) var var = + E - var - (3.6) The second erm on he righ hand side of equaion (3.6) reresens he influence of he bias in he auoregressive arameer esimae on he redicion error variance. The execed value of he auoregressive arameer esimae can be calculaed in erms of he zero and uni delay auocovariance esimaes R 0 and R : E ( a ) ( R ) ( R 0) E( R 0) R E = - E Ê cov ( R, R ) Ë Á ˆ ª R E 0 ( ) ( R ) ( R 0) E E ( R ) var 3 0 (3.7) in a second order aroximaion. A firs order aroximaion for he execed value only akes he firs erm on he righ hand side ino accoun. In equaion (3.7), he noaion for he Yule-Walker auocovariance esimaes R0, R is being used in a general sense. The redicion error variance will be considered here relaive o he variance of he sochasic rocess, which is given by var ( x ) = var( e ) Ê- a ˆ Á (3.8) Ë - a (Priesley, 994). For large values of, his can be aroximaed by

28 var ( x ) -8- var( e ) = - (3.9) a which is a consan for all, as execed for a saionary sochasic rocess. Subsiuing equaion (3.8) ino (3.6), he relaive redicion error variance can be calculaed as ( e ) ( x ) var var = - a - a - - a + ( E ( a ) - a) (3.0) - a In case of he ole lying closely o he uni circle, a will be close o. Aroximaing he fracions on he righ hand side of (3.0) yields ( e ) ( x ) var var Ê -ˆ = + ( E ( a ) - a) Á (3.) Ë For large values of, he relaive redicion error variance can be calculaed as ( e ) ( ) var var x = ( E( a ) - a) (3.) The relaive redicion error variance is herefore closely linked o he arameer esimae being eiher biased or unbiased, which in urn is closely conneced o he auocovariance esimaes being used in he various esimaion echniques. Therefore he echniques using unbiased auocovariance esimaes (leas squares and leas squares forward-backward) shall be reaed searaely from he Yule-Walker esimaion mehod, which uses biased auocovariance esimaes. Unbiased esimaion echniques: leas squares and leas squares forwardbackward The esimaes of he zero delay auocovariance in case of leas squares and leas squares forward-backward, being c and c + c ( ) resecively, are unbiased: 00 The rivial facor in case of he leas squares forward-backward aroach is negleced in his derivaion.

29 -9- ( R ) E var( e ) = R = - a 0 0 (3.3) The same holds for he uni delay auocovariance, which is esimaed by c 0 and c + c resecively: 0 0 ( R ) E a var( ) = R = - e - a (3.4) For large values of N, he covariance beween he zero and uni delay auocovariance esimaes for a firs order auoregressive rocess can be aroximaed by ( R R ) cov, 0 ( var( e )) 4a = - N - - a ( ) 3 (3.5) while he variance of he zero delay auocovariance esimae is aroximaed by ( R ) var 0 ( var( e )) = N - + a ( - a ) 3 (3.6) (Priesley, 994). Subsiuing equaions (3.3), (3.4), (3.5), (3.6) ino equaion (3.7) yields he following resul for he auoregressive arameer esimae: ( ) E â a = a - N - (3.7) So using unbiased auocovariance esimaes leads o a biased auoregressive arameer esimae. The bias in he leas squares and leas squares forward-backward esimae remains bounded hough, irresecive of he locaion of he ole. Equaion (3.7) can hen be subsiued ino (3.) o calculae he relaive redicion error variance: ( e ) 4a = ( x ) ( N -) var var (3.8)

30 -30- This means ha in case of leas squares and leas squares forward-backward, for large N he redicion error variance will aroach zero, irresecive of he locaion of he ole. For N = 00 and a = , he relaive redicion error variance according o equaion (3.8) can be calculaed as var var ( e ) ( ) x - = (3.9) Similar resuls can be derived for he leas squares forward-backward aroach. For large values of N, is relaive redicion error variance will aroach zero, irresecive of he ole locaion. Yule-Walker aroach Using he leas squares re- and oswindowed aroach ( Yule-Walker aroach), he auocovariance esimaes used for calculaing he auoregressive arameer esimaes are no longer unbiased, as hey equal he auocovariance esimaes roosed by Parzen (96) (equaion (.4)). The bias in he auocovariance esimaes increases for increasing delay: ( ) E R = N - N R (3.0) For large values of N, he covariance beween he zero and uni delay auocovariance esimaes for a firs order auoregressive rocess can now be aroximaed by ( R R ) cov, 0 ( var( e )) = - N 4a ( - a ) 3 (3.) while he variance of he zero delay auocovariance esimae is aroximaed by ( R ) var 0 ( var( e )) + a = N - a ( ) 3 (3.) (Priesley, 994).

31 -3- Subsiuing hese values ino equaion (3.7), he execed value of he auoregressive arameer esimae is given by ( ) 3 E( â) a N a a + a = - - (3.3) N - a This means ha he Yule-Walker esimae is biased as well, as can be execed from using biased auocovariance esimaes. I should be noed hough ha he Yule- Walker bias is no longer bounded. For a ª, ( ) E â and consequenly he bias will be exremely large. Subsiuing equaion (3.3) ino (3.), he relaive redicion error variance for he Yule-Walker aroach can be calculaed as ( e ) ( ) ( + a ) var Ê 3 var x N a a = Á + Ë N - a ˆ (3.4) I is seen from his equaion ha he relaive redicion error variance will aroach zero as N goes o infiniy. However, for finie values of N he relaive redicion error variance will become infinie as a aroaches. For N = 00 and a = , he relaive redicion error variance according o equaion (3.4) can be calculaed as ( e ) ( ) var var x = (3.5) These resuls show ha he relaive Yule-Walker redicion error variance becomes large for ole locaions close o he uni circle. Obviously, he examle daa used for calculaing he redicion error variance are exreme, and i can be quesioned wheher he aroximaions alied in his derivaion are sill valid for ole locaions so close o he uni circle. Noneheless, he examle daa do show how oles near he uni circle can lead o a large redicion error variance. Assuming ha he same effec will occur for larger model orders, he Yule-Walker esimaion echnique is unreliable as a means of daa redicion.

32 AUTOREGRESSIVE PROCESS OF ORDER TWO HAVING POLES NEAR THE UNIT CIRCLE In case of a second order auoregressive rocess, i will be much more difficul o give a mahemaical reamen of he redicion error variance. Therefore, insead of a mahemaical derivaion, simulaions were made of second order auoregressive rocesses wih oles near he uni circle. As boh of he auoregressive arameers in a second order rocess are real, he roos of he characerisic equaion (.30) will form a comlex conjugaed air ( ). Thus he locaion of he oles can be exressed as ( ) z = R ex ± ia (3.6) in which R denoes he absolue value and a he argumen of he ole above he real axis. The auoregressive rocess can hen be wrien as x - R cos( a ) x + R x = e (3.7) - - Simulaions were made for a values of 0, ¼, ½, ¾ and radians and for magniudes R aroaching he uni circle, using he leas squares no windowing aroach, he Yule-Walker aroach, he leas squares forward-backward aroach and Burg s mehod. Each simulaion consised of 04 daa oins, using a normally disribued urely random innovaion rocess having uni variance. As hese kinds of auoregressive rocesses will exhibi close o non-saionary behaviour, each recorded simulaion was receded wih 040 dummy simulaions o reven he occurrence of sar-u non-saionary behaviour. Each simulaion was carried ou en imes using he Malab for Windows rogram version 4.b. In each simulaion, he residual and he redicion error variance as well as he firs and second auoregressive arameer were esimaed, which were hereuon averaged over he number of simulaions. The residual variance was esimaed from equaion (.). The redicion error variance was esimaed using a second series of 04 daa samles. The firs series was used for esimaing he auoregressive arameers, while he second series was used for redicing he daa The case in which here are wo disinc real roos shall no be considered here.

33 -33- samles using he various arameer esimaes in order o esimae he redicion error variance from equaion (.4). a = 0 rad In his case he oles coincide on he osiive real axis and cluser around z = as hey aroach he uni circle. The auoregressive rocess is now given by x - R x - + R x - = e (3.8) The simulaion resuls for he residual variance are shown in figure. While leas squares, leas squares forward-backward and Burg s mehod sill yield a residual variance close o he acual value (being uniy) as he oles aroach he uni circle, he Yule-Walker mehod is no longer able o describe he auoregressive rocess correcly. Even for oles locaed a 0.0 from he uni circle, he residual variance in case of he Yule-Walker aroach is almos weny imes as large as in case of he leas squares aroach. Residual variance YW 0 0 LS, LSFB, Burg Disance o he uni circle Figure. Simulaion resuls for he residual variance in case of a = 0 rad using he various esimaion echniques (LS = Leas squares, YW = Yule-Walker, LSFB = Leas squares forward-backward, Burg = Burg s mehod).

34 -34- This resul does no aly o he residual variance only. The redicion error variance also becomes large in case of he Yule-Walker esimaion. The simulaion resuls are given in figure. Predicion error variance YW 0 0 LS, LSFB, Burg Disance o he uni circle Figure. Simulaion resuls for he redicion error variance in case of a = 0 rad using he various esimaion echniques. In hese simulaions, i was found ha he auoregressive arameer esimaes were no accurae in case of he Yule-Walker mehod. The firs and second auoregressive arameer esimaes and heir acual values are loed in figures 3 and 4 resecively. I is seen ha he second auoregressive arameer esimae goes o zero in case of he Yule-Walker mehod as he oles aroach he uni circle. This would mean ha he second order Yule-Walker esimaion leads o a firs order aroximaion of a second order rocess. The esimaed firs auoregressive arameer in case of a second order Yule-Walker esimaion is equal o he esimaed auoregressive arameer in case of a firs order Yule-Walker esimaion. This would mean ha in case of oles close o he uni circle Yule-Walker esimaes a lower order auoregressive model. For a discussion of he esimaed ole locaions in case of Yule-Walker, he reader is referred o he age 38.

35 -35- Firs auoregressive arameer - -. YW Acual, LS, LSFB, Burg Disance o he uni circle Figure 3. Simulaion resuls for he firs auoregressive arameer esimae in case of a = 0 rad using he various esimaion echniques (Acual = he acual value of he firs auoregressive arameer). Second auoregressive arameer Acual, LS, LSFB, Burg YW Disance o he uni circle Figure 4. Simulaion resuls for he second auoregressive arameer esimae in case of a = 0 rad using he various esimaion echniques.

36 -36- a = ¼ rad; a = ½ rad; a = ¾ rad In his case one of he oles lies above he real axis while he oher one lies below, since hey consiue a comlex conjugaed air. No maer how close he oles are locaed near he uni circle, boh he residual and he redicion error variance remain bounded for all of he esimaion echniques, including Yule-Walker. The execaions ha are saed hereabou in lieraure (e. g. Priesley, 994) are no suored by our simulaions. a = rad In his case he oles coincide on he negaive real axis and cluser around z = - as hey aroach he uni circle. The auoregressive rocess is now given by x + R x - + R x - = e (3.9) The simulaion resuls for he residual variance are shown in figure 5. While leas squares, leas squares forward-backward and Burg s mehod sill yield he correc residual variance (being uniy) as he oles aroach he uni circle, he Yule-Walker mehod is no longer able o describe he auoregressive rocess correcly. For oles locaed a 0.0 from he uni circle, he residual variance in case of he Yule-Walker aroach is more han weny imes as large as in case of he leas squares aroach. As was done in he case of a = 0 rad, he redicion error variance was calculaed using a second simulaed ime series. The simulaion resuls are given in figure 6.

37 -37- Residual variance YW 0 0 LS, LSFB, Burg Disance o he uni circle Figure 5. Simulaion resuls for he residual variance in case of a = rad using he various esimaion echniques. Predicion error variance YW 0 0 LS, LSFB, Burg Disance o he uni circle Figure 6. Simulaion resuls for he redicion error variance in case of a = rad using he various esimaion echniques.

38 -38- In hese simulaions, once again he auoregressive arameer esimaes were no accurae in case of he Yule-Walker mehod. The firs and second auoregressive arameer esimaes and heir acual values are loed in figures 7 and 8 resecively. Also in his case he second auoregressive arameer esimae goes o zero, while he firs auoregressive arameer esimae aroaches he auoregressive arameer of a firs order model. Once again alying a second order Yule-Walker esimaion yields a firs order model. Esimaed ole locaions Because of he Yule-Walker mehod guaraneeing he esimaed auoregressive model o be sable, he esimaed ole locaions are inside he uni circle. In case of oles locaed closely o he uni circle, i could be execed ha Yule-Walker yields ole locaion esimaes farher away from he uni circle in order of he auoregressive model o be sable. This migh exlain he oor Yule-Walker esimaion erformance in case of a = 0 rad and a = rad. This is however no he case. In case of Yule-Walker, one of he ole locaions is esimaed correcly, while he oher one is shifed owards he origin. If he oles are locaed closely o he uni circle, he laer may be osiioned a he origin, hereby suggesing ha ole-zero cancellaion occurs. This means ha a firs order model insead of a second order model is esimaed. Furhermore, he ole ha is esimaed correcly coincides wih he ole of he corresonding firs order model.

39 -39- Firs auoregressive arameer.8 Acual, LS, LSFB, Burg.6.4. YW Disance o he uni circle Figure 7. Simulaion resuls for he firs auoregressive arameer esimae in case of a = rad using he various esimaion echniques. Second auoregressive arameer Acual, LS, LSFB, Burg YW Disance o he uni circle Figure 8. Simulaion resuls for he second auoregressive arameer esimae in case of a = rad using he various esimaion echniques.

40 AUTOREGRESSIVE PROCESSES OF A GENERAL ORDER HAVING POLES NEAR THE UNIT CIRCLE As he Yule-Walker aroach may erform oorly in case of firs and second order auoregressive rocesses having oles near he uni circle, one migh exec similar resuls for general order auoregressive models. Since he oles of an auoregressive rocess are deermined hrough very comlicaed mahemaical rocesses saring from he auocovariance funcion, i is almos inracable o rea his roblem heoreically (Yamada, 986). Neverheless i is ossible o show ha he mahemaical circumsances in case of oles near he uni circle are such ha oor arameer esimaes in case of he Yule-Walker aroach are o be execed. Magniude of he arial correlaion coefficiens According o Cybenko (980), he locaion of he oles of he auoregressive rocess is linked o he magniude of he arial correlaion coefficiens beween he daa oins. All arial correlaion coefficiens are guaraneed o be less han uniy in absolue value. However, if he arial correlaion coefficiens are close o one in absolue sense, he underlying ime series (.) is close o non-saionariy (Box and Jenkins, 970). In he conex of linear filering heory, he oles deermined by he characerisic equaion (.30) are in his case inside bu close o he uni circle in he comlex lane (Oenheim, 978). This means ha he ransfer funcion relaing x o e as in equaion (.9) will be close o insabiliy in he filering sense. Thus in alicaions he condiion in which he arial correlaion coefficiens are close o ± means ha he underlying roblem o be modelled will be close o some anomalous behaviour (Cybenko, 980). The arial correlaion coefficiens can be calculaed for he firs and second order auoregressive rocesses ha were discussed in he revious chaers. A discussion of auocorrelaion and arial auocorrelaion coefficiens in connecion o auoregressive rocesses is given by Priesley (994). In he firs order case, he uni delay arial auocorrelaion coefficien equals he uni delay auocorrelaion coefficien: r = - a (3.30)