Nonlinear Time Series Modeling

Transcription

1 Nonlinear Time Series Modeling Richard A. Davis Colorado Sae Universiy h:// MaPhySo Worksho Coenhagen Seember 7 3, 4 MaPhySo Worksho 9/4

2 Par I: Inroducion o Linear and Nonlinear Time Series. Inroducion. Examles 3. Linear rocesses 3. Preliminaries 3. Wold Decomosiion 3.3 Reversibiliy 3.4 Idenifiabiliy 3.5 Linear ess 3.6 Predicion 4. Allass models 4. Alicaion of allass Noninverible MA model fiing Microsof Muddy Creek Seisomogram deconvoluion 4. Esimaion MaPhySo Worksho 9/4

3 Par II: Time Series Models in Finance. Classificaion of whie noise. Examles 3. Sylized facs concerning financial ime series 4. ARCH and GARCH models 5. Forecasing wih GARCH 6. IGARCH 7. Sochasic volailiy models 8. Regular variaion and alicaion o financial TS 8. univariae case 8. mulivariae case 8.3 alicaions of mulivariae regular variaion 8.4 alicaion of mulivariae RV equivalence 8.5 examles 8.6 Exremes for GARCH and SV models 8.7 Summary of resuls for ACF of GARCH & SV models MaPhySo Worksho 9/4 3

4 Par III: Nonlinear and NonGaussian Sae-Sace Models. Inroducion. Moivaion examles. Linear sae-sace models.3 Generalized sae-sace models. Observaion-driven models. GLARMA models for TS of couns. GLARMA exensions 3.3 Oher 3. Parameer-driven models 3. Esimaion 3. Simulaion and Alicaion 3.3 How good is he oserior aroximaion MaPhySo Worksho 9/4 4

5 Par IV: Srucural Break Deecion in Time Series. Piecewise AR models. Minimum descriion lengh MDL 3. Geneic algorihm GA 4. Simulaion examles 5. Alicaions EEG and seech examles 6. Alicaion o nonlinear models MaPhySo Worksho 9/4 5

6 References: Brockwell and Davis 99. Time Series: Theory and Mehods Brockwell and Davis. Inroducion o Time Series and Forecasing. Durbin and Kooman. Time Series Analysis by Sae- Sace Models. Embrechs, Klüelberg, and Mikosch 997. Modelling Exremal Evens. Fan and Yao. Nonlinear Time Series. Frances and van Dik. Nonlinear Time Series Models in Emirical Finance. Harvey 989. Forecasing, Srucural Time Series Models and he Kalman Filer. Rosenbla. Gaussian and Non-Gaussian Linear Time Series and Random Fields. Subba-Rao and Gabr 984. An Inroducion o Bisecral Analysis and Bilinear Time Series Models. Tong. Nonlinear Time Series Models; a dynamical sysems aroach. MaPhySo Worksho 9/4 6

7 Why nonlinear ime series models? Wha are he limiaions of linear ime series models?. Inroducion Wha key feaures in daa canno be caured by linear ime series models? Wha diagnosic ools visual or saisical sugges incomaibiliy of a linear model wih he daa? MaPhySo Worksho 9/4 7

8 Examle: Z,..., Z n ~ IID, Series Samle auocorrelaion funcion ACF: n h γˆ h ρˆ Z h where γ ˆ h n Z Z Z + h Z γˆ is he samle auocovariance funcion ACVF. MaPhySo Worksho 9/4 8

9 Theorem. If {Z }~ IID,, hen ρˆ, K, ˆ h' Z ρ Z Proof: see roblem 6.4 TSTM is aroximaely IID N,/n.. Samle ACF. Samle PACF MaPhySo Worksho 9/4 9

10 Cor. If {Z }~ IID, and E Z 4 <, hen ρˆ, K, ˆ h' Z ρ Z is aroximaely IID N,/n.. Residual ACF: Abs values. Residual ACF: Squares MaPhySo Worksho 9/4

11 Wha if E Z? For examle, suose {Z }~ IID Cauchy. 5. Series. Samle ACF Resul see TSTM 3.3: If {Z }~ IID Cauchy, hen n S ρˆ Z h,, ln n S. 5. S and S.5 are indeenden sable random variables. MaPhySo Worksho 9/4

12 How abou he ACF of he squares?. Residual ACF: Abs values. Residual ACF: Squares Resul: If {Z }~ IID Cauchy, hen n n ρˆ Z h S / ln S. 5 -., S.5 and S.5 are indeenden sable random variables. MaPhySo Worksho 9/4

13 Reversibiliy. The saionary sequence of random variables {X } is ime-reversible if X,...,X n d X n,...,x. Resul: IID sequences {Z } are ime-reversible. Alicaion: If lo of ime series does no look ime- reversible, hen i canno be modeled as an IID sequence. Use he fli and comare insecion es! Series MaPhySo Worksho 9/4 3

14 Reversibiliy. Does he following series look ime-reversible?. Series Residual Samle ACF: ACFAbs values. Residual ACF: Squares MaPhySo Worksho 9/

15 . Examles Closing Price for IBM //6-/3/ closing rice MaPhySo Worksho 9/ ime 5

16 Log reurns for IBM /3/6-/3/ blue96-98 *logreurns - - MaPhySo Worksho 9/ ime 6

17 Samle ACF IBM a 96-98, b 98- a ACF of IBM s half b ACF of IBM nd half ACF ACF Lag 3 4 Lag Remark: Boh halves look like whie noise? MaPhySo Worksho 9/4 7

18 Samle ACF of abs values for IBM a 96-98, b 98- a ACF, Abs Values of IBM s half b ACF, Abs Values of IBM nd half ACF ACF Lag 3 4 Lag Remark: Series are no indeenden whie noise? MaPhySo Worksho 9/4 8

19 ACF of squares for IBM a 96-98, b 98- a ACF, Squares of IBM s half b ACF, Squares of IBM nd half ACF ACF MaPhySo Worksho 9/ Lag Lag Remark: Series are no indeenden whie noise? Try GARCH or a sochasic volailiy model. 9

20 Examle: Pound-Dollar Exchange Raes Oc, 98 Jun 8, 985; Kooman websie day MaPhySo Worksho 9/4 log reurns exchange raes - 4 ACF lag ACF of squares ACF of abs values lag lag

21 Examle: Daily Ashma Presenaions 99: Jan Feb Mar Ar May Jun Jul Aug Se Oc Nov Dec Year Jan Feb Mar Ar May Jun Jul Aug Se Oc Nov Dec Year Jan Feb Mar Ar May Jun Jul Aug Se Oc Nov Dec Year Jan Feb Mar Ar May Jun Jul Aug Se Oc Nov Dec Year 993 Remark: Usually marginal disribuion of a linear rocess is coninuous. MaPhySo Worksho 9/4

22 Muddy Creek- ribuary o Sun River in Cenral Monana Muddy Creek: surveyed every 5.4 meers, oal of 5456m; 358 measuremens Degree AIC c bed elevaion disance m MaPhySo Worksho 9/4

23 Muddy Creek: residuals from olyd4 fi residuals deg disance m Minimum AIC c ARMA model: ARMA, Y.574 Y - + ε.3 ε -, {ε }~WN,.564 Some heory: Noncausal ARMA, model: LS esimaes of rend arameers are Y asymoically.743 Y - + ε efficien..3 ε - LS esimaes are asymoically inde of cov arameer esimaes. acf Blue samle Red model acf Blue samle Red model 3 4 MaPhySo Worksho 9/4 lag m 3 4 lag m 3

24 Muddy Creek con Summary of models fied o Muddy Creek bed elevaion: Degree AIC c ARMA AIC c, 59.67, 6.98, 6.3, 7.,.78, 4.68 MaPhySo Worksho 9/4 4

25 Examle: NEENe Ecosysem Exchange in Harvard Fores Abou half of he CO emied by humans accumulaes in he aomoshere Oher half is absorbed by sink rocesses on land and in he oceans NEE Rh + Ra GPP carbon flux GPP Gross Primary Producion hoosysynhesis Rh Heerorohic microbial resiraion Ra auorohic lan resiraion. The NEE daa from he Harvard Fores consiss of hourly measuremens. We will aggregae over he day and consider daily daa from Jan, 99 o Dec 3,. Go o ITSM Demo MaPhySo Worksho 9/4 5

26 3. Linear Processes 3. Preliminaries Def: The sochasic rocess {X,, ±, ±,...} defined on a robabiliy sace is called a discree-ime ime series. Def: {X } is saionary or weakly saionary if i. E X <, for all. ii. EX m, for all. iii. CovX, X +h γh deends on h only. Def: {X } is sricly saionary if X,...,X n d X +h,...,x n+h for all n and h, ±, ±, Remarks: i. SS + E X < weak saionariy ii. WS SS hink of an examle iii. MaPhySo Worksho 9/4 WS + Gaussian SS why? 6

27 3. Preliminaries con Def: {X } is a Gaussian ime series if X m,...,x n is mulivariae normal for all inegers m < n, i.e., all finie dimensional disribuions are normal. Remark: A Gaussian ime series is comleely deermined by he mean funcion and covariance funcions, m EX and γs, CovX s, X. If follows ha a Gaussian TS is saionary SS or WS if and only if m m and γs, γ-s deends only on he ime lag -s. MaPhySo Worksho 9/4 7

28 Def: {X } is a linear ime series wih mean if where {Z } ~ WN, and Imoran remark: As a reminder WN means uncorrelaed random variables and no necessarily indeenden noise nor indeenden Gaussian noise. X ψ Z, Proosiion: A linear TS is saionary wih i. EX, for all. ii. γ h ψ ψ + h and ρ h ψ ψ h / 3. Preliminaries con If {Z } ~ IID,, hen he linear TS is sricly saionary. ψ <. + ψ MaPhySo Worksho 9/4 8

29 Is he converse o he revious roosiion rue? Tha is, are all saionary rocesses linear? Answer: Almos. 3. Wold Decomosiion TSTM Secion 5.7 Examle: Se X A cosω + B sinω, ω,π, where A,B ~ WN,. Then {X } is saionary since E X, γh cosωh ~ Def: Le P n be he bes linear redicor oeraor ono he linear san of he observaions X n, X n-,.... For his examle, ~ P X X. n n n Such rocesses wih his roery are called deerminisic. MaPhySo Worksho 9/4 9

30 The Wold Decomosiion. If {X } is a nondeerminisic saionary ime series wih mean zero, hen X where i. ψ, Σ ψ <. ii. {Z } ~ WN, ψ iii. covz s,v for all s and ~ iv. P Z Z for all. ~ v. P V V for all s and. s vi. {V } is deerminisic. Z + The sequences {Z }, {V }, and {ψ } are unique and can be wrien as ~ Z X P X, ψ E X Z / E Z, V X ψ Z 3. Wold Decomosiion con V,. MaPhySo Worksho 9/4 3

31 3. Wold Decomosiion con Remark. For many ime series in aricular for all ARMA rocesses he deerminisic comonen V is for all and he series is hen said o be urely nondeerminisic. Examle. Le X U + Y, where {U } ~ WN, and is indeenden of Y~,τ. Then, in his case, Z U and V Y see TSTM, roblem 5.4. Remarks: If {X } is urely nondeerminisic, hen {X } is a linear rocess. Secral disribuion for nondeerminisic rocesses has he form F X F U + F V, where U ψ Z which has secral densiy f λ π ψ e iλ π ψ e iλ MaPhySo Worksho 9/4 3

32 3. Wold Decomosiion con ~ If E X P X >, hen F X F U + F V, is he Lebesque decomosiion of he secral disribuion funcion; F U is he absoluely coninuous ar and F V is he singular ar. Examle. Le X U + Y, where {U } ~ WN, and is indeenden of Y~,τ. Then F X dλ dλ + τ δ dλ π Kolmogorov s Formula. Clearly MaPhySo Worksho 9/4 πex{π π π π π ln f λ dλ}, > iff ln f λ dλ >. where E X ~ P X. 3

33 Examle TSTM roblem 5.3. sin X Z, {Z} ~ WN, τ, ψ. ψ π This rocess has a secral densiy funcion bu is deerminisic!! Examle see TSTM roblem 5.. Le and se I follows ha { Z } ~ WN, and X is he WD for {X }. Z. 5Z a If {ε }~IID N,, is {Z } IID? Answer? b If {ε }~IID,, is {Z } IID? Answer? MaPhySo Worksho 9/4 X Z ε ε, { ε} ~ WN, τ.5b ε 3. Wold Decomosiion con.5 3 X.5 X ε ε ε, +.5 ε ε +.5 ε ε 3 + L 33

34 3. Wold Decomosiion con Remark: In his las examle, he rocess {Z } is called an allass model of order. More on his ye of rocess laer. Go o ITSM Demo 3.3 Reversibiliy Recall ha he saionary ime series {X } is ime-reversible if X,...,X n d X n,...,x for all n. MaPhySo Worksho 9/4 34

35 The saionary ime series {X } is ime-reversible if X,...,X n d X n,...,x for all n. Theorem Breid & Davis 99. Consider he linear ime series {X } X 3.3 Reversibiliy, { Z } ~ IID, where ψz ±z r ψz - for any ineger r. Assume eiher a Z has mean and finie variance and {X } has a secral densiy osiive almos everywhere. or b /ψzπzσ π z, he series converging absoluely in some annulus D conaining he uni circle and πbx Σ π X - Z. Then {X } is ime-reversible if and only if Z is Gaussian. ψ Z MaPhySo Worksho 9/4 35

36 Remark: The condiion ψz ±z r ψz - on he filer recludes he filer from being symmeric abou one of he coefficiens. In his case, he ime series would be ime-reversible for non-gaussian noise. For examle, consider he series X Z 3.3 Reversibiliy con. 5Z + Z, { Z} ~ IID Here ψz -.5z + z z -.5 z - + z z ψz - and he series is ime-reversible. Proof of Theorem: Clearly any saionary Gaussian ime series is imereversible why?. So suose Z is nongaussian and assume a. If {X } ime-reversible, hen ψ B Z X d X. Z a Z ψ B ψ B ψ B MaPhySo Worksho 9/4 36

37 3.3 Reversibiliy con The firs equaliy akes a bi of argumen and relies on he secral reresenaion of {X } given by X e π, π] i λ dz λ, where Zλ is a rocess of orhogonal incremens see TSTM, Chaer 4. I follows, by he assumions on he secral densiy of {X } ha iλ X λ, ± ψ e dz miλ B ψ e π, π] is well defined. So ψ B. Z d Z a Z ψ B and, by he assumion on ψz, he rhs is a non-rivial sum. Noe ha a Why? The above relaion is a characerizaion of a Gaussian disribuion see Kagan, Linnik, and Rao 973. MaPhySo Worksho 9/4 37

38 3.3 Reversibiliy con Examle: Recall for he examle X ε ε, { ε } ~ IID,, τ and non-normal, he Wold decomosiion is given by X Z. 5Z, where Z ε 3.5 ε. By revious resul, {Z } canno be ime-reversible and hence is no IID. Remark: This heorem can be used o show idenifiabiliy of he arameers and noise sequence for an ARMA rocess. MaPhySo Worksho 9/4 38

39 3.4 Idenifiabiliy Moivaing examle: The inverible MA rocess X Z + θz, { Z} ~ IID,, θ <, has a non-inverible MA reresenaion, X ε + θ ε, { ε } ~ WN, θ, θ <. Quesion: Can he {ε } also be IID? Answer: Only if he Z are Gaussian. If he Z are Gaussian, hen here is an idenifiabiliy roblem, θ, θ, θ, θ <, give he same model. MaPhySo Worksho 9/4 39

40 3.4 Idenifiabiliy con For ARMA rocesses {X } saisfying he recursions, X φ X L φ X φ B X Z + θ θ B Z Z + Lθ q Z q, { Z } ~ IID,, casualiy and inveribiliy are yically assumed, i.e., φ z and θ z for z. By fliing roos of he AR and MA olynomials from ouside he uni circle o inside he uni circle, here are aroximaely +q equivalen ARMA reresenaions of X driven wih noise ha is whie no IID. For each of hese equivalen reresenaions, he noise is only IID in he Gaussian case. Boom line: For nongaussian ARMA, here is a disincion beween causal and noncausal; and inverible and non-inverible models. MaPhySo Worksho 9/4 4

41 3.4 Idenifiabiliy con Theorem Cheng 99: Suose he linear ime series X has a osiive secral densiy a.e. and can also be reresened as X Then if {X } is nongaussian, i follows ha for some osiive consan c. ψ Z, { Z} ~ IID,, ψ η Y, { Y} ~ IID, τ, η ψ c Y cz η +,, <, <. Proof of Theorem: As in he roof of he reversibiliy resul, we can wrie Z ψ B X η B Y ψ B a Y and Y b Z MaPhySo Worksho 9/4 4

42 3.4 Idenifiabiliy con Now le {Ys,} ~IID, Ys, d Y and se s - U a Y s,. s Clearly, {U } is IID wih same disribuion as Z. Consequenly, Y d bu s b a Y s,. s Since b as s, Which by alying Theorems 5.6. and 3.3. in Kagan, Linnik, and Rao 973, he sum above is rivial, i.e., here exiss inegers m and n such ha a m and b n are he only wo nonzero coefficiens. I follows ha MaPhySo Worksho 9/4 b Y bnz n, η ψ + n n. 4

43 3.5 Linear Tess Cumulans and Polysecra. We canno base ess for lineariy on second momens. A direc aroach is o consider momens of higher order and corresonding generalizaions of secral analysis. Suose ha {X } saisfies su E X k < for some k 3 and E + X X L X E X + h X + h L X h for all,,...,, h, and,..., k-. k h order cumulan. Coefficien, C k r,..., r k-, of i k z z z k in he Taylor series exansion abou,,, of χ z, K, zk ln E ex izx + izx + r + L+ iz k X + rk MaPhySo Worksho 9/4 43

44 3.5 Linear Tess con 3 rd order cumulan. X µ X µ X C3 r, s E + r + s µ If C3 r, s < r s hen we define he bisecral densiy or 3 rd order olysecral densiy To be he Fourier ransform, -π ω, ω π. f 3 irω isω ω, ω 3,, C r s e π r s MaPhySo Worksho 9/4 44

45 3.5 Linear Tess con k h - order olysecral densiy. Provided f k ω L k C r, K, r <, r r r k k, K, ω π k k : L r r r k C k r, K, r k e ir ω Lir k ω k, -π ω,..., ω k π. See Rosenbla 985 Saionary Sequences and Random Fields for more deails. MaPhySo Worksho 9/4 45

46 46 MaPhySo Worksho 9/4 Alied o a linear rocess. If {X } has he Wold decomosiion wih E Z 3 <, EZ 3 η, and Σ ψ <, hen where ψ : for <. Hence,, } ~ {, ψ IID Z Z X s r s r C + + ψ ψ ψ η, 3. 4, 3 ω ω ω + ω ψ ψ ψ π η ω ω i i i i e e e f 3.5 Linear Tess con

47 47 MaPhySo Worksho 9/4 The secral densiy of {X } is Hence, defining we find ha Tesing for consancy of φ hus rovides a es for lineariy of {X } see Subba Rao and Gabr 98.. ω ψ π ω i e f,,, 3 + ω ω ω ω ω ω ω ω φ f f f f., 6 π η ω ω φ 3.5 Linear Tess con

48 3.5 Linear Tess con Gaussian linear rocess. If {X } is Gaussian, hen EZ 3, and he hird order cumulan is zero why?. In fac C k for all k >. I follows ha f 3 ω, ω for all ω, ω [,π]. A es for linear Gaussianiy can herefore be obained by esimaing f 3 ω,ω and esing he hyohesis ha f 3 see Subba Rao and Gabr 98. MaPhySo Worksho 9/4 48

49 3.6 Predicion Suose {X } is a urely nondeerminisic rocess wih WD given by Then so ha ~ P X Z X X ψ ψ Z ~ P, X Z { Z. } ~ WN, Quesion. When does he bes linear redicor equal he bes redicor? Tha is, when does ~ P X E X X, X K?. MaPhySo Worksho 9/4 49

50 Answer. Need Z or, equivalenly, ~ P 3.6 Predicion con X E X X, X K? ~ P X X X, X, K E Z X, X, K. Tha is, BLP BP if and only if {Z } is a Maringale-difference sequence. Def. {Z } is a Maringale-difference sequence wr a filraion F an increasing sequence of sigma fields if E Z < for all and a Z is F measurable b EZ F - a.s. MaPhySo Worksho 9/4 5

51 3.6 Predicion con Remarks. An IID sequence wih mean zero is a MG difference sequence. A urely nondeerminisic Gaussian rocess is a Gaussian linear rocess. This follows by he Wold decomosiion and he fac ha he resuling {Z } sequence mus be IID N,. Examle While: Consider he noncausal AR rocess given by X X - + Z, where {Z }~IID PZ - PZ.5. Ieraing backwards in ime, we find ha X.5X.5 M X.5 Z.5Z +.5.5Z Z + +.5Z.5 Z + L. MaPhySo Worksho 9/4 5

52 5 MaPhySo Worksho 9/4 3.6 Predicion con is a binary exansion of a uniform, random variable. Noice ha from X, we can find X +, by loing off he firs erm in he binary exansion. This oeraion is exacly, X + X mod X, if X <.5, X -, if X >.5. * 3 * 3 * * 3, Z Z Z Z Z Z Z Z X L L Proeries:

53 4. Allass models Realizaion from an allass model of order 3 noise X ACF : allass A C F : allass ACF ACF model samle MaPhySo Worksho 9/4 3 4 Lag 3 4 Lag 53

54 4. Allass models con Causal AR olynomial: φzφ z Define MA olynomial: L φ z, φz for z. L θz z φz /φ z φ z - φ / φ for z MA olynomial is non-inverible. Model for daa {X } : φbx θb Z, {Z } ~ IID non-gaussian B k X X -k Examles: All-ass: X φx - Z φ Z -, φ <. All-ass: X φ X - φ X - Z + φ / φ Z - / φ Z - MaPhySo Worksho 9/4 54

55 Proeries: causal, non-inverible ARMA wih MA reresenaion X B φ B Z ψ Z φ φ B uncorrelaed fla secrum f X iω iω e φ e ω iω φ φ e π φ π zero mean daa are deenden if noise is non-gaussian e.g. Breid & Davis 99. squares and absolue values are correlaed. X is heavy-ailed if noise is heavy-ailed. MaPhySo Worksho 9/4 55

56 Esimaion for All-Pass Models Second-order momen echniques do no work leas squares Gaussian likelihood Higher-order cumulan mehods Giannakis and Swami 99 Chi and Kung 995 Non-Gaussian likelihood mehods likelihood aroximaion assuming known densiy quasi-likelihood Oher LAD- leas absolue deviaion R-esimaion minimum disersion MaPhySo Worksho 9/4 56

57 4. Alicaion of Allass models Noninverible MA models wih heavy ailed noise X Z +θ Z θ q Z -q, a. {Z } ~ IID nonnormal b. θz + θ z + + θ q z q No zeros inside he uni circle inverible Some zeros inside he uni circle noninverible MaPhySo Worksho 9/4 57

58 Realizaions of an inverible and noninverible MA rocesses Model: X θ B Z, {Z } ~ IIDα, where θ i B +/B + /3B and θ ni B + B + 3B ACF ACF Lag Lag MaPhySo Worksho 9/4 58

59 Alicaion of all-ass o noninverible MA model fiing Suose {X } follows he noninverible MA model X θ i B θ ni B Z, {Z } ~ IID. Se : Le {U } be he residuals obained by fiing a urely inverible MA model, i.e., So X U θˆbu ~ θ B θ i θ ~ B ni θ B ni ni Z BU, ~ θ ni is he inverible version of θ ni. Se : Fi a urely causal AP model o {U } ~ θ ni BU θ ni BZ. MaPhySo Worksho 9/4 59

60 Volumes of Microsof MSFT sock raded over 755 ransacion days 6/3/96 o 5/8/99 X *^5 6*^5 ^6 4 6 MaPhySo Worksho 9/4 6

61 Analysis of MSFT: Se : Logvolume follows MA4. X +.53B+.77B +.7B 3 +.B 4 U inverible MA4 Se : All-ass model of order 4 fied o {U } using MLE -dis:.68b +.9B.649B +.35B +.3B 3.B + 3.6B 3 4 U 4.96B 4 Z. ˆ ν 6.6 Model using R-esimaion is nearly he same. Conclude ha {X } follows a noninverible MA4 which afer refiing has he form: X +.34B+.374B +.54B B 4 Z, {Z }~IID 6.3 MaPhySo Worksho 9/4 6

62 a ACF of Squares of U b ACF of Absolue Values of U ACF..4.8 ACF Lag c ACF of Squares of Z 3 4 Lag d ACF of Absolue Values of Z ACF..4.8 ACF MaPhySo Worksho 9/4 Lag Lag 6

63 Summary: Microsof Trading Volume Two-se fi of noninverible MA4: inverible MA4: residuals no iid causal AP4; residuals iid Direc fi of urely noninverible MA4: +.34B+.374B +.54B B 4 For MCHP, inverible MA4 fis. MaPhySo Worksho 9/4 63

64 Muddy Creek: residuals from olyd4 fi residuals deg Minimum AIC c ARMA model: ARMA, Y.574 Y - + ε.3 ε -, {ε }~WN, disance m acf Blue samle Red model acf Blue samle Red model 3 4 MaPhySo Worksho 9/4 lag m 3 4 lag m 64

65 . Residual ACF: Abs values. Residual ACF: Squares Causal ARMA, model Y.574 Y - + ε.3 ε -, {ε }~WN, Residual ACF: Abs values. Residual ACF: Squares Noncausal ARMA, model: Y.743 Y - + ε.3 ε MaPhySo Worksho 9/

66 Examle: Seismogram Deconvoluion Simulaed waer gun seismogram {β k } wavele sequence Lii and Rosenbla, 988 {Z } IID refleciviy sequence -5 5 MaPhySo Worksho 9/ ime 66

67 Waer Gun Seismogram Fi Se : AICC suggess ARMA,3 fi fi inverible ARMA,3 via Gaussian MLE residuals no IID Se : fi all-ass o residuals order seleced is r. residuals aear IID Se 3: Conclude ha {X } follows a non-inverible ARMA MaPhySo Worksho 9/4 67

68 MaPhySo Worksho 9/ lag h 68 acf ACF of W lag h acf ACF of Z

69 Waer Gun Seismogram Fi con Recorded waer gun wavele and is esimae True Esimae Esimae inverible lag MaPhySo Worksho 9/4 69

70 Waer Gun Seismogram Fi con Simulaed refleciviy sequence and is esimaes MaPhySo Worksho 9/4 7

71 4. Esimaion for Allass Models: aroximaing he likelihood Daa: X,..., X n Model: X φ X + L+ φ + Z φ Z L φ Z / φ X r where φ r is he las non-zero coefficien among he φ s. Noise: z φ L z + + L+ φ z X φx φ X, where z Z / φ r. More generally define, z φ, φ z φ + L+ φ φ φ B X if n +,..., n +, + z, if n,..., +. Noe: z φ is a close aroximaion o z iniializaion error MaPhySo Worksho 9/4 7

72 Assume ha Z has densiy funcion f and consider he vecor z X,..., X, z φ,..., z φ, z φ,..., zn + φ,..., zn φ' Join densiy of z: indeenden ieces h z h X n f,..., X φ q z, z φ φ φ,..., z q h z φ n + φ,..., z n φ, and hence he oin densiy of he daa can be aroximaed by n h x f φqz φ φ q where qmax{ : φ }. MaPhySo Worksho 9/4 7

73 Log-likelihood: φ, ln / L n φq + where f z fz/. n ln f φ q z φ Leas absolue deviaions: choose Lalace densiy f z ex z and log-likelihood becomes Concenraed Lalacian likelihood Maximizing lφ is equivalen o minimizing he absolue deviaions MaPhySo Worksho 9/4 consan n ln κ n φ consan ln l n m n n φ z φ. n z φ / κ, z φ κ / φ q 73

74 Assumions for MLE Assume {Z } iid f z f z wih a scale arameer mean, variance furher smoohness assumions inegrabiliy, symmery, ec. on f Fisher informaion: ~ I f ' z / f z dz Resuls Le γh ACVF of AR model wih AR oly φ. and Γ [ γ- k],k nˆ φ MLE MaPhySo Worksho 9/4 φ D N, ~ I Γ 74

75 75 MaPhySo Worksho 9/4 Furher commens on MLE Le αφ,..., φ, / φ, β,..., β q, where β,..., β q are he arameers of df f. Se Fisher Informaion { } dz z f z f z f dz z f z f z f z dz z f z f z dz z f z f T f β β β β β β β β β β β β β β α α + + ; ; ; I ; ; ; ' L ; / ; ' Kˆ ; / ; ' Î,,

76 76 MaPhySo Worksho 9/4 Under smoohness condiions on f wr β,..., β q we have where Noe: is asymoically indeenden of and,, ˆ MLE N n D α α Γ ' ˆ ˆ ' ˆ ' ˆ ' ˆ ' ˆ ˆ L LK I LK L LK I L LK I L K L L I K I f f f f ˆ MLE φ ˆ MLE +, α ˆ MLE β

77 Asymoic Covariance Marix For LS esimaors of AR: n ˆ φ LS φ D N, Γ For LAD esimaors of AR: nˆ φ LAD φ D N, 4 f Γ For LAD esimaors of AP: nˆ φ LAD φ D N, Var Z f E Z Γ For MLE esimaors of AP: MaPhySo Worksho 9/4 nˆ φ MLE φ D N, Iˆ Γ 77

78 Lalace: LADMLE Sudens ν, ν >: Var Z f E Z Iˆ Suden s 3 : LAD: MLE: ν π ν Γ ν / 4 ν Γ ν + / 8Γ ν + / ν ν + 3 Iˆ LAD:.7337 MLE:.5 ARE:.7337/ MaPhySo Worksho 9/4 78

79 R-Esimaion: Minimize he obecive funcion n S φ ϕ z n + φ where {z φ} are he ordered {z φ}, and he weigh funcion ϕ saisfies: ϕ is differeniable and nondecreasing on, ϕ is uniformly coninuous ϕx ϕx Remarks: For LAD, ake MaPhySo Worksho 9/4 n R φ S φ ϕ z φ n + -, ϕ x, < x < /, / < x <. 79

80 Assumions for R-esimaion Assume {Z } iid wih densiy funcion f disr F mean, variance Assume weigh funcion ϕ is nondecreasing and coninuously differeniable wih ϕx ϕx Resuls Se ~ J ~ ~ If L > K, hen nˆ φ ϕ R s ds, φ K ~ F s ϕ s ds, ~ ~ D J K N, ~ ~ L K L ~ Γ f F s ϕ' s ds MaPhySo Worksho 9/4 8

81 Furher commens on R-esimaion ϕx x/ is called he Wilcoxon weigh funcion By formally choosing ϕ x -,, < x < /, / < x <. we obain ~ ~ J K ~ ~ L K Γ Var Z f E Z Γ. Tha is R LAD, asymoically. The R-esimaion obecive funcion is smooher han he LAD-obecive funcion and hence easier o minimize. MaPhySo Worksho 9/4 8

82 Obecive Funcions MaPhySo Worksho 9/4 R-esimaion LAD hi 8

83 83 MaPhySo Worksho 9/4 Summary of asymoics Maximum likelihood: R-esimaion Leas absolue deviaions: ~, ˆ MLE Γ D I N n φ φ Var, ˆ LAD Γ D Z E f Z N n φ φ ~ ~ ~ ~, ˆ R Γ D K L K J N n φ φ

84 Lalace: LADMLE ~ ~ R: J K 5 ~ ~ using ϕx x/, Wilcoxon L K 6 Sudens ν : LADMLE: / ν LAD R MLE LAD/R MLE/R MaPhySo Worksho 9/4 84

85 85 MaPhySo Worksho 9/4 Cenral Limi Theorem R-esimaion Think of u n / φφ as an elemen of R Define where R φ is he rank of z φ among z φ,..., z n- φ. Then S n u Su in disribuion on CR, where Hence,, - -/ -/ + ϕ ϕ n n n z n R n z n n R S φ φ φ φ u u u, ~ ~, ~, ' ' ~ ~ r r K J N K L S Γ φ + Γ φ N N u u u u ~ ~ ~ ~, ~ ~ ~ arg min ˆ arg min / Γ φ Γ φ r r D R n K L K J N K L S n S N u u φ φ

86 86 MaPhySo Worksho 9/4 Main ideas R-esimaion Define where F z is he df of z. Using a Taylor series, we have Also, Hence, - ~ -/ ϕ + ϕ n z n z n z z F n z z F S φ φ u u u u N u u u u u r D n z n z n K z z F n z z F n S Γ φ ϕ + ϕ - -/ ~ ' ' ' ' ' ~ ~ φ φ φ φ φ. / ~ ' ~ r n n o L S S + φ Γ u u u u. ~ ~, ~, ' ' ~ ~ r r D n K J N K L S Γ φ + Γ φ N N u u u u

87 Order Selecion: Parial ACF From he revious resul, if rue model is of order r and fied model is of order > r, hen n ˆφ / φˆ, LAD N, Var Z f E where is he h elemen of.,lad φˆ LAD Z Procedure:. Fi high order P-h order, obain residuals and esimae scalar, θ Var Z f E Z by emirical momens of residuals and densiy esimaes. MaPhySo Worksho 9/4 87

88 . Fi AP models of order,,..., P via LAD and obain -h coefficien ˆφ, for each. 3. Choose model order r as he smalles order beyond which he esimaed coefficiens are saisically insignifican. Noe: Can relace ˆφ wih if using MLE. In his case for > r, φˆ, MLE n / φˆ, MLE N,. Iˆ MaPhySo Worksho 9/4 88

89 AIC: or no? An aroximaely unbiased esimae of he Kullback-Leiber index of fied o rue model: AIC Var Z f L X f E Z E Z : ˆ, φ κˆ + Penaly erm for Lalace case: Var Z f E Z f E Z Penaly erm can be esimaed from he daa. MaPhySo Worksho 9/4 89

90 Samle realizaion of all-ass of order a Daa From Allass Model b ACF of Allass Daa X -4 - ACF c ACF of Squares 3 4 Lag d ACF of Absolue Values ACF ACF MaPhySo Worksho 9/4 Lag Lag 9

91 Simulaion resuls: relicaes of all-ass models model order arameer value φ.5 φ.3, φ.4 noise disribuion is wih 3 d.f. samle sizes n5, 5 esimaion mehod is LAD MaPhySo Worksho 9/4 9

92 9 MaPhySo Worksho 9/4 To guard agains being raed in local minima, we adoed he following sraegy. 5 random saring values were chosen a random. For model of order, k-h saring value was comued recursively as follows:. Draw iid uniform -,.. For,,, comue Selec o based on minimum funcion evaluaion. Run Hooke and Jeeves wih each of he saring values and choose bes oimized value.,...,, k k k φ φ φ φ φ φ φ φ φ φ,,,,, k k k k k k k M M M

93 Asymoic Emirical N mean sd dev mean sd dev %coverage rel eff* 5 φ φ Asymoic Emirical N mean sd dev mean sd dev %coverage 5 φ φ φ φ *Efficiency relaive o maximum absolue residual kurosis: n n z φ / v 4 3, v n n z φ z φ MaPhySo Worksho 9/4 93

94 R-Esimaor: Minimize he obecive fcn n S φ z n + where {z φ} are he ordered {z φ}. φ Emirical Emirical LAD N mean sd dev mean sd dev 5 φ φ φ φ φ φ MaPhySo Worksho 9/4 94