Transformation to approximate independence for locally stationary Gaussian processes

Transcription

1 ransformation to approximate independene for loally stationary Gaussian proesses Joseph Guinness, Mihael L. Stein We provide new approximations for the likelihood of a time series under the loally stationary Gaussian proess model. he likelihood approximations are valid even in ases when the evolutionary spetrum is not smooth in the resaled time domain. We desribe a broad lass of models for the evolutionary spetrum for whih the approximations an be omputed partiularly effiiently. In developing the approximations, we extend to the loally stationary ase the idea that the disrete Fourier transform is a deorrelating transformation for stationary time series. he approximations are applied to fit nonstationary time series models to high frequeny temperature data. For these data, we fit evolutionary spetra that are pieewise onstant in time and use a geneti algorithm to searh for the best partition of the time interval. Keywords: Loally stationary; periodogram; spetral analysis. 1 Introdution he lass of nonstationary time series models is a broad one. Even if we restrit the lass to univariate mean-zero Gaussian time series, we are still left with a lass that may be indexed by the set of all positive (or non-negative) definite ovariane funtions defined on the observation domain. One popular approah to defining an interpretable sublass of nonstationary time series models has been through the use of evolutionary spetra, due to Priestley (1965, 1981). Obtaining rigorous asymptoti results under this model is diffiult, and the asymptoti formulation of Dahlhaus (1996) has provided an important way forward in this regard. Speifially, let be the unit irle and N the natural numbers. For N and A a omplex-valued transfer funtion on [, 1] satisfying A(u, ν) = A(u, ν) for all (u, ν) [, 1], onsider the lass of real-valued stohasti proesses Z on 1,..., with Z (t) = A(t/, ν)e itν Ẑ(dν), (1) where Ẑ is a mean-zero omplex-valued, orthogonal measure with E Ẑ(dν) = dν and Ẑ(ν) = Ẑ( ν) to ensure Z is real. For large, Z is lose to stationary over time periods muh shorter than and, thus, the sequene of proesses Z may be alled a loally stationary sequene (Dahlhaus, 1996). Several authors have provided useful methods for estimating the evolutionary spetrum. Muh of the work has been sed on expressing the evolutionary spetrum in terms of a olletion of sis funtions. Neumann and von Sahs (1997) use a wavelet sis. Adak (1998) uses a windowed Fourier Basis. Dohono, Mallat and von Sahs (1998) use the osine paket transform. here are several papers, inluding Omo et al. (1), Omo et al. (), and Guo et al. (3) that use the smooth, loalized omplex exponential sis. Dahlhaus proposed estimation via the maximization of approximate Gaussian likelihoods, whih depend on a loal periodogram (Dahlhaus, 1997) and the preperiodogram (Dahlhaus, ) under some parametri model for the evolutionary spetrum. he preperiodogram is an estimate of the evolutionary spetrum introdued by Neumann and von Sahs (1997). Dahlhaus s () approximate likelihood is a funtion of the preperiodogram and has the attrative property that it redues to the Whittle likelihood (Whittle, 196) when the evolutionary spetrum is onstant as a funtion of time, i.e. when the proess is stationary. here may be some benefit, for both intuition and omputation, to restriting the form of A. Priestley (1965) desribed the lass of uniformly modulated proesses, whose transfer funtions may be expressed as A(t/, ν) = m(t/ )µ(ν) 1

2 in the loally stationary framework. Here, the variane of the proess is allowed to vary with time, but the underlying orrelation struture is stationary. For many environmental time series, inluding the temperature data that we analyze in this paper, the uniformly modulated model is not suffiiently flexible to adequately apture the ovariane struture exhibited in the data. We propose that a useful, interpretable, and flexible sublass of models for A is A(t/, ν) = K m k (t/ )µ k (ν). () k=1 Of ourse, this sublass may be made quite general if K is allowed to be infinite, but when K is taken to be a small integer, () may be onsidered a low-rank approximation to a partiular hoie of transfer funtion oming from a more general lass. he model in () possesses a lose relationship to the approah taken in hidden state modeling, i.e. when the proess an be desribed by a partiular state or superposition of several states at any given time. Here, the states are represented by the µ k s, and the ontribution of state k at time t is given by m k (t/ ). We assume A is real and positive, although Proposition 1 below holds for omplex A. A phase that is onstant in time is not identifiable, but a time-varying phase may be at least partially identifiable. We neglet the possibility of time-varying phase relationships, whih is a loss of generality. Other authors have impliitly reognized the diffiulty in estimating the phase by fousing on the estimation of the evolutionary spetrum. For example, both of Dahlhaus s (1997, ) likelihood approximations depend on A only through A. he rest of the paper is organized as follows. Setions and 3 develop the quadrati form and the log determinant approximations in the Gaussian likelihood, as well as desribe the omputational advantages that the model in () failitates. Setion ontains a numerial experiment demonstrating the auray and stability of the approximations. In Setions 5 and 6, we desribe the temperature data and the pieewise onstant in time model for A. Our optimization proedure, whih employs a geneti algorithm to searh for a partition of the time domain, is outlined in Setion 7. he results of the model fitting for the temperature data are given in Setion 8, and finally in Setion 9, we disuss the auray of the approximations for the models that we fit to the temperature data, and we ompare the auray to the likelihood approximations proposed by Dahlhaus. Quadrati Form Approximation he ovariane funtion of the proess desribed in (1) is given by K (s, t; A) = K (s, t) = ov{z (s), Z (t)} = A(s/, ν)a(t/, ν) e i(s t)ν dν. (3) Defining the matrix {C (A)} t,j+1 = π/ A(t/, ν j )e iνjt, with ν j = πj/ and H denoting the onjugate transpose operation, we might expet the ovariane funtion defined by D (s, t; A) = D (s, t) = {C (A)C (A) H } s,t = π A (s/, ν j ) A (t/, ν j ) e iνj(s t) () j= to approximate (3). We study the onvergene of the matrix (A) = D (, ) to the matrix Σ (A) = K (, ) with respet to the Frobenius matrix norm, F. he following proposition relaxes the results of Dahlhaus in that it does not require any smoothness of A in the resaled time domain, although it requires uniform smoothness of A in the frequeny domain. Proposition 1: If, for eah u, A(u, ν) is a omplex-valued twie ontinuously differentiable funtion on with respet to ν, and the seond partial derivative of A(u, ν) with respet to ν is uniformly bounded in u and ν, then Σ (A) (A) F = O(1) as.

3 A proof is given in Appendix A.1. he assumption that A is real and positive is not needed for Proposition 1. If, in addition, the ondition number of Σ (A) is bounded, the inverse matries onverge as well (see Horn and Johnson, 6, page 336), and thus the quadrati form term in the Gaussian likelihood an be approximated with 1 C (A) Z, (5) where Z is the 1 vetor of observations. In one sense, the inverse transformation in (5) an be thought of as a generalization of the idea that the disrete Fourier transform (DF) is a deorrelating transformation for stationary proesses. In the stationary ase, A(t/, ν) is onstant as a funtion of t, so we write A(t/, ν) = S(ν). A simple alulation onfirms that C (S) = (π) C (1/S) H (where (1/S)(ν) 1/S(ν)), so the inverse transformation is given by [ C (S) Z ]j = 1 1 e iνjt Z (t), S(ν j ) π whih is simply the DF of the observations saled by the square root of the spetral density, so in the stationary ase our transformation redues to the transformation given in some versions of the Whittle likelihood. here are several ways to ompute the inverse matrix-vetor produt in (5). One ould proeed by diretly onstruting and inverting the matrix C (A), but this is learly intratable for large, when even storing a matrix, let alone omputing its inverse, may not be possible. We propose that the most attrative method is to arrive at the inverse matrix-vetor produt using iterative methods. Convergene within a small number of iterations depends on our ability to find a good preonditioning matrix M, for whih MC (A) has a small ondition number relative to that of C (A). We have found that preonditioning with M = C (1/A) H works quite well in all of our examples. he speed of eah iteration depends on how fast C (A)w and Mw an be omputed for any vetor w. he model in () is well-suited to failitate effiient omputation of C (A)w. If w = (w,..., w ) is a 1 vetor, then [ π K ] [C (A)w] t = m k (t/ )µ k (ν j ) e iνjt w j = π j= K k=1 k=1 t= m k (t/ ) [µ k (ν j )w j ] e iνjt. herefore, the matrix-vetor produt an be omputed as a weighted sum of K inverse DFs, eah of whih an be omputed effiiently using the Fast Fourier ransform (FF). Furthermore, there is no need to store the matrix C (A). In Setion 6, we explore a speifi model for A for whih the preonditioning multipliation an be omputed using the FF and also does not require O( ) storage. Although this estimate omputes the exat inverse matrix-vetor produt, still present are the usual approximations that our in the spetral analysis of stationary time series. Speifially, approximation () effetively assumes that the proess is periodi with period, so when the beginning and the end of the series have different properties, the quadrati form estimate tends to have larger errors. his is true in both the stationary and nonstationary ases. We do not expet our approximation to solve this problem, but it should perform no worse in the nonstationary ase than the Whittle likelihood performs in the stationary ase. j= 3 Log Determinant Approximation Dahlhaus (, Proposition.5) shows that under suitable regularity onditions (inluding an assumption about some third order mixed partial derivatives of A), for any ɛ >, log det Σ (A) = π 1 π π log{π A(u, ν) }dν du + O( ɛ ). (6) 3

4 Let us instead start with the losely related approximation, log det{c (A)/ π} 1 log{b tj }, (7) t,j=1 where b tj = A (t/, ν j ) is the tj th entry defining the matrix B (A). his approximation has the virtues of being diretly appliable for numerial omputation and of being exat when A has the form A(u, ν) = m(u)µ(ν). Figures 1 and give some numerial results illustrating typial errors of the first order approximation given in (7) for four hoies of A. o study the approximation (7), it is helpful to onsider the deomposition A(u, ν) = m(u)µ(ν)r(u, ν), where m, µ and R are all positive-valued and bounded away from. his deomposition is not unique, and it should be hosen, roughly speaking, to minimize disontinuities in R. Suppose the following onditions on R hold: (I) For every u, R is absolutely ontinuous in ν with a uniformly bounded almost everywhere derivative. (II) Exept for possibly some finite number of values of u (not depending on ν), for every ν, R is absolutely ontinuous in u with a uniformly bounded almost everywhere derivative. At these exeptional u values, R may not be ontinuous in u, but for definiteness, we will assume R is right ontinuous in u for all ν. hen our numerial results suggest log det{c (A)/ π} = 1 log{b tj } + O(1) as. (8) j,t=1 hat suh a result might hold under suffiient smoothness onditions is perhaps not surprising in light of (6), whih would give an O(1) error for the log determinant if ɛ ould be set to. However, the onjetures that no smoothness assumptions are needed on m and µ and that R an have disontinuities in u might be somewhat unexpeted but is supported by the numerial results in Figures 1 and and in other examples we have tried. Suppose the following stronger ondition holds: (III) R is absolutely ontinuous in both u and ν with uniformly bounded almost everywhere derivative in both arguments, and R(, ν) = R(1, ν) for all ν. hen our numerial results suggest log det{c (A)/ π} = 1 log{b tj } + O( ) as. (9) j,t he assumption that R(, ν) = R(1, ν) for all ν may sometimes be plausible in ases for whih there is a substantial periodi omponent to the nonstationarity suh as for the high frequeny temperature data onsidered in Setion 5. In this example, the data enompass temperatures from a -hour period, so we might expet the beginning and the end of the series to have similar properties. In a wide variety of ases, the following refinement of (7) works quite well: defining b +1,j = b 1j, and σ (A) = 1 then approximate log det{c (A)/ π} by β t (A) = 1 j,t j log b t+1,j b tj { log b } t+1,j β t (A), b tj log det (A, γ) = 1 log{b tj } + γσ (A), (1) j,t

5 where γ appears to vary modestly with A with values between.5 and.35 working well. Note that if the approximation in (1) had an error of o( ) (with γ possibly depending on A), then it would explain both (8) and (9), sine under the onditions (I) and (II), Proposition 3 in Appendix A.3 shows that σ (A) = O(1) and under the stronger ondition (III), σ (A) = O( ). When R is identially 1, then σ (A) = and both (7) and (1) are exat. If R is identially 1, and m is onstant, then (1) is equivalent to the approximation given in some versions of the Whittle likelihood. Another feature of the approximation (1) (and of (7) for that matter) is that it is invariant to moving the last olumn of B (A) to the first olumn, an invariane that an be shown to hold also for the exat determinant. Proposition in Appendix A. gives a theoretial justifiation for approximation (1) with γ =.5 in a limited setting. he errors from approximations (7) and (1) are modest in all of the examples shown in Figures 1 and and all others we have onsidered, ranging from about.1 to 1 and either dereasing with or staying fairly onstant as varies. he orreted approximation (1) with γ =.5 always does better than (7) and sometimes muh better. For j = 1,, A j (1, ν)/a j (, ν) is not onstant in ν, so we only expet (8) and not (9) to hold for these funtions. he funtion A 1 has another disontinuity at t =.5: A 1 (.5 +, ν)/a 1 (.5, ν) is not onstant in ν, but the numerial evidene suggests (8) still holds in this ase and the refined approximation log det (A,.5) has error more than an order of magnitude smaller than (7) for all onsidered. Figure shows two funtions that are absolutely ontinuous in both arguments and satisfy A(1, ν)/a(, ν) onstant; we see that (7) provides exellent approximations for these funtions and log det (A,.5) is muh better still. Figure 3 plots the ratio of the error of (7) to.5σ (A) for the four funtions onsidered in Figures 1 and and it appears that the values are onverging as inreases but to somewhat different values for the different funtions. Beause even the approximation (7) is very aurate when A is smooth and A(1, ν)/a(, ν) is onstant, it makes more sense to set γ to a value that works well when these onditions do not hold (e.g., the funtions A 1 and A in Figure 1), so we reommend setting γ =.5. hese approximations do not and annot be expeted to work well for any funtion R. In partiular, disontinuities in ν for R ause serious problems for the approximations, although numerial experiments suggest that the errors an grow more slowly than for suh funtions, so that the approximations may still be of some use when there are suh disontinuities. We onsider disontinuities in frequeny for this interation term R (disontinuities in the main effet for frequeny an be aptured in the funtion µ and do not ause any problem) to be unlikely in pratie, at least for natural proesses. A simple numerial experiment o ompare our approximations with those proposed by Dahlhaus (1997, ), we present two simple examples in whih A depends on a single parameter, and we ompute the expetations of the exat loglikelihood and eah of the approximations over a range of parameter values. he purpose of this numerial experiment is to show an example in whih not only are our approximations sharper than Dahlhaus s, but the errors are more stable with respet to a hanging parameter, whih is a desirable property when maximizing the likelihood with respet to unknown parameters, for whih only differenes in loglikelihood matter. he exat negative loglikelihood for the loally stationary Gaussian time series is (minus a onstant / log(π)) L() = 1 log det Σ (A ) + 1 Z Σ (A ) Z. For evolutionary spetra that are pieewise onstant on K bloks in time, i.e. A (u, ν) = µ k, (ν) if u is in blok k, as will be the ase in the examples to follow, we interpret the loglikelihood approximation in Dahlhaus (1997) to be L a () = log det (A, ) + 1 π K k=1 J k (ν) B k µ k, (ν) dν, 5

6 where B k denotes the set of times in blok k, B k is the length of blok k, and J k is the ordinary periodogram over the kth blok of data. If we let K denote the ovariane funtion of the time series with parameter equal to the true value, the periodograms have expetation E(J k (ν)) = 1 π B k he loglikelihood approximation in Dahlhaus () is L b () = log det (A, ) + 1 π where I (t/, ν) is the preperiodogram, whih has expetation E(I (t/, ν)) = 1 π 1 t+1/±k/ s,t B k K (s, t)e iν(s t). t=1 I (t/, ν) A (t/, ν) dν, K ( t + 1/ k/, t + 1/ + k/ )e iνk, where x is the integer part of x. Disretizing the integrals in L a and L b into terms gives suffiient auray in this example. Our likelihood approximation is whose quadrati form has expetation L () = log det (A,.5) + 1 C (A ) Z, 1 tr(c (A ) Σ (A )C (A ) H ). In this numerial experiment, we ompare E(L approx ()) to E(L()) over a range of values for and for a few hoies of. Defining µ (ν) = 1/ π exp( os(ν)), the two models we onsider for this experiment are { A (1) 1/ π, t/ [,.5] (t/, ν) =, µ (ν), t/ (.5, 1] A () (t/, ν) = { 1/ π, t/ [,.5] (.75, 1] µ (ν), t/ (.5,.75] so that the dynami range of the spetrum over one segment inreases with. We take = 1, and we plot in Figure the ovariane funtion orresponding to A (1), noting that when, there is nonzero dependene aross bloks. Furthermore, the various approximations differ when beause the proess is nonstationary in those ases. Otherwise, when = all of the approximations are equal to the Whittle likelihood approximation, whih happens to be exat when = beause the proess is stationary white noise in that ase. In Figure 5, we plot the expeted values of the various likelihood approximations over a range of values of for three different values of the true parameter, =, 1,. Under both models A (1) and A (), all three approximations have a minimum at = when = and losely trak the exat loglikelihood over the range =.5 to 1. his is not surprising beause when is small, the model is lose to stationary, and the three approximations are nearly the same. When = 1, all three approximations perform well under both models, but our new approximation appears to improve on the existing approximations when beomes large. Finally, when =, our approximation ontinues to trak the exat loglikelihood, while the existing approximations fail. In these examples, when the likelihood approximations are not aurate, usually the error in approximating the quadrati form dominates. Under model A () our new approximation is nearly exat; the maximum absolute differene from the exat loglikelihood is less than.1 for all values of and that we studied, and sometimes muh smaller. We expet our approximations to perform well when the model has very short-range dependene at the beginning and the end of the series, as most of the error assoiated with replaing Σ (A) with (A) ours in the bottom left and upper right orners of the matrix (see the proof of Proposition 1). However, the approximation does worsen with larger values of., 6

7 able 1: Values of that minimize the expeted value of the exat loglikelihood and eah of the expeted loglikelihood approximations when the true value is. urv gives the value of the seond derivative at the minimum, as approximated by finite differenes. A (1) A () =. =.5 = 1. = 1.5 =. min urv min urv min urv min urv min urv E(L()) E(L a ()) E(L b ()) E(L ()) E(L()) E(L a ()) E(L b ()) E(L ()) In able 1 we display the value of that minimizes eah of the expeted likelihood approximations for five values of, as well as the seond derivative of eah expeted approximation at the minimum (approximated by finite differenes). For both models A (1) and A (), the existing approximations begin to fail to as inreases, in that they underestimate and overestimate the urvature at the minimum. Relative to the existing approximations, the new approximation is stable with respet to inreasing, and under model A (), the new approximation is nearly exat although it does worsen slightly as inreases. In this example, these results suggest that the new approximation is more stable than the existing approximations, and the new approximation is robust with respet to a model with a large dynami range on the interior of the time domain, as is the ase with A (). We note that Dahlhaus s () approximation was onstruted for a smoothly-varying A, whereas the examples here have jumps. Nevertheless, it is interesting to note that this approximation performs similarly to the blokwise periodogram approximation. 5 Desription of the data We implement the tehniques introdued in this paper to analyze a set of high-frequeny temperature data. Speifially, we will onsider two sets of hours of temperature data from the Southern Great Plains region of the Atmospheri Radiation Measurement (ARM) program, whih is reorded at regular one-minute intervals. All of the data an be aessed via the web at Figure 6 below plots hours ( = 1) of temperature data from the first two days in Otober 5 reorded at monitoring site EF-3, as well as the first differenes of those data, whih will be the fous of our analysis. We refer to the seond day as the normal day beause the temperature is low in the morning, warms up during the day and ools k down in the evening, and we refer to the first day as the unusual day due to its irregular temperature pattern. he first differenes, if onsidered to be a realization of a Gaussian proess, show a high degree of nonstationarity. o see this more learly, we an ompute and plot loal periodograms of the data (Figure 7). A loal periodogram is simply the periodogram of a window of data around the time of interest. Here we use 6 minute windows. he vertial axis refers to frequeny, and lighter olors on the image refer to higher power. he time series is plotted on top of the loal periodogram. It is evident that not only does the variane of the proesses hange over time, but the shape of the spetrum hanges as well. Dahlhaus (1997) uses a tapered version of a loal periodogram in his approximation of the quadrati form term in the Gaussian likelihood. he loal periodogram seems to be a good diagnosti and exploratory tool for finding nonstationarities, but as 7

8 an estimate of the evolutionary spetrum, it is highly window-dependent, whih is problemati when the spetral properties of the proess undergo sharp hanges in time. Neumann and von Sahs (1997) desribe the preperiodogram, whih is not window-dependent, as a starting point for their evolutionary spetrum estimator. he preperiodogram is the DF of a loal ovariane estimate and is similar to the Wigner-Ville spetrum. Dahlhaus () uses the preperiodogram instead of a loal periodogram in his likelihood approximation. 6 Modeling A simple nonstationary model for the first differene temperature proess is that its evolutionary spetrum is pieewise onstant in time. In this setting, the resaled time interval is partitioned into a number of bloks, and a single funtion of frequeny desribes the transfer funtion within eah blok. his model is a speial ase of the model in (). Indeed, if we have K bloks in the partition, and I k (t/ ) = 1 if t/ is a member of blok k and zero otherwise, then K A(t/, ν) = I k (t/ )µ k (ν) k=1 is pieewise onstant in time. In this ase, I k ontains the information about the partition, and µ k desribes the transfer funtion within blok k. Furthermore, for a 1 vetor u = (u 1,..., u ) the preonditioning transformation is given by [ C (1/A) H u ] = j = K t= k=1 K k=1 I k (t/ ) µ k (ν j ) u te iνjt 1 [I k (t/ )u t ]e iνjt. µ k (ν j ) his an also be omputed with K FFs and does not require O( ) storage. Not only is this model advantageous for the omputation of the quadrati form, there is a simple and easily omputed expression for the first order log determinant approximation and the seond order orretion. Speifially, log(b t+1,j /b tj ) is zero for all t exept at the breakpoints and t =, so we an ignore all other terms in σ (A). herefore, the log determinant approximation requires O( K) omputations rather than O( ) omputations. We will see that the pieewise onstant spetrum assumption is reasonable for some periods, where it seems that the properties of the proess undergo sharp hanges at several time points. For other time periods, however, the properties of the proess appear to hange slowly and ontinuously over a long period of time. In those ases, a transfer funtion that evolves ontinuously in time may be more appropriate. he model in () inludes suh funtions, and we desribe one speial ase in Setion 8. We parametrize the log of the µ k s as trigonometri polynomials, log µ k (ν) = N j= t= kj os(jν) (11) with N = 5. As an alternative, one ould parametrize with transfer funtions that orrespond to autoregressive proesses, whih would provide a way to ondut pieewise AR modeling without having to require independene among bloks, as is assumed in Davis et al. (6). 7 Optimization algorithm he optimization problem involves hoosing the partition and set of kj s to maximize the approximate likelihood. It is helpful to think of the problem as a ontinuous optimization (hoosing the kj s) nested inside of a disrete 8

9 optimization (hoosing the partition). We perform the optimization by searhing the spae of partitions, and for eah partition, we use ontinuous optimization tehniques to hoose the kj s that maximize or approximately maximize the likelihood for that partition. In this paper, we follow the example of Davis et al. (6), who use a geneti algorithm to searh for the breakpoints in pieewise stationary AR models. In their approah, the minimum desription length is the riterion funtion used to selet both the number of bloks in the partition and the breakpoints of the partition. We fix the number of bloks, and the riterion funtion we wish to maximize is the approximate likelihood desribed herein. he anonial form of the geneti algorithm starts with an initial population of individuals, partitions in this ase, and generates a new generation by rossing or mutating individuals from the urrent generation. he rossing operation involves seleting two parent individuals from the urrent generation and ombining their attributes to bear a hild with similar properties to its parents. he mutation operation involves seleting a single parent from the urrent generation and randomly perturbing some of its attributes to bear a hild with similar properties to its parent. In both operations parents are randomly hosen from the urrent population, and parents with high values of the riterion funtion (approximate likelihood in our ase) are more likely to be hosen than parents with low values of the riterion funtion. One the population of the new generation is large enough, that generation beomes the parent population for the next generation. New generations are reated in this way until some stopping riterion is met. Eah implementation of a geneti algorithm must therefore define the rossing and mutation operations. In our ase, we an desribe a partition with K bloks by its K 1 breakpoints. In the rossing operation, the hild is generated by uniformly sampling from the olletion of its parents breakpoints, subjet to the onstraint that there must be a minimum distane between breakpoints, 3 minutes in this appliation. he onstraint ensures that if both parents have a breakpoint at roughly the same loation, at most one of those two breakpoints an appear in the hild s partition. In the mutation operation, the hild is generated by randomly moving one or several of its parent s breakpoints, subjet to the same minimum distane onstraint. Our algorithm has population sizes of partitions, and eah hild, independently of all the other hildren, is generated by a rossing operation with probility.9 and by a mutation operation with probility.1. he initial population of partitions is generated by sampling uniformly over all partitions of the time domain with the fixed number of bloks, subjet to the minimum blok size onstraint desribed earlier. he algorithm is stopped after 1 generations. Of ourse, this is only half of the optimization proedure. We must still hoose the parameters defining the transfer funtions within eah blok to maximize the likelihood. For this parameterization, there is no simple form for obtaining the maximum likelihood parameter estimates, so we use an iterative optimization tehnique suh as onjugate gradient. hese tehniques are quite slow ompared to the time it takes to generate a new generation of partitions. We are helped by the fat that the gradient of the likelihood with respet to the parameters an be omputed fast using tehniques similar to those desribed for omputing the likelihood and by the fat that the Hessian is relatively sparse; the mixed partial derivatives with respet to parameters in different bloks are zero. However, the time required is large enough to prohibit maximizing the likelihood for eah partition in eah generation. o remedy this problem, we proeed as follows. In the initial generation of partitions, we hoose the parameters for eah partition by maximizing the approximate likelihood, and we store the partitions, along with the transfer funtion parameters, in a table. We then use this table to assign parameters to newly generated partitions in future generations. When a new partition is reated, for eah of its bloks, we find the partition in the table that has the most similar blok, take the parameters desribing the transfer funtion for that blok, and assign them to the blok of the urrent partition. his allows us to pik and hoose parameters from several different partitions in the table and assign them to the urrent partition. Every generations, we hoose the parameters again by maximizing the approximate likelihood, and we add the partitions from that generation, along with their parameters, to the table. hus, as we move along in the 9

10 algorithm, the table is filled in more densely, and we are more likely to find partitions in the table with bloks losely mathing the bloks of the urrent partition. 8 Optimization results o hoose an appropriate number of bloks for the partition, we ran our algorithm separately for varying numbers of bloks. For eah number of bloks, we repeated the optimization 1 times and reorded the estimate of A giving the highest approximate likelihood, whih means reording the partition and assoiated parameters. he analysis was repeated on the data from Otober 1 and, 5. In Figures 8 and 9, we show the best partitions from eah of the 1 runs for eah number of bloks. Eah best partition is represented as a row of irles plotted at the maximum loglikelihood for that partition. his allows us to see if the algorithm provided stable estimators of the partition and how muh the likelihood inreased when we added bloks to the partition. For the unusual day, it seems that the algorithm returned a stable estimate of the partition for as many as five bloks, but when we inreased the number of bloks, there tended to be some disagreement among the different runs, highlighting the inherent diffiulty of searhing a partition spae with more than a few bloks. For the normal day, the estimates were not quite as stable, perhaps suggesting that a pieewise onstant in time spetrum is not appropriate. As expeted, we saw large inreases in likelihood between the three- and four-blok models but small differenes in likelihood between the six- and seven-blok models. One may ask whether a model with a separate transfer funtion in eah blok like we have just onsidered gives a signifiantly better fit than a model with a single transfer funtion that is modulated by a different fator in eah blok. Reall that the models for A fit here ontain K bloks orresponding to K different spetra: A(t/, ν) = K I k (t/ )µ k (ν). k=1 We refer to this as the full model. We an ompare likelihoods and fitted values to those obtained from a uniformly modulated model for A, whih we refer to as the multipliative model: A(t/, ν) = K k I k (t/ )µ(ν). his is a speial ase of the uniformly modulated model, in whih the modulating funtion, m(t/ ) = k=1 K k I k (t/ ), k=1 is pieewise onstant in time. o ompare the fits obtained for these two models, we plot the maximum likelihood estimates for A for both models, as well as the ratio of the two estimates of A in Figures 1 and 11. Here we onsider only the best five-blok partition found for the full model. We found that for the unusual day, the full model inreased the loglikelihood by 39. over the uniformly modulated model, and for the normal day, we ahieved an inrease of 88.5 loglikelihood units. As disussed earlier, for the normal day, the properties of the proess appear to hange slowly over several hours, rather than undergo sharp jumps. Perhaps a model that allows the spetrum to hange ontinuously in time may be more appropriate. Consider the following model: A(t/, ν) = K I k (t/ ) exp [α k (t t k )/ ] µ k (ν), (1) k=1 where t k / is the time at whih blok k starts. his is again a speial ase of the model in (). Our maximum likelihood estimate of A under the model in (1) with K = 5 bloks inreased the loglikelihood by 7.37 over the full model. he estimate is plotted in Figure 1. 1

11 9 Numerial study of approximations he time series onsidered here are intentionally hosen to be short enough that an exat Gaussian likelihood an be omputed but long enough that our approximation gives a signifiant speed-up in omputation. It would be too time-onsuming to ompute and maximize exat Gaussian likelihoods within the geneti algorithm, but after the partition is hosen and fixed, maximizing the exat Gaussian likelihood one with respet to the transfer funtion parameters an be ompleted within a few minutes on a personal omputer. he exat negative loglikelihood is (minus a onstant / log(π)) L() = 1 log det Σ (A ) + 1 Z Σ (A ) Z. he details of the omputation of the exat likelihood under the model desribed by (1) are straightforward and left to the Appendix A.. Again, as in Setion, here are the approximations we onsider: L a () = log det (A, ) + 1 π L b () = log det (A, ) + 1 π K J k (ν) B k µ k (ν) dν, I (t/, ν) A (t/, ν) dν, k=1 t=1 L () = log det (A,.5) + 1 C (A ) Z, = arg min L(), a = arg min L a (), b = arg min L b (), = arg min L (). We take the partitions of the time interval to be known, so in the following, is the parameter vetor of oeffiients kj in (11). We evaluated our approximations in two ways. he first is how well we approximated the exat log determinant and quadrati form at for eah number of bloks. We ompared our seond order log determinant approximation to the first order log determinant approximation, whih is essentially what Dahlhaus (1997, ) proposes, only the integral is replaed with a sum. We also ompared our quadrati form approximation to that proposed in Dahlhaus (), whih is sed on the preperiodogram, and that proposed in Dahlhaus (1997), whih in this appliation we take to be a sum of integrals of ordinary blokwise periodograms. he seond evaluation we onsidered is how lose the parameters hosen by maximizing the approximate likelihoods ame to maximizing the exat likelihood. Speifially, we omputed L( ) L( approx ) for eah of the three approximations. his evaluates eah approximation sed on its ability to return an estimate that nearly maximizes the exat likelihood. Figures 13 and 1 show that for both days, while the two approximations of the log determinant are equivalent in the stationary (one blok) ase, the seond order approximation always gave an improvement over the first order approximation when there was more than one blok in the partition. Furthermore, the seond order approximation appears to retain a roughly equal level of auray regardless of the number of bloks, whih is as good of a result as one ould hope for with this approximation. Speifially, we would not expet the approximation to be more aurate in the nonstationary ase than it is in the stationary ase. It appears in this example that our quadrati form approximation is more stable than those proposed by Dahlhaus. For the normal day the preperiodogram and blokwise periodogram estimates are inaurate in the models with five or more bloks. In these ases, the issue is that the preperiodogram and the blokwise periodogram estimates find some low frequeny information in the blok that enompasses data roughly between hours 15 and 16, whih is a similar situation to that enountered in the numerial experiment in Setion. apering the observations within eah blok helps in these ases, but our approximation is still superior in the irumstanes we have studied. he third plot in Figures 13 and 1 shows that for these data, the error in the likelihood approximation is dominated by the quadrati form. Figure 15 shows that in all of our examples, maximizing our approximate likelihood always ame loser to maximizing the exat likelihood, although the Dahlhaus (1997) approximation performs well on the unusual day. 11

12 For the normal day, we do not inlude the estimates for more than four bloks when using Dahlhaus s likelihoods beause the quadrati form problem produed unreasonable estimates of A. 1 Conluding remarks We have provided a numerial example and an appliation with temperature data that suggest that our Gaussian likelihood approximations are sharper and more stable than those proposed by Dahlhaus. Although we have no theorems proving that our approximations are uniformly more aurate, they have performed better in the speifi examples we have studied. If the evolutionary transfer funtion an be written in the form in (), the approximations an be omputed effiiently. Otherwise, the approximations partiularly the quadrati form are omputationally intensive for long time series. However, the model in () is easily adaptable to diverse appliations by adjusting the form of the omponent transfer funtions, µ k, and their respetive modulating funtions, m k. he log determinant approximation may require alterations to failitate fast omputation in ases when the spetrum is not pieewise onstant in time. For example, when we approximate the log determinant term for the model in (1), we an ignore all the terms in σ (A) exept those at the breakpoints and end of the series. his should provide reasonable approximations when the spetrum does not vary too quikly in time between the breakpoints. he quadrati form term does not require suh an adjustment. However, more effort may be required to obtain a suitable preonditioner for the iterative algorithm, whih should not affet the auray of the quadrati form (as long as the iterative solver onverges) but will affet the speed of onvergene. Aknowledgements his work was initially supported by the United States Department of Energy, Offie of Siene, Offie of Biologial and Environmental Researh, Climate Change Researh Division, under ontrat DE-AC-6CH11357, as a part of the SiDAC program, and reeived subsequent support from US Department of Energy Grant DE-SC557. Appendix A.1 In this setion, we prove Proposition 1: Proposition 1: If, for eah u, A(u, ν) is a omplex-valued twie ontinuously differentiable funtion on with respet to ν, and the seond partial derivative of A(u, ν) with respet to ν is uniformly bounded in u and ν, then Σ (A) (A) F = O(1) as. Proof: We define f u,v (ν) = A(u, ν)a (v, ν). Beause A(u, ν) is twie ontinuously differentiable with respet to ν for eah u, f u,v is also twie ontinuously differentiable on for every u and v. herefore, the Fourier series of f u,v is uniformly onvergent (Körner, 1988, heorem 9.6), so we an write f u,v (ν) = k= u,v (k)e ikν. Furthermore, the Fourier oeffiients may be bounded by (Körner, 1988, Lemma 9.5) u,v (k) M 1 k, 1

13 where M 1 = sup u,v,ν / ν f u,v (ν). We also have f u,v (ν)dν = u,v () < beause f u,v is uniformly bounded on. Using the Fourier series representation, we may write the elementwise error as D (s, t) K (s, t) = π π u,v (k)e ikνj e iνj(s t) j= k= π = π u,v (k)e ikνj e iνj(s t) k= j= k= = π u,v ((t s) + l ) π u,v (t s) l= = π u,v (t s + l ) l =1 πm 1 t s + l, l =1 k= u,v (k)e ikν e iν(s t) dν u,v (k)e ikν e iν(s t) dν π where the order of summation and integration an be swithed beause of the absolute summability of the Fourier oeffiients. Now we ompute the square of the Frobenius norm. (A) Σ (A) F = t s k k= h=1 l =1 π πm 1 t s + l l =1 πm 1 k + l Sine k is between and, the largest term in the sum is the one with k in the denominator, so we rewrite 13

14 the sum as 1 (A) Σ (A) F = k k= h=1 k k= h=1 = k k=1 h=1 = k k=1 h=1 k k=1 h=1 k=1 j=1 ( πm 1 k + l=1 ( πm 1 k + l=1 ( πm 1 k + πm 1 k + l + πm 1 l l=1 ) ( πm1 k + π3 M 1 3 πm 1 l ( π M1 k + π6 M1 ) 9 ( 8π M 1 k 3 ( 8π M1 = j 3 ) + 8π6 M1 9 ) + 8π6 M1 9 8π M1 1 j 3 + 8π6 M1 9 j=1 ) = M1 (8π ζ(3) + 8π6 9 <. + l= ) l= πm 1 k + l πm 1 l ) ) he funtion ζ is the Riemann-Euler zeta funtion, and ζ(3) 1.. A. For the log determinant approximation (1) desribed in Setion 3, a value for γ of.5 has some theoretial justifiation. Let B (A) = {b tj } be as desribed in Setion 3 and P = {p tj } be the matrix whih performs the inverse DF (saled by / ), so that C (A) = πp B (A), where is the element-wise produt. Consider the speial ase in whih all entries of B (A) are 1 exept for those in the k th row (i.e. t = k) so that C (A) = π(p + e k v H ), where v is the vetor with j th element (b kj 1) p kj and e k is the unit vetor along the k th oordinate. Proposition : If b kj = e ɛhj, then with γ =.5, log det C (A)/ π = 1 log{b tj } + γσ (A) + O(ɛ 3 ) j,t as ɛ. Proof: By the matrix determinant lemma (Harville, 1997), det(c (A)/ π) = det(p )(1 + v H P e k). Beause P is a unitary matrix, its determinant is ±1, and its inverse is its onjugate tranpose. herefore, ( det C (A)/ ) π = 1 + v H P e k = (b kj 1). j=1 1

15 Now onsider what happens as ɛ. Expanding 1 j=1 b kj in a aylor series through order ɛ and writing h for 1 j=1 h j, we have ( det C (A)/ ) π = 1 + ɛ h + ɛ h j + O(ɛ 3 ). aking the log gives ( log det C (A)/ ) ɛ π = ɛ h + = ɛ h + ɛ j=1 h j 1 ɛ h + O(ɛ 3 ) j=1 (h j h) + O(ɛ 3 ) We have 1 j,k=1 log{b kj} = ɛ h, so that the simple approximation (7) aptures the O(ɛ) term in the log determinant. Furthermore, straightforward alulations yield σ (A) = ɛ j=1 (h j h), so that (1) has error O(ɛ 3 ) when γ =.5. j=1 A.3 In Setion 3, we onsidered the deomposition A(u, ν) = m(u)µ(ν)r(u, ν), where m, µ, and R are all positivevalued and bounded away from. In this appendix we explore the asymptotis of σ (A) in eah of two ases: Case 1: (I) For every u, R is absolutely ontinuous in ν with a uniformly bounded almost everywhere derivative. (II) Exept for possibly some finite number of values of u (not depending on ν), for every ν, R is absolutely ontinuous in u with a uniformly bounded almost everywhere derivative. At these exeptional u values, R may not be ontinuous in u, but for definiteness, we will assume R is right ontinuous in u for all ν. Case : (III) R is absolutely ontinuous in both u and ν with uniformly bounded almost everywhere derivative, and R(, ν) = R(1, ν) for all ν. Proposition 3: Under the onditions of Case 1, σ (A) = O(1), and under the onditions of Case, σ (A) = O( ). Proof: Reall that σ (A) = 1 { t=1 j=1 log b t+1,j b tj 1 } log b t+1,k. b tk k= Writing u t = t/ and using the deomposition of A and R(, ν) = R(1, ν) for all ν, it is easily shown that σ (A) = 1 { t= j= log R(u t+1, ν j ) R(u t, ν j ) 1 } log R(u t+1, ν k ). (13) R(u t, ν k ) Expanding log{r(u t+1, ν j )/R(u t, ν j )} in a aylor series gives ( log R(u t+1, ν j ) = R(u {R(ut+1 } ) t+1, ν j ) R(u t, ν j ), ν j ) R(u t, ν j ) + O. R(u t, ν j ) R(u t, ν j ) R(u t, ν j ) k= 15

16 Under the onditions of Case, using the uniformly bounded derivative ondition, and the fat that R is bounded away from, we know that R(u t+1, ν j ) R(u t, ν j ) = O( ). R(u t, ν j ) herefore, in (13) σ (A) is multiplied by the sum of terms, eah of whih is O( ), so σ (A) = O( ). Under the onditions of Case 1, R has finitely many disontinuities in u. herefore, { R(u t+1, ν j ) R(u t, ν j ) O( ) no disontinuities between u t and u t+1 = R(u t, ν j ) O(1) at least one disontinuity between u t and u t+1. In (13), there are finitely many t for whih the sum over j is O( ), so σ (A) = O(1). A. In order to ompute the exat Gaussian likelihood, we must onstrut the exat ovariane matrix, whih results from the ovariane funtion K (s, t) = A(s/, ν)a(t/, ν) e iν(s t) dν. Here, we ompute this integral numerially with the sum π J J A(s/, ν j )A(t/, ν j ) e iνj(s t), j= where ν j = πj/j, and J is a large integer (J = gives adequate auray in our examples). he omputation is made effiient with the FF, and the ovariane matrix is filled quikly by taking advantage of its blok oeplitz struture under the pieewise onstant spetrum model. After the ovariane matrix is onstruted, we ompute the Cholesky deomposition U. hen the log determinant term is simply the sum of the log of the diagonal entries of U, and the quadrati form term is Z Σ (A) Z = U Z. Finally, we ompute U Z with forward substitution. Referenes Adak, S. (1998) ime-dependent Spetral Analysis of Nonstationary ime Series. Journal of the Amerian Statistial Assoiation 93, Dahlhaus, R. (1996) On the Kullk-Leibler information divergene of loally stationary proesses. Stohasti Proesses and their Appliations 6, Dahlhaus, R. (1997) Fitting time series models to nonstationary proesses. he Annals of Statistis 5, Dahlhaus, R. () A likelihood approximation for loally stationary proesses. he Annals of Statistis 8, Davis, R. A., Lee,. C. M., Rodriguez-Yam, G. A. (6) Strutural Break Estimation for Nonstationary ime Series Models. Journal of the Amerian Statistial Assoiation 11, Donoho, D., Mallat, S., and von Sahs, R. (1998) Estimating Covarianes of Loally Stationary Proesses: Rates of Convergene of Best Basis Methods, ehnial Report 517, Stanford University, Dept. of Statistis. Guo, W., Dai, M., Omo, H. C., von Sahs, R. (3) Smoothing Spline ANOVA for ime-dependent Spetral Analysis. Journal of the Amerian Statistial Assoiation 98, Harville, D. A. (1997) Matrix Algebra From a Statistiian s Perspetive. Springer-Verlag, New York. Horn, R.A. and Johnson, C.R. (6) Matrix Analysis. Cambridge University Press, Cambridge. Körner,. W. (1988) Fourier Analysis. Cambridge University Press, Cambridge. Neumann, M. H., von Sahs, R. (1997) Wavelet thresholding in anisotropi funtion lasses and appliation to 16

17 adaptive estimation of evolutionary spetra. he Annals of Statistis 5, Omo, H. C., Raz, J. A., von Sahs, R., Malow, B. A. (1) Automati Statistial Analysis of Bivariate Nonstationary ime Series. Journal of the Amerian Statistial Assoiation 96, Omo, H. C., Raz, J., von Sahs, R., Guo, W. () he SLEX model of a non-stationary random proess. Annals of the Institute for Statistial Mathematis 5, 171. Priestley, M. B. (1965) Evolutionary spetra and non-stationary proesses. Journal of the Royal Statistial Soiety: Series B 7, 37. Priestley, M. B. (1981) Spetral Analysis and ime Series. Aademi Press, London. Whittle, P. (196) Gaussian estimation in stationary time series. Bulletin of the International Statistial Institute 39,

18 Figure Captions: Error Figure 1: Absolute approximation errors of log determinants for A 1 (u, ν) = exp{π ν(u u )} and A (u, ν) = exp{π ν sin(8.5πu)}. Symbol j orresponds to A j, blak to approximation (7) and gray to approximation (1) with γ =.5. 18

19 Error Figure : Absolute approximation errors of log determinants for A 3 (u, ν) = exp{1π ν(1 u.5 )} and A (u, ν) = exp{π ν sin(1πu)} + os (π ν sin(16πu)). Symbol j orresponds to A j, blak to approximation (7) and gray to approximation (1) with γ =.5. Error/Estimate Figure 3: Ratio of the error of the first order approximation to.5σ (A j ) for the four funtions onsidered in Figures 1 and. 19

20 .5 Covariane funtion orresponding to A 1 (1) K (s n,t) t Figure : Covariane funtion orresponding to A (1) with = 1. We plot K (s n, t) for s 1 = 5 (dots), s = 6 (+), s 3 = 61 (irles), and s = 7 ( ). he vertial line indiates the breakpoint in the proess. E(L approx ()) E(L approx ()) 1 b b a a a 11 b a 1 b a b 9 a a a a a a a a b b b b b b b 8 7a a a 6 a a a a a a a a a b b b b b b b b b b b b a a a a a a a a a a a a a b b b b b b b b b b b b b A (1) a b a b 11 a a b b 1 a a b 9 a a a a a a a b b b 8 b b a 7a a a 6 a a a a a a a a a b b b b b b b b b b b b b b b b b a a a a a a a a a a a a a b b b b b b b b b b b b b A () Figure 5: Expeted values of the negative loglikelihood approximations L a (), L b (), and L (), with true parameter = (blak), = 1 (blue), and = (magenta). he solid line indiates the expeted value of the exat loglikelihood L().

21 5 "Unusual" day temperature 3 "Normal" day temperature emp (Celsius) First differene of temperature.3 First differene of temperature emp (Celsius) ime (hours) ime (hours) Figure 6: emperature data π Frequeny π Frequeny "Unusual" day loal periodogram "Normal" day loal periodogram ime (hours) Figure 7: Loal periodograms with 6 minute windows for both days 1

22 "Unusual" day optimization results Differene of log likelihood ime (Hours) Figure 8: Optimization results for unusual day. Eah row of irles represents the best partition found with the geneti algorithm. "Normal" day optimization results Differene of log likelihood ime (Hours) Figure 9: Optimization results for normal day. Eah row of irles represents the best partition found with the geneti algorithm.

23 π Frequeny π Frequeny π Frequeny Multipliative model, log likelihood = Full model, log likelihood = Residual plot (ratio of spetra), full versus additive model, likelihood diff = ime (hours) Figure 1: Maximum likelihood estimate of the time-varying transfer funtion of the unusual day under the multipliative model (top), the full model (middle), and the ratio of the full model estimate to the multipliative model estimate (bottom). 3

24 Frequeny Frequeny Frequeny Multipliative model, log likelihood = π Full model, log likelihood = π Residual plot (ratio of spetra), full versus additive model, likelihood diff = 88.5 π ime (hours) Figure 11: Maximum likelihood estimate of the time-varying transfer funtion of the normal day under the multipliative model (top), the full model (middle), and the ratio of the full model estimate to the multipliative model estimate (bottom). π Log linear model, log likelihood = Frequeny ime (hours).13 Figure 1: Maximum approximate likelihood estimate of the model in (1) for the normal day.