EXTRAPOLATING TIME SERIES BY DISCOUNTED LEAST SQUARES by R. J. Duffin Report November, 1966

Transcription

1 EXTRAPOLATING TIME SERIES BY DISCOUNTED LEAST SQUARES by R. J. Duffin Report 66- Noveber, 966 University Libraries Carnegie Mellon University Pittsburgh PA

2 Extrapolating Tie Series by Discounted Least Squares Abstract An approxiating function is fitted to a tie series, such as daily observation. The fitting is carried out over all past tie by weighted least squares with an exponential weight factor The approxiating function is restricted to be a solution of a certain linear differential equation of the th order having constant coefficients. The solution which iniizes the least square expression can be continued into the future. In particular toorrow s extrapolated value is defined by this continuation. To obtain an explicit solution of the proble a forula is constructed which gives the extrapolated value as a linear cobination of the last observed values and the last extrapolated values. The coefficients of this extrapolation forula prove to be siply related to the coefficients of the differential equation. Another extrapolation forula is of vectorial nature. The coponents of a vector are independent functionals of the past observations. Then toorrow's vector is given as a linear function of today s vector and today s scalar observation. 3 CM i. HURT LIBRARY CARNEGIE-MELLON UNIVERSITY

3 EXTRAPOLATING TIME SERIES BY DISCOUNTED LEAST SQUARES* R. J. Duffin CARNEGIE INSTITUTE OF TECHNOLOGY This paper is concerned with extrapolation of an infinite sequence y.^y^.., of real (or coplex) nubers. This is accoplished by fitting the sequence (y n ) by a function p(n) taken fro a space of functions tered exponoials. for the fit is given by discounted least squares. The criteria This eans that p(n) is that exponoial which iniizes the 'error' expression E = I e n y n - p(n) 2. Here 6 is a positive constant tered the discount factor. Then the extrapolated value of the sequences at the point x is defined to be p(x). In the previous paper an exponoial was defined as an exponential polynoial of the for p(x) = I where the f>. are fixed coplex nubers assued distinct and nonzero. The coefficients d. are arbitrary coplex nubers so an -diensional vector space results. In this paper the definition of exponoial is extended so as to perit polynoial ters of the for p>*,x &*,..j,x ~ A*. It is seen, therefore, that the space of exponoials can be defined to be the solution set of a certain linear differential equation, jin jin- fl_b + e d = dx " dx" " ^Prepared under Research Grant DA-ARO-D-3-24-G68O, Ary Research Office (Durha).

4 3 where the coefficients e. are coplex constants. This space of exponoials will be denoted as XP. An iportant special case is defined by the equation d p/dx = 0. Then XP is the space of polynoials of degree less than. There are three reason why the extrapolation procedure just described can treat a large class of probles in applied atheatics. The first reason is that the discounted least squares criteria is suited to probles of echanics and econoics for which the progressive discount of the past sees natural. Another reason is that the space XP is invariant under an arbitrary translation of the x axis. This invariance property akes exponoials attractive functions for approxiating tie series. The third reason for the utility of this extraipolation ethod is that there is an underlying algebraic structure which is both interesting and significant. The bases. are arbitrary. The selection of the. and the discount factor 9 should take into account first the genesis of the data. Also, account ust be taken of the genesis of data error and the soothing property of the extrapolation. These questions are not treated in this paper. A central proble of this paper is the one step extrapolation of the sequence { y } to obtain an extrapolated value at n = 0. The extrapolated value is denoted as y* and is defined as y = p(0) When the iniization is carried out 3 it results that y* is given by a linear functional v* = 2 Q v. *o ^-. w n y n Here the coefficients Q do not depend on the sequence

5 y y... It is then natural to write and define y* as the predicted value of y, based only on the! previous values Yfc+i'Yk+2' # ' ## Theore 3b to follow gives a siple generating function to evaluate the coefficients Q Thereby the proble of one-step extrapolation is essentially solved. Nevertheless, the above forula is not satisfactory fro a coputational point of view. This is so because it is an infinite series and so an infinite eory is needed. This situation is reedied by Theore 2 which provides the following short eory forula, * = x T n y n+k + I x <f> n y +k - Here the T and the (b are constants which do not depend n. T n on the sequence y. Thus, this identity gives the extrapolation y* as a linear cobination of the previous values and the previous extrapolations. Consequently, the short eory forula is readily adaptible for coputer evaluation. Also of iportance are extrapolation forulas for other linear functionals of the sequence y-,y o,... For exaple c 2 p(l/2),p f (0),p ff (-2), j p(x)dx are linear functionals. We o ** ter such functions extrapolat o r s. Let w(x) be a vector whose coponents are linearly independent extrapolators. Then Theore 7 gives the following extrapolation forula w(x) = Aw(x + ) + by x+. where A is constant atrix and b is a constant vector. This is tered a very short eory forula because the extrapolation

6 5 is based only on one previous value of w and one previous value of y. The extrapolation of tie series based on a discounted least squares criteria has previously been treated by Duffin and Schidt [], Duffin and Whidden [2], and Morrison [3]. (Various other authors have proposed siilar extrapolation forulae but their work is not based on discounted least squares.) The present paper gives a ore general treatent of probles posed by references [], [2], and [3]. In particular the theores of this paper are aied at evaluating and interrelating the constants, p.,, Q, T, * A, and b by algebraic forulae. To begin the proof, let G(z) be a polynoial of degree in the variable z and let G (0) =. Then G(z) = 2L g,z j. j=0 3 Here the g. are arbitrary coplex constants except that g == and g ^ 0. Of central concern are functions p(x) of the real variable x satisfying the difference equation o = g Q p(x) + g^p(x + ].) + + g p(x + ). Let X denote the translation operation defined by the relation Xf(x) = f(x + ) # Then the difference equation can be written in operation for as 0 = G(X)p(x) = ir (l-^-.x)p(x) where the ^. are the roots of G(z) = 0. First suppose that x is restricted to the integers. If none of the roots are repeated it is seen that there are linearly independent

7 X solutions having the for p (x) = ^.. The general solution is a linear cobination of such solutions. If the root p. is repeated k ties there are k linearly independent solutions: x x k x B,',x6.,...,x B.. The general solution is again a linear 3 v 3. 3 cobination of such solutions and defines an -diensional vector space. This is the space of exponoials and it is denoted as XP. For soe probles of extrapolation it is required to continue exponoials to non-integral values. One way to do this is to let a B,,,, ^ r, x Bx 27riLx,. _ p = e and then define P> = e e where L is an integer. A f natural T choice for L is -IT < iag B + 2TTL <.TT. This choice iniizes oscillation. The abiguity here stes fro the fact that ultiplying a solution of the equation by an arbitrary function of period gives another solution. Since an exponoial satisfies the difference equation it is seen that prescribing the value of the exponoial at the integers l,2,..., deterines the value of the exponoial for all other positive integers. Thus the space XP is -diensional even if x is restricted to the integers l,2,...,. The approxiation schee eployed in this paper results fro ebedding the finite diensional space XP in an infinite Hilbert space of r discounted squares. This Hilbert space is denoted as DS. The eleents of DS are infinite sequences of coplex nubers y.^y^.., such that 2L y < CD. Here^ is a positive constant tered the discount factor. The discount factor is required to satisfy the inequality

8 0 _. < ; j =,2,...,. It is readily shown fro this inequality that exponoials are in DS. Thus the space XP is a finite diensional subspace of DS. Let v and y denote two eleents of DS then the Heritian bilinear for is defined as po [v,y] = X 0n Vn where v denotes the coplex conjugate of v. The nor of a eleent y of DS is defined as It is instructive in what follows to regard the sequence {y } as a tie series of observations'. Thus y can be regarded as the value of the observation at tie -x. To extrapolate the sequence {y } for values of n not a positive integer we first approxiate y by an eleent p of the subspace XP. By this is eant that p is chosen to iniize the expression GO E = (y - pll 2 = " e n y n - p(n) 2. As in the references [], [2], and [3] this is tered approxiation by discounted least squares. The following lea is aied at deterining p given a sequence (y }. Lea. Let p be the exponoial best approxiating y in the Hilbert space. Then for every exponoial r. ' [r,y] = [r,p]" Proof. This is erely a reforulation of the basic theore concerning Hilbert space which states that if p is the best approxiation

9 8 to y for p constrained to a subspace then p is the orthogonal projection of y into the subspace. By choosing a basis for the subspace XP the orthogonality relation stated by this lea leads to a syste of linear equations which could be used to deterine p as a unique linear cobination of the basis eleents. Then p is deterined uniquely at the positive integer points. The convention introduced above perits an exponoial, given at the positive integer points, to be deterined for all real x. Thus it is possible to define the extrapolation of the sequence y^y^**** at the point x to be p(x). Lea 2. Let r be a given exponoial and let k be a given nuber. Then there is a unique sequence of nubers c.c o...,.c r 2 ; such that the forula [r,v] = ^vfk + ) + c 2 v(k + 2)+-*-+ c v(k + ) holds for every exponoial v. Conversely given a nuber k and a sequence of nubers c l* C 2 3 ' *'' C there is a unique exponoial r for which the above forula holds» Proof. As is well known the linear functionals f(v) in an -diensional space such as XP have the for f(v) = [r,v]. Moreover, these linear functionals theselves for an -diensional space. Clearly the expression on the right side of the forula of the lea is a linear functional for any choice of constants c-,c ~,-...,c. Suppose that for soe choice c.v(k + ) +... c v (k + ) l z vanishes for all v in XP. However, an exponoial can be prescribed arbitrarily at a sequence of points obtained by successive unit translations to the right. Hence all the constants

10 c* ust be zero. This shows that v (k + ), v (k + 2),...,v (k. + ) are independent functionals. There can be no ore than independent functionals and consequently the left and right side of the forula represent the sae space of linear functionals. The proof of the lea follows fro this observation. Theore. Let r be a given exponoial and let k be a given nuber, then there is a unique sequence of nubers p^c.. #,c such that the forula [r,y] = Cjpflc + ) + c 2 p(k + 2) + «+ c p(k + ) holds for every y of the Hilbert space DS provided that p is the orthogonal projection of y into the exponoial subspace XP. Conversely given a sequence of nubers c n.c n^...c there =a ± y z is a unique exponoial r satisfying this forula. Proof. By Lea we have [r,y] = [r,p]. Then apply Lea 2 with v = p and the proof is seen to be coplete. A general proble of extrapolation is to extrapolate the sequence y.,y~,... to obtain a value for y at a point x 9 not a positive integer. This extrapolated value is denoted by the sybol ext(y ) and is defined as ext(y ) = p(x). The x x following corollary of Theore gives a forula for coputing ext(y x ).. Corollary. Let p(x) be the approxiating exponoial to the sequence y^y^*- - Then there is a kernel function q(x,n) p(x) = Ie n q(x,n)y such that CD n = [g,y]. For n fixed q(x,n) is an exponoial in x. For x fixed

11 0 q(x^n) is an exponoial in n. The kernel function ay depend on 0 but not on y. Proof. In Theore take k=x-l, c,=l, c 2 =0,...,c =0. This proves the forula of Corollary with q = r. To show that the kernel function is an exponoial in x take y = 0 except for n = n. This is seen to coplete the proof. A significant special case of the forula of Corollary is x = 0. This extrapolates the sequence y-i syj? one unit to the left to obtain ext(y ). This one step extrapolation is sufficiently iportant to warrant a special notation and we write y* = p(0). The general extrapolation forula becoes co Here q(n) = q(0,n) and q(n) is also tered a kernel. If the sequence y^y^... i s i n the Hilbert space DS then the sequences y +*Y + 2 > * s a^so ^n DS f r an Y positive or negative integer x. This follows fro the relation 00 OD As a natural extension of the previous notation let y* denote the extrapolation of the sequence y +^y.o^*»«to the point x. The extrapolation forula given by discounted least squares is oo ^x ~ LL g * y n+x* This forula involves all values of the sequence y.-j^y, 9 J and so is an infinite series. For this reason a forula of this nature ay be tered a long eory forula. It is now to be shown that y* is also given by a finite recursion relation and hence by a short eory forula.

12 Theore 2. Let yf be defined by the forula CD y* = n=l Then y * the one step extrapolation, is given by the recursion forula n=l n=l Here, for convenience of notation, f = g /g. Also 6 = y* - y and is tered the discrepancy. Proof. Let q =0 for n <. 0 and q< = q for n > 0 so y* = Z. n q'(nw +. -CD 00 -CD Since q(x) is a polynoial _ G(X)q(x) = 0.. Thus <*L g.q(x + j) = 0 o D or Z Q g.q(x + - j) = 0. A polynoial F(z) related to the polynoial G(z) plays an iportant role in what follows. It is defined as Q ^ o It is seen that F(X)q(n - x) = 0 for all n. Consequently F(X)q! (n - x) =0 for n > x + or n < x so CD F(X)0 X y* = ^ [ H f qi (n - x - j) ] % x n n=-oo j=0 D x+ n=x+l S y nn where the s^ are certain absolute constants. To evaluate these constants,, first set x = 0 in the last relation and obtain

13 Next let y, = r(k) an exponoial. Since the extrapolation of exponoials is error-free, y* = r(k) also. Note that f Q = so after substituting y. = r(k) the relation (*) can be written as r(0) = Z (s. - f.9 j )r(j). j=l 3 3 But since G(X)r(x) =0 it is also true that r(0) = - 2 Subtracting these two equations for 0 = Z (s. - f.g j + g.)r(j). 4 j J J r(0) gives But an exponoial can be defined arbitrarily on the integers,2,...,. Thus s. = f.-0-' - g, and relation (* ) becoes (f.e D - g.)y. - This is seen to be equivalent to the short eory forula stated in Theore 2. Corollary 2. The kernel function q(x) satisfies the constant coefficient difference equation G(X) q(x) = 0, an equation pf the th order. But q(x) does not satisfy any such equation of lower order. Proof. Suppose q(x) satisfied the equation G ft (X)q(x) =0 of order!t <. Then the proof of Theore 2 could be carried out with G T! replacing G. This would lead to a relation of the for f r(0) = H (fit» - ft <-9 j )r(j) j=l ^ 3 for every exponoial r(j). But an exponoial can be defined

14 3 arbitrarily at successive points. Thus take r(0) = but r(j) = 0 for j = l,..., tf. This contradicts the assued relation and the proof is coplete. So far the kernel function only been defined iplicitly. q(n) of the long forula has On the other hand the coefficients of the short forula are given explicitly in Theore 2. However^ since the short eory forula and the long eory forula are essentially equivalent it is possible to use the short forula to give an explicit procedure for evaluating q(n). Different ways of doing this are given in Theore 3a and Theore 3b to follow. Theore 3a. The kernel function q(n) satisfies the recursion forulae: n- q(n) = f - 6 n g - H f^q(n - j), < n < n n j== D q(n) =-Z. f.q(n - j) j=l 3. Thus q(2) = f 2 - q(3) = f 3-3 x 2 \ ^ ^ 2 2 l Proof. Let y. = 0 for all j except that y =. Then it 3 n follows directly fro the long forula that y* = G n q(n). It is also seen that: y* = 0 n ""-'q(n-j) if j < n -, y* = 0 if j >^ n. The short forula ay be written in the for Substituting the above special values of y. and y* in this forula leads directly to the forulae of Theore 3a. HUNT LIBRARY CARNEGIE-MELLOH UNIVERSITY

15 Theore 3b. The kernel function q(n) satisfies the generating 4 identity 'F(QZ) - < GO provided z is sall. Proof. 2 If z is sall the sequence l,z,z,... is in the Hilbert space DS. Thus, substituting y = z in the long forula gives oo y == /L q(n) z = z y*.. Then substituting y = z and, Yjt = z y* in the short forula ' y* = ^ (e^f. - gjz * - Ze-'f.z-'y^, j=l ^ 3 j=l ^ But f = g = so o o y"^ ^ f.0 z = 2> f.6 z - Z^. g z j=o D j=o D j=o j and the proof is coplete. We now turn fro one-step extrapolation to ulti-step extrapolation. Thus the two-step extrapolation of the sequence y-\ syy* i s given by ext (y.) = p(-l) etc. Theore 4a. Multistep extrapolators of the sequence Y-j^Ypj... are given by the long eory forulae: oo P(O) = "X ell q n Y n^ l oo CO

16 5 p(-3) = Here q is a condensed notation for the kernel q(n). - n Proof. Let Y = y for n > 0 and let Y = p(n) for n < 0. oo n n n Let E = T 0 n Y - P (n) where P (n) is an exponoial. Thus o oo E o = P (0).P(O) 2 + e n y n. P(n) 2. Clearly E is iniized when P(n) = p(n). Hence applying the one-step extrapolation forula to the sequence oo P(-D.- Vn-l Y,Y,,Y~,... gives But P so ' ' co (O) = Z «n CD OO p{.i, = I e n+ gi%y n + I e n+l %+y n. This proves the forula for the two-step extrapolator.,co Next consider E = > 6 Y - P(n)I and following siilar - /--^ n reasoning to that given above shows that oo p(-2) = L 6 n q n V 2, <*> P(-2) = eq lp (-i) + e 2 q 2 p(o) + 3 Substituting the series expression just derived for p(-l) is seen to prove the forula given for the three-step extrapolation. Further forulae are derived analogously and the proof is coplete.

17 Theore 4b. Multistep extrapolators of the sequence y 3y 3... are given by the short eory forulae; -p(0) = [ gy + I e n f 6 > L 'vi -I t 6 where g =0 and f = 0 JJ[ n >. Proof: Apply the short eory forula to the sequence Y,Y,,... which was introduced in the proof of Theore 4a. But Y* = p(-l) ' so * L~ ^n n- Also Y = p(0), /^ =0 and so -p(-d = g^o) + g y + ^ e n f 6. X il"-j» c ~~ u X PC-) = -g( I g n y n + I e n f n 6 n ) i l l - ^- G f n+l 6 n - This seen to prove the stated forulae for p(-l). Further forulae are derived analogously. Theore 5. The relation F( z) CD n ~ ^ e t(n)z

18 gives a one to one linear correspondence between polynoials of the for \ (z) = ^ a.z-' and conjugate exponoials t(n). Proof. Given such a polynoial L (z) then T(z) = H (z)/f( )z) 7 is a convergent power series if z is sufficiently sall. Thus write GD T(z) = ^ n t(n)z n -CD where t(n) =0 for n 0. Since %(z) = F(0z)T(z) we have oo Let j + n = k so Since o oo -oo $(z) = e k z k f.t(k - j) -oo o *C(z) is a polynoial of degree not exceeding it follows fro the above relation that f.t(k - j) = 0 for k >, or o o o f.t(x + - j) = 0 for x > 0, so f -it(x + j) =0 for x > 0. -j Hence G(X)t(x) = 0 and it follows that t(x) is an exponoial for x > 0. We can conclude fro this last result that the -diensional vector space p of polynoials of the for 't(z) is apped linearly into a space S of conjugate exponoials. Moreover, this is a one to one apping because power series are unique so S has

19 ' diension. However, the space XP of all conjugate exponoials has diension so S = XP and the proof is coplete.. Lea 3. Let (z) = S a.z-' be a given polynoial. Then there are polynoials H(z) = 2~ h j z and K * z ) such that 'C(z) = -H(z)G(z) + K(z)F(0z) Hence the syste of 2 equations i D 8 a i A ± = - L. 9i^j h j + - e f i_j k '* X - i - 2 can be used to find the coefficients h. and k.. Here Aj = D 3 + for i < and A. = 0 for i >. Proof. The roots of G(z) are { 6.} and the roots of F (Qz) are ( /^>.}. The discount factor 6 was choosen so that ount factor It follows that the roots of G(z) are outside a circle of radius /2 Q ' and the roots of F(0z) are inside this circle. Consequently G(z) and in the algebra F(Oz) cannot have a coon root so by a basic theore of polynoials *C(z)/z = -H Q (z)g(z) + K o (z)f(gz) where H (z) and K Q( Z ) are polynoials of degree less than the degree of G and F. Of course G and F are of degree so ultiplying through by z leads to the relation stated in the lea. Theore 6. Given an arbitrary exponoial t(n) let w(x) be the correspondinq extrapolat or

20 w(x) = e n t(n)y n+x. Then a short forula for this extrapolator w(x) = 7 Here h' and k; are coefficients defined by the polynoial 'l(z) which is the iage of t(n) according to Theore 5 and Lea 3. CO Proof. Let Q(z) = 2L q(n)z then according to Theore 3b n - Q(z) = G(z)/F(ez). "Let T(z) = ^ e t(n)z. Then according to Theore 5 we have T(z) = / C(z)/F(Gz). Then Lea 3 with X replacing z gives T(X) = H(X) [Q(X) - ] + K(X). Here X is interpreted as the translation operator. Operating on the function y.with the above identity we see that [Q(X) - l]y = x x 6 and so the short forula follows. This foralis is justified by the absolute convergence of the resulting series when y is in the Hilbert space and so the proof is coplete. A different proof for the existence of the short forula for w(x) results fro cobining Theore with k = - and Theore 4b. The short eory forulae given in Theore 2 and Theore 6 ay be tered th order forulae because the right sides are expressed as translation operators of order. Thus the extrapolator of Theore 6 can be written as Now an D is w(x) = (Ih.X j )6 x + ( I k j X D )y x. th order scalar difference equation is equivalent to a 9 first-order vector difference equation. This suggests that the

21 th order scalar extrapolation forula can be replaced by a 20 first order vector extrapolation forula. In fact such forulae are to be found in the paper by Morrison [3]. for the special case of polynoial extrapolation. Moreover, Morrison's paper indicates that first order vector extrapolators ay be advantageous in nuerical work because of econoy in eory. The following is a general theore on first order vector. extrapolation forula. Theore 7. Let w'(x),...,w (x) denote a set of independent extrapolators. Then there is a set of constants a.. and b.. _ Xj 2_ such that w. (x) = 2 a -M w -; (x + ) + b. This is tered a very short eory forula. co n Proof. Then w. (x)- = 2L 6 r. (n)y and the exponoials _ n"r~x f (n), fp(n),,..,f (n) for a basis for the space XP. Hence it is possible to find constants a.. such that Gr. (n.+ ) = "H a..r.(n) j i =,..., because 0r.(n + ) is also an exponoial. These identities are ultiplied by -0 y, and sued. This gives co ci^ Q r. (n + x)y,, n = c~ a..w. (x + ) n=l \ l J n+x+l r~- 3= in J J i CD 0 n r,(n) Let b. = 0r.() and this is seen to coplete the proof of the very short forula. Corollary 3. The extrapolation of y jat successive points is given by a very short eory forula.

22 Proof. The extrapolation of y n at successive points is 2 p(x + }, p(x + 2),...,p(x + ). By virtue of Theore these are independent extrapolators and the proof is coplete. Lea 4. Let x denote an arbitrary fixed point and let v(x) Q denote a function of the exponoial space XP. Then v(x o ) ' dx I ' ' Tisrr t x=x dx x=x o o are a set of independent linear functionals on the space XP. Proof. The above expressions are obviously linear functionals. If they were not independent then there would be an identity of the for - c.v (l) (x) =0; x = x =0 which holds for v in XP. But if v(x) is in XP so also is v(x + k) for every k. Hence the above identity actually holds for all x. It is a differential equation of order - and can have at ost - independent solutions. This is a contradiction, since there are independent functions in XP, Corollary 4. The extrapolation of y together with its derivatives up to order - evaluated of the sae point is given by the very short eory forula. Proof. The extrapolators are p(x), p* (x),...,p (x) evaluated at a certain point x = x. Then the corollary follows fro Lea 4 and Theore 7. REFERENCES. Duffin, R. J. and Schidt, Th. W., 'An extrapolator and scrutator, J. Math. Anal, and Appl. JL (960), Duffin, R. J. and Whidden, Phillips, 'An exponoial extrapolator!, J. Math. Anal, and Appl. 3^ (96),

23 22 3. Morrison, Noran, T Soothing and extrapolation of tie series by eans of discrete Laguerre Polynoials, SIAM Jour, (to appear).