A Stable Finite Dierence Ansatz for Higher Order Dierentiation of Non-Exact Data Bob Anderssen and Frank de Hoog, CSIRO Division of Mathematics and Statistics, GPO Box 1965, Canberra, ACT 2601, Australia and Markus Hegland, Computer Sciences Laboratory, RSISE, Australian National University, Canberra ACT 0200, Australia Abstract If standard central dierence formulas are used to compute second or third order derivatives from measured data even quite precise data can lead to totally unusable results due to the basic instability of the dierentiation process. Here an averaging procedure is presented and analysed which allows the stable computation of low order derivatives from measured data. The new method rst averages the data, then samples the averages and nally applies standard dierence formulas. The size of the averaging set acts like a regularization parameter and has to be chosen as a function of the grid size h. 1991 Mathematics Subject Classication. 65D25. 1
1. Introduction Let the given (observational or non-exact) data be dened by d := fd j = f(t j ) + j ; t j = jh; h = 1=n; j = 0; 1; 2; ; ng; (1) where f(t) denotes the underlying, but unknown, signal process and the j denote the (observational or non-exact) errors which are assumed to be identical and independently distributed normal random variables with E[ j ] = 0 and E[ j k ] = 2 jk, where jk denotes the Kronecker delta function and E[] the expectation operator. In the numerical dierentiation of non-exact data, the goal is to recover, from the given data d, an estimate of some (lower order) derivative f (p) (t) := d p f(t)=dt p ; p = 1; 2; or 3; (say); rather than f itself. Often, it is only the rst derivative that is required. This situation has been examined in considerable detail in the literature, under the assumption that higher order dierentiation is some natural generalization of the results for rst order dierentiation. Clearly, even the fractional dierentiation of non-exact data (cf. Anderssen (1976) and Hegland and Anderssen (1995)) can, on occasions, be given a similar interpretation. However, though not incorrect, this assumption glosses over important practical details which are the focus of this paper. Notation and Assumptions. The notation j will be used to denote the nite dierence value of the p-th derivative evaluated at the grid point jh. The function f will be assumed to have the smoothness required by the formulas presented below in terms of the dierentiation they involve. 2. The Finite Dierence Ansatz for First Order Dierentiation Often, in the past (cf. Andersen (1963)), and even today, data are dierentiated using a ruler to obtain an estimate of the rst derivative of f. It is fast and has a natural intuitive appeal. In fact, for the practitioner, who has just measured or calculated (graphically) some specic data the rst derivative of which must be estimated before their interpretation is possible, the ruler approach represents a realistic alternative (cf. Anderssen and Bloomeld (1974)). It is the reason why it is still used today, at least as a quick exploratory tool. However, its greatest drawback is (and was) its lack of objectivity in that the form of the rst derivative determined by a particular individual will be inuenced by their level of familiarity with the context within which that data have been derived. 2
Mathematically, numerical dierentiation was initially seen as simply a special case of constructing nite dierence approximations to derivatives. Consequently, the earliest methods proposed for the numerical dierentiation of accurate numerical, though not necessarily exact, data were nite dierence formulas. They were derived, in one way or another, through the manipulation of either the denition of a derivative or Taylor's expansion (theorem). They predate the computer (cf. Hartree (1952)), and relate closely to the earliest ideas about the numerical approximation of derivatives in dierential equations (cf. Richardson (1910)). However, the interconnection is not clear cut. On the one hand, forward, backwards and central dierence approximations take center stage in the numerical solution of dierential equations, whereas it is only the central (centered) dierence formulas (or, equivalently, centered moving-averages) f [1] j [m] = d j+m? d j?m ; m = 1; 2; ; j = m; m + 1; ; n? m; (2) 2mh which are the key to the numerical dierentiation of data. Normally, m is chosen to have the value 1, but the possibility of choosing a greater value has been implicitly examined by a number of authors within the context of optimizing the choice of h (cf. Conte and de Boor (1980; Section 7.1)). In part, the advantage of a central dierence formula, over the alternatives such as forward and backwards dierences, is the associated higher order of convergence, which, in turn, can be explained algebraically and graphically as a practical realization of the mean value theorem. In fact, the ruler dierentiation simply corresponds to an analogue realization of the mean value theorem applied directly to the data. Statistically, the approach adopted was quite dierent. Whereas the nite dierence formula approach moreorless determines the derivatives at the data points directly and explicitly from the data, the statistical approach is indirect. Here, one rst estimates statistically the parameters in an assumed parametric estimate ^f(t; ) of f(t), and then estimates f (1) (t) as ^f [1] (t; ): This statistical approach leads naturally to the following two important generalizations: (a) Non-Parametric Dierentiation. This is simply the non-parametric counterpart to the parametric procedure outlined above, where one replaces the specic choice of a parametric model for f(t) by a non-parametric functional characterizing its structure. For example, a popular choice for the non-parametric functional is the least squares smoothing spline criterion (cf. Wahba (1990)) f (t) = argfmin f 2H 1 [ nx j=0 (f(t j )? d j ) 2 + Z tn t 0 (d 2 f(t)=dt 2 ) 2 dt]g; (3) where H 1 denotes the Sobolev space of absolutely continuous rst derivatives. 3
(b) Fourier-Wiener Dierentiation. If it is assumed that the data has been generated by a stationary stochastic process, then discrete Fourier analysis and Wiener ltering can be applied directly to the data to recover an estimate of the (rst) derivative of f(t) (cf. Anderssen and Bloomeld (1974)). Though the major emphasis in Anderssen and Bloomeld (1974) was on the implementation of numerical dierentiation as a Wiener ltering process, they showed how, for given data, the Wiener ltering theory could be used to construct a type of centered moving-average (local dierentiator) for performing the dierentiation. In essence, these local dierentiators are simply central dierence formulas. However, the utility of such formulas does not appear to have been pursued in any great detail. In an independent study, Anderssen and de Hoog (1984) analysed the stability properties of multipoint nite dierence dierentiators of the form f [1] j = f [1] (t j ) = k=?r W k d j+k ; (4) where the W k ; k =?r;?r + 1; ; r? 1; r, denote appropriately chosen weights. They rst observed that, if the weights satisfy W k =?W?k ; k = 0; 1; 2; ; r; which implies that W 0 = 0, then the multipoint nite dierence dierentiators (4) are exact for constant data, and can be rewritten as the following sum of the central dierence dierentiators [k] f [1] j where f [1] j = k=1 w k f [1] j [k]; W k = w k =(2kh); k = 1; 2; ; r; (5) k=1 w k = 1: (6) In part, the goal of that paper was to show that the stabilization of such formulas was controlled by its length r. In fact, it was established that the choice of the length r must be related to the size of the step-length h so that r increases appropriately as h decreases. Consequently, under such circumstances, the dierentiator (5) can be given a regularization interpretation in which the length r and the weights w j play, respectively, the role of the regularization parameter and the regularization. A formal characterization of this fact was also derived. In a spline-type context, such results can be formalized using mollication (cf. Anderssen (1995)). Hegland and 4
Pragmatically, this result yields a natural ansatz for the construction of nite dierence formulas for the stabilized numerical dierentiation of one-dimensional observational or non-exact data; namely, The Finite Dierence Ansatz for First Order Dierentiation: \Choose as the local dierentiator, a weighted sum of the central dierence dierentiators j [m] so that, when it is applied to the data as a centered moving-average, an appropriately smooth estimate of the derivative f (p) (t) of the signal f(t) results." The purpose of this paper is an examination of the applicability of this ansatz to higher order dierentiation, when, for a xed m, the averaging is performed with respect to j. A natural motivation for this approach can be based on the advantages of performing repeated measurements in a statistical analysis. In fact, if, relative to the smoothness of f, the size of h is very small, then, for small r, the values of f [1] j+l[m]; l =?r;?r + 1; ; r? 1; r; can be viewed as repeated measurements of f [1] j [m]. 3. The Finite Dierence Operators Let: (a) D p denote the dierential operator of order p dened by D p = dp ; p = 1; 2; : dtp (b) I correspond to the unit interval [0; 1] on IR. (c) f be a real-valued function dened on I with sucient regularity such that exists. (d) G h denote the uniform grid of points D p f(t) = dp f dt p (t); t j = jh; j = 0; 1; 2; ; n; h = 1=n: 5
(e) f = f G h denote the restriction of f to the grid G h. (f) (p) h;m denote a family, parameterized by m, of dierence operators, dened in terms of their action on f, which approximate (D p f)(ih), i 2 I, in the sense that (p) h;mf(ih) = (D p f)(ih) + O((kh) 2 ); (7) where (p) h;mf(ih) only acts on the grid values f imj for j = 0; 1; 2;. For the rst and second derivatives, the natural counterparts of such second order nite dierence formulas are, respectively, (1) h;mf(ih) = (2) h;mf(ih) = f(ih + mh)? f(ih? mh) ; (8) 2mh f(ih + mh)? 2f(ih) + f(ih? mh) (mh) 2 : (9) Note. Clearly, for a given j, there will be an upper bound on the value of m which guarantees that 0 (j? m)h < (j + m)h 1. However, this is a rather technical matter which can be circumvented by assuming that, with respect to a given choice of j, the value of n is such as to generate a suciently ne grid which guarantees the application of any particular formula considered. In other words, it is assumed that one has sucient data to perform the relevant operations examined and discussed below. Such situations occur naturally in situations where the data is collected by a computer in an on-line monitoring scenario. 2 3.1 The Averaged Finite Dierence Formulas Let i [m] = (p) h;mf(ih): (10) Here, we examine the following averaging of these nite dierence dierentiation formulas i [m] = 1 j=?r The repeated measurement interpretation follows from the fact that i+j[m]: (11) i [m] = i [m]; (12) 6
i.e. the above averaging of the nite dierence formulas corresponds to the application of the chosen nite dierence formula (for a xed m) to the averaging of the data at the grid points (i + m)h, ih and (i? m)h by the formula f i h = 1 j=?r where ih corresponds to (i + m)h, im and (i? m)h, respectively. f ih+j ; (13) In order to guarantee that the errors generated by the application of the numerical dierentiation formulas (10) to the observational data fd j g remain uncorrelated, one must ensure that, respect to a given r, the value of m is suitably large. The simplest strategy is to replace m in (10) by kr + 1 to obtain i [kr + 1] = 1 j=?r i+j[kr + 1]; (14) and to constrain k to satisfy k 2. If m =, then every consecutive point about the grid point ih is utilized in the evaluation of (13). This clearly represents the most ecient use of the data. 4. Convergence and Stability The proof of convergence and stability exploits the data-averaging duality of the formula (11). Without loss of generality, attention is restricted to the situation where p is an even integer 2q. In fact, since, for suitably smooth functions f, f ij = f i jhf (1) i it follows that, with p = 2q, f i = 1 j=?r f i+j = f i + + (jh)2 p+1 X f (2) i + 2! l=3 r(r + 1)h2 f (2) i + 12 where the P 2l (r) denote polynomials of degree 2l in r. It, therefore, follows that 7 (1) l (jh) l l! qx l=2 f (l) i + O((jh) p+2 ); (15) P 2l (r)h 2l f (2l) i + O((rh) 2q+2 ); (16)
i [kr + 1] = i [kr + 1] = (p) + h;kr+1f(ih) + qx l=2 r(r + 1)h2 (p) 12 h;kr+1f (2) (ih) P 2l (r)h 2l (p) h;kr+1f (2l) (ih) + O (rh) 2q+2 ((kr + 1)h) 2q )! : (17) If one recalls that (p) h;kr+1f (2l) (ih) = f (2l+p) i + K((kr + 1)h) 2 f (2l+p+2) (); K = constant; where the value of the constant K depends on the explicit form of the nite dierence formula used and denotes the appropriate mean value, this last result becomes i [kr + 1] = f (p) i + K((kr + 1)h) 2 f (p+2) ( 2 ) r(r + 1)h2 + f (p+2) i + Kr(r + 1)h2 (krh) 2 f (p+4) ( 4 ) 12 12 + 1 qx P 2l+1 (r)h p+2l (f (p+2l) i + K((kr + 1)h) 2 f (p+2l+2) ( 2l+2 ) l=2 (rh) 2q+2 +O( ); (18) ((kr + 1)h) 2q where the 2, 4, etc denote the appropriate mean values. On recalling that (cf. Finney (1968), Section 5.8) 1 j=?r i+j = 1 p i ; where the i denote identically distributed Gaussian random variables with zero mean and variance 2, it follows that, because k 2, the i+(kr+1)j, for xed i and j = 0; 1; 2;, are independent and K (p) h;kr+1d(ih) = i [kr + 1] + q() ((kr + 1)h)?p i ; (19) 8
where the value of K depends on the nature of the nite dierence formula chosen. It follows immediately from (17) and (18) that i (i) krh! 0, (ii) rh! 0, (iii) r 1=2 (krh) p 1. [kr + 1] converges to f (p), if If it is assumed that krh = h s, then, from (iii), one obtains that r h?2ps. But, (ii) then implies that h 1?2ps must tend to zero, and thus that convergence is guaranteed if i 2ps < 1; k nite: (20) The size of the perturbation in the actual values obtained is clearly controlled by the value of k. In particular, the larger k the smaller the perturbation errors. Thus, for a very stable solution, one requires, as well as (i), (ii) and (iii), that (iv) k! 1. If it is again assumed that krh = h s, then it now follows that convergence is guaranteed if (2p + 1)s < 1: (21) 1 0.8 0.6 y 0.4 0.2 0 0.2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 x Figure 1: Data y = x 3 + x 9
5. Implementation and Exemplication Once, for a given p, the form of the local dierentiator has been chosen, implementation reduces to choosing the values of k and r. Clearly, the actual choice of k and r will depend of the nature of the observational data. The eciency of the proposed dierentiator shall now be demonstrated on synthetic data. Let f(x) = x 3 and the standard deviation be = 0:001 such that the data is The data is plotted for h = 0:01 in Figure 1. y i = (ih) 3 + i : (22) 80 60 40 20 y" 0 20 40 60 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 x Figure 2: Central dierence approximation of 2nd derivative with measurement error The second derivative for this example is f (2) (x) = 6x. In Figure 2 the values of the second dierences are plotted. One sees that the variance is huge such that no trend can be detected even thogh the the synthetic data is quite precise. Using the new dierentiator with r = 4 and k = 2 (which is minimal) one obtains the approximations for f (2) displayed in Figure 3. There is now an obvious trend and the points are fairly close to the expected points. However, due to the sampling procedure, there are much fewer data points in the derivative. But an interpolation of these points still gives a good result in this case. When the original data has higher curvature one might require more data points to reconstruct the derivative. 10
6 5 r=4, k=2 4 y" 3 2 1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 x Figure 3: Central dierence approximation of the averaged function References Chr. Andersen (1963) The Ruler Method - An Examination of a Method for Numerical Determination of Fourier Coecients, Acta Polytechnica Scandinavica, AP 324, 1963. R. S. Anderssen and P. Bloomeld (1974) Numerical dierentiation procedures for non-exact data, Numer. Math. 22 (1974), 157-182. R. S. Anderssen, Stable procedures for the inversion of Abel's equation, J.-Inst.-Math.-Appl. 17 (1976), no. 3, 329{342. R. S. Anderssen and F. R. de Hoog (1984) Finite dierence methods for the numerical dierentiation of non-exact data, Computing 33 (1984), 259-267. S.D. Conte and C. de Boor, Elementary numerical analysis: an algorithmic approach, 2nd ed. N.Y. McGraw-Hill, 1972. D. R. Hartree (1952) Numerical Analysis, Clarendon Press, Oxford, 1952. M. Hegland and R.S.Anderssen, 1996, A Mollication Framework for Improperly Posed Problems, submitted, Mathematics Research Report MRR 085-95, CMA, ANU. L. F. Richardson (1910) The approximate arithmetic solution by nite dierence of physical problems involving dierential equations with an application to the stresses in a masonary dam, Philos. Trans. Roy. Soc. London, Series A, 210 (1910), 307-357. G. Wahba (1990) Spline Models for Observational Data, SIAM, Philadelphia, Penn., 1990. 11