Howard Mark and Jerome Workman Jr.

Size: px

Start display at page:

Download "Howard Mark and Jerome Workman Jr."

Brittney Nelson
5 years ago
Views:

Derivatives in Spectroscopy Part IV Calibrating with Derivatives Howard Mark and Jerome Workman Jr.

discussion of the development of the Savitzky Golay method of using convolution functions to compute derivatives, concluding with the presentation of a general method to create the set of convolution

When performing quantitative calibrations using a derivative transform, several possible problems can arise.

1 Derivatives in Spectroscopy Part IV Calibrating with Derivatives Howard Mark and Jerome Workman Jr. 44 Spectroscopy 19(1) January 2004 Our previous three columns (1 3) contained discussion of the theoretical aspects of using derivatives in the analysis of spectroscopic data, followed by a discussion of the development of the Savitzky Golay method of using convolution functions to compute derivatives, concluding with the presentation of a general method to create the set of convolution coefficients for any desired order of derivative, using any degree of polynomial fitting function and number of data points. When performing quantitative calibrations using a derivative transform, several possible problems can arise. We ve already noted that one of these is the possibility that the data used to compute the derivative will be affected by interfering materials. We can do little in a column such as this one to deal with such arbitrary and sample-dependent issues. Therefore we will concentrate on those issues that are amenable to mathematical analysis; this consists mostly of the behavior of the computed derivative when there is noise on the data. Howard Mark serves on the Editorial Advisory Board of Spectroscopy and runs a consulting service, Mark Electronics (69 Jamie Court, Suffern, NY 10901). He provides assistance, training, and consultation in NIR spectroscopy, chemometrics, and statistical data analysis, as well as custom software and hardware development. He can be reached via at hlmark@prodigy.net. Jerome Workman Jr. serves on the Editorial Advisory Board of Spectroscopy and is chief technical officer and vice president of research and engineering for Argose, Inc. (Waltham, MA). He may be reached by at jworkman@argose.com. Most of our discussion so far has centered on the use of the two-point-difference method of computing an approximation to the true derivative, but since we have already brought up the Savitzky Golay method, it is appropriate here to consider both ways of computing derivatives, when considering how they behave when used for quantitative calibration purposes. In fact, the two-point method can be considered a special case of the more general Savitzky Golay concept because it can be considered the application of the set of convolution coefficients 1, 0, and 1 to the data. Of course, these convolution coefficients were created ad hoc, and not according to the general scheme that produces the Savitzky Golay set. Nevertheless, it is convenient to group them together for the purpose of further examination. We are also indebted to David Hopkins for invaluable discussions concerning the properties of the Savitzky Golay convolution coefficients (4). In our previous column we derived the expressions for the first and second derivatives of both the normal and Lorentzian band shapes (1). For the following discussion, however, we will address only the normal case because, as we will see, the Lorentzian case will parallel it closely. In that previous column, we used the standard generic formula for the normal distribution, ignoring the aspect of using it to describe the situation for quantitative analysis. The quantity of concern now is the signal-tonoise ratio (S/N) of the data that we will use to perform the calibration calculations. To deal with this systematically, the S/N must now be divided into two parts: the magnitude of the signal, and the magnitude of the noise. Then different situations can be compared by independently computing the signal and noise contributions to the final S/N that is operative on the calibration.

This diagram shows how, as the spacing at which the derivative is computed increases, the error in the approximation to the true derivative decreases, even for the same error in the data.

2 The signal part of S/N that concerns us is the way these expressions vary with the concentration of the analyte. Therefore, from equation 22 we obtain for the absorbance signal [25] For the first derivative we obtain Figure 11. This diagram shows how, as the spacing at which the derivative is computed increases, the error in the approximation to the true derivative decreases, even for the same error in the data. [26] We start with the simpler case, the signal. By investigating the behavior of the theoretical, ideal derivative, we avoid issues having to do with the different ways an approximation to the derivative can be obtained. The various approximations that can be obtained through the use of constructs such as the Savitzky Golay convolutions allow us to make tradeoffs between maximizing the signal, faithfully reconstructing the true derivative, and creating artifacts, but these issues are all obviated by considering the behavior of the theoretically ideal case. When we come to consider the noise, then as we shall see, the nature of the approximating method becomes very important, but for now we will ignore that. If the concentration of a material can vary, however, then according to Beer s law, the absorbance at any given wavelength will also be proportional to C, the concentration. Therefore to take the concentration into account we must modify (including changing the generic Y variable to A,to indicate absorbance) equations 1a, 6a, and 9a to Whereupon the first derivative becomes And the second derivative is [22] [23] [24] And for the second derivative we obtain [27] As we see from these equations, we have recovered the original expressions for the absorbance and the derivatives with respect to wavelength. The expression we used for the normal curve was the constant-area expression, but the continuation of the derivation for the change of the signal with respect to concentration will follow for the constant-height case, and for the Lorentzian curve, also. As we saw in reference 1, the maximum value of the first derivative decreases as, and the second derivative decreases as 2 compared with the absorbance, and therefore their derivatives with respect to concentration (which is the sensitivity to concentration changes) also decrease that way. Therefore we now turn to the noise part of S/N. As we saw just above, the two-point derivative approximation can be put into the framework of the Savitzky-Golay convolution functions, and we will therefore not treat them as separate methods. We have derived previously (5, 6) that the following expression relates the noise on data to the noise of a multiple of that data: 46 Spectroscopy 19(1) January 2004

3 Var (ax) a 2 Var (X) [28] and, of course, we know that variances add. Therefore, if we have several variables, each of them contaminated with some noise (whose variance is Var (X)), and they are multiplied by some constants, then the variance of the result is: Var (X net ) a 1 2 Var (X) + a 2 2 Var (X) + a 3 2 Var (X)... [29] Therefore, if X represents the spectrum, the various a i represent convolution coefficients and Var(X) represents a noise source that gives a constant noise level to the spectral values, then equation 29 gives the noise variance expected to be found on the computed resultant value, whether that is a smoothed spectral value or any order derivative computed from a Savitzky Golay convolution. Equation 29 can be simplified slightly: [30] The expression under the radical gives the multiplying factor for the noise standard deviation for the computed derivative (or smoothed spectrum, but that is not our topic here; we will address only the question of the effect on derivatives), and can be computed solely from the convolution coefficients themselves, independently of the effect of the convolution on the signal part of S/N. The nature of the convolution function matters, however, and so do the details of the way it is computed. To see this, let us begin by considering the two-point derivative we have been dealing with in most of this subseries of columns. For our first examination of the effect, let us consider that we are computing the derivative from adjacent data points spaced 1 nm apart (such as in our initial discussion of derivatives [1]). As we mentioned, the two-point first derivative is equivalent to using the convolution function { 1, 1}. We also treated this in our previous column, but it is worth repeating here. Therefore the multiplying factor of the spectral noise variance is , and the multiplying factor for the noise standard deviation is 2 1/2.Similarly, the second derivative is equivalent to a convolution function of {1, 2, 1}, making the multiplying factor for the standard deviation of the derivative calculated this way equal 6 1/2. Because we have noted earlier that the magnitude of the signal parts of S/N [dc/d (dx/d )] decreases with increasing derivative order, at this point it would appear that because the signal decreases and the noise increases when you take a derivative, you wind up losing from both parts of S/N. But things are not so simple. In this examination we have so far looked only at a derivative calculated from adjacent data points. What happens when we calculate a two-point derivative based on non-adjacent data points? We have already considered this question qualitatively in reference 3, when we noted that using the optimum spacing will result in an improved S/N for the derivative. Of course, improved in this case is in comparison to the derivative computed using adjacent data points; it must be determined on a case-bycase basis whether the improvement also carries over to the analytical signal. We can express the improvement semiquantitatively in a graph, as in Figure 11. Here we show true spectrum as the straight line representing the true derivative, and the measured absorbance data as the large Xs. Because the measured data are contaminated with random noise, they do not fall on the line representing the true spectrum. The diagram is set up in such a way, however, that the noise on the data from the two wavelengths representing spacing 1 and spacing 2 is the same. It is clear from this diagram that the computed approximation to the true derivative is better for the case of spacing 2, even though the noise is the same. There are several ways to express this in words. One way is to note that the error is spread over a larger X distance, and therefore has less effect at any one point. Another way is to note that for a derivative computation, the effective signal is the value of Y, and when X 2, Y is double the value 48 Spectroscopy 19(1) January 2004 Circle 50

4 Table I. Some of the Savitzky Golay convolution coefficients using a quadratic fitting function. 5-point 7-point 9-point 5-point 7-point 9-point first first first Index smooth smooth smooth derivative derivative derivative Normal factor of Y when X 1. Because the noise is the same, S/N therefore improves with an increase in the spacing. We learned in our prior column (3), however, that the improvement is linear with spacing only at very small values of X, at large values it decreases, levels off, and then eventually starts to get worse again. From a mathematical point of view, we can let X be the increment between adjacent measurement wavelengths. Then X = n X,where n is the number of wavelength increments over which the derivative is calculated. Then, because: The nature of the values involved caused the entries to be difficult to compare directly, therefore we recomputed them to eliminate the normalization factors, making the coefficients more easily comparable; we present these in Table II. For Table II we also computed the sums of the squares of the coefficients and present them in the last row. One trend is obvious: the more data included in the computation, the smaller the variance multiplying factor. This is computed derivative Y/ X Y/n X [31] Applying equation 28 to find the variation of the computed derivative we obtain: [32] and because X is a constant (with an assumed value of unity), and therefore its variance is zero, equation 32 becomes [33] Converting to standard deviations: [34] A similar expression can be developed for the second derivative, but we leave that as an exercise for the reader. We turn now to the effect of using the Savitzky Golay convolution functions. Table I presents a small subset of the convolutions from the tables. Because the tables were fairly extensive, the entries were scaled so that all of the coefficients could be presented as integers; we have previously seen this. January (1) Spectroscopy 49

5 Table II. The Savitzky Golay convolution coefficients multiplied out. All coefficients are for a quadratic fitting function. 5-point 7-point 9-point 5-point 7-point 9-point first first first Index smooth smooth smooth derivative derivative derivative SSK* *See text for the meaning of SSK. Table III. More Savitzky Golay convolution coefficients. Index 7-point 5-point first 5-point second smoothing derivative derivative, with quartic with cubic quadratic fitting function fitting function fitting function SSK expected for the case of smoothing; we know that the more data included in even an ordinary running smooth (that is, a running arithmetic average), the smaller the variance of the smoothed (averaged) result (reducing as the square root of the number of data points included in the average). Therefore it is not surprising to find it also happening with a weighted average, such as we find with a Savitzky Golay smooth. We see a similar effect from the first derivative; this can also be considered to be extended from the case of the twopoint derivative, where we showed earlier that the farther apart the points used are, the smaller the variance of the resulting derivative value. In the case of the Savitzky Golay convolution functions, however, the mechanism leading the reduced variance is slightly different than that of the twopoint derivative. In the Savitzky Golay case, the reduced variance is caused by the implicit smoothing effect of computing the function over multiple data points, just as it is in the case of explicit smoothing. There are several directions in which the convolutions can be varied; one is to increase the amount of data used, by using longer convolution functions as we demonstrated above. Another is to increase the degree of the fitting polynomial, and the third is to compute higher-order derivatives. In Table III we present a very small selection of the effect of potential variations. What can we learn from Table III? We can compare those sums of squared coefficients with the corresponding one in Table II using the same number of data points, and either: 1) The same order derivative with a lower-degree fitting polynomial, or 2) The same degree polynomial, for a lower-order derivative. For comparison one, we find two cases: seven-point smooth with quadratic versus quartic fitting function, and five-point first derivative with quadratic versus cubic fitting function. From these two comparisons we find that the noise of the derivative (of the same order and number of data points) increases as the degree of the fitting function increases. For comparison two, we find one case: five-point first derivative versus five-point second derivative, both using a quadratic fitting function. Here again, the noise increased with increasing derivative order. In fact, we see that the five-point first derivative using a cubic fitting function will have almost as high a noise level as the original data. Couple this with the fact we saw above, that the sensitivity to concentration of the first derivative is reduced compared to the sensitivity of the absorbance data itself, and we see that in this particular case, depending on the value of for the absorbance band, use of this form of computing the derivative may be worse than using the absorbance data, while using a different computation, such as a quadratic fitting function, may be better than the absorbance 50 Spectroscopy 19(1) January 2004

6 data. Therefore, the effect of using derivatives will depend very much, on a case-by-case basis, whether a particular computation will be beneficial or detrimental. For this reason, readers will find another very interesting exercise to compute the sums of the squares of the coefficients for several of the sets of coefficients, to extend these results to both higher order derivatives and higher degree polynomials, to ascertain their effect on the variance of the computed derivative for extended versions of these tables. Hopkins (7) has performed some of these computations, and has also coined the term RSSK/Norm for the ((coeff/ normalization factor) 2 ) in the Savitzky Golay tables. Because here we predivide the coefficients by the normalization factors, and we are not taking the square roots, we use the simpler term SSK (sum squared coefficients) for our equivalent quantity. Hopkins in the same paper has also demonstrated how the two-point computation of derivatives can also have an equivalent value of the RSSK/Norm, with results essentially equivalent to the ones we presented earlier. Table III in reference 7, particularly, shows how differences in the application of the derivative computation can cause the noise level of the computed derivative to be either greater or less than the noise of the absorbance spectrum from which they are computed. Acknowledgement The authors thank David Hopkins for valuable discussions regarding several aspects of the behavior of Savitzky Golay derivatives, and also for making sure we spelled Savitzky correctly! References 1. H. Mark and J. Workman, Spectroscopy 18(12), (2003). 2. H. Mark and J. Workman, Spectroscopy 18(9), (2003). 3. H. Mark and J. Workman, Spectroscopy 18(4), (2003). 4. D. Hopkins, Private communication (2002). 5. H. Mark and J. Workman, Spectroscopy 3(8), (1988). 6. H. Mark and J. Workman, Statistics in Spectroscopy (Academic Press, New York, 1991). 7. D. Hopkins, Near Infrared Analysis 2, 1 13 (2001). January (1) Spectroscopy 51

Chemometrics. Derivatives in Spectroscopy Part III Computing the Derivative. Howard Mark and Jerome Workman Jr.

Chemometrics. Derivatives in Spectroscopy Part III Computing the Derivative. Howard Mark and Jerome Workman Jr. Derivatives in Spectroscopy Part III Computing the Derivative Howard Mark and Jerome Workman Jr. Jerome Workman Jr. serves on the Editorial Advisory Board of Spectroscopy and is chief technical officer