Statistics of Accumulating Signal

Transcription

1 Instrument Science Report NICMOS Statistics of Accumulating Signal W.B. Sparks, STScI May 1998 ABSTRACT NICMOS detectors accumulate charge which can be non-destructively. The charge accumulation is a process, and out noise is present. Here, formulae are presented for the variance (i.e., error ) of two estimators of the underlying rate of accumulation of charge, or countrate, subject to these two processes only. The formulae are tested using Monte-Carlo simulations and are applied to the standard NICMOS MULTI- ACCUM sequences using the known NICMOS dark properties. These algorithms are essentially the current one employed by calnica, least-squares fit of a straight line, and the previous version, weighted mean of first differences. The usual formula for the error on the slope of the least-squares straight line is inapplicable because data points are not independent of previous data points. Also, the usual formula for the error on the weighted mean is inappropriate because of the presence of out noise (which cancels out to a degree depending on the timing intervals). The results can be used to infer: (i) the least-squares fit algorithm gives a significantly better estimate of the countrate than the weighted mean of first differences, in the noise dominated regime (ii) the uncertainty goes down approximately linearly with exposure time in the noise dominated regime which suggests longer integrations should be used where possible (iii) Multiple Initial and Final s (MIFs) improve the uncertainty by about 20% in regions of the detector where ampglow is small (cf. other sequences) (iv) there is little difference otherwise between any of the standard MULTIACCUM sequences. Tables are provided to allow optimal choice of sequence, and to show the characteristics of the uncertainties of MULTIACCUM data in various circumstances.these variances represent fundamental theoretical limits on the utility of the algorithms. Additional noise sources include cosmic rays and inaccuracies in the shading correction. A fairly high sensitivity to cosmic rays suggest using as many outs as possible, but the consequent accumulation of ampglow can add significant noise. 1

2 1. Model of a MULTIACCUM sequence In NICMOS detectors charge accumulates in each pixel as incident photons are detected, and this charge can be non-destructively. The accumulating charge between each out is expected to obey statistics, and, in addition, associated with each out there is electronic out noise that imposes a random error on the measurement of the charge within an individual pixel. Hence if outs are at times x i, and the actual accumulated charge is y i, then between x i 1 and x i the actual additional accumulated charge is y i y i 1 = p i, where p i is drawn from a distribution with expected value b.(x i 1 x i ) and b is the underlying countrate sought. For example, if the underlying source countrate is 0.1 electrons/sec, and x i 1 = 256 sec and x i = 384 sec, then the expected value (mean) of the accumulating charge in that interval is 12.8 electrons. For any particular integration, there will be an actual value of charge accumulated in that period which is randomly drawn from a distribution of mean=12.8. Hence, y i = p i + y i 1 = p i + p i 1 + p i 2... or in other words y i is a type of random walk. It should be stressed that once the interval of time has elapsed and p i is measured, the actual value p i propagates throughout the subsequent series; it is fixed thereafter and no longer contributes a random element. In addition to the statistical fluctuations associated with the component of the accumulating signal in which the actual accumulated charge is the sum of the preceding trials, there is an uncertainty associated with the measurement of that charge, namely the out noise. That uncertainty is assumed to be independent from one measurement to the next. Figure 1 illustrates. Figure 1: Charge y accumulates during the sequence, and is at times x i. Actual charge increments are p i and there is superimposed out noise. y Underlying countrate Charge y 3 y 2 Readout noise } p 2 } p 3 y 1 0 } p 1 0 x 1 x 2 x 3 x Time 2

3 Here, calculations are presented of the variance of estimators in use, or that have been used. The work does not attempt to calculate new estimators, nor reveal whether those used represent minimum variance estimates. Least squares fitting gives better answers than averaging first differences. 2. Model of a NICMOS MULTIACCUM observation This simple two component model is sufficiently flexible to capture the important elements of a NICMOS dark, see NICMOS ISR-026 by Skinner and Bergeron. The dark comprises (i) a time variable shading term, assumed here to be noiseless and known, (ii) a linear dark term which is conventional accumulating dark current, so that the amount of charge accumulated is proportional to the time since the last reset/, i.e. the elapsed exposure time, and (iii) ampglow, in which every individual out deposits a certain amount of charge into a pixel. The ampglow is strongly field dependent and varies from approximately electrons/ in the detector centers to ten times that in the corners of the field of view. The linear dark term is low, and is approx 0.05 electrons/sec. We assume the values are known and the external countrate corrected accordingly. In addition to internal dark current, there is external background emission from zodiacal light and thermal emission from the HST optics, depending on wavelength and filter. For example, in the F160W minimum background wide band filter, where NICMOS is maximally sensitive, the estimated background countrate is approx 0.09 electron/sec. Finally, there is emission from the source itself. Conversion from countrate to flux obviously depends on the filter used, whereas the statistics depend on the detection process itself, i.e., the countrate, and can therefore be scaled. For example, a countrate of 0.1 electron/sec using the F160W filter corresponds to 3.17 x 10 7 Jy or H = Variance calculations Least squares fit of straight line If y i = a+ bx i + ε i, the least squares fit which minimizes the weighted sum of the squares of the residuals χ 2 = ( w i ( y i a bx i ) 2 ) is Σ Σxy ΣxΣy 1 b = Σ Σx 2 ( Σx) 2 -- ( Σ Σxy ΣxΣy) where Σ denotes the sum of the weights, typically 1 w i = σ i fit, and Σxy etc. are implicitly above, and = ( ) 2. w i x i y i Σ Σx 2 Σx i, or w i = 1 for an unweighted 3

4 If the data points {x i,y i } are independent, then the uncertainty of the estimator b is σ b = Σ. This textbook formula is the one used in calnica at the time of writing. Now, suppose b is the estimator used, given the model above we can calculate the actual variance of b once the equation for it is recast in a form that isolates terms which are independent of one another. As in Figure 1, ignoring out noise for now, let where d is the linear dark, A the ampglow and x i the time interval between two outs. P(a) represents the distribution of mean a from which p i is drawn. The sums involved in calculating the slope may be recast in a form which groups together statistically independent terms. n y i i = 1 = y 1 + y 2 + y y n = p 1 + (p 1 +p 2 ) + (p 1 +p 2 +p 3 ) +... =np 1 + (n 1) p 2 + (n 2) p p n More generally, with weighting, Similarly, Σ x i y i = x 1 y 1 + x 2 y 2 + x 3 y 3... x n y n and y i = y i 1 + p i where p i Pb x ( i + d x i + A) n Σw i y i = p 1 w i + p 2 w i + p 3 w i + 1 n 2 = x 1 p 1 + x 2 (p 1 + p 2 ) + x 3 (p 1 + p 2 + p 3 ) x n = p 1 (x 1 + x x n ) + p 2 x i + p 3 x i x n p n n Σw i x i y i = p r w i x i r = 1 n i = r n 2 n 3 n 3 n P i i = 1 Substituting for Σy and Σxy above and using n 1 b= -- p r s s r 1 { x x r 1 } r = 1 n i = r = n i = 1 r 1 i = 1, we find 4

5 n Where S = w i, S r 1 = w i, x = w i x i w i, and x r 1 = w i x i w i. i = 1 r 1 i = 1 n i = 1 Hence, since p r are independent with variance (b + d) x i + A, the variance of b is σ 2 b n 2 r = 1 i = s 2 2 = s r 1 ( x x r 1 ) 2 [ b x i + d x i + A] Similarly, if we work with estimates y i of the y i which are inaccurate due to the presence of out noise r, we assume that the additional contribution to is , as given by the textbook variance for independent data, which adds in quadrature to give a final formula for the variance of the countrate estimator b σ 2 b n 1 = s 2 ( s r 1 ) 2 ( x x r 1 ) 2 [ b x i + d x i + A] 2 r = 1 For completeness, the variance of the intercept is, by a similar route and neglecting out noise, given by: n r 1 i = 1 nr r 1 i = 1 2 nr 2 σ b σ 2 a = n r = 1 s r ( sx x r 1 sxx) 2 n 2 where sx = w i x i and sxx = w i x i. i = 1 n i = 1 Weighted mean of first differences By first differences we mean the series y i y i 1 using the same notation as above. Each difference provides an estimate of the countrate, b i =(y i y i 1 ) / (t i t i 1 ). Then the estimator of the countrate is the weighted mean of the b i. By taking first differences, the component is now independent of the history of charge accumulation, and it is correct to adopt a simple weighting scheme depending on the individual uncertainties of each difference. During mid-1997, calnica version 2.2 attempted to take advantage of this fact and utilized a first differences approach. However, in regimes where out noise is important, this is not the case. To illustrate, suppose the only thing present is noise. The source countrate is zero, the dark is zero, and there is no ampglow. Then the underlying charge accumulated is always zero, however the measurement of that zero charge follows a Gaussian with standard deviation r (if r is the noise). Hence the set y i is a set of random numbers r x N(0,1) and N(0,1) is 5

6 the zero mean, unit variance normal distribution. Let the first differences be d i = y i y i 1, and the countrate estimates b i = d i / dt i with dt i = t i t i 1. On taking the first differences, if terms are independent the formal error on each difference is 2 r, and so the uncertainty on the mean would be ( 2 r) ( N ) with r the uncertainty on the measurement as noted above, and N the number of measurements. But note that when taking the mean of the differences, cancellation occurs in the out term: b 1 = (y 1 y 0 ) / dt 1 b 2 = (y 2 y 1 ) / dt 2 b 3 = (y 3 y 2 ) / dt 3... b N = (y N y (n 1) ) / dt N For example, if the dt i are all the same equal to unity, the average is <b > = Σ(b i ) / N =(y N y 0 ) / N, which has an uncertainty only 2 r N compared to the expected uncertainty on the mean in the case of independence of the uncertainty is less by a factor of 1 N than in the case of independent data. The amount of cancellation depends on the spacing of the out times, as shown below. Generally, we can calculate the variance, and incorporate the cancellation into an effective noise as follows: Let an individual count rate estimate from the first differences be d b i y i i y i 1 = = t i t i Then form a weighted mean 2 r N. That is d i b = w i t w = i b = w n n 1 w i t ( y i y ) i 1 Σw i i w y t n y n t ri Σw t i t i + 1 i i = 1 w i w i 1, then Consider the case where only noise is present, and the uncertainty on each y i is the same, r. Then 2 2 σ( b ) 2 r 2 w n w = t n 2 t 1 2 n 1 i = 1 w i w i t i t i + 1 ( Σw i ) 2 6

7 For out noise only, and weighting -----, we have = since the uncertainty on 2r 2 each b i is if r is the out noise per. Then substituting for w i, we obtain t i 2 which can be made equivalent to the representation which would be valid if the data were independent, 2 1 t w i = w i 2 i σ i 2r 2 n t 1 t n ( t i t i + 1 ) σ( b ) r2 i = 1 = Σ t i 2 t i r eff σ 2 = r2 2 Σ t i, if we define an effective out noise n t 1 t n ( t i t i + 1 ) i = 1 = r t i For example, if the data are equally spaced as in the example above, and r eff = r n. t i = t i + 1 = t Finally, therefore, in this scheme, we adopt a total variance that includes the component σ( b ) 2 1 t 2 = where w. w i = i d --- i 2 + r g eff 4. Validation of the formulae Monte Carlo simulations of the MULTIACCUM process were run for the standard NICMOS MULTIACCUM sequences and for each individual out within the sequences. The formulae presented in the previous section give excellent agreement with the empirically derived distributions from Monte-Carlo simulations, as may be seen in the tables and figures that follow. The countrate estimators have expected values equal to the mean countrate so they are unbiased, and this too is born out in the testing. 7

8 5. Results Caveats: The exact numerical details in the tables may differ from real NICMOS observations because (1) out noise is assumed per, not per pair of s, (2) only unweighted least squares regression line and a simple weighting were used (3) a zeroth is included in the fit. Nevertheless, the qualitative conclusions will remain valid, and quantitative comparison is likely to reveal only minor differences. The variance associated with the least-squares straight line estimator of count rate is substantially lower than for the weighted mean of first differences. The corresponding standard deviation is about 50% lower for the line fit in the low background out noise dominated regime. In a high countrate regime, the two methods are comparable. That is, least squares fitting is better than first differences. In the low countrate regime, the uncertainty on the countrate estimate declines linearly with time, as expected if the S/N has S t and N ~ constant. The cluster of final s in the MIF1024 sequence reduces the uncertainty by about 20% compared to the same sequence just prior to the final cluster of outs, in the low ampglow case. In other words, judicious choice of out timing can make a significant difference to the performance. In the high ampglow case, the extra s make little difference, and the overall uncertainty is higher by about a factor two. The countrate uncertainty in the low countrate (faint object) limit depends primarily on exposure time and not on the specific sequence chosen. Example: in Table 1, for a MIF1024 sequence and exposure 1024 sec, the uncertainty is using a least-squares fit prior to the final out cluster, and after. Hence a three sigma detection requires a countrate e/sec and e/sec respectively, or 2.61x10 7 Jy and 2.09x10 7 Jy for the F160W filter, or H=24.1 and respectively. The NICMOS exposure time calculator gives very similar answers, with S/N=3 for 0.1 e/s in slightly less than 1000sec. 6. Appendix: Tables of uncertainties for all the standard NICMOS MULTIACCUM sequences as a function of out number follow for various circumstances. Table 1 gives the minimum background faint object limit near the detector center, i.e. low ampglow, and Table 2 gives the same table for near the detector corners. Table 3 gives an example run at high source countrate. External thermal emission is not distinguished from source emission in this case. The columns are (1) sequence name (2) out number (3) old (incorrect) estimate of uncertainty from least squares fit of straight line (4) empirical estimate of the same quantity from Monte-Carlo simulation (5) revised formula for uncertainty of weighted mean of 8

9 first differences (6) empirical estimate of the same from Monte-Carlo simulation (7) revised analytical formula for uncertainty on least squares fit. THE FINAL COLUMN represents the current best estimate of the uncertainty associated with a particular observation. For Table 1: Output is sequence,, texp, els_an,els_emp,ewt_an,ewt_emp Empirical estimate uses n = 100 Parameters of run: Source countrate: electron/sec Readout noise: electron/sec Amp glow: electron/ Linear dark: electron/sec For Table 2: Output is sequence,, texp, els_an,els_emp,ewt_an,ewt_emp Empirical estimate uses n = 100 Parameters of run: Source countrate: electron/sec Readout noise: electron/sec Amp glow: electron/ Linear dark: electron/sec For Table 3: Source has H=19.4, or 16 micro Jy. Empirical estimate uses n = 100 Parameters of run: Source countrate: electron/sec Readout noise: electron/sec Amp glow: electron/ Linear dark: electron/sec Table 1. Minimum background uncertainty; no source MCAMRR MCAMRR MCAMRR MCAMRR

10 Table 1. Minimum background uncertainty; no source (Continued) MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF

11 Table 1. Minimum background uncertainty; no source (Continued) MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF

12 Table 1. Minimum background uncertainty; no source (Continued) MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF

13 Table 1. Minimum background uncertainty; no source (Continued) MIF MIF MIF MIF MIF MIF MIF MIF SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SPARS SPARS

14 Table 1. Minimum background uncertainty; no source (Continued) SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS

15 Table 1. Minimum background uncertainty; no source (Continued) SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP

16 Table 1. Minimum background uncertainty; no source (Continued) STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP

17 Table 1. Minimum background uncertainty; no source (Continued) STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP

18 Table 1. Minimum background uncertainty; no source (Continued) STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP

19 Table 1. Minimum background uncertainty; no source (Continued) STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP

20 Table 1. Minimum background uncertainty; no source (Continued) STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP Table 2. Minimum background uncertainty; no source; high ampglow MCAMRR MCAMRR MCAMRR MCAMRR

21 Table 2. Minimum background uncertainty; no source; high ampglow (Continued) MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MCAMRR MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF

22 Table 2. Minimum background uncertainty; no source; high ampglow (Continued) MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF

23 Table 2. Minimum background uncertainty; no source; high ampglow (Continued) MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF MIF

24 Table 2. Minimum background uncertainty; no source; high ampglow (Continued) MIF MIF MIF MIF MIF MIF MIF MIF SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SCAMRR SPARS SPARS

25 Table 2. Minimum background uncertainty; no source; high ampglow (Continued) SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS

26 Table 2. Minimum background uncertainty; no source; high ampglow (Continued) SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS SPARS STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP

27 Table 2. Minimum background uncertainty; no source; high ampglow (Continued) STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP

28 Table 2. Minimum background uncertainty; no source; high ampglow (Continued) STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP STEP