Model-based Correlation Measure for Gain and Offset Nonuniformity in Infrared Focal-Plane-Array Sensors

Model-based Correlation Measure for Gain and Offset Nonuniformity in Infrared Focal-Plane-Array Sensors César San Martin Sergio Torres Abstract In this paper, a model-based correlation measure between gain and offset nonuniformity in infrared focal plane array (FPA) imaging systems is developed. The correlation between the nonuniformity gain and offset is modeled by means of a generalized Gauss-Marov model. The propose model-based correlation is validated by using several real infrared video sequences and three well nown scene-based nonuniformity correction methods. Keywords: Gauss-Marov Model, Image Sequence Processing, Infrared FPA, Signal Processing 1. INTRODUCTION Infrared (IR) cameras use an IR sensor to digitize the information, and due to its high performance, the most employed integrated technology in IR sensors is the Focal Plane Array (FPA). An IR-FPA is a die composed of a group of photodetectors placed in a focal plane forming a matrix of X Y pixels, which gives the sensor the ability to collect the IR radiation. Nonuniformity noise in IR imaging sensors, which is due to pixel-to-pixel variation in the detectors responses, can considerably degrade the quality of IR images since it results in a fixed-pattern-noise (FPN) that is superimposed on the true image. Further, what maes matter worse is that the nonuniformity slowly varies over time, and depending on the FPA technology, this drift can tae from minutes to hours. In order to solve this problem, several scene-based nonuniformity correction (NUC) techniques have been developed 1,, 3, 4. Scene-based techniques perform the NUC Department of Electrical Engineering, Universidad de La Frontera. Casilla 54-D, Temuco, Chile. csmarti@ufro.cl Department of Electrical Engineering, University of Concepción. Casilla 160-C, Concepción, Chile. sertorre@udec.cl using only the video sequences that are being imaged, not requiring any ind of laboratory calibration technique. Our group has been active in the development of novel scene-based algorithms for NUC based on statistical estimation theory. In 5, 6 we have developed a Gauss-Marov model to capture the slow variation in the FPN and have utilized the model to adaptively estimate the nonuniformity in blocs of infrared video sequences using a Kalman Filter. In such wor the gain and offset are assumed uncorrelated. The principal contribution of this wor is to study and model the correlation between the nonuniformity gain and offset of each detector in the focal plane array. The model, called generalized Gauss-Marov model, is based in an experimental measure of the correlation between gain and offset nonuniformity. To estimate the correlation, we employ the detector parameters estimated by three well nown nonuniformity correction methods: a neural networ approach, a non-linear filtering method and a statistical algorithm. With those estimates the correlation function is obtained and the Gauss-Marov model is then generalized. To validate the proposed model experimental testing with several real infrared video sequences are performed. This paper is organized as follows. In Sections the IR-FPA read out model and the Gauss-Marov model are presented. In Section 3 the generalized Gauss-Marov model is developed. In Section 4 the correlation between gain and offset is measure and validate with sequence of real infrared data. In Section 5 the conclusions of the paper are summarized.. Gauss-Marov Model for The Gain and Offset Nonuniformity In this paper, we model the pixel-to-pixel variation in the detectors responses (nonuniformity) using the commonly used linear model for each pixel on the IR FPA. For the () th detector, the measured readout signal Y (n) at a given time n is represented by the approx-

imate linear relation Y (n) = X (1) (n)t (n) + X () (n) +V (n), where X (1) (n) and X () (n) are the gain and the offset of the th detector respectively and T (n) is the real incident infrared photon flux collected by the detector. The term V (n) represents the read-out noise associate with the th detector at time n. the subscript is omitted with the understanding that all operations are performed on a pixel by pixel basis. In practice, vectors of observations are captured over which the gain and offset are approximately constants. This is an acceptable assumption in applications where the observation vector is short. At a later time, other vectors of observation are captured for which these parameters may have drift. This slow random variation of the gain and offset between consecutive vectors of observation is modeled by a Gauss-Marov process. It is also considered that the gain and offset at th vector-time are uncorrelated. The driver noise for the gain and offset are assumed gaussian individually and mutually uncorrelated among them. Mathematically, the Gauss-Marov state equation is given by X +1 = Φ X + G W, (1) X (1) where X +1 = X () is the state vector at th vector time. Φ = is called the state transition α 0 0 β matrix between the states at th and + 1th time vector. α and β represent the level of drift in the gain and offset between consecutive vectors of observation, respectively. The parameter α (β ) is chosen according to the magnitude of the drift between the gain (offset) at state and the state + 1. G = 1 0 0 1 is a identity matrix that randomly relates the driving noise vector W to the state vector X. W = W (1) W (), W (1) and W () are the driving noise for the gain and offset process, respectively. Finally, to complete the Gauss-Marov model, the observation model for the detector response is given by Y = H X + V, () where Y is the observation vector at th bloc, and the length of observation vector is l. H = T (1) 1...... T (l ) 1 is the irradiance at th bloc. V is the vector of independent, additive temporal noise elements in the th bloc. The main assumption in the observation model is that the input irradiance T (n) is a uniformly distributed random variable in the range T min,t max constituting the range common to all detectors of possible irradiance levels prior to saturation. For a complete development for the gain and offset estimation of each detector in the FPA using an optimized recursion algorithm such that Kalman filter, see 5. 3. Generalized Gauss-Marov Model In this paper, we the model the correlation between the nonuniformity gain and offset as follow: X +1 = Φ X + G W, (3) where the transition matrix between the states at th α ρ and + 1th time vector is modified to Φ = where ρ and δ represent the cross correlation between the gain and offset in consecutive vectors of observations, respectively. The stability of the system (3) is given by λ i (Φ ) < 1, i.e., the product of the eigenvalues must be lower than 1. Note that if we multiplied (3) by X +1 l and applying the expectation value we obtain E X +1 X +1 l T = Φ E X X +1 l T +G E W W l T G T, (4) where l = 0,1,... For a complete nowledge of the generalized Gauss-Marov model we need to determine the G and Φ matrices. In the next sections we present the development of the (4) for two cases: when l = 0 and when l > 0. The first case allow to find the matrix G and the second case, permit to figure out an expression for the transition matrix Φ. 3.1. Solution for G : Case where l = 0 In this case, setting l = 0, using (3) for X +1, and replacing on (4) we obtain E X +1 X T +1 = Φ E X X T Φ T + G E and solve (5) for E W W T we obtain that δ β W W T G T, (5)

E W W T { } = G 1 E X +1 X +1 T Φ E X X T Φ T and assuming that X is a stationary random variable and E X +1 X +1 T = E X X T E X +1 X +1 T Φ E X X T σ = a σ ab σ ab σb Φ T = σ 1 σ 0 σ 0 σ where ( σ0 = (1 αβ δρ)σ ab αβσa ρβσb, σ 1 = 1 α ) σa αρσ a b ρ σb and σ = ( 1 β ) σb 1 βδσ a b δ σa g1, and considering G = g 1 1 we found that E W W T σw = a 0 0 σ wb i.e., W is uncorrelated only if g 1 and g 1 satisfies that g 1 g 1σ 1 σ 0 g 1 σ 0 σ with σ w a = σ 1 g 1σ 0 + g 1σ = 0, and σ w b = σ g 1 σ0 + g 1σ1. For example, if g 1 = 0 then g 1 = σ 0 σ and G = 1 σ 0 σ 0 1, (7) which implies that the W is uncorrelated. Similarly, the expectation of W can be obtained from (3) by means E W = G 1 I Φ E X = G 1 I Φ E X 0. 3.. Solution for Φ : Case where l > 0 In this case, replacing l > 0 on (4) we obtain E X +1 X T +1 l = Φ E X X +1 l T, (8) and expanding (8) on each elements of X, using correlation function notation and solve for Φ we obtain Φ = R X (l) R 1 (l 1), (9) X G 1 T, (6) where R X (l) = RX (1) X (1) (l) R X () X (1) (l) R X (1) X () (l) R X () X () (l). (10) Note that in 5 they assume that gain and offset are uncorrelated, α 0 = α 1 =... = α = α and β 0 = β 1 =... = β = β. Replacing this assumption on (9) results Φ = R X (1) X (1) (l) R X (1) X (1) (l 1) 0 0 R X () X () (l) R X () X () (l 1) = α 0, 0 β (11) this mean that the gain and offset are signals with exponential correlation function, i.e., following a Gauss- Marov model given in (1). Note that (11) correspond a particular case of (9), called the generalized transition matrix. In the next section we present the methodology to obtain an estimation of Φ using (9), i.e., based on estimate the correlation functions R X (l) and R X (l 1) obtained using some NUC published methods. 4. Estimation and Validation of the Generalized Model In this section we obtain an estimation of Φ from (9) using three NUC published methods: the neural networs approach 1,, 3, 4, nonlinear filtering 7, 8 and statistical algorithm 9. Briefly, the neural networs approach use a hidden layer lie a two-point NUC neurons that models the gain and offset of each detectors. This NUC technique is derived using and adaptive least-square (LMS) approach, maing it a comparison between neighboring pixels as new frames of data are input (the retina-lie processing). The second NUC method, nonlinear filtering, is used to obtain an individual estimation of gain and offset with recursive least-square (RLS) based on minimizing the error between a preliminary true scene estimation and the corresponding frame corrected for each pixel. This preliminary estimation of true irradiance is performed by a spatial nonuniformity filtering of readout data reducing the spatial signal resolution that is restored by the RLS algorithm. Both NUC methods, LMS and RLS based, perform the detector parameters estimation in a frame by frame basis. On the other hand, the statistical algorithm assumes that the irradiance at each detector is an uniformly distributed random variable and that the gain and offset are temporally constant within certain blocs of time. This mean that the gain and offset estimation is performed in a bloc of frame basis, i.e., when a new bloc of raw data arrives, the method perform the NU correction by a finite-impulse response filter and then,

the temporal filter is adapted to the changes in the detector parameters and the gain and offset are updated to the next arrivals of raw data. Now, to obtain an estimation for the gain and offset (X (1),X () ) we use real infrared data captured by the 18 18 InSb FPA cooled camera (Amber Model AE-418) operating in 3 5µm range. The sequences were collected at 1 PM with 3000 frames collected at a rate of 30 frames/s and each pixel was quantized in 16 bit integers. Using this raw data, we are able to estimate for each method the detector parameters correlation function R X (1) and R X (0) to obtain a suitable estimation of Φ. Additionally, in this wor we assume that Φ 0 = Φ 1 =... = Φ = Φ. Now, the correlation function between X i and X j with i = 1, and j = 1,, can be estimated by 10 ˆR n X (i) X ( j) (l ) = 1 n n 1 =0 X (i) ()X ( j) ( + l ) (1) where ˆR n X (i) X ( j)(l ) is the estimation for R X (i) X ( j) (1) and R X (i) X ( j) (0) when i = 1,, j = 1, and l = 0,1. In (1) the nowledge of X (1) and X () at each time is required. Since from the three NUC methods we have an estimation of the gain and offset we can express (1) by the following recursive algorithm ˆR n X (i) X ( j) (l ) = n 1 n ˆR n 1 X (i) X ( j)(l )+ 1 n X (i) (n 1)X ( j) (n 1+l ), (13) and then, using (9) we finally obtain the estimated Φ for each method. In summary, to obtain an estimation for Φ we needed to mae the following steps: i) using a real infrared sequence, estimate the gain and offset by three NUC methods, ii) estimate the correlation function for R X (1) and R X (0) using (13) and finally, iii) obtain the estimated Φ for each method using (11). To validate the parameters of each model, we considered two aspect: the NUC performance and the estimation error. For study the NUC capability of each model we use the roughness parameters defined by ρ( f ) = h f 1 + h T f 1 f 1, (14) where h is a horizontal mas, f 1 is the L 1 norm of frame f and represents discrete convolution. In the other hand, to validate the model we use equation () in order to calculate the mean-square error (MSE) between the raw data Y and the estimation Y ˆ ( Φ j ) for each model, given by 11 Table 1. Validation Results for different models Model λ i (φ j ) ρ( f ) MSE Φ 1 7.59 10 9.45 0.8844 Φ 0.353.17 0.8591 Φ 3 0.044.1313 0.1909 MSE = 1 N N 1 Y ˆ T Y ( Φ ) j Y ˆ Y ( Φ j ), (15) =0 where Y ˆ ( Φ j ) = H X = H Φ X 1 is the a priori estimate of Y ˆ based on the model Φ j for j = 1,,3. Following the previous procedure we obtain three generalized models corresponding to each NUC algorithm and they are given by: Φ 1 = 0.506 0.506 0.506 0.506 Φ = 0.8834 0.7384 0.7384 0.8835 Φ 3 = 0.6031 0.5505 0.15 0.1548 (16) (17) (18) where Φ 1, Φ and Φ 3 correspond to the model obtained with the neural networ approach, nonlinear filtering and statistical algorithm respectively. Note that all NUC methods presented are stable (see Table 1) and shows that X (1) and X () are correlated. This mean that the assumption that the gain and offset in consecutive vector of observation are uncorrelated is not valid. Even more, whereas LMS and RLS algorithm exhibit a large correlation, the statistical algorithm shows that the gain strongly is influenced by the gain and offset of the previous state, i.e., we can assume that the offset indirectly is estimate from the gain. This is a very important conclusion because it means that we need the nowledge of only one state to reduce NU. In Table 1, the λ i (Φ j ) for each model, the spatial average of ρ and MSE parameters are presented. We can observe that the best performance is obtained for the statistical algorithm ( Φ 3 ). Additionally, from equations (16),(17) and (18) we can observe that as the magnitude of the drift between the gain (offset) at state and state + 1 is increasingly different (i.e., the correlation between gain and offset is considered), the NU correction and the model estimation are improved. This in addition confirms the assumption of the gain and offset

(a) (b) (c) (d) Figure 1. The 0 th frame from the 1 PM sequence captured with the Amber infrared camera: a) the corrupted raw frame, the corrected frame with b) neural networ approach, c) nonlinear filtering, and d) statistical algorithm. It can be seen by using only the naed eyes that a good NUC is obtained from the three NUC methods. are temporally constant in a bloc of time. Finally, for illustration proposed, the LMS-based, RLS-based and the statistical algorithm were applied to the 0-frame image sequence, and the results is shown in Figure 1. In this figure, Fig. 1(a) correspond to the corrupted frames, Fig. 1(b), Fig. 1(c) and Fig. 1(d) shows the corrected frames with neural networ approach, nonlinear filtering, and statistical algorithm respectively. This figure clearly shows that the statistical method present a good reduction of FPN. 5. Conclusions The correlation between gain and offset nonuniformity in an infrared focal plane array has been analyzed. It was shown experimentally using real IR data that the gain and offset for each pixel in an infrared focal plane array are correlated. Indeed, when the asymmetry is more notable in the generalized transition matrix, i.e., the magnitude of the drift between the gain (offset) at consecutive bloc of frames is increasingly different, the method has shown an improved reduction in both nonuniformity and in the error estimation. 6. ACKNOWLEDGMENTS This wor was partially supported by Grants Milenio ICM P0-049, by FONDECYT 1060894 and Universidad de La Frontera, Proyecto DIUFRO 08-0048. The authors wish to than Ernest E. Armstrong (OptiMetrics Inc., USA) for collecting the data, and the United States Air Force Research Laboratory, Ohio, USA. References 1 D. Scribner, K. Sarady, M. Kruer: Adaptive Nonuniformity Correction for Infrared Focal Plane Arrays using Neural Networs. Proceeding of SPIE. 1541. (1991) 100 109. D. Scribner, K. Sarady, M. Kruer: Adaptive Retina-lie Preprocessing for Imaging Detector Arrays. Proceeding of the IEEE International Conference on Neural Networs. 3. (1993) 1955 1960.

3 S. Torres, E. Vera, R. Reeves, and S. Sobarzo: Adaptive Scene-Based Nonuniformity Correction Method for Infrared Focal Plane Arrays. Proceeding of SPIE. 5076 (003) 130 139. 4 E. Vera, S. Torres: Fast Adaptive Nonuniformity Correction for Infrared Focal Plane Arrays. Journal on Applied Signal Processing EURASIP. (005). 5 S. Torres, M. Hayat: Kalman Filtering for Adaptive Nonuniformity Correction in Infrared Focal Plane Arrays. The JOSA-A Opt. Soc. of America. 0 (003) 470 480. 6 S. Torres, J. Pezoa, and M. Hayat: Scene-based Nonuniformity Correction for Focal Plane Arrays Using the Method of the Inverse Covariance Form. OSA App. Opt. Inf. Proc. 4 (003) 587 5881. 7 F. Torres, S. Torres, and C. San Martn: A Recursive Least Square Adaptive Filter for Nonuniformity Correction of Infrared Image Sequences. LNCS. 3773 (005) 540-546. 8 F. Torres, C. San Martin, and S. Torres: A RLS Filter for Nonuniformity and Ghosting Correction of Infrared Image Sequences. Sequences. LNCS. 45 (006) 464-473. 9 M. Hayat, S. Torres, E. Amstrong, S. Cain, and B. Yasuda: Statistical Algorithm fo Nonuniformity Correction in Focal Plane Arrays. Applied Optics. 38 (1999) 773 780. 10 Charles W. Therrien : Discrete Random Signal and Statistical Signal Processing, Prentice Hall Press, USA, 199. 11 J. Pezoa, M. Hayat, S. Torres, and Md. Rahman: Multimodel Kalman Filtering for Adaptive Nonuniformity Correction in Infrared Sensors. JOSA A. 3 (006) 18 191.