Evaluation and Comparison of Mixed Effects Model Based Prognosis for Hard Failure

Transcription

1 IEEE TRANSACTIONS ON RELIABILITY, VOL. 62, NO. 2, JUNE Evaluation and Comparison of Mixed Effects Model Based Prognosis for Hard Failure Junbo Son, Qiang Zhou, Shiyu Zhou, Xiaofeng Mao, and Mutasim Salman, Senior Member, IEEE Abstract Failure prognosis plays an important role in effective condition-based maintenance. In this paper, we evaluate and compare the hard failure prediction accuracy of three types of prognostic methods that are based on mixed effect models: the degradation-signal based prognostic model with deterministic threshold (DSPM), with random threshold (RDSPM), and the joint prognostic model (JPM). In this work, the failure prediction performance is measured by the mean squared prediction error, and the power of prediction. We have analyzed characteristics of the three methods, and provided insights to the comparison results through both analytical study and extensive simulation. In addition, a case study using real data has been conducted to illustrate the comparison results as well. Index Terms Degradation signal, hard failure prognosis, mixed effects models, performance comparison. RUL DSPM RDSPM JPM FP, FN MSE AMSE ACRONYMS Remaining useful life Degradation signal-based prognostic model with deterministic threshold Degradation signal-based prognostic model with random threshold Joint prognostic model False positive, false negative Mean squared error Approximated mean squared error NOTATION True failure time for th unit Failure time Manuscript received July 30, 2012; revised November 28, 2012, December 16, 2012, January 28, 2013, and February 05, 2013; accepted February 05, Date of publication April 29, 2013; date of current version May 29, This work was supported by General Motors and the National Science Foundation under Grant Associate Editor: E. Zio. J. Son and S. Zhou are with the Department of Industrial and Systems Engineering, University of Wisconsin-Madison, WI, USA ( json5@wisc.edu; szhou@engr.wisc.edu). Q. Zhou is with the Department of Systems Engineering and Engineering Management, City University of Hong Kong, Kowloon, Hong Kong ( q.zhou@cityu.edu.hk). X. Mao and M. Salman are with the General Motors Research and Development ( xiaofeng.mao@gm.com; mutasim.a.salman@gm.com). Color versions of one or more of the figures in this paper are available online at Digital Object Identifier /TR Mean value of Censored time for th unit Estimated failure time for th unit Acceptable error range Predicted remaining useful life at prediction time instant by using model A Degradation signal at time instant Random measurement noise Vector of fixed-effect parameters for th unit Vector of random-effects parameters for th unit True degradation signal at time instant th unit Any general multivariate parametric distribution Deterministic degradation signal threshold Random degradation signal threshold for PDF of the random degradation signal threshold CDF of failure time Time instant when prediction is made CDF of failure time for th unit obtained by using model A Estimated CDF of failure time for th unit at prediction time instant by using model A PDF of failure time for th unit obtained by using model A Estimated PDF of failure time for th unit at prediction time instant by using model A Survival function for th unit obtained by using model A Estimated survival function for th unit at prediction time instant by using model A Estimated mean RUL for th unit by using model A EstimatedmedianRULfor th unit by using model A Vector of time-dependent regression functions /$ IEEE

2 380 IEEE TRANSACTIONS ON RELIABILITY, VOL. 62, NO. 2, JUNE 2013 Fig. 1. Two major types of prognosis methods. (a) Prognosis based on time-tofailure data; (b) prognosis based on degradation signals. Vector of random coefficients for the mixed effects model Hazard function for th unit Baseline hazard function Vector of time-invariant covariate for th unit Vector of coefficients for time-invariant covariates Vector of coefficients for the degradation signal Expectation under the distribution of estimated remaining useful life Expectation under the distribution of true remaining useful life Prior distribution of random variable Vector of population mean of random vector Variance-covariance matrix of random vector Variance of measurement noise Sample size of historical database Number of replication for Monte Carlo simulation I. INTRODUCTION P REDICTION for the remaining useful life (RUL) of an in-service unit plays a critical role in engineering practice. Asignificant amount of work exists in this field. Two types of data are often used in prognosis: (i) time-to-failure data as described in [1] [6] and the references therein, and (ii) degradation signals as described in [7] [13] and the references therein. Time-to-failure data are used to estimate the probability distributionoffailuretime,asshowninfig.1(a),andthenusedto predict the RUL. The prognosis based on time-to-failure data is more statistically rigorous when a large amount of time-tofailure data can be collected. However, the time-to-failure distribution obtained is for the entire population, rather than an individual unit. Thus, time-to-failure data are more suitable for system design optimization in terms of reliability performance, rather than prognosis on individual units. The degradation signal is a signal selected as the indicator of the system degradation. In the degradation signal based prognosis, a mathematical model is established to describe how the signal evolves, and then the RUL is determined based on the forecast of the degradation signal. This method was originally proposed for highly reliable units, because it is hard to get sufficient time-to-failure data for such units for traditional reliability analysis [7]. A popular degradation signal based method, denoted as the DSPM method, is to use a mixed effects model to describe the degradation path, and use a Bayesian updating approach to obtain the posterior distribution of the degradation signal of a specific unit, based on a prior distribution and the observed degradation path of that specific unit. The RUL is then calculated based on the posterior distribution. In this way, the prognosis for an individual unit can be achieved [8], [14]. In determining the RUL, the assumption of soft failure is required, as shown in Fig. 1(b). Soft failure refers to the failure defined as the degradation signal reaches a pre-specified threshold. However, in many situations, the soft failure assumption may not be valid. For example, an automotive battery fails when it cannot crank the engine and start it. It is known that a vital health indicator of the battery is its internal resistance, which is often used as the degradation signal to predict the RUL of the battery. However, the internal resistance level leading to cranking failure is not crisp. In other words, it is very difficult, and sometimes unreasonable, to define a constant threshold for it [15] [17], [40]. In such cases, the failure is referred to as hard failure, meaning the component keeps working until it breaks down. Despite its importance in practice, much less work exists for hard failure prognosis than for soft failure. One class of methods is to extend the conventional DSPM by using a random threshold [15] [17], [40]. These methods are initially designed for prognosis on the population level. However, the method in [40] can be used for individual unit prognosis through a straightforward extension. We denote this method as RDSPM. Another recent method for hard failure prognosis is the joint prognostic model (JPM) based method [18] [21]. The joint prognostic framework is composed of two stages: (i)theoffline modeling stage using the historical data from a large number of units, and (ii) the online prognosis and model updating stage using the newly collected signal from an in-service unit. In the JPM, a probabilistic hazard structure that depends on the degradation signal is used to describe the failure mechanism. No threshold is assumed in this method. Due to the popularity of DSPM, and the relatively limited works for hard failure prognosis, people tend to adopt the DSPM in practice even when the soft failure assumption does not exactly hold. Thus, it is highly desirable to evaluate under what conditions this practice is reasonable, and under what conditions exact hard failure models should be used. However, to the best of our knowledge, a comprehensive evaluation and comparison of the prognosis performance for hard failures are not available in the current literature. The goal of this paper is to provide such systematic evaluation and comparison for the three methods,dspm,rdspm,andjpm. Because prognosis is relatively under-developed compared to diagnosis, more focus on the prognosis has been on developing a new modeling scheme and methods [22], rather than

3 SON et al.: EVALUATION AND COMPARISON OF MIXED EFFECTS MODEL BASED PROGNOSIS FOR HARD FAILURE 381 on performance evaluation and comparison. A few performance analysis papers for prognostic methods are available in the survival analysis field in biostatistics [23] [27]. In those works, the most common ways to evaluate the prognostic model are the receiver operating characteristic (ROC) curve, and the corresponding area under the curve (AUC), yet their focus is not evaluating the RUL prediction accuracy. Rather, those works focus on determining the degradation signals significance [23], [24], [27]. The comparison studies are conducted using various types of different signals and their combinations, not among different models. Our work is different because our primary interest is not comparing different signals but evaluating and comparing different prognostic models in terms of their prediction accuracy. In the engineering field, several quantitative metrics have been proposed to evaluate the performance of prognostic algorithms [22]. To make the comparison relevant to engineering practice, appropriate performance metrics should be used. In this work, we select the mean squared error (MSE)ofthepredictedRUL and the false positive and false negative rate (FP/FN) as the prediction metrics. The definition of MSE is straightforward as given in (1), and it is one of the most widely used performance metrics in engineering [28]. The definitions of FP/FN rate and a relevant term, Power, are given in (2). These probabilities should be evaluated under the true RUL distribution. Fig. 2 illustrates this definition graphically. We use FP/FN rate as the performance metric because it is highly relevant to engineering practice. In reality, it is meaningless to predict the occurrence of the failure to be at an exact time point because it is always zero. Thus, the prediction of failure is often reported as a time interval in practice. The definition of FP/FN rate captures the characteristics of such interval prediction, and evaluates the probability of the correctness of the prediction. As shown in Fig. 2, the FN rate is actually the misdetection rate, while the FP rate is the false alarm rate. These two probabilities are critical to customer satisfaction and the maintenance cost. The MSE and FP/FN rate have very different characteristics. The MSE purely depends on the point estimate of the RUL, while the power depends on the shape of the true RUL distribution. In this paper, both metrics will be used in the performance evaluation and comparison. There are other performance metrics recently developed for prognostic models such as the performance index. The index is defined as a binary metric that evaluates whether the prediction accuracy at specific time instant falls within a specified -bound [22]. This metric has been used for visual evaluation of the prediction accuracy. Despite its intuitive interpretation, it is not suitable for evaluating multiple (1) (2) Fig. 2. Illustrative definition of FP/FN. units. For example, we need to show multiple plots to illustrate the prediction accuracy for different units [39]. The prediction power used in this paper is in fact conceptually similar to the index, but can be applied more easily to multiple units. Besides the index, other recently proposed performance metrics are RUL precision index, RUL accuracy-precision index, and RUL on-line steadiness index, etc. [35]. The first two can be obtained by the RUL confidence intervals computed at time instant, and the last one can be obtained by the standard error or the variance of the RUL estimate. In this study, the standard deviation and the confidence intervals of the RUL estimates will be provided. Thus, these metrics can be easily obtained as well. The rest of this paper is organized as follows. Section II addresses the detailed modeling procedure and structure of the three models, DSPM, RDSPM, and JPM. An analytical study is conducted in Section III for obtaining useful insights, and an extensive simulation comparison with more complicated models is followed in Section IV. A discussion section is provided in Section V to compare the methods under failure models other than hard failure. A case study has been done using real data to further illustrate the comparison result in Section VI. Finally, Section VII summarizes and concludes the paper. II. REVIEW OF DSPM, RDSPM, AND JPM A. Degradation Signal Based Prognostic Model With Fixed Threshold (DSPM) The modeling and parameter estimation for the DSPM have been well-studied [7]. In this method, the degradation signal path for the th unit is often expressed as a mixed effect model where is assumed to be the same for all units, and is varying from unit to unit. In this way, the model incorporates both the common characteristics among the population, and the individual characteristics for each unit. isassumedtofollow a multivariate normal distribution with unknown parameters to be estimated; however, any general multivariate distribution can be used as well. Measurement error is often assumed to follow a normal distribution with mean zero, and it can also be assumed to be generated by other stochastic (3)

4 382 IEEE TRANSACTIONS ON RELIABILITY, VOL. 62, NO. 2, JUNE 2013 processes such as the Brownian motion [8]. If we let the failure time, the distribution of is denote consists of two parts. The first part is a mixed-effect degradation signal model, (4) (8) In most cases, the distribution function of does not have a closed form due to the complexity of and.however, under certain simple model, and a relatively simple parametric probability density function for random-effect parameters, the closed form can be obtained. The mean RUL can be geometrically described as the area under the survival curve. The survival function (also called reliability function), defined as, can be easily obtained based on the cumulative distribution of. By integrating the survival function with respect to time, the mean RUL from DSPM can be estimated as. The detailed mathematics is shown in Section III for a linear degradation model. B. Degradation Signal Based Prognostic Model With Random Threshold (RDSPM) In this method, it is assumed that the threshold is a random variable following a certain distribution. Here, the failure threshold is assumed to follow a Gamma distribution, and is estimated from historical data. Under this assumption, we can formulate the failure time, often referred to as the first hitting time, as With this degradation path model, we can establish the model for time-to-failure data as the second part, where is mathematically defined as. This model can be viewed as an extension of the popular Cox proportional hazard model [32] that is widely used in reliability analysis [6]. Note that,, are the same for different units from the same population. They represent population characteristics while the individual specific information is contained in and.inthis way, the degradation signals and the time-to-failure data are integrated. The detailed parameter estimation procedures are given in the Appendix. Given the hazard function, the system survival function can be obtained as (9) (10) For the RUL prediction at time instant, we need to truncate the distribution at. This truncation can be done by replacing with, and adjusting the domain of integration accordingly. Further extending (5), the CDF of the RDSPM for the th unit at time can be expressed as (6) Because time instant can be obtained as (5) (6), the mean RUL at Also, the median residual life can be obtained by. C. Joint Prognostic Model (JPM) The JPM estimates the RUL distribution for individual units using both the time-to-failure data and the degradation signals. It (7) Once we have the survival function, the mean RUL estimation procedure is the same as that of the DSPM. An illustration is given in the following section, with a linear degradation path. III. PERFORMANCE EVALUATION AND COMPARISON UNDER LINEAR DEGRADATION MODELS In this section, we shall investigate the failure prediction accuracy of the DSPM and the JPM. Comparing with DSPM, RDSPM has one more layer of integration with respect to the threshold, making it analytically intractable. Hence, we leave the detailed discussion of the RDSPM to the numerical study section. As a first step, we analyze and compare JPM and DSPM to see how both models behave with respect to different parameter levels. In this study, the degradation signal is assumed to be a mixed effects model with linear signal propagation, as shown in (11). (11) The random coefficient varies from unit to unit, and it is assumed to follow the normal distribution ;and follows the normal distribution. This assumption is common in literature [7], [8]. The linear degradation signal is simple, yet it is still meaningful for both research and real-life applications. It has been used in many applications [8], [30]; and furthermore, the linear degradation signal model can handle the

5 SON et al.: EVALUATION AND COMPARISON OF MIXED EFFECTS MODEL BASED PROGNOSIS FOR HARD FAILURE 383 exponential degradation signal with a logarithm transformation [8]. With this linear degradation model, the hazard function of the system is assumed to be (12) In this model, the baseline hazard is assumed to be linear, and is the baseline coefficient. Under (11) and (12), we can derive the survival function at time instant as and (17) without give the prediction power, and AMSE for a specific unit. To obtain the overall performance, we need to take the expectation over the entire population (18) (19) Note that the is taken over the distribution of the random coefficient parameter. and the probability density function of the residual life is (13) A. Residual Life Prediction for the DSPM Given observations of the degradation path, denoted as, the mean RUL prediction at time based on the degradation model is given as (14) (20) where. The derivation of (14) is straightforward, and omitted here. Assume we have a RUL prediction at time for the th unit, as. Then, for the th unit, the prediction power, and the MSE of the prediction can be obtained based on (1), and (2) as and (15) (16) Note that, because of the observation noise, the RUL prediction is a random variable. In (15), the inner probability is taken under the true distribution of, while the outer expectation is taken under the distribution of.in(16),weneed to take the expectation on both and. A simple derivation shows that can be decomposed as. Note that the second term is determined solely by the characteristic of the system, and not by the accuracy of prediction. Thus, we may drop the second term, and simplify (16) to the approximated MSE (AMSE) expressed below. (17) In our study, we also noticed that the impact of the randomness of is not significant when the observation noise is moderate. Thus, the expectation over can be further dropped for both AMSE in (17) and power in (15) by assuming as a fixed value. By dropping the expectation over, the number of iterations to compute AMSE and power will reduce significantly. In the following studies, AMSE and the power without expectation over are used to save computation time. Equations (15), for, and the subscript represents the DSPM. To predict the mean RUL, we need to predict the future degradation signal path after time. The estimation procedure for the parameters of the mixed effects model in (11) has been well-studied [20], [21], [30]. In this section, we assume that a large amount of historical data is available, and we can get accurate estimates of the population parameters in this model, including the parameter, the distribution of with its own parameters of and, and the noise variance.theestimation errors in those terms are hence ignored here [33]. Thus, to predict the degradation signal path, we only need an estimate of the value for the specific th unit, and determine the threshold value. When the DSPM is needed to be applied without a pre-specified threshold,thevalueof is often taken as, where is the median life of the population [15]. In this study, we will follow the same way. The estimation of the value of is more involved. Unlike the estimation procedure for,the needs to be estimated by the restricted maximum likelihood (REML) approach [30]. After obtaining all the population-wise estimates, we need to update the parameters. The Bayesian updating method has been a popular method to estimate the value of given the observations of degradation and its population distribution as prior. Using the Bayesian method, we first obtain the posterior distribution of as where a normal distribution (21) is the density function of given, and is given as. After multiple steps of

6 384 IEEE TRANSACTIONS ON RELIABILITY, VOL. 62, NO. 2, JUNE 2013 algebra as in [8], we can find the posterior distribution of a normal distribution,where as Using this predicted hazard rate, we can get the survival function as (22) with. Using the posterior distribution of, we can get the predicted cumulative RUL distribution for unit as (23) where,and is the cumulative standard normal distribution. With (23), we can obtain the predicted probability density function of and the predicted survival function for unit as below, and the derivations are shown in the Appendix. (24) (25) Given the predicted survival function, the expected mean RUL, and the expected median RUL of the unit can be obtained as,and, respectively. In practice, both of them have been used as the prediction of RUL for the unit [2], [31]. Note that for a given unit with observation of its degradation signal until time, both and are fixed numbers. However, due to the measurement noise, even for the same unit, observed degradation signals will be different in different replications. Thus, and are in fact random variables. If the measurement noise is small as in most practical cases, the variation will be small. Using the RUL estimate, the performance metrics can be computed. B. Residual Life Prediction for the JPM Given observations of the degradation path,, the predicted hazard function under the JPM at time instant is given as (26) where the expectation is taken under the posterior distribution of, which can be obtained using the same Bayesian updating procedure as that described in Section III-A. Because the posterior distribution of is a normal distribution, follows a lognormal distribution. Based on this result, the expected value of the hazard rate function can be re-formulated as (27) (28) where the subscript indicates that the predicted survival function is under the JPM. Because the density function of the RUL is related to the hazard function and the survival function as, we can get the prediction of the density function under the JPM framework as (29) Given the predicted survival function, the expected mean RUL, and the expected median RUL of the unit can be obtained as,and, respectively. C. Comparison and Analysis of Prediction Accuracy In this section, we will evaluate and compare the prediction accuracy of the DSPM and the JPM. As mentioned in the previous section, both the expected mean RUL and the expected median RUL could be used as point estimates of the RUL. Here we will use the median RUL, i.e., and, as the expected RUL prediction for its simplicity. In this study, we set the baseline hazard coefficient as, acceptable error range as 10, and the standard deviation of the measurement error as is assumed to follow,andthe fixed parameter is. From the prediction method introduced above, we can see that the parameters which affect the performance of each prognostic model are the time instant of the prediction, the variance of random parameter,andthe measurement noise level. The time of prediction determines how much information the model can get from observations. The variance of a random parameter indicates how each individual differs from others. The measurement noise level influences the accuracy of the Bayesian update algorithm. We will investigate these parameters and see how the JPM and DSPM perform. It is worth mentioning that we have conducted many case studies under various parameter settings besides this specific setting. In all those cases, we found the conclusions to be consistent. Thus, this parameter set is used as an illustrative example. 1) Influence of Prediction Time : Table I shows the performance evaluation results for these two expected RUL prediction

7 SON et al.: EVALUATION AND COMPARISON OF MIXED EFFECTS MODEL BASED PROGNOSIS FOR HARD FAILURE 385 TABLE I COMPARISON UNDER THE SIMPLE MODEL STRUCTURE WITH DIFFERENT LEVEL OF TABLE II COMPARISON OF PERFORMANCE FOR A SPECIFIC UNIT methods. The power and the AMSE are computed and averaged using (18) and (19), and and are used for the expected RUL prediction for the DSPM and JPM, respectively. From TableI,wecanfind that the prediction accuracy gets better at the later stages of the unit s life, which is not surprising because a later prediction means more observations on the degradation path, and hence more information. We can also see that the performances of JPM and DSPM for this simple model are comparable, although JPM is slightly better. According to Table I, the variance of the prediction from DSPM is larger than that of the JPM, which means the performance of the DSPM varies more among different units. On the other hand, the performance of the JPM is more robust. 2) Influence of the Variance of Random Parameter : Larger values of means that the parameter has a larger variation. In this section, all the comparisons are made under the condition of. Table II shows three different cases, and the corresponding performances of two methods. The mean value of the random parameter is set to 0.1. Both and are within the range. As we can see from Table II, when the parameter of the specific in-serviceunit deviates from its mean value, the AMSE of DSPM gets significantly larger. On the other hand, the JPM is quite stable in all three cases in terms of the AMSE. This result indicates that the variance of the random parameter has a negative relationship with the performance of the prognostic models, especially for the DSPM. Table III further confirms this observation. Focusing on the last two rows of Table III, we can see the obvious deteriorating trend of the DSPM performance. The AMSE increases remarkably as the variance in increases. For the JPM, the performance is relatively stable. This phenomenon can be explained by the definition of.the is estimated by ; thus the DSPM can estimate the RUL accurately when the th in-service unit behaves similar to the population average. 3) Influence of the Measurement Noise : As mentioned before, in practice, the measurement error of the sensory device is reasonably small. Normally, it has no significant impact on the prediction accuracy. However, to make the comparison more complete, it is necessary to take a closer look at the measurement noise and its impact on both models. The updating algorithm depends on the observations of the degradation signal path; therefore, if the measurements are corrupted or not accurate, the overall prediction performance will be affected. Table IV summarizes the comparison results against different levels of measurement noise. Note that unreasonably large measurement error is not realistic. Both models are getting worse as measurement noise gets larger. Interestingly, the magnitude of the increment of AMSE from JPM is larger than that from DSPM. This result suggests that the JPM is more sensitive to the measurement noise. Please note that the AMSE and power reported in Table IV has been obtained without omitting in (15) and (17) because cannot be ignored for the analysis of the measurement noise level.

8 386 IEEE TRANSACTIONS ON RELIABILITY, VOL. 62, NO. 2, JUNE 2013 TABLE III COMPARISON UNDER THE LINEAR MODEL STRUCTURE WITH DIFFERENT LEVELS OF TABLE IV COMPARISON UNDER THE SIMPLE MODEL STRUCTURE WITH DIFFERENT LEVEL OF IV. NUMERICAL STUDIES FOR QUADRATIC DEGRADATION MODELS If the degradation signals and the hazard function do not have linear forms, it is difficult to obtain a closed-form estimation of the survival function. In this section, we conduct a series of simulation studies to evaluate and compare the prediction accuracy using a more general quadratic degradation model structure. A. Model Structure and Simulation Procedure In this study, we assumed that the true degradation signal path is modeled as a mixed-effects model given as (30) where isassumedtofollowamultivariate normal distribution. Besides the degradation signal, the hazard function is assumed in a more complex form as (31) where is the Weibull baseline hazard function, and is a time invariant covariate for the th unit which is assumed to be either 0 or 1, indicating two different groups of units. In reality, the parametric form of the true degradation signal model is usually unknown. Thus, we use a quadratic polynomial form to model the degradation path instead, as (32) and we assume that the hazard function is in the same form of thetruemodelas (33) In this simulation study, we set the true parameters as in Table V. Note that some parameters have been assigned small values to ensure that units will survive for a reasonably long time. For simplicity, we shall indicate all the population and individual parameters for the th unit as a vector. Please note that we have assumed a quadratic signal path model, while the true path model is not of that form. As we can see from Table V, we have set the value to zero. This setting indicates that the true failure time is not affected by,andis mainly dependent on the degradation signal with a strong positive relationship. In the simulation study, we have conducted the comparisons of DSPM, RDSPM, and JPM methods. All the true parameters are unknown to the three models, so have to be estimated. This condition makes the numerical study more realistic. The simulation consists of two parts. The firstpartistogenerate a historical sample of units with degradation signals and failure times based on the assumed true model, and then the population parameters in (32) and (33) are estimated based on the historical data. The key steps of the simulation are summarized in Tables VI and VII. Based on the simulated historical data, we can estimate the population parameters. Also, step [g]

9 SON et al.: EVALUATION AND COMPARISON OF MIXED EFFECTS MODEL BASED PROGNOSIS FOR HARD FAILURE 387 TABLE V TRUE PARAMETERS FOR THE SIMULATION STUDY TABLE VI SIMULATION ALGORITHM FOR GENERATING HISTORICAL DATA (PART 1) in Table VII is omitted because we have decided to omit in (15) and (17), except for the analysis of measurement noise level. In this study, we use the two-stage method for estimating the parameters for DSPM adopted in [7], [8], and the population parameter estimation technique for the JPM that can be found in [21]. For the RDSPM, we have an additional estimation step for the threshold distribution, which is assumed to be the Gamma distribution. Using historical failure records, we can fit the data to the Gamma distribution [33]. In the second part of the simulation, we first generate multiple instances of degradation signals and failure times according to the true model for unit.after that,wemaketherulpredictionusing the available degradation signals up to time. Then the calculation of the performance metrics is followed. The detailed simulation procedure isshownintablevii.thesubscript represents the unit for the prediction. B. Comparison Results 1) Performance Evaluation and Comparison With Large Sample Size: We used,and. In statistical inference, the size of 500 is often considered as a large sample size, and the variation due to the sample uncertainty will be small. Thus, in this simulation, we run part 1 of the simulation only once. Then the estimated DSPM, RDSPM, and JPM based on the historical data are used to evaluate the prediction accuracy. Table VIII shows the performances at different time points of prediction. At, the models have no information about the particular unit. Therefore, their performance reflects only the population behavior extracted from the historical data. Once individual specific information has been obtained, the performance gets better. It is interesting to note that at the early prediction stage, such as, or 20, the mean prediction power of the JPM is worse than that of DSPM, but it gets significantly better at the later stage. This result is because the JPM has an exponential structure in the hazard function, and hence it is more vulnerable to the accuracy of the parameter estimates, which depends largely on the amount of observation data. We also note that, although the RDSPM considers the randomness of the threshold, it performs similar to the DSPM, and worse than the JPM. We also investigated the influence due to the variance of the random coefficient and measurement noise. To analyze the impact of the variance of random coefficient,wehaveincreased it by multiplying a scaling constant. The results are obtained by changing from1to3where.for the measurement noise, we chose three different levels as well: 0.001, 0.01, and 0.1. Note that the measurement error of 0.01 is the same as that in Table VIII. The prediction time is fixed at 40. Table IX, and Table X summarize the results, respectively. Based on Table IX, we can see that the variance affects the overall performance, and the DSPM and RDSPM are more vulnerable to large variance. The AMSE from those two models consistently increasing as the variance increases. According to Table X, the performance of all three models gets worse as measurement noise increases. However, after a certain level of measurement noise, the ASME of the JPM gets worse significantly. The measurement noise is highly related to the accuracy of the estimated parameters. If the observed signals were corrupted or too noisy, it could be problematic for all three models, especially for the JPM due to its complex structure. This problem is a potential weakness of the JPM. Please note that those AMSE and power values reported in Table X were obtained including in (15) and (17) because we are considering the influence of measurement noises. 2) Impact of Sample Size of the Historical Dataset: In this section, we investigate the impact of the sample size of the historical dataset on the prediction accuracy. Table XI shows the performance with, and Table XII shows the performance at for different sample sizes. Because the purpose is to investigate the impact of sampling uncertainty, we need to repeat both Part 1 and Part 2 of the simulation algorithm. In other words, in the RUL prediction for each individual unit, we regenerate the historical database, and re-estimate the DSPM, RDSPM, and JPM model parameters. From these results, we can see that the JPM is more sensitive to the sample size. This result is not surprising because more parameters need to be estimated in the JPM. However, its performance is still better than the DSPM at the later stage of the prediction in terms of both the prediction power and the AMSE. In Table XII, an obvious trend can be found. As the sample size increases, the power increases. The AMSE tends to decrease because of the smaller estimation error, and the 95% confidence interval for the expected mean RUL gets narrower due

10 388 IEEE TRANSACTIONS ON RELIABILITY, VOL. 62, NO. 2, JUNE 2013 TABLE VII SIMULATION PROCEDURE FOR RUL PREDICTION AND PERFORMANCE EVALUATION (PART 2) TABLE VIII COMPARISON RESULTS to the reduced uncertainty in the parameter estimation. Notice that, even in the extreme case, the JPM is still better than the DSPM and the RDSPM in terms of both power and AMSE. V. DISCUSSION So far, our study has focused on the performance comparison under the hard failure case. As we can see from those results, the JPM performs well compare to the DSPM and the RDSPM. To make the comparison more comprehensive, we will extend the scope of the study in this section to obtain more insights. A. Comparison Under the Soft Failure Case Soft failure is a common type of failure in some applications. Most times, the threshold defining the soft failure is set by experts or reliability engineers or both. The soft failure has a very different nature from the hard failure due to the way the failure is defined. In this section, we have repeated the comparison study, and analyzed two different cases: (i)softfailurewith

11 SON et al.: EVALUATION AND COMPARISON OF MIXED EFFECTS MODEL BASED PROGNOSIS FOR HARD FAILURE 389 TABLE IX IMPACT OF VARIANCE OF RANDOM COEFFICIENT TABLE X IMPACT OF MEASUREMENT NOISE fixed threshold, and (ii) soft failure with random threshold. Case (ii) can be viewed as a generalization of (i). In this particular setting, we assumed that the random threshold in case (ii) follows a Gamma distribution. The DSPM, and the RDSPM are the methods specifically designed to handle case (i), and case (ii), respectively. Thus they are expected to perform well. The true degradation signal path is still assumed as that in (30), and unknown to all models. Table XIII summarizes the results. It can be seen that the DSPM and the RDSPM indeed outperform JPM significantly. B. Hard Failure Affected by Factors Other Than Degradation Signal It is generally known that the hard failure is more difficult to predict than the soft failure. Because soft failure depends solely on the degradation signal and its threshold, the prediction will be good as long as we have an accurate signal path model. However, it is hard to establish an accurate prognostic model for hard failure. The hard failure not only depends on the degradation signal, but possibly on some other factors as well. The flexibility of the JPM framework allows those factors to be easily incorporated. Table XIV illustrates the idea. The left columns are the results from a numerical study using simulated data with a factor other than the degradation signal affecting the failure time. Specifically, the left four columns are the results from the simulation study with,whichisdefined in (31). The right columns are the case where the hard failure time is determined by setting. It is evident that JPM has a clear advantage over the other two models which solely depend on degradation signal. C. Key Findings Based on our extensive experiments, and the comparisons conducted in Sections III, IV, and V, we can summarize some key findings. 1) The threshold-based methods (the DSPM, and the RDSPM) are easy to implement, and quite effective for soft failure cases. The idea of these two models is straightforward. They keep track of the degradation signal, and predict the time when the signal first hits the given threshold. For hard failure cases, these methods lack the ability of modeling the random correspondence

12 390 IEEE TRANSACTIONS ON RELIABILITY, VOL. 62, NO. 2, JUNE 2013 TABLE XI PERFORMANCE COMPARISON WITH SMALL HISTORICAL SAMPLE SIZE TABLE XII COMPARISON BETWEEN DIFFERENT SAMPLE SIZE AT between the failure and the degradation signal. Particularly, they are not preferred when the system failure is also impacted by factors other than the degradation signal. 2) The JPM relates the degradation signal to the failure through a probabilistic hazard structure, allowing it to handle hard failures. However, the degradation signal lies within an exponential part of the hazard function, making the JPM more sensitive to the accuracy of parameter estimates. Therefore, if the prediction is made at the early stage when observation data points are scarce, or if the signal noise level is high, the performance of the JPM is unreliable compared to the other two methods. 3) The JPM contains a baseline hazard and possibly other covariates which are not directly related to the degradation signal. This condition makes the JPM more flexible, but with the price of estimating more parameters. Thus, if the sample size of the historical data is small, the performance of JPM is unreliable. To make the JPM perform well, it is essential to have a reasonably large amount of historical data. VI. CASE STUDY BASED ON REAL DATA In this section, we used real data to evaluate both methods. The data contain the fail-to-crank times of 14 automotive batteries. The degradation signal is the internal resistance, which has been commonly used in battery life prediction [21]. These data were collected under the accelerated testing condition. The 14 batteries were made by two different manufacturers: eight were made by A, and the rest were made by B. A time-independent covariate was added to the model to account for their potential difference. The degradation signal paths of the 14 batteries and the histogram of the resistances at the failure time are shown in Fig. 3, which clearly shows that the failure does not depend on a fixed threshold. We used the same model structure shown in (32) and (33) to predict the failure time. For the DSPM, we used the mean resistance value at the week before the failure as the threshold. A quadratic form of the mixed effects model was selected to model the resistance signal propagation. Fig. 4 is the graphical illustration of the model fit. More rigorous goodness-of-fit measures are provided in Table XV, with statistical significance of

13 SON et al.: EVALUATION AND COMPARISON OF MIXED EFFECTS MODEL BASED PROGNOSIS FOR HARD FAILURE 391 TABLE XIII COMPARISON RESULTS UNDER DIFFERENT FAILURE TIME GENERATING MECHANISM Under the soft failure scenario, the RDSPM is identical to the DSPM TABLE XIV COMPARISON RESULTS WITH OTHER FACTORS AFFECTING THE FAILURE TIME Fig. 3. Resistance propagation of 14 batteries, and the histogram of resistance at failure. each estimated parameters. For the goodness-of-fit measure, the R-square values specifically designed for mixed effects model were used [41], [42]. Due to the limited sample size, we adopted a leave-one-out cross-validation approach to evaluate the performance. The prediction power is defined here as the (number of correct predictions)/14, and the MSE is evaluated over 14 predictions using (1). Correct predictions means that the true RUL is located within the acceptable error range of the predicted mean RUL. The acceptable error range was set to 2 weeks. The earliest failure time happened at the 6th week, thus we used a prediction time from 0 to 5. The final results are presented in Table XVI with their corresponding standard errors. We can see that the JPM performs better than the DSPM and RDSPM at the later stage of the prediction. The MSE of the JPM decreases remarkably as prediction time increases. However, the MSE of the DSPM and RDSPM has an increasing trend. Thistrendisanindicationthatthere might be other factors affecting the actual failure time which cannot be explained by the degradation signal. In real applications, it is almost impossible to identify all of the factors affecting the hard failure time. As we investigated in the discussion section, all of them, DSPM, RDSPM, and JPM, have their own advantages and limitations. However, as we can see in this particular case study with real data, the JPM performs better for the hard failure prognosis with current settings.

14 392 IEEE TRANSACTIONS ON RELIABILITY, VOL. 62, NO. 2, JUNE 2013 Fig. 4. Illustration of model fit using mixed effects model. TABLE XV GOODNESS-OF-FIT MEASURES AND SIGNIFICANCE TEST FOR ESTIMATED PARAMETERS TABLE XVI CASE STUDY COMPARISON RESULT VII. CONCLUSION In this paper, we evaluated and compared the prognosis accuracy in terms of both the prediction power and the AMSE for models under two major categories: the threshold-based methods (the DSPM, and the RDSPM), and the JPM. The DSPM is a popular prognosis method in the field of reliability engineering when there is a well-defined threshold for the degradation signal. The RDSPM is an extended form of the DSPM which accommodates the random threshold, allowing it to handle hard failures. In JPM, the predicted degradation signal is related to the system failure probability through a hazard function. Under the hard failure case, the comprehensive studies show that the JPM usually outperforms the threshold-based methods (DSPM and RDSPM) when we have a relatively large set of historical data, and the prediction is made at the later stage. Due to the fast development of sensing and information technology, it has become easier to collect large historical dataset. Thus, we envision that the JPM will find more applications in the future. On the other hand, when the historical data are scarce, or there is a given failure threshold for the degradation signal, the DSPM and the RDSPM still possesses an advantage. In summary, the threshold-based methods are suggested for soft failure prognosis. They are specifically designed and motivated by the soft failure case; thus they outperform the JPM under the soft failure scenario. However, for the hard failure prognosis, the DSPM and the RDSPM lack the flexibility of modeling the random correspondence between the failure and the degradation signal. Particularly, the threshold-based methods are not preferred when the system failure is also impacted by factors other than the degradation signal. Because of the flexible model structure of the JPM, it can handle the hard failure case more appropriately. However, the data availability plays an important role in the prognosis performance for JPM. Thus, in general, threshold-based methods are preferred at the early stage of prognosis even for the hard failure prediction. However, at later stage of prognosis, the JPM is preferred. We have also identified potential directions to extend the current work. In this work, we only compare two categories of prognosis methods that are based on the mixed effects degradation model. Some other interesting methods are not included to limit the scope of this paper. For example, the stochastic filtering-based method is a very important class of methods for prognosis. In those methods, the degradation states are assumed unobservable. Filtering techniques such as Kalman filtering and particle filtering are used to estimate the states, and then the RUL distribution is estimated based on the state estimates [35] [38]. Although it is often not easy to physically interpret the unobservable states and assign a linking function between the states and the observation, the filtering based methods are quite effective and computational efficient in many scenarios [14]. It would be interesting to rigorously compare and evaluate these filtering based methods. Another future direction is to

15 SON et al.: EVALUATION AND COMPARISON OF MIXED EFFECTS MODEL BASED PROGNOSIS FOR HARD FAILURE 393 determine the prediction interval which is related to the choice of acceptable error range. In practice, people tend to use the center-based interval prediction, using the mean or median value and the, or the equal-tail-probability prediction interval (e.g., confidence interval). For the online prognosis of individual units, those interval estimates, often times, do not yield the maximum prediction power. We will investigate along this line, and hope to report the results in the future. APPENDIX Parameter Estimation of the JPM: Basedontheconditional statistical independence assumption, the observed data likelihood for the th unit in the database can be written as (34) where represents the survival model, corresponds to the degradation model, and is a multivariate normal distribution with parameters. Also, is the event time (the unit either died at time or censored at time without knowing its actual time of death), and is an event indicator which takes a value of either 0, or 1 to indicate the unit has censored, or died, respectively. Here, is the set of all the parameters. The three components in (34) are defined as (39) where, is pre-specified threshold value for degradation signal, and indicates the cumulative standard normal distribution function. The failure distribution should be truncated because the time cannot be smaller than the prediction time instant. (40) (41) where,. The failure distribution is the first derivative of cumulative failure distribution. where. (42) (35) (36) ACKNOWLEDGMENT The authors would like to thank the editor and the referees for their valuable comments and suggestions. REFERENCES (37) where,, and. Also, denotes the baseline hazard rate, and is the vector of association parameters linking with the hazard function. Inthesamemanner, is the parameters linking the degradation signal with the hazard rate. To obtain the parameter estimates using the likelihood function introduced in (34), should be maximized. A more detailed estimation procedure can be found in [20]. Derivation of the Predicted p.d.f of : The cumulative failure distribution can be defined as below. (38) [1] W. Q. Meeker and L. A. Escobar, Statistical Methods for Reliability Data. Hoboken, NJ, USA: Wiley, [2] J.D.Kalbfleisch and R. L. Prentice, The Statistical Analysis of Failure Time Data. Hoboken, NJ, USA: Wiley, [3] J. F. Lawless, Statistical Models and Methods for Lifetime Data. Hoboken, NJ, USA: Wiley, [4] M.RausandandA.Høyland, System Reliability Theory: Models, Statistical Methods, and Applications. Hoboken, NJ, USA: Wiley, [5] A.K.S.Jardine,D.Lin,andD.Banjevic, Areviewonmachinery diagnostics and prognostics implementing condition-based maintenance, Mechan. Syst. Signal Process., vol. 20, no. 7, pp , [6] Y. Yuan, S. Zhou, C. Sievenpiper, and K. Mannar, Event log modeling and analysis for system failure prediction, IIE Trans., vol. 43, no. 9, pp , [7] C.J.LuandW.Q.Meeker, Using degradation measures to estimate a time-to-failure distribution, Technometrics, vol. 35, no. 2, pp , [8] N. Gebraeel, M. Lawley, R. Li, and J. Ryan, Residual-life distributions from component degradation signals: A Bayesian approach, IIE Trans., vol. 37, no. 6, pp , [9] N. Gebraeel, Sensory-updated residual life distributions for components with exponential degradation patterns, IEEE Trans. Autom. Sci. Eng., vol. 3, no. 4, pp , [10] K. Goebel and P. Bonissone, Prognostic information fusion for constant load systems, presented at the 8th Int. Conf. Inf. Fusion, 2005.