Assessing Reliability Using Developmental and Operational Test Data

Transcription

1 Assessing Reliability Using Developmental and Operational Test Data Martin Wayne, PhD, U.S. AMSAA Mohammad Modarres, PhD, University of Maryland, College Park Presented at the RAMS-04 Symposium Colorado Springs, Colorado Tuesday, January 8, 04, :5 AM

2 Agenda Background Overarching Model Framework Reliability growth in Developmental Testing (DT) Combing DT and Operational Testing (OT) Results Performance Comparisons Conclusions and Future Work

3 Background New military product development generally contains developmental reliability growth testing as design matures to final state Developmental testing environment may not completely represent operational usage environment Operators are different Loads and stresses may be different Reliability currently assessed in single operational test Final mature configuration Generally short and expensive tests 3

4 Problems with Current Operational Assessment Probability of Acceptance Failures T = 400 M = true but unknown reliability of the system and c = maximum number of allowable failures T = total demonstra<on test length True MTBF Short test lengths can lead to flat Operating Characteristic (OC) curves. This often results in test plans in which no failures or a single failure are allowable. Resource constraints and technology maturity may make demonstration infeasible 4

5 Proposed Alternative Assessment Utilize data available from developmental testing within Bayesian framework Account for reliability growth during development Account for differences in test environment/conduct Benefits include: Narrower probability intervals Reduced testing requirements, lower costs, etc. Can use additional data sources in reliability assessment 5

6 Overarching Framework Developmental Test (DT) Data Prior Distribution (Empirical Bayes) Failure Mode Posterior Result System Level Posterior DT Distribution Updated Prior Distribution Operational Test (OT) Data Posterior Distribution Results from DT form prior distribution for OT System Level Posterior OT Result 6

7 Likelihood for DT Reliability Growth Accounts for arbitrary corrective action strategy For each failure mode, i, in the system assume: Failure intensity is constant before and after corrective action n failures in demonstration test time T DT with n occurring before corrective action Corrective action at time v with Fix Effectiveness Factor (FEF) d Likelihood given by l( t i,,t i,,...,t i,ni,n i,n i, λ ) i = d i ( ) n i n i, n λ i i exp λ i v i ( d i )( T DT v i ) ( ) Likelihood allows for arbitrary reliability growth 7

8 Choice of Prior Distribution on Failure Intensity % Total Failure Rate **No<onal data p ( λ) = Γ α λ exp λ α ( α ) Provides inherent connection between failure modes Assumes failure mode failure intensities realized from common Gamma(α, ) distribution Gamma Follows vital-few, trivial-many construct Can use Empirical Bayes to estimate parameters p( λ i n i ) = Γ α n i λ i αn i ( ) v d i ( i )( T DT v i ) ( αn i ) exp λ i v d i ( i )( T DT v i ) 8

9 System Level Result System level estimate: Summing over m total failure modes Take limit as m becomes large n number of failures for each of the m observed failure modes, i. For m observed modes, system level mean is Observed E[ λ s ] = m ( d i )n i v i= i ( d i )( T v i ) where λ B = mα prior mean λ B T Unobserved Estimate includes contribution from unobserved modes 9

10 System Level Posterior Distribution Posterior can be simulated to determine approximate distribution Exactly Gamma if corrective actions are delayed Gamma approximation reasonable for arbitrary corrective action strategies µ = = E [ λs ] [ λs ] E[ λ ] Var s µ λs ~ Gamma α =, Can use mean and variance to develop system level posterior 0

11 Incorporating Operational Data Generally have increased failure intensity in operational environment DT conditions more benign, human factors, etc. Define MTBF as reciprocal of mode failure intensity λ Assume 00γ% decrease (degradation factor) in instantaneous MTBF (/λ) such that λ = γ λ DT ( ) OT Transformed prior found using properties of Gamma distribution λ OT γ = λ DT ( γ ) ~ Gamma α, γ ( ) Scaled prior accounts for reliability degradation

12 Marginal Posterior Distribution Assume n failures in operational test length T Treats degradation factor γ as nuisance parameter Marginal posterior development Compute joint posterior Compute marginal distribution by integrating over nuisance parameter ( ) = λ OT l t OT,,t OT,,,t OT,nOT, n OT λ OT p( λ OT n OT ) = Γ Λ,Γ Use Beta prior distribution for γ ( ) n OT exp λ OT T OT p( γ ) p( λ OT γ )l( t OT,,t OT,,,t OT,nOT,n OT λ ) OT γ p( γ ) p( λ OT γ )l( t OT,,t OT,,,t OT,nOT,n OT λ OT ) λ OT γ Marginal posterior probabilistically accounts for degradation

13 OT Posterior Assessment Posterior mean is scaled mean for Gamma distribution [ ] on slide 9 defined as, where,,,,,, µ µ µ µ µ µ λ = T b a a n F T b a a n F T n n E s Standard posterior mean for Gamma Ratio of Hypergeometric functions accounts for DT/OT degradation 3

14 System Level OT Posterior Distribution Exact system-level posterior can be simulated using Markov Chain Monte Carlo methods Posterior well approximated with Gamma distribution Use approximate Gamma to develop probability intervals p(λ) Can use mean and variance to develop system level posterior λ 4

15 Performance Comparisons Case Ini<al DT MTBF DT Length OT Length Mean Rela<ve Error 0.4 Simulation developed to examine relative error between model estimate and true value Classical Bayes 3 Full error distribution comparisons available in paper Bayesian approach performs better than current methods 5

16 Conclusions/Future Work Use of reliability data from developmental testing provides additional information that increases performance of overall reliability estimate Bayesian probabilistic approach provides flexibility Can utilize multiple information sources Can include additional sources of uncertainty Current/future efforts include: Modeling uncertainty on FEF values Developing prior information from additional data sources Analogous results for discrete systems 6