Treatment of Expert Opinion Diversity in Bayesian Belief Network Model for Nuclear Digital I&C Safety Software Reliability Assessment 20 International Topical Meeting on Probabilistic Safety Assessment and Analysis (PSA 20) Pittsburgh, PA, USA, September 24-28, 20 Ming Li 1, Sang Hun Lee 2, Seung Jun Lee 3, Tsong-Lun Chu 4, Athi Varuttamaseni 4, Meng Yue 4, Jaehyun Cho 5, and Hyun Gook Kang 2* 1 U.S. Nuclear Regulatory Commission, Washington, DC, USA 2 Department of Mechanical, Aerospace, and Nuclear Engineering, Rensselaer Polytechnic Institute, Troy, NY, USA 3 School of Mechanical and Nuclear Engineering, Ulsan National Institute of Science and Technology, Ulsan, Republic of Korea 4 Brookhaven National Laboratory, Upton, NY, USA 5 Integrated Safety Assessment Division, Korea Atomic Energy Research Institute, Daejeon, Republic of Korea Nuclear Plant Reliability and Information Lab.
Contents Introduction Proposed Framework: < BBN Model for NPP Safety Software Reliability Assessment > BBN Model Development High-level/Sub-level Structure BBN Model Parameter Quantification Expert Elicitation for BBN NPT Quantification Bayesian update of the BBN NPTs using Evidence Data Software Defect Estimation using BBN Model for Typical DPPS Software Conclusion 1/
Introduction Reliability quantification of safety software in digital I&C systems Analog system - Advanced features of digital system - Difficulties of supplying analog elements Digital system (PLD: Programmable Logic Device) S/W PSA CDF Safety measure To incorporate S/W reliability into the safety measure, S/W integrity needs to be quantified (ex. S/W failure probability on demand) Table 1. Existing safety-critical S/W reliability quantification methods QSRMs Description Related Works Software Reliability Growth Model (SRGM) Test-based method Bayesian Belief Network (BBN) Other Methods Time-based methods that use test data to estimate software failure rates employ statistical methods for the results of software testing, and possibly for operational data aggregates disparate software information (e.g. software failure data, software lifecycle activities) (1) Correlation approach; (2) Metrics-based method; (3) Context-based software risk model; etc. [Musa, 1987] [Lyu, 1996] [Miller et al., 1992] [May, 1995] [Gran, 2002] [Eom, 2009] [Neufelder, 2002] [Smidts and Li, 2004] [ASCA 2007] 2/
Basic Concept of Bayesian Belief Network A BBN is a probabilistic graphical model that uses Bayesian probability, a degree of belief in the occurrence of any event based on prior- and observed-evidence [1,2]. P V 1, V 2,., V n = P V i ppppppp V i n i=1, P V i ppppppp V i = P(V i ppppppp(v i ))P(V i ) P(ppppppp V i ) Example BBN model for Homes /Watson s wet lawn Prior probabilities based on initial degree of belief Yes No Sprinkler Rain Rain 0.2 0.8 Sprinkler 0.1 0.9 Holmes Lawn Wet Watson Lawn Wet NPT of the BBN model P(H_L_Wet = yes) = P(H_L_Wet Rain,Sprinkler) * P(Rain) * P(Sprinkler) + P(H_L_Wet no_rain,sprinkler) * P(no_Rain) * P(Sprinkler) + P(H_L_Wet Rain,no_Sprinkler) * P(Rain) * P(no_Sprinkler) + P(H_L_Wet no_rain,no_sprinkler) * P(no_Rain) * P(no_Sprinkler) P(H_L_Wet = yes) = 1*0.2*0.1 + 1*0.8*0.1 + 1*0.2*0.9 + 0*0.8*0.9 = 0.28 Rain Yes No Sprinkler Yes No Yes No H_L_Wet (yes) 1 1 1 0 H_L_Wet (no) 0 0 0 1 Rain Yes No W_L_Wet (yes) 1 0 W_L_Wet (no) 0 1 Inference from a single evidence 3/
Basic Concept of Bayesian Belief Network A BBN is a probabilistic graphical model that uses Bayesian probability, a degree of belief in the occurrence of any event based on prior- and observed-evidence [1,2]. n P V 1, V 2,., V n = i=1 P V i ppppppp V i, P V i ppppppp V i = P(V i ppppppp(v i ))P(V i ) P(ppppppp V i ) Example BBN model for Homes /Watson s wet lawn - When Holmes observes his lawn is wet (H_L_Wet), the evidence increases his belief on Rain at night from 0.2 to 0.714. Sprinkler Holmes Lawn Wet Rain Watson Lawn Wet P(Rain H_L_Wet) = P(H_L_WWW RRRR)P(RRRR) P(H_L_WWW) = [P(H_L_WWW RRRR,SSSSSSSSS)P(SSSSSSSSS)+P(H_L_WWW RRRR,nn_SSSSSSSSS)P(nn_SSSSSSSSS)]P(RRRR) P(H_L_WWW) P(Rain H_L_Wet) = 1 0.1+1 0.9 0.2 0.28 = 0.714 NPT of the BBN model updated belief Rain Yes No Sprinkler Yes No Yes No H_L_Wet (yes) 1 1 1 0 H_L_Wet (no) 0 0 0 1 Rain Yes No W_L_Wet (yes) 1 0 W_L_Wet (no) 0 1 initial belief (Prior) Inference from an observed evidence H_L_Wet becomes Evidence 4/
Previous BBN Models for Software Reliability Quantification Previous BBN models were developed which are specific to a particular safety software development environment. 2) LLNL: Assessment of Design Document of SW for a NPP 1) Halden project: Reliability assessment and acceptance of safety-critical software 3) KAERI: Assessment of Requirement Spec. of RPS Software Therefore, a BBN model that can be applied to (1) a general NPP safety-related software which follows standard SDLC, (2) and further to a specific safety software of NPP protection system needs to be developed. 5/
NPP Safety Software Development Life Cycle NPP safety software is typically designed based on IEEE Std. 1074 endorsed by RG 1.3 for complying with NRC regulations to assure safety software reliability. - Software defects can be introduced and removed at each SDLC phase - The residual software defects may induce the software failure on-demand Fig. 1. Basic Framework of SDLC processes [3] 6/
BBN Model Development High-level Structure High-level structure of a proposed BBN framework Requirement V&V Design V&V Implementation V&V Test V&V Installation V&V Total Defects Remaining in Requirement Total Defects Remaining in Design Total Defects Remaining in Implementation Total Defects Remaining in Test Total Defects Remaining in Installation Probability of Software Failure Requirement Development Design Development Implementation Development Test Development Installation Development Fig. 2. Overview of the BBN model The number of defects remaining in each SDLC phase is modeled a function of 1) The phase development quality (the developmental process adds defects), 2) The phase V&V quality (the V&V process removes defects). 7/
BBN Model Development Sub-level Structure At each SDLC phase, the states of Development quality determines: Defect density (the number of defects inserted per function point) (Defects introduced in current phase) = (Function Point) * (Defect Density) At each SDLC phase, the states of V&V quality determines: (1) Defect detection probability for defects introduced in the current phase (2) Defect detection probability for defects passed from the previous phases Fig. 3. BBN model for the Design 8/
BBN Model Development Sub-level Structure The model considers the Attribute which represents the quality in carrying out a collection of activities associated with the development and V&V qualities. The attributes were identified from software standard (IEEE-1012) and guidance (BTP- 14) and are modeled to be connected to the quality nodes in a diverging configuration. Fig. 4. Attributes nodes for Development quality in the Design phase Fig. 5. Attributes nodes for V&V quality in the Design phase 9/
BBN Parameter Quantification - Expert Elicitation Three phases of expert elicitation were conducted to estimate the BBN model. I : - Verify the causal relationship of the BBN nodes - Verify the model adequacy in capturing important attributes of SDLC II : - Estimate the prior distribution of the root nodes in BBN model - Estimate the NPTs of the child nodes in BBN model III: - Estimate specific evidence of the target system when applied to a particular safety software Fig. 6. Key steps in the BBN model evaluation 10/
BBN Parameter Quantification Initial Node Probability Table Expert elicitation (7 experts in the nuclear industry) is used to quantify the node probability table (NPT) of the BBN model parameters. To incorporate diverse experts opinion in a probabilistic manner, the NPTs was represented with the distribution of expert s opinions, rather than point estimates. Expert Elicitation on the node Number of FPs E1 E2 E3 E4 E5 E6 E7 < 100 FPs 0.3 0.1 0.1 0.1 0.5 0.3 0.6 100 FPs & < 1000 FPs 0.6 0.5 0.8 0.6 0.3 0.4 0.3 1000 FPs 0.1 0.4 0.1 0.3 0.2 0.3 0.1 NPT of Number of FP node in distribution form Number of FPs Distribution < 100 FPs Beta(1.59, 3.95) SDLC Expert Elicitation on the node V&V Complexity Low 5 th 50 th 95 th High E1 0.76 0.88 0.99 100 FPs & < 1000 FPs 1000 FPs Beta(4.07,4.03) Beta(2.79,10.22) NPT of Defect detection probability for defects introduced in current phase node in distribution form V&V Complexity High Medium Low Requirement Medium E2 0.47 0.68 0.89 E3 0.82 0.87 0.92 E4 0.66 0.83 0.99 E5 - - - E6 0.64 0.80 0.97 E7 0.42 0.75 1.00 Low Cumulative Distribution Function of Fitted Distribution 1 CDF of Fitted Distribution Derived 0.9 from Collected Expert Elicitation 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.4 0.5 0.6 0.7 0.8 0.9 1 Detection Probability of the Defects introduced at Current Require ment High Beta(1.81,1.40) Beta(4.09,1.06) Beta(3.74,0.59) Medium Beta(1.64,1.51) Beta(3.91,1.50) Beta(6.45,1.42) Low Beta(1.27,1.78) Beta(2.30,1.73) Beta(3.01,1.68) Design 11/
BBN Parameter Quantification Update of NPT with Reference In addition to the expert elicitation, other sources of evidence were used to Bayesian update the NPTs for Defect density and Defect detection probability nodes: (1 st stage) - Capers Jones, Applied Software Measurement: Global Analysis of Productivity and, Third edition, 2008 [4], In this study, CMM5 + 6σ, CMM4, Spiral software types were assumed to represent High, Medium, Low Development or V&V quality, respectively. Table 1. Selected Software Development Levels and Defect Characteristics [4] High Dev./V&V Medium Dev./V&V Table 2. Software Defect Allocation by SDLC phase Low Dev./V&V 12/
BBN Parameter Quantification Update of NPT with Reference NPT of defect density in each phase of SDLC was updated using conjugate prior based on the reference data on the defect potentials. Prior Prob.: Defect density derived from expert elicitation (Gamma dist.) Likelihood: Reference Data from Caper Jones [4] (Poisson dist.) Posterior Prob.: Updated defect density (Gamma dist.) Likelihood Table 3. Update of Conjugate Prior based on Evidence (Likelihood) Likelihood model parameters Conjugate prior Distribution Prior hyperparameters Posterior hyperparameters Poisson λ (rate) Gamma α, β α+ i x i, β+n Requireme nt Design Implement ation Test Installation and Check out Initial Defect Density NPT from Expert Elicitation Dev. Alpha Beta Mean Variance High 0.43 0.55 0.78 1.42 Medium 0.46 0.38 1.21 3.18 Low 0.41 0. 2.43 14.25 High 0.46 0.37 1.24 3.36 Medium 0.48 0.25 1.88 7.46 Low 0.50 0.15 3.28 21.58 High 0.50 0.33 1.51 4.55 Medium 0.52 0.20 2.67 13.59 Low 0.53 0.20 2.69 13.65 High 0.52 1.31 0.40 0.31 Medium 0.62 0.92 0.67 0.73 Low 0.41 0. 2.43 14.25 High 0.57 1.46 0.39 0.27 Medium 0.62 0.68 0.91 1.34 Low 0.58 0.38 1.52 4.01 Update with handbook data Evidence from Handbook Data 0.48 0.60 0.65 1.20 1.50 1.63 1.92 2.40 2.60 0.48 0.60 0.65 0 0 0 Updated NPT Alpha Beta Mean Variance 0.91 1.55 0.59 0.38 1.06 1.38 0.77 0.56 1.06 1. 0.91 0.78 1.66 1.37 1.21 0.88 1.98 1.25 1.58 1.26 2.12 1.15 1.84 1.60 2.42 1.33 1.82 1.37 2.92 1.20 2.44 2.04 3.13 1.20 2.61 2.18 1.00 2.31 0.43 0.19 1.22 1.92 0.64 0.33 1.06 1. 0.91 0.78 0.57 2.46 0.23 0.09 0.62 1.68 0.37 0.22 0.58 1.38 0.42 0.30 13/
BBN Parameter Quantification Update of NPT with Observations 2 nd Bayesian update of the NPTs for Defect density and Defect detection probability nodes with observations of actual development projects: Software development anomaly reports of IDiPS-RPS [5] and LOCS [6]. In this study, the defect estimates in the anomaly reports for both applications were assumed to be the number of defects detected in each SDLC phase, and used to update defect density NPT. x j P j,h V j,h + P j,m V j,m + P j,l V j,l = y j x j : number of defects in each SDLC phase y j : number of defects in KNICS/LOCS data P j,i : Defect Detection Probability at i V&V in j phase V j,i : Posterior distribution for KNICS/LOCS application at i V&V in j phase BP BP/CP/ATI P/COM BP/CP/ATI P/COM Defect density data from KNICS reports (y i ) ANR (High) ANR (Medium) ANR (Low) Estimated Defects, y j Requirement 2 2 2 6 Design 7 7 2 16 Implementation 1 1 1 3 Total 10 10 5 25 Integration 2 2 0 4 Validation (System testing) 2 1 1 4 x j estimated from KNICS data Number of Defects, x j Requirement 12.2476 Design 37.5438 Implementation 7.3886 Test 15.5686 Installation and Checkout* * The Defect density for Installation and Checkout was not updated using KNICS data since the system was not installed or checkout. - 14/
BBN Parameter Quantification Update of NPT with Observations 2 nd Bayesian update Based on the derived number of defects per FP for KNICS/LOCS application, the data (defect density per function point at each SDLC phase) was used to update using conjugate prior from the defect density result which was updated from handbook data. Likelihood Table 3. Update of Conjugate Prior based on Evidence (Likelihood) Likelihood model parameters Conjugate prior Distribution Prior hyperparameters Posterior hyperparameters Poisson λ (rate) Gamma α, β α+ i x i, β+n Requireme nt Design Implement ation Test Installation and Check out Updated Defect Density from 1 st stage update Developme nt Alpha Beta Mean Variance High 0.912 1.551 0.588 0.379 Medium 1.062 1.381 0.769 0.557 Low 1.064 1.1 0.909 0.777 High 1.656 1.368 1.210 0.885 Medium 1.976 1.253 1.578 1.259 Low 2.125 1.152 1.844 1.601 High 2.421 1.332 1.818 1.365 Medium 2.924 1.196 2.444 2.043 Low 3.129 1.197 2.615 2.184 High 1.000 2.306 0.434 0.188 Medium 1.223 1.924 0.636 0.330 Low 1.064 1.1 0.909 0.777 High 0.571 2.464 0.232 0.094 Medium 0.621 1.680 0.370 0.220 Low 0.580 1.380 0.420 0.304 Update with the Evidence from KNICS/LOCS data Evidence from KNICS Data 0.1670 0.2185 0.2584 0.5143 0.6704 0.7838 0.0981 0.1319 0.1411 0.1893 0.2773 0.3967 - - - Evidence from LOCS Data 0.026 0.033 0.036 0.087 0.110 0.126 0.080 0.103 0.110 0.055 0.078 0.099 0.040 0.063 0.072 Updated Result from KNICS/LOCS data Alpha Beta Mean Variance 1.079 2.551 0.423 0.166 1.280 2.381 0.538 0.226 1.323 2.1 0.609 0.281 2.0 2.368 0.916 0.387 2.646 2.253 1.5 0.522 2.909 2.152 1.351 0.628 2.519 2.332 1.080 0.463 3.056 2.196 1.391 0.633 3.270 2.197 1.489 0.678 1.189 3.306 0.360 0.109 1.500 2.924 0.513 0.5 1.461 2.1 0.673 0.310 0.571 2.464 0.232 0.094 0.621 1.680 0.370 0.220 0.580 1.380 0.420 0.304 15/
BBN Model Evaluation Based on the estimated NPTs, the number of software defects for typical digital protection system was estimated using the proposed BBN framework. By following FP counting rules [7], a sole-function safety software (low level external input + internal logic file + external output) has a representative size of 50 FPs. Considering the attributes are completed satisfactorily with respect to US regulatory licensing review, Medium quality is reasonably assumed for all attributes. Table 7. BBN model parameters for all Medium attribute quality given 50 FPs Number of Defects introduced in the current phase Detection probability for defects passed from previous phase Detection probability for defects introduced in current phase Number of Detected defects passed from previous phase Number of Detected defects introduced in current phase Defect density (Number of Defects per FP) Number of Defects remaining Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD Requirements 19.71 35.9 - - 0.79 0.16 - - 15.64 29.28 0.39 0.72 4.07 10.15 Design 42.61 52.56 0.46 0.26 0.79 0. 1.86 5.40 33.82 43.23 0.85 1.05 11.00.05 Implementation 49.45 56.96 0.48 0.25 0.84 0.15 5.25 9.66 41.49 49.03 0.99 1.14 13.71.99 Test 19.88 35.25 0.70 0.16 0.73 0.14 9.61 13.16 14.54 26.42 0.40 0.70 9.45 13.08 Installation/ Checkout 12.63 29.35 0.70 0.19 0.80 0.14 6.64 9.77 10.12 23.88 0.25 0.59 5.32 9.12 16/
Conclusion In this study, a practical BBN framework for quantifying the software failure probability of a NPP digital protection system was proposed. The BBN model was constructed by: (1) capturing NPP safety-related SDLC activity quality indicators and product information (2) establishing and quantifying the causal relationships between SDLC characteristics The BBN model parameter was estimated by: (1) aggregating largely diverse expert inputs using probability distribution of each cell of NPT (2) effectively integrating with other available sources of evidence Based on the proposed BBN model, the number of software faults for a typical NPP digital protection system was obtained. /
Reference [1] Heckerman, D., A tutorial on learning with Bayesian networks, Technical Report MSR-TR-95-06, Microsoft Research, Microsoft Corporation, 1995. [2] J. Pearl, Probabilistic Reasoning in Intelligent Systems, San Francisco CA: Morgan Kaufmann, 1988. [3] Schultz, David J. "IEEE standard for developing software life cycle processes." IEEE Std (1997): 1074-1997. [4] Capers Jones, Applied Software Measurement: Global Analysis of Productivity and, Third edition, 2008. [5] KAERI/TR-4092/2010, Reliability Assessment Method Of Reactor Protection System Software by Using V&V based Bayesian Nets, Korea Atomic Energy Research Institute. [6] Verification and Validation (V&V) Report for 2A Loop Instrumentation and Operating Control System, PLN-4681, Idaho National Laboratory, 2014. [7] International Function Point Users Group, http://www.ifpug.org. 18/
Thank you for your attention Q&A