Application of V&V 20 Standard to the Benchmark FDA Nozzle Model

Transcription

1 Application of V&V 20 Standard to the Benchmark FDA Nozzle Model Gavin A. D Souza 1, Prasanna Hariharan 2, Marc Horner 3, Dawn Bardot 4, Richard A. Malinauskas 2, Ph.D. 1 University of Cincinnati, Cincinnati, OH; 2 Food and Drug Administration, Silver Spring, MD; 3 ANSYS Inc., Evanston, IL; 4 Medical Device Innovation Consortium, St. Louis Park, MN V&V Symposium, Las Vegas, NV May 15,

2 Introduction CFD in medical device development and testing Valuable tool to evaluate the safety of devices against red blood cell damage (hemolysis) Example devices: Heart valves, Total artificial heart, and blood pumps Hemolysis estimated from CFD data using empirical relations Hemolysis index = C (shear stress) a (residence time) b (a, b, C empirical constants) Issue with CFD submission: Inadequate V & V 2

3 Study Goals 1. Assess the applicability of the V&V 20 standard to medical devices using a simplified medical device model 2. Evaluate the V&V process using the CAM (Credibility Assessment Method) CAM Developed by the V & V 40 committee to assess the credibility of the V & V activity 3

4 FDA Nozzle Model 12 mm 4 mm Designed as a part of FDA s round robin effort Contains flow features commonly seen in medical devices Availability of inter laboratory experimental and CFD data for Pressure, velocity, shear stress, and hemolysis

5 FDA Nozzle Model High Shear Stress 12 mm 4 mm Recirculation region HEMOLYSIS THROMBOSIS POTENTIAL 5

6 Nozzle model: V & V Process Context of Use 2 Assess the ability of hemolysis model to predict blood damage in the Nozzle model Hemolysis index = C (shear stress) a (residence time) b Context of Use 1 Assess the ability of CFD model to predict velocity and shear stress For this preliminary study, validation variable: Center line velocity Flow conditions: Re = 500, 2000, 3500, 5000, and 6500 Experimental data: Particle Image Velocimtry (PIV) data from 3 independent labs 6

7 5 Steps of V&V 20 Code Verification Solution Verification Input Parameter Uncertainty Experimental Result Uncertainty Validation 7

8 Step 1 Code Verification Establishing the correctness of the numerical code Use of Exact Analytical Solutions Method of Manufactured Solution (MMS) Verifies the full form of the governing equations using a manufactured solution and a source term Benchmark Problems Includes simplified flow problems (e.g. Couette flow between inclined plates) 8

9 Code verification: CAM Code Verification Software Quality Software Architecture and Logic Numerical Verification a. Software quality: Associated activities demonstrate that good software development practices were followed. Minimal evidence of established code review process and revision control performed manually. Code review process in place for checking the validity of the complete system; revision control performed through established process. Code review process in place with regression testing for checking the validity of the complete system; revision control performed through established process with audit capability. b. Software architecture and logic: Associated activities demonstrate that good software development practices were followed. Minimal test of all software components only comparing system level inputs and outputs. Unit testing conducted for some key intermediate checkpoints as well as system level inputs and outputs to avoid compensating error in code communication. Unit testing conducted for all intermediate checkpoints, as well as system level inputs and outputs to avoid compensating error in code communication. Justification: Use of commercial software (ANSYS CFX) 9

10 Code verification: CAM Software Quality Code Verification Software Architecture and Logic Numerical Verification a. Numerical verification: Is the code solving the equations correctly? Achieved level of credibility Reused components (3rd party software, previously developed in-house code, commercial code) with limited prior code verification documentation and no comparison with benchmark solutions. Numerical verification performed on the key units of the code using some established verification technique for e.g. analytical solution, method of manufactured solutions, numerical benchmark, etc. and some features and capabilities tested with benchmark solutions. Targeted level of credibility Numerical verification performed on all the units of the code using some established verification technique for e.g. analytical solution, method of manufactured solutions, numerical benchmark, etc. Order of numerical convergence determined and all important features and capabilities tested with rigorous benchmark solutions. Justification: i) Use of commercial software (ANSYS CFX) ii) velocity is the variable of interest 10

11 Step 2: Solution Verification Estimating the numerical uncertainty (u ) due to discretization using the Grid Convergence Index (GCI) method Mesh No. Refinement factor No. of cells No. of Nodes 1 (Coarse) 2 25,000 51, , , , ,202 4 (Fine) 2 1,600,000 3,212,402 Loc-1 Loc-2 Loc-3 fully-developed velocity inlet (parabolic) pressure outlet 11

12 Solution Verification - FDA Nozzle [Cont d] Uncertainty due to discretization Order of convergence (p) (%) Centerline velocity Centerline velocity Mesh set Loc-1 Loc-2 Loc-3 Loc-1 Loc-2 Loc-3 1, 2 & 3 (Coarse) E E E-01 2, 3 & 4 (Fine) E E E-01 12

13 CAM: Solution Verification Discretizat ion Error Use Errors Numerical Solver Error Targeted and achieved level of credibility a. Discretization Error: Associated activities address the error of discretization of the computational domain (e.g., spatial, temporal) and ensure the error is minimized. No convergence analysis was performed to quantify the discretization error. All applicable analyses were performed but models used did not address the same physics; no mass/ momentum/ energy balance check. All applicable convergence analyses were performed but not for variables of interest or location of interest; no estimation of discretization error; mass/ momentum/ energy balance check done. Refinement was performed with quantification of discretization error or uncertainty estimate for the variable of interest to account for discretization error; performed at the location of interest for the variable of interest; mass/ momentum/ energy balance check done. 13

14 Step 3 Input Parameter Uncertainty Analysis Same as Model validation section in CAM Estimating the simulation uncertainty due to uncertainty in the model input parameters ( ) Uncertainty analysis done using Sensitivity coefficient method S Simulation result (E.g. Nozzle Centerline Velocity) Input parameter (E.g. Flow rate, Viscosity, Turbulent Intensity) Corresponding experimental uncertainty in input parameter,,,,,,,,,, 2 [using second-order finite difference] perturbation size 10-9 < <

15 Input Parameter Uncertainty Analysis [Cont d] Input parameter uncertainty, from inter-laboratory experiments Input parameter (X) Mean Value % Uncertainty ( ) Flow rate, Q (m 3 /s) 2.02E % Dynamic viscosity, (Pa.s) 3.50E % Turbulent Intensity, TI (%) % Total number of simulations : 2n+1 = 7 (since we considered 3 input parameters) 15

16 Input Parameter Uncertainty - FDA Nozzle Simulation parameters Throat Reynolds number Re t = 2000 Use of commercial CFD software CFX, ANSYS, Inc. Turbulence Model SST Relative perturbation size ( ) for Q,, and TI 0.075% 16

17 Input Parameter Uncertainty CFD Data Locations % 6% 0% Q Flow rate Viscosity TI Turbulent Intensity % (Q) % ( %() 21 % 4 % 75 % 15% 4 % 81 % 17

18 CAM Model Validation System Configuration System Properties Validation Model Boundary Conditions Governing Equations I. Quantification of sensitivities: Sensitivity analyses were not performed Sensitivity analyses on expected key parameters were performed across the range of values expected in the context of use Comprehensive sensitivity analysis was performed across the range of values expected in the context of use ii. Quantification of uncertainties Uncertainties were not identified Uncertainties on expected key parameters were identified and quantified with a qualitative assessment of the effect on the model output Uncertainties on all key parameters were identified and quantified, and propagated to be reflected in the output Uncertainties were quantified and propagated to be reflected in the output 18

19 CAM Score Model Validation System Configuration System Properties Validation Model Boundary Conditions Governing Equations ii. Quantification of uncertainties Uncertainties were not identified Uncertainties were identified with limited quantification Uncertainties were quantified and are reflected in the output 19

20 Step 4 Experimental Uncertainty Comparator: Particle Image Velocimetry (PIV) Experiments performed at 3 independent labs Multiple dataset Multiple data points Geometry: All dimension within the set tolerance limit Boundary conditions: All boundary conditions and material properties measured Lab to lab variability was captured. But measurement uncertainty due to PIV not quantified 20

21 Experimental Data (Re = 2000) Large variability in the center line velocity near the reattachment region 21

22 CAM Score Comparator Validation Comparator Sample Characterization Control over test conditions Measurement uncertainty Degree to which the error of the measurement system is known Measurements are qualitative observations (e.g. imaging without quantification) with limited spatial / temporal monitoring. Measurements are obtained from instruments with known accuracy, and are monitored at critical locations. Measurements are obtained from instruments with known uncertainty (accuracy, repeatability, and reproducibility) and monitored against specific tolerances at critical locations. 22

23 Step 5 Validation reattachment point Error metric = (experiment mean CFD mean) ±u val 23

24 CAM Applicability to COU If the COU is to assess the velocity (everywhere) and shear predictions Relevance to COU: This is a function of the amount of overlap between the validation domain and the context of use. The output from the CM&S requires extrapolation from the validation domain The output from the CM&S mostly requires interpolation to the COU in the validation domain and less extrapolation because it is a close surrogate for COU Minimal differences exist between validation space and COU Future Work Extend this study to asses velocity (everywhere), shear stress and hemolysis predictions Challenge: Large experimental uncertainty in wall-shear and hemolysis measurements 24