Challenges In Uncertainty, Calibration, Validation and Predictability of Engineering Analysis Models

Transcription

1 Challenges In Uncertainty, Calibration, Validation and Predictability of Engineering Analysis Models Dr. Liping Wang GE Global Research Manager, Probabilistics Lab Niskayuna, NY 2011 UQ Workshop University of Minnesota, MN June 02, 2011 Team Members GE Global Research: Arun Subramaniyan, Nataraj Chennimalai, Xingjie Fang, Giridhar Jothiprasad, Martha Gardner, Amit Kale GE Aviation: Don Beeson, Gene Wiggs, and John Nelson 1

2 Outline Motivation History of Development How far are we along the path? GE capabilities Technical Challenges & Solutions Future Direction Summary 2

3 ... Motivation Why Model Calibration, Validation, Prediction & Uncertainty Quantification? What has been accomplished? Literature Review GE Eperience Input Factors Xs Possible technical solutions and future direction X 1 X2 Uncertainty Quantification UQ Input Factors Xs Etc. X1 X2 Etc. Deterministic Simulation Model parameters Model discrepancy Output Factors Ys Etc. θ δ Output Factors Ys Y2 Etc. Uncertainty: Aleatory Random and usually modeled by probability distributions. Methods include probability theory and classical statistics Epistemic Lack of knowledge. Methods include fuzzy logic or evidence/possibility theory Y1 3

4 History of Development How far are we along the path? One task at a time calibration, validation, prediction & uncertainty quantification - since the 1980s Calibration - data matching, or inverse problems, or parameter estimation - applied to heat transfer, fluid mechanics, solid mechanics, etc. Verification & Validation V&V - introduced by DoD, AIAA, ASME, National Labs Prediction - well-established physics models, calibrated empirical models, and meta-models Response Surface, Kriging, Gaussian Process, Radial Basis Function, etc Uncertainty quantification - Monte Carlo, First Order Second Moment, moments based, polynomial chaos, etc All tasks simultaneously - first introduced by Kennedy and O Hagan in 2001 Bayesian framework 4

5 Kennedy & O Hagan 2001 What is Bayesian Statistics? f θ y f y θ f θ f θ y = f y θ f θ L θ = f y θ Given data Kennedy & O Hagan Hybrid Model Formulation: f θ pdf Prior Likelihood function y i = η i, θ δ i ε i, i=1,2,,n θ θ pdf Posterior θ Observations from the physical system Output of a simulator, with design inputs and calibration parameters θ Discrepancy between the simulator and the physical system Observation measurement system error Build & calibrate Gaussian Process GP models for both η and δ... Specify beliefs about θ, δ through prior probability distributions Use Markov Chain Monte Carlo MCMC to obtain parameter estimates Similar approaches by Higdon et al. and Liu et al. Most implementations are for single output 5

6 Multiple Outputs Implementation by Los Alamos National Lab LANL - Higdon, William et al. Principal Components Analysis PCA for dimension reduction & efficiency improvements Correlated outputs η, θ = k w1, θ... k, 1 p p θ δ = d1v1... d p v p 1, k2 k pη and d, 1 d2,..., d pδ are the principal components k,..., w & v are the GP models for simulator and model correction δ δ η w η More Applicable to Real Problems with LANL Implementation 6

7 Maimum Likelihood Estimation MLE Alternative approach to Bayesian Xiong, Chen, Tsui and Apley Investigated three possible formulations y, Θ = η, θ ε y, Θ = η, θ δ ε y, Θ = η, θ δ ε best Implementation only for single output Sensitivity analysis prior to MLE optimization to avoid numerical instability 7

8 Model Inadequacy Correction & Prediction No Calibration Capture model inadequacy with no model calibration Wang et al. and Chen et al. y = η δ ε Closed form Bayesian posterior Solve GP hyper-parameters using MLE Improved efficiency for high dimensional design space Useful for well established physical models where calibration is not necessary or performed previously 8

9 V&V and Model Validation Metrics Current and desired state of validation metrics Oberkampf et al Quantitative Metrics using classical hypothesis testing, Bayes factor, frequentist s metrics, and area metrics Quantitative Metrics using Kennedy & O Hagan Bayesian Framework Chen et al. Preliminary elements of model validation Paez, Swiler, Mayes, Miller, et al International Modal Analysis Conference, Orlando, FL Epistemic uncertainty Paez & Swiler, Paez Most Common Desired State Customers Stakeholders Analysis Modelers Validation Analysts Eperimentalists 9

10 GE Capabilities Deterministic Inverse modeling since 2003 Methods development and implementation Efficient transient data matching using PCA based hybrid metamodels & zoning techniques Partial probabilistic data matching to update standard deviations Widely used across GE businesses Transient Analysis and Performance Heat Transfer and Fluid Systems GE90 78Xs GP Xs GEn 29 Xs Materials Design Acceleration Material Modeling 23 Xs, 35 TCs simultaneously Others: Transient cycle models 3D transient clearances Undercowl heat transfer Empirical model tuning Analysis time savings >50% Data mismatch reduced by half 10

11 11 Probabilistic Inverse modeling Bayesian Hybrid Modeling since 2006 Built on Kennedy & O Hagan Bayesian Method and LANL Implementation Efficiency improvement ~2X, fleibility, robustness Investigated possible formulations Key drivers of model inadequacy & insight to possible model improvements Validated with multiple benchmark problems GE Capabilities ε θ η ε δ η δ η η = = = =, y y y y ε δ θ η ε δ θ η ε δ θ η ε δ θ η = = = =,,,,, y y y y Kennedy & O Hagan

12 GE Capabilities Demonstration with challenging engineering problems P T _ ra tio Test data : y Test data uncertainty: ε BRM model: η,θ Design parameters: Model parameters: θ Missing vertical parts of high speed lines θ calibrated Hybrid Modeling Mean= Std= Test HM Mean 90% CI Confidence Bounds Performance Maps at Speed = 105% Model Discrepancy & Updating 1.6 Test 1.58 Hybrid Modeling Calibrated Simulator only 1.56 δ Corrected flow Corrected flow PT_ratio 105% speed-line δ % 2 Speed Lines 3 Speed Lines 105% 1.75 Test 1.7 Calibrated Simulator only Hybrid Modeling % 105% Mean= Std= Test 1.45 Calibrated Simulator only 1.4 Hybrid Modeling % Matches Test Data Well for Single & Multiple Speed Lines 12

13 GE Capabilities Transient AIR Model Benchmark problem 6 calibration parameters 9 Outputs Ys 52 time points in each transient > 3000 DoE points Only 52 DoE simulation points used for Hybrid modeling Calibrated Model ηt, θ Calibrated & Discrepancy Adjusted Model yt= ηt, θ δt δt Discrepancy BHM calibrates transient model accurately with very little data 13 13

14 GE Capabilities Demonstration with challenging engineering problems IPE Status Match TOW Uncertainty Improved calibration results by capturing model discrepancy More confidence in solution with probabilistic estimation Cycle Deck Characterizing model discrepancy and uncertainty in severity models Main effects able to point high thrust severity for improvement of current models Calibrated calibrated Simulator simulator discrepancy-adjusted Discrepancy-Adjusted discrepancy Discrepancy ZT49 Ehaust Gas Temperature ZPCN12 Fan Speed Histogram Counts Posterior Distribution of HPT Efficiency θ DE42DD Demonstration with IPE Status Match, TOW Uncertainty Severity Modeling, Cycle Deck Performance 14

15 GE Capabilities Etension to Model Validation GE90 Fan Blade Row Model Xs θs Combustion Dynamics Acoustic fluctuations p δ ε Calibrated Simulator Model Discrepancy δ Flame heat release fluctuations q δ at tested points δ at untested points Freq1 θ calibrated Computation v. Eperiment δ Enabler for Probabilistic Validation Metrics at Tested & Untested Points 15

16 GE Capabilities Going-forward Continue improving MCMC speed issues for high dimensional >100Xs and solving challenging engineering data matching applications Model Validation for All Engineering Models 3-year program Fleible to all models based on data availability, affordable & accurate, account for all types of uncertainty, probabilistic quantitative metrics co m b ustor e it T T PT Vr Vth K ω X s 2011 Hot Gas Path Heat Transfer 2012 Combustion Dynamics 2013 All Engineering Models g eom etry tip cleara n ce, c ore shift, film hole drilling Xs θs Acoustic fluctuations p θsε δ ε Aero purge flow δ = y η ε Flame heat release fluctuations q δ = y η, θ ε Mechanical δ = y η, θ ε Complicated Physics, Unknown Uncertainty, High Dimension Challenges Remain! 16

17 Technical Challenges and Solutions Curse of dimensionality Large number of calibration parameters. MCMC speed issues. Transient data matching PCA/Sensitivity, sparse matri inversion, adaptive convergence criteria for MCMC, sequential MC or other optimization techniques Source of Uncertainty Epistemic & Aleatory uncertainty. Probability, statistics, fuzzy logic or evidence/possibility theory Model inadequacy, uncertainty and characterization Identifiability of parameter calibration and model inadequacy Use as much knowledge as you can on the prior. Carefully choose the range. Uncertainty quantification of eperiments. Better understand model inadequacy and key drivers Post-processing to model discrepancy terms. Bayesian model comparison for possible suggestion to model improvement 17

18 Technical Challenges and Solutions Lack of data and etrapolation Limited test & simulation data No overlap between simulation & test data etrapolation Scientific knowledge, designer s belief prior Confounding Effects High-dimensionality coupled with lack of data Challenging mathematical issues Scientific knowledge, designer s belief prior Model validation and quantitative metrics Account for all source of uncertainty epistemic & aleatory Fleible for all analysis models - empirical, physics no calibration & metamodels based on data availability complete, partial and etrapolation Affordable & accurate Confidence and probabilistic metrics 18

19 Technical Challenges and Solutions Multiple sets of eperimental data Mied datasets multiple speed lines, multiple engine data Multiple modes of posterior distributions Numerical and speed issues Measurement error and uncertainty Uncertainty quantification for both epistemic and aleatory uncertainty Statistical analysis Outlier detection sensor failure Multi-physics & multi-fidelity models Direct simulations are prohibitively epensive Vast scale difference among the lowest atomic to the highest macroscopic scale Difficult to establish criteria & strategies when switching design space from one scale to another 19

20 Summary Advancements in model calibration, validation, prediction and uncertainty quantification in the past three decades Much research and many publications from industry, government, academia GE has been very active Technical challenges remain and are being worked 20