- PDF Free Download

Size: px

Start display at page:

Download ""

Eileen Bryan
5 years ago
Views:

ENGM 670 & MECE 758 Modeling and Simulation of Engineering Systems (Advanced Topics) Winter 2011 Lecture 10: Extra Material M.G. Lipsett University of Alberta http://www.ualberta.

htm Topics Decision Analysis Uncertainty Model verification Sensitivity Analysis Using Simulations Introduction to Bayes Risk Risk Analysis by simulating the effect of uncertainty on a

1 ENGM 670 & MECE 758 Modeling and Simulation of Engineering Systems (Advanced Topics) Winter 2011 Lecture 10: Extra Material M.G. Lipsett University of Alberta Topics Decision Analysis Uncertainty Model verification Sensitivity Analysis Using Simulations Introduction to Bayes Risk Risk Analysis by simulating the effect of uncertainty on a system with probabilistic variables (Monte Carlo simulation) Probability distribution functions, estimating probabilities and loss functions Probabilistic simulations of discrete-event systems and lumped-parameter systems Model validation (estimating parameters of constitutive relationships in linear and nonlinear systems) MG Lipsett,

2 Why Decisions Are Difficult Many factors affect decisions Technical Issues Financial Issues Management Issues People generally have a poor perception of risk Factors should be weighted to represent their relative importance Not every factor is equally important Appropriate relative weighting is difficult to do MG Lipsett, Decision Analysis Methods Structured methods to determine actions to manage risk Depends on good corporate information on costs, probabilities, and dependencies A facilitation process is used to ensure all scenarios are explored and considered consistently One example is Kepner-Tregoe's thinking processes: Situation Appraisal, Problem Analysis, Decision Analysis, Potential Problem Analysis, and Potential Opportunity Analysis MG Lipsett,

3 Basic Decision Analysis Process Gather experts and decision-makers Describe situation to the group Identify issues relevant to the situation Identify alternatives Usually established in advance (to have good supporting info available) Sometimes brainstorming is done during the process (free development of alternatives) followed by reality check and selection of a small set of realistic alternatives for analysis & decision Good simulations of how an option is expected to perform can be extremely helpful in decision making Select important factors and relative weighting factors For each option or event, estimate value for each factor in terms of costs and costs & benefits MG Lipsett, Example of Weighted Decision Factors Hydro run of river Contaminant Emissions CO 2 Radioactivity Land Use Water Impacts Waste Impacts Resource Availability Total Impact (Weighted) Hydro impoundment Manitoba purchase Wind Biomass Photovoltaic Nuclear Natural gas single cycle Natural gas combined cycle Natural gas cogeneration Gasification without CO 2 removal Gasification with 90% CO 2 removal Coal Oil Note: The Total Weighted Impact is calculated by applying a weight of 10 to contaminant emissions, 20 to greenhouse gases, and 1 to all other categories. Source: SENES and OPA MG Lipsett,

4 Where Decision Analysis Can Go Wrong A decision can go the wrong way for a number of reasons: Bad definition of the problem (trying to make a decision for the wrong problem, or poor definition of what the desired outcome should be and its specifications) Bad assumptions (trying to solve the wrong problem) Poor design of solution options (lack of creativity, constrained thinking) Poor modeling (not producing enough understanding of whether the proposed solution is likely to solve the problem) Poor presentation (communication of understanding is not sufficient to allow decision-makers to assess an option properly) Decision bias (decision-maker has undeclared preference for some options over others, perhaps including not solving the real problem) MG Lipsett, Responsibility for Good Decisions The engineering designer or manager can overcome all of these potential decision-making pitfalls (except the last one) by: careful problem analysis to define the actual need good creative engineering thinking use of model-based analysis for assessing solution options effective packaging of model results for understanding and effective assessment But: A good model, poorly communicated, is worse than useless. MG Lipsett,

5 Uncertainty To this point, we have assumed that we have perfect knowledge about a system. In reality, we may not know everything we need. Uncertainty comes in three forms an observed set of state variables that does not completely define the process state, A constitutive relationship that is not clearly defined, or for which the parameters are not well known, measurements of the system that are inaccurate All three forms occur in design, manufacturing, and other types of operations. Human operators and supervisors must make judgments based on incomplete information and past experience. Uncertainty in a process demands branching for exception handling based on monitoring the task execution. MG Lipsett, Sources of Uncertainty Assumptions about the model Model parameters Measurements of the inputs, outputs, and state variables of the system MG Lipsett,

6 Model Verification Verification is the process of checking that the model is a reasonable approximation of the system of interest A number of techniques can be used MG Lipsett, Model Verification Techniques MG Lipsett, Debug as you develop the simulation Have the simulation reviewed, preferably by more than one person Run the simulation under a number of inputs, and check that the output is reasonable (that is, the model behaves as you would expect) Run a trace to debug a DES; in a trace, the states and certain counters are displayed after each event occurs, allowing hand checks to be made of each step Run the simulation under simplified conditions, a benchmark case with known results, or a simple case with an analytical solution Display an animation of the simulation output Compare the input distribution descriptive statistics (sample mean and sample variance) to historical values, to make sure that the proper inputs are being used in the simulation 6

7 Model Parameters Uncertainty in the model parameters affects the output of the simulation Even if the structure of the governing equations or representations of discrete-events are correct, incorrect parameters drive the simulation to an erroneous set of outputs Parameter estimation can be used to identify the best guess for the parameters for a set of given data (Model validation and nonlinear parameter estimation are discussed later in this slide deck) MG Lipsett, Sensitivity Analysis In some systems there may be some underlying uncertainty in the parameters, especially for a new design In that case, the simulation itself can be used to test how a parameter affects the system behaviour Sensitivity analysis determines how important a variable or a parameter is to the system behaviour Uncertainty can be expressed as a range of values for an aspect of the model (or as a probability) Scenarios are often done (most likely / best /worst case) to give a range of expected results of a change in a value More precise scenario analysis can be done using simulations that test the probabilities using a distribution MG Lipsett,

8 Sensitivity of Individual Parameters For each parameter: Do the analysis first using the most likely value Realistic result Repeat the analysis, assigning the parameter its best-case value Optimistic result showing sensitivity of that parameter Repeat the analysis, assigning the parameter its worst-case value Pessimistic result showing sensitivity of that parameter Results are intuitively visualized using a Tornado Diagram MG Lipsett, Most Likely / Best Case / Worst Case The effect of variability of a parameter (such as a spring constant, transducer coefficient, etc.) can be assessed for its effect on system performance (such as the value of an output of the system) by analysing multiple cases for that alternative Choose the most likely value for the parameter, and a number on either side that represents the best-case value and the worst-case value Ideally these parameters bracket the risk associated with the uncertainty Assess the sensitivity of a parameter individually MG Lipsett,

9 Modeling General Uncertainty Scenarios Do three scenarios (realistic, optimistic, pessimistic) Do the analysis first using the most likely values for all the parameters. This gives the realistic result Repeat the analysis, assigning all the uncertain parameters their best-case values. This gives the most optimistic result Repeat the analysis, assigning all the uncertain parameters their worst-case values. This gives the most pessimistic result Useful for general assessment of the extreme range of possibilities Not very useful for understanding the effect of variability MG Lipsett, MG Lipsett, Risk Associated with An Event Or A Decision The consequence associated with a particular factor can be expressed as a cost (or a benefit, if the outcome is positive) Risk is any uncertainty that affects a system Risk depends on the conditions and time Uncertainties and risks have a time horizon Uncertainties will exist regarding future conditions and states of a system Uncertainties evolve over time Uncertainties become risks when they have consequences, that is, they affect measurable outputs of the system that are set against some standard Risk tends to reduce as time progresses: e.g., marketing uncertainty about how many units of a product goes to zero once all of the units have been sold. 9

10 Conditional Risk & Minimum Risk The conditional risk is defined as: for i = 1,,a, where a is number of possible actions α c is number of states of the system λ(α i w j ) is the loss function for taking an action α i when the state of the system is w j P(w j x) is the posterior probability that the state w j given that the feature vector x has been measured Select the action for which R is minimum (Bayes Risk): MG Lipsett, Expected Outcome for A Set of Probabilities From a set of independent probabilities and associated costs (or benefits), the expected outcome can be estimated easily A utility is planning to build a new electrical generating plant. There are some uncertainties about each technology choice, which will affect the useful life of each. The probabilities are as follows: Probabilities Useful Life Option A Option B Option A will cost $150k per megawatt. Option B will cost $300k per megawatt. Which has the lowest cost $/MWe/yr? MG Lipsett,

11 Probability Distributions How do we represent a range of possibilities? MG Lipsett, Probability Distribution Functions A random variable (RV), X, is a real numerical valued function defined over a sample space, S. X(s) assigns a real number to every outcome in S. A continuous random variable has a range with an infinite number of values on the real line and is typically (but not necessarily) defined over an infinite sample space. The probability density function (pdf) for a continuous random variable is a function that assigns a probability density to each and every value of the random variable. The probability distribution function (pdf) for a discrete random variable is a function that assigns a probability to each value of the random variable. A sampled sequence of a continuous RV is a set of discrete RVs. MG Lipsett,

12 Examples of PDFs Exponential Gaussian Discrete RVs (e.g. sampled data) have discrete PDFs Uniform Beta Gamma PDFs can be Multidimensional Weibull Many others MG Lipsett, Probabilistic Simulations In this course, we have studied model-based deterministic simulations of systems In reality, a system will have uncertainties In particular, the inputs to the system may be somewhat random Noise in a signal Belgian Blocks: automobile suspensions Flow-induced random vibrations of heat exchanger piping Irregularities in raw materials into a manufacturing process Random changes in financial conditions (such as interest rate) Random delays in a supply chain MG Lipsett,

13 Randomness in Technological Systems Some continuous probability distributions in manufacturing include: Interarrival times of orders, parts, or materials Loading and unloading times Processing times (including manual tasks such as inspection of products) Time for a machine to break under a defined operating condition Time for Initial set-up of a production line or to change a machine to do a different task MG Lipsett, Different Distributions for Different Processes Failure rates of machines often follow a gamma distribution, which has a lot of flexibility and a mean that is simpler to compute than the more general Weibull distribution Interarrival times often follow an exponential distribution (based on a Poisson process) If you are not sure what distribution to use, try to fit some data to a distribution If no data are available, then you will have to assume a distribution Look at the literature for similar types of processes MG Lipsett,

14 Estimating Probabilities and Loss Functions There are different places to get data related to probability distributions: Actuarial data (other systems) Empirical data (plant data, experiment) Probabilistic transformations (Jacobian) from other PDFs through system model MATLAB Statistics Toolbox can find the best fit of a number of different types of PDFs to a data set (including combinations) For example, normfit can be used for a single-variable (univariate) Normal (Gaussian) distribution MG Lipsett, Monte Carlo Simulation Monte Carlo simulation is a probabilistic analysis of a system Inputs are probabilities (distributions or functions) Process is the system model (differential equations or discrete-events) Output is the system response (or analysis result) Solution must be robust to the bandwidth of the inputs Output Inputs Process MG Lipsett,

15 Monte Carlo Method Monte Carlo simulation can be applied to deterministic problems (such as evaluating intractable integrals) More often, Monte Carlo simulation is used for probabilistic simulations, to show how random inputs result in random outputs, thereby revealing the system behaviour. This is like finding the frequency response function for a system by driving it with a Gaussian input which has equal signal power at all frequencies, thereby generating system output at all frequencies at which the system can respond. The key is to have a good idea of the nature of the probability distributions of each of the system inputs distribution (as discussed above) MG Lipsett, Monte Carlo with a Spreadsheet As we have seen, simple Monte Carlo Discrete-event simulation can be done using a spreadsheet Excel has a random-number generator and the ability to generate values from some well-known probability distributions, as well as the ability to generate summary statistics and charts for the outputs, subject to the following limitations: Simple data structures Calculations in cells cannot be very complex Number of iterations and number of variables must be fairly small so that solution time is reasonable and storage requirements are not excessive Lab #10 has a comprehensive example of using Excel for a simple queuing problem MG Lipsett,

16 Validating a Model Validation is the process of ensuring that the simulation is solving the model correctly The simulation must be solved in such a way that the simulation responds the way the system of interest would This is of particular concern when solving a set of stiff differential equations Validation relies on having a good understanding of what an actual system would do, which usually means having some data from the system of interest (or data for known elements that will be assembled for a new system) MG Lipsett, The Trouble with Data There are some potential problems with data: The data may not be representative of what you want to model: for example, a nuclear plant in normal operating conditions may not tell you the kind of system behaviour during an emergency. Data are in the wrong format or type, compared to how data would be used in the simulation The measurement method may be prone to error, methods of recording data may have errors, and data records may have round-off errors (e.g. resolution of scales) Data may be biased due to personal interpretation or bias Data may have the wrong units (short tons, long tonnes, metric tons ) MG Lipsett,

17 More Model Validation Issues Data errors fall into two types: Precision (how tight the measurements are) Accuracy (how close the measurements are to the actual) Depending on the system, data may not be collected under the same conditions, which affects reproducibility (and variance) Invasiveness means that the act of collecting data affects the system (there s noting like having a person with a clipboard watching you to improve your performance!) Temporal and spatial averaging may add a bias problem. Observer bias is an error that prevents someone from seeing a situation or a person objectively (such biases may be subtle and hard to detect!!) MG Lipsett, Non-linear LS Estimation Linear regression deals with fitting an observed set of data to a linear equation (i.e. y = ax + b). This method assumes that each observation can be described by the model fy is a linear combination of parameters of the form fy = x i1 θ 1 + x i2 θ 2 + x i3 θ 3 + where e is a random error with mean zero and variance. In this situation the parameters a and b (θ) can be solved for explicitly using the method of least squares. Many phenomena are described by nonlinear equations where the response is a nonlinear function of the independent variable(s) and parameter(s), making it difficult or impossible to solve for parameters explicitly. MG Lipsett,

18 MATLAB Implementation of NL-LSELSE In this (more general) case it is assumed that each observation can be described by the model but this model is not a linear function of the parameters θ The method of least squares is typically used to fit a set of data to this model. However, in many cases numerical methods must be employed in order to find the set of parameters that minimizes the residual. Fortunately, MATLAB has some built in tools to perform the numerical methods needed to solve nonlinear regression problems. The one presented here is nlinfit. MG Lipsett, MATLAB nlinfit Function MG Lipsett, nlinfit is used to find least-squares parameter estimates for nonlinear models. It uses the Gauss-Newton algorithm with Levenberg-Marquardt modifications for global convergence. This function is called as follows: >> beta = nlinfit(x,y,function,beta0) where beta is a vector of the parameters that is the solution to the nonlinear regression problem X is an n x p matrix of p predictors at each of n observations y is an n x 1 vector of observed responses function is a function handle to a function of the form yhat = modelfun(beta_hat,x) beta0 is an initial prediction of the parameters 18

19 Example (from MATLAB help file): The Hougen-Watson model for reaction kinetics is an example of a nonlinear model for which one may wish to estimate the parameters from an observed experiment. The form of this model is x β 3 1x2 β5 y = 1+ β x + β x + β x Given the data from an experiment, the unknown parameters β 1 β 5 can be found using MATLAB s nlinfit function as described in the next slides MG Lipsett, Example (2) First: the predictors should be in the form of an n x p matrix of p predictors at each of n observations. For example, for p=3 predictors x1, x2, x3: x1 x2 x3 Observation 1 x11 X12 x13 Observation 2 x21 x22 x23 Observation n xn1 xn2 xn3 Or in MATLAB >> X = [x11 x12 x13; x21 x22 x23; ; xn1 xn2 xn3]; MG Lipsett,

20 Example (3) Second: the observations should be in the form of an n x 1 vector In MATLAB >> observed_rates = [rate1; rate2; ; raten]; MG Lipsett, Example (4) Third: create an m-file to evaluate the rate given the predictors and an estimate of the parameters. function rate = hougen(beta,x) b1 = beta(1); b2 = beta(2); b3 = beta(3); b4 = beta(4); b5 = beta(5); x1 = X(:,1); x2 = X(:,2); x3 = X(:,3); rate = (b1*x2 - x3/b5)./(1+b2*x1+b3*x2+b4*x3); MG Lipsett,

21 Example (5) Fourth: call nlinfit to fit the model to the data A nonlinear estimation method searches for the solution, and so it requires an initial guess beta0 for the parameters beta: >>beta0 = [beta1; beta2 beta3; beta4; beta5]; You must ensure that the MATLAB current directory is set to the location where you saved the m-file with the function describing the equation of interest (here, hougen.m ) Then you call the nlinfit function: >> beta = nlinfit(x,observed_rates,@hougen,beta0) The vector beta will contain the best fit of the nonlinear equation in terms of the parameters and the data (predictors at different observations) MG Lipsett, Estimating Parameters of System Elements If it is possible to take a system apart and measure inputs and outputs for an element of a system, then it is straightforward to estimate the functional relationship between them (which, of course, is the constitutive relationship of the element) For example, a spring can be mounted in a testing machine, and the displacement measured at different forces applied to the element Note: if the element can exhibit any nonlinearities, then we have to have data across the entire range of inputs and outputs to characterise the parameters of the constitutive relationship completely Then, we simply plug the value of each parameter into the governing equations MG Lipsett,

22 Estimating Parameters (2) In most cases, we do not have the luxury of deconstructing the system, and we have to collect data from the system and attempt to estimate all of the parameters of the system at once. MG Lipsett, Summary We use models to give us predictive capability as to how a system may behave in some circumstances This understanding allows us to make better decisions than if we had to rely on our internal models of how a system may behave Good models need to be verified (and hopefully validated) For DES and lumped-parameter systems, parameter estimation gives us increased confidence; but the estimates depend on having good data for the system elements or events MG Lipsett,

Advanced Machine Learning Practical 4b Solution: Regression (BLR, GPR & Gradient Boosting)

Advanced Machine Learning Practical 4b Solution: Regression (BLR, GPR & Gradient Boosting) Professor: Aude Billard Assistants: Nadia Figueroa, Ilaria Lauzana and Brice Platerrier E-mails: aude.billard@epfl.ch,