A Comparative Evaluation of Models and Algorithms in Model-Based Fault Diagnosis

Transcription

1 A Comparative Evaluation of Models and Algorithms in Model-Based Fault Diagnosis L.A. Breedveld Parallel and Distributed Systems Group Faculty of Electrical Engineering, Mathematics, and Computer Science Delft University of Technology Master s thesis October, 2004

2 Abstract To increase safety and reliability, many devices are equipped with a diagnostic system. The diagnostic system is a software program that uses a model and observations of the system to create a diagnosis. Diagnosis is often a computational intensive activity. To build a good diagnostic system one has to make trade-offs between the required computing power and diagnostic accuracy. This thesis compares the diagnostic performance of various diagnostic techniques, and their required computing power. In particular, it studies the use of the following diagnostic techniques: Kalman filtering, hybrid mode estimation, and discrete reasoning. Kalman filtering reduces noise in a signal. Hybrid mode estimation uses multiple Kalman filters to find the current state of the system. Discrete reasoning uses discrete instead of continuous variables, for efficiency and modeling convenience. The diagnostic systems used for the tests were created using the Lydia toolset. Lydia is a system specification language. The model-based Lydia tools operate on this specification to perform various, diagnosis related, tasks. This thesis also describes the extension of the Lydia toolset with a software program that performs Kalman filtering and hybrid mode estimation. A simulation model of the Space Shuttle Main Engines was used as a case study on which to test diagnostic systems. Diagnostic systems were tested on their ability to detect failures of the rocket engine, and on the amount of CPU time they required to produce the diagnosis. The results of the test are as follows: The optimal variants of the diagnostic systems under study used about the same amount of CPU time. When the sensor noise is negligible, diagnostic systems that use discretization and discrete reasoning have the highest performance. When there is more sensor noise, the use of continuous domain techniques becomes necessary. Hybrid mode estimation, gives the highest average accuracy, however occasionally outputs spurious fault diagnoses. If this must be avoided, then discrete diagnosis has to be applied on the basis of Kalman filtered sensor readings. It was also necessary to apply an additional filtering of the diagnostic output, in order to decrease the number of false alarms.

3 Preface This report represents the results of a year s work in the field of model-based diagnosis. The research described in this paper has been done in the context of my graduation project at the Delft University of Technology. This research is part of the Lydia project which studies the diagnosis of systems by software. The project has been carried out in cooperation with the company Science & Technology. The reader will find that this report on model-based diagnosis is about a multi-disciplinary field. Beside the pure software aspects, topics from mathematics and physics were also important. This is when software becomes really useful, when it is applied to a particular field, in this case to the diagnosis of complex physical systems. Hopefully, this report will contribute to the development of more reliable and cost-efficient diagnostic systems. I would like to thank Arjan van Gemund and André Bos for their valuable guidance and advice during this project. Also a special thanks to my family for their help and encouragements. Thanks to God, who created and sustains everything by his power. Lucas Breedveld 1

4 Contents 1 Introduction Model-based diagnosis Diagnostic modules The role of a diagnostic system Thesis outline Related work Evaluating diagnostic systems Evaluation criteria Measuring diagnostic performance Measuring diagnostic accuracy Cost of inaccuracy, function Φ Cost of inaccuracy, function Γ Measuring computing resources costs Measuring development costs Implementation The Space Shuttle main engines Introduction The main engines Simulation Sensor placement The faults to be diagnosed The Lydia language and tools Basic syntax and semantics Dynamic systems Stochastic systems The Lydia toolset The Lydia simulator Simulating dynamic systems Update frequency Stabilization Simulating static systems Compress Connectors Global Continuous state estimation Kalman filter theory Kalman filter implementation Modeling options of the Kalman framework Evaluation setup Results

5 6.6 Conclusion Hybrid state estimation Hybrid observer theory Hybrid observer implementation Modeling options Results Discretization Test setup Results Discrete state estimation Discrete diagnosis Discrete modeling Using incomplete reasoners Compositionality Recursive influences Qualitative physics Custom thresholds Diagnostic options Results Complete diagnostic systems Module structures Test setup Results Diagnosis and supervisory control Choosing on one health state as diagnosis Filtering the diagnostic output Test setup Results Using no diagnosis filtering Using diagnosis filtering Conclusion Results of this project Recommendations A Lydia simulation models 71 A.1 The compress approach A.2 The connector approach A.3 The global approach B Diagnostic Lydia models 74 B.1 Hybrid diagnoser B.2 Zero threshold discrete diagnoser B.3 Custom threshold discrete diagnoser

6 Notation Domains and variables B Boolean domain {false, true or equivalent {0, 1 E n enumerated domain {0, 1,..., n 1 (B = E 2 ) R real continuous numbers R n n-dimensional vector of reals a scalar variable, or a constant a vector variable A matrix variable A T the transpose of A ˆx estimation of x P (A) probability (or relative frequency) that predicate A is true Special variables u control input x internal system state y measurements h health of the system 4

7 Chapter 1 Introduction To increase safety and reliability, many devices are equipped with a diagnostic system. Advanced systems, such as cars, spacecrafts and chemical plants, have a build-in diagnostic system to ensure safety and optimal performance. The diagnosis is automated; the diagnostic system is an (embedded) software program. The task of a diagnostic system is to prevent or detect faulty behavior of the system and to find the root cause of a failure. As systems grow larger and become more complex and more integrated with each other, the probability that one of its components fails becomes larger. Also, the impact of component failures increases, as now larger systems are depending on the component s functionality. Therefore, the need for accurate diagnostic systems is increasing. The task of a diagnostic software system can be expensive in terms of the required computing power. System behavior can be so complex that it becomes computationally infeasible to reason about every possible behavior. Although the cost of computing power is decreasing, it is still inevitable to make an optimal trade-off between diagnosis accuracy and speed. The research described in this report compares various diagnostic techniques in terms of their cost / performance ratio. It will give an indication of the expected gain in diagnostic performance, when switching to more advanced techniques. Furthermore, it will show the price for this performance gain, in terms of the required computing power. We will develop a method for measuring the diagnostic performance. This thesis will not provide a conclusive answer to the problem of choosing the most suitable diagnostic technique. However, by describing the evaluation method in detail, we ensure that the results can be extended by further research of other diagnostic techniques, or other application areas. 1.1 Model-based diagnosis In this report, we focus on the model-based approach to creating a diagnostic system. In this approach, a diagnostic system consists of two parts: a system model, and a reasoning engine (See Figure 1.1). The model contains a description of the behavior of the system in nominal and faulty situations. This model should be formal and standardized enough to be interpreted by a computer. If a model of the system is known, the rest of the diagnostic task can be automated. The reasoning engine contains general diagnostic algorithms that internally use the model of the system. These algorithms can be reused for multiple systems; the only thing that has to be changed is the model. The model can be written by domain experts, while the model-based algorithms are written by programming experts. In this project we will use the Lydia modeling language. The Lydia language was designed to be the model specification language used during model-based diagnosis. A model specification language is a bridge between the modeler, and the model-based algorithms. Internally, the model-based algorithms might have a very complex way to store and handle models. For the user this is not important, because he can specify the model in a (relatively) convenient modeling language. This language is automatically parsed and converted to a structure suitable for the model-based algorithms. A model-based algorithm compares the real system behavior with the model-predicted system behavior, to find out the health state of the system. A diagnoser gets information about the real system behavior from sensors. When the system is controlled, the diagnoser also needs information from the controller about its commands. The search for the health state that matches the observations, is a difficult search problem because there are many health states possible. For example, a system that contains 5

8 Control commands Controller The real system Measurements System model Reasoning engine Diagnosis Figure 1.1: Model-based diagnosis 14 components which can be either healthy or faulty, the total number of possible health states is Therefore, most algorithms do a best-first search, by starting with the most likely health state, and then moving to less likely states until enough probability mass has been covered. 1.2 Diagnostic modules On the implementation level, a diagnostic system often consists of smaller modules 1. The module structures that we will study have the simple form of a pipeline. The first module receives the observations, the last outputs the diagnosis. We will use the convention of writing a module in Small Capitals, while writing the name of a diagnostic system with a normal font. Figure 1.2 shows the three module structures that we will compare and evaluate in this report. We will now explain the basic principles behind these three module structures. When modeling a system, the most accurate way would be to model it in the continuous domain. Even a digital circuit contains real valued voltages and currents. However, for efficiency and convenience, it is often advisable to choose to model and diagnose in the discrete domain. Relations between variables are then expressed using qualitative expressions, instead of quantitative. We call the module that estimates the discrete health state using a discrete model and observations: Discrete Estimation. If we decide to use a discrete diagnostic module, then the input to this module must also be discrete. When the sensor readings have a real-valued character, these have to mapped to the discrete domain first. We call the module that performs this mapping: Discretization. The way discretization is applied greatly effects the diagnostic performance. Because the discrete domain is much smaller then the continuous domain, we must be careful to preserve important diagnostic information. From the two modules that we have discussed now, we build a diagnostic system that is called Discrete in Figure 1.2. Another problem with sensor readings is that they might be noise infected. If they are, then it is advisable to do a form of noise filtering. Filtering can be done most effectively if the filter is supplied with a model of the system. This model must be in the continuous domain, because the noise infected sensor readings are also in the continuous domain. The filter estimates, using the sensor readings, the value of the variable that the sensor is measuring. We call this module: Continuous Estimation, because it does estimation in the continuous domain (R). From the three modules that we have discussed now, we build a diagnostic system that is called Filtered in Figure 1.2. The module Continuous Estimation is configured by a continuous model. This model describes the continuous dynamics. Sometimes it is not certain which continuous dynamics are operating at the moment. This is true especially when a component fails. If a component fails, the continuous dynamics often change drastically. The number of different dynamics that can occur, can be modeled as being finite 1 We will use the term module to indicate a part of the diagnostic software, and the term component to indicate a physical part of the system 6

9 and expressed by a discrete variable. The module Hybrid Estimation models the system with both a discrete and a continuous part. The continuous part is tracked by multiple Continuous Estimationmodules. The discrete part is estimated, by comparing the performance of the continuous estimators. The continuous estimator that has made the correct assumption about the current discrete state of the system, will have the highest performance. This structure is used by the Hybrid diagnoser, as depicted in Figure 1.2. The discrete estimation is used as a diagnosis, while the continuous estimation is thrown away. 1. Discrete: observations DISCRETIZATION DISCRETE ESTIMATION diagnosis 2. Filtered: observations CONTINUOUS ESTIMATION DISCRETIZATION DISCRETE ESTIMATION diagnosis 3. Hybrid: observations CONTINUOUS ESTIMATION HYBRID ESTIMATION diagnosis Figure 1.2: Module structures In this report we will study diagnostic systems in terms of its modules. By examining modules separately, this examination can be more detailed and easier to understand, then if we would examine complete diagnostic system at once. However, to understand the discussion of a particular module we must remember that it has a role in a larger diagnostic system, as discussed in this section. At the end of the report, we will test the three complete diagnostic systems of Figure 1.2. We will compare the diagnostic accuracy of these systems, and the amount of processor time they require to produce a diagnosis. 1.3 The role of a diagnostic system The controller and the real system together form a feedback control loop. The controller uses sensor readings to adjust his control commands. For the long term, this task is performed by the supervisory controller (See Figure 1.3). The diagnosis of the diagnostic system is used by the supervisory controller as an advanced sensor that measures the health state of the system. Based on the diagnosis, the supervisory controller takes actions to move the system in a better state. For example, the supervisory controller can be a automated reconfiguration planner, which switches to backup components when the diagnosis indicates a fault has occurred. It can also be a human operator, who uses the diagnosis to choose when and what to repair. The diagnosis that the diagnostic system produces, eventually influences the system state. The diagnosis is used by the supervisory controller to take corrective actions. The diagnoser is created, in order to enable the supervisory controller to influence the system state positively. So in order to evaluate the performance of a diagnoser, the most accurate method would be to measure the effect of the diagnoser on the system state, when using the best available supervisory controller. Therefore, this report will also study the supervisory controller, and a part of the evaluation of diagnostic systems will be based on that. 1.4 Thesis outline This report is organized as follows. Chapter 2 starts with a description of the method that we will use to evaluate diagnostic systems. The subject of Chapter 3 will be the Space Shuttle Main Engines, which 7

10 Controller The real system repair, reconfiguration supervisory controller System model Reasoning engine Diagnosis Figure 1.3: The role of a diagnostic system is the system on which we will test diagnostic systems. Chapter 4 and 5 describes further characteristics of the test environment that we will use. Chapter 4 contains an introduction to the Lydia modeling language, and the available Lydia tools. In Chapter 5 we will discuss the properties of the Lydia simulator, and how the simulator can be used to simulate the Space Shuttle Main Engines. After the test environment has been discussed, we will examine the four diagnostic modules under study, in four separate chapters (Chapter 6 to 9). In these chapters, we will test the performance of the diagnostic modules as much as possible in separation of the other modules. However, the most reliable and interesting results are found in Chapter 10, that compares complete diagnostic systems built from the modules that were discussed in previous chapters. Chapter 11 also compares complete diagnostic system, but then from the perspective of supervisory control. The main contributions of this thesis are the following: ˆ Definition of a method for evaluation of diagnostic systems (Chapter 2) ˆ Study of modular modeling in the continuous domain (Chapter 5) and in the discrete domain (Chapter 9) ˆ Description of the creation of a software program that applies Kalman filtering (continuous estimation) and hybrid estimation (Chapter 6 and 7) ˆ The comparison of discrete versus continuous diagnostic systems (Chapter 10 and 11). 1.5 Related work The Space Shuttle Main Engines will be used as a case study on which to test diagnostic systems. The diagnosis of this engine has been done previously using a simple and fast real-time algorithm [16], and using a complex non-linear thermodynamic model [12]. The approach taken in this report is somewhere in between. Most of the diagnosis related literature is about one particular diagnostic approach, or about a set of techniques. One of the notable exceptions is the report of Dash and Venkatasubramanian [5] in which a relatively large group of diagnostic techniques are evaluated. The cited paper compares techniques such as Extended Kalman filtering, Neural networks, and Causal models. These diagnostic techniques are related, but not equal to those used in this report. Similar to this report, the paper provides a comparison of diagnostic techniques in terms of their accuracy, and their computational and modeling requirements. 8

11 Chapter 2 Evaluating diagnostic systems In this chapter a method is developed for the evaluation of diagnostic systems. Later in this report, we will apply the method described here. The goal is to describe an evaluation method that provide a repeatable and objective procedure to measure the benefit and costs of a diagnostic system. 2.1 Evaluation criteria The value of a diagnostic system depends on the following three important aspects: 1. Diagnosis accuracy: the measure in which the diagnosis produced by the diagnostic system agrees with the reality 2. Computing resources: the amount of CPU time and memory the algorithm of the diagnostic system needs. 3. Development costs: the cost of writing a model-based diagnostic program, and the cost of creating a model of the system to be diagnosed The next section contains a discussion on how to measure diagnostic performance. The sections after that discuss separately the measuring of the three aspects mentioned above. The last section completes the chapter with a discussion on the implementation of a program that evaluates diagnostic systems. 2.2 Measuring diagnostic performance For the most accurate testing of diagnostic systems, we should test the diagnostic capabilities on all systems that we want to diagnose. The number of systems that we want to diagnose will probably be large, so this approach would take too much time. In addition, the system to be diagnosed is sometimes not yet available; it is yet to be build. Therefore, it is advisable to choose one or more representative systems that share most of their characteristics with the real systems that must be diagnosed, and to test only on these systems. The goal of this research is to find diagnostic systems that perform well on systems of the domain of spacecraft, because the initiator of this research (Science & Technology) is working in this domain. Therefore, we choose to test diagnostic systems on one particular spacecraft sub-system, that is not too large, yet sufficiently complex to highlight the strength and weaknesses of the various diagnosers under study. We will test diagnostic systems on the Space Shuttle Main Engines (SSME). A description of the SSME is given in Chapter 3. We must bear in mind, however, that this is a case study, so it results should be interpreted with care. There is no guarantee that the performance ratio of tested diagnostic systems will be the same for other systems. The diagnostic systems will be tested on a simulator of the SSME. Figure 2.1 shows the setup that is used to evaluate a diagnostic system. The reasons for using a simulator are: ˆ Cost: Testing on the real system is more costly, then using a simulator. 9

12 health: h observations: Simulator u,y Diagnostic system health estimate: ^ h <executes> information flow Evaluator value(diagnoser) Figure 2.1: Evaluation setup ˆ Control: In a simulation model, it is easy to artificially introduce a fault, or decrease the sensor noise of a particular sensor. In the real system this would be much more difficult, or even impossible. ˆ Evaluation: It is easier to evaluate a diagnoser when using a simulation model. Because we have introduced the faults ourselves, we know exactly what the diagnosis should be. A property of a good diagnostic system, is that it performs well in all situations. In particular, the importance of the scenario where all components are healthy should not be underestimated. Diagnostic systems have been known to emit many false alarms in a scenario where all component were okay, with the result that the diagnostic system was put off. We define a scenario (s) as a time series of health vectors (h) and corresponding observations (u, y). A scenario starts at t = 0 and stops at t end. For the sake of simplicity, we assume the scenario takes timesteps of 1. s = < h(0), u(0), y(0) >, < h(1), u(1), y(1) >,..., < h(t end ), u(t end ), y(t end ) > (2.1) The evaluator supplies health vectors (h) to the simulator. In most scenarios, we will use a health vector that is constant; the fault is introduced at t = 0 s. The simulator simulates the system, and creates corresponding control commands (u) and sensor readings (y). These are supplied to the diagnostic system. For the diagnostic system the simulated health vectors are of course invisible. The cost of diagnostic system is the sum of its cost in all possible scenarios. A scenario must be weighted according to its probability: C(d) = s S P (s) C(d s) (2.2) where: C(d) : cost of diagnoser d P (s) : probability of scenario s C(d s) : cost of diagnoser d in scenario s S : the set of all scenarios While testing a diagnostic system, we will evaluate it on a smaller set of scenarios (S S), that are representative for the scenarios that occur in the real world. The probability of a scenario is then the probability that the scenario will be (of all other available scenarios) the best approximation of what will happen in reality. The set of scenarios that will be tested will focus on the coverage of possible health states. We will test the all-healthy scenario, and single and double faults. In this study, the health vector h will contain binary values. If h i = 1, than the i-th component is healthy, and if h i = 0, than the i-th component is 10

13 faulty. We will assume that component failures are independent of other failures. Using this assumption, the probability of a scenario can be calculated as follows: where: P (s) = n P (h i = h s,i ) (2.3) i h s,i : the health state of the i-th component in scenario s P (h i = 1) : the relative frequency that the i-th component is healthy. This is also called the a-priori probability: that is the probability before any observations are done. n : the number of components 2.3 Measuring diagnostic accuracy Diagnosis accuracy is the most important aspect of a diagnostic systems. Diagnostic accuracy is the measure in which the diagnosis produced by the diagnostic system agrees with the reality. It is more convenient to define a measure of the diagnostic inaccuracy, then of the accuracy. In this section we will develop two cost functions that measure the cost of diagnostic inaccuracy. The cost of an inaccurate diagnosis is difficult to estimate. It depends on the particular system, and also on how the diagnosis is further used. Both cost functions will be based on the assumption that a diagnosis is used by a supervisor to bring the system into a better state. If the diagnosis indicates that a component is faulty, while the component is healthy, then there is the cost of a false alarm (Abbreviated C F A ). On the other hand, if the diagnosis indicates that a component is healthy, while it is not, then there is the cost of a missed detection (Abbreviated C MD ). The exact cost of a false alarm and a missed detection will depend on the particular application. The evaluation of diagnostic systems will depend on the value of these parameters. For example, in a safe-critical system a false alarm will probably be much less costly than a missed detection, if it relates to the same component. If a missed detection is so costly, a diagnostic system that quickly emits an alarm must be valued higher, than a more conservative diagnostic system. The next two subsections will define two cost functions: Φ and Γ. Cost function Φ will define a measure of the cost of inaccuracy of the output of the diagnostic system. Cost function Γ defines a cost measure for use after the supervisory controller has acted on the diagnosis. For this purpose a simple model of the supervisory controller will be used Cost of inaccuracy, function Φ This first cost function estimates the cost of diagnostic inaccuracy. The following formula gives the calculation of the cost of a diagnostic system in a particular scenario. The formula contains two summations: one over all the timesteps in the scenario, the other one over all the (physical) components. For each timestep, and for each component, the estimated health must be equal to the real health. If not, then there is some cost: either the cost of a false alarm or of a missed detection. where: Φ(d s) = t 1 end t end + 1 t=0 n i C MD, ĥs,i(t) = 1 h s,i (t) = 0 C F A, ĥs,i(t) = 0 h s,i (t) = 1 0, otherwise Φ(d s) : the cost of inaccuracy of diagnoser d in scenario s t end : the last timestep of the scenario ĥ s,i (t) : the health estimation in scenario s at time t of the i-th component h s,i (t) : the real health in scenario s at time t of the i-th component C MD : the cost of a missed detection C F A : the cost of a false alarm n : the number of components (2.4) 11

14 The cost of multiple scenarios can be combined using Equation 2.2, where each scenario is weighted according to its relative probability. A further complication of diagnostic evaluation is that a diagnostic system might output multiple diagnoses for a single timestep. It is logical that the diagnostic output will differ for different timesteps. But also within one timestep, a diagnostic systems often outputs multiple diagnoses. It does so, when it is not certain which diagnosis is the correct one. For example, at a certain point in time, a diagnostic system for a four-component system, might output the following: probability health vector 0.45 [ 1, 1, 0, 1 ] 0.45 [ 1, 0, 1, 1 ] 0.10 [ 1, 0, 0, 1 ] If the probability mass does not add up to one, this must be corrected first by multiplying the probabilities by a suitable factor. Cost function Φ evaluates each diagnosis separately, and weights the cost according to their probability. The final cost function Φ is therefore: where: Φ(d) = t 1 end P (s) t end + 1 s S t=0 m(t) j P (j) n i C MD, ĥs,i,j(t) = 1 h s,i (t) = 0 C F A, ĥs,i,j(t) = 0 h s,i (t) = 1 0, otherwise m(t) : the number of outputted diagnoses at time t P (j) : the probability of the j-th outputted diagnosis, as assigned by the diagnostic system P (s) : the probability of the scenario s How much testing is required for function Φ? We will discuss how much testing we must do in order to get a reliable value of the cost of inaccuracy Φ of a diagnostic system. If we test a diagnostic system only for at most single faults, we would like to know how good that approximation is. This is important if we have calculated Φ for two diagnostic systems. Let us call these diagnostic systems A and B. If A has a lower Φ then B, we want to know if it is possible that by testing A and B also on double faults, this ordering between A and B will change. We know for sure that the ordering will not change if: when B performs perfectly on double faults, and A performs as worse as it can be on double faults, A will still have less inaccuracy costs than B. Therefore, it is useful to know the minimum and maximum costs on the scenarios that were not covered in the tests. The minimum cost on a scenario is zero. The maximum cost on a set of scenarios is a more complicated to calculate. Assume we have tested a diagnostic system on all scenarios with at most f faults. The maximum costs in the remaining scenarios can be calculated using the following formula: n ( ) n C max = P (h i i = 1) n i (1 P (h i = 1)) i ((n i) C F A + i C MD ) (2.6) i=f+1 The summation in the formula is over the number of faults. A scenario with i faults can be created by choosing i out of n components to be faulty. The scenario is weighted according its probability. The maximum cost is reached if for all components that are healthy an alarm is started ((n i) false alarms), and for all faulty components no alarm is started ( i missed detections). For a particular application, we need to fill in parameters to calculate the maximum inaccuracy cost. These parameters are the probability of a fault, the cost of missed detection and false alarm, and the number of components. Table 2.1 shows the maximum costs in not-covered scenarios, when we have covered scenarios with a particular maximum number of faults Cost of inaccuracy, function Γ The cost function Γ evaluates the inaccuracy of the diagnostic system, after the supervisory controller has acted on the diagnosis. For this purpose a simple model of a supervisory controller will be used. More 12 (2.5)

15 test covering Maximum extra cost nothing maximum no faults 8.77 maximum single faults 1.66 maximum double faults 0.18 maximum triple faults 0.01 Table 2.1: Maximum extra cost in scenarios not covered, when using cost function Φ (or Γ). Assumptions: P (h i = 1) = 0.98, C MD = 20, C F A = 1, n = 14 health: h observations: Simulator u,y Diagnostic system health estimate: ^ h Supervisory <executes> controller information flow Evaluator decisions: q value(diagnoser) Figure 2.2: Evaluation setup for cost function Γ details about this will be discussed in Chapter 11. The testing setup needed for function Γ is shown in Figure 2.2. The supervisory controller makes a zero or one decision (q i ) about a component, whether it is healthy or not. If it assumed to be faulty, the particular component is checked or repaired. Another interpretation of the decision of the supervisory controller, is that it starts an alarm of the components it thinks are faulty. It never changes its decision about a component. If a component is marked as faulty, it stays marked faulty. The cost of this behavior of the supervisory controller is measured by the Γ function. The Γ function is evaluated one time, at the end of the scenario, as opposed to the Φ function, which is evaluated at each time step. At the end of the scenario, the Γ evaluates the decision of the supervisory controller. In the following function we assume that if a component is faulty, it is faulty from t = 0 to the end of the scenario. where: Γ(d s) = n i C MD, q s,i = 1 h s,i = 0 C F A, q s,i = 0 h s,i = 1 t s,i,detect C DD, q s,i = 0 h s,i = 0 0, otherwise t s,i,detect : the time at which the supervisory controller detected a fault at component i-th and acted upon it q s,i = 1 : supervisory controller has done nothing with component i, it has assumed it was healthy q s,i = 0 : supervisory controller has checked or repaired component i, it has assumed it was faulty C DD : the cost of a detection delay, per second. The fault occurred at t = 0 s, and is detected t s,i,detect later. 13 (2.7)

16 The cost of multiple scenarios can be combined using Equation 2.2, where each scenario is weighted according to its relative probability. The effect of multiple diagnoses in a single timesteps, does not have to be considered, because this is handled by the supervisory controller. The final cost function Γ is therefore: Γ(d) = C MD, q s,i = 1 h s,i = 0 n C P (s) F A, q s,i = 0 h s,i = 1 (2.8) t s,i,detect C DD, q s,i = 0 h s,i = 0 s S i 0, otherwise How much testing is required for function Γ? There exists a perfect analogy between the amount of testing required for the Φ function and the amount of testing required for the Γ function. The results in Table 2.1 can be used without modification. 2.4 Measuring computing resources costs A diagnostic system uses computing resources to perform its task. In particular, it uses memory and CPU time. Due to technical reasons, the amount of memory used by the diagnostic system will not be not be measured, only the amount of CPU time. The CPU time is measured by the evaluator. Before the diagnostic system is called to produce a diagnosis, the evaluator gets the current time from the operating system. Then, when the diagnostic system has finished its output, the current time is fetched again, and the difference is calculated. The time needed by an empty diagnostic system will also be measured. This is a measurement of the overhead that is introduced by starting and stopping the diagnoser and by the evaluator itself. 2.5 Measuring development costs An important criteria for evaluating diagnosers, are the development costs. Often, the development costs outweighs the runtime costs of a diagnostic system. It is more expensive to create an accurate model or to develop and maintain a model-based algorithm, then to actually run it on the system. Therefore we will also look at the development costs of diagnostic systems, although it is not the main purpose of this research. The development costs include both the cost of the development of the model-based algorithm, and of the model itself. Not all of the development of the model-based algorithms that were used in this research, were done by the author of this report. Therefore, we will only give an indication of the cost of creating the model. The modeling costs will be measured according to the time that is needed to produce a model that is more or less on the performance upper bound of what is possible within a certain approach. 2.6 Implementation The evaluator is a software program called testdiag, which tests diagnosers. The program testdiag can also be useful during debugging, because it provides convenient ways to analyze the performance of a diagnostic system on various scenarios. The program consist of about 1100 lines of C-code. The operation of the program can be configured by a specification file and with command line options to control the following parameters: ˆ the simulation model to be used ˆ the diagnostic system to be evaluated ˆ parameters (such as sensor noise), and the range in which they must be tested ˆ which faults to insert and when ˆ how to summarize the results (for example: output the performance against the value of a parameter) 14

17 ˆ test performance using cost function Φ or Γ (before or after the supervisory controller) ˆ the probability that a component is faulty (used to weight scenarios properly) ˆ the cost of false alarms, missed detections, and detection delays The program testdiag uses the Lydia simulator (called lsim) to produce observation data. (For more information about the Lydia language and tools, see Chapter 4 and 5). The output data are passed on to the diagnostic system that is tested. A diagnostic system is configured in testdiag by the name of an executable file. This executable is the diagnostic system. During this project, this executable was a Linux shell script that calls the model-based algorithms, and converts to and from different data formats, if that was necessary. The output of the diagnoser is read and evaluated by the evaluator testdiag. Finally, the evaluator outputs a performance statistic for the diagnostic system. 15

18 Chapter 3 The Space Shuttle main engines The Space Shuttle Main Engines (SSME) [6, 4] will be used as a case study to test diagnostic algorithms. This system is sufficiently complex to highlight diagnostic principles, and the strengths and weaknesses of the various diagnosers under study. The evaluation setup includes a simulation model of the SSME on which the diagnosers will be tested. 3.1 Introduction One can easily image the importance of diagnostic algorithms for the Space Shuttle. Imagine you are seated in the Space Shuttle, counting down and waiting for the rocket engines below to start pushing you towards the sky. Can we be sure that this time the launch will go well? While you are waiting a controller mounted on the rocket engine is monitoring its health condition. Are all the valves and sensors functioning? A valve displacement of only 2% can result in serious damage to the engine. Finally the start command is given. The Main Fuel Valve is immediately ramped to fully open in two thirds of a second, allowing the liquid hydrogen to flow into the combustion chambers. Then, carefully, the liquid oxygen is provided, and the combustion is ignited. The controller does a safety check to verify that all is going well. If not, there is still the possibility to cancel the launch. Once the solid rocket boosters are ignited, the launch can not be canceled anymore. 3.2 The main engines The Space Shuttle has three main engines that are used to create thrust during the launch. During the first part of the launch they are assisted by the two large rocket boosters, that are dumped in sea after 16

19 hydrogen pump <power supply> <power supply> pump oxygen pump MFV FPOV MOV OPOV pump <power supply> <power supply> Fuel Preburner Oxygen Preburner liquid hydrogen Combustion chamber liquid oxygen gas Figure 3.1: Flowpaths and components of a SSME use. The main engines can be reused after multiple missions. The main engines will be the subject of our study. The Space Shuttle Main Engine (SSME) is one of the most efficient and powerful propulsion systems. Compared to its relative small weight, it produces an enormous amount of thrust. The engine has to operate under extreme conditions: the liquids are very cold, and the pump pressures are very high. The engines operate by burning liquid oxygen and liquid hydrogen. The liquid oxygen and hydrogen are stored in large external tanks, under a moderate pressure of P a (3.4 bar). For the combustion to operate properly, the pressure of the liquids has to be increased to about P a (280 bar). To pressurize the liquids, two sequential turbopumps are used. Before the oxygen and hydrogen are mixed in the main combustion chamber, they are first preburned. This is done in order to create an optimal mixture ratio, and it is used as a power source to operate the high pressure pumps. The preburner on the oxygen side is called the Oxygen Preburner, the preburner on the hydrogen side is called to Fuel Preburner. This name only indicates the placement of the preburner. The combustion in the Oxygen Preburner accelerates a turbine that is used to operate the second oxygen pump. Although the preburner turbine and the pump are mounted on the same shaft, the mixing of the combustion gases with the liquid oxygen must be prevented by all means. As Figure 3.1 shows, the flowpaths of the liquid and hydrogen are symmetrical to a certain degree. Both liquids are routed to two sequential turbopumps, and then routed to the two preburners. A difference is the placement of the valves, and the third oxygen flowpath that bypasses the preburners and goes directly to the Main Combustion Chamber through the Main Oxygen Valve (MOV). Most of the liquid oxygen is routed through this path. Not all flowpaths that exists in the engine are drawn in the figure, for example there exists a small hydrogen flowpath that is used to cool the nozzle of the engine. The figure shows the part that will be 17

20 100 OPOV FPOV MOV MFV Valve positioning command [%] Time [s] Figure 3.2: Valve commands during the startup sequence and the first few second feedback control. studied in the rest of this report. The valves of the engine are controlled by a controller that is mounted on the engine. The controller itself also receives commands from and sends data to the major data processing system of the Space Shuttle. During the start sequence, the valves have to be controlled very accurately in order to ensure a safe and reliable start. During the first few seconds the valves are commanded in a predefined pattern, later on two of the valves (OPOV and FPOV) are used to control the mixture ratio and the power level (See Figure 3.2). The other two valves (MOV and MFV) remain fully open. The valves are constructed in such a way, that when the valve is initially opened, only a small leakage flow passes through the valve. The normal flowpath is opened only after the valve is commanded more than 50% to open (Figure 3.3). 3.3 Simulation Arguably, the most accurate way to test diagnostic algorithms is to test them on the real system. Figure 3.4 shows one of the Space Shuttle Main Engines in a test environment. Although it would have been nice to have one of the engines at the ninth floor of the EEMCS 1 building in Delft, in this project the diagnostic algorithms were tested using a simulator. The simulation model of the SSME discussed in this section was not created with the purpose of being very accurate, but for the purpose of evaluating diagnostic algorithms on a complex test system. For an accurate thermodynamic model of the SSME the reader can refer to the model created by Rockwell International (the creators of the engine) [2, 12]. With respect to the simulation and diagnosis of the engine, it is interesting to note the strong interconnection of the components. Let us consider, for example, what will happen if the first hydrogen pump is stuck. First, the pressure out of the first pump will be lower, then the pressure out of the second pump will be lower as well, and as a result the hydrogen flow will be less. This means the Oxygen Preburner will receive less hydrogen, and produce less power. So the second oxygen pump will pump less fast, resulting in less oxygen flow, etcetera. The system stabilizes to a new equilibrium. To show how this is modeled, let us consider the pump and preburner component. The pump component increases the pressure (p) depending on flow (f) through the pump and the energy (E) provided by the preburner: p outlet = p inlet + E f + C 1 1 Electrical Engineering, Mathematics and Computer Science 18

21 flow percentage [%] valve positioning command [%] Figure 3.3: Effect of valve control commands on the flow. Figure 3.4: One of the Space Shuttle Main Engines in a test environment 19

22 where C 1 is a constant, which depends on the efficiency of the pump. The liquid flow depends on the outlet pressure of the pump, and the resistance of the system (R sys ): f = p outlet R sys The work done by the preburner depends on the liquid flow into the preburner. The preburner is modeled here as if it were a resistance. In reality, the preburner creates a back-pressure by preburning the liquids. E = k f 2 R preburn where k denotes the efficiency of the pump, which decreases when there is a large flow into the preburner (formula is not stated here). These equations together show one of the recursive influences. The recursive influences propagate iteratively through the system; there is some delay between the components. 3.4 Sensor placement Diagnostic systems will be tested on two different sensor setups. The first setup contains only the sensors that are needed for the normal operation of the SSME. The system needs five flow sensors (f1 to f5). These sensors are used to measure the mixture ratio in the preburners and the Main Combustion Chamber. One pressure sensor is used to measure the power level (p6) (Figure 3.5). No additional diagnostic sensors are used. The second sensor setup contains additional sensors at strategic places, in order to diagnose more faults. Four pressure sensors measure the outlet pressure of the four turbopumps (p7 to p10). Additional redundant sensors (f11 to f13, p14) are used to make it easier to distinguish between a valve and a sensor failure. hydrogen pump <power supply> <power supply> oxygen pump p7 p8 p10 p9 pump <power supply> MFV FPOV f1 f3, f11 Fuel Preburner MOV OPOV f4, f2 f12 Oxygen Preburner f5,f13 pump <power supply> Combustion chamber p6, p14 Figure 3.5: Sensor placement 3.5 The faults to be diagnosed Diagnostic systems will be tested on their ability to estimate the health of the components of the SSME. 14 different components are distinguished. 20

23 ˆ 4 turbopumps: The turbopump failures will be simulated as complete failures, where the outlet pressure equals the inlet pressure. Normally a turbopump is able to increase the pressure. ˆ 6 sensors: Sensor failures will also be simulated as complete failures, where the sensor output is zero (short-circuited). At the startup of the SSME, or when a nearby valve remains stuck-closed, a healthy sensor will also measure zero, so this fault is not as easy to detect as it seems. ˆ 4 valves: When a valve fails it will not respond to control commands. If this happens before the startup, then diagnosis is relatively easy, because then the valve will remain totally closed. We will also study the more subtle faults, when a valve is stuck half way between open and closed. 21

24 Chapter 4 The Lydia language and tools The Lydia language [19] and its toolset are used extensively in this project. The language is designed to be the model specification language used during model-based diagnosis. This section discusses its characteristics and its syntax. Lydia is a language in which one can specify the behavior of a system. This specification can be used for many tasks, such as diagnosis, simulation, and estimation. 4.1 Basic syntax and semantics The statements in the Lydia language are more or less similar to mathematical formulas. The statements in Lydia are declarative, not imperative. A statement like a = a + 1 is not allowed, because this can never be true. All statements in a Lydia model are always true. Let us consider the following Lydia model of an inverter 1 : // A d i g i t a l i n v e r t e r system i n v e r t e r ( bool in, bool out ) { out = not in ; The first line is a comment. The second line specifies the name of the system, and that in and out are boolean variables. The third line states that the output of an inverter is the inverse of the input. An important property of the Lydia language is that components of a system can be specified, so that they can be reused in different places. While building simulation or diagnosis models, it is advisable to try to use only one component specification for one physical component. This goal, however, is often difficult to attain [3, 18]. The following Lydia model describes how the inverter component, can be reused to build a system of three consecutive inverters. The intermediate variables b and c are used to specify how the three inverters are connected. system i n v e r t e r ( bool in, bool out ) { out = not in ; system t h r e e i n v e r t e r s ( bool in, bool out ) { i n v e r t e r ( in, b ) ; i n v e r t e r (b, c ) ; i n v e r t e r ( c, out ) ; This code is equivalent to the following code without components: system t h r e e i n v e r t e r s ( bool in, bool out ) { b = not in ; c = not b ; out = not c ; 1 An inverter is a digital component that inverts its input. If its input is 0, it outputs 1. And conversely, if its input is 1, its output is 0. 22