Data Driven Modelling for Complex Systems
|
|
- Janel Fox
- 5 years ago
- Views:
Transcription
1 Data Driven Modelling for Complex Systems Dr Hua-Liang (Leon) Wei Senior Lecturer in System Identification and Data Analytics Head of Dynamic Modelling, Data Mining & Decision Making (3DM) Lab Complex Systems & Signal Processing Research Group Department of Automatic Control & System Engineering University of Sheffield Sheffield, UK July 6-8, 017 HLW, Data Based Modelling, Slide 1 of 45 Data Driven Modelling Similar Terminologies: Black-box Modelling Data-based Modelling Learning from Data System Identification, etc. July 6-8, 017 HLW, Data Based Modelling, Slide of 45 1
2 Outline of the Talk 1) A quick selective review of system models Model types and generalised linear models ) Three classes of models White, grey and black-box models 3) Data-driven modelling Letting data speak and learning from data 4) Examples Using data-driven modelling techniques July 6-8, 017 HLW, Data Based Modelling, Slide 3 of 45 PART 1 A Quick Selective Review of System Models July 6-8, 017 HLW, Data Based Modelling, Slide 4 of 45
3 Linear vs Nonlinear Linear Systems Nonlinear Systems Linear systems are simple systems whose structure and behavior are regular and easy to understand. Nonlinear systems are complex systems whose structure and behavior are normally difficult to analyse. July 6-8, 017 Linear vs Nonlinear Models Linear Models Continuous-time model Continuous-time state space Continuous-time transfer function Discrete-time model Discrete-time state space Discrete-time transfer function (e.g. Z-transfer function) Nonlinear Models Continuous-time NL model Continuous-time NL state space Continuous-time transfer function for a nonlinear system? (No, not really!) Discrete-time NL model Discrete-time NL state space Discrete-time transfer function for nonlinear systems? (No, not really!) July 6-8, 017 Dr HL Wei, Data Based Modelling, Slide 6 of 45 3
4 Parametric & Nonparametric Models Parametric models ODE s (lumped parameter model) PDE s (distributed parameter model) AR(X) (AutoRegressive with exogenous inputs) ARMA (AautoregRessive Moving Average) NAR(X) (Nonlinear Autoregressive with exogenous inputs) NARMA(X) (Nonlinear Autoregressive Moving Average with exogenous inputs) Non-parametric approaches Correlation analysis (auto- and cross-correlation) FFT and Spectral analysis Time-frequency analysis Wavelet transform PCA, ICA, Bayesian, Gaussian, kernel methods, etc. July 6-8, 017 Dr HL Wei, Data Based Modelling, Slide 7 of 45 Linear-In-The-Parameters vs Linear-In-The-Variables Models Let x, z be independent ( input ) variables, y be the dependent variable, and e the noise signal; also let a, b, c be the model parameters. Then, y = a + bx y = a + bx + cx y = a + x b y = a + ln(x b /z c ) (linear in both) (linear-in-the-parameters) (linear-in-the-variables) (linear in neither) July 6-8, 017 Dr HL Wei, Data Based Modelling, Slide 8 of 45 4
5 Regression Model (1) The Simplest Case The simplest case of linear regression is used for line fitting y = ax + b (1) The unbiased estimates of a and b can be calculated using given data points (x k, y k ) (k=1,,, N). July 6-8, 017 Dr HL Wei, Data Based Modelling, Slide 9 of 45 Regression Model () The Multiple Regression Multiple linear regression model has a form below y f ( x1, x,, xn) e aixi e where x i is the ith input (independent) variable, a i is the associated model parameter. Given observational data points (x k, y k ) (k=1,,, N), the regression model becomes y k f ( x 1, k, x, k,, xn, k ) n i0 e k n i0 a x i i, k e k July 6-8, 017 Dr HL Wei, Data Based Modelling, Slide 10 of 45 5
6 Regression Model (3) The Vector and Matrix Form X 1, x1,1, x,1,, xm,1 y1 1, x1,, x,,, x m, y, y 1, x1, N, x, N,, x y m, N N a a a 0 1 T 1 T aˆ ( X X ) ( X y) m y y y 1 N a 0 a 0 a a x 0 1 a x 1 1,1 1, a x 1 1, N a a x a x,1, x a, N m a x m a m,1 x m m, x e e m, N 1 e N Normal equation y Xa e T T ( X X ) a X y T 1 T aˆ ( X X ) X y Least squares estimator July 6-8, 017 Regression Model (4) The General Form In general sense, regression model can be expressed in a linear-in-the-parameters form y a00 ( x1,, xn) a1 1( x1,, xn) amm ( x1,, xn) e a x) a ( x) a ( ) e 0 0 ( 1 1 m m x m i0 a i (x) e i whereφ i (x) are called model regressors or model terms formed by the n model variables through some linear or nonlinear manners. July 6-8, 017 Dr HL Wei, Data Based Modelling, Slide 1 of 45 6
7 Regression Model (5) The General Form: Examples Consider the model y = β 0 + β 1 logx 1 + β logx + ε φ 0 (x 1, x )=1 φ 1 (x 1, x )=log x 1 φ (x 1, x )=log x Consider a nonlinear IO system described by the model y( k) a1 y( k ) a y( k 1) u( k 1) a3 u ( k ) 3 a4 y ( k 1) a5 y( k ) u ( k ) e(k) July 6-8, 017 φ 1 (x(k)) φ (x(k)) φ 3 (x(k)) φ 4 (x(k)) φ 5 (x(k)) x 1 (k)= y(k-1) x (k)= y(k-) x 3 (k)= u(k-1) x 4 (k)= u(k-) x(k)=[x 1 (k), x (k), x 3 (k), x 4 (k)] T Regression Model (6) The General Form: Parameter Estimation Define the design matrix 0 ( x(1)) 0 ( x()) 0( x( N)) 1( x(1)) 1( x()) ( x( N)) 1 m( x(1)) ( ()) m x m( x( N)) Parameters can be estimated by least squares estimator T 1 T θˆ ( ) ( y) July 6-8, 017 Dr HL Wei, Data Based Modelling, Slide 14 of 45 7
8 Regression Model (7) Least Squares Method Is BLUE Least squares (LS) is BLUE Best (Minimum Variance) Linear Unbiased Estimator LS provides a unique solution if the design matrix is full rank in column. LS is equivalent to maximum-likelihood if noise is Gaussian. July 6-8, 017 Dr HL Wei, Data Based Modelling, Slide 15 of 45 PART Three Levels of Modelling for Complex Systems July 6-8, 017 HLW, Data Based Modelling, Slide 16 of 45 8
9 Three Classes of Boxes (Systems): White, Grey and Black-Boxes White-Box Grey-Box Black-Box All necessary information is completely known System information is partly known (the remaining part is unknown) No a priori information of the system is available July 6-8, 017 HLW, Data Based Modelling, Slide 17 of 45 White-, Grey- and Black-Box Modelling Approaches White-box Grey-box Black-box First principles e.g. physical, mechanical, and chemical laws First principles & data driven modelling techniques Data driven modelling (let data speak for themselves) July 6-8, 017 HLW, Data Based Modelling, Slide 18 of 45 9
10 x y Two Trivial Examples inputs 1 output z x +y z inputs x y z f (x,y) z (output) f(x,y) =??? x y z x y z=f(x,y) July 6-8, 017 White-Box Modelling: An Example - Pendulum Using Newton s second law of motion, along the tangential direction, we have ml mg sin( ) kl Taking the state variables as x 1, x we have the system state model Pictures above and below were from Richard Seto s lecture on Physics 000. x 1 x f ( x, x ) 1 1 g k f( x1, x) sin( x1 ) x L m x Using this model, we can do simulation and analysis etc. July 6-8,
11 Length Grey-Box Modelling A Simple Example - Spring a b c d e f g Experimental data Record Force (N) Length (in) 1, a , b , c , d , e , f , g d c a b e f Force Assume that the change in length of the spring is proportional to the force applied (Hooke s law),i.e., Length = a + b Force. With the known model structure, we can estimate the unknown model parameters a and b using some standard algorithm. July 6-8, 017 g When F = 5, L=? Grey-Box Modelling A Simple Example Parameter Estimation F L Data Model a =? L a b F b =? e1 L1 a 1 b F1 e e 1 L a 1 b F e e 3 a e4 b e L7 a1 b F7 e e6 a ˆ e T 1 T 7 ( A A) ( A L) bˆ L A θ e The suggested linear model L= F can only approximately represent the data, that is why there is an error term e in each of the above equations. July 6-8,
12 Black-Box Modelling White-Box: EVERYTHING of the system is known (including model structure, parameters, relevant governing laws and rules etc). Black-Box: NOTHING (or little) of the TRUE system model is known or can be known. What is available are recoded data of the system behaviour of interest. The objective is to learn a model or a set of models from data. Data Black-Box July 6-8, 017 HLW, Data Based Modelling, Slide 3 of 45 PART 3 Uncovering A Black-Box Letting Data Speak and Learning from Data July 6-8, 017 HLW, Data Based Modelling, Slide 4 of 45 1
13 ... Data Driven Modelling Black-Box Modelling: To model black-box systems, what we can do is to let data speak and learn from data. System Identification: A science of generating models from data with no (or very limited) a priori knowledge of the inherent system dynamics. Data Model Data Black-Box July 6-8, 017 Identification Toolkit Common Model Classes: Linear vs Nonlinear Parametric vs Nonparametric Continuous vs Discrete time Lumped vs Distributed Common Model Types: Regression, Differential eqns, Difference eqns, Artificial neural nets, Statistical models, Input-Output Models (1) Input signal Output signal Assumption: a) System input and output signals are measurable. b) The true system model structure is NOT known. Main Objective and Task: To generate a model that well represents the relationship and reveals the dynamics between the input and output. July 6-8, 017 HLW, Data Based Modelling, Slide 6 of 45 13
14 ... Input-Output Models () x1 () t x() t... x () n t y( t) f ( x ( t), x ( t),, x ( t)) e( t) 1 n yt () System identification aims to find a model that represents the relationship between system input and output such that e(t) is noise or error that inevitably exists in any data-driven modelling f( ) can be any arbitrary function but often is chosen to be those that are easily interpreted, or that have good properties and performance. x 1 (t), x (t),, x n (t) are called input variables (also called independent variables, explanatory variables, or simply predictors). y(t) is the output variable (also known as dependent variable). July 6-8, 017 HLW, Data Based Modelling, Slide 7 of 45 Input-Output Models (3) - ANNs Artificial Neural Networks (ANNs): ANNs are among the most popular approaches to data driven modelling. Appropriately trained ANNs can have a very good generalization property (ie prediction capability) However, ANN models are opaque and cannot be written down, and are difficult to interpret. ANN models may not be applicable in many real application scenarios. July 6-8, 017 HLW, Data Based Modelling, Slide 8 of 45 14
15 I/O Models (4) Linear Regression Linear regression models are of the form y( t) f ( x ( t), x ( t),, x ( t)) e( t) 1 a a x ( t) a x ( t)... a x ( t) e( t) n Note that in many data-based modelling tasks, the parameters a 0, a 1, a,, a n are unknown and need to be estimated from observed data. The estimates of these parameters can often be obtained by means of a least squares (LS) method. n n July 6-8, 017 HLW, Data Based Modelling, Slide 9 of 45 I/O Models (5): Generalized Linear Regression Consider a simple case with only 3 explanatory variables: x 1 (t), x (t), x 3 (t). A generalized linear model for such a case can be chosen as: y f ( x, x, x ) e 1 3 a a x a x a x a x x a x x a x x a x x a x x a x x a x x x a x x x e e is modelling error All linear model terms All cross product model terms of degree All cross product model terms of degree 3 All model terms of higher nonlinear degrees July 6-8, 017 HLW, Data Based Modelling, Slide 30 of 45 15
16 I/O Models (4) Linear Difference Eqns...u(t) Input signal y(t) Output signal In dynamic system modelling, the explanatory variables x 1 (t), x (t),, x n (t) are defined as the lagged system input and output variables; in this case, x1 ( t) y( t 1) y( t) f [ x1( t), x( t),, xn( t)] e( t)... a0 a1 y( t 1) a y( t ) a p y( t p) xp ( t) y( t p) 1 b1u ( t 1) bu( t ) bqu( t q) xp ( t) u( t 1)... et ( ) xpq( t) u( t q) This is usually referred to as an AutoRegressive with exogenous (ARX) model, where p and q are called model orders. July 6-8, 017 HLW, Data Based Modelling, Slide 31 of 45 I/O Models (5) Nonlinear Difference Eqns Consider a simple nonlinear difference equation: y( t) f [ y( t 1), y( t ), u( t 1)] e( t) We can use polynomials to approximate the nonlinear function f[ ] as below: All linear model y( t) a a y( t 1) a y( t ) a u( t 1) terms a y( t 1) y( t 1) a y( t 1) y( t ) a y( t 1) u( t 1) a y( t ) y( t ) a y( t ) u( t 1) a u( t 1) u( t 1) All model terms of higher nonlinear degrees All cross product model terms of degree et ( ) This is usually referred to as a Nonlinear AutoRegressive with exogenous input model (NARX). July 6-8, 017 HLW, Data Based Modelling, Slide 3 of 45 16
17 PART 4 Application Examples of Data-Driven Modelling July 6-8, 017 HLW, Data Based Modelling, Slide 33 of 45 Example 1: Bioethanol Production (1) Experimental Data Time Glucose Ethanol (hr) (mm) (mm) July 6-8, 017 Glucose and Ethanol (mm) Glucose Ethanol Hour We have applied the NARX modelling approach to the biomedical data, aiming to reveal the relationship between ethanol product and glucose, without using any a priori. 17
18 Example 1: Bioethanol Production () The identified NARX Model is Ethanol(t) = Ethanol(t-1) Glucose(t-) Glucose(t) Glucose(t-3) + e(t) Ethanol (mm) Measurement Model prediction Prediction error This is a simple model that can perfectly link the system output (ethonal) to the input (glucose) Hour July 6-8, 017 HLW, Data Based Modelling, Slide 35 of 45 Example : High Tide Forecast at the Venice Lagoon (1) A multiscale Cardinal B-spline NARMAX model was employed For a case study, the hourly water level data for year 199 were used for model estimation The identified model was used to predict water levels of year 1993 July 6-8, 017 HLW, Data Based Modelling, Slide 36 of 45 18
19 Example 3: (cont.) High Tide Forecast at the Venice Lagoon () Water Level [cm] Time [hr] 4 hours ahead prediction of water level at the Venice Lagoon in 1993 Thin solid line: measurements; thick dashed: model prediction July 6-8, 017 Example 3: Dst Index Prediction (1) Output signal y(t) = Dst(t) (disturbance storm time, [nt]) index Input variable u(t) = VBs(t) (solar wind rectified electric field [mv/m]) Training data Hourly Dst index and VBs data, March 1-31, 1979 No. of samples = 744 Test data Hourly Dst index and VBs data, April 1-30, 1979 No. of samples = 70 Identified model 3 y ( t) y ( t 1) y ( t 1) u( t 1) y ( t 1) y ( t 3) u( t ) y( t 1) u ( t 1) u( t ) July 6-8, 017 HLW, Data Based Modelling, Slide 38 of 45 19
20 Example 4: Dst Index Prediction () Training data: Hourly Dst index and VBs data, March 1-31, 1979 (~ 744) Test data: Hourly Dst index and VBs data, April 1-30, 1979 (~ 70) July 6-8, 017 HLW, Data Based Modelling, Slide 39 of 45 Example 4: Dst Index Prediction (3) Storm 1: PE = 91.48% (PE: Prediction Efficiency) Storm : PE = 9.17% All test data: PE = 93.34% July 6-8, 017 HLW, Data Based Modelling, Slide 40 of 45 0
21 The Understanding of Complex Systems Needs Data-Driven Modelling July 6-8, 017 HLW, Data Based Modelling, Slide 41 of 45 System Identification Has Many Applications for the Analysis of Complex Systems Chertsey BBC, Surry (9.04 am 13 th Feb 014) In practice, theoretical models are very difficult, if not impossible, to obtain (as they need first principles). Data-driven modelling provides a complementary but powerful tool for understanding complex systems. July 6-8, 017 1
22 Tips for Data Driven Modelling and Concluding Remarks July 6-8, 017 HLW, Data Based Modelling, Slide 43 of 45 Tips for Data Driven Modelling Always try the simplest possible models first (e.g. linear). If a simpler model works, then forget complex models. Keep in mind the main purpose of your modelling task. Transparent, parsimonious and easily interpretable models (e.g. regression models) are desirable if you are aiming to reveal dependency and interaction relationships between different explanatory variables/factors. If the modelling task is merely focused on prediction or classification, then either parametric models (e.g. linear regressons) or complicated opaque models (such as ANN models) can be an option. July 6-8, 017 Dr HL Wei, Data Based Modelling, Slide 44 of 45
23 Concluding Remarks System identification and data driven modelling, as powerful state-of-the-art approaches, have been widely applied to various areas of science and engineering. No particular modelling methods are always the best and/or universal for all applications, and therefore it is not possible or appropriate to claim that one approach is always better than all the others for solving all problems. July 6-8, 017 Dr HL Wei, Data Based Modelling, Slide 45 of 45 Key References 1. Wei, H.L., Billings, S.A., & Liu, J. (004) Term and variable selection for nonlinear system identification, International Journal of Control, 77, Wei, H.L., Billings, S.A., & Balikhin, M.A. (004) Prediction of the Dst index using multiresolution wavelet models Journal of Geophysical Research, 109, A Wei, H.L., & Billings, S.A. (006) Long term prediction of nonlinear time series using multiresolution models, International Journal of Control, 79, Wei, H.L., & Billings, S.A. (006) An efficient nonlinear cardinal B-spline model for high tide forecasts at the Venice Lagoon, Nonlinear Processes in Geophysics,13, Wei, H.L., Billings, S.A., & Balikhin, M.A.(006) Wavelet based nonparametric NARX models for nonlinear input-output system identification, International Journal of Systems Science, 37(15), Wei, H.L, Zhu, D., Billings, S.A., & Balikhin, M.A. (007) Forecasting the geomagnetic activity of the Dst index using multiscale radial basis function networks, Advances in Space Research, 40, Billings, S.A., Wei, H.L., & Balikhin, M.A. (007) Generalised multiscale radial basis function networks, Neural Networks, 0(10), Billings, S.A., & Wei, H.L.(008) An adaptive search algorithm for model subset selection and nonlinear system identification, International Journal of Control, 81, Wei, H.L., & Billings, S.A. (008) Model structure selection using an integrated forward orthogonal search algorithm assisted by squared correlation and mutual information, International Journal of Modelling, Identification and Control, 3, Billings, S.A. (013) Nonlinear system identification: NARMAX methods in the time, frequency, and spatio-temporal domains: John Wiley & Sons. 11. Ayala Solares, J. R., Wei, H.L., Boynton, R. J., Walker, S. N., & Billings, S. A. (016) Modelling and Prediction of Global Magnetic Disturbance in Near-Earth Space: a Case July 6-8, Study 017 for Kp Index using NARX Models, Space Weather, 14(10),
24 Acknowledgement We gratefully acknowledge that part of this work was supported by: EC Horizon 00 Research and Innovation Action Framework Programme (Grant No and grant title PROGRESS ). Engineering and Physical Sciences Research Council (EPSRC) (Grant No EP/I011056/1) EPSRC Platform Grant (Grant No EP/H00453X/1) July 6-8, 017 Thank You! Any Questions? July 6-8, 017 HLW, Data Based Modelling, Slide 48 of 45 4
This is a repository copy of Prediction of Kp Index Using NARMAX Models with A Robust Model Structure Selection Method.
This is a repository copy of Prediction of Kp Index Using NARMAX Models with A Robust Model Structure Selection Method. White Rose Research Online URL for this paper: http://eprints.whiterose.ac.uk/130274/
More informationPRediction Of Geospace Radiation Environment and Solar wind parameters
Page: 1 of 22 Ref. Ares(217)1318797-14/3/217 PRediction Of Geospace Radiation Environment and Solar wind parameters Work Package 3 Forecast of the evolution of geomagnetic indices Deliverable 3.4 Development
More informationChapter 4 Neural Networks in System Identification
Chapter 4 Neural Networks in System Identification Gábor HORVÁTH Department of Measurement and Information Systems Budapest University of Technology and Economics Magyar tudósok körútja 2, 52 Budapest,
More informationJ. Sjöberg et al. (1995):Non-linear Black-Box Modeling in System Identification: a Unified Overview, Automatica, Vol. 31, 12, str
Dynamic Systems Identification Part - Nonlinear systems Reference: J. Sjöberg et al. (995):Non-linear Black-Box Modeling in System Identification: a Unified Overview, Automatica, Vol. 3,, str. 69-74. Systems
More informationIdentification of Non-linear Dynamical Systems
Identification of Non-linear Dynamical Systems Linköping University Sweden Prologue Prologue The PI, the Customer and the Data Set C: I have this data set. I have collected it from a cell metabolism experiment.
More informationModelling and Prediction of Global Magnetic Disturbance in Near-Earth Space: a Case Study for Kp Index Using NARX Models
Modelling and Prediction of Global Magnetic Disturbance in Near-Earth Space: a Case Study for Kp Index Using NARX Models Jose Roberto Ayala Solares 1, Hua-Liang Wei 1, R. J. Boynton 1, Simon N. Walker
More informationNonparametric Bayesian Methods (Gaussian Processes)
[70240413 Statistical Machine Learning, Spring, 2015] Nonparametric Bayesian Methods (Gaussian Processes) Jun Zhu dcszj@mail.tsinghua.edu.cn http://bigml.cs.tsinghua.edu.cn/~jun State Key Lab of Intelligent
More information! # % & () +,.&/ 01),. &, / &
! # % & () +,.&/ ),. &, / & 2 A NEW METHOD FOR THE DESIGN OF ENERGY TRANSFER FILTERS Xiaofeng Wu, Z Q Lang and S. A. Billings Department of Automatic Control and Systems Engineering The University of Sheffield
More informationDimension Reduction (PCA, ICA, CCA, FLD,
Dimension Reduction (PCA, ICA, CCA, FLD, Topic Models) Yi Zhang 10-701, Machine Learning, Spring 2011 April 6 th, 2011 Parts of the PCA slides are from previous 10-701 lectures 1 Outline Dimension reduction
More informationGL Garrad Hassan Short term power forecasts for large offshore wind turbine arrays
GL Garrad Hassan Short term power forecasts for large offshore wind turbine arrays Require accurate wind (and hence power) forecasts for 4, 24 and 48 hours in the future for trading purposes. Receive 4
More informationChapter 2 System Identification with GP Models
Chapter 2 System Identification with GP Models In this chapter, the framework for system identification with GP models is explained. After the description of the identification problem, the explanation
More informationLearning from Data: Regression
November 3, 2005 http://www.anc.ed.ac.uk/ amos/lfd/ Classification or Regression? Classification: want to learn a discrete target variable. Regression: want to learn a continuous target variable. Linear
More informationApplica'on of NARMAX methodology to the forecast of radia'on environment in the Geospace.
Applica'on of NARMAX methodology to the forecast of radia'on environment in the Geospace. ssg.group.shef.ac.uk/progress/html This project has received funding from the European Union's Horizon 2020 research
More informationLinear Regression. Aarti Singh. Machine Learning / Sept 27, 2010
Linear Regression Aarti Singh Machine Learning 10-701/15-781 Sept 27, 2010 Discrete to Continuous Labels Classification Sports Science News Anemic cell Healthy cell Regression X = Document Y = Topic X
More informationPrediction of the Dst index using multiresolution wavelet models
JOURNAL OF GEOPHYSICAL RESEARCH, VOL. 109,, doi:10.1029/2003ja010332, 2004 Prediction of the Dst index using multiresolution wavelet models H. L. Wei, S. A. Billings, and M. Balikhin Department of Automatic
More informationLinear Models for Regression
Linear Models for Regression Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr
More informationCMU-Q Lecture 24:
CMU-Q 15-381 Lecture 24: Supervised Learning 2 Teacher: Gianni A. Di Caro SUPERVISED LEARNING Hypotheses space Hypothesis function Labeled Given Errors Performance criteria Given a collection of input
More informationProbabilistic Energy Forecasting
Probabilistic Energy Forecasting Moritz Schmid Seminar Energieinformatik WS 2015/16 ^ KIT The Research University in the Helmholtz Association www.kit.edu Agenda Forecasting challenges Renewable energy
More informationAUTOMATIC CONTROL COMMUNICATION SYSTEMS LINKÖPINGS UNIVERSITET AUTOMATIC CONTROL COMMUNICATION SYSTEMS LINKÖPINGS UNIVERSITET
Identification of Linear and Nonlinear Dynamical Systems Theme : Nonlinear Models Grey-box models Division of Automatic Control Linköping University Sweden General Aspects Let Z t denote all available
More informationNONLINEAR INTEGRAL MINIMUM VARIANCE-LIKE CONTROL WITH APPLICATION TO AN AIRCRAFT SYSTEM
NONLINEAR INTEGRAL MINIMUM VARIANCE-LIKE CONTROL WITH APPLICATION TO AN AIRCRAFT SYSTEM D.G. Dimogianopoulos, J.D. Hios and S.D. Fassois DEPARTMENT OF MECHANICAL & AERONAUTICAL ENGINEERING GR-26500 PATRAS,
More informationCONTROL SYSTEMS, ROBOTICS, AND AUTOMATION Vol. VI - Identification of NARMAX and Related Models - Stephen A. Billings and Daniel Coca
IDENTIFICATION OF NARMAX AND RELATED MODELS Stephen Department of Automatic Control and Systems Engineering, University of Sheffield, UK Keywords: Activation function, Artificial neural network, Correlation
More informationGaussian Process for Internal Model Control
Gaussian Process for Internal Model Control Gregor Gregorčič and Gordon Lightbody Department of Electrical Engineering University College Cork IRELAND E mail: gregorg@rennesuccie Abstract To improve transparency
More informationLecture 9. Time series prediction
Lecture 9 Time series prediction Prediction is about function fitting To predict we need to model There are a bewildering number of models for data we look at some of the major approaches in this lecture
More informationModeling Data with Linear Combinations of Basis Functions. Read Chapter 3 in the text by Bishop
Modeling Data with Linear Combinations of Basis Functions Read Chapter 3 in the text by Bishop A Type of Supervised Learning Problem We want to model data (x 1, t 1 ),..., (x N, t N ), where x i is a vector
More informationChap 1. Overview of Statistical Learning (HTF, , 2.9) Yongdai Kim Seoul National University
Chap 1. Overview of Statistical Learning (HTF, 2.1-2.6, 2.9) Yongdai Kim Seoul National University 0. Learning vs Statistical learning Learning procedure Construct a claim by observing data or using logics
More informationCONTROL SYSTEMS, ROBOTICS, AND AUTOMATION - Vol. V - Prediction Error Methods - Torsten Söderström
PREDICTIO ERROR METHODS Torsten Söderström Department of Systems and Control, Information Technology, Uppsala University, Uppsala, Sweden Keywords: prediction error method, optimal prediction, identifiability,
More information6.867 Machine learning
6.867 Machine learning Mid-term eam October 8, 6 ( points) Your name and MIT ID: .5.5 y.5 y.5 a).5.5 b).5.5.5.5 y.5 y.5 c).5.5 d).5.5 Figure : Plots of linear regression results with different types of
More informationEM-algorithm for Training of State-space Models with Application to Time Series Prediction
EM-algorithm for Training of State-space Models with Application to Time Series Prediction Elia Liitiäinen, Nima Reyhani and Amaury Lendasse Helsinki University of Technology - Neural Networks Research
More informationAUTOMATIC CONTROL COMMUNICATION SYSTEMS LINKÖPINGS UNIVERSITET. Questions AUTOMATIC CONTROL COMMUNICATION SYSTEMS LINKÖPINGS UNIVERSITET
The Problem Identification of Linear and onlinear Dynamical Systems Theme : Curve Fitting Division of Automatic Control Linköping University Sweden Data from Gripen Questions How do the control surface
More information1 Teaching notes on structural VARs.
Bent E. Sørensen February 22, 2007 1 Teaching notes on structural VARs. 1.1 Vector MA models: 1.1.1 Probability theory The simplest (to analyze, estimation is a different matter) time series models are
More information+ + ( + ) = Linear recurrent networks. Simpler, much more amenable to analytic treatment E.g. by choosing
Linear recurrent networks Simpler, much more amenable to analytic treatment E.g. by choosing + ( + ) = Firing rates can be negative Approximates dynamics around fixed point Approximation often reasonable
More informationEL1820 Modeling of Dynamical Systems
EL1820 Modeling of Dynamical Systems Lecture 9 - Parameter estimation in linear models Model structures Parameter estimation via prediction error minimization Properties of the estimate: bias and variance
More informationGaussian Processes. Le Song. Machine Learning II: Advanced Topics CSE 8803ML, Spring 2012
Gaussian Processes Le Song Machine Learning II: Advanced Topics CSE 8803ML, Spring 01 Pictorial view of embedding distribution Transform the entire distribution to expected features Feature space Feature
More informationIn this chapter, we provide an introduction to covariate shift adaptation toward machine learning in a non-stationary environment.
1 Introduction and Problem Formulation In this chapter, we provide an introduction to covariate shift adaptation toward machine learning in a non-stationary environment. 1.1 Machine Learning under Covariate
More informationClassic Time Series Analysis
Classic Time Series Analysis Concepts and Definitions Let Y be a random number with PDF f Y t ~f,t Define t =E[Y t ] m(t) is known as the trend Define the autocovariance t, s =COV [Y t,y s ] =E[ Y t t
More informationThe Kernel Trick, Gram Matrices, and Feature Extraction. CS6787 Lecture 4 Fall 2017
The Kernel Trick, Gram Matrices, and Feature Extraction CS6787 Lecture 4 Fall 2017 Momentum for Principle Component Analysis CS6787 Lecture 3.1 Fall 2017 Principle Component Analysis Setting: find the
More informationLast updated: Oct 22, 2012 LINEAR CLASSIFIERS. J. Elder CSE 4404/5327 Introduction to Machine Learning and Pattern Recognition
Last updated: Oct 22, 2012 LINEAR CLASSIFIERS Problems 2 Please do Problem 8.3 in the textbook. We will discuss this in class. Classification: Problem Statement 3 In regression, we are modeling the relationship
More informationECE 636: Systems identification
ECE 636: Systems identification Lectures 3 4 Random variables/signals (continued) Random/stochastic vectors Random signals and linear systems Random signals in the frequency domain υ ε x S z + y Experimental
More informationMachine Learning of Environmental Spatial Data Mikhail Kanevski 1, Alexei Pozdnoukhov 2, Vasily Demyanov 3
1 3 4 5 6 7 8 9 10 11 1 13 14 15 16 17 18 19 0 1 3 4 5 6 7 8 9 30 31 3 33 International Environmental Modelling and Software Society (iemss) 01 International Congress on Environmental Modelling and Software
More informationSystem Identification: From Data to Model
1 : From Data to Model With Applications to Aircraft Modeling Division of Automatic Control Linköping University Sweden Dynamic Systems 2 A Dynamic system has an output response y that depends on (all)
More informationTime Series: Theory and Methods
Peter J. Brockwell Richard A. Davis Time Series: Theory and Methods Second Edition With 124 Illustrations Springer Contents Preface to the Second Edition Preface to the First Edition vn ix CHAPTER 1 Stationary
More informationForecasting Data Streams: Next Generation Flow Field Forecasting
Forecasting Data Streams: Next Generation Flow Field Forecasting Kyle Caudle South Dakota School of Mines & Technology (SDSMT) kyle.caudle@sdsmt.edu Joint work with Michael Frey (Bucknell University) and
More information6.435, System Identification
SET 6 System Identification 6.435 Parametrized model structures One-step predictor Identifiability Munther A. Dahleh 1 Models of LTI Systems A complete model u = input y = output e = noise (with PDF).
More informationNONLINEAR IDENTIFICATION ON BASED RBF NEURAL NETWORK
DAAAM INTERNATIONAL SCIENTIFIC BOOK 2011 pp. 547-554 CHAPTER 44 NONLINEAR IDENTIFICATION ON BASED RBF NEURAL NETWORK BURLAK, V. & PIVONKA, P. Abstract: This article is focused on the off-line identification
More informationArtificial Neural Networks
Introduction ANN in Action Final Observations Application: Poverty Detection Artificial Neural Networks Alvaro J. Riascos Villegas University of los Andes and Quantil July 6 2018 Artificial Neural Networks
More informationCharacterisation of Near-Earth Magnetic Field Data for Space Weather Monitoring
Characterisation of Near-Earth Magnetic Field Data for Space Weather Monitoring Qingying Shu Space Glasgow Research Conference 28 October 2014 Acknowledgments This is an interdisciplinary PhD studentship
More informationRecent Advances in Bayesian Inference Techniques
Recent Advances in Bayesian Inference Techniques Christopher M. Bishop Microsoft Research, Cambridge, U.K. research.microsoft.com/~cmbishop SIAM Conference on Data Mining, April 2004 Abstract Bayesian
More informationEEG- Signal Processing
Fatemeh Hadaeghi EEG- Signal Processing Lecture Notes for BSP, Chapter 5 Master Program Data Engineering 1 5 Introduction The complex patterns of neural activity, both in presence and absence of external
More informationRoss Bettinger, Analytical Consultant, Seattle, WA
ABSTRACT DYNAMIC REGRESSION IN ARIMA MODELING Ross Bettinger, Analytical Consultant, Seattle, WA Box-Jenkins time series models that contain exogenous predictor variables are called dynamic regression
More informationComputational Learning Theory
09s1: COMP9417 Machine Learning and Data Mining Computational Learning Theory May 20, 2009 Acknowledgement: Material derived from slides for the book Machine Learning, Tom M. Mitchell, McGraw-Hill, 1997
More informationLecture 2: Univariate Time Series
Lecture 2: Univariate Time Series Analysis: Conditional and Unconditional Densities, Stationarity, ARMA Processes Prof. Massimo Guidolin 20192 Financial Econometrics Spring/Winter 2017 Overview Motivation:
More informationGaussian Process Regression: Active Data Selection and Test Point. Rejection. Sambu Seo Marko Wallat Thore Graepel Klaus Obermayer
Gaussian Process Regression: Active Data Selection and Test Point Rejection Sambu Seo Marko Wallat Thore Graepel Klaus Obermayer Department of Computer Science, Technical University of Berlin Franklinstr.8,
More informationAdvanced Machine Learning Practical 4b Solution: Regression (BLR, GPR & Gradient Boosting)
Advanced Machine Learning Practical 4b Solution: Regression (BLR, GPR & Gradient Boosting) Professor: Aude Billard Assistants: Nadia Figueroa, Ilaria Lauzana and Brice Platerrier E-mails: aude.billard@epfl.ch,
More informationDo Markov-Switching Models Capture Nonlinearities in the Data? Tests using Nonparametric Methods
Do Markov-Switching Models Capture Nonlinearities in the Data? Tests using Nonparametric Methods Robert V. Breunig Centre for Economic Policy Research, Research School of Social Sciences and School of
More informationSystem Identification and Optimization Methods Based on Derivatives. Chapters 5 & 6 from Jang
System Identification and Optimization Methods Based on Derivatives Chapters 5 & 6 from Jang Neuro-Fuzzy and Soft Computing Model space Adaptive networks Neural networks Fuzzy inf. systems Approach space
More informationMixed frequency models with MA components
Mixed frequency models with MA components Claudia Foroni a Massimiliano Marcellino b Dalibor Stevanović c a Deutsche Bundesbank b Bocconi University, IGIER and CEPR c Université du Québec à Montréal September
More informationMultitask Learning of Environmental Spatial Data
9th International Congress on Environmental Modelling and Software Brigham Young University BYU ScholarsArchive 6th International Congress on Environmental Modelling and Software - Leipzig, Germany - July
More informationIntroduction to System Modeling and Control. Introduction Basic Definitions Different Model Types System Identification Neural Network Modeling
Introduction to System Modeling and Control Introduction Basic Definitions Different Model Types System Identification Neural Network Modeling What is Mathematical Model? A set of mathematical equations
More informationEECE Adaptive Control
EECE 574 - Adaptive Control Basics of System Identification Guy Dumont Department of Electrical and Computer Engineering University of British Columbia January 2010 Guy Dumont (UBC) EECE574 - Basics of
More informationClassification for High Dimensional Problems Using Bayesian Neural Networks and Dirichlet Diffusion Trees
Classification for High Dimensional Problems Using Bayesian Neural Networks and Dirichlet Diffusion Trees Rafdord M. Neal and Jianguo Zhang Presented by Jiwen Li Feb 2, 2006 Outline Bayesian view of feature
More informationLecture : Probabilistic Machine Learning
Lecture : Probabilistic Machine Learning Riashat Islam Reasoning and Learning Lab McGill University September 11, 2018 ML : Many Methods with Many Links Modelling Views of Machine Learning Machine Learning
More informationDESIGNING A KALMAN FILTER WHEN NO NOISE COVARIANCE INFORMATION IS AVAILABLE. Robert Bos,1 Xavier Bombois Paul M. J. Van den Hof
DESIGNING A KALMAN FILTER WHEN NO NOISE COVARIANCE INFORMATION IS AVAILABLE Robert Bos,1 Xavier Bombois Paul M. J. Van den Hof Delft Center for Systems and Control, Delft University of Technology, Mekelweg
More informationNon-linear Measure Based Process Monitoring and Fault Diagnosis
Non-linear Measure Based Process Monitoring and Fault Diagnosis 12 th Annual AIChE Meeting, Reno, NV [275] Data Driven Approaches to Process Control 4:40 PM, Nov. 6, 2001 Sandeep Rajput Duane D. Bruns
More informationIntroduction to Machine Learning Prof. Sudeshna Sarkar Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur
Introduction to Machine Learning Prof. Sudeshna Sarkar Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Module 2 Lecture 05 Linear Regression Good morning, welcome
More informationParameter Estimation in a Moving Horizon Perspective
Parameter Estimation in a Moving Horizon Perspective State and Parameter Estimation in Dynamical Systems Reglerteknik, ISY, Linköpings Universitet State and Parameter Estimation in Dynamical Systems OUTLINE
More informationShould all Machine Learning be Bayesian? Should all Bayesian models be non-parametric?
Should all Machine Learning be Bayesian? Should all Bayesian models be non-parametric? Zoubin Ghahramani Department of Engineering University of Cambridge, UK zoubin@eng.cam.ac.uk http://learning.eng.cam.ac.uk/zoubin/
More informationMachine Learning Linear Classification. Prof. Matteo Matteucci
Machine Learning Linear Classification Prof. Matteo Matteucci Recall from the first lecture 2 X R p Regression Y R Continuous Output X R p Y {Ω 0, Ω 1,, Ω K } Classification Discrete Output X R p Y (X)
More informationOverview of sparse system identification
Overview of sparse system identification J.-Ch. Loiseau 1 & Others 2, 3 1 Laboratoire DynFluid, Arts et Métiers ParisTech, France 2 LIMSI, Université d Orsay CNRS, France 3 University of Washington, Seattle,
More informationNonlinear System Identification Using MLP Dr.-Ing. Sudchai Boonto
Dr-Ing Sudchai Boonto Department of Control System and Instrumentation Engineering King Mongkut s Unniversity of Technology Thonburi Thailand Nonlinear System Identification Given a data set Z N = {y(k),
More informationPREDICTING SOLAR GENERATION FROM WEATHER FORECASTS. Chenlin Wu Yuhan Lou
PREDICTING SOLAR GENERATION FROM WEATHER FORECASTS Chenlin Wu Yuhan Lou Background Smart grid: increasing the contribution of renewable in grid energy Solar generation: intermittent and nondispatchable
More informationGaussian Process priors with Uncertain Inputs: Multiple-Step-Ahead Prediction
Gaussian Process priors with Uncertain Inputs: Multiple-Step-Ahead Prediction Agathe Girard Dept. of Computing Science University of Glasgow Glasgow, UK agathe@dcs.gla.ac.uk Carl Edward Rasmussen Gatsby
More informationState Estimation of Linear and Nonlinear Dynamic Systems
State Estimation of Linear and Nonlinear Dynamic Systems Part I: Linear Systems with Gaussian Noise James B. Rawlings and Fernando V. Lima Department of Chemical and Biological Engineering University of
More informationNeutron inverse kinetics via Gaussian Processes
Neutron inverse kinetics via Gaussian Processes P. Picca Politecnico di Torino, Torino, Italy R. Furfaro University of Arizona, Tucson, Arizona Outline Introduction Review of inverse kinetics techniques
More informationECON 4160, Spring term Lecture 12
ECON 4160, Spring term 2013. Lecture 12 Non-stationarity and co-integration 2/2 Ragnar Nymoen Department of Economics 13 Nov 2013 1 / 53 Introduction I So far we have considered: Stationary VAR, with deterministic
More informationThe ARIMA Procedure: The ARIMA Procedure
Page 1 of 120 Overview: ARIMA Procedure Getting Started: ARIMA Procedure The Three Stages of ARIMA Modeling Identification Stage Estimation and Diagnostic Checking Stage Forecasting Stage Using ARIMA Procedure
More informationOn Moving Average Parameter Estimation
On Moving Average Parameter Estimation Niclas Sandgren and Petre Stoica Contact information: niclas.sandgren@it.uu.se, tel: +46 8 473392 Abstract Estimation of the autoregressive moving average (ARMA)
More informationElectric Load Forecasting Using Wavelet Transform and Extreme Learning Machine
Electric Load Forecasting Using Wavelet Transform and Extreme Learning Machine Song Li 1, Peng Wang 1 and Lalit Goel 1 1 School of Electrical and Electronic Engineering Nanyang Technological University
More informationCONTROL SYSTEMS, ROBOTICS, AND AUTOMATION Vol. VI - System Identification Using Wavelets - Daniel Coca and Stephen A. Billings
SYSTEM IDENTIFICATION USING WAVELETS Daniel Coca Department of Electrical Engineering and Electronics, University of Liverpool, UK Department of Automatic Control and Systems Engineering, University of
More informationECE521 Lecture 7/8. Logistic Regression
ECE521 Lecture 7/8 Logistic Regression Outline Logistic regression (Continue) A single neuron Learning neural networks Multi-class classification 2 Logistic regression The output of a logistic regression
More informationNew Machine Learning Methods for Neuroimaging
New Machine Learning Methods for Neuroimaging Gatsby Computational Neuroscience Unit University College London, UK Dept of Computer Science University of Helsinki, Finland Outline Resting-state networks
More informationBifurcation Analysis of the Aeroelastic Galloping Problem via Input-Output Parametric Modelling
Bifurcation Analysis of the Aeroelastic Galloping Problem via Input-Output Parametric Modelling Nicholas F. Giannelis 1 and Gareth A. Vio School of Aeronautical, Mechanical and Mechatronic Engineering
More informationLecture 3: Introduction to Complexity Regularization
ECE90 Spring 2007 Statistical Learning Theory Instructor: R. Nowak Lecture 3: Introduction to Complexity Regularization We ended the previous lecture with a brief discussion of overfitting. Recall that,
More informationTutorial on Gaussian Processes and the Gaussian Process Latent Variable Model
Tutorial on Gaussian Processes and the Gaussian Process Latent Variable Model (& discussion on the GPLVM tech. report by Prof. N. Lawrence, 06) Andreas Damianou Department of Neuro- and Computer Science,
More information1 Introduction to Generalized Least Squares
ECONOMICS 7344, Spring 2017 Bent E. Sørensen April 12, 2017 1 Introduction to Generalized Least Squares Consider the model Y = Xβ + ɛ, where the N K matrix of regressors X is fixed, independent of the
More informationOnboard Engine FDI in Autonomous Aircraft Using Stochastic Nonlinear Modelling of Flight Signal Dependencies
Onboard Engine FDI in Autonomous Aircraft Using Stochastic Nonlinear Modelling of Flight Signal Dependencies Dimitrios G. Dimogianopoulos, John D. Hios and Spilios D. Fassois Stochastic Mechanical Systems
More informationProbabilistic & Bayesian deep learning. Andreas Damianou
Probabilistic & Bayesian deep learning Andreas Damianou Amazon Research Cambridge, UK Talk at University of Sheffield, 19 March 2019 In this talk Not in this talk: CRFs, Boltzmann machines,... In this
More informationIdentification of ARX, OE, FIR models with the least squares method
Identification of ARX, OE, FIR models with the least squares method CHEM-E7145 Advanced Process Control Methods Lecture 2 Contents Identification of ARX model with the least squares minimizing the equation
More informationAn Introduction to Numerical Methods for Differential Equations. Janet Peterson
An Introduction to Numerical Methods for Differential Equations Janet Peterson Fall 2015 2 Chapter 1 Introduction Differential equations arise in many disciplines such as engineering, mathematics, sciences
More informationECE 521. Lecture 11 (not on midterm material) 13 February K-means clustering, Dimensionality reduction
ECE 521 Lecture 11 (not on midterm material) 13 February 2017 K-means clustering, Dimensionality reduction With thanks to Ruslan Salakhutdinov for an earlier version of the slides Overview K-means clustering
More informationRecurrent Neural Network Identification: Comparative Study on Nonlinear Process M.Rajalakshmi #1, Dr.S.Jeyadevi #2, C.Karthik * 3
ISSN (Online) : 2319-8753 ISSN (Print) : 2347-6710 International Journal of Innovative Research in Science, Engineering and Technology Volume 3, Special Issue 3, March 2014 2014 International Conference
More informationCSCE 478/878 Lecture 6: Bayesian Learning
Bayesian Methods Not all hypotheses are created equal (even if they are all consistent with the training data) Outline CSCE 478/878 Lecture 6: Bayesian Learning Stephen D. Scott (Adapted from Tom Mitchell
More informationData assimilation with and without a model
Data assimilation with and without a model Tim Sauer George Mason University Parameter estimation and UQ U. Pittsburgh Mar. 5, 2017 Partially supported by NSF Most of this work is due to: Tyrus Berry,
More informationCourse content (will be adapted to the background knowledge of the class):
Biomedical Signal Processing and Signal Modeling Lucas C Parra, parra@ccny.cuny.edu Departamento the Fisica, UBA Synopsis This course introduces two fundamental concepts of signal processing: linear systems
More informationFast Bootstrap for Least-square Support Vector Machines
ESA'2004 proceedings - European Symposium on Artificial eural etworks Bruges (Belgium), 28-30 April 2004, d-side publi., ISB 2-930307-04-8, pp. 525-530 Fast Bootstrap for Least-square Support Vector Machines
More informationSolar irradiance forecasting for Chulalongkorn University location using time series models
Senior Project Proposal 2102499 Year 2016 Solar irradiance forecasting for Chulalongkorn University location using time series models Vichaya Layanun ID 5630550721 Advisor: Assist. Prof. Jitkomut Songsiri
More informationARTIFICIAL INTELLIGENCE. Artificial Neural Networks
INFOB2KI 2017-2018 Utrecht University The Netherlands ARTIFICIAL INTELLIGENCE Artificial Neural Networks Lecturer: Silja Renooij These slides are part of the INFOB2KI Course Notes available from www.cs.uu.nl/docs/vakken/b2ki/schema.html
More informationSTA414/2104. Lecture 11: Gaussian Processes. Department of Statistics
STA414/2104 Lecture 11: Gaussian Processes Department of Statistics www.utstat.utoronto.ca Delivered by Mark Ebden with thanks to Russ Salakhutdinov Outline Gaussian Processes Exam review Course evaluations
More informationIntroduction to Machine Learning
10-701 Introduction to Machine Learning PCA Slides based on 18-661 Fall 2018 PCA Raw data can be Complex, High-dimensional To understand a phenomenon we measure various related quantities If we knew what
More informationMarkov chain optimisation for energy systems (MC-ES)
Markov chain optimisation for energy systems (MC-ES) John Moriarty Queen Mary University of London 19th August 2016 Approaches to the stochastic optimisation of power systems are not mature at research
More informationCHAPTER 6 CONCLUSION AND FUTURE SCOPE
CHAPTER 6 CONCLUSION AND FUTURE SCOPE 146 CHAPTER 6 CONCLUSION AND FUTURE SCOPE 6.1 SUMMARY The first chapter of the thesis highlighted the need of accurate wind forecasting models in order to transform
More information