Dynamic Systems Identification Part - Nonlinear systems Reference: J. Sjöberg et al. (995):Non-linear Black-Box Modeling in System Identification: a Unified Overview, Automatica, Vol. 3,, str. 69-74. Systems modelling from data
Nonlinear dynamic systems u(k) Static models Input/output data Interpolation or extrapolation Dynamic model Input/output data expanded with delayed input and output data Learning for one-stepahead prediction Validation for multi-stepahead prediction yˆ ( k ) = f ( yˆ( k ), yˆ( k ),..., u( k ),...) Z. Z - -n u(k-) u(k-n). Nonlinear model Gaussian Process Model ^ y(k-) ^ y(k) ^ y(k-n) Systems modelling from data Z Z. Z - -n y(k) ^
RBF network Example 7 I/O data, basis functions distributed randomly in operating area Original function Approksimation with basis functions Systems modelling from data 3
Experimental modelling ling of nonlinear systems 99s: ANN = nonlinear systems identification Rule: Do not estimate what you already know! white box model, grey box model (physical modelling, semi-physical modelling), black box model Nonlinear black-box box models: artificial neural networks, fuzzy models, wavelet models, etc. Systems modelling from data 4
Experimental modelling ling of nonlinear systems Used terms (system theory vs. ANN/Machine learning: estimate, identify = train, learn validate = generalize model structure = network estimation data, identification data = training set validation data = generalization set overfit = overtraining Systems modelling from data 5
Some practical concerns about nonlinear systems identification Identification procedure cannot/must not be fully automatised! Neccessary: S/W, I/O data. We need experience on similar identification cases. Computer simulations of similar cases. Systems modelling from data 6
Experimental modelling ling of nonlinear systems Nonlinear systems identification problem: y ( k) = g( u( k ), u( k ),..., y( k ), y( k ),...) + v( k) yˆ ( k θ ) = g ( ψ ( k ), θ ) ψ(t) ) = ve vector of regressors θ = vector of parameters subproblems Selection of regressors ψ(k) Selection of nonlinear mapping g(ϕ) Systems modelling from data 7
Regressors linear systems A( z ) y( z B( z ) C( z ) ) = u( z ) + e( z F ( z ) D ( z ) ) FIR (A=F=D=,C=) ARX (F=C=D=) OE (A=C=D=) ARMAX (F=D=) BJ (A=) State-space x k) Ax ( = ( k ) + Bu( k ) T yˆ( k) =θψ( k, θ) Systems modelling from data 8
Regressorssors nonlinear systems regressorssors ψ determine different models: NFIR: NARX: NOE: NARMAX: NBJ: State-space u ( k i ) u ( k i ), y ( k i ) u ( k i ), yˆ ( k i ) yˆ( k) = g( ψ( k), θ) u ( k i ), y ( k i ), ε ( k i ) = y ( k i ) yˆ ( k i ) u( k i), y( k i), ε ( k i), ε ( k i) = y( k n) yˆ ( k i) n n other possible regressors Systems modelling from data 9
Nonlinear mappings g( ψθ, ) = α g ( ψ) Fourier series (scalar case) g k (ψ) is a basis function Known structures: wavelet functions B splines ARTIFICIAL NEURAL NETWORKS Multilayer perceptron radial basis function network etc FUZZY MODELS k k Systems modelling from data
Forms of known structures: Neural networks with sigmoid activation function g σ β γ ( ψ) = ( ψ + ) k k k Radial basis function networks Fuzzy models g r β γ ( ψ) = ( ( ψ )) k k k = j j g( ψ) y ( µ ( ψ)) A γ - position β - direction scale Systems modelling from data
u(k) Multilayer networks Recurrent networks Z. Z - -n u(k-) u(k-n). Nonline ar Process Mode l y(k) ^ ϕ ( k) = g( ψ( k i), θ) Algorithms for estimation of parameters: ^ y(k-) ^ y(k) y(k-n) ^ Gauss-Newton algorithms are very efficient, Off-line identification, as well as on-line (recursive), Gradient optimisation methods time consuming. Z Z. - Z -n Systems modelling from data
Subproblems: Systematic selection of regressors u(k) - static nonlinearity u(k-i) u(k-i),y( ),y(k-i)... Selection of basis function Most of them are universal approximators There exist no exact criteria for basis function selection except subjective ones Radial functions for low number of regressors (e.g. wavelet function for max. 3 regressors) ridge functions for larger number of regressors (e.g. neural networks with sigmoid functions) Fuzzy models, where heuristic knowledge exsists. Models order (n+, Takan s theorem); Systems modelling from data 3
Measures of model quality Measure of model quality (example) V ( θ) = E y( t) g( ψ( t), θ) = λ + E g ( ψ( t)) g( ψ( t), θ) Three sources of differences with true system: noise e(t); ; variance λ=e(e (t)) bias V = E g t g t ˆ * ( θν, ψ( )) ( ψ( )) Variance of estimation V = λ dimθ N Systems modelling from data 4
There is always a limitation to model fit. Performance Bias Variance # parameters Systems modelling from data 5
Recommendatins for practice: Look at the data (detection of nonlinearity, time constants). Try simple things first (try simple structures and model orders first, at first linear models and small number of estimation parameters). Look into the physics (ideas for regressor ors). Validation and estimation data. Center and scale the data. The bias-variance trade-of off. f. The notion of efficient number of parameters (shrinking). Systems modelling from data 6
Sampling time selection (same rule as for linear systems). Sampling time should be selected to grasp all interesting process dynamics. Input signal selection (look at magnitude distribution and input/output distribution of the data).5 Input/output Vhodno /ihodni pairs pari podatkov of data 8.4.3. 6 4 y. -. -. -.3-3 -.4 - -.8 -.6 -.4 -...4.6.8 u Systems modelling from data 7
Model validation yˆ ( k) One-step-ahead prediction = g( y( k ), y( k ),... u( k ), u( k ),...) Simulation nnvalid yˆ ( k) = g( yˆ( k ), yˆ( k ),... u( k ), u( k ),...) nnsimul crossco hist Systems modelling from data 8
Model validation Validation of residuals for one-step-ahead Auto correlation function of prediction error prediction 8 Histogram of prediction errors.5 -.5 5 5 5 lag Cross correlation fct of u and prediction error.4 6 4 -.5 - -.5.5.5. -. -.4 - lag Systems modelling from data 9
Model validation yˆ ( k) yˆ ( k) The consistency of input/output response One-step-ahead prediction = g( y( k ), y( k ),... u( k ), u( k ),...) Simulation = g( yˆ( k ), yˆ( k ),... u( k ), u( k ),...) 3 Output (dashed) and simulated output (solid) - -3-4 3 4 5 time (samples) Systems modelling from data
Model validation Modela reduction (pruning) 4 Network after having pruned 59 weights 3 Model purposiveness or usefulness Systems modelling from data
Example of identification of st order system Mathematical model of the process with parameters ( 3 ) ( ) ( ) y( k) = y( k ).5tanh y k + u k u input signal y output signal Systems modelling from data
Nonlinearity of the system 4 ) y(k+) -4 y(k) -3 - u(k) 3 Systems modelling from data 3
Estimation signal and response Validation signal and response Input estimation signal Vhodni signal za identifikacijo - 3 4 5 6 7 8 9 Time Cas Odziv sistema Response za identifikacijo - Input validation signal Vhodni signal za vrednotenje 3 4 5 6 7 8 9 Time Cas - 3 4 5 6 7 8 9 Time Cas Odziv sistema Response za vrednotenje - 3 4 5 6 7 8 9 Cas Systems modelling from data 4 Time
Histograms of magnitudes and input/output data 3 Estimation input signal histogram Porazdelitev amplitud na vhodnem signalu za identifikacijo 35 Validation input signal histogram Porazdelitev amplitud na vhodnem signalu za vrednotenje 5 3 5 5 5 5 5 -.5 - -.5.5.5 Amplituda signala Magnitude -.5 - -.5.5.5 Amplituda signala Magnitude 4 y(k+) -4 y(k) 3 - -3 Systems modelling from u(k) data 5
Neural network, regressors, structure and parameters Used software: NNSYSID Toolbox for Matlab Regressors: y(k-), u(k-) Structure: ARX (model error method) Optimisation method: Levenberg-Marquardt W -.5588 -.6 -.953.555.499 -.867 = -.549.39.4768.3366 -.9.8379.84.384.73 W = [.54.7784.8.74.448 -.58] Systems modelling from data 6
Model validation (one-step-ahead prediction) 4 4 y(k+) y(k+) -4-4 y(k) -3 - u(k) 3 y(k) -3 - u(k) 3 Systems modelling from data 7
Validation of residuals (one-step-ahead prediction).5 napaka -.5 - y(k) -3 - u(k) 3 Systems modelling from data 8
Model response and original system s response on validation data (simulation, not one-step-ahead prediction) Odziv sistema Simulation na signal za response vrednotenje - simulacija.5.5 -.5 - -.5 3 4 5 6 7 8 9 Systems modelling Time Cas from data 9
Residuals of simulated valdiation data 4 8 Porazdelitev pogreška Residuals Avtokorelacija autocorrelation izhodnega pogreška 6 4 -.6 -.4 -...4.6.8.5.8.6.4 Cross-correlation between e and u Križna korelacija vhodnega signala in izhodnega pogreška. -.5 - -8-6 -4 4 6 8 Tau [s] -. -.4 -.6 -.8 - - -8-6 -4 4 6 8 Tau [s] Systems modelling from data 3
Examination assignment Select a discrete nonlinear system of the st order, identify it and make a written report. The report should contain the following items: mathematical model of the original system with all parameters; Simulation scheme or Matlab code that enables rerun of data acquisition (no masks); Show the nonlinearity of the system in 3D plot; Estimation signal and response, sampling time; Validation signal and response, sampling time; Systems modelling from data 3
Histograms of magnitudes and input/output data pairs; Type of neural network and optimisation method; Figure of the final network structure; Values of parameters (weights); Figure of comparison between original system simulation response and model simulation response on validation signal. Simulation residuals validation on validation data (the figure of residuals, residuals histogram, φee, φue ). Eksperiments should be repeatable based on your report only. Reports should not be longer than 6 pages. Systems modelling from data 3