Capabilities and Prospects of Inductive Modeling Volodmr STEPASHKO Prof., Dr. Sci., Head of Department INFORMATION TECHNOLOGIES FOR INDUCTIVE MODELING International Research and Training Centre of the Academ of Sciences of Ukraine 1
Laout 1. Historical aspects of IM 2. International events on IM 3. Attempt to define IM: what is it? 4. IM destination: what is this for? 5. IM explanation: basic algorithms and tools 6. Basic Theoretical Results 7. IM compared to ANN and CI 8. Real-world applications of IM 9. Main centers of IM research 10. IM development prospects 2
1. Historical aspects of IM 1968 First publication on GMDH: Iвахненко O.Г. Метод групового урахування аргументів конкурент методу стохастичної апроксимації // Автоматика. 1968. 3. С. 58-72. Terminolog evolution: heuristic self-organization of models (1970s) inductive method of model building (1980s) inductive learning algorithms for modeling (1992) inductive modeling (1998) GMDH: Group Method of Data Handling MGUA: Method of Group Using of Arguments 3
A.G.Ivakhnenko: GMDH originator 4
Main scientific results in inductive modelling theor: Foundations of cbernetic forecasting device construction Theor of models self-organization b experimental data Group method of data handling (GMDH) for automatic construction (self-organization) of model for complex sstems Method of control with optimization of forecast Principles of noise-immunit modelling from nois data Principles of polnomial networks construction Principle of neural networks construction with active neurons 5
Academician Ivakhnenko Originator of the scientific school of inductive modelling Author of 44 monographs and numerous articls Prepared more than 200 Cand. Sci (Ph.D.) and 27 Doct. Sci 6
2. International events on IM 2002 Lviv, Ukraine 1 st International Conference on Inductive Modelling ICIM 2002 2005 Kiv, Ukraine 1 st International Workshop on Inductive Modelling IWIM 2002 2007 Prague, Czech Republic 2 nd International Workshop on Inductive Modelling IWIM 2007 2008 Kiv, Ukraine 2 nd International Conference on Inductive Modelling ICIM 2008 2009 Krnica, Poland 3 rd International Workshop on Inductive Modelling IWIM 2009 2010 Yevpatoria, Crimea, Ukraine 3 rd International Conference on Inductive Modelling ICIM 2009 Zhukn (near Kiv, Ukraine) Annual International Summer School on Inductive Modelling 7
3. Attempt to define IM: what is it? IM is MGUA / GMDH IM is a technique for model self-organization IM is a technolog for building models from nois data IM is the technolog of inductive transition from data to models under uncertaint conditions: small volume of nois data unknown character and level of noise inexact composition of relevant arguments (factors) unknown structure of relationships in an object 8
4. IM destination: what is this for? IM is used for solving the following problems: Modelling from experimental data Forecasting of complex processes Structure and parametric identification Classification and pattern recognition Data clasterization Machine learning Data Mining Knowledge Discover 9
5. IM explanation: algorithms and tools Basic Principles of GMDH as an Inductive Method Given: data sample of n observations after m input x 1, x 2,, x m and output variables Find: model f(x 1, x 2,, x m,θ) with minimum variance of prediction error GMDH Task: f arg minc( f ), C( f ) model qualit criterion, I f I set of models Basic principles of the GMDH as an inductive method: 1. generation of variants of the graduall complicated structures of models 2. successive selection of the best variants using the freedom of decisions choice 3. external addition (due to the sample division) as the selection criterion Sample Part А Generation of models f I Part В Calculation of criterion С( f ) f * C min 10
Main stages of the modeling process D A T A (s a m p le, a p rio r in fo rm a tio n ) C h o ic e o f a m o d e l c la s s S tru c tu re g e n e ra tio n P a ra m e te r e s tim a tio n C rite rio n m in im iz a tio n A d e q u a c a n a l s is F in is h in g th e p ro c e s s 11
GMDH features Model Classes: linear, polnomial, autoregressive, difference (dnamic), nonlinear of network tpe etc. Parameter estimation: Least Squares Method (LSM) Model structure generators: GMDH Generators Sorting-out Iterative Exhaustive search Directed search Multilaered Relaxative 12
13 Main generators of models structures 1. Combinatorial: 1 1, 1,, 1,, ) ( s s j s i s l s F l i m s x X θ ) ) 2. Combinatorial-selective: 3. Selective (multilaered iterative): 2 2 5 2 4 3 2 1 1 1, ; 1,, 0,1,...;, ) ( ) ( F r j r i r j r i l r j l r i l r l C l F j i r + + + + + ϑ ϑ ϑ ϑ ϑ ),...,, ( ; 1,...,2, 2 1 m m v v v d d d d v X θ )
14 External Selection Criteria Given sample: W (X ), X [nxm], [nx1] Division into two subsamples: n n n X X X W W W B A B A B A B A + ; ; ;,,,, ) ( 1 B W A G X X X G T G G T G G θ ) Parameter estimation for a model Xθ: Regularit criterion: 2 B W A X W X CB θ θ ) ) Unbiasedness criterion: 2 A B B B X AR θ )
IM tools Information Technolog ASTRID (Kiv) KnowledgeMiner (Frank Lemke, Berlin) FAKE GAME (Pavel Kordik at al., Prague) GMDHshell (Oleksi Koshulko, Kiv) 15
6. Basic Theoretical Results f * arg min C( f ). f F F set of model structures С criterion of a model qualit Structure of a model: ) f ), θ ( X f Estimation of parameters: ) θ f arg Q criterion of the qualit of model parameters estimation f min θ R f m ) Q( θ f ). 16
Main concept: Self-organizing evolution of the model of optimal complexit under uncertaint conditions Main result: Complexit of the optimum forecasting model depends on the level of uncertaint in the data: the higher it is, the simpler (more robust) there must be the optimum model Main conclusion: GMDH is the method for construction of models with minimum variance of forecasting error 17
6 J(s σ 2 ) σ 2 2,0 σ 2 1,5 6 J(σ 2 s) s 4 s 3 s 2 5 σ 2 1,0 5 s 1 4 4 s 0 3 3 2 J b (s) J(s 0) σ 2 0,5 2 1 1 0 σ 2 0 0 1 2 3 4 s σ 2 кр(2,3) σ 2 кр(1,2) σ 2 кр(0,1) σ 2 0 0 0,5 1 1,5 2 2,5 Illustration to the GMDH theor 18
7. IM compared to ANN and CI Selective (multilaered) GMDH algorithm: x 1 f 1 g 1 x 2 f 2 g 2 f x 3 f 3 g 3 x 4 f 4 g 4 m C 2 m F C 2 F F 19
Optimal structure of the multilaered net x 1 f 1 x 2 g 2 f x 3 f 3 x 4 m f 4 C 2 m F g 4 C 2 F F 20
8. Real-world applications of IM 1. Prediction of tax revenues and inflation 2. Modelling of ecological processes activit of microorganisms in soil under influence of heav metals irrigation of trees b processed wastewaters water ecolog 3. Sstem prediction of power indicators 4. Integral evaluation of the state of the complex multidimensional sstems economic safet investment activit ecological state of water reservoirs power safet 5. Technolog of informative-analtical support of operative management decisions 21
9. Main centers of IM research IRTC ITS NANU, Kiv, Ukraine NTUU KPI, Kiv, Ukraine KnowledgeMiner, Berlin, German CTU in Prague, Czech Sichuan Universit, Chengdu, China 22
10. IM development prospects The most promising directions: 1. Theoretical investigations 2. Integration of best developments of IM, NN and CI 3. Paralleling 4. Preprocessing 5. Ensembling 6. Intellectual interface 7. Case studes 23
THANK YOU! Volodmr STEPASHKO Address: Prof. Volodmr Stepashko, International Centre of ITS, Akademik Glushkov Prospekt 40, Kiv, MSP, 03680, Ukraine. Phone: +38 (044) 526-30-28 Fax: +38 (044) 526-15-70 E-mail: stepashko@irtc.org.ua Web: www.mgua.irtc.org.ua 24