Model Selection in Functional Networks via Genetic Algorithms

Transcription

1 Model Selection in Functional Networs via Genetic Algoritms R E Pruneda and B Lacruz Abstract Several statistical tools and most recentl Functional Networs (FN) ave been used to solve nonlinear regression problems One of te tass associated wit all of tese metodologies consists of discovering te functional form of te contribution of te eplanator variables to te response variable In tis paper, we tacle tis problem using functional networ models (FNs) Since tese models usuall involve from a moderate to ig number of parameters, a genetic algoritm (GA) for model selection is proposed After an introduction of FNs and GAs, te performance of te proposed metodolog is assessed using a simulation stud as well as a real-life data set Kewords: functional networs, genetic algoritms, nonlinear regression Introduction Functional networs ave been applied to reproduce te relationsip between a response variable Y and one or more predictor variables X, X 2,, X in [3], [4] and [5] In tis paper, we consider tat te relationsips among tese variables can be written as f(y ) = (X, X 2,, X ) + ɛ () were ɛ is a random error wose epected value is assumed to be 0 Our purpose is to discover te structure of te transformations f and in () Functional networs can be seen as te grapical representation of a functional equation (an equation were te unnowns are functions), wic provides a better understanding of te properties of te model at and Te principal steps to wor wit functional networs consist of () selecting te topolog (guaranting te uniqueness) and a set of basic functions to approimate te unnown functions in te model and (2) learning, wic involves to coose a criterion to estimate te parameters and a procedure to select te best model Dpt of Matematics, Universit of Castilla-La Manca and Dpt of Statistical Metods, Universit of Zaragoza (Spain) E- mails: rosapruneda@uclmes, lacruz@unizares Te autors are partiall supported b te Ministr of Science and Tecnolog of Spain troug CICYT/FEDER Project MTM , b te Junta de Comunidades de Castilla-La Manca, troug project PAI and Gobierno de Aragón (troug consolidated researc group Stocastic Models) In tis paper, we present two basic functional networ topologies: te additive model and te additive general model, wic are briefl described in Sections 2 and 22 Te advantage of tese two models is tat te estimation problem can be solved using te constrained least squares criterion, wic leads to solve a linear sstem of equations Moreover, te set of basic functions cosen to approimate te unnown functions is te polnomial famil of linearl independent functions Φ = {, t, t 2,, t q } Model selection is tacled via Genetic Algoritms (GAs) Te are euristic searc algoritms based on te evolutionar ideas of natural selection and genetics An introduction can be found in [7] Previous wors in functional networs consider forward-bacward or eaustive searcing metods ([3], [4] and [5]) But, because of te computational cost, tese procedures are useless wen te number of parameters is large Te rest of te paper is structured as follows In Section 2 two basic functional networ models are introduced Genetic algoritms are described in Section 3 togeter wit te strategies for its application to our particular problem Te performance of te proposed tecniques is sowed in Sections 4 and 5, were a simulation stud and a real-life data set are presented 2 Some functional networ models We propose to approimate () using te additive and te general additive functional networ models Te additive model approimates in () b a sum of functions, one on eac eplanator variable, wic allows us to analse te contribution of eac predictor separatel Te general additive model also includes te interactions among tem 2 Additive model Te additive functional equation is f() = ( ) + 2 ( 2 ) + + ( ), (2) wic leads to te functional networ sowed in Figure To estimate f and,, in (2) we consider linear

2 u 2 2 u 2 u + Figure : Te additive functional networ combinations of basic functions, tat is, ˆf() = j= a 0j 0j () and ĥi( q i i ) = a ij ij ( i ) (3) j= and te problem is reduced to estimate te parameters a ij, i, j If te set of basic functions is te polnomial famil, equation (2) can be approimated b j= a 0j j = a + q j= f - q a j j + + a j j, (4) j= were a is te constant term, wic is included just once for avoiding identifiabilit problems Note tat te number of parameters in tis model is i=0 q i + If q i = q, i, ten te number of parameters is q (+)+ 22 Te General Additive Model A more general form of approimating in () is considered wen we use te general additive functional equation f() = q r = q r = c r r r ( ) r ( ), (5) were c rr are unnown parameters and eac belongs to te famil of basic functions An eample of its corresponding functional networ model, for = 2 and q = q 2 = q, is sown in Figure 2 2 q q c c q c q c qq Figure 2: Te general additive functional networ for = 2 and q = q 2 = q To estimate te unnown function in (5) we just need to approimate f as in (3) Ten, te problem is reduced to + f - estimate te parameters a 0j and c r r, j, r,, r If te set of basic functions is te polnomial famil, equation (5) can be approimated b j= a 0j j = q r =0 q r =0 c r r r r (6) Note tat te number of parameters in tis model is + i= (q i+) Wen q i = q, i, te number of parameters is q + (q + ) 3 Genetic Algoritms Te number of parameters in a functional networ model depends on te number of eplanator variables and te number of elements in te set of basic functions In te additive model, te dimensionalit of te problem grows linearl wit te number of eplanator variables, but wen te general additive model is considered, te dimensionalit grows eponentiall wit te number of eplanator variables Ten, wen te number of parameters is from moderate to ig, an euristic searc metod must be implemented for model selection In tis paper, a genetic algoritm is considered Its purpose is to select te optimal subset of parameters to compose a model wic provides a good approimation to () A genetic algoritm starts wit a random set of initial models Eac model is represented b a string of binar caracters, called cromosomes As an eample, let us consider te model selection problem of an additive functional networ wit two eplanator variables f() = ( ) + 2 ( 2 ), were f, and 2 are approimated b polnomials of degree 3 Eac model is ten represented b a cromosome of lengt 0 Te cromosome v = [] represents te complete model, tat is, all te terms are included in te model, α + α α 3 3 = β 0 + β + β β β β β And, te cromosome v = [000000] represents te linear model, tat is, onl te linear terms in, and 2 are included in te model, α = β 0 + β + β 2 2 Eac model of te initial set is evaluated b te adjusted R-squared criterion (Ra), 2 wic penalizes models wit a ig number of parameters: n Ra 2 i= = e2 i /(n p) n i= ( ˆf( i ) ˆf( (7) i )) 2 /(n ), were n is te sample size, p is te number of parameters in te model, e i is te i-t residual (for eample, for te additive model, e i = ˆf( i ) ĥ( i ) ĥ( i ), were ˆf and ĥi are obtained as in (3)) and ˆf() = n n i= ˆf( i )

3 Net, te best cromosomes of te initial population, in te sense of tose wit igest R 2 a, are selected Ten, a new population is obtained b appling genetic operations as crossover and mutation Tis population is again evaluated and te process is repeated for some specified number of additional generations or wen te evaluation function does not improve an more Te size of te initial population and te probabilities of mutation and crossover ave to be cosen b te user See [7] for furter details 4 Assessing Performance Using Simulation To assess te performance of te proposed metod we use a set of simulated data from te model Y 2 = X 2 + X 2 2 X X 2 + ɛ, (8) were X and X 2 are independent U[0, 2], and ɛ is N[0, 0] and independent of X and X 2 We ave estimated te model b appling te general additive functional networ model, described in Section 22, wit two eplanator variables All te functions ave been approimated b tird degree polnomials Te genetic algoritm described in Section 3, wit populations of size 700, and crossover and mutation probabilities equal to 04, and 0, respectivel, as been applied undredfold In tis eample, te models are represented b cromosomes of 9 bits Te cromosome representing te true model (8), including te constant term, is [ ] Te mapping of tis cromosome and te corresponding terms in te functional networ model can be depicted as follows: Table sows te terms included in te 7-t best selected models, ordered b te value of te evaluation function R 2 a Te are models wose R 2 a is greater tan 079 Te constant term does not appear in te table, since it is alwas included in te model Model number 5 is te true model wit R 2 a = Te rest of te models ave R 2 a values ver close to tat Note tat all te models contain a number of terms less tan 6 (including te constant term), far awa from te maimum, 9 Moreover, most of tem (0/7) ave eactl 4 parameters, te same as te true model We can define a simple measure of te compleit of te model b adding te powers of te terms included on it It is sown in te last column of Table Wit te elp of tis measure, Table : Selected Models to Approimate te Simulated Model in (8) 2 Interactions R 2 Compleit 2, , , , , 3, , 3 2 2, , we can coose te model wit smallest compleit among tose wit igest R 2 a In tis case, tis model is number 5, te true model 5 Assessing Performance Using Real data Boston Housing data set contains 506 observations of 3 continuous variables and binar valued variable related wit ousing values in suburbs of Boston Te purpose is to find te best fitting functional form and, in particular, to determine te pattern of te influence of air pollution on ousing values as measured b 5 Te variables are: : Median value of owner-occupied omes in dollar 000 s, : Per capita crime rate b town, 2 : Percentage of residential land zoned for lots over 25,000 sqft, 3 : Percentage of non-retail business acres per town, 4 : Carles River dumm variable (= if tract bounds river; 0 oterwise), 5 : Nitric oides concentration (parts per 00 millions), 6 : Average number of rooms per dwelling, 7 : Percentage of owner-occupied units built prior to 940,

4 8 : Weigted distances to five Boston emploment centres, 9 : Inde of accessibilit to radial igwas, 0 : Full-value propert-ta rate per dollar 0,000, : Pupil-teacer ratio b town, 2 : (B 063) 2 were B is te proportion of blacs b town, 3 : Proportion of lower status of te population Tis data set was created b Harrison and Rubinfeld, [6], and it is analzed in [] and [2], among oters In [6] and [] a linear model is proposed were, 8, 9 and 3 are transformed b logaritms and 5 and 6 are squared In [2] te ACE algoritm is applied to te transformed variables suggested in [6] Te conclude tat te best model onl need 6, 0, and 3, as predictors, wit a milder transformation for (different tan logaritmic), a transformation for 6 different from squared one and some transformation for 0 We appl an additive functional networ model Eac function is approimated b tird degree polnomials Te GA proposed in Section 3 is applied Te models are represented b cromosomes of 42 bits Table 2 sows te best models (Ra 2 > 068) obtained b repeating undredfold te GA All tese models include a complete transformation in and te constant term Note tat an model contain 6, 0 or 2 Most of te models suggest a complete transformation of, 2, 3 and 3 and include just one term of 5, 7, 9 and Note tat 5 appears in all te models squared or cubic, as it was found b Harrison and Rubinfeld [2] Breiman, L, and Friedman, JH, Estimating Optimal Transformations for Multiple Regression and Correlation (wit discussions), Journal of te American Statistical Association, V80, N39, pp , 985 [3] Castillo, E, Cobo, A, Gutiérrez, JM and Pruneda, RE, An Introduction to Functional Networs wit Applications, Kluwer Academic Publisers: New Yor, 998 [4] Castillo, E, Hadi, A S, Lacruz, B, Optimal Transformations in Multiple Linear Regression Using Functional Networs, Lecture Notes in Computer Science V2084, Part I, pp [5] Castillo, E, Hadi, A S, Lacruz, B and Gutiérrez, JM, Some Applications of Functional Networs in Statistics and Engineering, Tecnometrics, V43, pp 024, 200 [6] Harrison, D and Rubinfeld, D L, Hedonic Housing Prices and te Demand for Clean Air, Journal of te Environmental Economics and Management, N5 pp 8-02, 978 [7] Micalewicz, Z, Genetic Algoritms + Data Structures = Evolution Programs, 3rd Edition, Springer, 999 [8] UCI ML Repositor Database ttp://wwwicsuciedu/ mlearn/mlsummartml Attending to te compleit measure introduced in Section 4, te best model is number 4 followed b number 9 Bot give complete transformations of and 3, do not transform 7 and include 2 5 and 3 9 Te just differ in te transformations suggested for 3 and 8 6 Conclusions and Future Wor A genetic algoritm is presented as a powerful tool to select te terms involved in a functional networ model Te GAs solve te computational problems wic appear in model selection wit a moderate to ig number of parameters Te obtained models are simple and provide satisfactor approimations References [] Belsle, D A, Ku, E and Welsc, R E Regression Diagnostics Identifing Influential Data and Sources of Collinearit, Jon Wile and Sons, 980

5 Table 2: Selected Models GA Terms in te selected model R2 Compleit, 3 3, 2 3, , , , , 2 2, , 2 3, , , , , 2 3, , 2 3, , 2 3, , , 3 3, , , 3 2 2, , , , , 3 2, 3 2 3, 2 3, , , , 2 3, , , , , , , , , , 2, 3 2, 3 2 3, 2 3, ,