Introduction. A Bayesian Network Approach to Forecasting

Transcription

1 From: FLAIRS-00 Proceedngs. Copyrgh 000, AAAI ( All rghs reserved. Inferencng Bayesan Neworks From Tme Seres Daa Usng Naural Selecon Andrew J. Novoblsk The Unversy of Texas a Arlngon P.O. Box 438 Gulf Shores, AL (334) andyn@novoech.com Farhad A. Kamangar, Ph.D. Compuer Scence & Engneerng Dep. Unversy of Texas a Arlngon Arlngon, TX 7609 (87) kamangar@cse.ua.edu Absrac Ths paper descrbes a new framework for usng naural selecon o evolve Bayesan Neworks for use n forecasng me seres daa. I exends curren research by nroducng a ree based represenaon of a canddae Bayesan Nework ha addresses he problem of model denfcaon and ranng hrough he use of naural selecon. The framework consrucs a modfed Naïve Bayesan classfer by searchng for relaonshps whn he daa ha wll produce a model for he underlyng process generang he me seres daa. Expermenal resuls are presened ha compare forecass n he presence of mulple sources of nformaon made usng he naurally seleced belef nework versus a random walk. Inroducon Recen leraure n he felds of Daa Mnng and forecasng has ncluded arcles oulnng new approaches for forecasng me seres daa ulzng Bayesan Neworks[]..A Bayesan Nework s a dreced acyclc graph wh nodes represenng he arbues of he model and dreced lnks represenng a causal relaonshp beween paren and chld. Togeher, hs nformaon represens he dependence beween varables and gves a concse specfcaon of he jon probably of he model []. Sgnfcanly, provdes a whe box approach o represenng relaonshps ha exs whn he doman beng modeled and can handle nferencng n he absence of complee nformaon. One area of parcular neres s he way n whch boh he srucure and values of nework represenaons can be learned from he daa alone. To hs end, a sgnfcan amoun of work has been done on learnng he nework srucure gven an denfed se of arbues of he daa o be modeled. However, n he case of forecasng a me seres, each pas value becomes a poenal arbue for use n he nework model descrbng he process ha produced he me seres. Ths ncrease n compuaonal complexy, plus he ably o enhance he forecas model usng daa sources exernal o he me seres beng forecas, underscores he need for an auonomous mehod for denfyng he forecas model. Examples of frameworks ha address hese ssues n he Copyrgh 000, Amercan Assocaon for Arfcal Inellgence ( All Rghs Reserved. conex of usng Bayesan Neworks o forecas me seres daa nclude [3,4]. The framework presened here reles upon evoluonary compung o derve a vald forecasng model from he me seres o be forecas by usng Genec Programmng. Genec Programmng [5] s a search mehod based on he mechancs of naural selecon [6] ha has been adaped o manpulae ree based models. There s a srong poenal, as evdenced n compleed work such as Kennedy e. al.[7] and De Jong e. al.[8], for evoluonary programmng o have a posve mpac on he denfcaon of approprae Bayesan nework models hrough correc encodng of he nework srucure and proper choces of fness funcons. One of he mos promsng aspecs of usng a genec programmng framework for he denfcaon of Bayesan neworks s he closeness he wo approaches already share. For example, Ramon and Sebasan [9] pon ou ha learnng boh he Belef Ne's parameers and srucure from avalable daa can be vewed as a search problem for denfyng a suable Bayesan nework for modelng he sample daa. Heckerman also dscusses hs by ponng ou several mehods for selecng canddae nework represenaons, n addon o dscussng varous mehods for evaluaon he fness of he learned represenaon wh respec o he sample daa. A Bayesan Nework Approach o Forecasng The Evoluonary Bayesan Nework approach ulzes a Bayesan nework o model a sochasc process. The nework combnes condonally dependen and ndependen nformaon o produce a model of he saonary process producng he frs dfference n he me seres beng forecased. Consder he me seres Y ha s passed hrough a dfferencng mechansm o produce he seres. The seres can hen be represened by he model: + = m(,,..., a, a ) [Eq. ] where a, a -, are a se of ndependen, dencally dsrbued random varables and m() s a funcon boh he prevous values of and he random varables a. One of he smples models m() s ha of he Random Walk:

2 + = + a [Eq. ] The Random Walk, frs dscussed by Bacheler n 900 [0], has been used as he de Faco sandard of comparson for mehods ha forecas fuure values n he sock marke. The predcve model for he Random Walk s: ˆ + = + E[ a ] [Eq 3] Assumng ha a s normally dsrbued wh mean equal o 0, Eq 3 becomes: ˆ + [Eq 4] = and wll be used as he baselne model predcor ha he Bayesan Nework model predcor s compared o. The Bayesan model represenaon of Eq s: δ max P( δ...) [Eq 5], = + mn( ) δ max( ) where mn() and max() are he mnmum and maxmum possble values of and δ s a connuous varable represenng he possble values for a me +. Alhough possble o work wh predcors ha use he connuous value δ, he framework dscussed n hs paper reles on he se of dscree values D, conanng elemens {d, d, }. The values of D are defned for he predcve model by dvdng up he range (mn(),max()) no N dscree ranges R wh boundares λ,such ha: mn( ) < R λ [Eq. 6] λ < R λ for =,,N- [Eq. 7] λ N < R N < max( ) [Eq. 8] wh d defned as: d = λ [Eq. 9] λ + λ d = for =,,N- [Eq. 0] N d = λ N = max() [Eq. ] Gven he values of D, he predcve model s defned as a Bayesan Classfer. The classfer s raned usng a se of arbues ha relae values of d wh pror values of and reurns he predced value for d by ulzng he maxmum a poseror hypohess o creae: ˆ ˆ d, max P( d...) [Eq. ] + = d D Inally, he smplfyng assumpon s made ha he pror values of are condonally ndependen gven he value of a me. Based on hs assumpon, Eq 4 can be rewren as he Nave Bayes Classfer: ˆ + = dˆ, max P( d ) P( j d ) [Eq. 3] d D j= The compuaonal mpac of usng he Nave Bayes classfer s mporan. I grealy reduces he amoun of sorage and number of calculaons requred o represen he Bayesan nework. Ths s sgnfcan n an naural selecon envronmen where all members of he populaon mus be creaed, raned, and hen esed for fness durng each generaon. However, he smplfyng assumpon of condonal ndependence beween pror values of means ha he MAP hypohess wll only be reurned when hs assumpon s rue. Ths s no he case when a process has an auoregressve componen o. Therefore, s necessary o suppor he case where he value of s condonally dependen on a subse of he values of a pror mes. The framework suppors he possbly of auoregressve componens by movng from a Nave Bayes classfer o a smple Bayes classfer, defned as: ˆ + = dˆ,maxp( d l, l,...) P( k P j d )) ( ) d D k L j= j L [Eq. 4] where L s he subse of lags {l, l, l 3, }from K, he se all lags {0,,,}. The las mprovemen he framework makes o he basc classfer s he applcaon of Occam's Razor o he number of erms used whn he classfer. Ths subse of K, K ' s hen used gvng: ˆ + = dˆ,max P( d d D l, l,...) k L P( k) ) j j L K ' P( j d ) [5] Fgure shows a sample classfer, n nework form, for + ha reles on he condonally dependen erms { -, -3 } and he condonally ndependen erms { -, -5 }. Naural Selecon Gven he desred form of he Evolved Bayesan Nework model, s necessary o specfy a ree srucure ha wll sasfy he requremens of he naural selecon process. Specfcally:. The ree srucure mus allow for he possble selecon of any - no eher he condonally ndependen or condonally dependen secon of he nework.. The ree mus allow for he dynamc specfcaon of range mappng (posonng of λ's) durng he quanzaon process of ndvdual saes for each varable node. 3. The ree mus no nhb he genec operaors used n he naural selecon process. The ree used by he naural selecon framework o evolve populaons of member Bayesan Neworks s bes descrbed as a se of rees whn a ree srucure. The roo ree, shown n Fgure, s used o descrbe he member varables conaned whn he nework and he condonal relaonshps beween hem. Each node conans a sngle neger arbue ha wll be used o deermne some aspec of he nework based on he node's

3 poson relave o he roo. The frs chld node always represens, whle each of he remanng chldren represen he remander of he - 's used n defnng he predcng model. Specfcaon of he - beng represened by a parcular node s encoded n ha node's value Fgure -A Smplfed Bayesan Nework for Selecng d Dependen -5 Independen + -5 Fgure - A Sample Roo Tree and he Bayesan Nework represens. Each node conaned whn he roo ree conans a subree ha encodes he λs used o paron he nerval (mn(), max()) no dscree caegores. Durng he naural selecon process, hgh level belef neworks can be modfed by addng or removng subrees o he mmedae chld level of he roo ree. Enhancemens o he low level quanzaon nformaon s made by aachng new subrees o he chldren of he nodes represenng varables n he nework. Snce he value of he node n he genec encodng s he same n all cases, he lsed consrans are me. The Genec Programmng Framework The Genec Programmng framework akes a seres o be forecas, a se of relaed daa seres o be used n predcng he forecas value, and a se of parameers ha drve he process. The man algorhm mpors he daa (Y ). convers he seres o be forecas no he desred dfference model ( ) and hen dvdes he daa no ranng, es, and evaluaon subses. Nex, he evoluonary search mechansm s nvoked and reles upon Neca, a commercal off he self (COTS) produc wh a programmable Applcaons Programmer Inerface, o buld, ran, and provde probables for he query node durng forecasng. A seres of ndependen runs are compleed, wh each run conssng of applyng he reproducons, crossover, and muaon operaors o produce a seres of generaons of poenal Bayesan Neworks. The bes ndvdual across all generaons of all runs s kep and used for he fnal forecas. Once he fnal forecas s made, s compared agans a Random Walk as a relave performance measuremen. Boh he resulng forecas values and performance sascs are preserved for pos processng purposes. The evoluonary par of he algorhm derves from he use of reproducon, crossover, and muaon o generae a new populaon from an old one. The choce of usng crossover or reproducon o add members o he new populaon s based on he desred percenage of new members generaed usng reproducon. For example, 30% of he new populaon s o be generaed by reproducon, hen he remanng 70% wll be generaed by crossover. Reproducon s he selecon of an ndvdual o be carred forward as-s no he nex generaon. The random selecon of ndvduals s made usng a probably dsrbuon based on he lnear fness rankng [] of he populaon. Members wh he bes fness values are hgher ranked and herefor seleced more ofen. Crossover s he creaon of wo new members by randomly selecng wo exsng members (possbly he same member wce) of he curren populaon n he same manner as reproducon. Then a sngle lnk n each of he ndvduals s randomly seleced and broken o creae an upper and lower paral ree. The upper ree of he frs ndvdual s combned wh he lower ree of he second ndvdual and he upper ree of he second ndvdual s combned wh he lower ree of he frs ndvdual wo creae wo new members for he nex generaon. Muaon s defned as randomly changng he value of a node whn he ree represenaon of an ndvdual. Ths operaon occurs wh a very low probably and s nended o dsplace he ndvdual f becomes suck n a local mnmum. In order for he Evolvng Bayesan Nework framework o ulze he gven ree represenaon o encode he Bayesan nework was necessary o add one adapaon relaed o crossover and muaon. Snce crossover could occur beween arbrary pons, s possble ha he ac of crossover or muaon could change eher he number of nodes n he nework or he organzaon of he caegores ha each node was broken no. Therefore, he member rees had o be scanned for valdy. If a ree was found o be nvald was kep n he populaon bu gven a very low fness rankng. Invald rees were added o he end of he ranked ls afer beng sub ranked by her complexy, wh he leas complex beng gven a beer rank poson. Once he ree s ransformed no a nework, ranng commences. Fness s compued by usng he ndvdual Bayesan Nework o selec he mos lkely value of he me seres for he nex me perod gven nformaon from prevous values of he me seres. Tranng s

4 accomplshed makng a subse of he daa avalable o he Neca case based learnng funconaly afer he canddae ree was defned. Once ranng s compleed, he ndvdual s measured for fness, usng he fness funcon, F(), defned as: F ( member ) ( ˆ = MNSE, ) [Eq. 6] where MNSE, he Mean Normalzed Square Error, s compued as: ˆ = ˆ MNSE (, ) = max( ) mn( ) [Eq. 7] Fness s measured by usng he raned nework o forecas values for he subse of se asde for esng. Ths process s hen repeaed for he seleced number of runs, generaons, and populaon sze unl he bes Bayesan Nework s produced by he framework. The resulng Bayesan Nework s hen used o forecas values for he remanng subse of se asde for he purpose of evaluang he forecasng mehod agans a Random walk. The success of he search process depends on new ndvduals beng creaed from he mos relavely f n he curren populaon. A fness value for an enre member s consdered o be shared by each member of he power se of ndvdual sub rees conans. These subrees, or schemaa, are hen combned durng he generaon of a new populaon o creae, n heory, a more robus populaon ha wll produce ncreasngly f ndvduals. Unforunaely, hs echnque s no guaraneed o converge o he bes f ndvdual and herefore requres he use of large populaons randomly sampled o creae he nal populaon such ha as many schemaa as possble are represened. Expermenal Resuls Table shows he overall resuls of forecasng for he GM me seres gven he number of records o hold back for esng, and he mehod used o provde he forecas. Each forecas consss of 40 one sep ahead predcons usng a nework ha was evolved usng a me seres wh 94 values. Avalable hsorcal daa was delvered as a sngle record conssng of he 9 prevous values of he varable. The HLD column ndcaes he number of records held back for he esng of he nework. The remanng 94 HLD records were used n he naural selecon of he Bayesan Nework and s supervsed ranng. Each expermenal forecas was made usng a populaon of sze 5, evolved hrough 5 generaons, wh he naural selecon process repeaed fve mes per forecas. Ths was suffcen o allow he process o acheve convergence when 70% of he populaon esed beer han he random walk. In addon o he unvarae forecas, represened by MAP n he METHOD column, expermenal resuls were compled for he mulvarae mehod M-MAP (allowng he framework o buld a nework ulzng nformaon from oher secures) and for he random walk, RWALK. HLD METHOD GNS RNS MNSE TRN MNSE EVL MAPE EVL 40 MAP % 40 M-MAP % 40 RWALK % 60 MAP % 60 M-MAP % 60 RWALK % 80 MAP % 80 M-MAP % 80 RWALK % Table - Expermen Resuls for Forecasng he Fuure Value of GM Sock Prces. The las hree columns conan nformaon on he forecasng error obaned when comparng he acual value of he seres o he forecas value produced by he bes ndvdual produced by he evoluonary process. Ths nformaon has been spl no hree columns; MSNE TRN, MSNE EVL, and MAPE EVL. MSNE TRN conans he Mean Square Normalzed Error for he ndcaed number of records held back for he esng of he raned nework. MSNE EVL conans he MSNE for he fnal se of records used o evaluae he raned nework on nformaon no seen durng he selecon process. The MAPE EVAL column provdes he Mean Average Percen Error beween he rue and forecas value for he ranng record se and he evaluaon record se of he mos f member of he evolved populaon. MAPE was provded as an acceped sandard of error evaluaon and was no used n he process of naurally selecng he Evolved Bayesan Nework used o provde he forecass. Fgure 3 shows he complee daase for he GM Sock Prce me seres. Ths seres shows a generally rsng rend over me. For he GM forecas, he wnnng Bayesan nework from he naural selecon process, shown n Fgure 4 reled upon he prevous prce of GM sock, n addon o prevous prces from he GE and IBM sock as well. Ths nework was used o generae he forecas shown n fgure 5. Fgure 6 shows he Average Percenage error for each of he forecas values generaed durng he evaluaon phase of he framework. Dscusson Overall, he framework for naurally selecng Bayesan Neworks ouperformed he Random walk, bu no by a large margn. The bes forecas was delvered by an Evolved Bayesan Nework wh a holdback of greaer han or equal o 60 records, 0 records more han he number forecas durng he evaluaon phase of he framework. Addonally, he bes MAPE EVAL value belonged o he same forecas ha conaned he bes MSNE EVAL value. The forecasng resuls for he bes naurally seleced Bayesan Nework were whn 4% for frs 7 values and 5% for remanng values over he daa

5 held back for evaluaon of he seleced me seres. I s neresng o noe ha he seleced forecas model holds whn he noed error olerance over an exended perod of me whou requrng reranng of he nework. Prce* GM Sock Prce Tme Fgure 3 - The GM Sock Prce Tme Seres. GM -0 IBM -5 IBM -9 IBM - The framework for naurally selecng Bayesan Neworks was also able o separae he classfcaon of daa pons no relevan arbues from he process of usng he dscovered arbued n he hgh level forecasng model. Ths resuled n allowng he operaors nvolved wh he genec programmng based search of he model space o cause changes n he quanzaon levels of seleced arbues o occur more frequenly han he organzaon and addon/removal of he arbues hemselves. In essence, he naural selecon process would move he search o a hgh probably area of success, and hen allow for refnemen of he seleced model o oban he bes soluon n ha localy. 6% 5% 4% 3% % GM GE -5 Fgure 4 - The Bayesan Nework Model for he Prce of GM Sock. Acual GM Forecas GM % 0% Tme Fgure 6 - The Average Percen Error of he Forecased GM Sock Prces. Prce Tme Fgure 5 - The Forecased Prce of GM Sock. Based on he nal resuls, would be neresng o see f he seleced nodes and her accompanyng probables could be mapped drecly o he coeffcens of an ARIMA model wh some degree of uly. I would also be neresng o sudy he effec on he applcably of usng he Model Averagng mehod s each of he avalable hypohess were sgnfcanly dfferen. Conclusons The problem of denfyng and ranng a model ha can be used o forecas he value of a me seres gven hsorcal daa has been he focus of much research. The mehod dscussed n hs paper combnes naural selecon wh a Bayesan Model. The resulan forecasng framework shares smlar model componens wh he Box and Jenkn's ARIMA model and has been expermenally shown o evolve Bayesan Neworks ha perform beer han a random walk References [] Heckerman, D., A. Mamdan, and M. Wellman Real World Applcaons of Bayesan Neworks. Communcaons of he ACM 38, no. 5: [] Russell, Suar, and Peer Norvg Arfcal Inellgence: A Modern Approach. Englewood Clffs: Prence Hall. [3] Abamson, B. 99. ARCO: An Applcaon of Belef Neworks o he Ol Marke. In Proceedngs of The Uncerany n Arfcal Inellgence Conference: -8. [4] Dagum, P., A. Galper, E. Horvz, and A. Sever Unceran Reasonng and Forecasng. Inernaonal Journal of Forecasng, no. : [5] Koza, John Genec Programmng: On he Programmng of Compuers by Means of Naural Selecon. Cambrdge: MIT Press. [6] Holland, John. 99. Adapaon n Naural and Arfcal Sysems: An Inroducory Analyss wh Applcaons o Bology, Conrol, and Arfcal Inellgence. Cambrdge: MIT Press. [7] Kennedy, H., C. Chnnah, P. Bradbeer, and L. Morss The Consrucon and Evaluaon of Decson Trees In a Comparson of Evoluonary and Concep Learnng Mehods. In Seleced Papers from he AISB Inernaonal Workshop Evoluonary Compung. Sprnger-Verlag. [8] De Jong, K., W. Spears, and D. Gordon Usng Genec Algorhms for Concep Learnng. San Dego, CA: Naval Research Lab Techncal Repor. [9] Ramon, M., and P. Sebasan Learnng Bayesan Neworks from Incomplee Daabases. In Proceedngs of he Threenh Conference on Uncerany n Arfcal Inellgence [0] Bacheler, Lous The Random Characer of Sock Marke Prces. Eded by Paul Cooner. Theory of Speculaon. MIT Press. [] Mller, B., and D. Goldberg Genec Algorhms, Selecon Schemes, and he Varyng Effecs of Nose. In Evoluonary Compuaon Journal 4, no. :