Time-line Hidden Markov Experts and its Application in Time Series Prediction

Transcription

1 me-ne Hdden arkov Exers and s Acaon n me eres Predcon X. Wang P. Whgham D. Deng he Informaon cence Dscusson Paer eres Number 3/3 June 3 IN 7-64

2 Unversy of Oago Dearmen of Informaon cence he Dearmen of Informaon cence s one of sx dearmens ha make u he choo of Busness a he Unversy of Oago. he dearmen offers courses of sudy eadng o a maor n Informaon cence whn he BCom BA and Bc degrees. In addon o undergraduae eachng he dearmen s aso srongy nvoved n osgraduae research rogrammes eadng o Com A c and PhD degrees. Research roecs n saa nformaon rocessng connecons-based nformaon sysems sofware engneerng and sofware deveomen nformaon engneerng and daabase sofware mercs dsrbued nformaon sysems mumeda nformaon sysems and nformaon sysems secury are arcuary we suored. he vews exressed n hs aer are no necessary hose of he dearmen as a whoe. he accuracy of he nformaon resened n hs aer s he soe resonsby of he auhors. Coyrgh Coyrgh remans wh he auhors. Permsson o coy for research or eachng uroses s graned on he condon ha he auhors and he eres are gven due acknowedgmen. Reroducon n any form for uroses oher han research or eachng s forbdden uness ror wren ermsson has been obaned from he auhors. Corresondence hs aer reresens work o dae and may no necessary form he bass for he auhors fna concusons reang o hs oc. I s key however ha he aer w aear n some form n a ourna or n conference roceedngs n he near fuure. he auhors woud be eased o receve corresondence n connecon wh any of he ssues rased n hs aer or for subsequen ubcaon deas. Pease wre drecy o he auhors a he address rovded beow. Deas of fna ourna/conference ubcaon venues for hese aers are aso rovded on he Dearmen s ubcaons web ages: h:// Any oher corresondence concernng he eres shoud be sen o he DP Coordnaor. Dearmen of Informaon cence Unversy of Oago P O Box 56 Dunedn NEW ZEALAND Fax: ema: ds@nfoscence.oago.ac.nz www: h://

3 me-ne Hdden arkov Exers and Is Acaon n me eres Predcon Xn Wang Peer Whgham Da Deng Dearmen of Informaon cence Unversy of Oago Dunedn New Zeaand {xnw whgham ddeng}@nfoscence.oago.ac.nz Absrac A moduarsed connecons mode based on he xure of Exers E agorhm for me seres redcon s nroduced. A se of connecons modues earn o be oca exers over some commony aearng saes of a me seres. he dynamcs for mxng he exers s a arkov rocess n whch he saes of a me seres are regarded as saes of a H. Hence here s a arkov chan aong he me seres and each sae assocaes o a oca exer. he sae ranson on he arkov chan s he rocess of acvang a dfferen oca exer or acvang some of hem smuaneousy by dfferen robabes generaed from he H. he sae ranson roery n he H s desgned o be me-varan and condona on he frs order dynamcs of he me seres. A modfed Baum Wech agorhm s nroduced for he ranng of he me-varan H and has been roved ha by E rocess he kehood funcon w converge o a oca mnmum. Exermens wh wo me seres show hs aroach acheves sgnfcan mrovemen n he generasaon erformance over goba modes. Key Words: me eres redcon; xure of Exers; H; Connecons ode; Execaon and axmzaon; Gauss Probaby Densy Dsrbuon;

4 ŷ X D : Dynamc suaon defned for a me seres a me : D X X X. b σ π a : ae ranson robaby from sae a me - o sae a me. : Lkehood funcon of observng. : Probaby of observng wh. α β : A me seres aso a seres of observaons for H. {y y... y }. y : Vaue of a me seres a me. : Lengh of a me seres. X : γ ode nu a me s a vecor by embeddng L revous vaues of a me seres. X [y - y -... y -L ]. : Ouu from exer a me. : he number of saes n H aso he number of oca exers and he number of cusers exraced by fuzzy cuserng echnque. : Dsrbuon funcon of robaby densy n H. I s assumed Gauss n he aer. : Varance of Gauss dsrbuon. : ae seres of a arkov chan aong a me seres. {s s... s }. s : he vaue of a me. ζ: ace of sae vaues s can be aken. : : Probaby of beng n sae a. Vaue of H arameers used for E-se n he modfed Baum-Wech agorhm. : Varabe for H arameers or he arameers o be esmaed n -se n he modfed Baum-Wech agorhm. ξ: ace of H arameer can be aken. : Probaby of observng {y y... y } and endng u n sae a me. : Probaby of observng {y y... y } from me and sarng wh sae s a me. : Probaby of beng n sae a me when observng.. η : Probaby of beng n sae a me and beng n sae a me when observng. Fgure. Ls of he symbos used n he aer.

5 Inroducon In he fed of me seres anayss he modeng echnques can be dvded generay no wo caegores: oca modeng or nonaramerc modeng and goba modeng or aramerc modeng. Loca modes such as neares neghbour agorhm are formed n each se reed on amoun of daa. he hosohy s fndng segmens of he me seres ha cosey resembe he segmen of he curren on. Goba modes such as auoregresson modes and connecons modes are usuay consruced o f he whoe rocess of a me seres by mnmzng he squared error. One robem of oca modes s ha hey canno gve a goba descron of he me seres. However s aso no easy o consruc a snge goba mode o reresen a me seres recsey esecay when he me seres show some comex feaures such as chaos. o dea wh hese robems a ye of mode caed xure of Exers E aearng. he E was deveoed on dvde-andconquer rnce wh he dea ha dvdng a comex robem no some sme ones and deang wh each of hem searaey.. Connecons E modes he E was nroduced o connecons socey by Jacobs Jordan e a. 99 n 99. he man on s ranng some " sub-modes" n oca envronmens o make hem become "exers" over he oca envronmens and combnng he exers by some agorhms o generae fna ouu. Generay here are hree man mode srucures deveoed based on E: GE Gaed Exers and Herarchca xure of Exers Jordan e a. 99; Jordan e a. 99; Jordan e a. 994; Wegend e a. 996 HE Hdden arkov Exers Wegend e a. and IOH Inu/Ouu Hdden arkov ode Bengo e a. 995; Bengo 996. GE combnes he exers by a gang nework whch s usuay a ner Jordan e a. 99; Jordan e a. 99; Jordan e a. 994 or a non-near Wegend e a. 996 feed forward nework. In boh HE and IOH he exers are hosed by a H bu he sae ranson robabes n he IOH are generaed from a se of recurren neworks caed sae ranson nework.. E mode n me seres redcon For me seres modeng he benefs of E ncude ha on one sde coud be used o exrac regmes or saes from comex me seres on he oher sde akng a sub-mode o f each sae eads he ocased modeng more effcen and recse. Jus as Wegend sad "Exracng regme nformaon does no sacrfce redcon accuracy. In conrary we can oban beer redcons snce he exers can ruy be exers n her regon as oosed o coverng everyhng oory" Wegend e a ome eary works for me seres modeng ncude AR hreshod Auoregresson ong e a. 98 CAR Cassfcaon and Regresson rees Breman e a. 984 and AR uvarae Adave Regresson nes Fredman 99. hese modes smy s he nu sace no some regons and f hem ocay wh regresson modes. In connecons socey a he GE HE and IOHE modes have been used for me seres redcon Wegend e a. 996; Wegend e a. ; Bengo. hese modes have frmy sasca background as he mode-ranng s a rocess of maxmsng he robaby of observng he me seres nsead of mnmzng he squared error. In addon o hese modes here are aso some oher modes based on E: HE Lehr e a. 999 and he mode nroduced n Kohmorgen e a.. Boh modes are based on H and ryng o descrbe he sae ranson rocess more recsey. Anoher mode whch shoud 3

6 be cassfed o G mode s he mode nroduced n Cao 3 where a O s used as gang nework raher han a feed forward nework. For dynamca me seres s commony o defne hem n erms of a sae sace descron: s g s w y f s v where s R s s he sae of he me seres w v are whe nose. For such me seres hdden arkov mode s arorae for reresenng he underyng dynamca rocess. o Hs have been ouary adoed for me seres anayss. ome sgnfcan work abou H for me seres anayss ncude he foows: Hamon aed he dea of swchng regmes o mode condona varance of economca me seres where auoregressve condona heeroskedascy ARCH s used o mode he varance bu s arameers are regme-reaed and earned by E agorhm Hamon 989; 99; 996; Hamon e a Wegend combned H and nonnear feed forward neworks o redc robaby densy dsrbuon for a fnanca me seres: &P5 Wegend e a.. Bengo emoyed dfferen IOHEs whch ake near and near neworks as exers for fnanca reurn seres redcon Bengo. In addon o he acaons n economca fed Lehr Lehr e a. 999 and Kohmorgen used her modes for chaoc me seres segmenaon. Usuay H works n a E srucured mode where moderae he exers o reresen he sae ranson. When he mode s aed for redcon a robem face o s ha s unabe o descrbe he ransons a each me on recsey snce he sae ranson roery s defned over he whoe rocess. Hence he dffcuy s ha on one hand eoe nend o ake he advanage of H o mode me seres more accuraey on he oher hand he goba roery makes he ransons n same robaby and resus n nferor redcons. Here we nroduce a mode n E srucure named "HE" me-ne Hdden arkov Exers for on redcon. I has a smar rocess o a HE Wegend e a. bu he sae ranson roery s me-varan. ha means raher han hodng a ranson robaby for he whoe rocess he HE ocazes he ranson roery a each me on and modes from he dynamc suaon of he me seres. Hence he sae ranson a a me on s avaabe when he dynamc suaon of he me seres s known. he ranng rocess for HE ncudes decomosng a comex me seres no some sme and commony aeared saes earnng each of hem ocay by an exer and consrucng a frs order H o moderae he exers for observng he whoe seres wh maxmum robaby. Fnay a connecons mode s emoyed o earnng he mevaran ranson robaby. In he rocess of redcon a exers wh he same nus ake ar n redcon bu he reave conrbuons of hem are deermned by he H. he aer s organsed as foows: n secon we gve a descron abou he HE and comare wh oher E modes. he agorhms for he mode ranng are rovded n secon 3 and he redcon rocess s gven n secon 4. In secon 5 we es he mode wh wo me seres: Laser daa and Leuven daa foowed by dscusson and concuson n secon 6. In aendx we resen he deas abou a modfed Baum-Wech agorhm for mode ranng and gve he exanaon for s convergence. 4

7 H ae ran. Ne. Exer Exer Exer Fgure. HE wh oca exers moderaed by a H. "ae ran. Ne." eans "sae ranson Nework". s dfferenang oeraon. HE ode he archecure of HE s shown n Fgure he exers n HE have smar meanngs as ha n E bu hey are no med o arcuar srucures such as near or non-near feed forward nework whch s ofen he case n GE HE and IOH bu no aways arorae for me seres modeng. he exers may be any ye of connecons modes or regresson modes deendng on he suaby for a arcuar robem. Each exer resonds o a me seres sae exraced by a Fuzzy cuserng echnque accordng o he dynamca suaon of he me seres X X X so ha he daa sames ha have smar feaures are cusered no he same grous. hs aows he exers o be reavey sme o earn and herefore o have good generazaon roeres. he ae ranson Nework s a connecons mode used o ma ocazed sae ransons. I akes X as nu o esmae sae ranson so as o race he exers defned by some changng aerns of a me seres. he H combnes exers by he ror robaby of beng n each sae whch s generaed by he sae ranson robabes and revous sae saus. herefore he mxng rocess s deermned by boh neror nformaon and exeror nformaon. A rocessng dagram s gven n Fgure 3. In he HE assumes ha for each sae he robaby densy for observaons s Gaussan. I akes he ouu of he exer as he condona mean and aduss he varance n ranng rocess o f he nose eve on he sae. hs no ony aows a dsrbuon exanaon for he exer s ouu a each me on bu aso aves he way for usng Bayes aw o cacuae oseror robaby n onese-ahead redcon. o he HE has a cosed-oo re-correcon rocess. he advanages of HE over GE ncude ha he HE has dynamcs n exers-mxng rocess aows sae saus re-esmang by Bayes aw and no maon for exer srucure. Comarng wh HE boh of hem ake H as he dynamcs n exers-mxng however HE s abe o descrbe he sae ranson robabes a each on. o IOH HE has Gaussan exanaon for ouu of each exer whereas he sae nework n IOH us generaes condona mean and s exers mus be LPs. Ahough he modes n Lehr e a. 999 and Kohmorgen have Gauss assumon hey have a resrcon ha a exers share a same varance vaue and he aer one akes he vaue as he varance of mxed ouu. For me seres redcon as exers f her regons wh dfferen nose eves hey canno have same varance vaue. Even hough he nose eves of a exers are same he varance of he mxed ouu mus be smaer han ha of any snge exer. Oherwse he mxure woud be a faed one. Anoher robem wh he modes s ha for sae esmaon hey eher use sofmax- 5

8 funcon ccuagh e a. 989 Ps y ex{ [ y ŷ X ] } ex{ [ y ŷ X ] } P s Lehr e a. 999 or Bayes aw bu coverng us a emora neghbourhood Kohmorgen whou consder of he robaby derved from he rocess:. here s a sgnfcan dfference beween HE and above modes ha a he modes exce HE ake nu X o search exers or sae ransons. ha means he regons of he exers or he sae ranson are deermned by he nu oson whereas HE uses he dfferenaon of nu X. nce HE defnes he exers domans by some changng aerns akng X o race sae ranson woud be more effcen and recse. Prevousy he auhors have exermened wh defnng exers and modeng sae ransons by X bu generazaon erformance accuracy and convergence seed were nferor o he curren mode. 3 ode ranng he ranng rocess and he roducons n each se s shown n Fgure aes exracon he dynamca suaon of a me seres a me on s defned as foows: [ ] D X X X y y y... y y 3 y 3 L L Fuzzy C-means cuserng Bezdek 98; Bezdek e a. 99 s aed o cuser a me ons no grous accordng o he feaure defned n equaon 3. he rocess cacuaes he cuser membersh degree µ whch s defned as he degree o whch vecor X beongs o cuser and udaes cuser cenres V eravey o make he foowng obecve funcon reach a mnmum: ŷ ŷ ŷ... ŷ Ps Ps... Ps- Ps Ps- Ps Ps A[a] Exer Exer ae ran. Ne. Exer - Exer X... X Fgure 3. Dagram of E. s s he sae saus a me. Ps [Ps Ps Ps ]. Ps s he robaby of beng n a me. ŷ s he ouu from exer. Σ denoes summang oeraon. ʘ denoes mucaon beween wo vaues. denoes mucaon beween marxes. Ohers are same as Fgure. 6

9 ranng es Producons ae Exracon Fuzzy embersh Degree Ina Exer-ranng Exer Neworks Ina H-earnng H &Gauss Parameers ae Probaby for Refned H-earnng Exer Neworks Refned Exer-ranng H &Gauss Parameers ae ranson odeng ae ranson Nework Fgure 4. ranng rocess and he Producons. O [ µ ] D V 4 he cuserng rocess cassfes he me ons no cusers. hese cusers are caed saes of he me seres and are aso regarded as saes of he H. Hence µ may be nerreed as he degree o whch he on beongs o a secfed sae. 3. Exer ranng In boh na exer-ranng and refned exer-ranng oca exers are raned by corresondng daa sames cusered by sae exracon. In na exer-ranng a reavey hgh hreshod K s aed o he fuzzy membersh degrees o exrac he me ons ha are srongy feaured wh a arcuar sae. hese me ons are hen used o ran a oca exer so as o nk he exer o he sae. he hosohy behnd he rocess s ha some me ons ha beong o a sae of he me seres o a hgh degree shoud be exraced o ran a modue o make a oca exer for he sae. In he refned exerranng foowng na H-earnng a reavey ower hreshod K s aed for he robabes a me on beongs o each sae. hs aows he mode o assgn he on no a sae or wo saes f robabes for he wo saes are a over he hreshod. hs rocess cassfes a me ons no saes and uses hem o ran he corresondng exers. Comarng wh na exer-ranng he refned exer-ranng re-rans he exers wh reavey wder range of sames o make hem cover her domans oay. 3.3 H earnng Aong wh he exer ranng rocess he H earnng aso ss no wo rocesses: an na one and refned one. he dfference beween hem s ha he frs one s based on he erformance of he na-raned exers and he aer one s on he refned-raned exers. he erformance s deermned by he ouu from each exer whch rovdes he condona mean for he assumed Gauss dsrbuon on he corresondng H sae. he H wh me-varan ranson roery s earn by a modfed Baum-Wech agorhm based on E rnce Baum e a. 97; Demser e a. 977; Rabner

10 uosng he robaby dsrbuon of he H s Gaussan. For sae b y y s X [ y ] ŷ X σ e 5 πσ As each sae s assocaed wh a oca exer he number of cuser s aso he number of saes and he number of execs.... ;... ; s ζ. mar o Baum e a. 97; Lorace 98 he kehood funcon for geng observaons wh curren arameers and o-be-omsed arameers ξ s defned as foows: og P 6 ζ Gven a arcuar sae sequence he robaby of geng s: π s a s s bs y P 7 where π { π } s he robaby of beng n sae s s ζ a. As n HE he sae s ranson robaby s me-varan a me s defned as he robaby of ransng from sae s - a me - o sae s a me : s s. Hence he kehood funcon becomes: ζ ζ as s og π og a ogb y 8 s s s ζ In order o esmae he arameers for obanng he maxmum of he kehood funcon he foowng wo ses shoud be reeaed. Execaon E-se In E-se a forward and a backward rocess are erformed. In he forward rocess a robaby of observng he ara sequence and endng u n sae { y y... y}... a me s defned as: α y y... ys. In he backward rocess we defne β s for he robaby of sarng wh sae s a me o observe. he robaby of beng n sae a me for observng he whoe { y y... y } sequence of observaons s defned as γ s. hen he hree ars n funcon become see Aendx A: og s π 9 og a s s 3 og b y s s 8

11 axmsaon -se By maxmsng he funcon udang formuas are derved as foows: ~ s π a~ s s 3 s ~ σ [ γ y yˆ X γ ] 4 By he modfed Baum-Wech agorhm funcon and P can converge o oca maxmums Aendx B. 3.4 ae ranson modeng he sae ranson roery s a seres of marx enres corresondng o he hdden sae sequence for each on of he me seres. In HE when he me seres dynamca suaon changes here w be a change of he exer ha s bes for modeng curren suaon or a change of he rooron ha each exer conrbues o he ouu. ha means he exer ha was bes for he recedng suaon may no onger be bes for he curren suaon or ha he degree of fness for each exer o he curren suaon changes. Consequeny he H w exerence a ranson on sae whch may be eher a ranson from one sae o anoher or a ara ranson from beng a sae n some robaby o a new vaue. Hence we can ma he sae ranson robabes from he dynamca suaon of he me seres. Here RBF srucured ae ranson Nework erforms he modeng Fgure 3. 4 Predcon he ror robaby for each sae s aken as he combnng coeffcen for each exer and herefore one-se-ahead redcon s avaabe by he foowng ses: A me wh esmaed sae ranson robabes he ror robaby for each sae s deermned as: a P s P s 5 Combne exer by he ror robabes o make redcon: yˆ P s ˆ 6 y X 9

12 he oseror robaby for each sae can be obaned by Bayes aw and w be aken as he saus of he recedng sae for nex se redcon. P s P s P s y s P s 7 y s P s 5 Exermens he HE mode has been esed usng wo chaoc me seres: Laser daa ana Fe me eres Predcon and Anayss Comeon and Leuven daa K. U. Leuven me eres Predcon Comeon. Laser daa s a ow dmenson dmenson.~. chaoc ow nose daa. Chaoc usaons more or ess foow he heoreca Lorenz mode of a wo eve sysem Hübner e a In he exermens we ake daa ons for mode ranng and foowng 5 daa ons for esng. he embeddng dmenson s 5 and he me deay s. Leuven Comeon daa s generaed from he foowng comuer generazed Chuas crcu uykens e a. 997: x& α[ x h x ] x& x x x3 x& 3 βx For he K. U. Leuven daa we ake frs daa sames as ranng daa nex 5 sames for es. Embeddng dmenson s and me deay s. For each daa se an RBF nework and one-hdden-ayer LP are used searaey as exers. he RBF nework has exonena ransfer funcon. he LP akes "og-gmod" ransfer funcon and raned by Back-roagaon agorhm. A exermens are conduced based on he "same srucure" and "same-scae" rnce. I means ha n he exermens a goba mode s comared wh a HE mode whose exers have he same nework srucure same number of hdden ayers and same number of neurons wh he goba mode. In oher words boh modes have he same number of degrees of freedom. For exame f he HE mode uses exers n RBF srucure and each of hem has hdden neurons w comare o a snge RBF nework ha has 4 hdden neurons. In he exermens wh he Laser daa HE mode been esed wh exers n RBF and LP srucure searaey. For he Leuven daa he number of exers n HE s hree for boh exer srucures. he quay of redcon n he exermens was evauaed by he RE Rooed ean quared Error and NE Normazed ean quared Error.

13 RE 8 y yˆ NE y y ŷ y 9 For he ranng rocess n Fgure 5 and Fgure 6 we gave he sae saues of boh me seres measured n robabes Hence by he end of se refned H-earnng he robaby of each on beng n each sae was deermned. Here we us gve on o for Laser daa n Fgure 5 and o 4 for Leuven daa n Fgure 6. Pon of Laser rang Daa Probaby Beongng o ae Probaby Beongng o ae Fgure 5. Probaby for each sae of Laser ranng daa from on o..5 4 Pon of Leuven ranng Daa Probaby Beongng o ae Probaby Beongng o ae Probaby Beongng o ae Fgure 6. Probaby for each sae of Leuven ranng daa from on o 4.. For he redcon erformance abe and abe show he redcon resus from he HE modes wh boh RBF and LP as exers and her corresondng goba modes. here are conssen mrovemens on he redcon quay over he corresondng snge goba modes. In erms of sae redcon Fgure 7 and Fgure 8 show he ror robabes of beng n each sae for boh daa ses. From he vew of xure of Exers hey are he gang coeffcens for each exer n he rocess of redcon. Here we show ony he robabes for he RBF srucured exers. For he behavour of each exer we dsay he redcon errors from each exer n he HE modes n Fgure 9 and Fgure. Here we us gve he redcon errors for boh me seres when HE has RBF exers.

14 RBF LP nge ode HE ode N. H. N. RE NE N. H. N RE NE /Exer / Exer / Exer / Exer abe. Predcon RE and NE on Laser daa from goba mode and HE mode wh RBF nework and LP as exer searaey. "N. H. N." eans "Number of Hdden Neurons". RBF LP nge ode HE ode N. H. N. RE NE N. H. N. RE NE / Exer / Exer / Exer / Exer.8.96 abe. Predcon RE and NE on Leuven daa from goba mode and HE mode wh RBF nework and LP as exer searaey. "N. H. N." eans "Number of Hdden Neurons". Laser Daa Probaby for ae Probaby for ae Fgure 7. Pror Probaby for each sae on Laser daa. he exers are n RBF srucure Leuven Daa Probaby for ae Probaby for ae Probaby for ae Fgure 8. Pror Probaby for each sae on Leuven daa. he exers are n RBF srucure.

15 e e e Predcon Error of Exer RBF on Laser Daa Predcon Error of Exer RBF on Laser Daa Predcon Error of xed Exer RBF on Laser Daa Fgure 9. Predcon errors of a exers and mxed exer for Laser daa e e e. e Predcon Error of Exer RBF on Leuven Daa Predcon Error of Exer RBF on Leuven Daa Predcon Error of Exer 3 RBF on Leuven Daa Predcon Error of ed Exer RBF on Leuven Daa Fgure. Predcon errors of a exers and mxed exer for Leuvan daa 6 Concuson and Dscusson he exermens show he E mode hosed by a H wh me-varan ranson roery can be aed o enhance he quay of one-se-ahead me seres redcons. Usng he same nework scae and srucure such as he number of hdden neurons and number of hdden ayers or he same number of degrees of freedom he HE mode can generae beer redcons han a goba mode for he shown daa ses. Addonay has been demonsraed ha a connecons nework can be used o mode he sae ransons aong a me seres. One queson wh he mode s he comung cos and convergence roery of ranng a seres of nu-reaed ranson robaby marxes. As he udang of he ranson robabes s smuaneousy haenng a every me on he me cos s oerabe and convergence seed s exedous. For exame by he modfed Baum-Wech agorhm he H earnng wh ons Laser daa akes abou 5 seconds n aab aform on a 3

16 Hz PC and he earnng rocess needs abou -- eraons o ge o convergence Fgure. Covergence rocess of funcon Anoher sgnfcan queson wh HE s how o choose he number of exers and herefore he number of saes. From our exerence he number has a drec mac on redcon quay and he comexy of he modes. ore oca exers aow a me seres o be modeed more recsey.e. he ranng daa can be more accuraey descrbed however arger errors aear on he saeranson modeng. hs rade-off has o be acheved manuay by ra and error. Fuure work w consder how hs rade-off can be quanfed and uned whou manua nervenon o acheve an arorae eve of generazaon. 5 5 Fgure. Convergng of Funcon for Laser daa when HE has RBF exers. References Baum L.E. Pere. oues G. and Wess N. 97. A maxmzaon echnque occurrng n he sasca anayss of robabsc funcons of arkov chans Annas of ahemaca ascs Bengo. and Frascon P An nu ouu H archecure. In Advances n Neura Informaon Processng ysems G. esauro ourezky.d.. and Leen.K. ed. Cambrdge A I Press Bengo. Frascon Paoo 996. Inu/Ouu Hs for sequence rocessng ransacons on Neura Neworks IEEE Bengo. Lauzon V. Ducharme. R.. Exermens on he acaon of IOH o mode fnanca reurn seres. IEEE ransacons on Neura Neworks 3-3. Bezdek J. 98. Paern Recognon wh Fuzzy Obecve Funcon Agorhms Penum Press. Bezdek J. and Pa Fuzzy odes for Paern Recognon IEEE Press. New ork Breman L. Fredman J. H. Oshen R. A. and one P. J Cassfcaon and Regresson rees CA Wadsworh Inernaona Grou. Cao L. 3. uor vecor machnes exers for me seres forecasng Neurocomung Demser A. Lard N. and Rubn D axmum kehood from ncomee daa va he E agorhm Journa of he Roya asca ocey eres B -38. Fredman J. 99. uvarae Adave Regresson nes Annas of ascs 9-4. Hamon J. D A new aroach o he economc anayss of nonsaonary me seres and he busness cyce Economerca Hamon J. D. 99. Anayss of me seres subec o changes n regme Journa of Economercs Hamon J. D. and usme R Auoregressve Condona Heeroskedascy and Changes n Regme Journa of Economercs Hamon J. D ecfcaon esng n arkov-swchng me-seres modes Journa of Economercs

17 Hübner U. Wess C. O. Abraham N. B. and ang D Lorenz-ke Chaos n NH 3-FIR Lasers. In me eres Predcon: Forecasng he Fuure and Undersandng he Pas A.. Wegend Gershenfed N. A. ed. A Addson-Wesey Jordan. I. and Jacobs R. A. 99. Adave mxures of oca exers Neura Comuaon Jordan. I. and Jacobs R. A. 99. Herarches of adave exers. In Advances n Neura Informaon Processng ysems J. oody Hanson. & Lmann R. ed Jordan. I. and Jacobs R. A Herarchca mxures of exers and he E agorhm Neura Comuaon Lehr. Pawezk K. Kohmorgen J. and uer K.-R Hdden arkov xures of Exers wh an Acaon o EEG Recordngs from ee heory n Boscences Lorace L. A. 98. axmum kehood esmaon for muvarae observaons of arkov ource IEEE ransacons on Informaon heory Kohmorgen J. üer K.-R. Rweger J. Pawezk K.. Idenfcaon of Nonsaonary Dynamcs n Physoogca Recordngs Boogca Cybernecs ccuagh P. and Neder J. A Generased near odes onograhs on ascs and Aed Probaby econd edn London Chaman and Ha. Rabner L. R A uora on hdden arkov modes and seeced acaons n seech recognon Proceedngs of he IEEE uykens J. A. K. Huang A. and Chua L. O A famy of n-scro aracors from a generazed Chuas crcu Inernaona Journa of Eecroncs and Communcaons ong H. and Lm K hreshod auoregresson Lm Cyces and cycca daa Journa of he Roya asca ocey B Wegend A.. angeas. and rvasava A. N Nonnear gaed exers for me seres: dscoverng regmes and avodng overfng Inernaona Journa of Neura ysems Wegend A.. and h.. Predcng Day Probaby Dsrbuons of &P5 Reurns Journa of Forecasng Wofgang H. 99. Aed Nonaramerc Regresson Cambrdge Cambrdge Unversy Press. 5

18 Aendx A Parameer udang n he modfed Baum-Weh agorhm. We defne: s γ s s β α β α β α A- We defne η as he robaby of beng n sae a me and beng n sae a me for observng he whoe sequence of observaons wh arameers. s s η s s y b a y b a β α β α A- Now reurn o funcon for axmsaon. As he arameers π and are ndeendeny s no hree erms n equaon 9 we can omse each erm ndvdua. he frs erm n funcon s: a σ s s og π og π ζ A-3 Addng he Lagrange muer δ and usng he consran ha o maxmse. π s og π δ π π A-4 We can ge he udang formua for π : s ~ π A-5 he second erm n funcon becomes: ζ s s og a s s a og 6

19 [ og a s s og a s s... og a s s ] [ ] og a s s og a s s... og a s s... [ og a s s og a s s... og a s s ] for [ og a s s og a s s... og a s s ] [ ] og a s s og a s s... og a s s... [ og a s s og a s s... og a s s ] for [ og a s s og a s s... og a s s ] [ ] og a s s og a s s... og a s s... [ og a s s og a s s... og a s s ] for A-6 In a smar way we use he Lagrange muer δ and he consran ha a hen ge: a a δ A-7 hen we can ge foowng formua: s s δ A-8 a o he udang funcon for a can be goen as foows: s s a ~ s η A-9 γ he hrd erm n funcon becomes: 3 og b s y og b y s ζ 7

20 o maxmse we ge: ~ σ s s y ŷ X [ γ y ŷ X ] A- γ Aendx B Convergence roery of he modfed Baum-Wech agorhm Convergence characer of modfed Baum-Wech agorhm s smar o ha of cassca Baum-Wech agorhm. For he ssue of esmang arameers o ge a se of observaons wh maxmum ossby he odfed Baum-Wech agorhm n each E cyce maxmses he kehood funcon and esmaes corresondng arameers n he -se. hs s a rocess of fndng crca on of funcon based on arameers vaue. For Gauss robaby densy b Larace Lorace 98 has roven ha has a unque goba maxmum as a funcon of and hs maxmum s he one and ony one crca on. Hence here s wh he equay of. Wh hs concuson we can rove ha here are: P P wh equay of see heorem. By E eraon boh funcons monoonc ncrease wh udang and converge o a oca maxmum see heorem. o by he modfed Baum-Wech agorhm he kehood funcon and he robaby funcon coud converge o oca maxmum. heorem. By each E eraon for ξ here s: wh equay of. Proof: We defne as he vaue of ha ges he maxmum of n h eraon and as he curren vaue of he arameers used n he E se. As defned n equaon 6 og ζ In he h eraon by se here s: B- For any osve scaar a here s: og a a so we can ge: 8

21 og ζ ζ [ ] ζ B- As here s: and when heorem. funcon and P funcon converge o oca maxmum. In oher words: f s a crca on of funcon s aso a crca on of. ha s: If: hen: Proof: ζ og ζ B-3 ζ o f: here s:. 9