Deep Belief Network using Reinforcement Learning and its Applications to Time Series Forecasting

Transcription

1 Deep Belef Network usng Renforcement Learnng and ts Applcatons to Tme Seres Forecastng Takaom HIRATA, Takash KUREMOTO, Masanao OBAYASHI, Shngo MABU Graduate School of Scence and Engneerng Yamaguch Unversty Tokwada -16-1, Ube, Yamaguch JAPAN {v003we, wu, m.obayas, Kunkazu KOBAYASHI School of Informaton Scence and Technology Ach Prefectural Unversty 15-3 Ibaragabasama, Nagakute, Ach JAPAN Abstract. Artfcal neural networks (ANNs) typfed by deep learnng (DL) s one of the artfcal ntellgence technology whch s attractng the most attenton of researchers recently. However, the learnng algorthm used n DL s usually wth the famous error-backpropagaton (BP) method. In ths paper, we adopt a renforcement learnng (RL) algorthm Stochastc Gradent Ascent (SGA) proposed by Kmura & Kobayash nto a Deep Belef Nets (DBN) wth multple restrcted Boltzmann machnes nstead of BP learnng method. A longterm predcton experment, whch used a benchmark of tme seres forecastng competton, was performed to verfy the effectveness of the novel DL method. Keywords: deep learnng, restrcted Boltzmann machne, stochastc gradent ascent, renforcement learnng, error-backpropagaton 1 Introducton A tme seres s a data strng to be observed n a temporal change n a certan phenomenon. For example, foregn currency exchange rate, stock prces, the amount of ranfall, the change of sunspot, etc. There s a long hstory of tme seres analyss and forecastng [1], however, the predcton study of tme-seres data n the real world s stll on the way because of the nonlnearty and the nose affecton. Artfcal neural networks (ANNs) have been wdely used n pattern recognton, functon approxmaton, nonlnear control, tme seres forecastng, and so on, snce 1940s []-[11]. After a learnng algorthm named error-backpropagaton (BP) was proposed by Rumelhart, Hnton & Wllam for mult-layer perceptrons (MLPs) [], adfa, p. 1, 011. Sprnger-Verlag Berln Hedelberg 011

2 ANNs had ts heyday from 1980s to 1990s. As a successful applcaton, ANNs are utlzed as tme seres predctors today [3]-[11]. Especally, a deep belef net (DBN) composed by restrcted Boltzmann machnes (RBMs) [6] and Mult-Layer Perceptron (MLP) [] was proposed n [8]-[10] recently. Deep learnng (DL), whch s a knd of novel ANNs tranng method, s attractng the most attenton of artfcal ntellgent (AI) researchers. By a layer-by-layer tranng algorthm and stack structures, bg data n the hgh dmensonal space s able to be classfed, recognzed, or sparsely modeled by DL [6]. However, although there are varous knds of deep networks such as auto-encoder, deep Boltzmann machne, convolutonal neural network (CNN), etc., the learnng algorthm used n DL s usually wth the famous BP method. In ths paper, a renforcement learnng method named stochastc gradent ascent (SGA) proposed by Kmura & Kobayash [13] s ntroduced to the DBN wth RBMs as the fne-tunng method nstead of the conventonal BP learnng method. The error between the output of the DBN and the sample s used as reward/punshment to modfy the weghts of connectons of unts between dfferent layers. Usng a benchmark named CATS data used n the predcton competton [4] [5], the effectveness of the proposed deep learnng method was confrmed by the tme seres forecastng results. DBN wth BP Learnng (The Conventonal Method) In [8], Kuremoto et al. frstly appled Hnton & Slakhutdnov s deep belef net (DBN) wth restrcted Boltzmann machnes (RBMs) to the feld of tme seres forecastng. In [9] and [10], Kuremoto, Hrata, et al. constructed a DBN wth RBMs and a mult-layer perceptron (MLP) to mproved the prevous tme seres predctor wth RBMs only. In ths secton, these conventonal methods are ntroduced..1 DBN wth RBM and MLP Restrcted Boltzmann machne (RBM) RBM s a Boltzmann machne consstng of two layers of the vsble layer and the hdden layer, no connectons between neurons on the same layer. It s possble to extract features to compress the hgh-dmensonal data n low-dmensonal data by performng an unsupervsed learnng algorthm. Each unt of the vsble layer has a symmetrc connecton weghts wth respect to the unt each of the hdden layer. Couplng between the unts are b-drecton. The value of the connecton weghts between unts s the same n both drectons. Unt of the vsble layer has a bas b, the hdden layer b. All neurons n the vsble layer and the hdden layer stochastcally output 1 or 0, accordng to ts probablty wth sgmod functon (Eq. (1) and Eq. ()). 1 p( h 1 v) (1) 1 exp( b w v )

3 1 p( v 1 h) () 1 exp( b w h ) Usng a learnng rule whch modfes the weghts of connectons, RBM network can reach a convergent state by observng ts energy functon: E( v, h) b v b h w v h (3) Detals of the learnng algorthm of RBM can be found n [8]., DBN wth RBM and MLP DBN s a mult-layer neural network whch usually composes multple RBMs [6] [8]. DBN can extract the feature of features of hgh-dmensonal data, so t s also called deep learnng, and appled to many felds such as dmensonalty reducton, mage compresson, pattern recognton, tme seres forecastng, and so on. A DBN predcton system s proposed recently by composng of plural RBMs n [8], and another DBN predcton system usng RBM and MLP s proposed n [9] (see Fg. 1). However, all of these DBNs used the learnng algorthm proposed n [6],.e. RBM s unsupervsed learnng and BP learnng, supervsed learnng as fne-tunng conventonally. Fg. 1. A structure of DBN composed by RBM and MLP 3 DBN wth SGA (The Proposed Method) BP s a powerful learnng method for the feed-forward neural network, however, t modfes the model strctly accordng to the teacher sgnal. So the robustness of the ANNs bult by BP s restrcted. Meanwhle, noses usually exst n the real tme-seres data. Therefore, we consder that predcton performance may be mproved by usng a renforcement learnng (RL) algorthm to modfy the ANN predctors accordng to the rewards/punshment correspondng to a good/bad output. That s, f the absolute error between the output and the sample s small enough (e.g. usng a threshold), then t s consdered as a good output, and a postve reward s used to modfy parameters of ANNs. In [7], MLP and a self-organzed fuzzy neural network (SOFNN) wth a RL algorthm called stochastc gradent ascent (SGA) [13] were shown ther prorty to the conventonal BP learnng algorthm. Here, we ntend to nvestgate the case when SGA s used n DBN nstead of BP.

4 Fg..The structure of MLP desgned for SGA 3.1 The structure of ANNs wth SGA In Fg., a MLP type ANN s desgned for SGA [7]. The man dfference to the conventonal MLP s that the output of network s gven by two unts whch are the parameters of a stochastc polcy whch s a Gaussan dstrbuton functon.sga Learnng algorthm for ANNs The SGA algorthm s gven as follows [7] [13]. 1. Observe an nput x( on tme t.. Predct a future data y ( as yˆ ( accordng to a probablty 1 ( yˆ( ) ( y ˆ( W, x( ) exp( ) wth ANN models whch are constructed by parameters W,, w, w, v }. { 3. Receve a scalar reward/punshment r t by calculatng the predcton error. 1 r t 1 f ( y( ) ( else (4) Where ( MSE (, MSE ( t ( y( ), β s a postve constant. t 4. Calculate characterstc elgblty e ( and elgblty trace D (. (W) (5) D ( e( D ( t 1) (6) Where 0 1 s a dscount factor, w denotes th element of the nternal varable vector W. Calculate ( : w ( ( r b) D ( (7) w t

5 Where b denotes the renforcement baselne. 5. Improve the polcy ( yˆ( x(, W ) by renewng ts nternal varable W. W W W (8) Where α s the learnng rate. 6. For the next tme step t 1, f the predcton error MSE( s converged enough then end the learnng process, else, return to step 1. Fg. 3. Tme seres data of CATS 4 Predcton Experments and Results 4.1 CATS benchmark tme seres data CATS tme seres data s the artfcal benchmark data for forecastng competton wth ANN methods [4] [5].Ths artfcal tme seres s gven wth 5,000 data, among whch 100 are mssed (hdden by competton the organzers). The mssed data exst n 5 blocks:elements 981 to 1,000;elements 1,981 to,000;elements,981 to 3,000; elements 3,981 to 4,000; elements 4,981 to 5,000. The mean square error E 1 s used as the predcton precson n the competton, and t s computed by the 100 mssng data and ther predcted values as followng: E 1 { ( y( ) ( y( ) ( y( ) ( y( ) 100 t981 t1981 t981 t3981 where y ˆ( s the long term predcton result of the mssed data. t4981 ( y( ) } (9)

6 4. Optmzaton of meta parameters The number of RBM that consttute the DBN and the number of neurons of each layer affect predcton performance serously. In ths paper, these meta parameters are optmzed usng random search method [1]. In the optmzaton of the ANN structure, random search s known to exhbt hgher performance than the grd search. The meta parameters of DBN and ther exploraton lmts are shown as followng: the number of RBMs: 0-3; the number of neurons n each layers: -0; learnng rate of each RBM: ; learnng rate of SGA : ; dscount factor: ; constant β n Eq. (8) Fg. 4. The predcton results of dfferent methods for CATS (Block 1). Table 1. The comparson of performance between dfferent methods usng CATS data Method E 1 DBN(SGA) (proposed) 170 DBN(BP)+ARIMA (10) 44 DBN (9) (BP) 57 Kalman Smoother (The best of IJCNN '04) (5) 408 DBN (8) ( RBMs) 115 MLP (8) 145 A herarchcal Bayesan Learnng Scheme for Autoregressve Neural 147 Networks (The worst of IJCNN '04) (4) 4.3 Experments Result The predcton results of the frst blocks of CATS data are shown n Fg. 4. Comparng to the conventonal learnng method of DBN,.e., usng Hnton s RBM unsu-

7 pervsed learnng method [6] [8] and back-propagaton (BP), the proposed method whch used the renforcement learnng method SGA nstead of BP showed ts superorty accordng to the measure of the average predcton precson E 1. Addtonally, the result by the proposed method acheved the top of rank of all prevous studes such as MLP wth BP, the best predcton of CATS competton IJCNN 04 [5], the conventonal DBNs wth BP [8] [9], and hybrd models [10] [11]. The detals are shown n Table 1.The optmal parameters obtaned by random search method are shown n Table. Table. Parameters of DBN used for the CATS data (Block 1) DBN wth SGA (proposed) DBN wth BP [9] The number of RBMs 3 1 Learnng rate of RBM Structure of DBN (the number of neurons n each layer) Learnng rate of SGA or BP Dscount factor Coeffcent Concluson In ths paper, we proposed to use a renforcement learnng method stochastc gradent ascent (SGA) to realze fne-tunng of a deep belef net (DBN) composed by multple restrcted Boltzmann machnes (RBMs) and mult-layer perceptron (MLP). Dfferent from the conventonal fne-tunng method usng the error backpropagaton (BP), the proposed method used a rough udgment of renforcement learnng concept,.e., good output owns postve rewards, and bad one wth negatve reward values, to modfy the network. Ths makes the avalable of a sparse model buldng, avodng to the over-fttng problem whch occurs by the lost functon wth the teacher sgnal. Tme seres predcton was appled to verfy the effectveness of the proposed method, and comparng to the conventonal methods, the DBN wth SGA showed the hghest predcton precson n the case of CATS benchmark data. The future work of ths study s to nvestgate the effectveness to the real tme seres data and other nonlnear systems such as chaotc tme seres data. Acknowledgment: Ths work was supported by JSPS KAKENHI Grant No and No

8 References: 1. G. E. P. Box, D. A. Perce, Dstrbuton of Resdual Autocorrelatons n Autoregressve- Integrated Movng Average Tme Seres Models, Journal of the Amercan Statstcal Assocaton, Vol. 65, No. 33, Dec., 1970, pp D. E., Rumelhart, G. E., Hnton, G. E. & R. J. Wllams, Learnng representaton by backpropagatng errors, Nature, Vol. 3, No. 9, 1986, pp M. Casdagl, Nonlnear predcton of chaotc tme seres, Physca D, Vol.35, 1981, pp A. Lendasse, E. Oa, O. Smula, M. Verleysen, Tme Seres Predcton Competton: The CATS Benchmark, Proceedngs of Internatonal Jont Conference on Neural Networks (IJCNN'04), 004, pp A. Lendasse, E. Oa, O. Smula, M. Verleysen, Tme Seres Predcton Competton: The CATS Benchmark, Neurocomputng, Vol. 70, 007, pp G.E. Hnton, R.R. Salakhutdnov, Reducng the dmensonalty of data wth neural networks, Scence, Vol.313, 006, pp T. Kuremoto, M. Obayash, M. Kobayash, Neural Forecastng Systems. In Renforcement Learnng, Theory and Applcatons (ed. C. Weber, M. Elshaw and N. M. Mayer), Chapter 1, pp. 1-0, 008, INTECH. 8. T. Kuremoto, S. Kmura, K. Kobayash, M. Obayash, Tme seres forecastng usng a deep belef network wth restrcted Boltzmann machnes, Neurocomputng, Vol.137, No.5, Aug. 014, pp T. Kuremoto, T. Hrata, M. Obayash, S. Mabu, K. Kobayash, Forecast Chaotc Tme Seres Data by DBNs, Proceedngs of the 7th Internatonal Congress on Image and Sgnal Processng (CISP 014), Oct. 014, pp T. Hrata, T. Kuremoto, M. Obayash, S. Mabu, Tme Seres Predcton Usng DBN and ARIMA, Internatonal Conference on Computer Applcaton Technologes (CCATS 015), Matsue, Japan, Sep. 015,pp G.P. Zhang, Tme seres forecastng usng a hybrd ARIMA and neural network model, Neurocomputng, Vol.50, 003, pp J. Bergstra, Y. Bengo, Random Search for Hyper-Parameter Optmzaton, Journal of Machne Learnng Research, Vol. 13, 01, pp H. Kmura, S. Kobayash, Renforcement learnng for contnuous acton usng stochastc gradent ascent, Proceedngs of 5 th Intellgent Autonomous Systems, 1998, pp