Investigation of artificial neural network models for streamflow forecasting

Transcription

1 19th Iteratioal Cogress o Modellig ad Simulatio, Perth, Australia, 1 16 December Ivestigatio of artificial eural etwork models for streamflow forecastig H. D. Tra a,b, N. Muttil a ad B. J. C. Perera c a School of Egieerig ad Sciece, Victoria Uiversity, Victoria b Istitute for Sustaiability ad Iovatio, Victoria Uiversity, Victoria c Faculty of Health, Egieerig ad Sciece, Victoria Uiversity, Victoria dug.tra@vu.edu.au Abstract: Time series forecastig is the use of a model to forecast future evets based o kow past evets. Accurate forecasts of time series variables at differet time scales are becomig icreasigly ecessary to facilitate mitigatio of egative impacts of climate chage, maximisatio of system beefits from improved plaig ad maagemet, ad miimisatio of system failure risks i all social, ecoomical ad evirometal activities. Examples iclude hourly forecasts of raifall (1-48 hours ahead), which are useful for flood warig, ad mothly forecasts of streamflow (1-1 moths ahead), which are beeficial for plaig ad operatio of water supply systems. There are may other applicatios i fiace ad ecoomics such as tourism demad forecastig ad stock forecastig. Artificial eural etwork (ANN) models, which are cosidered as a category of the data-drive techiques, have bee widely used i streamflow forecastig. Several distiguishig features of ANN models make them valuable ad attractive for forecastig tasks. First, there are few a priori assumptios about the models as opposed to model-drive techiques. They lear from examples ad capture the fuctioal relatioships amog the data eve if the uderlyig relatioships are too complex to specify. Secod, ANN models ca geeralize after learig from the sample data preseted to them. Third, ANN models are uiversal fuctioal approximators for ay cotiuous fuctio to the desired accuracy. Fourth, ANN models have flexible structures that allow multi-iput ad multi-output modellig. This is particularly importat i streamflow forecastig where iflows at multiple locatios are cosidered withi a catchmet. This paper ivestigated two basic ANN models, amely, feed forward eural etwork (FFNN) ad layered recurret eural etwork (LRNN) for streamflow forecastig i a attempt to uderstad why ANN models were used successfully i some streamflow forecastig studies but ot always. I our study, two hypothetical ad two real datasets were used to test performace of two differet ANN models usig feed forward ad layered recurret structures. Furthermore, a existig iput selectio techique usig partial mutual iformatio (PMI) approach, which ca remove the isigificat iputs ad thus potetially ehace the performace of ANN models, is also ivestigated. The results showed that the PMI approach correctly idetified sigificat iputs of the two hypothetical datasets. However, the forecastig performace of FFNN ad LRNN were ot ehaced, whe PMI idetified iputs were used i compariso to usig all iputs. The LRNN did ot outperform the FFNN, although it is expected to perform better. Performace of both FFNN ad LRNN models are related to oise level ad autoregressive feature of time series data. Keywords: artificial eural etworks, streamflow forecastig, iput selectio 1099

2 Tra et al., Ivestigatio of artificial eural etwork models for streamflow forecastig 1. INTRODUCTION Forecastig streamflow is a importat task sice it ca help i short term operatio of water supply systems as well as providig early warig of river floodig. The geeral difficulty associated with streamflow forecastig is the o-liear ad o-statioary characteristics which are ofte ecoutered i most streamflow time series data (Coulibaly et al. 001). Artificial eural etworks (ANN) models, which are cosidered as a category of the data-drive techiques, have bee widely used i streamflow forecastig. Several distiguishig features of ANN models make them valuable ad attractive for forecastig tasks (Maier ad Dady 000; Samarasighe 006). First, there are few a priori assumptios about the models as opposed to model-drive techiques. They lear from examples ad capture the fuctioal relatioships amog the data eve if the uderlyig relatioships are too complex to specify. Secod, ANN models ca geeralize after learig from the sample data preseted to them. Third, ANN models are uiversal fuctioal approximators for ay cotiuous fuctio to the desired accuracy. Fourth, ANN models have flexible structures that allow multi-iput ad multi-output modellig. This is particularly importat i streamflow forecastig where iflows at multiple locatios are cosidered withi a catchmet. There are may differet types of ANN architectures ad amog them, feed forward ad recurret eural etworks have recetly gaied attetio i literature. Feed forward etworks are static, ad are the most commo form applied i hydrology due to its simple framework (Güldal ad Togal 010). The static etworks ca oly simulate the short-term memory structures withi processes. I cotrary, the recurret eural etworks provide a represetatio of dyamic iteral feedback loops to store iformatio for later use ad to ehace the efficiecy of learig. This paper ivestigated two basic ANN models usig feed forward ad recurret structures for streamflow forecastig i a attempt to uderstad why ANN models were used successfully i some streamflow forecastig studies but ot always. This icosistecy was observed i a recetly published study by Wag et al. (009) ivolvig similar ANN models for forecastig streamflow time series at two differet locatios, oe demostratig a high E (Nash-Sutcliffe efficiecy coefficiet) of 0.87, while the other showig a medium E of 0.61, betwee observed ad forecasted values o validatio datasets. I our study, two hypothetical ad two real datasets were be used to test performace of these ANN models. The modellig difficulty of all four datasets was assessed by comparig with the well-kow multiple liear regressio (MLR) model. Furthermore, a iput selectio techique usig partial mutual iformatio approach (May et al. 008), which ca remove the isigificat iputs ad thus potetially ehace the performace of ANN models, was also ivestigated.. DATASETS Four time series datasets were used i this study to ivestigate the performace of the two ANN models. The first two datasets were hypothetical time series data ad the last two datasets were real streamflow time series data. The descriptio of 4 datasets ad their time-series plots are give below. (1) AR9 (hypothetical data) Figure 1a This dataset is a widely used hypothetical dataset with autoregressive feature (May et al. 008) i which Y t depeds o its three time-lags ad a radom oise: Yt = 0.3Y t 1 0.6Yt 4 0.5Yt 9 + et[ N[0,1 ]] (1) () YX (hypothetical data) Figure 1b This dataset does ot have the autoregressive feature sice Y t depeds o a idepedet time series X t, whose values follow ormal distributio, ad a radom oise (Muñoz ad Czerichow 1998): Yt = 0.5X t 1 0.6X t 4 0.X t X t X t 6 + et[ N[0,1/ 0] ] () X t = N [0,1] (3) (3) US dataset Figure a 1100

3 Tra et al., Ivestigatio of artificial eural etwork models for streamflow forecastig This dataset is based o a observed time series of daily discharge from the gauge i U.S. from 006 to 008 (Guve 009). (4) Eildo dataset Figure b This dataset is based o a mothly time series of Eildo river flow i the North-Wester regio of Victoria (Australia), from Jauary, 1891 to Jue, 007. Figure 1 Time series plots of hypothetical AR9 ad YX datasets Figure Time series plots of real daily flow (m 3 /s) for USdata dataset ad mothly total flow (m 3 ) for Eildo dataset 3. PARTIAL MUTUAL INFORMATION (PMI) Mutual iformatio (MI) measures the depedece betwee two variables X ad Y by usig their joit distributios ad margial desities (Sharma 000). I other words, the MI measures the reductio i ucertaity of Y due to the kowledge of the variable X. Based o the MI approach, the partial mutual iformatio (PMI) techique was developed to better reveal the true relatioship betwee X ad Y by removig the iteractive relatioships with oe or more other variables (May et al. 008): ' f ' ' ( x, y) x = x E[ x z] (5) ' ' X, Y ' ' PMI = f ' ' ( x, y )l dx dy X, Y ' (4) f ' ( x) fy ( y) X ' y = y E[ y z] (6) where X ad Y are geeralized to represet Y t ad lagged time Y t-h with time step h coditioal o Z which is a set of remaiig time-lag variables. I performig the PMI techique, the iput variable that has the highest coditioal PMI value at each iteratio is added to the selectio set. Three statistical tests icludig bootstrap, Akaike iformatio criteria (AIC) ad Hampel distace detailed i May et al. (008) were also employed i this study to elimiate the occurrece by chace of the depedecy betwee iput variables ad the output variable idetified by the PMI techique. The bootstrap is used to test the quality of statistical estimates based o a sample of data. The AIC is based o pealizig model complexity ad the Hampel test is based o outlier detectio, which determies whether a give value, x, is sigificatly differet to other values. 4. NEURAL NETWORK MODELS May types of artificial eural etworks have bee developed i the last few decades startig from the well kow multilayer perceptro structure (Samarasighe 006). Each perceptro is a sigal processig uit 1101

4 Tra et al., Ivestigatio of artificial eural etwork models for streamflow forecastig ad a etwork of coected perceptros is capable of performig classifyig ad forecastig tasks. The classifyig ad forecastig tasks are doe through a -step process of traiig (i.e. learig from sample data) ad simulatig (i.e. geeralizig to classify or forecast). Multilayer perceptro etworks are further classified ito static feed-forward eural etworks (FFNN) ad dyamic recurret eural etworks (RNN) with regards to how sigals are trasferred betwee layers of perceptros. I the static FFNN, sigals essetially propagate i the forward directio, while a backward propagatio of so-called feedback sigals ca occur i the RNN (Güldal ad Togal 010). The static FFNN with a typical layout as show i Figure 3a, ca achieve satisfactory aalytical outcomes if sufficiet data are available. Nevertheless, i cosideratio of the temporal or short-term dyamic property, the static eural etwork has difficulty i recogizig ad predictig the reality. The FFNN is ivestigated i this study. The defiitio of a RNN, so-called dyamic eural etwork, is that at least oe feedback lik is added to the static eural etwork as show i Figure 3b. The RNN allows sigals to propagate i both forward ad backward directios, which offers the etwork dyamic memories. I other words, RNN has additioal feed back loops either from the output or hidde layer to the iput layer that delay ad store iformatio from the previous time step that is ot preset i the architecture feed forward etworks. The presece of these feed back loops has a profoud impact o the learig capability ad performace of the eural etwork, which allows the etwork to capture the true hidde dyamic memories or persistece compoets of oliear time-series systems (Carcao et al. 008). The RNN has bee proved to be a powerful method for hadlig complex systems such as oliear time-varyig systems. A layered recurret eural etwork (LRNN), which is a form of RNN, is also ivestigated ad compared with the FFNN i this study. Figure 3 A typical layout of feed-forward eural etworks (FFNN) (adapted from Carcao et al. 008) ad recurret eural etworks (RNN) (adapted from Güldal ad Togal 010) 5. PERFORMANCE INDICATORS Model performace i this study was evaluated usig three performace idicators, the mea square error (MSE), Nash-Sutcliffe coefficiet (E) ad the mea absolute percetage error (MAPE), as show i Equatios 7-9 (Carcao et al. 008). The MSE quatifies the differeces betwee observed ad predicted values alog the time series axis ad pealises for the large differeces because of its square power. This allows the ANN models to better capture peak values of time series data. The MSE is commoly used as the traiig objective to be miimized i ANN models. E captures the goodess of fit of the model of iterest by comparig it with a aïve model i which the mea value is used as predicted values. The value of E lies betwee oe ad mius ifiity. A value of uity implies that the model exactly matches observatios, zero implies that the model is o better tha assumig the mea value, ad a egative value idicates that the mea values of the observed time series would be a better predictor tha the model. However, the limitatios of MSE ad E are that they are uable to provide the practical assessmet of differece betwee predicted ad observed values. For example, MSE provides similar value of 4 for the differece betwee observatio of 1000 ad predictio of 100 as well as betwee observatio of 1 ad predictio of 3 i which it is obviously that the former predictio is very good while the latter predictio is very poor. The MAPE was therefore used i this study to provide scale-free percetage differece betwee observed ad predicted values (Coulibaly et al. 001). 110

5 Tra et al., Ivestigatio of artificial eural etwork models for streamflow forecastig MSE = 1 1 MAPE = ( Q obsevred Q Q predicted Q ) obsevred predicted 1 Qobserved (7) E ( Qobsevred Qpredicted ) 1 (8) ( Q Q ) 1 = 1 observed (9) where is umber of data poit. observed 6. RESULTS The PMI tool was obtaied from May et al. (008), ad was used to select model iputs from potetial 15 iputs obtaied through time lagged observatios for four datasets. That is, 15 past values of observatios (i.e. streamflow) were used as the model iputs to forecast curret output as oe-step ahead forecastig. The feed-forward eural etwork (FFNN), layered recurret eural etwork (LRNN) ad multiple liear regressio (MLR) models, which are available i the Matlab software, were used i this study. Both eural etwork models used oly oe hidde layer for ease of compariso. All four datasets were divided ito traiig (60%), validatio (0%) ad testig (0%) datasets. The testig dataset is the last part of the time series ad was kept i time series order. The remaiig 80% of dataset was the radomly divided ito traiig ad validatio datasets to have better chace of represetative learig data for FFNN ad LRNN models. The MLR model used the whole 80% dataset without splittig ito a validatio dataset. The traiig ad validatio datasets were used i the traiig process i FFNN ad LRNN models, i which the traiig dataset was used to provide learig kowledge, while the validatio dataset was used to avoid over-fittig. The testig dataset was used oly i assessig the forecastig performace of MLR, FFNN ad LRNN models. All four datasets were scaled betwee [-1, 1] ad the used i the traiig ad testig. The traiig process for FFNN ad LRNN models was carried out usig Leveberg-Marquardt algorithm. Least square method was used for the calibratio of the MLR model. The sigle objective of miimizig mea square error (MSE) was used i the traiig process of all models. The remaiig two performace idicators (i.e. E ad MAPE) were the computed usig the traiig (i.e. combiatio of traiig ad validatio) ad testig outcomes. The suitable umber of hidde euros for FFNN ad LRNN models was selected usig trial ad error. 6.1 Iput selectio The results of iput selectio usig the PMI method are show i Table 1. As ca be see, the PMI method correctly idetified three lagged iputs of two hypothetical datasets (i.e. AR9 ad YX). For the two real datasets, the selected iputs are show but could ot be assessed sice the true iputs are ukow. Table 1 Result of iput selectio usig PMI techique Dataset True Iputs PMI_selected iputs AR9 Y t-1, Y t-4, Y t-9 Y t-1, Y t-4, Y t-9 YX X t-1, X t-4, X t-6 X t-1, X t-4, X t-6 USdata ukow Q t-1 Eildo ukow Q t-1, Q t-3, Q t Forecastig performace As stated i Sectio 1, the MLR model was used to assess how difficult it was to model the four datasets usig ANN models. The results of the MLR model are show i the last colum of Tables ad 3 i which the MLR was ru usig all 15 past values as model iputs to predict the curret values. The E values computed from the MLR models are withi the rage ( ) for all datasets, which idicates some modellig difficulty. 1103

6 Tra et al., Ivestigatio of artificial eural etwork models for streamflow forecastig The forecastig performaces of FFNN ad LRNN models are show i Table for traiig dataset ad Table 3 for testig dataset. It should be oted that the MLR model was ru oly for the case of all 15 iputs for all datasets for comparative purposes. I compariso with the E of MLR model o four datasets, both FFNN ad LRNN slightly outperformed the MLR mode with the exceptio of US dataset. For AR9 dataset, there is o sigificat differece i model performace as demostrated through 3 performace idicators for both traiig ad testig datasets (all iputs or PMI-selected iputs) i FFNN ad LRNN models. For remaiig datasets, the performace of both FFNN ad LRNN models were also ot ehaced whe PMI-selected iputs was used. Although the LRNN model is theoretically better tha the FFNN model i hadlig dyamic time series data, the performace of the LRNN i compariso with the FFNN model as evideced from this study is ot up to its expectatio. Table Traiig performaces of FFNN ad LRNN Traiig FFNN LRNN MLR dataset Hidde Hidde euros euros E AR AR9_pmi YX YX_pmi USdata USdata_pmi Eildo Eildo_pmi Table 3 Testig performaces of FFNN ad LRNN Traiig FFNN LRNN MLR dataset Hidde Hidde euros euros E AR AR9_pmi YX YX_pmi USdata USdata_pmi Eildo Eildo_pmi Both FFNN ad LRNN models produced good forecastig performace for the hypothetical YX dataset ad reduced performace with the hypothetical AR9 dataset. By examiig the data geeratig mechaisms of AR9 ad YX datasets, it is suggested that the reaso could be related to the autoregressive feature of the AR9 data as well as the oise level (typical oise level i AR9 dataset ad small oise level i YX dataset). Both real datasets also showed the autoregressive feature through the sigificat relatio betwee curret ad past values (Table 1) ad the performace of FFNN ad LRNN models were also ot high o these two real datasets. Further similar testig of more complex or more challegig autoregressive datasets are therefore required to clarify the oise level ad the relatioships betwee autoregressive feature ad the performace of ANN models. Scatter plots betwee observed ad predicted values for Eildo dataset by FFNN ad LRNN models are show i Figure 4. These plots idicate that both FFNN ad LRNN models uder-predict high flow values. This is aother limitatio of ANN models that also require future attetio. Techiques such as wavelet ad sigular spectrum aalysis described recetly i literature could be used to decompose autoregressive time 1104

7 Tra et al., Ivestigatio of artificial eural etwork models for streamflow forecastig series ito time series compoets that cotai importat features such as tred ad oise of the autoregressive data. These time series compoets ca be used as model iputs to ANN modellig. These techiques will be sought i the future. (c) (d) Figure 4 Scatter plots of observed ad predicted flow (mothly m 3 ) for Eildo dataset usig 15 lagged iputs (a ad c) ad PMI-selected iputs (b ad d) with FFNN (a ad b) ad LRNN (c ad d) models 7. CONCLUSIONS This study ivestigated feed forward eural etwork (FFNN) ad layered recurret eural etwork (LRNN) models ad a o-liear iput selectio techique usig partial mutual iformatio (PMI) approach for modelig ad forecastig dyamic time series data with oise. Two hypothetical datasets with ad without autoregressive feature were used together with two real datasets of daily ad mothly streamflow. The results showed that the PMI approach correctly idetified sigificat iputs of the two hypothetical datasets. However, the forecastig performace of FFNN ad LRNN were ot ehaced whe PMI idetified iputs were used i compariso to usig all iputs. The LRNN did ot outperform the FFNN, although it is expected to perform better. Iitial fidigs idicated that performace of both FFNN ad LRNN are related to oise level ad auto-regressive feature of time series data. Future works will focus o further similar testig of more complex or more challegig autoregressive datasets. REFERENCES Carcao, E C, Bartolii, P, Muselli, M & Piroddi, L 008, 'Jorda Recurret Neural Network versus IHACRES i Modellig Daily Streamflows,' Joural of Hydrology, 36, Coulibaly, P, Actil, F & Bobee, B 001, 'Multivariate Reservoir Iflow Forecastig Usig Temporal Neural Networks,' Joural of Hydrologic Egieerig, 6(5), Güldal, V & Togal, H 010, 'Compariso of Recurret Neural Network, Adaptive Neuro-Fuzzy Iferece System ad Stochastic Models i Egirdir Lake Level Forecastig,' Water Resources Maagemet, 4, Guve, A 009, 'Liear Geetic Programmig For Time-Series Modellig of Daily Flow Rate,' Joural of Earth Systems ad Sciece, 118(), Krauser, P & Boyle, D P 005, 'Compariso of Differet Efficiecy Criteria for Hydrological Model Assessmet,' Advace GeoSciece, 5, Maier, H R & Dady, G C 000, 'Neural Networks for The Pedictio Ad Forecastig of Water Resources Variables: A Review of Modellig Issues ad Applicatios,' Evirometal Modellig ad Software, 15(1), May, R J, Maier, H R, Dady, G C & Ferado, T M K G 008, 'No-Liear Variable Selectio For Artificial Neural Networks Usig Partial Mutual Iformatio,' Evirometal Modellig & Software, 3(10-11), Muñoz, A & Czerichow, T 1998, 'Variable Selectio Usig Feedforward Ad Recurret Neural Networks,' Egieerig Itelliget Systems for Electrical Egieerig ad Commuicatios, 6(), Samarasighe, S 006, Neural Networks for Applied Scieces ad Egieerig, Auerbach Publicatios, New York. Sharma, A 000, 'Seasoal to Iteraual Raifall Probabilistic Forecasts For Improved Water Supply Maagemet: Part 1-A Strategy For System Predictor Idetificatio,' Joural of Hydrology, 39(1-4), Wag, W-C, Chau, K-W, Cheg, C-T & Qiu, L 009, 'A Compariso of Performace of Several Artificial Itelligece Methods for Forecastig Mothly Discharge Time Series,' Joural of Hydrology, 374(3),