FORECASTING CHINA S FOREIGN TRADE VOLUME WITH A KERNEL-BASED HYBRID ECONOMETRIC-AI ENSEMBLE LEARNING APPROACH

Transcription

1 Jrl Syst Sci & Complexity (2008) 21: 1 19 FORECASTING CHINA S FOREIGN TRADE VOLUME WITH A KERNEL-BASED HYBRID ECONOMETRIC-AI ENSEMBLE LEARNING APPROACH Lean YU Shouyang WANG Kin Keung LAI Received: 20 August 2007 / Revised: 7 September 2007 c 2008 Springer Science + Business Media, LLC Abstract Due to the complexity of economic system and the interactive effects between all kinds of economic variables and foreign trade, it is not easy to predict foreign trade volume. However, the difficulty in predicting foreign trade volume is usually attributed to the limitation of many conventional forecasting models. To improve the prediction performance, the study proposes a novel kernel-based ensemble learning approach hybridizing econometric models and artificial intelligence (AI) models to predict China s foreign trade volume. In the proposed approach, an important econometric model, the co-integration-based error correction vector auto-regression (EC-VAR) model is first used to capture the impacts of all kinds of economic variables on Chinese foreign trade from a multivariate linear analysis perspective. Then an artificial neural network (ANN) based EC-VAR model is used to capture the nonlinear effects of economic variables on foreign trade from the nonlinear viewpoint. Subsequently, for incorporating the effects of irregular events on foreign trade, the text mining and expert s judgmental adjustments are also integrated into the nonlinear ANN-based EC-VAR model. Finally, all kinds of economic variables, the outputs of linear and nonlinear EC-VAR models and judgmental adjustment model are used as input variables of a typical kernel-based support vector regression (SVR) for ensemble prediction purpose. For illustration, the proposed kernel-based ensemble learning methodology hybridizing econometric techniques and AI methods is applied to China s foreign trade volume prediction problem. Experimental results reveal that the hybrid econometric-ai ensemble learning approach can significantly improve the prediction performance over other linear and nonlinear models listed in this study. Key words Artificial neural networks, error-correction vector auto-regression, foreign trade prediction, hybrid ensemble learning, kernel-based method, support vector regression. Lean YU Shouyang WANG Institute of Systems Science, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing , China. yulean@amss.ac.cn; sywang@amss.ac.cn. Kin Keung LAI Department of Management Sciences, City University of Hong Kong, Tat Chee Avenue, Kowloon, Hong Kong, China. mskklai@cityu.edu.hk. This research is supported by the National Natural Science Foundation of China under Grant Nos and , the Knowledge Innovation Program of the Chinese Academy of Sciences under Grant Nos , , and , and the Strategy Research Grant of City University of Hong Kong under Grant No

2 2 LEAN YU SHOUYANG WANG KIN KEUNG LAI 1 Introduction Since the policy of reform and openness was initiated in 1978, China s economy has developed very fast [1]. From 1978 to 2006, the average GDP growth rate arrives at 9.6% according to the national statistical data [2]. Currently, China has been one of the fastest developing nations around the world. If it keeps growing at an average of 4% to 6% annually, China will overtake Japan as Asia s largest economy by 2040 [3]. In China s economy, the most dynamic and important part is the external sector. In 1978, the foreign trade volume of China was about 19.8 billion US dollar, but in 2006, it reached about 1.76 trillion US dollar, increasing about 89 times in 28 years, according to official figures [4]. The growth of China s foreign trade is much higher than the growth of GDP, which is already an amazing speed in the world. Up to 2004, China has emerged as the world s third largest trade country and might become the second largest trade country at the end of 2007 [5]. Currently China is regarded as the world factory in many people s eyes around the world. The trademark Made in China appears almost every corner in the world. China s foreign trade has a strong influence on the global economy. Domestically, China s foreign trade is the most important sector in the national economy. In 2006, the openness of economy (the ratio of foreign trade in GDP) exceeds 65 percent. About one-fourth of China s industrial labors are employed by the foreign trade related industries. Moreover, as a developing country, the rapid development of foreign trade helps China to bring in foreign advanced technologies and its badly needed capitals. Also, it helps China to gain the advanced management skills and experience, and even to reform its legal system. The foreign trade not only plays an important role in China s economic growth and employment, but also the continuous stability and prosperity of China s society. Thus China s government pays much attention to foreign trade. As a transition economy, China s government has been playing a key role in its economic development. For accelerating the development of foreign trade, the government not only has issued a series of favorable policies, but also shaped various social and economic policies based on the performance of the foreign trade sector. One important policy, for example, is export rebate tax. Every March of the year, the highest legislation authority of China, the National People s Congress (NPC), will hold its annual grand session. In this session, the delegates will audit the fiscal plan of the central government. The export rebate tax was one of important components in the fiscal plan. To make a feasible fiscal plan, it is necessary to make an accurate prediction for foreign trade volume. Therefore, accurate foreign trade volume forecasting has some important implications for Chinese macroeconomic policymaking. A typical example is that the national fiscal program about export rebate tax is easy to be established and to be implemented if export trade forecasting is accurate. On the contrary, inaccurate foreign trade volume forecasting will lead to some unexpected results. For example, if forecasting is larger than the actual exports, some financial fund will be otiose and cannot exert its function. If forecasting is smaller than the actual exports, the financial subsidies cannot cover the rebate tax and thus affecting the enthusiasm of foreign trade enterprises [1]. Another example is the international negotiation. As China is getting deeper involved in the global economy, there is more and more trade conflicts. And a better understanding of its own foreign trade will definitely give a better China s government position at the negotiation table. So, accurate foreign trade forecasting can reduce uncertainty and blindness in macroeconomic policymaking and trade conflict solving and thus obtaining good economic and social benefits. However, it is difficult to predict the foreign trade accurately due to the complexity of economic system and interactive effects of the economic variables on foreign trade. For this purpose, this paper attempts to propose a novel ensemble forecasting approach hybridizing econometric models and artificial intelligence (AI) models to predict China s foreign trade volume. The

3 FORECASTING TRADE WITH A HYBRID ENSEMBLE LEARNING APPROACH 3 basic idea of the ensemble forecasting approach is to use each model s unique feature to capture different patterns in the data. Both theoretical and empirical findings suggest that combining different methods can be an effective and efficient way to improve forecast performances. In their pioneering work on ensemble forecasting, Bates and Granger [6] showed that a linear combination of forecasts would give a smaller error variance than any of the individual methods. Since then, the studies on this topic (i.e., ensemble or combined forecasts) have expanded dramatically. Makridakis et al. [7] claimed that combining several single models has become common practice in improving forecasting accuracy ever since the well-known M-competition in which a combination of forecasts from more than one model often leads to improved forecasting performance. Likewise, Pelikan et al. [8] and Ginzburg and Horn [9] proposed combining several feed-forward neural networks to improve time series forecasting accuracy. More literature can be referred to a comprehensive review and annotated bibliography provided by Clemen [10]. Actually, the combined forecasting model is equivalent to the ensemble forecasting model in the general sense. But the word ensemble is preferablely relative to combined in terms of the explanation of Yu et al. [11]. Although some ensemble techniques including linear ensemble (e.g., simple averaging [12], weighted averaging [13], and stacked regression [14] ) and nonlinear ensemble (e.g., neural-networkbased nonlinear ensemble [11] ) have been presented, there are still some difficulties in the ensemble forecasting. First of all, for linear ensemble techniques, it is not necessarily suitable for all the situations. In some cases, it is hard to capture nonlinear patterns hidden in different ensemble members with linear ensemble techniques. Secondly, for nonlinear ensemble techniques, there is only one nonlinear ensemble way: neural-network-based nonlinear ensemble [11]. Nevertheless, it is well-known to us that there are some shortcomings such as local minima and over-fitting in neural network training and learning. Based upon the above two aspects, it is necessary to introduce a new ensemble forecasting approach in an attempt to overcome the two main difficulties mentioned above. The new ensemble forecasting approach will utilize the kernel-based support vector machine (SVM) technique to combine different single models for integration purpose. In terms of the above descriptions, the basic process of the proposed ensemble forecasting approach consists of four stages. First of all, an important econometrical model, the cointegration-based error correction vector auto-regression (EC-VAR) model is used to analyze the impacts of the economic variables on Chinese foreign trade from a multivariate linear analysis perspective. The main reason of selecting EC-VAR model is that linear models seem to be easier to capture the linear patterns hidden the data according to the findings of Yu et al. [11]. Second, an artificial neural network (ANN) based EC-VAR model is used to capture the nonlinear effects of economic variables on foreign trade from the nonlinear viewpoint. Third, for incorporating the effects of irregular events on foreign trade, the text mining and expert s judgmental adjustments are also integrated into the nonlinear ANN-based EC-VAR model. Finally, all kinds of economic variables, the outputs of linear and nonlinear EC-VAR models and judgmental adjustment model are used as input variables of a kernel-based SVM model for ensemble forecasting purpose. In the ensemble learning approach, econometric models and AI models are included simultaneously. In a sense, the proposed ensemble forecasting approach is actually a hybrid ensemble learning strategy. The main objectives of this study reflect the following two-fold: 1) to show how to predict foreign trade volume using the proposed hybrid econometric-ai ensemble learning approach; and 2) to display how various methods compare in their accuracy in forecasting foreign trade volume. In view of the two objectives, this paper mainly describes the building process of the proposed hybrid ensemble learning model and the application of the proposed ensemble forecasting approach in foreign trade volume forecasting, and meanwhile comparing forecasting

4 4 LEAN YU SHOUYANG WANG KIN KEUNG LAI performance with different evaluation criteria. The rest of this study is organized as follows. Section 2 describes the building process of the proposed hybrid econometric-ai ensemble learning method in detail. For further illustration, the proposed ensemble learning methodology is applied to Chinese export trade prediction in Section 3. Finally, some main conclusions and future research directions are contained in Section 4. 2 Hybrid Econometric-AI Ensemble Learning Methodology In this section, an overall building process of the hybrid econometric-ai ensemble learning methodology is described in detail. First of all, the co-integration technique and error correction vector auto-regression (EC-VAR) model are briefly reviewed. Then an ANN-based EC-VAR model is presented to explore nonlinear patterns in the complex economic phenomena. To incorporate the effects of irregular events into prediction, text mining technique and expert s judgmental adjustments are also considered into the ANN-based EC-VAR model. In order to integrate almost all information and implied knowledge, a kernel-based support vector regression (SVR) model is used for nonlinear ensemble purpose. 2.1 Co-Integration Technique and EC-VAR Model At one time conventional wisdom was that non-stationary variables should be differenced to make them stationary before including them in multivariate models. However, this situation was changed when Engle and Granger [15] introduced the concept of co-integration in They showed that it is quite possible for a linear combination of integrated variables to be stationary. In this case the variables are said to be co-integrated. Consider a set of variables in long-run equilibrium (static equilibrium) when β 1 x 1t + β 2 x 2t β n x nt = 0. The equilibrium error is then e t = β T X t. If the equilibrium is meaningful it must be the case that the error is stationary. For this, co-integration technique is popular and widely used in many domains since Interested readers can refer to Engle and Granger [15] for more details. If co-integration relationship between variables exists, an error correction vector autoregressive (EC-VAR) model can be formulated for prediction purpose. Actually, an EC-VAR model can be seen as a co-integration-based forecasting model, which is represented by k 0 k m j y t = α i y t i + β j,i x j,t i + γ EC t 1 + ξ t, (1) j=1 i=0 where y t is dependent variable, y t i is the lag term, x t i is the lag terms of the independent variables, EC t 1 is the co-integration relation, or error correction term, and ξ t is the Gaussian noise, and α, β, γ are the coefficients of different variables and lag terms, respectively. If one would like to build an EC-VAR model, the procedure below should be followed. First of all, stationarity testing is performed. If the time series are stationary, VAR is appropriate for modeling. If the time series are not stationary, co-integration relationship checking is required. If there are co-integration relationships among different time series, the EC-VAR model may be constructed. Generally, an EC-VAR model can lead to a better understanding of the nature of any nonstationarity among the different component series and can also improve longer term forecasting performance over an unconstrained model. The advantage of the EC-VAR model is that it is fully driven by data, which make it popular in academics and practitioners. More details about EC-VAR model can be referred to [16].

5 FORECASTING TRADE WITH A HYBRID ENSEMBLE LEARNING APPROACH ANN-Based Nonlinear EC-VAR model In the above co-integration-based EC-VAR model, the EC is only a linear error correction term. But many people believed that the errors of component series may contain much nonlinearity, a linear error correction is often not sufficient in a sense. For this reason, Hendry and Ericsson [17] proposed a nonlinear EC-VAR model with the following computational form: k 0 k m j y t = α i y t i + β j,i x j,t i + γ f(ec t 1 ) + ξ t, (2) j=1 i=0 where f( ) is a nonlinear function, EC t 1 denotes a long-term co-integration relationship, and other symbols are similar to Equation (1). Hendry and Ericsson [17] used a thrice function of EC t 1 to predict the currency demand of UK and obtained good performance. In this study, we used the ANN to construct a nonlinear function f ( ) to create an ANN-based nonlinear EC- VAR model. That is, the proposed ANN-based EC-VAR model applies the sigmoid function to determine the nonlinear function f ( ) as the error correction term. In this paper, the sigmoid function is represented by 1 f(ec t 1 ) =, (3) 1 + e (w ECt 1+b) where w and b are unknown parameters determined by ANN training. Usually, the feed-forward neural network (FNN) can realize this goal. Hornik et al. [18] and White [19] found that a threelayer FNN with an identity transfer function in the output unit and logistic functions in the middle-layer units can approximate any continuous function arbitrarily well given a sufficient amount of middle-layer units. Generally, the ANN-based EC-VAR model is performed with the following two steps [1] : 1) Determine the co-integration relationship between the variables, and 2) The co-integration relationship is placed into the VAR equation with a nonlinear function instead of linear function or a constant. Particularly, this paper uses the sigmoid function as the nonlinear function. 2.3 Text Mining and Judgmental Adjustment-Based Nonlinear EC-VAR Model In the complex economic system, the interactive effects are often generated from different sources. Besides the related variables and nonlinear interactions, some important irregular events, for example, 911 terrorist attack in 2001 and SARS in 2003, have a significant impact on the world economy and foreign trade. To further improve the prediction performance, the text mining and expert s judgmental adjustment are incorporated into the forecasting model. Generally text mining refers to the process of extracting interesting and non-trivial information and knowledge from unstructured text [20]. Interested readers can refer to [21] for more details about text mining. In this study, the main goal of the text mining is to retrieve related information from various sources and the human experts provide the judgmental adjustment information for these important information. Within such a framework, the equation (2) can be extended as k 0 k m j y t = α i y t i + β j,i x j,t i + γ f(ec t 1 ) + δ JA + ξ t, (4) j=1 i=0 where δ is an indicator parameter, i.e.,

6 6 LEAN YU SHOUYANG WANG KIN KEUNG LAI { 1, if irregular events occurred, δ = 0, otherwise, JA represents the expert s judgmental adjustment (JA) for a specified irregular event, and other symbols are the same as the above. 2.4 Kernel-Based Nonlinear Hybrid Ensemble Forecasting Model Considering various different economic factors and judgmental adjustments, we have obtained many different forecasting equations. In order to capture the effects of different economic variables and increase forecasting accuracy, the subsequent task is to combine these different prediction results into an aggregated output in an appropriate ensemble strategy. Suppose there are n individual predictions, ˆf1 (x), ˆf 2 (x),, ˆf n (x), provided by different forecasting models, the main question of ensemble forecasting is how to combine (ensemble) these different single forecasts into an aggregate forecast ŷ = ˆf(x), which is assumed to be a more accurate prediction. The general form for such an ensemble predictor can be defined as ŷ = ˆf(x) = n w i ˆfi (x), (5) where w i denotes the assigned weight of ˆf i (x), and in general the sum of the weight is equal to one. In ensemble forecasting, how to determine ensemble weights is a key issue. As earlier mentioned, there are a variety of methods for determining ensemble weights in the past studies. Generally, there are two kinds of ensemble strategies: linear ensemble and nonlinear ensemble, which are described below. Typically, linear ensemble strategies include two categories: the simple averaging [12] and the weighted averaging [13]. Weighted averaging has three main methods: the simple mean squared error (MSE) approach [12], stacked regression (modified MSE) approach [14], and variance-based weighted approach [22]. Simple averaging is one of the most frequently used ensemble approaches that are easy to understand and implement [12]. Some experiments have shown that this approach by itself can lead to improved performance and it is an effective approach to improve prediction performance. Usually, the simple averaging method for ensemble forecasting is defined as ˆf(x) = n w i ˆfi (x) = 1 n n ˆf i (x), (6) where the weight of each individual predictor w i = 1 n. Although simple averaging is an easy-to-do ensemble approach, it treats each member equally, i.e., it does not stress different ensemble members that can make more contribution to the final generalization. That is, it does not take into account the fact that some predictor may be more accurate than others. If the variances of ensemble members are very different, we do not expect to obtain a better result by using simple averaging [23]. In addition, since the weights in the combination are so unstable, a simple averaging may not be the best choice in practice [24]. The simple MSE approach estimates the linear weight parameter w i in equation (5) by minimizing the MSE [12], that is, for i = 1, 2,, n, { m ( m w opt,i = arg min w i j=1 ( w T i } ) 2 ˆf i (x j ) d ji (x j ) = j=1 ) 1 m ˆf i (x j )fi T (x j ) d ji (x) ˆf i (x j ), (7) j=1

7 FORECASTING TRADE WITH A HYBRID ENSEMBLE LEARNING APPROACH 7 where d(x) is the expected value. The simple MSE solution seems to be reasonable, but, as Breiman [14] has pointed out, this approach has two serious problems in practice: 1) the data are used both in the training of each predictor and in the estimation of w i, and 2) individual predictors are often strongly correlated since they try to predict the same task. Due to these problems, this approach s generalization ability will be poor. The stacked regression method was proposed by Breiman [14] in order to solve the problems associated with the previous simple MSE method. Thus, the stacked regression method is also called the modified MSE method. This approach utilizes cross-validation data to modify the simple MSE solution, i.e., w opt,i = arg min w i { m j=1 } ( w T i g i (x j ) d ji (x j ) ) 2, i = 1, 2,, n, (8) where g i (x j ) = ( (1) (2) (m) ˆf i (x j ; D cv ), ˆf i (x j ; D cv ),, ˆf i (x j ; D cv ) ) T R M is a cross-validated version ˆf i (x j ) R M and D cv is the cross-validated data. Although this approach overcomes the limitations of the simple MSE method, the solution is based on the assumption that the error distribution of each validated set is normal [23]. In practice, however, this normal assumption does not always hold and thus this approach does not lead to the optimal solution in the Bayes sense [23]. The variance-based weighting ensemble approach estimates the weight parameter w i by minimizing error variance σi 2[22] assuming that all predictors are error-independent, i.e., w opt,i = arg min w i { n ( wi σi 2 ) ) } 2, i = 1, 2,, n, (9) under the constraints n w i = 1 and w i 0. Using the Lagrange multiplier, the optimal weights are w opt,i = ( σ 2 i ) 1 n ( ), i = 1, 2,, n. (10) σ 2 1 j j=1 The variance-based weighting method is based on the assumption of error independence. Moreover, as earlier mentioned, individual predictors are often strongly correlated for the same task. This indicates that this approach has serious drawbacks for minimizing error-variance when single predictors with strong correlation are included in the ensemble members. From the above descriptions, we find that the ensemble forecasting based on linear techniques is insufficient. Therefore, the emerging nonlinear ensemble technique is a promising solution to determine the optimal weight for neural ensemble predictor. Currently, one typical nonlinear ensemble approach: the artificial neural network (ANN) based nonlinear ensemble method [1,11] is presented. This approach uses meta neural networks for ensemble purposes. Interested readers can refer to [11] for more details. Experiment results obtained show that the neural network-based nonlinear ensemble approach consistently outperforms the other linear ensemble approach. However, there are several shortcomings to the neural network-based nonlinear ensemble approach. First, a neural network often traps into local minima due to the drawbacks of neural network algorithm. Second, a neural network can easily exhibit the overfitting problem because of too many learning epochs. Finally, neural network architecture and type heavily depend upon the user s experience, thus affecting final prediction performance. To

8 8 LEAN YU SHOUYANG WANG KIN KEUNG LAI avoid these negative effects, a new kernel-based nonlinear model, the support vector regression (SVR) model, is adopted to combine the ensemble members into an aggregate forecast. The support vector machine (SVM) is an elegant tool for solving pattern recognition and regression problems. Over the past decades, it has attracted many researchers attention from the neural network and mathematical programming community. The main reason for this phenomenon is its ability to provide excellent generalization performance. Readers can refer to [25] for more details. Simply speaking, in an SVR model, one has to estimate the functional dependence of the dependent variable y on a set of independent variables x. The model assumes, like other regression problems, that the relationship between the independent and dependent variables is given by a deterministic function f plus some additive noise y = f(x) + ε. (11) The current task is then to find a functional form for f that can correctly predict new cases that the SVR has not been presented before. This can be achieved by training the SVR model on a sample set, i.e., training set a process that involves the sequential optimization of an error function. Concretely speaking, assume that x is an input vector and z is a feature space vector which is related to x by a transformation, z = φ(x). Let the training set D: {x i, d i }, consist of m data points where x i is the i-th input pattern and d i is the corresponding target value, d i R. The goal of the SVR is to estimate a function f(x) that is as close as possible to the target values for every sample, and meantime is as flat as possible for good generalization. The function f(x) is represented using a linear function in the feature space f(x) = w φ(x) + b, (12) where b denotes the bias. As in all SVR designs, we define the kernel function k(x, ˆx) = φ(x) φ(ˆx), where denotes inner product in the z space. Thus, all computations will be done using only the kernel function. This inner-product kernel helps in taking the dot product of two vectors in the feature space without having to construct the feature space explicitly. Mercer s theorem [25] explains the conditions under which this kernel operator is useful for SVR designs. In a sense, the SVR can be viewed as a kernel-based learning method. Similar to ANN-based nonlinear ensemble forecasting model, the SVR-based nonlinear ensemble forecasting model can also be seen as a nonlinear information processing system that can be represented as ŷ = ˆf(x) = ϕ ( ˆf(x1 ), ˆf(x 2 ),, ˆf(x n ) ), (13) where ( ˆf(x 1 ), ˆf(x 2 ),, ˆf(x n )) is the output of single predictors, ŷ is the aggregated output or ensemble forecasting result, ϕ(x)is a nonlinear function determined by SVR training. Note that many related economic variables and their lag terms can also affect the foreign trade. To our knowledge and experience, these related variables and their lag terms have some significant effects on the foreign trade prediction and they should be included into the ensemble forecasting. In a sense, the ensemble forecasting in this study has become a hybrid ensemble learning problem. Concretely speaking, suppose that there are k lag terms (y t i, i = 1, 2,, k), m related variables (x j, j = 1, 2,, m), and p forecasts provided by different forecasting models, then hybrid ensemble learning approach can be represented as ŷ t = ϕ(y t 1,, y t k ; x 1, x 2,, x m ; ŷ 1, ŷ 2, ŷ p ; ω) + ξ t, (14) where ω is a vector of all parameters and φ( ) is a nonlinear ensemble function.

9 FORECASTING TRADE WITH A HYBRID ENSEMBLE LEARNING APPROACH 9 Determining the nonlinear function φ( ) is quite challenging. In this study, a standard SVR with a Gaussian kernel is employed to realize nonlinear mapping. Actually the study uses the k lag terms (y t i, i = 1, 2,, k), m related variables (x j, j = 1, 2,, m), and p forecasts as inputs of SVR model to construct the hybrid ensemble forecasting methodology. Figure 1 gives an intuitive illustration for the proposed hybrid ensemble forecasting approach. Note that the SVR is also sensitive to parameters similar to the artificial neural networks. For this problem, we use trial and error method to determine the best parameters to obtain stable results. (y t 1, y t 2,, y t k ) (x 1, x 2,, x m) ŷ 1 ŷ 2 ŷ p Input vector z = (y t 1,, y t k ; Input layer x 1, x 2,, x m; ŷ 1, ŷ 2,, ŷ p) Transformation layer K(z 1,z) K(z 2,z) K(z s,z) w 1 w 2 w s Nonlinear transformation by Kernel function K(z i, z) Output layer ŷ ŷ = s wik(zi, z) + b Figure 1 An intuitive illustration for the hybrid ensemble forecasting model From Figure 1, it is easy to find that the structure of SVR is similar to that of ANN. In the same way, the SVR also consists of an input layer, middle layer or transformation layer and output layer. The difference between SVR and ANN is that every node of the middle layer is a support vector transformed by a kernel function. Usually, the Gaussian function is used as a kernel function. Note that the SVR could overcome the important drawbacks of ANN e.g., local minima because the SVM adopts margin maximization or structural risk minimization principle. In the above hybrid ensemble learning process, it should be noted that all variables and all single models including linear co-integration-based EC-VAR, ANN-based EC-VAR, and judgmental adjustment-based nonlinear EC-VAR models are included into the final ensemble learning. It is reasonable for hybrid ensemble learning because these variables and single models are uncorrelated from calculation view, while the un-correlation of ensemble members can improve the performance of ensemble learning according to the previous research results in ensemble learning [27 28]. It is interesting to examine the underlying idea of the proposed hybrid econometric-ai ensemble learning methodology. For a complex and difficult forecasting problem, one single linear or nonlinear model is inadequate to model it due to interactive effect of multiple factors. Through the linear co-integration-based EC-VAR, ANN-based EC-VAR and judgmental adjustment-based nonlinear EC-VAR models, the linear patterns, nonlinear patterns, and irregular patterns are also found. In order to formulate a comprehensive understanding, the explored patterns are aggregated into a single prediction, as indicated in equation (14) [1]. From the above analysis, it is obvious that the conventional linear econometric models are

10 10 LEAN YU SHOUYANG WANG KIN KEUNG LAI insufficient for complex prediction problems because some nonlinear and irregular patterns are not captured by them. From this viewpoint, it is not hard to understand why conventional linear econometric models cannot always predict some complex and difficult economic problems well. In this sense, using such a hybrid econometric-ai ensemble learning methodology, some complex prediction problems are easy to be implemented based on different linear and nonlinear models as well as nonlinear ensemble techniques [1]. 3 Experimental Analysis For verification purpose, the proposed kernel-based ensemble learning approach hybridizing econometric techniques and AI methods is applied to China s foreign trade volume prediction problem due to its important implications on macroeconomic policymaking. In this section, we first describe the experimental data and evaluation criteria. Then several typical forecasting models and related parameters are presented. Finally, the experimental results are reported in terms of the constructed models. 3.1 Experimental Data and Evaluation Criteria Due to the complexity of economic system, the effects of the economic variables (or factors) on foreign trade are realized from many different ways, including direct, indirect, and some interactive ways. Therefore we must select some representative variables to construct the forecasting model. For this purpose, the following principles are worth considering in the process of model variable selection. a) Theoretical interpretability. This requires that the selected variables must have an explanation power for the constructed model. That is, the variable can explain why it affects the foreign trade from the theoretical viewpoint. b) Representative. This principle requires us to find some representative variables to construct a foreign trade forecasting model. That is to say, the selected variables are some important variables affecting foreign trade. c) Availability. This requires that the related data are available for selected variables. According to the above three principles, we can select some main variables affecting Chinese foreign trade in terms of trade partners and competitors, as shown in Table 1. Table 1 Some main factors (or variables) affecting Chinese foreign trade Category China Trade partners Trade competitors Export and import indicator The total amount of Chinese foreign trade Import amount of main trade partners Export amount of main trade competitors The nominal Renminbi (RMB) exchange Hong Kong dollar, Japanese yen, Korean exchange rates rate against Japanese yen, Ger- won, Taiwan new dollar, against the US US dollar man mark, Korean and East Asian currency dollar won, Singapore dollar, Taiwan new dollar. rates Economic situation and investment Financial and monetary situation Inflation rates, GDP, FDI Currency supply, foreign exchange reserve CPI, unemployment rate, GDP CPI and GDP of these countries

11 FORECASTING TRADE WITH A HYBRID ENSEMBLE LEARNING APPROACH 11 The data used in this study are monthly covered from January 1985 to December 2006, which is from various sources. The data of Chinese foreign trade volume (including export, import, and total trade volume), foreign reserve, and foreign direct investment are collected from Commerce Statistics of the Ministry of Commerce of China [4], while the exchange rates data are from Federal Reserve Economic Data of United States ( We take the data from January 1985 to December 1999 with a total of 180 observations as training samples for model construction and the remainder is used as the testing samples (84 observations) for model verification. For space consideration, the original data are not listed here and are obtained from the corresponding sources. In addition, for comparison, two typical indicators, normalized root mean squared error (NRMSE) and directional statistics (D stat ) are used in this study. Given N pairs of the actual values (or targets, y t ) and predicted values (ŷ t ), the NRMSE which normalizes the RMSE by dividing it through the standard deviation of respective series can be defined as N NRMSE = t=1 (y t ŷ t ) 2 N t=1 (y t ȳ t ) = 1 1 N (y t ŷ t ) 2 σ N 2, (15) where σ is the estimated standard deviation of the data and ȳ t the mean. Clearly, the NRMSE only measure prediction in terms of levels. That is, accuracy in goodness-of-fit is only one of the most important criteria for forecasting models the others are the earnings generated from improved decisions. For macroeconomic policymakers, the aim of forecasting is to support or improve decisions so as to make right decision. But in foreign trade forecasting, improved decisions often depend on correct forecasting directions between the actual and predicted values, y t and ŷ t, respectively. In a sense, the latter is more important than the former. Basically, directional statistics (D stat ) [11] can be expressed as D stat = 1 N a t 100%, (16) N t=1 where a t =1 if (y t+1 y t )(ŷ t+1 y t ) 0, and a t =0 otherwise, and N is the number of the testing samples. In the experiments, all econometric models (e.g., EC-VAR) are implemented via the Eviews software package, which is produced by Quantitative Micro Software Corporation. The AI models (e.g., ANN and SVR) are built using the Matlab software package, which is produced by Mathworks Laboratory Corporation. The hybrid ensemble learning approaches involves the econometric and AI models and thus it uses both of the software packages. 3.2 Forecasting Models and Their Parameter Estimation Based on the training data and related variables, different models can be constructed for prediction in terms of the previous descriptions Co-integration-based EC-VAR model As previously mentioned, many factors influence Chinese foreign trade. Usually, the most popular method for handling this situation is vector auto-regressive (VAR) model due to its intuition and simplicity. However, it should be noted that most macroeconomic time series are not stationary. Furthermore, the forecasting performance of VAR model is not satisfactory for Chinese foreign trade at all. The main reason is that VAR is a typical linear model, but economic system is a complex system including many nonlinear and irregular patterns, thus the goodness-of-fit of linear model for macroeconomic variables is very poor. In order to improve the t=1

12 12 LEAN YU SHOUYANG WANG KIN KEUNG LAI prediction performance and analyze the relationships among economic variables, co-integration technique proposed by Engle and Granger [15] should be used. If co-integration relationship between variables exists, an error correction vector auto-regressive (EC-VAR) model can be formulated for forecasting purpose. Using the monthly data and some important variables listed above, the co-integration equation and error correction term for EC-VAR model can be formulated, as shown below. lex t = α i lex t i + β 1i lfr t i + β 2i usrmb t i + β 3i usjp t i + β 4i lusavr t i + β 5i luseu t i + β 6i lussg t i + γ EC t 1 + ξ t, (17) EC t 1 = lex + η 1 lfr + η 2 jusrmb + η 3 usjp +η 4 lusavr + η 5 luseu + η 6 lussg + η 7 t + c, (18) where α, β, γ, η are the parameters determined by model and is the difference operator, lex denotes the logarithm of monthly Chinese export volume, lfr is the logarithm of Chinese foreign exchange reserve, usrmb represents the exchange rate of RMB against US dollar, usjp denotes the exchange rate of Japanese yen against US dollar, lusavr represents the logarithm of the average exchange rate of US dollar, luseu is the logarithm of euros against US dollar, lussg is the logarithm of Singapore dollar against US dollar, t is time factor, k is the variable lag order, EC is the error correction terms, and c is a constant. It should be noted that the variables of equations (17) and (18) are selected by many experiments. In terms of the experimental data and reported results, the optimal lag order is determined to be 6 according to the parameter significance testing. Although there is a slight difference for different variables, it is rational for the selected variables to have sixth-order lag effects. Using the experimental data, the estimated parameters can be found in Table 2. Table 2 Estimated parameters for EC-VAR model i α β β β β β β η γ c ANN-based nonlinear EC-VAR model China s economy grows very fast, and it also undergoes a quick structural change. The linear econometric forecasting models are usually hard to capture these structural changes. Moreover one important drawback of the EC-VAR model is that the forecasting performance is poor sometimes when nonlinearity exists. To improve the model robustness, some researchers (e.g., Hendry and Ericsson [17] ) begin trying to build a nonlinear EC-VAR model with the form of equation (2).

13 FORECASTING TRADE WITH A HYBRID ENSEMBLE LEARNING APPROACH 13 As mentioned in Section 2.2, the sigmoid function is used to capture the nonlinear patterns hidden the data. That is, the sigmoid function is used for error correction term. Particularly, this paper uses a sigmoid function with an expected value 0.5 as the error correction term in the nonlinear EC-VAR model. That is, the error correction term is 1 f(ec t 1 ) = e (w ECt 1+b) In this error correction term, parameters w and b are determined by a typical three-layer feedforward neural network. In this way, an ANN-based nonlinear EC-VAR model is constructed for China s foreign trade volume forecasting. Using the real data and related variables, the ANN-based nonlinear EC-VAR model can be represented in the following form: lex t = α i lex t i + β 1i lfr t i + β 2i usrmb t i + + β 3i usjp t i + β 4i lusavr t i + β 5i luseu t i β 6i lussg t i + γ f(ec t 1 ) + ξ t, (19) 1 f(ec t 1 ) = 0.5, (20) 1 + e (w ECt 1+b) where all parametric symbols are the same as the explanations of equations (17) (18). Using the experimental data, the estimated parameters are shown in Table 3. Table 3 Estimated parameters for ANN-based nonlinear EC-VAR model i α β β β β β β γ w b Incorporating text-mining and judgmental adjustment into EC-VAR model It is well known that the quantitative forecasting models are based on historical data. The basic hypothesis behind the quantitative forecasting models is that history might repeat. But there are many unexpected irregular events which happen rarely, some may never happen again. However, such events have a huge impact on world economy, for example 911 terrorist attacks in the United States. Therefore to further improve the forecasting performance, text mining technique and expert s judgmental adjustment should be included into the forecasting model. In this study, the main goal of the text mining is to collect related text information affecting foreign trade variability from various sources and to provide the judgmental adjustment information for improving foreign trade forecasting.

14 14 LEAN YU SHOUYANG WANG KIN KEUNG LAI As earlier mentioned, text mining and expert judgmental adjustment are used to explore the effects of irregular events on Chinese foreign trade. We can use text-mining technique to search some irregular and infrequent events and include them into the quantitative models for improving prediction performance. For example, the SARS in 2003, an important irregular event, has a significant impact on Chinese foreign trade in May and June Such an event is hard to be captured by the quantitative forecasting model, thus we are required to consult the related domain experts (e.g., trade experts) to quantify the effect of SARS event on Chinese foreign trade. Experts can give a judgmental adjustment value according to their experience. Within such a framework, a judgmental adjustment (JA) based nonlinear EC-VAR model can be represented by lex t = α i lex t i β 1i lfr t i + β 3i usjp t i + β 2i usrmb t i β 4i lusavr t i + β 5i luseu t i β 6i lussg t i +γ f(ec t 1 )+δ JA + ξ t, (21) where δ is the indicator paramter, and other parameters are same to the explanation of equations (17) (20). Note that the estimated parameters in equation (21) are identical to those of equations (19) (20) because the same experimental data are used. Actually, the idea of using text mining and expert judgmental adjustment to improve the prediction performance is not new, this practice has been used to improve the crude oil price forecasting, and this study only uses the similar method. Interested readers can refer to [26] for more details. As an illustrative example, we use text mining technique to collect some important irregular trade events occurring in 2006 and meantime summarize the effects of these irregular events on China s foreign trade according to the judgment of related trade experts in Ministry of Commerce of China [4], as shown in Table Nonlinear hybrid ensemble forecasting model Considering various different factors and judgmental adjustment, we can formulate different forecasting models with high accuracy. Besides different prediction models, some economic variables and their lag terms also affect the foreign trade volume. In order to capture the effects of different variables on Chinese foreign trade and increase forecasting accuracy, a nonlinear hybrid ensemble learning approach is proposed for this purpose. As described in Section 2.4, the proposed nonlinear hybrid ensemble forecasting model can be represented as ŷ t = ϕ( y t 1,, y t k0 ; x 1,t 1, x 1,t 2,, x 1,t k1 ; x 2,t 1, x 2,t 2,, x 2,t k2 ; ; x m,t 1, x m,t 2,, x m,t km ; ŷ 1, ŷ 2, ŷ p ; ω) + ξ t. Determining the function φ( ) is quite challenging. In this study, kernel-based support vector regression (SVR) is employed to realize nonlinear mapping. Actually, the SVR training is a process of searching for optimal ensemble weights. Using the experimental data, the corresponding parameters can be determined. In such a way, a kernel-based nonlinear hybrid econometric-ai ensemble learning approach can be generated for Chinese foreign trade volume forecasting problem. (22)

15 FORECASTING TRADE WITH A HYBRID ENSEMBLE LEARNING APPROACH 15 Table 4 The effects of some irregular events on China s export trade in 2006 Time Irregular Events Direction Ranges 2006/12 First strategic economic dialogue between China and US to % harmonize two nations economic policy 2006/11 Beijing holds forum on China-Africa cooperation and South % South cooperation is strengthened Russian Premier signed 12 economic cooperation agreements + 3 8% with Chinese Premier Chinese foreign exchange reserve exceeds one trillion US dollars + 7 9% 2006/10 US GDP of the third quarter grows 1.6% + 2 4% 2006/09 China largely adjusts the customs policy, especially for policy 4 10% of export rebate tax 2006/08 American unemployment rate arrives at 4.7% 2 5% 2006/07 China and India reopen frontier trade port after 44 years + 3 7% 2006/06 GDP of Euro section grows 1.9% in the first quarter of % 2006/04 The economic and trade forum between Taiwan and Mainland + 5 9% is held in Beijing. 2006/02 Chinese exceeds Japan in foreign exchange reserve for the first time and becomes the world largest foreign exchange reserve country + 5 8% 3.3 Forecasting Results and Model Comparisons Using the above experimental design and related data, the kernel-based nonlinear hybrid econometric-ai ensemble learning approach can be used for Chinese foreign trade volume forecasting problem. For intuitive purpose, Figure 2 gives graphical representations of the forecasting results for China s export trade volume using the proposed approach. From the figure, we can roughly find that the total forecasting performance is very promising and satisfactory. Monthly export trade volume 100 Million USD Actual Predicted Figure 2 Month Forecasting results of hybrid ensemble learning approach (01/ /2006) In order to verify the effectiveness of the proposed kernel-based hybrid ensemble learning

16 16 LEAN YU SHOUYANG WANG KIN KEUNG LAI approach, single econometric model (EC-VAR), single AI models (single ANN and SVR), hybrid models (ANN-based EC-VAR and ANN & JA-based EC-VAR), four linear ensemble methods (Simple averaging, simple MSE, stacked regression, and variance-based weighting), and two nonlinear ensemble approaches (ANN-based nonlinear ensemble and Kernel-based nonlinear hybrid ensemble) are used to predict Chinese foreign trade volume for model comparison purpose. Note that all parameters of ANN and SVR model are determined by trial and error in the experiments. According to the previous experimental descriptions, the computational results are reported in Tables 5 and 6 from the viewpoint of level prediction and direction prediction. In the two tables, a clear comparison of various methods for China s foreign trade volume forecasting is given via NRMSE and D stat. Generally speaking, the results obtained from the two tables also indicate that the prediction performance of the proposed kernel-based nonlinear hybrid ensemble forecasting model is better than those of the single econometric and AI models and other ensemble forecasting models. Table 5 The NRMSE comparison with different forecasting models Models Export volume Import volume Total volume NRMSE Rank NRMSE Rank NRMSE Rank Single EC-VAR Single ANN Single SVR ANN-based EC-VAR ANN & JA-based EC-VAR Simple averaging Simple MSE Stacked Regression Variance-based weighting ANN-based Ensemble Kernel-based hybrid ensemble Table 6 The D stat comparison with different forecasting models Models Export volume Import volume Total volume D stat (%) Rank D stat (%) Rank D stat (%) Rank Single EC-VAR Single ANN Single SVR ANN-based EC-VAR ANN&JA-based EC-VAR Simple averaging Simple MSE Stacked Regression Variance-based weighting ANN-based Ensemble Kernel-based hybrid ensemble Focusing on the NRMSE indicator in Table 5, our proposed kernel-based nonlinear hybrid econometric-ai ensemble approach performs the best in all the cases, followed by the