SELECTION OF ARX MODELS ESTIMATED BY THE PENALIZED WEIGHTED LEAST SQUARES METHOD

Transcription

1 九州大学学術情報リポジトリ Kyushu University Institutional Repository SELECTION OF ARX MODELS ESTIMATED BY THE PENALIZED WEIGHTED LEAST SQUARES METHOD Qin, Pan Faculty of Mathematics, Kyushu University Nishii, Ryuei Faculty of Mathematics, Kyushu University 秦, 攀九州大学数理学研究院 西井, 龍映九州大学数理学研究院 出版情報 :Bulletin of informatics and cybernetics. 42, pp.35-43, 2-2. Research Association of Statistical Sciences バージョン :published 権利関係 :

2 SELECTION OF ARX MODELS ESTIMATED BY THE PENALIZED WEIGHTED LEAST SQUARES METHOD by Pan Qin and Ryuei Nishii Reprinted from the Bulletin of Informatics and Cybernetics Research Association of Statistical Sciences,Vol.42 FUKUOKA, JAPAN 2

3 Bulletin of Informatics and Cybernetics, Vol. 42, 2 SELECTION OF ARX MODELS ESTIMATED BY THE PENALIZED WEIGHTED LEAST SQUARES METHOD By Pan Qin and Ryuei Nishii Abstract We consider the selection problem of auto-regressive time-series models with exogeneous variables (ARX) estimated by Penalized Weighted Least Squares (PWLS) method. AIC and BIC are developed based on the maximum likelihood estimation. Therefore, in this research, we evaluate GIC for the ARX models estimated by PWLS. In a numerical experiment, the model selected by GIC shows an excellent performance, especially, in a target region. Key Words and Phrases: AIC, auto-regressive time-series models with exogeneous variables, BIC, GIC, MLE, Model selection, System identification.. Introduction Auto-Regressive time-series models with exogeneous variables (ARX) is useful to describe dynamic behavior of a system or process in the discrete time domain. Selecting an appropriate model structure is most crucial for a successful modeling application. Akaike s Information Criterion (AIC) (Akaike (974)) and Bayesian Information Criterion (BIC) (Schwarz (978)) can be used for the ARX model evaluation and selection when the models are obtained by the Maximum Likelihood Estimation (MLE). However, various estimation procedures other than MLE may also be utilized for ARX models in practical applications. In this paper, the ARX models estimated by Penalized Weighted Least Squares (PWLS) method are taken into consideration. The reason for using PWLS is two-fold: In many modeling applications, the different measurements should be assigned different weights (Ljung (996)). For example, if we want to construct an ARX model which has good prediction performance on some specified regions, then the measurements in such regions should be put on more weight. On the other hand, regularization techniques are often used in the estimation procedures to control the trade-off between the smoothness of the model and the goodness of fitting to the data (Konishi and Kitagawa (28)). Especially for the estimation of ARX models, Ljung (996, Sect. 7.4) pointed out the importance of regularization techniques in the cases that the Hessian is an ill-conditioned matrix and/or the model is specified by too many parameters in comparison with the sample size. Therefore, the selection of ARX Faculty of Mathematics, Kyushu University, 744 Motooka, Nishi-Ku, Fukuoka , Japan. pan@math.kyushu-u.ac.jp Faculty of Mathematics, Kyushu University, 744 Motooka, Nishi-Ku, Fukuoka , Japan. nishii@math.kyushu-u.ac.jp

4 36 P. Qin and R. Nishii models estimated by PWLS is not only to find the proper auto-regressive order and variable subsets but to find an appropriate regularization parameter. Because the ARX models obtained by PWLS are different from the ones obtained by MLE, AIC and BIC are not applicable. Nevertheless, this situation may be remedied by the Generalized Information Criterion (GIC) (Konishi and Kitagawa (996)). GIC is a general framework for constructing information criteria in the context of functional statistics and can be applied to evaluate statistical models constructed by various types of estimation procedures other than MLE. In this research, we investigate what GIC can offer for evaluating and selecting the ARX model estimated by PWLS. The rest of the paper is organized as follows. Section 2 describes the ARX models estimated by PWLS. GIC for selecting the ARX models estimated PWLS will be specified in Section 3. In Section 4, numerical results are given. Finally, the discussion is stated in Section ARX Models estimated by PWLS Let t denote the present time, then an ARX model for -step-ahead prediction can be formulated as the following: m n y(t + ) = α i y(t i + ) + β ju(t j + ) + e(t), e(t) N(, σ 2 ), () i= j= for t =,, N, where y(t) R is the output (target variable), u(t) R p denotes p external inputs (explanatory variables), α i R and β j R p are coefficients, is a natural number, m and n are the ARX model orders satisfying m n. In this paper, we estimate the ARX models by minimizing the following function: where J P W LS = [ w t y(t + ) h t ξ ] 2 + λξ Kξ (2) h t = [y(t) y(t ) y(t m + ) u(t) u(t ) u(t n + ) ] : (m + np), ξ = [ α α 2 α m β β 2 β n ] : (m + np) and w t > are weights, K is an (m + pn) (m + pn) positive-semi definite matrix, and λ > is a regularization parameter. The function (2) can be also reformulated as follows: J P W LS = (y Hξ) W (y Hξ) + λξ Kξ (3) where y = ( y( + ) y(2 + ) y(n + ) ) : N, H = ( ) h h 2 h N : N (m + pn), w w 2 W = : N N. w N

5 Selection of ARX Models Estimated by the Penalized Weighted Least Squares Method 37 By solving the equation J P W LS / ξ =, the PWLS estimator ˆξ P W LS which minimizes J P W LS is given as the following: ˆξ P W LS = (H W H + λk) H W y. (4) The selection problem of the ARX models achieved by ˆξ P W LS is to find the proper regularization parameter λ and the orders m and n. Note that MLE for ARX models is obtained by ˆξ MLE = (H H) Hy (Ljung (996)). It is obvious that ˆξ P W LS is different from ˆξ MLE. Thus AIC and BIC cannot be applied to the selection of ARX models estimated by ˆξ P W LS. In the following section, GIC will be derived for the ARX models estimated by ˆξ P W LS. 3. GIC for the ARX models estimated by PWLS In this section, we will calculate GIC for the ARX models estimated by PWLS. Define θ = (ξ, σ) and let f(y(t + ) ˆθ P W LS ) denote an estimated ARX model, in which ˆθ P W LS is an M-estimator obtained by the solution of the following implicit equation: ψ(y(t + ), ˆθ P W LS ) = (5) with ψ being referred to as ψ function (Huber (24)). Then GIC for the models obtained by ˆθ P W LS is given by the following theorem in the context of functional statistics. Theorem 3.. GIC for f(y(t+ ) ˆθ P W LS ) can be calculated as follows: GIC = 2 log f(y(t + ) ˆθ P W LS ) + 2tr ( R Q ), (6) where matrices R and Q given by R = N ψ(y(t + ), θ) θ ˆθ, (7) θ= P W LS Q = ψ(y(t + ), N ˆθ log f(y(t + ) θ) P W LS ) θ ˆθ. (8) θ= P W LS For the proof, the readers are referred to Konishi and Kitagawa (996). An ARX model means that a conditional distribution of y(t + ) can be given by the normal distribution having mean h tξ and variance σ 2. Based on this fact, we construct a ψ function as follows: ψ(y(t + ), θ) = { w t log f(y(t + ) θ) } θ 2 λ ξ Kξ (9)

6 38 P. Qin and R. Nishii = σ 2 H W (y Hξ) tr(w )λ Kξ tr(w ) + σ σ 3 (y Hξ) W (y Hξ) = () (tr(w where λ = λ/ ) )σ 2. By solving the equation (), the estimator ˆθ P W LS for ARX models are obtained as ( ) ΞH W y ˆξP W LS ) ˆθ P W LS = = ) ˆσ P W LS (y Hˆξ P W LS W (y Hˆξ P W LS tr(w ) where Ξ = ( H W H + tr(w )ˆσ 2 P W LSλ K ). Then, the GIC for the ARX models obtained by ˆθ P W LS is given as the following proposition. Proposition 3.2. GIC for the ARX models achieved by ˆθ P W LS are calculated as follows: { GIC = 2 log f(y(t + ) ˆθ P W LS ) where [ ( ˆσ P 2 Ξ a W LSw t tr a 2tr(W ) ) ( bt λ Kˆξ P W LS c t ) ( ) ]} bt, () c t a = 2tr(W )ˆσ P W LS λ Kˆξ P W LS, (2) ) b t = (y(t + ) h tˆξ P W LS h t, (3) ˆσ 2 P W LS c t = ˆσ P W LS + ˆσ 3 P W LS Proof. Based on (), R in (7) can be derived as follows: R = ψ(y(t + ), θ) N θ θ= P W LS = ˆσ 2 H W H 2tr(W )λ K P W LS N 2 ˆσ P 3 (y Hˆξ P W LS ) W H W LS ( y(t + ) h tˆξ P W LS ) 2. (4)

7 Selection of ARX Models Estimated by the Penalized Weighted Least Squares Method 39 = = N ˆσ 2 P W LS N ˆσ 2 P W LS tr(w ) ˆσ P 2 W LS 2 ˆσ P 3 H W (y Hˆξ P W LS ) W LS 3 ˆσ 4 P W LS Ξ (y Hˆξ P W LS ) W (y Hˆξ P W LS ) 2tr(W )ˆσ P W LS λ Kˆξ P W LS 2tr(W )ˆσ P W LS λ ˆξ P W LSK 2tr(W ) ( Ξ a a 2tr(W ) ). (5) with a defined by the formula (2). Q in (6) can be derived as follows Q = ψ(y(t + ), N ˆθ log f(y(t + ) θ) P W LS ) θ ˆθ θ= P W LS { = log f(y(t + ) θ) } w t 2 λ ξ Kξ N θ ( ) log f(y(t + ) θ) θ θ= ˆθ P W LS = N w t ˆσ 2 P W LS (y(t + ) h tˆξ P W LS ) h t λ Kˆξ P W LS + ˆσ P W LS ˆσ P 2 W LS + ˆσ P W LS ˆσ 3 P W LS (y(t + ) h tˆξ P W LS ) 2 (y(t + ) h tˆξ P W LS ) ˆσ 3 P W LS h t (y(t + ) h tˆξ P W LS ) 2 = N w t ( bt λ Kˆξ P W LS c t ) ( bt c t ) (6) where b t and c t are defined by the formulas (3) and (4) respectively. Then, substitute R and Q in (6) by (5) and (6), the conclusion in Proposition 3.2. can be proved. Finally, we can select the ARX models estimated by PWLS through choosing the regularization parameter λ and ARX model orders m and n which minimize GIC in Proposition 3.2..

8 4 P. Qin and R. Nishii 4. Numerical example The dataset acquired from a winding process (Moor (997)) are used in the numerical study. Figure shows a scheme of the winding process. The main part of the plant is composed of a plastic web that is unwinded from first reel r (unwinding reel), goes over the traction reel r 2 and is finally rewinded on the rewinding reel r 3. The reel r is driven by the motor M and the reel r 3 is driven by the motor M 3. The motors are controlled by the input currents I and I 3 (A) respectively. S, S 2 and S 3 (rad/s) denote the angular speeds of the three reels and are measured by dynamo tachometers respectively. T (kg) is the tension between r and r 2 and is measured by a tension meter. The inputs S, S 2, S 3, I and I 3 and the output T are shown in Figure 2. S I r M T r 2 r 3 S 2 S 3 M 3 I 3 Figure : A winding system. Large tension will lead to the damage on the plastic web. Therefore, we want to construct ARX models which have good performance on prediction of large tension ( T (t).5kg in this study). Figure 3 depicts the samples of tension satisfy T (t).5kg. In the numerical study, we select two ARX models obtained by MLE based on AIC and BIC respectively. Let w t =, an ARX model estimated by Penalized Least Squares (PLS) is selected by GIC. Two ARX models estimated by PWLS with w t = T (t) and T (t) are selected by GIC. Let λ = (.) δ tr(w ) for δ =,,, then λ can be selected by selecting δ based on GIC. The order m of ARX models is selected from m =,, 5 and n is selected from n =,, m. We used the identity matrix as matrix K in PLS and PWLS. There are 25 samples which are sampled by Hz in the dataset. The samples are divided into two parts. The first 25 samples are used for estimating the ARX models and the rest samples are used for testing the prediction performance of the estimated models. The numerical results are listed in Table. Numerals in bold face denote the maximum coefficients of determination in each setting. In the numerical results, the coefficient of determination Rf 2 is calculated from all the 25 test samples of tension and their predictions. Rl 2 is calculated from the 45 test samples of tension which satisfy T (t).5kg and their predictions. Rf 2 shows

9 Selection of ARX Models Estimated by the Penalized Weighted Least Squares Method 4 T (kg) S (rad/s) S 2 (rad/s) S 3 (rad/s) I (A) I 3 (A) Output Input Input Input Input Input Time(sec) Figure 2: Inputs and output data 3 2 Tension(kg) Time(sec) Figure 3: The samples of tension in the test data (starts indicate the samples satisfying T (t).5kg)

10 42 P. Qin and R. Nishii Table : The selected ARX models and the coefficients of determination for =, 2, 3 IC w i Estimator m n δ λ R 2 f All test data R 2 l T (t).5kg (N=25) (N=45) AIC MLE BIC MLE GIC PLS GIC T (t) PWLS GIC T (t) PWLS AIC MLE BIC MLE GIC PLS GIC T (t) PWLS GIC T (t) PWLS AIC MLE BIC MLE GIC PLS GIC T (t) PWLS GIC T (t) PWLS the prediction performance for all testing samples and Rl 2 shows the performance of predicting the large tension in the test samples. Table indicates that the ARX models estimated PLS shows the similar performance of models estimated by MLE. However, the models estimated by PWLS with w t = T (t) work well on predicting large tension than other selected models. 5. Discussion In this paper, we addressed the selection of the ARX models estimated by PWLS. Because Akaike s AIC and Schwarz s BIC cover only models obtained by MLE, they cannot be applied to the ARX models estimated by PWLS. However, GIC is applicable to models achieved by various estimation methods in the context of functional statistics. Therefore we specified GIC for the ARX models estimated by PWLS. Based on the specified GIC, the proper regularization parameters λ and the orders m and n can be selected for the ARX models estimated by PWLS. It is worth noting that the specified GIC can be extended to the ARX models estimated by PLS naturally. The numerical experiments were conducted to examine the performance of the specified GIC, and GIC selected excellent models. Acknowledgement The authors are very grateful for the reviewer and the editor of BIC for those careful reading and valuable comments. This research is motivated by the joint research between Mazda Motor Corporation and Kyushu University. The authors would like to thank Mazda Motor Corporation for supporting the research.

11 Selection of ARX Models Estimated by the Penalized Weighted Least Squares Method 43 References Akaike, H. (974). A New Look at the Statistical Model Identification, IEEE Transtraction on Automatic Control AC-9, De Moor, B.L.R. (ed.) (997). DaISy: Database for the Identification of Systems, URL: smc/daisy/. [Used dataset: Data from a test setup of an industrial winding process, Process Industry Systems, 97-3.], Department of Electrical Engineering, ESAT/SISTA, K.U. Leuven, Belgium Huber, P. (24). Robutst Statistics, Wilely, New York. Konishi, S and Kitagawa, G. (996). Generalized Information Criteria in Model Selection, Biometrika 83, Konishi, S. and Kitagawa, G. (28). Information Criteria and Statistical Modeling, Springer. Ljung, L. (996). System Identification, Theory for the User, Prince Hall PTR. Schwarz, G. (978). Estimating the dimension of a model, Annals of Statistics 6, Received October 3, 29 Revised March 25, 2