Moving Least Squares Support Vector Machines for weather temperature prediction

Transcription

1 ESANN 7 proceedngs, European Symposum on Artfcal Neural Networks, Computatonal Intellgence and Machne Learnng. Bruges (Belgum), -8 Aprl 7, doc.com publ., ISBN Avalable from Movng Least Squares Support Vector Machnes for weather temperature predcton Zahra Karevan, Yunlong Feng and Johan A. K. Suykens KU Leuven, ESAT-STADIUS Kasteelpark Arenberg, B- Leuven, Belgum Abstract. Local learnng methods have been nvestgated by many researchers. Whle global learnng methods consder the same weght for all tranng ponts n model fttng, local learnng methods assume that the tranng samples n the test pont regon are more nfluental. In ths paper, we propose Movng Least Squares Support Vector Machnes (M-) n whch each tranng sample s nvolved n the model fttng dependng on the smlarty between ts feature vector and the one of the test pont. The expermental results on an applcaton of weather forecastng ndcate that the proposed method can mprove the predcton performance. Introducton In local learnng, nstead of usng all samples to tran the model, only those whch are n the regon of the test pont are used for model fittng [, ]. For example, n [] Polynomal Local regresson has been nvestgated. Besdes, concernng seasonal behavor of weather varables, n [], local learnng methods have been employed to deploy the local structure of the data to pursue better performance. Accurate weather forecastng s a major area of nterest wthn the field of clmate nformatcs. Numercal Weather Predcton (NWP) s the most wdely used method n the state-of-the-art approaches for weather forecastng. However, one major drawback of NWP s that t s an ntense method n terms of computatonal complexty []. Recently, data-drven approaches have been nvestgated to acheve relable weather predcton. In our prevous work [], Least Squares Support Vector Machne has been used for weather forecastng and the experments revealed that the performance s compettve wth state-of-the-art methods n weather forecastng. The objectve of ths paper s to nvestgate a soft localzaton n the framework of called Movng (M-). In the proposed method, we deploy the samples n the tranng set for model fittng whle ther mpact on the model s determned by ther smlarty to the test pont feature vector. Ths localzaton s done by definng a weghted cost functon n the optmzaton problem n the prmal problem. Moreover, to obtan a good generalzaton, tunng the parameters s done by usng Movng Cross Valdaton (M-CV) n whch each sample affects the valdaton error based on ts smlarty to the test pont. In ths study, n order to evaluate the proposed method, we conduct our experments on an applcaton of forecastng the maxmum and mnmum temperature of Brussels for to days ahead. 77

2 ESANN 7 proceedngs, European Symposum on Artfcal Neural Networks, Computatonal Intellgence and Machne Learnng. Bruges (Belgum), -8 Aprl 7, doc.com publ., ISBN Avalable from Movng Least Squares Support Vector Machnes In global learnng methods, the same weghts are consdered for all data ponts n the tranng data whle local learnng algorthms assume that the samples n the the test pont vcnty are more nfluental for model fittng. In Movng Least Squares (MLS), t s assumed that the tranng samples may not have smlar mportance n functon estmaton. Thus, each tranng pont has a weght whch s based on the dstance between the tranng sample and the test pont []. In ths paper, a soft localzaton n the framework of Least Squares Support Vector Machnes (s) s proposed. dffers from Support Vector Machnes (SVMs) n the sense that nstead of quadratc programmng n SVM, results n solvng a set of lnear equatons by takng equalty constrants and least squares loss [7]. Consderng x Rd, y R and ϕ : Rd Rh where ϕ( ) beng the feature map, the model n prmal space s formulated as: y x (x) w xt ϕ(x) + b x, () where b x R and w x Rh are estmated for a gven x. Note that y x (x) s dependng on x explctly and mplctly snce not only t s a functon of x, but also the optmzaton problem n prmal space s done for any fixed x. Gven s (x) as a non-negatve smlarty measure between the th tranng data feature vector and any fixed x, the optmzaton problem for tranng M model n prmal space s as follows: (w x, b x, e x ) mn w,b,e s. t. T w w + T γ N s (x)e, () y w ϕ(x ) + b + e,,..., N. Note that Weghted, proposed n [8], uses a smlar formulaton to obtan robust estmaton for regresson problems. Here a dfferent functon s estmated for any gven fixed x value. In the proposed method, the smlarty crteron s (x) can be ether a bnary or a real value. The former mples that only part of the tranng samples n the vcnty of the test pont s used to tran the model. Nevertheless, for real valued smlarty, all samples n the tranng set are nvolved for learnng the data whle ther nfluence on the model fittng s determned by ther smlarty to the test pont. In ths paper, we use the Gaussan smlarty crteron s (x) xt x + exp( x x /h ) and the cosne-based smlarty functon s (x) x x where x s the L-norm of the vector x. Note that, the former has a bandwdth parameter (h) to be tuned, whle the latter has no tunng parameter. As a specfic case, f s (x) s set equal to one, () represents the classcal formulaton n prmal space. From the Lagrangan L(w, b, e; α) wt w + γ s (x) e α (wt 78

3 ESANN 7 proceedngs, European Symposum on Artfcal Neural Networks, Computatonal Intellgence and Machne Learnng. Bruges (Belgum), -8 Aprl 7, doc.com publ., ISBN Avalable from ϕ(x ) + b + e y ), the optmalty condtons can be expressed as () below. L α ϕ(x ), w w L α, b () L α γs (x)e,,..., N, e L y wt ϕ(x ) + b + e,,..., N, α where α R are the Lagrange multplers. Assumng y [y,, yn ]T and N [,, ]T and after elmnatng w and e, the dual problem s wrtten as follows b TN, () N Ω + Sγ (x) α y where Sγ (x) s a dagonal matrx equal to Sγ (x) dag([ s (x)γ ; ; sn (x)γ ]) and Ω s the kernel matrx where based on Mercer s theorem [9]: Ωj ϕ(x )T ϕ(xj ) K(x, xj ), j,,..., N. () In ths paper, the Radal Bass Functon (RBF) s used as a kernel functon whch s formulated n (). In ths case, the regularzaton parameter γ and the kernel bandwdth σ are tunng parameters. K(x, xj ) exp( x xj /σ ). () Consderng α,x and b x as the soluton of the lnear equaton system (), the M- model as a functon estmator s formulated as follows: y x (x) N α,x K(x, x ) + b x. (7) Here the notaton of the soluton (α,x, b x ) stresses that () needs to be solved for any gven x. Although the proposed method can mprove the performance, t has ts own drawback. The proposed method s a tme consumng approach snce the model fittng should be done for each test pont ndependently. These methods are not sutable when the sze of the test set s large. On the other hand, n the case of tme seres problems, one may leverage the local learnng methods snce each tme there s only one test pont to be predcted.. Tunng the parameters In order to obtan a good generalzaton for the models, one may use k-fold CrossValdaton (CV) for tunng the parameters. Obvously, for the proposed method CV s not an deal tunng approach, snce the model s not traned properly for the regons whch are not close to the test pont. 79

4 ESANN 7 proceedngs, European Symposum on Artfcal Neural Networks, Computatonal Intellgence and Machne Learnng. Bruges (Belgum), -8 Aprl 7, doc.com publ., ISBN Avalable from In [], the authors suggested Robust Cross Valdaton to deal wth outlers and non-gaussan nose. In ths paper, we use k-fold Movng Cross Valdaton (M-CV), n whch each sample n the valdaton set affects the valdaton error based on the smlarty between ts feature vector to the test pont. Let k be the number of folds and Nv be the number of samples n the vth fold. The k-fold M-CV error for any fixed x s calculated as follows k v v s (x) ErrorM CV (x) s (x) err, (8) where err s a performance evaluaton method. In ths paper, the Mean Square Error (MSE) and Mean Absolute Error (MAE) are used to measure err.. Experments Dataset In ths study, data have been collected from the Weather Underground webste and nclude real measurements for weather elements such as temperature and humdty for ctes ncludng Brussels, Lege, Antwerp, Amsterdam, Endhoven, Dortmund, London, Frankfurt, Gronngen, Dubln, and Pars. The performance of the proposed method s evaluated on two test sets: () from md-november to md-december (Nov/Dec) and Fg. : Ctes ncluded n the () from md-aprl to mn-may model (Apr/May). The data cover a tme perod from begnnng 7 to md and comprse 98 measured weather varables for each day. The total number of features s equal to 98 lag, where lag s a tunng parameter whch ndcates the number of prevous days consdered n the model. Note that the number of tranng samples was determned by the number of days from the begnnng of 7 untl the day before the test date (vares from 89 to 7 ponts).. Results and dscusson In ths secton, we evaluate the proposed method on an applcaton of weather forecastng and as a case study, the predcton of the mnmum and maxmum temperature n Brussels for to days ahead was consdered. The tunng parameters ncludng the lag varable, the kernel bandwdth (σ) and the regularzaton constant (γ) of M-, and the kernel bandwdth parameter n the case of Gaussan smlarty (h) were tuned usng M-CV. We run our experments tmes and n Table, the average MAE and MSE of the predcton of the proposed method s compared wth those of for two test sets. As t s shown, mostly, M- outperformed. Furthermore, t can be 8

5 ESANN 7 proceedngs, European Symposum on Artfcal Neural Networks, Computatonal Intellgence and Machne Learnng. Bruges (Belgum), -8 Aprl 7, doc.com publ., ISBN Avalable from seen that usng the RBF smlarty crteron tends to be more advantageous than cosne-based one. Test set Days ahead Nov/Dec Apr/May Temp. Mean Absolute Error MM- RBF Cosne Mean Square Error MM- RBF Cosne Table : Average MAE and MSE of the predctons by and M- based on Cosne and RBF smlarty s (x) for test sets Nov/Dec and Apr/Nov. Fgure represents a comparson between the performance of Weather Underground predctons and the one of the data-drven models over both test sets together for the mnmum and maxmum temperature n Brussels. As t s depcted, for mnmum temperature, data-drven models mostly outperformed Weather Underground predctons and for maxmum temperature the performances were compettve. Concluson The am of the present research was to propose a Movng method n whch each tranng sample nfluence on the model fittng s dependng on the smlarty between ts feature vector and the one of the test pont. The proposed method was evaluated based on temperature predcton n Brussels for to days days ahead. The expermental results suggest that utlzng local learnng can mprove the accuracy of forecastng. In addton, data-drven methods have shown compettve performance wth the state-of-the-art methods n weather forecastng. 8

6 ESANN 7 proceedngs, European Symposum on Artfcal Neural Networks, Computatonal Intellgence and Machne Learnng. Bruges (Belgum), -8 Aprl 7, doc.com publ., ISBN Avalable from Weather Underground M- (RBF) M- (Cosne) Weather Underground M- (RBF) M- (Cosne) MAE for. Temp. MAE for. Temp..... Days ahead Days ahead Fg. : MAE of the predctons for Weather Underground, and M wth RBF and cosne based smlarty for. and. temperature Acknowledgment The research leadng to these results has receved fundng from the European Research Councl under the European Unon s Seventh Framework Programme (FP7/7-) / ERC AdG A-DATADRIVE-B (99). Ths paper reflects only the authors vews and the Unon s not lable for any use that may be made of the contaned nformaton. Research Councl KUL: CoE PFV// (OPTEC), BIL/T; PhD/Postdoc grants Flemsh Government: FWO: projects: G.77. (Structured systems), G.88N (Tensor based data smlarty); PhD/Postdoc grant ds Medcal Informaton Technologes SBO IWT: POM II SBO Belgan Federal Scence Polcy Offce: IUAP P7/9 (DYSCO, Dynamcal systems, control and optmzaton, -7). References [] Le on Bottou and Vladmr Vapnk. Local learnng algorthms. Neural Computaton, ():888 9, 99. [] Clve Loader. Local regresson and lkelhood. Sprnger Scence & Busness Meda,. [] Janqng Fan and Irene Gjbels. Local polynomal modellng and ts applcatons: monographs on statstcs and appled probablty, volume. CRC Press, 99. [] Zahra Karevan and Johan A. K. Suykens. Clusterng-based feature selecton for black-box weather temperature predcton. In Int. Jont Conf. on Neural Networks,. [] Peter Bauer, Alan Thorpe, and Glbert Brunet. The quet revoluton of numercal weather predcton. Nature, (77):7,. [] Davd Levn. The approxmaton power of movng least-squares. Mathematcs of Computaton of the Amercan Mathematcal Socety, 7():7, 998. [7] Johan A. K. Suykens, Tony Van Gestel, Jos De Brabanter, Bart De Moor, and Joos Vandewalle. Least Squares Support Vector Machnes. World Scentfc,. [8] Johan A. K. Suykens, Jos De Brabanter, Lukas Lukas, and Joos Vandewalle. Weghted least squares support vector machnes: robustness and sparse approxmaton. Neurocomputng, 8():8,. [9] James Mercer. Functons of postve and negatve type, and ther connecton wth the theory of ntegral equatons. Phlosophcal Transactons of The Royal Socety of London. Seres A, contanng papers of a mathematcal or physcal character, pages, 99. [] Jos De Brabanter, Krs Pelckmans, Johan A. K. Suykens, Joos Vandewalle, and Bart De Moor. Robust cross-valdaton score functons wth applcaton to weghted least squares support vector machne functon estmaton. Techncal report, KU Leuven,. 8