SUPPORT VECTOR MACHINES FOR CLASSIFICATION AND MAPPING OF RESERVOIR DATA

Transcription

1 SUPPOR VECOR MACHINES FOR CLASSIFICAION AND MAPPING OF RESERVOIR DAA Mkhal Kaevsk Stephae Cau Patrck Wog Aleksey Pozdukhov Mchel Maga Syed Shbl IDIAP RR-0-04 Jauary 200 o be publshed as a chapter of the Sprger book o soft computg for reservor charactersato

2 2

3 Support Vector Maches for Classfcato ad Mappg of Reservor Data M. Kaevsk,3,4, A. Pozdukhov 2, S. Cau 3, M. Maga 4, P.M. Wog 5, S.A.R. Shbl 6 IDIAP Dalle Molle Isttute of Perceptual Artfcal Itellgece, Smplo 4, Case Postale 592, 920 Martgy, Swtzerlad, kaevsk@dap.ch 2 Moscow State Uversty ad IBRAE, Moscow, Russa 3 INSA, Roue, Frace, scau@sa-roue.fr 4 Uversty of Lausae, mchel.maga@mp.ul.ch 5 Uversty of New South Wales, Australa 6 Ladmark Graphcs Abstract. Support Vector Maches (SVM) s a ew mache learg approach based o Statstcal Learg heory (Vapk-Chervoeks or VC-theory). VCtheory has a sold mathematcal backgroud for the depedeces estmato ad predctve learg from fte data sets. SVM s based o the Structural Rsk Mmsato prcple, amg to mmse both the emprcal rsk ad the complexty of the model, provdg hgh geeralsato abltes. SVM provdes o-lear classfcato SVC (Support Vector Classfcato) ad regresso SVR (Support Vector Regresso) by mappg the put space to hgh-dmesoal feature space usg kerel fuctos, where the optmal solutos are costructed. he paper presets the revew ad cotemporary developmets of the advaced methodology based o Support Vector Maches (SVM) for the aalyss ad modellg of spatally dstrbuted formato. he methodology developed combes the power of SVM wth well kow geostatstcal approaches ad tools cludg exploratory data aalyss ad exploratory varography. Real case studes (classfcato ad regresso) are based o reservor data wth 294 vertcally averaged porosty data ad 2D sesmc velocty ad ampltude. A porosty classfcato ad regresso maps are geerated usg SVC/SVR ad the results are compared wth geostatstcal models. Itroducto Support Vector Maches (SVM) s a ew mache learg approach based o Statstcal Learg heory (Vapk-Chervoeks or VC-theory). VC-theory has a sold mathematcal backgroud for the depedeces estmato ad predctve learg from fte data sets. SVM s based o the Structural Rsk Mmsato 3

4 prcple, amg to mmse both the emprcal rsk ad the complexty of the model, provdg hgh geeralsato abltes. It ca be appled for regresso ad probablty desty fucto estmato ad hece t s sutable for solvg may reservor charactersato problems. SVM provdes o-lear classfcato by mappg the put space to hgh-dmesoal feature space usg kerel fuctos, where the maxmal separatg margs are costructed. Usg dfferet kerels we obta learg maches aalogous to the well-kow archtectures (e.g., RBF eural etworks, multlayer perceptros). he performace of the SVM ca be mproved by kerel modfcato a data-depedet way. It allows to buld very flexble models to solve wde varety of classfcato ad regresso tasks. I the preset study radal bass fucto kerel s maly used. By varyg SVM hyper-parameters (parameters that are tued by the user outsde the mache) t was possble to cover wde rego of possble solutos from overfttg to oversmoothg. he paper presets the revew ad cotemporary developmets of the advaced methodology based o Support Vector Maches (SVM) for the aalyss ad modellg of spatally dstrbuted formato. he methodology developed for the spatal data combes the power of SVM wth well kow geostatstcal approaches ad tools cludg exploratory data aalyss ad exploratory varography. We wll preset results usg a reservor data set wth 294 vertcally averaged porosty data. A porosty map s geerated usg SVM ad the results are compared wth geostatstcal models ad smulatos. he preset study develops the deas of adaptato of Support Vector Maches to spatal data preseted (Kaevsk et al 999, Kaevsk ad Cau 2000). utorals, publcatos, software, data, lst o SVM applcatos (cludg refereces o speach recogto, patter recogto ad mage classfcato, obect detecto, fucto approxmato ad regresso, boformatcs, tme seres predctos, data mg, etc.) ca be foud o ( 200). 2 Support Vector Maches Classfcato Let us preset short descrpto of SVM applcato to the classfcato problems. Detaled theoretcal presetato of the SVM ca be foud (Burgess 998 ad Vapk 998) o whch the presetato below s based. radtoal troducto to the SVM classfcato s the followg: ) bary (2 class) classfcato of learly separable problem; 2) bary classfcato of learly o-separable problem, 3) o-lear bary problem 4) geeralsatos to the mult-class classfcato problems. Frst results o applcato of Support Vector Classfers (bary classfcato of polluto data, mult-class classfcato of evrometal sol types data) ca be foud (Kaevsk et al 999, Kaevsk et al 2000a,b). he followg problem s cosdered. A set S of pots (x ) s gve R 2 (we are workg a two dmesoal x = [x, x 2 ] space). Each pot x belogs to 4

5 ether of two classes ad s labeled by y {-,+}. he obectve s to establsh a equato of a hyper-plae that dvdes S leavg all the pots of the same class o the same sde whle maxmsg the mmum dstace betwee ether of the two classes ad the hyper-plae maxmum marg hyper-plae. Optmal hyper-plae wth the largest margs betwee classes s a soluto of the costraed optmsato problem cosdered below. 2. Learly separable case 2 Let us remd that data set S s learly separable f there exst W R, b R such that, Y ( W X + b) +, =,... N () he par (W,b) defes a hyper-plae of equato ( W X + b ) = 0 Learly separable problem: Gve the trag sample {X, Y } fd the optmum values of the weght vector W ad bas b such that they satsfy costrats Y ( W X + b) +, =,... N (2) Ad the weght vector W mmses the cost fucto (maxmsato of the margs) F( W ) = W W / 2 (3) he cost fucto s a covex fucto of W ad the costrats are lear W. hs costraed optmzato problem ca be solved by usg Lagrage multplers. Lagrage fucto s defed by L( W, b, α) = W X / 2 α N = where Lagrage multplers α 0 [ Y ( W X + b) ] he soluto of the costraed optmsato problem s determed by the saddle pot of the Lagraga fucto L( W, b, α) whch has to be mmsed wth respect to W ad b ad to be maxmsed wth respect to α. Applcato of optmalty codto to the Lagraga fucto yelds W = N = α Y X (4) 5

6 N = α Y = 0 (5) hus, the soluto vector W s defed terms of a expaso that volves the N trag data. Because of costraed optmsato problem deals wth a covex cost fucto, t s possble to costruct dual optmsato problem. he dual problem has the same optmal value as the prmal problem, but wth the Lagrage multplers provdg the optmal soluto. he dual problem s formulated as follows: maxmse the obectve fucto N N Q( α ) = α (/ 2) α α Y Y X X (6) = = Subect to the costrats N = α Y = 0 (7) α 0, =,...N (8) Note that the dual problem s preseted oly terms of the trag data. Moreover, the obectve fucto Q(α) to be maxmsed depeds oly o the put patters the form of a set of dot products {X X } =,2, N. After determg optmal Lagrage multplers α 0, the optmum weght vector s defed by (4) ad the bas s calculated as follows b = W X S, for Y ( s) = + Note that from the Kuh-ucker codtos t follows that [ Y ( W X + b) ] = 0 α (9) Oly α that ca be ozero ths equato are those for whch costrats are satsfed wth the equalty sg. he correspodg pots X, called Support Vectors, are the pots of the set S closest to the optmal separatg hyper-plae. I may applcatos umber of support vectors s much less that orgal data pots. he problem of classfyg a ew data pot X s smply solved by computg F( X ) = sg( W X + b) (0) wth the optmal weghts W ad bas b. 6

7 2.2 SVM classfcato of o-separable data: Soft marg classfer I case of learly o-separable set t s ot possble to costruct a separatg hyper-plae wthout allowg classfcato error. he marg of separato betwee classes s sad to be soft f trag data pots volate the codto of lear separablty ad the prmal optmsato problem s chaged by usg slack varables. Problem s posed as follows: gve the trag sample {X,Y } fd the optmum values of the weght vector W ad bas b such that they satsfy costrats Y ( W X + b) + ξ, ξ 0, () he weght vector W ad the slack varables ξ mmse the cost fucto N F( W ) = W W / 2 + C ξ (2) = where C s a user specfed parameter (regularsato parameter s proportoal to /C). he dual optmsato problem s the followg: gve the trag data maxmse the obectve fucto (fd the Lagrage multplers) N N Q( α ) = α (/ 2) α α Y Y X X (3) = = Subect to the costrats (7) ad 0 α C, =,...N (4) Note that ether the slack varables or ther Lagrage multplers appear the dual optmsato problem. he parameter C cotrols the trade-off betwee complexty of the mache ad the umber of o-separable pots. he parameter C has to be selected by the user. hs ca be doe usually oe of two ways: ) C s determed expermetally va the stadard use of a trag ad testg data sets, whch s a form of re-samplg; 2) It s determed aalytcally by estmatg VC dmeso ad the by usg bouds o the geeralsato performace of the mache based o a VC dmeso (Vapk 998). 2.3 SVM o-lear classfcato I most practcal stuatos the classfcato problems are o-lear ad the hypothess of lear separato the put space s too restrctve. he basc dea of Support Vector Maches s ) to map the data to a hgh dmesoal feature space (possbly of fte dmeso) va a o-lear 7

8 mappg ad 2) costructo of a optmal hyper-plae (applcato of the lear algorthms descrbed above) for separatg features. he frst tem s agreemet of Cover s theorem o the separablty of patters whch states that put multdmesoal space may be trasformed to a ew feature space where the patters are learly separable wth hgh probablty, provded: ) the trasformato s o-lear; 2) the dmesoalty of the feature space s hgh eough (Hayk 999). Cover s theorem does ot dscuss the optmalty of the separatg hyper-plae. By usg Vapk s optmal separatg hyper-plae VC dmeso s mmsed ad geeralsato s acheved. Let us remd that the lear case the procedure requres oly the evaluato of dot products. ϕ ( x) deote a set of o-lear trasformato from the put Let { } =,... m space to the feature space; m s a dmeso of the feature space. No-lear trasformato s defed a pror. I the o-lear case the optmsato problem the dual form s followg: gve the trag data maxmse the obectve fucto (fd the Lagrage multplers) N N Q( α ) = α (/ 2) α α Y Y K( X X ) (5) = = Subect to the costrats (7) ad (4). he kerel (5) s m = K( X, Y) = ϕ ( X ) ϕ( Y) = ϕ ( X ) ϕ ( Y ) (6) hus, we may use er-product kerel K(X,Y) to costruct the optmal hyperplae the feature space wthout havg to cosder the feature space tself explct form. he optmal hyper-plae s ow defed as N f ( X ) = α Y K( X, X ) + b (7) = Fally, the o-lear decso fucto s defed by the followg relatoshp: [ K( X, X + b] F( X ) = sg W ) (8) he requremet o the kerel K(X, X ) s to satsfy Mercer s codtos (Vapk 998). hree commo types of Support Vector Maches are wdely used: Polyomal kerel K ( X, X ) ( + ) p = X X (9) 8

9 where power p s specfed a pror by the user. Mercer s codtos are always satsfed. Radal bass fucto RBF kerel s defed by 2 { X / 2 } 2 K( X, X ) = exp σ (20) X Where the kerel badwdth σ (sgma value) s specfed a pror by the user. I geeral, Mahalaobs dstace ca be used. Mercer s codtos are always satsfed. wo-layer perceptro { β X X + } K ( X, X ) tah β (2) = 0 0 Mercer s codtos are satsfed oly for some values of β 0 β. For all three kerels (learg maches), the dmesoalty of the feature space s determed by the umber of support vectors extracted from the trag data by the soluto to the costraed optmsato problem. I cotrast to RBF eural etworks, the umber of radal bass fuctos ad ther cetres are determed automatcally by the umber of support vectors ad ther values. I the preset study oly the results obtaed wth the RBF kerel are preseted. 2.4 Mult-class classfcato If there s a bary classfer, the mult-class (M class) classfcato problem ca be solved by the dfferet reductos of prmary problem to several dchotomes. (Mayoraz ad Alpayd 998, Westo ad Watks 998, Vapk 998). he most evdet method s oe-to-rest or oe-agast-all classfcato whe M bary classfcato models, oe per each class s developed. hus, M decso fuctos are derved, oe for each class. Fal classfcato label for valdated pot s assged by y arg max y K ( x, x ) + b ( m ) ( m ) = λ (22) m Secod possblty s par-wse classfcato whe M(M-) bary classfcato models are developed. Aother way s drect geeralsato of SVM to M-class problems. he ma dsadvatage of ths method s that the QPproblem sze becomes very large. Oe-to-rest ad par-wse schemes seem to gve satsfactory results the geostatstcal applcatos. 3 Spatal Data Mappg wth Support Vector Regresso Assume z R s a varable to be predcted based o some geographcal observatos (x,y). Our work ams at estmatg a depedece betwee z ad the 9

10 geographcal co-ordates based o emprcal data (samples) S =(x,y,z,ε ), =,, where x,y, - are the geographcal co-ordates of samples z - s the observed or measured quatty. It s assumed to be the realsato of a radom varable Z wth a ukow probablty dstrbuto P x,y (Z). ε - s the measuremet accuracy for the observato z deotes the sample sze 3. Predcto problem 3.. he ε-sestve cost fucto Assumg f s a predcto fucto (.e. a fucto used to predct the value of Z kowg the geographcal co-ordates), we defe the cost of choosg ths partcular fucto for a gve decso process. Frst, for a gve observato (x,y,z) we defe the ε-sestve cost fucto: f ( x, y) z ε f f ( x, y) z > ε C {( x, y), z, ε, f } = (23) 0 otherwse where ε characterses some acceptable error. Now, for all possble observatos we defe the global or geeralsato error also kow as the tegrated predcto error IPE: IPE( f ) = E ( C(( x, y,), z, ε, f )) ω( x, y) dxdy (24) Z where ω(x,y) s some exteral measure, dcatg the relatve mportace of a mstake at pot (x,y). I case of o-homogeeous motorg etworks ths fucto ca take to accout spatal clusterg. Usually ω(x,y) =, so that all postos are assumed to be equally mportat. Our approach s a cost drve modellg. For the ε-sestve cost fucto t s possble to compute the best predcto fucto (.e. the oe mmsg the IPE). For ω(x,y) =, ths target fucto s such that: x, y ( Z) dz = Px, y ( Z) z< r ( x, y) ε z r( x, y) + ε P dz (25) hs fucto equlbrates the tals of the dstrbuto. For r(x,y) s the codtoal meda fucto. ε = 0 soluto 0

11 3..2 No symmetrcal cost fucto he same calculato ca be doe for asymmetrc cost fucto. For some practcal applcato, t may appear that the errors uder a certa level are ot as much mportat as the errors above (over-estmatos ad uder-estmatos are ot equvalet). I ths case the cost fucto should be the followg C d a( f ( x, y) z ε a ) (( x, y), z, ε, f ) = b( z f ( x, y) ε u ) 0 f (f(x, y) - z) (f(x, y) - z) otherwse > ε < ε u a where a ad b are parameters cotrollg the asymmetry of the cost fucto. I ths case r s (x,y) the target fucto mmsg the IPE s defed from the followg relatoshp: bpx, y ( Z) dz = apx, y ( Z) z< r + s ( x, y) ε l z rs ( x, y) ε a It equlbrates the weghted tals. Other robust cost fuctos are detaled (Vapk, 998, chapter ). dz 3.2 Emprcal Rsk Mmsato ad Structural Rsk Mmsato 3.2. Fucto Modellg Let us assume ths soluto s a fucto that ca be decomposed to two dfferet compoets: a tred plus a remag radom process. A ce way to take to accout ths pror, s to look for the soluto a fuctoal space that ca be decomposed to two orthogoal subspaces, oe modellg the tred, whle the other oe deals wth the remag radom process. Assume H s such a Hlbert space. Assume K (x,y) s a bass of the tred compoet ad ϕ k, k=,..m s a orthoormal bass of the remag part (ote that m ca be fty) m f ( x, y) = wkϕ k ( x, y) + β K ( x, y) k = J = (26) he complexty of the soluto ca be tued through w 2 =Σ k=..m w k 2 (Vapk 998). hus, a relevat strategy to mmse IPE s to mmse the emprcal error together wth matag w 2 small. hs ca be obtaed by mmsg the followg cost fucto:

12 2 mmze w 2 subect to f ( x, y ) - Z ε, for =,... (27) But, ufortuately, some data may le outsde of ths epslo tube due to ose or outlers makg these costrats too strog ad mpossble to fulfl. I ths case Vapk suggests to troduce so called slack varables ξ, ξ *. hese varables measure the dstace betwee the observato ad the ε tube (see the example Fgure 2.). he dstace betwee the observato ad the ε ad ξ, ξ * s llustrated by the followg example: mage you have a great cofdece your measuremet process, but the varace of the measured pheomea s large. I ths case, ε has to be chose a pror very small whle the slack varables ξ, ξ * are optmsed ad thus ca be large. Remember that sde the epslo tube ([f(x,y)- ε, f(x,y)+ ε ]) cost fucto s zero. Fgure. Support vector regresso. Explaato of the ε tube ad slack varables. Note that by troducg the couple (ξ, ξ * ) the problem has ow 2 ukow varables. But these varables are lked sce oe of the two values s ecessary equals to zero. Ether the slack s postve (ξ * = 0) or egatve (ξ = 0). hus, z [f(x,y)- ε -ξ, f(x,y)+ ε +ξ * ]. Now, we are lookg for a soluto mmsg at the same tme ts complexty (measured by w 2 ) ad ts predcto error (represeted by max (ξ, ξ * )= ξ + ξ * ). I ths case, let us troduce a user specfed trade off parameter C betwee these two cotradctory obectves. hat leads us to the followg problem: 2

13 mmse 2 * ω + C ( ξ + ξ ) 2 subect to = f ( x, y ) Z ε ξ * f ( x, y ) + Z ε ξ * ξ, ξ 0 for =,... (28) Dual formulato A classcal way to reformulate a costrat based mmsato problem s to look for the saddle pot of Lagraga L: * L( w, ξ, ξ α 2 2 * ) = w + C( ξ + ξ ) α ( = = * * α ( f ( x, y ) Z + ε + ξ ) = = Z f ( x, y ) + ε + ξ ) * * ( η ξ + η ξ ) * * where α, α, η, η are Lagraga multplers assocated wth the costrats. hey ca be roughly terpreted as a measure of the fluece of the * costrats the soluto. A soluto wth α = α = 0 ca be terpreted as the correspodg data pot has o fluece o ths soluto. At the mmum the dervatve of the Lagraga equals to zero (Kuh-ucker codtos). hus t ca be checked that: w * = ( α α ) ϕ ( x, y ) k k = η = C α * * = C α η for =,..., for =,..., for k =,...m hese varables ca be removed from the orgal formulato of the mmsato problem to get the dual formulato of the problem: maxmse - 2 subect to = = = = m * * ( α α ) ϕ ( x, y k ) ϕ k ( x, y ) ( α α ) * * ε ( α + α ) + Z ( α α ) * ( α α ) K ( x, y ) = 0 for K 0 α, α C * k = = for,... =,... m 3

14 3.2.3 he ature of the soluto o solve the problem wthout specfyg fuctos ϕ k t s ecessary to choose ϕ k such that: m ( ) ϕ k ( x, y ) ϕ k ( x, y ) = G ( x, y ),( x, y ) (29) k = hs s the case reproducg kerel Hlbert space, where G s the reproducg kerel. Fuctos ϕ k are the ege fuctos of G. I ths case the soluto ca be formulated the followg form: f ( x, y) = w G(( x, y),( x, y )) + β K ( x, y) (30) = = * wth w = ( α α ). Note that the fucto ϕ k has dsappeared. hs soluto oly depeds o the kerel fucto G. Note also that here at least oe of alphas s equalled to zero depedg of the observed value z, above or uder the ε-tube. Remark: the soluto proposed equato (30) s the same as the regresso sple ad krgg estmates (sce they are postve defte ad reproducg kerels ca be terpreted as covarace fucto (Wahba 990). he dfferece betwee these methods les the uderlyg hypotheses ad thus the way weghts (29) are estmated. I the SVR framework the regularsato s ot performed o w but o the represetato of the fucto some feature space. hs s a way to defe a regularsato prcple that guaratees a explct boud o the IPE error. From the practcal pot of vew, due to L type mmsato, may of the w ca be ether zero or C. w s zero whe assocated measuremet pot les wth the ε-tube ad thus has o fluece o the estmato. hs pot s useless for the estmato ad ca be removed wthout chagg the result. w s equals to C whe the assocated measuremet pot s too far from the ε-tube. I ths case, the fluece of the pot s bouded at C. Aother way to formulate ths remark s to establsh the lk betwee SVR ad sparse approxmato (Gros 998). m Kerel choce As the case of classfcato the practcal choce for the kerel s the Gaussa kerel: = + σ (3) where σ deotes the badwdth of the kerel. I ths case = ad the tred fucto K s a costat. 4

15 3.2.5 Hyper parameters For practcal mplemetato the hyper parameters of the method have to be tued. hese parameters are the followg: C: although ofte recommeded as very large, geostatstcal applcatos show a great deal of depedece o ths parameter. It has to be tued carefully. ε : f o addtoal formato s avalable the easest way to tue t s to put t small comparso to stadard devato of data. See below detals o fluece of the epslo o trag ad mappg. I geeral, t ca be related to error measuremets ad/or small scale varatos ot resolved by samplg etwork usually descrbed by ugget effect varogram. σ: the badwdth of kerel. Here aga the IPE of the proposed soluto s very sestve to ths parameter. More geerally, the performace of the soluto s sestve to the dstace matrx used the kerel 4 Case studes. Descrpto of data Let us lst the ma phases (steps) of the classfcato/regresso studes appled by usg SVC/SVR:. Vsualsato of data. Motorg etwork aalyss ad descrpto. Uderstadg of data clusterg. 2. Exploratory data aalyss. Uvarate statstcal aalyss, outlers detecto, data trasformato ad data pre-processg, tred detecto, etc. 3. Exploratory structural aalyss (varography). Uderstadg ad modellg of spatal correlatos. 4. Splttg data to data sets: rag, estg, Valdato. 5. rag of SVC/SVR wth dfferet models. Selecto of the optmal SVM hyper-parameters. 6. Patter completo (categorcal data mappg). Regresso, spatal predctos of cotuous varable. 7. Statstcal aalyss ad varography of the resduals. 8. Uderstadg ad terpretato of the results. 9. Coclusos. Because of the large dffereces magtude, both porosty ad co-ordate values were re-scaled to betwee zero ad oe before ay calculatos were performed. All mappg ad classfcato results wll thus be preseted usg such re-scaled values; however,, t s uderstood that the orgal raw values ca be obtaed by performg a smple back-trasform. Batch statstcs ad data post plots are preseted below. I the preset paper two case studes are cosdered detal: Bary classfcato of porosty data. o pose ths problem orgal cotuous data were trasformed to low ad hgh level of porosty. Idcator cut correspods to the level of 0.5 (about mea value): porosty data hgher/less 5

16 tha 0.5 are coded as class + ad -. he results of SVC bary classfcato are compared wth dcator krgg. he geeralsato of the bary task s a mult class classfcato problem (Mayoraz ad Alpayd 998, Westo ad Watks 998, Kaevsk et al. 2000b). Revew o geostatstcal approach for spatal data classfcato ca be foud (Atkso ad 2000). Spatal predctos/mappg of porosty data. Support Vector Regresso model s developed for the spatal predctos of cotuous porosty data. Results of the SVR mappg are compared wth ordary krgg. From the begg orgal data were splt several tmes to two data sets: 200 ad 94 measuremets. he frst data set was used to develop SVM models (trag data set) ad the secod oe (valdato data set) was used to valdate the results. Because motorg etwork s ot clustered, radom splttg was used ( case of clustered motorg etworks spatal declusterg procedures ca be used to have represetatve testg data set). Aother proportos betwee data sets were used as well. Batch statstcs of the etre data set (294 measuremets): mmum = 0.0; Q /4 = ; meda = 0.55; Q 3/4 = 0.69; max =.000e+00; mea = 0.53; varace = 0.048; skewess = 0.2; kurtoss = Post plots of trag ad valdato data sets are preseted Fgure 2. Fgure 2. Presetato of trag data set as area-of-fluece polygos. Post plot of testg ( + ) ad valdato ( O ) data sets. 6

17 A mportat phase of spatal data aalyss (despte of the methods used) deals wth descrpto of spatal cotuty usg exploratory varography (Chles ad Delfer 999). he most wdely used measure of spatal cotuty for the spatal fucto Z(x) s a semvarogram 2 { } { } ( ) γ ( x, h) = Var Z ( x) Z ( x + h) = E Z ( x) Z ( x + h) = γ ( h) 2 (32) where h s a separato vector betwee pots space. I case of trsc hypotheses semvarogram (varogram) depeds oly o separato vector betwee pars. he emprcal estmate of the semvarogram s gve by N ( h) γ ( h) = + 2 N ( h) ( Z ( x) Z ( x h) ) = where N(h) s a umber of pars separated by vector h. Varogram rose semvarogram computed for the dfferet separato vectors for the trag data s preseted Fgure 3. Geometrcal asotropy s preset the Northeast ad Southwest tredg drectos. 2 (33) Fgure 3. Varogram rose of trag data. I case of secod order statoary regoalzed radom fucto the relatoshp betwee covarace fucto C(h) ad varogram s followg: γ(h)=c(0)-c(h). 7

18 Behavour of the varogram ear the org at small dstaces descrbes the smoothess of the fucto ad characterses the relatoshp betwee radom ad spatally structured parts of formato. I the preset study the varography s wdely used to cotrol the qualty of models performace. 4. Classfcato of reservor data I the preset paper oly the bary classfcato problem s cosdered. Orgal data were trasformed to dcators (2 classes) ad splt to trag, testg ad valdato data set used to develop a model, to tue hyper-parameters (kerel badwdth ad regularsato parameter C) ad to valdate the model. he splttg was performed several tmes dfferet proportos. 4.. Bary classfcato wth Support Vector Maches he partcular case of data splttg to trag (cludes 50 trag ad 50 testg data pots) ad valdato data (94 data pots) sets s preseted Fgure 4. he problem s clearly o-lear. Valdato data represets dfferet regos classes. Fgure 4. Bary (2 classes) classfcato problem. + post plot of valdato data. I order to fd optmal hyper-parameters comprehesve search was carred out by computg trag ad testg error surfaces depedg o kerel badwdth ad C parameter. he optmal choce s the oe wth low values of trag ad testg errors ad small values of Support Vectors. 8

19 he behavour of the error surfaces s followg: rag error s small ad eve zero the rego of small kerel badwdths overfttg rego. All data pots are mportat (overfttg) ad are Support Vectors: I ths rego geeralsato s bad ad testg error s hgh. estg error ad umber of Support Vectors do ot deped o C parameter. For the trag error at hgher values of C overfttg s acheved at larger values of kerel badwdths (see Fgures 5-7). I the rego of hgh values of kerel badwdths (comparable wth the scale of the rego) there s a oversmoothg. rag error s hgh ad testg error after reachg some mmum at optmal termedate values of badwdth s also creasg. I ths rego the umber of Support Vectors s also slowly creasg. A optmal rego s reached at termedate values of kerel badwdth ad C parameter. I our case the optmal parameters were the followg: kerel badwdths about 0. ad C=0. Fgure 5. SVM bary classfcato. Estmate of trag error surface. he classfcato soluto wth the optmal hyper-parameters s preseted Fgure 8. Valdato data are post plot as well. I the followg secto of the paper the same problem s solved wth dctor krgg. Let us remd that the classcal output of SVC s determstc classfcato, case of dcator krgg output s a probablty map to be above or below the threshold. 9

20 Fgure 6. SVM bary classfcato. Estmate of testg error surface. Fgure 7. SVM bary classfcato. Number of Support Vectors surface. 20

21 Fgure 8. SVM optmal classfcato alog wth valdato data post plot. Flled crcles belog to the valdato data of whte zoe class, empty crcles belog to valdato data of coloured class. Number of Support Vectors ( + ) equals 56. hus, order to make classfcato, SVC eeds oly 56 data pots (they are Support Vectors). rag error was 4.6%, testg error = 8% ad valdato error = %. Oly at the border of decso surface where there s the bggest ucertaty classfcato, SVC has some problems wth classfcato of valdato data. I fact, t should be take to accout that data ca be cotamated by ose ad t s ot ecessary to follow exactly trag ad valdato classes for the partcular realsato of the regoalzed fucto Bary classfcato wth dcator krgg I order to compare the results of SVM bary classfcato wth geostatstcal approach dcator krgg was used. Idcator krgg s a krgg appled to the dcator trasformed data: = (34) where z k s a threshold. I terms of probablty dcator ca be represeted as 2

22 { } { } E I( x, z ) = P Z( x) z = F( z ) (35) k k k hus, the output of the dcator krgg spatal predctos s terpreted as a probablty to be below threshold. It gves a probablstc terpretato of the bary classfcato problem. he dcator krgg s a BLUE Best Lear Ubased Estmator appled to the dcators (Deutsch ad Jourel 997). he basc equatos of the dcator krgg wrtte terms of covarace fucto are followg: F IK (, z { }) = λ I (, z ) (36) k = k λ k C µ α β I ( x β xα, zk ) + = k CI ( x xα, zk ), =,..., (37) β = λk 0 β β = = (38) After exploratory varography based o data, covarace fuctos/varograms should be modelled. hs s performed by fttg the theoretcally vald models to the expermetal oes. he results of dcator krgg are preseted Fgure 9 alog wth valdato data post plot. k Fgure 9. Results of dcator krgg (probablty to belog to class O ) alog wth valdato data post plot. 22

23 he output of dcator krgg s a probablstc map to be above or below threshold. I our case the terpretato s to belog to oe or aother class. he soluto of dcator krgg s more varable, because of exacttude propertes of IK the soluto follows trag data pots. he same kd of soluto ca be obtaed by SVC by reducg kerel badwdth movg to overfttg rego. Aother commets s related to asotropy. I case of SVC sotropc kerel was used. I case of IK asotropc varogram model was developed takg to accout asotropc spatal correlatos. Next step the developmet of SVC deals wth the mplemetato of asotropc kerels ad/or pre-processg of data (e.g., co-ordates trasformatos). Fally, other kerels ca be appled as well (see Vapk 998, where wde choce of kerels s preseted). 4.2 Support Vector Regresso I the preset secto the problem of reservor data mappg spatal regresso usg SVR s cosdered SVR rag I case of SVR there are three hyper-parameters ad error cubes should be aalysed to fd the optmal soluto. Comprehesve search a 3D hyperparameter space was performed. Some 2D errors surfaces wth fxed C parameter are preseted Fgures 0-2. he same dscusso as the case of classfcato cocerg overfttg ad oversmoothg regos s applcable as well. he optmal parameters were chose takg to accout trag ad testg errors, umber of Support Vectors. Fgure 0. Estmate of SVR trag error surface. C=

24 Fgure. SVR testg error surface. C= Fgure 2. Surface of the umber of Support Vectors. C= A mportat phase of the trag procedure deals wth uderstadg how much useful formato was extracted by SVR from data ad what s left. I terms of spatal data ad geostatstcs useful formato s spatally structured formato. Spatal structures are descrbed bascally by varograms. hat s why 24

25 varographc tools are effcet to uderstad ad to expla the results. I the preset study they were used to cotrol the performace of SVR SVR Mappg he two partcular results of Support Vector Regresso mappg are preseted Fgures 3 ad 4. It should be oted that by varyg hyper-parameters t was possble to develop models of very dfferet complexty, coverg regos from overfttg to oversmoothg. A terestg oversmoothg case deals wth large scale modellg so called detredg. No-learty ad flexblty of SVR hghly smplfes detredg problem. he qualty of detredg ca be cotrolled wth geostatstcal tools, cludg varography. Actually, herarchy of SVR models ca be developed to extract asotropc formato from data at dfferet scales ad dfferet regos. Oe possblty could be mxtures of SVR, aother oe local SVR models. A mportat questo, ot elaborated ths paper, deals wth fluece of data pre-processg: lear ad o-lear trasformatos of spatal co-ordates ad data. It seems that case of asotropc structures data pre-processg ca make them more sotropc ad less Support Vectors wll be ecessary, perhaps leadg to better geeralsato propertes. hs problem should be studed wth a well defed smulated data sets. Fgure 3. SVR porosty mappg. Kerel badwdth = 0., epslo parameter = 0.0, all trag data are Support Vectors ( O ). 25

26 Fgure 4. SVR porosty mappg. Kerel badwdth = 0., e parameter = 0.08, the umber of Support Vectors ( O ) equals 50. he results o valdato data by usg SVR(C=0000, kerel badwdth = 0., ε = 0.08) are preseted Fgure 7. Let us remd that oly 50 (!) data (Support Vectors) were used to get almost the same qualty of the model as OK. Here we ca pose a terestg questo about the use of SVR motorg etwork desg ad redesg. he methodologcal work ths drecto should be related to the developmets of correspodg obectve fuctos. Let us remd that case of OK krgg varace s ofte used to optmse motorg etwork. A aalogue of estmato varace ca be derved for the SVR based o the trag resduals. hs approach was appled wth Geeral Regresso Neural Networks (Kaevsk 999) Geostatstcal Mappg. Ordary Krgg Ordary krgg OK was used as a geostatstcal model for the porosty mappg. Ordary krgg s a BLUE model also based o the aalyss ad modellg of spatal correlato structures varography ad s descrbed by the followg system of equatos ( umber of data measuremets): 26

27 * Z ( x ) = 0 w = = w = = w ( x ) Z( 0 x γ µ γ 0 =,, ) I accordace wth geostatstcal methodology deep structural aalyss exploratory varography, ad modellg were carred out. he ma atteto durg varogram fttg was pad to the drectos whch drft s eglgble. Geostat Offce was used at all stages of geostatstcal aalyss ad modellg. he result of ordary krgg mappg of porosty data s preseted Fgure 5. he same OK model was used to estmate valdato data. he results of the valdato for SVR models ad OK are preseted Fgure 6. hey are qute good. Fgure 5. Porosty mappg wth ordary krgg. 27

28 Fgure 6. Valdato results. SVR ad ordary krgg. he qualty of mappg ca be qualtatvely descrbed by omdrectoal varograms of the resduals ( see Fgure 7.). SVR trag resduals demostrate pure ugget effect all spatally structured formato was extracted by SVR model from data. Nugget effect of the trag resduals correspods to the ugget effect of raw data. he varograms of the valdato resduals both of OK model ad SVR have pure ugget effect as well. It meas good results o valdato data. Fgure 7. Omdrectoal varograms of raw data, SVR trag resduals, SVR valdato resduals, krgg valdato resduals. 28

29 5 Cocluso he paper presets adaptato of the SVM algorthms Support Vector Classfcato ad Support Vector Regresso to the spatally dstrbuted reservor data. wo problems were cosdered detal: ) bary classfcato of spatal categorcal data ad 2) spatal regresso/mappg of porosty data. he basc deas of SVM trag by usg errors surfaces was demostrated. I was show that ear the optmal soluto the umber of Support Vectors s rather low that s a good dcato for low geeralsato/valdato error. he obtaed results are promsg that was demostrated wth valdato data both cases. he results were compared wth geostatstcal approach dcator krgg case of classfcato ad ordary krgg case of regresso. he future developmets of the preset work deal wth the study of kerel types (polyomal, MLP- lke, sples, etc.) o the trag procedures ad fal results. A mportat ssue s related the problems of estmato of predcto varace (lke krgg varace geostatstcs). hs problem ca be solved partly by usg trag resduals. Fally, a geeralsato of the SVM to the multvarate case, whe qualty ad quatty of formato o dfferet varables dffer s of great mportace for wder applcato of SVM approach to evrometal data. 6 Ackowledgemets he work was supported part by INAS grats ad Geostat Offce software tools were used for the exploratory data aalyss, varography ad presetato of the results. 7 Refereces Atkso P. M., ad Lews P. Geostatstcal classfcato for remote sesg: a troducto. Computers ad Geoseces, vol. 26 pp , Burgess C. A tutoral o Support Vector Maches for patter recogto. Data mg ad kowledge dscovery, 998. Cherkassky V ad F. Muler. Learg from data. Wley Iterscece, N.Y. 998, 44 p. Crsta N. ad Shawe-aylor J. A Itroducto to Support Vector Maches ad other kerel-based learg methods. Cambrdge Uversty Press, pp. Deutsch C.V. ad A.G. Jourel. GSLIB. Geostatstcal Software Lbrary ad User s Gude. Oxford Uversty Press, New York, 997. Glard N. Kaevsk, E Mayoraz, M Maga. Spatal Data Classfcato wth Support Vector Maches. Accepted for Geostat 2000 cogress. South Afrca, Aprl Gros F. A equvalece betwee sparse approxmato ad support vector maches. Neural Computato, 0(), pp , 998. Hayk S. Neural Networks. A Comprehesve Foudato. Secod Edto. Macmlla College Publshg Compay. N.Y.,

30 Kaevsk M., N. Glard, M. Maga, E. Mayoraz. Evrometal Spatal Data Classfcato wth Support Vector Maches. IDIAP Research Report. IDIAP-RR-99-07, 24 p., 999. ( Kaevsk M. Spatal Predctos of Sol Cotamato Usg Geeral Regresso Neural Networks. It. J. o Systems Research ad Iformato Systems, Volume 8, umber 4. Specal Issue: Spatal Data: Neural ets/statstcs. Guest Edtors Dr. Patrck Wog ad Dr. om Gedeo.Gordo ad Breach Scece Publshers pp Kaevsk M.. S. Cau. Spatal Data Mappg wth Support Vector Regresso. IDIAP Reasearch Report; RR a. ( Kaevsk M., A. Pozdukhov, S. Cau, M. Maga. Advaced spatal data aalyss ad modellg wth Support Vector Maches. IDIAP Research Report, RR-00-3, 2000b. Kaevsk M, V. Demyaov, S. Cherov, E. Saveleva, A. Serov, V. mo, M. Maga. Geostat Offce for Evrometal ad Polluto Spatal Data Aalyss. Mathematsche Geologe, N3, Aprl 999, pp Mayoraz E. ad E. Alpayd Support Vector Mache for Multclass Classfcato, IDIAP- RR 98-06, 998 ( Westo J., Watks C. Mult-class Support Vector Maches. echcal Report CSD-R , 9p, 998. Vapk V. Statstcal Learg heory. Joh Wley & Sos, 998. Wahba G. Sple Models for Observatoal Data. No. 59 regoal coferece seres appled mathematcs, SIAM Phladelpha, Pesylvaa