Nonlinear Robust Regression Using Kernel Principal Component Analysis and R-Estimators

Transcription

1 IJCSI Internatona Journa of Comuter Scence Issues, Vo. 8, Issue 5, o, Setember 0 ISS (Onne: onnear Robust Regresson Usng Kerne rnca Comonent Anayss and R-Estmators Anton Wbowo and Mohammad Ishak Desa acuty of Comuter Scence and Informaton Systems Unverst eknoog Maaysa, 830 UM Johor Bahru, Johor Maaysa Abstract In recent years, many agorthms based on kerne rnca comonent anayss (KCA have been roosed ncudng kerne rnca comonent regresson (KCR. KCR can be vewed as a non-nearzaton of rnca comonent regresson (CR whch uses the ordnary east squares (OLS for estmatng ts regresson coeffcents. We use CR to dsose the negatve effects of mutconearty n regresson modes. However, t s we known that the man dsadvantage of OLS s ts senstveness to the resence of outers. herefore, KCR can be narorate to be used for data set contanng outers. In ths aer, we roose a nove nonnear robust technque usng hybrdzaton of KCA and R-estmators. he roosed technque s comared to KCR and gves better resuts than KCR. Keywords: Kerne rnca comonent anayss, kerne rnca comonent regresson, robustness, nonnear robust regresson, R-estmators.. Introducton Kerne rnca comonent anayss (KCA has been roosed to be used for nonnear systems by mang an orgna nut sace nto a hgher-dmensona feature sace, see [3, 9, 7, 8] for the detaed dscusson, and becomes an attractve agorthm because t does not nvove nonnear otmzaton, t s as sme as the rnca comonent anayss (CA and t does not need to secfy the number of rnca comonents ror to modeng comared to other nonnear methods. In recent years, many nonnear agorthms based on KCA have been roosed ncudng kerne rnca comonent regresson (KCR whch can be vewed as a non-nearzaton of rnca comonent regresson (CR and dsose the effects of mutconearty n regresson modes [6, 8, 4, 5, 6, ]. However, KCR was constructed based on the ordnary east squares (OLS for estmatng ts regresson coeffcents whch was senstve to the resence of outers. An observaton s caed outer f t does not foow the OLS based near regresson mode. When we use OLS to estmate regresson coeffcents then outers have a arge nfuence to the redcton vaues snce squarng resduas magnfes the effect of the outers. herefore, KCR can be narorate to be used when outers are resent. In revous years, severa technques have been deveoed to overcome the negatve effects of outers such as R-estmators whch was a robust method based on the ranks of the resduas [7, ]. However, the revous works aed t for tackng the effect of outers n the near regresson mode. We shoud notce that the estmate of regresson coeffcents usng R-estmators s obtaned through sovng a nonnear otmzaton robem. o obtan the estmate of regresson coeffcents, we can use severa technques for sovng ths nonnear otmzaton robems such as genetc agorthms (GAs, smuated anneang and artce swarm otmzaton [, 4, 5,, 3,, 3]. However, ayng R-estmators n the ordnary regresson st yeds near modes whch have mtatons n acatons. In ths aer, we roose a nove robust technque usng hybrdzaton of KCA and R-estmators to overcome the mtaton of KCR and R-estmators n the near regresson. We use KCA to dsose the effects of mutconearty n regresson and to construct nonnearty of redcton mode by transformng orgna data nto a hgher-dmensona feature sace and erform a kerne trck to have a mute near regresson n ths sace. hen, we erform R-estmators n ths near regresson and sove the otmzaton robems of the R-estmators for obtanng a nonnear robust regresson. We refer the roosed method as the robust kerne rnca comonent R regresson (R-KCRR. We organze the rest of manuscrt as foows: Secton, we revew theores and methods of R-estmators and

2 IJCSI Internatona Journa of Comuter Scence Issues, Vo. 8, Issue 5, o, Setember 0 ISS (Onne: KCA, foowed by R-KCRR and ts agorthm. In Secton 3, we comare the erformance of the roosed method usng severa data sets. nay, concusons are gven n Secton 4. and ( = Θ ( /( (.5b a +. heores and Methods. R-Estmators whch are caed the Wcoxon and Van der Waerden scores, resectvey, wth Θ s the nverse of cumuatve norma dstrbuton functon. he ordnary mute near regresson mode s gven by y = Xβ + e (. ( ( ( C = / Ψ Ψ R and, ( ψ ( x ψ ( x ψ ( x... R ( x = 0. where ~ y = y y y, R X = x x x R ~ ( + x = (, = ( x x x R X X R wth s vector wth a eements equa to one, + β = β β β R s a vector of regresson ( 0 coeffcents, ( e = e e e R s vector of resduas and R s the set of rea numbers and =,,,. When we use OLS to fnd the estmate of β, say ˆβ, then the estmate s found by mnmzng = e, (. where x e y β x = x. he souton can be found by sovng the foowng near equaton = and ( ˆ X Xβ = X y (.3 However, t s we known that the redcton of the OLS based regresson w be dstorted when outers are resent. o overcome the resence of outers, we can use R-estmators whch mnmze. KCA Assume that we have a functonψ : R, where s the feature sace whch t s an Eucdean sace wth dmenson (. hen, we defne the matrces K = ΨΨ R where Ψ = and assume that = ψ he reaton of egenvaues and egenvectors of the matrces C and K were studed by Schokof et a. [8]. Let ˆ be the rank of Ψ where ˆ mn(, whch mes that both rank(k and rank( Ψ Ψ are equa to ˆ. It s evdent that the egenvaues of K are nonnegatve rea numbers snce the matrx K s symmetrc and ostve semdefnte []. Let λ λ... λ ~ λ~ λ + ˆ > r r λ ˆ + = = λ = 0 be the egenvaues of K and B = ( b b... b be the matrx of the corresondng normazed egenvectors b (s =,,, s of K. hen, et α = b λ and a = Ψ α for =,,, ˆ. he egenvectors a, however, cannot be found excty snce we do not know Ψ Ψ excty. However, we can obtan the rnca comonent of ψ ( x corresondng to nonzero egenvaues of Ψ Ψ by usng a kerne trck. he -th rnca comonent of ψ ( x ( =,,, ˆ as gven as foows: where = a R s the rank of ( R e (.4 e and ( a s a score functon whch s monotone and satsfes ( = 0. he common choce of a ( are = ( = ( / (.5a a + a ( x a = ( x ψ ( x (.6a ψ α ψ = where α s the -th eement of α. Accordng to Mercer heorem, f we choose a contnuous, symmetrc and ostve semdefnte kerne κ : R R R then there exsts ϕ : R such that κ ( x, x = ϕ j ( x ϕ( x j [0, 7]. Instead of choosng ψ excty, we choose a kerne κ and emoy the

3 IJCSI Internatona Journa of Comuter Scence Issues, Vo. 8, Issue 5, o, Setember 0 ISS (Onne: corresondng functon ϕ as ψ. Let K = κ x, x then K and α ( =,,, j ( j ˆ are excty known now. herefore, Eq. (.6a s aso excty known and can be wrtten as ( x a = κ( x, x (.6b ψ α =.3 onnear Robust Regresson Usng KCA and R-Estmators he centered mute near regresson n the feature sace s gven by U = ( U U and = ( ϑ ϑ ( ˆ ( ˆ ϑ, ( ˆ ( ˆ where szes of U ( ˆ, U ( ˆ, ϑ ( ˆ and ϑ ( ˆ are ˆ, ( ˆ, ˆ and ( ˆ, resectvey. he mode (3.3 can be wrtten as y U + ~. = e 0 ( ˆ ϑ ( ˆ + U( ˆ ϑ( ˆ (.9 It s easy to verfy that U ϑ U ϑ 0 ( ˆ ( ˆ ( ˆ ( ˆ = ( whch mes U ˆ ϑ ( ˆ s equa to 0. ( Consequenty, the mode (.9 reduces to y = Ψ γ + ~ 0 e (.7 y = U + ~. ( ˆ ϑ ( ˆ (.0 e 0 where γ = ( γ γ... γ s a vector of regresson coeffcents n the feature sace, ~ e s a vector of random errors and y = ( I ( / y where I s 0 the dentty matrx. Snce the rank of Ψ Ψ s equa to ˆ, then the remanng ( ˆ egenvaues of Ψ Ψ are zero. Let λ k ( k = ˆ, ˆ + +,..., be the zero egenvaues of Ψ Ψ and a be the normazed egenvectors of k Ψ Ψ corresondng to λk. urthermore, we defne A = ( a a... a. It s evdent that A s an orthogona matrx, that s, A = A -. It s not dffcut to verfy that where A Ψ ΨA = D D D = O ( ˆ O, O λ λ... 0 D = ( ˆ, λ ˆ and O s a zero matrx. By usng AA = I, we can rewrte the mode (.7 as y = Uϑ + ~ 0 e where U = ΨA and ϑ = A γ. Let (.8 where U = ΨA ( ˆ ( ˆ = KΓ( ˆ and Γ ( ˆ = ( α, α,, α. It s evdent that the eements of U ( ˆ are the rnca comonents of ψ ( x for =,,,. hen, f we ony use the frst rˆ ( ˆ, α ˆ vectors of α,,, mode (.0 becomes y + = U ε 0 r ( ϑ ˆ ( ˆ, (. r where ε = ( ε,,, ε ε s a vector of resduas nfuenced by drong the term U ( ˆ ϑ ( ˆ n mode (., resectvey. We usuay dsose of the term U ( ˆ ϑ ( ˆ for tackng the effects of mutconearty on the CA based regressons where the number rˆ s caed the retaned number of nonnear rnca comonents (Cs for the KCR. We can use the rato λ λ ( =,,, ˆ for detectng the resence of mutconearty on U (. If λ λ ˆr s smaer than, say < /000, then we consder that mutconearty exsts on U ( rˆ []. Let us consder mode (. agan. We can see that mode (. has the same structure wth mode (. whch mes that we can drecty ay R-estmators n mode (.. or ths urose, we defne rˆ U ( ˆ = ( u u r u R and obtan = y u ϑ. hen, we mnmze ε o ( rˆ = ( ~, (.5 a ε R o fnd the estmators of ϑ ( rˆ, where R ~ s the rank of ε, by usng a nonnear otmzaton sover.

4 IJCSI Internatona Journa of Comuter Scence Issues, Vo. 8, Issue 5, o, Setember 0 ISS (Onne: ˆ * Let ϑ ( r be the estmator of ϑ ( rˆ usng the above R- estmators. hen, the redcton vaue of y wth the frst rˆ vectors of α, α,, α usng R-estmators, say ˆ ~y, s gven by ˆ ~ * y = y + KΓ ( ˆ ϑ( ˆ (.6 r r and the resdua between y and y~ s gven by ~ ε = y ~. y (.7. Gven a vector x R, the redcton of the R- KCRR wth the frst rˆ vectors of α, α, α s gven by, ˆ g ( ˆ( x = y + d (,. r κ x x = We shoud note that ths agorthm works under the assumton ψ ( x = 0. When ψ ( x 0 = =, we reace K by K = K EK KE+EKE n Ste 4, where E s the matrx wth a eements equa to /. he redcton of the R-KCRR wth the frst rˆ vectors of α, α, α s gven by g, ˆ ( ˆ( x = y + d (, (.8 r κ x x = ˆ and g( r ˆ s a * where ( d d d = Γ( rˆ ϑ( rˆ functon from R nto R. We summarze the above rocedures of the R-KCRR as foows:. Gven ( y x x x for =,,...,.. Cacuate y = ( y and y = ( I ( /. 3. Choose a kerne κ : R R R and a functon a : R R. 4. Construct K j = κ( x, x j and K = ( K j. 5. Dagonaze K. Let rank(k = ˆ and λ λ... λ > r~ λ r~ λ + ˆ λ ˆ =... = λ = 0 be the egenvaues of K and + b b... b be the corresondng normazed egenvectors b (s =,,, of K. 6. Choose rˆ ( ˆ s 0 y and construct α = b λ for =,,, rˆ. hen, defne Γ ( = α, α,, α. rˆ ( rˆ y u ϑ o ˆr 7. Cacuate U ( rˆ = KΓ( rˆ and et ε = (. 8. Let R ~ be the rank of ε. 9. Sove robem (.5 usng a nonnear otmzaton sover and et ˆ * ϑ ( r be souton of ( Cacuate ( d d = Γ ˆ * ( ϑ(. d rˆ rˆ abe : Growth of the Son of the Count de Montheard. Age (yr, mth [day] Heght (cm Age (yr, mth [day] Heght (cm , , ,7[] 40., ,0 4.6,6 8.,6 4.9,0 90.0,0 49.9,6 9.8, , , , , , , , ,6[0] 69. 5,0.7 5,0[] ,7.7 5,6[8] , ,3[8] 8.4 6,6[9].9 6,6[6] , ,0[] , ,[9] , ,5[5] , ,7[4] , Case Studes 3. Data Sets We generated data sets from a trgonometrc functon and snc functon to test the erformances of KCR and R-KCRR. he generated data from the trgonometrc functon and snc functon are gven as foows: f ( x = 4.5sn(x +.5cos( x, (3. wth x [ π : 0.5: π ] and [ π : 0.: π ] x ;

5 IJCSI Internatona Journa of Comuter Scence Issues, Vo. 8, Issue 5, o, Setember 0 ISS (Onne: sn( x f ( x = 5 x f x 0 otherwse. (3. wth x [ 8 : 0.5: 8] and x [ 6 : 0.3: 6], resectvey. he notaton [ z : : z] stands for [ z, z +, z +,, z] where s a rea number. Generay, the generated data from the those functons can be wrtten as y ( = f x + e where =,,,. We aso generate y ( = f x + e where j =,,, t ; where t s a ostf nteger. he random noses e and e are rea numbers generated by a normay dstrbuted random wth zero mean and standard devaton σ and σ, resectvey, wth σ [, σ 0,]. or shake of comarsons, we set σ and σ equa to 0. and 0.3, and ca the set of {( x, } y and {( x, y } the tranng data set and the testng data set, resectvey. In addton, we aso used a subset of the famous set of observaton taken on the heght of the son of the Count de Montbeard between 959 and 977 [9]. Ony the frst ten years of data were used n ths anayss. he growth of son data are gven n the abe. In these data, we artfcay generate the testng data by the reaton x jt = 0. rand( + x and j y jt = 0.5 rand( + y j where rand( generates a random number whch s unformy dstrbuted n the nterva (0,. hen, we comare the erformance of the above methods usng the three data sets wth and wthout outers. or ths urose, we generated 00 sets of the tranng data and 00 sets of the testng data. urthermore, we use the mean absoute error (MAE to estmate the redcton error for the tranng data set whch s gven by MAE = ( y y~. (3.3 he MAE s aso used to redcton error of the testng data sets and denoted by MAEt. In ths case studes, outers are created artfcay by movng some ( x, y s and ( x, y s away from desgnated ocatons. We generate eght otenta outers for each of the frst, second and thrd data sets where the ostons of outers n x -drecton and x t - drecton are chosen randomy n the doman of x and doman of x, resectvey. he ostons of outers = n y -drecton and y t -drecton are randomy seected n nterva [ 0, 0] from the correct ostons of y and y, resectvey. 3. Resuts In these case studes, we used the Wcoxon and Van der Waerden scores for R-estmators and the standard genetc agorthm (GA for sovng the otmzaton robem of R-estmators. hen, we used the Gaussan kerne κ( x, z = ex( x z ρ wth arameter ρ s equa to fve for both KCR and R-KCRR. We nvoved the estmate of ϑ by usng KCR, say ( rˆ ϑ ˆ n the nta ouaton of GA. In the nta, ( ˆr ouaton, the -th gene of the other chromosomes (or canddate soutons of ϑ s randomy chosen by the ( ˆr formuae ( ( ϑ ˆ + 30 rand( 5 (3.4 ( r where ϑˆ ( r s the -th eement of ϑˆ ( r and =,,, ˆr. or the sake of comarsons, the numbers of ouaton, maxmum teratons, mutaton rate and seecton rate are 50, 000, 0. and 0.5, resectvey. or each chromosome n any ouaton we sort ε n the descendng order, say ε [ ] and ε[ ] ε[ ] ε[ ], to determne ts rank. hen, we defne c k = ε[ 0. k ] where k=,, 3, 4 and rank ε s gven by 5 4 ~ R 3 = f c ε f c ε < c f c 3 ε < c f c 4 ε < c f ε < c. 4 3 (3.5 As the resuts, the three ots of the redctons of KCR and R-KCRR corresondng to the three data sets are resented n gure, gure and gure 3, resectvey. We can see that the redctons of R- KCRR are ess dstorted by the resence of outers comared to KCR. abe ustrates the redcton errors of KCR and R-KCRR. In the case of data wth outers, R-KCRR wth Wcoxon and Van der Waeden scores gve ower MAEs and MAEts comared to KCR. he MAEs R-KCRR wth Wcoxon scores for the trgometrc, snc and growth of son are.457,.450 and whereas the corresondng MAEts are.363,.674 and ,

6 IJCSI Internatona Journa of Comuter Scence Issues, Vo. 8, Issue 5, o, Setember 0 ISS (Onne: resectvey. he MAEs R-KCRR wth Van der Waeden score for the trgometrc, snc and growth of son are.4774,.4609 and whereas the corresondng MAEts are.57,.7709 and , resectvey. abe summarzes MAEs and MAEts of the three data sets wthout outers. In ths case, we can see that both KCR and R-KCRR erform we. abe : MAE and MAEt for KCR and R-KCRR wth outers (Wc=Wcoxon, VDW=Van der Waerden. Data Method MAE MAEt an exct mute near regresson and used R- estmators on ths near mode to have a robust regresson. hen, we soved the otmzaton robem of R-estmators usng GA for obtanng the estmate of regresson coeffcents. In ths aer, we used Wcoxon and Van der Waerden scores on R-estmators. We summarzed severa mortant onts reatng to our cases studes. rst, the redctons of R-KCRR are ess dstorted and gve smaer MAEs and MAEts comared to KCR when outers are resent n thedata. Second, wthout outers, both R-KCRR and KCR erform equay we. rgono- Metrc ( r ˆ = 0 Snc ( r ˆ = 3 Growth of Son ( r ˆ = 4 KCR R-KCRR Wc R-KCRR VDW KCR R-KCRR Wc R-KCRR VDW KCR R-KCRR Wc R-KCRR VDW abe 3: MAE and MAEt for KCR and R-KCRR wthout outers (Wc=Wcoxon, VDW=Van der Waerden. Data Method MAE MAEt rgono- Metrc ( r ˆ = 0 Snc ( r ˆ = 3 Growth of Son ( r ˆ = 4 KCR R-KCRR Wc R-KCRR VDW KCR R-KCRR Wc R-KCRR VDW KCR R-KCRR Wc R-KCRR VDW gure : KCR (Back and R-KCRR (red usng Wcoxon scores wth ρ and rˆ equa to 5 and 0, resectvey. he back crces are trgonometrc data wth random noses: (a tranng data, (b testng data. 4. Concusons We have roosed a nove robust regresson usng the hybrdzaton of KCA and R-estmators. Our method yeds a nonnear robust redcton and can dsose the effects of mutconearty n regresson mode. he roosed method was erformed by transformng orgna data nto a hgher dmensona feature sace and creatng a mute near regresson n the feature sace. After that, we erformed a kerne trck to have

7 IJCSI Internatona Journa of Comuter Scence Issues, Vo. 8, Issue 5, o, Setember 0 ISS (Onne: Acknowedgments he authors sncerey thank to Unverst eknoog Maaysa and Mnstry of Hgher Educaton (MOHE Maaysa for Research Unversty Grant (RUG wth vot number Q.J We aso thank to he Research Management Center (RMC UM for suortng ths research roject. gure : KCR (Back and R-KCRR (red usng Van der Waerden scores wth ρ and rˆ equa to 5 and 3, resectvey. he back crces are snc data wth random noses: (a tranng data, (b testng data. gure 3: KCR (Back and R-KCRR (red usng Wcoxon scores wth ρ and rˆ equa to 5 and 3, resectvey. he back crces are the growth of son data wth random noses: (a tranng data, (b testng data. References [] H. Anton, Eementary Lnear Agebra, John Wey and Sons, Inc., 000. [] M.B. Aryanezhad and M. Hemat, A new genetc agorthm for sovng nonconvex nonnear rogrammng robems, Aed Mathematcs and Comutaton, 86:86 94, 008. [3] J. Cho, J. Lee, S.W Cho, D. Lee, and I. Lee, aut dentfcaton for rocess montorng usng kerne rnca comonent anayss, Chemca Engneerng Scence, ages 79 88, 005. [4] M. Gen, R. Cheng, and L. Ln, etwork Modes and Otmzaton Mutobjectve Genetc ALgorthm Aroach, Snger, 008. [5] R.L. Haut and S.E. Haut, ractca Genetc Agorthms, John Wey and Sons, 004. [6] L. Hoegaerts, J.A.K. Suykens, J. Vandewae, and B. De Moor, Subset based east squares subsace n reroducng kerne hbert sace, eurocomutng, ages 93 33, 005. [7]. Huber, Robust Statstcs, John Wey and Son Inc, 98. [8] A.M. Jade, B. Srkanth, B.D Kukar, J. Jog, and L. rya, eature extracton and denosng usng kerne ca, Chemca Engneerng Scences, 58: , 003. [9] C. Lu, C. Zhang,. Zhang, and W. Zhang, Kerne based symmetrca rnca comonent anayss for face cassfcaton, eurocomutng, 70:904 9, 007. [0] H. Q. Mnh,. yog, and Y. Yao, Mercer s theorem, feature mas, and smoothng, Lecture otes n Comuter Scence, Srnger Berng, 4005/006, 009. [] D. C. Montgomery, E. A. eck, and G. G. Vnng, Introducton to Lnear Regresson, Wey-Interscence, 006. [] M.S. Osman, Mahmoud A. Abo-Snn, and A.A. Mousa, A combned genetc agorthm-fuzzy ogc controer (ga-fs n nonnear rogrammng, Aed Mathematcs and Comutaton, 70:8 840, 005. [3] C. H. ark, W. I. Lee, W. Suck, and A. Vautrn, Imroved genetc agorthm for mutdscnary otmzaton of comoste amnates, Chemometrcs and Integent Laboratory Systems, 68: , 008. [4] R. Rosa, M. Groam, L. J. rejo, and A. Cchok, Kerne ca for feature extracton and de-nosng n nonnear regresson, eura Comutng and Acatons, ages 3 43, 00. [5] R. Rosa and L. J. rejo, Kerne arta east squares regresson n reroducng kerne hbert sace, Journa of Machne Learnng Research, :97 3, 00. [6] R. Rosa, L. J. rejo, and A. Cchok, Kerne

8 IJCSI Internatona Journa of Comuter Scence Issues, Vo. 8, Issue 5, o, Setember 0 ISS (Onne: rnca comonent regresson wth em aroach to nonnear rnca comonent extracton, echnca Reort, Unversty of asey, UK, 00. [7] B. Schokof, A. Smoa, and K.R. Muer, onnear comonent anayss as a kerne egenvaue robem, eura Comutaton, 0:99 39, 998. [8] B. Schokof and A.J. Smoa, Learnng wth kernes, he MI ress., 00. [9] G.A.. Seber and C.J. Wd, onnear Regresson, John Wey and Sons, Inc., 998. [0] S..Svanandam and S..Deea, Introducton to Genetc Agorthms, Srnger, 008. [] S. Sumath,. Hamsarya, and. Surekha, Evoutonary Integence, Srnger, 008. [] A. Wbowo and Y. Yamamoto, A note of kerne rnca comonent regresson, o aear n Comutatona Mathematcs and Modeng, Srnger, 0. [3] X. Yu and M. Gen, Introducton to Evoutonary Agorthms, Srnger, 00. Anton Wbowo s currenty workng as a senor ecturer n the facuty of comuter scence and nformaton systems, UM. He receved B.Sc n Math Engneerng from Unversty of Sebeas Maret (US Indonesa and M.Sc n Comuter Scence from Unversty of Indonesa. He aso receved M. Eng and Dr. Eng n System and Informaton Engneerng from Unversty of sukuba Jaan. Hs nterests are n the fed of comutatona ntegence, oeratons research and data anayss. Mohamad Ishak Desa s a rofessor n the facuty of comuter scence and nformaton systems, UM. He receved hs B.Sc. n Mathematcs from UKM n Maaysa, a ostgraduate doma n system anayss from Aston Unversty, UK. He aso receved a M.A. n Mathematcs from Unversty of Inos at Srngfed, USA and then, a hd n oeratons research from Saford Unversty, UK. He s currenty the Head of the Oeratons and Busness Integences Research Grou n UM. Hs nterests are oeratons research, otmzaton, ogstc and suy chan, and comutatona ntegence.