OPERATOR-VALUED KERNEL RECURSIVE LEAST SQUARES ALGORITHM

Size: px

Start display at page:

Download "OPERATOR-VALUED KERNEL RECURSIVE LEAST SQUARES ALGORITHM"

Maximilian Davis
6 years ago
Views:

1 3rd Europen Sgnl Processng Conference EUSIPCO OPERATOR-VALUED KERNEL RECURSIVE LEAST SQUARES ALGORITM P. O. Amblrd GIPSAlb/CNRS UMR 583 Unversé de Grenoble Grenoble, Frnce. Kdr LIF/CNRS UMR 779 Ax-Mrselle Unversé Mrselle, Frnce ABSTRACT The pper develops recursve les squre lgorhms for nonlner flerng of mulvre or funconl d srems. The frmework reles on kernel lber spces of operors. The resuls generlze o hs frmework he kernel recursve les squres developed n he sclr cse. We prculrly propose wo possble exensons of he noon of pproxme lner dependence of he regressors, whch n he conex of he pper, re operors. The developmen of he lgorhms re done n nfne-dmensonl spces usng mrces of operors. The lgorhms re esly wren n fne-dmensonl sengs usng block mrces, nd re llusred n hs conex for he predcon of bvre me seres. Index Terms kernel RLS, operor-vlued kernels, vecorvlued RKS, mulsk lernng, funconl d nlyss 1. INTRODUCTION In hs pper we consder he problem of onlne nonlner flerng for funconl d. Funconl d nlyss dels wh pplcons for whch d re nurlly modeled s funcons [1,, 3]. For exmple, hs occurs n sensor neworks where ech sensor mesures, sy, he emperure over me, nd he gol s o predc he whole feld of emperure over me. Anoher relevn pplcon could be hyperspecrl mges nlyss, for whch ech pxel cn be descrbed s whole funcon of he wvelengh. In some pplcon, for exmple ocen monorng, funconl d re vlble n srem. For exmple, buoy n he pcfc ocen regulrly sends he mesures hs recorded durng some perod. Assmlon of hs funconl d needs o be done onlne. In hs work we consder flerng usng reproducng kernel lber spce RKS pproch. Ths frmework hs been proved exremely powerful for sclr problems [4, 5] snce llows nonlner processng usng lner echnques. owever, for mulvre or funconl d, he usul heory of RKS s no ppropre snce s dedced o one dmensonl oupu d. I would be possble o del wh produc of dfferen spces, bu hs would no ke correcly no ccoun nercons beween vrbles. Ths hs led o he developmen of reproducng kernel lber spces of operors [6]. Mny works hve been done n hs conex nd some pplcons hs been developed, especlly n he mchne lernng nd sscs communes [7, 8, 9, 10] o ce some bu few. Some work concernng he onlne processng of funconl d usng RKS of operors re emergng, such s [11, 1]. The conrbuons of hs pper re wo operor-vlued kernel recursve les squre lgorhms, ovkrls, especlly for mulvre nd funconl d flerng pplcons. These re developed for nfne-dmensonl d, nd obvously prculrzed for mulvre d. The lgorhms rely on wo dfferen defnons of pproxme lner dependence for operors. Erler works On-lne kernel bsed lernng hs rced lo of enon n he he ls decde, n boh he mchne lernng nd he sgnl processng communes. The recen revew [13] gves n nce vew of he feld, especlly for pplcons n flerng. In prllel nd especlly n he mchne lernng communy, mny works hve developed he use of operor-vlued kernel lber spces nd her pplcon n mulsk lernng nd funconl d nlyss. In hs pper, we focus on onlne lernng wh operor-vlued kernels. In prculr, we exend he semnl work n [14], snce our resuls recover he usul krls when he oupus re sclrs. In he followng secon, we mke precse he conex of he pper nd recll some necessry merls on operor-vlued reproducng kernel lber spces. Secon 3 s he core of he pper. We presen sprsfcon procedure whch generlzes he pproxme lner dependence procedure of he sclr cse. Two possble exensons re proposed whch led o wo versons of he lgorhm. A smple llusron s provded n he ls secon. Fnlly, dscusson concludes he pper by hghlghng remnng problems nd some fuure developmens.. NOTATION AND BACKGROUND We move he work by regresson problem n hgh-dmensonl spces. Gven d se {x, y } N =1, we look for n f so h good model for he d s y = fx + ε. ere, we suppose h x nd y belongs o some lner spces X nd whch re no necessrly fne-dmensonl he fc h X s lner spce could even be relxed. For exmple, he spces cn be some funconl spces such s L I, I beng some nervl of he rel lne. In hs cse, he problem s regresson problem for funconl d, nd f s hen n operor from X o whch hs o be nferred from he lernng se {x, y } N =1. Anoher mporn exmple n sgnl processng s he problem of predcon of mulvre me seres z = fz,..., z M + ε. ere, f z R nz, he pplcon f s n operor from R Mnz o R nz nd hs o be chosen so s o mnmze he predcon errors. When s R or C, powerful pproch o solve hese knd of problems s o look for f no RKS. Ths llows o model f s hghly nonlner funcon whle he sme me uhorzng s esmon usng very effcen lner lgorhms. Ths hs led o lo of works n he ls wo decdes [4]. Exendng he frmework when he oupu spce s muldmensonl or nfne-dmensonl hs genered some work n he ls decde [7, 8, 9]. The exenson consders spce of operors nsed of funconl spce, nd hs requres o swch from sclr-vlued kernels o operor-vlued kernels. We now formlly recll he dfferen noons we need /15/$ IEEE 41

2 3rd Europen Sgnl Processng Conference EUSIPCO If K nd K re wo lber spces, LK, K s he Bnch spce of bounded lner operors from K o K. In he cse of K = K, we se LK = LK, K. lber spces wh reproducng operor-vlued kernels [6]. Consder now lber spce of bounded nd hence connuous operors from X o, equpped wh n nner produc. s clled RKS f δ fg : R Σ δ fg Σ := g Σf s connuous f, g X. Ths generlzon o operors of he connuy of he evluon funconl for funcons, defnng propery of usul RKS [5]. Applcon of Resz represenon heorem hen llows o show h here exss n operor-vlued kernel K, : X X L such h g Σf = Kf, g Σ, f, g X. Noe h Kf,.g s n operor from X nd belongs o. The precedng defnng propery cn be seen s reproducng propery snce g Kf, hk = Kh, g Kf, k. K hs he followng symmery propery Kf, hg k = Kh, k Kf, g = Kh, fk g. Fnlly, Kf, h, f, h X, hs posveness propery: n N, {f, g } n =1 Kf, f jg gj = Kfj, g j Kf, g,j,j = j Kf j, g j 0. The properes of symmery nd posveness re he defnng properes of kernel. We cll K n operor-vlued kernel on X. I cn be shown [7, 9] h kernel deermnes unque RKS. Then mos of he consrucons done n he funcon cse cn be rnsposed n hs conex. In prculr, he represener heorem cn be proved s n [4] o show h for he problem f = rg mn f cx, y, fx + λω f, where c s cos funcon nd Ω srcly ncresng posve penly, ny mnmzer cn be wren s f = Kx, z, z,. 3. OPERATOR-VALUED KRLS The m here s o lern n operor h lnks x X o y bsed on d srem {x, y }, 1. We consder n operorvlued kernel K nd s ssoced RKS LX,. Afer observons, we look for he member f of whch solves f = rg mn Σ y Σx. =1 Thnks o he represener heorem, he mnmzer hs he form Kx, z, nd he progrm becomes z = rg mn z y Kx j, x z j. =1 Workng n produc spces wh he usul ssoced nner produc wh mrces of operors llows o wre hs s z = rg mn z y Kz. where K L s mrx of operors whose, jh block s K j = Kx, x j. Under some ssumpons, soluon o hs progrm cn be obned ncludng some regulrzon f needed. Two problems occur here when delng wh srem of d. The frs s how o ncorpore new observon n he lernng procedure, nd he second s he lner ncrese n dmensons of he mrx of operors K nd of he vecor of operors y. The frs problem s solved by formulng recursve soluon o he opmzon problem. The second cn be solved by sprsfcon procedure whch keeps he mos represenve d we hve seen up o me. Equvlenly hs corresponds o pproxmng he mrx of operors K by lower dmensonl mrx. Ths problem s well know n kernel bsed lernng pproches for lrge d ses [4]. ere, snce we fce d srem, nce soluon o solve boh problems s o recursvely buld dconry h correcly represens he d srem. Ths de ws proposed n [14] o solve he sclr-vlued krls. vng bul dconry up o me 1, he new dum x s ncorpored n he dconry f cn no be predced from he members of he dconry. Ths s clled pproxme lner dependence ALD. We develop hs de n he operor conex. I urns ou h hs noon for operor s no s smple s n he sclr cse. We provde wo possble vews of ALD Onlne sprsfcon v ALD Assume h me sep, fer hvng observed 1 smples {x } =1, we hve colleced dconry conssng of subse of he rnng smples D = { x j} m, where by consrucon {K x j, } m re lner ndependen feure operors. Ths obvously requres o defne he noon of lner ndependence for feure operors. We gve wo possble defnons. Lnerly dependen operors. Followng [7], we wll sy h {Kx, } re lnerly ndependen, f ny y j cn be unquely wren s Kxj, xc, c. ence, when new dum x s observed, we es f Kx, y cn be wren s m =1 K, x c, nd hs for ll y. Approxme Lner Dependence proceeds by performng men squre predcon nd by esng f he error s smll or no. Le = 1,..., m where j. The men squre error of predcon reds δ y = mn m K x j, j Kx, y = mn K m K x y + Kx, x y y. ere, K L m s mrx of operors whose, jh block s K j = K x, x j. Lkewse, K x L m, s row of operors whose h block s K x = Kx, x. Le be he operor n L m, defned by = K K x. The mnmum s hen reched y = K K x y, where snds for he djon. The ALD es consss n comprng δ y o zero prcclly o user defned hreshold. The condon δ y > 0, y s obvously edous o verfy. owever, snce he very menng of ALD s o es wheher he new dum x s worh consderng n he 1 4

3 3rd Europen Sgnl Processng Conference EUSIPCO predcon of y, seems nurl o use he es wh y = y. Thus, he new dum s declred ALD f δ y = Kx, x y y K x y y δ 0, where δ 0 s user defned hreshold. Noe h from hese equons, he sclr-vlued ALD condon n [14] cn be recovered by choosng y = 1, nd does no depend on y. Globlly lnerly dependen operors. A second choce consders h he d re dependen f c R, > 1 no ll zero such h ckx, = 0. To mplemen hs, when new dum x s colleced, we predc he operor Kx, L, usng lner combnon of he operors K x,, = 1,..., m ken from he curren dconry. To es globl lner dependence we compre m δ = mn jk x j, Kx, o user prescrbed hreshold δ 0. The norm used bove s n operor norm. I cn be he usul operor norm Kx, = sup y Kx,.y / y, or he lber-schmd norm f we suppose Kx, o be lber-schmd. In hs cse, le e k be ny orhonorml sysem of, hen Kx, S = k Kx, e k. Recll h he spce S, of lber-schmd operors from o s n lber spce when endowed wh he nner produc 1, := k 1e k e k. δ hen reds m δ = mn jk x j, Kx, S m = mn jk x j, e k Kx, e k. k Le Tr K = k Ke k ek, K he mrx wh enres K j = Tr K x, x j nd k he vecor wh enres k = Tr Kx, x. Then we ge δ = mn Tr Kx, x k + K. The mnmum s ned for = K k nd hence δ = Tr Kx, x k K k. In he cse of fne-dmensonl spces, he rce s obvously very esy o clcule. Noe h f = R hs pproch lso reduces o he frmework developed n [14]. 3.. Algorhm 1: okrls Gven d up o me, we look for he operor f such h f = rg mn f =1 fx y. The represener heorem for operor RKS mples f = Kx, z,, where z,. If he ALD sprsfcon echnque s used, recll h Kx, y m K x j,,jy. We hve for ny y nd ny j, hnks o he reproducng propery f x j y = m,m j Kx, x jz y α,β=1 j,β K x β, x α op,αz y. To keep smple noons, se z α =,αz. Defne A L m, o be he mrx of operors whose j, block s A j = j,. Noe h he block A j = 0 s soon s > m j. Then he column f x 1,..., f x of equls A K. Therefore, he progrm nd s soluon re z = rg mn z m y A Kz = K A A A y. Ths form cn be urned no recursve lgorhm. The recurson however depends on he resul of he ALD es. Suppose we hve obned z, nd h he new dum x, y s cqured. The dervon s done s n he sclr cse. We jus provde here he resulng recurson equons. 1. Kx, s ALD δ y δ 0, he dconry s unchnged, D = D, m = m, K = K. The mrces of operors A nd P := A A re upded v A = A P = P P I P P, nd he soluon z s upded by z = z + K P I P y K x z.. Kx, s no ALD δ y > δ 0, he dconry s upded s D = D x, m = m + 1, he mrces of operors A nd P re upded by I s he deny operor A = A 0 0 I nd P = P 0 0 I The mrx of operors K cn be upded v K K K x = K x Kx, x = K + K K x Z K Kx Z Kx K. K K x Z where Z = Kx, x K K x K x. The vecor z s upded by he recurson z = z K K x Z y K x z Z y K x z To sum up he lgorhm, for ech 1, frs evlue nd δ, nd pply em 1 bove f ALD s rue δ δ 0 or em f no Algorhm : Globl okrls When he globl ALD condon s used, he srucure of he lgorhm s lle b lered. For he ske of llusron, we presen n he fne-dmensonl seng for whch = R ny. In hs suon, he mrx K s pproxmed by K A I ny K A I ny, where K s he n y n y block mrx whose s, h block s he n y n y mrx Kx s, x, where I ny s he n y dmensonl deny mrx, nd snds for he Kronecker produc. Lkewse, K s he m n y m n y block mrx wh defnng blocks K x, x j. A s he m 1,..., where s obned by Z., 43

4 3rd Europen Sgnl Processng Conference EUSIPCO he globl ALD creron. When sprsfcon bsed on he globl ALD s used, we hve z = rg mn y z A I ny K z [ ] = K A A A I ny y. Ths form s esly urned no recursve formul, long he lnes developed prevously. When he new dum s no ALD, he srucure of he lgorhm s he sme s n lgorhm 1. owever, he srucure s lle b lered when he new dum s ALD. In h cse, P = A A nd z re upded ccordng o okrls, σ pred for x nd y Trce of error covrnce 0.06 GokRLS, T=I GokRLS,T=Cov y okrls q = P 1 + P, P = P q P, z = z + K q y I ny K z. If n y = 1 s esy o verfy h lgorhm reduces lso o he sclr-vlued krls [14]. Complexy Anlyss. A sndrd mplemenon of okrls nd Globl okrls requres On 3 ym operons when dm = n y <. Thus, okrls lgorhms sgnfcnly lleve he compuonl boleneck of bch operor-vlued les squres, whch kes On 3 y 3 me. The compuonl cos of boh okrls lgorhms s domned by he cos of fndng he mrx Z used o compue K he upde of n he non-ald cse. owever, s mporn o noe h he cos of evlung he ALD condon n he Globl okrls lgorhm s much less hn h of okrls. For he former he ol number of erons s of On ym, wheres for he ler s of On 3 ym. Whle ALD nvolves he mulplcon of block mrces of sze n ym n ym nd n ym n y, globl ALD only nvolves compung he rce of he operor oupued by he kernel nd he mulplcon of smple mrces of sze m m nd m AN ILLUSTRATION We perform smple expermen o llusre he okrls lgorhms. By no mens hs smll expermen s nended o be complee vldon of he lgorhms. The numercl sbly of he lgorhms hs o be esblshed, especlly becuse he okrls re nended o be ppled o funconl d for whch he dmensons wll be hgh. Bu hs s beyond he scope of he pper. We pply he lgorhms o he nonlner predcon of mulvre me seres. We consder he couplng of Glss-Mckey lke models. The bvre me seres s defned by x = x 0.4 x x x 10 4 y = 0.6y + 0.8y + 0.4x 1 + y 10 + ε y, y y 3 + ε x, where ε x, nd ε y, re..d. zero-men Gussn noses of vrnce 10. Bsed on he observon of he me seres up o me, he m s o esme x +1, y +1 onlne usng okrls. The kernel used here s he smple seprble kernel operor Kx, y = kx, yt where k s sclr kernel on X we chose he Gussn kernel n our smulons nd T s consn operor. A smple choce for T s he deny operor. In hs cse however, he okrls corresponds o he evluon of sclr-vlued krls n prllel. Furhermore, hs choce does no llow o use he nercons h my ex beween Fg. 1. Lef: Sndrd devon curves for he okrls for ech componens of he bvre process. Rgh: Trce of he covrnce of he error of predcon of he bvre process for he okrls nd GokRLS wh T = cov oupu, nd GokRLS wh T = I. he componens of he oupu here, correlons beween x +1 nd y +1. A neresng choce o ke hese nercons no ccoun s o mke T dependen on he oupu. A smple wy of proceedng s o choose T s he covrnce of he oupu. Ths s esy o mplemen snce he covrnce cn be lern onlne. The d used re {x = x, y = 4; y = x +1, y +1} 5. We plo n fgure 1, lef, he sndrd devon of he error of predcon z +1 ẑ +1, where ẑ +1 s he esmon by he onlne lgorhm for ech componen z = x nd z = y. The lgorhm s he okrls wh T beng he covrnce of he oupu s descrbed bove. In he rgh plo, we depc he rce of he covrnce of he error of predcon for he okrls nd GokRLS wh T s bove, nd for he GokRLS wh T = I. Ths ls cse corresponds o sclrvlued krls n prllel. For boh plo we only depc he 104 frs erons. The prmeers vrnce of he Gussn kernel, hreshold of he ALD es of ech lgorhm were chosen so h hey use on verge m = 60 regressors n he dconry fer 104 erons. The curves re obned by vergng over 1000 ndependen snpshos. As cn be seen, convergence hs no complely seled bu s lmos reched fer 104 erons. Noe h n he lef plo, he lower bound for he curves re 0.1 whch s he sndrd devons of he dynmcl nose chosen n he model. The convergence of he okrls wh he chosen kernel s hus ssfcory. The rgh plo llows o qunfy he dfference beween okrls nd GokRLS. The gn s sgnfcn for he okrls he expense of hgher compuonl complexy. The plo llows lso o show he mprovemen when consderng he covrnce operor of he oupu s T nsed of I, s llusred wh he GokRLS for he wo cses. Dscusson. The smple exmple developed prevously shows he mpornce of couplng he oupus v he choce of he kernel. Ths couplng clerly mproves predcon over he pplcon of krls n prllel T = I. owever, hs exmple s very smple nd we re currenly developpng exmples n much hgher dmensons for funconl vlued sgnls. Fnlly, we hve no elucded relshonshps beween he wo forms of ALD we hve suded, queson of gre heorecl nd prccl neres. Concernng fuure works, we nend o develop sldng ovkrls nd n ovklms, sudy he convergence of he lgorhms, nd pply ll hese lgorhms o Grnger cusly n hgh dmensonl spces. 44

5 3rd Europen Sgnl Processng Conference EUSIPCO 5. REFERENCES [1] J. O. Rmsy nd B. W. Slvermn, Appled funconl d nlyss: Mehods nd cse sudes, Sprnger, 00. [] F. Ferry nd P. Veu, Nonprmerc Funconl D Anlyss, Sprnger Verlg, 006. [3] D. Bosq nd D. Blnke, Inference nd predcon n lrge dmensons, John Wley & Sons: Chcheser, UK, 007. [4] B. Schölkopf nd A. J. Smol, Lernng wh kernels, MIT Press, Cmbrdge, M, USA, 00. [5] I. Senwr nd A. Chrsmnn, Suppor vecor mchnes, Sprnger, 008. [6] E. Senkene nd A. Tempel mn, lber spces of operorvlued funcons, Lhunn Mhemcl Journl,, no. 4, pp , [7] C. A. Mcchell nd M. Ponl, On lernng vecor-vlued funcons, Neurl Compuon, vol. 17, pp , 005. [8]. Ln, Nonlner funconl models for funconl responses n reproducng kernel lber spces, The Cndn Journl of Sscs, vol. 35, pp , 007. [9]. Kdr, E. Duflos, P. Preux, S. Cnu, nd M. Dvy, Nonlner funconl regresson: funconl RKS pproch, n.w. Teh nd M. Terngon Eds., Proceedngs of The Threenh Inernonl Conference on Arfcl Inellgence nd Sscs AISTATS, JMLR: W&CP 9, Ch Lgun, Srdn, Ily, 010, pp [10] N. Lm, F. d Alché Buc, F. Aulc, nd G. Mchlds, Operor-vlued kernel-bsed vecor uoregressve models for nework nference, Mchne lernng, vol. 99, no. 3, pp , 014. [11] G. Plloneo, F. Dnuzzo, nd G. De Ncolo, Byesn onlne mulsk lernng of gussn processes, IEEE Trns. on PAMI, vol. 3, no., pp , 010. [1] J. Audffren nd. Kdr, Onlne lernng wh operorvlued kernels, n Proceedngs of he 3h Symposum on Arfcl Neurl Neworks ESANN, 015. [13] K. Slvks, P. Boubouls, nd S. Theodords, Acdemc Press Lbrry n Sgnl Processng: Volume 1, Sgnl Processng Theory nd Mchne Lernng, chper ch. 17, Onlne lernng n reproducng kernel lber spces, pp , Elsever, 014. [14] kov Engel, She Mnnor, nd Ron Mer, The kernel recursve les squres lgorhm, IEEE Trnscons on Sgnl Processng, vol. 5, no. 8, pp ,

Supporting information How to concatenate the local attractors of subnetworks in the HPFP

Supporting information How to concatenate the local attractors of subnetworks in the HPFP n Effcen lgorh for Idenfyng Prry Phenoype rcors of Lrge-Scle Boolen Newor Sng-Mo Choo nd Kwng-Hyun Cho Depren of Mhecs Unversy of Ulsn Ulsn 446 Republc of Kore Depren of Bo nd Brn Engneerng Kore dvnced