Theoretical Analysis of Cross-Validation for Estimating the Risk of the k-nearest Neighbor Classifier

Transcription

1 Joural of Machi Larig Rsarch Submittd 09/15; Rvisd 07/18; Publishd 11/18 Thortical Aalysis of Cross-Validatio for Estimatig th Ris of th -Narst Nighbor Classifir Alai Cliss Laboratoir d Mathématiqus UMR 854 CNRS-Uivrsité d Lill Iria Modal Projct-tam, Lill F Villuv d Ascq Cdx, Frac cliss@math.uiv-lill1.fr Trista Mary-Huard maryhuar@agroparistch.fr INRA, UMR 030 / UMR 810 Géétiqu Quatitativ t Évolutio L Moulo, F Gif-sur-Yvtt, Frac UMR MIA-Paris, AgroParisTch, INRA, Uivrsité Paris-Saclay F-75005, Paris, Frac Editor: Hui Zou Abstract Th prst wor aims at drivig thortical guaratis o th bhavior of som crossvalidatio procdurs applid to th -arst ighbors NN rul i th cotxt of biary classificatio. Hr w focus o th lav-p-out cross-validatio LpO usd to assss th prformac of th NN classifir. Rmarably this LpO stimator ca b fficitly computd i this cotxt usig closd-form formulas drivd by Cliss ad Mary-Huard 011. W dscrib a gral stratgy to driv momt ad xpotial coctratio iqualitis for th LpO stimator applid to th NN classifir. Such rsults ar obtaid first by xploitig th coctio btw th LpO stimator ad U-statistics, ad scod by maig a itsiv us of th gralizd Efro-Sti iquality applid to th L1O stimator. O othr importat cotributio is mad by drivig w quatificatios of th discrpacy btw th LpO stimator ad th classificatio rror/ris of th NN classifir. Th optimality of ths bouds is discussd by mas of svral lowr bouds as wll as simulatio xprimts. Kywords: Classificatio, Cross-validatio, Ris stimatio 1. Itroductio Th -arst ighbor NN algorithm Fix ad Hodgs, 1951 i biary classificatio is a popular prdictio algorithm basd o th ida that th prdictd valu at a w poit is basd o a majority vot from th arst labld ighbors of this poit. Although quit simpl, th NN classifir has b succssfully applid to may difficult classificatio tass Li t al., 004; Simard t al., 1998; Schirr ad Slay, 003. Efficit implmtatios hav b also dvlopd to allow dalig with larg datasts Idy ad Motwai, 1998; Adoi ad Idy, 006. Th thortical prformacs of th NN classifir hav b alrady xtsivly ivstigatd. I th cotxt of biary classificatio prlimiary thortical rsults dat bac to c 018 Alai Cliss ad Trista Mary-Huard. Lics: CC-BY 4.0, s Attributio rquirmts ar providd at

2 Cliss ad Mary-Huard Covr ad Hart 1967; Covr 1968; Györfi Th NN classifir has b provd to b waly uivrsally cosistt by Sto 1977 as log as = + ad / 0 as +. For th 1NN classifir, a asymptotic xpasio of th rror rat has b drivd by Psaltis t al Th sam stratgy has b succssfully applid to th NN classifir by Sapp ad Vatsh Hall t al. 008 study th ifluc of th paramtr o th ris of th NN classifir by mas of a asymptotic xpasio drivd from a Poisso or biomial modl for th traiig poits. Mor rctly, Caigs t al. 017 poitd out som limitatios suffrd by th classical NN classifir ad dducd a improvd vrsio basd o a local choic of i th smi-suprvisd cotxt. I cotrast to th aformtiod rsults, th wor by Chaudhuri ad Dasgupta 014 focuss o th fiit-sampl framwor. Thy typically provid uppr bouds with high probability o th ris of th NN classifir whr th bouds ar ot distributio-fr. Altrativly i th rgrssio sttig, Kulari ad Posr 1995 drivd a stratgy ladig to a fiit-sampl boud o th prformac of 1NN, which has b xtdd to th wightd NN rul 1 by Biau t al. 010a,b s also Brrtt t al., 016, whr a wightd NN stimator is dsigd for stimatig th tropy. W rfr itrstd radrs to Biau ad Dvroy 016 for a almost thorough prstatio of ow rsults o th NN algorithm i various cotxts. I umrous if ot all practical applicatios, computig th cross-validatio CV stimator Sto, 1974, 198 has b amog th most popular stratgis to valuat th prformac of th NN classifir Dvroy t al., 1996, Sctio 4.3. All CV procdurs shar a commo pricipl which cosists i splittig a sampl of poits ito two disjoit substs calld traiig ad tst sts with rspctiv cardialitis p ad p, for ay 1 p 1. Th p traiig st data srv to comput a classifir, whil its prformac is valuatd from th p lft out data of th tst st. For a complt ad comprhsiv rviw o cross-validatio procdurs, w rfr th itrstd radr to Arlot ad Cliss 010. I th prst wor, w focus o th lav-p-out LpO cross-validatio. Amog CV procdurs, it blogs to xhaustiv stratgis sic it cosidrs ad avrags ovr all th p possibl such splittigs of {1,..., } ito traiig ad tst sts. Usually th iducd computatio tim of th LpO is prohibitiv, which givs ris to its surrogat calld V fold cross-validatio V-FCV with V /p Gissr, Howvr, Stl 009; Cliss ad Mary-Huard 011 rctly drivd closd-form formulas rspctivly for th bootstrap ad th LpO procdurs applid to th NN classifir. Such formulas allow for a fficit computatio of th LpO stimator. Morovr sic th V-FCV stimator suffrs th sam bias but a largr variac tha th LpO o Cliss ad Robi, 008; Arlot ad Cliss, 010, LpO with p = /V strictly improvs upo V-FCV i th prst cotxt. Although big favord i practic for assssig th ris of th NN classifir, th us of CV coms with vry fw thortical guarats rgardig its prformac. Morovr probably for tchical rasos, most xistig rsults apply to Hold-out ad lav-o-out L1O, that is LpO with p = 1 Kars ad Ro, I this papr w rathr cosidr th gral LpO procdur for 1 p 1 usd to stimat th ris altrativly th classificatio rror rat of th NN classifir. Our mai purpos is th to provid distributio-fr thortical guarats o th bhavior of LpO with rspct to iflutial paramtrs such as p,, ad. For istac w aim at aswrig qustios such as: Dos

3 Prformac of CV to Estimat th Ris of NN thr xist ay rgim of p = p with p som fuctio of whr th LpO stimator is a cosistt stimat of th ris of th NN classifir?, or Is it possibl to dscrib th covrgc rat of th LpO stimator dpdig o p? Cotributios. Th mai cotributio of th prst wor is two-fold: i w dscrib a w gral stratgy to driv momt ad xpotial coctratio iqualitis for th LpO stimator applid to th NN biary classifir, ad ii ths iqualitis srv to driv th covrgc rat of th LpO stimator towards th ris of th NN classifir. This w stratgy rlis o svral stps. First xploitig th coctio btw th LpO stimator ad U-statistics Korolju ad Borovsich, 1994 ad th Rosthal iquality Ibragimov ad Sharahmtov, 00, w prov that uppr boudig th polyomial momts of th ctrd LpO stimator rducs to drivig such bouds for th simplr L1O stimator. Scod, w driv w uppr bouds o th momts of th L1O stimator usig th gralizd Efro-Sti iquality Bouchro t al., 005, 013, Thorm Third, combiig th two prvious stps provids som isight o th itrplay btw p/ ad i th coctratio rats masurd i trms of momts. This fially rsults i w xpotial coctratio iqualitis for th LpO stimator applyig whatvr th valu of th ratio p/ 0, 1. I particular whil th uppr bouds icras with 1 p / + 1, it is o logr th cas if p > / + 1. W also provid svral lowr bouds suggstig our uppr bouds caot b improvd i som ss i a distributio-fr sttig. Th rmaidr of th papr is orgaizd as follows. Th coctio btw th LpO stimator ad U-statistics is clarifid i Sctio, whr w also rcall th closd-form formula of th LpO stimator applid to th NN classifir Cliss ad Mary-Huard, 011. Ordr-q momts q of th LpO stimator ar th uppr boudd i trms of thos of th L1O stimator. This stp ca b applid to ay classificatio algorithm. Sctio 3 th spcifis th prvious uppr bouds i th cas of th NN classifir, which lads to th mai Thorm 3. charactrizig th coctratio bhavior of th LpO stimator with rspct to p,, ad i trms of polyomial momts. Drivig xpotial coctratio iqualitis for th LpO stimator is th mai cocr of Sctio 4 whr w highlight th strgth of our stratgy by comparig our mai iqualitis with coctratio iqualitis drivd with lss sophisticatd tools. Fially Sctio 5 xploits th prvious rsults to boud th gap btw th LpO stimator ad th classificatio rror of th NN classifir. Th optimality of ths uppr bouds is first provd i our distributio-fr framwor by stablishig svral w lowr bouds matchig th uppr os i som spcific sttigs. Scod, mpirical xprimts ar also rportd which support th abov coclusios.. U-statistics ad LpO stimator.1. Statistical framwor Classificatio W tacl th biary classificatio problm whr th goal is to prdict th uow labl Y {0, 1} of a obsrvatio X X R d. Th radom variabl X, Y has a uow joit distributio P X,Y dfid by P X,Y B = P X, Y B ] for ay Borlia st B X {0, 1}, whr P dots a rfrc probability distributio. I what follows o particular distributioal assumptio is mad rgardig X. To prdict th labl, o aims at buildig a classifir ˆf : X {0, 1} o th basis of a st of radom variabls 3

4 Cliss ad Mary-Huard D = {Z 1,..., Z } calld th traiig sampl, whr Z i = X i, Y i, 1 i rprst copis of X, Y draw idpdtly from P X,Y. I sttigs whr o cofusio is possibl, w will rplac D by D. Ay stratgy to build such a classifir is calld a classificatio algorithm, ad ca b formally dfid as a fuctio A : 1 {X {0, 1}} F that maps a traiig sampl D oto th corrspodig classifir A D = ˆf F, whr F is th st of all masurabl fuctios from X to {0, 1}. Numrous classifirs hav b cosidrd i th litratur ad it is out of th scop of th prst papr to rviw all of thm s Dvroy t al for may istacs. Hr w focus o th -arst ighbor rul NN iitially proposd by Fix ad Hodgs 1951 ad furthr studid for istac by Dvroy ad Wagr 1977; Rogrs ad Wagr Th NN algorithm For 1, th NN classificatio algorithm, dotd by A, cosists i classifyig ay w obsrvatio x usig a majority vot dcisio rul basd o th labls of th closst poits to x, dotd by X 1 x,..., X x, amog th traiig sampl X 1,..., X. I what follows ths arst ighbors ar chos accordig to th distac associatd with th usual Euclida orm i R d. Not that othr adaptiv mtrics hav b also cosidrd i th litratur s for istac Hasti t al., 001, Chap. 14. But such xampls ar out of th scop of th prst wor, that is our rfrc distac dos ot dpd o th traiig sampl at had. Lt us also mphasiz that possibl tis ar bro by usig th smallst idx amog tis, which is o possibl choic for th Sto lmma to hold tru Biau ad Dvroy, 016, Lmma 10.6, p.15. Formally, giv V x = { 1 i, X i { X 1 x,..., X x }} th st of idics of th arst ighbors of x amog X 1,..., X, th NN classifir is dfid by 1, if 1 A D ; x = f i V x Y i = 1 Y ix > 0.5 D ; x := 0, if 1 Y ix < 0.5,.1 B0.5, othrwis whr Y i x is th labl of th i-th arst ighbor of x for 1 i, ad B0.5 dots a Broulli radom variabl with paramtr 1/. Lav-p-out cross-validatio For a giv sampl D, th prformac of ay classifir ˆf = A D rspctivly of ay classificatio algorithm A is assssd by th classificatio rror L ˆf rspctivly th ris R ˆf dfid by L ˆf = P ˆfX Y D, ad R ˆf ] = E P ˆfX Y D. I this papr w focus o th stimatio of L ˆf ad its xpctatio R ˆf by us of th Lav-p-Out LpO cross-validatio for 1 p 1 Zhag, 1993; Cliss ad Robi, 008. LpO succssivly cosidrs all possibl splits of D ito a traiig st of cardiality p ad a tst st of cardiality p. Dotig by E p th st of all possibl substs of {1,..., } with cardiality p, ay E p dfis a split of D ito a traiig sampl D = {Z i i } ad a tst sampl Dē, whr ē = {1,..., } \. For a giv classificatio algorithm A, th fial LpO stimator of th prformac of A D = f is th avrag ovr all possibl splits of th classificatio rror stimatd o ach tst st, that is 1 1 R p A, D = 1 p p {A D X i Y i},. E p i ē 4

5 Prformac of CV to Estimat th Ris of NN whr A D is th classifir built from D. W rfr th radr to Arlot ad Cliss 010 for a dtaild dscriptio of LpO ad othr cross-validatio procdurs. I th squl, th lgthy otatio R p A, D is rplacd by R p, i sttigs whr o cofusio ca aris about th algorithm A or th traiig sampl D, ad by R p D if th traiig sampl has to b pt i mid. Exact LpO for th NN classificatio algorithm Usually du to its smigly prohibitiv computatioal cost, LpO is ot applid xcpt with p = 1 whr it rducs to th wll ow lav-o-out. Howvr i svral cotxts such as dsity stimatio Cliss ad Robi, 008; Cliss, 014 or rgrssio Cliss, 008, closd-form formulas hav b drivd for th LpO stimator wh applid with projctio ad rl stimators. Th NN classifir is aothr istac of such stimators for which fficitly computig th LpO stimator is possibl. Its computatio rquirs a tim complxity that is liar i p as prviously stablishd by Cliss ad Mary-Huard 011. Lt us brifly rcall th mai stps ladig to th closd-form formula. 1. From Eq.. th LpO stimator ca b xprssd as a sum ovr th obsrvatios of th complt sampl of probabilitis: p p {A D X i Y i} = p p {A D X i Y i} 1 {i/ } E p i/ E p = 1 P A D X i Y i i / P i /. p Hr P mas that th itgratio is mad with rspct to th radom variabl E p, which follows th uiform distributio ovr th p possibl substs i E p with cardiality p. For istac P i / = p/ sic it is th proportio of subsampls with cardiality p which do ot cotai a giv prscribd idx i, which quals 1 p / p. S also Lmma D.4 for furthr xampls of such calculatios.. For ay X i, lt X 1,..., X +p 1, X +p,..., X 1 b th ordrd squc of ighbors of X i. This list dpds o X i, that is X 1 should b otd X i,1. But this dpdcy is sippd hr for th sa of radability. Th y i th drivatio is to coditio with rspct to th radom variabl R i which dots th ra i th whol sampl D of th -th ighbor of X i i th D. For istac R i = j mas that X j is th -th ighbor of X i i D. Th P A D X i Y i i / = +p 1 j= P A D X i Y i R i = j, i / P R i = j i /, whr th sum ivolvs p trms sic oly X,..., X +p 1 ar cadidats for big th -th ighbor of X i i at last o traiig subst. 3. Obsrv that th rsultig probabilitis ca b asily computd s Lmma D.4: P i / = p P R i = j i / = j P U = j P A D X i Y i V i = j, i / = 1 Y j 1 F +1 ] H + Yj 1 FH 5 1 ],

6 Cliss ad Mary-Huard with U Hj, j 1, p 1, H HN j i, j N j i 1, 1, ad H HN j i 1, j N j i, 1, whr F H ad F H rspctivly dot th cumulativ distributio fuctios of H ad H, H dots th hyprgomtric distributio, ad N j i is th umbr of 1 s amog th j arst ighbors of X i i D. Th computatioal cost of LpO for th NN classifir is th sam as that of L1O for th + p 1NN classifir whatvr p, that is Op. This cotrasts with th usual p prohibitiv computatioal complxity smigly suffrd by LpO... U-statistics: Gral bouds o LpO momts Th purpos of th prst sctio is to dscrib a gral stratgy allowig to driv w uppr bouds o th polyomial momts of th LpO stimator. As a first stp of this stratgy, w stablish th coctio btw th LpO ris stimator ad U-statistics. Scod, w xploit this coctio to driv w uppr bouds o th ordr-q momts of th LpO stimator for q. Not that ths uppr bouds, which rlat momts of th LpO stimator to thos of th L1O stimator, hold tru with ay classifir. Lt us start by itroducig U-statistics ad rcallig som of thir basic proprtis that will srv our purposs. For a thorough prstatio, w rfr to th boos by Srflig 1980; Korolju ad Borovsich Th first stp is th dfiitio of a U-statistic of ordr m N as a avrag ovr all m-tupls of distict idics i {1,..., }. Dfiitio.1 Korolju ad Borovsich Lt h : X m R dot ay masurabl fuctio whr m 1 is a itgr. Lt us furthr assum h is a symmtric fuctio of its argumts. Th ay fuctio U : X R such that U x 1,..., x = U hx 1,..., x = whr m, is a U-statistic of ordr m ad rl h. 1 h x i1,..., x im m 1 i 1 <...<i m Bfor clarifyig th coctio btw LpO ad U-statistics, lt us itroduc th mai proprty of U-statistics our stratgy rlis o. It cosists i rprstig ay U-statistic as a avrag, ovr all prmutatios, of sums of idpdt variabls. Propositio.1 Eq. 5.5 i Hoffdig With th otatio of Dfiitio.1, lt us dfi W : X R by W x 1,..., x = 1 r r h x j 1m+1,..., x jm,.3 j=1 whr r = /m dots th itgr part of /m. Th U x 1,..., x = 1 W x! σ1,..., x σ, σ whr σ dots th summatio ovr all prmutatios σ of {1,..., }. 6

7 Prformac of CV to Estimat th Ris of NN W ar ow i positio to stat th y rmar of th papr. All th dvlopmts furthr xposd i th followig rsult from this coctio btw th LpO stimator dfid by Eq.. ad U-statistics. Thorm.1. For ay classificatio algorithm A ad ay 1 p 1 such that a classifir ca b computd from A o p traiig poits, th LpO stimator R p, is a U-statistic of ordr m = p + 1 with rl h m : X m R dfid by h m Z 1,..., Z m = 1 m m 1 { A Di m X i Y i }, whr D i m dots th sampl D m = Z 1,..., Z m with Z i withdraw. Not for istac that wh A = A dots th NN algorithm, th cardiality of D i m has to satisfy p, which implis that 1 p 1. Proof of Thorm.1. From Eq.., th LpO stimator of th prformac of ay classificatio algorithm A computd from D satisfis R p A, D = R p, = 1 1 p p E p = 1 1 p p E p 1 {A D X i Y i} i ē i ē v E p+1 1 {v= {i}} 1 {A D X i Y i}, sic thr is a uiqu st of idics v with cardiality p + 1 such that v = {i}. Th R p, = 1 1 p p v E p+1 1 {v= {i}} 1 {i ē} 1 { }. A Dv\{i} X i Y i E p Furthrmor for v ad i fixd, E p 1 {v= {i}} 1 {i ē} = 1 {i v} sic thr is a uiqu st of idics such that = v \ i. O gts R p, = 1 1 p p = 1 p+1 v E p+1 by oticig p p = p! p! p! =! p 1! p! 1 {i v} 1 { } A Dv\{i} X i Y i 1 p + 1 v E p+1 = p { }, A Dv\{i} X i Y i i v p+1. 7

8 Cliss ad Mary-Huard Th rl h m is a dtrmiistic ad symmtric fuctio of its argumts that dos oly dpd o m. Lt us also otic that h m Z 1,..., Z m rducs to th L1O stimator of th ris of th classifir A computd from Z 1,..., Z m, that is h m Z 1,..., Z m = R 1 A, D m = R 1, p+1..4 I th cotxt of tstig whthr two biary classifirs hav diffrt rror rats, this fact has alrady b poitd out by Fuchs t al W ow driv a gral uppr boud o th q-th momt q 1 of th LpO stimator that holds tru for ay classifir as log as th followig xpctatios ar wll dfid. Thorm.. For ay classifir A, lt A D ad A Dm b th corrspodig classifirs built from rspctivly D ad D m, whr m = p + 1. Th for vry 1 p 1 such that a classifir ca b computd from A o p traiig poits, ad for ay q 1, ] q] ] q ] E Rp, E Rp, E R1,m E R1,m..5 Furthrmor as log as p > / + 1, o also gts for q = E Rp, E Rp, ] ] ] ] E R1,m E R1,m m.6 for vry q > ] q] E Rp, E Rp, Bq, γ ] q R 1,m E R1,m q max q γ E m, m Var R1,m m,.7 whr γ > 0 is a umric costat ad Bq, γ dots th optimal costat dfid i th Rosthal iquality Propositio D.. Th proof is giv i Appdix A.1. Eq..5 ad.6 straightforwardly rsult from th Js iquality applid to th avrag ovr all prmutatios providd i Propositio.1. If p > / + 1, th itgr part /m bcoms largr tha 1 ad Eq..6 bcoms bttr tha Eq..5 for q =. As a cosquc of our stratgy of proof, th right-had sid of Eq..6 is qual to th classical uppr boud o th variac of U-statistics which suggsts it caot b improvd without addig furthr assumptios. Uli th abov os, Eq..7 is drivd from th Rosthal iquality, which abls us to uppr boud a sum r ξ i q of idpdt ad idtically ctrd radom variabls i trms of r ξ i q ad r Varξ i. Lt us rmar that, for q =, both trms of th right-had sid of Eq..7 ar of th sam ordr as Eq..6 up to costats. Furthrmor usig th Rosthal iquality allows taig advatag of th itgr part /m wh p > / + 1 uli what w gt by usig Eq..5 for q >. I particular it provids a w udrstadig of th bhavior of th LpO stimator wh p/ 1 as highlightd latr by Propositio 4.. 8

9 Prformac of CV to Estimat th Ris of NN 3. Nw bouds o LpO momts for th NN classifir Our goal is ow to spcify th gral uppr bouds providd by Thorm. i th cas of th NN algorithm A 1 itroducd by.1. Sic Thorm. xprsss th momts of th LpO stimator i trms of thos of th L1O stimator computd from D m with m = p + 1, th xt stp cosists i focusig o th L1O momts. Drivig uppr bouds o th momts of th L1O is achivd usig a gralizatio of th wll-ow Efro-Sti iquality s Thorm D.1 for Efro- Sti s iquality ad Thorm 15.5 i Bouchro t al. 013 for its gralizatio. For th sa of compltss, w first rcall a corollary of this gralizatio that is provd i Sctio D.1.4 s Corollary D.1. Propositio 3.1. Lt ξ 1,..., ξ dot idpdt Ξ-valud radom variabls ad ζ = fξ 1,..., ξ, whr f : Ξ R is ay masurabl fuctio. With ξ 1,..., ξ idpdt copis of th ξ i s, thr xists a uivrsal costat κ 1.71 such that for ay q, ζ Eζ q κq fξ 1,..., ξ i,..., ξ fξ 1,..., ξ i,..., ξ q/. Th applyig Propositio 3.1 with ζ = R 1 A, D m = R 1,m L1O stimator computd from D m with m = p + 1 ad Ξ = R d {0, 1} lads to th followig Thorm 3.1. It cotrols th ordr-q momts of th L1O stimator applid to th NN classifir. Thorm 3.1. For vry 1 1, lt A Dm m = p + 1 dot th NN classifir lart from D m ad R 1,m b th corrspodig L1O stimator giv by Eq... Th for q =, ] ] 3/ E R1,m E R1,m C 1 m ; 3.1 for vry q >, ] q ] E R1,m E R1,m C q q m q/, 3. with C 1 = + 16γ d ad C = 4γ d κ, whr γd is a costat arisig from Sto s lmma, s Lmma D.5 that grows xpotially with dimsio d, ad κ is dfid i Propositio 3.1. Its proof dtaild i Sctio A. rlis o Sto s lmma Lmma D.5. For a giv X i, it provs that th umbr of poits i D i havig X i amog thir arst ighbors is ot largr tha γ d. Th dpdc of our uppr bouds with rspct to γ d s xplicit costats C 1 ad C iducs thir strog dtrioratio as th dimsio d grows sic γ d 4.8 d 1. Thrfor th largr th dimsio d, th largr th rquird sampl siz for th uppr boud to b small at last smallr tha 1. Not also that th ti braig stratgy basd o th smallst idx i th prst wor is chos so that it surs Sto s lmma to hold tru. 9

10 Cliss ad Mary-Huard I Eq. 3.1, th asir cas q = abls to xploit xact calculatios ] of rathr tha uppr bouds o th variac of th L1O stimator. Sic E R1,m = R ris of A D p th NN classifir computd from D p, th rsultig 3/ /m rat is a strict improvmt upo th usual /m that is drivd from usig th sub-gaussia xpotial coctratio iquality provd by Thorm 4.4 i Dvroy t al By cotrast th largr q arisig i Eq. 3. rsults from th difficulty to driv a tight uppr boud for th xpctatio of } q with q >, whr D i 1{ A Di m X i A Di,j m X i rsp. D m i,j dots th sampl D m whr Z i has b rsp. Z i ad Z j hav b rmovd. W ar ow i positio to stat th mai rsult of this sctio. It follows from th combiatio of Thorm. coctig momts of th LpO ad L1O stimators ad Thorm 3.1 providig a uppr boud o th ordr-q momts of th L1O. Thorm 3.. For vry p, 1 such that p+, lt R p, dot th LpO ris stimator s. of th NN classifir A D dfid by.1. Th thr xist ow costats C 1, C > 0 such that for vry 1 p, for q =, ] ] 3/ E Rp, E Rp, C 1 p + 1 ; 3.3 m for vry q >, ] q ] E Rp, E Rp, C q q p+1 q/, 3.4 with C 1 = 18κγ d π ad C = 4γ d κ, whr γd dots th costat arisig from Sto s lmma Lmma D.5. Furthrmor i th particular sttig whr / + 1 < p, th for q =, for vry q >, ] ] 3/ E Rp, E Rp, C 1 p + 1 p+1, 3.5 ] q ] E Rp, E Rp, p + 1 Γ q 3/ max p + 1 p+1 q, p + 1 p+1 q 3 q/ 3.6, whr Γ = max C 1, C. 10

11 Prformac of CV to Estimat th Ris of NN Th straightforward proof is dtaild i Sctio A.3. Lt us start by oticig that both uppr bouds i Eq. 3.3 ad 3.4 dtriorat as p grows. This is o logr th cas for Eq. 3.5 ad 3.6, which ar spcifically dsigd to covr th stup whr p > / + 1, that is whr /m is o logr qual to 1. Thrfor uli Eq. 3.3 ad 3.4, ths last two iqualitis ar particularly rlvat i th stup whr p/ 1, as +, which has b ivstigatd by Shao 1993; Yag 006, 007; Cliss 014. Eq. 3.5 ad 3.6 lad to rspctiv covrgc rats at wors 3/ / for q = ad q / q 1 for q >. I particular this last rat bcoms approximatly qual to / q as q gts larg. O ca also mphasiz that, as a U-statistic of fixd ordr m = p + 1, th LpO stimator has a ow Gaussia limitig distributio, that is s Thorm A, Sctio Srflig, 1980 m Rp, E Rp, ] L N 0, σ + 1, whr σ 1 = Var gz 1 ], with gz = E h m z, Z,..., Z m ]. Thrfor th uppr boud giv by Eq. 3.5 is o-improvabl i som ss with rspct to th itrplay btw ad p sic o rcovrs th right magitud for th variac trm as log as m = p + 1 is assumd to b costat. Fially Eq. 3.6 has b drivd usig a spcific vrsio of th Rosthal iquality Ibragimov ad Sharahmtov, 00 statd with th optimal costat ad ivolvig a balacig factor. I particular this balacig factor has allowd us to optimiz th rlativ wight of th two trms btw bracts i Eq This lads us to claim that th dpdc of th uppr boud with rspct to q caot b improvd with this li of proof. Howvr w caot coclud that th trm i q 3 caot b improvd usig othr tchical argumts. 4. Expotial coctratio iqualitis This sctio provids xpotial coctratio iqualitis for th LpO stimator applid to th NN classifir. Our mai rsults havily rly o th momt iqualitis prviously drivd i Sctio 3, amly Thorm 3.. I ordr to mphasiz th gai allowd by this stratgy of proof, w start this sctio by succssivly provig two xpotial iqualitis obtaid with lss sophisticatd tools. W th discuss th strgth ad wass of ach of thm to justify th additioal rfimts w itroduc stp by stp alog th sctio. A first xpotial coctratio iquality for R p A, D = R p, ca b drivd by us of th boudd diffrc iquality followig th li of proof of Dvroy t al. 1996, Thorm 4.4 origially dvlopd for th L1O stimator. Propositio 4.1. For ay itgrs p, 1 such that p +, lt R p, dot th LpO stimator. of th classificatio rror of th NN classifir A D dfid by.1. Th for vry t > 0, P Rp, E Rp, > t t 8+p 1 γ d. 4.1 whr γ d dots th costat itroducd i Sto s lmma Lmma D.5. 11

12 Cliss ad Mary-Huard Th proof is giv i Appdix B.1. Th uppr boud of Eq. 4.1 strogly xploits th facts that: i for X j to b o of th arst ighbors of X i i at last o subsampl X, it rquirs X j to b o of th + p 1 arst ighbors of X i i th complt sampl, ad ii th umbr of poits for which X j may b o of th + p 1 arst ighbors caot b largr tha + p 1γ d by Sto s Lmma s Lmma D.5. This rasoig rsults i a rough uppr boud sic th domiator i th xpot xhibits a + p 1 factor whr ad p play th sam rol. Th raso is that w do ot distiguish btw poits for which X j is amog or abov th arst ighbors of X i i th whol sampl although ths two stups lad to highly diffrt probabilitis of big amog th arst ighbors i th traiig sampl. Cosqutly th dpdc of th covrgc rat o ad p i Propositio 4.1 ca b improvd, as cofirmd by forthcomig Thorms 4.1 ad 4.. Basd o th prvious commts, a sharpr quatificatio of th ifluc of ach ighbor amog th + p 1 os lads to th xt rsult. Thorm 4.1. For vry p, 1 such that p +, lt R p, dot th LpO stimator. of th classificatio rror of th NN classifir A D dfid by.1. Th thr xists a umric costat > 0 such that for vry t > 0, max P Rp, E Rp, > t, P E Rp, R t p, > t xp ], p p 1 1 with = 104κ1+γ d, whr γ d is itroducd i Lmma D.5 ad κ 1.71 is a uivrsal costat. Th proof is giv i Sctio B.. Uli Propositio 4.1, taig ito accout th ra of ach ighbor i th whol sampl abls us to cosidrably rduc th wight of p compard to that of i th domiator of th xpot. I particular, lttig p/ 0 as + with assumd to b fixd for istac mas th ifluc of th + p factor asymptotically gligibl. This would allow for rcovrig up to umric costats a similar uppr boud to that of Dvroy t al. 1996, Thorm 4.4, achivd with p = 1. Howvr th uppr boud of Thorm 4.1 dos ot rflct th right dpdcis with rspct to ad p compard with what has b provd for polyomial momts i Thorm 3.. I particular it dtriorats as p icrass uli th uppr bouds drivd for p > / + 1 i Thorm 3.. This drawbac is ovrcom by th followig rsult, which is our mai cotributio i th prst sctio. Thorm 4.. For vry p, 1 such that p+, lt R p, dot th LpO stimator of th classificatio rror of th NN classifir ˆf = A D dfid by.1. Th for vry t > 0, ] ] max P Rp, E Rp, > t, P E Rp, R t p, > t xp p + 1, 4. 1

13 Prformac of CV to Estimat th Ris of NN whr = 4 max C, C 1 with C1, C > 0 dfid i Thorm 3.1. Furthrmor i th particular sttig whr p > / + 1, it coms ] max P Rp, E Rp, xp 1 mi p + 1 p + 1 ] > t, P E Rp, R p, > t t 4Γ 3/, p + 1 p + 1 p + 1 1/3 t, 4Γ 4.3 whr Γ ariss i Eq. 3.6 ad γ d Lmma D.5. dots th costat itroducd i Sto s lmma Th proof has b postpod to Appdix B.3. It ivolvs diffrt argumts for drivig th two iqualitis 4. ad 4.3 dpdig o th rag of valus of p. Firstly for p /+1, a simpl argumt is applid to driv Iq. 4. from th two corrspodig momt iqualitis of Thorm 3. charactrizig th sub-gaussia bhavior of th LpO stimator i trms of its v momts s Lmma D.. Scodly for p > / + 1, w rathr xploit: i th appropriat uppr bouds o th momts of th LpO stimator giv by Thorm 3., combid with ii Propositio D.1 which stablishs xpotial coctratio iqualitis from gral momt uppr bouds. I accordac with th coclusios draw about Thorm 3., th uppr boud of Eq. 4. icrass as p grows uli that of Eq Th bst coctratio rat i Eq. 4.3 is achivd as p/ 1, whras Eq. 4. turs out to b uslss i that sttig. Howvr Eq. 4. rmais strictly bttr tha Thorm 4.1 as log as p/ δ 0, 1 as +. Not also that th costats Γ ad γ d ar th sam as i Thorm 3.1. Thrfor th sam commts rgardig thir dpdc with rspct to th dimsio d apply hr. I ordr to facilitat th itrprtatio of th last Iq. 4.3, w also driv th followig propositio provd i Appdix B.3 which focuss o th dscriptio of ach dviatio trm i th particular cas whr p > / + 1. Propositio 4.. With th sam otatio as Thorm 4., for ay p, 1 such that p +, p > / + 1, ad for vry t > 0 P R ] Γ p, E Rp, > p + 1 3/ p+1 whr Γ > 0 is th costat arisig from 3.6. t + p+1 t 3/ p + 1 Th prst iquality is vry similar to th wll-ow Brsti iquality Bouchro t al., 013, Thorm.10 xcpt th scod dviatio trm of ordr t 3/ istad of t for th Brsti iquality. With rspct to, th first dviatio trm is of ordr 3/ /, which is th sam as with th Brsti iquality. Th scod dviatio trm is of a somwhat diffrt ordr, that is p + 1/, as compard with th usual 1/ i th Brsti iquality. t, 13

14 Cliss ad Mary-Huard Nvrthlss w almost rcovr th / rat by choosig for istac p 1 log /, which lads to log /. Thrfor varyig p allows to itrpolat btw th / ad th / rats. Not also that th dpdc of th first sub-gaussia dviatio trm with rspct to is oly 3/, which improvs upo th usual rsultig from Iq. 4. i Thorm 4. for istac. Howvr this 3/ rmais crtaily too larg for big optimal v if this qustio rmais widly op at this stag i th litratur. Mor grally o strgth of our approach is its vrsatility. Idd th two abov dviatio trms dirctly rsult from th two uppr bouds o th momts of th L1O stablishd i Thorm 3.1. Thrfor ay improvmt of th lattr uppr bouds would immdiatly lad to hac th prst coctratio iquality without chagig th proof. 5. Assssig th gap btw LpO ad classificatio rror 5.1. Uppr bouds First, w driv w uppr bouds o diffrt masurs of th discrpacy btw R p, = R p A, D ad th classificatio rror L ˆf or th ris R ˆf = E L ˆf ]. Ths bouds o th LpO stimator ar compltly w for p > 1, som of thm big xtsios of formr os spcifically drivd for th L1O stimator applid to th NN classifir. Thorm 5.1. For vry p, 1 such that p ad +, lt R p, dot th LpO ris stimator s. of th NN classifir ˆf = A D dfid by.1. Th, ] E Rp, R ˆf 4 p π, 5.1 ad E Rp, R ˆf ] 18κγ d 3/ π p p π 5. Morovr, E Rp, L ˆf ] p π 5.3 I cotrast to th rsults i th prvious sctios, a w rstrictio o p ariss i Thorm 5.1, that is p. This coms from th us of Lmma D.6 provd by Dvroy ad Wagr 1979b, which givs a uppr boud o th L 1 stability of th NN classifir wh p obsrvatios ar rmovd from th traiig sampl D. Actually this uppr boud oly rmais maigful as log as 1 p. 14

15 Prformac of CV to Estimat th Ris of NN Proof of Thorm 5.1. Proof of 5.1: With ˆf = AD, Lmma D.6 immdiatly provids E Rp, L ˆf ] E = L ˆf ] E L ˆf ] 1{A E D X Y } 1 {A D = P A D X AD X 4 p π X Y } ] Proof of 5.: Th proof combis th prvious uppr boud with th o stablishd for th variac of th LpO stimator, that is Eq E Rp, E L ˆf ] ] ] ] ] = E Rp, E Rp, + E Rp, E L ˆf ] 18κγ d 3/ π p p, π which cocluds th proof. Th proof of Iq. 5.3 is mor itricat ad has b postpod to Appdix C.1. ] Kpig i mid that E Rp, = RA D p, th right-had sid of Iq. 5.1 is a uppr boud o th bias of th LpO stimator, that is o th diffrc btw th riss of th classifirs built from rspctivly p ad poits. Thrfor, th fact that this uppr boud icrass with p is rliabl sic th classifirs A D p+1 ad A D ca bcom mor ad mor diffrt from o aothr as p icrass. Mor prcisly, th uppr boud i Iq. 5.1 gos to 0 providd p / dos. With th additioal rstrictio p, this rducs to th usual coditio / 0 as + s Dvroy t al., 1996, Chap. 6.6 for istac, which is usd to prov th uivrsal cosistcy of th NN classifir Sto, Th mootoicity of this uppr uppr boud with rspct to ca sm somwhat uxpctd. O could thi that th two classifirs would bcom mor ad mor similar to ach othr as icrass ough. Howvr it ca b provd that, i som ss, this dpdc caot b improvd i th prst distributio-fr framwor s Propositio 5.1 ad Figur 1. Not that a uppr boud similar to that of Iq. 5. ca b asily drivd for ay ordr-q momt q at th pric of icrasig th costats by usig a + b q q 1 a q + b q, for vry a, b 0. W also mphasiz that Iq. 5. allows us to cotrol th discrpacy btw th LpO stimator ad th ris of th NN classifir, that is th xpctatio of its classificatio rror. Idally w would hav lid to rplac th ris R ˆf by th prdictio rror L ˆf. But with our stratgy of proof, this would rquir a additioal distributio-fr coctratio iquality o th prdictio rror of th NN classifir. To th bst of our owldg, such a coctratio iquality is ot availabl up to ow. Uppr boudig th squard diffrc btw th LpO stimator ad th prdictio rror is prcisly th purpos of Iq Provig th lattr iquality rquirs a compltly diffrt stratgy which ca b tracd bac to a arlir proof by Rogrs ad Wagr 15

16 Cliss ad Mary-Huard 1978, s th proof of Thorm.1 applyig to th L1O stimator. Lt us mtio that Iq. 5.3 combid with th Js iquality lad to a lss accurat uppr boud tha Iq Fially th appart diffrc btw th uppr bouds i Iq. 5. ad 5.3 rsults from th compltly diffrt schms of proof. Th first o allows us to driv gral uppr bouds for all ctrd momts of th LpO stimator, but xhibits a wors dpdc with rspct to. By cotrast th scod o is xclusivly ddicatd to uppr boudig th ma squard diffrc btw th prdictio rror ad th LpO stimator ad lads to a smallr. Howvr v if probably ot optimal, th uppr boud usd i Iq. 5. still abls to achiv miimax rats ovr som Höldr balls as provd by Propositio Lowr bouds Bias of th L1O stimator Th purpos of th xt rsult is to provid a coutr-xampl highlightig that th uppr boud of Eq. 5.1 caot b improvd i som ss. W cosidr th followig discrt sttig whr X = {0, 1} with π 0 = P X = 0 ], ad w dfi η 0 = P Y = 1 X = 0 ] ad η 1 = P Y = 1 X = 1 ]. I what follows this two-class grativ modl will b rfrrd to as th discrt sttig DS. Not that i th 3 paramtrs π 0, η 0 ad η 1 fully dscrib th joit distributio P X,Y, ad ii th distributio of DS satisfis th strog margi assumptio of Massart ad Nédélc 006 if both η 0 ad η 1 ar chos away from 1/. Howvr this favourabl sttig has o particular ffct o th forthcomig lowr boud xcpt a fw simplificatios alog th calculatios. Propositio 5.1. Lt us cosidr th DS sttig with π 0 = 1/, η 0 = 0 ad η 1 = 1, ad assum that is odd. Th thr xists a umric costat C > 1 idpdt of ad such that, for all / 1, th NN classifirs A D ad A D 1 satisfy E L A D L A D 1 ] C Th proof of Propositio 5.1 is providd i Appdix C.. Th rat / i th righthad sid of Eq. 5.1 is th achivd udr th grativ modl DS for ay /. As a cosquc this rat caot b improvd without ay additioal assumptio, for istac o th distributio of th X i s. S also Figur 1 blow ad rlatd commts. Empirical illustratio To furthr illustrat th rsult of Propositio 5.1, w simulatd data accordig to DS, for diffrt valus of ragig from 100 to 500 ad diffrt valus of ragig from 5 to 1. Figur 1 a displays th volutio of th absolut bias E L A D L A D 1 ] as a fuctio of, for svral valus of plai curvs. Th absolut bias is a odcrasig fuctio of, as suggstd by th uppr boud providd i Eq. 5.1 which is also plottd dashd lis to as th compariso. Th o-dcrasig bhavior of th absolut bias is ot always rstrictd to high valus of w.r.t., as illustratd i Figur 1 b which 16

17 Prformac of CV to Estimat th Ris of NN corrspods to DS with paramtr valus π 0, η 0, η 1 = 0., 0., 0.9. I particular th o-dcrasig bhavior of th absolut bias ow appars for a rag of valus of that ar smallr tha /. Not that a rough ida about th locatio of th pa, dotd by pa, ca b dducd as follows i th simpl cas whr η 0 = 0 ad η 1 = 1. For th pa to aris, th two classifirs basd o ad rspctivly 1 obsrvatios hav to disagr th most strogly. This rquirs o of th two classifirs say th first o to hav tis amog th arst ighbors of ach labl i at last o of th two cass X = 0 or X = 1. With π 0 < 0.5, th tis will most lily occur for th cas X = 0. Thrfor th discrpacy btw th two classifirs will b th highst at ay w obsrvatio x 0 = 0. For th ti situatio to aris at x 0, half of its ighbors hav to b 1. This oly occurs if i > 0 with 0 th umbr of obsrvatios such that X = 0 i th traiig st, ad ii 0 η η 1 = /, whr 0 rsp. 1 is th umbr of ighbors of x 0 such that X = 0 rsp. X = 1. Sic > 0, o has 0 = 0 ad th last xprssio boils dow to = 0η 1 η 0 η 1 1/ For larg valus of, o should hav 0 π 0, that is th pa should appar at pa π 0η 1 η 0 η 1 1/ I th sttig of Propositio 5.1, this rasoig rmarably yilds pa, whil it lads to pa 0.4 i th sttig of Figur 1 b, which is clos to th locatio of th obsrvd pas. This also suggsts that v smallr valus of pa ca aris by tuig th paramtr π 0 clos to 0. Lt us mtio that vry similar curvs hav b obtaid for a Gaussia mixtur modl with two disjoit classs ot rportd hr. O th o had this mpirically illustrats that th / rat is ot limitd to DS discrt sttig. O th othr had, all of this cofirms that this rat caot b improvd i th prst distributio-fr framwor. Lt us fially cosidr Figur 1 c, which displays th absolut bias as a fuctio of whr = Cof for diffrt valus of Cof, whr dots th itgr part. With this choic of, Propositio 5.1 implis that th absolut bias should dcras at a 1/ rat, which is supportd by th plottd curvs. By cotrast, pal d of Figur 1 illustrats that choosig smallr valus of, that is = Cof, lads to a fastr dcrasig rat Ma squard rror Followig a xampl dscribd by Dvroy ad Wagr 1979a, w ow provid a lowr boud o th miimal covrgc rat of th ma squard rror s also Dvroy t al., 1996, Chap. 4.4, p.415 for a similar argumt. 17

18 Cliss ad Mary-Huard Bias Bias a b Bias Bias Cof c Cof d Figur 1: a Evolutio of th absolut valu of th bias as a fuctio of, for diffrt valus of plai lis. Th dashd lis corrspod to th uppr boud obtaid i 5.1. b Sam as prvious, xcpt that data wr gratd accordig to th DS sttig with paramtrs π 0, η 0, η 1 = 0., 0., 0.9. Uppr bouds ar ot displayd i ordr to fit th scal of th absolut bias. c Evolutio of th absolut valu of th bias with rspct to, wh is chos such that = Cof dots th itgr part. Th diffrt colors corrspod to diffrt valus of Cof. d Sam as prvious, xcpt that is chos such that = Cof. 18

19 Prformac of CV to Estimat th Ris of NN Propositio 5.. Lt us assum is v, ad that P Y = 1 X = P Y = 1 = 1/ is idpdt of X. Th for = 1 odd, it rsults E R1, L ˆf ] = 1 0 t P R1, L ˆf ] > t dt 1 8 π 1 From th uppr boud of ordr / providd by Iq. 5.3 with p = 1, choosig = 1 lads to th sam 1/ rat as that of Propositio 5.. This suggsts that, at last for vry larg valus of, th / rat is of th right ordr ad caot b improvd i th distributio-fr framwor Miimax rats Lt us coclud this sctio with a corollary, which provids a fiit-sampl boud o th gap btw R p, ad R ˆf = E L ˆf ] with high probability. It is statd udr th sam rstrictio o p as th prvious Thorm 5.1 it is basd o, that is for p. Corollary 5.1. With th otatio of Thorms 4. ad 5.1, lt us assum p, 1 with p, +, ad p / + 1. Th for vry x > 0, thr xists a vt with probability at last 1 x such that R ˆf R p, 1 p 1 x + 4 π p, 5.4 whr ˆf = A D. Proof of Corollary 5.1. Iq. 5.4 rsults from combiig th xpotial coctratio rsult drivd for R p,, amly Iq. 4. from Thorm 4. ad th uppr boud o th bias, that is Iq R ˆf R ] ] R E p, ˆf E Rp, + Rp, R p, 4 π p + p + 1 x Not that th right-had sid of Iq. 5.4 could b usd to driv bouds o R ˆf that sm similar to cofidc bouds. Howvr w do ot rcommd doig this i practic for svral rasos. O th o had, Iq. 5.4 rsults from th rpatd us of coctratio iqualitis whr umric costats ar ot optimizd at all. This would lad to rquir a larg sampl siz for th dviatio trms to b small i practic. O th othr had, xplicit umric costats such as i Corollary 5.1 xhibit a dpdc o γ d 4.8 d 1, which bcoms xpotially larg as d icrass. Provig that this dpdc ca b wad or ot rmais a compltly op qustio at this stag. Nvrthlss o ca highlight that, for a giv, icrasig d will quicly ma th dviatio trm largr tha 1, whras both R ˆf ad R p, blog to 0, 1]. 19

20 Cliss ad Mary-Huard Th right-most trm of ordr / i Iq. 5.4 rsults from th bias. This is a cssary pric to pay which caot b improvd i th prst distributio-fr framwor accordig to Propositio 5.1. Bsids combiig th rstrictio p with th usual cosistcy costrait / = o1 lads to th coclusio that small valus of p w.r.t. hav almost o ffct o th covrgc rat of th LpO stimator. Waig th y rstrictio p would b cssary to pottially uac this coclusio. I ordr to highlight th itrst of th abov dviatio iquality, lt us dduc a optimality rsult i trms of miimax rat ovr Höldr balls Hτ, α dfid by { Hτ, α = g : R d R, gx gy τ x y α}, with α ]0, 1 ad τ > 0. I th followig statmt, Corollary 5.1 is usd to prov that, uiformly with rspct to, th LpO stimator R p, ad th ris R ˆf of th NN classifir rmai clos to ach othr with high probability. Propositio 5.3. With th sam otatio as Corollary 5.1, for vry C > 1 ad θ > 0, thr xists a vt of probability at last 1 C 1 o which, for ay p, 1 such that p, +, ad p /+1, th LpO stimator of th NN classifir satisfis 1 θ R ˆf ] L θ 1 C 4 log R ˆf 4 p L π R p A, D L 1 + θ R ˆf ] L + θ 1 C 4 log R ˆf + 4 p L π, 5.5 whr L dots th classificatio rror of th Bays classifir. Furthrmor if o assums th rgrssio fuctio η blogs to a Höldr ball Hτ, α for som α ]0, mid/4, 1 rcall that X i R d ad τ > 0, th choosig = = 0 α α+d lads to R p A, D L + R ˆf L, a.s Iq. 5.5 givs a uiform cotrol ovr of th gap btw th xcss ris R ˆf L ad th corrspodig LpO stimator R p ˆf L with high probability. Th dcrasig rat i C 1 of this probability is dirctly rlatd to th log factor i th lowr ad uppr bouds. This dcrasig rat could b mad fastr at th pric of icrasig th xpot of th log factor. I a similar way th umric costat θ has o prcis maig ad ca b chos as clos to 0 as w wat, ladig to icras o of th othr dviatio trms by a umric factor θ 1. For istac o could choos θ = 1/ log, which would rplac th log by a log. Th quivalc stablishd by Eq. 5.6 rsults from owig that this choic = mas th NN classifir achiv th miimax rat α α+d ovr Höldr balls Yag, This holds tru for α ]0, 1 as log as d 4. Howvr if d < 4 th miimax rat is oly achivd ovr ]0, d/4. This limitatio rsults from th dpdc of th dviatio trms with rspct to i Eq. 5.5, which is ot optimal ad should b furthr improvd. 0

21 Prformac of CV to Estimat th Ris of NN Proof of Propositio 5.3. Lt us dfi K as th maximum valu of ad assum x = C log for som costat C > 1 for ay 1 K. Lt us also itroduc th vt Ω = { Th P Ω c ] 1 C 1 1 K, K x = =1 R ˆf R p A, D x + 4 p π 0, as +, sic a uio boud lads to K =1 C log = K C log C 1 log = 1 C 1 Furthrmor combiig for a, b > 0 th iquality ab a θ + b θ /4 for vry θ > 0 with a + b a + b, it rsults that hc Iq x θ R ˆf L + θ 1 4 R ˆf L x θ R ˆf L + θ 1 4 R ˆf C log, L Lt us ow prov th xt quivalc, amly 5.6, by mas of th Borl-Catlli lmma. First Yag 1999 combid with Thorm 7 i Chaudhuri ad Dasgupta 014 provid that th miimax rat ovr th Höldr ball Hτ, α is achivd by th NN classifir with =, that is R ˆf L α α+d, whr a b mas thr xist umric costats l, u > 0 such that l b a u b. Morovr it is th asy to chc that D 1 := θ 1 4 R ˆf L C log Cθ d 3α α+d log = o + R ˆf L, D := p = 0 d α+d = o + R ˆf L. Bsids 1 C 1 P Rp A, D L R ˆf L Ωc ] P R ˆf L > θ + D 1 + D R ˆf L Rp A, D L = P R ˆf 1 L > θ + D 1 + D R ˆf. L }. 1

22 Cliss ad Mary-Huard Lt us ow choos ay ɛ > 0 ad itroduc th squc of vts {A ɛ} 1 such that Rp A, D L A ɛ = R ˆf 1 L > ɛ. Usig that D 1 ad D ar gligibl with rspct to R ˆf L as +, thr xists a itgr 0 = 0 ɛ > 0 such that, for all 0 ad with θ = ɛ/, θ + D 1 + D R ˆf L ɛ. Hc Rp A, D L P A ɛ ] P R ˆf 1 L > θ + D 1 + D R ˆf 1 L C 1 Fially, choosig ay C > lads to + =1 P A ɛ ] < +, which provids th xpctd coclusio by mas of th Borl-Catlli lmma. 6. Discussio Th prst wor provids svral w rsults quatifyig th prformac of th LpO stimator applid to th NN classifir. By xploitig th coxio btw LpO ad U- statistics Sctio, th polyomial ad xpotial iqualitis drivd i Sctios 3 ad 4 giv som w isight o th coctratio of th LpO stimator aroud its xpctatio for diffrt rgims of p/. I Sctio 5, ths rsults srv for istac to coclud to th cosistcy of th LpO stimator towards th ris or th classificatio rror rat of th NN classifir Thorm 5.1. Thy also allow us to stablish th asymptotic quivalc btw th LpO stimator shiftd by th Bays ris L ad th xcss ris ovr som Höldr class of rgrssio fuctios Propositio 5.3. It is worth mtioig that th uppr-bouds drivd i Sctios 4 ad 5 s for istac Thorm 5.1 ca b miimizd by choosig p = 1, suggstig that th L1O stimator is optimal i trms of ris stimatio wh applid to th NN classificatio algorithm. This obsrvatio corroborats th rsults of th simulatio study prstd i Cliss ad Mary-Huard 011, whr it is mpirically show that small valus of p ad i particular p = 1 lad to th bst stimatio of th ris for ay fixd, whatvr th lvl of ois i th data. Th suggstd optimality of L1O for ris stimatio is also cosistt with rsults by Burma 1989 ad Cliss 014, whr it is provd that L1O is asymptotically th bst cross-validatio procdur to prform ris stimatio i th cotxt of low-dimsioal rgrssio ad dsity stimatio rspctivly. Altrativly, th LpO stimator ca also b usd as a data-dpdt calibratio procdur to tu, by choosig th valu ˆ p which miimizs th LpO stimat. For istac

23 Prformac of CV to Estimat th Ris of NN i classificatio, LpO ca b usd to gt th valu of ladig to th bst prdictio prformac. I this cotxt th valu of p th splittig ratio ladig to th bst NN classifir ca b vry diffrt from p = 1. This is illustratd by th simulatio rsults summarizd by Figur i Cliss ad Mary-Huard 011 whr p has to b largr tha 1 as th ois lvl bcoms strog. This phomo is ot limitd to th NN classifir, but xtds to various stimatio/prdictio problms Brima ad Spctor, 199; Arlot ad Lrasl, 01; Cliss, 014. If w tur ow to th qustio of idtifyig th bst prdictor amog svral cadidats, choosig p = 1 also lads to poor slctio prformacs as provd by Shao 1993, Eq. 3.8 with th liar rgrssio modl. For th LpO, Shao 1997, Thorm 5 provs th modl slctio cosistcy if p/ 1 ad p + as +. For rcovrig th bst prdictor amog two cadidats, Yag 006, 007 provd th cosistcy of CV udr coditios rlatig th optimal splittig ratio p to th covrgc rats of th prdictors to b compard, ad furthr rquirig that mip, p + as +. Although th focus of th prst papr is diffrt, it is worth mtioig that th coctratio rsults stablishd i Sctio 4 ar a sigificat arly stp towards drivig thortical guarats o LpO as a modl slctio procdur. Idd, xpotial coctratio iqualitis hav b a y igrdit to assss modl slctio cosistcy or modl slctio fficicy i various cotxts s for istac Cliss 014 or Arlot ad Lrasl 01 i th dsity stimatio framwor. Still thortically ivstigatig th bhavior of ˆ p rquirs som furthr ddicatd dvlopmts. O first stp towards such rsults is to driv a tightr uppr boud o th bias btw th LpO stimator ad th ris. Th bst ow uppr boud currtly availabl is drivd from Dvroy ad Wagr 1980, s Lmma D.6 i th prst papr. Ufortuatly it dos ot fully captur th tru bhavior of th LpO stimator with rspct to p at last as p bcoms larg ad could b improvd i particular for p > as mphasizd i th commts followig Thorm 5.1. Aothr importat dirctio for studyig th modl slctio bhavior of th LpO procdur is to prov a coctratio iquality for th classificatio rror rat of th NN classifir aroud its xpctatio. Whil such coctratio rsults hav b stablishd for th NN algorithm i th fixd-dsig rgrssio framwor Arlot ad Bach, 009, drivig similar rsults i th classificatio cotxt rmais a challgig problm to th bst of our owldg. Acowldgmts Th authors would li to tha th associat ditor ad rviwrs for thir highlightful commts which gratly hlpd to improv th prstatio of th papr. This wor was partially fudd by th BFast PEPS CNRS. 3

24 Cliss ad Mary-Huard Appdix A. Proofs of polyomial momt uppr bouds A.1. Proof of Thorm. Th proof rlis o Propositio.1 that allows to rlat th LpO stimator to a sum of idpdt radom variabls. I th followig, w distiguish btw th two sttigs q = whr xact calculatios ca b carrid out, ad q > whr oly uppr bouds ca b drivd. Wh q >, our proof dals sparatly with th cass p / + 1 ad p > / + 1. I th first o, a straightforward us of Js s iquality lads to th rsult. I th scod sttig, o has to b mor cautious wh drivig uppr bouds. This is do by usig th mor sophisticatd Rosthal s iquality, amly Propositio D.. A.1.1. Exploitig Propositio.1 Accordig to th proof of Propositio.1, it ariss that th LpO stimator ca b xprssd as a U-statistic sic with R p, = 1 W Z! σ1,..., Z σ, σ W Z 1,..., Z = ad h m Z 1,..., Z m = 1 m m 1 m m a=1 h m Za 1m+1,..., Z am 1 { A Di m X i Y i } = R 1, p+1, with m = p + 1 whr A Di m. dots th classifir basd o sampl D i m = Z 1,..., Z i 1, Z i+1,..., Z m. Furthr ctrig th LpO stimator, it coms ] R p, E Rp, = 1! σ W Z σ1,..., Z σ, whr W Z 1,..., Z = W Z 1,..., Z E W Z 1,..., Z ]. Th with h m Z 1,..., Z m = h m Z 1,..., Z m E h m Z 1,..., Z m ], o gts ] q] E Rp, E Rp, E W Z1,..., Z q ] Js s iquality 1 m = E m q h m Zi 1m+1,..., Z im q q m = E h m Zi 1m+1,..., Z im. m A.1 4

25 Prformac of CV to Estimat th Ris of NN A.1.. Th sttig q = If q =, th by idpdc it coms ] q] m E Rp, E Rp, Var h m Zi 1m+1,..., Z im m which lads to th rsult. A.1.3. Th sttig q > = m m Var h m Zi 1m+1,..., Z im ] 1 = Var R1 A, Z 1, p+1, m If p / + 1: A straightforward us of Js s iquality from A.1 provids ] q] E Rp, E Rp, m 1 m = E R1, p+1 E E hm Zi 1m+1,..., Z im q] R1, p+1 ] q]. If p > / + 1: Lt us ow us Rosthal s iquality Propositio D. by itroducig symmtric radom variabls ζ 1,..., ζ /m such that 1 i /m, ζ i = h m Zi 1m+1,..., Z im hm Zi 1m+1,..., Z im, whr Z 1,..., Z ar i.i.d. copis of Z 1,..., Z. Th it coms for vry γ > 0 q m m E h m Zi 1m+1,..., Z im E ζ i q, which implis q m E h m Zi 1m+1,..., Z im Bq, γ max γ m E ζ i q m ], E ζi q ]. Th usig for vry i that E ζ i q ] q E hm Zi 1m+1,..., Z im q ], 5

26 Cliss ad Mary-Huard it coms q m E h m Zi 1m+1,..., Z im Bq, γ max q γ ] q ] E R1,m E R1,m, m q Var R1,m. m Hc, it rsults for vry q > ] q] E Rp, E Rp, q+1 ] Bq, γ max q q ] q/ γ E R1,m E R1,m, Var R1,m m m q, which cocluds th proof. A.. Proof of Thorm 3.1 Our stratgy of proof follows svral idas. Th first o cosists i usig Propositio 3.1 which says that, for vry q, hm Z 1,..., Z m q m κq h m Z 1,..., Z m h m Z 1,..., Z j m,..., Z, q/ j=1 whr h m Z 1,..., Z m = R 1,m by Eq..4, ad h m Z 1,..., Z m = h m Z 1,..., Z m E h m Z 1,..., Z m ]. Th scod ida cosists i drivig uppr bouds of j h m = h m Z 1,..., Z j,..., Z m h m Z 1,..., Z j,..., Z m by rpatd uss of Sto s lmma, that is Lmma D.5 which uppr bouds by γ d th maximum umbr of X i s that ca hav a giv X j amog thir arst ighbors. Fially, for tchical rasos w hav to distiguish th cas q = whr w gt tightr bouds ad q >. A..1. Uppr boudig j h m For th sa of radability lt us ow us th otatio D i = D m i s Thorm.1, ad lt D i j dot th st Z 1,..., Z j,..., Z whr th i-th coordiat has b rmovd. Th, j h m = h m Z 1,..., Z m h m Z 1,..., Z j,..., Z m is ow uppr boudd by j h m 1 m m { } 1 A i j Di X i Y { } i A Di j X i Y i 1 m m { } i j A Di X i A D i. A. j X i 6