In the classical period up to the 1980 s, research on regression

Transcription

1 O obust egesso wth hgh-dmesoal edctos Nouedde El Kaou, Deek Bea, Pete Bckel, Chghway Lm, ad B Yu Uvesty of Calfoa, Bekeley, ad Natoal Uvesty of Sgaoe Submtted to Poceedgs of the Natoal Academy of Sceces of the Uted States of Ameca We study egesso M-estmates the settg whee, the umbe of covaates, ad, the umbe of obsevatos, ae both lage but We fd a exact stochastc eesetato fo the dstbuto of β = agm β R = ρy X β at fxed ad ude vaous assumtos o the objectve fucto ρ ad ou statstcal model A scala adom vaable whose detemstc lmt ρκ ca be studed whe / κ > 0 lays a cetal ole ths eesetato Futhemoe, we dscove a o-lea system of two detemstc equatos that chaactezes ρκ Iteestgly, the system shows that ρκ deeds o ρ though oxmal mags of ρ as well as vaous asects of the statstcal model udelyg ou study Seveal classcal esults statstcs ae ueded I atcula, we show that whe / s lage eough, least-squaes becomes efeable to least-absolute-devatos fo double exoetal eos obust egesso ox fucto hgh-dmesoal statstcs cocetato of measue Abbevatos: EPE, exected edcto eo; L =, equal law; LAD, least absolute devatos; d, deedet detcally dstbuted; fd, fte dmesoal I the classcal eod u to the 980 s, eseach o egesso models focused o stuatos fo whch the umbe of covaates was much smalle tha, the samle sze Least squaes egesso LSE was the ma fttg tool used but ts sestvty to outles came to the foe wth the wok of Tukey, Hube, Hamel ad othes statg the 950 s Gve the model Y = X β 0+ɛ ad M-estmato methods descbed the abstact, t follows fom the dscusso [7] 70 fo stace that, f the desg matx X a matx whose -th ow s X s o sgula, ude vaous egulaty codtos o X, ρ, ψ = ρ ad the d eos {ɛ } =, β s asymtotcally omal wth mea β 0 ad covaace matx Cρ, ɛx X Hee Cρ, ɛ = E ψ ɛ /[E ψ ɛ] ad ɛ has the same dstbuto as ɛ s It follows that, fo fxed, the elatve effcecy of M estmates such as LAD, to LSE, does ot deed o the desg matx Thus, LAD has the same advatage ove LSE fo heavy taled dstbutos as the meda has ove the mea I ecet yeas thee has bee geat focus o the case whee ad ae commesuate ad lage Geatest atteto has bee ad to the sase case whee the umbe of ozeo coeffcets s much smalle tha o Ths has bee acheved by addg a l tye of ealty to the quadatc objectve fucto of LSE, the case of the LASSO Ufotuately, these tyes of methods esult based estmates of the coeffcets ad statstcal feece, as oosed to edcto, becomes oblematc Hube [6] was the fst to vestgate the egme of lage wth Hs esults wee followed u by Potoy [9] ude weake codtos Hube showed that the behavo foud fo fxed essted egos such as / 0 ad 3 / 0 That s, estmates of coeffcets ad cotasts wee asymtotcally Gaussa ad elatve effceces of methods dd ot deed o the desg matx Hs agumets wee, at, heustc but well cofmed by smulato He also oted out a susg featue of the egme, / κ > 0 fo LSE; ftted values wee ot asymtotcally Gaussa He was uable to deal wth ths egme othewse, see the dscusso o 80 of [6] I ths ae we ted to, at heustcally ad wth comute valdato, aalyze fully what haes obust egesso whe / κ < We do lmt ouselves to Gaussa covaates but eset gouds that the behavo holds much moe geeally We also vestgate the sestvty of ou esults to the geomety of the desg matx We have chose to use heustcs because we beleve successful geeal oofs by us o othes wll eque a geat deal of tme ad ehas ema uesolved We oceed the mae of Hube [6] who also develoed hghly lausble esults buttessed by smulatos, may of whch have ot yet bee establshed goously We fd that : the asymtotc omalty ad ubasedess of estmates of coodates ad cotasts whch, ulke ftted values, have coeffcets deedet of the obseved covaates esst these stuatos; ths haes at scale /, as the fxed case, at least whe the mmal ad maxmal egevalues of the covaace of the edctos stay bouded away fom 0 ad esectvely These fdgs ae obtaed by usg tcate leave-oe-out etubato agumets both fo the data uts ad edctos; exhbtg a a of maste equatos fom whch the asymtotc mea squae edcto eo ad the coect exessos fo asymtotc vaaces ca be ecoveed; 3 showg that these two quattes deed a olea way o /, the eo dstbuto, the desg matx ad the fom of the objectve fucto, ρ It s woth otg that ou fdgs ued the classcal tuto that the deal objectve fucto s the egatve logdesty of the eo dstbuto That s, we show that whe / s lage eough, t becomes efeable to use least-squaes athe tha LAD fo double exoetal eos We llustate ths ot Fgue 3 The Ma esults secto cotas a detaled esetato of ou esults We gve some examles ad suotg Reseved fo Publcato Foototes ad the vecto defg the cotasts has om bouded away fom 0 ad wwwasog/cg/do/0073/as PNAS Issue Date Volume Issue Numbe 7

2 smulatos the Examles secto We eset ou devato the last secto Ma esults We cosde the followg obust egesso oblem: let β be β = agm β R ρy X β [] = Hee X R, Y = X β 0 + ɛ, whee β 0 R, ɛ s a adom scala eo deedet of the vecto X R ρ s a covex fucto We assume that the as {ɛ } = ad {X } = ae deedet Futhemoe, we assume that X s ae deedet Ou am s to chaacteze the dstbuto of β As we wll dscuss late, ou aoach s ot lmted to ths stadad obust egesso settg: we ca, fo stace, shed lght o smla questos weghted egesso The followg lemma s easly show by usg the otatoal vaace of the Gaussa dstbuto see SI Lemma Suose that X = λ X, whee X s ae d N 0, Σ, wth Σ of ak, ad {λ } = ae o zeo scalas, deedet of {X } = Call βρ; β0, Σ the soluto of Equato [ ] Whe >, we have the stochastc eesetato βρ; β 0, Σ L = β 0 + βρ; β 0, Id β 0 Σ / u, whee u s ufom o the shee of adus R ad s deedet of βρ; 0, Id β 0 Futhemoe, βρ; β 0, Id β 0 L = βρ; 0, Id I lght of ths esult, t s clea that we just eed to udestad the dstbuto of βρ; 0, Id to udestad that of βρ; β 0, Σ Result Suose that ρ s a o-lea covex fucto Let us call ρ, = βρ; 0, Id We assume that X = λ X, whee X ae d N 0, Id ad {λ } = ae o-zeo scalas, deedet of {X } = We also assume that Y = ɛ e β 0 = 0 ad {ɛ } = ae deedet of {X } = The, ude egulaty codtos o {ɛ } =, {λ } = ad ρ, ρ, has a detemstc lmt obablty as ad ted to fty whle / κ < We call ths lmt ρκ Let us call ẑ ɛ = ɛ + λ ρκz, whee Z N 0, ae d ad deedet of {ɛ} = ad {λ } = We ca deteme ρκ though solvg [ E ox cλ ρ] ẑ ɛ lm = = κ, E λ lm [ẑ ɛ ox cλ ρẑ ɛ] = = κ ρκ, [S] whee c s a ostve detemstc costat to be detemed fom the above system The exectatos above ae take wth esect to the jot dstbuto of {ɛ } =, {λ } = ad {Z } = The ox abbevato efes to the oxmal mag whch s stadad covex otmzato see [8] Oe of ts def- to s ox c ρx = agm y R ρy + x y Coollay Imotat secal case Whe fo all, λ =, ad ɛ s ae d, the same coclusos hold but the system chaactezg ρκ becomes: f ẑ = ɛ + ρκz, whee ɛ has the same dstbuto as ɛ ad s deedet of Z N 0,, { E [ox c ρ] ẑ ɛ = κ, E [ẑ ɛ ox c ρẑ ɛ] = κρκ [S] c The asymtotc omalty the fd covegece sese ad ubasedess of β s a cosequece of Result ad Lemma Note the comlcated teacto of ρ, the dstbuto of ɛ, the dstbuto of the X s ad κ detemg ρκ I the fxed κ = 0 case, X N 0, Σ, the cotbuto of the desg s just Σ, whch detemes the coelato matx of β I geeal the coelato stuctue s the same but the vaaces also deed o the desg As ou agumets wll show, we exect that the esults coceg ρκ detaled Result wll hold whe the assumtos of omalty o {X } = ae elaced by assumtos o cocetato of quadatc foms X Results o fd covegece of β also aea lkely to hold ude these weakeed estctos The dffeece betwee the systems of equatos chaactezg ρκ Result ad Coollay hghlghts the motace of the geomety of the edctos, X, the esults As a matte of fact, f we cosde the case whee Σ = Id ad λ s ae d wth E λ =, both stuatos the X s have covaace Id ad ae ealy othogoal to oe aothe; howeve, the settg of Result, X / s close to λ - hece vaable wth - wheeas the settg of Coollay, the X s all have almost the same om ad hece ae ea a shee The motace of the geomety of the vectos of edctos ths stuato s hece a geealzato of smla heomea that wee hghlghted [3] fo stace Futhe examles of a dffeet atue detaled [4] 7 llustate the fudametal motace of ou mlct geometc assumtos o the desg matx Ou aalyss also exteds to the case whee ρ s elaced by ρ, whee fo stace ρ = w ρ the weghted egesso case as log as {w } = s deedet of {X } = Oe smly eeds to elace ρ by ρ the system [S] above ad take exectato wth esect to these quattes, too We efe the teested eade to [4], 6 Examles We llustate the qualty of ou esults o a few umecal examles, showg the motace of both the objectve fucto ad the dstbuto of the eos the behavo of ρκ Fo smlcty, we focus oly o the case whee λ = fo all, e the case of Gaussa edctos a examle wth λ adom s the SI We also assume that ɛ s ae d Least-squaes I ths case, ρx = x / ad ψx = x Hece, ox c ρ = x Elemetay comutatos the show that +c c = κ/ κ We also fd that l κ = κ/ κσɛ, whee σɛ s the vaace of ɛ Natually, the case of least-squaes, oe ca use esults coceg Wshat dstbuto [] as well as the exlct fom of β to vefy mathematcally that ths exesso s coect We also ote that ths case, the dstbuto of ɛ does ot matte, oly ts vaace Meda egesso LAD Ths case, whee ρ takes values ρx = x, s substatally moe teestg ad eveals the motace of the teacto betwee objectve fucto ad eo dstbuto Clealy, we fst have to comute the ox of the fucto ρ It s well-kow ad ot dffcult to show that ths ox s the soft-thesholdg fucto Moe fomally, usg the otato x + = maxx, 0, we have, fo ay t > 0, ox t ρy = sgy y t + I ths subsecto, we use the otato l stead of ρ We wte βρ; β0, Σ stead of βρ; β 0, Σ; {ɛ } =, {λ } = fo smlcty wwwasog/cg/do/0073/as Footle Autho

3 Relatve eo of E L, vs Lκ comuted fom system, Gaussa eos, 000 smulatos = that the secod equato the system [S] becomes κl κ = s h κ + c κ, [ ] ] = s [h κ + κ Φ + κ, 00 whee h s the fucto such that fo t [0, ], E,/L L κ ht = t π Φ [ + t]/ ex [Φ [ + t]/] / Fally, callg ζ the fucto such that fo t [0, ], f ϕ deotes the stadad omal desty, ζt = Φ t ϕ[φ t] Φ t t, / Fg Relatve eos: E l, l, Gaussa eos, 000 smulatos κ Case of Gaussa eos futhe maulatos show that we ca solve fo s as a fucto of κ ad theefoe fo l κ Ou fal exesso s that, whe the ɛ s ae d N 0, σ ɛ, l κ = κ ζ[ + κ]/ σɛ ζ[ + κ]/ Fgue comaes ths exesso fo l κ to E l, obtaed by smulatos The comaso s doe by comutg elatve eos A fgue comag the actual values, whch ae also of teest, s the SI Let us call s = l κ + σ ɛ Whe ɛ s ae d N 0, σ ɛ, ẑ ɛ N 0, s The fst equato of ou system [S] theefoe becomes P Z > c/s = κ, whee Z N 0, Hece, c/s = Φ + κ/, whee Φ s the quatle fucto fo the stadad omal dstbuto We ow tu ou atteto to the secod equato the system [S] We have [y ox t ρy] = y y t + t y t Usg the fact that c/s = Φ + κ/, comutatos show Case of eos wth symmetc dstbuto We call f,ɛ the desty of ẑ ɛ ad do the deedece of ρκ o ρ ad κ fom ou otatos fo smlcty The fst equato of system [S] stll eads P ẑ ɛ > c = κ Let us call F,ɛ F,ɛ the cdf of ẑ ɛ ad F,ɛ = F,ɛ Let us deote by the fuctoal vese of F,ɛ Itegato by ats, symmety of f,ɛ as well as the above chaactezato of c fally lead to the mlct chaactezato of l κ deoted smly by fo shot the ext Relatve eo of E L, vs κ comuted fom system, double exoetal eos, 000 smulatos L = E L,/ E L, ad L κ/ κ comuted fom system, double exoetal eos, 000 smulatos L = E L,/ L κ E L,/ E L, Aveage ove 000 smulatos Pedcto fom heustcs / / Fg Relatve eos: E l, l, double exoetal eos, 000 κ smulatos Fg 3 Pedcto e l κ/l κ vs ealzed value of E l, /E l,, double exoetal eos Susgly, accodg to ths measue, t becomes efeable to use oday least-squaes athe tha l -egesso whe the eos ae double-exoetal ad κ s suffcetly lage Footle Autho PNAS Issue Date Volume Issue Numbe 3

4 equato κ = 4 F,ɛ κ/ x F,ɛxdx σ ɛ [] We ote assg that + σɛ = 4 x F 0,ɛxdx; theefoe the evous equato ca be ewtte κ = F,ɛ 4 κ/ 0 x F,ɛxdx, a coveet equato to wok wth umecally whe κ s small Case of double exoetal eos We ow eset a comaso of smulato esults to umecal solutos of System [ S] whe the eos ae double exoetal It should be oted that ths case the cdf F,ɛ takes values [ ] t / F,ɛt = Φ + e e t Φ [ t + ] [ ] e t t Φ It s also clea ths case that σ ɛ = We used all ths fomato to solve Equato [] fo, by dog a dchotomous seach Fgue llustates ou esults by showg the elatve eos betwee E l, comuted fom smulatos ad umecal solutos of system [ S] wth aoate aametes A fgue comag the actual values s the SI Othe objectve fuctos We have caed out smla comutatos ad valdatos of esults fo othe objectve fuctos, cludg the Hube objectve fuctos, the objectve fuctos aeag quatle egesso, as well as l 3 ad l 5 objectve fuctos - the latte two moe fo the aalytcal tactablty tha fo the statstcal teest We efe the eade to [4] fo detals Futhe emaks The chaactezatos of ρκ allows us to comae the efomace of vaous egesso methods fo vaous eo dstbutos Oe mathematcal ad statstcal cosequece s that we ca otmze ove ρ to mmze ρκ whe the dstbuto of the eos s gve ad log-cocave ad we ae the setu of Gaussa edctos We have doe ths the comao ae [] Qute deedetly, we ca vestgate the efomace of say meda egesso vs least squaes fo a age of values of κ I the case of double exoetal eos, t s wellkow see eg [7] that meda egesso s twce as effcet as least-squaes whe κ s close to 0 As ou smulatos ad comutatos llustate, ths s ot the case whe κ s ot close to zeo Ideed, whe κ > 3 o so, l κ < l κ fo double exoetal eos Ths should seve as cauto agast usg atual maxmum-lkelhood methods hgh-dmeso sce they tu out to be subotmal eve aaetly favoable stuatos Devato We ow tu ou atteto to the devato of the system of equatos [ S] eseted Result Ou aoach hges o a double leave-oe out aoach, the use of cocetato oetes of ceta quadatc foms ad the Shema-Woodbuy-Moso fomula of lea algeba We focus o the case β 0 = 0 ad Σ = Id Lemma guaatees that we ca do so wthout loss of geealty Note that ths case Y = ɛ We call βρ; 0, Id smly β fom ow o We also assume that ρ has two devatves We call the esduals R = ɛ X β ad use the otato X = {X j} j Recall that ψ = ρ We ote that ude ou assumtos β satsfes the gadet equato Xψɛ X β = 0 [3] I the devatos that follow, we wll use eeatedly the fact that f X ae d N 0, Id ad A s a sequece of detemstc symmetc matces, ude mld codtos o the gowth of tace A k wth k N, we have as ad gow su =,, X A X tace A = op May methods ca be used to show ths cocetato esult A atculaly smle oe s to comute the secod ad fouth cumulats of X A X It shows that the esult holds as soo as tace A tace A = o/, a mld codto Ths cocetato esult s easly exteded to the case whee A s adom but deedet of X The evous esult also exteds easly to X = λ X, ude mld codtos o λ s, to yeld su =,, X A X λ tace A = op [4] Leavg-out oe obsevato Let us call β the usual leave oe out estmato e the estmato we get by ot usg X, Y ou egesso oblem It solves X jψɛ j X j β = 0 [5] j Note that whe {X } = ae deedet, β s deedet of X Fo all j, j, we call j, j, = ɛ j X j β [6] Whe j, these ae the esduals fom ths leave-oe-out stuato Fo j =,, s the edcto eo fo obsevato Itutvely, t s clea that ude egulaty codtos o ρ ad ɛ s, whe X s ae d, fo j, R j j, ths meas statstcally that leave-oe-out makes sese O the othe had, t s easy to covce oeself by lookg eg at the least-squaes stuato that, s vey dffeet fom R hgh-dmeso The exaso we wll get below wll deed cofm ths fact a moe geeal settg tha least-squaes Takg the dffeece betwee Equatos [3] ad [5] we get, afte usg Taylo exasos fo j ad tucatg the exaso at fst ode, X ψɛ X β + j ψ j, X jx j β β 0 We call S = j ψ j, X jx j Ths suggests that β β S X ψɛ X β [7] Note that S s deedet of X Hece, multlyg the evous exesso by X, we get, usg the aoxmato gve Equato [4] whch amouts to assumg that S s ce eough, R, λ tace S ψr Exeece adom matx theoy as well as the fom of the matx S suggest that tace S should have a detemstc lmt aga ude codtos 3 o ρ, λ s ad ɛ s 3 to hel wth tuto, ote that the least squaes case, S = j X j X j, a samle covaace matx multled by 4 wwwasog/cg/do/0073/as Footle Autho

5 The, by symmety betwee the obsevatos, all tace S ae aoxmately the same, e, whe ad ae lage, tace S c Hece, R, λ cψr [8] Sce R,[] = β X +V γ β S, the evous equato eads [ ] ψ,[] V V γ β S β ψ,[] V X 0 Note that sce X ad β ae deedet whe X s ae deedet ad deedet of {ɛ } = ad {λ } =, much ca be sad about the dstbuto of, Howeve, at ths ot the devato t s ot clea what the value of c should be Callg S = ψ,[] V V, ad u = ψ,[] V X, Leavg-out oe edctolet us cosde what haes whe we leave the -th edcto out Because we ae assumg that X s N 0, Id ad β 0 = 0, all the edctos lay a symmetc ole, so we ck the -th to smlfy otatos Thee s othg atcula about t ad the same aalyss ca be doe wth ay othe edctos Let us call γ R the coesodg otmal egesso vecto fo the loss fucto ρ We use the otatos ad attos [ X = V X ], β = [ βs β ] We have V R Natually, γ satsfes We call = V ψɛ V γ = 0,[] = ɛ V γ, e the esduals based o edctos Note that {,[] } = s deedet of {X } = ude ou assumtos because V s deedet of X ad the X s ae d It s tutvely clea that 4 R,[], fo all, sce addg a edcto wll ot hel us much estmatg β 0 = 0 Hece the esduals should ot be much affected by the addto of oe edcto Takg the dffeece of the equatos defg β Equato [3] ad γ, we get [ ] X ψɛ X β V ψɛ 0 V γ = 0 Ths -dmesoal equato seaates to a scala ad a vecto equato, amely, X ψɛ X β = 0, V [ψr ψ,[] ] = 0 Usg a fst-ode Taylo exaso of ψr aoud ψ,[] ad otg that R,[] = V γ β S X β, we ca tasfom the fst equato above to [ X ψ,[] + ψ,[] V γ β ] S X β 0 Ths gves the ea detty X[ψ,[] + ψ β,[] V γ β S ] X ψ,[] Wokg smlaly o the equatos volvg V, we get ψ,[] V [R,[] ] 0 we see that γ β S β S u Usg ths aoxmato the evous equato fo β, we have fally β Xψ,[] X ψ,[] u S u [9] Aoxmato of ths deomato Let us wte matx fom u S u = X AX, whee A = D / P V D /, P V = D / V V DV V D / ad D s a dagoal matx wth D, = ψ,[] Note that P V s a ojecto matx of ak geeal Let us call ξ the deomato of β dvded by We have ξ = X D AX Let us call S = S ψ,[] V V Usg the Shema-Moso-Woodbuy fomula see [5], 9 ad the SI, we see that P V, = We +ψ,[] V [S] V otce that S ca be aoxmated by a matx M whch s deedet of V by usg ou leave-oe-edctoout obsevatos fo whch V M V /λ = tace M + o P by Equato [4] these two aoxmatos atually eque some egulaty codtos o ρ, etc so that M s ce eough Hece, P V, = + + λ ψ,[] tace [S ] op Theefoe, usg the aoxmatos,[] R ad tace [S ] c because tace [S ] tace [S ] usg Shema-Moso-Woodbuy, we also have P V, = + ψ,[] V [S] V + λ cψ,[] Sce P V s a ak ojecto matx, we have tace P V = = PV,, ad theefoe = + λ cψ,[] = + op Usg cocetato oetes of X codtoal o {λ } =, we have ξ = tace D λd AD λ + o P = λ ψ,[] P V, + o P = 4 ude egulaty codtos o ρ, ɛ s ad λ s Footle Autho PNAS Issue Date Volume Issue Numbe 5

6 Relacg P V, by ts aoxmate value, we get ξ = c So fally, = λ ψ,[] + cλ ψ,[], = Xψ,[] / β c /c + cλ ψ,[] c λ ψ,[] X [0] Usg aga ψ,[] ψr, we see that [ E β E c λ ψ R ], [] = assumg that we ca take exectatos all these aoxmatos Fom aoxmatos to fuctoal system Ou aoxmatos coceg the esduals ad β shed cosdeable lght o them Ou focus s ow o β Fom Equato [ 8] we got the aoxmato =, R + cλ ψr Recall that fo a covex, oe ad closed fucto ρ, whose subdffeetal we call ψ, ad t > 0, ox t ρ = Id + tψ It s a motat fact that the ox s deed a fucto ad ot a mult-valued mag, eve whe ρ s ot dffeetable eveywhee We theefoe get the aoxmato R ox cλ ρ, Recallg Equato [6] ad usg the deedece of β ad X, we have, L = ɛ + λ β Z, whee Z s N 0, ad deedet of ɛ, λ ad β We ow ague that β s asymtotcally detemstc Usg the elatosh betwee β ad β Equato [7] ad takg squaed oms, we see that β β + β S X ψr + X S X ψ R Assumg that the smallest egevalue of S / emas bouded, whch s automatcally satsfed wth hghobablty fo stogly covex fuctos ρ, we see that β β s O P /, ovded β emas bouded ad ψ ad ψ do ot gow too fast at fty Alyg the Efo-Ste equalty, we see that va β = O/ f we take squaed exectatos ou aoxmatos It follows that β s asymtotcally detemstc These agumets suggest that as ad become lage, L, = ɛ + λ ρκz + o P, whee Z N 0,, deedet of λ ad ɛ, ad ρκ s detemstc We also ote that Z ae d, sce X ae Sce cλ ψr, R, ox cλ ρ,, we see that Equato [ ] ow becomes asymtotcally κ ρκ = = E λ [ ], ox cλ ρ,, whee the exectatos ae ove the jot dstbuto of λ s, ɛ s ad Z s We ote that ou agumets do ot deed o deedece of λ s o ɛ s, though both famles of adom vaables eed to be deedet of {X } = Ths s the secod equato of System [ S] We ow ecall that usg the fact that the matx P V above was a ojecto matx, we had agued that asymtotcally = κ + + cλ op ψr = We obseve that ox cλ, = theefoe ad +cλ ψox cλ [, ] +cλ ψr ox cλ, Ths allows us to coclude that ude egulaty codtos, = E ox cλ, = κ Ths s the fst equato of ou System [S] A ote o o-dffeetable ρ s Oe of the aeals of ou systems [S] ad [S] s that they yeld exessos eve the case of o-dffeetable ρ, sce the ox s well-defed Howeve, we deved the systems assumg smoothess of ρ To go aoud ths hudle, oe ca aoxmate ρ by a famly ρ η of smooth covex fuctos such that ρ η ρ as η 0 a aoate sese Itutvely t s qute clea that ρη κ should ted to ρκ as η teds to 0 ude aoate egulaty codtos o ɛ s ad λ s We the just eed to take lmts ou systems to justfy them fo o-dffeetable ρ s ACKNOWLEDGMENTS Bea gatefully ackowledges suot fom NSF gat DMS VIGRE Bckel gatefully ackowledges suot fom NSF gat DMS El Kaou gatefully ackowledges suot fom a Alfed P Sloa Reseach Fellowsh ad NSF gat DMS CAREER Yu gatefully ackowledges suot fom NSF Gats SES CDI, DMS ad CCF Bea, D, Bckel, P, El Kaou, N, ad Yu, B 0 Otmal objectve fucto hgh-dmesoal egesso Submtted to PNAS Eato, M L 007 Multvaate statstcs Isttute of Mathematcal Statstcs Lectue Notes Moogah Sees, 53 Isttute of Mathematcal Statstcs Ret of the 983 ogal 3 El Kaou, N 00 Hgh-dmesoalty effects the Makowtz oblem ad othe quadatc ogams wth lea costats: sk udeestmato A Statst 38, El Kaou, N, Bea, D, Bckel, P, Lm, C, ad Yu, B 0 O obust egesso wth hgh-dmesoal edctos Techcal eot 5 Ho, R A ad Johso, C R 994 Tocs matx aalyss Cambdge Uvesty Pess, Cambdge Coected et of the 99 ogal 6 Hube, P J 973 Robust egesso: asymtotcs, cojectues ad Mote Calo A Statst, Hube, P J ad Rochett, E M 009 Robust statstcs Wley Sees Pobablty ad Statstcs Joh Wley & Sos Ic, Hoboke, NJ, secod edto 8 Moeau, J-J 965, Poxmté et dualté das u esace hlbete Bull Soc Math Face 93, Potoy, S 984, Asymtotc behavo of M-estmatos of egesso aametes whe / s lage I Cosstecy A Statst, s 6 wwwasog/cg/do/0073/as Footle Autho