Asymptotic statistics for multilayer perceptron with ReLu hidden units

Transcription

1 ESANN 8 proceedigs, Europea Symposium o Arificial Neural Neworks, Compuaioal Ielligece ad Machie Learig. Bruges (Belgium), 5-7 April 8, i6doc.com publ., ISBN Available from hp:// Asympoic saisics for mulilayer percepro wih ReLu hidde uis Joseph Rykiewicz Uiversie Paris I - SAMM 9 rue de olbiac, Paris - Frace Absrac. We cosider regressio models ivolvig mulilayer percepros (MLP) wih recified liear ui (ReLu) fucios for hidde uis. I is a difficul ask o sudy saisical properies of such models for several reasos: A firs difficuly is ha hese acivaio fucios are o differeiable everywhere, a secod reaso is also ha i pracice hese models may be heavily overparamerized. I geeral, he esimaio of he parameers of he MLP is doe by miimizig a cos fucio, we focus here o he sum of square errors (SSE) which is he sadard cos fucio for regressio purpose. I his framework, we ca characerize he asympoic behavior of he SSE of esimaed models which give iformaio o he possible overfiig of such models. his ask is doe usig rece mehodology iroduced o deal wih models wih a loss of ideifiabiliy which is very flexible. So, we do have o assume ha a rue model exis or ha a fiie se of parameers realize he bes regressio fucio. Iroducio Feed-forward eural eworks are well kow ad popular ools. hese eworks have gaied i populariy sice he surge of Deep Learig which provides ousadig pracical resuls. Deep eural eworks combie a cascade of muliple layers of o liear processig uis ad he ReLu fucio is ow oe of he mos popular acivaio fucios for such eworks (see Lecu e al.[4]). Eve if hese eworks work very well i pracice, very few heoreical resuls are available abou such complex models. We propose i his paper o fill a lile bi his gap, hece we focus oly o shallow eworks wih oly oe hidde layer, bu we deal wih ReLu acivaio fucios for he hidde layer. We may hope, ha our mehodology may be exeded o more complex eworks. his paper is orgaized as follows: Firsly, we give a geeral iequaliy for he differece of he sum of square errors (SSE) of he esimaed regressio model ad he SSE of he heoreical bes regressio fucio i our model. A se of geeralized derivaive fucios is a key ool i derivig such iequaliy. Uder suiable codiios, checked by MLP wih ReLu hidde uis, we provide he asympoic disribuio for he differece of SSE eve if hese models are o differeiable everywhere ad if he parameers characerizig he bes regressio fucio are o uique ad belog o a ifiie se. 49

2 ESANN 8 proceedigs, Europea Symposium o Arificial Neural Neworks, Compuaioal Ielligece ad Machie Learig. Bruges (Belgium), 5-7 April 8, i6doc.com publ., ISBN Available from hp:// he model For a observaio x Rd, a MLP fucio wih k hidde uis ca be wrie: fθ (x) = β + k ai φ bi + wi x i= wih θ = (β, a,, ak, b,, bk, w,, wd,, wkd ) Rk++k d he pa rameer vecor of he model, ad wi := (wi,, wid ). Le us deoe Θ k++k d R he possible bouded se of parameers. he rasfer fucio φ will be assumed o be a ReLu fucio: φ(z) = max(, z) for z R. Noe ha his fucio is o differeiable wih respec o z =. We observe a radom sample of idepede ad ideically disribued radom vecors: (, Y ),, (, Y ), from he disribuio P of a vecor (, Y ), wih Y a real radom variable. he regressio model ca be wrie as: Y = f () + ε, E (ε ) =, E ε = σ <. () where f is he bes regressio fucio which belogs o he se {fθ, θ Θ}: f = arg mi ky fθ ()k, θ Θ where, a geeral radom variable Z, sz kg(z)k := g(z) dp (z) is he L orm for ageeral square iegrable fucio g. Le us wrie Θ he se of parameers realizig he bes regressio fucio f : θ Θ, fθ = f. Noe ha we do o assume ha Θ is a fiie se which meas ha loss of ideifiabiliy ca occur, his is he case if he MLP has reduda hidde uis (see Fukumizu [] or Rykiewicz [5]). A aural esimaor of f is he leas square esimaor (LSE) fθ ha miimizes he SSE: fθ = arg mi θ Θ (Y fθ ( )). () = fθ is expeced o coverge o he fucio f uder suiable codiios. Now, le us iroduce geeralized derivaive fucios: dθ (x) = fθ (x) f (x), fθ 6= f. kfθ () f ()k (3) Noe ha hese fucios are always defied eve if he fucios fθ are o differeiable everywhere. We give ow he mai resuls of his paper. 49

3 ESANN 8 proceedigs, Europea Symposium o Arificial Neural Neworks, Compuaioal Ielligece ad Machie Learig. Bruges (Belgium), 5-7 April 8, i6doc.com publ., ISBN Available from hp:// Upper boud for he SSE his lemma is prove i Rykiewicz [5], i gives a very geeral upper boud for he sum of square errors. Lemma. Le ε be he error Y f ( ), for all regressio fucios fθ, θ Θ wih fθ 6= f ad dθ defied i (3), he P ε dθ ( ) = (Y f ( )) (Y fθ ( )) P (d ( )). = = = θ Usig his lemma, we ca he give he asympoic behavior of he SSE uder fairly geeral assumpios.. Approximaio of he SSE Firs, we recall ha a family of radom sequeces {Y (g), g G, =,, } is said o be uiformly op () if for every δ > ad ε > here exiss a cosa N (δ, ε) such ha P sup Y (g) < ε δ g G for all N (δ, ε). Defie he limi se of derivaives D as he se of fucios d L (P ) such ha oe ca fid a sequece (θ ) Θ saisfyig kfθ () f ()k ad kd dθ k. Wih such (θ ), defie, for all [, ], f = fθ, where < +. We hus have ha, for ay d D, here exiss a parameric pah (fθ ) α wih α a sricly posiive real umber, such ha for ay [, α], 7 kfθ () f ()k is coiuous, eds o as eds o ad kd dθ k as eds o. Usig he reparameerizaio kfu () f ()k = u, for ay d D, here exiss a parameric pah (fu ) u α such ha: Z (fu f ud) dp = o(u ). (4) (5) Now, le us iroduce some assumpios: B- Le u be defied as (4), he map u 7 P (Y fu ()) admis a secod-order fu ()) aylor expasio wih sricly posiive secod derivaive P (Y u a u =. B- he se of geeralized derivaive fucios S = {dθ, θ {Θ\Θ }} is a Dosker class (see va der Vaar [6], for defiiio of Dosker class). 493

4 ESANN 8 proceedigs, Europea Symposium o Arificial Neural Neworks, Compuaioal Ielligece ad Machie Learig. Bruges (Belgium), 5-7 April 8, i6doc.com publ., ISBN Available from hp:// he followig heorem is prove i Rykiewicz [5]. heorem. Uder (B-) ad (B-) P supfθ,θ Θ (Y f ( )) (Y f ( )) = θ = o P + op (). supd D max = ε d( ); Eve whe he se of possible regressio fucios F may be heavily overparamerized or o differeiable, his heorem proves he ighess of he SSE, if he se S is a Dosker class. Noe ha assumpio B- is rue for MLP wih ReLu hidde uis eve if he fucios fθ are o differeiable everywhere because i ivolves oly differeiabiliy i quadraic mea as i Le Cam [3]. Now, usig he same reparamerizaio echique ad followig he same ideas ha i Rykiewicz [5], we ca prove assumpio B- ad give he geeral descripio of he asympoic behavior of he SSE: Reparameerizaio. If k is he miimal umber of hidde uis o ge he bes fucio f, he he wriig of f wih a eural ework wih k hidde uis is uique, up o some permuaios: f = β + k ai φ wi x + bi. (6) i= So, for a θ Θ, if fθ = f, a vecor of iegers = (i ) i k + exiss so ha k k < < < k + k ad, up o permuaios, we have w = = w = if >, wi + = = wi+ = wi i k, P i+ bi + = = bi+ = bi i k,. j=i + aj = ai i k P Moreover, β + i= ai φ(bi ) = β if > else β = β. Pi+ P For i k, le us defie si = j= a ai ad, if i+ aj 6=, le i + j i + P aj. If i+ a =, q will be se a. Now, le us wrie us wrie qj = Pi+ j j i + i + aj P γ = β + i= ai φ(bi ) β if > else γ = β β. he, we ge he reparameerizaio θ 7 (Φ, ψ ) wih k + k + Φ = γ, (wj )j=, (bj )j=, (si )ki=, (aj )kk + +, k + k ψ = (qj )j=, (w, b ) i i i=+k +. Wih his parameerizaio, for a fixed, Φ is a ideifiable parameer ad all he o-ideifiabiliy of he model will be i ψ. Namely, fθ will be equal o: Pk Pi fθ = (γ + β ) + i= (si + ai ) j= qj φ(wj x + bj ) i + Pk + i= + aj φ(wi x + bi ). k + 494

5 ESANN 8 proceedigs, Europea Symposium o Arificial Neural Neworks, Compuaioal Ielligece ad Machie Learig. Bruges (Belgium), 5-7 April 8, i6doc.com publ., ISBN Available from hp:// So, for a fixed, f(φ,ψ ) = f if ad oly if Φ = (, w,, w,, w,, w, b,, b,, bk,, bk, {z } {z } k {z k} {z } k + k k + k,,,, ). {z } {z } k k k + We ge he he followig expasio for he umeraor of geeralized derivaive fucios: Lemma.3 For a fixed, i he eighborhood of he ideifiable parameer Φ : f(φ,ψ ) (x) f (x) = (Φ Φ ) f(φ,ψ ) (x) + o(kf(φ,ψ ) f k ), wih Pk (Φ Φ ) f(φ,ψ ) (x) = γ + i= si φ(wi x + bi ) Pk Pi+ + i= j=i + qj wj wi xai IR+ (wi x + bi ) Pk Pi+ + i= j=i + qj bj bi ai IR+ (wi x + bi ) Pk + i= + ai φ(wi x + bi ) k + where IR+ is he idicaor fucio of R+ : IR+ (z) = if z < ad IR+ (z) = if z Now, wih his reparameerizaio, he proposiio of Rykiewicz [5] shows ha he assumpio (B-) is rue for our model. Fially, we ca give he asympoic behavior of he SSE: heorem.4 Le he map Ω : L (P ) L (P ) be defied as Ω(f ) = kffk. Uder he assumpios B- ad B-, a ceered Gaussia process {W (d), d D} wih coiuous sample pahs ad a covariace kerel P (W (d )W (d )) = P (d d ) exiss so ha lim = (Y f ( )) (Y fθ ( )) = σ sup (max {W (d); }). d D = he idex se D is defied as D = D, he uio rus over ay possible vecor of iegers = (,, k + ) Nk + wih k k < < < k + k ad Pk Pk D = Ω γ + i= i φ(wi + bi ) + i= IR+ (wi + bi )(ζi + αi ) Pk + i= + µi φ(wi + bi ), k + γ,,, k, α,, αk R, µk + +,, µk R+ ; ζ,, ζk Rd, (wk ++, bk ++ ),, (wk, bk ) Θ\ (w, b ),, (wk, bk ). 495

6 ESANN 8 proceedigs, Europea Symposium o Arificial Neural Neworks, Compuaioal Ielligece ad Machie Learig. Bruges (Belgium), 5-7 April 8, i6doc.com publ., ISBN Available from hp:// his heorem shows ha he degree of over-fiig is bouded i probabiliy, bu depeds o he size of he asympoic se D. Hece, he over-fiig ha occurs i he over-realizable case is for he exreme values of he ipu weighs as i a MLP wih sigmoidale acivaio fucios (see Hagiwara ad Fukumizu []). I order o reduce he over-fiig we eed o corol he size of he limi fucios i D, his ca be doe by reducig he size of he ipus weighs (wi, bi ) i k eiher by L pealizaio (weigh decay mehod) or L pealizaio (Lasso mehod). However, oe ha he size of he weighs has o be large eough so ha Θ coais some parameers of Θ, so we eed o fid a rade-off for his pealizaio. 3 Coclusio MLP models have bee used for may years, bu have evolved dramaically hese las years. he ReLu acivaio fucio is ow oe of he mos popular eve if he saisical properies of models usig hese fucios are o well kow. his paper is a aemp o fill his gap. By usig moder heory which deals wih over-parameerized models we ca give he asympoic behavior of he SSE for MLP wih oe hidde layer usig ReLu fucios. I is clear ha he eworks used i pracice are deeper ad he layer afer he hidde uis is ofe a poolig fucio like he mea, he maximum or a orm, bu poolig fucios ca also be see as cosrais over parameers ad so over asympoic se D. Fially, our mehodology seems o be promisig o give some saisical udersadig of models ivolved i deep learig. Refereces [] Fukumizu, K., Likelihood raio of uideifiable models ad mulilayer eural eworks, A. Sais. 3 (3) [] Hagiwara, K. ad Fukumizu K. Relaio bewee weigh size ad degree of over-fiig i eural ework regressio. 8. Neural Neworks : [3] Le Cam, L., O he assumpios used o prove asympoic ormaliy of maximum likelihood esimaors, als of Mahemaical Saisics 4 (97) [4] LeCu, Ya, Begio, Y. ad Hio, G., Deep learig, Naure. 5 (7553) (5) [5] Rykiewicz, J., Asympoics for Regressio Models Uder Loss of Ideifiabiliy, Sakhya A, 78 () (6) [6] va der Vaar, A.W., Asympoic saisics, Cambridge uiversiy press (998). 496