A diluted version of the perceptron model

Transcription

1 Stochastic Processes and their Applications ) A diluted version of the perceptron model David Márquez-Carreras a Carles Rovira a Samy Tindel b a Facultat de Matemàtiques Universitat de Barcelona Gran Via Barcelona Spain b Institut Elie Cartan Université de ancy 1 BP Vandoeuvre-lès-ancy France Received 28 February 2006; received in revised form 27 February 2007; accepted 27 February 2007 Available online 6 March 2007 Abstract This note is concerned with a diluted version of the perceptron model. We establish a replica symmetric formula at high temperature which is achieved by studying the asymptotic behavior of a given spin magnetization. Our main task will be to identify the order parameter of the system. c 2007 Elsevier B.V. All rights reserved. MSC: 60G15; 82D30 Keywords: Spin glasses; Perceptron model; Magnetization; Order parameter 1. Introduction A wide number of spectacular advances have occurred in the spin glasses theory during the last past years and it could easily be argued that this topic at least as far as the Sherrington Kirkpatrick model is concerned has reached a certain level of maturity from the mathematical point of view: the cavity method has been set in a clear and effective way in [9] some monotonicity properties along a smart path have been discovered in [4] and these elements have been combined in [10] in order to obtain a completely rigorous proof of the Parisi solution [7]. However there are some canonical models of mean field spin glasses for which the basic theory is far from being complete and this paper proposes to study the high temperature behavior of one of them namely the diluted perceptron model which can be described as follows: for Corresponding author. Tel.: ; fax: addresses: davidmarquez@ub.edu D. Márquez-Carreras) carles.rovira@ub.edu C. Rovira) tindel@iecn.u-nancy.fr S. Tindel) /$ - see front matter c 2007 Elsevier B.V. All rights reserved. doi: /j.spa

2 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) consider the configuration space Σ = { 1 1} and for σ = σ 1... σ ) Σ define a Hamiltonian H M σ ) by H M σ ) = ) η k u g ik γ ik σ i. 1) k M i In this Hamiltonian M stands for a positive integer such that M = α for a given α 0 1); u is a bounded continuous function defined on R; {g ik i 1 k 1} and {γ ik i 1 k 1} are two independent families of independent random variables g ik following a standard Gaussian law and γ ik being a Bernoulli random variable with parameter γ which we denote by B γ ). Finally {η k k 1} stands for an arbitrary family of numbers with η k {0 1} even if the case of interest for us will be η k = 1 for all k M. Associated with this Hamiltonian define a random Gibbs measure G on Σ whose density with respect to the uniform measure µ is given by Z 1 M exp H M σ ) ) where the partition function Z M is defined by Z M = exp H M σ ) ). σ Σ In the sequel we will denote by f the average of a function f : Σ n R with respect to dg n i.e. f = Z n M f σ 1... σ n ) exp ) H M σ l ). l n σ 1...σ n ) Σ n The measure described above is of course a generalization of the usual perceptron model which has been introduced for neural computation purposes see [5]) and whose high temperature behavior has been described in [9 Chapter 3] or [8] for an approach based on convexity properties of the Hamiltonian. Indeed the usual perceptron model is induced by a Hamiltonian Ĥ M on Σ given by Ĥ M σ ) = ) 1 u k M 1/2 g ik σ i 2) i where we have kept the notation introduced for Eq. 1). Thus our model can be seen as a real diluted version of 2) in the sense that in our model each condition i g ikγ ik σ i 0 only involves in average a finite number of spins uniformly in. It is worth noticing at that point that this last requirement fits better to the initial neural computation motivation since in a one-layer perceptron an output is generally obtained by a threshold function applied to a certain number of spins that does not grow linearly with the size of the system. Furthermore our coefficient γ is arbitrarily large which means that the global interaction between spins is not trivial. Another motivation for the study of the system induced by 1) can be found in [2]. Indeed in this latter article a social interaction model is proposed based on a Hopfield-like or perceptronlike) diluted Hamiltonian with parameter and M where represents the number of social agents and M the diversity of these agents the number of interactions of each agent varying with the dilution parameter. However in [2] the equilibrium of the system is studied only when M is a fixed number. The result we will explain later on can thus be read as follows: as soon as the diversity M does not grow faster than a small proportion of the capacity of the social interaction system is not attained.

3 1766 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) Let us turn now to a brief description of the results contained in this paper: in fact we will try to get a replica symmetric formula for the system when M is a small proportion of which amounts to identify the limit of 1 logz M) when M = α. This will be achieved as in the diluted SK model studied through the cavity method see [3] for a study based on monotonicity methods) once the limiting law for the magnetization σ i is obtained. This will thus be our first aim and in order to obtain that result we will try to adapt the method designed in [9 Chapter 7]. However in our case the identification of the limiting law for σ i will be done through an intricate fixed point argument involving a map T : P P where P stands for the set of probability measures on [ 1 1]) which in turn involves a kind of Pλ) Pµ) measure for two independent Poisson measures Pλ) and Pµ). otice that this kind of complexity inherent to diluted inhomogeneous systems is also illustrated e.g. in the context of random assignments in [1]. For sake of readability we will give the details of almost) all the computations we will need in order to establish our replica symmetric formula but it should be mentioned at that point that our main contribution with respect to [9 Chapter 7] is that construction of the invariant measure. More specifically our paper is divided as follows: At Section 2 we will establish a decorrelation result for two arbitrary spins. amely setting U = u for αu small enough we will show that E [ σ 1 σ 2 σ 1 σ 2 ] K for a constant K > 0. At Section 3 we will study the asymptotic behavior of the magnetization of m spins where m is an arbitrary integer. Here again if αu is small enough and in the particular case of interest where all η k = 1 we will see that ] E [ σ i z i i m K m3 where z 1... z m is a family of i.i.d. random variable with law µ αγ and µ αγ is the fixed point of the map T alluded to above whose precise description will be given at the beginning of Section 3. Finally at Section 4 we obtain the replica symmetric formula for our model where all η k = 1): set )) V p = exp u g im σ i dµ αγ x 1 ) dµ αγ x p ) i p Gγ ) = α log exp γ ) γ p p! E p=0 x 1...x p ) [ ]) V p+1 V p where x means integration with respect to the product measure ν on { 1 1} p such that σi dν = x i. Let F : [0 1] R + be defined by F0) = log 2 αu0) and F γ ) = Gγ ). Then if αu is small enough we will get that 1 E [ logz M ) ] Fγ ) K for a finite constant K. All these results will be described in greater detail in the corresponding sections.

4 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) Spin correlations As in [9 Chapter 7] the first step towards a replica symmetric formula will be to establish a decorrelation result for two arbitrary spins in the system. However a much more general property holds true and we will turn now to its description: for j let T j be the transformation of Σ n that for a configuration σ 1... σ n ) in Σ n exchanges the j-th coordinates of σ 1 and σ 2. More specifically let f : Σ n R with n 2 and let us write for j ) f = f σ 1 j c σ j 1 ; σ 2 j c σ j 2 ;... ; σ n j c σ j n where for l = 1... n σ l j c = σ l 1... σ l j 1 σ l j+1... σ l ). Then define f T j by f T j σ 1... σ n ) = f ) σ 1 j c σ j 2 ; σ 2 j c σ j 1 ;... ; σ n j c σ j n. 3) For j 1 we will call U j the equivalent transformation on Σ n 1. Definition 2.1. We say that Property P γ 0 B) is satisfied if the following requirement is true: let f and f be two functions on Σ n depending on m coordinates such that f 0 f T = f and there exists Q 0 such that f Q f ; then if γ γ 0 we have E f f m Q B for any Hamiltonian of the form 1) uniformly in η. Set now U = u. With Definition 2.1 in hand one of the purposes of this section is to prove the following theorem. Theorem 2.2. Let γ 0 be a positive number and U be small enough so that ) 4U αγ0 2 e4u e αγ 0e 4U 1) 3 + 2γ 0 + αγ0 2 + γ 0 3 )e4u < 1. 4) Then there exists a number B 0 γ 0 U ) such that if γ γ 0 the property P γ 0 B 0 ) holds true for each 1. In the previous theorem notice that the value of γ 0 has been picked arbitrarily. Then we have to choose U which also contains implicitly the temperature parameter accordingly. Let us also mention that the spin decorrelation follows easily from the last result: Corollary 2.3. Assuming 4) there exists K > 0 such that for all γ < γ 0 E σ 1 σ 2 σ 1 σ 2 K. Proof. It is an easy consequence of property P γ 0 B 0 ) applied to n = 2 f f σ 1 σ 2 ) = σ1 1σ 2 1 σ 2 2). = 1 and We will prepare now the ground for the proof of Theorem 2.2 which will be based on an induction argument over. A first step in this direction will be to state the cavity formula for our model: for σ = σ 1... σ ) Σ we write ρ ρ 1... ρ 1 ) = σ 1... σ 1 ) Σ 1.

5 1768 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) Then the Hamiltonian 1) can be decomposed into H M σ ) = ) η k γ k u g ik γ ik σ i + g k σ H 1M ρ) k M i 1 with H 1M ρ) = ) ηk u g ik γ ik σ i and ηk = η k1 γ k ). 5) k M i 1 ote that in H 1M the coefficients η k = η k 1 γ k ) are not deterministic and hence H 1M is not really of the same kind as H M. However this problem can be solved by conditioning on {γ k k M}. Then given the randomness contained in the γ k the expression H 1M ρ) is a Hamiltonian of a 1)-spin system with γ ik B γ 1 ) where γ = γ 1 and so γ γ γ 0. Thus given a function f : Σ n R we easily get the following decomposition of the mean value of f with respect to G n : with f = Av f ξ Avξ )) ξ = exp η k γ k u g ik γ ik σi l + g k σ l 7) l n k M i 1 and with f defined for a given f : Σ n 1 f = ρ 1...ρ n ) Σ n 1 ρ 1...ρ n ) Σ n 1 f ρ 1... ρ n ) exp R by ) H 1M ρl ) l n exp ). H 1M ρl ) l n otice also that in expression 6) Av stands for the average with respect to the last component of the system namely if f = f ρ 1 σ 1... ρn σ n ) then Av f ρ 1... ρ n ) = 1 2 n σ j =±1 j n f ρ 1 σ 1... ρn σ n ). Let us introduce now a little more notation: in the sequel we will have to take expectations for a fixed value of ξ given at 7). Let us denote thus by E γ the expectation given γ k k M and define ] E γ [ ] = E γ [ g k g ik γ ik i 1 k D1 M 8) where D1 M is given by D M 1 = {k M : γ k = 1}. 6)

6 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) One has to be careful about the way all these conditioning are performed but it is worth observing that the set D1 M is not too large: indeed it is obvious that setting A for the size of a set A we have and thus D M 1 = k M γ k E D M 1 = M γ = αγ. Let us go on now with the first step of the induction procedure for the proof of Theorem 2.2: in P γ 0 B) we can assume without loss of generality that f and f depend on the coordinates 1... m 1. Moreover since f ξ Q f ξ we have Av f ξ Av f ξ QAv f ξ and hence Av f ξ Av f ξ Q. We now define the following two events: Ω 1 = { p m 1 k D M 1 : γ pk = 1} = { p m 1 k M : γ pk = γ k = 1} Ω 2 = { j 1 k 1 k 2 D M 1 : γ jk 1 = γ jk2 = 1} = { j 1 k 1 k 2 M : γ jk1 = γ jk2 = γ k1 = γ k2 = 1}. These two events can be considered as exceptional. Indeed it is readily checked that PΩ 1 ) α γ 2 Thus if Ω = Ω 1 Ω 2 we get m 1) PΩ 2) α 2 γ PΩ) αγ 2 m 1) + α 2 γ 4 and using this fact together with 10) we have E f f = E Av f ξ Av f ξ Av f ) ξ = E 1 Ω Av f ξ Q αγ 2 m 1) + α 2 γ 4 + E 1 Ω c + E 1 Ω c 9) 10) Av f ) ξ Av f ξ Av f ) ξ Av f ξ. 11) Consequently in order to prove Theorem 2.2 we only need to bound accurately the expectation of the right-hand side of 11) by means of the induction hypothesis. To this end we will introduce some new notation and go through a series of lemmas: set J 1 = { j 1 : γ jk = 1 for some k D M 1 }

7 1770 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) and observe that when Ω 1 does not occur J 1 {1... m 1} =. Define J 1 = cardj 1 ) and write an enumeration of J 1 as follows: J 1 = { j 1... j J1 }. Lemma 2.4. Let U j be the transformation defined at 3) and f : Σ n f T = f. When Ω 1 does not occur we have Av f ξ) j J 1 U j = Av f ξ. R such that Proof. The proof of this lemma can be done following the steps of [9 Lemma 7.2.4] and we include it here for sake of readability. Set T = j J 1 T j. Since f depends only on the coordinates {1... m 1 } and this set is disjoint from J 1 we have f T = f. Moreover f T T = f T = f. On the other hand ξ only depends on J 1 {} and using we obtain Hence ξσ 1 σ 2... σ n ) = ξσ 2 σ 1... σ n ) ξ T T = ξ. f ξ) T T = f ξ and since T 2 = Id we get Finally f ξ) T = f ξ) T. 12) Av[ f ξ) T ] = Av f ξ 13) Av[ f ξ) T ] = Av f ξ) U j. j J 1 14) The proof is now easily concluded by plugging 13) and 14) into 12). Let us now go on with the proof of Theorem 2.2: thanks to Lemma 2.4 when Ω 1 does not occur we can write [ Av f ξ = 1 Av f ξ Av f ξ) ] U js = 1 f s 15) 2 2 s J 1 1 s J 1 with f s = Av f ξ) U jl Av f ξ) U jl. l s 1 l s otice that U 2 j = Id and that f s enjoys the same kind of antisymmetric property as f since f s U js = f s.

8 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) Define R 1 = D1 M. Then recalling relation 9) we have R 1 = D M 1 = k M γ k and let us enumerate as k 1... k R1 the values k M such that γ k = 1. We also define I I 1 R 1 as follows: I 1 v = { j 1 : γ jk v = 1} for v R 1 and observe that we trivially have J 1 = v R 1 I 1 v. Moreover when Ω 2 does not occur we have I 1 v 1 I 1 v 2 = if v 1 v 2. Then on Ω c we get J 1 = CardJ 1 ) = Iv 1. 17) v R 1 Furthermore it is easily checked that for each v and conditionally on the γ k the quantity I 1 v is a binomial random variable with parameters 1 and γ which we denote by Bin 1 γ ). With all this notation in mind our next step will be to bound f s in function of f in order to get a similar condition to that of Definition 2.1: Lemma 2.5. Recall that U = u. Then on Ω c for j s Iv 1 we have where f s ˆQAv f ξ ˆQ 4QU exp 4U R 1 ). Proof. Let us decompose ξ as ξ = ξ ξ with )) ξ = exp η k γ k u g ik γ ik σi m + g k σ m 3 m n k M i 1 )) ξ = exp η k γ k u g ik γ ik σi m + g k σ m. 18) k M i 1 m 2 Thus ξ ξ exp ) ) u g ik v γ ik v σi m + g k v σ m m 2 i 1 and hence v R 1 ξ exp 2U R 1 ) Av f ξ Av f ξ ) exp 2U R 1 ). 19) 16)

9 1772 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) On the other hand since f only depends on {1... m 1 } we have f T jl = f for any l J 1 on Ω c which yields f s = Av f ξ) U jl U jl Av f ξ) l s 1 l s = Av f ξ) T jl f ξ) ) T jl l s 1 l s = Av f ξ T jl ξ )) T jl 20) l s 1 l s where we have used the fact that J 1 can be written as J 1 = { j 1... j J1 }. Moreover for any l by construction of ξ we have ξ T jl = ξ. Thus ξ T jl ξ [ T jl = ξ ξ T jl ξ ] T jl. 21) l s 1 l s l s 1 l s Set now Γ = sup σ ξ T jl ξ l s 1 l s T jl = sup ξ ξ T js. σ Then from 20) and 21) and invoking the fact that f Q f we get f s Γ Av f ξ ) QΓ Av f ξ. 22) We now bound Γ : recall that ξ is defined by 18) and thus ξ = v R 1 ξ v with )) ξ v = exp η k v u g ik v γ ik v σi m + g k v σ m. m 2 i 1 Recall now that we have assumed that j s Iv 1. Therefore we have j s I 1 v if v v according to the fact that Iv 1 I 1 v = on Ω c. Hence and ξ v T js = ξ v ξ ξ T js = ξ v ξ v T js ) v v ξ v. 23) On the other hand since e x e y x y e a for x y a we obtain ξ v ξ v T js 4U e 2U 24) and we also have the trivial bound ξ v e 2U. 25)

10 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) Thus plugging 24) and 25) into 23) we get Γ 4U e 2U R 1. Combining this bound with 19) and 22) the proof is now easily completed. We are now ready to start the induction procedure on P γ 0 B) which will use the following elementary lemma whose proof is left to the reader). Lemma 2.6. Let R be a random variable following the BinM γ ) distribution and λ be a positive number. Then [ E Re λr] αγ e λ e αγ eλ 1) 26) [ E R 2 e λr] α 2 γ 2 e 2λ e αγ eλ 1) + αγ e λ e αγ eλ 1). 27) Let us proceed now with the main step of the induction: Proposition 2.7. Assume that P 1 γ 0 B) holds for 2 and γ γ 0. Consider f and f as in Definition 2.1. Then [ f ] E f m Q αγ 2 + α 2 γ 4 + 4BΥα γ U )) 28) where Υα γ U ) = U αγ 2 e 4U e αγ e4u 1) 3 + 2γ + αγ 2 + γ 3 )e 4U ). Proof. Using 11) and 15) we have [ f ] ) E f Q αγ 2 m 1) + α 2 γ E f s Ω c. Av f ξ s J 1 However on Ω c the functions f s and Av f ξ depend on m 1 + J 1 coordinates. Since γ γ and m 1 + J 1 m1 + J 1 ) the definition of the expectation E γ the property P 1 γ 0 B) 17) and Lemma 2.5 imply [ ] [ f s [ ] ] fs E 1 Ω c = E 1 Ω c E γ Av f ξ s J 1 Av f ξ s J 1 [ ] m 1 + J 1 )B ˆQ E 1 Ω c 1 s J 1 m ] [1 Ω c J J 1 ) e 4U R 1 Recall that according to 16) we have J 1 v R 1 I 1 v 4 1 B QU E 8 m B QU E [1 Ω c J J 1 ) e 4U R 1 ].

11 1774 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) and that the quantity R 1 is a BinM γ ) random variable. Thus [ ] { [ ]} { } E 1 Ω c J 1 e λr 1 = E E 1 Ω c J 1 e λr 1 R 1 = E e λr 1 E [1 Ω c J 1 R 1 ] [ ] γ E R 1 e λr 1 [ ] { [ ]} E 1 Ω c J 1 2 e λr 1 = E E 1 Ω c J 1 2 e λr 1 R 1 { = E e λr 1 E [1 Ω c J 1 2 ]} R1 [ ] γ + γ 2 )E R 1 + R1 2 ) eλr 1. 29) The proof of this proposition is now easily concluded by applying the previous bounds together with Lemma 2.6 to the quantity ] E [1 Ω c J J 1 ) e 4U R 1. We can turn now to the main aim of this section: Proof of Theorem 2.2. The result is now an immediate consequence of 4) and Proposition 2.7 applied to B = B 0 = αγ 2 + α 2 γ 4 1 ε where ε satisfies 4U αγ 2 0 e4u e αγ 0e 4U 1) 3 + 2γ 0 + αγ γ 3 0 )e4u ) < ε < 1. Before closing this section we will give an easy consequence of Theorem 2.2: we will see that as grows to the Gibbs measure G taken on a finite number of spins looks like a product measure. To this end let us denote by the average with respect to the product measure ν on Σ 1 such that i 1 σ i dνρ) = σ i. Equivalently for a function f on Σ 1 we can write f = f σ σ 1 1 ) where σi i is the i-th coordinate of the i-th replica ρ i. Recall also that for v R 1 Iv 1 has been defined as I 1 v = {i 1 : γ ik v = 1}. We now introduce the enumeration {i1 v... i v Iv 1 } of this set. Furthermore given the randomness contained in the γ k the law of Iv 1 is a Bin 1 γ ). Proposition 2.8. Assume 4) and γ γ 0 and consider Θ = exp η kv u g i v p k v σ i v p + g kv σ. v R 1 p Iv 1

12 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) Then when Ω does not occur we have E γ Avσ Θ AvΘ Avσ Θ AvΘ 2B 0 J 1 1) J e2u 1) where the conditional expectation E γ has been defined at 8). Remark 2.9. The quantity Θ appears naturally in the decomposition of the Hamiltonian H M. Indeed on Ω2 c we have H M σ ) = ) η k u g ik γ ik σ i k M = k D M 1 = k D M 1 η k u η k u i ) g ik γ ik σ i + η k u i 1 k D1 M ) g ik γ ik σ i + η kv u i 1 v R 1 Observe also that ξ defined by 7) evaluated for n = 1 gives ξ = Θ. ) g ik γ ik σ i + g k σ i 1 g i v p k v σ i v p + g kv σ. p Iv 1 Proof of Proposition 2.8. The proof is similar to that of Proposition in [9] and we include it here for sake of completeness: On Ω c since the sets Iv 1 are disjoint the values i v p for any v and p are different and we can write Iv 1 = J 1 { } j 1... j J1. v R 1 Set f = f σ 1 j 1... σ 1 j J1 ) Avσ Θ Let us also define for 2 l J 1 f j l = f σ σ j l j l σ 1 j l+1... σ 1 j J1 ) f = f σ 1 j 1... σ 1 j J1 ) AvΘ. and f jl in a similar way. Then Avσ Θ E γ Avσ Θ AvΘ AvΘ f j = E 1 f j J1 γ f j1 f j J1 [ f jl 1 E γ f ] j l f 2 l J 1 jl 1 f jl [ f j E l 1 f j l γ f jl 1 + E f ] j γ l f jl 1 f jl. 30) f jl 1 f jl 2 l J 1 Let us concentrate now on the first term of the right-hand side of 30) since the other term can be bounded similarly: observe that for 2 l J 1 we have f j l = f j l 1 with e 2U e 2U.

13 1776 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) Furthermore it is easily seen that f j l 1 f j l enjoys the antisymmetric property assumed in Definition 2.1. Thus applying P 1 γ 0 B 0 ) we get f j E l 1 f j l γ f jl 1 B 0 J 1 + 1) e 2U 1) 1 which ends the proof. 3. Study of the magnetization For the non-diluted perceptron model in the high temperature regime the asymptotic behavior of the magnetization can be summarized easily: indeed it has been shown in [6] that σ 1 converges in L 2 to a random variable of the form tanh 2 z r) where r is a solution to a deterministic equation and z 0 1). Our goal in this section is to analyze the same problem for the diluted perceptron model. However in the current situation the limiting law is a more complicated object and in order to present our asymptotic result we will go through a series of notations and preliminary lemmas. Let P be the set of probability measures on [ 1 1]. We start by constructing a map T : P P in the following way: for any integer θ 1 let τ 1... τ θ ) be θ arbitrary integers. Then for k = 1... θ let t k be the cumulative sum of the τ k ; that is t 0 = 0 and t k = ˆk k τˆk for k 1. Let also {ḡ ik i k 1} and {ḡ k k 1} be two independent families of independent standard Gaussian random variables. Define then a random variable ξ θτ by ξ θτ = ξ θτ σ 1... σ tθ ε) ) θ τk = exp u ḡ ik σ tk 1 +i + ḡ k ε. 31) k=1 i=1 Whenever θ = 0 set also ξ θ = 1 which is equivalent to the convention 0 k=1 w k = 0 for any real sequence {w k ; k 0}. Consider now x = x 1... x θ k=1 τ k ) with x i 1 and a function θ τ k f : { 1 1} k=1 R. We denote by f x the average of f with respect to the product measure ν on { 1 1} θ k=1 τ k such that σ i dνδ) = x i where δ = σ 1... σ θ k=1 τ k ). Using this notation when θ 1 we define T θτ : P P such that for µ P T θτ µ) is the law of the random variable Avεξ θτ X Avξ θτ X where X = X 1... X θ k=1 τ k ) is a sequence of i.i.d. random variables of law µ independent of the randomness in ξ θτ and Av denotes the average over ε = ±1. otice that when θ = 0 T θτ µ) is the Dirac measure at point 0. Finally we can define the map T : P P by T µ) = κθ τ 1... τ θ )T θτ µ) 33) θ 0 τ 1...τ θ 0 32)

14 with D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) αγ αγ )θ κθ τ 1... τ θ ) = e e θγ θ! τ l γ l θ τ 1! τ θ! 34) and where the coefficients α γ are the parameters of our perceptron model. We will see that the asymptotic law µ of the magnetization σ 1 will satisfy the relation µ = T µ). Hence a first natural aim of this section is to prove that the equation µ = T µ) admits a unique solution: Theorem 3.1. Assume 2U e 2U αγ 2 < ) Then there exists a unique probability distribution µ on [ 1 1] such that µ = T µ). Remark 3.2. otice that 4) implies 35). In order to establish the fixed point argument for the proof of Theorem 3.1 we will need a metric on P and in fact it will be suitable for computational purposes to choose the Monge Kantorovich transportation-cost distance or equivalently the total variation distance) for the compact metric space [ 1 1] ): for two probabilities µ 1 and µ 2 on [ 1 1] the distance between µ 1 and µ 2 will be defined as dµ 1 µ 2 ) = inf E X 1 X 2 36) where this infimum is taken over all the pairs X 1 X 2 ) of random variables such that the law of X j is µ j j = 1 2. This definition is equivalent to say that dµ 1 µ 2 ) = inf dx 1 x 2 )dζx 1 x 2 ) with dx 1 x 2 ) = x 2 x 1 where this infimum is now taken over all probabilities ζ on [ 1 1] 2 with marginals µ 1 and µ 2 see Section 7.3 in [9] for more information about transportation-cost distances). Finally throughout this section we also use a local definition of distance between two probabilities with respect to an event Ω: d Ω µ 1 µ 2 ) = inf E X 1 X 2 )1 Ω 37) where this infimum is as in 36). Proof of Theorem 3.1. Assume that θ 1 and τ k 1 for some k = 1... θ. Then using arguments similar to those in Lemma in [9] we can prove for 1 i θ k=1 τ k that Avεξ θτ x x i Avξ θτ 2U e 2U 38) x with x = x 1... x θ k=1 τ k ). Then if y = y 1... y θ k=1 τ k ) the bound 38) implies that Avεξ θτ x Avξ θτ x Avεξ θτ y Avξ θτ y 2U e 2U θ τ k k=1 i=1 x tk +i y tk +i. 39) Remark that if θ = 0 or θ 0 but τ k = 0 for any k = 1... θ then the left-hand side of 39) is zero.

15 1778 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) Let now X Y ) be a pair of random variables such that the laws of X and Y are µ 1 and µ 2 respectively µ 1 and µ 2 are independent of the randomness in ξ θτ ). Consider independent copies X i Y i ) i θk=1 τ k of this couple of random variables. Then if X = X i ) i θk=1 τ k and Y = Y i ) i θk=1 τ k we have that Avεξ θτ X Avξ θτ X d) = T θτ µ 1 ) and Avεξ θτ Y Avξ θτ Y d) = T θτ µ 2 ). Hence applying 39) for x = X and y = Y and taking first expectation and then infimum over the choice of X Y ) we obtain dt θτ µ 1 ) T θτ µ 2 )) 2U e 2U dµ 1 µ 2 ) θ τ k. 40) Finally recall see [9 Lemma 7.3.2]) that for a given sequence {c n ; n 1} of positive numbers such that n 1 c n = 1 and two sequences {µ n ν n ; n 1} of elements of P we have d c n µ n ) c n ν n c n d µ n ν n ). 41) n 1 n 1 n 1 Applying this elementary result to c θτ = κθ τ 1... τ θ ) µ θτ = T θτ µ 1 ) and ν θτ = T θτ µ 2 ) we get dt µ 1 ) T µ 2 )) κθ τ 1... τ θ )dt θτ µ 1 ) T θτ µ 2 )) θ 0 τ 1...τ θ 0 θ 2U e 2U κθ τ 1... τ θ ) τ k dµ 1 µ 2 ) = 2U e 2U k=1 θ 0 τ 1...τ θ 0 k=1 αγ αγ )θ e θ! θ 0 = 2U e 2U αγ 2 dµ 1 µ 2 ) θγ ) dµ 1 µ 2 ) where we have used the fact that the mean of a Poisson random variable with parameter ρ is ρ. Then under assumption 35) T is a contraction and there exists a unique probability distribution such that µ = T µ). otice that the solution to the equation µ = T µ) depends on the parameters α and γ. Furthermore in the sequel we will need some continuity properties for the application α γ ) µ αγ. Thus we will set µ = µ αγ when we want to stress the dependence on the parameters α and γ and the following holds true: Lemma 3.3. If α γ ) and α γ ) satisfy 35) then dµ αγ µ α γ ) 4 [ γ γ α γ e γ γ + αγ α γ e αγ α γ ]. Proof. Since µ αγ = T αγ µ αγ ) and µ α γ = T α γ µ α γ ) using the triangular inequality and Theorem 3.1 we have dµ αγ µ α γ ) dt αγ µ αγ ) T αγ µ α γ )) + dt αγ µ α γ ) T α γ µ α γ )) 1 2 dµ αγ µ α γ ) + dt αγ µ α γ ) T α γ µ α γ )).

16 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) So dµ αγ µ α γ ) 2dT αγ µ α γ ) T α γ µ α γ )) and we only need to deal with dt αγ µ α γ ) T α γ µ α γ )). However Lemma in [9] which is a direct consequence of 41) implies that dt αγ µ α γ ) T α γ µ α γ )) 2 καγ θ τ 1... τ θ ) κ α γ θ τ 1... τ θ ) θ 0 τ 1...τ θ 0 2V 1 + V 2 ) with κ defined in 34) and V 1 = θ 0 V 2 = θ 0 e θγ τ 1...τ θ 0 e α γ τ 1...τ θ 0 τ l γ l θ θ! τ 1! τ θ! α γ ) θ θ! τ 1! τ θ! e θγ γ e αγ αγ ) θ e α γ α γ ) θ l θ τ l e θγ γ τ l l θ. ow following the arguments of 7.53) in [9] we get V 1 = 1 e αγ αγ ) θ e α γ α γ ) θ αγ α γ e αγ α γ θ! θ 0 V 2 γ γ e γ γ θe α γ α γ ) θ θ! θ 0 = α γ γ γ e γ γ which ends the proof of this lemma. From now on we will specialize our Hamiltonian to the case of interest for us: Hypothesis 3.4. The parameters η k k = 1... M in the Hamiltonian 1) are all equal to one. This assumption being made we can now turn to the main result of the section: Theorem 3.5. Let γ 0 be a positive number such that 4U αγ 2 0 e4u e αγ 0e 4U 1) 3 + 2γ 0 + αγ γ 3 0 )e4u ) < 1 42) and assume that there exists a positive number C 0 satisfying C 0 α γ 6 0 U e 2U 1. Then for any γ γ 0 given any integer m we can find i.i.d. random variables z 1... z m with law µ αγ such that [ ] E σ i z i K m3 44) i m for a constant K > 0 independent of m. 43)

17 1780 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) Remark 3.6. The two conditions in the above theorem are met when the following hypothesis is satisfied: there exists L > 0 such that { )} L U α γ0 6 exp 8U + αγ 0 e 4U 1 < 1. As in the case of Theorem 2.2 the proof of Theorem 3.5 will require the introduction of some notation and preliminary lemmas. Let us first recast relation 44) in a suitable way for an induction procedure: consider the metric space [ 1 1] m equipped with the distance given by dx i ) i m y i ) i m ) = x i y i. i m We also denote by d the transportation-cost distance on the space of probability measures on [ 1 1] m defined as in 36). Define now ) D M m γ 0 ) = sup d L σ 1... σ m ) µ m αγ 45) γ γ 0 where LX) stands for the law of the random variable X. Then the statement of Theorem 3.5 is equivalent to say that under Hypothesis 43) we have D M m γ 0 ) K m3 for any fixed integer m 1. It will also be useful to introduce a cavity formula for m spins which we proceed to do now: generalizing some aspects of the previous section we consider for p {1... m} the random sets and D M p = {k M : γ p+1k = 1} m FM m = Dp M. p=1 We also define the following two rare events: satisfying Ω 1 = { k M p 1 p 2 m : γ p1 +1k = γ p2 +1k = 1} Ω 2 = { i m k 1 k 2 F m M : γ ik 1 = γ ik2 = 1} P Ω 1 ) αγ 2 m 2 and P Ω 2 ) α2 γ 4 m 2. 46) Then the following properties hold true: first for a fixed k if Ω 1 c is realized we have Card{p m γ p+1k = 1} 1. Moreover still on Ω c 1 for p 1 p 2 D M p 1 D M p 2 = ;

18 and hence D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) m R m FM m = Dp M = γ p+1k. k M p m p=1 Actually notice that we always have R m m Dp M. p=1 Let us introduce now an enumeration of F m M : F m M = {k 1... k Rm } and for any v R m set I m v = { j m : γ jk v = 1}. Then on Ω 2 c we get I m v 1 I m v 2 = if v 1 v 2 47) and we can also write J m = v R m I m v = v R m { j m : γ jkv = 1}. Let us separate now the m last spins in the Hamiltonian H M : if Ω 1 c ρ = σ 1... σ m ) we have the following decomposition: H M σ ) = H mm ρ) + log ξ p=1 k Dp M i m is realized for with H mm ρ) = ) u g ik γ ik σ i k FM m )c i m ) m ξ = exp u g ik γ ik σ i + g p+1k σ p+1. 48) Observe that in the last formula H mm ρ) is not exactly the Hamiltonian of a m)-spin system changing γ into γ because the set FM n is not deterministic. But this problem will be solved again by conditioning upon the random variables {γ p+1k p = 1... m k M}. For the moment let us just mention that the m cavity formula will be the following: given f on Σ we have 1 Ω 1 c f = 1 Ω 1 c Av f ξ Avξ where is the average with respect to H mm and Av is the average with respect to last m spins. Moreover in the last formula we have kept the notation ξ from Section 2 which hopefully 49)

19 1782 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) will not lead to any confusion. Finally we denote by L 0 the law of a random variable conditioned by {γ p+1k p = 1... m k M} and by E γm the associated conditional expectation. We can start now stating and proving the lemmas and propositions that will lead to the proof of Theorem 3.5. Recall that given x = x 1... x m ) x i 1 and a function f on Σ m f x means the average of f with respect to the product measure ν on Σ m such that σi dνρ) = x i for 0 i m. Recall also that γ = γ m. Then as a direct consequence of the definition of the operator T θτ we have the following result: Lemma 3.7. Let X = X 1... X m ) be an independent sequence of random variables where the law of each X l is µ αγ. Set w p = Avσ p+1ξ X Avξ X p = 1... m. Then on Ω c = Ω 1 Ω 2 ) c we have L 0 w 1... w m ) = T D M 1 I m k k DM 1 )µ αγ ) T D M m I m k k DM m )µ αγ ). We will now try to relate the random variables w p with the magnetization of the m last spins. A first step in that direction is the following lemma where we use the random value of the parameter α associated with the Hamiltonian of a m)-spin system. Lemma 3.8. On Ω c set Γ m = d L 0 w 1... w m ) L 0 w 1... w m )) where for p = 1... m w p = T D M p I m k k DM p )µ α γ ) Then on Ω c we have Γ m 2U e 2U γ 0 R m mα Proof. Using 40) we obtain m Γ m 2U e 2U p=1 k Dp M with α = M R m m. { } R m mα Rm exp γ 0 I m k dµ αγ µ α γ ). k=1 I m k. The proof of this lemma is then easily finished thanks to Lemma 3.3 and taking the following equality into account: γ α α = γ R m mα. otice that we have introduced the random variables w p for the following reason: given the randomness contained in the {γ p+1k p = 1... m k M} w p can be interpreted as w p = Avσ p+1ξ X p = 1... m Avξ X where X = X 1... X m ) is an independent sequence of random variables with law µ α γ.

20 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) Lemma 3.9. Consider Z = σ 1... σ m ) and denote u p = Avσ p+1ξ Z Avξ Z p = 1... m. Then on Ω c d L 0 u 1... u m ) L 0 w 1... w m )) 4D m F m M )c J m γ 0 ) U e 2U where the quantity D has been defined at relation 45). Proof. As in 38) we can obtain for any i m Avσ p+1 ξ x x i Avξ 2U e 2U. 50) x But in fact these derivatives are vanishing unless i Ip m Iv m v; k v D M p for some p = 1... m. Indeed on Ω c from 47) we have I m p 1 I m p 2 = if p 1 p 2. Then for a given p {1... m} we can decompose ξ into ξ = ξ p ξ p with ξ p = exp ) u g ik γ ik σ i + g p+1k σ p+1 k Dp M i m ) = ξ p {σ i i Ip m } σ p+1 ) ξ p = ξ p {σ i i J m \ Ip m } σ p+1 p m p p. Then Avσ p+1 ξ x Avξ x p=1 = Avσ p+1ξ p x Av ξ p x Avξ p x Av ξ p x and clearly the derivative x i ow invoking inequality 50) we get m Avσ p+1 ξ X u p m Avξ X = Avσ p+1ξ p x Avξ p x is zero when i does not belong to Ip m for any p {1... m}. p=1 i Ip m X i σ i 2U e 2U. Then the definition of E γm and 45) easily yield m X i σ i 2D m FM m )c J m γ 0 ) E γm p=1 i Ip m which ends the proof.

21 1784 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) Set now for 1 p m ū p = Avσ p+1ξ Avξ. 51) Then ū p is closer to the real magnetization in the sense that ū p = σ p+1 on Ω c and the following lemma claims that the distance between ū p and u p vanishes as. Lemma For 1 p m let ū p be defined by 51). Then on Ω c we have d L 0 ū 1... ū m ) L 0 u 1... u m )) 2B 0 J m 2 1 m + 1 e2u 1) where the constant B 0 has been defined in the previous section. Proof. The computations can be leaded here almost like in the proof of Proposition 2.8 and the details are left to the reader. We will now identify the law of the ū p in terms of laws of the type T µ αγ ): Lemma Recall that d Ω c has been defined by relation 37). Then for m 1 set δ m = d Ω c Lū 1... ū m ) a b 1 v 1 )... b m v m )) b) v) ) T b1 v 1 µ αγ ) T bm v m µ αγ ) where we have used the following conventions: for j m v j is a multi-index of the form v j = v j 1... v j b j ); the first summation b) is over b j 0 for j = 1... m; the second one v) is over v j 1... v j b j 0 for j = 1... m; and ab 1 v 1 )... b m v m )) is defined by a b 1 v 1 )... b m v m )) = P D M j = b j I m k k DM j ) = v j j m). Then under the conditions of Lemma 3.10 we have with δ m c 1 m) c 1 m) = 4 U e 2U E D m F m M )c J m γ 0 ) ) + 2B 0 E J m 2 1 m + 1 e2u 1) + 2U e 2U γ 0 E R m mα J m exp { γ0 }) R m mα. Proof. This result is easily obtained by combining Lemmas and taking expectations. With Lemma 3.11 in hand we can see that the remaining task left to us is mainly to compare the coefficients ab 1 v 1 )... b m v m )) with the coefficients κ αγ b j v j ). This is done in the following lemma.

22 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) Lemma With the conventions of Lemma 3.11 we have m a b 1 v 1 )... b m v m )) κ αγ b j v j ) ml 0γ ). 52) b) v) j=1 Proof. In fact it is easily seen that we only need to prove that ab v) κ αγ b v) L 0γ ) tv 0 with v = v 1... v b ). However notice that ) M γ ) b ab v) = 1 γ ) M b b b and recall that Then with l=1 m v l κ αγ b v) = e αγ αγ ) b e bγ γ ) l b b! v 1! v b!. bv 0 ab v) καγ b v) A + B A = e αγ αγ ) b Ā bv b! bv 0 Ā bv = e bγ γ v ) l l b b m v 1! v b! v l=1 l B = B b ) m γ b v bv 0 l=1 l B b = e αγ αγ ) b b! v l ) γ ) vl 1 γ ) vl 1 γ ) m vl ) M γ ) b 1 γ ) M b. b ) γ ) vl 1 γ ) m vl ) m vl ow following the estimates for the approximation of a Poisson distribution by a Binomial given in [9 Lemma 7.4.6] we can bound Ā bv and B b by a quantity of the form c. The proof is then easily finished. Let us relate now the law of ū 1... ū m ) with µ m αγ. Lemma We have d Ω c Lū 1... ū m ) µ m αγ ) c 2 m) with c 2 m) = 4 U e 2U E D m F m M )c J m γ 0 ) )

23 1786 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) b) v) E J m B 0 m + 1 e2u 1) + 2m2 L 0 γ 0 ) + 2U e 2U γ 0 E R m mα J m exp { γ0 R m mα }). Proof. otice that invoking relation 33) and Theorem 3.1 we get ) m κ αγ b j v j ) T b1 v 1 µ αγ ) T bm v m µ αγ ) = µ m αγ. j=1 Then the results follows easily from Lemmas 3.11 and 3.12 Lemma in [9] and the triangular inequality. We are now ready to end the proof of the main result concerning the magnetization of the system. Proof of Theorem 3.5. First of all notice that by symmetry we have L σ 1... σ m ) = L σ m+1... σ ). Furthermore thanks to 46) and 49) and Lemma 3.3 we can write ) D M m γ 0 ) = sup d L σ 1... σ m ) µ m αγ γ γ 0 sup γ γ 0 d Ω c L Avσ m+1 ξ Avξ... Avσ ξ Avξ ) ) µ m αγ + 2m3 αγ αγ 0 2) Avσ m+1 ξ sup d Ω γ γ c L... Avσ ) ) ξ µ m 0 Avξ Avξ αγ + 2m3 αγ αγ 0 2) + 4mαγ { 0 mγ0 } { mαγ0 }) γ 0 exp + exp. Then Lemma 3.13 implies D M m γ 0 ) 4 U e 2U E D m FM m )c J m γ 0 ) ) + 2B 0 E J m 2 1 m + 1 e2u 1) + 2U e 2U γ 0 E R m mα J m exp + 2m2 L 0 γ 0 ) It is readily checked as we did in 29) that E J m ) m αmγ m3 αγ0 4 expγ 0). E J m 2 ) m γ 0 + γ 2 0 )αmγ 0 + αmγ 0 ) 2 ) { γ0 }) R m mα

24 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) E J m 3 ) m γ 0 + 3γ γ 3 0 )αmγ 0 + 3αmγ 0 ) 2 + αmγ 0 ) 3 ). Thus using the fact that R m Y where Y BmM γ ) together with the trivial bound R m M there exists a constant K 0 1 such that D M m γ 0 ) 4 U e 2U E D m FM m )c J m γ 0 ) ) [ ] K 0 m 3 αγ0 4 exp 2 3 γ 0) + L 0 γ 0 ) +. 53) ow we are able to prove by induction over that [ ] 2K 0 m 3 αγ0 4 exp 3 2 γ 0) + L 0 γ 0 ) D M m γ 0 ) for all m 2. Indeed in order to check the induction step from 1 to notice that FM m )c M and that E J m 3) 25 m m3 αγ0 6 ). So using also that P J m ) E J m 2) 16m2 αγ0 4 2 and by our induction hypothesis and 53) we have [ ] K 0 m 3 αγ0 4 exp 3 2 γ 0) + L 0 γ 0 ) D M m γ 0 ) + 4 U e 2U 2K 0E J m 3) [ ] M m γ0 4 exp 2 3 γ 0) + L 0 γ 0 ) m [ ] K 0 m 3 αγ0 4 exp 2 3 γ 0) + L 0 γ 0 ) [ ] 2α γ0 4 exp 2 3 γ 0) + L 0 γ 0 ) + 4 U e 2U 50K 0m 3 αγ m3 αγ m3 αγ Finally since M < m the proof easily follows from hypothesis 43). 4. Replica symmetric formula ow that the limiting law of the magnetization has been computed we can try to evaluate the asymptotic behavior of the free energy of our system namely p γ ) = 1 [log E exp H M σ ) ) )]. σ Σ

25 1788 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) To this end set [ ]) Gγ ) = α log exp γ ) γ p V p! E p+1 p=0 V p where )) V p := exp u g im σ i dµ αγ x 1 ) dµ αγ x p ) i p x 1...x p ) and x means integration with respect to the product measure ν on { 1 1} p such that σ i dν = x i. Then the main result of this part states that: Theorem 4.1. Set F such that F γ ) = Gγ ) and F0) = log 2 αu0). Then if γ γ 0 and 42) and 43) hold true we have p γ ) Fγ ) K where K does not depend on γ and. Since p 0) = log 2 αu0) the proof of the theorem is a consequence of the following proposition. Proposition 4.2. If γ γ 0 and 42) and 43) hold we have p γ ) Gγ ) K where p γ ) is the right derivative of p γ ). Proof. We divide the proof into two steps. Step 1. We will check that p γ ) G1 γ ) K. 54) where G 1 γ ) is defined as [ ) )) ] αe log exp u g im γ im σ i + g M σ u g im γ im σ i. i Following the method used in Lemma in [9] we introduce the Hamiltonians HM 1 σ ) = ) u g ik γ ik + δ ik ) σ i k M i i HM 2 σ ) = ) u g ik min1 γ ik + δ ik )) σ i k M i where {δ ik } 1 i 1 k M is a family of i.i.d. random variables with Pδ ik = 1) = δ Pδ ik = 0) = 1 δ. We also assume that this sequence is independent of all the random sequences

26 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) previously introduced. Observe that the random variables min1 γ ik + δ ik )) are i.i.d. with Bernoulli law of parameter γ where γ γ + δ γ δ. Set now for j = 1 2 p j δ) = 1 ) [log )] E exp H j M σ ). σ Σ Obviously p 2 δ) = p γ ) and our first task will be to show that p 1 δ) p2 δ) is of order δ 2 : notice that p 1 δ) p2 δ) = 1 ] [log exp H E M 1 σ ) + H M 2 σ )) 2 where 2 denotes the average for the Gibbs measure defined by the Hamiltonian HM 2. Consider now YM 1 = ik γ ikδ ik. Since γ ik + δ ik = min1 γ ik + δ ik ) + γ ik δ ik on the set {YM 1 = 0} we have H M 1 = H M 2. So we can write p 1 δ) p2 δ) = 1 ] [1 E {Y 1 M =1} log exp H M 1 σ ) + H M 2 σ )) ] [1 E {Y 1 M 2} log exp H M 1 σ ) + H M 2 σ )) 2. Using that and PY M 2) = 1 1 γ δ ) M 2 M 1 γ δ ) M 1 γ δ 2 2 α2 δ 2 γ 2 PYM 1 = 1) = M 1 γ δ ) M 1 γ δ 2 αγ δ 2 it is easily checked that p 1 lim δ) p2 δ) K δ 0 + δ 55) which means that we can evaluate the difference p 1 δ) p γ ) instead of p 2 δ) p γ ). However following the same arguments as above we can write p 1 δ) p γ ) = 1 ] [log exp H E M 1 σ ) + H Mσ )). We consider now Y M = ik δ ik. otice that on the set {Y M = 0} H M = HM 1. So we can write p 1 δ) p γ ) = 1 ] [1 E {YM =1} log exp HM 1 σ ) + H Mσ )) + 1 ] [1 E {YM 2} log exp HM 1 σ ) + H Mσ )) V 1 δ) + V 2 δ). Let us bound now V 1 δ) and V 2 δ): since PY M 2) = 1 1 δ ) M M 1 δ ) M 1 δ α M 1)δ2

27 1790 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) we have V 2 δ) 2α 2 M 1)U δ 2. On the other hand using a symmetry argument we get V 1 δ) = M 1 δ ) M 1 δ [ 2 ) )) ] E log exp u g im γ im σ i + g M σ u g im γ im σ i. Hence we obtain that i p 1 lim δ) p γ ) V 1 δ) + V 2 δ) = lim δ 0 + δ δ 0 + δ [ ) )) ] = αe log exp u g im γ im σ i + g M σ u g im γ im σ i. 56) Finally since p γ ) = i i lim p γ ) p γ ) p 2 γ γ + γ = lim δ) p γ ) γ δ 0 + δ 1 γ ) putting together 55) and 56) we obtain 54). Step 2. Let us check now that i Gγ ) G 1 γ ) K. 57) To this end set ) )) Ψ := exp u g im γ im σ i + g M σ u g im γ im σ i i and let us try to evaluate first E[Ψ]: notice that )) exp u g im γ im σ i + g M σ exp H M 1 σ ) ) σ Σ i Ψ = exp H M σ ) ) σ Σ )) exp u g im γ im σ i + g M σ = i )) exp u g im γ im σ i i where M 1 denotes the usual average using the Hamiltonian H M 1. Set B p := { 1 i=1 γ im = p γ M = 0} and B := {γ M = 1} and let us denote by E M the conditional M 1 M 1 i

28 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) expectation given {γ im 1 i }. Then [ ] 1 E [Ψ] = E 1 Bp E M [Ψ] + E [1 B E M [Ψ]] = 1 p=0 p=0 1 where ) V p := u g im σ i. i p p ) γ ) p 1 γ ) p+1 E [ expvp+1 ) M 1 expv p ) M 1 ] + γ e2u 58) Set X p = σ 1... σ p ). Then using the triangular inequality and following the same arguments as in Proposition 2.8 we get for a strictly positive constant K [ ] [ ] E expvp+1 ) M 1 expvp+1 ) Xp+1 E expv p ) M 1 expv p ) Xp [ ] expvp+1 = E ) M 1 expv p ) Xp expv p+1 ) Xp+1 expv p ) M 1 expv p ) M 1 expv p ) Xp e 3U E [ expv p+1 ) M 1 expv p+1 ) Xp+1 + expv p ) M 1 expv p ) Xp ] e 3U p2 K. Consider now some i.i.d. random variables z 1... z p of law µ αγ such that 44) holds. Set Y p = z 1... z p ). Then following the same arguments as above we get for a strictly positive constant K [ ] [ ] expvp+1 E ) Xp+1 expvp+1 ) Yp+1 E expv p ) Xp expv p ) Yp e 3U E [ expv p+1 ) Xp+1 expv p+1 ) Yp+1 + expv p ) Xp expv p ) Yp ] e 3U p3 K where in the last inequality we have used 44) and the fact that x i expv p ) x e U. otice that if W is a random variable with law Bin 1 γ ) then EW 3 ) K where K does not depend on. So putting together 58) 60) we get 1 ) [ ] 1 γ ) p E [Ψ] = 1 γ ) p expvp+1 ) Yp+1 E + K p expv p ) Yp. p=0 59) 60)

29 1792 D. Márquez-Carreras et al. / Stochastic Processes and their Applications ) Using now arguments similar to those used in the proof of Lemma 3.11 we get [ ] E [Ψ] = exp γ ) γ p expvp+1 p! E ) Yp+1 + K expv p ) Yp. 61) p=0 Finally once 61) has been obtained 57) can be established following the method used in Proposition in [9] the remaining details being left to the reader. Acknowledgement The first and second authors were partially supported by MEC grant MTM References [1] D. Aldous The ζ2) limit in the random assignment problem Random Structures Algorithms 18 4) 2001) [2] R. Cont M. Lowe Social distance heterogeneity and social interactions Preprint. [3] S. Franz F. Toninelli The Kac limit for diluted spin glasses Internat. J. Modern Phys. B ) 2004) [4] F. Guerra F. Toninelli The thermodynamic limit in mean field spin glass models Comm. Math. Phys ) 2002) [5] J. Hertz A. Krogh R. Palmer Introduction to the Theory of eural Computation Addison-Wesley Publishing Company [6] D. Márquez-Carreras C. Rovira S. Tindel Asymptotic behavior of the magnetization for the perceptron model Ann. Inst. H. Poincaré Probab. Statist. 42 3) 2006) [7] M. Mezard G. Parisi M.A. Virasoro Spin Glass Theory and Beyond World Scientific [8] M. Shcherbina B. Tirozzi Rigorous solution of the Gardner problem Comm. Math. Phys ) 2003) [9] M. Talagrand Spin Glasses: A Challenge for Mathematicians Springer [10] M. Talagrand The Parisi formula Ann. of Math ) 2006)