Akaike Information Criterion for ANOVA Model with a Simple Order Restriction

Transcription

1 Akaike Information Criterion for ANOVA Mode with a Simpe Order Restriction Yu Inatsu * Department of Mathematics, Graduate Schoo of Science, Hiroshima University ABSTRACT In this paper, we consider Akaike information criterion (AIC) for ANOVA mode with a simpe ordering (SO) θ θ 2 θ where θ,..., θ are popuation means. Under an ordinary ANOVA mode without any order restriction, it is we known that an ordina AIC, whose penaty term is 2 the number of parameters, is an asymptoticay unbiased estimator of a risk function based on the expected K-L divergence. However, in genera, under ANOVA mode with the SO, the ordina AIC has an asymptotic bias which depends on unknown popuation means. In order to sove this probem, we cacuate the asymptotic bias, and we derive its unbiased estimator. By using this estimator we provide an asymptoticay unbiased AIC for ANOVA mode with the SO, caed AIC SO. A penaty term of the AIC SO is simpy defined as a function of maximum ikeihood estimators of popuation means. Key Words: Order restriction, Simpe ordering, AIC, ANOVA.. Introduction In rea data anaysis, anaysts can consider many statistica modes. Nevertheess, in many cases, we assume that considered modes satisfy some reguarity conditions. For exampe, in the case of deriving a maximum ikeihood estimator (MLE), we often assume that the MLE is a soution of a ikeihood equation. If this assumption hods, in genera, the MLE has good properties such as consistency and asymptotic normaity. Furthermore, if additiona mid conditions hod, statistics based on the MLE have aso good properties, e.g., Akaike information criterion (AIC) becomes an asymptoticay unbiased estimator of a risk function based on the expected K-L divergence, and a penaty term of AIC can be simpy expressed as 2 the number of parameters. In addition, it can be shown that the nu distribution of a ikeihood ratio statistic converges to chi-squared distribution. Thus, when certain reguarity conditions hod, we can get good modes (or * Corresponding author E-mai address: d44576@hiroshima-u.ac.jp

2 statistics) from the viewpoint of usefuness and estimation accuracy. On the other hand, when some parameters of the considered mode are restricted, reguarity conditions are not satisfied. In particuar, when the parameters θ,..., θ k are restricted as θ θ 2 θ k, this restriction is caed a simpe ordering (SO), and the SO is very important in appied statistics. The main advantage of considering order restrictions is that some information can be refected in the mode. As a resut, we can expect that estimation accuracy can be improved. For exampe, et X,..., X k be independent random variabes, and et X i N(µ i, σ 2 /n i ) where n i > 0. Then, when the assumption of the SO is true, the MLE of µ i for the mode with the SO is more efficient compared with the MLE for the mode without any restriction. Specificay, the MLE ˆµ i of µ i for the non-restricted mode is given by ˆµ i = X i. On the other hand, under the assumption of the SO, from Robertson et a. (988) the MLE ˆµ i,so can be obtained as v ˆµ i,so = min max j=u n jx j v v;i v u;u i j=u n, (i =,..., k). j For these MLEs, Brunk (965), Lee (98) and Key (989) showed that k (a) i= n ie(ˆµ i µ i ) 2 > k i= n ie(ˆµ i,so µ i ) 2, (b) E(ˆµ i µ i ) 2 > E(ˆµ i,so µ i ) 2, (i =,..., k), (c) P( ˆµ i,so µ i t) > P( ˆµ i µ i t), (t > 0, i =,..., k), respectivey. Furthermore, from Hwang and Peddada (994), the resut of (c) was extended to the case of eiptica distributions. Thus, considering order restrictions yieds good estimators from the viewpoint of estimation accuracy. However, modes with order restrictions are not easy to use. Anraku (999) considered AIC for k-custers ANOVA mode with the SO, and showed that a genera AIC, whose penaty term is 2 the number of parameters, is not an asymptoticay unbiased estimator of a risk function. Furthermore, its asymptotic bias depends on unknown parameters. Moreover, Yokoyama (995) considered a parae profie mode with a random-effects covariance structure proposed by Yokoyama and Fujikoshi (993). Variance parameters of the random-effects covariance structure are restricted as the SO, and Yokoyama (995) investigated the ikeihood ratio test for testing the adequacy of this structure. In this test, they showed that the nu distribution of the ikeihood ratio test statistic does not necessariy converge to chi-squared distribution. In addition, they aso showed that the imiting distribution of the test statistic depends on unknown variance parameters. As can be seen from these two exampes, derived resuts from the mode with order restrictions are not easy to use even if the assumed restriction is very simpe such as the SO. Based on these, in this paper we focus on AIC for ANOVA mode with the SO. Deriving an unbiased estimator of the asymptotic bias which depends on unknown 2

3 parameters, we propose AIC for ANOVA mode with the SO, caed AIC SO. Finay, we woud ike to reca that AIC is defined as AIC = 2(ˆx) + 2p, (.) where ˆx is the MLE of a parameter x, ( ) is a og-ikeihood function and p is the number of independent parameters. Hereafter, in order to avoid confusion, if ˆx is derived based on the mode without any order restriction, we refer to the AIC given by (.) as an ordina AIC. Simiary, if ˆx is derived based on the mode with a order restriction, we refer to the AIC given by (.) as a pseudo AIC (paic). The remainder of the present paper is organized as foows: In Section 2, we derive MLEs of parameters and a risk function for ANOVA mode with the SO. In Section 3, we define severa notations, and we provide one important emma for cacuating the asymptotic bias. In Section 4, we provide AIC for ANOVA mode with the SO, caed AIC SO. In Section 5, we introduce different AIC SO s for severa specia cases. In Section 6, we confirm that performance of the AIC SO through numerica experiments. In Section 7, we concude our discussion. Technica detais are provided in Appendix. 2. ANOVA mode with a simpe order restriction Let X ij be a observation variabe on the jth individua in the ith custer, where i =,..., k and j =,..., N i. Here, et k 2 and N = N +, N k, and et N k 6 > 0. Moreover, assume that X,..., X knk are mutuay independent random variabes. In this setting, we consider the mode X ij N(θ i, σ 2 ), (2.) where θ,..., θ k and σ 2 > 0 are unknown parameters. Furthermore, we assume that the parameters θ,..., θ k are restricted as θ θ 2 θ k. (2.2) Thus, the restriction (2.2) is the SO. Let Θ be a set defined as Θ = {(θ,..., θ k ) R k θ θ 2 θ k }. Then, the mode (2.) with the restriction (2.2) is equa to ANOVA mode whose mean parameters are restricted on Θ. Here, we put θ = (θ,..., θ k ). In addition, et θ = (θ,,..., θ k, ) and σ 2 denote the true parameters of θ and σ 2, respectivey. Finay, for the true parameters θ and σ, 2 we assume that θ Θ and σ 2 > 0. 3

4 2.. Maximum ikeihood estimator In this subsection, we derive MLEs of unknown parameters for the mode (2.) with the SO. Let N = (N,..., N k ). Suppose that X is an N-dimensiona vector which has a X ij, (i =,..., k, j =,..., N i ). In other words, X can be written as X = (X,..., X ij,..., X knk ). Furthermore, for any i with i k, define X i = N i N i j= X ij, σ 2 = N k N i (X ij X i ) 2. (2.3) i= j= Hence, Xi and σ 2 are the sampe mean and variance, respectivey. We put X = ( X,..., X k ). Note that under the ordina ANOVA mode without any order restriction, the MLEs of θ and σ 2 are X and σ 2, respectivey. Here, since X ij s are mutuay independent, from normaity of X ij, a og-ikeihood function (θ, σ 2 ; X) can be expressed as (θ, σ 2 ; X) = N 2 og(2πσ2 ) 2σ 2 = N 2 og(2πσ2 ) 2σ 2 k N i (X ij θ i ) 2 i= j= k N i (X ij X i ) 2 2σ 2 i= j= k N i ( X i θ i ) 2. (2.4) i= Here, for any a = (a,..., a p ) R p and b = (b,..., b p ) R p >0, we define a b = p b i a 2 i. (2.5) Note that (2.5) is a compete norm. Then, for any σ 2 > 0, the maximization probem of i= (θ, σ 2 ; X) on Θ is equa to the minimization probem of H(θ) = k N i ( X i θ i ) 2 = X θ 2 N, (2.6) i= on Θ. Needess to say, this minimization probem is equa to the minimization of H (θ) = H(θ) = X θ N on Θ. Reca that the norm N is the compete norm, and the set Θ is the non-empty cosed convex set. Therefore, for any X R k, there exists a unique point ˆθ in Θ such that ˆθ minimizes H (θ) on Θ, (see, e.g., Rudin, 986). This impies that existence and uniqueness for the MLE ˆθ = (ˆθ,..., ˆθ k ) of θ hod. Moreover, from Robertson et a. (988), for any i with i k, ˆθ i is given by ˆθ i = min v;i v max u;u i 4 v j=u N X j j v j=u N. (2.7) j

5 On the other hand, it is easiy checked that the MLE ˆσ 2 of σ 2 can be obtained by differentiating the function ( ˆθ, σ 2 ; X) with respect to (w.r.t.) σ 2 as ˆσ 2 = N k N i (X ij X i ) 2 + N i= j= k N i ( X i ˆθ i ) 2, i= because ( ˆθ, σ 2 ; X) is a concave function w.r.t. σ Risk function and bias Let X be a random variabe, and et X i.i.d. X. Then, a risk function based on the expected Kuback-Leiber divergence can be defined as = E EE 2( ˆθ, ˆσ 2 ; X ) N og(2πˆσ 2 ) + Nσ2 ˆσ 2 + k i= N i(θ i, ˆθ i ) 2 ˆσ 2 R (say). (2.8) Note that in the case of the ordina ANOVA mode, a risk function R is given by R = EE 2( X, σ 2 ; X ). Since the maximum og-ikeihood ( ˆθ, ˆσ 2 ; X) can be written as ( ˆθ, ˆσ 2 ; X) = N 2 og(2πˆσ2 ) N 2, (2.9) if we estimate the risk function R by 2( ˆθ, ˆσ 2 ; X), then the bias B, which is the difference between the expected vaue of 2( ˆθ, ˆσ 2 ; X) and R, can be expressed as B = ER { 2( ˆθ, ˆσ 2 ; X)} = E k Nσ 2 i= ˆσ 2 + E N i(θ i, ˆθ i ) 2 ˆσ 2 Next, we evauate the bias B. Let S and T be random variabes defined by N. (2.0) S = σ 2 k N i i= j=(x ij X i ) 2, T = σ 2 k N i ( X i ˆθ i ) 2. i= Note that S is distributed as χ 2 N k where χ2 N k is the chi-squared distribution with N k degrees of freedom. Furthermore, since X,..., X knk are independenty distributed as norma distributions, we obtain S X where the notation means that and are mutuay independent. In addition, from (2.7), ˆθ is a function of the random vector X. Thus, T is aso a function of X and it hods that S T. Using S and T, it hods that N ˆσ 2 /σ 2 = S + T and we obtain Nσ 2 ˆσ 2 = N 2 N ˆσ 2 /σ 2 = N 2 S + T = N 2 S + T/S. (2.) 5

6 In addition, noting that ( + x) = x + c x 2 where x 0 and 0 c, (2.) can be expanded as Nσ 2 ˆσ 2 = N 2 S N 2 T S 2 + C N 2 T 2 S 3, where C is a random variabe with 0 C. Hence, from S χ 2 N k have E Nσ 2 ˆσ 2 = N 2 N k 2 N 2 ET (N k 2)(N k 4) + E C N 2 T 2 = N + k O(N ) ET + O(N )ET + E C N 2 T 2 Simiary, using ( + y) = c y where y 0 and 0 c, we get k i= N i(θ i, ˆθ i ) 2 ˆσ 2 = N k i= N i(θ i, ˆθ i ) 2 σ 2 = N k i= N i(θ i, ˆθ i ) 2 S + T σ 2 S + T/S = N k i= N i(θ i, ˆθ i ) 2 σ 2 C NT k i= N i(θ i, ˆθ i ) 2 S σ 2 S 2 = N k i= N i(θ i, X i + X i ˆθ i ) 2 σ 2 C NT k i= N i(θ i, ˆθ i ) 2 S σ 2 S 2 = N k i= N i(θ i, X i ) 2 σ 2 2N k i= N i( X i θ i, )( X i ˆθ i ) S σ 2 S + NT k NT i= N i(θ i, ˆθ i ) 2 C S σ 2 S 2, S 3 and S T, we S 3. (2.2) where C is a random variabe with 0 C. Here, for any i with i k, it hods that S X i, S ˆθ i, S T and X i N(θ i,, σ 2 /N i ). Therefore, we obtain k i= E N i(θ i, ˆθ i ) 2 ˆσ 2 Nk = N k 2 2N N k 2 E + NET N k 2 E = k + O(N ) C NT σ 2 2N N k 2 E +ET + O(N )ET E σ 2 k N i ( X i θ i, )( X i ˆθ i ) i= k i= N i(θ i, ˆθ i ) 2 σ 2 S 2 k N i ( X i θ i, )( X i ˆθ i ) i= C NT σ 2 k i= N i(θ i, ˆθ i ) 2. (2.3) S 2 6

7 Thus, from (2.2) and (2.3), it hods that E k Nσ 2 i= ˆσ 2 + E N i(θ i, ˆθ i ) 2 ˆσ 2 = N + 2(k + ) 2N N k 2 E σ 2 k N i ( X i θ i, )( X i ˆθ i ) + J, (2.4) i= where J is given by J = O(N ) + O(N )ET + E C N 2 T 2 S 3 E C NT σ 2 k i= N i(θ i, ˆθ i ) 2. S 2 Here, from the definition of ˆθ, it hods that X ˆθ N X θ N for any θ Θ. Moreover, from the assumption, the true vaue θ satisfies θ Θ. Thus, it hods that X ˆθ N X θ N and T = σ 2 k i= N i ( X i ˆθ i ) 2 = σ 2 ( X ˆθ N ) 2 σ 2 ( X θ N ) 2 = σ 2 k N i ( X i θ i, ) 2 K, i= (say), where K χ 2 k. Therefore, by using the above inequaity, we get 0 ET EK = k and ET = O(). In addition, noting that 0 C, we have E C N 2 T 2 N 2 T 2 E = S 3 S 3 N 2 ET 2 (N k 2)(N k 4)(N k 6) O(N )EK 2 = O(N )(2k + k 2 ) = O(N ). This impies E C N 2 T 2 = O(N ). S 3 Noting that the triange inequaity θ ˆθ N θ X N + X ˆθ N and X ˆθ N X θ N, we obtain θ ˆθ N 2 θ X N. Hence, since 0 C and T K, the foowing inequaity hods: E C NT k i= N i(θ i, ˆθ i ) 2 σ 2 S 2 k NT i= E N i(θ i, ˆθ i ) 2 σ 2 S 2 N T (N k 2)(N k 4) E σ 2 ( θ ˆθ N ) 2 O(N )E4K 2 = O(N ). 7

8 This impies E C NT σ 2 k i= N i(θ i, ˆθ i ) 2 = O(N ). S 2 Thus, from the definition of J, we obtain J = O(N ). From (2.0) and (2.4), the bias B can be expressed as B = 2(k + ) 2N N k 2 E σ 2 k N i ( X i θ i, )( X i ˆθ i ) + O(N ). (2.5) i= Hence, in order to correct the bias up to the order of N, we must cacuate the expected vaue in (2.5). 3. Notation and main emma In this section, we provide the emma to cacuate the expected vaue in (2.5). First, we define severa notations. 3.. Notation Let be an integer with 2 and et n,..., n be positive numbers. We put n = (n,..., n ). For any -dimensiona vector x = (x,..., x ) R, and for any i, j with i j, we write x i,j = (x i,..., x j ). Note that x i,j is a (j i + )-dimensiona vector whose the sth eement is x i+s where s j i +. In particuar, x i,i = x i and x, = x. Let x i,j = j x s, s=i x (n) i,j = j s=i n sx s j s=i n s = j s=i n sx s ñ i,j = n i,j x i,j ñ i,j. For simpicity, we often represent x (n) i,j as x i,j. Note that x i,i = x i. Next, et A be a set defined by A = {(a,..., a ) R a a 2 a } = {(a,..., a ) R t, a t a t+ }, and et A and A be sets defined by A = {(x,..., x ) R x = x 2 = = x }, and A = {(x,..., x ) R x < x 2 < < x } = {(x,..., x ) R t, x t < x t+ }. 8

9 We define A = R. Moreover, for any integer i with i, we write W i = {(w,..., w i ) N i t i, w t < w t, w 0 = 0, w i = }. Hence, for exampe, in the case of = 2, the sets W 2 and W 2 2 are given by W 2 = {(2) }, W 2 2 = {(, 2) }, and in the case of = 3, the sets W 3, W 3 2 and W 3 3 are given by W 3 = {(3) }, W 3 2 = {(, 3), (2, 3) }, W 3 3 = {(, 2, 3) }. Furthermore, in the case of = 4, the sets W 4, W 4 2, W 4 3 and W 4 4 are given by W 4 = {(4) }, W 4 2 = {(, 4), (2, 4), (3, 4) }, W 4 3 = {(, 2, 4), (, 3, 4), (2, 3, 4) }, W 4 4 = {(, 2, 3, 4) }. Note that the number of eements of Wi is C i. Aso note that, for any eement w = (w,..., w i ) in Wi, w is an i-dimensiona vector and w i =. From the definitions of W and W, W has the unique eement w = () and W has the unique eement w = (,..., ). Furthermore, for any i (i =,..., ) and for any w Wi, we define a set A i (w) as foows. First, in the case of i =, W has the unique eement w = (), and we define A (w) = {(x,..., x ) R x = x 2 = = x } = A. On the other hand, in the case of 2 i, for any eement w = (w,..., w i ) in W i, we define A i(w) = {(a,..., a ) A t i, a wt < a wt+, 0 s i, w 0 = 0, a +ws = a ws+ }, = {(x,..., x ) R t i, x wt < x wt+, 0 s i, w 0 = 0, x +ws = = x ws+ }. (3.) Thus, from (3.), the eement x = (x,..., x ) in A i (w) satisfies x = = x w < x +w = = x w2 < < x +wi = = x. In particuar, when i =, W has the unique eement w = (w,..., w ) = (,..., ), and it hods that A (w) = {(x,..., x ) R x < x 2 < < x } = A. Here, we provide severa exampes. When = 2, A 2 (w) and A 2 2(w) can be expressed as A 2 (w) = A 2 = {x R 2 x = x 2 }, A 2 2(w) = A 2 2 = {x R 2 x < x 2 }. 9

10 In addition, when = 3, for each s (s =, 2, 3), A 3 s(w) can be expressed as A 3 (w) = A 3 = {x R 3 x = x 2 = x 3 }, A 3 2(w) = {x R 3 x < x 2 = x 3 }, (if w = (, 3) W 3 2 ) A 3 2(w) = {x R 3 x = x 2 < x 3 }, (if w = (2, 3) W 3 2 ) A 3 3(w) = A 3 3 = {x R 3 x < x 2 < x 3 }. Therefore, in genera, it hods that and A = i= w;w Wi A i(w), (3.2) (i, w) (i, w ) A i(w) A i (w ) =. (3.3) Next, for any i and j with i j, we define a matrix D (n) i,j. First, when i = j, et D (n) i,j be a matrix and et D (n) i,j be a (j i) (j i + ) matrix and et the sth row of D (n) i,j = 0. On the other hand, when i < j, et D (n) i,j ( s j i) be defined by ( ) n i,i+s ñ, n i+s,j. (3.4) i,i+s ñ i+s,j Hence, for exampe, when = 4, D (n) i,j s ( i j 4) are given by D (n), = D(n) 2,2 = D(n) 3,3 = D(n) 4,4 = 0, D (n),2 = D(n) 2,3 = D(n) 3,4 = ( ), D (n) ( n 2,3 = n n 2 n +n 2 n D (n),4 = n 3 n 2 +n 3 n 2 +n 3 n +n 2 ) (, D (n) n 3 2,4 = n 2 n 2 n 3 n 4 n 2 +n 3 +n 4 n 2 +n 3 +n 4 n 3 n 2 +n 3 +n 4 n 4 n +n 2 n 3 +n 4 n +n 2 n n 2 n +n 2 +n 3 n 2 n +n 2 +n 3 n 3 n +n 2 +n 3 n 3 +n 4 n 4 n 3 +n 4 n 3 +n 4 n 3 n 2 +n 3 n 2 +n 3. ), For simpicity, we often represent D (n) i,j as D i,j. Finay, we define a function η (n). Let η (n) be a function from R to A, and et η (n) (x) be defined by η (n) (x) = argmin x y 2 n, y A for any x = (x,..., x ) R. For simpicity, we often represent η (n) as η. Note that η (x) is we-defined because (R, n ) is a Hibert space and A is the non-empty cosed convex set (see, e.g., Rudin, 986). Aso note that η (x) is an -dimensiona vector. Let 0

11 η (x)s be a sth eement of η (x) ( s ). Then, from Robertson et a. (988), η (x)s can be expressed as η (x)s = min v;v s max u;u s v j=u n jx j v j=u n j = min v;v s max u;u s x u,v. (3.5) In addition, we define η (x) = x Main emma The foowing emma hods. Lemma 3.. Let k be an integer with k 2, and et n,..., n k be positive numbers. Let ξ,..., ξ k be rea numbers, and et be a positive number. Suppose that x,..., x k are independent random variabes, and x i N(ξ i, /n i ), (i =,..., k). We put n = (n,..., n k ), ξ = (ξ,..., ξ k ) and x = (x,..., x k ). Then, it hods that Proof. See Appendix. k E i= k = (k i)p i= n i (x i ξ i )(x i η (n) k (x)i) η (n) k (x) w;w W k i A k i (w). 4. AIC for ANOVA mode with the simpe ordering In this section, we derive AIC for ANOVA mode (2.) with the SO. First, we cacuate the expected vaue in (2.5). From (2.3), X,..., X k are mutuay independent, and for any i, with i k, it hods that X i N(θ i,, σ /N 2 i ). Furthermore, from (2.7) the MLE ˆθ can be expressed as ˆθ = η (N) k ( X). Hence, from Lemma 3., the expected vaue in (2.5) can be written as E σ 2 k N i ( X i θ i, )( X i ˆθ i ) = E i= σ 2 k i= k = (k i)p ˆθ i= N i ( X i θ i, )( X i η (N) k ( X)i) w;w W k i Thus, noting that Q = O(), substituting Q into (2.5) yieds B = 2(k + ) A k i (w) = Q. (say) 2N N k 2 Q + O(N ) = 2(k + ) 2Q + O(N ). (4.)

12 Therefore, in order to correct the bias up to the order of N, we ony have to add 2(k + ) 2Q to 2( ˆθ, ˆσ 2 ; X). However, it is easiy checked that Q depends on the true vaues θ,,..., θ k, and σ. 2 Thus, we must estimate Q. Here, et k ˆM = {ˆθ i }, i= and et ˆm = # ˆM, (4.2) where the notation # ˆM means that the number of eements of ˆM. From the definition of ˆm, ˆm is a discrete random variabe, and its possibe vaues are to k. For exampe, if ˆθ = = ˆθ k, then ˆm =, and if ˆθ < ˆθ 2 < < ˆθ k, then ˆm = k. Simiary, if ˆθ < ˆθ 2 = = ˆθ k, then ˆm = 2. Here, from the definitions of ˆm and A k i (w), we have ˆθ w;w W k i A k i (w) ˆm = i. This impies Ek ˆm = k k (k i)p( ˆm = i) = (k i)p ˆθ i= i= w;w Wi k A k i (w) = Q. Thus, k ˆm is an unbiased estimator of Q. Therefore, from (4.) we obtain E2( ˆm + ) = E2(k + ) 2(k ˆm) = 2(k + ) 2Q = B + O(N ). Hence, adding 2( ˆm + ) (instead of 2(k + ) 2Q) to 2( ˆθ, ˆσ 2 ; X), we obtain AIC for ANOVA mode with the SO, caed AIC SO. Theorem 4.. Let ( ˆθ, ˆσ 2 ; X) be the maximum og-ikeihood given by (2.9), and et ˆm be the random variabe given by (4.2). Then, the AIC SO is defined by AIC SO := 2( ˆθ, ˆσ 2 ; X) + 2( ˆm + ). In addition, for the risk function R given by (2.8), it hods that EAIC SO = R + O(N ). Remark 4.. The AIC SO is derived under the order restriction (2.2). However, we can aso derive the AIC SO even if we change a part of inequaities in (2.2) to =. For exampe, when k = 4 we can derive the AIC SO for the mode (2.) with θ = θ 2 θ 3 = θ 4. (4.3) 2

13 In this case, putting N = N + N 2, N2 = N 3 + N 4, θ = θ 2 = µ and θ 3 = θ 4 = µ 2, and repacing X,..., X N, X 2,..., X 2N2 Y,..., Y N, X 3,..., X 3N3, X 4,..., X 4N4 Y 2,..., Y 2N 2, the mode (2.) under (4.3) is equa to the mode Y ij N(µ i, σ 2 ), (i =, 2, j =,..., Ni ) under the restriction µ µ 2. Hence, by using the same argument, we can derive the AIC SO. Remark 4.2. The AIC SO is an asymptoticay unbiased estimator of the risk function R and the order of the bias is N. Simiary, for the ordina ANOVA mode without any order restriction, the ordina AIC is aso an asymptoticay unbiased estimator of the risk function R, and the order of the bias is N. Thus, the AIC SO is as good as the AIC from the viewpoint of estimation accuracy of risk functions. In addition, the penaty term of AIC SO is 2( ˆm + ), and from (4.2), ˆm is simpy defined as the function of the MLE. Therefore, aso from the viewpoint of usefuness, the AIC SO is as good as AIC. 5. AIC SO for severa specia cases In this section, we provide the AIC SO for severa specia cases. 5.. AIC SO when the true variance σ 2 is known In this subsection, we assume that the true variance σ 2 is known in ANOVA mode (2.). Then, under this assumption and the SO, the MLEs ˆθ,..., ˆθ k of θ,..., θ k are given by (2.7) because (2.6) does not depend on the variance parameter. Furthermore, in this case, the risk function based on the K-L divergence, R can be written by repacing ˆσ 2 with σ 2 in (2.8) as R = EE 2( ˆθ, σ ; 2 X ) k = E N og(2πσ ) 2 i= + N + N i(θ i, ˆθ i ) 2 = N og(2πσ 2 ) + N + E σ 2 k i= N i(θ i, X i + X i ˆθ i ) 2 k = N og(2πσ ) 2 i= + N + k 2Q + E N i( X i ˆθ i ) 2 σ 2. (5.) 3 σ 2

14 Note that under the ordina ANOVA mode without the SO, when σ 2 is known the risk R is given by R = EE 2( X, σ 2 ; X ). Here, from (2.4), the maximum og-ikeihood ( ˆθ, σ 2 ; X) can be expressed as ( ˆθ, σ 2 ; X) = N 2 og(2πσ2 ) 2σ 2 k N i (X ij X i ) 2 i= j= 2σ 2 k N i ( X i ˆθ i ) 2. (5.2) i= Hence, the bias B which is the difference between the expected vaue of 2( ˆθ, σ 2 ; X) and R, can be expressed as B = ER { 2( ˆθ, σ ; 2 X)} = N + k 2Q E σ 2 k N i (X ij X i ) 2 i= j= = N + k 2Q (N k) = 2k 2Q. Reca that the random variabe ˆm given by (4.2) satisfies Ek ˆm = Q. Therefore, we obtain the foowing coroary. Coroary 5.. Let ( ˆθ, σ ; 2 X) be the maximum og-ikeihood given by (5.2), and et ˆm be the random variabe given by (4.2). Then, under ANOVA mode (2.) with the SO and known variance σ, 2 the AIC SO is given by AIC SO = 2( ˆθ, σ ; 2 X) + 2 ˆm. Moreover, it hods that EAIC SO = R, where R is the risk function given by (5.). Remark 5.. When the true variance σ 2 is known, the AIC SO is an unbiased estimator of the risk function R. In addition, under the ordina ANOVA mode without the SO, when σ 2 is known the ordina AIC is an unbiased estimator of the risk function R AIC SO with known variance weights In this subsection, we consider the foowing mode: X ij N(θ i, ι i σ 2 ), (i =,..., k, j =,..., N i ), (5.3) where θ,..., θ k and σ 2 are unknown parameters, and ι,..., ι k are known positive weights. Furthermore, aso in this mode, we assume the SO given by (2.2) for the parameters θ,..., θ k. Here, et X i = N i N i j= X ij, σ 2 = N k ι i= i N i j= (X ij Xi ) 2. 4

15 Note that under the ordina ANOVA mode (5.3) without the SO, the MLEs of θ i and σ 2 are given by Xi and σ 2, respectivey. In this setting, we put X = ( X,..., Xk ). Next, define n i = N i /ι i, for any i ( i k). Then, Xi is distributed as N(θ i, σ 2 /n i ). Therefore, putting n = (n,..., n k ) and ι = (ι,..., ι k ), the og-ikeihood function (θ, σ 2 ; X, ι) can be written by (θ, σ 2 ; X, ι) = k N i og ι i N 2 2 og(2πσ2 ) 2σ 2 = 2 = 2 i= k i= k i= N i og ι i N 2 og(2πσ2 ) 2σ 2 N i og ι i N 2 og(2πσ2 ) 2σ 2 k ι i= i k ι i= i k ι i= i N i j= N i j= N i j= (X ij θ i ) 2 (X ij Xi ) 2 2σ 2 k n i ( Xi θ i ) 2 i= (X ij Xi ) 2 2σ 2 X θ 2 n. Thus, by using the same argument as in Subsection 2., the MLEs of θ i and σ 2, ˆθi and ˆσ 2, respectivey, are give by ˆθ i = min v;i v max u;u i ˆσ 2 = N k ι i= i j= v j=u n j Xj v j=u n, (i =,..., k), j N i (X ij Xi ) 2 + N k n i ( Xi ˆθi ) 2. Next, we put ˆθ = (ˆθ,..., ˆθk ) and R 2 = EE 2(ˆθ, ˆσ 2 ; X, ι) where R 2 is the risk function. Furthermore, the maximum og-ikeihood (ˆθ, ˆσ 2 ; X, ι) is given by (ˆθ, ˆσ 2 ; X, ι) = 2 i= k N i og ι i N 2 og(2πˆσ 2 ) N 2. (5.4) i= Moreover, by using the same argument as in Subsection 2.2, the bias B 2 { 2(ˆθ, ˆσ 2 ; X, ι)} can be expressed as Here, define B 2 = 2(k + ) 2N N k 2 E M = σ 2 k n i ( Xi θ i, )( Xi ˆθi ) + O(N ). i= = ER 2 k {ˆθi }, m = #M. (5.5) i= Then, we obtain the foowing coroary. Coroary 5.2. Let (ˆθ, ˆσ 2 ; X, ι) be the og-ikeihood given by (5.4), and et m be the random variabe given by (5.5). Then, under ANOVA mode (5.3) with the SO, the 5

16 AIC SO is given by AIC SO = 2(ˆθ, ˆσ 2 ; X, ι) + 2(m + ). Furthermore, it hods that EAIC SO = R 2 + O(N ) where R 2 is the risk function defined by R 2 = EE 2(ˆθ, ˆσ 2 ; X, ι). Remark 5.2. Under ANOVA mode (5.3) with the SO, when σ 2 is known, the AIC SO can be derived as AIC SO = 2(ˆθ, σ 2 ; X, ι) + 2m. Furthermore, for the risk function R 3 = EE 2(ˆθ, σ 2 ; X, ι), it hods that EAIC SO = R Mutivariate ANOVA mode with the SO Let V (i) j = (V (i) j,..., V (i) jp ) be a p-dimensiona random vector on the jth individua in the ith custer, where i =,..., k and j =,..., N i. Here, et k 2, p 2 and N = N + + N k. In this setting, we assume N k 6 > 0. Moreover, we assume that V (),..., V (k) N k are mutuay independent. Then, we consider the foowing mode V (i) j N p (ω + δ i a, I p + ρaa ), ( > 0, + ρa a > 0), (5.6) where ω = (ω,..., ω p ), and δ,..., δ k, and ρ are unknown parameters. In addition, I p is a p p unit matrix, and a = (a,..., a p ) is a known non-zero vector. Here, without oss of generaity, we may assume that δ = 0. Moreover, the parameters δ,..., δ k are restricted as δ δ 2 δ k. (5.7) In other words, we consider the SO restriction for the parameters δ,..., δ k. For exampe, under the mode (5.6), when a = p this mode is a parae profie mode considered by Yokoyama and Fujikoshi (993), where p is a p-dimensiona vector of ones. Next, we decompose the mode (5.6). Let h,..., h p be p-dimensiona vectors with h uh u = 0, (u u ), h uh u = and h = (a a) /2 a. Define H 2 = (h 2,..., h p ) and H = (h, H 2 ). Then, considering H V (i) j we get h V (i) j N(h ω + (a a) /2 δ i, + ρa a), ( i k, j N i ), (5.8) and H 2V (i) j N p (H 2ω, I p ), ( i k, j N i ). (5.9) Here, we repace h V (i) j with Y ij. In addition, we put h ω + (a a) /2 δ i = ϑ i and + ρa a = ς 2. Then, the mode (5.8) is equa to Y ij N(ϑ i, ς 2 ), (ς 2 > 0, i k, j N i ), (5.0) 6

17 and the parameters ϑ,..., ϑ k are restricted as ϑ ϑ 2 ϑ k. Furthermore, since H 2V ()..., H 2V (k) N k are independent and identicay distributed, putting H 2ω = (µ,..., µ p ) = µ the mode (5.9) can be expressed as Z st N(µ s, ), ( > 0, s p, t N). (5.) Note that Y,..., Y knk, Z,..., Z (p )N are mutuay independent. Aso note that the parameters µ,..., µ p are not restricted. Here, et Ȳ i = N i Z s = N N i j= Y ij, ς 2 = N N Z st, = t= k N i (Y ij Ȳi) 2, (i =,..., k), i= j= p N(p ) s= N (Z st Z s ) 2, (s =,..., p ), and et Ȳ = (Ȳ,..., Ȳk). Since µ s and are not restricted, it is easiy checked that t= the MLEs of µ s and are Z s and, respectivey. Next, we put Y = (Y,..., Y knk ), Z = (Z,..., Z (p )N ) and ϑ = (ϑ,..., ϑ k ). Then, the og-ikeihood function (ϑ, ς 2 ; Y ) of Y, is given by (ϑ, ς 2 ; Y ) = N 2 og(2πς2 ) 2ς 2 = N 2 og(2πς2 ) 2ς 2 k N i (Y ij ϑ i ) 2 i= j= k N i (Y ij Ȳi) 2 2ς 2 i= j= k N i (Ȳi ϑ i ) 2. i= Simiary, the og-ikeihood function (µ, ; Z) of Z is given by (µ, N(p ) ; Z) = og(2π ) p 2 2 s= N (Z st µ s ) 2. By using the same argument as in Subsection 2., the MLEs of ϑ i and ς 2, ˆϑ i and ˆς 2 can be expressed as t= ˆϑ i = min v;i v max u;u i ˆς 2 = N v j=u N jȳj v j=u N, j k N i (Y ij Ȳi) 2 + N i= j= k N i (Ȳi ˆϑ i ) 2, i= respectivey. Note that the joint og-ikeihood of Y and Z, (ϑ, ς 2, µ, ; Y, Z) satisfies that (ϑ, ς 2, µ, ; Y, Z) = (ϑ, ς 2 ; Y ) + (µ, ; Z) because Y and Z are independent. Here, et Y and Z be random vectors satisfying (Y, Z ) i.d.d. (Y, Z). Then, 7

18 the risk function R 4 can be written as R 4 = EE 2( ˆϑ, ˆς 2, Z, ; Y, Z ) = EE 2( ˆϑ, ˆς 2 ; Y ) + EE 2( Z, ; Z ), where, ˆϑ = ( ˆϑ,..., ˆϑ k ) and Z = ( Z,..., Z p ). In order to cacuate the bias which is the difference between the expected vaue of 2( ˆϑ, ˆς 2, Z, ; Y, Z) and R 4, it is sufficient to cacuate and Here, it is easiy checked that On the other hand, define EE 2( ˆϑ, ˆς 2 ; Y ) + 2( ˆϑ, ˆς 2 ; Y ), EE 2( Z, ; Z ) + 2( Z, ; Z). EE 2( Z, ; Z ) + 2( Z, ; Z) = 2p + O(N ). M = k { ˆϑ i }, m = #M. (5.2) i= Then, using the same argument as in Section 4, we have EE 2( ˆϑ, ˆς 2 ; Y ) + 2( ˆϑ, ˆς 2 ; Y ) = 2(m + ) + O(N ). (5.3) Therefore, we obtain the foowing coroary. Coroary 5.3. Under the mode (5.6) with the order restriction (5.7), the AIC SO is given by AIC SO = 2( ˆϑ, ˆς 2, Z, ; Y, Z) + 2(m + + p). Furthermore, it hods that EAIC SO = R 4 + O(N ). Remark 5.3. Under the mode (5.6) with the order restriction (5.7), when both and ρ are known (i.e., both ς 2 and are known), the AIC SO can be derived as AIC SO = 2( ˆϑ, ς 2, Z, ; Y, Z) + 2(m + p ). Moreover, for the risk function R 5 = EE 2( ˆϑ, ς 2, Z, ; Y, Z ), it hods that EAIC SO = R 5. We introduced six modes thus far. In other words, the mode (2.) when σ 2 is unknown (Case A), and known (Case B). Moreover, the mode (5.3) when σ 2 is unknown (Case C), and known (Case D). Finay, the mode (5.0) and (5.) when both ς 2 and are unknown (Case E), and known (Case F). The properties of the AIC SO and the ordina AIC for these six modes are summarized in Tabe 5.. 8

19 5.4. Comparison of the AIC SO and the pseudo AIC (paic) under certain candidate modes Let k be an integer with k 2, and et Wi k be the set defined as in Subsection 3. where i is an integer with i k. Moreover, for any i (i =,..., k) and for any w Wi k, we define a set Ck i (w) as foows. First, in the case of i =, Wk has the unique eement w = (k), and we define C k (w) = {(x,..., x k ) R k x = x 2 = = x k } = C k. On the other hand, in the case of 2 i k, for any eement w = (w,..., w i ) in Wi k, we define Ci k (w) = {(x,..., x k ) R k t i, x wt x wt+, 0 s i, w 0 = 0, x +ws = = x ws+ }.(5.4) Thus, from (5.4), the eement x = (x,..., x k ) in Ci k (w) satisfies x = = x w x +w = = x w2 x +wi = = x k. In particuar, when i = k, Wk k has the unique eement w = (w,..., w k ) = (,..., k), and it hods that Ck k (w) = {(x,..., x k ) R k x x 2 x k } = Ck k. Here, et X st be independent random variabes where s =,..., k and t =,..., N s. Then, for any i with i k and for any w Wi k, we consider ANOVA mode X st N(θ s, σ 2 ), (s =,..., k, t =,..., N s ), with θ = (θ,..., θ k ) Ci k(w). For exampe, when k = 5, i = 3 and w = (w, w 2, w 3 ) = (, 3, 5) C3, 5 above mode is equa to ANOVA mode with θ θ 2 = θ 3 θ 4 = θ 5. Reca that the number of eements of Wi k is k C i. Hence, it hods that k #Wi k = 2 k. i= This impies that we can consider 2 k modes. In this subsection, these modes are candidate modes. Next, we consider the AIC SO and the pseudo AIC (paic) for these candidate modes. Reca that the paic is defined as paic = 2(ˆθ) + 2p, 9

20 where ( ) is a maximum og-ikeihood, p is the number of independent parameters in the candidate mode, and ˆθ is the MLE which is derived under the order restricted mode. In this setting, we define the minimum AIC SO mode and the minimum paic mode. Let M,..., M 2 k be candidate modes, and et AIC SO (M q ) and paic(m q ) be vaues of the AIC SO and the paic in the candidate mode M q, respectivey. Then, we define that the candidate mode M q is the minimum AIC SO mode if M q satisfies the foowing two conditions: (m) For any candidate mode M q, it hods that AIC SO (M q ) AIC SO (M q ). (m2) For any candidate mode M q with AIC SO (M q ) = AIC SO (M q ), it hods that #(M q ) #(M q ) where #(M j ) is the number of independent parameters in the candidate mode M j, (j =,..., 2 k ). Simiary, by repacing AIC SO with paic in the conditions (m) and (m2), we aso define the minimum paic mode. Then, the foowing theorem hods. Theorem 5.. Let k ( 2) be an integer, and et M,..., M 2 k be candidate modes defined as in Subsection 5.4. Then, the minimum AIC SO mode is equa to the minimum paic mode. (q) (q) Proof. Let M q be the minimum AIC SO mode, and et ˆθ,..., ˆθ k be the MLEs of (q) θ,..., θ k in the mode M q, respectivey. First, we consider the case of not ˆθ ˆθ (q) k = =. Hence, there exists a number i (2 i k) and natura numbers w,..., w i with w < < w i where w i = k such that and ˆθ (q) w < < ˆθ (q) (q) w j + = = ˆθ w j = wj s=w j + N X s s wj s=w j + N, (j =,..., i), (5.5) s ˆθ (q) w i. Note that w 0 = 0. Here, et (q) be a 2 maximum og-ikeihood in the mode M q. Furthermore, from (5.5) it hods that ˆm = i. Therefore, AIC SO (M q ) can be written as AIC SO (M q ) = (q) + 2( + i). Moreover, from the definition of the minimum AIC SO mode, the mode M q is ANOVA mode with θ w0 + = = θ w θ w + = = θ w2 θ wi + = = θ wi. In this mode, the number of independent parameters is i +. Thus, paic(m q ) is aso (q) + 2( + i). Hence, we get AIC SO (M q ) = paic(m q ). On the other hand, from the 20

21 definition of M q it hods that paic(m q ) = AIC SO (M q ) = min AIC SO (M u ) u 2 k min u 2 k, u q AIC SO (M u ). (5.6) Furthermore, from the definitions of the AIC SO and the paic, it is cear that AIC SO (M u ) paic(m u ). Therefore, combining this inequaity and (5.6), we obtain paic(m q ) min AIC SO (M u ) min paic(m u ). u 2 k, u q u 2 k, u q Hence, for any candidate mode M u, it hods that paic(m q ) paic(m u ). In addition, for any candidate mode M u with paic(m q ) = paic(m u ), it hods that #(M q ) #(M u ). In fact, if paic(m q ) = paic(m u ) and i = #(M u ) < #(M q ) = i, it hods that AIC SO (M q ) = paic(m q ) = paic(m u ) AIC SO (M u ). However, since i = #(M u ) < #(M q ) = i, this impies that M q is not the minimum AIC SO mode. This is a contradiction. Hence, for any candidate mode M u with paic(m q ) = paic(m u ), it hods that #(M q ) #(M u ). Therefore, the minimum paic mode is M q. Simiary, by using the same argument, we can aso prove the case (q) (q) of ˆθ = = ˆθ k. Reca that the AIC SO is the asymptoticay unbiased estimator of the risk function. Furthermore, in genera, the paic is the asymptoticay biased estimator of the risk function. However, Theorem 5. means that the minimum AIC SO mode based on the AIC SO is equa to the minimum paic mode based on the paic athough the AIC SO and paic are asymptoticay unbiased and biased estimators, respectivey. In other words, when we consider the mode seection probem for these candidate modes using the AIC SO or the paic, we may use the paic. Remark 5.4. Needess to say, for these candidate modes, we can aso use the AIC SO. Here, we woud ike to note that, in genera, the resut of Theorem 5. does not hod when the number of candidate modes is smaer than 2 k. For exampe, when we consider the nested candidate modes, in genera, the minimum AIC SO mode is not equa to the minimum paic mode. 6. Numerica experiments Let X ij be a random variabe distributed as N(θ i, σ 2 /N i ) where i 4, j N i and N = = N 4. Moreover, et N = N + N 2 + N 3 + N 4. In this section, we consider 2

22 the foowing four candidate modes: Mode : ANOVA mode with θ = θ 2 = θ 3 = θ 4, Mode 2 : ANOVA mode with θ θ 2 = θ 3 = θ 4, Mode 3 : ANOVA mode with θ θ 2 θ 3 = θ 4, Mode 4 : ANOVA mode with θ θ 2 θ 3 θ 4. Thus, these four modes are nested. From 00,000 monte caro simuation runs, we compare performance of the AIC SO and the paic. In the qth simuation, where ( (q) (q) ˆθ ˆθ ˆσ 2(q) AIC SO be MLEs of the parameters θ,..., θ 4 q 00000), et,aic SO,..., 4,AIC SO and and σ 2 for the minimum AIC SO mode, respectivey. Simiary, et (q) (q) ˆθ,pAIC,..., ˆθ 4,pAIC and ˆσ 2(q) paic be MLEs of the parameters θ,..., θ 4 and σ 2 for the minimum paic mode, respectivey. Here, since the risk function of ANOVA mode with the SO is given by (2.8), the estimator 4 R(ˆθ,..., ˆθ 4, ˆσ 2 ) = N og(2πˆσ 2 ) + Nσ2 i= ˆσ 2 + N i(θ i, ˆθ i ) 2 ˆσ 2, is an unbiased estimator of the risk function. Based on this, we evauate performance of the AIC SO and the paic as PE AICSO = PE paic = R(ˆθ (q) (q),aic SO,..., ˆθ 4,AIC SO, q= q= ˆσ 2(q) AIC SO ), R(ˆθ (q) (q),paic,..., ˆθ 4,pAIC, ˆσ 2(q) paic). Thus, the PE AICSO and the PE paic are estimated vaues of risk functions for the minimum AIC SO mode (the mode seected by using the AIC SO ) and the minimum paic mode (the mode seected by using the paic), respectivey. Next, in this simuation, we consider the foowing true modes: Case : θ = θ 2 = 2, θ 3 = θ 4 = 2.8, σ 2 = 2, Case 2 : θ =.5, θ 2 =.8, θ 3 = 2., θ 4 = 2.4, σ 2 = 2, Case 3 : θ = θ 2 = θ 3 = θ 4 = 2.5, σ 2 = 2. In Case, Mode 3 and 4 incude the true mode, and in Case 2, Mode 4 incudes the true mode. Moreover, in Case 3, Mode, 2, 3 and 4 incude the true mode. For these cases, we set N = 40 and N = 200. The vaues of the PE AICSO cases 3 are given in Tabe , respectivey. and the PE paic in the From Tabe , we can see that the AIC SO is an asymptoticay unbiased estimator of the risk function. Reca that from the definitions of the AIC SO and the paic, the vaue of the AIC SO is equa to or smaer than that of the paic. We can confirm that this 22

23 inequaity hods for a cases. Moreover, for the vaues of the PE AICSO and the PE paic, from Tabe 6. and 6.2, we can see that the PE AICSO is smaer than the PE paic in Case and 2. Thus, compared with the paic, mode seections using the AIC SO are better from the viewpoint of the risk for the seected mode in Case and 2. On the other hand, in case 3, the mode seection using the paic is better. 7. Concusion In this paper, we derived the AIC SO for ANOVA mode with the simpe order restriction. We showed that the AIC SO is the asymptoticay unbiased estimator of the risk function. Furthermore, we aso showed that if the true variance is known, the AIC SO is the unbiased estimator. We woud ike to note that since the penaty term of the AIC SO is simpy defined as the function of the MLEs, the AIC SO is very usefu for anaysts. Thus, from the viewpoint of usefuness and estimation accuracy of the risk function, the AIC SO is as good as the ordina AIC. Furthermore, Theorem 5. shows that under certain candidate modes, the seected modes based on minimizing the AIC SO and the paic are the same mode. In addition, from numerica experiments we coud confirm that the AIC SO is an unbiased estimator of the risk function. Appendix In this section, we define severa notations. Next, we show seven emmas, Lemma A G, and using Lemma F and Lemma G we prove Lemma 3.. First, we define the inequaity of vectors. Let x = (x,..., x p ) and y = (y,..., y p ) be p-dimensiona vectors, and et 0 p be a p-dimensiona vector of zeros. Then, define x 0 p i {,..., p}, x i 0, x y x y 0 p. Furthermore, for some proposition P, we define an indicator function {P } as {P } = { if P is true 0 if P is not true. Appendix A: Lemma A and its proof Lemma A. Let be an integer with 2, and et n,..., n be eements of R >0. Let n = (n,..., n ), and et x = (x,..., x ) be a vector of R. Then, the foowing (i), (ii) and (iii) hod: 23

24 (i) For a integers a, b and c with a b < c, it hods that and x a,b x a,c x a,b x b+,c x a,c x b+,c, x a,b < x a,c x a,b < x b+,c x a,c < x b+,c. (A.) (A.2) (ii) Let i be an integer with 2 i, and et w,..., w i be integers with w < w 2 < < w i and w i =. Put w 0 = 0. Then, if x +w0,w < x +w,w 2 < < x +wi,w i (A.3) is true, for a integers s and t with s < t i, it hods that x +ws,w s < x +ws,w t. (A.4) (iii) Let i and j be integers with i < j. Then, it hods that x i,b x b+,j, ( b N with i b < j) D i,j x i,j 0 j i. (A.5) Proof. First, we prove (i). Let a, b and c be integers with a b < c. In this setting, we show x a,b < x a,c x a,b < x b+,c. Let x a,b < x a,c, i.e., x a,b x a,c < 0. Then, we get x a,b x a,c = = = = = b j=a n jx j ñ a,b b j=a n jx j j=a ñ a,b c j=a n jx j ñ a,c b j=a n jx j + c j=b+ n jx j ñ a,c b ( n j x j ) c j=b+ n jx j ñ a,b ñ a,c ñ a,c b ) c (ña,c ñ a,b j=b+ n j x j n jx j ñ a,b ñ a,c ñ a,c b ( ) c ñb+,c j=b+ n j x j n jx j ñ a,b ñ a,c ñ a,c ( b j=a n c jx j j=b+ n ) jx j j=a j=a = ñb+,c ñ a,c ñ a,b ñ b+,c Hence, noting that ñ b+,c /ñ a,c is positive, we have x a,b x a,c < 0 ñb+,c ñ a,c ( x a,b x b+,c ) < 0 = ñb+,c ñ a,c ( x a,b x b+,c ). x a,b x b+,c < 0 x a,b < x b+,c. 24

25 Therefore, it hods that x a,b < x a,c x a,b < x b+,c. Moreover, by considering its contraposition, x a,b x a,c x a,b x b+,c aso hods. In addition, noting that x a,b x b+,c ñ a,b ñ b+,c x a,b ñ a,b ñ b+,c x b+,c b c ñ b+,c n j x j ñ a,b n j x j ñ b+,c ñ b+,c j=a j=b+ b c c c n j x j + ñ b+,c n j x j ñ a,b n j x j + ñ b+,c n j x j j=a c n j x j ñ a,c j=a j=b+ c j=b+ n j x j j=b+ j=b+ c j=a n jx j ñ a,c c j=b+ n jx j ñ b+,c x a,c x b+,c, we get x a,b x b+,c x a,c x b+,c. Finay, by considering its contraposition, we aso get x a,b < x b+,c x a,c < x b+,c. Thus, it hods that (A.) and (A.2). Next, we prove (ii). Assume that (A.3) is true. Let s and t be integers with s < t i. When t = 2 and s =, from (A.3) it hods that x +w,w < x +w,w 2. Moreover, from (A.2) x +w,w < x +w,w 2 yieds x +w,w < x +w,w 2. Thus, we get x +ws,w s < x +ws,w t. Hence, if t = 2, (ii) is proved. Therefore, we consider the case of t 3. Since (A.3) is true, we obtain x,w < < x +wt 3,w t 2 < x +wt 2,w t < x +wt,w t. (A.6) Here, using (A.2) and the ast inequaity of (A.6), x +wt 2,w t < x +wt,w t, we get x +wt 2,w t < x +wt 2,w t. (A.7) Thus, if s = t, (ii) is proved. Finay, we consider the case of s < t, i.e., there exists q with q 2, such that s = t q. Here, we put v = t. Note that from (A.7) the inequaity x +wv,w v < x +wv,w t hods. In this setting, (ii) is proved as foows:. Combining x +wv,w v < x +wv,w t and the inequaity x +wv 2,w v < x +wv,w v in (A.6), we obtain x +wv 2,w v < x +wv,w t. 2. Again, by using (A.2), we get x +wv 2,w v < x +wv 2,w t. 3. Here, if v = s, (ii) is proved. On the other hand, if s < v, repacing v with v, and we go back to the step. Therefore, using above method we obtain (ii). 25

26 Finay, we prove (iii). Let i and j be integers with i < j, and et b be an integer with i b < j, i.e., i b j. We put s = b i +. Note that s j i. Reca that from (3.4), the sth row of the matrix D i,j is given ( ) n i,i+s ñ, n i+s,j. i,i+s ñ i+s,j Therefore, the sth eement of the vector D i,j x i,j can be expressed as ( ) n i,i+s ñ, n i+s,j x i,j i,i+s ñ i+s,j ( ) = n i,i+s ñ, n i+s,j (x i,i+s i,i+s ñ, x i+s,j ) i+s,j = n i,i+s x i,i+s ñ i,i+s n i+s,j x i+s,j ñ i+s,j = x i,i+s x i+s,j = x i,b x b+,j. Hence, if D i,j x i,j 0 j i, then we obtain x i,b x b+,j 0, i.e., x i,b x b+,j. On the other hand, if x i,b x b+,j, i.e., x i,b x b+,j 0 for any integer b with i b < j, then, we get D i,j x i,j 0 j i. Thus, (A.5) hods. Appendix B: Lemma B and its proof Lemma B. Let be an integer with 2, and et n,..., n R >0 and n = (n,..., n ). Let x = (x,..., x ) R, and et i be an integer with 2 i. In addition, et w,..., w i N, and et w < w 2 < < w i where w i =. Put w 0 = 0. Assume that η (x) = = η (x)w, η (x)w + = = η (x)w 2, η (x)w i + = = η (x)w i,. and η (x)w j = x +wj,w j, ( j i). (B.) Moreover, aso assume that x,w < x +w,w 2 < < x +wi,w i. (B.2) Then, the foowing two propositions hod: (i) Let s be an integer with < s i. If the inequaity D +wt,w t x +wt,w t 0 wt w t (B.3) 26

27 hods for any integer t with s t i, then, the foowing inequaity aso hods: D +ws 2,w s x +ws 2,w s 0 ws w s 2, (B.4) (ii) where we define 0 0 = 0. For any integer t with t i, it hods that D +wt,w t x +wt,w t 0 wt w t. (B.5) Proof. First, we prove (i). We woud ike to reca that, from (3.5) η (x)w s is given by η (x)w s = min v;v w s max x u,v. u;u w s (B.6) Here, assume that ( v > w s s.t. v = argmin v;v w s Note that the assumption (B.7) is equa to max u;u w s x u,v ). (B.7) min v;v w s max x u,v = u;u w s max x u,v u;u w. s (B.8) Then, from (B.6) and (B.8) we have Furthermore, noting that η (x)w s = max x u,v. u;u w s max u;u w s x u,v x +ws 2,v we aso get η (x)w s x +ws 2,v. (B.9) Incidentay, since v satisfies the inequaity v > w s, there exists a number t such that s t i and + w t v w t. Based on this, we consider the foowing two cases: Case : x +ws 2,w t < x +wt,v, Case 2 : x +ws 2,w t x +wt,v. It is cear that Case is the negation of Case 2. Next, we show that both Case and Case 2 are fase. In fact, if Case is true, i.e., the inequaity x +ws 2,w t < x +wt,v is true, from (A.2) we obtain x +ws 2,w t < x +ws 2,v. Thus, using this inequaity and (B.9) we get η (x)w s > x +ws 2,w t. (B.0) Reca that we assume the inequaity (B.2). Hence, from (A.4) it hods that x +ws 2,w s < x +ws 2,w t. (B.) 27

28 Therefore, combining (B.0) and (B.), we obtain η (x)w s > x +ws 2,w s. (B.2) However, from the assumption (B.), it hods that η (x)w s = x +ws 2,w s. This resut and (B.2) contradict. Hence, Case is fase. Next, we consider Case 2. Suppose that x +ws 2,w t x +wt,v is true. Then, from (A.) we have x +ws 2,v x +wt,v. Combining this inequaity and (B.9), we get η (x)w s x +wt,v. (B.3) Here, when v = w t, from (B.3) it hods that η (x)w s x +wt,w t. On the other hand, when + w t v < w t, from the assumption (B.3), it hods that D +wt,w t x +wt,w t 0 wt w t. Using this inequaity and (A.5) we obtain x +wt,v x +v,w t. Again, from (A.) it hods that x +wt,v x +wt,w t. Substituting this inequaity into (B.3) yieds η (x)w s x +wt,w t. Thus, in both cases it hods that η (x)w s x +wt,w t. (B.4) Here, reca that from the assumption (B.), the equaity η (x)w s = x +ws 2,w s hods. Therefore, combining this equaity and (B.4) it hods that x +ws 2,w s x +wt,w t. (B.5) Note that from the definitions of s and t, the inequaity s t hods. Thus, (B.5) and (B.2) contradict. Therefore, Case 2 is fase. Hence, both Case and Case 2 are fase. This impies that the assumption (B.7) is not true. Thus, we obtain in other words, it hods that ( ) argmin max x u,v v;v w s u;u w s = w s, η (x)w s = min v;v w s max x u,v = u;u w s max x u,ws. u;u w s (B.6) Therefore, from (B.6) it hods that η (x)w s x r,ws for any integer r with + w s 2 < r w s. Again, using η (x)w s = x +ws 2,w s we get x +ws 2,w s x r,ws. Moreover, from (A.) we have x +ws 2,r x r,ws. Here, we repace r with b. Then, b is the integer satisfying + w s 2 b < w s and it hods that 28

29 x +ws 2,b x b+,ws. Thus, from (A.5), this impies D +ws 2,w s x +ws 2,w s 0 ws w s 2. Consequenty, (B.4) is proved, This impies that (i) hods. Next, we prove (ii). Since we have aready proved the first proposition (i), in order to prove (ii), it is sufficient to prove that D +wi,w i x +wi,w i 0 wi w i. Here, we consider η (x)w i. Reca that from (3.5) η (x)w i is given by Noting that w i =, we get η (x)w i = min max x u,v. v;v w i u;u w i η (x)w i = min max x u,v = max x u,wi. v;v w i u;u w i u;u w i (B.7) Aso note that, from (B.) the equaity η (x)w i = x +wi,w i hods. Therefore, using this equaity and (B.7) we obtain x +wi,w i x r,wi for any integer r with + w i < r w i. Again, by using the same argument as in the proof of (i), we have D +wi,w i x +wi,w i 0 wi w i. Thus, combining this resut and (i), (ii) is proved. Appendix C: Lemma C and its proof Lemma C. Let be an integer with 2, and et n,..., n R >0, n = (n,..., n ), ξ,..., ξ R and > 0. Let x,..., x be independent random variabes, and for any integer s with s, et x s N(ξ s, /n s ). Put x = (x,..., x ). Then, the foowing four propositions hod: (i) R = i= w;w Wi η (A i(w)), η (A i(w)) η (A i (w )) =, ((i, w) (i, w )). (ii) For the set A (w) = A, it hods that x η (A (w)) D, x, 0. (C.) Moreover, for any integer i with 2 i and for any eement w = (w,..., w i ) Wi, it hods that x η (A i(w)) 0 t i, D +wt,w t+ x +wt,w t+ 0 ρt,w, 0 s i 2, x +ws,w s+ < x +ws+,w s+2, (C.2) where w 0 = 0 and, ρ t,w = w t+ w t. 29

30 (iii) For any integer i with i and for any eement w = (w,..., w i ) W i, it hods that x η (A i(w)) 0 t i, η (x) + w t = = η (x)w t+ = x +wt,w t+, where w 0 = 0. (iv) For any integer i with i, it hods that P ( x η (A i(w)) ) = P η (x) A i(w). w;w W i w;w W i Proof. First, we prove (i). From the definition of the function η ( ), we get η (R ) A. Hence, from (3.2) and (3.3) it is cear that (i) hods. Second, we prove (iv). Here, note that from (i) it hods that η (A i (w)) η (A i (w )) =. Thus, the events x η (A i (w)) and x η (A i (w )) are disjoint. Therefore, from the definition of the probabiity we obtain P(x η (A i(w))) = P x η (A i(w)). w;w W i w;w W i Furthermore, from the inverse image, we aso get x η (A i(w)) η (x) w;w W i w;w W i A i(w). Hence, (iv) is proved Next, we prove (ii). First, we prove the right-arrow in (C.). x η (A (w)), i.e., η (x) A (w). Then, from the definition of A (w), we get η (x) = η (x)2 = = η (x) ˆα (say). This impies that η (x) = ˆα. In addition, from the definition of the function η it hods that min x y 2 y A n = x η (x) 2 n = x ˆα 2 n. (C.3) Moreover, noting that A (w) A we have min x y 2 y A n min x y 2 y A (w) n = min x α 2 n. α R Here, note that the norm x α 2 n is a convex function with respect to (w.r.t.) α on R. Thus, there exists a unique point α min which maximizes x α 2 n w.r.t. α. Therefore, we obtain min x y 2 y A n x α min 2 n. (C.4) 30

31 Hence, combining (C.3) and (C.4), the inequaity x ˆα 2 n x α min 2 n hods. Therefore, from uniqueness of α min we obtain ˆα = α min. On the other hand, α min can be obtained by differentiating the function x α 2 n w.r.t. α as α min = x, because the function x α 2 n is the convex function. Thus, it hods that η (x) = η (x)2 = = η (x) = x,. (C.5) Here, reca that η (x)s is given by (3.5). Hence, η (x) can be written as η (x) = min v x,v. (C.6) Moreover, from (C.5) we get η (x) = x,. Therefore, combining this equaity and (C.6), it hods that x,v x, for any integer v with v <. Thus, from (A.) we obtain x,v x v+,. Hence, from (A.5) this impies that D, x, 0. Consequenty, the right-arrow in (C.) is proved. Next, we prove the eft-arrow in (C.). Let D, x, 0. Then, from (A.5) it hods that x,v x v+, for any integer v with v <. Again, from (A.) we have x,v x,. Hence, combining this resut and (C.6) we get η (x) = x,. On the other hand, from (3.5), η (x) can be expressed as η (x) = max u x u,. (C.7) Here, since the inequaity x,v x v+, hods, from (A.) we obtain x, x v+,. This resut and (C.7) yied η (x) = x,. Thus, it hods that η (x) = η (x). In addition, from the definition of η we have η (x) η (x). Therefore, combining this inequaity and the equaity η (x) = η (x), we get η (x) = = η (x). This impies that η (x) A (w), i.e., x η (A (w)). Hence, the eft-arrow in (C.) is proved. Therefore, (C.) is proved. Next, we prove (C.2). First, we show the right-arrow in (C.2). Let i be an integer with 2 i, and et w = (w,..., w i ) be an eement with w Wi. Here, we put w 0 = 0. Furthermore, assume that x η (A i (w)). Note that w i =. Then, since η (x) A i (w) the foowing equaities hod: η (x) = = η (x)w ˆδ, η (x)w + = = η (x)w 2 ˆδ 2, η (x)w i + = = η (x)w i ˆδ i,. (say). Moreover, the inequaity ˆδ < < ˆδ i aso hods. We put ˆδ = (ˆδ,..., ˆδ i ). Then, η (x) can be written by using ˆδ as η (x) = (ˆδ w w 0,..., ˆδ i w i w i ). From the definitions 3