A Note on the Applied Use of MDL Approximations

Transcription

1 A Note on the Applied Use of MDL Approxiations Daniel J. Navarro Departent of Psychology Ohio State University Abstract An applied proble is discussed in which two nested psychological odels of retention are copared using Miniu Description Length (MDL). The standard Fisher inforation approxiation (FIA) to the Noralized Maxiu Likelihood (NML) is calculated for these two odels, with the result that the full odel is assigned a saller coplexity, even for oderately large saples. A geoetric interpretation for this behavior is considered, along with its practical iplications. The Miniu Description Length (MDL) principle (Rissanen 1978; see also Grünwald 1998) has attracted interest in applied fields, because it allows coparisons between non-nested and isspecified odels without requiring restrictive assuptions. Of particular interest is the Noralized Maxiu Likelihood (NML) criterion, NML = p(x θ (X)) p(y θ (Y )) dy where θ (X) denotes the axiu likelihood estiate (MLE) for the data X. While the NML has any desirable properties (Rissanen 2001), a practical difficulty is that the noralization ter is difficult to evaluate. Consequently, an approach based on Fisher inforation (FIA) is often used in its place (e.g. Rissanen 1996). This criterion is given by, FIA = ln p(x θ (X)) + k ( ) N I(θ) +ln 2 ln dθ + o(1) 2π where k denotes the nuber of free paraeters, N denotes the saple size, and I(θ) denotes the expected Fisher inforation atrix of saple size one. The second and third ters are often referred to as a coplexity penalty. Under regularity conditions 1 (Rissanen 1996, p ), it can be shown that the FIA asyptotically converges to ln(nml). The practical advantage to the FIA is that it only requires integration over the paraeter space, and that the Fisher inforation atrix is soeties easier to find than the axiu likelihood. However, while the optiality of the NML criterion holds for all N, the FIA is only guaranteed asyptotically, which can be probleatic in soe applications. This note presents one such case. Since applied researchers are often forced by necessity to use asyptotic easures such as the FIA, it is useful to take note of circustances under which they are inappropriate, and give consideration to the reasons. 1 The regularity conditions, ost iportantly the asyptotic norality of the MLE, are satisfied for odels that constitute copact (i.e. closed and bounded) subsets of an exponential faily, such as those discussed here. 1

2 The Applied Proble The applied proble originates in the study of huan forgetting ( retention ; see Navarro, Pitt & Myung, in press). In a typical retention experient, participants are presented with a list of words, and asked to recall the later. Measureents at different ties after stiulus presentation produce a retention curve : The probability of accurate recall is initially high, but tends towards zero over tie. A retention odel takes the for p(c t) =f(t, θ) wherep(c t) denotes the probability of correct recall at tie t, andf(t, θ) is the hypothesized for of the retention curve, paraetrized by θ. Obviously, f(t, θ) ust lie on [0, 1] for all t. Additionally, t is noralized to lie between 0 and 1. The classic retention odel is the exponential (EX) odel, f(t, θ) = a exp( bt), where θ =(a, b) such that a [0, 1] denotes the initial retention probability, and b gives the decay rate. While definitive bounds on b are not easy to specify, experience with a large database suggests that [0, 100] is reasonable (see Rubin & Wenzel 1996 or Navarro et al., in press). A second retention odel is provided by Wickelgren s (1972) strength-resistance (SR) theory, which hypothesizes that f(t, θ) =a exp( bt w )whereθ =(a, b, w). Iportantly, w is constrained to lie on [0, 1], since w<0 results in an increasing retention function, and w>1resultsina faster-than-exponential decay, neither of which is consistent with the theory. Note that the EX odel is a special case of the SR odel, and by inspection, the NML denoinator ter is always saller for the EX odel than for the SR odel, irrespective of saple size. Fisher Inforation Matrices In any retention experient, continuous easureents are ipractical, so retention is easured at soe nuber of fixed tie intervals t 1,...,t. Thus the observed data ay be treated as N observations fro an -variate binoial distribution, where the i-th Bernoulli probability is described by f(t i,θ). A standard result (e.g., Schervish 1995, p ; Su, Myung & Pitt, in press) allows the uv-th eleent of the Fisher inforation atrix of saple size one for these odels to be written I uv (θ) = i=1 1 f(t i,θ)(1 f(t i,θ)) f(t i,θ) f(t i,θ). θ u θ v The partial derivatives of f(t i,θ)aresiple. FortheEXodel, f(t i,θ)/ a =exp( bt i )and f(t i,θ)/ b = at i exp( bt i ). For the SR odel, the partial derivatives are f(t i,θ)/ a = exp( bt w i ), f(t i,θ)/ b = at w i exp( btw i ), and f(t i,θ)/ w = abt w i ln(t i)exp( bt w i ). Substitution into the Fisher inforation forula yields [ (1/a) I(a, b) = i=1 y(i) i=1 t ] iy(i) i=1 t iy(i) a i=1 t2 i y(i) for the EX odel, where y(i) =1/(exp(bt i ) a). For the SR odel, [ (1/a) i=1 z(i) i=1 tw i z(i) b i=1 tw i (ln t i)z(i) I(a, b, w) = i=1 tw i z(i) a i=1 t2w i z(i) ab i=1 t2w i (ln t i )z(i) b i=1 tw i (ln t i)z(i) ab i=1 t2w i (ln t i )z(i) ab 2 i=1 t2w i (ln t i ) 2 z(i) where z(i) =1/(exp(bt w i ) a). ] Sall Saple Behavior The experiental design was assued to consist of 8 evenly spaced t i values, though the results are consistent across a range of designs. Without a closed for for the integral in the FIA, nuerical estiates were obtained using Monte Carlo ethods (e.g. Robert & Casella 1999). 2

3 EX odel SR odel Fisher inforation approxiation Saple size, N Figure 1: The coplexity assigned by the Fisher inforation approxiation (FIA) for the EX and SR odels (before taking logariths). Given the low diensionality of the integrals and the unnecessarily extensive sapling (10 8 saples were taken), even siple Monte Carlo ethods provide adequate results. The integral ter I(θ) dθ for the EX odel is approxiately , whereas for the SR odel the value is approxiately Substituting this into the FIA expression indicates that the SR odel has a higher estiated coplexity only when N 2096, as shown in Figure 1. This presents a substantial difficulty for the applied proble, since not one of the 77 data sets considered by Navarro et al. (in press) had a saple size this large. Therefore, every data set that could possibly be observed would be better fit by SR, yet the nested EX odel would be penalized ore severely for excess coplexity. It is worth considering the source of this proble. Following Myung, Balasubraanian and Pitt (2000) and Balasubraanian (1997), the coplexity ters in the FIA can be viewed as approxiations to (the logarith of) the ratio of two Rieannian volues. The first is the volue occupied by the odel in the space of probability distributions (assuing the Fisher inforation etric), given by V f = I(θ) dθ. The second volue is that of a little ellipsoid around θ, intended to quantify the size of a region of appreciable fit, given by V c =(2π/N) k/2 J(θ ) / I(θ ) where J(θ )= 1/N [ t 2 ln p(c t)/ θ u θ v is the observed inforation. In general, the observed inforation J(θ ) will differ fro the expected ]θ=θ inforation I(θ ), but for the current purposes it suffices to note that there are data sets for which they are very siilar, yielding J(θ ) I(θ ). As these are binoial odels, an exaple of such a data set would be one in which C i Nf(t i,θ ) for all i since the data are alost identical to their expected values at the MLE paraeters. Note that these are the data sets that are well fit by the odel, and are thus precisely those of ost interest to applied research. In such cases, V c (2π/N) k/2. When the observed inforation atrix closely approxiates the expected inforation atrix, it is reasonable to view the coplexity penalty for the FIA as approxiately equivalent to ln (V f /V c ). At N = 1 we observe that, for the EX odel, V c =2π< V f. The little ellipsoid is saller than the odel, as it should be. However, by expanding the EX odel into a third diension, one obtains the SR odel, for which V c =(2π) 3/2 > V f.thevolue of the three diensional ellipsoid is now larger than the volue of the entire odel. Taken together, these observations suggest that the extension of the odel along the new diension induced by the addition of the w paraeter is so tiny that the sall ellipsoid now protrudes 3

4 extensively, like a arble ebedded in a sheet of paper. Most of the region of good fit no longer lies within the region occupied by the odel. In short, until N gets very large, θ does not see to lie sufficiently within the odel anifold to support the Gaussian approxiations that underlie the FIA. As an aside, it does not appear that the proble can be entirely solved by incorporating the J(θ ) / I(θ ) ter. After all, by judicious choice of t and C, data sets could be chosen for which the observed inforation is precisely equal to the expected inforation even at sall saples. For instance, if t =(.25,.5,.75, 1) and N = 16, then the data set C =(8, 4, 2, 1) yields MLE paraeters a =1andb = ln 16 for both odels, and w = 1 for SR. In both cases, f(t i,θ )=C i /N for all i: The data take on their expected values at the MLE. Accordingly, V c =(2π/N) k/2 for these data. Even so, V f 0.07 for the SR odel, while V f 3.39 for the EX odel, indicating that the little ellipsoid is not well-located for the SR odel. Discussion Asyptotic approxiations are useful in practice only insofar as they can be relied on to give the right answers. Clearly, if V c < V f is not satisfied, the standard FIA expression is not necessarily reliable. In psychology, for instance, it is rare to find studies with N>2000, yet V c can still reain saller than V f even at this saple size. This well-locatedness requireent is entioned by Balasubraanian (1997) as a condition of his asyptotic expansions, but is not generally thought of as a regularity condition because it holds asyptotically (i.e. V c tends to zero as N tends to infinity). In finite saples, soe care is needed. While observing V c <V f does not guarantee that θ is well-located, observing that V c >V f does iply that it is not. It ay be possible to use this observation to forulate approxiations that perfor better in sall saples, though this question is left to ore theoretically-inclined researchers. In any case, it is clear that the source of the current difficulty does not lie with the MDL principle itself: The NML criterion does not suffer fro this proble at any N. It is only that the o(1) ter in the asyptotic criterion can be large enough to ake the approxiation ipractical for saller saples. While the point is soewhat obvious, it does iply the need to take care in the use of the criterion. To that end, it is worth ensuring that V c <V f before using the approxiation. Acknowledgeents This research was supported by NIH grant R01-MH57472, and by a grant fro the Office of Research at OSU. The author would like to thank Peter Grünwald, Michael Lee, In Jae Myung, and Mark Pitt for helpful suggestions, and two anonyous reviewers for coents that substantially iproved the anuscript. References Balasubraanian, V. (1997). Statistical inference, Occa s razor and statistical echanics on the space of probability distributions. Neural Coputation, 9, Grünwald, P. (1998). The Miniu Description Length Principle and Reasoning under Uncertainty. Ph.D. Thesis, ILLC Dissertation Series DS , CWI, the Netherlands. Myung, I. J., Balasubraanian, V., & Pitt, M. A. (2000). Counting probability distributions: Differential geoetry and odel selection. Proceedings of the National Acadey of Sciences USA, 97, Navarro, D. J., Pitt, M. A. & Myung, I. J. (in press). Assessing the distinguishability of odels and the inforativeness of data. Cognitive Psychology. 4

5 Rissanen, J. (1978). Modeling by the shortest data description. Autoatica, 14, Rissanen, J. (1996). Fisher inforation and stochastic coplexity. IEEE Transactions on Inforation Theory 42, Rissanen, J. (2001). Strong optiality of the noralized ML odels as universal codes and inforation in data. IEEE Transactions on Inforation Theory 47, Robert, C. P. & Casella, G. (1999). Monte Carlo Statistical Methods. Springer: New York. Rubin, D. C. & Wenzel, A. (1996). One hundred years of forgetting: A quantitative description of retention. Psychological Review, 103, Schervish, M. J. (1995). Theory of Statistics. New York: Springer. Su, Y., Myung, I. J. & Pitt, M. A. (in press). Miniu description length and cognitive odeling. In P. Grünwald, I. J. Myung, I. J.,& M. A. Pitt(Eds.) Advances in Miniu Description Length: Theory and Applications. MIT Press. Wickelgren, W. A. (1972). Trace resistance and decay of long-ter eory. Journal of Matheatical Psychology, 9,