The Use of MDL to Select among Computational Models of Cognition

Transcription

1 The Ue of DL to Select among Computational odel of Cognition In J. yung, ark A. Pitt & Shaobo Zhang Vijay Balaubramanian Department of Pychology David Rittenhoue Laboratorie Ohio State Univerity Univerity of Pennylvania Columbu, OH Philadelphia, PA {myung.1, Abtract How hould we decide among competing explanation of a cognitive proce given limited obervation? The problem of model election i at the heart of progre in cognitive cience. In thi paper, inimum Decription Length (DL) i introduced a a method for electing among computational model of cognition. We alo how that differential geometry provide an intuitive undertanding of what drive model election in DL. Finally, adequacy of DL i demontrated in two area of cognitive modeling. 1 odel Selection and odel Complexity The development and teting of computational model of cognitive proceing are a central focu in cognitive cience. A model embodie a olution to a problem whoe adequacy i evaluated by it ability to mimic behavior by capturing the regularitie underlying oberved data. Thi enterprie of model election i challenging becaue of the competing goal that mut be atified. Traditionally, computational model of cognition have been compared uing one of many goodne-of-fit meaure. However, ue of uch a meaure can reult in the choice of a model that over-fit the data, one that capture idioyncracie in the particular data et (i.e., noie) over and above the underlying regularitie of interet. Such model are conidered complex, in that the inherent flexibility in the model enable it to fit divere pattern of data. A a group, they can be characterized a having many parameter that are combined in a highly nonlinear fahion in the model equation. They do not aume a ingle tructure in the data. Rather, the model contain multiple tructure; each obtained by finely tuning the parameter value of the model, and thu can fit a wide range of data pattern. In contrat, imple model, frequently with few parameter, aume a pecific tructure in the data, which will manifet itelf a a narrow range of imilar data pattern. Only when one of thee pattern occur will the model fit the data well. The problem of over-fitting data due to model complexity ugget that the goal of model election hould intead be to elect the model that generalize bet to all data ample that arie from the ame underlying regularity, thu capturing only the regularity, not the noie. To achieve thi goal, the election method mut be enitive to the complexity of a model. There are at leat two independent dimenion of model complexity. They are the number of free parameter of a

2 model and it functional form, which refer to the way the parameter are combined in the model equation. For intance, it eem unlikely that two one-parameter model, y = θx and y = x θ, are equally complex in their ability to fit data. The two dimenion of model complexity (number of parameter and functional form) and their interplay can improve a model fit to the data, without necearily improving generalizability. The trademark of a good model election procedure, then, i it ability to atify two oppoing goal. A model mut be ufficiently complex to decribe the data ample accurately, but without over-fitting the data and thu loing generalizability. To achieve thi end, we need a theoretically well-jutified meaure of model complexity that take into account the number of parameter and the functional form of a model. In thi paper, we introduce inimum Decription Length (DL) a an appropriate method of electing among mathematical model of cognition. We alo how that DL ha an elegant geometric interpretation that provide a clear, intuitive undertanding of the meaning of complexity in DL. Finally, application example of DL are preented in two area of cognitive modeling. 1.1 inimum Decription Length The central thei of model election i the etimation of a model generalizability. One approach to aeing generalizability i the inimum Decription Length (DL) principle [1]. It provide a theoretically well-grounded meaure of complexity that i enitive to both dimenion of complexity and alo lend itelf to intuitive, geometric interpretation. DL wa developed within algorithmic coding theory to chooe the model that permit the greatet compreion of data. A model family f with parameter θ aign the likelihood f(y θ) to a given et of oberved data y. The full form of the DL meaure for uch a model family i given below. k N DL = ln f ( y θ $ ) + ln + ln d I θ det ( θ) 2 2π where $ θ i the parameter that maximize the likelihood, k i the number of parameter in the model, N i the ample ize and I(θ) i the Fiher information matrix. DL i the length in bit of the hortet poible code that decribe the data with the help of a model. In the context of cognitive modeling, the model that minimize DL uncover the greatet amount of regularity (i.e., knowledge) underlying the data and therefore hould be elected. The firt, maximized log likelihood term i the lack-of-fit meaure, and the econd and third term contitute the intrinic complexity of the model. In particular, the third term capture the effect of complexity due to functional form, reflected through I(θ). We will call the latter two term together the geometric complexity of the model, for reaon that will become clear in the remainder of thi paper. DL arie a a finite erie of term in an aymptotic expanion of the Bayeian poterior probability of a model given the data for a pecial form of the parameter prior denity [2]. Hence in eence, minimization of DL i equivalent to maximization of the Bayeian poterior probability. In thi paper we preent a geometric interpretation of DL, a well a Bayeian model election [3], that provide an elegant and intuitive framework for undertanding model complexity, a central concept in model election. 2 Differential Geometric Interpretation of DL From a geometric perpective, a parametric model family of probability ditribution form a Riemannian manifold embedded in the pace of all probability

3 ditribution [4]. Every ditribution i a point in thi pace, and the collection of point created by varying the parameter of the model give rie to a hyper-urface in which ``imilar'' ditribution are mapped to ``nearby'' point. The infiniteimal ditance between point eparated by the infiniteimal parameter difference dθ i i 2 k i j given by d = gij ( θ) dθ dθ where g ij (θ) i the Riemannian metric tenor. The i, j = 1 Fiher information, I ij (θ), i the natural metric on a manifold of ditribution in the context of tatitical inference [4]. We argue that the DL meaure of model fitne ha an attractive interpretation in uch a geometric context. The firt term in DL etimate the accuracy of the model ince the likelihood f ( y θ $ ) meaure the ability of the model to fit the oberved data. The econd and third term are uppoed to penalize model complexity; we will how that they have intereting geometric interpretation. Given the metric g ij = I ij on the pace of parameter, the infiniteimal volume element on the parameter manifold i k i dv = dθ det I( θ) dθ det I( θ). The Riemannian volume of the parameter i = 1 manifold i obtained by integrating dv over the pace of parameter: V = dv = dθ det I( θ) In other word, the third term in DL penalize model that occupy a large volume in the pace of ditribution. In fact, the volume meaure V i related to the number of ditinguihable probability ditribution indexed by the model. 1 Becaue of the way the model family i embedded in the pace of ditribution, two different parameter value can index very imilar ditribution. If complexity i related to volume occupied by model manifold, the meaure of volume hould count only different, or ditinguihable, ditribution, and not the artificial coordinate volume. It i hown in [2,5] that the volume V achieve thi goal. 2 While the third term in DL meaure the total volume of ditribution a model can decribe, the econd term relate to the number of model ditribution that lie cloe to the truth. To ee thi, taking a Bayeian perpective on model election i helpful. Uing Baye rule, the probability that the truth lie in the family f given the oberved data y can be written a: Pr( f y) = A( f, y) dθ w( θ) Pr( y θ ) Here w(θ) i the prior probability of the parameter θ, and A(f, y) = Pr(f)/Pr(y) i the ratio of the prior probabilitie of the family f and data y. Bayeian method aume that the latter are the ame for all model under conideration and analyze the ocalled Bayeian poterior P = dθ w( θ) Pr( y θ). f Lacking prior knowledge, w hould be choen to weight all ditinguihable ditribution in the family equally. Hence, w(θ) = 1/V. For large ample ize, the likelihood function f ( y θ $ ) localize under general condition to a multivariate 1 Roughly peaking, two probability ditribution are conidered inditinguihable if one i mitaken for the other even in the preence of an infinite amount of data. A careful definition of ditinguihability involve ue of the Kullback-Leibler ditance between two probability ditribution. For further detail, ee [3,4]. 2 Note that the parameter of the model are alway aumed to be cut off in a manner to enure that V i finite.

4 Gauian centered at the maximum likelihood parameter $ θ (ee [3,4] and citation therein). In thi limit, the integral for P f can be explicitly carried out. Performing the integral and taking a log given the reult ln P = ln f ( y θ $ ) + ln( V / C ) + O( 1 / N) where C = ( 2π / N) k / 2 h( θ $ ) f where h( θ $ ) i a data-dependent factor that goe to 1 for large N when the truth lie within f (ee [3,4] for detail). C i eentially the volume of an ellipoidal region around the Gauian peak at f ( y θ $ ) where the integrand of the Bayeian poterior make a ubtantial contribution. In effect, C meaure the number of ditinguihable ditribution within f that lie cloe to the truth. Uing the expreion for C and V, the DL election criterion can be written a DL = ln f ( y θ $ ) + ln( V / C ) + term ubleadingin N (The ubleading term include the contribution of h( θ $ ) ; ee [3,4] regarding it role in Bayeian inference.) The geometric meaning of the complexity penalty in DL now become clear; model which occupy a relatively large volume ditant from the truth are penalized. odel that contain a relatively large fraction of ditribution lying cloe to the truth are preferred. Therefore, we refer to the lat two term in DL a geometric complexity. It i alo illuminating to collect term in D a f y DL = ln ( θ $ ) ( V / C ) = ln " max " ( normalized imized likelihood ) Written thi way, DL elect the model that give the highet value of the maximum likelihood, per the relative ratio of ditinguihable ditribution (V /C ). From thi perpective, a better model i imply one with many ditinguihable ditribution cloe to the truth, but few ditinguihable ditribution overall. 3 Application Example Geometric complexity and DL contitute a powerful pair of model evaluation tool. When ued together in model teting, a deeper undertanding of the relationhip between model can be gained. The firt meaure enable one to ae the relative complexitie of the et of model under conideration. The econd build on the firt by uggeting which model i preferable given the data in hand. The following imulation demontrate the application of thee method in two area of cognitive modeling: information integration, and categorization. In each example, two competing model were fitted to artificial data et generated by each model. Of interet i the ability of a election method to recover the model that generated the data. DL i compared with two other election method, both of which conider the number of parameter only. They are the Akaike Information Criterion (AIC; [6]) and the Bayeian Information Criterion (BIC; [7]) defined a: AIC = 2ln f ( y θ $ ) + 2k; BIC = 2ln f ( y θ $ ) + k ln N. 3.1 Information Integration In a typical information integration experiment, a range of timuli are generated from a factorial manipulation of two or more timulu dimenion (e.g.,, viual and auditory) and then preented to participant for categorization a one of two or more poible repone alternative. Data are cored a the proportion of repone in one

5 category acro the variou combination of timulu dimenion. For thi comparion, we conider two model of information integration, the Fuzzy Logical odel of Perception (FLP; [8]) and the Linear Integration odel (LI; [9]). Each aume that the repone probability (p ij ) of one category, ay A, upon the preentation of a timulu of the pecific i and j feature dimenion in a two-factor information integration experiment take the following form: FLP: p ij θiλj LI ( )( ) ; : p θi + λ = ij = θ λ + 1 θ 1 λ 2 i j i j where è i and ë j (i=1,..,q 1 ; j=1,..,q 2: 0 < è i, ë j < 1) are parameter repreenting the correponding feature dimenion. The imulation reult are hown in Table 1. When the data were generated by FLP, regardle of the election method ued, FLP wa recovered 100% of the time. Thi wa true acro all election method and acro both ample ize, except for DL when ample ize wa 20. In thi cae, DL did not perform quite a well a the other election method. When the data were generated by LI, AIC or BIC fared much more poorly wherea DL recovered the correct model (LI) acro both ample ize. Specifically, under AIC or BIC, FLP wa elected over LI half of the time for N = 20 (51% v. 49%), though uch error were reduced for N = 150 (17% v 83%). Table 1: odel Recovery Rate for Two Information Integration odel Sample Size Selection ethod Data from: odel fitted: FLP j LI AIC/BIC FLP 100% 51% N = 20 LI 0% 49% DL FLP 89% 0% LI 11% 100% AIC/BIC FLP 100% 17% N = 150 LI 0% 83% DL FLP 100% 0% LI 0% 100% That FLP i elected over LI when a method uch a AIC wa ued, even when the data were generated by LI, ugget that FLP i more complex than LI. Thi obervation wa confirmed when the geometric complexity of each model wa calculated. The difference in geometric complexity between FLP and LI wa 8.74, meaning that for every ditinguihable ditribution for which LI can account, FLP can decribe about e ditinguihable ditribution. Obviouly, thi difference in complexity between the two model mut be due to the functional form becaue they have the ame number of parameter. 3.2 Categorization Two model of categorization were conidered in the preent demontration. They were the generalized context model (GC: [10]) and the prototype model (PRT: [11]). Each model aume that categorization repone follow a multinomial probability ditribution with p ij (probability of category C J repone given timulu X i ), which i given by

6 j C ij J GC: pij = ; PRT: pij = K k C K In the equation, ij i a imilarity meaure between multidimenional timuli X i and X j, ij i a imilarity meaure between timulu X i and the prototypic timulu X J of category C J. Similarity i meaured uing the inkowki ditance metric with the metric parameter r. The two model were fitted to data et generated by each model uing the ix-dimenional caling olution from Experiment 1 of [12] under the Euclidean ditance metric of r = 2. A hown in Table 2, under AIC or BIC, a relatively mall bia toward chooing GC wa found uing data generated from PRT when N = 20. When DL wa ued to chooe between the two model, there wa improvement over AIC in correcting the bia. In the larger ample ize condition, there wa no difference in model recovery rate between AIC and DL. Thi outcome contrat with that of the preceding example, in which DL wa generally uperior to the other election method when ample ize wa mallet. Table 2: odel Recovery Rate for Two Categorization odel Sample Size Selection ethod ik Data from: odel fitted: GC ij K ik PRT AIC/BIC GC 98% 15% N = 20 PRT 2% 85% DL GC 96% 7% PRT 4% 93% AIC/BIC GC 99% 1% N = 150 PRT 1% 99% DL GC 99% 1% PRT 1% 99% On the face of it, thee finding would ugget that DL i not much better than the other election method. After all, what ele could caue thi reult? The only circumtance in which uch an outcome i predicted under DL i when the functional form of the two model are imilar (recall that the model have the ame number of parameter), thu minimizing the differential contribution of functional form in the complexity term. Calculation of the geometric complexity of each model confirmed thi upicion. GC i indeed only lightly more complex than PRT, the difference being equal to 0.60, o GC can decribe about two ditribution (e ) for every ditribution PRT can decribe. Thee imulation reult together demontrate uefulne of DL and the geometric complexity meaure in teting model of cognition. DL enitivity to functional form wa clearly demontrated in it uperior model recovery rate, epecially when the complexitie of the model differed by a nontrivial amount. 4 Concluion odel election in cognitive cience can proceed far more confidently with a clear undertanding of why one model hould be preferred over another. A geometric

7 interpretation of DL help to achieve thi goal. The work carried out thu far indicate that DL, along with the geometric complexity meaure, hold coniderable promie in evaluating computational model of cognition. DL chooe the correct model mot of the time, and geometric complexity provide a meaure of how different the two model are in their capacity or power. Future work i directed toward extending thi approach to other clae of model, uch a connectionit network. Acknowledgment and Author Note.A.P. and I.J.. were upported by NIH Grant H V.B. wa upported by the Society of Fellow and the ilton Fund of Harvard Univerity, by NSF grant NSF-PHY and by the DOE grant DOE-FG02-95ER The preent work i baed in part on [5] and [13]. Reference [1] Rianen, J. (1996) Fiher information and tochatic complexity. IEEE Tranaction on Information Theory, 42, [2] Balaubramanian, V. (1997) Statitical inference, Occam razor and tatitical mechanic on the pace of probability ditribution. Neural Computation, 9, [3] ackay, D. J. C. (1992). Bayeian interpolation. Neural Computation, 4, [4] Amari, S. I. (1985) Differential Geometrical ethod in Statitic. Springer- Verlag. [5] yung, I. J., Balaubramanian, V., & Pitt,. A. (1999) Counting probability ditribution: Differential geometry and model election. Proceeding of the National Academy of Science USA, 97, [6] Akaike, H. (1973) Information theory and an extenion of the maximum likelihood principle, in B. N. Petrox and F. Caki, Second international ympoium on information theory, pp Akademiai Kiado, Budapet. [7] Schwarz, G. (1978) Etimating the dimenion of a model. The Annal of Statitic, 6, [8] Oden, G. C., & aaro, D. W. (1978) Integration of featural information in peech perception. Pychological Review, 85, [9] Anderon, N. H. (1981) Foundation of Information Integration Theory. Academic Pre. [10] Noofky, R.. (1986) Attention, imilarity and the identificationcategorization relationhip. Journal of Experimental Pychology: General, 115, [11] Reed, S. K. (1972) Pattern recognition and categorization. Cognitive Pychology, 3, [12] Shin, H. J., & Noofky, R.. (1992) Similarity-caling tudie of dot-patten claification and recognition. Journal of Experimental Pychology: General, 121, [13] Pitt,. A., yung, I. J., & Zhang, S. (2000). Toward a method of electing among computational model of cognition. Submitted for publication.