The Dual of the Maximum Likelihood Method

Transcription

1 Department of Agricltral and Resorce Economics University of California, Davis The Dal of the Maximm Likelihood Method by Qirino Paris Working Paper No by Qirino Paris All Rights Reserved. Readers may make verbatim copies of this docment for non-commercial prposes by any means, provided that this copyright notice appears on all sch copies. Giannini Fondation of Agricltral Economics

2 The Dal of the Maximm Likelihood Method Qirino Paris* University of California, Davis Smmary The Maximm Likelihood method estimates the parameter vales of a statistical model that maximize the corresponding likelihood fnction, given the sample information. This is the primal approach that, in this paper, is presented as a mathematical programming specification whose soltion reqires the formlation of a Lagrange problem. A remarkable reslt of this setp is that the Lagrange mltipliers associated with the linear statistical model (regarded as a set of constraints) is eqal to the vector of residals scaled by the variance of those residals. The novel contribtion of this paper consists in developing the dal model of the Maximm Likelihood method nder normality assmptions. This model minimizes a fnction of the variance of the error terms sbject to orthogonality conditions between the errors and the space of explanatory variables. An intitive interpretation of the dal problem appeals to basic elements of information theory and establishes that the dal maximizes the net vale of the sample information. This paper presents the dal ML model for a single regression and for a system of seemingly nrelated regressions. Key Words: Maximm likelihood, primal, dal, vale of sample information, SUR. JEL Classification: C1, C3 * Qirino Paris is a professor at the University of California, Davis. May

3 1. Introdction In general, to any problem stated in the form of a maximization (minimization) criterion, there corresponds a dal specification in the form of a minimization (maximization) goal. This strctre mst apply also to the Maximm Likelihood (ML) approach, one of the most powerfl and widely sed statistical methodologies. An intitive description of the Maximm Likelihood principle is fond in Kmenta (2011, pp ): Given a sample of observations abot random variables, the qestion is: To which poplation does the sample most likely belong? The answer can be fond by defining the likelihood fnction as the joint probability distribtion of the data, treated as a fnction of the nknown coefficients. The Maximm Likelihood Estimator (MLE) of the nknown coefficients consists of the vales of the coefficients that maximize the likelihood fnction. (Stock and Watson, 2011). That is, the maximm likelihood estimator selects the parameter vales that give the observed data sample the largest possible probability of having been drawn from a poplation defined by the estimated coefficients. In this paper, we concentrate on data samples drawn from normal poplations and deal with linear statistical models. The ML method, then, is to estimate the vales of the mean and variance of that normal poplation that will maximize the likelihood fnction given the sample information. The novel contribtion of the paper consists in developing the dal specification of the Maximm Likelihood method (nder normality) and in giving it an intitive interpretation. In section 2 we present the traditional Maximm Likelihood method in the form of a primal nonlinear programming model. This specification is a natral step toward the discovery of the dal strctre of the Maximm Likelihood method. It differs 2

4 from the traditional way to deal with the ML approach becase the estimates of parameters and errors are compted simltaneosly rather then seqentially. A remarkable reslt of this setp is that the Lagrange mltipliers associated with the linear statistical model (regarded as a set of constraints) is eqal to the vector of residals scaled by the variance of those residals. Section 3 derives and interprets the dal specification of the ML method. It trns ot that a crcial dal constraint is represented by a nontraditional definition of the variance! 2 as the inner prodct of the vector of sample information y and the vector of errors divided by the nmber of observations. The intitive interpretation of the dal problem appeals to the basic elements of information theory and establishes that the dal maximizes the net vale of the sample information (NVSI), as defined and illstrated in section 3. Section 4 presents a discssion of the dal specification of the seemingly nrelated regressions (SUR) model. Aside from the more complex strctre of the SUR specification and of the corresponding likelihood fnction, the process of deriving the dal problem is analogos to the single eqation methodology. Also the intitive interpretation of the SUR dal appeals to basic elements of information theory and establishes the maximization of the net vale of sample information. Althogh the nmerical soltion of the dal ML method is feasible (given a stable nonlinear programming software), the main goal of this paper is to present and interpret the dal specification of the ML method becase it completes the nderstanding of the role played by residals. It trns ot, in fact, that residals in the primal model assme the role of shadow prices in the dal model, as discssed in section 3. Section 5 sggests an implementable nmerical specification of the dal SUR model. 3

5 2. The Primal of the Maximm Likelihood Method (Normal Distribtion) Consider a linear statistical model y = X! + (1) where y is an (n!1) vector of sample data (observations), X is a (n! k) matrix of predetermined vales,! is an (k!1) vector of nknown parameters, and is an (n!1) vector of random errors that is assmed to be independently and normally distribted as N(0,! 2 I n ),! 2 > 0. The vector of observations y is known also as the sample information. Then, ML estimates of the parameters and errors can be obtained by maximizing the logarithm of the corresponding likelihood fnction with respect to (,!," 2 ) sbject to the constraints represented by model (1). We called this specification the primal ML method. Specifically, Primal: maxlog (,!," 2 y) = # n 2 log2$ # n 2 log" 2 # % 2" (2) 2 sbject to y = X! +. (3) Traditionally, the constraints (3) are sbstitted for the error vector in the likelihood fnction (2) and an nconstrained maximization calclation will follow. This algebraic maniplation, however, obscres the path toward a dal specification of the problem nder stdy. Therefore, we maintain the strctre of the ML model as in relations (2) and (3) and proceed to state the corresponding Lagrangean fnction by selecting an (n! 1) vector! of Lagrange mltipliers of constraints (3) to obtain L(,!," 2,#) = $ n 2 log2% $ n 2 log" 2 $ & 2" $ # & (y $ X! $ ). (4) 2 4

6 Following Kmenta (2011, pp ), the maximization of the Lagrangean fnction (4) reqires taking partial derivatives with respect to (,!," 2,#), setting them eqal to zero, in which case we signify that a soltion of the reslting eqations is an estimate of the corresponding parameters and errors:!l! = " # + $ (5) 2!L!" = X # $ (6)!L!" = # n 2 2" + $ 2 2" (7) 4!L = #y + X$ +. (8)!" First order necessary conditions (FONC) are given by eqating (5)-(8) to zero and signifying that the reslting soltion vales are ML estimates. We obtain ˆ! = û ˆ " 2 (9) X! ˆ" = 0 (10)! ˆ 2 = û" û / n (11) y = X ˆ! + û (12) where ˆ! 2 > 0. A first remarkable observation is that the Lagrange mltipliers ˆ! are defined in terms of the estimates of the primal variables and! 2. Indeed, the Lagrange mltipliers ˆ! are eqal to û p to a scalar ˆ! 2. This recognition will simplify the statement of the corresponding dal problem becase we can dispense with the symbol!. 5

7 In fact, eqation (10) can be restated eqivalently as the following orthogonality condition between the residals of model (1) and the space of predetermined variables X! ˆ" = X! û # ˆ = 0. (13) 2 The meaning of a Lagrange mltiplier (or dal variable) is that of a marginal sacrifice (shadow price) which is a measre of tightness of the corresponding constraint. Within the context of this paper, the optimal vales of the Lagrange mltipliers ˆ! are identically eqal to the residals û scaled by ˆ! 2. Hence, the symbol û takes on a doble role: as a vector of residals in the primal ML problem [(2)-(3)] and as a vector of shadow prices in the dal ML problem (defined in the next section). Primal ML Figre 1. Representation of the antilog of the primal objective fnction (2) for one normal variable with zero mean and! 2 = Figre 1 illstrates the familiar shape of the likelihood fnction as represented in the primal objective fnction (2) for one normal random variable. 6

8 Frthermore, note that the mltiplication of eqation (12) by the vector ˆ! (replaced by its eqivalent representation given in eqation (9)) and the se of the orthogonality condition (13) prodces ˆ"! y = ˆ"! X ˆ# + ˆ"! û ˆ! y $ ˆ = ˆ! û 2 $ ˆ 2 ˆ! y = ˆ! û. (14) This means that a ML estimate of! 2 can be obtained eqivalently as û! ˆ 2 = " û n = û " y n, (15) a comptationally sefl reslt. Relation (15), in fact, is of paramont importance for stating an operational specification of the dal ML model becase the presence of the vector of sample observations y in the estimate of! 2 is the only place where this sample information exercises its strctral effects in the dal model by limiting the range of the residals û, as clarified in the next section. 3. The Dal of the Maximm Likelihood Method The statement of the dal specification of the ML approach follows the general rles of dality theory in mathematical programming. The dal objective fnction to be minimized is the Lagrangean fnction stated in (4) with the appropriate simplifications allowed by its algebraic strctre and the information derived from FONCs (9) and (10). The dal constraints are all the FONCs that are different from the primal constraints. This leads to the following specification: Dal: min L * (,! 2 ) = " n 2 log2# " n 2 log! 2 " $ y! + $ 2 2! (16) 2 7

9 sbject to X! " = 0 (17) 2! 2 = " y n. (18) The soltion of the dal ML problem [(16)-(18)] by any reliable nonlinear programming software (e.g., GAMS) prodces estimates of (,!," 2 ) that are identical to those obtained from the primal ML problem. The dal ML estimate of the parameter vector! is obtained as a vector of Lagrange mltipliers associated to the orthogonality constraints (17). The last two terms of the dal objective fnction can be rearranged to express a qantity that has intitive appeal, that is! " y # + " 2 2# =! ( " y! " / 2). (19) 2 # 2 Using simple terminology of information theory, the expression in the nmerator of the second ratio of eqation (19) can be given the following interpretation. The linear statistical model (1) can be regarded as the decomposition of a message into a signal and noise, that is message = signal + noise (20) where message! y, signal! X", and noise!. Let s recall that the doble role of incldes also that of shadow price of the constraints defined by model (1). Hence, the inner prodct! y can be regarded as the gross vale of the sample information while the expression! / 2 can be regarded as a cost fnction of noise. In smmary, therefore, the relation (! y "! / 2 ) can be interpreted as the net vale of sample information (NVSI) that is maximized in the dal objective fnction. 8

10 It is well known that, nder the assmptions of model (1), least-sqares estimates of the parameters are also maximm likelihood estimates. Hence,! / 2 is the primal objective fnction (to be minimized) of the least-sqares method while (! y "! / 2 ) corresponds to the dal objective fnction (to be maximized) of the least-sqares approach. An optimal soltion of either the primal or the dal ML problems reslts in min LS =! 2 =! y "! 2 = max NVSI û ˆLS =! û 2 = û! û y "! û 2 = ˆNVSI where LS stands for Least Sqares. The role of shadow price taken on by û is jstified by the derivative of the least-sqares fnction with respect to the vector of sample information y, that is! ˆLS!y = û indicating that the change in the Least-Sqares objective fnction de to an infinitesimal change in the vector of constraints (sample information in model (1)) is the meaning of the Lagrange mltipliers that, in the LS case, are eqal to the vector of residals. Note also that, in view of constraint (18), relation (19) can be reformlated as! " y # + " 2 2# =!n + " 2 2# 2 a reslt that jstifies the se of the alternative definition of the variance! 2 (21) in constraint (18), that follows from eqation (14). That definition, in fact, provides the only effective presence of the sample information y in the dal model. 9

11 Dal ML Figre 2. Representation of the antilog of the dal objective fnction (16) for one normal variable with zero mean and! 2 = The Dal of the Seemingly Unrelated Regressions Model A seemingly nrelated regressions model consists of M linear eqations defined by a sample of T observations for each eqation; y m = X m! m + m (22) where m = 1,..., M, y m is a (T!1 ) vector of sample observations, X m is an (T! K m ) matrix of predetermined vales,! m is a ( K m!1) vector of regression coefficients, and m is a (T!1 ) vector of distrbances. We assme that the distrbances m are normally distribted with mean zero, E( mt ) = 0, t = 1,...,T and covariance matrix E( m! m ) = " mm I T where I T is an identity matrix of order ( T! T ). Frthermore, we assme that the distrbances in different eqations are contemporaneosly correlated, that is E( m! p ) = " mp I T, (m, p = 1,..., M ). 10

12 In order to simplify the notation and to facilitate the derivatives of the likelihood fnction we write model (22) in a more compact fashion, as in Schmidt (1976)! " y 1 y 2! y M # $ =! X 1 X 2! X " M! & # & $ & & " & % 1 0! 0 0 % 2! 0!!! 0 0! % M # ' ' ' ' $ ' +! 1 2! " M # $ (23) and, finally: Y = X! + U (24) M M where Y is a ( T! M ) matrix, X is a (T!" K m ) matrix,! is a (! K m " M ) matrix, and U is a (T! M ) matrix. Given the assmptions, the error covariance is an ( M! M ) matrix m m # %! = % % % $ % " 11 " 12! " 1M " 12 " 22! " 2 M!!! " 1M " 2 M! " MM & ( ( (. (25) ( '( The logarithm of the likelihood fnction of the above SUR specification constittes the primal objective fnction to be maximized sbject to the linear system of constraints stated in eqation (24). Analytically, the primal problem becomes Primal: maxlog!(u,!,") = # MT 2 log(2$ ) # T 2 log " # 1 2 TraceU"#1 U % (26) sbject to Y = X! + U. (27) The interpretation of the primal problem corresponds to the traditional maximm likelihood approach where the goal is to select those vales of the parameters (!," ) that maximize the probability of the sample Y. In the above nontraditional specification of the ML approach, the interpretation of the primal problem shold be integrated with the 11

13 statement that the goal is to select those vales of matrices ( U,!," ) that maximize the probability of the sample Y. The path toward the dal of problem [(26),(27)] begins with the statement of the Lagrangean fnction and the calclation of its first derivatives. We select a (T! M ) matrix of Lagrange mltipliers! associated with constraints (27) and write L(U,!,", #) = $ MT 2 log(2% ) $ T 2 log " $ 1 2 TraceU"$1 U & $ Trace #&(Y $ X! $ U) = $ MT 2 log(2% ) + T 2 log "$1 $ 1 2 Trace"$1 U& U $ Trace #&(Y $ X! $ U) (28) with first partial derivatives!l!u = "U#"1 + $ (29)!L!" = X # $ (30)!L!" = T #1 2 " # 1 2 U$ U (31)!L = #Y + X$ + U. (32)!" MLE of the nknown parameters are obtained by eqating relations (29)-(32) to zero and solving for (! U,!!,! ",! #)!! =!U!" #1 (33) 0 = X!"! (34)!U!! = " U! T (35) Y = X!! +! U. (36) 12

14 The dal specification takes advantage of the strctre of the Lagrangean fnction (28) and the FONCs (33), (34) and (36). Note that, sing (33) and (34), and mltiplying (36) by the matrix of Lagrange mltipliers!!, we obtain and, therefore,!"! Y =!"! X #! +!"! U! (37)!$ %1 U!! Y = $! %1 U!! U! Trace!! "1 U! # Y = Trace!! "1 U! #!U TraceY!! "1 U! # = Trace!U!! "1 U! # Trace!$ # Y = TraceY! "1 U! # Trace!$ #!U = Trace!U!! "1 U! # (38) Hence, the Lagrangean fnction (28) can be evalated and simplified as L(!U,!!) = " MT 2 log(2# ) " T 2 log!! " 1 2 Trace!U!! "1!U $ " Trace!% $ Y + Trace!% $ X &! + Trace!% $!U = " MT 2 log(2# ) " T 2 log!! " 1 2 Trace!U!! "1!U $ " TraceY!! "1!U $ + Trace!U!! "1!U $ = " MT 2 log(2# ) " T 2 log! " TraceY! "1!U $ Trace U!! "1!U $ This leads to the statement of the dal specification of the seemingly nrelated regressions model as Dal: min L(U,!) = " MT 2 log(2# ) " T 2 log! " TraceY!"1U $ TraceU!"1 U $ (39) sbject to 0 = X! U" #1 (40)! = U" U T. (41) The dal constraints (40) and (41) are a generalization of constraints (17) and (18). Note that, from eqation (37), the error covariance matrix cold have been stated, in analogy to constraint (18), as 13

15 ! = U" Y T. (42) This is, in fact, a more stable specification of the covariance matrix for the actal comptation of the ML estimates sing the dal approach. The estimate of the parameter matrix B is obtained as the matrix of Lagrange mltipliers of the orthogonality constraints (40). Since Lagrange mltipliers have the meaning of shadow prices and!! =!U!" #1, the last two terms of the dal objective fnction (39) can be interpreted as the net vale of the sample information (NVSI) (analogos to the single eqation example), with TraceY! "1 U # being the gross vale of sample information and TraceU! "1 U # / 2 the cost of noise: NVSI = TraceY! "1 U # " TraceU! "1 U # / 2. (43) This qantity is maximized in the dal problem de to the negative sign appearing in front of it. 5. Nmerical Implementation of the Dal SUR Model The main goal of this paper was aimed at the analysis and intitive nderstanding of the other (dal) side of the ML method. It did not consist in the proposal that ML estimates be obtained by solving the dal model nmerically. This goal, however, can easily be achieved with a stable nonlinear software sch as GAMS. At present time, traditional econometric packages cannot be sed for this task becase they do not deal explicitly with the error strctre of econometric models. The principal step toward a nmerical implementation of the dal ML SUR model is the acknowledgement of two facts: first, it is necessary to define explicitly the 14

16 determinant of the! matrix. Second, it is necessary to compte the inverse of! that mst be garanteed to be symmetric and positive definite. The determinant of a sqare matrix A can be compted by appealing to its Cholesky factorization A = LDL T where L is a nit lower trianglar matrix, L T is its transpose, and D is a diagonal matrix with positive elements on the main diagonal. It follows that the determinant of matrix A is the prodct of the diagonal elements of the D matrix. The inverse of matrix A is comptable by stating explicitly the relation AA!1 = I. Then, a proper coding for GAMS of the dal ML SUR problem (39)-(41) can be stated as minl(i) =! MT 2! T T M M T M M 2 log[det(")]! $ $ $ y tm# mp tp + $ $ $ tm # mp tp / 2 (44) t!1 m=1 p=1 t!1 m=1 p=1 sbject to M T "" x tk tp! pm = 0, m = 1,..., M;k = 1,... " K m (45) p=1 t=1 M k=1 T # &! = %" tm y tp / T (, $ t=1 ' m, p = 1,..., M (46)! = LDL T (47) M det(! ) = " d mm (48) m=1!! "1 = I (49) where! mp is the mth and pth element of the inverse matrix! "1. The ML estimate of the matrix of parameters! is obtained as a matrix of Lagrange mltipliers associated with the system of constraints (45). The Cholesky factorization of constraint (47) defines the symmetry and the positive definiteness of the covariance matrix!. The diagonal matrix D is reqired to have positive elements on its main diagonal. A very important aspect of solving highly nonlinear models by nmerical methods is a proper scaling of the data 15

17 series. As a rle of thmb, a desirable process is to scale all the data series arond a nit vale. The comptation of the covariance matrix as in constraint (46) is reqired by the fact that it is the only place where the sample information appears in the dal model. This is becase the objective fnction term involving the vector of sample information y corresponds to T M M!!! y tm " mp tp = TM. (50) t=1 m=1 p=1 Eqation (50) is nothing else that the mltivariate extension of eqation (21) for the single variable case. This dal ML formlation was solved for a SUR linear model (with intercepts) whose sample data inclded for dependent variables and five explanatory variables. 6. Conclsion A statistical model may be intitively regarded as the decomposition of a sample of messages into signal and noise. When the noise is distribted according to a normal density, the ML method maximizes the probability that the sample belong to a given normal poplation defined by the ML estimates of the model s parameters. All this is well known. This paper has analyzed and interpreted the dal of the ML method nder nivariate and mltivariate normal assmptions. It trns ot that the dal fnction is a convex fnction of noise. Hence, a convenient interpretation is that the dal of the ML method minimizes a cost fnction of noise. This cost fnction is defined by the sample variance (in the nivariate case) or the sample covariance matrix (in the mltivariate case) of noise. Eqivalently, the dal ML method maximizes the net vale of the sample information. The choice of an economic 16

18 terminology for interpreting the dal ML method is jstified by the doble role played by the symbols representing the residal terms of the estimated statistical model. These symbols may be interpreted as residal qantities in the primal specification and shadow prices in the dal model becase they correspond to the Lagrange mltipliers of the sample observations. Under the assmption of this paper, the least-sqares method leads to ML estimates of the model s parameters. Therefore, also in this case, the dal of the leastsqares approach corresponds to the maximization of the net vale of the sample information. The nmerical implementation of the dal ML method brings to the fore, by necessity, a neglected definition of the sample variance (in the nivariate case) and the sample covariance matrix (in the mltivariate case). In both cases, it is necessary to state the definition of the variance and covariances as the prodct of the sample information and the residals (noise), as revealed by eqations (15) and (42), becase it is the only place where the sample information appears in the dal model. The dal approach to ML estimates may or may not be of any practical importance over the traditional estimation procedres. This depends on the researcher s personal preferences for nmerical algorithms. It completes, however, or knowledge of what exists on the other side of the ML fence, a territory that has revealed interesting vistas on old statistical landscapes. References Benoit, Commandant. (1924), "Notes sr ne Méthode de Résoltion des Éqations Normales Provenant de l'application de la Méthode des Moindres Carrés a n Systéme d'éqations Linéaires en Nombre Inférier a Celi des Inconnes.- 17

19 Application de la Méthode a la Résoltion d'n Systéme Defini d'éqations Linéaires, (Procédé d Commandant Cholesky)." Bll. Géodésiqe 2(April, May, Jne 1924): Brooke, A., Kendrick, D. and Meeras, A. (1988), GAMS A User s Gide, Redwood City, CA: The Scientific Press. Kmenta, J. (2011), Elements of Econometrics Second Edition, Ann Arbor: The University of Michigan Press. Schmidt, P. (1976), Econometrics, New York: Marcel Dekker. Stock, J. H. and Watson, M. W. (2011), Introdction to Econometrics Third Edition, Boston: Addison-Wesley. 18