The Data-Constrained Generalized Maximum Entropy Estimator of the GLM: Asymptotic Theory and Inference

Transcription

1 Entropy 03, 5, ; do:0.3390/ Artcl OPE ACCESS ntropy ISS Th Data-Constrand Gnralzd Maxmum Entropy Estmator of th GLM: Asymptotc Thory and Infrnc Ron Mttlhammr,, cholos Scott Cardll and Thomas L. Marsh 3 3 Economc Scncs and Statstcs, Washngton Stat Unvrsty, Pullman, WA 9964, USA Salford Systms, San Dgo, CA 96, USA; E-Mal: scardll@gocougs.wsu.du Economc Scncs and IMPACT Cntr, Washngton Stat Unvrsty, Pullman, WA 9964, USA; E-Mal: tl_marsh@wsu.du Author to whom corrspondnc should b addrssd; E-Mal: mttlha@wsu.du; Tl.: ; Fax: Rcvd: 7 Aprl 03; n rvsd form: 3 Aprl 03 / Accptd: 7 May 03 / Publshd: 4 May 03 Abstract: Maxmum ntropy mthods of paramtr stmaton ar appalng bcaus thy mpos no addtonal structur on th data, othr than that xplctly assumd by th analyst. In ths papr w prov that th data constrand GME stmator of th gnral lnar modl s consstnt and asymptotcally normal. Th approach w ta n stablshng th asymptotc proprts concomtantly dntfs a nw computatonally ffcnt mthod for calculatng GME stmats. Formula ar dvlopd to comput asymptotc varancs and to prform Wald, llhood rato, and Lagrangan multplr statstcal tsts on modl paramtrs. Mont Carlo smulatons ar provdd to assss th prformanc of th GME stmator n both larg and small sampl stuatons. Furthrmor, w xtnd our rsults to maxmum cross-ntropy stmators and ndcat a varant of th GME stmator that s unbasd. Fnally, w dscuss th rlatonshp of GME stmators to Baysan stmators, pontng out th condtons undr whch an unbasd GME stmator would b ffcnt. Kywords: gnralzd maxmum ntropy; gnralzd maxmum cross-ntropy; asymptotc Thory; GME computaton; unbasd GME; GME as Baysan stmaton MSC 00 Cods: 6

2 Entropy 03, Introducton Informaton thortc stmators hav bn rcvng ncrasng attnton n th conomtrc-statstcs ltratur [ 7]. In othr wor, [3] proposd an nformaton thortc stmator basd on mnmzaton of th Kullbac-Lblr Informaton Crtron as an altrnatv to optmally-wghtd gnralzd mthod of momnts stmaton. Ths spcfc stmator handls waly dpndnt data gnratng mchansms and undr rasonabl rgulatory assumptons t s consstnt and asymptotcally normally dstrbutd. Subsquntly, [] proposd an nformaton thortc stmator basd on mnmzaton of th Crss-Rad dscrpancy statstc as an altrnatv approach to nfrnc n momnt condton modls. In [] dntfd a spcal cas of th Crss-Rad statstc th Kullbac-Lblr Informaton Crtron (.g., maxmum ntropy) as bng prfrrd ovr othr stmators (.g., mprcal llhood) bcaus of ts ffcncy and robustnss proprts. Spcal ssus of th ournal of Economtrcs (March 00) and Economtrc Rvws (May 008) wr dvotd to ths partcular topc of nformaton stmators. Hstorcally, nformaton thortc stmators hav bn motvatd n svral ways. Th Crss-Rad statstc drctly mnmzs an nformaton basd concpt of closnss btwn th stmatd and mprcal dstrbuton []. Altrnatvly, th maxmum ntropy prncpl s basd on an axomatc approach that dfns a unqu objctv functon to masur uncrtanty of a collcton of vnts [8 0]. Intrst n maxmum ntropy stmators stms from th prospct to rcovr and procss nformaton whn th undrlyng samplng modl s ncompltly or ncorrctly nown and th data ar lmtd, partal, or ncomplt [0]. To dat th prncpl of maxmum ntropy has bn appld n an abundanc of crcumstancs, ncludng n th flds of conomtrcs and statstcs [ 7], conomc thory and applcatons [8 4], accountng and fnanc [5 7], and rsourcs and agrcultural conomcs [8 3]. Morovr, wdly usd conomtrc softwar pacags ar now ncorporatng procdurs to calculat maxmum ntropy stmators n thr latst rlass (.g., SAS, SHAZAM, and GAUSSX). In most cass, rgorous nvstgaton of small and larg sampl proprts of nformaton thortc stmators hav laggd far bhnd mprcal applcatons [3]. Excptons nclud [ 3] who xamnd nformaton thortc altrnatvs to gnralzd mthod of momnts stmaton; [4] who drvd th statstcal proprts of th gnralzd maxmum ntropy stmator n th contxt of modlng multnomal rspons data; and, [0] who provdd asymptotc proprts for th momnt-constrand gnralzd maxmum ntropy (GME) stmator for th gnral lnar modl (showng t s asymptotcally quvalnt to ordnary last squars). An altrnatv nformaton thortc stmator of th gnral lnar modl (GLM), yt to b rgorously nvstgatd, but that has arsn n mprcal applcatons (.g., [4]), s th purly data-constrand formulaton of th gnralzd maxmum ntropy stmator [0]. In a purly data-constrand formulaton th rgrsson modl tslf, as opposd to momnt condtons of t, rprsnts th constranng functon to th ntropy objctv functon. In th maxmum ntropy framwor, unl ordnary last squar or maxmum llhood stmators of th GLM, momnt constrants ar not ncssary to unquly dntfy paramtr stmats. Morovr, thr xsts dstnct dffrncs btwn th data and momnt constrand vrsons of th GME for th GLM. For [0] hav shown th data-constrand GME stmator to b man squar rror supror to th momnt-constrand GME stmator of th GLM n slctd Mont Carlo xprmnts. Our papr contrbuts to th conomtrc ltratur n svral ways. Frst, rgularty condtons ar dntfd that provd a sold foundaton from whch to dvlop statstcal proprts of th data constrand GME stmator of th GLM and hypothss tsts on modl paramtrs. Gvn th rgularty

3 Entropy 03, condtons, w dfn a condtonal maxmum ntropy functon to rgorously prov consstncy and asymptotc normalty. As dmonstratd n ths papr th data-constrand GME stmator s not asymptotcally quvalnt to th momnt-constrand GME stmator or ordnary last squars stmator. Howvr, th GME stmator s shown to b narly asymptotcally ffcnt. Morovr, w drv formula to comput th asymptotc varanc of th proposd stmator. Ths allows us to dfn classcal Wald, Llhood Rato, and Lagrang Multplr tsts for tstng hypothss about modl paramtrs. Scond, thortcal xtnsons to unbasd, cross ntropy, and Baysan stmaton ar also dntfd. Furthr, w dmonstrat that th GME spcfcaton can b xtndd from fnt-dscrt paramtr and rror spacs to nfnt-contnuous paramtr and rror spacs. Altrnatv formulatons of th data constrand GME stmator of th GLM undr slctd rgularty condtons, and th mplcatons to proprts of th stmator, ar also dscussd. Thrd, to complmnt th thortcal rsults, Mont Carlo xprmnts ar usd n comparng th prformanc of th data-constrand GME stmats to last squars stmats for small and mdum sz sampls. Th prformanc of th GME stmator s tstd rlatv to slctd dstrbutons of th rrors, to th usr suppld supports of th paramtrs and rrors, and to ts robustnss to modl msspcfcaton. Mont Carlo xprmnts ar also prformd to xamn th sz and powr of th Wald, Llhood Rato, and Lagrang Multplr tst statstcs. Fourth, nsght nto computatonal ffcncy and gudlns for sttng boundars of paramtrs and rror support spacs ar dscussd. Th condtonal maxmum ntropy formulaton utlzd n proof of asymptotc proprts provds a bass for nw computatonally ffcnt mthod of calculatng GME stmats. Th approach nvolvs a nonlnar sarch ovr a K-vctor of coffcnt paramtrs, whch s much mor ffcnt than numrcal approachs proposd lswhr n th ltratur. Fnally, practcal gudlns for sttng boundars of paramtrs and rror support spacs ar analyzd and dscussd.. Th Data-Constrand GME Formulaton Lt Y X rprsnt th gnral lnar modl wth Y bng an dpndnt varabl vctor, X bng a fxd K matrx of xplanatory varabls, β bng a K vctor of paramtrs, and ε bng an vctor of dsturbanc trms (All of our rsults can b xtndd to stochastc X. For xampl, f X s d wth Var( X ), a postv dfnt matrx, thn th asymptotc proprts ar dntcal to thos dvlopd blow). Th GME rul for dfnng th stmator of th unnown β n th gnral lnar modl formulaton s gvn by ˆ Zpˆ wth pˆ ( pˆ ˆ,, p K ) drvd from th followng constrand maxmum ntropy problm: p ( p,, p K ) K Max p ln( p) w ln( w) p, w :, subjct to: Y XZp Vw p w p [0], w [0],,.

4 Entropy 03, In th prcdng formulaton, th matrcs Z and V ar K KM and matrcs of support ponts for th β and ε vctors, rspctvly, as: z 0 0 v z 0 0 v 0 Z and V, 0 0 z 0 0 v K whr z ( z,, zm) s a M vctor such that z z zm and ( z, zm),, K, and smlarly v ( v,, v) s a vctor such that v v v and ( v, v),, (n thr orgnal formulaton, [0] rqurd to b contand n a fxd ntrval wth arbtrarly hgh probablty. Hr w assum such an vnt occurs wth probablty). Th M p vctors and th w vctors ar wght vctors havng nonngatv lmnts that sum to unty and ar usd to rprsnt th β and ε vctors as Zp, for p ( p,, p K ), and Vw, for w ( w,, w ). Th basc prncpl undrlyng th stmator ˆ Zpˆ for β s to choos an stmat that contans only th nformaton avalabl. In ths way th maxmum ntropy stmator s not constrand by any xtranous assumptons. Th nformaton usd s th obsrvd nformaton contand n th data, th nformaton contand n th constrants on th admssbl valus of β, and th nformaton nhrnt n th structur of th modl, ncludng th choc of th supports for th s. In ffct, th nformaton st usd n stmaton s shrun to th boundary of th obsrvd data and th paramtr constrant nformaton. Bcaus th objctv functon valu ncrass as th wghts n p and w ar mor unformly dstrbutd, any dvaton from unformty rprsnts th ffct of th data constrants on th wghtng of th support ponts usd for rprsntng β and. Ths fact also motvats th ntrprtaton of th GME as a shrnag-typ stmator that n th absnc of constrants on β wll shrn ˆ to th cntrs of th supports dfnd n th spcfcaton of Z. W nxt stablsh consstncy and asymptotc normalty rsults for th GME stmator undr gnral rgularty condtons on th spcfcaton of th stmaton problm. 3. Consstncy and Asymptotc ormalty of th GME Estmator Rgularty Condtons. To stablsh asymptotc rsults for th GME stmator, w utlz th followng rgularty condtons for th problm of stmatng β n Y X. R. Th 's ar d wth c c for som δ > 0 and larg nough fnt postv c c. R. Th pdf of, f ( ), s symmtrc around 0 wth varanc. R3. ( L, H), for fnt L and H,,, K. R4. X has full column ran. R5. ( XX ) s O() and th smallst gnvalu of ( XX ) for som > 0, and, whr s som postv ntgr. ( ) R6. XX, a fnt postv dfnt symmtrc matrx. ot that condton R stats that th support of s contand n th ntror of som larg nough closd fnt ntrval [ c, c ]. Condton R3 stats that th tru valu of paramtr can b nclosd

5 Entropy 03, wthn som opn ntrval ( L, H ). Th condtons R4-R6 on X ar famlar analogus to typcal assumptons mad n th last squars contxt for stablshng asymptotc proprts of th last squars stmator of β. W utlz condton R6 to smplfy th dmonstraton of asymptotc normalty, but th rsult can b stablshd undr war condtons, as alludd to n th proof. Fnally, our proof of th asymptotc rsults wll utlz symmtry of th dsturbanc dstrbuton, whch s th contnt of condton R. Rformulatd GME Rul. Th asymptotc rsults ar drvd wthn th contxt of th followng rprsntaton of th GME modl, rprsntd n scalar notaton to facltat xposton of th proof. Th GME rprsntaton dscrbd blow s compltly consstnt wth th formulaton n Scton undr th condton that th support ponts rprsntd by th vctor v ar chosn to b symmtrcally dsprsd around 0. W us th sam vctor of support ponts for ach of th 's, consstnt wth th d natur of th dsturbancs, and so hncforth v rfrs to th common th scalar support pont n th dvlopmnt blow. Th rprsntaton s also mor gnral than th rprsntaton n Scton II n th sns that dffrnt numbrs of support ponts can b usd for th rprsntaton of dffrnt paramtrs. Th constrand maxmum ntropy problm s as follows: subjct to: K Max p ln( p ) w ln( w ) b, p, w C. C3. c v v v c C. z p b, L z z z H,,, K vw y X b( b),,, C4. v v (thus for odd v 0) C5. C6. p,,, K w,,, As wll bcom apparnt, th nonngatvty rstrctons on p and () w ar nhrntly nforcd by th structur of th optmzaton problm tslf, and thus nd not b xplctly ncorporatd nto th constrant st. Asymptotc Proprts. Th followng thorm stablshs th consstncy and asymptotc normalty of th GME stmator of β n th GLM. Thorm. Undr rgularty condtons R-R5, th GME stmator ˆ Zpˆ s a consstnt stmator of β. Wth th addton of rgularty condton R6, th GME stmator s asymptotcally normally dstrbutd as for approprat dfntons of,, and. a ˆ ~,

6 Entropy 03, 5 76 Proof. Dfn th maxmzd ntropy functon, condtonal on b pwb, : ( C) ( C6), as: K F( ) Max p ln( p ) w ln( w ) () Th optmal valu of w ( w,, w ) n th condtonally-maxmzd ntropy functon s gvn by: w( ) argmax w ln( w ), w: C6, vw ( ) whch s th maxmzng soluton to th Lagrangan: Th optmal valu of L w ln( w ) w v w ( ). w w w s thn: ( ( )) v w( ( )) w( ( )),,,, ( ( )) vm whr ( ( )) s th optmal valu of th Lagrangan multplr undr th condton b, and v w ( ). It follows from th symmtry of th v s around zro that: vm m m vw( ( ( ))) vw( ( ( ))) (4) Smlarly, th optmal valu of p ( p,, p ) n th condtonally-maxmzd ntropy functon s gvn by: p ( ) arg max p ln( p ), p: C5, zp whch s th maxmzng soluton to th Lagrangan: L p p p z p. p p ln( ) (3) Th optmal valu of p s thn: ( ) z p( ),,, K, ( ) zm m whr ( ) s th optmal valu of th Lagrangan multplr undr th condton b. Substtutng th optmal solutons for th p s and w s nto () obtans th condtonal maxmum valu functon: (5)

7 Entropy 03, 5 76 Dfn th gradnt vctor of K ( ) z m F( ) ( ) ln m ( ( )) vm ( ( )) ( ) ln. m F( ) F( ) as G( ) so that: F() G () ( ) ( ()) X,,, K, and thus G( ) ( ) X ( ( )), whr ( ) and (()) ar K and vctors of Lagrangan multplrs. It follows that th Hssan matrx of F( ) s gvn by: F( ) G( ) H ( ) ( ) 0 0 ( ) 0 0 (()) X. K( ) 0 0 Rgardng th functonal form of th drvatvs of th Lagrangan multplrs apparng n th dfnton of ( ) H, t follows from (C) that: so that from (3): w( ( ( ))) ( ( )), ( ( )) ( ) K Thn, from (C) ( ) X ( ( )) w ( ( ( ))) ( ) (), and thus:. H X X ( ) for. w ( ( ( ))) ( ) Also, basd on (C): ( ) zp, so that:

8 Entropy 03, H X. ( ) w ( ( ( ))) ( ) z p Bcaus th dnomnators of th trms n th dfnton of th H s ar postv valud, t follows that H ( ) s a ngatv dfnt matrx, bcaus XX s postv dfnt. ow consdr th cas whr, so that: ar d wth man zro, and thus: ( ) y X w ( ( ( ))),,, m ( ( )) ( ( )) m ar d wth man zro. Bcaus s boundd n th ntror of [, ], th rang of ( ( )) ( ) s boundd as wll. In addton, ( ( )) s symmtrcally dstrbutd around zro bcaus th s ar so dstrbutd, and, from (4): ( ( )) ( ( )) ( ( )) m ( ( )) m (6) m m It follows that E( ( ( ))) 0, th ( ) ( ( )) s ar d, and ( ) has fnt varanc, say Var( ( )). Thn, usng a multvarat vrson of Lapounov's cntral lmt thorm, and gvn condton R6 (asymptotc normalty can b stablshd wthout rgularty condton R6. In fact, th bounddnss proprts on th X-matrx statd n R5 would b suffcnt. S [33] for a rlatd proof undr th war rgularty condtons). 3.. Consstncy G X d ( ) ( ( ) ( ( ))) ([0], ) For any τ, rprsnt th condtonal maxmum valu functon, F( ), by a scond ordr Taylor srs around β as: F( ) F( ) G( ) ( ) ( ) H( )( ) (7) whr ls btwn τ and β. Th valu of th quadratc trm n th xpanson can b boundd by: ( ) H( )( ) ( H( )) s (8) whr s ( H( )) dnots th smallst gnvalu of H( ) and a a [34]. Th K

9 Entropy 03, smallst gnvalu xhbts a postv lowr bound gvn by ( ) s XX C whatvr th valu of. Th valu of th lnar trm n th xpanson s boundd n probablty; that s, 0 and for ( ), thr xsts a fnt A( ) such that: bcaus P G( ) ( ) A( ), (9) d G( ) ([0], ). It follows from Equatons (7) (9) that, for all 0, P( Max ( F( )) F( )) as. Thus ˆ p arg max( F( )), and th GME stmator : of β s consstnt. 3.. Asymptotc ormalty Expand G(b) n a Taylor srs around β, whr ˆ arg max F( ) s th GME stmator of β, to obtan: ˆ G( ) G( ) H( )( ˆ ) (0) whr s btwn ˆ and β. In gnral, dffrnt ponts wll b rqurd to rprsnt th dffrnt coordnat functons n G( ˆ ). At th optmum, G( ˆ ) [0] and ˆ s a consstnt stmator of β; p thrfor, and: ˆ d ( ) H( ) G( ), whr d dnots quvalnc of lmtng dstrbutons. Usng ( ), not that: w( ( )) X X H ( ) 0 P, w( ( )) whr w ( ( )),,, ar d. It follows from R6 that p H ( ) wth d E. Rcallng that G( ) ([0], ), Slutsy's Thorm [34] mpls that: ˆ d ( ) ([0], ) ot that holdng th support of constant, on can rduc th ntrval (c, c ). As 0, th asymptotc varanc of ( ˆ ) may tnd to zro, but cannot grow wthout bound. For xampl, f at 0, 0 such that P( ) ( c ), all ( c, c ) ( P( ) ( c ) all ( c, c )), thn lm 0. 0

10 Entropy 03, Also not that, for larg sampls, th paramtrs rlanc on th supports vanshs. In contrast, th supports on th rrors nflunc th computd covaranc matrx. Fnally, for non-homognous rrors, th covaranc matrx stmator could b adjustd followng a standard Wht s covaranc corrcton Cross-Entropy Extnsons To xtnd th prvous asymptotc rsults to th cas of cross-ntropy maxmzaton [0], frst suppos that z z and/or for som. Lt z,,, and,,, dnot th dstnct valus among th z s and s, rspctvly, and lt a and dnot th rspctv multplcts of th valus z and. From Equatons (3) and (5), w ( ( ( ))) wm( ( ( ))) f m and p ( ) pm( ) f z zm. Thus, th maxmzaton problm gvn by Equaton () and Condtons C-C6 s quvalnt to: K p w max pln w ln bpw,, a wth obvous changs bng mad to C-C6. Th only altratons ndd to th prcdng proof ar: () w ( ( ( ))) w ( ( ( ))),,,, and ( ( )) ( ( )) m m m ( ) z a p ( ),,, K. (3) m a ( ) zm m () (3) Mor gnrally, th sam rprsntaton ()-(3) appls for any a 0, 0. Furthrmor, Equatons () and (3) ar homognous of dgr zro n (,, ) and ( a,, a ), rspctvly. Thus, wthout loss of gnralty, th normalzaton condtons: and can b mposd. Usng Equatons (), (), and (3), w hav charactrzd th maxmum cross ntropy soluton. Upon substtuton of Equatons () (3) n th approprat argumnts, all rsults, ncludng th rsults n th nxt scton on statstcal tstng, apply to th maxmum cross-ntropy paradgm. 4. Statstcal Tsts Th GME stmator ˆ Zpˆ s consstnt and asymptotcally normally dstrbutd. Thrfor, asymptotcally vald normal and tst statstcs can b usd to tst hypothss about β. For mprcal mplmntaton of such tsts a consstnt stmat of th asymptotc covaranc matrx of ˆ wll b rqurd. An stmat of s straghtforwardly obtand by calculatng ˆ ˆ M( ) ( XX ) M( ), whr: a

11 Entropy 03, M ( ˆ ) X X ˆ ˆ w( ( ( ))) ( ) An stmat of th varanc,, of th s can b constructd as th asymptotc covaranc matrx of ˆ can b stmatd by: ˆ ˆ ˆ ˆ Var( ) ˆ ( ) M ( ) ( X X ) M ( ).. ˆ ( ) ( ( )). Thn ˆ ˆ Altrnatvly, ξ can b stmatd by: ˆ ˆ ( ) w ( ( ( ˆ ))) ( ˆ ). Thn: ˆ ˆ ˆ ( ) Var( ) ( X X ). ˆ ( ˆ ) 4.. Asymptotcally ormal Tsts Bcaus T Z ˆ 0 s asymptotcally (0,) undr th null hypothss Var ( ˆ ) statstc T z can b usd to tst hypothss about th valus of th s. H, th 0 0 : 4.. Wald Tsts Wald tsts of lnar rstrctons on th lmnts of β can b xprssd n th usual form. Lt H0 : R r b th null hypothss to b tstd, whr R s a L K matrx wth ran ( R) L K. Thn ˆ d ( R r) (0, R( ) R ). Thus, th Wald tst statstc has a lmtng dstrbuton as: d L T ( R ˆ r)( R( Var( ˆ )) R) ( R ˆ r) W undr th null hypothss H 0. Smlarly, for nonlnar rstrctons g( ) [0], whr g( ) s a g ( b) contnuously dffrntabl L-dmnsonal vctor functon wth q b and ran ( q( )) L K, t follows that: ˆ ˆ ˆ ˆ ˆ d T g( )( q( ) Var( ) q( )) g( ) Llhood Rato Tsts W To stablsh a psudo-llhood rato tst of functonal rstrctons on th β vctor, frst not that: ˆ d F( ) F( ) G( ) G( ), L

12 Entropy 03, whch follows from Equatons (7) and (0) and th fact that H p ( ). Thus: ˆ ( ˆ) ˆ ˆ ˆ ( ) d F( ) F( ) K. ow lt ˆ R b a rstrctd GME stmator of β. Thus, ˆ arg max( Fb ( )) for a lnar null hypothss H ˆ 0 : R r, or R arg max( F( b)) for a gnral null hypothss H 0 : g( ) [0]. As bfor, lt L = ran ( R) Thn: undr th null hypothss. Lagrang Multplr Tsts bg : ( b) 0 R brb : r K for a lnar hypothss or L = ran ( q( )) K for a gnral hypothss. ˆ ( ˆ ) ˆ ˆ ˆ ˆ ( ) d F( ) F( R) L Dfn R, r, g,, and ˆ R as abov. Thn a Lagrangan multplr tst of functonal rstrctons on β can b basd on th fact that: ˆ ˆ d G( )( ) ( ) R XX G R L ˆ ( ˆ ) undr th null hypothss. 5. Mont Carlo Smulatons R A Mont Carlo xprmnt was conductd to xplor th samplng bhavor of tst stuatons basd on th Gnralzd Maxmum Entropy Estmator. Th data wr gnratd basd on a lnar modl contanng an ntrcpt trm, a dchotomous xplanatory varabl, and two contnuously masurd xplanatory varabls. Th rsults of th Mont Carlo xprmnt also add addtonal prspctv to smulaton rsults rlatng th bas and man squar rror to th maxmum ntropy stmator gnratd prvously by [0]. Th lnar modl Y = Xβ + s spcfd as Y X X 3X 3, whr X s a dscrt random varabl such that (, ) X ~d Brnoull(.5), obsrvatons on th par of xplanatory random varabls, that ar cnsord at th man ±3 X X 3 ar gnratd from d outcoms of standard dvatons, and outcoms of th dsturbanc trm ar dfnd as U 6, whr d ~ U Unform(0,). Th support ponts for th dsturbanc trms wr spcfd as V = ( 0, 0, 0)' (rcall C and C3) for all xprmnts. Thr dffrnt sts of support ponts wr spcfd for th β-vctor, gvn by:

13 Entropy 03, Z Z I II , , and: Z III (rcall C). Th support ponts n Z I wr chosn to b most favorabl to th GME stmator, whr th lmnts of th tru β-vctor ar locatd n th cntr of thr rspctv supports and th wdths of th supports ar rlatvly narrow. Th supports rprsntd by Z II ar tltd to th lft of β and β and to th rght of β 3 and β 4 by unt, wth th wdths of th supports bng th sam as thr countrparts n Z I. Th last st of supports rprsntd by Z III ar wdr and ffctvly dfn an uppr bound of 0 on th absolut valus of ach of th lmnts of β. To xplor th rspctv szs of th varous tsts prsntd n Scton IV, th hypothss H0: c was tstd usng th T Z tst, and th hypothss H 0 : c, 3 d was tstd usng th Wald, psudo-llhood, and Lagrang Multplr tsts, wth c and d st qual to th tru valus of β and β 3,.., c = and d =. Crtcal valus of th tsts wr basd on thr rspctv asymptotc dstrbutons and a 0.05 lvl of sgnfcanc. An obsrvaton on th powr of th rspctv tsts was obtand by prformng a tst of sgnfcanc whrby c = d = 0 n th prcdng hypothss. All scnaros wr analyzd usng 0,000 Mont Carlo rpttons, and sampl szs of n = 5, 00, 400, and,600 wr xamnd. In th cours of calculatng valus of th tst statstcs, both unrstrctd and rstrctd (by β = c and/or β 3 = d) GME stmators ndd to b calculatd. Thrfor, bas and man squar rror masurs rlatng to ths and th last squars stmators wr calculatd as wll. Mont Carlo rsults for th tst statstcs and for th unrstrctd GME and OLS stmators ar prsntd n Tabls and, rspctvly, whl rsults rlatng to th rstrctd GME and OLS stmators ar prsntd n Tabl 3. Bcaus th choc of whch asymptotc covaranc matrx to us n calculatng th T Z and Wald tsts was nconsquntal, only th rsults for th scond suggstd covaranc matrx rprsntaton ar prsntd hr. Rgardng proprts of th tst statstcs, thr bhavor undr a tru H 0 s consstnt wth th bhavor xpctd from th rspctv asymptotc dstrbutons whn n s larg (sampl sz of 600), thr szs bng approxmatly.05 rgardlss of th choc of support for β. Th szs of th tsts rman wthn 0.0 of thr asymptotc sz whn n dcrass to 400, xcpt for th Lagrang Multplr tst undr support Z II, whch has a slghtly largr sz. Across all support chocs and rangng ovr all sampl szs from small to larg, th szs of th T Z and Wald tsts rman n th rang; for Z I supports and small sampl szs, th szs of th tsts ar substantally lss than Rsults wr smlar for th psudo-llhood and Lagrang Multplr tsts, xcpt for th cass of Z II support and n 00, whr th sz of th tst ncrasd as hgh as 0.36 for th psudo-llhood tst and 0.73 for th Lagrang multplr tst whn n = 5.

14 Entropy 03, Tabl. Rjcton Probablts for Tru (, 3 ) and Fals ( 3 0) Hypothss. Supports T z WALD Psudo-Llhood Lagrang Multplr H 0 H 0 H 0 H 0 Z I β = β = 0 β = β = 0 β = β = 0 β = β = 0 β 3 = β 3 = 0 β 3 = β 3 = 0 β 3 = β 3 = 0 n = n = n = n = Z II n = n = n = n = Z III n = n = n = n = Th powrs of th tsts wr all substantal n rjctng fals null hypothss xcpt for th T Z tst n th cas of Z II support and th smallst sampl sz, th lattr rsult bng ndcatv of a notably basd tst. Ovrall, th choc of support dd mpact th powr of tsts for rjctng th rrant hypothss, although th ffct was small for all but th T Z tst. In th cas of unrstrctd stmators and th most favorabl support choc (Z I ), th GME stmator domnatd th OLS stmator n trms of MSE, and GME suprorty was substantal for sampl szs of n 00 (Tabl ). Th GME-Z I stmator and, of cours, th OLS stmator, wr unbasd, wth th GME-Z I stmator xhbtng substantally smallr varancs for smallr n. Th choc of support has a sgnfcant ffct on th bas and MSE of th GME stmator for small sampl szs. thr th GME-Z II or GME-Z III stmator domnats th OLS stmator, although th GME-Z III stmator s gnrally th bttr stmator across th varous sampl szs. Whn n = 5, th GME-Z II stmator offrs notabl mprovmnt ovr OLS for stmatng thr of th four lmnts of β, but s sgnfcantly wors for stmatng β. For largr sampl szs, th GME-Z II stmator s gnrally nfror to th OLS stmator. Although th cntrs of th Z III support ar on avrag furthr from th tru β s than ar th cntrs of th Z II support, th wdr wdths of th formr rsult n a supror GME stmator. Th rsults for th rstrctd GME stmators n Tabl 3 ndcat that undr th rrant constrants 3 0, th GME domnats th OLS stmator for all sampl szs and for all support chocs. Th suprorty of th GME stmator s substantal for smallr sampl szs, but dsspats as sampl sz ncrass. Th rsults suggst a msspcfcaton robustnss of th GME stmator that dsrvs furthr nvstgaton.

15 Entropy 03, Tabl. E( ˆ ) and Man Squar Error Masurs Unrstrctd Estmators. Estmator β = β = β 3 = β 4 = 3 E( ˆ ) MSE E( ˆ ) MSE E( ˆ 3) MSE E( ˆ 4) MSE GME-Z I n = n = n = n = GME-Z II n = n = n = n = GME-Z III n = n = n = n = OLS n = n = n = n = Tabl 3. E( ˆ ) and Man Squar Error Masurs Rstrctd Estmators Undr th Errant Rstrcton 3 0. Estmator β = β 4 = 3 E( ˆ ) MSE E( ˆ 4) MSE GME-Z I n = n = n = n = GME-Z II n = n = n = n = GME-Z III n = n = n = n = OLS n = n = n = n =

16 Entropy 03, 5 77 Asymmtrc Error Supports W prsnt furthr Mont Carlo smulatons to show that rgularty condton R, whch assums symmtry of th dsturbanc trm, s not a ncssary condton for dntfcaton of th GME slop paramtrs. It s dmonstratd blow that f th supports of th rror dstrbuton asymmtrc, thn only th ntrcpt trm of th GME rgrsson stmator s asymptotcally basd. Th Mont Carlo xprmnts that follow ar dntcal to thos abov xcpt for spcfcaton of th usr suppld support ponts for th rror trms and th undrlyng tru rror dstrbuton. To llustratv th mpact of asymmtrc rrors, xprmnts ar basd on on st of support ponts symmtrc about zro, VI ( 0,0,0), and two sts of support ponts not symmtrc about zro, VII ( 5,5,5) and VIII ( 5,0,5). Th support V II s a smpl translaton of V I by fv postv unts n magntud and rtanng symmtry cntrd about 5. Th asymmtrc support V III translats th truncaton ponts by fv postv unts n magntud, but rtans th cntr support pont 0. Th tru rror dstrbuton s gnratd n two ways: a symmtrc dstrbuton spcfd as a (0,) dstrbuton truncatd at ( 3,3) and an asymmtrc dstrbuton spcfd as a Bta(3,) translatd and scald from support (0,) to ( 3,3) wth man 0.6. Supports on th paramtr coffcnts trms ar rtand as Z I, provdng symmtrc support ponts about th tru coffcnt valus. Mont Carlo xprmnts prsntd n Tabl 4 and 5 ar gnratd for sampl szs 5, 00, and 400 wth,000 rplcatons for ach sampl sz. Consdr whn th tru dstrbuton s symmtrc about zro. Slop coffcnts for rror supports that ar not symmtrc about zro appar basd n smallr sampl szs. Howvr, th bas and MSE of th slop coffcnts dcras as th sampl szs ncrass. xt, suppos th tru dstrbuton s asymmtrc. For symmtrc and asymmtrc supports only th ntrcpt trms ar prsstntly basd, dvrgng from th tru paramtr valus as th sampl sz ncrass. Ths rsults dmonstrat th robustnss of GME slop coffcnts to asymmtrc rror dstrbutons and usr suppld supports. Tabl 4. Man and MSE of,000 Mont Carlo Smulatons wth Tru Dstrbuton Symmtrc. Symmtrc and Asymmtrc Error Supports and Coffcnt Support Z I. Estmator β = β = β 3 = β 4 = 3 E(β ) MSE E(β ) MSE E(β 3 ) MSE E(β 4 ) MSE GME-Z I,V I GME-Z I,V II GME-Z I,V III OLS

17 Entropy 03, 5 77 Tabl 5. Man and MSE of 000 Mont Carlo Smulatons wth Tru Dstrbuton Asymmtrc. Symmtrc and Asymmtrc Error Supports and Coffcnt Support Z I. β = β = β 3 = β 4 =3 Estmator E(β ) MSE E(β ) MSE E(β 3 ) MSE E(β 4 ) MSE GME-Z I,V I GME-Z I,V II GME-Z I,V III OLS Furthr Rsults Unbasd GME Estmaton. It s apparnt from th proof of th thorm n Scton 3 that th p p ln( ) trms ar asymptotcally unnformatv. It s nstructv to not that f ths trms ar dltd from th GME objctv functon and th rsultng objctv functon s thn maxmzd through choosng b and w subjct to constrants C C4 and C6, th rsultng GME stmator s n fact unbasd for stmatng β. Ths follows bcaus th s ar d man zro and symmtrcally dstrbutd around zro, and th nw stmator, say, s such that s a symmtrc functon of th s. Baysan Analogus. As pontd out by [35] maxmum ntropy mthods can b motvatd as an mprcal Bays rul. W xpand on thr analogy by notng a strong formal paralll to th tradtonal Baysan framwor of nfrnc. In partcular, on can vw ntropy analogu to th log of a non-normalzd Baysan pror and K p w ln( p ) as th maxmum ln( w ) as th maxmum ntropy analogu to th non-normalzd log of th probablty dnsty rnl or log-llhood functon. For any gvn st of support ponts Z and V, w can dfn functons f and f by: z K p( b)ln( p( b)) w( x)ln( w( x)) f ( b ) and f ( x) z K p( )ln( p( )) v d v w( y)ln( w( y)) Thn for d ~ f, th maxmum llhood stmator of β s, and f on adds prors ~ f, thn ˆ dy nd

18 Entropy 03, s th Baysan postror mod stmator of β. W not th followng consquncs of ths quvalncs. Frst, f th support ponts v,, v can b chosn so that f s vry clos to th tru dstrbuton of, thn th GME stmator should b narly asymptotcally ffcnt. Scond, n fnt sampls th pror nformaton nfluncs ˆ such that ˆ s gnrally not unbasd. Thrd, th support ponts usd n th GME stmator hav no partcular rlatonshp to th ponts of support of th dstrbuton of a dscrt random varabl. Th dstrbutons f and f ar absolutly contnuous for any choc of Z and V. Th prvous Mont Carlo rsults llustrat th Baysan-l charactr of th maxmum ntropy rsults. Th GME wth rasonably narrow ponts of support cntrd on th tru valus of β domnatd th OLS stmator and was somtms far bttr. On th othr hand, th GME prformd poorly whn th ponts of support wr smlarly narrow and ms-cntrd by only on-ghth th rang of th ponts of support. In th lattr cas, man squard rrors wr oftn much wors than OLS and bass wr oftn substantal. Fnally, wdr ponts of support, vn though thy wr th most ms-cntrd of th cass xamnd, wr qut smlar to OLS rsults for modrat to larg sampl szs, and provdd som dgr of mprovmnt ovr OLS for small sampls. Fnally, th GME approach s a spcal cas of gnralzd cross ntropy, whch ncorporats a rfrnc probablty dstrbuton ovr support ponts. Ths allows a drct mthod of ncludng pror nformaton, an to a Baysan framwor. Howvr, n a classcal sns, th mprcal stmaton stratgs ar nhrntly dffrnt. GME Calculaton Mthod. Th condtonal maxmum ntropy formulaton () utlzd n th proof of asymptotc rsults rprsnts th bass for a computatonally ffcnt mthod of obtanng GME stmats. In partcular, maxmzng F( ) through choc of τ nvolvs a nonlnar sarch ovr a vctor of rlatvly low dmnson (K) as opposd to sarchng ovr th (KM + ) dmnsonal spac of (p,w) valus. In th procss of concntratng th objctv functon, not that th ndd Lagrang multplr functons ( K) and ( ( )) can b xprssd as lmntary functons for thr support ponts or lss, and stll xst n closd form (usng nvrs hyprbolc functons) for support vctors havng fv lmnts. As a pont of comparson, th calculaton of GME stmats n th Mont Carlo xprmnt wth =,600 was compltd n a mattr of sconds on a 33 mhz prsonal computr. Such a calculaton would b ntractabl, lt alon ffcnt, n th spac of (p,w) valus. W not furthr that th dual algorthm of [0] would stll nvolv a sarch ovr a spac of dmnson =,600, whch would b nfasbl hr and n othr problms n whch th numbr of data ponts s larg. 7. Conclusons W hav shown that th data-constrand GME stmator of th GLM s consstnt and asymptotcally normal as long as th coffcnts and rrors oby th constrants of th constrand maxmum ntropy problm. Furthrmor, w hav dmonstratd th possblty that th GME stmator can b asymptotcally ffcnt. Thus, dpndng on th dstrbuton of th rrors, GME may b mor or lss ffcnt than altrnatvs such as last squars. W prformd Mont Carlo tsts showng that th qualty of th GME stmats dpnds on th qualty of th supports chosn. Th Mont Carlo rsults suggst that GME wth wd supports wll oftn prform bttr than OLS whl provdng som robustnss to msspcfcaton.

19 Entropy 03, W hav shown how all th convntonal typs of asymptotc tsts can b calculatd for GME stmats. In th Mont Carlo study ths asymptotc tsts prformd xtrmly wll n sampls of 400 or mor. In smallr sampls th tsts prformd lss wll, partcularly whn th supports wr narrow, although som of th rsults wr qut accptabl. W hav also dmonstratd that all our rsults can b appld to a maxmum cross-ntropy stmator. Whl our focus has bn on asymptotc proprts, w hav also shown how th ntropy trms nvolvng th coffcnts play a rol analogous to a Baysan pror. Furthrmor, ths trms ar asymptotcally unnformatv and can b omttd f th rsarchr wshs to us an unbasd GME stmator. Rfrncs. Imbns, G.; Spady, R.; ohnson, P. Informaton Thortc Approachs to Infrnc n Momnt Condton Modls. Economtrca 998, 66, Imbns, G. A w Approach to Gnralzd Mthod of Momnts Estmaton, Harvard Insttut of Economc Rsarch Dscusson Papr o. 633; Harvard Unvrsty: Cambrdg, MA, USA, Ktamura, Y.; Stutzr, M. An Informaton-Thortc Altrnatv to Gnralzd Mthod of Momnts Estmaton. Economtrca 997, 65, Crss,.; Rad, T.R.C. Multnomal goodnss of ft tsts.. Roy. Stat. Soc B 984, 46, Pomp, B. On Som Entropy Masurs n Data Analyss. Chaos Solutons Fractals 994, 4, Sdnfld, T. Entropy and Uncrtanty. Phlo. Sc. 986, 53, udg, G.G.; Mttlhammr, R.C. An Informaton Thortc Approach to Economtrcs; Cambrdg Unvrsty Prss: Cambrdg, UK, Shannon, C.E. A mathmatcal thory of communcaton. Bll Syst. Tch.. 948, 7, ayns, E.T. Informaton Thory and Statstcal Mchancs. In Statstcal Physcs; Ford, K., Ed.; Bnjamn: w Yor, Y, USA, 963; p Golan, A.; udg, G.; Mllr, D. Maxmum Entropy Economtrcs: Robust Estmaton wth Lmtd Data; Wly & Sons: w Yor, Y, USA, Zllnr, A.; Hghfld, R.A. Calculaton of maxmum ntropy dstrbuton and approxmaton of margnal postror dstrbutons.. Economtrcs 998, 37, Soof, E. Informaton Thortc Rgrsson Mthods. In Applyng Maxmum Entropy to Economtrc Problms (Advancs n Economtrcs); Fomby, T., Hll, R.C., Eds.; Emrald Group Publshng Lmtd: London, UK, Ryu, H.K. Maxmum ntropy stmaton of dnsty and rgrsson functons.. Economtrcs 993, 56, Golan, A.; udg, G.; Prloff,.M. A maxmum ntropy approach to rcovrng nformaton from multnomal rspons data. ASA 996, 9, Vnod, H.D. Maxmum Entropy Ensmbls for Tm Srs Infrnc n Economcs. Asan Econ. 006, 7, Holm,. Maxmum ntropy Lornz curvs.. Economtrcs 993, 59, Marsh, T.L.; Mttlhammr, R.C. Gnralzd Maxmum Entropy Estmaton of a Frst Ordr Spatal Autorgrssv Modl. In Spatal and Spatotmporal Economtrcs (Advancs n Economtrcs); Pac, R.K., LSag,.P., Eds; Emrald Group Publshng Lmtd: London, UK, 004.

20 Entropy 03, Krbs, T. Statstcal Equlbrum n On-Stp Forward Loong Economc Modls. ET 997, 73, Golan, A.; udg, G.; Karp, L. A maxmum ntropy approach to stmaton and nfrnc n dynamc modls or countng fsh n th sa usng maxmum ntropy. EDC 996, 0, Kattuman, P.A. On th sz Dstrbuton of Establshmnts of Larg Entrprss: An Analyss for UK Manufacturng; Unvrsty of Cambrdg: Cambrdg, UK, Calln,.L.; Kwan, C.C.Y.; Yp, P.C.Y. Forgn-Exchang Rat Dynamcs: An Emprcal Study Usng Maxmum Entropy Spctral Analyss.. Bus. Econ. Stat. 985, 3, Bllaccco, A.; Russo, A. Dynamc Updatng of Labor Forc Estmats: ARES. Labor 99, 5, Sngupta,.K. Th maxmum ntropy approach n producton frontr stmaton. Math. Soc. Sc. 99, 5, Frasr, I. An applcaton of maxmum ntropy stmaton: Th dmand for mat n th Untd Statd Kngdom. Appl. Econ. 000, 3, Lv, B.; Thl, H. A Maxmum Entropy Approach to th Choc of Asst Dprcaton.. Accountng Rs. 978, 6, Stuzr, M. A smpl nonparamtrc approach to drvatv scurty valuaton.. Fnanc. 996, 5, Buchn, P.W.; Klly, M. Th Maxmum Entropy Dstrbuton of an Asst Infrrd from Opton Prcs.. Fnanc. and Quant. Anal. 996, 3, Prcl, P.V. Last squars and ntropy: A pnalty functon prspctv. Am.. Agr. Econ. 00, 83, Pars, Q.; Howtt, R. An Analyss of Ill-Posd Producton Problms Usng Maxmum Entropy. Am.. Agr. Econ. 998, 80, Lnc, S.H.; Mllr, D.. Rcovrng Output-Spcfc Inputs from Aggrgat Input Data: A Gnralzd Cross-Entropy Approach. Am.. Agr. Econ.998, 80, Mllr, D..; Plantnga, A.. Modlng Land Us Dcsons wth Aggrgat Data. Am.. Agr. Econ. 999, 8, Frnandz, L. Rcovrng Wastwatr Tratmnt Objctvs: An Applcaton of Entropy Estmaton for Invrs Control Problms. In Advancs n Economtrcs, Applyng Maxmum Entropy to Economtrc Problms; Fomby, T., Hll, R.C., Eds.; a Prss Inc.: London, UK, Wht, H. Asymptotc Thory for Economtrcans; Acadmc Prss: w Yor, Y, USA, Rao, C.R. Lnar Statstcal Infrnc and Its Applcatons, nd d; Wly & Sons: w Yor, Y, USA, Mllr, D.; udg, G.; Golan, A. Robust Estmaton and Condtonal Infrnc wth osy Data; Unvrsty of Calforna: Brly, CA, USA, by th authors; lcns MDPI, Basl, Swtzrland. Ths artcl s an opn accss artcl dstrbutd undr th trms and condtons of th Cratv Commons Attrbuton lcns (