As evident from the full-sample-model, we continue to assume that individual errors are identically and

Transcription

1 Maxmum Lkelhood smao Greee Ch.4; App. R scrp modsa, modsb If we feel safe makg assumpos o he sascal dsrbuo of he error erm, Maxmum Lkelhood smao (ML) s a aracve alerave o Leas Squares for lear regresso models. ve beer, ML ca also be used for o-lear models. I s hus a more geerally applcable esmao sraegy ha Ordary Leas Squares. As wh ay ecoomerc esmao, we sar wh a spulaed populao model, cludg dsrbuo assumpos for he error erm. Sckg wh he CLRM, we have ( ) y = xβ + ε ε ~, y = Xβ + ε ε ~, I () As evde from he full-sample-model, we coue o assume ha dvdual errors are decally ad depedely dsrbued (..d) followg a ormal dsrbuo wh mea ad varace. Le's collec he ukow parameers βad o a sgle vecor θ. The, he desy for a sgle observao (loosely raslaed as "he probably of observg a gve observao") codoal o θ (ad, of course also o x, whch we wll acly assume hroughou) s gve by y xβ f y θ = π exp () Sce he ε 's are depede, so are he y 's. We ca hus wre he desy for he ere sample ( he "sample desy" or "sample dsrbuo") as a produc of dvdual deses,.e. f ( y θ) = f ( y θ) = ( π ) exp ( y xβ ) = = = ( π ) exp y Xβ ( y Xβ) (3) The las equaly follows from he fac ha exp(a)*exp(b)=exp(a+b), ad usg he er produc-of vecor rule for he summao of squared producs. Our focus wll le o he esmao of he parameer vecor θ. For ha purpose, we ca erpre f y θ as a fuco of θ, gve he daa. Ths chage of esmao focus s expressed va a chage oao. Specfcally, we wll wre () as l ( θ y ) = l θ, ad refer o as he lkelhood fuco for a

2 sgle observao. Aalogously, we wll wre (3) as L( θ y) = L for he ere sample, or smply he sample lkelhood. θ, ad call he lkelhood fuco By coveo ad for mahemacal coveece (e.g. o avod very large umbers), we work wh he lkelhood log form. My oao wll be as follows: l ( θ ) s called he "lkelhood fuco" (LF) for a sgle observao. l l ( θ) s he log-lkelhood fuco (LLF) for a sgle observao. L θ s he sample LF. l L θ s he LLF for he sample. To derve he sample LLF, you ca ake oe of wo roues: () ake he log of he LF for a dvdual observaos, he add over dvduals (sce l(a*b) = l(a) + l(b)), or compue he sample LF frs, he ake he log. For he ormal regresso model: y xβ l l ( θ) = l π l ( ) l L l l ( θ) = π ( ) ( y Xβ) ( y Xβ) (4) Oce he LLF s specfed, he goal s o fd he θha maxmzes hs LLF for he sample. Thus, we rea he sample LLF as a objecve fuco, whch we are ryg o maxmze. Noe: Mos dvdual desy ordaes comprsed L( θ) wll be much smaller ha, so L( θ) self, beg a produc of hese ordaes usually akes a value bewee ad. The log of hs wll hus be egave. Therefore, maxmzg l L( θ ) meas fdg a value of θ ha ges he LLF "as close o zero as possble" from below. The frs dervave of l l ( θ) s he dvdual level score fuco or "grade", deoed as g = = sample equvale s gve as g ( θ) g θ. These grade expressos always have he same θ. The dmeso as θ. Aga, you have wo choces for compug g ( θ ) : () Compue he dvdual grade, he add up over observaos, or compue he sample grade drecly from l L( θ ). The frs order codos for opmzao requre he sample grade o equal zero a he soluo values for θ. For he ormal regresso model: ( θ) ( Xβ) l L X y β = = l L θ y Xβ y Xβ + 4 (5) Srcly speakg, he lkelhood fuco s oly proporoal o he sample dsrbuo. Ths meas oly cludes elemes of he sample dsrbuo ha coa he parameers of eres. Ay pars of he sample dsrbuo ha ca be mulplcavely separaed from hose elemes are gored he lkelhood fuco. However, whchever esmae of θ ha maxmzes he sample dsrbuo wll also maxmze he lkelhood fuco, ad vce versa. Thus, hs dsco ca be egleced for praccal purposes. Here I wll follow our ma exbook ad use he same mahemacal expresso for sample dsrbuo ad lkelhood.

3 3 The frs order codos are also called "Lkelhood quaos". They lead o he maxmum lkelhood esmaors ˆ e'e β = X'X X y ad ˆ = where e = y Xβ ˆ (6) Clearly, he soluo for he coeffce vecor s decal o he oe derved from he OLS problem. The esmae for dffers slghly from he OLS soluo as does o correc he deomaor for degrees of freedom (k). To assure a maxmum, we eed o exame he properes of he Hessa marx of secod dervaves. We H θ ), he add up over all could aga derve he hs expresso for a sgle observao (deoed observaos, or compue he sample Hessa H ( θ) drecly from he sample grade. For he ormal regresso model: ( θ) l L( θ) l L X X Xε 4 β β β H ( θ) = where l L l L = ε = y Xβ θ θ ε X εε β 4 6 ( ) (7) I ca be show ha all egevalues for hs Hessa are egave, hus H ( θ) s egave defe, ad we have deed a maxmum. Regulary ad Relaed Properes of ML smao For a fuco f ( y ) = θ o be ameable o ML esmao, mus sasfy he followg "regulary codos" (see Greee p. 55, alhough he expresses hese codos dfferely). The frs hree dervaves of l ( ) f y θ w.r..θ are couous ad fe for almos all y ad for all elemes of θ, ad he dervaves are egrable (so we ca derve he expecaos of he frs ad secod dervaves - see below). The suppor of y does o deped o θ. (So o eleme θ ca deoe a boud of he dsrbuo of y ) 3. The rue value of θ les a closed ad bouded "compac se". (.e. here ca' be dscouous "ses" of caddaes for he soluo of θ) If hese regulary codos are sasfed, wo eresg properes of ML esmao arse. These are he score dey ad he formao marx dey.

4 4 Score dey The score dey saes ha he expecao of he grade (w.r.. y ) a he rue values of he parameers s zero. Ths holds for boh a dvdual grade ad he sample grade. I mahemacal erms: y ( ) y g θ = g θ = (8) Le's verfy hs for he ormal regresso model: ( Xβ) X y y = y Xβ y Xβ + 4 (9) Frs erm: ( Xβ) X y = X ( y Xβ) y Xβ = y Xβ = Xβ Xβ = ad Secod erm: y Xβ y Xβ + = y Xβ ( y Xβ) ad ( y Xβ) ( y Xβ) = ( y y y Xβ + β X Xβ) = ( y y) ( y ) Xβ + β X Xβ = β X Xβ where ( y y) ( y y) = ( Xβ + ε) ( Xβ + ε) = ( β X Xβ + β Xε + εε ) = β X Xβ + So: y Xβ y Xβ = β X Xβ β X Xβ + = y Xβ y Xβ + = + = 4 ad Iformao Marx Idey The Iformao Marx s he egave of he expecao of he Hessa. A he dvdual level, we wll I θ. Aleravely, deoe as I ( θ ). If we sum hs over, we oba he sample Iformao marx we ca derve I ( θ ) by drecly akg he expecao (w.r.. y) of he sample Hessa:

5 5 ( Xβ) X X X y 4 I ( θ) = ( H ( θ) ) = y = y Xβ X y Xβ ( y Xβ) 4 6 X X X X = 6 4 () As we wll lear shorly, he verse of I ( θ ) s he mos effce esmaor for he varace-covarace marx of θ. The Iformao Marx Idey saes ha a he rue parameer values, he varace of he grade s equal o he formao marx,.e. I ( θ y ) = Var g y g ad g θ = θ θ I ( θ y) = Var g = g g θ y θ ( θ) () Ths s a b edous o show for he ormal regresso model, bu we wll llusrae hs equaly oher examples. Compuaoal Implemeao of ML For mos ecoomerc problems aalycal soluos for ˆβ ad assocaed sascs are dffcul o derve. Isead, we le he compuer solve he problem usg a "erave algorhm". The basc cocep s as follows:. Defe some (kx) sarg vecor θ (e.g. usg OLS, or resuls form a dffere daa se, or form a relaed bu smpler model, ec). Also, choose a "covergece crero" ha deermes whe he algorhm s compleed (ex: sop f he chage l L( θ y ) by movg from oe caddae θ o aoher s smaller ha "c", where c s some small umber, usually bewee. ad... Move from θ o θ (ad, more geerally, from θ o θ + ) usg he followg rule: θ+ = θ + λ () where s a k x k marx of k dreco vecors (oe for each eleme of θ), ad scalar λ s he "sep sze" ha deermes how far alog he drecoals we move ul we deerme ha we have reached θ. 3. valuae l L( θ y ) ad l ( θ y) ad deerme f he dfferece ( l L( θ y) - l ( ) L + θ y )< L + ( whch case he move from θ o θ + has brough us o a hgher po o l L) or o. If o,

6 6 repea seps ) ad 3) ul he dfferece <. Togeher, Seps ) ad 3) form a sgle "erao". 4. Coue ul l ( ) l ( ) L θ y L θ y < c (or some oher covergece crero s sasfed). The + las caddae θ s your ML soluo. Now we eed o address he choce of λ ad. As descrbed more deal Greee's Appedx, a popular dreco marx ha works well may praccal applcaos s ( H ) g = θ θ (3) Ths s called "Newo's Mehod". The le search parameer λ s eher chose ex ae (.e. λ =.5), or deermed a each erao by sasfyg ( θ λ ) l L + λ = (4) As evde from (3) Newo's approach requres he evaluao of he grade ad Hessa a each erao. If he aalycal forms for g ad H ca be easly derved, or f good umercal approxmaos are avalable, hs s o a problem. However, some cases H may be dffcul o compue or eve approxmae. I such suaos he H grades" (OPG), gve as ( θ ) ( θ ) ( θ ) where ( θ ) θ erm (3) ca be replaced by he "ouer produc of g ( θ ) ( θ ) g = = xk OPG G G G g θ Ths procedure s o que as accurae as usg he acual Hessa, bu works suffcely well may applcaos. We wll re-vs he OPG shorly whe we alk abou esmao of he asympoc varace of ˆβ. (5) smag he Asympoc Varace of he ML smaor To recap, for he OLS esmaor b we were able o derve he exac ("fe sample") varace as V ( b) = ( X X ), whch we he approxmaed by s ( X X ). However, s was show o be ubased, whch s aoher fe sample propery. I oher words, we ever had o resor o large-sample (or "asympoc") heory o derve V(b). Ths s dffere for he varace of he ML esmaor, V ( β ˆ ). For hs cosruc, o fe sample resuls are avalable. All esmaors for V ( β ˆ ) are asympoc aure,.e. hey coverge o he rue value as he sample sze goes o fy. Thus, hey become more relable wh larger sample sze.

7 The mos commoly used esmaor for V ( βˆ ) s he verse of he egave Hessa a he soluo value,.e. ˆ V ( ˆ ) = H ( ˆ ) β β (6) A secod esmaor ha does o requre he compuao of he Hessa s he vered OPG a he soluo value,.e. Vˆ ˆ ˆ β = G G (7) Ths approach s also kow as he "BHHH" esmaor, amed afer he foudg auhors Berd. e al. (974). I s ceraly covee, bu ca be very accurae for smaller samples. For a relaed dscusso see Greee p Noes for R Implemeao Scrps modsa ad b esmae he same CLRM based o wage daa as scrp modsb va ML. As we wll lear shorly, f he CLRM assumpos are sasfed, OLS ad ML should produce bascally decal resuls uder large sample szes. Ths s deed he case for hs example. Scrp modsa calls R's bul- opmzao roue ("opm"), whch uses a umercal grade ad Hessa o solve he opmzao problem. Ths s covee for he (may) saces whe he aalycal grade ad / or Hessa are dffcul o compue or program. You ca also ru opm wh a user-suppled grade (ad le jus he Hessa be derved umercally). See he opm maual (ype "?opm" R) for deals. As a geeral rule, he more aalycal compoes you ca supply, he faser ad more accurae your algorhm wll be. A fully aalycal mplemeao of ML s gve scrp modsb, where we code up our ow Newo roue based o aalycal grade ad Hessa o fd he maxmum of he log-lkelhood fuco.