Discrete Choice Modeling

Transcription

1 Dscrete Choce Modelg Wllam Greee Ster School of Busess New York Uversty [Topc 6-Nolear Models] 1/87

2 6. NONLINEAR MODELS [Topc 6-Nolear Models] 2/87

3 Nolear Models Ageda Estmato Theory for Nolear Models Estmators Propertes M Estmato Nolear Least Squares Maxmum Lkelhood Estmato GMM Estmato Mmum Dstace Estmato Mmum Ch-square Estmato Computato Nolear Optmzato Nolear Least Squares Newto-lke Algorthms; Gradet Methods (Backgroud: JW, Chapters 12-14, Greee, Chapters 7,12-14) [Topc 6-Nolear Models] 3/87

4 What s the Model? Ucodtoal characterstcs of a populato Codtoal momets: E[g(y) x]: meda, mea, varace, quatle, correlatos, probabl tes Codtoal probabltes ad destes Codtoal meas ad regressos Fully parametrc ad semparametrc specfcatos Parametrc specfcato: Kow up to parameter θ Codtoal meas: E[y x] = m(x, θ) [Topc 6-Nolear Models] 4/87

5 What s a Nolear Model? Model: E[g(y) x] = m(x,θ) Objectve: Lear about θ from y, X Usually estmate θ Lear Model: Closed form; = h(y, X) Nolear Model Not ecessarly wrt m(x,θ). E.g., y = exp(θ x + ε) θˆ Wrt estmator: Implctly defed. h(y, X, )=0, E.g., E[y x]= exp(θ x) ˆθ [Topc 6-Nolear Models] 5/87

6 What s a Estmator? Pot ad Iterval ˆθ= f(data model) I( ˆθ ) =θ± ˆ samplg varablty Classcal ad Bayesa ˆθ= E[ θ data,pror f( θ )] = expectato from posteror I( ˆθ ) = arrowest terval from posteror desty cotag the specfed probablty (mass) [Topc 6-Nolear Models] 6/87

7 Parameters Model parameters The parameter space: Estmators of parameters The true parameter(s) exp( y / θ) Example : f(y xβx ) =, θ = exp( ) Model parameters : β θ Codtoal Mea: E(y xβx ) = θ = exp( ) [Topc 6-Nolear Models] 7/87

8 The Codtoal Mea Fucto m( x, θ ) = E[y x] for some θ Θ. 0 0 A property of the codtoal mea: 2 E (y m( x, θ)) s mmzed by E[y x] y,x (Proof, pp , JW) [Topc 6-Nolear Models] 8/87

9 M Estmato Classcal estmato method 1 ˆθ= arg m q( data =1, θ) Example : Nolear Least squares ˆ 1 θ= arg m [y -E(y, θ)] 2 x =1 [Topc 6-Nolear Models] 9/87

10 A Aalogy Prcple for M Estmato 1 The estmator ˆθ mmzes q= q(data 1, θ) = The true parameter θ mmzes q*=e[q(data, θ)] 0 The weak law of large umbers: 1 P q= q(data 1, θ) q*=e[q(data, θ)] = [Topc 6-Nolear Models] 10/87

11 Estmato 1 P q= q(data 1, θ) q*=e[q(data, θ)] = Estmator P q q* ˆθ mmzes q True parameter θ mmzes q* P Does ths mply θ θ 0? Yes, f... ˆ 0 [Topc 6-Nolear Models] 11/87

12 Idetfcato Uqueess : If θ θ, the m(x, θ) m(x, θ) for some x Examples whch ths property s ot met: (1) (Multcollearty) (2) (Need for ormalzato) E[y x] = m( β x/ σ) (3) (Idetermacy) m(x, θ)= β +β x +β 1 2 x whe β = 0 β4 3 3 [Topc 6-Nolear Models] 12/87

13 Cotuty q(data, θ) s (a) Cotuous θ for all data ad all (b) Cotuously dfferetable. Frst dervatves are also cotuous (c) Twce dfferetable. Secod dervatves must be ozero, though they eed ot be cotuous fuctos of θ. (E.g. Lear LS) θ [Topc 6-Nolear Models] 13/87

14 1 Cosstecy P q= q(data 1, θ) q*=e[q(data, θ)] = Estmator P q q* ˆθ mmzes q True parameter θ mmzes q* P Does ths mply θ θ 0? ˆ 0 Yes. Cosstecy follows from detfcato ad cotuty wth the other assumptos [Topc 6-Nolear Models] 14/87

15 Asymptotc Normalty of M Estmators Frst order codtos: N (1/) Σ=1q(data, ˆθ ) = 0 θ ˆ 1 ˆ N q(data, θ) = Σ=1 θ ˆ 1 N = Σ ˆ ˆ =1g(data, θ ) = g(data, θ) For ay ˆθ, ths s the mea of a radom sample. We apply Ldberg-Feller CLT to assert the lmtg ormal dstrbuto of g(data, ˆθ ). [Topc 6-Nolear Models] 15/87

16 Asymptotc Normalty A Taylor seres expaso of the dervatve g(data, ˆθ ) = g(data, θ ) + H( θ )( ˆθ θ ) = q(data, θ ) H( θ ) = Σ= 1 θ θ θ = some pot betwee ˆθ ad θ The, ( ˆθ θ ) = [ H( θ )] g(data, θ ) ad ( ˆθ θ ) = [ H( θ)] g θ0 0 (data, ) [Topc 6-Nolear Models] 16/87

17 Asymptotc Normalty ( ˆθ θ ) = [ H( θ )] g(data, θ ) H θ [ ( )] coverges to ts expectato (a matrx) g(data, θ 0 ) coverges to a ormally dstrbuted vector (Ldberg-Feller) Imples lmtg ormal dstrbuto of (ˆθ θ ). 0 Lmtg mea s 0. Lmtg varace to be obtaed. Asymptotc dstrbuto obtaed by the usual meas. [Topc 6-Nolear Models] 17/87

18 Asymptotc Varace ˆ [ ( )] (data, ) a 1 θ θ 0 + H θ g θ 0 Asymptotcally ormal Mea = θ 0 Asy.Var[ ˆ] [ ( )] Var[ (data, )] [ ( )] 1 1 θ = H θ0 g θ0 H θ0 (A sadwch estmator, as usual) What s Var[ g(data, θ )]? 0 1 E[ g (data, θ0 ) g (data, θ0 ) '] Not kow what t s, but t s easy to estmate. 1 1 Σ= 1g(data, ˆθ) g(data ˆ, θ) ' [Topc 6-Nolear Models] 18/87

19 Estmatg the Varace Asy.Var[ ˆ] [ H( )] Var[ g(data, )] [ H( )] Estmate [ H( 1 1 θ = θ0 θ0 θ0 1 m(data, ˆθ) θ θ ˆˆ 2 1 θ0)] wth = m(data θ θ, ˆ) m(data, ˆ) Estmate Var[ g(data, θ0)] wth = 1 θ θ ˆ ˆ 2 E.g., f ths s lear least squares, (1/2) Σ (y -x β) m(data, ˆθ ) = (1 / 2)(y x b) = = 1 ( XX/) = 1 m(data, ˆθ) θ θ ˆˆ 2 =1 m(data θ θ, ˆ) m(data, ˆ) 2 N 2 = (1 / ) Σ θ θ ˆ ˆ = 1exx [Topc 6-Nolear Models] 19/87

20 Nolear Least Squares Gauss-Marquardt Algorthm q = the codtoal mea fucto = m( x, θ) m( x, θ) 0 g = = x = 'pseudo regressors ' θ Algorthm - terato ˆθ = ˆ θ + [ X 'X ] X 'e (k+1) (k) [Topc 6-Nolear Models] 20/87

21 Applcato - Icome Germa Health Care Usage Data, 7,293 Idvduals, Varyg Numbers of Perods Varables the fle are Data dowloaded from Joural of Appled Ecoometrcs Archve. Ths s a ubalaced pael wth 7,293 dvduals. They ca be used for regresso, cout models, bary choce, ordered choce, ad bvarate bary choce. Ths s a large data set. There are altogether 27,326 observatos. The umber of observatos rages from 1 to 7. (Frequeces are: 1=1525, 2=2158, 3=825, 4=926, 5=1051, 6=1000, 7=987). Note, the varable NUMOBS below tells how may observatos there are for each perso. Ths varable s repeated each row of the data for the perso. (Dowloaded from the JAE Archve) HHNINC = household omal mothly et come Germa marks / (4 observatos wth come=0 were dropped) HHKIDS = chldre uder age 16 the household = 1; otherwse = 0 EDUC = years of schoolg AGE = age years [Topc 6-Nolear Models] 21/87

22 Icome Data Desty INCOME Kerel desty estmate for INCOME [Topc 6-Nolear Models] 22/87

23 Expoetal Model f(icome Age,Educ,Marred) 1 HHNINC exp = θ θ θ = exp(a + a Educ + a Marred + a Age) E[HHNINC Age,Educ,Marred] Startg values for the teratos: E[y othg else]=exp(a ) 0 Start a = loghhninc, a = θ = a = a 3 = 0 [Topc 6-Nolear Models] 23/87

24 Covetoal Varace Estmator ˆ 2 = 1 Σ [y m(x, θ)] ( X 0 X 0 ) #parameters 1 Sometmes omtted. [Topc 6-Nolear Models] 24/87

25 Estmator for the M Estmator q = (1 / 2)[y exp( x β )] = (1 / 2)(y θ) g H 2 2 = e θx = y θ xx Estmator s [ Σ H] [ Σ gg ][ Σ H] N -1 N N -1 =1 =1 =1 2 = [ Σ y θxx ] [ Σ e θ xx ][ Σ y θxx ] N -1 N 2 N -1 =1 =1 =1 Ths s the Whte estmator. See JW, p [Topc 6-Nolear Models] 25/87

26 Computg NLS Reject; hhc=0$ Calc ; b0=log(xbr(hhc))$ Names ; x=oe,educ,marred,age$ Nlsq ; labels=a0,a1,a2,a3 ; start=b0,0,0,0 ; fc=exp(a0 x) ; lhs=hhc;output=3$ Create; theta = exp(x'b) ; e=hhc-theta ; g2=(e*theta)^2 ; h=hhc*theta$ Matrx; varm = <x'[h] x> * x'[g2]x * <x'[h] x> $ Matrx; stat(b,varm,x)$ [Topc 6-Nolear Models] 26/87

27 Iteratos Covergece e X X X X e -1 ' gradet ' = 0 0 ( 0 ' 0 ) 0 ' 0 Beg NLSQ teratos. Learzed regresso. Iterato= 1; Sum of squares= ; Gradet= Iterato= 2; Sum of squares= ; Gradet= Iterato= 3; Sum of squares= ; Gradet= E-02 Iterato= 4; Sum of squares= ; Gradet= E-04 Iterato= 5; Sum of squares= ; Gradet= E-06 Iterato= 6; Sum of squares= ; Gradet= E-08 Iterato= 7; Sum of squares= ; Gradet= E-10 Iterato= 8; Sum of squares= ; Gradet= E-12 Iterato= 9; Sum of squares= ; Gradet= E-14 Iterato= 10; Sum of squares= ; Gradet= E-16 Iterato= 11; Sum of squares= ; Gradet= E-17 Iterato= 12; Sum of squares= ; Gradet= E-19 Iterato= 13; Sum of squares= ; Gradet= E-21 Covergece acheved [Topc 6-Nolear Models] 27/87

28 NLS Estmates User Defed Optmzato Nolear least squares regresso LHS=HHNINC Mea = Stadard devato = Resduals Sum of squares = Varable Coeffcet Stadard Error b/st.er. P[ Z >z] Covetoal Estmates Costat EDUC MARRIED AGE Recomputed varaces usg sadwch estmator for M Estmato. B_ B_ B_ B_ [Topc 6-Nolear Models] 28/87

29 Hypothess Tests for M Estmato Null hypothess: c( θ)= 0 for some set of J fuctos (1) cotuous c( θ ) (2) dfferetable; = R( θ ), J K Jacoba θ (3) fuctoally depedet: Rak R( θ) = J Wald: gve ˆθ, Vˆ =Est.Asy.Var[ ˆθ], W=Wald dstace ˆ ˆ -1 =[ c( θ)- c( θ)]{ R( θ) V( θ) R ( θ)} ' [ c( θ)- c( θ)] ch-squared[j] [Topc 6-Nolear Models] 29/87

30 Chage the Crtero Fucto 1 P q= q(data θ θ = 1, ) q*=e[q(data, )] Estmator ˆθ mmzes q Estmator ˆθ c mmzes q subject to restrctos c( θ)=0 ˆc q ˆq. c D 2(q ˆ q ) ch squared[j] [Topc 6-Nolear Models] 30/87

31 Score Test LM Statstc Dervatve of the objectve fucto (1 / ) Σ=1 q(data, θ ) Score vector = = g(data, θ ) θ Wthout restrctos g(data, ˆθ ) = 0 Wth ull hypothess, c( ˆθ ) mposed g(data, ˆθ c ) geerally ot equal to 0. Is t close? (Wth samplg varablty?) c c 1 c Wald dstace = [ g (data, ˆθ )]'{Var[ g(data, ˆθ )]} [ g(data, ˆθ )] D LM ch squared[j] [Topc 6-Nolear Models] 31/87

32 Expoetal Model f(icome Age,Educ,Marred) 1 HHNINC exp = θ θ θ = exp(a + a Educ + a Marred + a Age) Test H : a = a = a = [Topc 6-Nolear Models] 32/87

33 Wald Test Matrx ; Lst ; R=[0,1,0,0 / 0,0,1,0 / 0,0,0,1] ; c=r*b ; Vc = R*Varb*R' ; Wald = c'<vc>c $ Matrx R has 3 rows ad 4 colums d D D D D D D D D Matrx C has 3 rows ad 1 colums Matrx VC has 3 rows ad 3 colums d d d d d D D D D-07 Matrx WALD has 1 rows ad 1 colums [Topc 6-Nolear Models] 33/87

34 Chage Fucto Calc ; M = sumsqdev $ (from urestrcted) Calc ; b0 = log(xbr(hhc)) $ Nlsq ; labels=a0,a1,a2,a3;start=b0,0,0,0 ; fc=exp(a0+a1*educ+a2*marred+a3*age) ; fx=a1,a2,a3? Fxes at start values ; lhs=hhc $ Calc ; M0 = sumsqdev $ [Topc 6-Nolear Models] 34/87

35 Costraed Estmato Nolear Estmato of Model Parameters Method=BFGS ; Maxmum teratos=100 Start values: D+01 1st dervs D-10 Parameters: D+01 Itr 1 F=.4273D+03 gthg=.2661d-10 * Coverged NOTE: Covergece tal teratos s rarely at a true fucto optmum. Ths may ot be a soluto (especally f tal teratos stopped). Ext from teratve procedure. 1 teratos completed. Why dd ths occur? The startg value s the mmzer of the costraed fucto [Topc 6-Nolear Models] 35/87

36 Costraed Estmates User Defed Optmzato Nolear least squares regresso LHS=HHNINC Mea = Stadard devato = Resduals Sum of squares = Varable Coeffcet Stadard Error b/st.er. P[ Z >z] A A (Fxed Parameter)... A (Fxed Parameter)... A (Fxed Parameter)... --> calc ; m0=sumsqdev ; lst ; df = 2*(m0 - m) $ DF = [Topc 6-Nolear Models] 36/87

37 LM Test Fucto : q = (1 / 2)[y exp(a + a Educ...)] Dervatve : LM statstc 0 1 g = e θx LM=( Σ g)[ Σ gg ] ( Σ g) 1 = 1 = 1 = 1 All evaluated at ˆa 0 = log(y),0,0,0 2 [Topc 6-Nolear Models] 37/87

38 LM Test Name ;x=oe,educ,marred,age$ Create ;theta=exp(x'b);e=hhc-theta$ Create ;g=e*theta ; g2 = g*g $ Matrx ; lst ; LM = 1'[g]x * <x'[g2]x> * x'[g]1 $ Matrx LM has 1 rows ad 1 colums [Topc 6-Nolear Models] 38/87

39 Maxmum Lkelhood Estmato Fully parametrc estmato Desty of y s fully specfed The lkelhood fucto = the jot desty of the observed radom varable. Example: desty for the expoetal model 1 y f(y x) β= exp, θ = exp( x ) θ θ E[y x ]= θ, Var[y x ]= θ 2 NLS (M) estmator examed earler operated oly o E[y x ]= θ. [Topc 6-Nolear Models] 39/87

40 The Lkelhood Fucto 1 y f(y = θ = x) β exp, exp( x ) θ θ Lkelhood = f(y,..., y x,..., x ) By depedece, y L( β data)= exp, θ = =1 exp( xβ) θ θ The MLE, βˆ, maxmzes the lkelhood fucto MLE [Topc 6-Nolear Models] 40/87

41 Log Lkelhood Fucto 1 y f(y = θ = x) β exp, exp( x ) θ θ 1 y L( β data)= exp, θ = =1 exp( xβ) θ θ The MLE, βˆ, maxmzes the lkelhood fucto logl( β data) s a mootoc fucto. Therefore The MLE, βˆ MLE MLE y logl( β data)= -logθ =1 θ, maxmzes the log lkelhood fucto [Topc 6-Nolear Models] 41/87

42 Codtoal ad Ucodtoal Lkelhood Ucodtoal jot desty f(y, x θδ, ) θ= our parameters of terest δ = parameters of the margal desty of x Ucodtoal lkelhood fucto L( θδ, y, X)= f(y, x θδ, ) =1 f(y, x θδ, ) = f(y x, θδ, )g( x θδ, ) L( θδ, y, X )= f(y x, θδ, )g( x θδ, ) =1 Assumg the parameter space parttos logl( θδ, y, X )= logf(y x, θ ) + logg( x δ) =1 =1 = codtoal log lkelhood + margal log lkelhood [Topc 6-Nolear Models] 42/87

43 Cocetrated Log Lkelhood ˆ θ maxmzes logl( θ data) Cosder a partto, θ=( βα, ) two parts. logl Maxmum occurs where = 0 β α Jot soluto equates both dervatves to 0. If logl/ α=0 admts a mplct soluto for α MLE terms of β, α ˆ =αβ ˆ( ˆ), the wrte MLE logl ( βαβ, ( ))=a fucto oly of β. c Cocetrated log lkelhood ca be maxmzed for β, the the soluto computed for α. The soluto must occur where α ˆ =αβ ˆ ( ˆ),so restrct the search to ths subspace of the parameter space. MLE [Topc 6-Nolear Models] 43/87

44 A Cocetrated Log Lkelhood Fxed effects expoetal regresso: θ = exp( α+ x β) logl = ( logθ y / θ ) logl α = 1 t= 1 = ( α+ ( x β) y exp( α x β)) = 1 t= 1 T = 1 y exp( α x β)( 1) t= 1 T T T t t t t t t t t = T + y exp( α x β) t= 1 t t T = T + exp( α ) y exp( x β) = 0 t= 1 t t t t T Σ t= 1 y t / exp(x tβ) Solve ths for α = log = α ( β) T T Σ c t= 1y t / exp(x tβ) Cocetrated log lkelhood has θ = t exp(x tβ) T [Topc 6-Nolear Models] 44/87

45 ML ad M Estmato logl( θ ) = log f(y x, θ) = 1 ˆθ MLE = argmax log f(y 1 x, θ) = 1 = argm - log f(y 1 x, θ) = The MLE s a M estmator. We ca use all of the prevous results for M estmato. [Topc 6-Nolear Models] 45/87

46 Regularty Codtos Codtos for the MLE to be cosstet, etc. Augmet the cotuty ad detfcato codtos for M estmato Regularty: Three tmes cotuous dfferetablty of the log desty Fte thrd momets of log desty Codtos eeded to obta expected values of dervatves of log desty are met. (See Greee (Chapter 14)) [Topc 6-Nolear Models] 46/87

47 Cosstecy ad Asymptotc Normalty of the MLE Codtos are detcal to those for M estmato Terms proofs are log desty ad ts dervatves Nothg ew s eeded. Law of large umbers Ldberg-Feller cetral lmt apples to dervatves of the log lkelhood. [Topc 6-Nolear Models] 47/87

48 Asymptotc Varace of the MLE Based o results for M estmato Asy.Var[ ˆθ ] MLE ={-E[Hessa]} {Var[frst dervatve]}{-e[hessa]} logl logl logl = -E Var -E θ θ θ θ θ [Topc 6-Nolear Models] 48/87

49 The Iformato Matrx Equalty Fudametal Result for MLE The varace of the frst dervatve equals the egatve of the expected secod dervatve. 2 logl -E = The Iformato Matrx θ θ Asy.Var[ ˆθ ] MLE logl logl logl = -E -E -E θ θ θ θ θ θ 2 logl = -E θ θ 1 [Topc 6-Nolear Models] 49/87

50 Three Varace Estmators Negatve verse of expected secod dervatves matrx. (Usually ot kow) Negatve verse of actual secod dervatves matrx. Iverse of varace of frst dervatves [Topc 6-Nolear Models] 50/87

51 Asymptotc Effcecy M estmator based o the codtoal mea s semparametrc. Not ecessarly effcet. MLE s fully parametrc. It s effcet amog all cosstet ad asymptotcally ormal estmators whe the desty s as specfed. Ths s the Cramer-Rao boud. Note the mpled comparso to olear least squares for the expoetal regresso model. [Topc 6-Nolear Models] 51/87

52 Ivarace Useful property of MLE If λ=g( θ) s a cotuous fucto of θ, the MLE of λ s g( ˆθ ) MLE E.g., the expoetal FE model, the MLE of λ = exp(- α) s exp(- αˆ ),MLE [Topc 6-Nolear Models] 52/87

53 Applcato: Expoetal Regresso Expoetal (Loglear) Regresso Model Maxmum Lkelhood Estmates Depedet varable HHNINC Number of observatos Iteratos completed 10 Log lkelhood fucto Number of parameters 4 Restrcted log lkelhood Ch squared Degrees of freedom Varable Coeffcet Stadard Error b/st.er. P[ Z >z] Mea of X Parameters codtoal mea fucto Costat EDUC MARRIED AGE NLS Results wth recomputed varaces usg results for M Estmato. B_ B_ B_ B_ [Topc 6-Nolear Models] 53/87

54 Varace Estmators LogL = logθ y / θ, θ= exp( xβ ) = 1 logl g = = x + (y / θ ) x = [(y / θ) 1] x β = 1 = 1 = 1 Note, E[y x ] = θ, so E[ g]= 0 2 logl H = = (y θ = 1 / ) x x β β E[ H] = x x = - X'X (kow for ths partcular model) [Topc 6-Nolear Models] 54/87

55 Three Varace Estmators Berdt-Hall-Hall-Hausma (BHHH) = [(y ˆ =1 1 / θ) 1] gg xx = Based o actual secod dervatves 1 1 H = (y ˆ =1 1 / θ) xx = Based o expected secod dervatves { E H } =1 = x = ( ) = 1 x X'X [Topc 6-Nolear Models] 55/87

56 Varace Estmators Loglear ; Lhs=hhc;Rhs=x ; Model = Expoetal create;theta=exp(x'b);h=hhc/theta;g2=(h-1)^2$ matr;he=<x'x> ; ha=<x'[h]x> ; bhhh=<x'[g2]x>$ matr;stat(b,ha);stat(b,he);stat(b,bhhh)$ Varable Coeffcet Stadard Error b/st.er. P[ Z >z] B_ ACTUAL B_ B_ B_ B_ EXPECTED B_ B_ B_ B_ BHHH B_ B_ B_ [Topc 6-Nolear Models] 56/87

57 Hypothess Tests Trty of tests for ested hypotheses Wald Lkelhood rato Lagrage multpler All as defed for the M estmators [Topc 6-Nolear Models] 57/87

58 Example: Expoetal vs. Gamma Gamma Dstrbuto: f(y x, θ,p) = P exp( y / θ)y θ P 1 Γ(P) Expoetal: P = 1 P > 1 [Topc 6-Nolear Models] 58/87

59 Log Lkelhood logl =Σ P logθ log Γ(P) y / θ+ (P 1)log y Γ (1) = 0! = 1 = 1 logl =Σ logθ y / θ = 1 + (P 1) log y (P 1) log θ log Γ(P) =Σ logθ y / θ+ (P 1)log(y / θ) log Γ(P) = 1 = Expoetal logl + part due to P 1 [Topc 6-Nolear Models] 59/87

60 Estmated Gamma Model Gamma (Loglear) Regresso Model Depedet varable HHNINC Number of observatos Iteratos completed 18 Log lkelhood fucto Number of parameters 5 Restrcted log lkelhood Ch squared Degrees of freedom 4 Prob[ChSqd > value] = Varable Coeffcet Stadard Error b/st.er. P[ Z >z] Mea of X Parameters codtoal mea fucto Costat EDUC MARRIED AGE Scale parameter for gamma model P_scale [Topc 6-Nolear Models] 60/87

61 Testg P = 1 Wald: W = ( ) 2 / = Lkelhood Rato: logl (P=1)= logl P = LR = 2( ) = Lagrage Multpler [Topc 6-Nolear Models] 61/87

62 Dervatves for the LM Test logl =Σ logθ y / θ+ (P 1)log(y / θ) log Γ(P) logl β logl P = 1 =Σ (y / θ P) x =Σ g = 1 = 1 x, =Σ log(y / θ) ψ (P) =Σ g = 1 = 1 P, ψ(1)= For the LM test, we compute these at the expoetal MLE ad P = 1. x [Topc 6-Nolear Models] 62/87

63 Ps Fucto = dlogγ(p)/dp 5 Ps(p) = Log Dervatve of Gamma Fucto 0-5 Ps Fucto PV [Topc 6-Nolear Models] 63/87

64 Dervatves g x,(1) g x,(educ) g = = = g x,(marred), g g = whe P s urestrcted 1 g x,(age) g P, = whe P = 1 the gamma model [Topc 6-Nolear Models] 64/87

65 Score Test Test the hypothess that the dervatve vector equals zero whe evaluated for the larger model wth the restrcted coeffcet vector. 1 Estmator of zero s g= g =1 Statstc = ch squared = g [Var g] 1 1 Use =1 gg -1 g (the 's wll cacel). ch squared = ( ) -1 ( g ) gg =1 =1 g =1 [Topc 6-Nolear Models] 65/87

66 Calculated LM Statstc Create ;theta=exp(x b) ; g=(hhc/theta 1) Create ;gp=log(hhc/theta)-ps(1)$ Create ;g1=g;g2=g*educ;g3=g*marred;g4=g*age;g5=gp$ Namelst;gg=g1,g2,g3,g4,g5$ Matrx ;lst ; lm = 1'gg * <gg'gg> * gg'1 $ Matrx LM ? Use bult- procedure.? LM s computed wth actual Hessa stead of BHHH logl;lhs=hhc;rhs=oe,educ,marred,age;model=g;start=b,1;maxt=0$ LM Stat. at start values [Topc 6-Nolear Models] 66/87

67 Clustered Data ad Partal Lkelhood Pael Data: y x, t = 1,..., T t t Some coecto across observatos wth a group Assume margal desty for y x = f(y x, θ) Jot desty for dvdual s f(y,..., y X ) f(y x, θ) 1,T t= 1 t t T t t t t T "Pseudo loglkelhood" = log f(y x, θ) = 1 t= 1 = log f(y x, θ) = 1 t= 1 Just the pooled log lkelhood, gorg the pael aspect of the data. Not the correct log lkelhood. Does maxmzg wrt θ work? Yes, f the margal desty s correctly specfed. T t t t t [Topc 6-Nolear Models] 67/87

68 Iferece wth Clusterg (1) Estmator s cosstet (2) Asymptotc Covarace matrx eeds adjustmet Asy.Var[]=[Hessa] Var[gradet][Hessa] H = T = 1 t= 1 g = g, where g = g Terms t = 1 t= 1 t g H Est.Var[ ˆθ ] = -1-1 T are ot depedet, so estmato of the varace caot be doe wth = 1 t= 1 ( t ) ( t ) 1 T T ( = Hˆ = ) { T 1 t 1 t = 1( g ˆ t= 1 t ) ( ) }( ) T g ˆ = t ' t 1 = 1 Hˆ t= 1 t T T = 1 t= 1 t= 1 T gg But, terms across are depedet, so we estmate Var[ g] wth g g ' PMLE (Geerally serts a correcto term /(-1) before the mddle term.) t t 1 [Topc 6-Nolear Models] 68/87

69 Cluster Estmato Expoetal (Loglear) Regresso Model Maxmum Lkelhood Estmates Covarace matrx for the model s adjusted for data clusterg. Sample of observatos cotaed 7293 clusters defed by varable ID whch detfes by a value a cluster ID Varable Coeffcet Stadard Error b/st.er. P[ Z >z] Mea of X Parameters codtoal mea fucto Costat EDUC MARRIED AGE Ucorrected Stadard Errors Costat EDUC MARRIED AGE [Topc 6-Nolear Models] 69/87

70 O Clusterg The theory s very loose. That the margals would be correctly specfed whle there remas correlato across observatos s ambguous It seems to work pretty well practce (ayway) BUT It does ot mply that oe ca safely just pool the observatos a pael ad gore uobserved commo effects. The correcto redeems the varace of a estmator, ot the estmator, tself. [Topc 6-Nolear Models] 70/87

71 Robust Estmato If the model s msspecfed some way, the the formato matrx equalty does ot hold. Assumg the estmator remas cosstet, the approprate asymptotc covarace matrx would be the robust matrx, actually, the orgal oe. Asy.Var[ ˆθ ] = [ E[Hessa]] Var[gradet][ E[Hessa]] MLE 1 1 (Software ca be coerced to computg ths by tellg t that clusters all have oe observato them.) [Topc 6-Nolear Models] 71/87

72 Two Step Estmato ad Murphy/Topel Lkelhood fucto defed over two parameter vectors =1 logl= logf(y x,z, θδ, ) (1) Maxmze the whole thg. (FIML) (2) Typcal Stuato: Two steps 1 y E.g., f(hhninc educ,marred,age,ifkds)= exp, θ θ θ = exp( β +β Educ +β Marred +β Age +β Pr[IfKds]) If[Kds age,bluec] = Logstc Regresso Pr[IfKds]=exp( δ +δ Age +δ Bluec) /[1 + exp( δ +δ Age +δbluec)] (3) Two step strategy: Ft the stage oe model ( δ) by MLE frst, sert the results logl( θδ, ˆ) ad estmate θ. [Topc 6-Nolear Models] 72/87

73 Two Step Estmato (1) Does t work? Yes, wth the usual detfcato codtos, cotuty, etc. The frst step estmator s assumed to be cosstet ad asymptotcally ormally dstrbuted. (2) The asymptotc covarace matrx at the secod step that takes ˆ δ as f t were kow s too small. (3) Repar to the covarace matrx by the Murphy Topel Result (1983,2002) [Topc 6-Nolear Models] 73/87

74 Murphy-Topel - 1 logl ( δ) defes the frst step estmator. Let Vˆ = Estmated asymptotc covarace matrx for ˆδ log f,1(..., δ) ˆ 1 g ˆ ˆ,1 =. ( V1 mght = [ Σ=1g,1g,1] ) δ logl( θδ, ˆ) defes the secod step estmator usg the estmated value of δ. Vˆ = Estmated asymptotc covarace matrx for ˆθ ˆδ log f,2(..., θδ, ˆ) ˆ 1 g ˆ ˆ,2 =. ( V2 mght = [ Σ=1g,2g,2] ) θ Vˆ s too small [Topc 6-Nolear Models] 74/87

75 Murphy-Topel - 2 Vˆ 1 1,1,1 1 =1,1,1 2 = Estmated asymptotc covarace matrx for ˆδ g = log f (..., δ) / δ. ( Vˆ mght = [ Σ gˆ gˆ ] ) Vˆ g h C R = Estmated asymptotc covarace matrx for ˆθ ˆδ = log f (..., θ, ˆδ ) / θ. ( Vˆ mght =,2,2 2 = log f (..., θ, ˆδ ) / δ ˆ,2,2 = Σ = Σ =1,2,2 =1,2,1 [ Σ gˆ gˆ ] ) 1 =1,2,2 gˆ hˆ (the off dagoal block the Hessa) gˆ gˆ (cross products of dervatves for two logl's) M&T: Corrected ˆ ˆ ˆ ˆ ˆ ˆ ˆ V2 = V2 + V 2 [ CV 1 C' - CV 1 R' - RV 1 C' ] V 2 [Topc 6-Nolear Models] 75/87

76 Applcato of M&T Names ; z1=oe,age,bluec$ Logt ; lhs=hhkds ; rhs=z1 ; prob=prfkds $ Matrx ; v1=varb$ Create ; g1=hhkds-prfkds$ Names ; z2 = oe,educ,marred,age,prfkds Loglear; lhs=hhc;rhs=z2;model=e$ Matrx ; v2=varb$ Create ; g2=hhc*exp(z2'b)-1$ Create ; h2=g2*b(5)*prfkds*(1-prfkds)$ Create ; varc=g1*g2 ; varr=g1*h2$ Matrx ; c=z2'[varc]z1 ; r=z2'[varr]z1$ Matrx ; q=c*v1*c'-c*v1*r'-r*v1*c ; mt=v2+v2*q*v2;stat(b,mt)$ [Topc 6-Nolear Models] 76/87

77 M&T Applcato Multomal Logt Model Depedet varable HHKIDS Varable Coeffcet Stadard Error b/st.er. P[ Z >z] Mea of X Characterstcs umerator of Prob[Y = 1] Costat AGE BLUEC Expoetal (Loglear) Regresso Model Depedet varable HHNINC Varable Coeffcet Stadard Error b/st.er. P[ Z >z] Mea of X Parameters codtoal mea fucto Costat EDUC MARRIED AGE PRIFKIDS B_ B_ B_ B_ B_ Why so lttle chage? N = 27,000+. No ew varato. [Topc 6-Nolear Models] 77/87

78 GMM Estmato 1 N gβ ( )= Σ= m 1 (y,x,β ) N Asy.Var[ gβ ( )] estmated wth 1 1 N W= βσm = 1m,x (y,x,β, ) (y ) N N The GMM estmator of β the mmzes 1 N 1 1 N q = Σ= 1m (y β,x 'W, ) m Σ= 1, x (y,β ) N N. ˆ -1 1 gβ ( ) Est.Asy.Var[ βgmm] = [ G'W G], G = β [Topc 6-Nolear Models] 78/87

79 GMM Estmato-1 GMM s broader tha M estmato ad ML estmato Both M ad ML are GMM estmators. 1 = log f(y x, β) gβ ( ) for MLE = 1 β 1 = E(y x, β) gβ ( ) e = 1 for NLSQ β [Topc 6-Nolear Models] 79/87

80 GMM Estmato - 2 Exactly detfed GMM problems 1 N Whe gβ ( ) = Σ = m 1 (y,x,β ) = 0 s K equatos N K ukow parameters (the exactly detfed case), the weghtg matrx 1 N 1 1 N q = Σ= 1m (y β,x 'W, ) Σ= 1 (y N m, x,β ) N s rrelevat to the soluto, sce we ca set exactly gβ ( ) = 0 so q = 0. Ad, the asymptotc covarace matrx (estmator) s the product of 3 square matrces ad becomes [ G'W G] = G WG' [Topc 6-Nolear Models] 80/87

81 Optmzato - Algorthms Maxmze or mmze (optmze) a fucto F( θ) Algorthm = rule for searchg for the optmzer Iteratve algorthm: (k+ 1) (k) = θ + Update (k+ 1) (k) (k) Dervatve (gradet) based algorthm θ θ (k) = θ + Update( g ) Update s a fucto of the gradet. Compare to 'dervatve free' methods (Dscotuous crtero fuctos) [Topc 6-Nolear Models] 81/87

82 Algorthms Iterato Geeral structure: g λ (k) (k) W (k) Optmzato θ = θ + Update (k+1) (k) (k) = θ +λ (k+1) (k) (k) (k) (k) = dervatve vector, pots to a better value tha θ θ (k) = drecto vector = 'step sze' = a weghtg matrx W Algorthms are defed by the choces of g λ (k) ad W (k) [Topc 6-Nolear Models] 82/87

83 Algorthms (k) (k) (k) -g 'g (k) Steepest Ascet: λ =, W = I (k) (k) (k) g 'H g g H (k) (k) = frst dervatve vector = secod dervatves matrx Newto's Method: λ = -1, W = [ H ] (k) (k) (k) 1 ( Sometmes called Newto-Raphso. Method of Scorg: λ = -1, W = [E[ H ]] (k) (k) (k) 1 (Scorg uses the expected Hessa. Usually feror to Newto's method. Takes more teratos.) BHHH Method (for MLE) : λ (k) (k) (k) = -1, W = [ Σ = g g '] (k) 1 1 [Topc 6-Nolear Models] 83/87

84 Le Search Methods Squeezg: Essetally tral ad error (k) λ = 1, 1/2, 1/4, 1/8,... Utl the fucto mproves Golde Secto: Iterpolate betwee (k) ad (k-1) Others : May dfferet methods have bee suggested λ λ [Topc 6-Nolear Models] 84/87

85 Quas-Newto Methods How to costruct the weghtg matrx: Varable metrc methods: W W E, W I (k) (k 1) (k 1) (1) = + = Rak oe updates: (k) (k 1) (k 1) (k 1) W W a a ' = + (Davdo Fletcher Powell) There are rak two updates (Broyde) ad hgher. [Topc 6-Nolear Models] 85/87

86 Stoppg Rule Whe to stop teratg: 'Covergece' (1) Dervatves are small? Not good. Maxmzer of F() s the same as that of f(), but the dervatves are small rght away. (2) Small absolute chage parameters from oe terato to the ext? Problematc because t s a fucto of the stepsze whch may be small. (3) Commoly accepted 'scale free' measure = g [ H ] g (k) (k) 1 (k) [Topc 6-Nolear Models] 86/87

87 For Example Nolear Estmato of Model Parameters Method=BFGS ; Maxmum teratos= 4 Covergece crtera:gthg.1000d-05 chg.f.0000d+00 max db.0000d+00 Start values: D D D D D+01 1st dervs D D D D D+05 Parameters: D D D D D+01 Itr 1 F=.3190D+05 gthg=.1078d+07 chg.f=.3190d+05 max db =.1042D+13 Try = 0 F=.3190D+05 Step=.0000D+00 Slope= D+07 Try = 1 F=.4118D+06 Step=.1000D+00 Slope=.2632D+08 Try = 2 F=.5425D+04 Step=.5214D-01 Slope=.8389D+06 Try = 3 F=.1683D+04 Step=.4039D-01 Slope= D+06 1st dervs D D D D D+04 Parameters: D D D D D+00 Itr 2 F=.1683D+04 gthg=.1064d+06 chg.f=.3022d+05 max db =.4538D+07 Try = 0 F=.1683D+04 Step=.0000D+00 Slope= D+06 Try = 1 F=.1006D+06 Step=.4039D-01 Slope=.7546D+07 Try = 2 F=.1839D+04 Step=.4702D-02 Slope=.1847D+06 Try = 3 F=.1582D+04 Step=.1855D-02 Slope=.7570D st dervs D D D D D-06 Itr 20 F=.1424D+05 gthg=.1389d-07 chg.f=.1155d-08 max db =.1706D-08 * Coverged Normal ext from teratos. Ext status=0. Fucto= D+05, at etry, D+05 at ext [Topc 6-Nolear Models] 87/87