Composite Likelihood

Transcription

1 Composite Likelihood Nancy Reid January 30, 2012 with Cristiano Varin and thanks to Don Fraser, Grace Yi, Ximing Xu

2 Background parametric model f (y; θ), y R m ; θ R d likelihood function L(θ; y) f (y; θ) why likelihood? maximum likelihood estimator is consistent and asymptotically efficient ˆθ. N{θ, j 1 (ˆθ)} j(θ) = l (θ); l(θ) = log L(θ) likelihood ratio, or log-likelihood difference, captures asymmetry in the model w(θ) = 2{l(ˆθ) l(θ)}. χ 2 d combine with prior for Bayesian inference EM algorithm for computing maximum likelihood estimate Composite Likelihood Brown, January

3 ... background warning: likelihood methods need regularity conditions on the model can have poor finite sample behaviour for large numbers of parameters priors also very tricky with large numbers of parameters finite sample corrections may be advisable difficulty with construction of the likelihood function inversion of large covariance matrices intractable integrals; awkward normalization constants combinatorial explosion nuisance components difficult to specify lower dimensional marginal or conditional distributions may be tractable combine these to form composite likelihood Composite Likelihood Brown, January

4 Terminology Model f (y; θ), y R m, θ R p Events A 1,..., A K ; sub-densities f (y A k ; θ) Composite log-likelihood K K cl(θ; y) = w k log f (y A k ; θ) = w k l(θ; y A k ) k=1 k=1 w k weights to be determined composite likelihood is a type of: pseudo-likelihood (spatial modelling); quasi-likelihood (econometrics); limited information method (psychometrics)... Composite Likelihood Brown, January

5 Example: spatial generalized linear models generalized linear geostatistical models E{Y (s) u(s)} = g{x(s) T β + u(s)}, s S R d, d 2 Diggle & Ribeiro, 2007 random intercept u( ) is a realization of a stationary GRF, mean 0, covariance cov{u(s), u(s )} = σ 2 ρ(s s ; α) m observed locations y = (y 1,..., y m ) with y i = y(s i ) likelihood function m L(θ; y) = f (y i u i ; θ) f (u; θ) du R n }{{} 1... du m i=1 MVN(0,Σ) no factorization into lower dimensional integrals, as with independent observations from the usual GLMMs Composite Likelihood Brown, January

6 ... spatial generalized linear models L(θ; y) = R n i=1 n f (y i u i ; θ)f (u; θ)du 1... du n simulation methods, MCMC, MCEM, etc., costly O(m 3 ) pairwise likelihood L pair (θ; y) = f (y i u i ; θ)f (y j u j ; θ)f 2 (u i, u j ; θ)du i du j R 2 {(i,j) S δ } Heagerty & Lele (1998), Varin (2008) comments: product of bivariate integrals accurate quadrature approximations available use only close pairs: S δ = {(i, j) : s i s j < δ} computational cost O(n) Composite Likelihood Brown, January

7 Composite conditional likelihoods Besag (1974) pseudo-likelihood m L C (θ; y) = f (y r {y s : y s neighbour of y r ; θ) r=1 pairwise conditional m m L C (θ; y) = f (y r y s ; θ) full conditional L C (θ; y) = time series r=1 s=1 Molenbergs & Verbeke (2005); Mardia et al. (2009) L C (θ; y) = m f (y r y ( r) ; θ) r=1 m f (y r y r 1 ; θ) r=1 Composite Likelihood Brown, January

8 Composite marginal likelihoods independence likelihood pairwise likelihood L ind (θ; y) = L pair (θ; y) = pairwise differences L diff (θ; y) = m m f (y r ; θ) r=1 m r=1 s=r+1 m 1 m r=1 s=r+1 optimal combination of L ind and L pair Chandler & Bate (2007) f (y r, y s ; θ) Cox & Reid (2004); Varin (2008) f (y r y s ; θ) Curriero & Lele (1999) Composite Likelihood Brown, January

9 Derived quantities composite log-likelihood cl(θ; y) = log L C (θ; y) = K k=1 w kl k (θ; y) composite score u(θ; y) = θ cl(θ; y) = K k=1 w k θ l k (θ; y) E{u(θ; Y )} = 0 sensitivity matrix H(θ) = E θ { θ u(θ; Y )} variability matrix J(θ) = var θ {u(θ; Y )} Godambe information G(θ) = H(θ)J 1 (θ)h(θ) H(θ) J(θ) Composite Likelihood Brown, January

10 Inference Sample y 1,..., y n independent from f (y; θ) or f i (y; θ) n Composite log-likelihood cl(θ; y) = cl(θ; y i ) maximum composite likelihood estimator ˆθ CL = arg max cl(θ; y) u(ˆθ CL ; y) = 0 i=1 Asymptotic consistency, normality n(ˆθ CL θ) L N p {0, G 1 (θ)}, n, m fixed if n fixed and m, need assumptions on replication examples include time series and spatial data decaying correlations G(θ) = H(θ)J 1 (θ)h(θ) Composite Likelihood Brown, January

11 ... inference θ = (ψ, λ), ψ R q, λ R p q inference based on maximum likelihood estimator ˆθ CL. N p {θ, G 1 (ˆθ CL )} = ˆψ CL. N q {ψ, G ψψ (ˆθ CL )} CL log-likelihood ratio statistic w CL (ψ) = 2{cl(ˆθ CL ) cl( θ CL )} L q j=1 λ jzj 2 θ CL = {ψ, ˆλ CL (ψ)}, constrained mle Z j N(0, 1) λ j eigenvalues of (H ψψ ) 1 G ψψ Kent (1982) G(θ) = H(θ)J 1 (θ)h(θ) Composite Likelihood Brown, January

12 ... inference ˆθ CL not fully efficient unless G(θ) = H(θ)J 1 (θ)h(θ) = i(θ) efficiency ρ cl(θ) is not a log-likelihood function logliknorm rho.vals Composite Likelihood Brown, January

13 ... inference efficiency of ˆθ CL can be pretty high, in many applications careful choice of weights can improve efficiency of ˆθ CL in special cases weights can be used to incorporate sampling information, including missing data Yi, 12, Molenberghs, 12, Briollais & Choi,12 w CL (ψ) can be re-scaled to. χ 2 q Chandler & Bate (2007), Salvan et al. 11, 12 (wip) log likelihoods rho=0.5, n=10, q=5 log likelihoods rho=0.8, n=10, q= rho rho Composite Likelihood Brown, January

14

15 y N(µ1, σ 2 R), R rs = ρ

16 Model selection Akaike Information Criterion AIC CL = 2cl n (ˆθ CL ) + 2 tr{j(ˆθ)h 1 (ˆθ)} Bayesian Information Criterion Varin & Vidoni (2005) BIC CL = 2cl n (ˆθ CL ) + log(n) tr{j(ˆθ)h 1 (ˆθ)} effective number of parameters Gao & Song (2010) tr{h(θ)g 1 (θ)} = tr{j(θ)h 1 (θ)} G(θ) = H(θ)J 1 (θ)h(θ) Composite Likelihood Brown, January

17 Some surprises Y N(µ, Σ) ˆµ CL = ˆµ, ˆΣ CL = ˆΣ 1 ρ... ρ Y N(µ1, σ 2 ρ 1... ρ R), R = ρ... ρ 1 ˆθCL = ˆθ, G(θ) = i(θ), G(θ) = H(θ)J 1 (θ)h(θ) H(θ) J(θ) H(θ) = var(score), J = E( θ Score) Y (0, R): ˆρ CL ˆρ; a.var(ˆρ CL ) > a.var(ˆρ) efficiency improvement, nuisance parameter is unknown CL can be fully efficient, even if H(θ) J(θ) Mardia et al (2008); Xu, 12 Composite Likelihood Brown, January

18 ... some surprises Godambe information G(θ) can decrease as more component CLs are added pairwise CL can be less efficient than independence CL this can t always be fixed by weighting Xu, 12 parameter constraints can be important Example: binary vector Y, exp(βy j + βy k + θ jk y j y k ) P(Y j = y j, Y k = y k ) {1 + exp(βy j + βy k + θ jk y j y k )} this model is inconsistent parameters may not be identifiable in the CL, even if they are in the full likelihood Yi, 12 Composite Likelihood Brown, January

19 Applications: spatial and space-time data conditional approaches seem more natural condition on neighbours in space condition on small number of lags (in time) some form of blockwise components often proposed Stein et al, 04; Caragea and Smith, 07 fmri time series Kang et al, 12 air pollution and health effects Bai et al, 12 computer experiments: Gaussian process models Xi, 12 spatially correlated extremes joint tail probability known joint density requires combinatorial effort (partial derivatives) composite likelihood based on joint distribution of pairs, triples seems to work well Davison et al, (2012); Genton et al., 12, Ribatet, 12 Composite Likelihood Brown, January

20 Spatial extremes vector observations (X 1i,..., X di ), i = 1,..., n example, wind speed at each of d locations component-wise maxima Z 1,..., Z m ; Z j = max(x j1,..., X jn ) Z j are transformed (centered and scaled) general theory says Pr(Z 1 z 1,..., Z m z m ) = exp{ V (z 1,..., z m )} function V ( ) can be parameterized via Gaussian process models example V (z 1, z 2 ) = z 1 1 Φ{(1/2)a(h) + a 1 (h) log(z 2 /z 1 )} + z 1 2 Φ{(1/2)a(h) + a 1 (h) log(z 1 /z 2 )} Z (h) = (z 1, z 2 ), Z (0) = (0, 0), a(h) = h T Ω 1 h Composite Likelihood Brown, January

21 ... spatial extremes Pr(Z 1 z 1,..., Z d z m ) = exp{ V (z 1,..., z m )} to compute log-likelihood function, need the density combinatorial explosion in computing joint derivatives of V ( ) Davison et al. (2012, Statistical Science) used pairwise composite likelihood compared the fits of several competing models, using AIC analogue described above applied to annual maximum rainfall at several stations near Zurich Composite Likelihood Brown, January

22 Davison et al, 2012 Composite Likelihood Brown, January

23 ... Davison et al, 2012 Composite Likelihood Brown, January

24 Network tomography, Liang & Yu, 2003 X = (X 1,..., X m ) network dynamics e.g. traffic flow counts; node delays Y = (Y 1,..., Y m ) measurement vector m m Y = AX, A known routing matrix, entries 1 m m components X j are independent, with density f j ( ; θ j ) Composite Likelihood Brown, January

25 ... applications time series a case of large m, fixed n need new arguments re consistency, asymptotic normality consecutive pairs: consistent, not asy. normal AR(1): consecutive pairs fully efficient; all pairs terrible (consistent, highly variable) MA(1): consecutive pairs terrible Davis and Yau (2011) genetics: estimation of recombination rate somewhat similar to time series but correlation may not decrease with increasing length suggesting all possible pairs may be inconsistent joint blocks of short sequences seems preferable linkage disequilibrium family based sampling Larribe and Fearnhead (2011); Choi and Briollais, 12 Composite Likelihood Brown, January

26 ... applications Gaussian graphical models Gao and Massam, 12 symmetry constraints have a natural formulation in terms of elements of concentration matrix conditional distribution of y j y ( j) multivariate binary data for multi-neuron spike trains CL as a working likelihood in maximization by parts latent variable models in psychometrics Amari (IMS,12) Bellio, 12 Moustaki, 12, Maydeu-Olivares, 12 many linear and generalized linear models with random effects multivariate survival data... Composite Likelihood Brown, January

27 What don t we know? Design marginal vs. conditional choice of weights down-weighting distant observations choosing blocks and block sizes Uncertainty estimation Ĵ(ˆθ CL ) = var{ cl(θ)/ θ} need replication; need lots of replication perhaps estimate G(ˆθ CL ) or var(ˆθ CL ) directly bootstrap, jackknife Composite Likelihood Brown, January

28 ... what don t we know? Identifiability (1): does there exist a model compatible with a set of marginal or conditional densities? Identifiability (2): what if different components are estimating different parameters? Robustness: CL uses low-dimensional information: is this a type of robustness? find a class of models with same low-d marginals Xu, 12 classical perturbation of starting model (using copulas?) Joe, 12 random effects models might be amenable to theoretical analysis Jordan, 12 asymptotic theory for large m (long vectors of responses), small n relationship to Generalized Estimating Equations Composite Likelihood Brown, January

29 Aspects of robustness model robustness univariate and bivariate margins only for example means, variances, association parameters similar in flavour to generalized estimating equations GEE: mean structure primary computational robustness composite log-likelihood functions are smoother than log-likelihood functions easier to maximize, easier to work with especially in high dimension cases Liang and Yu (2003) robust to missing data mechanisms: Yi, Zeng and Cook (2010) access to multivariate distributions: e.g. mv extremes Davison et al. (2012) Composite Likelihood Brown, January

30 Robustness of consistency working model {f (y; θ); θ Θ} true model g(y) Model Full Likelihood Composite Likelihood Correctly specified f (y; θ 0 ) = g(y) f k (y; θ 0 ) = g k (y) for all k ˆθ ML θ 0 ˆθ CL θ 0 Misspecified f (y; θ) g(y), f k (y; θ) g k (y) for some k ˆθ ML θml ˆθ CL θ Top row efficiency Ximing Xu, U Toronto Composite Likelihood Brown, January

31 ... robustness of consistency working model {f (y; θ); θ Θ} true model g(y) Model Full Likelihood Composite Likelihood Correctly specified f (y; θ 0 ) = g(y) f k (y; θ 0 ) = g k (y) for all k ˆθ ML θ 0 ˆθ CL θ 0 Misspecified f (y; θ) g(y), f k (y; θ) g k (y) for some k ˆθ ML θml ˆθ CL θ Ximing Xu, U Toronto Composite Likelihood Brown, January

32 ... robustness of consistency example (Andrei and Kendziorski, 2009): Y 1 N(µ 1, σ1 2), Y 2 N(µ 2, σ2 2 ), ε N(0, 1) Y 3 = Y 1 + Y 2 + by 1 Y 2 + ɛ full likelihood for multivariate normal is a mis-specified model, ˆb = 0 composite conditional likelihood based on normal distribution for f (Y 3 Y 2, Y 1 ) consistent estimate of b example (Arnold and Xu): f (Y ) = Φ p (Y ; µ, Σ) + g(µ, Σ)( p i=1 Y i1{ Y i t}) sub-distributions of dimension k < p are multivariate normal pairwise likelihood estimator nearly identical to mle assuming incorrect N p (µ; Σ) model Composite Likelihood Brown, January

33 Some dichotomies conditional vs marginal pairwise vs everything else unstructured vs time series/spatial weighted vs unweighted it works vs why does it work? vs when will it not work... Composite Likelihood Brown, January

34

35