Nonparametric Bayesian modeling for dynamic ordinal regression relationships

Transcription

1 Nonparametric Bayesian modeling for dynamic ordinal regression relationships Athanasios Kottas Department of Applied Mathematics and Statistics, University of California, Santa Cruz Joint work with Maria DeYoreo (Department of Statistical Science, Duke University) 10th Conference on Bayesian Nonparametrics NCSU, Raleigh, NC June 22-26, / 23

2 Motivation Regression modeling for one or more ordinal categorical responses recorded over discrete time Focus on applications, including problems in ecology and the environmental sciences, where it is natural/necessary to model the joint stochastic mechanism for the response(s) and covariates Motivating application: study of dynamically evolving natural selection surfaces in evolutionary biology Data example: maturity (recorded on an ordinal scale),, and for Chilipepper rockfish, collected over 15 years along the coast of California 2 / 23

3 Motivation Regression modeling for one or more ordinal categorical responses recorded over discrete time Focus on applications, including problems in ecology and the environmental sciences, where it is natural/necessary to model the joint stochastic mechanism for the response(s) and covariates Motivating application: study of dynamically evolving natural selection surfaces in evolutionary biology Data example: maturity (recorded on an ordinal scale),, and for Chilipepper rockfish, collected over 15 years along the coast of California 2 / 23

4 Motivation Regression modeling for one or more ordinal categorical responses recorded over discrete time Focus on applications, including problems in ecology and the environmental sciences, where it is natural/necessary to model the joint stochastic mechanism for the response(s) and covariates Motivating application: study of dynamically evolving natural selection surfaces in evolutionary biology Data example: maturity (recorded on an ordinal scale),, and for Chilipepper rockfish, collected over 15 years along the coast of California 2 / 23

5 Modeling through latent continuous responses Assume each ordinal response represents a discretized version of an underlying latent continuous response 1 k ordinal variables Y = (Y 1,..., Y k ), with y j (continuous) covariates X = (X 1,..., X p ) {1,..., C j }, and p Assume Y j = l if-f γ j,l 1 < Z j γ j,l, for j = 1,..., k, and l = 1,..., C j (with γ j,0 = and γ j,cj = ) Multivariate normal distribution for Z = (Z 1,..., Z k ) multivariate ordinal probit model symmetric, unimodal latent response distribution with mean x T β implies restrictive effects of covariates on the probability response curves computational challenges in estimating cut-off points 1 e.g., Albert and Chib, / 23

6 Modeling through latent continuous responses Assume each ordinal response represents a discretized version of an underlying latent continuous response 1 k ordinal variables Y = (Y 1,..., Y k ), with y j (continuous) covariates X = (X 1,..., X p ) {1,..., C j }, and p Assume Y j = l if-f γ j,l 1 < Z j γ j,l, for j = 1,..., k, and l = 1,..., C j (with γ j,0 = and γ j,cj = ) Multivariate normal distribution for Z = (Z 1,..., Z k ) multivariate ordinal probit model symmetric, unimodal latent response distribution with mean x T β implies restrictive effects of covariates on the probability response curves computational challenges in estimating cut-off points 1 e.g., Albert and Chib, / 23

7 Modeling through latent continuous responses Assume each ordinal response represents a discretized version of an underlying latent continuous response 1 k ordinal variables Y = (Y 1,..., Y k ), with y j (continuous) covariates X = (X 1,..., X p ) {1,..., C j }, and p Assume Y j = l if-f γ j,l 1 < Z j γ j,l, for j = 1,..., k, and l = 1,..., C j (with γ j,0 = and γ j,cj = ) Multivariate normal distribution for Z = (Z 1,..., Z k ) multivariate ordinal probit model symmetric, unimodal latent response distribution with mean x T β implies restrictive effects of covariates on the probability response curves computational challenges in estimating cut-off points 1 e.g., Albert and Chib, / 23

8 Objectives For univariate responses, more general methods have been explored, relaxing either the distributional or linearity assumption 1 In the multivariate setting, complications arise from issues of constrained covariance matrices and inference for the cut-offs, and methods for general Bayesian inference are limited In contrast to semiparametric approaches, our aim is flexible modeling and inference for the ordinal regression relationships and for the response distribution 1 e.g., Newton et al., 1996; Mukhopadhyay and Gelfand, 1997; Denison et al., 2002; Chib and Greenberg, / 23

9 Objectives For univariate responses, more general methods have been explored, relaxing either the distributional or linearity assumption 1 In the multivariate setting, complications arise from issues of constrained covariance matrices and inference for the cut-offs, and methods for general Bayesian inference are limited In contrast to semiparametric approaches, our aim is flexible modeling and inference for the ordinal regression relationships and for the response distribution 1 e.g., Newton et al., 1996; Mukhopadhyay and Gelfand, 1997; Denison et al., 2002; Chib and Greenberg, / 23

10 Objectives For univariate responses, more general methods have been explored, relaxing either the distributional or linearity assumption 1 In the multivariate setting, complications arise from issues of constrained covariance matrices and inference for the cut-offs, and methods for general Bayesian inference are limited In contrast to semiparametric approaches, our aim is flexible modeling and inference for the ordinal regression relationships and for the response distribution 1 e.g., Newton et al., 1996; Mukhopadhyay and Gelfand, 1997; Denison et al., 2002; Chib and Greenberg, / 23

11 The nonparametric mixture model We use a version of implied conditional regression 1 modeling the joint latent response-covariate distribution f (z, x) inference for f (z x), and for Pr(Y x), implied through f (z, x) and f (x) Dirichlet Process (DP) mixture model for f (z, x): f (z, x G) = N(z, x µ, Σ)dG(µ, Σ), G α, ψ DP(α, G 0 ( ψ)) DP constructive definition 2 : f (z, x G) = r=1 p rn(z, x µ r, Σ r ), where θ r = (µ r, Σ r ) iid G 0, and the weights p 1, p 2,... are determined through stick-breaking: stick-breaking proportions β s iid beta(1, α), s = 1, 2,... p 1 = β 1, and p r = β r 1 r m=1 (1 βm), for r = 2, 3,... 1 Nadaraya, 1964; Watson, 1964; Müller et al., Sethuraman, / 23

12 The nonparametric mixture model We use a version of implied conditional regression 1 modeling the joint latent response-covariate distribution f (z, x) inference for f (z x), and for Pr(Y x), implied through f (z, x) and f (x) Dirichlet Process (DP) mixture model for f (z, x): f (z, x G) = N(z, x µ, Σ)dG(µ, Σ), G α, ψ DP(α, G 0 ( ψ)) DP constructive definition 2 : f (z, x G) = r=1 p rn(z, x µ r, Σ r ), where θ r = (µ r, Σ r ) iid G 0, and the weights p 1, p 2,... are determined through stick-breaking: stick-breaking proportions β s iid beta(1, α), s = 1, 2,... p 1 = β 1, and p r = β r 1 r m=1 (1 βm), for r = 2, 3,... 1 Nadaraya, 1964; Watson, 1964; Müller et al., Sethuraman, / 23

13 The nonparametric mixture model We use a version of implied conditional regression 1 modeling the joint latent response-covariate distribution f (z, x) inference for f (z x), and for Pr(Y x), implied through f (z, x) and f (x) Dirichlet Process (DP) mixture model for f (z, x): f (z, x G) = N(z, x µ, Σ)dG(µ, Σ), G α, ψ DP(α, G 0 ( ψ)) DP constructive definition 2 : f (z, x G) = r=1 p rn(z, x µ r, Σ r ), where θ r = (µ r, Σ r ) iid G 0, and the weights p 1, p 2,... are determined through stick-breaking: stick-breaking proportions β s iid beta(1, α), s = 1, 2,... p 1 = β 1, and p r = β r 1 r m=1 (1 βm), for r = 2, 3,... 1 Nadaraya, 1964; Watson, 1964; Müller et al., Sethuraman, / 23

14 Ordinal regression functions Flexible model for f (z, x) flexible inference for Pr(Y x) Implied regression functions provide a nonparametric extension of probit regression (with random covariates): Pr(Y = (l 1,..., l k ) x; G) = r=1 γk,lk γ1,l1 w r (x) N(z m r (x), S r )dz γ k,lk 1 γ 1,l1 1 with covariate dependent weights w r(x) p rn(x µ x r, Σxx r ) and covariate dependent probabilities, where m r(x) = µ z r+σ zx r (Σ xx µ x r) and S r = Σ zz r Σ zx r (Σ xx r ) 1 Σ xz r r ) 1 (x 6 / 23

15 Ordinal regression functions Flexible model for f (z, x) flexible inference for Pr(Y x) Implied regression functions provide a nonparametric extension of probit regression (with random covariates): Pr(Y = (l 1,..., l k ) x; G) = r=1 γk,lk γ1,l1 w r (x) N(z m r (x), S r )dz γ k,lk 1 γ 1,l1 1 with covariate dependent weights w r(x) p rn(x µ x r, Σxx r ) and covariate dependent probabilities, where m r(x) = µ z r+σ zx r (Σ xx µ x r) and S r = Σ zz r Σ zx r (Σ xx r ) 1 Σ xz r r ) 1 (x 6 / 23

16 Model properties Provided C j > 2, both µ and Σ are identifiable in the induced mixture kernel for (Y, X), under fixed cut-off points The prior model has large support again under fixed cut-offs it assigns positive probability to all Kullback-Leibler (KL) neighborhoods of a mixed ordinal-continuous distribution, p 0(x, y), as well as to all KL neighborhoods of the implied conditional distribution, p 0(y x) Identifiability result + KL property obtained under fixed cut-offs computational advant over parametric models 7 / 23

17 Model properties Provided C j > 2, both µ and Σ are identifiable in the induced mixture kernel for (Y, X), under fixed cut-off points The prior model has large support again under fixed cut-offs it assigns positive probability to all Kullback-Leibler (KL) neighborhoods of a mixed ordinal-continuous distribution, p 0(x, y), as well as to all KL neighborhoods of the implied conditional distribution, p 0(y x) Identifiability result + KL property obtained under fixed cut-offs computational advant over parametric models 7 / 23

18 Model properties Provided C j > 2, both µ and Σ are identifiable in the induced mixture kernel for (Y, X), under fixed cut-off points The prior model has large support again under fixed cut-offs it assigns positive probability to all Kullback-Leibler (KL) neighborhoods of a mixed ordinal-continuous distribution, p 0(x, y), as well as to all KL neighborhoods of the implied conditional distribution, p 0(y x) Identifiability result + KL property obtained under fixed cut-offs computational advant over parametric models 7 / 23

19 Model properties Interactions and dependence between covariates are implicit in joint modeling framework Inference for inverse relationships covariate distribution across ordinal responses values, f (x Y = y) Model can accommodate directly continuous covariates as well as discrete covariates which have some ordering modification of methodology to handle nominal categorical covariates 8 / 23

20 Model properties Interactions and dependence between covariates are implicit in joint modeling framework Inference for inverse relationships covariate distribution across ordinal responses values, f (x Y = y) Model can accommodate directly continuous covariates as well as discrete covariates which have some ordering modification of methodology to handle nominal categorical covariates 8 / 23

21 Model properties Interactions and dependence between covariates are implicit in joint modeling framework Inference for inverse relationships covariate distribution across ordinal responses values, f (x Y = y) Model can accommodate directly continuous covariates as well as discrete covariates which have some ordering modification of methodology to handle nominal categorical covariates 8 / 23

22 Hierarchical model for the data y ij = l iff γ j,l 1 < z ij γ j,l, i = 1,..., n, j = 1,..., k ind. (z i, x i ) {µ l, Σ l }, L i N(µ Li, Σ Li ), N L i p iid p l δ l (L i ), l=1 i = 1,..., n i = 1,..., n p α GD((1, 1,..., 1), (α, α,..., α)) (µ l, Σ l ) ψ iid N(µ l ; m, V)IW(Σ l ; ν, S), l = 1,..., N and the full model is completed with conditionally conjugate priors on ψ = (m, V, S) and α 9 / 23

23 Ozone concentration data example Data set comprising 111 measurements of ozone concentration (ppb), wind speed (mph), radiation (langleys), and temperature (degrees Fahrenheit) Ozone concentration recorded on continuous scale To construct an ordinal response: define high as above 100 ppb, medium as (50, 100] ppb, and low as less than 50 ppb Comparison of inferences from the model for (Y, X) with those from a DP mixture of normals model for (Z, X) 10 / 23

24 Ozone concentration data example Data set comprising 111 measurements of ozone concentration (ppb), wind speed (mph), radiation (langleys), and temperature (degrees Fahrenheit) Ozone concentration recorded on continuous scale To construct an ordinal response: define high as above 100 ppb, medium as (50, 100] ppb, and low as less than 50 ppb Comparison of inferences from the model for (Y, X) with those from a DP mixture of normals model for (Z, X) 10 / 23

25 Ozone concentration data example Data set comprising 111 measurements of ozone concentration (ppb), wind speed (mph), radiation (langleys), and temperature (degrees Fahrenheit) Ozone concentration recorded on continuous scale To construct an ordinal response: define high as above 100 ppb, medium as (50, 100] ppb, and low as less than 50 ppb Comparison of inferences from the model for (Y, X) with those from a DP mixture of normals model for (Z, X) 10 / 23

26 Ozone data pr(low) pr(medium) pr(high) radiation radiation radiation temperature temperature temperature wind speed wind speed wind speed Figure: Posterior mean (solid) and 95% interval estimates (dashed) for Pr(Y = l x m; G) (black) compared to Pr(γ l 1 < Z γ l x m; G) (red). 11 / 23

27 Ozone data pr(low) pr(medium) pr(high) temperature temperature temperature radiation radiation radiation Figure: Posterior mean estimates for Pr(Y = l x 1, x 2 ; G), for l = 1, 2, 3, corresponding to low (left), medium (middle) and high (right). Red represents a value of 1, white represents / 23

28 Extension to dynamic ordinal regression modeling Focusing on a univariate ordinal response, we seek to extend to a model for Pr t (Y x), for t T = {1, 2,... } Build on the earlier framework by extending to a prior model for {f (z, x G t ) : t T }, and thus for {Pr(Y x; G t ) : t T } Motivating application: data from NMFS on female Chilipepper rockfish collected between 1993 and 2007 along the coast of California three ordinal levels for maturity: immature (1), pre-spawning mature (2), and post-spawning mature (3) measured in millimeters recorded on an ordinal scale: j implies the fish was between j and j + 1 years of (data range 1 to 25) incorporate into the model in the same fashion with the maturity variable 13 / 23

29 Extension to dynamic ordinal regression modeling Focusing on a univariate ordinal response, we seek to extend to a model for Pr t (Y x), for t T = {1, 2,... } Build on the earlier framework by extending to a prior model for {f (z, x G t ) : t T }, and thus for {Pr(Y x; G t ) : t T } Motivating application: data from NMFS on female Chilipepper rockfish collected between 1993 and 2007 along the coast of California three ordinal levels for maturity: immature (1), pre-spawning mature (2), and post-spawning mature (3) measured in millimeters recorded on an ordinal scale: j implies the fish was between j and j + 1 years of (data range 1 to 25) incorporate into the model in the same fashion with the maturity variable 13 / 23

30 Extension to dynamic ordinal regression modeling Focusing on a univariate ordinal response, we seek to extend to a model for Pr t (Y x), for t T = {1, 2,... } Build on the earlier framework by extending to a prior model for {f (z, x G t ) : t T }, and thus for {Pr(Y x; G t ) : t T } Motivating application: data from NMFS on female Chilipepper rockfish collected between 1993 and 2007 along the coast of California three ordinal levels for maturity: immature (1), pre-spawning mature (2), and post-spawning mature (3) measured in millimeters recorded on an ordinal scale: j implies the fish was between j and j + 1 years of (data range 1 to 25) incorporate into the model in the same fashion with the maturity variable 13 / 23

31 Rockfish data t=1993 t=1994 t= t= t= t= t= t= t= t= t= t= Figure: Bivariate plots of versus at each year of data, with data points colored according to maturity level: red level 1; green level 2; blue level / 23

32 DDP model extension To retain model properties at each t, use DDP prior for {G t : t T } 1 Time-dependent weights and atoms: f (z, x G t ) = r=1 { (1 β r,t ) r 1 m=1 β m,t } N(z, x µ r,t, Σ r ) Stochastic process with beta(α, 1) marginals for the {β r,t : t T }: B = { ( β t = exp ζ2 + ηt 2 ) } : t T 2α where ζ N(0, 1) and, independently, {η t : t T } is an AR(1) process with N(0, 1) marginals (η t η t 1, φ N(φη t 1, 1 φ 2 ), with φ < 1) Vector autoregressive model for the {µ r,t : t T } 1 MacEachern, 2000; Taddy, 2010; Nieto-Barajas et al., / 23

33 DDP model extension To retain model properties at each t, use DDP prior for {G t : t T } 1 Time-dependent weights and atoms: f (z, x G t ) = r=1 { (1 β r,t ) r 1 m=1 β m,t } N(z, x µ r,t, Σ r ) Stochastic process with beta(α, 1) marginals for the {β r,t : t T }: B = { ( β t = exp ζ2 + ηt 2 ) } : t T 2α where ζ N(0, 1) and, independently, {η t : t T } is an AR(1) process with N(0, 1) marginals (η t η t 1, φ N(φη t 1, 1 φ 2 ), with φ < 1) Vector autoregressive model for the {µ r,t : t T } 1 MacEachern, 2000; Taddy, 2010; Nieto-Barajas et al., / 23

34 DDP model extension To retain model properties at each t, use DDP prior for {G t : t T } 1 Time-dependent weights and atoms: f (z, x G t ) = r=1 { (1 β r,t ) r 1 m=1 β m,t } N(z, x µ r,t, Σ r ) Stochastic process with beta(α, 1) marginals for the {β r,t : t T }: B = { ( β t = exp ζ2 + ηt 2 ) } : t T 2α where ζ N(0, 1) and, independently, {η t : t T } is an AR(1) process with N(0, 1) marginals (η t η t 1, φ N(φη t 1, 1 φ 2 ), with φ < 1) Vector autoregressive model for the {µ r,t : t T } 1 MacEachern, 2000; Taddy, 2010; Nieto-Barajas et al., / 23

35 DDP model extension To retain model properties at each t, use DDP prior for {G t : t T } 1 Time-dependent weights and atoms: f (z, x G t ) = r=1 { (1 β r,t ) r 1 m=1 β m,t } N(z, x µ r,t, Σ r ) Stochastic process with beta(α, 1) marginals for the {β r,t : t T }: B = { ( β t = exp ζ2 + ηt 2 ) } : t T 2α where ζ N(0, 1) and, independently, {η t : t T } is an AR(1) process with N(0, 1) marginals (η t η t 1, φ N(φη t 1, 1 φ 2 ), with φ < 1) Vector autoregressive model for the {µ r,t : t T } 1 MacEachern, 2000; Taddy, 2010; Nieto-Barajas et al., / 23

36 Hierarchical model for the data y t,i = j γ j 1 < z t,i γ j, t s c, i = 1,..., n t u t,i = j log(j) < w t,i log(j + 1), t s c, i = 1,..., n t {y t,i = (z t,i, w t,i, x t,i )} {µ l,t }, {Σ l }, {L t,i } n t N(µ Lt,i,t, Σ Lt,i ) t s c η l,t η l,t 1, φ N(φη l,t 1, 1 φ 2 ), i=1 {L t,i } {η l,t }, {ζ l } n t t s c i=1 l=1 N p l,t δ l (L t,i ) {ζ l }, {η l,1 } ind. N(0, 1), l = 1,..., N 1 l = 1,..., N 1, t = 2,..., T µ l,1 m 0, V 0 N(m 0, V 0 ), l = 1,..., N µ l,t µ l,t 1, Θ, m, V N(m + Θµ l,t 1, V), l = 1,..., N, t = 2,..., T with priors on α, φ, Θ, and ψ = (m, V, D) Σ l ν, D iid IW(Σ l ; ν, D), l = 1,..., N 16 / 23

37 Rockfish data t= t= t= t= t= t= t= t= t= t= t= t= t= t= t= Figure: Posterior mean estimates for the bivariate density of and across all years. 17 / 23

38 Rockfish data t=1993 t=2000 t= Figure: Posterior mean and 95% interval bands for the expected value of over (continuous), across three years. Overlaid are the data (in blue) and the estimated von Bertalanffy growth curves (in red). 18 / 23

39 Rockfish data t=1993 t=1994 t=1995 pr t (y x) pr t (y x) pr t (y x) t=1996 t=1997 t=1998 pr t (y x) pr t (y x) pr t (y x) t=1999 t=2000 t=2001 pr t (y x) pr t (y x) pr t (y x) t=2002 t=2003 t=2004 pr t (y x) pr t (y x) pr t (y x) Figure: Posterior mean and 95% interval bands for the ordinal probability curves associated with : immature (solid); pre-spawning mature (dashed); post-spawning mature (dotted). 19 / 23

40 Rockfish data t=1993 t=1994 t=1995 pr t (y u*) pr t (y u*) pr t (y u*) t=1996 t=1997 t=1998 pr t (y u*) pr t (y u*) pr t (y u*) t=1999 t=2000 t=2001 pr t (y u*) pr t (y u*) pr t (y u*) t=2002 t=2003 t=2004 pr t (y u*) pr t (y u*) pr t (y u*) Figure: Posterior mean and 95% interval bands for the ordinal probability curves associated with : immature (solid); pre-spawning mature (dashed); post-spawning mature (dotted). 20 / 23

41 Rockfish data at 90% maturity at 90% maturity year year Figure: Posterior mean and 90% intervals for the smallest value of above 2 years at which probability of maturity first exceeds 90% (left), and similar inference for (right). 21 / 23

42 Conclusions Modeling framework for ordinal regression problems with a small to moderate number of covariates, and for settings where modeling the joint response-covariate distribution is appropriate (or necessary) DeYoreo, M. & Kottas, A. (2015). "Bayesian nonparametric modeling for multivariate ordinal regression." (under review) DeYoreo, M. & Kottas, A. (2015). "Modeling for dynamic ordinal regression relationships: An application to estimating maturity of rockfish in California." (submitted for publication) Binary responses require a different model due to identifiability constraints DeYoreo, M. & Kottas, A. (2015). "A fully nonparametric modeling approach to binary regression." (revised) 22 / 23

43 Conclusions Modeling framework for ordinal regression problems with a small to moderate number of covariates, and for settings where modeling the joint response-covariate distribution is appropriate (or necessary) DeYoreo, M. & Kottas, A. (2015). "Bayesian nonparametric modeling for multivariate ordinal regression." (under review) DeYoreo, M. & Kottas, A. (2015). "Modeling for dynamic ordinal regression relationships: An application to estimating maturity of rockfish in California." (submitted for publication) Binary responses require a different model due to identifiability constraints DeYoreo, M. & Kottas, A. (2015). "A fully nonparametric modeling approach to binary regression." (revised) 22 / 23

44 Conclusions Modeling framework for ordinal regression problems with a small to moderate number of covariates, and for settings where modeling the joint response-covariate distribution is appropriate (or necessary) DeYoreo, M. & Kottas, A. (2015). "Bayesian nonparametric modeling for multivariate ordinal regression." (under review) DeYoreo, M. & Kottas, A. (2015). "Modeling for dynamic ordinal regression relationships: An application to estimating maturity of rockfish in California." (submitted for publication) Binary responses require a different model due to identifiability constraints DeYoreo, M. & Kottas, A. (2015). "A fully nonparametric modeling approach to binary regression." (revised) 22 / 23

45 MANY THANKS!!! 23 / 23