Bridging the two cultures: Latent variable statistical modeling with boosted regression trees

Transcription

1 Bridging the two cultures: Latent variable statistical modeling with boosted regression trees Thomas G. Dietterich and Rebecca Hutchinson Oregon State University Corvallis, Oregon, USA 1

2 A Species Distribution Modeling Problem: ebird data 12 bird species 3 synthetic species 3124 observations from New York State, May-July covariates 2

3 Two Cultures Probabilistic Graphical Models Occupancy Models MacKenzie, et al., 2002 Flexible Nonparametric Models Boosted Regression Trees Friedman, 2001 Elith et al, 2006 Elith, Leathwick & Hastie,

4 Occupancy-Detection Model Occupancy features (e.g. elevation, vegetation) Probability of occupancy (function of X i ) Probability of detection (function of W it ) Detection features (e.g. time of day, effort) o i d it X i Z i Y it W it t=1,,t i=1,,m Visits Sites True (latent) presence/absence Z i ~ Bern(o i ) Observed presence/absence Y it Z i ~ Bern(Z i d it ) 4 MacKenzie, et al, 2006

5 Parameterizing the model X i o i Z i Y it d it W it t=1,,t i=1,,m Z i ~P(Z i X i ): Species Distribution Model P Z i = 1 X i = o i = F(X i ) occupancy probability y it ~P(y it z i, w it ): Observation model P Y it = 1 Z i, W it = Z i d it d it = G(W it ) detection probability 5

6 Standard Approach: Log Linear (logistic regression) models log F X i 1 F X i = β 0 + β 1 X i1 + + β J X ij log G W it 1 G W it Fit via maximum likelihood = α 0 + α 1 W it1 + + α K W itk Can apply hypothesis tests to assess importance of covariates H 0 : β 1 = 0 H a : β 1 > 0 6

7 Results on Synthetic Species with Nonlinear Interactions Predictions exhibit high variance because model cannot fit the nonlinearities well 7

8 A Flexible Predictive Model Predict the observation y it from the combination of occupancy covariates x i and detection covariates w it Boosted Regression trees log P Y it=1 X i,w it P Y it =0 X i,w it = β 1 tree 1 X i, W it + + β L tree L (X i, W it ) Fitted via functional gradient descent Model complexity is tuned to the complexity of the data Number of trees Depth of each tree 8

9 P(Z) Results Systematically biased because it does not capture the latent occupancy Underestimates occupancy at occupied sites to fit detection failures Much lower variance than the Occupancy-Detection model, because it can handle the non-linearities 9

10 Two Cultures: Summary Probabilistic Graphical Models Flexible Nonparametric Models Advantages Supports latent variables Supports hypothesis tests on meaningful parameters Disadvantages Model must be carefully designed (interactions? non-linearities?) Data must be transformed to match modeling assumptions (linearity, Gaussianity) Model has fixed complexity so either under-fits or over-fits Advantages Model complexity adapts to data complexity Easy to use off-the-shelf Disadvantages Cannot support latent variables Cannot provide parametric hypothesis tests 10

11 The Dream Probabilistic Graphical Models Flexible Nonparametric Models Flexible Nonparametric Probabilistic Models 11

12 A Simple Idea: Parameterize F and G as boosted trees log log F X 1 F X = f0 (X) + ρ 1 f 1 (X) + + ρ L f L (X) G W 1 G W = g0 W + η 1 g 1 W + + η L g L (W) Perform functional gradient descent in F and G 12

13 Results: OD-BRT Occupancy probabilities are predicted very well 13

14 Interpreting Non-Parametric Models: Partial Dependence Plots Simulate manipulating one variable (e.g., Distance of Survey) Visualize the predicted response 14 Distance of survey

15 Partial Dependence Plot Synthetic Species 3 OD-BRT correctly captures the bimodal detection probability 15 Time of Day

16 Partial Dependence Plot Blue Jay vs. Time of Day 16 Time of Day

17 Partial Dependence Plot Blue Jay vs. Duration of Observation 17 Effort in Hours

18 Summary: We can have our cake (latent variables, interpretable submodels) and eat it too (have flexible, easy-to-use modeling tools) Probabilistic Graphical Models Flexible Nonparametric Models Flexible Nonparametric Probabilistic Models Easier to use More accurate 18

19 accuracy Concluding Remarks With limited data, the most accurate predictive model is much simpler than the true model Predictive accuracy on a single data set is not a sufficient criterion for a scientific model Most Accurate Predictive Model Predictive Accuracy of True Model complexity 19

20 Acknowledgements Liping Liu: Boosted Regression Trees in OD models Steve Kelling and colleagues at the Cornell Lab of Ornithology National Science Foundation Grants , , , and

21 21

22 Supporting Materials 22

23 Regression Trees Interactions are captured by the if-then-else structure of the tree Nonlinearities are approximated by piecewise constant functions Tree can be flattened into a linear model: Y 1 = 5 X 1 3 X 2 0 X 2 0 Y 1 = 3 Y 1 = 8 Y 1 = 1 Y 1 = 5 I X 1 3, X I X 1 3, X 2 < I x 1 < 3, X I(X 1 < 3, X 2 < 0) 23

24 Functional Gradient Descent Boosted Regression Trees Friedman (2000), Mason et al. (NIPS 1999), Breiman (1996) Fit a logistic regression model as a weighted sum of regression trees: log P Y = 1 P Y = 0 = tree0 (X) + η 1 tree 1 (X) + + η L tree L (X) When flattened this gives a log linear model with complex interaction terms 24

25 L2-Tree Boosting Algorithm Let F 0 X = f 0 (X) = 0 be the zero function For l = 1,, L do Construct a training set S l = X i, Y i i=1 where Y is computed as Y i LL F = how we wish F would change at X i F F=F l 1 X i Let f l = regression tree fit to S l F l F l 1 + η l f l The step sizes η l are the weights computed in boosting This provides a general recipe for learning a conditional probability distribution for a Bernoulli or multinomial random variable N 25

26 Alternating Functional Gradient Descent Loss function L(F, G, y) F 0 = G 0 = f 0 = g 0 = 0 For l = 1,, L For each site i compute zi = L(F l 1 x i, G l 1, y i )/ F l 1 x i Fit regression tree f l M to x i, zi i=1 Let F l = F l 1 + ρ l f l For each visit t to site i, compute y it = L F l x i, G l 1 w it, y it / Gl 1 w it Fit regression tree g l to w it, y it Let G l = G l 1 + η l g l M,T i i=1,t=1 26 Hutchinson, Liu, Dietterich, AAAI 2011

27 Multiple Visit Data Site A (forest, elev=400m) B (forest, elev=500m) C (forest, elev=300m) D (grassland, elev=200m) True occupancy (latent) Visit 1 (rainy day, 12pm) Detection History Visit 2 (clear day, 6am) Visit 3 (clear day, 9am)

28 Covariates 28

29 Synthetic Species 2 F and G nonlinear o i log = 2 x o i + 3 xi 2x i i d it log = exp( 0.5w 4 1 d it ) + sin(1.25w 1 it + 5) it 29

30 Predicting Occupancy Synthetic Species 2 30

31 Open Problems Sometimes the OD model finds trivial solutions Detection probability = 0 at many sites, which allows the Occupancy model complete freedom at those sites Occupancy probability constant (0.2) Log likelihood for latent variable models suffers from local minima Proper initialization? Proper regularization? Posterior regularization? How much data do we need to fit this model? Can we detect when the model has failed? 31