SP Experimental Designs - Theoretical Background and Case Study

Transcription

1 SP Experimental Designs - Theoretical Background and Case Study Basil Schmid IVT ETH Zurich Measurement and Modeling FS2016

2 Outline 1. Introduction 2. Orthogonal and fractional factorial designs 3. Efficient designs 4. Pivot designs 5. Testing a design: A case study 6. Conclusions SP Experimental Designs 2

3 Introduction Explain how the variation of certain attributes affects the outcome of interest (causal relationship), applying a statistically efficient and effective framework (maximum amount of information with minimum amount of resources) Kuhfeld (1994): The best approach to design creation is to use the computer as a tool along with traditional design skills, not as a substitute for thinking about the problem SP Experimental Designs 3

4 A brief history : While serving as surgeon on HMS Salisbury, James Lind carried out a systematic clinical trial to compare patients with scurvy (lack of vitamin C disease) Entry requirements to reduce exogenous variation 12 seamen were assigned to 6 treatment groups, each receiving a different diet over a 2-week-period Other examples: Agriculture, marketing, economics Sir Ronald Fisher (1935): Experiments are experience carefully planned in advance, and designed to form a secure basis of new knowledge : manipulation/variation of (existing) attributes formation of attribute levels observation/measurement of outcomes SP Experimental Designs 4

5 Experimental design In contrast to revealed preference (RP) data, stated preference (SP) data are generated by some systematic and planned design process SP data may provide insights into a hypothetical market for which no RP data is available Formulation of statistical hypotheses to be tested Specification of the number of experimental units (observations) required and the population from which they will be sampled Specification of the randomization procedure for assigning the experimental units to the attribute levels: Sources of variation among the units are distributed over the entire experiment Determination of the statistical analysis that will be performed (discrete choice, multivariate regression,...) SP Experimental Designs 5

6 Orthogonal designs x y: Two attribute vectors x and y are said to be (strictly) orthogonal if the inner product is zero cov(x, y) = E(x E(x)) E(y E(y)) = 0 Correlations between attributes are zero and attribute levels appear equally often in combination with all other attribute levels (balance) the effects of interest can be estimated efficient and stochastically independent Full factorial orthogonal design with 2 attributes x and y à 3 levels (3 2 possible combinations; orthogonally coded): Choice set x y SP Experimental Designs 6

7 Fractional factorial designs Full factorial design: Experiment size explodes with increasing attributes and levels. E.g. 10 attributes with 3 levels: 3 10 possible attribute combinations (= degrees of freedom) Full factorial designs are, by definition, perfectly orthogonal in all main-effects and higher order interactions Use an optimal subset of a full factorial Orthogonality can be maintained under the assumption that some effects (often higher order interactions) are zero However, interactions might be highly correlated with main effects: U i = α X tt,i + β X tc,i + γ X tt,i X tc,i + ɛ i (1) U i X tc,i = β + γ X tt,i (2) SP Experimental Designs 7

8 Fractional factorial designs Assume a full factorial design with 10 attributes à 3 levels (3 10 combinations): To estimate all 10 main-effects, one needs at least 20 choice sets (10*(3-1) degrees of freedom) Hence, 45 two-way ((10-1)*10/2) as well as many higher order interactions ( degrees of freedom) are ignored Practical considerations: Main-effects typically account for 70-90% of explained variance, two-way interactions for 5-15% Limit # of attribute levels: Often 2-5 levels Limit # of attributes: Often 6-16 attributes Only allow some two-way interactions to be different from 0 (e.g. travel time x travel cost) Block-design: Divide fractional factorial into groups with the same # of choice sets in a statistically efficient way SP Experimental Designs 8

9 Block-designs Typically, a respondent receives between 6 and 15 choice sets (response burden and cognitive fatigue) Even fractional designs often include more choice sets than what the researcher wants to assign to each respondent Correlation between blocks and attributes should be minimized. Otherwise, one respondent gets all blocks with e.g. high travel times Common mistake: Assign first x choice sets to block b Orthogonal blocking: Block number is uncorrelated with attributes Good news: Most software automatically assign choice sets to each block specified by the researcher SP Experimental Designs 9

10 Some important definitions Unlabeled experiment: A choice experiment where alternatives have no intrinsic meaning (e.g. route 1 vs. route 2) Labeled experiment: A choice experiment where the alternatives are labeled. Model parameters can be estimated for each alternative independently (e.g. car vs. train vs. bus) Generic effect: The same model parameter for all alternatives in the utility function (e.g. travel cost) Alternative-specific effect: Different model parameters for each alternative in the utility function (e.g. travel time car vs. travel time bus vs. travel time train) Own vs. cross effect: If cross effects are present, the IID error assumption is violated SP Experimental Designs 10

11 Example of unlabeled experiment SP Experimental Designs 11

12 Example of labeled experiment SP Experimental Designs 12

13 Orthognonal fraction of full factorial (example) 4 attributes with 3 levels, 3 unlabelled choice alternatives, possible attribute level combinations: Minimum of 8 choice sets to estimate all 4 (generic) main-effects Smallest orthogonal fraction = choice sets Set TT1 TC1 AC1 QU1 TT2 TC2 AC2 QU2 TT3 TC3 AC3 QU redundant alternatives weakly dominant alternatives dominant alternatives SP Experimental Designs 13

14 Problems with orthogonal designs Reasons for moving away from orthogonal designs (OD): For some problems, an OD does not exist (e.g. for limited, by the researcher predefined number of choice sets) in general, ODs require a larger sample size and lead to larger choice sets Behaviorally plausible choice scenarios: ODs may include dominant/weakly dominant/redundant choice sets no information gain When working with preference constraints, orthogonality cannot be maintained Need for more sophisticated approaches: Efficient experimental designs SP Experimental Designs 14

15 Efficient designs: Some basic concepts Efficiency: For given design requirements (violating strict orthogonality), minimize the variances of parameter estimates, which are taken from the variance-covariance matrix of a design D-Efficient GLM Designs: No prior information about the parameter values (signs, magnitude) Efficiency convergence towards orthogonality D-Efficient MNL Designs: Efficiency measures depend on the unknown parameter values one wants to estimate In many cases, one has some sound knowledge about the sign and relative values of the design attributes (e.g. travel cost and travel time both have a negative effect on utility, leading to a positive value of time) SP Experimental Designs 15

16 Example Orthogonal design with travel time and travel cost (2 alternatives, 3 levels): Quadrants 1 and 3 dominate quadrants 2 and 4 SP Experimental Designs 16

17 Example 2 Cost_MIV - Cost_PT Time_MIV - Time_PT WLS Predictions Efficient design with travel time and travel cost (2 alternatives, 3 levels): Elimination of dominant alternatives SP Experimental Designs 17

18 Efficient designs: Some basic concepts Main question: How can the researcher make use of prior information in order to increase the efficiency (minimize standard errors of the attributes, i.e. more robust results) and reduce the sample size requirements? Example 1: Orthogonal designs make no use prior information time and cost attributes are uncorrelated Example 2: Efficient design with no dominant alternatives automatically leads to a negative correlation between time and cost forces respondents to trade-off and increases the amount of preference information given sample size D-Efficient MNL approach: Use expected parameter distributions with µ k and σ k to calculate the optimal design SP Experimental Designs 18

19 D-Efficient GLM designs Find a design matrix Z, with rows selected from a Q x k matrix X where n Q, that is optimal in some sense. Z is an n x k matrix, where k is the number of parameters and n is the number of choice sets in the actual experiment Row-based Federov algorithm (R-package AlgDesign): Selection from a predefined candidature set (after exclusion of dominant/redundant alternatives, etc.) Optimization criterion: Maximize k-th root of the determinant of the normalized dispersion matrix M Ω 1 Assumption: Observations are independent and error terms are normally distributed ( max. Z ) 1 D Efficiency = det Z k n (3) SP Experimental Designs 19

20 D-Efficient MNL designs Asymptotic variance-covariance (AVC) matrix for discrete choice models depends on the true parameter values Starting point: Need to make assumptions about the model, utility functions and parameter values Design matrices Z are created using a column-based swapping algorithm: Selection of attribute levels over all choice situations for each attribute Optimization criteria: Minimize k-th root of the determinant of the AVC matrix Ω ( min. D Error = det Ω(Z, β)) 1 k (4) SP Experimental Designs 20

21 Some remarks on D-Efficient designs Large number of different algorithms and optimization criteria exist (focus on D-Efficiency as most common approach in the literature) Eliminating undesirable choice sets has to be done manually by using preference constraints GLM designs: Can be created in the open-source software R. Robust towards misspecification of priors and often as efficient as MNL designs MNL designs: Created in the commercial software NGENE. Easier to implement, more assistance and possibilities. Priors usually come from the literature, intuition and pre-test studies. Misspecification can be minimized by assuming a random distribution of priors (Bayesian approach) SP Experimental Designs 21

22 An example of a design strategy 9 attributes with 3 levels (3 9 full factorial), 2 labeled alternatives, 32 choice sets with 4 blocks, estimation of all linear main effects, quadratic effects and 6 selected two-way interactions ( degrees of freedom) Polynomial and interaction effects have to be specified in the utility function of a design No weakly dominant alternatives (i.e. all attribute values of one alternative in choice set s are strictly better or equal: a 1 a 2 or a 1 a 2 ) No strongly dominant travel time relative to travel cost alternatives or vice versa (i.e. a 1,cost a 2,cost and a 1,time a 2,time or vice versa) Weak priors to determine the direction of expected effects SP Experimental Designs 22

23 Efficient design (example) 4 attributes with 3 levels, 3 unlabelled choice alternatives, possible attribute level combinations: Minimum of 8 choice sets to estimate all 4 (generic) main-effects Weak priors, exclusion of all dominant choice sets Free choice about the number of choice sets (# choice sets > df ) Set TT1 TC1 AC1 QU1 TT2 TC2 AC2 QU2 TT3 TC3 AC3 QU no more dominant/weakly dominant/redundant choice sets SP Experimental Designs 23

24 Some general remarks Experimental design creation is a research topic on its own (Rose and Bliemer, 2009; Quan et al., 2011) If priors are misspecified, one might run into troubles. Be careful when using priors! Use attributes, values and trade-off variations that are plausible Make sure that there are some overlapping values of generic attributes between alternatives (pivot designs) Order effects: Randomize order of alternatives across respondents in the questionnaire Carefully introduce respondents to the (hypothetical) scenario and explain the attributes you are presenting to them SP Experimental Designs 24

25 Pivot designs It is preferable to base variations around values for observed behavior (state-of-the-art in transportation research): Calculate design with placeholder values (e.g. 1,2,3) and replace them by relative changes (e.g. 0.7, 1.1, 1.5) Combination of RP data with variations given by the design respondents can better identify with the presented choice scenarios; much more variation in the attribute levels Possible to include one reference alternative in the choice sets (e.g. bike travel time, whose value is not varied) Problems: If reference values are (highly) dominant, the respondents will more likely choose the respective alternative (only in labeled experiments) Correlation between attributes; skewness SP Experimental Designs 25

26 Pivot designs: Trade-off distribution Example where MIV is often cheaper and faster than PT modification of reference values needed! SP Experimental Designs 26

27 Testing a design: A case study Once you have your design, you should test the performance of estimating the coefficients of interest, based on simulation of a more or less hypothetical population Define priors for the attribute weights of the utility function based on recent similar studies Simulate error structure (GEV) for the utility function taking into account the panel structure of the designs Calculate individual utilities and determine the chosen alternatives for each simulated subject Estimate the parameters for the simulated data and compare the results with the a-priori assumptions SP Experimental Designs 27

28 Testing a design: Pivot approach Experimental design: 9 attributes with 3 levels (3 9 full factorial), 2 labeled alternatives, 32 choice sets (8 per subject) Reference values taken from a Swiss mode choice experiment: Total travel time: For PT alternative = travel time without access and egress time; for MIV alternative = travel time + parking search time Total travel cost: For PT alternative = ticket price; for MIV alternative = fuel cost + parking cost Number of transfers: PT alternative only Attribute Effect code: Travel time (MIV and PT) [%] Travel cost (MIV and PT) [%] Delay prob. (MIV and PT) [%] Walking / waiting time (MIV and PT) [min.] Number of transfers (PT) [#] SP Experimental Designs 28

29 Testing a design: A-priori Coefficients Prior values for the individual weights of attribute k, β ik N(µ k, σ k ), and alternative-specific constants are simulated based on results obtained from the linear model in the BMVI Zeitkostenstudie (Axhausen et al., 2014) For each simulated individual i, the same β ik is used over all 8 choice sets, representing the panel structure of the experiment Coefficient Mean SD Type ASC MIV Alternative-specific β timemiv Alternative-specific β timept Alternative-specific β cost Generic β delaymiv Alternative-specific β delaypt Alternative-specific β walk Generic β transferspt Alternative-specific VOT MIV VOT PT 15.0 CHF / h 14.2 CHF / h Number of simulated 400 coefficient vectors β ik SP Experimental Designs 29

30 Testing a design: Utility Function The random utility model framework (RUM) assumes that in each choice set s, individual i perceives utility U ijs for each alternative j among the full set of alternatives J (MIV and PT), given the attributes X ijs, and chooses the one that maximizes utility. U ijs has an observed component V ijs and an unobserved component ɛ ijs : U ijs = V ijs + ɛ ijs (5) where and ɛ ijs GEV (0, 1, 0) (6) K V ijs = β ik X ijsk (7) k=1 SP Experimental Designs 30

31 Testing a design: Choice simulation The chosen alternatives choice is are calculated as follows: if U is,miv > U is,pt : choice is = { MIV else PT (8) Snippet of a simulated discrete choice data set: id set alt choice block time cost walk delay transfers (min.) (CHF) (min.) (min.) (#) MIV PT MIV PT MIV SP Experimental Designs 31

32 Testing a design: Estimation For given randomly drawn subsets of RP data, simulated β ik coefficient vectors and simulated error terms ɛ ijs, the models are estimated for 3 different designs. This process is repeated 2000 times to get insights into the distributions of coefficients (robustness), variances (precision) and values of time The between-design differences of E(β k ) and E(SE k ) with respect to the a-priori parameters are small Design approach: GLM MNL: β 0 MNL: β k E(β k ) E(SE k ) E(β k ) E(SE k ) E(β k ) E(SE k ) ASC MIV * β timemiv β timept β cost β delaymiv * β delaypt * β walk * β transferspt VOT MIV VOT PT SP Experimental Designs 32

33 Conclusions No substantial differences between the different design approaches: Designs are robust and reproduce the a-priori values well From a behavioral perspective, one should always exclude dominant and weakly dominant alternatives! Personal suggestion: Create an efficient design by... carefully thinking about your research question and aims assigning about 8 choice sets to a respondents and using a block-design (total # of choice sets 1.5 df ) using MNL approach with zero (or weak) priors, excluding undesired choice sets by manually setting preference conditions updating your design after a pre-test study SP Experimental Designs 33