Properties of Preference Questions: Utility Balance, Choice Balance, Configurators, and M-Efficiency

Transcription

1 Ppeties f Pefeence Questins: Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency by Jhn R. Hause and Olivie Tubia August, 3 Jhn R. Hause is the iin Pfess f Maketing, Slan Schl f Management, Massachusetts Institute f Technlgy, E56-34, 38 Memial Dive, Cambidge, MA 4, (67) 53-99, fax (67) , jhause@mit.edu. Olivie Tubia is a gaduate student at the Maketing Gup, Massachusetts Institute f Technlgy, E56-345, 38 Memial Dive, Cambidge, MA 4, (67) , fax (67) , tubia@mit.edu. This eseach was suppted by the MIT Slan Schl f Management and MIT s Cente f Innvatin in Pduct Develpment. This pape may be dwnladed fm the Vitual Custme Initiative s website, mitslan.mit.edu/vc. This website als cntains demnstatins f many f the pefeence measuement questins discussed in this pape, including cnfiguats. Special thanks t Eic Badlw, Theds Evgeniu, and Duncan Simeste f insightful cmments and Ely Dahan wh was instumental in develping the cnfiguat with which we cllected the data f the stylized fact n wam-up questins. Lim Weisbeg pvided the web design f the cnfiguat and the examples f paied-cmpaisn and chice-based questins. The auths ae listed in alphabetical de. Cntibutins wee equal and synegistic.

2 Ppeties f Pefeence Questins: Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency Abstact This pape uses stylized mdels t exple efficient pefeence questins. The fmal analyses pvide insights n citeia that ae cmmn in widely-used questin-design algithms. We demnstate that the citein f metic utility balance leads t inefficient questins, abslute biases, and elative biases amng patwths. On the the hand, chice balance f statedchice questins (the analgy f utility balance f metic questins) can lead t efficiency unde sme cnditins, especially f discete featues. When at least ne fcal featue is cntinuus, chices ae imbalanced at ptimal efficiency. This chice imbalance declines as espnses becme me accuate and nn-fcal featues incease. The analysis f utility- and chice-balance pvide insights n the elative success f metic- and chice-based adaptive plyhedal methds and suggest futue impvements. We then use the fmal mdels t exple a new fm f data cllectin that has becme ppula in e-cmmece and web-based maket eseach cnfiguats. We study the stylized fact that questins nmally used as wam-up questins in cnjint analysis ae supising accuate in pedicting cnsume behavi that is elevant t e-cmmece applicatins. F these applicatins, new data suggest that wam-up questins ae, pehaps, me useful manageially than paied-cmpaisn tadeff questins. We examine and genealize this insight t demnstate the advantages f matching tadeff, cnfiguat, and ange questins t the elevant manageial decisins. This futhe genealizes t M-efficiency (manageial efficiency), an altenative citeia f questin design algithms. Designs which minimize D-es and A-es may nt minimize M D - and M A -es which measue pecisin elevant t manageial decisins. We pvide examples in which simple mdificatins t questin design incease M-efficiency. eywds: Cnjint analysis, efficient questin design, adaptive questin design, Intenet maket eseach, e-cmmece, pduct develpment.

3 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency. Mtivatin Pefeence measuement is cucial t manageial decisins in maketing. Methds such as cnjint analysis, vice-f-the-custme methds, and, me ecently, cnfiguats, ae used widely f pduct develpment, advetising, and stategic maketing. In espnse t this manageial need, maketing scientists have develped many sphisticated and successful algithms t select questins efficiently. F example, Figue illustates tw ppula fmats that have been studied widely. Figue Ppula Fmats f Cnjint Analysis (a) Metic paied-cmpaisn fmat (laptp bags) (b) Stated-chice fmat (Executive Educatin) F the metic-pais fmat (Figue a) questins can be selected with efficient designs, with Adaptive Cnjint Analysis (ACA), with plyhedal methds (uhfield, Tbias, and Gaatt 994; Sawtth ; Tubia, Simeste, Hause, Dahan 3). F the stated-chice fmat (Figue b), ecent advances adapt questins based n petests, manageial beliefs, pi questins and d s eithe by selecting an aggegate design that is eplicated f all subsequent espndents by adapting questin design f each espndent (Aa and Hube ; Hube and Zweina 996; Jhnsn, Hube and Bacn 3; anninen ; Sand and Wedel,, Tubia, Hause, and Simeste 3). These questin-design methds ae used with a vaiety f estimatin methds (egessin, Hieachical Bayes estimatin, mixed lgit mdels, analytic cente appximatins, and hybid methds) and have pven t pvide me efficient and me accuate data with which t estimate patwths.

4 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency Figue Example Cnfiguats (a) Cnfiguat fm Dell.cm (b) Innvatin Tlkit f Industial Flavings (c) Design Palette f Pickup Tucks (d) Use Design f Laptp Cmpute Bags Recently, as pefeence measuement has mved t web-based questinnaies, e- cmmece fims and academic eseaches have expeimented with a new fmat in which espndents ( e-cmmece custmes) select featues t cnfigue a pefeed pduct. The bestknwn e-cmmece example is Dell.cm (Figue a); hweve, the fmat is becming ppula n many websites, especially autmtive websites (e.g., kbb.cm s aut chice advis, vehix.cm, autweb.cm). F maketing eseach, cnfiguats ae knwn vaiusly as innvatin tlkits, design palettes, and use design fmats. They have been instumental in the design f flavings f Intenatinal Flav and Fagance, Inc (Figue b), new tuck platfms f Geneal Mts (Figue c), and new messenge bags f Timbuk (Figue d). (Figues fm Dahan and Hause ; Thmke and vn Hippel, Uban and Hause 3; vn Hippel ). Respndents find cnfiguats easy t use and a natual means t expess pefeences. Hweve, because cnfiguats fcus n the espndents pefeed pduct, they have nly

5 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency been used t estimate patwth estimates when espndents ae asked t epeat the tasks with vaying stimuli (e.g., Liechty, Ramaswamy and Chen ). In this pape, we examine questin design f all thee fms f pefeence measuement: metic pais, stated chices, and cnfiguats. We use fmal mdeling methds t examine key ppeties that have been used by the algithms ppsed t date. With these mdels we seek t undestand basic ppeties in de t help explain why sme algithms pefm well and thes d nt. These mdels als pvide insight n the le f cnfiguats in pefeence measuement and indicate the stengths and weaknesses f this new fm f data cllectin elative t me-standad methds. The mdels themselves ae dawn fm empiical expeience and epesent, in a stylized manne, the basic ppeties f questin design. The fmal mdels als pvide insight n the elatinship f questin design t manageial use suggesting, f example, that sme measuement fmats ae best f develping a single pduct, sme f pduct-line decisins, and sme f e-cmmece decisins. These insights lead t evised manageial citeia (M A -e and M D -e). The pape is stuctued as fllws: We examine fist the cncept f utility balance as applied t the metic paiedcmpaisn fmat and demnstate that this citein leads t bth bias and inefficiency f that fmat. This analysis als lends insight n ecent cmpaisns between ACA and plyhedal methds (Tubia, et. al. 3). We then examine a elated cncept f the stated-chice fmat. The liteatue ften labels this cncept utility balance, but, f the pupse f this pape, we label it chice balance t indicate its elatinship t stated-chice data. Ou analyses indicate that, unlike utility-balance f metic data, chice balance f stated-chice data can lead t impved efficiency unde sme cnditins. In the cases, ptimal efficiency equies unbalanced chices. Ou analyses bth eflect insights fm ecent algithmic papes and pvide new insights n hw chice balance changes as a functin f the chaacteistics f the pblem. We then examine data that suggests that cnfiguat questins pvide me useful infmatin f e-cmmece decisins than tadeff questins (Figue ). We exple this bsevatin with a stylized mdel t suggest hw questin fmats ae best matched t manageial decisins. These insights genealize t M-efficiency. 3

6 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency We define u scpe, in pat, by what we d nt addess. Althugh we daw expeience fm and seek t pvide insight f new algithms, algithmic design is nt the fcus f this pape. N d we wish t ente the debate n the elative meits f metic-ating stated-chice data. F nw we leave that debate t the auths such as Eld, Luviee and Davey (99) Me (3). We study bth data fmats (and cnfiguats) and use fmal mdels t gain insights n all thee fms f pefeence questins. We d nt addess hieachical mdels f espndent hetegeneity, but athe fcus n each espndent individually. We ecgnize the pactical imptance f hieachical mdels, but abstact fm such estimatin t fcus n the basic ppeties. The ecent emegence f individual-specific questin designs and/ diffeentiated designs suggest that u insights shuld extend t such analyses (Jhnsn, Hube, and Bacn 3; Sand and Wedel 3; Tubia, Hause, and Simeste 3). But that emains futue eseach.. Efficiency in Questin Design Befe we begin u analyses, we eview biefly the cncepts f efficient questin design. The cncept f questin efficiency has evlved t epesent the accuacy with which patwths ae estimated. Efficiency fcuses n the standad es f the estimates. Let p be the vect f patwths t be estimated and pˆ the vect f estimated patwths. Then, if pˆ is (appximately) nmally distibuted with vaiance Σ, the cnfidence egin f the estimates is an ellipsid defined by ( p p)' Σ ˆ ( p pˆ), Geene (993, p. 9). F mst estimatin methds, Σ depends upn the questins that the espndent is asked and, hence, an efficient set f questins minimizes the cnfidence ellipsid. This is implemented as minimizing a nm f the matix, Σ. A-es ae based n the tace f Σ, D-es n the deteminant f Σ, and G-es n the maximum diagnal element. Because the deteminant is ften the easiest t ptimize, mst empiical algithms wk with D-efficiency (uhfeld, Tbias, and Gaatt 994, p. 547). G- efficiency is aely used. Because the deteminant f a matix is the pduct f its eigenvalues and the tace is the sum f its eigenvalues, A-es and D-es ae ften highly celated (uhfeld, Tbias, and Gaatt 994, p. 547, Aa and Hube, p. 74). In this pape we use bth, chsing the ci- In pactice, Σ is unknwn and is eplaced with its estimated value. F simplicity we assume that Σ is knwn, which shuld change neithe the insights n the esults f u analyses. 4

7 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency tein which leads t the mst tanspaent insight. In sme cases the esults ae easie t explain if we wk diectly with Σ ecgnizing det(σ) = [det( Σ )]. 3. Utility Balance f the Metic Paied-Cmpaisn Fmat The metic pais fmat (Figue a) is widely used f cmmecial cnjint analysis (Geen, iege, Wind, p. S66; Te Hfstede, im and Wedel, p. 59). Althugh the fmat has been used academically and cmmecially f ve 5 yeas, its mst cmmn applicatin is in the adaptive cmpnent f ACA (Geen, iege and Agawal 99, Hause and Shugan 98, Jhnsn 99). 3 Metic utility balance is at the heat f ACA. With the gal that a difficult chice pvides bette infmatin f futhe efining utility estimates, ACA pesents t the espndent pais f cncepts that ae as nealy equal as pssible in estimated utility (Ome 999, p. ; Sawtth Sftwae,, p., espectively). This citein has appeal. F example, in the study f espnse latencies, Haaije, amakua, and Wedel (, p. 38) suggest that espndents take me time n chice sets that ae balanced in utility and, theefe, make less epne chices. Geen, iege and Agawal (99, p. 6) quting a study by Hube and Hansen (986) ascibe bette pedictive ability t geate espndent inteest in these difficult-t-judge pais. Finally, Shugan (98) pvides a they t suggest that the cst f thinking is invesely pptinal t the squae f the utility diffeence. Thee ae clea advantages in tems f espndent inteest, attentin, and accuacy f metic utility-balanced questins. Hweve, ecent simulatin and empiical evidence suggests that altenative citeia pvide questin designs that lead t me accuate patwth estimates (Tubia, et. al. 3). Futheme, adaptatin implies endgeneity in the design matix, which, in geneal, leads t bias. Thus, we exple whethe nt the citein f metic utility balance has advese chaacteistics, such as systematic biases, that ffset the advantages in espndent attentin. We begin with a stylized mdel. The tace f a matix des nt necessaily equal the invese f the tace f the invese matix, thus an empiical algithm that maximizes the tace f Σ - des nt necessaily minimize the tace f Σ. Hweve, the eigenvalues ae elated, especially when the eigenvalues f Σ ae simila in magnitude, as they ae in empiical situatins. 3 F example, D. Lynd Bacn, AMA s Vice Pesident f Maket Reseach and Executive Vice Pesident f NFO Wldgup cnfims that metic pais emain ne f the mst ppula fms f cnjint analysis. Sawtth Sftwae claims that ACA is the mst ppula fm f cnjint analysis in the wld, likely has the lagest installed base, and, in, accunted f 37% f thei sales. Pivate cmmunicatins and Maketing News, 4//, p.. 5

8 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency A Stylized Mdel F simplicity we cnside pducts with tw featues, each f which is specified at tw levels. We assume futhe that the featues satisfy pefeential independence such that we can wite the utility f a pfile as an additive sum f the patwths f the featues f the pfile (eeney and Raiffa 976, p. -5; antz, et. al. 97). Withut lss f geneality, we scale the lw level f each featue t ze. Let p be the patwth f the high level f featue and let p be the patwth f the high level f featue. F these assumptins thee ae fu pssible pfiles which we wite as {, }, {, }, {, }, and {, } t indicate the levels f featues and, espectively. With this ntatin we have the tue utilities given by: u(, ) = u(, ) = p u(, ) = p u(, ) = p p F metic questins in which the espndent pvides an inteval-scaled espnse, we mdel espnse e as an additive, ze-mean andm vaiable, e. F example, if the espndent is asked t cmpae {, } t {, }, the answe is given by a p p e. We dente = the pbability distibutin f e with f (e). We dente the estimates f the patwths, estimated fm espnses t the questins, by ˆp and ˆp. Adaptive (Metic) Utility Balance Withut lss f geneality, cnside the case whee p > p > p such that the ff- > diagnal questin ({,} vs. {,}) is the mst utility-balanced questin. We assume that this dinal elatinship is based n pi questins. We label the e assciated with the mst utility-balanced questin as eub and label es assciated with subsequent questins as eithe e e. We nw cnside an algithm which adapts questins based n utility balance. Adaptive metic utility balance implies the fllwing sequence. Fist questin: p p = p p e ˆ ˆ ub p ˆ = p e e ub < p p p ˆ = p e e ub p p Secnd questin: if (Case ) Suppse that e ub p p, then: E [ pˆ = p ] = p E[ e ] if (Case ) E[ pˆ ˆ p ] = E[ p ] p p E[ eub eub p p] = p E[ eub eub p p] > 6

9 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency Suppse that e ub < p p, then: E [ pˆ = p ] = p E[ e ] E[ pˆ ˆ p ] = E[ p] p p E[ eub eub < p p] = p E[ eub eub < p p] > Thus, unde bth Case and Case, ne f the patwths is biased upwads f any ze-mean distibutin that has nn-ze measue belw p - p. The intuitin undelying this pf is that the secnd questin in the adaptive algithm depends diectly n the andm e a diect vesin f endgeneity bias (Judge, et. al. 985, p. 57). We demnstate hw t genealize this esult with a pseud-egessin fmat f ˆ p = ( X ' X ) ( X ' a) = p ( X ' X ) X ' e, whee a and e ae the answe and e vects and X is the questin matix. Then the secnd w f X depends n the magnitude f eub elative t p - p. Specifically, the fist w f X is [,-] and its secnd w can be witten as [b, -b] whee b = if es ae small and b = if es ae lage. The bias, B, is then given by the fllwing equatin whee k =. ( ) ( ) b e ub ek b eub () E[ B ] = E[( X ' X ) X ' e] = E( ) = E( ) beub ek beub Equatin illustates the phenmenn clealy: b = if e ub is small (me negative) and b = if e ub is lage (me negative), hence eithe ˆp ˆp is biased upwad. These aguments genealize eadily t me than tw featues and me than tw questins. The basic insight is that the es fm ealie questins detemine the late questins (X) in a systematic way the essence f endgeneity bias. Relative Bias Bias alne des nt necessaily impact manageial decisins because patwths ae unique nly t a psitive linea tansfmatin. Thus, we need t establish that the bias is als elative, i.e., the bias depends n the magnitude f the tue patwth. Because the bias is p- ptinal t e ub we need t shw that p E[ eub eub p p] < t demnstate that bias is geate f smalle (tue) patwths than it is f lage (tue) patwths. We d s by diect diffeentiatin in the Appendix. Thus, we can state the fllwing ppsitin. 7

10 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency Ppsitin. F the stylized mdel, n aveage, metic utility balance biases patwths upwad and des s diffeentially depending upn the tue values f the patwths. When we expand the adaptive questins t a me cmplex wld, in which thee ae me than tw featues, the intuitin still hlds; hweve, the magnitude f the bias depends upn the specific manne by which metic utility balance is achieved and n the accuacy f the espndent s answes. As an illustatin, we built a 6-questin simulat f a tw-patwth pblem in which, afte the fist questin, all subsequent questins wee chsen t achieve metic utility balance based n patwths estimated fm pi questins. Based n 6, egessins each, biases anged fm % f elatively lage tue patwths t 3% f elatively small patwths. 4 T examine the pactical significance we cite data fm Tubia, et. al. (3) wh simulate ACA questin design f a -patwth pblem. In thei data, biases wee appximately 6.6% f the magnitudes f the patwths at eight questins when aveaged acss the cnditins in thei expeimental design. The biases ae significant at the.5 level. The Effect f Metic Utility Balance n Efficiency Althugh metic utility balance leads t bias, it might be the case that adaptatin leads t geate efficiency. F D-es, minimizing the deteminant f Σ is the same as maximizing the deteminant f Σ. F metic data, Σ = X ' X (f apppiately centeed X). Fm this expessin we see tw things immediately. Fist, as anninen (, p. 7) and uhfeld, et. al. (994, p. 547) nte, f a given set f levels, efficiency tends t push featues t thei exteme values. (These statements assume that the maximum levels ae selected t be as lage as is easnable f the empiical pblem. Levels which ae t exteme isk additinal espndent e nt explicitly acknwledged by the statistical mdel used in questin design. See Delquie 3). Secnd, pefect metic utility balance impses a linea cnstaint n the clumns f X because X p =. This cnstaint induces X X t be singula. When X X is singula, the deteminant will be ze and D-es incease withut bund. Impefect metic utility balance leads t 4 The simulat is available fm the website listed in the acknwledgements sectin f this pape. The simulat is designed t demnstate the phenmenn; we ely n Tubia, et. al. (3) f an estimate f the empiical magnitude. The simulat chses x unifmly n [,] and x based n utility balance. The abslute values f the espnse es ae appximately half thse f the dependent measue. The magnitude f the bias changes if the espnse es change, but the diectin f the bias emains. 8

11 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency det(x X) and extemely lage D-es. A-es behave similaly. Thus, geate metic utility balance tends t make questins inefficient. 5 Insights n the Relative Pefmance f Metic Plyhedal Methds The analysis f metic utility balance pvides insights n the elative pefmance f metic plyhedal methds. Metic plyhedal methds fcus n asking questins t educe the feasible set f patwths as apidly as pssible by slving an eigenvalue pblem elated t Snnevend ellipsids. These ellipsids fcus questins elative t the axis abut which thee is the mst uncetainty. Plyhedal methds d nt attempt t impse metic utility balance. In algithms t date, plyhedal methds ae used f (at mst) the fist n questins whee n equals the numbe f patwths. Hweve, metic plyhedal methds als impse the cnstaint that the ws f X be thgnal (Tubia, et. al. 3, Equatin A9). Thus, afte n questins, X will be squae, nn-singula, and thgnal ( XX ' I implies X ' X I ). 6 Subject t scaling, this thgnality elatinship minimizes D-e (and A-e). Thus, while the adaptatin inheent in metic plyhedal methds leads t endgenus questin design, the thgnality cnstaint appeas t vecme the tendency f endgeneity t incease es. Tubia, et. al. (3) ept at mst a % bias f metic plyhedal questin selectin, significantly less than bseved f ACA questin selectin. Summay Metic utility balance may etain espndent inteest and encuage espndents t think had abut tadeffs, hweve, these benefits cme at a pice. Metic utility balance leads t bias, elative bias, and lweed efficiency. On the the hand, the inheent thgnality citein (athe than utility balance) in metic plyhedal methds enables metic plyhedal questin design t achieve the benefits f adaptatin with significantly less bias and, appaently, withut educed efficiency. 4. Chice Balance f the Stated-Chice Fmat In the stated-chice fmat (Figue b), tw pfiles that have the same utility will be equally likely t be chsen, thus chice balance is elated t utility balance. Hweve, because 5 The simulat cited in the acknwledgements sectin f this pape als cntains an example in which metic utility balance can be impsed. The me clsely utilities ae balanced, the wse the fit. 6 Othgnal ws assume that XX =αi whee α is a pptinality cnstant. Because X is nn-singula, X and X ae invetible, thus X XX X - = αx X - X X = αi. 9

12 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency the undelying statistical mdels ae fundamentally diffeent, the implicatins f chice balance f stated-chice questins ae diffeent fm the implicatins f metic utility f metic paiedcmpaisn questins. Chice balance is an imptant citein in many f the algithms used t custmize stated-chice questins. F example, Hube and Zweina (996) begin with thgnal, levelbalanced, minimal velap designs and demnstate that swapping and elabeling these designs f geate chice balance inceases D p -efficiency. Aa and Hube (, p. 75) test these algithms in a vaiety f dmains with hieachical Bayes methds cmmenting that the maximum D p -efficiency is achieved by sacificing thgnality f bette chice balance. 7 Ome and Hube (, p. 8) futhe advcate chice balance. Tubia, Hause, and Simeste (3) use chice-balance t incease accuacy in adaptive chice-based questins. Finally, the cncept f chice balance is gemane t the pinciple, used in the cmpute adaptive testing liteatue, that a test is mst effective when its items ae neithe t difficult n t easy, such that the pbabilities f cect espnses ae clse t.5 (Ld 97). T undestand bette why chice balance impves efficiency while metic utility balance degades efficiency, we ecgnize that, f lgit estimatin f patwths with stated-chice data, Σ depends upn the tue patwths and, implicitly, n espnse e. Specifically, McFadden (974) shws that: q J i i i () Σ = R ( xij xik Pik )' Pij ( xij J i= j= k= k= whee R is the effective numbe f eplicates; J i is the numbe f pfiles in chice set i; q is the numbe f chice sets; x ij is a w vect descibing the j th pfile in the i th chice set; and P ij is the pbability that the espndent chses pfile j fm the i th chice set. Equatin has a me tanspaent fm when all chice sets ae binay, in paticula: q (3) Σ = R ( xi xi )' Pi ( Pi )( xi xi ) = R whee d i i= i= J x ik P ik ) q d ' P ( P is the w vect f diffeences in the featues f the i th chice set. In Equatin 3 it is clea that chice balance (P i ½) tends t incease a nm f Σ i i i ) d i. Hweve, the chice pb- 7 D p -efficiency ecgnizes that Σ depends n the tue patwths. See Equatin.

13 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency abilities (P i ) ae a functin f the featue diffeences ( d i ), thus pue chice balance may nt maximize eithe the deteminant ( the tace) f Σ. Fist ecgnize that the tue chice pbabilities in the lgit mdel ae functins f the utilities, d p. If we hld the utilities cnstant, we als hld the chice pbabilities cnstant. i Subject t this linea utility cnstaint, we can incease efficiency withut bund by inceasing the abslute magnitudes f the d i s. This esult is simila t that f the metic-pais fmat; the expeimente shuld chse featue diffeences that ae as lage as easnable subject t the linea cnstaint impsed by d i p and subject t any es (utside the mdel) intduced by lage featue diffeences (Delquie 3). anninen () explits these ppeties effectively by selecting ne cntinuus featue (e.g. pice) as a fcal featue with which t manipulate d p, while teating all emaining featues as discete by setting them t thei maximum minimum levels. She epts substantial inceases in efficiency f bth binay and multinmial chice sets (p. 3). anninen uses numeical means t select the ptimal levels f the cntinuus fcal featue. We gain futhe insight with fmal analysis. In paticula, we examine the ptimality cnditins f the fcal featue, hlding the thes cnstant. F geate tanspaency we use binay questins and the tace f Σ q d ik P i i= k = (4) tace( Σ ) = i athe than the deteminant. 8 The tace is given by: If we assume, withut lss f geneality, that the fcal featue is the th featue, then the fistde cnditins f the fcal featue ae: ( P i P (5) i d i = p dik f all i k = whee d ik is the level f the k th featue diffeence in the i th binay questin. We fist ule ut the tivial slutin f d ik = P i ) f all k. Nt nly wuld this imply a chice between tw identical altenatives, but the secnd-de cnditins imply a minimum. A slightly me inteesting case ccus when pefect chice balance is aleady achieved by the - 8 F a fcal featue, the tace and the deteminant have elated maxima, especially if all but the nn-fcal featue ae set t thei minimum maximum values.

14 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency discete featues. We etun t this slutin late, hweve, in the fcal-featue case it ccus with ze pbability (i.e., n a set f ze measue). The me inteesting slutins f a cntinuus fcal featue equie that d i be nn-ze and, hence, that P P i i Pi. Thus, when di can vay cntinuusly, Equatin 5 implies that ptimal efficiency equies chice imbalance. Thee ae tw equivalent slutins t Equatin 5 cespnding t a ight-t-left switch f the tw altenatives. That is, the sign f d i must match the sign f. Thus, withut lss f geneality, we examine situatins wee di is psitive (see Appendix). We plt the maginal impact f d i n tace( Σ ) f illustative levels f the nn-fcal featue diffeence. 9 Fllwing Aa and Hube () and Tubia, Hause and Simeste (3), we use the magnitude f the patwths as a measue f espnse accuacy highe magnitudes indicate highe espnse accuacy. Figue 3 illustates that efficiency is maximized f nn-ze fcal featue diffeences. Figue 3 als illustates that the ptimal level f the fcal featue diffeence depends upn the level f the nn-fcal featues and upn the magnitude. i i Chice balance ( P P ) is a nn-linea functin f the featue diffeences. Nnetheless, we can identify the systematic impact f bth nn-fcal featues and espnse accuacy. In paticula, fmal analyses eveal that chice balance inceases with geate espnse accuacy and with geate levels f the nn-fcal featue diffeences. We fmalize these insights in tw ppsitins. The pfs ae in the Appendix. P i Ppsitin. As espnse accuacy inceases, chice balance inceases and the level diffeence in the fcal featue deceases. Ppsitin 3. As the levels f the nn-fcal featues incease, chice balance inceases. 9 F lw magnitudes, the patwths f bth featue ae set equal t. in these plts. F high magnitudes they ae dubled. The lw level f the nn-fcal featue (abslute value) is.5. The high level is 3..

15 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency Figue 3 Optimal Efficiency and Chice Imbalance Efficicency (tace) 4 3 Efficicency (tace) Featue diffeence Lw levels nn-fcal featues High levels nn-fcal featues Featue diffeence Lw magnitude High magnitude (a) Lw vs. high nn-fcal featue levels (b) Lw vs. high magnitude in lgit mdel Quantal Featues In many applicatins, all featues ae binay they ae eithe pesent absent. Examples include fu-wheel steeing n autmbiles, cell-phne hldes n laptp bags, and aut fcus n cameas. In these cases, the fcal-featue methd cannt be used f maximizing efficiency. Hweve, discete-seach algithms ae feasible t maximize a nm n. Aa and Hube () and Sand and Wedel () pvide tw such algithms. Sand and Wedel () extend these cncepts t a mixed lgit mdel. An inteesting (and cmmn) case ccus when the expeimente limits the numbe f featues that vay, pehaps due t cgnitive lad (Sawtth Sftwae 996, p. 7; Shugan 98). Σ F a given numbe f binay featues (±), k d ik is fixed. In this case, it is easy t shw that Equatin 4 is maximized when chices ae as balanced as feasible. This same phenmenn applies when the eseache equies that all featues vay. Insights n the Relative Pefmance f Chice-based Plyhedal Methds Like metic plyhedal methds, chice-based plyhedal methds ely upn Snnevend ellipsids. Hweve, f stated-chice data, espndents answes pvide inequality cnstaints athe than equality cnstaints. Rathe than selecting questins vects paallel t the lngest axis f the ellipsid, chice-based plyhedal methds use the ellipsid t identify taget pat- If thee ae a maximum numbe f featues that can be nn-ze athe than a fixed numbe, it is pssible, althugh less likely, that a maximum will might ccu f less than the maximum numbe f featues. This case can be checked numeically f any empiical pblem. Nnetheless, the qualitative insight hlds. 3

16 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency wths at the extemes f the feasible egin. The chice-based algithm then slves a utility maximizatin (knapsack) pblem t identify the featues f the pfiles. The knapsack pblem assues that the cnstaints g (appximately) thugh the analytic cente f the egin. This citein pvides appximate a pii chice balance. (By a pii we mean that, given the pi distibutin f feasible pints, the pedicted chices ae equal.) In this manne, the algithm achieves chice balance withut equiing utility balance thughut the egin. Utility balance nly hlds f the analytic cente f the egin a set f ze measue. The applicatins f chice-based plyhedal methds t date have fcused n discete featues implemented as binay featues. F discete featues, chice balance achieves high efficiency. F cntinuus featues, Ppsitin suggests algithmic impvements. Specifically, the knapsack pblem might be mdified t include tace( Σ ) in the bjective functin. Hweve, unlike a knapsack slutin, which is btained apidly, the evised math pgam may need t ely n new, ceative heuistics. Summay Chice balance affects the efficiency f stated-chice questins quite diffeently than metic utility balance affects metic paied-cmpaisn questins. This, despite the fact that chice balance is elated t dinal utility balance. If at least ne featue is cntinuus, then that featue can be used as a fcal featue and the mst efficient questins equie chice imbalance (and nn-ze levels f the fcal featue). Hweve, we shw fmally that me accuate espnses and highe nn-fcal featues lead t geate chice balance. If featues ae discete and the numbe which can vay is fixed, then chices shuld be as balanced as feasible. Finally, this analysis helps explain the elative pefmance f stated-chice plyhedal methds and aggegate custmizatin methds. 5. Cnfiguats In Sectins 3 and 4 we expled tw ppula fmats f pefeence questins suggesting that, f the metic paied-cmpaisn fmat, utility balance led t bias, elative bias and inefficiency while, f the chice-based fmat, chice balance culd be efficient f discete featues but nt necessaily f cntinuus fcal featues. We nw exple a elatively new fmat, cnfiguats, in which espndents fcus n the featues ne at a time and ae nt asked t cmpae pfiles in the taditinal sense. Neithe utility balance n chice balance apply t cnfiguats. Nnetheless, analysis f this new fmat leads t insights that genealize t all thee questin 4

17 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency fmats and helps us undestand which fmat is mst efficient in which situatins. In cnfiguats, espndents ae typically given a pice f each featue level and asked t indicate thei pefeences f that pice/featue cmbinatin (eview Figue ). Cnfiguat questins ae equivalent t asking the espndent whethe the patwths f the featues (p k s) ae geate than less than the stated pices. These data appea quite useful. F example, Liechty, Ramaswamy and Chen () use ten epeated cnfiguat measues pe espndent t estimate successfully hieachical Bayes pbit mdels. Stylized Fact: Wam-up Questins d Supisingly Well in Pedicting Featue Chice Figue 4 pesents an empiical example. In data cllected by Tubia, et. al. (3), espndents wee shwn five laptp bags wth appximately $ and allwed t chse amng thse bags. Vaius cnjint methds based n metic paied-cmpaisn questins cllected pi t this task pedicted these pfile chices quite well. Figue 4 Wam-up Questins ae Effective Pedicts f Cnfiguat Tasks Pecent f Featues Pedicted Cectly With Initial Questins Withut Initial Questins Pedicted Pecentile f Chsen Pfile With Initial Questins Withut Initial Questins Numbe f Cnjint Questins Numbe f Cnjint Questins (a) Ability t pedict featue chice (b) Ability t pedict pefeed pfile Befe cmpleting the metic paied-cmpaisn task, in de t familiaize espndents with the featues, espndents answeed wam-up questins. They wee asked t indicate the pefeed level f each featue. Such wam-up questins ae cmmn. We e-analyzed Tubia, et. al. s data in tw ways. One cnjint estimatin methd did nt use the wam-up questins. The the cnjint estimatin methd included mntnic cnstaints based n the wam-up questins. One mnth afte the initial cnjint-analysis questinnaie, the espndents wee In the methd labeled with initial questins, the wam-up questins whee used t detemine which f tw featue levels wee pefeed (e.g., ed black). In the methd labeled withut initial questins, such deing data 5

18 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency ecntacted and given a chance t custmize thei chsen bag with a cnfiguat. Figue 4 plts the ability f the cnjint analyses t pedict cnfiguat chice as a functin f the numbe f metic paied-cmpaisn questins that wee answeed. F bth pedictive citeia, the ability t pedict cnfiguat chice is supising flat when initial wam-up questins ae included. (Pedictins ae less flat f fecasts f the chice f five laptp bags, Tubia, et. al. 3, Table 6.) F cnfiguat pedictins, the nine wamup questins appea t pvide me usable infmatin than the sixteen paied-cmpaisn questins. This empiical bsevatin is elated t bsevatins by Evgeniu, Bussis, and Zachaia (3) whse simulatins suggest the imptance f dinal cnstaints. While, at fist, Figue 4 appeas t be cunte-intuitive, it becmes me intuitive when we ealize that the wam-up questins ae simila t cnfiguat chice and the paied-cmpaisn questins ae simila t the chice amng five (Paet) pfiles. We fmalize this intuitin with u stylized mdel. Stylized Mdel t Explain the Efficiency f Wam-up Questins F tanspaency assume metic wam-up questins. This enables us t cmpae the wam-up questins t paied-cmpaisn questins withut cnfunding a metic-vs.-dinal chaacteizatin. We cnside the same thee questins used peviusly, ewiting e as e f mnemnic expsitin f paied-cmpaisn tadeff questins. Withut lss f geneality we incpate featue pices int the definitin f each featue. We assume p nt assume p > p as we did befe. The thee questins ae: (Q) p ˆ = p e (Q) p ˆ = p e (Q3) p p = p p e ˆ ˆ t > p > ub, but need We nw cnside stylized estimates btained fm tw questins. In u stylized wld f tw featues at tw levels, Q and Q can be eithe metic wam-up questins metic paied-cmpaisn questins. Q3 can nly be a metic paied-cmpaisn questin. Assuming all questins ae unbiased, we examine which cmbinatin f questins pvides the geatest pecisin (efficiency): Q Q, QQ3, QQ3. wee nt available t the estimatin algithm. In this paticula plt, f cnsistency, we use the analytic-cente methd f Tubia, et. al., hweve, the basic insights ae nt limited t this estimatin methd. t 6

19 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency Cnside featue. Because featue pice is incpated in the featue, if p k, then featue chice is cectly pedicted if pˆ k is nn-negative. (We use simila easning f featue.) Simple calculatins (see Appendix) eveal that the mst accuate cmbinatin f tw questins is QQ if and nly if Pb[ e p ] > Pb[ e e p ]. Since p is a negative k k k t k k numbe, these cnditins hld f mst easnable density functins. F example, in the Appendix we demnstate that, if the es ae independent and identically distibuted nmal andm vaiables, then this cnditin hlds. Ppsitin 4. F ze-mean nmally distibuted (i.i.d.) es, the wam-up questins (QQ) pvide me accuate estimates f featue chice than d pais f questins that include a metic paied-cmpaisn questin (Q3). Intuitively, metic wam-up questins pase the es me effectively than tadeff questins when the fcus is n featue chice athe than hlistic pducts. The questins fcus pecisin whee it is needed f the manageial decisins. We pvide a me fmal genealizatin f these cncepts in the next sectin. The intuitin des nt depend upn the metic ppeties f the wam-up questins and applies equally well t dinal questins. We invite the eade t extend Ppsitin 4 t dinal questins using the M-efficiency famewk that is expled in the next sectin. 6. Genealizatins f Questin Fcus: M-Efficiency The intuitin dawn fm Ppsitin 4 suggests that it is best t match pefeence questins caefully t the manageial task. F example, metic wam-up questins ae simila t cnfiguat questins and may be the mst efficient f pedicting featue chice. On the the hand, tadeff questins ae simila t the chice f a single pduct fm a Paet set. Estimating patwths fm tadeff questins may be best if the manageial gal is t pedict cnsume chice when a single new pduct is launched int an existing maket. We nw examine this me geneal questin. We begin with the stylized mdel f tw questins and then suggest a genealizatin t an abitay numbe f questins. 7

20 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency Stylized Analysis: Matching Questin Selectin t Manageial Fcus With metic questins (and pefeential independence) we can ask ne me independent questin, which we call a ange questin. We then cnside thee stylized pductdevelpment decisins. (Q4) p p = p p e ˆ ˆ Launch a single pduct The pduct-develpment manage has ne sht at the maket and has t chse the single best pduct t launch, pehaps because f shelf-space the limitatins. The manage is mst inteested in the difficult tadeffs, especially if the tw featues ae substitutes. The manage seeks pecisin n p p. Launch a pduct line (platfm decisin) The pduct-develpment manage is wking n a platfm fm which t launch multiple pducts, pehaps because f synegies in pductin. The manage is mst inteested in the ange f pducts t ffe (the beadth f the platfm). The manage seeks pecisin n p p. Launch a cnfiguat-based e-cmmece website The pduct-develpment manage plans t sell pducts with a web-based cnfiguat, pehaps because f flexible manufactuing capability. Hweve, thee is a cst t maintaining a bad inventy and thee is a cgnitive lad n the cnsume based n the cmplexity f the cnfiguat. The manage seeks pecisin n and n p. p F these stylized decisins we expect intuitively that we gain the mst pecisin by matching the questins t the manageial decisins. Table illustates this intuitin. F decisins n a single pduct launch, we gain the geatest pecisin by including tadeff questins (Q3); f decisins n pduct lines we gain the geatest pecisin by including ange questins (Q4), and f decisins n e-cmmece we gain the geatest pecisin by fcusing n featuespecific questins (Q and Q). These analytical esults, which match questin fmat t manageial need (f metic data), ae cmpaable in spiit t the empiical esults f Mashall and Badlw () wh examine cnguency between the fmats f the pfile and validatin data (metic vs. anking vs. chice) in hybid cnjint analysis. Mashall and Badlw s analyses suggest that pfile data equie less augmentatin with self-explicated data when they ae cnguent with the validatin data. Such questins ae feasible with a metic fmat. They wuld be eliminated as dminated altenatives in an dinal fmat such as stated-chice questins. 8

21 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency Table E Vaiances Due t Altenative Questin Selectin Tadeff Questins Range Questins Featue Fcus Single-pduct launch σ 3σ σ Pduct-line decisins 3σ σ σ e-cmmece website 3 σ 3 σ σ M-Efficiency The insights fm Table ae simple, yet pweful, and genealize eadily. In paticula, the stylized manageial decisins seek pecisin n Mp whee M is a linea tansfm f p. F example, if the manage is designing a cnfiguat and wants pecisin n the valuatin f featues elative t pice, then the manage wants t cncentate pecisin nt n the patwths pe se, but n specific cmbinatins f the patwths. We indicate this manageial fcus with a matix, M c, whee the last clumn cespnds t the pice patwth. This fcus matix pvides citeia, analgus t D-es and A-es, by which we can design questins. M-Efficiency f Metic Data the estimate f M c = We begin with metic data t illustate the citeia. F metic data, the standad e f Mp is pptinal t Σ = M ( X ' X ) M ' (Judge, et. al. 985, p. 57). By defining efficiency based n Σ M athe thanσ, we fcus pecisin n the patwths that matte mst t the manage s decisin. We call this fcus manageial efficiency (M-efficiency) and define M-es as fllws f suitably scaled X matices. M M D -e = q det( M ( X ' X ) M ' ) / n 9

22 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency M A -e = n M X X M tace q / ) ' ) ' ( ( The manageial fcus affects D-es and A-es diffeently. M D -e is defined by the deteminant f the mdified cvaiance matix and, hence, fcuses n the gemetic mean f the estimatin vaiances (eigenvalues). All else equal, the gemetic mean is minimized if the estimatin vaiances ae equal in magnitude. On the the hand, M A -e is defined by the tace f the cvaiance matix and, hence, fcuses n the aithmetic mean f the estimatin vaiances. Thus, if the manage wants diffeential fcus n sme cmbinatins f patwths and is willing t accept less pecisin n the cmbinatins, then M A -es might be a bette metic than M D - es. T illustate M-efficiency, we pvide an example in which we accept highe A- and D- es in de t educe M A - and M D -es. Cnside an thgnal aay, X. F this design, X X = I 6, whee I 6 is a 6x6 identify matix. The thgnal aay minimizes bth D-es and A-es. It des nt necessaily minimize eithe M D - M A -es as we illustate with the fivefeatue cnfiguat matix, M c. = c M = X

23 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency T incease M-efficiency, we petub X X M M clse t M cm c while maintaining full M M [ 6 c c ank. We chse XM such that X X = α βi ( β ) M M ] whee α and β ae scalas, β [,], and α is chsen t maintain cnstant questin magnitudes. 3 By cnstuctin, X M is n lnge an thgnal aay, hence bth A-es and D-es will incease. Hweve, M A - and M D -es decease because X M is me fcused n the manageial pblem. F example, with β = 4/5, M A -es decease by ve %; M D -es decease, but nt as damatically. Lage β s tun ut t be bette at educing M D -es, but less effective at educing M A -es. Deceases in M A -efficiency ae als pssible f squae M matices. Hweve, we cannt impve M D -es f squae M matices, because, f squae M, the questin design that minimizes D-e will als minimize M D -e. This is tue by the ppeties f the deteminant: det( Σ M ) = det(m(x X) - M ) = det(m)det((x X) - )det(m ) = det(m )det(m) det((x X) - ) = det(m M) det((x X) - ). As an illustatin, suppse the manage wants t fcus n cnfiguat pices (as in M c ), but als wants t fcus n the highest pice psted in the cnfiguat. T captue this fcus, we make M squae by adding anthe w t M c, [ -]. T decease M A - e we unbalance X t make pice me salient in the questin design. F example, a 4% unbalancing deceases M A -es by 5% with nly a slight incease in M D -es (<%). We chse these examples t be simple, yet illustative. We can ceate lage examples whee the deceases in M-es ae much lage. Ou petubatin algithms ae simple; me cmplex algithms might educe M-es futhe. M-efficiency appeas t be an inteesting and elevant citein n which t fcus futue algithmic develpment. M-Efficiency f Stated-Chice Questins The same aguments apply t stated-chice questins. The M-efficient citeia becme eithe the deteminant (M D -e) the tace (M A -e) f M(Z PZ) - M whee Z is a pbability-balanced X matix and P is a diagnal matix f chice pbabilities. See Aa and Hube (, p. 74) f ntatin. As in Sectin 4, thee will be a tadeff between chice balance and the levels f fcal featues. F M-es, that tadeff will be defined n cmbinatins f the featues athe than the featues themselves. 3 Recall that efficiency citeia tend t push featue levels t the extemes f the feasible space. Thus, t cmpae diffeent designs we must attempt t keep the magnitudes f the featues cnstant. In this example, we implement that cnstaint by keeping the tace f X M X M equal t the tace f X X. Ou gal is t pvide an example whee M- efficiency mattes. We can als ceate examples f the cnstaints the details wuld be diffeent, but the basic insights emain.

24 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency Summay f Sectins 5 and 6 Cnfiguat questins, based n analgies t e-cmmece websites, ae a elatively ecent fm f pefeence questins. Empiical data suggest that such questins ae supisingly gd at pedicting featue chice, in lage pat because they fcus pecisin n manageial issues that ae imptant f e-cmmece and mass custmizatin. This insight genealizes. Intuitively, pecisin (efficiency) is geatest when the questin fmat is matched t the manageial decisin. M A -es (and M D -es) pvide citeia by which match fmat t manageial elevance. These citeia can be used t mdify questin-design algithms. Thee emain the challenges f new algithmic develpment. Existing algithms have been ptimized and tested f D- and D p -efficiency. Hweve, Figue 4, Table, and the cncept f M-es have the ptential t efcus these algithms effectively. 7. Summay and Futue Reseach In the last few yeas geat stides have been made in develping algithms t select questins f bth metic paied-cmpaisn pefeence questins and f stated-chice questins. Algithms have been develped f a wide ange f estimatin methds including egessin, lgit, pbit, mixed lgit, and hieachical Bayes (bth metic and chice-based). In sme cases the algithms fcus n static designs and in the cases n adaptive designs. This existing eseach has led t pefeence questins that ae me efficient and me accuate as evidenced by bth simulatin and empiical expeiments. Summay In this pape, we use stylized mdels t gain insight n the ppeties f algithms and the citeia they ptimize. The mdels abstact fm the cmplexity f empiical estimatin t examine ppeties tanspaently. In key cases, the stylized mdels pvide insight n empiical facts. While sme f the fllwing insights fmalize that which was knwn fm pi algithmic eseach, many insights ae new and pvide a basis f futhe develpment: Metic utility balance leads t elative biases in patwth estimates. Metic utility balance is inefficient. Chice balance, while elated t utility balance, affects stated-chice questins diffeently than utility balance affects metic questins. When featues ae discete and the numbe f featues which can vay is fixed, chice balance is a cmpnent f ptimal efficiency.

25 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency Othewise, ptimal efficiency implies nn-ze chice balance. As espnse accuacy and/ the levels f nn-fcal featues incease, ptimal efficiency implies geate chice balance. Wam-up (cnfiguat) questins ae supisingly accuate f pedicting featue chice because they fcus pecisin n the featue-chice decisin. This cncept genealizes. Tadeff, ange, cnfiguat questins pvide geate efficiency when they ae matched t the apppiate manageial pblem. M-es pvide citeia by which bth metic and stated-chice questins can be fcused n the manageial decisins that will be made based n the data. Simple algithms (petubatin and/ unbalancing) can decease M-es effectively. The fmal mdels als pvide a they with which t undestand the accuacy f ecently ppsed plyhedal methds. Specifically, Metic plyhedal methds appea t avid the bias and inefficiency f utility balance by maintaining thgnality while fcusing questins whee patwths estimates ae less pecise. T date, stated-chice plyhedal methds have been applied f discete featues. Hence, thei use f appximate chice balance makes them clse t efficient. Hweve, stated-chice plyhedal methds f cntinuus featues might be impved with algithms that seek nn-ze chice balance. Futue Reseach This pape, with its fcus n stylized mdels, takes a diffeent tack than ecent algithmic papes. The tw tacks ae cmplementay. Ou stylized mdels can be extended t explain simulatin esults in Andews, Ainslie and Cuim (), empiical esults in Me (3), t exple the efficient use f self-explicated questins (Te Hfstede, im and Wedel ). On the the hand, insights n utility balance, chice balance, and M-efficiency can be used t impve algithms f bth aggegate custmizatin and plyhedal methds. Finally, inteest in cnfiguat-like questins is gwing as me maketing eseach mves t the web and as e-cmmece manages demand maketing eseach that matches the decisins they make daily. We have nly begun t undestand the natue f these questins. 3

26 Utility Balance, Chice Balance, Cnfiguats, and M-Efficiency, Refeences Refeences Andews, Rick L., Andew Ainslie, and Iman S. Cuim (), An Empiical Cmpaisn f Lgit Chice Mdels with Discete vesus Cntinuus Repesentatins f Hetegeneity, Junal f Maketing Reseach, 39, (Nvembe), Aa, Neeaj and Jel Hube (), Impving Paamete Estimates and Mdel Pedictin by Aggegate Custmizatin in Chice Expeiments, Junal f Cnsume Reseach, 8, (Septembe), Dahan, Ely and Jhn R. Hause (), "The Vitual Custme," Junal f Pduct Innvatin Management, 9, 5, (Septembe), Delquie, Philippe (3), Optimal Cnflict in Pefeence Assessment, Management Science, 49,, (Januay), -5. Evgeniu, Theds, Cnstantins Bussis, and Gigs Zachaia (3), Genealized Rbust Cnjint Estimatin, Wking Pape, (Fntainebleau, Fance: INSEAD), May. Eld, Tey, Jdan Luviee, and ishnakuma S. Davey (99), An Empiical Cmpaisn f Ratings-Based and Chice-based Cnjint Mdels, Junal f Maketing Reseach 9, 3, (August), Geen, Paul E, Abba iege, and Manj. Agawal (99), Adaptive Cnjint Analysis: Sme Caveats and Suggestins, Junal f Maketing Reseach, pp. 5-.,, and Yam Wind (), Thity Yeas f Cnjint Analysis: Reflectins and Pspects, Intefaces, 3, 3, Pat, (May-June), S56-S73. Geene, William H. (993), Ecnmetic Analysis, E, (Englewd Cliffs, NJ: Pentice-Hall, Inc.) Haaije, Rinus, Wagne amakua, and Michel Wedel (), Respnse Latencies in the Analysis f Cnjint Chice Expeiments, Junal f Maketing Reseach 37, (August), Hause, Jhn R. and Steven M. Shugan (98), Intensity Measues f Cnsume Pefeence, Opeatins Reseach, 8,, (Mach-Apil), Hube, Jel and David Hansen (986), Testing the Impact f Dimensinal Cmplexity and Affective Diffeences f Paied Cncepts in Adaptive Cnjint Analysis, Advances in Cnsume Reseach, 4, Melanie Wallendf and Paul Andesn, eds., Pv, UT: Assciatins f Cnsume Reseach, and laus Zweina (996), The Imptance f Utility Balance in Efficient Chice Designs, Junal f Maketing Reseach, 33, (August), R