Measuring Consumer Preferences Using Conjoint Poker

Toubia et al.: Measuring Consumer Preferences Using Conjoint Poker141Marketing Science 31(1), pp. 138–156, 2012 INFORMSFigure 3Examples of Hands (from Weakest to Strongest): One Pair (a), Straight (b), Double Straight (c), Flush (d), Straight Flush (e), andDouble Flush (f)(b)(a) One card(c)(d)(e)(f)—Double flush (strongest): all three cards have thesame level on two attributes.In the card selection stage, each player is asked toindicate his or her preferred card from that hand.1This information is used to provide incentives torespondents, as described next.2.2. IncentivesAt the end of the experiment, one player is selectedrandomly, and one of the rounds played by thatplayer is selected randomly. If the player won thatround, then he or she wins the product on his or herpreferred card. If price is one of the attributes, thenthe player also receives the difference between a preset amount of money and the price of that product.If the player was tied for best in that round, he orshe receives the reward with a probability equal to1 divided by the number of players in the tie. If the1In our implementation, players were asked to choose one card.Other implementations may introduce a “no-choice” option.player lost that round, then he or she receives nothing (except for any potential nominal fee paid to allrespondents). In cases in which the experiment alsoinvolves an external validity task, there is a positiveprobability that the incentives will be based on thattask instead (see the setup of our experiments in §3).2For the sake of argument, let us assume risk neutrality in the hand selection stage (this assumptionwill be relaxed in our final model). We define the utility of a card as the utility of the product on that cardplus the utility of the difference between the presetamount of money and the price of that product, ifapplicable. Given the incentive structure, the expectedutility derived by a respondent from a given round ifplaying a given hand, h, is then proportional to the2This mechanism is based on the “random lottery procedure,”which has been widely used and validated in experimental economics (Starmer and Sugden 1991). See Ding et al. (2005, 2009) forother applications of this procedure to incentive-compatible conjoint analysis.

Toubia et al.: Measuring Consumer Preferences Using Conjoint Poker142Marketing Science 31(1), pp. 138–156, 2012 INFORMSprobability of winning that round with h, multipliedby the utility of the respondent’s preferred card in h:Expected utility if play hand h Probability of winningthe round with h Utility of preferred card in h.This implies a trade-off. On the one hand, playersshould play strong hands to increase the probabilitythat they will receive a prize. On the other hand, players should also play hands that contain cards that theylike in order to increase the utility from the potentialprize. This implies that the optimal strategy is neitherto always play the strongest hand nor to play a handthat contains the card with the highest utility in eachround. In particular, it may be optimal to play a handthat is not the strongest hand but contains a highutility card or to play a hand that does not containone’s favorite card but has a higher chance of winningthe round.The above expression was provided only to illustrate the basic trade-off faced by CP participants. Inreality, we do not expect all consumers to be risk neutral in the hand selection stage. In the next section,we develop a choice model that captures this trade-offwithout assuming risk neutrality.2.3. Consumer Choice ModelWe now propose a choice model that captures thehand selection and card selection stages of CP. Leti index consumers, r index the rounds in the game,h index the possible hands available to players, andj index the profiles (i.e., cards). When there are fourcards per round and when each hand consists of threecards, each player has a choice between four possible hands in each round. We index these four handsby h 811 21 31 49 and the four cards by j 811 21 31 49,where h 1 corresponds to the hand made of profiles 811 21 39, h 2 corresponds to profiles 821 31 49,etc. With a slight abuse of notation, we write j hif card j is present in hand h. Let hi1 r be the handselected by consumer i in round r in the hand selection stage, and let ji1 r be the profile selected by thatconsumer in the card selection stage among the profiles in hand hi1 r . To estimate partworths from thechoices made by consumers, we construct a likelihoodfunction for Pr({hi1 r 1 ji1 r }). We havePr48hi1 r 1 ji1 r 95 Pr4hi1 r 5 Pr4ji1 r hi1 r 50(1)The second component, Pr(ji1 r hi1 r 5, corresponds tothe card selection stage in which consumer i choosesone profile from a set and may be modeled simplyusing logistic probabilities: exp4xj i 5 if j hi1 r 1 Pj 0 hi1 r exp4xj 0 i 5(2)Pr4j hi1 r 5 0otherwise1where i corresponds to the partworths for consumer i, and xj is an appropriately coded row vectorthat captures the attribute levels in profile j.The other term in Equation (1), Pr(hi1 r 5, correspondsto the hand selection stage, which is unique to CP.Here also, the consumer needs to make a choicebetween a set of possible alternatives (i.e., hands).In the case of risk neutrality, the expected utilityfrom choosing a hand is proportional to product ofthe probability of winning with that hand and theutility of the preferred card in that hand. Note thatwhereas a utility intercept is not identified from thecard selection stage alone in the absence of a “nochoice” option, such an intercept is identified fromthe hand selection stage because of the possibility ofnot winning anything in the round. In particular, ifwe normalize the utility of not winning anything (i.e.,of losing the round) to 0, then the expected utilityobtained by a risk-neutral consumer i from playinghand h is equal to Vhi1 r Phw r 4 i maxj h 8xj i 95, wherePhw r is the probability of winning round r by playing hand h, and i is an intercept that captures consumer i’s utility from winning a prize in the game. Ina two-player game, Phw r is equal to the probabilitythat the other player’s best hand is strictly weakerthan h plus half of the probability that the otherplayer’s best hand is exactly as good as h (ties are broken randomly). In our experiments, each consumerplayed against the computer, the computer’s cardswere drawn randomly without replacement from theset of all possible cards, and the computer alwaysplayed the strongest hand in each round. The resulting winning probabilities Phw r are computed in closedform in Appendix A.The above expression assumed risk neutrality inthe hand selection stage; i.e., the utility of the preferred card in a hand was assumed to have a proportional influence on the player’s evaluation of thehand. Risk-averse (respectively, risk-seeking) behavior is obtained when that the player’s evaluation ofthe hand is a concave (respectively, convex) functionof the utility of the preferred card in the hand. Making the standard assumption of constant relative riskaversion gives rise to the following expression: i Vhi1 r Phw r i max8xj i 9 1(3)j hwhere i is the risk-aversion parameter for consumer i. Risk neutrality is obtained when i 1, i 1 gives rise to risk aversion in hand selection, and i 1 gives rise to risk-seeking behavior.We then model consumer i’s choice in the handselection stage asPr4h5 Pexp4 · Vhi1 r 5h0 rexp4 · Vhi10 r 51(4)

Toubia et al.: Measuring Consumer Preferences Using Conjoint Poker143Marketing Science 31(1), pp. 138–156, 2012 INFORMSwhere is a logit scale parameter that capturesthe possibility that the response error is different inthe hand selection and card selection stages (e.g.,because of misrepresentations of the probabilities ofwinning).32.4. Design AlgorithmWhen estimating partworths from conjoint analysisdata, the statistical efficiency of the estimates, typically measured by their asymptotic covariance matrix,depends on the profiles shown to consumers. Inthe case of CBC, this asymptotic covariance matrixdepends further on the partworth estimates themselves (Huber and Zwerina 1996). Under mild conditions the asymptotic covariance matrix is equal tothe inverse of the information matrix (see, for example, McFadden 1974, Newey and McFadden 1994).A series of statistical efficiency measures have beenproposed, typically based on various types of matrixnorms (e.g., determinant, trace) applied to the information matrix, and various design algorithms havebeen proposed to select profiles that maximize efficiency. In CBC, a common efficiency measure isD-efficiency (based on the determinant norm), anda common approach for creating D-efficient designsis to obtain some prior information on the partworths and, given that information, to apply a setof operators to transform a nonefficient design intoa D-efficient design (see, for example, Huber andZwerina 1996; Sandor and Wedel 2001, 2002, 2005).Four properties characterize D-efficient CBC designs:level balance (the levels of an attribute occur withequal frequency), orthogonality (any two levels of different attributes appear in profiles with frequenciesequal to the product of their marginal frequencies),minimal overlap (each attribute level repeats itselfwithin each choice set with minimal probability), andutility balance (profiles in each choice are similarlyattractive).This design approach may be extended to the construction of efficient CP designs. The key difference isin the computation of the information matrix. Givenour choice model above, the log-likelihood function isequal toL4data X1 1 1 1 5XX log4Pr4hi1 r 55ir log4Pr4ji1 r hi1 r 55 constant03An alternative version of the game would be such that players stillwin some amount of money if they lose the round. The same modeling framework could be applied to a situation like this by simply replacing Vhi1 r Phw r 4 i maxj h 8xj · i 95 i with Vhi1 r Phw r 4 i maxj h 8xj · i 95 i 41 Phw r 5 · i , where i would capture the utilityfrom winning the amount of money offered when the round is lost.Taking the Hessian of the likelihood function withrespect to , we find that the expected value ofthe information matrix ì is proportional to (seeAppendix B for details):4 X X rì z̃hr Pr4hr 5z̃Thrrhr r Pr4hr 5X zj hr Pr4j hr 5zj hTr 1(5)j hrwhere Pr4hr 5 is given by (4), Pr4j hr 5 is given by (2),andXz̃rhr · ïVhrr · ïVhr0r Pr4h0r 51h0rzj hr xjT XxjT0 Pr4j 0 hr 51j 0 hrïVhrr Phwr r 4 xjh 5 1 xjT 1rhrwhere jh r argmax4xj 50j hrThe D-efficiency of any CP design may be computedfrom the determinant of the corresponding information matrix using the same formulas as the ones usedto compute the D-efficiency of a CBC design givenits information matrix. Moreover, the same operators (e.g., swapping, relabeling) used to improve theD-efficiency of a CBC design may also be used toimprove the D-efficiency of a CP design.One important difference between CP and CBC isthat efficient CP designs tend to have more leveloverlap compared with efficient CBC designs such asthose considered here (i.e., standard logit designs).5Minimal overlap occurs when attribute levels arerepeated within a choice set as little as possible. Forexample, in our experiments we achieved no overlapin our CBC design; i.e., each level of each attributeappeared exactly once in each choice set, leading tomaximal design efficiency. In contrast, some overlapis actually desirable in CP designs. Indeed, in a CPdesign with no level overlap, all hands in all roundsare equally strong (all hands are double straights, andPhw r is constant for all r and h), and the intercept iand risk-aversion parameter i are not identified (allhands have the same probability of winning).In our field study, we followed Huber and Zwerina(1996) and conducted a pretest from which we4The expression in Equation (5) assumes a homogeneous partworthvector (Huber and Zwerina 1996). In Appendix B, we providethe information matrix for the mixed logit model. This informationmatrix involves integrals and does not have a closed form (Sandorand Wedel 2002). For computational simplicity, the designs usedin our experiments assume a homogeneous partworth vector (i.e.,standard logit model).5Sandor and Wedel (2002) showed that efficient mixed logit designstend to have more level overlap compared with efficient standardlogit designs.

Toubia et al.: Measuring Consumer Preferences Using Conjoint Poker144Marketing Science 31(1), pp. 138–156, 2012 INFORMSobtained prior estimates (our pretest had 56 and 58respondents in the CBC and CP conditions, respectively). We then applied the relabeling and swappingoperators to improve the D-efficiency of a design thatwas selected as a starting point (see, for example,Huber and Zwerina 1996; Sandor and Wedel 2001,2002, 2005). For CBC, we used a D0 -efficient designwith no overlap as a starting point (a D0 -efficientis a D-efficient design assuming that all partworthsare equal to 0). This D0 -efficient design was obtainedusing the standard cyclic approach of Bunch et al.(1994): starting with an orthogonal design, a set ofchoice alternatives was constructed by adding cyclically generated alternatives to each set. For CP, weused a perturbed version of that D0 -efficient designas a starting point, where the perturbation was performed in order to introduce some amount of leveloverlap.63.Experiment DetailsWe now report the results of three experimentsthat compared incentive-compatible CP to incentivecompatible CBC. Our first study was a betweensubjects online experiment that enabled us to comparethe partworth estimates across the two methods andprovided indirect evidence that CP participants considered more of the profile-related information presented to them compared with CBC participants.Studies 2 and 3 are within-subjects eye-tracking studies that enabled us to measure more directly theamount of information considered by respondentsunder the two methods.3.1. SetupThe setup of all three studies was similar. CBC and CPwere implemented on the same online platform withthe same user interface. See Figure 2 for screenshotsof the CP interface and Figure 4 for screenshots of theCBC interface. We used mini laptops as our productcategory. Each of our product profiles was a different customized version of the Dell Inspiron Mini 11zlaptop computer. The display (1106" high-definition6The cyclic approach of Bunch et al. (1994) is such that if attributea is at level l in the qth profile of the orthogonal design, then it isat level l in the first card in the qth question of the choice design,level l 1 in the second card, etc. In other words, starting from theorthogonal design, the level of each attribute is incremented by 1for each new card in the choice set. In our perturbed version, it wasincremented with probability 1 t instead of 1. Therefore if attributea was at level l in the qth profile of the orthogonal design, it wasat level l in the first card in the qth round of Conjoint Poker. Thevalue in the second card was then equal to l 1 with probability1 t and to l with probability t. The value in the third card wasthen equal to that in the second card plus 1 with probability 1 tand equal to that in the second card with probability t, etc. We usedt 00350WLED display), processor (1 3 GHz Intel Celeron 743processor), RAM (2 GB), battery (28WHr lithium-ionbattery), operating system (Genuine Windows VistaHome Edition), and webcam (integrated 1.3 M pixelwebcam) were held constant. Six attributes were varied, with four levels each: design (four different colorschemes: promise pink, obsidian black, jade green,or ice blue), warranty (1-year limited, 1-year limitedwith in-home service after remote diagnosis, 2-yearlimited with in-home service after remote diagnosis,or 3-year limited with in-home service after remotediagnosis), McAfee SecurityCenter antivirus (30-day,15-month, 24-month, or 36-month subscription), harddrive (120 GB, 160 GB, 250 GB, or 320 GB), accessory(Logitech black cordless mouse, Logitech red cordlessmouse, Linksys wireless router, or Creative Labs headphones), and price ( 500, 550, 600, or 650). Giventhis price range, and given the fact that Dell laptopsare customizable, we were able to offer any productprofile as an incentive (like in Ding et al. 2005).7In each study, the flow of the experiment was asfollows, for each respondent:1. Instructions: Detailed instructions (using pictureillustrations) were displayed on the introductory page.Care was taken to make the instructions in both conditions as symmetric as possible. In addition, we createda slide show for each condition with a shorter version of the instructions, which we embedded at the topof the page (using http://www.authorstream.com—slide shows and instructions are available from theauthors upon request). The slide shows in both conditions used similar language (e.g., in both conditions,profiles were referred to as “cards”). Throughout theexperiment, a link to the instructions was available toparticipants.2. External validity task: As an external validity task,participants were asked to select one card from aset of eight. See Figure 5 for a screenshot. The format of the external validity task and the set of eightprofiles were identical across conditions and participants. These eight profiles were randomly selectedsubject to a level-balancing constraint (each levelof each attribute appears exactly twice across profiles) and such that exactly two of the four orderedattributes (warranty, security software, hard drive,price) were at one of the two most attractive levels ineach profile. This last constraint was added to avoiddominance.3. Main task: Participants in the CBC condition weregiven 20 CBC questions, each with four alternatives.Participants in the CP condition were asked to play20 CP rounds against the computer. (CP participantswere given two practice rounds between Steps 27Although appealing, this property is not required. See, for example, Ding (2007) and Dong et al. (2010).

Toubia et al.: Measuring Consumer Preferences Using Conjoint PokerMarketing Science 31(1), pp. 138–156, 2012 INFORMSFigure 4One CBC Questionand 3.) See Figures 2 and 4 for screenshots from eachcondition. The CBC and CP designs were selected tobe D-efficient as explained above. The only exceptionis the second eye-tracking study (Study 3), in whichFigure 5145External Validity Taskboth designs were identical. Within each condition,the design was identical across respondents.4. Follow-up questionnaire: Participants were administered a short follow-up survey that measured

Toubia et al.: Measuring Consumer Preferences Using Conjoint Poker146their feedback on the task, their knowledge of regularpoker, and how much they would need to be paid inorder to participate in a similar study in t

Toubia et al.: Measuring Consumer Preferences Using Conjoint Poker 140 Marketing Science 31(1), pp. 138–156, 2012 INFORMS Figure 2 One Round of Conjoint Poker Step 2. Hand selection stage : Player creates a three-card hand. (Clicking on a