Experimental design issues in choice-based conjoint applied to patient choice in healthcare

As the use of CBC analysis in the area of healthcare has increased over the past decade, there has been some debate over the lack of standard methodologies. Some researchers have looked at the comparison between available designs in CBC analyses to choose the optimal one in the healthcare setting. Kinter et al. compared experimental design approaches in conjoint analysis applied to patient-centered outcomes research [15]. They compared the parameters and predictions obtained from an orthogonal design to a D-efficient design using a randomized trial in patients with schizophrenia. No differences between the two designs were observed in an overall test of equality for the parameters in the study. Further, the authors state that while more studies on the statistical efficiency of competing design strategies are needed, a more pressing research problem is to document the impact the experimental design has on respondent efficiency. An interesting hypothetical trial experiment was performed by Najafzadeh et al. [2]. In this study, a group of 341 patients with cardiovascular disease completed an all discrete choice experiment questionnaire. The purpose was to quantify the patients’ preference for various outcomes of anticoagulant therapy with a hypothetical new drug, old drug or no drug. For example, this study found that on an average, patients assigned the same value to a 1% increased risk in fatal bleeding event as they did to a 3% increase in cardiovascular death event.

Recently, Groenewoud et al. investigated the preferences of a different patient population: patients with knee arthrosis, chronic depression or Alzheimer’s disease, using discrete choice experiments [3]. The study found that in these three patient groups, healthcare providers were chosen based on very different criteria. Specifically, patients with knee arthrosis considered the most valued attributes for their choice to be effectiveness and safety, while the group of chronically depressed patients valued continuity of care and their relationship with the therapist. For the patients with Alzheimer’s disease, the most valued attribute (as chosen by the patient’s representative) was the expertise of the healthcare provider. Thus, the study concluded that the quality of information regarding healthcare providers needs to be significantly improved, such that patients or their representatives could best make informed choices.

Patients healthcare decisions may be influenced by the software used by the researcher to administer the CBC survey. As such, the introduction of a new approach to CBC analysis, an adaptive choice-based conjoint analysis program, could be appealing to physicians and researchers. Cunningham et al. propose a software designed to provide a survey process more engaging than that of the traditional conjoint analysis survey [1]. They use a two-stage approach: a screening phase, in which the patient builds his own levels of the available attributes, followed by a choice phase. The attributes and levels identified in the screening phase are then advanced to the choice phase. The set of choices from this phase advances to the next choice phase and so on until a winner is determined. The authors suggest that this approach, although more time-consuming initially, may be preferred by patients due to the increased level of engagement it requires.

CBC and discrete choice experiments are also becoming increasingly popular in the health economics area. A literature review performed by De Bekker-Grob et al. focuses on experimental design issues, estimation procedures and validity of responses as main topics [16]. The authors note that the designs used became more statistically efficient and that there is an increased use of odds ratios and probabilities, especially at the policy-making level. There is room for improvement in terms of incorporating interaction terms in the design and analysis of these experiments as well as in better explaining the risk of different choices.

The lack of standard methodologies has served as a barrier to the more widespread use of CBC in healthcare. The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Good Research Practices for Conjoint Analysis Task Force was established to identify good research practices for CBC applications in healthcare. In a report by the task force, Bridges et al. highlighted ten important stages of the development and publication of an application of conjoint analysis or a discrete-choice experiment in healthcare, including [17]: “research question; attributes and levels; construction of tasks; experimental design; preference elicitation; instrument design; data-collection plan; statistical analyses; results and conclusions; and study presentation.” Harrison et al. performed a thorough review of the literature that incorporated risk attributes in healthcare discrete choice experiments [18]. The authors concluded that while there were some examples of good practice in the presentation of risk, there was a lot of room for improvement in the areas of training and quality of background materials to support risk communication. It is clear that there is more to be done in consistently applying the good practices defined by the ISPOR Good Research Practices for Conjoint Analysis Task Force in the healthcare arena.

In a valuation study, it is possible for some profiles to be better than others and for some profiles to be worse than others. Profiles that are better than others on all attributes are dominating profiles while those that are worse are dominated. If a choice set has a dominating profile the respondent will always choose that profile and nothing is learned about the relative value the respondent attaches to an increment in one attribute versus an increment in another. Similarly, if a choice set has a dominated profile, it will never be selected. Therefore, choice sets should not have dominating or dominated profiles; such choice sets are called Pareto optimal (PO) subsets. Formally, a subset T of S is said to be a PO subset if for every two distinct profiles (x₁. x₂. …, x_m), (y₁. y₂. …, y_m) Є T, there exist subscripts i and j (i ≠ j) such that x_i < y_i and x_j > y_j. That is, no profile dominates another profile in PO sets.

For example, consider a new medication to control post-chemotherapy symptoms with four side effects (attributes): post-chemotherapy fatigue, nausea, vomiting and drowsiness. Each attribute has two levels of intensity, high and low. If asked to choose the level of each side effect separately the respondent will naturally select the low level for each side effect. However, due to constraints such as budget and production technology, it may not be feasible for drug makers to expend resources on reducing all four side effects. Drug makers need to know which of these four side effects are the most important to reduce in order to improve the quality of life for cancer patients. They can then make informed decisions about where resources should be concentrated. In such an environment, respondents are required to make tradeoffs and these tradeoffs cannot be determined by asking about each side effect separately. For this purpose, we create ‘medication profiles’ of the side effects of the medication and ask the respondent to make a choice. If a profile has all four side effects at the low level, it will ‘dominate’ other profiles that have one or more side effects at the high level and respondents will always choose the dominating profile. Provided there are no such dominating or dominated profiles, different respondents will select different profiles and we can gain insight into the relative importance of these side effects for the quality of life of cancer patients.

At the first stage of the medication investigation, we consider only two levels: the low level (0) and high level (1) for each side effect. If we present a choice set with the profiles {0000, 1000, 0100, 0010}, the first profile with all four side effects at the low level dominates the other profiles. The respondent will always choose the first profile from this choice set. Nothing is learned about the relative value the patient attaches to a reduction in one side effect versus a reduction in another. Similarly, if we present a choice set with the profiles {0011, 1000, 0100, 0010}, the first profile is dominated and will not be selected. For this reason, choice sets should have no dominating or dominated profiles.

Raghavarao and Wiley showed that $S_l=\{(x_1, x_2, \mathrm{\dots } x_n)|{{\displaystyle \Sigma }}_{i=1}^n x_i=l\}$ , where the levels of the attributes in each profile sum to the same number l are PO [19]. For example, in the above medication investigation, S₂.#x00A0;= {1100, 1010, 1001, 0110, 0101, 0011} would be a PO choice set with no dominating or dominated profiles, since the levels of the attributes in each profile sum to the same number. Note that any subset of a PO subset is itself a PO subset.

In discrete choice experiments, we often have several options for selecting choice sets to estimate the main effects of the attributes and their interactions. We would like to use a plan that is optimal in some sense. Several optimality criteria exist in experimental design literature, such as A-optimality, D-optimality and E-optimality. However, since each choice set may not have the same number of profiles, IPP is often used as an optimality criterion to compare designs with different number of profiles. IPP is defined as the reciprocal of the average variance of estimated main effects divided by the number of profiles used in the experiment. For details on optimal designs and the IPP criteria, see Raghavarao et al. [20].

Optimal experiments for estimating the main effects of attributes based on two consecutive choice sets, S_l and S_l+1, has been examined in the literature. More recently, Xiao and Chitturi examined the IPP of nonconsecutive choice sets, S_l and S_n-l, and showed that the IPP can be maximized under certain conditions [21]. They further showed that nonconsecutive choice sets have higher IPP than consecutive choice sets for experiments with four or more attributes. The advantage of using nonconsecutive choice sets is that it often reduces the number of profiles in each choice set and makes the experiment smaller under certain conditions. Experimenters using PO choice sets can choose designs based on multiple criteria such as the number of profiles, IPP, A-, D- and E-optimality. However, since choice sets may not have the same number of profiles, we recommend IPP as the optimality criterion to compare designs with different number of profiles.

Although PO choice sets and high IPP are desirable in CBC, choice sets based on these criteria may be too large and impractical for real-life applications. If the number of attributes or attribute levels are large, usually more than six, the profiles in a single choice set may be too numerous for respondents to make precise decisions. Therefore, several methods have been developed to reduce choice set sizes. Raghavarao et al. discuss each method in detail, with interesting applications [20]. It is also possible that experimenters are only interested in estimating the main effects of attributes and a subset of the interactions, rather than estimating all interactions related to the attributes. Chen and Chitturi give smaller designs based on PO choice sets that are capable of estimating main effects and subsets of interactions [14].

Subsetting choice sets is an option in a situation where the number of existing profiles is not too large and can be divided into a number of small choice sets which the investigator considers appropriate for the experiment. This is the simplest method of forming smaller sets from large choice sets by dividing each choice set into smaller sets to conduct the study. Of course, this is achieved at the cost of an increased number of choice sets. If the larger choice sets are PO, the smaller subsets will also be PO. For example, in an experiment with four attributes at three levels (0, 1, 2), we can estimate all main effects using the following two consecutive choice sets S₂ (ten profiles) and S₃ (16 profiles).

However, if we want to use smaller choice sets, we can divide the profiles in choice sets S₂ and S₃ into seven PO choice sets of four profiles or less as given below.

For a given situation, the smallest number of such profiles can be calculated under the additional condition of estimability of the maximum possible order of interactions [22].

Another method of reducing the choice set sizes is to subset the attributes and/or attribute levels into smaller sets. The investigator may have different goals in a particular design and, as such, the cases illustrated for this approach are differentiated by the interactions of lower or higher order that are of interest and can be estimated in a specific context. When subsetting the attributes in an experiment, we divide the total number of attributes into overlapping subsets and form separate main-effect plans based on the attributes in each subset. A limitation of this approach is that while all main effects are estimable, we can only estimate two-factor interactions if both attributes are in the same subset. Similarly, attribute levels in an experiment can be divided into overlapping subsets of levels and we can construct PO choice sets for all attributes using the levels in each subset. For example, if an experiment with four attributes at five levels {0, 1, 2, 3, 4} is planned, it can be run as two experiments: the first with levels {0, 1, 2} and the second with levels {2, 3, 4} leading to a significant reduction in choice set sizes.

If the experiment is constructed using a BIBD, we can generate smaller designs, which will keep the IPP unchanged, as described in Raghavarao et al. [20]. Specifically, in the original BIBD design, we can choose a set of profiles where each one is formed by taking the presence (absence) of a symbol as the high (low) level of the corresponding attribute in the same profile of the design. Using this technique, we then create another choice set from the complement BIBD design. It has been shown by Raghavarao and Zhang that the newly created choice sets have the same IPP as the original designs [23].

Choice set sizes can also be reduced by applying a cyclic construction, creating s choice sets of size s each from an s^s experiment, in other words, an experiment with s factors at s levels each. The example described in Raghavarao et al. starts with an orthogonal Latin Square design, which is rotated cyclically to create balanced designs [20]. This strategy can be applied to create a main-effect plan in s PO choices of size s, which seems to be a methodology reasonably applicable in the drug development area. Specifically, consider the safety profile of a drug in a Phase III trial. At this point in its development, the experimental drug has a profile already observed in previous clinical trial phases, which must be tested and confirmed. In the oncology area, the safety of specific (prespecified to follow in the protocol of the study) adverse experiences (AE) are measured by the standard National Cancer Institute toxicity grades. The grades are 1, 2, 3 and 4. Grade 1 marks mild toxicity while 4 marks a life-threatening adverse event. We consider the attributes to be the following AEs: fatigue, dizziness, constipation and anorexia. This becomes a 4⁴ design, for which we can reduce the choice set sizes using the cyclic permutation of the 4 levels. As the toxicity grades are ordered, the patient should only be offered profiles that are not dominating or dominated. The example considered here is applicable to any drug tested in a Phase III clinical trial where the safety profile includes the AE levels as none, mild, moderate and severe. The experimenter should carefully choose the AEs of interest that define the potential profile of the drug as suggested by the previous phases of the drug development.

In this paper, we examine experimental design issues in discrete choice experiments, such as the construction of tasks using PO choice sets, IPP and reducing choice set size, as applied to patient choice in healthcare. CBC analysis is finding widespread applications in the healthcare industry, where it is used to understand how patients reveal their preferences through choice. Such studies can help healthcare professionals better understand patient choice in various fields, including healthcare policy, regulatory policy, drug development, healthcare intervention, doctor–patient communications and health technology assessment. The ISPOR Good Research Practices for Conjoint Analysis Task Force highlighted the important stages in the development of an application of conjoint analysis or discrete-choice experiment in healthcare [17]. However, there is more to be done to better understand experimental design issues and develop standard methodologies for CBC in healthcare. Researchers conducting CBC analysis or discrete choice experiments in the area of healthcare should consider experimental design issues such as PO choice sets, IPP and reducing choice set sizes, when designing the study.

This work was presented at the Joint Statistical Meetings in Vancouver, BC, Canada (31 July 2018).

The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

No writing assistance was utilized in the production of this manuscript.