Common values in assessing health outcomes from disease and injury: disability weights measurement study for the Global Burden of Disease Study 2010

Introduction

The Global Burden of Diseases, Injuries, and Risk Factors Study (GBD) aims to quantify health losses from a wide array of diseases and injuries. These losses are expressed in units of disability-adjusted life-years (DALYs), which account for both premature mortality—measured as years of life lost (YLLs)—and time spent in states of reduced health—measured as years lived with disability (YLDs). Al though the term disability has many meanings in different contexts,^1–4 in the GBD, disability refers to any short-term or long-term loss of health.⁵

For the latest revision of the GBD (GBD 2010), YLDs have been computed for 1160 sequelae resulting from 289 disease and injury causes, by multiplying the number of people living with each sequela by an associated disability weight. Extensive efforts were made to standardise and streamline the list of health con sequences across diseases, and as a result the 1160 sequelae have been mapped into a set of 220 distinct health states that capture the most salient differences in symptoms and functioning.⁶ The disability weight for a health state is a number on a scale from zero to one that represents the severity of health loss associated with the state. A value of 0 implies that a health state is equivalent to full health, and a value of 1 implies that a state is equivalent to death.

The previous comprehensive estimation of the global burden of disease (undertaken in the final revision of GBD 1990, which was published in 1996⁷) used the judgments of a small group of health-care profess ionals to establish disability weights for 483 sequelae of 131 diseases and injuries. These disability weights were used widely in WHO’s revisions of the GBD for 1999–2002, and 2004,^8–14 and in several national and subnational burden of disease studies.^15–22 Additions and amendments to the 1996 GBD weights have been assimilated selectively, largely on the basis of the Dutch Disability Weights study,^23–25 which adapted the GBD measurem ent approach from the 1996 study, with specific modifications to the descriptions of health states and addition of several states.

In view of the widespread use of GBD weights and the centrality of disability weights to the comparable measurement of disease burden across diverse causes, there has been much commentary and debate about the 1996 GBD weights and their derivatives, with critics challenging several aspects of the work.^26–40 Four broad topics dominate the debate: how to define the construct being measured; what methods of measurement of this construct to use to elicit responses from individuals or groups; whose responses should be elicited; and how universal the resulting weights are.

The starting point for measurement of disability weights should be a clear definition of the construct to be measured. Two distinct possibilities are to measure welfare loss or health loss. Welfare loss is a broad construct, and while health outcomes undoubtedly affect wellbeing generally, there are probably many additional influences from factors unrelated to health. A debate in health economics continues about whether to take overall wellbeing or health as the quantity to be maximised in health-policy choices,^41–43 with a parallel discussion in philosophy addressing the so-called separate spheres argument about appropriate accounting for health and non-health consequences in evaluation of health-care priorities.⁴⁴ However, a broad recognition that health might be afforded a special status that distinguishes it from other elements of wellbeing pervades many strains of discourse in public policy and international law. For example, pro tection of the right to health was most famously articulated in the International Covenant on Economic, Social and Cultural Rights, which affirmed “the right of everyone to the enjoyment of the highest attainable standard of physical and mental health”.⁴⁵ Most governments have ratified international treaties that include the right to health,⁴⁶ and provisions for health or the right to health appear in 135 national constitutions.⁴⁷ Although some have argued that the burden of disease should be quantified in terms of overall welfare loss because health and wellbeing are not separable,⁴⁸ others have challenged this view.^43,49,50

The measurement of disability weights in GBD 1990 did not clearly distinguish between health and wellbeing,⁷ but an evolution in the conceptual thinking behind the GBD has subsequently made explicit the aspiration to quantify health loss rather than welfare loss.⁵¹ This choice now distinguishes the GBD from other strains of research into weighing of health consequences; therefore, we draw attention to important implications of this difference when relevant.

A related set of issues concerns the specific measurement methods used to elicit judgments about health or welfare from individuals or groups. Measurement of disability weights in the 1996 GBD revision⁷ was partly based on a technique called the person trade-off, which locates comparisons between health outcomes within a resource allocation framework. Broad debate about methods pertains not only to measurement of the burden of disease, but also to related efforts to quantify outcomes in economic evaluations of health interventions.^52–55 Key issues range from technical concerns about the psychometric properties of different measures^55,56 to conceptual and ethical concerns about some methods, including the specific formulation of the person trade-off in the 1996 GBD weights.^26,57 Methods appropriate for eliciting judgments about welfare loss should be adapted to study health loss.⁴² There are ongoing debates about the relevance of specific values and judgments, such as time preference, risk aversion, or inequality aversion that could complicate the interpretation of responses to different elicitation methods.^52,58,59

A third question in discussions of disability weights is whose judgments should be used to derive these weights. Three respondent groups have been considered: health-care professionals, individuals who experience a health state, or the general public. Arguments for various respondent groups are both principled and pragmatic.^60–63 The 1996 GBD weights⁷ used health-care professionals on the basis that they would have knowledge of a diverse set of health states and would be able to make comparative judgments. Individuals in a health state have the most intimate knowledge of the reductions in function associated with that state; however, their comparative judgments with other health states will be based on asymmetric information.^62,64 Additionally, the capacity of individuals with some chronic disorders to adapt to their circumstances could lead to underestimation of the health loss associated with a particular state.^7,65,66 Much of the scientific literature about health-state weights uses the responses of the general public, on the basis of the argument that, in a democratic society, the views of the general public are relevant in comparative assessments that inform public policy.^61,67

Finally, the universality of disability weights has been much discussed. A central theme in some critiques of DALYs and in broader discussions of disability has been the contextualisation of disability within a particular social and cultural environment,^3,36 which raises questions about the possibility of significant cross-cultural variability in disability weights.^27,28,30,68 The universality of disability weights could largely depend on the specific construct that is chosen. A reasonable hypothesis is that the construct of health loss associated with different health states is more universal than is the construct of welfare loss. Welfare loss might be strongly affected by social context, support networks, and a myriad of individual preferences that might be less pertinent to measures of health loss. For example, the same health loss could be associated with different welfare losses in societies with disability and health insurance compared with those without such insurance. Fundamentally, questions about the universality of measures of health loss and welfare loss are empirical ones, but the evidence needed for systematic scientific investigation of these questions remains limited.

Beyond the disability weights measurement study from the 1996 revision of GBD 1990, several other investig ations of disability weights have added to both the empirical basis for weighting of health outcomes and the conceptual and philosophical debates about these measures. Some of these studies have focused on estimation of weights for many conditions with an adaptation of the 1996 GBD approach, as in the Dutch study,^23–25 the European Disability Weights Project,^68–70 and some national burden of disease studies.^71–73 Several others have focused on techniques to obtain these weights in culturally diverse settings.^74–76 Various studies have pro vided critical per spectives and new empirical measurem ents for disability weights pertaining to specific dis orders or categories—eg, depression,⁷⁷ suicidality,⁷⁸ stroke,⁷⁹ injuries,^80,81 oral health,^82,83 and neglected tropical diseases.^39,84–87

For GBD 2010, we have undertaken a compre hensive re-estimation of disability weights through a large-scale empirical study. This report describes the design, implementation, and results of the GBD 2010 disability weights measurement study, which yields novel disability weights with measurements of uncert ainty for the 220 unique health states arising from the array of disease and injury causes in the GBD. Addressing the four themes of persistent debate and responding to critiques of previous efforts to measure disability weights for the GBD, we focused on eliciting judgments about health loss rather than welfare loss; used a new, standardised approach to measurement with simple paired com parison questions; included a major emphasis on surveyi ng respondents from the general public; and used primary data collection in diverse communities to examine hypotheses about cultural variation in assess ments of disability.

Methods

Results

Table 1 shows the characteristics of the 13 902 participants in the household surveys and the 16 328 participants in the web survey. The web survey included respondents from 167 countries, 27 of which had at least 50 respondents (figure 1). 7180 (44%) participants in the web survey were from the USA. 10 579 (93%) of 11 320 selected respondents in the four face-to-face household surveys participated, and the probability of response was at least 88% in each site. 3323 (69%) of 4833 selected respondents in the US telephone survey participated. Of all 20 465 people who entered the web survey and were eligible to participate, 13 391 (65%) completed the full questionnaire and an additional 2937 (14%) completed at least part of the survey. As we had expected, respondents in the web-based survey had much higher levels of educational attainment than respondents in any of the household surveys.

Figure 2 presents a series of heat maps offering a visual comparison of the paired comparison response probabilities in each of the household surveys and in the analogous version of the web survey. Each heat map displays a matrix of response probabilities for the 108×108 different pairwise comparisons from the set of 108 health states common to all surveys. The degree of measurement error across surveys varied, with the noisiest results in Bangladesh (figure 2); however, the surveys produced fairly similar information content within the paired comparison responses overall.

Probit regression analyses were run on a pooled dataset, providing 379 492 paired comparison responses, and on datasets for each individual survey. Comparing results from regression analyses for each individual survey with pooled data, we confirmed a high degree of consistency between sites (figure 3). Pearson’s correlation coefficients between individual survey results and the results from the pooled analyses were 0·9 or higher in all cases except Bangladesh (figure 3). Comparing the web survey, which had the most highly educated respondents, with the survey in Tanzania, which had respondents with the lowest average educational levels, we noted a correlation of 0·89. Comparisons based on Lin’s concordance correlation coefficient were virtually identical to the Pearson’s results in most cases; the largest discrepancy between the two measures appeared in Bangladesh, where Lin’s coefficient was 0·72.

Table 2 shows the estimated disability weights and 95% uncertainty interv als for all 220 health states in the GBD 2010. Most were located on the mild end of the severity scale (figure 4). 58 (26%) unique states had disability weights below 0·05, implying that 1 year of life lost because of prem ature mortality is equivalent to more than 20 years lived in the specific health state. States with weights below 0·01 included those associated with mild anaemia, mild hearing or vision loss, secondary infertility, and several others (table 2). The outcomes with the highest disability weights were acute schizo phrenia and severe multiple sclerosis (table 2).

We identified 126 states in the present study that had corresponding sequelae in WHO’s update of the GBD for 2004.¹⁴ Some were directly comparable because the health states were specified consistently in the two studies. For the remainder, health-state definitions did vary between the new and old versions (eg, the old study had one sequela for multiple sclerosis but ours included mild, moderate, and severe multiple sclerosis states), which implies that the comparisons are between outcomes that are similar but not identical (appendix).

There is a broad pattern of agreement between the old and new weights, particularly in the moderate to severe range (figure 5). Overall, the correlation between the old and new weights was 0·70 (figure 5). However, in the range below 0·2—where most of the new weights reside—many states have substantially lower weights in our analysis than previously, which could have important implications for estimates of the overall burden asso ciated with specific causes. In some cases, decreased weights in our study reflect a more detailed disaggregation of mild outcomes. For example, WHO’s update for 2004¹⁴ combined mild and moderate chronic obstructive pulmonary disease (with a weight of 0·17), whereas we have recorded large differences in the disability weight for mild (0·015) versus moderate disease (0·19). In other cases when the description of outcomes is fairly unamb iguous, we nevertheless have reported substantially lower weights than previously—eg, infertility (previously 0·18; cur rently 0·01), amputated fingers (previously 0·10; currently 0·03), moderate to profound hearing loss (previously 0·12–0·33; currently 0·02–0·03), and blindn ess (prev iously 0·60, currently 0·20). However, several states are associated with substantially higher weights in our study—eg, untreated epilepsy (previously 0·15; currently 0·42), illicit drug use disorders (previously 0·25 overall, with no dis aggre gation for specific drugs; currently ranging as high as 0·64 for opioid dependence), and acute low back pain (previously 0·06, currently 0·27).

Discussion

This study, to our knowledge, is the largest empirical effort to date to measure disability weights for a wide array of health outcomes across a diverse range of populations. Several important findings emerged. First, we have demonstrated that it is feasible to elicit assessments about a wide variety of health outcomes in virtually any population, even where educational attain ment is low. Second, we have established the utility of a new approach to elicit this information with much simpler questions than have been used before and fairly straight-forward methods of data analysis. Third—and perhaps most importantly—we have reported compelling evidence that contradicts the prevailing hypothesis that assessments of disability must vary widely across samples with diverse cultural, educational, environmental, or demographic circumstances.

The design of this new study responds to critiques of past efforts to measure disability weights for the GBD in several ways. The main way to elicit assessments of health in our study was simple paired comparison questions, which were comprehensible by people with little or no formal schooling who were well represented in our sample. This method creates opportunities to acquire input from a broad cross-section of respondents who are usually excluded from such measurement exercises. Indeed, a major strength that distinguishes this study from others is that the new approach accommodates data collection through population-based household surveys rather than from expert panels. The reliance on health professionals has been a frequent target for criticism of existing disability weights,^27,38,39 which is consistent with the premise that these weights should reflect a broad social understanding of the relative importance of different aspects of health. Respondents in this study were asked to compare lay descriptions that emphasised the consequences of one condition. By contrast, health-care professionals might unwittingly take into account the typical comorbidities seen with particular conditions; their judgments of severity could reflect not only the health state being described, but also what usually goes alongside because of related conditions.

Data from these surveys provide extensive empirical evidence for critical questions about the extent of variation in comparative health assessments across different settings. Recent discussions about disability have emphasised the importance of recognising the effects of contextual differences on levels of disability experienced by people with a particular health condition in various settings. Our findings suggest that concerns about pervasive variations in health assessments across settings due to these differences in contextual factors could be overstated. In our study, a high degree of cons istency was reported in paired comparison responses across surveys with highly heterogeneous respondents. For example, the web-based survey was done in a non-random, self-selected, highly educated sample, and yet the analyses of these results showed that the correlation with the representative population sample from Tanz ania where educational attainment was lowest was strong. Although some variation between the results of individual surveys and those from the pooled analysis of data from all respondents remained, even the lowest correlation was roughly 0·75.

This high degree of consistency contrasts with arguments and results in several commentaries and small empirical studies.^27,28,38 One previous study²⁷ compared condition rankings elicited from 241 key informants in 14 countries and showed that some differences between countries and types of informant groups were large enough to question the universality of disability assessments. Our study—with larger samples in representative household surveys—yielded higher correlation coefficients overall than did this previous investigation. While the somewhat different outcomes in these two studies might elude definitive explanation, we could speculate that the selection of key informants can introduce important heterogeneities across settings that are magnified in relatively small samples, whereas larger, population-based samples are less likely to show biases from selection. Likewise, our focus on the con struct of health loss as opposed to welfare loss might have been an important driver of the consistency of results. A further potential contributor to the high level of agreement in our study was the focus of the lay descriptions on functional consequences rather than on the aetiological cause. Another major exercise in the comparative evaluation of disability assessments across countries showed fairly high levels of agreement bet ween western European countries.⁶⁸ Our study rei nforces this finding in a more diverse array of settings.

There are important limitations of our study. First, while we included surveys in a set of countries that reflect widely divergent cultural perspectives and levels of economic development, and did the household surveys on the basis of stratified random sampling within particular locations, we do not suggest that the five household surveys in this study comprise a random sample of the world’s population, and we certainly make no such claims about the web-based survey. Results might have been different had we chosen another set of countries for the household surveys. Likewise, the web-based survey includes many respondents from North America, Australia, and western Europe, but few respondents from Africa or the Middle East; results could change with a different geographical concentration of respondents. In terms of individual respondent characteristics, our sample included fairly few respondents aged 70 years or older, and the web survey had a heavy skew towards female respondents.

While the high correlations across settings give support to the proposition that there is a broadly shared set of common values for health losses that transcends narrow cultural and geographical boundaries, replication of this study in other settings will undoubtedly add nuance to the understanding of which aspects of judgments about health are most closely shared across settings and which are most variable. Indeed, the broad patterns of consistency overall could mask important examples of specific conditions for which ratings do vary across settings. More detailed examination of this variation on a case-by-case basis and reports of further empirical evidence will no doubt lead to interesting findings and generate new hypotheses about specific notions of health that are shared broadly versus those that diverge in important ways. In terms of the measures of consistency examined, it is also important to note that while our results point to a high degree of consistency in responses on direct comparisons between health states, the study did not include population health equivalence questions in the household surveys. Therefore, we are not able to draw conclusions about whether respondents make similar trade-offs between fatal and non-fatal outcomes across countries.

One of the most important limitations is that our study assigns weights to conditions presented as brief descriptions in lay language. With the aim to develop accurate but parsimonious lay descriptions for all conditions, some aspects of health states were inevitably omitted in the interest of simplicity, comprehensibility, and feasibility. While we made great efforts in this study to ensure consistency in language across conditions and to avoid ambiguous terms, some variability remained; differences in disability weights could be sensitive to this variation. For some examples such as illicit drug use disorders, we recorded notably higher results than in previous studies. We suspect that part of the reason for this result was that the descriptions of illicit drug use disorders attributed a functional outcome to a particular cause, which was deliberately avoided in other health states. Other work has indicated that labelling of alternatives in discrete choice experiments can substantially alter respondent choice.⁹⁵ In our study, explicit reference to drug use disorders as the cause of the functional outcomes described could have biased disability weights upwards partly by introducing confounding considera tions of comorbidities that could be more common in individuals with drug use disorders than in those with out. Moreover, with references to behaviours that might invoke moral or legal judgments or social disapproval, the descriptions could have led some respondents to regard the outcome less favourably than if they had focused exclusively on health consequences.

In some cases when disability weights were surprisingly low (eg, for profound hearing loss), the result might have been sensitive to omission of aspects of the health state that were pertinent to assessments about the severity of health loss, such as anxiety or depression. Further empirical work is needed to know whether addition of aspects like depression would bridge the difference between the existing weights and the new ones. In addition to affecting the average weights estimated in the study, parsimony in description of health outcomes could have exaggerated the consistency of responses across settings—eg, by eliminating additional variation in the weighing of omitted elements. Parsimony could also produce an opposite effect, namely inflation of variation, by introducing cultural differences in the ways that respondents impute omitted aspects of the health outcomes on the basis of their own knowledge and local circumstances.

In other cases, we noted differences from previous GBD weights that are harder to explain in terms of specific aspects of the lay descriptions. For example, intellectual impairments and development disorders were rated more mildly here than previously. The general public might consider intellectual disability to be less of a reduction in health than does a group of highly educated professionals. The disability weight for blindness is also substantially lower in our study than in previous GBD iterations. The decreased weight could be partly related to the use of a health construct rather than a welfare construct. In cases such as these when the difference between old and new weights is large, further empirical work to try to identify the reasons for these differences would be of high interest.

In the future, it will be worthwhile to consider extending the approach used to elicit comparisons for outcomes described in a generically standardised way—eg, with one of the many popular health status meas urement instruments such as SF-12 or EQ-5D.^40,96–99 A drawback of such an approach is that these instru ments tend to focus parsimoniously on a few core dimensions of health, which might not capture the most salient functional consequences and symptoms assoc iated with the wide array of health outcomes in the GBD. Nevertheless, the approach would allow linkage with representative household surveys that elicit both generic functional health descriptors and specific diag noses, which would in turn allow the characterisation of the distribution of health states associated with particular sequelae in the population. In our study, a focus on one typical case description or at most a small number of discrete severity levels could have obscured important variation in the range of health states associated with particular sequelae. Another benefit of linking descript ions to generic health status measurement systems is that it would allow correction of functional health information associated with a specified health problem for the presence of comorbidities. In GBD 2010, a related method has been used in a more limited way to determine the distribution of individuals with a part icular diagnosis across different health states as well as the fraction of time spent asymptomatic.¹⁰⁰

Despite these limitations, we emphasise that this study provides an uncommonly extensive empirical basis for understanding views of health across a diverse set of populations. It establishes a new set of disability weights that we hope will supply a crucial resource for assessments of burden of disease and for analyses of the comparative effectiveness and cost-effectiveness of interventions. We suggest that the new set of weights offer a substantially more rigorous and credible basis for weighing outcomes in these analyses than did the weights that were measured previously, which continue to be widely used. Data in this study offer new opport unities for empirically based inquiry into the possibility of substantial cross-cultural heterogeneity in assessm ents of disability relating to specific health conditions. Notably, we did not observe evidence to support the hypothesis that comparative assessments of health at a global level are undermined by extensive cultural vari ation. On the contrary, we have reported strong evidence that many aspects of individuals’ assessments of health outcomes seem to reflect common values, affirming universal aspirations for averting negative health out comes such as pain or depression and for enjoying high levels of functioning in domains of health such as mobility.

Supplementary Material