Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Blinded Outcome Assessment Was Infrequently Used and Poorly Reported in Open Trials

Abstract

Objective

Unblinded outcome assessment can lead to biased estimates of treatment effect in randomised trials. We reviewed published trials to assess how often blinded assessment is used, and whether its use varies according to the type of outcome or assessor.

Design and setting

A review of parallel group, individually randomised phase III trials published in four general medical journals (BMJ, Journal of the American Medical Association, The Lancet, and New England Journal of Medicine) in 2010.

Main outcome measures

Whether assessment of the primary outcome was blinded, and whether this differed according to outcome or assessor type.

Results

We identified 258 eligible trials. Of these, 106 (41%) were reported as double-blind, and 152 (59%) as partially or fully open-label (that is, they included some groups who were unblinded, such as patients, those delivering the intervention, or those in charge of medical care). Of the 152 open trials, 125 required outcome assessment. Of these 125 trials, only 26% stated that outcome assessment was blinded; 51% gave no information on whether assessment was blinded or not. Furthermore, 18% of trials did not state who performed the assessment. The choice of outcome type (e.g. instrument measured, rated, or naturally occurring event) did not appear to influence whether blinded assessment was performed (range 24-32% for the most common outcome types). However, the choice of outcome assessor did influence blinding; independent assessors were blinded much more frequently (71%) than participant (5%) or physician (24%) assessors. Despite this, open trials did not use independent assessors any more frequently than double-blind trials (17% vs. 18% respectively).

Conclusions

Blinding of outcome assessors is infrequently used and poorly reported. Increased use of independent assessors could increase the frequency of blinded assessment.

Introduction

A key component of randomised controlled trials is the assessment of patient outcomes, which involves assigning an outcome value to each trial participant. It is often recommended that outcome assessors are blinded to treatment allocation, as failure to do so can lead to systematic differences between treatment groups, resulting in biased estimates of the treatment effect [110]. Previous reviews have found that unblinded outcome assessment can lead to estimates of treatment effect that are exaggerated between 27% and 68%, depending on outcome type [13].

Despite the wide body of evidence supporting blinded outcome assessment to prevent bias in estimated treatment effects, it is still unclear how often it is used in practice. Most previous reviews have largely focused on specific disease areas [9, 11] or specific subsets of trials [12, 13], such as those that used at least some element of blinding. We therefore undertook a review of trials published in general medical journals to assess how often blinded outcome assessment was used, with a focus on trials that were not fully blinded.

Methods

We included parallel group, individually randomized, controlled trials which were published in one of four major medical journals in 2010 (BMJ, Journal of the American Medical Association, The Lancet, and New England Journal of Medicine). Pilot and phase I or II trials, as well as articles that reported only secondary analyses were excluded. Trials were identified from the electronic table of contents for each journal. One reviewer determined whether trials met the eligibility criteria for all trials identified; a second reviewer assessed this for a subset of trials (n = 61), and agreement was 100%. Full details of the search strategy and inclusion/exclusion criteria have been published elsewhere [14]. We chose to review phase III articles from high impact general medical journals to focus on trials that were likely to have the greatest impact on patient care.

We extracted data onto a standardised form, and all trials were independently assessed by two different reviewers. Disagreements between reviewers were resolved by discussion, or by a third reviewer if necessary. For each trial, information was extracted on the blinding status of the trial, how the primary outcome was recorded, who assessed the primary outcome, and whether the assessor was blinded to treatment allocation. We identified the primary outcome as follows: (a) if only one outcome was identified as the primary, we used this; (b) if no outcomes were identified as the primary, we used the first outcome presented in the results section of the abstract; and (c) if multiple outcomes were identified as primary, we used the first of these outcomes presented in the results section of the abstract.

We classified trials as either being reported as double-blind or partially (or fully) open-label (referred to as ‘open’). We categorised trials as being ‘reported as double-blind’ if this was explicitly stated in the article, if the article stated that everyone involved in the study was blinded, or if they used a placebo or sham treatment that was described as being identical to the intervention in terms of appearance. We categorised trials as being ‘reported as open’ when it was explicitly stated, or implied through the description of the interventions, that at least some trial personnel were not blinded to treatment allocation. This included (but was not limited to) participants, those administering the intervention, those providing medical care apart from the intervention, and those assessing outcomes.

Definition of outcome and assessor types

Outcomes were grouped into the following categories: instrument measured, rated, naturally occurring event, action-based event, all-cause mortality, other, or a composite of multiple outcome types. Full details are shown in Table 1. We defined instrument measured, rated, naturally occurring and action-based events as requiring assessment. We defined all-cause mortality as not requiring assessment. For outcomes defined as ‘other’, we decided on a case by case basis whether they required assessment.

Assessors were classified as follows: participant, carer or physician, and independent assessors. Carers or physicians were people who provided some aspect of medical care, or helped to deliver the intervention. Independent assessors were those who, apart from assessing the outcome, had no other involvement with participants. For each outcome, we defined the assessor as the person who primarily recorded or judged the outcome. For example, for rated outcomes, the assessor was the person who made the rating; for naturally occurring events, the assessor was the person who decided whether the event had occurred or not; for action-based events, the assessor was the person who made the decision which led to the event; and for instrument measured outcomes, the assessor was the person who operated the instrument which directly provided outcome results.

We defined outcome assessment as being blinded if the article either (a) stated that blinded outcome assessment had been performed; or (b) identified who assessed the outcome, and identified this person as being blinded. We defined outcome assessment as being unblinded if the article either (i) stated that the assessment was not blinded; or (ii) identified the person who performed the assessment, and identified this person as being unblinded. When the assessment did not fall into either category, we listed it as unclear.

Results

Trial characteristics

In total, 258 trials met our eligibility criteria and were included in our review. General trial characteristics are shown in Table 2. Overall, 106 trials (41%) were reported as double-blinded, and 152 (59%) as partially or fully open-label (hereafter referred to as ‘open’). Of the trials reported as double blind, the majority used a pharmacological intervention (n = 97/106, 92%). In comparison, the majority of open trials used a non-pharmacological intervention (n = 100/152, 66%). Open trials often had poor descriptions of who was blinded; 61% did not state whether participants were blinded, and 68% did not state whether carers or physicians were blinded.

Use of blinded outcome assessment

Results are shown in Table 3 and in Fig 1. In total, 27 open trials used an outcome that did not require formal assessment (26 all-cause mortality, 1 other). Among the 125 open trials where outcome assessment was required, only 33 (26%) stated that they used blinded outcome assessment; 64 (51%) did not report whether assessment was blinded or unblinded. Furthermore, 22 trials (18%) did not state who performed the assessment.

Blinding status by outcome or assessor type

Results are shown in Tables 4 and 5. In open trials, the choice of outcome type appeared to have little effect on whether blinded outcome assessment was used. The proportion of trials using blinded assessment for instrument measured outcomes, rated outcomes, naturally occurring events, or composite outcomes varied between 24–32%. The one exception was action-based events, where only one open trial (11%) used blinded assessment.

thumbnail
Table 4. Blinding of different outcome types in trials reported as partially or fully open-label.

https://doi.org/10.1371/journal.pone.0131926.t004

thumbnail
Table 5. Blinding status by assessor type in trials reported as partially or fully open-label.

https://doi.org/10.1371/journal.pone.0131926.t005

Conversely, the rates of blinded outcome assessment varied substantially depending on the choice of assessor. Only one open trial (5%) which used patient assessment and 6 trials (24%) which used a carer or physician assessment used blinded assessment; in comparison, 15 trials (71%) using an independent assessor used blinded assessment.

There did not appear to be any difference between double-blind and open trials in the use of independent assessors (double-blind 18% vs. open 17%). However, there may have been differences across specific outcomes; for example, open trials used independent assessment for 43% of naturally occurring events, compared with 27% of double-blind trials.

Discussion

There is substantial evidence to suggest that unblinded outcome assessment can lead to biased estimates of treatment effect. It is therefore recommended that blinded outcome assessment is used to avoid this source of bias. Our review identified 125 partially or fully open-label trials which required outcome assessment. Despite recommended practice, only 26% of trials used blinded assessment. The true figure may be higher, but is difficult to ascertain for certain due to poor reporting; over half of trials did not provide any information on whether outcome assessors were blind or not. Furthermore, 18% of trials did not state who performed the assessment.

We found that lack of blinded assessment was not associated with the outcome type (apart from action-based outcomes), but did differ according to the assessor. Assessment was blinded in only 5% of trials using patient assessment, and 24% using physician assessment. In contrast, assessment was blinded in 71% of trials using an independent assessor.

Despite the fact that using an independent assessor can substantially increase the feasibility of using blinded assessment, and thus reduce the potential for bias, we found that open trials were no more likely than double-blinded trials to use independent assessment. This surprising result has been noted before by Dechartes et al [15], who found that adjudication committees (typically a committee consisting of clinical experts not involved in patient care) were equally likely to be used in trials of low and high risk of ascertainment bias.

Overall, reporting of various aspects of blinding in open trials was poor. Very few trials explicitly stated whether patients or carers were blinded. This is similar to what has been found in previous studies [13, 16], and suggests that guidelines for better reporting have not been as well adopted as they should be. Given that randomised trials are often used in treatment guidelines, it is essential that the methodology of these trials is clearly reported so that others can adequately judge the relative merits of each trial.

Problematic reporting was not limited to open trials. Amongst self-reported double-blind trials, only 49% reported whether participants were blinded, 58% whether carers or physicians are blinded, and 54% whether outcome assessment was blinded. Previous research has found that some self-reported double-blind trials are actually partially open [13], and we found one trial which described itself as double-blind but actually had unblinded carers or physicians. Given the lack of details surrounding blinding in our review, it is possible that a number of other self-reported double-blind trials were actually open. It is therefore important even for double-blind trials to carefully describe who was (or was not) blinded.

Only 9 of 109 non-pharmacological trials (8%) used a double-blind design. This is not surprising given the challenges associated with blinding patients and physicians in these scenarios [17]. However, useful methods of blinding in these circumstances have previously been documented by Boutron et al [12], and could be more frequently adopted.

It is unclear why so few open trials used blinded outcome assessment. In some circumstances, blinded assessment is not feasible, for example patient reported outcomes in trials when patients are unblinded due to the nature of the intervention. In some cases when blinded outcome assessment is not possible, it may be possible to modify the outcome definition to reduce the risk of bias [18]. However, in many cases blinded assessment is possible [11, 12, 19]. For example, in most trials it is possible to have an independent person who is not otherwise involved in the medical care for a specific patient (and is therefore blinded to their treatment allocation) to assess their wellbeing. Often, photos or recordings can be taken and sent to an independent adjudication committee for assessment, and with new technological developments, this is a particularly promising way of ensuring blinded assessment.

The use of independent assessors may also be helpful in double-blind trials where the integrity of the blind is uncertain. For example, in some double-blind trials, the methods used to blind participants and investigators may fail, and they may become aware of the treatment allocation. Conversely, patients or physicians may be able to guess their treatment allocation based on their symptoms or side-effects. It is unclear how often this inadvertent unblinding occurs in practice, as it is rarely tested or reported [20, 21]. Therefore, more frequent use of independent assessors could also be considered in double-blind trials where the integrity of the blind is uncertain.

There are some limitations to our study. The articles we reviewed were written prior to the release of the 2010 CONSORT statement [22], and reporting may have subsequently improved since then. Secondly, we included only trials from high impact general medical journals; results may not be generalizable to lower impact or specialist journals.

Conclusions

Blinded outcome assessment was poorly reported, and was infrequently used. Investigators should consider using independent assessors more often to increase the feasibility of blinded assessment.

Author Contributions

Conceived and designed the experiments: BCK. Performed the experiments: BCK SR SC. Analyzed the data: BCK. Contributed reagents/materials/analysis tools: BCK SR SC. Wrote the paper: BCK SR SC.

References

  1. 1. Hrobjartsson A, Thomsen AS, Emanuelsson F, Tendal B, Hilden J, Boutron I, et al. Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. BMJ. 2012;344:e1119. pmid:22371859
  2. 2. Hrobjartsson A, Thomsen AS, Emanuelsson F, Tendal B, Hilden J, Boutron I, et al. Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors. CMAJ: Canadian Medical Association journal = journal de l'Association medicale canadienne. 2013;185(4):E201–11.
  3. 3. Hrobjartsson A, Thomsen AS, Emanuelsson F, Tendal B, Rasmussen JV, Hilden J, et al. Observer bias in randomized clinical trials with time-to-event outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. International journal of epidemiology. 2014.
  4. 4. Linde K, Witt CM, Streng A, Weidenhammer W, Wagenpfeil S, Brinkhaus B, et al. The impact of patient expectations on outcomes in four randomized controlled trials of acupuncture in patients with chronic pain. Pain. 2007;128(3):264–71. pmid:17257756
  5. 5. Noseworthy JH, Ebers GC, Vandervoort MK, Farquhar RE, Yetisir E, Roberts R. The impact of blinding on the results of a randomized, placebo-controlled multiple sclerosis clinical trial. Neurology. 1994;44(1):16–20. pmid:8290055
  6. 6. Poolman RW, Struijs PA, Krips R, Sierevelt IN, Marti RK, Farrokhyar F, et al. Reporting of outcomes in orthopaedic randomized trials: does blinding of outcome assessors matter? The Journal of bone and joint surgery American volume. 2007;89(3):550–8. pmid:17332104
  7. 7. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA: the journal of the American Medical Association. 1995;273(5):408–12.
  8. 8. Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008;336(7644):601–5. pmid:18316340
  9. 9. Liu CJ, LaValley M, Latham NK. Do unblinded assessors bias muscle strength outcomes in randomized controlled trials of progressive resistance strength training in older adults? American journal of physical medicine & rehabilitation / Association of Academic Physiatrists. 2011;90(3):190–6.
  10. 10. Hrobjartsson A, Emanuelsson F, Skou Thomsen AS, Hilden J, Brorson S. Bias due to lack of patient blinding in clinical trials. A systematic review of trials randomizing patients to blind and nonblind sub-studies. International journal of epidemiology. 2014;43(4):1272–83. pmid:24881045
  11. 11. Karanicolas PJ, Bhandari M, Taromi B, Akl EA, Bassler D, Alonso-Coello P, et al. Blinding of outcomes in trials of orthopaedic trauma: an opportunity to enhance the validity of clinical trials. The Journal of bone and joint surgery American volume. 2008;90(5):1026–33. pmid:18451395
  12. 12. Boutron I, Guittet L, Estellat C, Moher D, Hrobjartsson A, Ravaud P. Reporting methods of blinding in randomized trials assessing nonpharmacological treatments. PLoS medicine. 2007;4(2):e61. pmid:17311468
  13. 13. Haahr MT, Hrobjartsson A. Who is blinded in randomized clinical trials? A study of 200 trials and a survey of authors. Clin Trials. 2006;3(4):360–5. pmid:17060210
  14. 14. Kahan BC, Morris TP. Reporting and analysis of trials using stratified randomisation in leading medical journals: review and reanalysis. BMJ. 2012;345:e5840. pmid:22983531
  15. 15. Dechartres A, Boutron I, Roy C, Ravaud P. Inadequate planning and reporting of adjudication committees in clinical trials: recommendation proposal. Journal of clinical epidemiology. 2009;62(7):695–702. pmid:19135860
  16. 16. Montori VM, Bhandari M, Devereaux PJ, Manns BJ, Ghali WA, Guyatt GH. In the dark: the reporting of blinding status in randomized controlled trials. Journal of clinical epidemiology. 2002;55(8):787–90. pmid:12384193
  17. 17. Boutron I, Tubach F, Giraudeau B, Ravaud P. Blinding was judged more difficult to achieve and maintain in nonpharmacologic than pharmacologic trials. Journal of clinical epidemiology. 2004;57(6):543–50. pmid:15246122
  18. 18. Kahan BC, Cro S, Dore CJ, Bratton DJ, Rehal S, Maskell NA, et al. Reducing bias in open-label trials where blinded outcome assessment is not feasible: strategies from two randomised trials. Trials. 2014;15(1):456.
  19. 19. Karanicolas PJ, Bhandari M, Walter SD, Heels-Ansdell D, Guyatt GH. Radiographs of hip fractures were digitally altered to mask surgeons to the type of implant without compromising the reliability of quality ratings or making the rating process more difficult. Journal of clinical epidemiology. 2009;62(2):214–23 e1. pmid:18778914
  20. 20. Bello S, Moustgaard H, Hrobjartsson A. The risk of unblinding was infrequently and incompletely reported in 300 randomized clinical trial publications. Journal of clinical epidemiology. 2014;67(10):1059–69. pmid:24973822
  21. 21. Fisher S, Greenberg RP. How sound is the double-blind design for evaluating psychotropic drugs? The Journal of nervous and mental disease. 1993;181(6):345–50. pmid:8501455
  22. 22. Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c869. pmid:20332511