Keywords
Publication bias, transparency, audit
This article is included in the All trials matter collection.
Publication bias, transparency, audit
The results of clinical trials are used to make informed choices with patients about medical treatments. However, there is extensive and longstanding evidence that the results of clinical trials are routinely withheld from doctors, researchers, and patients. A current systematic review of all cohort studies following up registered trials, or trials with ethical approval, shows that approximately half fail to publish their results1. Evidence from an earlier review shows that studies with “negative” or non-significant results are twice as likely to be left unpublished2. Legislation, such as FDA Amendment Act 2007 (http://www.fda.gov/RegulatoryInformation/Legislation/SignificantAmendmentstotheFDCAct/FoodandDrugAdministrationAmendmentsActof2007/default.htm), which requires trials to post summary results on clinicaltrials.gov within 12 months of completion, have been widely ignored, with a compliance rate of one in five3,4. The FDA is entitled to impose fines of $10,000 a day on those breaching this law, but has never yet done so5,6. This public health problem has also been the subject of extensive campaigning. For example, the AllTrials campaign is currently supported by 89,000 individuals and 700 organisations, including major funders, professional bodies, patient organisations and government bodies (http://www.alltrials.net/).
Previous work suggests that some sponsors, companies, funders, and research sites may perform better than others5,7. In any sector, audit of the best and worst performers can be used to improve performance, allowing those with a poor performance to learn from those doing better. To be effective, however, audit should be repeated, and ideally ongoing8.
All work on publication bias to date relies on a single sweep of labour-intensive manual searches9,10, or a single attempt to automatically match registry entries to published papers using registry identification number11. Manual matching comes at high cost and does not give ongoing feedback. We therefore set out to: develop an online tool that automatically identifies trials with unreported results; present and rank the prevalence of publication failure, broken down by sponsor; and maintain the service, updating the data automatically, so that companies and research institutes are motivated to improve their performance.
The methods used by the online tool are as follows. Raw structured data on all studies in clinicaltrials.gov are downloaded in XML format. Studies are kept if they: have a study type “interventional” (excluding observational studies); have a “status” of “completed”; have a completion date more than 24 months ago, and after Jan 1 2006; are phase 2, 3, 4, or “n/a” (generally a device or behavioural intervention); no application to delay results posting has been filed (ascertained from the firstreceived_results_disposition_date tag); are conducted by a sponsor who has sponsored more than 30 trials (to exclude trials conducted by minor sponsors and make the ranking in the tool more informative).
Results are then sought for all included studies, using two methods. First the tool checks for structured results posted directly in clinicaltrials.gov, ascertained by the presence of the firstreceived_results_date tag. Secondly, the tool searches for the nct_id (registry ID number) of the trial in PubMed’s Secondary Source ID field. Since 2005, all trials with a registry ID in the body of the journal article text should have that ID replicated in this field (https://www.nlm.nih.gov/bsd/policy/clin_trials.html). However, since in our experience approximately 1.5% of PubMed records include a valid nct_id list in the abstract, but not the Secondary Source ID field, our tool additionally searches for this ID in the title or abstract text. We exclude results published before the completion date of the trial, or results that have the words “study protocol” in the title.
A final filter is then applied, with the aim of excluding publications reporting protocols or additional analysis and commentary, rather than trial results; after experimenting with the standard validated PubMed “therapy” filters (both broad and narrow) and a rudimentary search for “study protocol”, the former was used. A comparison of the three methods is reported in the accompanying iPython notebook [https://github.com/ebmdatalab/trialstracker]12.
Accepting that an automated tool cannot produce results with the accuracy of a manual search, we also performed some rudimentary checks of the output of the automated search against existing manual search cohorts. The overall prevalence of unreported studies found by the tool was compared against three previous studies on publication bias. In addition, disparities on individual studies found to be unreported by the tool were compared against the underlying data from a recent publication bias cohort study conducted using clinicaltrials.gov data.
The output data is then shared through an interactive website at https://trialstracker.ebmdatalab.net allowing users to rank sponsors by number of trials missing, number of trials conducted, and proportion of trials missing. Users can click on a sponsor name to examine the number and proportion of trials completed and reported from each year for that sponsor. The site URL changes as users focus on each organisation’s performance, so that users can easily share insights into the performance of an individual company or institution. By default sponsors are sorted by the highest number of unreported trials, rather than the highest proportion, in order to initially focus on larger and more well-known organisations. The site is designed responsively to be usable on mobile, tablet or desktop devices.
For transparency and replication, all code for the tool, with comments and all data sources, is available as an iPython notebook12. All software is shared as open source, under the MIT license. A full CSV is shared containing all data, including all studies before our filters are applied, allowing others to conduct additional analyses or sensitivity analyses with different filtering methods.
The TrialsTracker tool was successfully built and is now running online at https://trialstracker.ebmdatalab.net. Sample screenshots are presented in Figure 1 and Figure 2.
Since Jan 2006, trial sponsors included in our dataset have completed 25,927 eligible trials, of which 11,714 (45.2%) have failed to make results available. Table 1 to Table 4 report the sponsors with the top five highest number of unreported trials, the highest number of eligible trials, the highest proportion of unreported trials, and the lowest proportion of unreported trials. In total, 2390/8799 (27.2%) trials with sponsors classed as “industry” were identified as unreported; 122/470 (26.0%) trials with sponsors classed as “US Fed” were identified as unreported; 361/996 (36.2%) trials with sponsors classed as “NIH” were identified as unreported; 8841/15662 (56.4%) trials with sponsors classed as “other” were identified as unreported. We find that 8.7 million patients were enrolled in trials that are identified as unreported.
A previous paper automatically matching registry entries to PubMed records and clinicaltrials.gov results found 55% had no evidence of results11, consistent with our overall findings. A previous manual audit (of which BG is co-author) found 56% of trials conducted in the University of Oxford reported results; our method also found 56% for the same institution9. A previous manual audit examined 4347 trials across 51 academic medical centres7. We compared their individual study data against ours and found that 2562 trials (62.6%) in their cohort were also in ours, but note that their study only represented 2% of our total cohort. For studies in both cohorts we found 60% reported results, while they found 66%. Of studies in both cohorts: 1149 were found “reported” by both; 534 studies were found “unreported” by both; 497 were found “reported” by their method and “unreported” by ours; 382 were found “unreported” by theirs and “reported” by ours.
The tool was successfully built, and is now fully functional online. We found non-publication rates consistent with those from previous work using manual searches, and reasonable consistency with individual study matches from a previous manual cohort. A wide range of publication failure rates were apparent in the data.
Our tool is the first to provide live ongoing interactive monitoring of failure to publish the results of clinical trials. The method of automatic matching has strengths and weaknesses. It can be run automatically, at a lower unit cost than a manual search, and therefore allows coverage of more trials than any traditional cohort study. It also permits repeated re-analysis at minimal additional marginal cost compared to a manual search.
In corollary, the efficiency of automatic matching also brings challenges around specificity and sensitivity. Firstly, there may be false adjudications of non-publication, i.e. if a trial’s results paper does not include its registry identifier. However, since 2005 all major medical journals (through the International Committee of Medical Journal Editors; http://icmje.org/recommendations/browse/publishing-and-editorial-issues/clinical-trial-registration.html) have required trials to be registered, and all trials should include their registry ID in the text. Therefore, in our view, the responsibility for results being undiscoverable, when the registry ID is not included by the trialists, lies solely with the trialists; research that is hard to discover is not transparently reported. We hope that in the future better methods for probabilistic record linkage will also be available for wider use13. Secondly, there may be false positives, where a study identified through ID matching and then filtered, is in fact not reporting results. We have used standard filters to account for this, and we are keen to improve our method in the light of concrete constructive feedback. Our checks for consistency against overall prevalence findings and individual study data from previous research to a large extent exclude gross errors in prevalence figures.
Notably there are specific additional methods for linking clinicaltrials.gov records to PubMed records that we tried and rejected. Some trials have a link to a PubMed record directly in the clinicaltrials.gov results_reference tag, which ClinicalTrials documentation (https://prsinfo.clinicaltrials.gov/definitions.html) suggests indicates results from a publication. We found 2263 eligible trials had such tags, but no summary results on ClinicalTrials.gov. However, on manual examination, we found these are often erroneous, and commonly report results of unrelated studies from several years previously. In discussion, clinicaltrials.gov staff confirmed that this field is neither policed nor subject to substantial editorial control (personal communication with Annice Bergeris).
Our findings are consistent with previous work on publication bias1, finding that approximately half of trials fail to report results. Previous studies have used 2007 as their start date for expecting results to be made available, reflecting the FDA Amendment Act 2007. We did not use this date, as this legislation has been widely ignored5,6, and because we regard sharing results as an ethical obligation, not a legal one. Our methods accept results posting at any time after study completion, and any sponsor posting results for any trial since 2006 will find their results improve in our live data.
We have previously argued that live ongoing monitoring of trials transparency will help to drive up standards, especially if this information is used by clinicians, policymakers, ethics committees, regulators, patients, patient groups, healthcare payers, and research funders, to impose negative consequences on those who engage in the unethical practice of withholding trial results from doctors, researchers, and patients14. Recent comments by US Vice President Joe Biden threatened to withhold financial support from publicly-funded researchers who fail to report clinical trial results, suggesting some consequences may arise6. We would be happy to collaborate or work with organisations seeking to get a better understanding of their own failure to publish, and wishing to act on this data.
We have also previously argued that medicine has an “information architecture” problem; all publicly accessible documents and data on all clinical trials should be aggregated and indexed for comparison and gap identification, and that good knowledge management and better use of trial identifiers will facilitate this15. At present, medicine faces serious shortcomings in this area. With 75 trials and 11 systematic reviews being published every day on average16 better knowledge management must be a priority.
We have shared all our underlying data so that others can explore in detail non-publication for specific studies, interventions, companies, funders, sponsors, or institutions that interest them. We believe that research work on research methods and reporting should go beyond identifying the overall prevalence of problems, and identify individual people and organisations who are performing poorly, in order to both support and incentivise them to improve. That is only possible with ongoing monitoring and feedback on individual studies, an approach we have taken on other projects such as COMPare17,18. We hope that others will also pursue this model of audit and feedback, and assess its impact on performance.
We have designed, built, and launched an easily accessible online service that identifies sponsors who have failed in their duty to make results of clinical trials available.
Website available at: https://trialstracker.ebmdatalab.net
Latest source code: https://github.com/ebmdatalab/trialstracker
Archived source code as at the time of publication: DOI: 10.5281/zenodo.16352212
License: MIT license
BG conceived the project; both authors developed the analyses, trial matching and filtering methods; APS wrote the data-analysis script and built the interactive website; BG drafted the manuscript; both authors revised the manuscript; both authors are guarantor.
BG has received research funding from LJAF, the Wellcome Trust, the NHS National Institute for Health Research, the Health Foundation, and the WHO. BG is co-founder of the AllTrials campaign on trials transparency. BG receives personal income from speaking and writing for lay audiences on the misuse of science. APS receives income as a freelance software developer.
BG is funded by the Laura and John Arnold Foundation (LJAF) to conduct work on research integrity; APS is employed on this grant.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
We are grateful for constructive discussions on design and impact with Jess Fleminger, Carl Heneghan and Sile Lane.
Views | Downloads | |
---|---|---|
F1000Research | - | - |
PubMed Central
Data from PMC are received and updated monthly.
|
- | - |
Competing Interests: No competing interests were disclosed.
Competing Interests: I was joint co-author on a paper which was cited by this manuscript. Dr Goldacre has cited my work in a statement given to a House of Commons select committee, and has given a statement to me in support of an application which I made to the University of Nottingham which supported the impact of my work in this field.
Alongside their report, reviewers assign a status to the article:
Invited Reviewers | ||
---|---|---|
1 | 2 | |
Version 1 03 Nov 16 |
read | read |
Provide sufficient details of any financial or non-financial competing interests to enable users to assess whether your comments might lead a reasonable person to question your impartiality. Consider the following examples, but note that this is not an exhaustive list:
Sign up for content alerts and receive a weekly or monthly email with all newly published articles
Already registered? Sign in
The email address should be the one you originally registered with F1000.
You registered with F1000 via Google, so we cannot reset your password.
To sign in, please click here.
If you still need help with your Google account password, please click here.
You registered with F1000 via Facebook, so we cannot reset your password.
To sign in, please click here.
If you still need help with your Facebook account password, please click here.
If your email address is registered with us, we will email you instructions to reset your password.
If you think you should have received this email but it has not arrived, please check your spam filters and/or contact for further assistance.
Recent studies demonstrating high levels of disclosure of industry-sponsored research1-4 have arguably had little impact on public confidence, perhaps because ... Continue reading TrialsTracker: an important step on the trials transparency trail
Recent studies demonstrating high levels of disclosure of industry-sponsored research1-4 have arguably had little impact on public confidence, perhaps because they were conducted and funded by industry, limited in scope to a single institution or geography, and immediately appeared outdated. To restore public confidence in research funded by the pharmaceutical industry, a method of assessing clinical trials disclosure is needed that is independent, global and up-to-date. In TrialsTracker, Powell-Smith and Goldacre have created a resource with the potential to be just that.
TrialsTracker5 is a semi-automated system that identifies completed studies in the world’s largest clinical trials registry, ClinicalTrials.gov,6,7 and then identifies which have had their results posted to the registry or published in a peer-reviewed journal indexed by PubMed. The strengths of TrialsTracker include: providing comparative data (the reported rate of disclosure of industry-sponsored studies of 72.8% comparing favourably with that for non-industry-sponsored studies of 45.6%); clarifying trends (disclosure rates having nearly doubled between 2006 and 2008, from 33.2% to 65.1%, before tailing off); and making data available for further analysis and checking. Assessments of disclosure rates by geographical region, type of intervention (e.g. drug, vaccine or medical device) and phase of clinical development may also be possible within the TrialsTracker data set.
As highlighted by other commenters on the article, TrialsTracker misses results disclosed on clinical trials registries hosted by individual pharmaceutical companies and other funding, regulatory and governmental institutions, with the likely outcome that trial disclosure rates will be under-estimated. While Powell-Smith and Goldacre showed that overall disclosure rates calculated from TrialsTracker were consistent with those reported previously, when comparing at the trial level the results of TrialsTracker with those of a previous manual audit, results were concordant for only 65.7% of trials, with TrialsTracker incorrectly describing 14.9% as reported and incorrectly describing 19.4% as unreported. However, comparison of TrialsTracker results with the findings of a study published last year provides reassurance that TrialsTracker might be accurate, for some industry sponsors at least.1 The study, by Mooney and Fay, showed that, of 76 eligible Pfizer-sponsored clinical trials that were completed in 2010, 65 had been published by April 2015, and a further 4 were in preparation or submitted for publication. Only 7 studies (9.2%) were not going to be published (all of which had been terminated early owing to recruitment challenges). In comparison, TrialsTracker lists 77 Pfizer studies completed in 2010, only 2 (2.6%) of which were classified as undisclosed by October 2016 in publications or the registry.5 Further comparison of the results obtained with each approach would enable additional validation of the TrialsTracker methodology.
It would be helpful if Powell Smith and Goldacre could address in their article some other methodological aspects: how often the data are updated and how sponsors of more than 30 studies are identified. It is also worth considering what the implications of excluding sponsors of 30 studies or fewer might be. Analysing data from a recent study of nearly 70 000 clinical trials registered on ClinicalTrials.gov in 2005–2014,8 it can be calculated that sponsors of 30 studies or fewer (70.2% of all sponsors) are responsible for 54.7% of registered trials. If sponsors of a large number of studies are better at disclosing their data, excluding sponsors of smaller numbers of studies will lead to an understatement of the true extent of the failure to disclose results.
The authors’ characterization of studies as undisclosed when the results have been disclosed in a registry other than ClinicalTrials.gov is clearly erroneous. However, it is difficult not to have sympathy with their plight, particularly when one of them is leading an initiative that aims to link all publicly available information on individual clinical trials across multiple platforms.9 Additional methods to link multiple registries with published results may be helpful,10 but the increasing availability of clinical trials data (e.g. from reports submitted to regulatory agencies) makes this a moving target.11,12 The validity of TrialsTracker could potentially be improved by the authors tracking trials reported on registries other than ClinicalTrials.gov, by sponsors posting results on ClinicalTrials.gov, by ClinicalTrials.gov linking to results posted on other registries, and by journals or PubMed allowing ClinicalTrials.gov registration numbers to be added to the abstract or associated meta-data of published articles retrospectively (ideally without generating an erratum). Working together to optimize the TrialsTracker methodology, diverse stakeholders can maximize transparency and strengthen trust in evidence arising from clinical trials.
References
Recent studies demonstrating high levels of disclosure of industry-sponsored research1-4 have arguably had little impact on public confidence, perhaps because they were conducted and funded by industry, limited in scope to a single institution or geography, and immediately appeared outdated. To restore public confidence in research funded by the pharmaceutical industry, a method of assessing clinical trials disclosure is needed that is independent, global and up-to-date. In TrialsTracker, Powell-Smith and Goldacre have created a resource with the potential to be just that.
TrialsTracker5 is a semi-automated system that identifies completed studies in the world’s largest clinical trials registry, ClinicalTrials.gov,6,7 and then identifies which have had their results posted to the registry or published in a peer-reviewed journal indexed by PubMed. The strengths of TrialsTracker include: providing comparative data (the reported rate of disclosure of industry-sponsored studies of 72.8% comparing favourably with that for non-industry-sponsored studies of 45.6%); clarifying trends (disclosure rates having nearly doubled between 2006 and 2008, from 33.2% to 65.1%, before tailing off); and making data available for further analysis and checking. Assessments of disclosure rates by geographical region, type of intervention (e.g. drug, vaccine or medical device) and phase of clinical development may also be possible within the TrialsTracker data set.
As highlighted by other commenters on the article, TrialsTracker misses results disclosed on clinical trials registries hosted by individual pharmaceutical companies and other funding, regulatory and governmental institutions, with the likely outcome that trial disclosure rates will be under-estimated. While Powell-Smith and Goldacre showed that overall disclosure rates calculated from TrialsTracker were consistent with those reported previously, when comparing at the trial level the results of TrialsTracker with those of a previous manual audit, results were concordant for only 65.7% of trials, with TrialsTracker incorrectly describing 14.9% as reported and incorrectly describing 19.4% as unreported. However, comparison of TrialsTracker results with the findings of a study published last year provides reassurance that TrialsTracker might be accurate, for some industry sponsors at least.1 The study, by Mooney and Fay, showed that, of 76 eligible Pfizer-sponsored clinical trials that were completed in 2010, 65 had been published by April 2015, and a further 4 were in preparation or submitted for publication. Only 7 studies (9.2%) were not going to be published (all of which had been terminated early owing to recruitment challenges). In comparison, TrialsTracker lists 77 Pfizer studies completed in 2010, only 2 (2.6%) of which were classified as undisclosed by October 2016 in publications or the registry.5 Further comparison of the results obtained with each approach would enable additional validation of the TrialsTracker methodology.
It would be helpful if Powell Smith and Goldacre could address in their article some other methodological aspects: how often the data are updated and how sponsors of more than 30 studies are identified. It is also worth considering what the implications of excluding sponsors of 30 studies or fewer might be. Analysing data from a recent study of nearly 70 000 clinical trials registered on ClinicalTrials.gov in 2005–2014,8 it can be calculated that sponsors of 30 studies or fewer (70.2% of all sponsors) are responsible for 54.7% of registered trials. If sponsors of a large number of studies are better at disclosing their data, excluding sponsors of smaller numbers of studies will lead to an understatement of the true extent of the failure to disclose results.
The authors’ characterization of studies as undisclosed when the results have been disclosed in a registry other than ClinicalTrials.gov is clearly erroneous. However, it is difficult not to have sympathy with their plight, particularly when one of them is leading an initiative that aims to link all publicly available information on individual clinical trials across multiple platforms.9 Additional methods to link multiple registries with published results may be helpful,10 but the increasing availability of clinical trials data (e.g. from reports submitted to regulatory agencies) makes this a moving target.11,12 The validity of TrialsTracker could potentially be improved by the authors tracking trials reported on registries other than ClinicalTrials.gov, by sponsors posting results on ClinicalTrials.gov, by ClinicalTrials.gov linking to results posted on other registries, and by journals or PubMed allowing ClinicalTrials.gov registration numbers to be added to the abstract or associated meta-data of published articles retrospectively (ideally without generating an erratum). Working together to optimize the TrialsTracker methodology, diverse stakeholders can maximize transparency and strengthen trust in evidence arising from clinical trials.
References