Comparison of published orthopaedic trauma trials following registration in Clinicaltrials.gov

Abstract

Background

After the Food and Drug Administration Modernization Act of 1997, the registration of all clinical trials became mandatory prior to publication. Our primary objective was to determine publication rates for orthopaedic trauma trials registered with ClinicalTrials.gov. We further evaluated methodological consistency between registration and publication.

Methods

We searched Clinical Trials.gov for all trials related to orthopaedic trauma. We excluded active trials and trials not completed by July 2009, and performed a systematic search for publications resulting from registered closed trials. Information regarding primary and secondary outcomes, intervention, study sponsors, and sample size were extracted from registrations and publications.

Results

Of 130 closed trials, 37 eligible trials resulted in 16 publications (43.2%). We found no significant differences in publication rates between funding sources for industry sponsored studies and nongovernment/nonindustry sponsored studies (p > 0.05). About half the trials (45%) did not include the NCT ID in the publication. Two (10%) publications had major changes to the primary outcome measure and ten (52.6%) to sample size.

Conclusions

Registration of orthopaedic trauma trials does not consistently result in publication. When trials are registered, many do not cite NCT ID in the publication. Furthermore, changes that are not reflected in the registry of the trial are frequently made to the final publication.

Introduction

ClinicalTrials.gov (CTG) was developed by the U.S. National Institutes of Health in collaboration with the Food and Drug Administration (FDA) in February 2000 [1]. It was established after the FDA Modernization Act of 1997, which made it a requirement for the Department of Health and Human Services to establish a registry of all clinical trials of experimental treatments for serious or life-threatening diseases or conditions [2]. It offers current information on clinical trials for a wide range of diseases and conditions, allowing doctors, researchers, and patients to locate clinical trials conducted worldwide. CTG currently contains over 94,000 registered trials from 174 countries [3].

The database summarizes each registered trial and gives information including but not limited to the purpose of the study, recruiting status, criteria for patient participation, and location of the trial and specific contact information. Furthermore, additional information such as research study design and condition under study is outlined to help potential participants consider a particular trial.

Each listing on the registry is given a unique National Clinical Trials Identifier (NCT ID) with the prefix 'NCT' and an 8-digit number that is used to locate and identify a specific trial. We believe that the NCT ID should be included in any final publication to allow the reader to evaluate the strength of the trial by comparing it to the original plans as outlined in the registry. We performed a review of trials registered in CTG and relating to orthopaedic trauma evaluating publication rates and consistency of reporting. We hypothesized that the incidence of registered trials not published would be high despite mandatory reporting of clinical trials, and there would be methodological differences between publication and registration methodology.

Methods

Eligibility criteria

Trials that were considered eligible to be included in our analysis met the following criteria: 1) registered on clinicaltrials.gov registry, 2) registered up to July 2009, and 3) trials that were reported as closed. Ongoing trials were excluded from analysis.

Search strategy

We performed a search of orthopaedic trauma trials with the http://www.ClinicalTrials.gov database in duplicate (MJ, HS) using the search terms (orthopaedic trauma OR orthopedic trauma). We also performed a search using the conditions listed on CTG including "wounds and injuries" and "fracture, bone," "radius fracture," "tibia fracture" and "sprains and strains." This resulted in 264 registered trials up to and including July 2010. The search was further narrowed to closed studies with the understanding that studies still actively recruiting participants are unlikely to have publications (n = 134). Refining the search resulted in 130 closed studies. Of these, we decided to review studies with an estimated completion date up to and including July 2009 to allow adequate time for preparation and submission of a manuscript and therefore excluded those trials with a completion date after July 2009 (n = 67). Seven studies did not indicate a completion date or estimated completion date in the database and were thus excluded. We also excluded non-randomized and observational studies. Application of this criteria resulted in 37 studies for subsequent analysis. Discrepancies in eligibility were resolved by consensus. Agreement for final inclusion between reviewers was excellent. (Kappa = 0.79; 95% confidence interval, 0.46 to 0.94).

Because our results from our previous search terms provided us with a larger number of trials to analyze, we believe our search strategy is sufficient.

Publication rate

Each of the 130 closed studies were first assessed to determine whether or not the investigators listed any publications within the trial registry. If none were listed, a search using key terms from the trial and the primary investigator was performed using the Pubmed, EMBASE and Ovid Medline databases. Suspected publications were confirmed with the registry based on NCT ID, publication date, authors, number of participants, primary and secondary outcome measures, and study location.

Data abstraction

After identifying a published study, two of us (HS, MJ) extracted data from both the CTG database and the publication including authors, year of publication, NCT ID and its presence in the paper, trial status (complete, active, not recruiting etc.), study start date, registration date and completion date, study sponsor, primary and secondary outcome measures, study phase and design, study location, inclusion and exclusion criteria, and sample size.

We evaluated the agreement between the methodology reported in the registry and that of the publication. Major discrepancies were defined using a modification of criteria previously reported by Chan et al. [4]: (1) a pre-specified primary outcome was reported as secondary or was not labeled as either primary or secondary in the publication; (2) a pre-specified primary outcome was omitted from the published article; (3) a new primary outcome was introduced in the published article.

Statistical analysis

Study characteristics were presented with descriptive statistics. Association between trial sponsor and incidence of publication was anlayzed with the Fisher exact test. All p values were two tailed with a level of significance set at 0.05.

Kappa statistics were used to calculate the agreement between reviewers on study eligibility.

Results

Characteristics of registered trials

The 37 eligible orthopaedic trauma trials included in CTG spanned North and South America, Europe, Asia and Africa. Most trials evaluated surgical interventions, however there were also drug and therapeutic interventions such as the use radiation. Further details of the eligible trials can be found in Table 1.

Table 1 Details of Eligible Trials

Publication rate

In total, 16 of 37 (43.2%) registered trials with an estimated completion date up to and including July 2009 were confirmed to have publications. Four interim publications [5–8] defined as ongoing trials with publications prior to the study completion were also identified for a total of 20 publications included in our analysis of consistency with trial registration. Of the 4 interim publications, 1 was sponsored by industry [7], and the remaining 3 by a nongovernment/nonindustry source [5, 6, 8]. Nongovernment/nonindustry sponsored trials had the highest publication rate, with 9 of 19 (47.3%) trials resulted in publication [9–17]. In comparison, 6 of 14 (42.9%) industry sponsored trials resulted in publication [18–23], and 1 of 4 (25%) government sponsored trials [24] were published. There were no significant differences in publication rates for industry sponsored studies and nongovernment/nonindustry sponsored studies (p = 0.92), between government sponsored and nongovernment/nonindustry sponsored trials (p = 0.6) or industry sponsored trials (p = 0.62).

Fifteen of the 20 publications were from a trial with a status of "completed" in CTG [9–19, 21–24]. The remaining publications were listed as active but not recruiting [5–8], and terminated in the database [20]. One additional study had a published abstract but no complete manuscript [25]. We verified this by confirming that the title, author, outcome measures, and study dates were the same as those mentioned in the registered trial. Of the 20 publications, the NCT ID was included in 11 (55%) [8, 9, 11, 12, 14, 15, 17, 18, 20, 23, 24].

Characteristics of published trials can be found in Table 2.

Table 2 Characteristics of Published Orthopaedic Trauma Trials

Consistency between trial registration and publication

We compared the study sample size proposed on CTG with the actual study enrolment reported in the publication. There was one study that did not include a proposed sample size in the registry [10]. In 9 of the remaining 19 studies (47.3%), enrolment as stated in CTG exactly matched that of the final publication [5, 7–9, 12, 16–18, 24]. Of the 10 publications (52.6%) with an inconsistent sample size, 7 had a smaller sample than the one reported in CTG [6, 11, 13, 15, 19, 20, 22], and 3 had a larger sample [14, 21, 23]. The sample size discrepancy in all publications differed from the original figure by a minimum of ± 6%.

We compared the agreement between the primary and secondary outcome measures as stated in the publication and on CTG. The registry listed a primary outcome measure for all trials and secondary outcome measures for 18 trials. One trial did not list a secondary outcome measure in the registry or publication [24]. Two of the 20 publications (10%) had a major discrepancy with the registry. Each of these publications did not state the primary outcome measure despite identifying one in the registry [6, 18]. All but two of the remaining 18 publications were consistent with the registry with regards to primary and secondary outcomes [5, 7–12, 15–17, 19–23]. Two publications did not report a secondary outcome measure in the manuscript that had been listed in the registry [13, 14].

Discussion

As a result of our search of orthopaedic trauma trials registered in CTG, we found that only 16 of 37 (43.2%) trials resulted in publication as of March 2011. Furthermore, final results of completed trials were not reported in the registry, and many publications were inconsistent in sample size and reporting of outcomes with the original report in CTG. We are unable to determine the reason for trials failing to result in publication. Several possibilities exist, including that the data was never analyzed, the journals elected not to publish, the authors elected not to publish the findings because they failed to substantiate the study hypothesis or believed the results contradicted the hypothesis. Another possibility may be that papers were not accepted for publication because of non-significant results. Regardless, the results of this study represent a large potential for publication bias in the orthopaedic trauma literature.

Historically, studies with statistically significant outcomes have been more likely to be submitted for publication and accepted by journals. Dickersinet al. [26] found that trials with statistically significant results were 2.9 times more likely to be published. Several authors have recently reported similar problems in the orthopaedic literature [27, 28]. Publication bias may lead to wasted resources due to unnecessary duplicate studies, as well as potential harm to study participants [29].

Nongovernment/nonindustry sponsored studies demonstrated the highest rate of publication (47.3%), while industry sponsored trials were similar (42.9%). Only one of four (25%) completed government sponsored trials were published. This is contrary to the results of several studies that have found that in general, industry sponsored studies have the lowest rates of publication [30].

Almost half (45%) of trials did not include the NCT ID in their publication. Including the NCT ID in the publication should be made mandatory as it allows the reader to locate and identify the trial on CTG and compare the original study design with the published design. Published manuscripts where the power or primary outcome measures have been changed can alert the reader to concerns of study validity.

We also found that 2 (10%) publications had major changes to the primary outcome measure and 10 (52.6%) to sample size. Discrepancies between trial registry and publication have been observed in previous studies and are a concern as they may compromise the integrity of the conclusions that can be drawn from the study [31]. Simply adding patients to a study may take a clinically insignificant finding, and make it statistically significant for the reader. Mandatory registration would allow journal editors to examine trial information when reviewing the manuscript to uncover such discrepancies.

We reviewed requirements for authors in various orthopaedic surgery journals to verify whether the journals support trial registration prior to a publication. We found that the Journal of Arthroplasty, British Journal of Bone and Joint Surgery, Journal of Orthopaedic Surgery, and the Journal of Orthopaedic Trauma did not include information about requirements related to the registration of clinical trials. Acta Orthopaedica Scandinavica instructs authors to comply with the CONSORT guidelines for reporting of clinical trials [32]. Although these guidelines ensure that the author submits the registration number and the name of the registry in the publication, they do not require the author to register a clinical trial in a registry. Clinics in Orthopaedic Surgery, the American Journal of Bone and Joint Surgery, and BioMed Central explicitly stated that registration of all clinical trials was required. Improving instructions for authors submitting manuscripts to journals, and making trial registration a necessity is an essential step to move towards reliable reporting of clinical trial results [33, 34].

The limitations of our study include the uncertainty of the time it may take between submission of a paper to a journal and approval for publication. A potential weakness is that although we reviewed studies until March 2011 (publication time between July 2009-March 2011), it is possible that there are completed trials in print that are currently awaiting publication. Some journals may publish studies at a faster pace than others. It may be that there were studies that were published without our knowledge, completed early, or submitted before their completion date. Furthermore, it may be that changes to trial information on CTG simply may not be updated frequently enough and may cause the available information to be misleading. Also, it is possible that studies that did not focus on orthpaedic trauma but included some aspect of it were overlooked because they did not correspond with the search terms on CTG.

Our study also has several strengths. We conducted an extensive search in duplicate for publications of closed studies using several different search methods. We also conducted exhaustive searches for interim publications, and publications resulting from studies listed as terminated or withdrawn.

Conclusions

The results of our review of orthopedic trauma trials on ClinicalTrials.gov indicate that registration does not consistently result in publication or disclosure of results. Although only 10% of the publications had major discrepancy between trial registry and publication, smaller changes are frequently made to the final presentation of the data that are not reflected in the registry of the trial. When trials are registered, a great number of them do not cite the registration number in the publication, making it impossible for the reader to evaluate the study conclusions in relation to the original plans for the trial. We suggest all journals should make registration of clinical trials mandatory for publication.