High impact nutrition and dietetics journals’ use of publication procedures to increase research transparency

Abstract

Background

The rigor and integrity of the published research in nutrition studies has come into serious question in recent years. Concerns focus on the use of flexible data analysis practices and selective reporting and the failure of peer review journals to identify and correct these practices. In response, it has been proposed that journals employ editorial procedures designed to improve the transparency of published research.

Objective

The present study examines the adoption of editorial procedures designed to improve the reporting of empirical studies in the field of nutrition and dietetics research.

Design

The instructions for authors of 43 journals included in Quartiles 1 and 2 of the Clarivate Analytics’ 2018 Journal Citation Report category Nutrition and Dietetics were reviewed. For journals that published original research, conflict of interest disclosure, recommendation of reporting guidelines, registration of clinical trials, registration of other types of studies, encouraging data sharing, and use of the Registered Reports were assessed. For journals that only published reviews, all of the procedures except clinical trial registration were assessed.

Results

Thirty-three journals published original research and 10 published only reviews. Conflict of interest disclosure was required by all 33 original research journals. Use of guidelines, trial registration and encouragement of data sharing were mentioned by 30, 27 and 25 journals, respectively. Registration of other studies was required by eight and none offered Registered Reports as a publication option at the time of the review. All 10 review journals required conflict of interest disclosure, four recommended data sharing and three the use of guidelines. None mentioned the other two procedures.

Conclusions

While nutrition journals have adopted a number of procedures designed to improve the reporting of research findings, their limited effects likely result from the mechanisms through which they influence analytic flexibility and selective reporting and the extent to which they are properly implemented and enforced by journals.

Introduction

The rigor and integrity of the published research literature in the field of nutrition studies has come into serious question in recent years. One reason for this is the publicity surrounding the retraction of more than a dozen papers produced by researchers at a major nutrition laboratory at one of the top universities in the United States [1, 2]. Such a large-scale retraction of papers casts doubts on the peer review process of the journals that published this research, and is especially concerning as many of these were high impact journals with presumably stringent peer review procedures and standards.

In addition, some nutrition researchers have begun to question the quality of published research both within the discipline as a whole and within specific areas of specialization. A report from an American Society for Nutrition advisory committee highlighted the threats to research integrity that arise from competing interests among researchers, specifically the type of selective statistical analyses that are conducted and reported and the conclusions investigators draw from these (e.g., withholding unfavorable findings from publication) [3]. In a more detailed examination of such issues in the field of childhood obesity interventions, Brown et al. identified ten statistical and methodological errors frequently found in the published literature that were associated with exaggerated claims about program effectiveness in changing diet, increasing exercise and weight reduction [4]. These included reporting results only for secondary outcome variables in the light of null findings for primary outcomes, reporting only subgroup analyses in the light of no main effects, data dredging for spurious statistically significant results, using one-tailed tests of statistical significance, and claiming that null results were nonetheless “clinically significant”. Bero and colleagues also identified bias in outcome reporting in a series of systematic reviews focused on nutrition research funded by the food industry [5,6,7]. They found the reporting of results in such studies tended to be skewed in favor of the sponsoring industry, and that authors’ financial conflicts of interest were frequently undisclosed in journal publications. Such reporting bias and lack of transparency is especially problematic as it can lead to the development of inappropriate dietary guidelines [8].

In the field of nutritional epidemiology, Ioannidis contends that “good scientific principles” are lacking, with investigators being free to report selective results from multiple analyses they have conducted, very few of which will have been prespecified [9]. In a study of published case-control and cohort studies that examined the association between 40 randomly selected ingredients from recipes and cancer, Schoenfeld and Ioannidis found that the vast majority reported either an increased or decreased risk based on very weak statistical evidence [10]. They attributed this to publication bias and selective reporting of positive results in publications.

The concerns raised regarding the quality and integrity of research published in the field of nutrition research reflect those that have arisen within many academic disciplines over the past decade, and that have been described in terms of a “credibility crisis” [11, 12]. The response to this crisis has been, in part, to call for increased rigor and transparency in the editorial and peer review process used by academic journals [13,14,15]. This has involved requiring full and detailed disclosure of conflicts of interest, use of guidelines such as the Consolidated Standards for Reporting Trials (CONSORT) [16] and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [17] when writing-up manuscripts for publication, preregistration of hypotheses, study designs and measures, and sharing of data and code as a condition of publication. Those concerned with the quality and integrity of nutrition research have also suggested that such procedures be introduced by academic journals within the discipline [3, 4].

Six studies have examined journal adoption of three editorial procedures designed to improve the quality of published research in the fields of pediatric research [18, 19], surgery [20], emergency medicine [21], orthopedic and general medicine [22], and addiction [23]. Clinical trial registration and recommendation of the CONSORT guidelines were assessed in all six studies, with adoption of the procedure ranging from 23 to 86% for the former and 20 to 81% for the latter [18,19,20,21,22]. Other reporting guidelines, such as PRISMA, were less likely to be recommended in the instructions for authors of these journals. Conflict of interest disclosure was examined in three studies and required by 61% of online and 78% of print pediatrics journals [18, 19], and 97% of addiction journals [23].

To our knowledge, no existing study has assessed the adoption of editorial procedures designed to improve the reporting of empirical studies in the field of nutrition research. The current study is intended to fill this gap in existing knowledge. It is an exploratory and descriptive study in which no hypotheses were prespecified or tested.

Materials and methods

In line with previous studies of journal editorial procedures [18, 20,21,22,23], Clarivate Analytics Journal Citation Report (JCR) was used to identify high impact nutrition and dietetics journals. Specifically, 21 journals ranked in Quartile 1 (Q1) and 22 in Quartile 2 (Q2) of the 2018 JCR category Nutrition and Dietetics were selected [24]. The impact factor of each of these 43 journals was also obtained from the JCR webpage. Q1 journals are those with impact factors greater than 75% of journals within that particular JCR category, while Q2 journals have impact factors greater than 50% of journals within the category. The 43 Q1 and Q2 journals were selected so that their instructions for authors could be identified and reviewed in a timely manner (i.e., before they became outdated and were replaced online) and this number was considered to be reasonable in terms of representing the high impact journals within the field of nutrition and dietetics research. However, recognizing the limitations of journal rankings based on impact factor [25], the Scimago Journal Ranking (SJR) h-index of each of the 43 journals was also identified. This expresses the number of articles (h) published by the journal that have received at least h citations [26].

The instructions for authors of each of the 43 journals was located on the journal webpage on the Internet. In addition, if a journal’s instructions for authors cited additional documents that prospective authors were required to consult (e.g., publisher ethics guidelines), these too were located. Instructions for authors and related documents that were available as a Portable Document Format (PDF) were downloaded. If the author instructions and related documents on webpages were unavailable as a PDF, they were copied and pasted into a Word document. All the material reviewed was downloaded between April 3 and October 14, 2019. The downloaded instructions for authors of each journal, along with any relevant additional materials cited in these, were reviewed separately by each of the two authors and mention of each publication procedure was recorded. The two reviewers met to discuss their ratings and any discrepancies were resolved. Specifically, where there was disagreement about the presence or absence of requirements and recommendations in a particular set of instructions for authors, they re-reviewed these during the meeting and came to agreement as to whether the procedure was or was not described in the documents. The Center for Open Science list of journals that have adopted the Registered Reports publication format was also reviewed for the inclusion of any of the 43 nutrition and dietetics journals on December 19, 2019 [27].

For journals that published both original research and systematic reviews the following six procedures were assessed: (i) conflict of interest disclosure, (ii) recommendation of specific reporting guidelines when writing-up the results of studies for publication (e.g., CONSORT, PRISMA) or reference to the EQUATOR Network or FAIRshare repositories of guidelines [28, 29], (iii) registration of clinical trials in a registry such as Clinical Trials.gov [30] (iv) registration of other types of studies in a registry such as PROSPERO (for systematic reviews) [31], (v) encouraging or requiring data sharing, and (vi) use of the Registered Reports publishing format [27]. For journals that exclusively published systematic reviews and meta-analyses, five of the six procedures were considered relevant, the exception being clinical trial registration. In addition, in the case of two of the remaining five, it was anticipated that fewer guidelines would be relevant to journals that only publish reviews (e.g., PRISMA and Meta-analysis of Observational Studies in Epidemiology (MOOSE) [32]) and that just one registry (PROSPERO) would likely be utilized. These procedures were selected based on those used in previous studies of journal editorial practices [18,19,20,21,22,23], as well as those described in the broader literature on measures to reduce publication and reporting bias and promote research transparency [33,34,35].

The analyses presented are descriptive data pertaining to the number of publication procedures mentioned in journal instructions to authors.

Results

Table 1 shows the JCR impact factor, SJR h-index and publisher of the 43 journals that appeared in the top two quartiles (Q1 and Q2) of the 2018 JCR Nutrition and Dietetics category. Ten of the journals were judged to be exclusively focused on review articles from the aims described in their instructions for authors, and these were reviewed separately from the 33 journals that focused on original research (in addition to reviews).

Table 1 2018 JCR Q1 and Q2 Nutrition and Dietetics Journals Arranged by JCR and Scimago Impact Factors and with Publisher^a

Table 2 shows which of the five publication procedures was mentioned in the instructions for authors of each of the ten journals that exclusively publish review articles. Conflict of interest disclosure was required by all journals, but two of the other four procedures were only mentioned by three or four journals and two were not mentioned by any journal in its instructions for authors.

Table 2 Specification in Instructions for Authors of Five Publication Procedures Designed to Improve Research Integrity by 2018 JCR Q1 and Q2 Nutrition and Dietetics Journals that Publish Reviews (n = 10)^a

Table 3 shows which of the six publication procedures was mentioned in the instructions for authors of the 33 nutrition and dietetics journal that published articles reporting original research. The range was 1 to 5, and the mean was 3.7. Twenty-three (70%) of the journals discussed four or more of the procedures in their instructions for authors.

Table 3 Specification in Instructions for Authors of Six Publication Procedures Designed to Improve Research Integrity by 2018 JCR Q1 and Q2 Nutrition and Dietetics Journals that Publish Original Research (n = 33)^a

All of the 33 journals required authors to disclose conflicts of interest. The second most widely recommended publication procedure was use of guidelines when writing-up results from empirical studies, with 30 of the 33 journals mentioning at least one specific set of guidelines and one mentioning the EQUATOR Network in their instructions to authors or related documents. Registration of clinical trials was also common, with 27 journals requiring this of investigators. Compared to clinical trials, registration of research using other study designs was much less common, with just eight journals requiring this of investigators. Of these, seven required registration of systematic reviews in PROSPERO and one (Journal of Nutrition) required registration of observational studies, but did not specify a particular registry.

Discussion of data sharing appeared in the instructions for authors of 25 of the 33 journals. For the vast majority, this involved encouraging data sharing for all types of studies. The three exceptions to this were Nutrition and Diabetes for which data sharing was a condition of publication, and Obesity and Applied Physiology Nutrition and Metabolism which both required a data sharing statement, but only for clinical trials. None of the journals had adopted the Registered Reports publication format at the time the review was conducted.

Table 4 shows the specific guidelines and guideline repositories recommended in each of the 30 journals that discussed these in their instructions for authors. Ten journals mentioned the EQUATOR Network repository of guidelines. Five of these journals also mentioned the FAIRsharing repository and all but one also mentioned a specific set of guidelines. A total of 19 specific guidelines were recommended, with the number mentioned by each journal that discussed these ranging from one to 12. Of the 29 that recommended a specific set of guidelines, 27 mentioned CONSORT, 21 ARRIVE, 19 PRISMA and 12 STROBE. Each of the other 15 guidelines was mentioned by fewer than 10 journals.

Table 4 Mention by Journals of the EQUATOR Network and Specific Guidelines (n = 30)^a

Discussion

The requiring of Registered Reports has not been adopted at all within the discipline’s high impact academic peer-reviewed outlets, and registration of studies other than clinical trials is seldom required. Notably however, and consistent with similar studies conducted in other academic disciplines, publication procedures such as conflict of interest disclosure, clinical trial registration, recommendation of guidelines and data sharing have become standard editorial requirements in nutrition and dietetic journals.

Nevertheless, given this generally encouraging picture, the question arises as to why there is such concern about the quality and integrity of published research within the field of nutrition and dietetics studies [3,4,5,6]. In answer, the limited effects on research quality and transparency of the publication procedures examined in this study can best be explained by the mechanisms through which they influence analytic flexibility and selective reporting and the extent to which they are properly implemented and enforced. For example, conflict of interest disclosure does not directly affect flexible data analysis and selective reporting of results. Rather, it alerts readers to the fact that one or more authors of a paper has a competing interest that may create a preference for results of a certain kind (e.g., those showing a positive effect of an intervention program). In addition, journal conflict of interest disclosure policies almost exclusively focus on financial conflicts, which is a potential limitation, especially in applied research fields where affiliation and confirmation biases may predispose investigators to favor certain types of results over others. In light of such influences, it has been argued that there is a need to focus on conflicts of interest beyond those of a financial nature, such as career-related advancement and ideological conflicts [3, 36]. The scope of these may be fairly wide in the field of nutrition and dietetics research. For example, a number of the papers retracted by the Cornell University Food and Brand Lab focused on meal sizes and the structure of eating habits, ideas for which its director was well-known and about which he had written a popular book [37]. Kroger and colleagues [3] propose that investigators who have published a book relevant to the topic of their research should report this as a financial conflict of interest.

However, even in the absence of such a book, it is likely that investigators will have a strong affiliation to the theories they have developed and their prior discoveries regarding these, and therefore their ability to objectively conduct replication studies may be limited [38]. How such affiliation bias should be covered by conflict of interest disclosures remains unclear, and it is questionable whether this is even feasible and desirable [39]. Accordingly, conflict of interest disclosures may not be the most efficient editorial procedure to employ when attempting to reduce analytic flexibly and selective outcome reporting. In the absence of being able to create a list of every possible “interest” that might influence how studies are conceptualized and conducted, it is probably easier to put in place publication procedures such as prospective registration and Registered Reports that make such practices difficult to engage in, whatever the motivation for their use.

Indeed, study registration has been presented by its advocates as a more direct way to address these problems [40] and, as was found in the present study, mandatory registration is becoming widespread among academic journals, at least in the case of randomized clinical trials. However, its application beyond this type of study has been minimal [21, 23], and this was found to be the case in the current analysis of nutrition and dietetics journals. The two main limitations of study registration are that a large proportion of registered studies are retrospectively registered and many investigators do not adhere to key elements of their registered analysis plan when analyzing data and writing-up results for publication [41,42,43,44]. So, while it is encouraging that more than 80% of journals required clinical trial registration and we would recommend all journals require registration of studies using other research designs (e.g., systematic reviews, observational studies), it is essential that editors insist registration is prospective and not retrospective and that they require investigators to demonstrate they adhered to their prospectively registered analysis plans [15, 45].

Lack of adherence and suboptimal oversight is also a problem with study guidelines and data sharing, both of which tend to be recommended or encouraged, rather than required, by journals. As shown in the current study, there now exist a lot of reporting guidelines, but it is unlikely reviewers check manuscript against all the items appearing in these when reviewing manuscripts, and research shows adherence to many of the specific items described in CONSORT, PRISMA and STROBE remains poor [46,47,48]. If journals are to recommend use of reporting guidelines, they should ensure that there is some formal procedure used to assess adherence built into the peer-review process, with either authors or reviewers completing a check-list of compliance.

While three-quarters of the nutrition and dietetics journals in the current study discussed data sharing in their instructions for authors, only one required it as a condition of publication. We recommend all journals require a clear data sharing statement from authors giving precise reasons for their decisions as to how they are going to share their data, and justifying any decision not to share data. If data are housed in a third-party repository, the means of accessing these should be specified; if they are not stored in such a repository, a clear procedure for obtaining the data from the authors should be described. The type of voluntary policies favored by nutrition and dietetics journals have been shown to result in less than one in ten investigators making raw empirical data accessible [49,50,51,52]. Moreover, even when data are shared by investigators, someone has to reanalyze them and identify discrepancies and irregularities in order for analytic flexibility and selective reporting to be identified. For this to have a deterrent effect, a great deal of such reanalysis of published studies would need to take place, something which is unlikely in the prevailing research culture that places little value on replication studies [53]. Also, in the absence of a prespecified analysis plan, there is no way to recreate the study as initially designed and identify deviations from this in a replication study. Reproducibility also requires adherence to a prespecified analysis plan that describes the computational procedures used in the original research, as well as access to the original data and code [54].

Limitations

There are a number of limitations of this study that should be noted. First, not all nutrition and dietetics journals were included and it is possible that other journals, such as those included in Quartiles 3 and 4 of the JCR, used more or less of any one of the particular publication procedures that were examined. Second, the instructions for authors and related materials were reviewed by just two individuals who may each have failed to identify a publication procedure in the course of their review. Third, the paper presents a snap-shot of the state of the discipline, and journals may have subsequently adopted one or more of the publication procedures assessed since the time the review was conducted. For example, we are aware that one of the journals included in our sample (the International Journal of Eating Disorders) has subsequently introduced Registered Reports as an option for authors. Fourth, the study presents no data as to whether the procedures examined were actually enforced by the journals that adopted them and adhered to by the authors that publish in these journals. As noted above, adherence and enforcement are essential to the effective operation of editorial policies in improving research transparency and quality.

Conclusions

In conclusion, publication procedures that lock investigators into prespecified hypotheses, methods, measures and statistical analyses have the greatest potential to limit analytic flexibility and selective reporting of results. Of the procedures examined in this study that best accomplish this, one (Registered Reports) was not used by any of the nutrition and dietetics journals reviewed and the other (registration) was rarely required outside of clinical trials. None of the journals that published only reviews and meta-analyses required that these be preregistered.

We recommend that all nutrition and dietetics journals adopt one or both of the procedures that lock-in data analyses and allow differentiation of genuine confirmatory hypothesis testing from other types of research. Journals could then clearly distinguish (either by having separate sections or including some indicator on the title page of each paper) confirmatory research from research that purports to be but cannot demonstrate that the hypotheses tested were pre-specified. A third category of genuine exploratory research could also be included. The journal instructions for authors could clearly specify the types of claims that can be made for studies within each of these categories. This simple procedure would prevent investigators who employ flexible data analysis practices and selective reporting from presenting the results of these as genuine hypothesis testing [13]. The goal would be, over time, to move more investigators to preregister their hypotheses and analysis plans or publish using the registered reports format in order that their papers be published in the more prestigious confirmatory category that allowed stronger statements about the validity and implications of the study. At present, nutrition and dietetics journals make no distinction between results that emerge from flexible data analysis and selective reporting and those that do not, and therefore there is no incentive for investigators to prespecify their hypotheses and analysis plans.