Keywords
Causality, publication bias, questionable research practice, reporting bias, research design, selective reporting
This article is included in the Research on Research, Policy & Culture gateway.
Causality, publication bias, questionable research practice, reporting bias, research design, selective reporting
Selective reporting of research findings presents a large-scale problem in science, substantially affecting the validity of the published body of knowledge (Bouter et al., 2016; Dwan et al., 2014; van den Bogert et al., 2017). Reporting bias (publication bias or outcome reporting bias) occurs when the decision to report depends on the direction or magnitude of the findings. In clinical research, registration of trials prior to data collection is used to prevent selective reporting with some success (Chan et al., 2017; Gopal et al., 2018). However, it is insufficiently effective because despite registration or publication of the study protocol, trial results often remain partially or completely unpublished (Jones et al., 2013) and selective reporting of “positive findings” also occurs among trials registered at, for example, clinicaltrials.gov (Dechartres et al., 2016).
Although many epidemiological studies have described the occurrence of selective reporting, very few studies have targeted its causes. In particular there is little high-quality evidence on effective interventions. To develop effective interventions against reporting bias, we need a good understanding of possible contributions of actors involved (such as academic environment, editors, researchers) and of possible mechanisms. We also need clear hypotheses of how causes may be interrelated and how actors are involved.
We recently developed a taxonomy of putative determinants of selective reporting abstracted from the literature. We used qualitative content analyses of empirical and non-empirical studies until we reached saturation, which indicates that the categories likely cover all important putative determinants of selective reporting. This resulted in 12 categories (Table 1).
Determinant category | Description | Examples |
---|---|---|
A. Motivations | ||
Preference for particular findings | A particular preference motivates a focus on finding results that match preferences, mostly statistically significant or otherwise positive findings, wishful thinking and acting | Significance chasing, finding significant results, larger effect size, suppressing publication of unfavourable results, not being intrigued by null findings |
Prejudice (belief) | A conscious or unconscious belief that may be unfounded, and of which one may or may not be aware | Prior belief about efficacy of treatment, author reputation or gender bias in the phase of review |
B. Means | ||
Opportunities through poor or flexible study design* | Attributes of study design relating to power and level of evidence provide much leeway in how studies are performed and in interpretation of their results | Not a controlled or blinded study, study protocol unavailable, small sample size |
Limitations in reporting and editorial practices | Constraints and barriers to the practice of reporting relevant detail | Journal space restrictions, author writing skills |
C. Conflicts and balancing of interests | ||
Relationship and collaboration issues | Intellectual conflict of interest between reporting and maintaining good relationships | Disagreements among co-authors and between authors and sponsors, sponsors prefer to work with investigators who share the sponsor’s position |
Dependence upon sponsors | Financial conflict of interest resulting in lack of academic freedom | Requirements and influence of funding source with financial interests in study results |
Doubts about reporting being worth the effort | Weighing investment of time and means versus likelihood of gain through publication | Anticipating disappointment of yet another rejection or low chances of acceptance of a manuscript, belief that findings are not worth the trouble |
Lack of resources, including time | Insufficient manpower or finances | Lack of time resulting from excessive workload, or lack of personnel due to life events |
D. Pressures from science and society | ||
Academic publication system hurdles | Various hurdles to full reporting related to submission and processing of manuscripts (other than reporting) including those that represent an intellectual conflict of interest | Solicited manuscripts, authors indicating non- preferred reviewers, editor’s rejection rate |
High-risk area and its development | Area of research or discipline or specialty including its historical development and competitiveness, the currently dominant paradigms and designs, and career opportunities | Ideological biases in a research field, area with much epidemiological research versus clinical or laboratory research (“hard sciences”), humanities, experimental analytic methods, “hot” fields, publication pressure in the specific field |
Unfavourable geographical or regulatory environment | Geographical or regulatory environment that affects how research is being performed | Continents under study included North America, Europe and Asia; few international collaborations; no governmental regulation of commercially sponsored research |
Potential harm | Publishing data can harm individuals | Risk of bioterrorism, or confidentiality restriction |
In the literature review we also found some instances of hypothesized effect modification of the determinants of selective reporting, so that the effects of determinants are assumed not to be simply additive. For example, “Outcomes could be deemed post hoc to have little clinical relevance if they fail to show significant findings and may thus be omitted when accommodating space limitations” (Chan & Altman, 2005). In this case, a preference, namely statistically significant findings, combined with editorial practices lead to reporting bias. Similarly, Ioannidis (2005) hypothesized that a focus on preferred, positive findings could result in reporting of non-reproducible findings (only) if there is also an opportunity to do so through flexibility in study designs and freedom in reporting on it. That is, he concludes that “The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true” because “Flexibility increases the potential for transforming what would be ‘negative’ results into ‘positive’ results.“
Along these lines, we hypothesize that the combination of two of the most common categories in our review (van der Steen et al., 2018) –– i.e., focusing on preferred findings and employing a poor or flexible study design, suffices to cause bias through selective reporting. Inspired by Rothman’s (1976) framework of necessary, sufficient and component causes, through multiple discussions, we inductively derived Figure 1 which shows clusters covering these and the ten other categories of determinants and their possible interrelationships. The two categories are part of clusters A (motivations) and B (means). We view both clusters A and B as necessary causes, that is, they are both part of any sufficient cause of reporting bias. There is also effect modification between A and B because reporting bias is not possible with A or B alone. Note that the preference need not be authors’ preference; it may also be that of a reviewer or editor. In addition to clusters A and B, we propose clusters C and D containing categories of component causes which are discussed in the next section.
Poor or flexible study design may offer the means for selective reporting in addition to limitations in reporting and editorial practices (cluster B in Figure 1). In parallel, we placed “prejudice” in cluster A together with “preference for particular findings” because both may, whether consciously or not, represent a motivation for behaviour that leads to reporting bias. The possible motivations, wishes and beliefs in cluster A are different concepts that may result in “wishful thinking” (Bastardi et al., 2011) and in motivated reasoning around the interpretation of scientific findings (e.g. to serve political interests; Colombo et al., 2016; Kraft et al., 2015). Persons may or may not be fully aware of their motivations and the resulting behaviour may or may not be intentional (Greenland, 2009). At the root of reporting bias may thus lay a basic human attitude, the very natural tendency to make public our successes (Dickersin & Min, 1993).
The pertinence of the second necessary cause (cluster B)––multiple opportunities to select what to analyse or report––is illustrated by the many degrees of freedom that researchers have but should not be tempted to use (in performing psychological research: Wicherts et al., 2016). The necessary causes thus represent having a motive (preference or prejudice; cluster A) and the means (opportunities in study design or reporting; cluster B). Together they form a sufficient cause for reporting bias.
Obviously, researchers and editors are key stakeholders because commonly they decide what is actually being reported and what is not. It can be argued that researchers are the most important because a single editor’s decision is not decisive for non-publication or selective publication. Researchers are actors in three of the four categories in clusters A and B that represent the necessary causes, while editors are key players in only one category (in cluster B; Figure 1). Note that we assume actors in the field are capable.
After a series of rejections researchers may doubt whether reporting is worth the effort under the pressure of lack of resources such as time. Balancing effort and output is placed in cluster C (component cause conflicts and balancing of interests; Figure 1). Cluster C also includes relationship and collaboration issues and dependence upon sponsors. Cluster C thus represents conflicts of interests, individuals and teams juggling with harmony in relationships and time investments.
Other component causes represent pressures from science and society (cluster D), so from the wider environment. The individual researcher has less control over type C, and in particular type D causes, than over motivations (A) and means (B). C and D cannot fully control or explain individuals’ decisions, but they may shape motivations (A) and means (B). When this is the case the effect on reporting bias of the categories in cluster C or D is mediated through the categories contained in cluster A or B. For example, important news is selectively reported but what is deemed important news is shaped by the development within a scientific domain (cluster C; Preston et al., 2004). Also, researchers’ collaborations or relations with sponsors may nudge them to selectively report the preferences of others. A final example is academic publication system hurdles (cluster D) and dependence upon sponsors (cluster C) leading to reporting bias through their impact on the combination of a preference for positive findings and the opportunities that flexible designs offer.
We propose a theoretical framework of reporting bias by relating and ordering 12 determinant categories that we derived from the literature (van der Steen et al., 2018). We further combined these categories in four clusters (A-D).
The model has more layers and is more refined than we anticipated when we wrote a protocol to develop a taxonomy of determinants of selective reporting and their interrelationships. We then expected a central role for preferences for particular “positive” findings only (van der Steen et al., 2018 Supplement 1, Figure 1). However, having the means is necessary as well. Although the determinants in our model are mostly based on research in the biomedical area, the model fits well with the “Desire-Belief-Opportunity” (DBO) model that analytical sociologists use to explain various phenomena (Hedström, 2005) and which we came across after developing our theoretical framework. Desire and Belief concur with the two motivations in cluster A, while opportunities (alternative actions available to the actor) represent the means in cluster B.
Theory may guide the development of interventions as research often does not systematically consider contextual and individual factors that influence delivery of an intervention. Thus, theory may help avoid an ad hoc or data-driven approach to attempts to reduce reporting bias. Although one might assume that interventions addressing reporting bias effectively will be complex, the removal of a single necessary cause is obviously effective. For example, a potentially very effective measure that funders and (medical) ethics committees could adopt is systematic monitoring of all written research outputs and comparing the outcomes reported therein to the corresponding research protocols and statistical analysis plans and eventual amendments. This would require that these organizations make submission to them or to a publicly available repositories mandatory. The approach would become practically feasible if software comparing protocols to publications becomes available (ter Riet & Bouter, 2016). In the jargon of this paper, this approach would eliminate the necessary cause ‘means.’ Given suitable negative reinforcements (punishments) following incomplete reporting, such measures may also reduce motivation to report selectively. Similarly, elements from the component causes contained in cluster C and D that are highly prevalent and strongly modify the combined effect of cluster A and B may be prioritized targets. Mediators can also be good candidates for intervention. For example, component causes contained in cluster C may mediate the impact of elements of D on elements of clusters A or B.
In addition to informing the development of interventions that are subsequently evaluated, our framework may also help to identify high risk scientific fields. For example, areas where designs offer considerable flexibility or where the researchers’ degrees of freedom are combined with strong beliefs or a mission to disseminate particular outcomes. Of course, based on research, our theoretical framework may need to be adapted. Motive and means may be stable clusters but the C and D type causes may change as science changes. Future work may also help to refine the framework’s relevance for specific disciplinary fields (e.g., non-clinical biomedical research). Nevertheless, because the causal pathways seem plausible, were derived from the literature on selective reporting and is congruent with theory developed in the social sciences (Hedström, 2005), we feel that the current work can already help to design further research on the effectiveness of interventions.
PLOS ONE Supplement 2 to article van der Steen et al., 2018. Determinants of selective reporting abstracted from the selected literature. “S2 File. Dataset with determinants.” In Excel available from: https://doi.org/10.1371/journal.pone.0188247.s003 (van der Steen et al., 2018)
PLOS ONE Supplement 3 to article van der Steen et al., 2018. Categories of determinants of selective reporting with literature references. “S3 file. References to the 64 articles included in the determinant analysis, per category.” In Word available from: https://doi.org/10.1371/journal.pone.0188247.s004 (van der Steen et al., 2018)
Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).
The writing of the article was supported by personal grants to JTvdS from the Netherlands Organisation for Scientific Research (NWO; Innovational Research Incentives Scheme: Vidi grant number 917.11.339), and from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Consolidator grant agreement No 771483), and by Leiden University Medical Center, Leiden, The Netherlands.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Views | Downloads | |
---|---|---|
F1000Research | - | - |
PubMed Central
Data from PMC are received and updated monthly.
|
- | - |
Is the work clearly and accurately presented and does it cite the current literature?
Yes
Is the study design appropriate and is the work technically sound?
Partly
Are sufficient details of methods and analysis provided to allow replication by others?
Partly
If applicable, is the statistical analysis and its interpretation appropriate?
Not applicable
Are all the source data underlying the results available to ensure full reproducibility?
No source data required
Are the conclusions drawn adequately supported by the results?
Partly
Competing Interests: No competing interests were disclosed.
Reviewer Expertise: Meta-research; research integrity; systematic reviews; social sciences; science policy; education; experimental and quasi-experimental methods.
Is the work clearly and accurately presented and does it cite the current literature?
Partly
Is the study design appropriate and is the work technically sound?
Partly
Are sufficient details of methods and analysis provided to allow replication by others?
Yes
If applicable, is the statistical analysis and its interpretation appropriate?
Not applicable
Are all the source data underlying the results available to ensure full reproducibility?
No source data required
Are the conclusions drawn adequately supported by the results?
Partly
References
1. Baker M: 1,500 scientists lift the lid on reproducibility. Nature News. 2016. Reference SourceCompeting Interests: No competing interests were disclosed.
Reviewer Expertise: research ethics, psychology, medical informatics
Alongside their report, reviewers assign a status to the article:
Invited Reviewers | ||
---|---|---|
1 | 2 | |
Version 2 (revision) 17 Jul 19 |
read | read |
Version 1 12 Mar 19 |
read | read |
Provide sufficient details of any financial or non-financial competing interests to enable users to assess whether your comments might lead a reasonable person to question your impartiality. Consider the following examples, but note that this is not an exhaustive list:
Sign up for content alerts and receive a weekly or monthly email with all newly published articles
Already registered? Sign in
The email address should be the one you originally registered with F1000.
You registered with F1000 via Google, so we cannot reset your password.
To sign in, please click here.
If you still need help with your Google account password, please click here.
You registered with F1000 via Facebook, so we cannot reset your password.
To sign in, please click here.
If you still need help with your Facebook account password, please click here.
If your email address is registered with us, we will email you instructions to reset your password.
If you think you should have received this email but it has not arrived, please check your spam filters and/or contact for further assistance.
Comments on this article Comments (0)