Issues Related to Empty Reviews

In examining the literature concerning empty reviews, we offer the following summary of the core issues. First, empty reviews may relate to an area of study which is very new. Cooper asserts that research syntheses should concern topics for which there is already a body of evidence [4]. Where it is important to identify new interventions and gaps in knowledge, systematic reviews provide direction for targeted research and in some countries, may be required as part of large grant applications for trials.

Second, in some circumstances, reviews may focus on questions that are highly specific. For example, they may restrict the population by age, context, diagnostic criteria, or intervention criteria. In the case that studies meeting these specified criteria have not been conducted, there are no includable studies.

Third, many empty reviews may be the result of overly stringent methodological inclusion criteria imposed in the interest of higher quality evidence. These criteria may involve study selection based on specific designs, outcome measures, or comparison conditions which may not be available in existing primary studies.

The issue of empty reviews was introduced to the literature in 2007 by Lang and colleagues, who suggested that guidelines were needed for reporting of empty reviews in order to prevent reviewers from deriving unsubstantiated implications for practice, or from simply concluding that no eligible studies were found [3]. Lang et al. further suggest that in the case of empty reviews, authors should note observations from ineligible articles and abstracts. In response, Green et al., while acknowledging that “a specific structure for the reporting of empty reviews and providing information for further research could be helpful,” argue that basing conclusions on studies which do not meet inclusion criteria specified in the review protocol increases the risk of bias of the review and may, indeed, mislead readers [5]. In an editorial supporting the inclusion of empty systematic reviews in Evidence-based Communication Assessment and Intervention, Schlosser and Sigafoos concur with Green and colleagues' position, and encourage commentators for the journal “to highlight this potential for biases if an empty review over-reached in their analysis and interpretation of excluded studies.” [6]

The Cochrane Handbook for Systematic Reviews of Interventions sets policy and provides specific guidance for the reporting of Cochrane systematic reviews but does not yet provide specific guidance for the reporting of empty reviews [2]. As a result, the reporting of empty reviews may be inconsistent.

Study Aims

The objectives of the present study were to provide a description of empty reviews and their general characteristics in The CDSR and across topic areas as defined by Cochrane Review Groups. To explore the extent to which empty reviews are reported in The CDSR, we first identified all reviews without included studies and examined their frequency and proportion overall as well as within Cochrane Review Groups. Second, to examine the persistence of empty reviews, we analyzed time since original publication of identified reviews. Third, to assess the level of existing, but non-includable, research related to topics of empty reviews, we examined the number of excluded studies reported by each of these reviews across The CDSR and within Cochrane Review Groups. Finally, to assess the possibility of future updating with eligible studies, we examined time since last update of these reviews as well as numbers of reported on-going studies and studies awaiting assessment. In this way, we hoped to establish whether the prevalence and general characteristics of empty reviews varied systematically across Cochrane Review Groups. We assumed that inconsistencies in prevalence and characteristics across Cochrane Review Groups might suggest the necessity of general guidelines for the reporting of empty reviews in The CDSR.

Summary of Results

This study examined empty systematic reviews in The CDSR, the world's largest library of systematic reviews. Almost 9% of The CDSR consisted of empty reviews, with nearly 85% of Cochrane Review Groups hosting at least one empty review. Nearly half of empty reviews had been published since 2008, whereas 28 (7.4%) reviews were 10 years or older. Nearly half of empty reviews had been updated within the last 2 years, but 15% of empty reviews had not been updated within the past 5 years. The number of excluded studies listed within empty reviews was highly variable (M = 9.7, SD = 14.5) although more than 25% of these reviews did not report any excluded studies. Further, only 10% of empty reviews reported on-going studies and fewer than 5% of empty reviews reported studies awaiting assessment. Finally, nearly a quarter of empty reviews did not reference any excluded studies, on-going studies or studies awaiting assessment. We found considerable variation across Cochrane Review Groups in terms of numbers and proportions of empty reviews hosted, and at least some variation in time since update, number of excluded studies reported, number of ongoing studies reported, and number of studies awaiting assessment reported.

Implication of Results

Nearly 9% of systematic reviews published in The CDSR on August 15, 2010 had no studies meeting inclusion criteria. Some of these reviews had remained without included studies for more than 10 years. Findings related to age of reviews provide at most a rough estimate of how long empty reviews persist, because studies meeting inclusion criteria may be found on update. The method to identify empty reviews only accounted for those which were empty on August 15, 2010. Reliable information about how long empty reviews remain empty will require examination of all Cochrane reviews to identify previously empty reviews in future research.

The majority of empty reviews (84.8%) were last assessed as up-to-date within the past 5 years, and nearly half (48.7%) had been updated within the past 2 years. Kristiansen reported that although there were about 3200 published Cochrane reviews as of November 2001, there were only 100–200 updates per year [7]. Clarke and colleagues disputed Kristiansen's numbers but admitted that the Collaboration had room to improve its updating practices [8]. To the extent that update rates across The CDSR are similar to those found by Kristiansen, empty reviews would appear to be updated at least as often as reviews with included studies.

Some of the observed differences in reporting of empty reviews across Cochrane Review Groups may relate to differing levels of urgency of questions addressed within topic areas, the ways in which questions are posed or the stringency of inclusion criteria considered across Cochrane Review Groups. On the other hand, observed differences in proportions of empty reviews across Cochrane Review Groups might suggest differing editorial practices and informal policies related to the acceptability of empty reviews.

It may be that many empty reviews result from the problem outlined by Cooper, that is to say that the authors of these reviews are attempting to bring together evidence in the topic area that is immature and, arguably, not currently suitable for review [4]. On the other hand, it may be that the priorities of health care decision-makers and those of researchers do not fit together very well. In either case, the absence of evidence in the empty review might help stimulate appropriate research, resulting in eventual updating of empty reviews with eligible studies.

Explanations for the existence and persistence of these empty reviews remain unknown and warrant further analysis, perhaps through continued monitoring of presently identified empty reviews over time or through detailed qualitative analysis of types of questions posed by the reviews, their inclusion criteria, or the types of conclusions derived from the absence of included studies. Examination of the breadth of questions addressed and inclusion and exclusion criteria may be helpful to understanding the genesis of empty reviews. In addition, examination of quantity and quality of excluded studies reported, reasons for their exclusion, their incorporation in the discussion of implications for practice, and any caveats related to the use of evidence from inferior studies may assist in identifying optimal, and less than optimal, strategies for reporting empty reviews. As the next step in our investigation of empty reviews in The CDSR, we have begun to examine the extent to which these reviews contain specific recommendations for practice and whether these recommendations reflect the absence of included studies in the review or if they incorporate information from other sources. We anticipate being able to report the findings of our continued explorations as subsequent steps are completed.

Limitations

Although very unlikely, it is possible that empty reviews were missed in our search and that the number of these reviews is thus slightly underrepresented. Analysis of the age of empty reviews will require detailed examination of the history of all Cochrane systematic reviews and remains for future research. In addition, the analysis of update status of empty reviews may reflect the update status of all systematic reviews in The CDSR, and not reveal anything unique to empty reviews. Further, our assumption that reporting of excluded studies, ongoing studies, or studies awaiting assessment in empty reviews suggests greater promise of eventual update with includable studies may not be valid, although this point warrants further exploration. In particular, studies awaiting assessment may never be assessed, and ongoing studies, when finished, may not meet inclusion criteria.

Perhaps the most important limitation of our study was the examination of empty reviews at only one point in time. While this snapshot of The CDSR permitted analysis of incidence and prevalence of empty reviews across The CDSR and within Cochrane Review Groups, it did not allow examination of reviews which were previously empty but which have since been updated with eligible studies. Further, this snapshot examination of empty reviews did not permit detection of differences in patterns of updating across Cochrane Review Groups. Thus, our inability to observe the life cycle of empty reviews limits our ability to speculate about the reasons for inconsistencies in incidence and prevalence, updating patterns, or reporting differences across groups.

Conclusions

The stated purpose of Cochrane reviews is to help healthcare providers, consumers, researchers, and policy makers “make well-informed decisions about health care… by providing a reliable synthesis of the available evidence on a given topic… considering all the evidence on the effect of an intervention” [2]. Review authors are currently not given specific guidance on how to report empty reviews, and it is clear that such guidance may be necessary. Work toward development of guidance for reporting empty reviews might benefit from a consensus meeting of systematic review contributors and other stakeholders, informed in part by this study and future explorations. Provisional guidance emerging from such a meeting would then require an iterative revision process following methodology outlined by Moher [9]. Guidance for reporting empty reviews might include clear instructions for the incorporation of potentially important information not always considered in systematic reviews of interventions, as well as specific direction on whether to discuss and how to present the findings of non-included studies. Such guidance may even have implications for the reporting of reviews which have included studies.