Introduction

The quantity of articles published in scholarly journals increased by 200 to 300% from the early 1980s to the late 1990s [1]. Although it seems likely that also the number of available reviewers increased [2], concerns remain about increases in reviewing load [3]–[5], and the possibility of a peer review system in crisis [6]. Such concerns often suggest that the peer review system is susceptible to the tragedy of the commons [7], in which individual behaviour can worsen the situation of a group [8] (though the exact analogy is questionable, depending on if the commons is managed or unmanaged [9]). The peer review system also has been characterised as a “reciprocal altruistic system”, since the same researchers are both authors and reviewers [3]. Ideally, researchers should balance the “withdrawals” they make from the “peer review commons” as authors by contributing the appropriate number of reviews [3], [6].

There are suggestions that some of the most active publishing scientists tend to be least likely to review [8], [10], though such individuals may be quite rare [3]. We know, however, of no quantitative assessment of this phenomenon. More broadly, we know of no quantitative studies of how well individual researchers balance the number of reviews they contribute to the peer review system with the number of reviews they require of the system.

A recent survey based study found that the although individual researchers' “decline to review rate” (number of declined review requests divided number of requests) increased with the number of papers published per year, so did the number of review requests, such that number of reviews also increased [11]. However, the information reported did not address whether individuals balance their “withdrawals” and contributions to the peer review system, since it concerned only number of published papers. Nor did it address the variability among individuals in their balance of withdrawals and contributions. Finally, there was considerable variation in the reported relationship between number of reviews provided and number of papers published.

Here we report a quantitative comparison of individual researchers' balance of reviewing and submission activities for data from four journals of the British Ecological Society over the period 2003–2010.

Methods

Results

There was considerable variation in the relationship between the number reviews provided by researchers, and the number required by their submissions (figure 1). Assuming that required reviews are distributed evenly among coauthors, a total of 1796 (44%) researchers provided more than twice as many reviews as required by their submissions. A total of 814 (20%) researchers provided less than half as many reviews as required by their submissions. These proportions were quite invariant to variation in the year in which researchers first submitted an article (figure 1 inset). The observed variation is much greater than one would expect by chance (i.e., if researchers knew the proportion of review requests they should accept, and accepted requests with this probability) (figure 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. Individual researcher's peer review balance.

Perfect balance would be if researchers reside on the 1∶1 line, since then they provide the same number of reviews as their submissions require. Red data points indicate individuals whose over- or under-contribution is unlikely to have resulted from chance (p>0.05, binomial test). Blue data points indicate p>0.05, though note that tests for individuals close to the origin have low statistical power. Panel (a) corresponds with the assumption that reviews are distributed equally among co-authors, (b) with the assumption that the submitting author is responsible for all reviews. Inset: dynamics of proportion of researchers over-contributing by more than double (upper line) or under-contributing by less than half (lower line). Axes of the main plot are square root scaled to better illustrate variation close to the origin. Y-values in both panels, and x-values in panel b are slightly jittered to assist visualization of otherwise overlaying points.

https://doi.org/10.1371/journal.pone.0092896.g001

If we instead assume that the submitting author is responsible for all required reviews, the proportions change considerably, to 12% providing twice as many reviews as required, and 52% providing half as many as required. Note that the sum of these two proportions is unaffected by the assumption (64% in both cases).

If we return to the assumption that required reviews are evenly distributed among authors, of the 814 reviewers who provided fewer than half as many reviews as required to balance their submissions, 668 did not provide all the reviews they were invited to. This latter group was, in total, responsible for a shortfall of 979 reviews, from a total number of reviews required of 6807.

Researchers who submitted more papers had higher review completion rates (i.e., the number of reviews completed divided by the number of review invitations received) (figure 2). This positive relationship was stronger for researchers who had entered the database relatively recently (e.g., researchers submitting their first manuscript in years 2008–2010, compared to those who submitted their first article in years 2003–2005). This implies that these more recently-added researchers (perhaps students and postdocs) respond more sensitively to greater numbers of requests to review, by accepting a greater proportion of those requests.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. Individual researchers review completion rate, with separate graphs corresponding to the year the researchers entered the database.

Lines are predictions from a quasibinomial response variable model with number of submissions, year of first submission, and their interaction, as continuous explanatory variables. The data analysed was limited to individual researchers with less than six submission, in order to avoid comparing across very different ranges of number of submissions. The model was part of an exploratory analysis, so use of p-values is inappropriate. X-axis values are jittered slightly

https://doi.org/10.1371/journal.pone.0092896.g002

Discussion

That so many researchers provided more reviews than the number required to balance their submissions is encouraging for the peer review system. That somewhere between 20% and 52% of researchers substantially under-contribute also is encouraging. Also encouraging is the tendency for more frequent submitters to also complete a greater proportion of invited reviews.

There was considerable variation among individuals, with some providing many more, and some many fewer reviews than required by their submissions. There are numerous explanations for individuals over- or under-contributing. Quantitative analysis of what drives the behaviour of individual researchers is beyond the scope of our study. With the limited data available to us, we cannot distinguish the relative importance of alternative explanations for inter-individual variation, although we can rule out chance variation. We can also rule out lack of opportunity to review as an explanation for under-contributions, since a large majority of under-contributors declined at least one invitation to review. Applying theories of the evolution and maintenance of cooperation [15], [16] to the peer review system is an intriguing possibility, and below we identify some testable hypotheses.

Over-contributors may be motivated by a sense of professional obligation, by the incentives that some journals offer (e.g., the receipt of free access and public recognition of one's contribution offered by BES journals), by access to papers before publication, and by the opportunity to learn from the comments of the editor and other reviewers. The propensity for researchers who submit more to accept a higher proportion of reviews suggests that, at least on average, researchers believe that submitting a lot requires reviewing a lot (figure 2). The greater strength of this relationship for more recent researchers further suggests that the younger researchers may feel the obligation to balance their submissions and reviews more strongly than the older ones. Alternatively, it could be that these individuals (younger and submitting large numbers of mss) are simply more active researchers in all respects, and that these individuals receive relatively few requests to review so can accept and complete a high proportion. Under-contributors may contribute to the peer review system in other ways (e.g., editing for journals). Or, they may allocate their time to other tasks, in particular tasks that they have strong incentives or obligations to perform (e.g., publishing papers). Under-contribution is understandable given the incentives that many researchers face. Academics must “publish or perish”—not “review or perish”.