https://orcid.org/0000-0003-2005-0497
Figures
Abstract
Background
Retraction of published research can reduce the dissemination of incorrect or misleading information, but concerns have been raised about the clarity and rigor of the retraction process. Failure to clearly and consistently retract research has several risks, for example discredited or erroneous research may inform health research studies (e.g. clinical trials), policies and practices, potentially rendering these unreliable.
Objective
To investigate consistency and clarity of research retraction, based on a case study of retracted Covid-19 research.
Study design
A cross-sectional study of retracted Covid-19 articles reporting empirical research findings, based on searches of Medline, Embase and Scopus on 10th July and 19th December 2020.
Key results
We included 46 retracted Covid-19 articles. The number eligible for inclusion nearly doubled, from 26 to 46, in five months. Most articles (67%) were retracted from scientific journals and the remainder from preprint servers. Key findings: (1) reasons for retraction were not reported in 33% (15/46) of cases; (2) time from publication to retraction could not be determined in 43% (20/46) of cases; (3) More than half (59%) of retracted Covid-19 articles (27/46) remained available as original unmarked electronic documents after retraction (33% as full text and 26% as an abstract only). Sources of articles post-retraction were preprint servers, ResearchGate and, less commonly, websites including PubMed Central and the World Health Organization. A retracted journal article which controversially claimed a link between 5G technology and Covid-19 remains available in its original full text from at least 60 different websites.
Conclusions
The retraction process is inconsistent and often ambiguous, with more than half of retracted Covid-19 research articles remaining available, unmarked, from a wide range of online sources. There is an urgent need to improve guidance on the retraction process and to extend this to cover preprint servers. We provide structured recommendations to address these concerns and to reduce the risks that arise when retracted research is inappropriately cited.
Citation: Frampton G, Woods L, Scott DA (2021) Inconsistent and incomplete retraction of published research: A cross-sectional study on Covid-19 retractions and recommendations to mitigate risks for research, policy and practice. PLoS ONE 16(10): e0258935. https://doi.org/10.1371/journal.pone.0258935
Editor: Eleanor Ochodo, Stellenbosch University and Kenya Medical Research Institute, KENYA
Received: January 21, 2021; Accepted: October 10, 2021; Published: October 27, 2021
Copyright: © 2021 Frampton et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting information files.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Retraction of published research is essential for correcting errors in the scientific literature and maintaining trust in research [1–4]. Published research may be retracted for various reasons, including misconduct (fraud, plagiarism, duplicate publication), honest error (mistakes, legitimate artefacts or re-interpretation of data in light of new facts), author disputes, ethical issues, or failure to comply with publishing standards [5–15].
The proportion of published research articles that have been retracted increased steadily during the 1970s to 2000s, across a wide range of scholarly disciplines [1, 2, 5–8, 10, 16], but the rate of increase slowed after 2012 [1, 2]. The frequency of retraction reflects a balance between the rate at which flawed or erroneous research is published and the extent to which the scientific community (authors and their institutions, editors and publishers, and readers of the research) can detect and act upon inappropriate publications. Retraction rates have been found to vary with the scholarly discipline [8, 16], journal impact factor [3, 4, 6, 14] and country of authorship [5, 7].
Retraction was first listed as a specific publication category in Medline in 1980, and the National Library of Medicine has provided bibliographic indexing of retracted articles with links to their retraction notices since 1984. Web of Science has listed retracted articles since around 2000; and the International Committee of Medical Journal Editors (ICMJE) have recommended since at least 1997 that a retraction notice should be provided clearly stating the reason for retraction [10]. However, studies investigating retractions during the 1980s to 2010s found that the reasons for retraction were ambiguous or not reported for 5% to 20% of retracted articles [4, 5, 7, 8, 10–12, 14, 15, 17–19]. Furthermore, many retracted articles remained available post-retraction without being marked as retracted [4, 10, 17, 18, 20–22] and retracted articles have been cited in the academic literature post-retraction without acknowledgement of their retracted status [3, 7, 8, 12, 14, 23–27]. The time from publication to retraction has varied considerably, ranging from several months to more than 10 years [5, 7, 8, 12–15, 18], and occasionally more than 20 years [5]. Studies have also shown that retraction policies and the clarity and consistency of retraction notices have varied widely across academic journals [4, 6, 7, 10, 19, 28]. Retraction of research may be requested by authors, editors, journals, publishers, or other parties such as an author’s institution. The majority of retractions (59% to 81%) assessed were made by authors [10, 12, 15], whilst the person or organisation responsible for retraction was unclear or not reported in 5% to 53% of retractions [10, 15, 19].
In response to these inconsistencies the Committee on Publication Ethics (COPE) published guidelines in 2009 to improve consistency and clarity of the retraction process [29]. The COPE guidelines state that notices of retraction should clearly identify the retracted article and be linked with it; the retraction notices should be available freely, state the reason for retraction and who is retracting; and publications should be retracted as soon as possible after the editor is concerned that a publication is seriously flawed. The COPE guidelines also state that “prompt retraction should minimise the number of researchers who cite the erroneous work, act on its findings, or draw incorrect conclusions, such as from ‘double counting’ redundant publications in meta-analyses or similar instances” [29]. In a further development to improve understanding of the retraction process and promote good practice, a not-for-profit organisation, Retraction Watch, was launched in 2010 and now hosts a database of over 80,000 retracted scientific articles, as well as blogs discussing key articles and issues with retractions [30].
The Covid-19 pandemic has resulted in an enormous proliferation of scientific research and scientific articles [31, 32]. For instance, searches for Covid-19 articles on PubMed found that the publication rate had already reached 59 articles per day by late April 2020 [33], with almost 10 published every working hour by early May 2020 [32]. To date, more than 100,000 COVID-19 articles have been produced [34]. Concerns have been raised that the “rush to publish” Covid-19 research has led to a decline in scientific rigor, resulting in the mass publication of poor quality research [32, 35], creating extensive research waste [36], and leading to inappropriate and harmful information dissemination and decision making [37]. A number of high-profile Covid-19 studies have been retracted after publication [37–44].
Retraction of research is associated with several risks, including that discredited or erroneous work may continue to be cited, and may inform health research studies (e.g. clinical trials), policies and practices, as well as causing research waste if research, policies or practices have to be suspended or corrected. Examples of consequences arising from retractions of Covid-19 research are provided in S1 Appendix.
When reading the scientific literature on Covid-19 we came across several anomalies in the retraction of Covid-19 research studies whereby we were unable to reconcile information on why and/or when some retractions had taken place. Several studies investigating the characteristics of retracted Covid-19 research have also reported ambiguity and inconsistency of the retraction process [38, 40, 43–45]. However, the prevalence of these issues is unclear. We therefore conducted a cross-sectional study of the Covid-19 articles that had been retracted up to 19th December 2020, to clarify the extent of inconsistencies in the process for retracting Covid-19 research and whether any recommendations could be made to address them. We focused on Covid-19 retractions because (i) the relatively high rate of retractions provides a contemporaneous and sizeable cohort for analysis, and (ii) retractions could have important implications for decision-making in a pandemic if not communicated clearly and consistently.
Materials and methods
Research question, aim and objectives
Our research question is “How consistent and transparent is the process for retracting Covid-19 research from scientific journals and preprint servers?”. Our aim was to investigate the consistency and transparency of the process for retracting Covid-19 research articles, based on a representative sample of publication sources, to help determine whether any specific changes in the retraction process are necessary. We focused on issues relating to the retraction process that occur once a decision has been made to retract an article, i.e. we did not investigate ways to prevent retractions (which would require a different type of research study). Our specific objectives were (1) to investigate the completeness of reporting of the process for retracting COVID-19 articles and the accessibility of articles after retraction during 2020; and (2) if necessary and appropriate, to provide recommendations on how specific aspects of the retraction process could be improved.
Methodological approach
We conducted a cross-sectional study of Covid-19 research articles retracted during 2020. To ensure objectivity and reduce the risk of introducing bias in our methods, we conducted a systematic search to identify a representative sample of the retracted articles; specified inclusion criteria that the articles should meet; required at least two researchers to assess each included article; and employed an iterative approach to data extraction (details below).
Searches
We conducted searches for retracted articles in Ovid Medline, Ovid Embase and Scopus bibliographic databases on 10th July and 19th December 2020. We used a combination of key terms, and MeSH and EMTREE index terms, relating to retraction, combined with Covid-19 terms from published Covid-19 search strings (the EPPI Centre’s Covid-19 living systematic map of the evidence [46] search strategy for Medline; and a modified version of the Centers for Disease Control and Prevention Library search string for Covid-19 for Scopus [47]. Our Medline search strategy is shown in S2 Appendix.
Article selection
Many of the Covid-19 articles have been posted on preprint servers [31, 39, 48–52]. Given the increasing importance of preprint servers as a means of promptly disseminating research we included retractions both from academic journals and preprint servers.
Articles were eligible for inclusion in our analysis if they met all of the following criteria: (1) The article had Covid-19 as its main focus. (2) The article had been made available either on a preprint server or was published in a scientific journal but had been subsequently retracted (described either as retracted or withdrawn). Articles which were described as “accepted” by journals and were available online as full text were eligible. (3) The article reported empirical research, was a review of empirical research, or reported case studies or case series. Letters, commentaries and opinion papers were excluded unless they reported a description of the methods and results of empirical research. We also excluded a retracted fake paper on Covid-19 that had been written specifically to test the rigor of a suspected predatory journal’s peer review and editorial practices [53].
We included articles classified as a “temporary removal”, provided that the article had not been reinstated or replaced by the time of our final assessment (19th December 2020).
Data extraction
Each of the eligible studies for analysis was read by the research team (GF, LW, DAS) and a draft table of study characteristics was prepared as a data extraction template. The template was modified iteratively as relevant information from the studies was encountered on each read. The final template (S3 and S4 Appendices) was then approved by the research team and applied to all the included studies to ensure consistency of the extracted data. One researcher (GF or LW) extracted data into the template and this was then checked by a different researcher (LW or DAS). Any disagreements in interpretation were resolved through consultation between all researchers.
Results
We identified 26 relevant retracted Covid-19 articles from our search on 10th July 2020 and a further 20 unique articles from our search on 19th December 2020, i.e. the number of eligible articles had nearly doubled in a 5-month period, giving a total of 46 retracted articles in total for analysis. Our searches gave very similar results to those of Retraction Watch [54]: we included only two articles [55, 56] that were not in the Retraction Watch list at 19th December 2020.
Characteristics of the retracted articles
Thirty-one articles (67%) were published in 31 unique scientific journals and 15 were published on the preprint servers bioRxiv, [57] medRxiv [52] and SSRN [48] (S3 Appendix). Of the 31 articles published by scientific journals, 8 were retracted by the authors, 14 were retracted by the journal (i.e. the editors and/or publisher), 6 were retracted by an unspecified combination of the authors and journal, while the retractor was unclear for 3 articles. Where reported, the articles hosted on preprint servers were retracted by the authors (Table 1).
Characteristics of retractions
Characteristics of the retractions are summarised in Table 1. Seven articles were retracted due to duplication or plagiarism [58–64]. The remaining 39 articles (85%) each reported a unique study.
Papers were retracted for a wide range of reasons. Where reported, the reasons for retraction included some relatively minor issues as well as major problems with study conduct or data analysis. Thus, the “weight” of a retraction (i.e. it’s importance in relation to ensuring good standards of research rigor) differed across studies.
Unclear reasons for retraction (N = 15).
COPE guidelines state that the reason for retraction should always be reported [29]. However, for 15 of the 46 Covid-19 retractions (33%) this information was not provided (Table 1). Scientific journals rather than preprint servers were responsible for most of these unclear retractions (12 and 3 retractions respectively).
Retraction due to misconduct and noncompliance with standards (N = 10).
Ten retractions (22%) were related to research misconduct or noncompliance with research or publication standards (other than plagiarism or duplication). These included two high-profile studies by Mehra et al. [65, 66] retracted from the Lancet and New England Journal of Medicine (NEJM) due to concerns about the credibility of the Surgisphere company which had supplied the datasets for both studies [41, 67]. The articles retracted due to issues of misconduct or noncompliance with standards were equally split (5 each) between preprint servers and scientific journals (4 articles).
Retraction to update an article with new information (N = 9).
Nine retractions (20%) were stated to have been retracted to enable an article to be updated with new information, for example based on comments received during peer review. At the time of our final assessment (19th December 2020) none of these articles had been replaced and it is unclear whether, and when, they would be updated. Most (7/9) of these articles were retracted from preprint servers and in nearly all (8/9) cases it was the article authors who had retracted them. The one retraction requested by a scientific journal was retracted due to concerns that the conclusions were based on theory not field epidemiology data and thus further research was needed [68].
Retraction due to errors in data or analyses (N = 4).
Four articles (9%) were retracted due to errors in data or analyses, all of which had initially been published in scientific journals, although in 3 cases it was the article authors who requested the retraction. It is unclear how the authors came to be aware of these errors.
Retraction due to duplicate publication or plagiarism (N = 7).
Six articles were duplicates of previously published papers, [58–60, 62–64] and one [61] was retracted due to plagiarism, all of which were retracted by journals. Two of the duplicate articles reflect self-plagiarism [58, 59] whilst the remaining four were accidental repeat duplications by four different journals [60, 62–64].
Other reasons for retraction (N = 1).
One article on COVID-19 was retracted from a journal on the authors’ request because the authors became aware that their paper was not the first to document mortality in a pregnant COVID-19 patient [69]. Instead of deleting one line from their abstract to correct this (“To our knowledge, this is the first report of maternal death with confirmed COVID-19 infection”) they retracted the full article.
Time from publication to retraction
The time period between publication of an article and its retraction was calculable for only 26 of the 46 retracted articles (57%) (S4 Appendix). Where available, the median time from publication to retraction was longer for articles published in scientific journals (45.5 days; range 7 to 217 days; n = 14) than for those published on preprint servers (10.5 days; range 9 to 34 days; n = 12). Three of the four articles published on the bioRxiv preprint server were retracted after only one day, [87] two days [96] and four days [88]. This very short timescale from publication to retraction suggests lack of adequate preparation of manuscripts, perhaps reflecting the lack of detailed author guidelines for preprint servers.
Availability of original research articles after retraction
Of the 46 retracted Covid-19 articles, 27 (59%) continued to be available as original unmarked versions from online sources after they had been retracted (Table 2). Fifteen articles (33%) remained available as original full-text articles whilst the remaining 12 (26%) were available only as abstracts. The host preprint servers where the articles had originally been published (medRxiv, bioRxiv and SSRN) and the ResearchGate academic networking platform were the key sources of these original full text articles and abstracts (for further details see S3 Appendix). Some articles which had been retracted relatively early in the course of the Covid-19 pandemic, in January and February 2020 [87, 96, 97], had been available as original unmarked full-text versions for more than 8 months after retraction at our check on 19th December 2020 (Table 2).
The availability of full text articles and abstracts shown in Table 2 is a snapshot of a dynamic situation. For example, a search of Google Scholar showed that a journal article on “5G Technology and induction of coronavirus in skin cells” by Fioranelli et al. 2020 [89] retracted on 24th July 2020 was still available on 5th September 2020 from up to 47 different online sources, which increased to 60 online sources by 19th December 2020. This article remained available from 60 online sources at 26th June 2021, although three of the original sources were no longer available whilst a further three new online sources emerged (S5 Appendix). The Fioranelli article is highly controversial, claiming an unsubstantiated notion that 5G Technology influences Covid-19 infection.
In some cases unmarked original versions of retracted articles remained available post-retraction at reputable online sites: a full-text article that had been removed from the journal website continued to be available from PubMed Central (PMC) [56] (though not available from the sister PubMed site), whilst two retracted abstracts continued to be available in the WHO COVID Literature Collection [104, 105].
For eight of the 12 retracted articles that had been published on the bioRxiv and medRxiv preprint servers [85–87, 91, 94, 95, 97, 100], clicking on the “Download PDF” link gave access to an unmarked original version of the full text. Moreover, for three articles the “Info/History” tab on these preprint servers provided several different unmarked original versions of the full text article [85–87].
Inappropriate citation of retracted articles
Due to the relatively short timescale of our study we could not investigate the citation patterns of retracted Covid-19 research. However, we noted that an editorial published in the NEJM [106] continued to cite a study [66] that had been retracted from the same journal on 4th June 2020, with the editorial remaining uncorrected at 26th June 2021, more than 12 months after the retraction.
Discussion and recommendations
We identified several problems with the way that Covid-19 articles have been retracted, notably lack of clarity on the timing of and reasons for retractions, and continued availability of retracted articles, often from multiple sources. Concerns had already been raised across a range of scholarly disciplines prior to the Covid-19 pandemic regarding ambiguity in the reasons for retraction [4, 5, 7, 8, 10–12, 14, 15, 17–19], continued availability of published research after retraction [4, 10, 17, 18, 20–22] and continued citation of retracted research [3, 7, 8, 12, 14, 23–27], suggesting that COPE guidelines were not adequately adhered to prior to 2020. However, these studies did not compare retractions before and after the COPE guidelines [29] were published and so the impact of the guidelines is difficult to determine. Our findings, together with those of other studies on Covid-19 retractions [38–40, 44], indicate that improvements in adherence to retraction guidelines are urgently needed.
The underlying reasons for the continued lack of clarity and consistency in the retraction of Covid-19 journal articles are unclear but could in part reflect inadequate awareness of the existence of the COPE guidelines, and inadequate editorial oversight of the large amounts of Covid-19 research published during the pandemic [39, 107]. The lack of clarity and consistency in the retraction of articles from preprint servers is perhaps not surprising since there are no guidelines on the process for retracting articles from preprint servers (the COPE guidelines [29] refer only to retractions from academic journals). Since preprint servers have become an important route for disseminating Covid-19 research ideas and findings [48–51] there is a need for guidance on retractions to be extended to cover preprint servers.
Risks of incomplete and ambiguous Covid-19 retractions
Continued availability of research articles after retraction is a serious problem because it creates uncertainty as to which version(s) of published articles can be trusted, potentially undermining scientific research and public confidence in medicine and other sciences. During the Covid-19 pandemic, stakeholders including the public, news organisations and scientists have had to make sense of rapidly-emerging and often contradictory information in order to make critical decisions whilst inaccurate information has also been circulating, e.g. on social media channels and the Internet. The paper by Fioranelli et al. illustrates the ease with which inaccurate information can be disseminated by failing to completely retract discredited ideas (S5 Appendix).
Retraction can have serious consequences, depending on the nature of the retracted information and the impact that it has on end-users of the research. Retracted articles may inform health agency guidance, systematic reviews, meta-analyses, or prediction models, potentially rendering their conclusions and recommendations unreliable. Clinical research (e.g. controlled trials) may be suspended or adjusted based on findings from retracted articles, with a range of implications for patients, investigators and health agencies. Retracted articles may be described in detail in newspapers or other popular media, potentially disseminating unreliable information to the public. And retractions can in turn trigger further retractions or corrections (“domino effect”), potentially creating confusion around which versions of publications are reliable. Examples of each of these issues and consequences arising from the retraction of Covid-19 articles are provided in S1 Appendix.
Key retraction problems and solutions
Incomplete retraction: The problem.
The continued availability of more than half of the retracted COVID-19 articles as unmarked pre-retraction versions defeats the purpose of the retraction process, which is to prevent the dissemination of discredited or unreliable science [29]. As illustrated by the links in S5 Appendix, unmarked original versions of retracted articles shared by online sites may be downloaded directly via a search engine, bypassing any contextual information at the host websites. The COPE guidelines do not provide any specific recommendations to prevent the dissemination of articles after they have been retracted.
Incomplete retraction: Solutions.
All copies of a retracted article, including both the abstract and the full text (and any accompanying materials), should be clearly marked as retracted [50]. A prudent marking approach for retracted articles would be to use a strong (e.g. bold red) watermark stating “retracted” or “withdrawn” that diagonally covers each full page of the article so that the retraction status of all parts of an article cannot be in doubt (several of the retracted articles included in our study provide examples of this [58, 59, 61, 65, 101]). Clear watermarking of all retracted articles is a logical failsafe to ensure that the retracted status of an article will remain clear if it becomes detached from its retraction notice or contextual information at a host website.
Once a decision has been made to retract an article, it should be the authors’ responsibility to ensure that all copies of the article documents (abstract, full text, supplementary material) that they have made available are clearly marked as retracted. This applies to copies of the article on preprint servers and websites, including personal websites, ResearchGate and other academic networking platforms, as well as any copies of the original article stored on a computer. Journal editors should check that any linked documents, such as editorials, are updated as necessary to reflect the retraction. Ideally, editors (or at a broader level publishers) should have a standard operating procedure to ensure that no documents are missed from consideration when an article is retracted.
Unclear reasons for retraction: The problem.
One third of the retracted Covid-19 articles did not report the reason for retraction, which prevents lessons from being learnt, and could lead to the unnecessary repetition of errors. Failure to disclose the reasons for retraction makes it more difficult to identify and ameliorate misconduct.
Lack of explanation of the reasons for retraction also creates ambiguity around which versions of an article can be trusted, particularly if an article is retracted temporarily and later reinstated. For example, an article reporting on factors associated with SARS-CoV-2 infection [84] was temporarily removed from the Journal of Hospital Infection and later reinstated (S3 Appendix) with the same citation details but no explanation of why the article had been retracted, when the reinstatement was made, or whether the reinstated version differed in any way from the retracted version. This could lead to two different versions of the article in circulation which would be indistinguishable in bibliographies.
Unclear reasons for retraction: Solutions.
The COPE guidelines [29] recommend that the “reason(s) for retraction” should be stated, and that the retraction notice should “be objective, factual, and avoid inflammatory language” [29]. We understand that some publishers currently appear to have a contrary policy to this. For example, Elsevier does not require authors to provide a reason when requesting retraction of articles from their scientific journals, and we note that Elsevier had the largest market share of the journal articles from which COVID-19 articles had been retracted (S4 Appendix). Publishers should ensure that the reason for the retraction is published in the retraction notice.
The COPE guidelines state that retraction notices should be linked to retracted articles where possible but do not state the reverse, i.e. that retracted articles should also be linked to their retraction notices. Publishers of articles should, ideally, ensure that a retracted article is directly linked to its retraction notice, e.g. via an embedded hyperlink.
Unclear timing of retractions: The problem.
We were unable to determine the retraction date for 43% (20/46) of the retracted Covid-19 articles (S4 Appendix). For these articles it is not possible to accurately determine either the time from publication to retraction or the time that an article remains available post-retraction.
Perhaps unsurprisingly given their relatively large market share of the journals included in our study, the publisher Elsevier accounted for around half of the retracted articles for which the date of retraction was not reported (S4 Appendix), although five articles retracted from journals published by Elsevier did report the retraction date, suggesting an inconsistent policy.
Unclear timing of retractions: Solutions.
The COPE guidelines [29] state that “notices of retraction should be published promptly to minimise harmful effects” but they do not specify that the date of the retraction and of the retraction notice should be provided. The date of publication and date of retraction should both be clearly stated, alongside the reason for retraction, in the retraction notice posted by a scientific journal or on a preprint server.
General recommendations on retraction from the pre-Covid-19 literature
As well as the COPE guidelines published in 2009, prior to the Covid-19 pandemic several recommendations had been made regarding how the retraction process should be improved, and our findings suggest that these recommendations remain pertinent:
- Better mechanisms are needed for raising researchers’ awareness of retracted literature [8, 14], including better systems for linking retractions to their retraction notices [6].
- Scientists and their students should develop the habit of looking for corrections or retractions of works they cite in their publications [25]. Journals should require prospective authors to declare that they have checked their manuscript’s reference list for retracted articles [108]. Computer software should be implemented to check reference lists for retracted articles free of charge in any database or the internet [24].
- A better vocabulary for talking about retractions is needed and a database for classifying retractions would be useful [6]. Standardisation of the wording of retraction notices may be helpful [3], for example using a standard retraction form with a checklist of retraction reasons [11].
- Automated methods to deliver article status updates (such as CrossMark) may reduce persistence of error in the scientific literature [22].
- Retracted articles should be cited appropriately, with negative (i.e. critical) citation encouraged [9].
- General reforms that have been suggested include increasing the responsibility of journal editors; and providing greater transparency and clarity around mistakes in research [6]. An example of how journals could take ownership of their retraction data and use self-critical reflection to improve standards is provided by Fang et al. [3].
Recommendations on retraction based on our findings
Below we provide comprehensive recommendations on good practice for the retraction of published articles, bringing together the findings from our study on Covid-19 retractions, recommendations made by other authors prior to the Covid-19 pandemic summarised above, and the COPE guidelines on retraction [29]. These recommendations are broadly applicable, not limited to Covid-19 research, and cover articles published on preprint servers as well as those published in academic journals.
For any recommendations to be acted upon, the relevant stakeholders (authors, preprint managers, journal editors and journal publishers, as well as the wider scientific community, including authors of reviews, meta-analyses and guidelines) must first be aware of the problem. It is important therefore that the limitations in the retraction process are widely communicated. Our recommendations are more detailed, focused and extensive than those provided by the COPE guidelines and we hope that they will be useful to the Committee on Publication Ethics and other stakeholders when updating future retraction guidelines.
Recommendations for study authors, journal editors and publishers, and preprint managers.
Table 3 provides our recommendations for (i) authors of retracted articles, (ii) journal editors and publishers, and (iii) preprint server managers to address the specific problems of (a) continued availability of retracted articles, specifically the need to ensure that all retracted articles are clearly marked as such; (b) lack of reporting of the reasons for retraction; (c) lack of reporting of the date of retraction; and (d) ambiguity in who retracted an article.
Recommendations for all authors of articles, including authors of reviews, meta-analyses, guidelines and prediction models.
Table 4 provides our recommendations for authors of all articles.
The COPE guidelines on retraction [29] acknowledge that secondary research articles, including systematic reviews and meta-analyses may themselves need to be corrected or retracted if they include retracted studies, but as far as we are aware there is no agreed process for ensuring that this happens in a consistent way, or indeed at all. Scientific journals should require authors to demonstrate that they have checked for any retractions when preparing their manuscript.
To ensure transparency in systematic reviews and meta-analyses, any articles that are otherwise eligible but excluded due to having been retracted should be specified as retracted in the list of excluded studies when following standard PRISMA [112] principles.
Future research needs
Raise awareness of research retraction issues.
An extensive literature has developed in relation to limitations of the retraction process, but to our knowledge the critical problem that most retracted COVID-19 articles continue to be accessible as unmarked original versions of the abstract or full text is not widely known. It is important therefore to raise awareness of this problem, to support the recommendations to reduce dissemination of retracted research.
Encourage the checking of retraction status.
The feasibility of developing common standards for encouraging authors to check for retractions before submitting their manuscript for publication should be investigated. Existing reporting checklists, such as CONSORT [113], STROBE [114], SPIRIT [115], PRISMA [112] and others [116] could be updated to require authors to declare that they have checked for retractions when preparing their manuscript. We note that Retraction Watch is currently listed as a resource by Cochrane Training [117] but there is not a requirement that authors of reviews should check that none of their included studies and cited articles have been retracted.
Guidelines for retractions from preprint servers and other online sources.
Given the proliferation of preprint servers and their popularity as sources of scientific publication [31, 39, 48–51] a set of good practice publishing standards should be established for preprint servers and the Internet. We understand that the COPE has a broad remit that covers the spectrum of scientific publication, not limited to academic journals, and it seems logical for the COPE guidelines on retraction to be extended to these online publication sources. However, as our study confirms, some of the existing COPE guidelines are not consistently followed and it is imperative that the reasons for this are investigated and understood in order to improve the likelihood of future compliance. Furthermore, as discussed above, current COPE guidelines are insufficiently detailed in some areas.
Preprints are seen as an important way of soliciting open peer review and scientific journals have been working increasingly in collaboration with preprint servers to improve the peer review of scientific research prior to publication [52, 118–120]. Of the 100 top-ranked clinical journals, 86% now allow authors to submit their articles to preprint servers prior to publication in the journal [50]. Journals which work in association with preprint servers could have a role to play in helping to ensure that appropriate guidelines for retraction are applied by the preprint servers that they are linked with.
Link articles to information on their retraction status.
As illustrated by the hyperlinks in S5 Appendix, unmarked electronic versions of retracted articles can easily be downloaded without needing to visit the host website for any contextual information. Many researchers may never see retractions or corrections because they just download the electronic copies of the articles they need and never again consult the original source [6]. This problem was recognised over a decade ago and was one of the factors that led to the development of the CrossMark button for electronic articles [6]. Clicking on an article’s CrossMark button gives the reader instant access to information on whether the content has been updated, corrected or retracted [110] and saves readers time compared to the other ways of checking retractions shown in Table 4. Unfortunately, CrossMark is not available on many electronic articles. Initiatives to promote wider uptake of CrossMark, or other information linking systems for electronic articles are needed.
Limitations of this study
Our recommendations concern good practice in handling retractions once a decision to retract an article has been made; but they do not address ways to avoid the need for retractions in the first place, such as by ensuring that reporting standards are adhered to prior to publication [121] or by optimising peer review [43], as this is beyond the scope of our study. A list of legitimate reasons for retracting articles is provided in the COPE guidelines on retraction [29].
The increasing frequency of COVID-19 retractions means that our study, which was up to date at 19th December 2020, could be quickly superseded with new retractions [122]. However, the critical finding, that most of the retracted COVID-19 articles continue to be available after retraction, would remain valid unless a substantial rapid change in the behaviour of authors, preprint server managers, journal editors and publishers occurs. This seems very unlikely in the absence of general awareness of the problem and a current lack of recommendations to address it [54].
Continued citation of retracted articles was a well-known problem prior to the Covid-19 pandemic [3, 5, 7–9, 14, 18, 20, 23, 24, 26, 27], with some articles being cited for more than a decade after their retraction [20, 24, 27]. However, we were unable to include a meaningful analysis of citation patterns in our study due to the relatively short time that has elapsed since Covid-19 articles were retracted and the unclear timing of retractions for 43% of the articles.
Finally, our study was not based on a pre-specified and peer-reviewed protocol. This was due to the limited timescale we had available to conduct the work on this fast-moving topic. We protected our methods from becoming overly subjective or biased in the absence of a protocol by using an iterative approach to data extraction to ensure all articles were assessed consistently and fairly. All extracted data were checked by at least two researchers to minimise subjectivity and errors, and all data supporting our conclusions and recommendations are provided in the accompanying Tables and Appendices together with references and links to their original sources.
Conclusions
More than half of retracted Covid-19 articles remain available, unmarked, after retraction, from a wide range of online sources. The process for retracting covid-19 research is inconsistent and often unclear, with key information about the reasons for and timing of retraction frequently missing or ambiguous. Our findings suggest an urgent need to improve guidance on the retraction process, such as updating the COPE guidelines, and extending guidance to cover retractions from preprint servers as well as from journals. We provide targeted evidence-based recommendations to help authors of retracted articles, journal editors and publishers, and preprint server managers to achieve this. We also provide recommendations for authors of any research documents on how to reduce the risk of inappropriately citing retracted articles. Our case study was on Covid-19 retractions but our findings are consistent with those of studies in other scholarly disciplines and our recommendations are therefore applicable generally, not limited to Covid-19 research.
Supporting information
S1 Appendix. Examples of research waste and other consequences of Covid-19 retractions.
https://doi.org/10.1371/journal.pone.0258935.s001
(PDF)
S3 Appendix. Characteristics of retracted Covid-19 articles.
https://doi.org/10.1371/journal.pone.0258935.s003
(PDF)
S4 Appendix. Timing of publication and retraction of Covid-19 articles.
https://doi.org/10.1371/journal.pone.0258935.s004
(PDF)
S5 Appendix. Sources of unmarked copies of the retracted paper by Fioranelli et al [89].
https://doi.org/10.1371/journal.pone.0258935.s005
(PDF)
Acknowledgments
We thank colleagues in the Wessex Institute, University of Southampton, and four reviewers for providing comments on a draft of the manuscript.
References
- 1. Brainard J, You J. What a massive database of retracted papers reveals about science publishing’s ‘death penalty’. Science. 2018.
- 2. Brainard J. Rethinking retractions. Science. 2018;362(6413):390–3. pmid:30361352
- 3. Fang FC, Casadevall A. Retracted Science and the Retraction Index. Infection and Immunity. 2011;79(10):3855–9. pmid:21825063
- 4. Marcus A, Oransky I. What studies of retractions tell us. Journal of Microbiology & Biology Education. 2014;15(2):151–4. pmid:25574267
- 5. Bozzo A, Bali K, Evaniew N, Ghert M. Retractions in cancer research: a systematic survey. Research Integrity and Peer Review. 2017;2:5:1–7. pmid:29451549
- 6. Van Noorden R. Science publishing: the trouble with retractions. Nature. 2011;478:26–8. pmid:21979026
- 7. Fang FC, Steen RG, Casadevall A. Misconduct accounts for the majority of retracted scientific publications. Proceedings of the National Academy of Sciences. 2012;109(42):17028–33. pmid:23027971
- 8. Grieneisen ML, Zhang M. A comprehensive survey of retracted articles from the scholarly literature. PLoS ONE. 2012;7(10):e44118. pmid:23115617
- 9. Bolboacă SD, Buhai D-V, Aluaş M, Bulboacă AE. Post retraction citations among manuscripts reporting a radiology-imaging diagnostic method. PLoS ONE. 2019;14(6): e0217918. pmid:31194762
- 10. Wager E, Williams P. Why and how do journals retract articles? An analysis of Medline retractions 1988–2008. Journal of Medical Ethics. 2011;37(9):567–70. pmid:21486985
- 11. Decullier E, Huot L, Samson G, Maisonneuve H. Visibility of retractions: a cross-sectional one-year study. BMC Research Notes. 2013;6:238:1–6. pmid:23782596
- 12. Budd JM, Sievert M, Schultz TR. Phenomena of retraction: reasons for retraction and citations to the publications. JAMA. 1998;280:296–7. pmid:9676689
- 13. Nath SB, Marcus SC, Druss BG. Retractions in the research literature: misconduct or mistakes? Medical Journal of Australia. 2006;185:152–4. pmid:16893357
- 14. Redman BK, Yarandi HN, Merz JF. Empirical developments in retraction. Journal of Medical Ethics. 2008;34:807–9. pmid:18974415
- 15. Huh S, Kim SY, Cho H-M. Characteristics of retractions from Korean medical journals in the KoreaMed database: A bibliometric analysis. PloS One. 2016;11(10):e0163588:1–7. pmid:27706245
- 16. Lu SF, Jin GZ, Uzzi B, Jones B. The retraction penalty: evidence from the web of science. Scientific Reports. 2013;3:3146:1–5. pmid:24192909
- 17. Steen RG. Retractions in the scientific literature: is the incidence of research fraud increasing? Journal of Medical Ethics. 2011;37(4):249–53. pmid:21186208
- 18. Budd JM, Sievert ME, Schultz TR, Scoville C. Effects of article retraction on citation and practice in medicine. Bulletin of the Medical Library Association. 1999;87(4):437–43. pmid:10550028
- 19.
Vuong Q-H. The limitations of retraction notices and the heroic acts of authors who correct the scholarly record: An analysis of retractions of papers published from 1975 to 2019. Learned Publishing. 2020;33(2):119–30.
- 20. Kim SY, Yi HJ, Cho H-M, Huh S. How many retracted articles indexed in KoreaMed were cited 1 year after retraction notification. Science Editing. 2019;6(2):122–7.
- 21. Pfeiffer MP, Snodgrass GL. Medical school libraries’ handling of articles that report invalid science. Academic Medicine. 1992;67(2):109–13. pmid:1546986
- 22. Davis P. The persistence of error: a study of retracted articles on the Internet and in personal libraries. Journal of the Medical Library Association. 2012;100(3):184–9. pmid:22879807
- 23. Neale AV, Dailey RK, Abrams J. Analysis of citations to biomedical articles affected by scientific misconduct. Science and Engineering Ethics. 2010;16:251–61. pmid:19597966
- 24. Korpela KM. How long does it take for the scientific literature to purge itself of fraudulent material?: the Breuning case revisited. Current Medical Research and Opinion. 2010;26(4):843–7. pmid:20136577
- 25. Garfield E. How to avoid spreading error. Scientists must search for corrections. The Scientist. 1987;1(19):255–7.
- 26. Pfeifer MP, Snodgrass GL. The continued use of retracted, invalid scientific literature. JAMA. 1990;263(10):1420–3. pmid:2406475
- 27. Schneider J, Ye D, Hill AM, Whitehorn AS. Continued post-retraction citation of a fraudulent clinical trial report, 11 years after it was retracted for falsifying data. Scientometrics. 2020;125:2877–913.
- 28. Marcus A. A scientist’s fraudulent studies put patients at risk. Science. 2018;362(6413):394. pmid:30361354
- 29.
Committee on Publication Ethics (COPE). Retraction Guidelines. Version 2 2009. https://publicationethics.org/node/19896
- 30.
Retraction Watch. Retraction Watch. Tracking retractions as a window into the scientific process: Center for Scientific Integrity; 2020 [11th January 2020]. https://retractionwatch.com/.
- 31.
Fraser N, Brierley L, Dey G, Polka JK, Pálfy M, Nanni F, et al. Preprinting the COVID-19 pandemic [preprint pending peer review]. bioRxiv preprint server. 2020:1–34.
- 32. Bonini S, Maltese G. COVID-19 Clinical trials: Quality matters more than quantity. Allergy. 2020;75(10):2542–7. pmid:32434272
- 33. Kambhampati SBS, Vaishya R, Vaish A. Unprecedented surge in publications related to COVID-19 in the first three months of pandemic: A bibliometric analytic report. Journal of Clinical Orthopaedics and Trauma. 2020;11:S304–S6. pmid:32405191
- 34.
PRIMER. COVID-19 Primer 2021 [14th January 2021]. https://covid19primer.com/dashboard.
- 35. Alexander PE, Debono BV, Mammen MJ, Iorio A, Aryal K, Deng D, et al. COVID-19 coronavirus research has overall low methodological quality thus far: case in point for chloroquine/hydroxychloroquine. Journal of Clinical Epidemiology 2020;123:120–6. pmid:32330521
- 36. Glasziou P, Sanders S, Hoffmann T. Waste in Covid-19 research. BMJ. 2020;369:m1847:1–2. pmid:32398241
- 37. Ioannidis JPA. Coronavirus disease 2019: the harms of exaggerated information and non-evidence-based measures. European Journal of Clinical Investigation. 2020;50(4):e13223:1–5.
- 38. Yeo-Teh NSL, Tang BL. An alarming retraction rate for scientific publications on Coronavirus Disease 2019 (COVID-19). Accountability in Research. 2020:1–7.
- 39. Bramstedt KA. The carnage of substandard research during the COVID-19 pandemic: a call for quality. Journal of Medical Ethics. 2020;46:803–7. pmid:33004545
- 40. Teixeira da Silva JA. Silently withdrawn or retracted preprints related to Covid-19 are a scholarly threat and a potential public health risk: theoretical arguments and suggested recommendations. Online Information Review. 2020.
- 41. Piller C, Travis J. Authors, elite journals under fire after major retractions. Science (New York, NY). 2020;368(6496):1167–8. pmid:32527807
- 42. Soltani P, Patini R. Retracted COVID-19 articles: a side-effect of the hot race to publication. Scientometrics. 2020;125:819–22.
- 43. Teixeira da Silva JA, Bornemann-Cimenti H, Tsigaris P. Optimizing peer review to minimize the risk of retracting COVID-19-related literature. Medicine, Health Care, and Philosophy. 2020;Nov 20 (epub ahead of print):1–6. pmid:33216274
- 44.
Retraction Watch. Retraction Watch 2020. https://retractionwatch.com/.
- 45. Boschiero MN, Carvalho TA, Marson FAL. Retraction in the era of COVID-19 and its influence on evidence-based medicine: is science in jeopardy? Pulmonology. 2020;25:25. pmid:33281107
- 46.
EPPI Centre. COVID-19: a living systematic map of the evidence 2020 [cited 2020]. http://eppi.ioe.ac.uk/cms/Projects/DepartmentofHealthandSocialCare/Publishedreviews/COVID-19Livingsystematicmapoftheevidence/tabid/3765/Default.aspx.
- 47.
CDC Library. COVID-19 Scopus search alert: Centers for Disease Control and Prevention; 2020 [cited 2020 10/07/2020]. https://www.cdc.gov/library/researchguides/2019novelcoronavirus/scopussearchalert.html.
- 48. Navabi Nouri S, Cohen YA, Madhavan MD, Slomka PJ, Iskandrian AE, Einstein AJ. Preprint manuscripts and servers in the era of coronavirus disease 2019. Journal of Evaluation in Clinical Practice. 2020;October 2020:1–6.
- 49.
The Economist. High speed science 2020 [updated 9th May 2020; cited 2021]. 64–65. https://www.economist.com/leaders/2020/05/09/speeding-up-science-during-the-pandemic.
- 50. Flanagin A, Fontanarosa PB, Bauchner H. Preprints Involving Medical Research—Do the Benefits Outweigh the Challenges? JAMA. 2020;324(18):1840–3. pmid:33170226
- 51. Majumder MS, Mandl KD. Early in the epidemic: impact of preprints on global discourse about COVID-19 transmissibility. Lancet Global Health. 2020;8:e627–e30. pmid:32220289
- 52. Rawlinson C, Bloom T. New preprint server for medical research. BMJ. 2019;365:l2301:1–2. pmid:31167753
- 53. Oodendijk W, Rochoy M, Ruggeri V, Cova F, Lembrouille D, Trottinetta S, et al. SARS-CoV-2 was unexpectedly deadlier than push-scooters: could hydroxychloroquine be the unique solution? Asian Journal of Medicine and Health. 2020;18(9):14–21.
- 54.
Retraction Watch. Retracted coronavirus (COVID-19) papers 2020 [7 July 2020]. https://retractionwatch.com/retracted-coronavirus-covid-19-papers/.
- 55. An X-S, Li X-Y, Shang F-T, Yang S-F, Zhao J-Y, Yang X-Z, et al. [RETRACTED] Clinical characteristics and blood test results in COVID-19 patients. Annals of Clinical and Laboratory Science. 2020;50(3):299–307. Retraction notice in: Annals of Clinical and Laboratory Science.2020 Jul;50(4):560. pmid:32581016
- 56. Irshad M, Khattak SA, Hassan MM, Majeed M, Bashir S. WITHDRAWN: How perceived threat of Covid-19 causes turnover intention among Pakistani nurses: A moderation and mediation analysis. International Journal of Mental Health Nursing. 2020;30(1):350-. Withdrawal notice at: pmid:32779356
- 57.
Sever R, Roeder T, Hindle S, Sussman L, Black K-J, Argentine J, et al. bioRxiv: the preprint server for biology. bioRxiv [Preprint Server]. 2019:833400.
- 58. Huang Y, Zhao N. [RETRACTED] Mental health burden for the public affected by the COVID-19 outbreak in China: Who will be the high-risk group? Psychology, Health & Medicine. 2020;26(1):23–4. Retraction notice at: Psychology, Health & Medicine. https://doi.org/10.1080/13548506.2020.1833631
- 59. Huang Y, Zhao N. RETRACTED: Chinese mental health burden during the COVID-19 pandemic. Asian Journal of Psychiatry. 2020;51: 102052:1–3. Retraction notice at: Asian Journal of Psychiatry.2020.54:102482. https://doi.org/10.1016/j.ajp.2020.102482 pmid:32361387
- 60. Mulvey JJ, Magro CM, Ma LX, Nuovo GJ, Baergen RN. WITHDRAWN: A mechanistic analysis placental intravascular thrombus formation in COVID-19 patients. Annals of Diagnostic Pathology. 2020;46: 151529. Duplicate of article in: Annals of Diagnostic Pathology. 2020. 46:151530. pmid:32361635
- 61. Singh A. [RETRACTED] Noninvasive versus invasive ventilation: One modality cannot fit all during COVID-19 outbreak. Korean Journal of Anesthesiology. 2020;73(4):359–61. Retraction notice in: Korean Journal of Anesthesiology. 2020. 73(5):468. https://ekja.org/journal/view.php?doi=10.4097/kja.20227.r1
- 62. Argyropoulos KV, Serrano A, Hu J, Black M, Feng X, Shen G, et al. WITHDRAWN: Association of initial viral load in SARS-CoV-2 patients with outcome and symptoms. American Journal of Pathology. 2020. Duplicate of: American Journal of Pathology. 190(9);1881–1887: https://doi.org/10.1016/j.ajpath.2020.07.001. Withdrawal notice at: https://doi.org/10.1016/j.ajpath.2020.06.012 pmid:32650002
- 63. El-Assaad I, Hood-Pishchany MI, Kheir J, Mistry K, Dixit A, Halyabar O, et al. Complete Heart Block, Severe Ventricular Dysfunction, and Myocardial Inflammation in a Child With COVID-19 Infection. JACC: Case Reports. 2020;2(9):1351–5. Retraction notice at: https://www.sciencedirect.com/science/article/pii/S2666084920305854?via%3Dihub pmid:32835278
- 64. Hedima EW, Adeyemi MS, Ikunaiye NY. WITHDRAWN: Community pharmacists: On the frontline of health service against COVID-19 in LMICs. Research in Social and Administrative Pharmacy. 2020. Duplicate of: Research in Social and Administrative Pharmacy. 2021.17(1):1964–1966. https://doi.org/10.1016/j.sapharm.2020.04.013. Withdrawal notice at: pmid:32340893
- 65. Mehra MR, Desai SS, Ruschitzka F, Patel AN. RETRACTED: Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis. Lancet. 2020;(published online May 22). Retraction notice at: Lancet.395(10240):1820 https://doi.org/10.1016/S0140-6736(20)31324-6
- 66. Mehra MR, Desai SS, Kuy S, Henry TD, Patel AN. Retraction: Cardiovascular Disease, Drug Therapy, and Mortality in Covid-19. N Engl J Med. New England Journal of Medicine. 2020;382(26):2582. Retraction notice at: New England Journal of Medicine.2020 June 4. 382:2582.
- 67. Servick K, Enserink M. The pandemic’s first major research scandal erupts. Science. 2020;368(6495):1041–2. pmid:32499418
- 68. Zhuang GH, Shen MW, Zeng LX, Mi BB, Chen FY, Liu WJ, et al. [WITHDRAWN: Potential false-positive rate among the ’asymptomatic infected individuals’ in close contacts of COVID-19 patients]. Zhonghua Liu Xing Bing Xue Za Zhi. 2020;41(4):485–8. Withdrawal notice not available. pmid:32133832
- 69. Karami P, Naghavi M, Feyzi A, Aghamohammadi M, Novin MS, Mobaien A, et al. WITHDRAWN: Mortality of a pregnant patient diagnosed with COVID-19: A case report with clinical, radiological, and histopathological findings. Travel Medicine and Infectious Disease. 2020:101665. Withdrawal notice at: pmid:32283217
- 70.
Patel A, Desai S. [RETRACTED] Ivermectin in COVID-19 Related Critical Illness. SSRN Preprint Server. 2020. Watermarked retracted document at: https://www.isglobal.org/documents/10179/6022921/Patel+et+al.+2020+version+1.pdf
- 71.
Patel AN, Desai SS, Grainger DW, Mehra MR. [RETRACTED] Usefulness of ivermectin in Covid-19 illness. Online source. 2020. Watermarked retracted document at: https://www.isglobal.org/documents/10179/6022921/Patel+et+al.+2020+version+2.pdf/
- 72.
Raharusuna P, Priambada S, Budiarti C, Agung E, Budi C. [RETRACTED] Patterns of COVID-19 Mortality and Vitamin D: An Indonesian Study. SSRN Preprint server. 2020. No retraction notice available.
- 73. Dumantepe M, Aydin S, Yildiz E, Okur HK, Kocagoz AS, Gundogdu Y, et al. WITHDRAWN: Subsegmental Thrombus in COVID-19 Pneumonia: Immuno-Thrombosis or Pulmonary Embolism? Data Analysis of Hospitalized Patients with Coronavirus Disease. Heart, Lung & Circulation. 2020;24:24. Withdrawal notice at: pmid:32917551
- 74.
Luowei L, Zheng H, Shanliang X, Hao Y, Xinping J, Hui W, et al. [A new coronal form of aerosol transmission in public transportation Epidemiological investigation of viral pneumonia cluster epidemic situation.] [Practical Preventive Medicine]. 2020;1–3. No withdrawal notice available
- 75.
Kanwar BA. WITHDRAWN: Proposal for Initiative of Evidence-based Treatment of COVID-19 Patients with Worsening Hypoxia. American Journal of Respiratory and Critical Care Medicine. 2020;Online ahead of print. Withdrawal notice at: https://pubmed.ncbi.nlm.nih.gov/33026820/
- 76. Zhang X, Jiang Z, Yuan X, Wang Y, Huang D, Hu R, et al. WITHDRAWN: Nurses reports of actual work hours and preferred work hours per shift among frontline nurses during coronavirus disease 2019 (COVID-19) epidemic: A cross-sectional survey. International Journal of Nursing Studies. 2020:103635. Withdrawal notice at: pmid:32425239
- 77.
Etemadifar M, Aghababaee A, Sedaghat N, Rayani M, Nouri H, Abhari A, et al. WITHDRAWN: Incidence and mortality of COVID-19 in Iranian multiple sclerosis patients treated with disease-modifying therapies. Revue Neurologique. 2020;Online ahead of print. Withdrawal notice at: https://doi.org/10.1016/j.neurol.2020.08.001
- 78. Vavougios GD. WITHDRAWN: Selenium—associated gene signatures within the SARS-CoV-2—host genomic interaction interface. Free Radical Biology & Medicine. 2020;15:15. Withdrawal notice at: pmid:32679367
- 79. Bility MT, Agarwal Y, Ho S, Castronova I, Beatty C, Biradar S, et al. WITHDRAWN: Can Traditional Chinese Medicine provide insights into controlling the COVID-19 pandemic: Serpentinization-induced lithospheric long-wavelength magnetic anomalies in Proterozoic bedrocks in a weakened geomagnetic field mediate the aberrant transformation of biogenic molecules in COVID-19 via magnetic catalysis. Science of the Total Environment. 2020;Online October 2020: 142830. Withdrawal notice at: https://doi.org/10.1016/j.scitotenv.2020.142830
- 80. Alhowary AaA, Aleshawi AJ, Othman AR, Obeidat AY, Khader YS. WITHDRAWN: Mechanical ventilation in COVID-19: Is it due to patient or virology factors? Annals of Medicine and Surgery. 2020. Withdrawal notice at: pmid:32837717
- 81. Wei J, Lei P, Yang H, Fan B, Qiu Y, Zeng B, et al. WITHDRAWN: Analysis of thin-section CT in patients with coronavirus disease (COVID-19) after hospital discharge. Clinical Imaging. 2020. Withdrawal notice at: pmid:32425336
- 82. Kumar D. [RETRACTED] Corona Virus Killed by Sound Vibrations Produced by Thali or Ghanti: A Potential Hypothesis. Journal of Molecular Pharmaceuticals and Regulatory Affairs. 2020;2(2). Retraction notice at: https://core.ac.uk/download/pdf/328107419.pdf
- 83. Wang XF, Yuan J, Zheng YJ, Chen J, Bao YM, Wang YR, et al. [Retracted: Clinical and epidemiological characteristics of 34 children with 2019 novel coronavirus infection in Shenzhen]. Zhonghua Er Ke Za Zhi. 2020;58(0):E008. No retraction notice available. pmid:32062875
- 84. Maltezou HC, Dedoukou X, Tsonou P, Tseroni M, Raftopoulos V, Pavli A, et al. TEMPORARY REMOVAL: Hospital factors associated with SARS-CoV-2 infection among healthcare personnel in Greece. Journal of Hospital Infection. 2020;22:22.
- 85.
Gaibazzi N, Martini C, Mattioli M, Tuttolomondo D, Guidorrossi A, Suma S, et al. [WITHDRAWN] Lung disease severity, coronary artery calcium, coronary inflammation and mortality in Coronavirus disease 2019. medRxiv Preprint Server. 2020. Withdrawal notice at: https://www.medrxiv.org/content/10.1101/2020.05.01.20087114v3
- 86.
Gaibazzi N, Tuttolomondo D, Guidorossi A, Botti A, Tedeschi A, Martini C, et al. [WITHDRAWN] Smoking Prevalence is Low in Symptomatic Patients Admitted for COVID-19. medRxiv Preprint Server. 2020. Withdrawal notice at: https://www.medrxiv.org/content/10.1101/2020.05.05.20092015v3
- 87.
Chen Z, Zhang W, Lu Y, Guo C, Guo Z, Liao C, et al. [WITHDRAWN] From SARS-CoV to Wuhan 2019-nCoV: Will History Repeat Itself? bioRxiv Preprint Server. 2020:2020.01.24.919241. Withdrawal notice at: https://www.biorxiv.org/content/10.1101/2020.01.24.919241v4.article-info
- 88.
Parves MR, Riza YM, Mahmud S, Islam R, Ahmed S, Evy BA, et al. [WITHDRAWN] Analysis of Ten Microsecond simulation data of SARS-CoV-2 dimeric main protease. bioRxiv Preprint Server. 2020. Withdrawal notice at: https://www.biorxiv.org/content/10.1101/2020.04.10.036020v2
- 89. Fioranelli M, Sepheri A, Roccia MG, Jafferani M, Olisova OY, Lomonosov KM, et al. RETRACTED: 5G Technology and induction of coronavirus in skin cells. Journal of Biological Regulators and Homeostatic Agents. 2020;34(4):Epub ahead of print. Retraction notice at: https://www.biolifesas.org/biolife/2020/07/27/retracted-5g-technology-and-induction-of-coronavirus-in-skin-cells/
- 90. Beato-Vibora PI. RETRACTED: No deleterious effect of lockdown due to COVID-19 pandemic on glycaemic control, measured by glucose monitoring, in adults with type 1 diabetes. Diabetes Technology and Therapeutics. 2020. Retraction notice at: pmid:32396400
- 91.
Cercy SP. [WITHDRAWN] Psychiatric predictors of COVID-19 outcomes in a skilled nursing facility cohort. medRxiv Preprint Server. 2020. Withdrawal notice at: https://www.medrxiv.org/content/10.1101/2020.05.23.20099671v2
- 92.
Editor and Publisher of International Journal of Mental Health Nursing. Withdrawn: How perceived threat of Covid-19 causes turnover intention among Pakistani nurses: A moderation and mediation analysis. International Journal of Mental Health Nursing. 2020.
- 93.
Chu P, Zhou Z, Gao Z, Cai R, Wu S, Sun Z, et al. [WITHDRAWN] Computational analysis suggests putative intermediate animal hosts of the SARS-CoV-2. bioRxiv Preprint Server. 2020. Withdrawal notice at: https://www.biorxiv.org/content/10.1101/2020.04.04.025080v2
- 94.
Siyu C, Xia M, Wen W, Cui L, Yang W, Liu S, et al. [WITHDRAWN] Mental health status and coping strategy of medical workers in China during The COVID-19 outbreak. medRxiv Preprint Server. 2020. Withdrawal notice at: https://www.medrxiv.org/content/10.1101/2020.02.23.20026872v2
- 95.
Davido B, Lansaman T, Bessis S, Lawrence C, Alvarez J-C, Mascitti H, et al. [WITHDRAWN] Hydroxychloroquine plus azithromycin: a potential interest in reducing in-hospital morbidity due to COVID-19 pneumonia (HI-ZY-COVID)? medRxiv Preprint Server. 2020. Withdrawal notice at: https://www.medrxiv.org/content/10.1101/2020.05.05.20088757v2
- 96.
Pradhan P, Pandey AK, Mishra A, Gupta P, Tripathi PK, Menon MB, et al. [WITHDRAWN] Uncanny similarity of unique inserts in the 2019-nCoV spike protein to HIV-1 gp120 and Gag. bioRxiv Preprint Server. 2020. Withdrawal notice at: https://www.biorxiv.org/content/10.1101/2020.01.30.927871v2
- 97.
Yang Y, Lu Q, Liu M, Wang Y, Zhang A, Jalali N, et al. [WITHDRAWN] Epidemiological and clinical features of the 2019 novel coronavirus outbreak in China. medRxiv Preprint Server. 2020. Withdrawal notice at: https://www.medrxiv.org/content/10.1101/2020.02.10.20021675v2
- 98.
Adjodah D, Dinakar K, Fraiberger SP, Rutherford GW, Glidden DV, Gandhi M. [WITHDRAWN] Decrease in Hospitalizations for COVID-19 after Mask Mandates in 1083 U.S. Counties. medRxiv Preprint Server. 2020. Withdrawal notice at: https://www.medrxiv.org/content/10.1101/2020.10.21.20208728v2.
- 99.
Chen P, Lei J, Wy Y, Liu G, B Z. [WITHDRAWN] Liver impairment associated with disease progression in COVID 19 patients. Liver International. 2020;Online ahead of print. Withdrawal notice at: https://onlinelibrary.wiley.com/doi/abs/10.1111/liv.14481 and https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7262073/
- 100.
Kim MS, Jang S-W, Park Y-K, Kim B-o, Hwang T-H, Kang SH, et al. [WITHDRAWN] Treatment response to hydroxychloroquine, lopinavir/ritonavir, and antibiotics for moderate COVID 19: A first report on the pharmacological outcomes from South Korea. medRxiv Preprint Server. 2020. Withdrawal notice at: https://www.medrxiv.org/content/10.1101/2020.05.13.20094193v2
- 101. Bae S, Kim MC, Kim JY, Cha HH, Lim JS, Jung J, et al. [RETRACTED] Effectiveness of Surgical and Cotton Masks in Blocking SARS-CoV-2: A Controlled Comparison in 4 Patients. Annals of Internal Medicine. 2020;173(1):W22–W3. Retraction notice in: Annals of Internal Medicine. 173(1);79. https://doi.org/10.7326/L20-0745 pmid:32251511
- 102. Wang X, Xu W, Hu G, Xia S, Sun Z, Liu Z, et al. RETRACTED ARTICLE: SARS-CoV-2 infects T lymphocytes through its spike protein-mediated membrane fusion. Cellular & Molecular Immunology. 2020. Retraction notice at: Cellular & Molecular Immunology. 2020. 17:894. https://www.nature.com/articles/s41423-020-0498-4 pmid:32265513
- 103. Nouvier M, Chalencon E, Novle-Catin E, Pelletier S, Hallonet P, Charre C, et al. [RETRACTED] First viral replication of Covid-19 identified in the peritoneal dialysis fluid of a symptomatic patient. Bulletin de la Dialyse à Domicile. 2020;3(1):54503. Retraction notice at: Bulletin de la Dialyse à Domicile. 2020. 3(1):54713. https://bdd.rdplf.org/index.php/bdd/article/view/54713
- 104.
Adjodah D, Dinakar K, Fraiberger S, Rutherford GW, Glidden DV, Gandhi M. Decrease in Hospitalizations for COVID-19 after Mask Mandates in 1083 U.S. Counties. WHO COVID-19 Global literature on coronavirus disease. 2020;Article ID ppmedrxiv-20208728.
- 105. Fioranelli M, Sepehri A, Roccia MG, Jafferani M, Olisova OY, Lemonosov KM, et al. 5G Technology and induction of coronavirus in skin cells. WHO COVID-19 Global literature on coronavirus disease. 2020;covidwho-646363.
- 106. Jarcho JA, Ingelfinger JR, Hamel MB, D’Agostino RB, Harrington DP. Inhibitors of the renin–angiotensin–aldosterone system and Covid-19. NEJM. 2020;382:2462–4. pmid:32356625
- 107. Ledford H, Van Noorden R. High-profile coronavirus retractions raise concerns about data oversight. Nature. 2020;582(7811):160. pmid:32504025
- 108. Drury NE, Karamanou DM. Citation of retracted articles: A call for vigilance. The Annals of Thoracic Surgery. 2009;87(2):670.
- 109. Dotto L, Kinalski M, Machado PS, Pereira GKR, Sarkis-Onofre R, Fernandes dos Santos MB. The mass production of systematic reviews about COVID-19:An analysis of PROSPERO records. Journal of Evidence Based Medicine. 2021;14:56–64. pmid:33595200
- 110.
Crossref. Crossmark factsheet. 2021. https://www.crossref.org/pdfs/about-crossmark.pdf
- 111.
Zotero. Retracted item notifications with Retraction Watch integration 2021 [11th January 2021]. https://www.zotero.org/blog/retracted-item-notifications/.
- 112. Page MJ, Moher D, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ. 2021;372:n160:1–36. pmid:33781993
- 113. Schultz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c332. pmid:20332509
- 114. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Medicine. 2007;4(10):e296. pmid:17941714
- 115. Chan A-W, Tetzlaff JM, Altman DG, Laupacis A, Gøtzsche PC, Krleža-Jerić K, et al. SPIRIT 2013 Statement: Defining standard protocol items for clinical trials. Annals of Internal Medicine. 2013;158(3):200–7. pmid:23295957
- 116.
EQUATOR Network. EQUATOR Network. Enhancing the quality and transparency of health research. Reporting guidelines 2021 [May 2021]. https://www.equator-network.org/reporting-guidelines/
- 117.
Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, et al. Chapter 4: Searching for and selecting studies May 2021. https://training.cochrane.org/resource/tsc-induction-mentoring-training-guide/trials-search-co-ordinator-handbook/9-professional-0
- 118.
PLOS. Community Comments Continued: An update on the preprint commenting pilot at PLOS: PLOS (Public Library of Science); 2020 [cited 2021]. from: https://theplosblog.plos.org/2020/10/community-comments-continued-an-update-on-the-preprint-commenting-pilot-at-plos/.
- 119. Kleinert S, Horton R. Preprints with The Lancet: joining online research discussion platforms. Lancet. 2018;391(10139):2482–3. pmid:29976456
- 120. Kleinert S, Horton R. Preprints with The Lancet are here to stay. Lancet. 2020;396(10254):805. pmid:32950077
- 121. Benchimol EI, Moher D, Ehrenstein V. Retraction of COVID-19 pharmacoepidemiology research could have been avoided by effective use of reporting guidelines. Clinical Epidemiology. 2020;12:1403–20. pmid:33376409
- 122. Lu X, Zhang M, Qian A, Tang L, Xu S. Lung ultrasound score in establishing the timing of intubation in COVID-19 interstitial pneumonia: A preliminary retrospective observational study. PLoS One. 2020;15(9): e0238679:1–10. pmid:32881950