Original Investigation

Retracted publications in pharmacy systematic reviews

Sarah Jane Brown¹, Caitlin J. Bakker, AHIP², Nicole R. Theis-Mahon, AHIP³

doi: http://dx.doi.org/10.5195/jmla.2022.1280

Volume 110, Number 1: 47-55
Received 05 2021: Accepted 07 2021

ABSTRACT

Objective:

Systematic reviews and other evidence syntheses, the pinnacle of the evidence pyramid, embody comprehensiveness and rigor; however, retracted data are being incorporated into these publications. This study examines the use of retracted publications in the field of pharmacy, describes characteristics of retracted publications cited by systematic reviews, and discusses factors associated with citation likelihood.

Methods:

Using data from Retraction Watch, we identified retracted publications in the pharmacy field. We identified all articles citing these retracted publications in Web of Science and Scopus and limited results to systematic reviews. We classified the retraction reason, determined whether the citation occurred before or after retraction, and analyzed factors associated with the likelihood of systematic reviews citing a retracted publication.

Results:

Of 1,396 retracted publications, 283 were cited 1,096 times in systematic reviews. Most (65.0%) (712/1096) citations occurred before retraction. Citations were most often to items retracted due to data falsification or manipulation (39.2%), followed by items retracted due to ethical misconduct including plagiarism (30.4%), or concerns about or errors in data or methods (26.2%). Compared to those not cited in systematic reviews, cited items were significantly more likely to be retracted due to data falsification and manipulation, were published in high impact factor journals, and had longer delays between publication and retraction.

Conclusions:

Further analysis of systematic reviews citing retracted publications is needed to determine the impact of flawed data. Librarians understand the nuances involved and can advocate for greater transparency around the retraction process and increase awareness of challenges posed by retractions.

Keywords: retraction of publication as topic; systematic reviews as topic; research; pharmacy; evidence-based pharmacy practice; publishing.

INTRODUCTION

Retracted publications present a complicated problem for systematic reviews. Systematic reviews, meta-analyses, and other evidence syntheses are considered to be the pinnacle of the evidence pyramid as they embody comprehensive literature searching and rigorous methodological quality assessment [1]. Because few health care providers have capacity to search for and read multiple original research articles to address a clinical question, systematic reviews are intended to serve as easily identifiable evidence upon which clinical decisions can be made. This has resulted in an inherent expectation among clinicians that these works can be relied upon consistently. However, despite the structure, rigor, and importance of systematic reviews, flawed research could be incorporated into these publications. Though retracted publications comprise a small segment of the scientific and medical literature, their presence in systematic reviews holds potential to cause harm by influencing patient care and future research [2].

A recent example highlights the potential impact retracted data could have on patient care. The COVID-19 pandemic, the accompanying growth in published COVID-19 research, and the need for a robust evidence base make a timely illustration of the danger fraudulent data pose and the potential for significant and direct patient harm. As of April 19, 2021, Retraction Watch had identified 105 articles on COVID-19 that were retracted for a variety of reasons since the beginning of the COVID-19 pandemic in January 2020 [3]. Several high-profile COVID-19 drug therapy studies were retracted due to suspected data fraud, creating significant implications for patient care and treatment [4]. Although the COVID-19 pandemic may have amplified issues, poor-quality and duplicative research is not unique to the pandemic. While the aforementioned articles were retracted and discredited promptly, this is not a consistent pattern in the scientific literature. Previous research shows that the time between publication and retraction is between 2 and 3.5 years [5–7]. Even for studies that do not result in retraction, methodological issues may remain. Irrelevant questions, poor study design, and biased reporting are known problems, with an estimated 85% of research generating waste [8]. Economic factors and commercial motives in the pharmaceutical industry may also lead to a focus on short-term gains and less rigor [9].

Retracted publications have been documented and analyzed in a variety of biomedical disciplines, including dentistry [10–12], radiation and oncology [13–15], mental health [16], emergency medicine [17], obstetrics and gynecology [18,19], and surgery [20,21]. Previous studies show that a greater proportion of drug therapy articles are retracted for reasons of misconduct and fraud compared with other biomedical studies [22]. One recent cross-sectional study comparing clinical trials data reporting through publications versus ClinicalTrials.gov found that 74% of studies included at least one discrepancy between the publication and the registry [23]. Earlier research found that only 13.4% of clinical trials reported summary results in ClinicalTrials.gov within one year of trial completion, and only 38.3% reported results at any time [24].

This is concerning because these studies constitute the foundational science that higher evidence is built upon, and providers in all health care fields rely on pharmaceutical and drug literature to identify the most appropriate drug therapies for their patients. If flawed data are incorporated into the higher levels of evidence that these providers seek out, there is a potential for risk to patients' health and safety. The broad reach of pharmaceutical literature combined with the greater proportion of instances of scientific misconduct as compared to other health care disciplines leads us to conclude that the influence of retracted publications in pharmacy systematic reviews deserves further scrutiny [22].

Retractions serve an essential function in scientific literature by allowing science to course correct [11,16,25]. They are also the most visible mechanism to indicate scientific misconduct [26]. Publications may be retracted for a variety of reasons that range from administrative publication errors and mistakes to scientific fraud and misconduct; a single paper may also be retracted for more than one reason [6]. The rise in retractions can be attributed to a number of factors, including increased awareness of retractions and their impact, greater adoption of retraction policies, and adherence to publishing and retraction guidelines such as those issued by the Committee on Publication Ethics (COPE) and the International Committee of Medical Journal Editors (ICMJE) [27–29].

Even with guidelines and increased awareness in identifying retracted publications, there are no standard practices for identifying and labeling these publications. In a study of the representation of retracted publications in mental health, records for 144 retracted articles were examined across seven platforms [16]. Almost half (40%) of those records did not indicate that the article had been retracted, and only 10 of the 144 retracted papers were noted as such across all platforms. These inconsistencies are echoed in later research that assessed 150 retracted articles for compliance with ICMJE's seven recommendations for retracting an article [30]. Only 70 of the 150 articles met all ICMJE criteria.

These inconsistencies may contribute to the continued citation of retracted publications [12]. The array of reasons a publication may be retracted also adds complexity to the issue because portions of a retracted study may still hold value. Continued citation is not inherently problematic as citation does not always equate to endorsement [31]. However, previous research shows that retracted publications are consistently cited positively, implying endorsement of the findings [12]. The aim of this study was to explore the presence of retracted publications in systematic reviews in the pharmaceutical and drug therapy literature.

METHODS

We obtained a list of all retracted publications in the Retraction Watch Database [32] from inception to May 2019, courtesy of the Center for Scientific Integrity. Retraction Watch is a searchable database of retracted publications and retraction notices, as well as a web presence that highlights stories of retracted publications. For this study, we used a subset of these data and limited our dataset to publications within the Retraction Watch subject classifications of pharmacology, toxicology, and drug design. These subject classifications were chosen due to their applicability to pharmaceutical sciences.

We performed a cited reference search of each retracted publication in both Scopus and Web of Science. These citation indices were chosen for their broad disciplinary scope and ability to perform a cited reference search using a known item. A cited reference search for each retracted item was performed in both databases to ensure a complete set of citing publications. Our searches in each database were limited to the document types “article” and “review.” We then created a single EndNote file with the exported results of our cited reference searches for each retracted item, removing any duplicate citations found. After performing this task for each retracted publication, we then uploaded all results to Rayyan [33] for screening to identify systematic and other evidence-based syntheses (hence referred to as systematic reviews) that cited the retracted publications. Articles were independently screened for inclusion by two reviewers in two phases: title and abstract and full-text screening. Disputes were resolved by consensus. Following full-text screening, we performed a final confirmatory screening to ensure all articles met inclusion criteria.

Systematic reviews were included if they met one or more of the following criteria: self-identified as a systematic review, scoping review, rapid review, meta-analysis, or clinical practice guideline; were PRISMA compliant [34]; or included a detailed methods section outlining multiple databases searched and terms used. Items were excluded if they self-identified as a narrative review, self-identified as consensus-based guidelines, or full text was not available in English. Systematic reviews that had been retracted were excluded from our analysis.

For each retracted publication, we assigned a single reason for the retraction. We mapped the reasons for retraction listed by Retraction Watch and to the taxonomy outlined by Bar Ilan and Halevi [35]: (1) administrative error, such as journals erroneously publishing articles twice or publishing nonfinal versions; (2) ethical misconduct, such as plagiarism, peer review fabrication, or authorship disputes; or (3) scientific distortion, which includes data errors, fabrication, and manipulation. Bar Ilan and Halevi's initial classification of scientific distortion included both intentional and unintentional errors. We further subdivided this classification into (3a) scientific distortion due to falsification or manipulation, describing situations of willful manipulation or falsification of data and (3b) scientific distortion concerns or errors, describing cases where there were errors or other controversy in data, but intention to distort was not confirmed. These updated categories added nuance to the scientific distortion category and isolated intentional manipulation of data from other errors or unintentional mistakes. Where papers were retracted for more than one reason, we assigned the reason for retraction that was the most problematic. For example, a publication retracted for both data fabrication as well as an administrative error was classified as scientific distortion.

Journal impact factor (JIF) quartiles from Clarivate's Journal Citation Reports (JCR) were also added to the dataset of the retracted publications and their citing systematic reviews. Quartile rankings compare a journal's JIF to other journals in its JCR subject area and assign a quartile ranking (Q1, Q2, Q3, Q4). For example, a Q1 journal would have an impact factor within the top 25% of the JIF distribution of its JCR subject area [36]. The use of quartile rankings, as opposed to the JIFs themselves, accounts for disciplinary differences in citation and publishing practices.

Finally, we identified whether the citation to the retracted publication occurred prior to or following retraction. Citation occurred after the retraction if the retraction notice had been published for at least six months prior to the publication of the citing article. A six-month window was used to account for the length of time between manuscript submission and publication, as the median time between submission to online publication is estimated to be 125 days [37].

We compared retracted publications that were and were not cited in systematic reviews. Chi-square tests were conducted to examine relationships between reason for retraction, JIF quartiles, and having been cited in a systematic review, and an independent samples t-test was conducted to compare length of time from publication to retraction between the two groups. Chi-square tests were also conducted to examine relationships between reason for retraction, JIF quartiles, and whether the citation occurred before or after retraction. Statistical tests were performed using R version 3.6.0 [38].

RESULTS

Retracted publications in the pharmaceutical literature

Of the 1,396 retracted publications identified, 312 retracted publications were cited 32,559 times. The 1,396 publications were most frequently retracted due to ethical misconduct (553/1396, 39.6%), followed by concerns or errors regarding scientific distortion (383/1396, 27.4%) and scientific distortion due to falsification or manipulation (292/1396, 20.9%). Retracted publications were most frequently published in JIF Q1 journals (505/1396, 36.2%), Q2 journals (322/1396, 23.1%), or journals that had not been assigned a JIF (115/1396, 8.2%).

Of the 312 retracted publications that were cited, 283 were cited 1,096 times in systematic reviews. When considering the differences between retracted items that were cited in systematic reviews and those that were not cited in systematic reviews, we found statistically significant differences in their reason for retraction (X²(4,1396)=83.46, p<.001), JIF quartile (X²(4,1396)=28.45, p<.001), and timing of retraction (t(365.05)=-10.805, p<.001). Retracted publications cited in systematic reviews were associated with retraction due to scientific distortion—falsification or manipulation and having been published in JIF Q1 or Q2 journals. Retracted publications cited in systematic reviews had a significantly longer period between publication and notice of retraction than those not cited (3.2 years versus 7.0 years; Table 1).

Systematic reviews citing retracted publications

Of the 1,096 systematic review citations to retracted publications, 712 occurred prior to retraction of the publication, while 384 occurred after the publication's retraction. Considering citations that occurred prior to retraction and those that occurred following retraction, there was a statistically significant difference in the JIF quartile (X²(4,1096)=29.81, p<.0001) and the reason that the cited item was retracted (X²(4,1096)=64.221, p<.0001).

Systematic reviews that cite retracted publications after retraction were more likely to be published in JIF Q1 journals and less likely to be published in journals with no JIF. Publications retracted due to ethical misconduct were associated with citation in systematic reviews after their retraction, while publications retracted due to scientific distortion—falsification or manipulation were associated with being cited in systematic reviews before the retraction (Table 2).

DISCUSSION

Our findings show that retracted publications continue to be cited in systematic reviews in the field of pharmacy. Almost a quarter (20.3%) of the retracted publications in this sample were cited by one or more systematic reviews, and these citations occurred both before and after the issuance of the retraction notice. Because retractions are not consistently or clearly represented across resources [16], it is possible that systematic review authors are unaware that a particular publication has been retracted. Further investigation into the nature of these citations is needed to determine how retracted publications are utilized to support systematic review findings.

Systematic reviews and other evidence-based syntheses occupy the top spot in the evidence hierarchy [1], and librarians teach students to look for systematic reviews to answer clinical questions. Although the reputational and career implications of retraction are recognized by faculty [39], as are the ethical issues of student plagiarism and research misconduct and their implication for the student's academic career [40], the clinical and educational implications of retractions as potential evidence have not been fully addressed in health sciences education. If further analysis of citations to retracted publications showed that systematic reviews regularly incorporate flawed data into their findings and recommendations, what do health sciences students, faculty, and health care providers need to know about retractions and how to incorporate retractions into one's knowledge base? Librarians may be uniquely positioned to play a broader educational role in teaching students about retractions and how retracted data can potentially influence a systematic review. We can teach updated critical appraisal skills and incorporate techniques for assessing the level of rigor applied to a single systematic review. Librarians can also influence the broader awareness of retractions among health sciences students and faculty.

When assessing scientific literature, researchers may use metrics such as the JIF as an indicator of journal quality or rigor [41]. Our investigation found that both the retracted publications and systematic reviews citing the publication after the retraction was issued were more likely to be in JIF Q1. The use of the JIF as a proxy for scientific quality and rigor has previously been questioned [42,43]. Our results support further questioning of this assumption, as well as the assumption that retracted publications are more frequently the problem of less rigorous, less impactful journals, and emphasize the importance of critical appraisal of individual articles.

Critical appraisal involves revisiting previously read materials and integrating new findings into one's understanding. This may be particularly important in the case of fundamentally flawed research, as the retraction process takes time. Fraud and ethical misconduct in scientific studies typically take longer to discover than blatant plagiarism or administrative errors [6], and this is likely because concerns need to be raised and investigations need to be conducted thoroughly [35]. In our sample, retracted publications that were cited in systematic reviews had a significantly longer period between publication and notice of retraction than those that were not cited (3.2 years versus 7.0 years). The lag time between publication and retraction creates more opportunities for retracted data to be used in systematic reviews. This may be less of an issue for reviews that are on a regular update schedule, such as clinical practice guidelines and Cochrane systematic reviews, but does present a problem for standalone systematic reviews.

The need for regular updates of systematic reviews is well established, although challenging in practice. In one study of 100 systematic reviews published between 1995 and 2005, it was found that 23% of reviews were out of date within two years of publication, 15% within one year, and 7% were outdated by the time they were published [44]. While the discussion surrounding the need for updates has largely centered around the emergence of new and contradictory evidence, the inclusion of potentially erroneous data in the initial systematic review heightens the urgency of the need for regular updates. This problem may present an opportunity for librarians to advocate for updates to both systematic review guidance for authors and guidance for publishers to address this problem.

Some systematic review guidance encourages authors to search for and consider retractions [45], while other guidelines do not mention retractions at all [46]. Searching for retractions should be built into the systematic review workflow and can be accomplished through known-item search using resources such as the Retraction Watch Database [32]. However, while procedures to identify retractions among search results may be integrated into the systematic review process, little direction currently exists as to how the retraction should be handled when it comes to inclusion in the systematic review. While it may seem prudent to exclude a retracted publication due to fabrication of data, others have argued that excluding a publication based on its retracted status alone would not align with best practices if that item had met the inclusion and exclusion criteria specified in the systematic review protocol [47]. If systematic review authors can determine the reason for retraction and its potential impact on the systematic review's findings, it allows them to assess the trustworthiness of the data and make decisions about the inclusion of the study. However, retraction notices are often vague, lacking in sufficient detail to determine exactly which parts of the paper may be problematic and the nature of the concerns [48].

The concerning degree of variability in how retractions are represented across databases and a lack of full adherence to COPE and ICMJE guidelines has been well reported [16,30]. Journal retraction policies vary, and retraction notices may list an explicit reason for retraction. Ambiguity and a lack of context or detail may obscure the severity of the problem [10,49,50]. For journal editors, clearer guidance on describing the reason for retraction and better adherence to COPE guidelines may address the opacity and inconsistency in retraction notices. Librarians, as chief negotiators with publishers, can advocate for better transparency around the retraction process and more context and detail in the issued retraction notices.

Literature in the field of pharmacy and the confidentiality around drug development present some unique considerations. Investigations into drug therapy literature have found that the proportion of retractions due to scientific misconduct are significantly higher when compared to the broader biomedical literature [22]. Nearly half of all retracted publications in the field of anesthesia and analgesia are due to fraud or misconduct, although just four authors are responsible for nearly 60% of the retracted anesthesia literature [51]. There is a lack of a culture of sharing and transparency in the pharmaceutical sciences, as drug development data are often proprietary and manufacturers have business interests in keeping it confidential. However, clinical trial data do not fall under the same legal protections as data about product development [52]; studies have shown that data are not reported to ClinicalTrials.gov in a timely manner despite obligations to do so [24,53]. Another study of a sample of 489 ICMJE-affiliated journals found that 56% merely referred to the guidelines without specifying a data-sharing policy [54]. This lack of transparency and explicit guidance may contribute to flawed data entering the peer-reviewed literature and their subsequent citation. One possible solution to assist authors in assessing data, and in preventing the use of flawed data, would be to publish and share data before publication. This increases transparency and reproducibility and has the potential for errors or flaws to be caught before appearing in the literature [49].

This transparency would benefit both information users and information creators, including those whose work has been retracted. There is a stigma around retractions [55]. Retractions imply culpability and have an inherently negative connotation and impact on authors, potentially penalizing honesty. Greater clarity around the reason for retraction and a more nuanced taxonomy for describing reasons for retractions may provide more context and encourage authors to correct unintentional errors. Several researchers have argued for publishers to establish a self-retraction process that would be initiated by the author to correct these types of errors and mistakes and distinguish this type of retraction from one originating from fraud or misconduct [55,56]. This more nuanced approach to retractions, combined with greater transparency in retraction notices and processes, could contribute to enabling users to make informed decisions about when and how they might incorporate retracted articles into their research and practice.

This is not a comprehensive list of retracted publications cited by systematic reviews in the field of pharmacy. The Retraction Watch Database does not currently have subject headings devoted to pharmacy practice. As such, this sample of articles is more representative of basic and pharmaceutical sciences. Pharmacists and those interested in pharmacy practice rely on literature that may not be represented by the titles included in this study since these categories are not necessarily inclusive of social and administrative pharmacy practice. We did not assess the quality of the systematic reviews that were included in our analysis, nor did we assess whether the retracted publications were noted as such in the systematic reviews. As a result, the systematic reviews included in this sample may be of lower quality or may indicate that the publication in question has been retracted.

In conclusion, the presence of retracted publications in pharmacy systematic reviews raises concerns. In this study, 20.3% of retracted publications were cited in systematic reviews. Of these citations, 39.3% of citations were to items retracted due to data falsification or manipulation. This is of particular concern for the field of pharmacy, with implications for human, animal, and environmental health. Further analysis of the systematic reviews citing retracted publications is needed, including an assessment of the nature of the citation and the methodological quality of the systematic review, to understand the scale of the problem and determine the impact of flawed data on systematic review conclusions and recommendations.

This article has been approved for the Medical Library Association's Independent Reading Program <http://www.mlanet.org/page/independent-reading-program>.

ACKNOWLEDGMENTS

We wish to acknowledge Dr. Ivan Oranksy and the Center for Scientific Integrity for providing the data underlying this manuscript.

DATA AVAILABILITY STATEMENT

The data underlying this publication are available through the Center for Scientific Integrity's Retraction Watch Database and are subject to the provisions of a Data Use Agreement. Individuals may request access to this data by contacting the Retraction Watch Database at team@retractionwatch.com.

REFERENCES

1. Paul M, Leibovici L. Systematic review or meta-analysis? Their place in the evidence hierarchy. Clin Microbiol Infect. 2014 20(2):97–100. DOI: http://dx.doi.org/10.1111/1469-0691.12489
.

2. Steen RG. Retractions in the medical literature: how many patients are put at risk by flawed research? J Med Ethics. 2011 Nov;37(11):688–92. DOI: http://dx.doi.org/10.1136/jme.2011.043133
.

3. Retraction Watch. 2021 [cited 2021]. Available from: https://retractionwatch.com/retracted-coronavirus-covid-19-papers/.

4. Ledford H, Van Noorden R. High-profile coronavirus retractions raise concerns about data oversight. Nature. 2020 Jun;582(160). DOI: http://dx.doi.org/10.1038/d41586-020-01695-w
.

5. Foo JY. A retrospective analysis of the trend of retracted publications in the field of biomedical and life sciences. Sci Eng Ethics. 2011 Sep;17(3):459–68. DOI: http://dx.doi.org/10.1007/s11948-010-9212-8
.

6. Steen RG, Casadevall A, Fang FC. Why has the number of scientific retractions increased? PLoS One. 2013 8(7):e68397. DOI: http://dx.doi.org/10.1371/journal.pone.0068397
.

7. Trikalinos NA, Evangelou E, Ioannidis JP. Falsified papers in high-impact journals were slow to retract and indistinguishable from nonfraudulent papers. J Clin Epidemiol. 2008 May;61(5):464–70. DOI: http://dx.doi.org/10.1016/j.jclinepi.2007.11.019
.

8. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–89. DOI: http://dx.doi.org/10.1016/s0140-6736(09)60329-9
.

9. Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JPA, Salman RA-S, Chan A-W, Glasziou P. Biomedical research: increasing value, reducing waste. Lancet. 2014;383(9912):101–4. DOI: http://dx.doi.org/10.1016/s0140-6736(13)62329-6
.

10. Faggion CM, Jr. More detailed guidance on the inclusion/exclusion of retracted articles in systematic reviews is needed. J Clin Epidemiol. 2019 Dec;116:133–4. DOI: http://dx.doi.org/10.1016/j.jclinepi.2019.07.006
.

11. Nogueira TE, Goncalves AS, Leles CR, Batista AC, Costa LR. A survey of retracted articles in dentistry. BMC Res Notes. 2017 Jul;10(1):253. DOI: http://dx.doi.org/10.1186/s13104-017-2576-y
.

12. Theis-Mahon NR, Bakker CJ. The continued citation of retracted publications in dentistry. J Med Libr Assoc. 2020 Jul;108(3):389–97. DOI: http://dx.doi.org/10.5195/jmla.2020.824
.

13. Bozzo A, Bali K, Evaniew N, Ghert M. Retractions in cancer research: a systematic survey. Res Integr Peer Rev. 2017;2:5. DOI: http://dx.doi.org/10.1186/s41073-017-0031-1
.

14. Rosenkrantz AB. Retracted publications within radiology journals. AJR Am J Roentgenol. 2016 Feb;206(2):231–5. DOI: http://dx.doi.org/10.2214/AJR.15.15163
.

15. Wasiak J, Hamilton DG, Foroudi F, Faggion CM, Jr. Surveying retracted studies and notices within the field of radiation oncology. Int J Radiat Oncol Biol Phys. 2018 Nov 1;102(3):660–5. DOI: http://dx.doi.org/10.1016/j.ijrobp.2018.06.028
.

16. Bakker C, Riegelman A. Retracted publications in mental health literature: discovery across bibliographic platforms. J Librariansh Inf Sci. 2018;6(1):eP2199. DOI: http://dx.doi.org/10.7710/2162-3309.2199
.

17. Chauvin A, De Villelongue C, Pateron D, Yordanov Y. A systematic review of retracted publications in emergency medicine. Eur J Emerg Med. 2019 Feb;26(1):19–23. DOI: http://dx.doi.org/10.1097/mej.0000000000000491
.

18. Bennett C, Chambers LM, Al-Hafez L, Michener CM, Falcone T, Yao M, Berghella V. Retracted articles in the obstetrics literature: lessons from the past to change the future. Am J Obstet Gynecol MFM. 2020 Nov;2(4):100201. DOI:http://dx.doi.org/10.1016/j.ajogmf.2020.100201
.

19. Chambers LM, Michener CM, Falcone T. Plagiarism and data falsification are the most common reasons for retracted publications in obstetrics and gynaecology. BJOG. 2019 Aug;126(9):1134–40. DOI: http://dx.doi.org/10.1111/1471-0528.15689
.

20. King EG, Oransky I, Sachs TE, Farber A, Flynn DB, Abritis A, Kalish JA, Siracuse JJ. Analysis of retracted articles in the surgical literature. Am J Surg. 2018 Nov;216(5):851–5. DOI: http://dx.doi.org/10.1016/j.amjsurg.2017.11.033
.

21. Yan J, MacDonald A, Baisi LP, Evaniew N, Bhandari M, Ghert M. Retractions in orthopaedic research: a systematic review. Bone Joint Res. 2016 Jun;5(6):263–8. DOI: http://dx.doi.org/10.1302/2046-3758.56.BJR-2016-0047
.

22. Samp JC, Schumock GT, Pickard AS. Retracted publications in the drug literature. Pharmacotherapy. 2012 Jul;32(7):586–95. DOI: http://dx.doi.org/10.1002/j.1875-9114.2012.01100.x
.

23. Talebi R, Redberg RF, Ross JS. Consistency of trial reporting between ClinicalTrials.gov and corresponding publications: one decade after FDAAA. Trials. 2020 Jul 23;21(1):675. DOI: http://dx.doi.org/10.1186/s13063-020-04603-9
.

24. Anderson ML, Chiswell K, Peterson ED, Tasneem A, Topping J, Califf RM. Compliance with results reporting at ClinicalTrials.gov. N Engl J Med. 2015 Mar 12;372(11):1031–9. DOI: http://dx.doi.org/10.1056/NEJMsa1409364
.

25. Atlas MC. Retraction policies of high-impact biomedical journals. J Med Libr Assoc. 2004 92(2):242–50. https://pubmed.ncbi.nlm.nih.gov/15098054.

26. Hesselmann F, Graf V, Schmidt M, Reinhart M. The visibility of scientific misconduct: a review of the literature on retracted journal articles. Curr Sociol. 2017 Oct;65(6):814–45. DOI:http://dx.doi.org/10.1177/0011392116663807
.

27. COPE Council. COPE Retraction guidelines—English [Internet]. Version 2: November 2019 ed. Committee on Publication Ethics; 2019. <http://dx.doi.org/10.24318/cope.2019.L4
>.

28. International Committee of Medical Journal Editors. Corrections, retractions, republications and version control [Internet]. [cited April 21 2021]. http://www.icmje.org/recommendations/browse/publishing-and-editorial-issues/corrections-and-version-control.html>.

29. Resnik DB, Wager E, Kissling GE. Retraction policies of top scientific journals ranked by impact factor. J Med Libr Assoc. 2015 Jul;103(3):136–9. DOI: http://dx.doi.org/10.3163/1536-5050.103.3.006
.

30. Suelzer EM, Deal J, Hanus KL. Challenges in discovering the retracted status of an article. Discussion paper for Reducing the Inadvertent Spread of Retracted Science: Shaping a Research and Implementation Agenda 2020. http://hdl.handle.net/2142/108367.

31. Garfield E. Can citation indexing be automated? In: Stevens ME, Giuliano VE, Heilprin LB, editors. Statistical association methods for mechanical documentation, symposium proceedings. Miscelleaneous publication 269. Washington, DC National Bureau of Standards; 1965. p. 189–92.

32. Retraction watch database [Internet]. Center for Scientific Integrity. [cited May 2019]. Available from: http://retractiondatabase.org/RetractionSearch.aspx.

33. Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. Syst Rev. 2016 5(1):210. DOI: http://dx.doi.org/10.1186/s13643-016-0384-4
.

34. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009 Jul;339:b2535. DOI:http://dx.doi.org/10.1136/bmj.b2535
.

35. Bar-Ilan J, Halevi G. Temporal characteristics of retracted articles. Scientometrics. 2018 116(3):1771–83. DOI: http://dx.doi.org/10.1007/s11192-018-2802-y
.

36. Web of Science. Journal Citation Reports: learn the basics [Internet]. [cited 24 Feb 2021].<https://clarivate.libguides.com/jcr/basics>.

37. Himmelstein DS, Powell K. Analysis for “the history of publishing delays” blog post v1.0. Zenodo. 3 Feb 2016. DOI: 10.5281/zenodo.45516.

38. R Core Team. R: a language and environment for statistical computing 3.6.0. Vienna, Austria: R Foundation for Statistical Computing; 2019 [cited 26 Apr 2021]. <https://www.R-project.org/>.

39. Azoulay P, Bonatti A, Krieger JL. The career effects of scandal: evidence from scientific retractions. Res Policy. 2017 46(9):1552–69. DOI: http://dx.doi.org/10.1016/j.respol.2017.07.003
.

40. Sonfield MC. Academic plagiarism at the faculty level: legal versus ethical issues and a case study. J Acad Ethics. 2014;12(2):75–87. DOI: http://dx.doi.org/10.1007/s10805-014-9205-3
.

41. Bakker C, Cooper K, Langham-Putrow A, McBurney J. Qualitative analysis of faculty opinions on and perceptions of research impact metrics. College & Research Libraries. 2020;81(6). DOI: 10.5860/crl.81.6.896.

42. Brito R, Rodriguez-Navarro A. Evaluating research and researchers by the journal impact factor: is it better than coin flipping? J Informetr. 2019 Feb;13(1):314–24. DOI: http://dx.doi.org/10.1016/j.joi.2019.01.009
.

43. Moher D, Naudet F, Cristea IA, Miedema F, Ioannidis JPA, Goodman SN. Assessing scientists for hiring, promotion, and tenure. PLoS Biol. 2018 Mar;16(3):e2004089. DOI: http://dx.doi.org/10.1371/journal.pbio.2004089
.

44. Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med. 2007 Aug;147(4):224–33. DOI: http://dx.doi.org/10.7326/0003-4819-147-4-200708210-00179
.

45. Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland L. Chapter 4: Searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, and Welch VA, editors. Cochrane handbook for systematic reviews of interventions. Version 6.1: Cochrane; 2020.

46. Aromataris E, Munn Z. JBI Manual for evidence synthesis 2020. Available from: https://jbi-global-wiki.refined.site/space/MANUAL.

47. Barnish M. The inclusion of retracted studies in systematic reviews: a response to Gray et al. Int J Nurs Stud. 2018 Apr;80:90–1. DOI: http://dx.doi.org/10.1016/Ujnurstu.2017.12.013
.

48. Grey A, Avenell A, Bolland M. Timeliness and content of retraction notices for publications by a single research group. Account Res. 2021 May;5:1–15. DOI: http://dx.doi.org/10.1080/08989621.2021.1920409
.

49. Moylan EC, Kowalczuk MK. Why articles are retracted: a retrospective cross-sectional study of retraction notices at BioMed Central. BMJ Open. 2016 Nov;6(11):e012047. DOI: http://dx.doi.org/10.1136/bmjopen-2016-012047
.

50. Wager E, Williams P. Why and how do journals retract articles? An analysis of Medline retractions 1988-2008. J Med Ethics. 2011 Sep;37(9):567–70. DOI: http://dx.doi.org/10.1136/jme.2010.040964
.

51. Nair S, Yean C, Yoo J, Leff J, Delphin E, Adams DC. Reasons for article retraction in anesthesiology: a comprehensive analysis. Can J Anaesth. 2020 Jan;67(1):57–63. DOI: http://dx.doi.org/10.1007/s12630-019-01508-3
.

52. Kesselheim AS, Mello MM. Confidentiality laws and secrecy in medical research: improving public access to data on drug safety. Health Aff. 2007 Mar-Apr;26(2):483–91. DOI: http://dx.doi.org/10.1377/hlthaff.26.2.483
.

53. Riley IL, Boulware LE, Sun JL, Chiswell K, Que LG, Kraft M, Todd JL, Palmer SM, Anderson ML. Factors associated with reporting results for pulmonary clinical trials in ClinicalTrials.gov. Clin Trials. 2018 Feb;15(1):87-94. DOI: http://dx.doi.org/10.1177/1740774517740352
.

54. Siebert M, Gaba JF, Caquelin L, Gouraud H, Dupuy A, Moher D, Naudet F. Data-sharing recommendations in biomedical journals and randomised controlled trials: an audit of journals following the ICMJE recommendations. BMJ Open. 2020 May 30;10(5):e038887. DOI: http://dx.doi.org/10.1136/bmjopen-2020-038887
.

55. Teixeira da Silva JA, Al-Khatib A. Ending the retraction stigma: encouraging the reporting of errors in the biomedical record. Res Ethics. 2019:251–9. DOI: http://dx.doi.org/10.1177/1747016118802970
.

56. Fanelli D. Set up a ‘self-retraction’ system for honest errors. Nature. 2016 531(7595):415. DOI: http://dx.doi.org/10.1038/531415a
.

Sarah Jane Brown, 1 sjbrown@umn.edu, Assistant Librarian, Health Sciences Library, University of Minnesota Libraries, Minneapolis, MN

Caitlin J. Bakker, AHIP, 2 cjbakker@umn.edu, Associate Librarian, University of Minnesota Libraries, Minneapolis, MN

Nicole R. Theis-Mahon, AHIP, 3 theis025@umn.edu, Associate Librarian, Health Sciences Library, University of Minnesota Libraries, Minneapolis, MN

Copyright © 2022 Sarah Jane Brown, Caitlin J. Bakker, Nicole R. Theis-Mahon

This work is licensed under a Creative Commons Attribution 4.0 International License.

Journal of the Medical Library Association, VOLUME 110, NUMBER 1, January 2022