Building capacity to encourage research reproducibility and #MakeResearchTrue
DOI:
https://doi.org/10.5195/jmla.2018.273Keywords:
Reproducibility, Program Evaluation, Replicability, Collaborations, Conference PlanningAbstract
Background: Research into study replication and reporting has led to wide concern about a reproducibility crisis. Reproducibility is coming to the attention of major grant funders, including the National Institutes of Health, which launched new grant application instructions regarding rigor and reproducibility in 2015.
Study Purpose: In this case study, the authors present one library’s work to help increase awareness of reproducibility and to build capacity for our institution to improve reproducibility of ongoing and future research.
Case Presentation: Library faculty partnered with campus research leaders to create a daylong conference on research reproducibility, followed by a post-conference day with workshops and an additional seminar. Attendees came from nearly all schools and colleges on campus, as well as from other institutions, nationally and internationally. Feedback on the conference was positive, leading to efforts to sustain the momentum achieved at the conference. New networking and educational opportunities are in development.
Discussion: Libraries are uniquely positioned to lead educational and capacity-building efforts on campus around research reproducibility. Costs are high and partnerships are required, but such efforts can lead to positive change institution-wide.References
Ioannidis JPA. Why most published research findings are false. PLOS Med. 2005 Aug;2(8):e124. DOI: http://dx.doi.org/10.1371/journal.pmed.0020124.
Begley CG, Ellis LM. Drug development: raise standards for preclinical cancer research. Nature. 2012 Mar 28;483(7391):531–3. DOI: http://dx.doi.org/10.1038/483531a.
Open Science Collaboration. Estimating the reproducibility of psychological science. Science. 2015 Aug 28;349(6251):aac4716. DOI: http://dx.doi.org/10.1126/science.aac4716.
Baker M, Dolgin E. Cancer reproducibility project releases first results. Nature. 2017 Jan 18;541(7637):269–70. DOI: http://dx.doi.org/10.1038/541269a.
National Institutes of Health, Agency for Healthcare Research and Quality. Implementing rigor and transparency in NIH & AHRQ research grant applications [NOT-OD-16-011] [Internet]. Washington, DC: The Institutes; 9 Oct 2015 [cited 13 Nov 2017]. <https://grants.nih.gov/grants/guide/notice-files/NOT-OD-16-011.html>.
Collins FS, Tabak LA. Policy: NIH plans to enhance reproducibility. Nature. 2014 Jan 30;505(7485):612–3. DOI: http://dx.doi.org/10.1038/505612a.
Goodman SN, Fanelli D, Ioannidis JPA. What does research reproducibility mean? Sci Transl Med. 2016 Jun 1;8(341):341ps12. DOI: http://dx.doi.org/10.1126/scitranslmed.aaf5027.
Hines WC, Su Y, Kuhn I, Polyak K, Bissell MJ. Sorting out the FACS: a devil in the details. Cell Rep. 2014 Mar 13;6(5):779–81. DOI: http://dx.doi.org/10.1016/j.celrep.2014.02.021.
Neimark J. Line of attack. Science. 2015 Feb 27;347(6225):938–40. DOI: http://dx.doi.org/10.1126/science.347.6225.938.
Jones PM, Chow JTY, Arango MF, Fridfinnson JA, Gai N, Lam K, Turkstra TP. Comparison of registered and reported outcomes in randomized clinical trials published in anesthesiology journals. Anesth Analg. 2017 Jul 10. DOI: http://dx.doi.org/10.1213/ane.0000000000002272.
Head ML, Holman L, Lanfear R, Kahn AT, Jennions MD. The extent and consequences of p-hacking in science. PLOS Biol. 2015 Mar;13(3):e1002106. DOI: http://dx.doi.org/10.1371/journal.pbio.1002106.
Methods in full. Nature. 2007 Feb 15;445(7129):684. DOI: http://dx.doi.org/10.1038/445684a.
Simera I, Moher D, Hirst A, Hoey J, Schulz KF, Altman DG. Transparent and accurate reporting increases reliability, utility, and impact of your research: reporting guidelines and the EQUATOR Network. BMC Med. 2010 Apr 26;8:24. DOI: http://dx.doi.org/10.1186/1741-7015-8-24.
Hartung DM, Zarin DA, Guise JM, McDonagh M, Paynter R, Helfand M. Reporting discrepancies between the ClinicalTrials.gov results database and peer-reviewed publications. Ann Intern Med. 2014 Apr 1;160(7):477–83. DOI: http://dx.doi.org/10.7326/m13-0480.
Fusenig NE, Capes-Davis A, Bianchini F, Sundell S, Lichter P. The need for a worldwide consensus for cell line authentication: experience implementing a mandatory requirement at the International Journal of Cancer. PLOS Biol. 2017 Apr;15(4):e2001438. DOI: http://dx.doi.org/10.1371/journal.pbio.2001438.
Trafimow D. Editorial. Basic Appl Soc Psychol. 2014 Jan;36(1):1–2. DOI: http://dx.doi.org/10.1080/01973533.2014.865505.
Aschwanden C. Failure is moving science forward [Internet]. FiveThirtyEight; 24 Mar 2016 [cited 24 May 2017]. <https://fivethirtyeight.com/features/failure-is-moving-science-forward/>.
Baker M. 1,500 scientists lift the lid on reproducibility. Nature. 2016 May 26;533(7604):452–4. DOI: http://dx.doi.org/10.1038/533452a.
Harris RF. Rigor mortis: how sloppy science creates worthless cures, crushes hope, and wastes billions. New York, NY: Basic Books; 2017.
Yong E. How the GOP could use science’s reform movement against it. Atlantic [Internet]. Apr 2017 [cited 24 May 2017]. <https://www.theatlantic.com/science/archive/2017/04/reproducibility-science-open-judoflip/521952/>.
Ioannidis JPA. How to make more published research true. PLOS Med. 2014 Oct;11(10):e1001747. DOI: http://dx.doi.org/10.1371/journal.pmed.1001747.
Center for Open Science. Open science framework [Internet]. Charlottesville, VA: The Center, University of Virginia; 2017 [cited 24 May 2017]. <https://osf.io>.
Zhao S, Lackey MJ, Schmick D, Rethlefsen ML. Research reproducibility 2016: home [Internet]. Salt Lake City, UT: Spencer S. Eccles Health Sciences Library, University of Utah; 5 Jan 2017 [cited 24 May 2017]. <http://campusguides.lib.utah.edu/utahrr16>.
Zhao S, Lackey MJ, Rethlefsen ML. Reproducibility of research: get started [Internet]. Salt Lake City, UT: Spencer S. Eccles Health Sciences Library, University of Utah; 29 Apr 2017 [cited 24 May 2017]. <http://campusguides.lib.utah.edu/reproducibility/start>.
Spencer S. Eccles Health Sciences Library. Research reproducibility conference [Internet]. Salt Lake City, UT: The Library, University of Utah; 2016 [cited 24 May 2017]. <http://library.med.utah.edu/e-channel/portfolio/research-reproducibility-conference/>.
Lackey MJ, Rethlefsen ML. How to make science more reliable? tips for systematic reviews. In: Kiefer J. The Scope [Internet]. Salt Lake City, UT: University of Utah Health Sciences; 13 Jul 2016 [cited 12 Nov 2017]. <https://healthcare.utah.edu/the-scope/shows.php?shows=0_49l4nen3>.
Schmick D. How to make science more reliable: tips for managing data. In: Kiefer J. The Scope [Internet]. Salt Lake City, UT: University of Utah Health Sciences; 1 Jul 2016 [cited 12 Nov 2017]. <https://healthcare.utah.edu/the-scope/shows.php?shows=0_dy2ih9ht>.
Librarian’s role in reproducibility of research [MLA ’17 symposium] [Internet]. Springshare; 22 May 2017 [cited 24 May 2017]. <http://mlasymposium.libguides.com/2017>.
Department of Biomedical Informatics. DeCART: data science for the health sciences [Internet]. Salt Lake City, UT: University of Utah; 2017 [cited 13 Jun 2017]. <https://datascience4health.bmi.utah.edu/>.
National Academies of Sciences Engineering and Medicine. Fostering integrity in research. Washington, DC: National Academies Press; 2017. DOI: https://dx.doi.org/10.17226/21896.
Kirtley S. Increasing value and reducing waste in biomedical research: librarians are listening and are part of the answer. Lancet. 2016 Apr 16;387(10028):1601. DOI: http://dx.doi.org/10.1016/S0140-6736(16)30241-0.
