Rigor and reproducibility instruction in academic medical libraries
DOI:
https://doi.org/10.5195/jmla.2022.1443Keywords:
reproducibility, replicability, library instruction, education, rigor and reproducibility, NIH requirementsAbstract
Background: Concerns over scientific reproducibility have grown in recent years, leading the National Institutes of Health (NIH) to require researchers to address these issues in research grant applications. Starting in 2020, training grants were required to provide a plan for educating trainees in rigor and reproducibility. Academic medical centers have responded with different solutions to fill this educational need. As experienced instructors with expertise in topics relating to reproducibility, librarians can play a prominent role in providing trainings, classes, and events to educate investigators and trainees, and bolstering reproducibility in their communities.
Case Presentations: This special report summarizes efforts at five institutions to provide education in reproducibility to biomedical and life sciences researchers. Our goal is to expand awareness of the range of approaches in providing reproducibility services in libraries.
Conclusions: Reproducibility education by medical librarians can take many forms. These specific programs in reproducibility education build upon libraries’ existing collaborations, with funder mandates providing a major impetus. Collaborator needs shaped the exact type of educational or other reproducibility support and combined with each library’s strengths to yield a diversity of offerings based on capacity and interest. As demand for and complexity of reproducibility education increases due to new institutional and funder mandates, reproducibility education will merit special attention.
References
Baker M. 1,500 scientists lift the lid on reproducibility. Nature. 2016 May 1;533(7604):452–4.
Trouble at the lab. The Economist [Internet]. 2013 Oct 18 [cited 2021 Nov 11]. https://www.economist.com/briefing/2013/10/18/trouble-at-the-lab.
Carroll AE. Science needs a solution for the temptation of positive results. The New York Times [Internet]. 2017 May 29 [cited 2021 Nov 11]. https://www.nytimes.com/2017/05/29/upshot/science-needs-a-solution-for-the-temptation-of-positive-results.html.
Vedantam S, Penman M. When great minds think unalike: inside science’s “replication crisis.” NPR [Internet]. 2016 May 24 [cited 2021 Nov 11].
Goodman SN, Fanelli D, Ioannidis JPA. What does research reproducibility mean? Sci Transl Med. 2016 Jun 1;8(341):341ps12-341ps12.
Plesser HE. Reproducibility vs. replicability: a brief history of a confused terminology. Front Neuroinformatics. 2018;11:76.
Patil P, Peng RD, Leek JT. A statistical definition for reproducibility and replicability [Internet]. 2016 Jul [cited 2021 Nov 15] p. 066803. https://www.biorxiv.org/content/10.1101/066803v1.
Kenett RS, Shmueli G. Clarifying the terminology that describes scientific reproducibility. Nat Methods. 2015 Aug;12(8):699.
NOT-OD-15-103: enhancing reproducibility through rigor and transparency [Internet]. [cited 2022 Jan 11]. https://grants.nih.gov/grants/guide/notice-files/not-od-15-103.html.
NOT-OD-20-033: NIH and AHRQ announce upcoming changes to policies, instructions and forms for research training grant, fellowship, and career development award applications [Internet]. [cited 2021 Nov 15]. Available from: https://grants.nih.gov/grants/guide/notice-files/NOT-OD-20-033.html.
Download: reproducibility and replicability in science [Internet]. The National Academies Press [cited 2021 Nov 15]. https://www.nap.edu/download/25303.
Barba LA. Terminologies for reproducible research. arXiv180203311 Cs. 2018 Feb 9 [cited 2021 Nov 15]. http://arxiv.org/abs/1802.03311.
Guidance: rigor and reproducibility in grant applications [Internet]. [cited 2021 Nov 16]. https://grants.nih.gov/policy/reproducibility/guidance.htm.
Roundtable on data science postsecondary education: a compilation of meeting highlights. Washington, DC.: National Academies Press; 2020.
Lapane KL, Dube CE. Rigor and reproducibility training for first year medical students in research pathways. Clin Transl Sci. 2021 May;14(3):1026–36.
Millman KJ, Brett M, Barnowski R, Poline J-B. Teaching computational reproducibility for neuroimaging. Front Neurosci. 2018;12:727.
Willig J, Croker J, Wallace B, Dempsey D, Wallace B, Redden D. 2440 Teaching rigor, reproducibility, and transparency using gamification. J Clin Transl Sci. 2018 Jun;2(S1):61–61.
OSF [Internet]. [cited 2021 Oct 5]. https://osf.io/meetings/ResearchReproducibility2020.
Chopik WJ, Bremner RH, Defever AM, Keller VN. How (and whether) to teach undergraduates about the replication crisis in psychological science. Teach Psychol. 2018 Apr 1;45(2):158–63.
Frank MC, Saxe R. Teaching replication. Perspect Psychol Sci. 2012 Nov 1;7(6):600–4.
Pownall M. Pre-registration in the undergraduate dissertation: a critical discussion. Psychol Teach Rev. 2020 Jan;26(1):71–6.
FORRT. Introducing a Framework for Open and Reproducible Research Training (FORRT) [Internet]. OSF Preprints. 2019 [cited 2021 Jan 25]. https://osf.io/bnh7p/.
Steeves V. Reproducibility librarianship. Collab Librariansh [Internet]. 2017 Jul 11;9(2). https://digitalcommons.du.edu/collaborativelibrarianship/vol9/iss2/4.
Rethlefsen ML, Lackey MJ, Zhao S. Building capacity to encourage research reproducibility and #MakeResearchTrue. J Med Libr Assoc. 2018 Jan;106(1):113–9.
Sayre F, Riegelman A. The reproducibility crisis and academic libraries. Coll Res Libr [Internet]. 2018 Jan [cited 2021 Jan 19];79(1). http://crl.acrl.org/index.php/crl/article/view/16846.
Oliver JC, Kollen C, Hickson B, Rios F. Data science support at the academic library. J Libr Adm. 2019 Apr 3;59(3):241–57.
NYU Langone Health_Institution_7-12-21.docx [Internet]. [cited 2021 Nov 16]. https://central.nyumc.org/research/site/grants-lib/Boilerplate Text/NYU Langone Health.
Bowen RS. Understanding by design [Internet]. Center for Teaching [cited 2021 Nov 16]. https://cft.vanderbilt.edu/guides-sub-pages/understanding-by-design/.
LaPolla F, Hao Y, Surkis A, Bakker C, Mentnech T, Exner N. Rigor and reproducibility instruction in academic medical libraries appendix [Internet]. Zenodo; 2021. https://zenodo.org/communities/2021rigor_reproducibility_libraries_paper/?page=1&size=20.
Knowles MS. The adult learner : the definitive classic in adult education and human resource development. 6th edition. Routledge, 2005.
LaPolla FWZ, Contaxis N, Surkis A. Piloting a long-term evaluation of library data workshops. J Med Libr Assoc. 2021 Oct 5;109(3):490–6.
Heidi G. GRRR – Grand Rounds for Research Reproducibility [Internet]. [cited 2021 Nov 16]. https://library.med.utah.edu/blog/eccles/2017/09/29/grrr-grand-rounds-for-research-reproducibility/.
Research Reproducibility 2018: Conference [Internet]. [cited 2021 Nov 16]. https://library.med.utah.edu/blog/eccles/event/research-reproducibility-2018-conference/.
Steeves V. UUtah 2018 Reproducibility Immersive Course [Internet]. [cited 2021 Nov 16]. https://vickysteeves.gitlab.io/2018-uutah-repro/.
Vicky Steeves / 2018-uutah-repro [Internet]. GitLab. [cited 2021 Nov 16]. https://gitlab.com/vickysteeves/2018-uutah-repro.
McCormack WT, Garvan CW. Team-based learning instruction for responsible conduct of research positively impacts ethical decision-making. Account Res. 2014 Jan 2;21(1):34–49.
Many analysts, one data set: making transparent how variations in analytic choices affect results - R. Silberzahn, E. L. Uhlmann, D. P. Martin, P. Anselmi, F. Aust, E. Awtrey, Š. Bahník, F. Bai, C. Bannard, E. Bonnier, R. Carlsson, F. Cheung, G. Christensen, R. Clay, M. A. Craig, A. Dalla Rosa, L. Dam, M. H. Evans, I. Flores Cervantes, N. Fong, M. Gamez-Djokic, A. Glenz, S. Gordon-McKeon, T. J. Heaton, K. Hederos, M. Heene, A. J. Hofelich Mohr, F. Högden, K. Hui, M. Johannesson, J. Kalodimos, E. Kaszubowski, D. M. Kennedy, R. Lei, T. A. Lindsay, S. Liverani, C. R. Madan, D. Molden, E. Molleman, R. D. Morey, L. B. Mulder, B. R. Nijstad, N. G. Pope, B. Pope, J. M. Prenoveau, F. Rink, E. Robusto, H. Roderique, A. Sandberg, E. Schlüter, F. D. Schönbrodt, M. F. Sherman, S. A. Sommer, K. Sotak, S. Spain, C. Spörlein, T. Stafford, L. Stefanutti, S. Tauber, J. Ullrich, M. Vianello, E.-J. Wagenmakers, M. Witkowiak, S. Yoon, B. A. Nosek, 2018 [Internet]. [cited 2021 Nov 16]. https://journals-sagepub-com.ezproxy.med.nyu.edu/doi/10.1177/2515245917747646.
2021 Summer Seminar Series – RCR [Internet]. [cited 2021 Nov 16]. https://research.ufl.edu/rcr/rcr-training/2021-summer-seminar-series/.
Ye H. Data organization in spreadsheets [Internet]. 2020 Aug 19 [cited 2021 Nov 16]. https://zenodo.org/record/3991311.
Broman KW, Woo KH. Data organization in spreadsheets. Am Stat. 2018 Jan 2;72(1):2–10.
Marwick B, Boettiger C, Mullen L. Packaging Data analytical work reproducibly using R (and friends). Am Stat. 2018 Jan 2;72(1):80–8.
Briney K, Coates H, Goben A. Foundational practices of research data management. Res Ideas Outcomes. 2020 Jul 27;6:e56508.
Data organization in spreadsheets [Internet]. [cited 2021 Nov 16]. https://uf-repro.github.io/data-organization/.
The Carpentries [Internet]. The Carpentries. [cited 2022 Jan 12]. https://carpentries.org/index.html.
Software Carpentry [Internet]. Software Carpentry. [cited 2022 Jan 12]. http://software-carpentry.org//index.html.
ReproducibiliTea [Internet]. ReproducibiliTea. [cited 2021 Nov 16]. https://reproducibilitea.org/.
Promoting an open research culture [Internet]. [cited 2021 Nov 16]. https://www.science.org/doi/full/10.1126/science.aab2374.
Reporting guidelines [Internet]. The EQUATOR Network. [cited 2021 Nov 16]. https://www.equator-network.org/reporting-guidelines/.
NIH. NOT-OD-21-013: Final NIH Policy for Data Management and Sharing [Internet]. [cited 2021 Sep 28]. https://grants.nih.gov/grants/guide/notice-files/NOT-OD-21-013.html.
NLM. NLM Reproducibility Workshop [Internet]. [cited 2022 Jan 12]. https://nlm-repro.github.io/.
Data management, curation & reproducibility [Internet]. [cited 2022 Jan 12]. http://ischool.illinois.edu/degrees-programs/courses/is477.
Reproducibility [Internet]. School of Information Sciences. [cited 2022 Jan 12]. http://ischool.illinois.edu/research/areas/reproducibility.
Giustini D, Read KB, Deardorff A, Federer L, Rethlefsen ML. Health sciences librarians’ engagement in open science: a scoping review. J Med Libr Assoc. 109(4):540–60.
Initiative CZS. Advancing diversity and inclusion in scientific open source [Internet]. Medium. 2021 [cited 2022 Jan 24]. https://cziscience.medium.com/advancing-diversity-and-inclusion-in-scientific-open-source-eaabe6a5488b.
Benjamin R. Race after technology : abolitionist tools for the new Jim code. Cambridge, UK ; Medford, MA: Polity; 2019.
Haibe-Kains B, Adam GA, Hosny A, Khodakarami F, Waldron L, Wang B, McIntosh C, Goldenberg A, Kundaje A, Greene CS, Broderick T, Hoffman MM, Leek JT, Korthauer K, Huber W, Brazma A, Pineau J, Tibshirani R, Hastie T, Ioannidis JPA, Quackenbush J, Aerts HJWL. Transparency and reproducibility in artificial intelligence. Nature. 2020 Oct;586(7829):E14–6.
Downloads
Published
Versions
- 2023-10-02 (2)
- 2022-12-08 (1)
Issue
Section
License
Copyright (c) 2022 Fred Willie Zametkin LaPolla
This work is licensed under a Creative Commons Attribution 4.0 International License.
