Xan Goodman, AHIP, MLIS, John Watts, MLS, Rogelio Arenas, MA, Rachelle Weigel, BA, Tony Terrell, PhD
doi: http://dx.doi.org/10.5195/jmla.2018.400
Received August 2016: Accepted July 2017
ABSTRACT
Objective
This article describes the collection and analysis of annotated bibliographies created by first-year health sciences students to support their final poster projects. The authors examined the students’ abilities to select relevant and authoritative sources, summarize the content of those sources, and correctly cite those sources.
Methods
We collected images of 1,253 posters, of which 120 were sampled for analysis, and scored the posters using a 4-point rubric to evaluate the students’ information literacy skills.
Results
We found that 52% of students were proficient at selecting relevant sources that directly contributed to the themes, topics, or debates presented in their final poster projects, and 64% of students did well with selecting authoritative peer-reviewed scholarly sources related to their topics. However, 45% of students showed difficulty in correctly applying American Psychological Association (APA) citation style.
Conclusion
Our findings demonstrate a need for instructors and librarians to provide strategies for reading and comprehending scholarly articles in addition to properly using APA citation style.
Information literacy instruction provides an essential foundation for students who are contemplating careers in the health sciences. Throughout their undergraduate experiences, health sciences students engage with health sciences and scientific literature in increasingly sophisticated ways. Evidence shows that first-year students entering university environments struggle with the content of scholarly and scientific literature [1–3]. These students also lack sophisticated research skills to help them navigate resources that are available through university libraries [1–5]. Thus, first-year health sciences students must build foundational skills to help them develop critical thinking and lifelong learning skills that will be valuable throughout their college careers and beyond [4, 5].
This study was designed to examine the abilities of first-year undergraduate health sciences students to select relevant and authoritative sources, follow and use American Psychological Association (APA) citation style for sources, and summarize selected sources to support their research topics for a final poster project. By characterizing the information literacy skills of first-year health sciences students, the authors aimed to guide librarians’ efforts to improve their information literacy instruction.
This study focused on students enrolled in “Inquiry and Issues in Health Sciences” (HSC 100), a required first-year seminar for students who intend to major in the health sciences at the University of Nevada, Las Vegas. Students in this course often select a major in nursing, pre-dental, pre-medical, or other health sciences majors. HSC 100 is a high-enrollment gateway course that meets twice a week in a large auditorium-style classroom. The culminating assignment for this course is a research poster exploring a health sciences topic of the student’s choice. Each student individually develops a research topic, seeks scholarly sources addressing the topic, assembles the required elements of the project on a standard tri-fold poster board, and presents the research to the class during a large poster session on the final day of class.
To prepare students for finding the required research articles, librarians provide an hour-long instruction session with a lecture on locating scholarly research articles using library databases and Google Scholar. Librarians also discuss various characteristics of a scholarly article, including its purpose, general structure, and intended audience. Students are pointed to an online research guide, which includes links to recommended databases, citation information, and the lecture presentation materials from the instruction session.
This study was deemed exempt from review by the Institutional Review Board of the University of Nevada, Las Vegas (protocol #1311-4646M). The team—which included a health sciences librarian, teaching and learning librarian, and library technician—attended poster sessions each semester from fall 2013 to spring 2015. Each team member was assigned an area of the poster session from which to collect data. Using iPads provided by the library, team members took photos of each poster in their designated areas. We agreed that photos should be taken of each entry of the annotated bibliography as well one holistic photo of the poster to provide context for the annotations. After each poster session, we transferred photographs from the iPads to a password-protected file drive.
A graduate student assistant was hired to de-identify the images, removing all student information using Adobe Photoshop software. From the 6 semesters of poster presentations, we obtained images from 1,253 posters. A total of 413 posters had legible images and were analyzed using the sampling methods described below.
Based on Oakleaf’s recommendations to avoid rater fatigue [6], we randomly selected approximately one-third of the legible subset of posters to form a representative sample. As we expected that each rater could carefully score 30 posters in 1 week, we scored one-third of the sample to prevent rater exhaustion and still provide a robust sample for scoring. Randomization was performed using the RANDOM.ORG integer generator by creating a series of integers from 1 to 413 with no repeats. Each integer corresponded to a folder number that represented a single poster. Ten randomly sampled posters were used for the rubric norming process, and 120 randomly sampled posters underwent formal scoring.
In alignment with Mertler’s model for rubric development [7], we began developing the HSC 100 information literacy rubric by defining the learning outcomes of the assignment. The assignment required students to: (1) locate peer-reviewed research articles; (2) evaluate the sources for relevance to their topics; (3) summarize the sources, including their research methods and conclusion; and (4) cite the sources using APA style. To design the rubric, we reviewed guidelines for information literacy rubrics [6, 8] and settled on four performance levels (1–4) representing a range of student abilities to meet each of the four rubric criteria (Table 1).
Table 1 Information literacy rubric
| Performance level | Competent | Developing | Beginning | Indeterminate or poor |
| Rating | 4 | 3 | 2 | 1 |
| Relevance | Clearly and specifically defines topic. Selected sources contribute directly to topic/argument/debate. | General topic is understood but may demonstrate a need to specify. Topic is covered generally, as opposed to in depth. | Sources peripherally touch on topic or go on tangents from main idea. Sources tease at topic without ever addressing it or answering a question. | Sources are of little to no significance or relevance to the topic. |
| Authority | Choice of material demonstrates a discerning eye for scholarly and non-scholarly sources. | Selection demonstrates a significant understanding of scholarly material but may include some questionable sources (e.g., op-ed, secondary source, magazine, WebMD). | Selections demonstrate significant obstacles in discerning appropriate sources. Sources may be too similar or ambiguous as to whether they come from scholarly sources. | Selections come from spurious sources (e.g., Wikipedia, blogs, forum posts). Sources use conjecture or anecdotal data. |
| Summary | Student demonstrates an ability to determine the strengths and weaknesses of sources or exactly what it answers or does not answer. | Student demonstrates competence in evaluating material but may overreach conclusions from the information. | Student demonstrates significant lack in comprehending the meaning of findings or how to apply findings correctly. | Uses information to reinforce unsubstantiated point. Pushes subjective or impartial narrative while citing information. |
| Citation | Correctly follows American Psychological Association (APA) format on all citations. Cites all sources. Knows information that is appropriate to cite (common knowledge vs. attribution). | Consistently uses APA format with little to no errors. Demonstrates competence but a need for improvement in discerning appropriate places to cite. | Citations show an inability to adhere to APA or a different citation format altogether. Student demonstrates significant lack of understanding as to where to place citations. | Has haphazardly selected or has no citations. Shows inconsistent or no use of APA. |
Before scoring posters using the rubric, we met to discuss the application of the rubric to student work and to determine inter-rater reliability. We practiced rating 3 posters together and achieved consensus on the guidelines for applying the rubric; these 3 posters were excluded from further analysis. To measure inter-rater reliability, we randomly selected 10 posters for each rater to score independently. We achieved a Krippendorff’s alpha [9] value of 0.853 among all 4 raters, which was sufficient to allow independent scoring [10].
Each rater scored 30 posters within 1 week to prevent bias in scoring due to fatigue. Scores on each rubric criterion were summed to provide an overall score for each poster that ranged from 4 to 16. Scores were analyzed using SPSS software.
We scored a total of 120 posters. Of the 4 rubric criteria, students scored lowest in the areas of summary and citation and highest in the areas of relevance and authority (Table 2).
| Performance level | Competent | Developing | Beginning | Indeterminate or poor | ||||||
| Rating | 4 | 3 | 2 | 1 | Mean | |||||
| n | (%) | n | (%) | n | (%) | n | (%) | n | (SD) | |
| Relevance | 62 | (51.7%) | 33 | (27.5%) | 22 | (18.3%) | 3 | (2.5%) | 3.2 | (0.85) |
| Authority | 79 | (65.8%) | 25 | (20.8%) | 13 | (10.8%) | 3 | (2.5%) | 3.5 | (0.78) |
| Summary | 41 | (34.2%) | 42 | (35.0%) | 33 | (27.5%) | 4 | (3.3%) | 3.0 | (0.86) |
| Citation | 38 | (31.7%) | 28 | (23.3%) | 48 | (40.0%) | 6 | (5.0%) | 2.8 | (0.94) |
Our results suggested that first-year students, although new to using scholarly sources, had a good grasp on how to locate scholarly articles for an annotated bibliography assignment. Approximately half of the students were competent in their abilities to locate scholarly articles that were relevant to the topics they covered in their final posters. These findings demonstrate that these first-year students knew how to navigate library databases, Google Scholar, and possibly other Internet search engines to locate scholarly articles, which is consistent with previous studies [11–13]. Students also performed well in selecting scholarly peer-reviewed sources that were written by authors with expertise or authority related to their health topics, which is consistent with previous studies [12, 13].
Over half of the students demonstrated an early stage of developing proficiency with summarizing articles, suggesting that summarizing an article including all the required elements for an assignment is a complex skill, which is in agreement with previous studies [14, 15]. A final summary should be well written, be concise, and critically connect the scholarly articles to the student’s research topic. However, our students showed varying abilities in summarizing articles, perhaps because they were novices in both reading and summarizing scholarly articles. Therefore, librarians can provide instruction on how to identify parts of an article to help improve students’ summarization skills. Based on these findings, our library instruction team altered HSC 100 instruction to allow students more in-class time to practice reading scholarly articles, recognizing sections of an article, understanding the meaning of each section, and summarizing articles.
We found that first-year health sciences students struggled the most with citing articles, which is consistent with previous studies [16–19]. It is important for students to credit others’ ideas to avoid acts of intentional or unintentional plagiarism. Although students had access to an online guide with links to guidance on how to properly cite sources using APA style, we found that students needed more help in understanding how to cite scholarly articles. Citation is a complex skill that requires students to understand the parts of a citation to render correct output. Automatic citation generators might contribute to students’ inability to understand how to correctly apply a particular style to a citation [16, 18]. Librarians can help improve student performance by providing additional consultation appointments or tutorials to increase student proficiency with APA citation style.
In future research on undergraduate information literacy skills, researchers could conduct a longitudinal study to examine annotated bibliographies created by health sciences students at critical points throughout their undergraduate education, tracking individual students from a first-year seminar course through their capstone or culminating experiences. Such a research design could include refined rubrics designed to assess article summary, citation, and selection skills, with a targeted focus on students’ abilities to articulate the strengths and weaknesses of scholarly articles. It may also be valuable to perform a pre-assessment of student skills to draw stronger connections between the role of information literacy instruction and student outcomes.
1 Head AJ. Learning the ropes: how freshmen conduct course research once they enter college [Internet]. Seattle, WA: Information School, University of Washington; 2013 [cited 5 Dec 2013]. <http://www.projectinfolit.org/uploads/2/7/5/4/27541717/pil_2013_freshmenstudy_fullreportv2.pdf>.
2 Ivanitskaya LV, Hanisko KA, Garrison JA, Janson SJ, Vibbert D. Developing health information literacy: a needs analysis from the perspective of preprofessional health students. J Med Libr Assoc. 2012 Oct;100(4):277–83. DOI: http://dx.doi.org/10.3163/1536-5050.100.4.009.
3 Rangachari PK, Rangachari U. Information literacy in an inquiry course for first-year science undergraduates: a simplified 3C approach. Adv Physiol Educ. 2007 Jun 1;31(2):176–9. DOI: http://dx.doi.org/10.1152/advan.00092.2006.
4 Benedict J. Using popular-press nutrition books to develop critical reading skills of first-year college students. J Nutr Educ Behav. 2013 Mar;45(2):188–90.
5 Salisbury F, Karasmanis S. Are they ready? exploring student information literacy skills in the transition from secondary to tertiary education. Aust Acad Res Libr. 2011;42(1):43–58. DOI: http://dx.doi.org/10.1080/00048623.2011.10722203.
6 Oakleaf MJ. Using rubrics to collect evidence for decision-making: what do librarians need to learn? Evid Based Libr Inf Pract. 2007;2(3):27–42. DOI: http://dx.doi.org/10.18438/B8WS3W.
7 Mertler CA. Designing scoring rubrics for your classroom. Practical Assess Res Eval. 2001 Dec;7(25). (Available from: <http://pareonline.net/getvn.asp?v=7&n=25.>. [cited 20 Nov 2017].)
8 Rinto EE. Developing and applying an information literacy rubric to student annotated bibliographies. Evid Based Libr Inf Pract. 2013;8(3):5–18. DOI: http://dx.doi.org/10.18438/B8559F.
9 Krippendorff K. Reliability in content analysis: some common misconceptions and recommendations. Hum Commun Res. 2004 Jul;30(3):411–33.
10 Stemler SE. A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assess Res Eval. 2004 Mar;9(4). (Available from: <http://PAREonline.net/getvn.asp?v=9&n=4>. [cited 20 Nov 2017].)
11 Head AJ. Finding context: what today’s college students say about conducting research in the Digital Age [Internet]. Seattle, WA: Information School, University of Washington; 2009 [cited 4 Feb 2009]. <http://www.projectinfolit.org/uploads/2/7/5/4/27541717/2009_final_report.pdf>.
12 Head AJ, Eisenberg MB. Lessons learned: how college students seek information in the Digital Age [Internet]. Seattle, WA: Information School, University of Washington; 2009 [cited 1 Dec 2009]. <http://www.projectinfolit.org/uploads/2/7/5/4/27541717/pil_fall2009_finalv_yr1_12_2009v2.pdf>.
13 Borelli S, Johnson CM. Information evaluation instruction: a three term project with a first year experience course. Commun Inf Lit. 2012;6(2):173–90.
14 Mittermeyer D. Incoming first year undergraduate students: how information literate are they? Educ Inf. 2005;23(4):203–32, DOI: http://dx.doi.org/10.3233/EFI-2005-23401.
15 Crawford J, Irving C. Information literacy: the link between secondary and tertiary education project and its wider implications. J Libr Inf Sci. 2007;39(17) 17–26.
16 Ludovico C, Wittig C. A universe of information, one citation at a time: how students engage with scholarly sources. J Libr Inf Serv Distance Learn. 2015;9(1):30–9. DOI: http://dx.doi.org/10.1080/1533290X.2014.946343.
17 Lwong ET. Mapping information literacy outcomes and learning experiences of health sciences undergraduate students. Partnership Can J Libr Inf Pract Res. 2014;9(1):1–17.
18 Nash S, Wilson MC. Value-added bibliographic instruction: teaching students to find the right citations. Ref Serv Rev. 1991;19(1) 87–92.
19 Park S, Mardis LA, Ury CJ. I’ve lost my identity – oh, there it is…in a style manual teaching citation styles and academic honesty. Ref Serv Rev. 2011;39(1) 42–57.
Xan Goodman, MLIS, AHIP, xan.goodman@unlv.edu, Health Sciences Librarian and Assistant Professor, University of Nevada, Las Vegas, 4505 South Maryland Parkway, Box 457014, Las Vegas, NV 89154-7014
John Watts, MLS, john.watts@unlv.edu, Head of Knowledge Production and Associate Professor, University of Nevada, Las Vegas, 4505 South Maryland Parkway, Box 457034, Las Vegas, NV 89154-7014
Rogelio Arenas, MA, arenasr@unlv.nevada.edu, PhD Candidate, Anthropology Department and University Libraries, and Information Literacy Fellow, University of Nevada, Las Vegas, 4505 South Maryland Parkway, Box 457014, Las Vegas, NV 89154-7014
Rachelle Weigel, BA, rachelle.weigel@unlv.edu, Library Technician II, University of Nevada, Las Vegas, 4505 South Maryland Parkway, Box 457014, Las Vegas, NV 89154-7014
Tony Terrell, PhD, tony.terrell@unlv.edu, Director of Advising and Recruitment, Honors College, University of Nevada, Las Vegas, 4505 South Maryland Parkway, Box 457014, Las Vegas, NV 89154-7014
*Based on a presentation at MLA ‘16, the 116th Annual Meeting of the Medical Library Association; Toronto, Canada; May 16, 2016. ( Return to Text )
Articles in this journal are licensed under a Creative Commons Attribution 4.0 International License.
This journal is published by the University Library System of the University of Pittsburgh as part of its D-Scribe Digital Publishing Program and is cosponsored by the University of Pittsburgh Press.
Journal of the Medical Library Association, VOLUME 106, NUMBER 1, January 2018