Assessment of knowledge and skills in information literacy instruction for rehabilitation sciences students: a scoping review

Objective This scoping review investigates how knowledge and skills are assessed in the information literacy (IL) instruction for students in physical therapy, occupational therapy, or speech-language pathology, regardless of whether the instruction was given by a librarian. The objectives were to discover what assessment measures were used, determine whether these assessment methods were tested for reliability and validity, and provide librarians with guidance on assessment methods to use in their instruction in evidence-based practice contexts. Methods A scoping review methodology was used. A systematic search strategy was run in Ovid MEDLINE and adapted for CINAHL; EMBASE; Education Resources Information Center (ERIC) (EBSCO); Library and Information Science Abstracts (LISA); Library, Information Science & Technology Abstracts (LISTA); and Proquest Theses and Dissertations from 1990 to January 16, 2017. Forty articles were included for data extraction. Results Three major themes emerged: types of measures used, type and context of librarian involvement, and skills and outcomes described. Thirty-four measures of attitude and thirty-seven measures of performance were identified. Course products were the most commonly used type of performance measure. Librarians were involved in almost half the studies, most frequently as instructor, but also as author or assessor. Information literacy skills such as question formulation and database searching were described in studies that did not involve a librarian. Conclusion Librarians involved in instructional assessment can use rubrics such as the Valid Assessment of Learning in Undergraduate Education (VALUE) when grading assignments to improve the measurement of knowledge and skills in course-integrated IL instruction. The Adapted Fresno Test could be modified to better suit the real-life application of IL knowledge and skills.


INTRODUCTION
Since the term evidence-based medicine was first coined in the mid-1990s [1], the integration of evidence-based practice (EBP) in health professional education has continued to grow. While EBP has its roots in medicine, it is now established in the required competencies of allied health professions such as physical therapy, occupational therapy, and speech-language pathology [2][3][4]. Along with this growth has come the establishment of information literacy (IL) instruction to support EBP [5][6][7].
behaviors [11][12][13] and can be applied to IL instruction. Knowledge in the context of EBP is "an awareness of the sources of evidence and an understanding of the evidence itself" [13]. Skills include the application of this knowledge [12]. An IL example would be the knowledge of how a database functions and the skill of how to conduct searches in this database. Attitudes refer to the recognition of the need to use certain knowledge or skills while working on an academic project or in clinical practice [11]. This differs slightly from attitude in the context of measures of attitude, which are selfreported perceived learning [14]. An EBP behavior is when someone "applies the knowledge and skills to solve [an] issue in practice" [11]. These categories help instructors align their teaching and assessment methods with learning outcomes.
IL knowledge and skills are necessary at every step of the EBP cycle. The Association of College and Research Libraries Information Literacy Competency Standards of "Determine, Access, Evaluate, Apply, and Ethics" correlate with the five EBP steps of "Formulate, Search, Appraise, Incorporate, Evaluate" [5] and provide a structure for teaching the IL knowledge and skills needed for EBP [6,7]. Librarians are experts in such knowledge and skills as question formulation (patient-interventioncomparison-outcome [PICO] format and other research question formats), database searching, resource selection, article selection, and article appraisal (critical appraisal), and can be the model for future clinicians who will use such knowledge and skills in practice [15].
As recommended by good IL practice, librarians teaching in EBP contexts "routinely state specific instructional goals, explain rationales for teaching methods, identify ways they expect students to be impacted by IL instruction, and detail the effect of library instruction…on student success" [16]. This last point, examining the effect of instruction on student success, can be the most difficult to implement, though good assessment is important to help instructors understand if students are learning what the instructor intends them to learn [17].
Studies that looked at how librarians included assessment in their practice revealed that the assessment methods were primarily indirect assessments or assessment of attitude, which was defined as "self-reporting of perceived skills, learning, behaviors, or attitudes" [14]. Warner found in her review of the library assessment literature that there were "serious limitations to the instruments most typically used, primarily that they have failed to adequately assess student learning" [18]. In interviewing eighteen librarians, Cull observed that "most librarians…did not objectively assess student learning at all but only solicited student reactions to instructional content, methods, and instructor effectiveness, typically using brief end-of-class student evaluation forms" [19]. Two other studies also noted that self-reported attitude assessments were the most frequently cited form of evaluation in library instruction [14,20].
While assessments of attitude are important for understanding how students feel about their own learning and about the instructor's teaching, students are not good evaluators of their own skill levels [21][22][23]. It is important, therefore, to assess their learning with measures of performance that document knowledge, skills, or behaviors based on actual student work [14]. Another aspect to consider is whether an assessment measure has been tested for reliability and validity. Validated assessment measures increase the accuracy and the reliability of results, facilitate comparison across different studies [24], and allow a richer understanding of the effectiveness of the instruction.
The importance of well-developed assessment measures is clear, but applying this knowledge can be a challenge, since librarians and other EBP instructors may not know which assessment measures would best align with their learning outcomes and instructional methods. In an effort to assist instructors with this choice, literature reviews in library and information science and health sciences education have summarized the assessment measures being used in IL and EBP learning contexts, but not all educational contexts and measure types have been well addressed. These reviews leave major gaps in the literature.
The authors found two gaps in the library and information science literature. First, all of the reviews conducted by librarians looked only at studies where librarians were involved in the design or delivery of instruction and assessment [14,[25][26][27][28][29]. Given the alignment of IL instruction objectives and EBP instruction objectives [5][6][7], it is valuable to consider studies where instruction and assessment of IL outcomes took place without explicit librarian involvement, as these studies might provide jmla.mlanet.org 106 (1) January 2018 Journal of the Medical Library Association librarians with examples of assessment methods used in health sciences education and novel opportunities for assessment.
Second, none of the reviews that librarians conducted covered the assessment of rehabilitation sciences students. The earliest review was conducted in 2006, before the Adapted Fresno Test for occupational therapy was created and does not have much content from rehabilitation sciences [25]. Two more recent reviews provided excellent overviews of the assessments that librarians were conducting but were broader in subject coverage, were narrower in date coverage, and did not provide specific recommendations for assessment measures in EBP instruction [14,26]. Three other recent reviews specifically focused on instruction in health sciences but still left a gap for assessment in rehabilitation sciences. One included teaching methods but not assessment measures [27]; the other two recommended the Fresno Test, but no or few studies outside of medicine were included [28,29].
We found one major gap in the health sciences, in which none of the reviews focused on the performance assessment of students in physical therapy, occupational therapy, or speech-language pathology. The reviews in medicine focused on studies with medical students and clinicians [11,12], and the reviews in rehabilitation sciences focused on practicing occupational and physical therapists [13,30] or only found measures of attitude [31]. Three reviews found the Fresno Test [12] or the Adapted Fresno Test [13,30] as the only validated assessments of performance, but they were used with medical students or rehabilitation sciences clinicians. Assessments of rehabilitation sciences clinicians or of medical students and clinicians are not sufficient. Rehabilitation sciences professionals approach practice differently than medical professionals, and "discipline-specific measures that address a profession's own EBP needs and concerns may be warranted" [13,30]. Students will not have the same level of EBP skills and knowledge as an expert clinician, and assessments of students should reflect this difference [15].
To address these gaps in the literature, this scoping review investigates how knowledge and skills are assessed in the IL instruction of physical therapy, occupational therapy, and speech-language pathology students, regardless of whether the instruction was designed and delivered by a librarian.
This project had three main objectives: (1) to determine what assessment methods were being used, (2) to determine whether these assessment methods had been tested for reliability and validity, and (3) to provide librarians with guidance on assessment methods that could be used in their own instruction. We chose to report on both measures of attitude as well as measures of performance, though our focus was on the latter.

METHODS
We chose a scoping review rather than a systematic review methodology for this review, because we addressed a broad question, "aiming to summarize and disseminate research findings" as proposed by Arksey and O'Malley in their definition and outlined in the stages below [32].

Stage 1: Identifying the research question
The original research question stated: How are knowledge and skills being assessed in the IL instruction of nursing and allied health students? After the first round of full-text screening, we narrowed the scope to include only physical therapy, occupational therapy, and speech-language pathology students, and we removed nursing from the research question due to reasons explained in stage 3 below.

Stage 2: Identifying relevant studies
A systematic search strategy was constructed by one author and reviewed by the second author as recommended by the PRESS standard [33]. This strategy was then run in MEDLINE (Ovid) and adapted for CINAHL; EMBASE (Ovid); Education Resources Information Center (ERIC) (EBSCO); Library and Information Science Abstracts (LISA); Library, Information Science & Technology Abstracts (LISTA); and Proquest Theses and Dissertations from 1990 to January 16, 2015. Four thousand ninety-seven articles were found, with 2,747 articles remaining after duplicates were removed. No limits for language or publication type were applied at the searching stage. EndNote citation software was used for duplicate removal and screening. The MEDLINE search strategy can be found in the supplemental appendix.

Stage 3: Selecting studies
The two authors each independently screened the titles and abstracts of all 2,747 articles, resulting in the exclusion of 2,143 articles. We then screened the remaining 604 full-text articles for eligibility. A third librarian was available to resolve any disagreements that were not resolved by discussion between the 2 authors; however, external consultation was not needed.
To be included in the review, the studies had to describe the assessment of IL skills in undergraduate or graduate physical therapy, occupational therapy, speech-language pathology, nursing, or general allied health academic degree programs. We defined IL skills as any instruction on question formulation, searching, database use, critical appraisal, or any library skills that pertained to these aspects of EBP. A librarian did not have to be involved in the instruction or assessment, except in studies that focused exclusively on critical appraisal, in which case librarian involvement was a requirement for inclusion, in order to keep the focus on areas of librarian expertise.
We excluded articles if they were not in English or in French. One hundred thirty-five studies were initially included for data extraction. Due to the large number of included studies, we conducted a further round of screening to put aside the articles that pertained to nursing or allied health disciplines other than physical therapy, occupational therapy, or speech-language pathology. This paper reports on the physical therapy, occupational therapy, and speech-language pathology portion of the data set. The final round of screening resulted in 33 articles for data extraction. For the update, the first author conducted a search from January 17, 2015, to January 16, 2017, and found 238 references, 141 after duplicates were removed. We screened these references using the new criteria, yielding 7 studies for inclusion. The PRISMA flow diagram ( Figure 1) details the screening process [34]. The two authors independently extracted the data from the forty included articles using an Excel spreadsheet with predefined fields for the following information in each study: country of study, degree level of learners, health profession of learners, librarian involvement (none or as instructor, assessor, author), measures of attitude used, measures of performance used, testing of measures for reliability or validity, instructional context of the IL instruction, knowledge and skills taught, and stated learning outcomes.
To determine whether the study assessed IL knowledge and skills, we had to explicitly define what IL knowledge and skills were important for EBP [15]. Using the study by Shaneyfelt et al. [12] as a model, and our own experience and research in EBP and IL to further refine, we used the following definitions while coding the articles: Knowledge and/or skills in question formulation (PICO format and other research question formats); database searching (any mention of databases or the mechanics of searching); search quality (explicit content addressing appraisal of a search); search development (explicit content addressing how to develop a good search); resource selection (how to choose a database or another resource); article selection (content to select based on study design and relevance to topic); and article appraisal (critical appraisal) To categorize the assessments found, we used the definitions in Schilling and Applegate: Measures of performance are tests, course products, and portfolios; measures of attitude are self-report surveys, interviews, focus groups. [14] We had hoped to report on whether the assessments were formative (in-process) measures or summative (at the end) and at what stage in the EBP cycle they took place, but the articles did not report enough information to make coding these data points possible.
We resolved any differences in data charting by discussion, with both authors returning to the original article to make sure that the data matched the article. No formal quality assessment was undertaken as the goal was to find all assessments being used, regardless of the quality of the study in which they were used.

Stage 5: Collating, summarizing, and reporting the results
We used the data charted above to collate the data into descriptive statistics and summarize the qualitative themes that emerged. The qualitative themes arose from discussions between the two authors.

RESULTS
Of the 40 included studies, the level of learners were undergraduate (n=15), graduate (n=19), or mixed undergraduate and graduate (n=6). Most of the studies were conducted in the United States (n=25) with other studies in Canada (n=4), Australia (n=5), continental Europe (n=3), Republic of Ireland (n=2), and the United Kingdom (n=1). The studies were evenly distributed across the included health professions: physical therapy (n=11), occupational therapy (n=11), mixed physical and occupational therapy programs (n=11), and speech-language pathology (n=7). Table 1 provides more information. Three major themes emerged as the authors discussed their findings: types of measures used, type and context of librarian involvement, and skills and outcomes described.

Theme: Types of measures used
Thirty-four measures of attitude found in 30 studies were collated into the following categories: selfreport surveys (n=25), focus groups (n=2), informal feedback (n=2), and interviews (n=1) [40]. Most of the self-report surveys were developed locally. Eight studies used one of the following published instruments: evidence-based practice profile (EBP2) [63,67,68]; knowledge, attitude, and behavior (KAB) questionnaire [80]; practice-scholar selfefficacy survey [52]; EBP competency scale [75]; selfevaluation in EBP [8]; and EBP self-efficacy and task value [77].  Thirty-seven measures of performance in 34 studies were collated into the following categories: course products (n=24); non-IL standardized tests, defined as tests developed for EBP contexts (n= 6, using 4 tests and 1 rubric); IL standardized tests, defined as tests developed for IL contexts (n=2, using the research readiness self-assessment [RRSA] [73]); and locally developed tests (n=5). The course products varied widely; for example, there were 7 searching assignments, where the search strategy was the main focus; 5 case-based inquiry [86] assignments, where students applied the 5 EBP steps to a patient scenario; 5 critically appraised topic assignments, where students summarized the evidence on a given topic; and 4 research project assignments. One study adapted the Valid Assessment of Learning in Undergraduate Education (VALUE) rubric to assess an EBP assignment [82].
The non-IL standardized tests were all originally tested for reliability and validity in separate published studies: the Adapted Fresno Test (for occupational therapy), the Dutch Modified Fresno Test (for speech-language pathology, in Dutch only), the knowledge of research evidence competencies (K-REC) test, and the California critical thinking skills test (CCTST). Table 2 provides more details on all of the published measures.

Theme: Type and context of librarian involvement
Librarian involvement was nearly equally divided, with almost half (n=19, 47.5%) of the studies involving a librarian. In these studies, the librarian was most frequently involved as an instructor (n=15), but the librarian was also involved as an author (n=14), as an assessor (n=12), and as a course designer (n=2). Though librarian involvement was not explicitly stated in the other half (n=21, 52.5%) of the studies, the students received instruction and/or assessment of searching and other IL skills. All but one of the included studies involving librarians used some measure of performance. The full breakdown of librarian involvement can be seen in Table 3.
The majority of the studies (n=33, 82.5%) had IL knowledge and skills being taught in a courseintegrated context by either a librarian or the instructor, though there were studies with one-shot instruction (n=2) and studies where the delivery context was unclear (n=2) or where no course was directly associated (n=3). While not all studies had clearly stated learning outcomes, 9 studies that did not describe librarian involvement included explicit statements regarding knowledge and skills often taught by a librarian, such as "effectively search research literature and judiciously select relevant evidence" [48] and "enable students to develop skills in formulating answerable clinical questions and finding and evaluating the research evidence" [70]. Of the 3 studies that used the Adapted Fresno Test or the Modified Dutch Fresno Test as the assessment for EBP knowledge and skills, both of which include a section on assessing the "finding the evidence" portion of the EBP cycle, none of the studies described the involvement of a librarian.

Theme: Skills and outcomes described
Eighteen of the studies described the instructional content of the lectures, workshops, and course materials, and another 8 studies did not describe the content but had learning outcomes that described EBP knowledge and skills. For the purpose of summarizing the knowledge and skills taught, we merged the descriptions of instructional content and of learning outcomes, even though stating in an outcome that a course will teach certain concepts does not necessarily mean that the stated knowledge or skill has been taught. We have reported the specific knowledge and skills (as described in the methods) in Table 4. Most of these 26 studies (n=23, 88.5%) described database searching knowledge or skills, including 9 studies that did not involve a librarian. The second most common content topic of these 26 studies was question formulation (n=21, 80.8%), with librarians involved in 10 of the studies.

DISCUSSION
The forty included studies revealed a wide variety of attitude and performance measures being used in rehabilitation sciences student contexts. The published self-report surveys are worth investigating for adaptation in library contexts; however, a deeper investigation and discussion of these measures is beyond the intended scope of this review. These measures may provide guidance for librarians who want valid and reliable ways of understanding students' perceptions of IL teaching and learning.   Of the validated tests of performance, the Adapted Fresno Test [51] and the Modified Fresno Test [90] have the most potential for instructional librarians to incorporate into their own assessments, as they have already been adapted to occupational therapy and physical therapy and have components for evaluating the first two steps of the EBP cycle.
This review did not find an example study using the Modified Fresno Test (other than the validation study), but it was included here as a suggested assessment measure for those teaching physical therapy students. Unfortunately, there is no validated English version for speech-language pathology. The benefit of the adapted or Modified Fresno Test is that it evaluates a real-life application of knowledge and skills [91], testing the student's ability to take a case through the five EBP steps. One drawback of the adapted or Modified Fresno Test is that these measures were originally designed to be taken as a stand-alone exam, where the student is asked to describe how to search instead of providing an actual search history. As such, the adapted or jmla.mlanet.org 106 (1) January 2018 Journal of the Medical Library Association Modified Fresno Test equates describing how to search to actually using a database and does not test the student's ability to apply the knowledge and skills that librarians teach in IL instruction.
The K-REC test [65], while based on the Fresno Test, primarily tests knowledge and does not cover IL knowledge well. The CCTST [42] is a commercially available test and does not directly test EBP knowledge. The only IL standardized test included in our review, the RRSA [73], was not used in a course-integrated instruction setting, and it is unclear if it tests the effectiveness of IL instruction in EBP contexts.
The large number of assignments that focused on IL knowledge and skills provide many examples for librarians designing their own assignments for use in courses. Course assignments are an excellent method for librarians to become involved in the assessment cycle, especially when class time is limited. While time-consuming to develop, a valid and reliable grading rubric is one way for librarians to give more rigor to assignments used as assessment measures [92].
Our review did not find any examples of grading rubrics that measure skills in a meaningful way and are ready for use by interested librarians. Thomas et al. used a comprehensive and instructive EBP reference model [93] as a grading rubric for their EBP case-inquiry, but it would be impractical for librarians to use for grading assignments due to its complexity [10]. Grant and Brettle as well as Evenson used simple yes/no rubrics [56,57]. The adaptation of the VALUE rubric to the graduate physical therapy student context by Turbow and Evener might be the most useful [82], as it assessed multiple skills in the EBP cycle and could be widely adopted by librarians who are grading case-based inquiries, critically appraised topics, or research project assignments. However, librarians using the VALUE rubric would most likely need to create more detailed requirements for the points awarded for each category in order to make it an effective assessment tool. It is our opinion that more studies are needed to verify the VALUE rubric's reliability and validity in health sciences and graduate student contexts.
The fact that all but one of the included studies involving librarians used some measure of performance suggests that health sciences librarians are doing performance-based assessment. Most of these assessment measures were course assignments; therefore, their impact on learning can be more difficult than tests to quantify, unless they are graded in a systematic manner. The use of grading rubrics, as discussed above, and clearly defined learning outcomes aligned with these rubrics would allow better demonstration of the impact of assignments.
The fact that the majority of the included studies described course-integrated instruction suggests that a place has already been made in these curricula for EBP instruction and the accompanying IL knowledge and skills. We had hoped to make a connection between the learning outcomes of a course, the knowledge and skills taught by the instructional activities, and the assessment measures used in order to better understand how the knowledge and skills were assessed. However, most of the studies did not describe in sufficient detail all three components to make these connections. Instead, as it emerged that IL skills were often taught or stated as outcomes in courses where librarians were not mentioned, we collated what knowledge and skills were taught and by whom.
It was striking to discover the number of courses that reported learning outcomes that addressed searching and database use but that did not have a librarian involved in the course. This trend suggests that there are still barriers to instructors recognizing the value and expertise of librarians in EBP instruction. As experts in IL, librarians can model the proper knowledge and skills, particularly in the first three EBP steps, to help future clinicians reach eventual expertise and behavior in clinical practice [15].
With course learning outcomes already aligned with IL outcomes, there are many opportunities for librarians to work with instructors to develop course content and assessments to better teach to these outcomes, if instructors are open to this collaboration. In cases where the librarian is not involved at all, the time has already been made in the curriculum and course instructors may be open to assistance from the librarian expert in designing and teaching the content. In cases where the librarian is involved in the instruction, but not the assessment, there may be an assignment or other assessment measure that the librarian could use to assess IL knowledge and skills. With some modifications, portions of the adapted or Modified Fresno Test could be used by librarians to evaluate question formulation and database searching more effectively. Librarians could work with instructors to integrate the entire adapted or Modified Fresno Test into the course assessment. Another option would be for librarians to use the VALUE rubric as modified by Turbow and Evener [82] with existing assignments.
The limitations of our study were as follows: It was possible that we missed relevant studies due to the large and ever-changing body of literature. We also did not contact authors for additional data in cases where studies lacked detail about such points as the involvement of a librarian or specific learning objectives.
This review confirmed for us that useful assessment tools in EBP instruction take time to develop and implement, on the part of the librarian and in the context of larger course programs. Assessment measures that test knowledge can be useful, especially if that is the only measure that a librarian can integrate into a course. However, knowledge tests will not evaluate whether instructional content such as hands-on practice with question formulation and database searching is effective in skill development. Assessments that measure skills, such as grading rubrics or the adapted or Modified Fresno Test, better indicate the effectiveness of instruction on student learning. Future areas of research could focus on validating modifications of the adapted or Modified Fresno Test that better assess searching skills and on further validating the VALUE rubric in EBP contexts.
Health sciences librarians spend a great deal of time developing and delivering instruction, and it is important to find measures that assess this instruction in an authentic way. More studies need to be done on the long-term retention of IL knowledge and skills. Given the lifelong learning characteristics of rehabilitation sciences professional programs, measuring long-term knowledge retention and skill development is essential to proving the value of IL instruction. jmla.mlanet.org