Question formulation skills training using a novel rubric with first-year medical students

Objective: The authors used an assessment rubric to measure medical students' improvement in question formulation skills following a brief evidence-based practice (EBP) training session conducted by a health sciences librarian. Method: In a quasi-experimental designed study, students were assessed using a rubric on their pre-instructional skills in formulating answerable EBP questions, based on a clinical scenario. Following their training, they were assessed using the same scenario and rubric. Student pre- and post-test scores were compared using a paired t-test. Results: Students demonstrated statistically significant improvement in their question formulation skills on their post-instructional assessments. The average score for students on the pre-test was 45.5 (SD 11.1) and the average score on the post-test was 65.6 (SD 5.4) with an average increase of 20.1 points on the 70-point scale, p<0.001. Conclusion: The brief instructional session aided by the rubric improved students' performance in question formulation skills.


INTRODUCTION
The pivotal first step in the evidence-based practice (EBP) process hinges on formulating an answerable question. A focused question will guide one efficiently to finding the needed evidence. Conversely, a poorly constructed question will misdirect a search for the needed evidence, thereby wasting precious time. As two EBP pioneers once observed, "fuzzy questions tend to lead to fuzzy answers" [1]. EBP can be defined as "a way of providing health care that is guided by a thoughtful integration of the best available scientific knowledge with clinical expertise" [2]. EBP serves as a decisionmaking framework in clinical practice. The "evidence" in EBP can include original research articles, point-of-care resources, diagnostic tests, physical exams, or patient histories [3]. After the first step of formulating an answerable question, steps two through four consist of searching for the evidence, critically appraising the evidence, and making a decision on applying the evidence to the patient. A fifth, more reflective step, involves evaluating one's performance.
EBP has had a deep and sustained influence on the health professions, and most medical school curricula now include teaching EBP [4]. The approaches employed for teaching EBP are diverse [5,6]. EBP textbooks stress the importance of clearly stated questions for correctly launching the EBP process and offer practical advice to learners. Health sciences librarians frequently teach medical students how to compose questions in the context of their EBP coursework [7]. Librarians are natural collaborators in teaching EBP question formulation skills, having worked for well over a century in the broader area of question formulation [8][9][10].
EBP instructors commonly employ a structure for formulating questions that consists of patient, intervention, comparison, and outcome (PICO) elements. The patient element might include some clinically relevant demographic characteristic like the patient's age and pertinent medical problems; intervention consists of the treatment under consideration; comparison relates to the control treatment; and outcome pertains to results of clinical importance [11,12]. The widely cited Fresno test [13][14][15][16] of evidence-based medicine [17] that assesses learners' EBP skills and knowledge incorporates PICO, albeit for only two brief segments. The Berlin questionnaire for evaluating learned EBP skills models the PICO structure in a similar way [18], and an assessment tool by Wyer et al. also uses PICO [19]. Some practitioners have suggested alternatives to PICO. For example, the SPIDER question formulation format relates to qualitative instead of quantitative EBP studies [20]. Davies inventoried an impressive array of alternatives to PICO, including some cited in this article, although these alternatives focus primarily upon evidence-based library and information practice rather than the broader health care professions context of EBP [21].
A growing body of evidence points to the limitations to PICO despite its common use. A team of early EBP physicians created the PICO structure in 1995 with no known input by health sciences librarians or other information professionals [11]. An exploratory study by Huang et al. compared actual clinicians' questions with PICO and found PICO did not represent those questions well. Importantly, they also determined that PICO was most suitable for treatment EBP questions, rather than all other types of EBP questions [22]. However, only roughly half of EBP questions relate to treatment [23,24], so the PICO format may not adapt well to diagnosis, prognosis, epidemiology, or other types of EBP questions. Schardt et al. found no statistical difference between using and not using PICO search templates for searching in PubMed [25], whereas Hoogendam et al. found PICO to be deficient for launching a timed PubMed search [26]. Furthermore, a 2018 review of whether PICO improved the quality of searches in a variety of databases proved inconclusive [27].
A Cochrane Collaboration-sponsored systematic review on interventions to teach learners how to formulate questions underscored the importance of the topic, stating that "formulating questions is fundamental to the daily life of a healthcare worker" [28]. The systematic review reported that PICO question formulation interventions, most involving residents or practicing clinicians while excluding medical students, produced mixed results. The authors of the Cochrane systematic review selected only four randomized controlled trials on question formulation. These four key studies employed different educational interventions and units of measurement, had methodological issues, and/or produced inconclusive results [29][30][31][32].
The Cochrane systematic review, plus the authors' observations of students regularly struggling with the PICO format for many of the same reasons reported in the literature, provided the rationale for improving instruction on question formulation skills at our medical school. Therefore, we evaluated the effect of a new approach to training first-year medical students on question formulation that included a brief instructional session and a novel rubric intended to overcome perceived limitations of existing approaches.

Population
We performed a quasi-experiment pre-/post-test study [33,34] with 107 first-year medical students who were enrolled in a required "Quantitative Medicine One" course at the University of New Mexico's School of Medicine to assess the impact of the new EBP instructional approach on formulating an EBP question. This required course incorporated biostatistics, epidemiology, and EBP topics. The course began during the sixth month of the first year of medical school. None of the 107 students had previously learned EBP question formulation skills. The first 3 authors-consisting of a faculty librarian/informatacist, physician epidemiologist, and hospitalist-served as the faculty instructors for the course. The University of New Mexico Institutional Review Board approved this study (19-008).

Instructional intervention
All 107 students took a pre-test to gauge their preinstruction skills in formulating questions on their first day of class. The interventional instruction occurred 28 days later, and the post-test occurred within 30 hours after the intervention.
The pre-test presented students on the first day of the course with a case vignette of an elderly patient with Parkinson's disease: Formulate a question based on this clinical vignette that, when answered, will lead to the best treatment of this patient.
The same 107 students received training on question formulation for 25 minutes as part of an overview of the EBP process in an active learning, large studio classroom setting. At the beginning of the large group session, students were asked, "Why do you think that formulating answerable questions will be important for your individual professional education and for your career?" Small groups consisting of about 8 students, seated at tables, listed their answers to this question on whiteboards. The groups reported their rationales, which the faculty librarian tied together for real-time thematic analysis. He described a systematic review on questions raised by clinicians at the point of care [35], stressing the possible number of questions raised per patient as a means of underscoring the importance of question formulation skills.
About halfway through this large group session, the faculty librarian introduced the rubric (Table 1), as previous studies suggested that students were more likely to accept a rubric in the earlier, formative stages of learning a new skill [36]. The interactive mode of instruction offered practical tips for using the rubric as a checklist. The twenty-fiveminute segment of class on question formulation guided students through the steps of focusing on the problem or disease, amplifying the critical details, and composing a free-standing question as guided by the rubric. The faculty librarian gave them examples of well-formulated questions composed by medical students in previous years to facilitate selfconfidence among the current students and contrasted these model questions with less productive questions. Students first worked alone and then with a partner in their small groups in a pair-share team to formulate their own diagnosis, treatment, or prognosis types of EBP questions. Within thirty hours of the large group session, all students participated in labs consisting of twenty-five to thirty students in which the faculty librarian briefly showed them examples of diagnosis, treatment, and prognosis EBP questions and fielded any questions. He then gave them five minutes to formulate an answerable question based on the same clinical vignette as that in the pre-test. Students were allowed to use their rubrics as checklists during the post-test. The remainder of the session revolved around searching for evidence, primarily in PubMed. The instructors scored the pre-and post-tests using the rubric to measure the degree of student improvement, with the post-test results passed along to the students before they completed a similar but graded assignment on question formulation without the clinical vignette prompt. Later in the course, the instructors briefly introduced the PICO format to familiarize students with it, in case they encounter it later in their careers.

The rubric
We scored students' formulated questions using the rubric ( Table 1). The prototype of the current rubric was initially developed in 2013 through a series of trial and error approaches involving first-and second-year medical students. It was improved iteratively across 2014-2018 through formal and informal feedback from students and faculty.
The current version of the rubric features three major sections. The first section evaluates students' focus on the main problem or disease. This focus on the disease or problem prepares the learner to translate this concept to effective literature searching by finding the appropriate Medical Subject Heading (MeSH) in PubMed that best represents the problem or disease. The second section evaluates students' amplification of essential clinical details, such as descriptive adjectives (semantic qualifiers), scales, temporality, or population. These adjectives can alert the practitioner about which studies to include or exclude in matching the evidence to the specific patient, and details concerning the population can inform the search by including or excluding certain populations, using filters related to age groups, and using key MeSH terms, such as those related to socioeconomic factors, class, risk factors, or protective factors. The third section determines if the learner has composed a question that a reader can understand without the reader knowing the broader clinical context. In other words, "the question stands by itself."

Statistical analysis
Students' mean pre-and post-test scores were analyzed with a paired t-test using Stata, version 15 (College Station, TX).

RESULTS
The 107 students who underwent the brief training and use of the rubric improved their question formulation skills by a statistically significant margin. The average score for students on the pretest was 46 (standard deviation [SD] 11, 95% confidence interval [CI] 43-48), whereas the average score on the post-test was 66 (SD 5, 95% CI 65-67). Students did not perform well on the pre-test, as expected, largely because approximately half of the students did not mention the disease.

DISCUSSION
We found that question formulation skills taught using this new instructional intervention and rubric yielded a significant improvement of 20.1 points on average on a 70-point scale. Those who have taught students, residents, and practitioners know that teaching others how to formulate answerable questions can be far more challenging than it might first appear to be. Question formulation expects the learner to initially use inductive logic and then transition to using deductive logic [37][38][39]. The wording of a question also can introduce unintended ambiguity and confusion [40,41].
This study differed from previous library and EBP studies on question formulation in several important ways. Our medical students were new to EBP question formulation, whereas populations in other studies likely had prior question formulation training, particularly with PICO. Second, we used a new approach to question formulation to overcome perceived limitations of the PICO structure. Third, the rubric evolved in a collaborative context involving librarian, basic sciences, and clinical faculty members. Fourth, this new approach and rubric easily accommodated diagnosis, prognosis, and epidemiology as well as treatment types of EBP questions.
The interval of no more than thirty hours between the intervention and the post-test minimized the likelihood of biases such as regression to the mean or secular changes [42]. This brief interval also prevented threats to internal validity such as the historical artifact or maturation [43][44][45]. The faculty librarian used a near identical approach in teaching question formulation and the same rubric with physician assistant and medical residents during most of these same years, 2014-2019. Learners in these other education programs demonstrated similar improvements in formulating questions, although this study with medical students represented the first rigorous research study to measure improved student performance.
As this study involved medical students at one specific medical school, it likely has limited generalizability. The same instructional protocols and rubric have been used at our medical school and other degree programs at our institution for several years with similar outcomes however, so our results appear to remain consistent and, thereby, reliable in this institution.
Although we might have preferred using an experimental design, this curriculum posed too many opportunities for contamination, because students belong to several concurrent academic groups during the first two years at this medical school. This multiple group membership would place intervention and control students in daily contact, thereby risking contamination of the control participants by the intervention participants. This study could not measure students' long-term retention of question formulation skills because the curricular schedule scattered students among clinical assignments following the course. Furthermore, the study design did not allow a distinction between the separate effects of the teaching and rubric.
Future studies are needed to replicate these results, and multicenter studies could greatly extend their generalizability. It would be interesting to measure the effect of the instruction alone and rubric alone in isolation from one another, although this experiment would involve considerable logistic obstacles due to possible contamination of controls. Furthermore, as the faculty librarian created the rubric to improve subsequent searches, future studies could seek to demonstrate a connection between question formulation and search quality.
This prospective quasi-experiment involving pre-and post-test scores demonstrated that a brief instructional session in question formulation skills accompanied by a new rubric improves students' skills. Question formulation skills help students become lifelong learners and more immediately appear to improve their critical thinking skills [46]. This new approach places greater emphasis on the needed elements for question formulation than the popular Fresno test. Also, the instructional session improves upon the PICO approach by expanding its versatility beyond treatment types of EBP questions to include epidemiology, diagnosis, risk, harm, and prognosis types of questions. Therefore, this study might signal a way forward to diversify how trainers teach EBP question formulation skills.

DATA AVAILABILITY STATEMENT
De-identified score data are available in the University of New Mexico's Digital Repository at https://digitalrepository.unm.edu/hsc_hslic/1/. jmla.mlanet.org