Using educational prescriptions to teach medical students evidence-based medicine.

Abstract Purpose: To evaluate feasibility and impact of evidence-based medicine (EBM) educational prescriptions (EPs) in medical student clerkships. Methods: Students answered clinical questions during clerkships using EPs, which guide learners through the “four As” of EBM. Epidemiology fellows graded EPs using a rubric. Feasibility was assessed using descriptive statistics and student and fellow end-of-study questionnaires, which also measured impact. In addition, for each EP, students reported patient impact. Impact on EBM skills was assessed by change in EP scores over time and scores on an EBM objective structured clinical exam (OSCE) that were compared to controls from the prior year. Results: 117 students completed 402 EPs evaluated by 24 fellows. Average score was 7.34/9.00 (SD 1.58). 69 students (59%) and 21 fellows (88%) completed questionnaires. Most students thought EPs improved “Acquiring” and “Appraising”. Almost half thought EPs improved “Asking” and “Applying”. Fellows did not value grading EPs. For 18% of EPs, students reported a “change” or “potential change” in treatment. 56% “confirmed” treatment. EP scores increased by 1.27 (95% CI: 0.81–1.72). There were no differences in OSCE scores between cohorts. Conclusions: Integrating EPs into clerkships is feasible and has impact, yet OSCEs were unchanged, and research fellows had limitations as evaluators.


Introduction
To provide the highest quality care to patients in the everchanging field of medicine, physicians must be trained in lifelong learning. These skills include the ''four As'' to practicing evidence-based medicine (EBM), namely the ability to: (1) Ask answerable clinical questions, (2) Acquire evidence to answer those questions, (3) Appraise evidence, and (4) Apply evidence to patient care (Straus et al. 2005). Studies suggest that although medical students' self-perceived competence in these areas is high, their actual performance is poor (Caspi et al. 2006). These knowledge gaps have led the Association of American Medical Colleges (AAMC) to explicitly state in their Learning Objectives for Medical Student Education that students must be able to apply the principles of EBM to patient care (2010).
There is no clear evidence about the best method to teach EBM to medical students (Ilic & Maloney 2014). Studies suggest that one brief training session can improve searching skills (Gruppen et al. 2005) that EBM training embedded in a clerkship can improve EBM skills as measured by selfassessments (Dorsch et al. 2004), and that EBM knowledge and skills can be equally improved with computer-or lecture-based instruction (Davis et al. 2008). But these studies are limited in that they were not able to evaluate all four As of practicing EBM. Instead, they assessed individual components, such as Acquiring or Appraising. These studies as well as other reviews (Shaneyfelt et al. 2006;Oude Rengerink et al. 2013) also highlight the lack of validated tools to test the four As of practicing EBM. For example, although Shaneyfelt describes two tools that have been assessed for validity in evaluating EBM skills, these tools are not designed for use in the care of actual patients (Shaneyfelt et al. 2006), so they do not allow for the evaluation of all EBM steps, because they do not include an assessment of the application of evidence to patient care.

Practice points
To provide quality care, physicians must be lifelong learners, trained in the practice of evidence-based medicine (EBM A model for teaching EBM that was recently proposed suggests integrating formal EBM training across both preclerkship and clerkship instruction to optimize the transfer of EBM skills (Maggio et al. 2015). One tool that can enable this, and evaluate all four steps of practicing EBM, is the EBM educational prescription (EP), a novel web-based tool that offers formal EBM training in the context of clinical care by guiding learners through the four steps of EBM to address clinical questions in real time. The EBM EP website (http://ebm.wisc.edu/ep/) and grading criteria were initially designed to evaluate the EBM competency of medical residents, and have been tested for validity as an evaluation tool at multiple residency programs in the USA (Feldstein et al. 2009(Feldstein et al. , 2011. In our study, we evaluate the feasibility of incorporating the web-based EBM EP into medical student clerkships and the impact on student EBM skills and patient care.

Overview
At the University of Pennsylvania Perelman School of Medicine, the basic science blocks occupy the first year and a half of the medical school curriculum, and the clinical clerkships occupy the second semester of the second year and the first semester of the third year (January to December). At the time of this study, students' EBM curriculum included an Introduction to Epidemiology course that provided lectures and small group sessions on study design, bias, confounding, chance, and diagnostic testing, and occurred during the first semester of their first year. They then had a Clinical Decision Making course in the second semester of their first year that provided lectures on EBM, decision analysis, prediction rules, and cost analysis, and small group sessions alongside a year-long series of pathophysiology blocks that occupied the second semester of their first year and the first semester of their second year. In these small group sessions, they practiced their critical appraisal and decision-making skills using clinical content from their pathophysiology blocks.
During the intervention period, medical students were required to complete at least one EBM EP per clerkship for clinical questions arising on their Internal Medicine, Family Medicine and Pediatrics clerkship blocks. Prior to performing EPs, students received an information sheet about the EPs and the study from their clerkship directors. The students were also asked to complete an online orientation for the EP.
Graders of the EPs were physician fellows enrolled in the University of Pennsylvania's Master of Science in Clinical Epidemiology (MSCE) program (2015). Evaluation of EPs was a mandatory part of the MSCE fellows' first year ''Professional Development'' curriculum. All second semester first-year fellows and first semester second-year fellows participated. Prior to their evaluations of the student EPs, the fellows received online training on evaluating EPs, as well as an orientation lecture on the EPs and practicing EBM. Graders also received an information sheet, and introductory and follow-up emails regarding the EP.
The study received expedited approval and a HIPAA waiver from our Institutional Review Board.

EBM educational prescription
The EBM EP is a web-based tool that guides learners through the four As of EBM (Supplement 1). We made minor changes (e.g. changing the term ''resident'' to ''learner'') to adapt it for medical students. Learners use the EP to define a clinical question (''Ask''), document a search strategy (''Acquire''), ''Appraise'' the identified evidence, report the results, and describe the ''Application'' of the evidence to their particular patient. The EP also contains a question about whether the evidence changed patient care: ''Did (performing the EBM EP) change your patient's management?'' The answer choices included ''No Effect on Plan'', ''Reinforced/Confirmed Plan'', ''Would Have Changed Plan if Had Information Earlier'', or ''Changed Plan''. Questions about the time required to complete the EP, as well as the usefulness of the EP (from ''Extremely Useful'' to ''Not at All'') and learner satisfaction with the EP process (from ''Extremely Satisfied'' to ''Not at All'') were also included at the end of each EP. The EP website has built-in EBM resources to help learners complete their EPs, including resources to help appraise and interpret studies. Learners can view all of their completed EPs.
The EP grading form is a web-based tool that allows graders to evaluate learners' EPs (Supplements 2A and 2B). EPs are evaluated in five areas: question formation; searching; evidence appraisal; application to patients; and overall competence. A built-in grading rubric provides anchors for grading, using a scale from 1 (not yet competent) to 9 (superior). The EP grading form has the same built-in resources as the learner pages. Fellows entered the time required to grade each EP and were also asked to provide written feedback to students about their EPs.

Study population
All medical students at the University of Pennsylvania Perelman School of Medicine who completed their clinical clerkships from January to December 2010 were included in the intervention group. Students who did not complete all core clerkships within the calendar year (e.g. MD-PhD students) were excluded. In addition, all medical students who completed their clerkships from January 2009 to December 2009 were included in the study as historic controls for a comparison of performance on an end-of-year EBM objective structured clinical exam (OSCE). The EBM OSCE directly followed the students' end-of-year Clinical Skills Inventory evaluation, and took place in the same location. Students had 25 minutes to complete the EBM OSCE. The historic controls completed the EBM OSCE in Spring 2010, while those in the intervention group completed the exam in Spring 2011. The intervention group and historic controls received similar EBM educations during their basic science blocks, except the intervention cohort received an introductory lecture on the EBM EP just prior to their clerkships.

Other data sources
Baseline characteristics of the two classes of medical students (2009 and 2010) taking the end-of-year EBM exam as well as the fellows were obtained from administrative databases. Baseline data for the medical students included age, gender, race, participation in a dual degree program, and clerkship grades in which they performed EPs. The grades from the three clerkships were averaged to create a ''gradepoint average''. Baseline data for the fellows included age, gender, race, graduate degrees, and clinical specialty.
Students and fellows also completed an end-of-year questionnaire through SurveyMonkey (SurveyMonkey Inc., Palo Alto, CA). Questionnaires included quantitative questions based on a Likert scale and free text, and asked about their attitudes toward the EP, barriers to using the EP, impact of the EP on student EBM skills, and suggestions for improvement (Supplements 3 and 4).
The end-of year EBM exam was based on a previously validated OSCE (Frohna et al. 2006). The OSCE tested the four As of EBM, and had a maximum score of 120, with 30 points for correctly Asking answerable questions, 25 points for correctly Acquiring evidence, 30 points for correctly Appraising studies (Appraise 1), 15 points for correctly calculating a number needed to treat (Appraise 2), and 20 points for correctly Applying evidence to patient care (Fliegel et al. 2002;Frohna et al. 2006) (Supplement 5). The OSCE and scoring rubric were modified for this study by adding the final 20 point section on applying evidence to patient care. Each EBM OSCE was graded by two independent evaluators. Inter-rater reliability of OSCE scoring was measured using quadratic weighted kappa scores.

Analysis of EP feasibility, impact, and predictive validity
Feasibility was evaluated by examining the number of EPs performed per student, average self-reported time to complete EPs, average number of EPs rated by graders, average self-reported time to evaluate EPs, and average competency rating for each EP category (Table 1). Descriptive statistics were also used to analyze student and grader questionnaire responses to understand their attitudes toward the EP, barriers to using the EP, and suggestions for improvement. Impact of performing EPs was evaluated by examining changes in individual student's EP competency ratings over time and comparing EBM OSCE scores of the intervention and control cohorts. Changes in students' competency ratings were estimated using a mixed-model regression controlling for within-person effects. Differences in mean total OSCE scores as well as scores for the individual components of the test were analyzed using t-tests. The association between number of EPs performed and overall OSCE scores was examined, controlling for student performance on their last EP. Descriptive statistics were used to analyze student and grader questionnaire responses about the impact of the EP on student EBM skills and to estimate the percentage of EPs that caused a change in management and the types of changes that occurred.
Predictive validity of the EP was evaluated by comparing students' average and most recent EP competency ratings with their EBM OSCE scores and mean gradepoint average across their Internal Medicine, Family Medicine and Pediatrics clerkships.

Results
One hundred seventeen students had 402 EPs evaluated by fellows (mean 3.4 per student). Five students completed 2 EPs during the year, 57 completed 3 EPs, 54 completed 4 EPs, and 1 student completed 5 EPs. Three hundred and seventy-seven EPs (93.8%) addressed foreground questions, which traditionally ask for specific information to directly inform a clinical decision, and define the Population, Intervention, Comparator, and Outcome (i.e. the PICO) of interest. Twenty-five EPs (6.2%) addressed background questions, which traditionally ask for general information, such as the pathophysiology of a disorder, or the mechanism of action or dosing of a drug. The majority (300, 81.5%) involved a question about therapy. Thirty-four (9.2%) examined a diagnostic question.
Students' average overall competency score was 7.34 (standard deviation [SD] 1.58) out of 9. Students took a median of 90 minutes (interquartile range [IQR] 60-120) to complete EPs and fellows took a median of 20 minutes (IQR 15-30) to grade EPs. The EPs changed or would have changed plans if they had the information sooner 18% of the time, and 56% confirmed the plan. The change in overall competency score over one year was significant at 1.27 points (95% confidence interval [CI]: 0.81-1.72).
In general, students perceived the EPs to favorably impact their ability to search databases, and find and appraise evidence (Supplementary Figure  S1). Approximately 40% thought the EP also improved their ability to ask clinical questions and apply evidence in practice, and viewed the EP as valuable. Students' most frequently cited barriers to using the EP included time (33%), personal attitude toward EBM (26%), faculty knowledge of EBM (20%), and their own comfort with appraising evidence (19%).
Twenty-one fellows (88%) completed questionnaires. Although most fellows believed students' EPs improved, and over one-third believed that EPs strengthened learners' EBM skills and was a valuable tool for evaluating learners' EBM skills, the majority did not believe that evaluating EPs was a valuable experience for them, nor would they recommend the experience continue as part of their clinical epidemiology training (Supplementary Figure S2). The most commonly cited barrier by fellows (86%) was the time required to grade EPs.
One hundred and eighteen students met eligibility criteria for the control cohort and had OSCEs available for grading. There were no meaningful demographic differences between the intervention and control cohorts (Table 2), nor were there differences in mean overall OSCE scores (means of 87.1 and 86.2 out of 120, respectively, p ¼ 0.66) ( Figure  1). There were also no differences in the mean scores on the individual components of the OSCE. The inter-rater reliability of the OSCE graders was excellent with a quadratic weighted kappa of 0.93.
There were no significant associations between number of EPs performed and overall OSCE scores. There were also no significant associations between EP scores and OSCE scores or clerkship ''gradepoint averages''.

Discussion
To our knowledge, this is the first study to evaluate the use of a web-based EBM educational prescription integrated into a medical student clerkship. We found that it was feasible to incorporate the EP into clerkships, and students reported impact on patient care and EBM skills. However, scores did not increase on an EBM OSCE for those who used the EP throughout their clerkships, and using fellows who were not involved in the clinical care of patients cared for by the medical students had limitations.
Our findings are similar to those of earlier evaluations of EBM EPs used in the context of Internal Medicine residencies. In a pilot using paper EPs, housestaff reported that the EPs improved their EBM skills, but that time was a barrier to performing EPs (Feldstein et al. 2009). In a later study examining web-based EPs across five Internal Medicine residency sites, housestaff took half the time of our medical students to complete their EPs, yet scored almost a full point lower on average than our students (Feldstein et al. 2011). Housestaff in that study reported an impact of the EPs on their care plans similar to what our medical students reported, except a smaller percentage of the students believed the EP changed the plan, suggesting that students in general may believe they have less decision-making authority or impact on the plan compared to housestaff. Examples from EPs in which students noted an impact on the patients management include: adding spironolactone to the medication regimen of a patient with systolic dysfunction; discontinuing a proton pump inhibitor (PPI) in a patient with recurrent pneumonia and no clear indication for a PPI; prescribing a high dose statin for secondary stroke prevention; and recommending watchful waiting instead of antibiotics in a child with uncomplicated acute otitis media.
More than 70% of housestaff thought the EP was a valuable experience, which is much higher than our student ratings, and may be related to the perceived impact of the EPs on patient care. Housestaff and students perceptions about barriers were similar, with time and comfort with evidence resources being the most cited, although housestaff perceived time as a greater barrier, and comfort with resources as a lesser barrier than students. The findings about barriers have been described elsewhere (Green & Ruff 2005).
Both students and fellows expressed concern about the approach of the EBM EP curriculum. Although both were supportive of the use of EPs in real time to increase EBM skills and improve patient care, both were concerned that the fellows evaluating the EPs were not involved in the care of patients whom the EPs were targeting. This disconnect between the evaluators and clinical care may have led to delays in evaluator feedback, and lower scores from students when evaluating the impact of the EP on their ability to use evidence in practice, and the impact of the EP on their future approach to patient care. Overall, the fellows  did not value the experience, and recommended that it not be continued as part of their training in subsequent years. As a result, the EP curriculum adopted a new approach in calendar year 2012. Students' EPs were evaluated by clinicians responsible for the medical students' training during their clerkships, including clerkship directors and teaching attendings. For example, in some clerkships, students presented their completed EPs to their attendings, who used EP score cards with the grading rubric in real time to assess the students' competency. This approach has consistently received favorable ratings over the years (Appendix 1) and has been adopted by other clinical services.
Our study had limitations. First, we were unable to assess whether the EBM EP curriculum was associated with more long-term behaviors of students, including asking more clinical questions, performing more literature searches, or practicing EBM more consistently. Second, the significant improvement in EP scores over time identified by the study may have been the result of maturation bias, rather than the result of performing EPs over time.
In addition, because inter-rater reliability was not calculated for the fellows who graded the EPs, and because EPs were graded as they were performed and fellows were not blinded to when during the year the EPs were submitted, fellows may have been biased in their assessment of the EPs, providing higher scores over time despite unchanged performance on the EPs. The availability of a grading rubric to anchor grading likely minimized such bias. Third, the end-of-year EBM exam that we used in this study was based on a previously validated EBM OSCE, but was not validated itself. However, there were no significant differences in the overall exam scores or the scores on the individual domains, even when we examined only those domains that were part of the original EBM OSCE. Finally, the study was conducted in a single medical student class at a single institution, thus the generalizability may be limited. However, the program was implemented in a usual teaching setting during inpatient and outpatient clerkships, which may improve its generalizability. Further studies in other medical schools would be needed to estimate the feasibility and impact of this program with more certainty, and to understand how to best integrate the approach into different teaching programs.

Conclusion
Integrating an EBM EP into medical student clerkships is feasible, and students reported impact on patient care and EBM skills. Medical schools might consider this as one method to teach and evaluate EBM as required by the AAMC. These results may also be generalizable to other medical students nationally and internationally, but the program would need to be evaluated in additional settings to better estimate its feasibility and impact.