A model of four hierarchical levels to train Chinese residents' teaching skills for "practice-based learning and improvement" competency.

Abstract Objectives: The current study focused on validating a protocol for training and auditing the resident’s practice-based learning and improvement (PBLI) and quality improvement (QI) competencies for primary care. Methods: Twelve second-year (R2), 12 first-year (R1) and 12 postgraduate year-1 residents were enrolled into group A, B and C, respectively, as trainees. After three training protocols had been completed, a writing test, self-assessed questionnaire and mini-OSTE and end-of-rotation assessment were used in auditing the PBLI competency, performance and teaching ability of trainees. Results: Baseline expert-assessed PBLI and QI knowledge application tool writing scores were low for the R1 and R2 residents. After three training protocols, PBLI and QI proficiencies, performance and teaching abilities were improved to similar levels cross the three training levels of residents based on the expert-assessed writing test-audited assessments and on the faculty and standardized clerk-assessed end-of-rotation-/mini-OSTE-audited assessments. Conclusion: The different four-level hierarchical protocols used to teach group A, B and C were equally beneficial and fitted their needs; namely the different levels of the trainees. Specifically, each level was able to augment their PBLI and QI proficiency. This educational intervention helps medical institutions to train residents as PBLI instructors.


Introduction
Quality improvement (QI) is the common element and goal of practice-based learning and improvement (PBLI) and system-based practice competencies in medical education [1,2]. After entering postgraduate training, residents are expected to take up responsibility for teaching ACGME (Accreditation Council for Graduate Medical Education) six-core competencies, including PBLI/QI to young clerks [3]. Research has documented that carrying out teaching also helps residents to learn [4]. The ACGME and American Board of Medical Specialties require the PBLI/QI competency for board certification in all residency specialties [1,5,6]. Questionnaires distributed by our institution have reported that there are problems teaching the PBLI/QI competency to young physicians, effectively.
The fundaments of the PBLI and QI competencies, examining current practices and applying a systemic and evidence-based approach to improving primary care qualities, overlap [7]. Medical education is now placing emphasis on adequate training of every young physician at the beginning of their carriers in PBLI/QI competencies. However, improving familiarity with the PBLI/QI competencies requires knowledge, theory and experience-based strategies that are practiced repeatedly. Accordingly, such training responsibility is a challenge to clinical instructors who are fully occupied by clinical practice, research and other education responsibilities. It is therefore not surprising that the need has been increasing steadily for an instructor training program for residents.
Being closer in age to the clerks, the near-peer status of residents makes them more approachable when there is a need to ask "stupid questions", compared with a faculty member [8]. Thus, optimizing the resident's mastery of the PBLI/QI competencies and helping the residents as teachers might help to solve the poor promotion of these two domains among clerks.
Using a comprehensive approach, our study aim was to establish and validate a four hierarchical levels-based protocol for training and auditing the teaching abilities of residents for PBLI/QI competencies.

Materials and methods
The detail description is shown in the Supporting Materials and Methods.

Grouping
From 2013 January to 2014 October, 12 second-year residents (R 2 ), 12 first-year residents (R 1 ) and 12 postgraduate year-1 (PGY 1 ) residents were enrolled to form groups A, B and C of this study, respectively (Figures 1 & 2). Ethical approval was obtained from the Ethics Committee of our institution, and care was taken to apply the World Medical Association Declaration of Helsinki principle of research.

Recruitment and training of faculty raters and standardized clerks
Fourteen faculty raters who had wide experience of teaching and auditing PBLI/QI competency of trainees were invited from another medical center to take part in this study [9]. We recruited 12 standardized clerks and trained each for 2 h in role playing. After 2-h consensus sessions of viewing the videotapes of the pilot scenarios, 2 of the 14 faculties were in charge of training of the PBLI/QI teaching skills of the residents via workshops.

Design
The four-level hierarchical protocol for training and auditing of the trainee's clinical teaching skills consisted of, "knows, knows how, shows how and does". Accordingly, our whole protocols were designed to follow the four-level hierarchical protocol. Within the "group B" protocol, the R 1 residents received a 1-h interactive lecture which followed by given by 30-min of opened discussion guided two qualified faculty members (Figures 1 & 2). Within the "group C" protocol, the PGY 1 residents receiving a 30-min audio/visual presentation by qualified faculty (this targeted the "knows" level of the four-level hierarchical protocol). Over the next 30 min, small group role playing and debriefing regarding the PBLI's teaching skills were undergone (this targeted the "knows how" level of the four-level hierarchical protocol). Subsequently, a 30-min. real-time classroom demonstration on how to teach PBLI/QI competency was arranged (this targeted the "shows how" level of the four-level hierarchical protocol). We believed that the group A trainees (R 2 residents) had undergone long-term exposure to clinical practice and this was able to drive them to learn and carry out PBLI spontaneously. So, the senior group A trainees (R 2 residents) directly underwent auditing of their PBLI teaching skills (this targeted the "does" level of the four-level hierarchical protocol) via mini-OSTE (Figures 1 & 2). Figure 1. The training and auditing four-level hierarchical protocols for different trainees (group A for R 2 residents, group B for R 1 residents and group C for PGY 1 residents) for clinical teaching.
Auditing the first and second "knows and knows how" levels of hierarchical protocol Before and after finishing the PBLI workshop and the mini-OSTE, all trainees (PGY 1 /R 1 /R 2 ) were asked to complete the PBLI and QI knowledge application tool (QIKAT-R) writing test within 15 min, which included three questions related to two clinical scenarios [7,10,11]. Clinical scenarios and scoring system of PBLI and QIKAT-R writing test were according to previous studies [7,9]. Briefly, QIKAT-R answers were scored from 0 (low) to 5 (high); thus the three answers generated a cumulative score that could range from 0 to 15 points (Supplement Table 1) [1,12,13]. The average scores of PBLI/QI scenarios were converted into percentages for further analysis; these were called the PBLI/QI writing scores. Simultaneously, three experts individually scored the trainee's 36 PBLI and QIKAT response to hypothetical clinical scenarios. After the initial scoring, the three experts met to resolve substantial differences in their assigned ratings. Then, all residents were asked to fill-in a self-assessed questionnaire for PBLI proficiency (Table 1).

Auditing the third "show how" level of hierarchical protocol
Based on a definition of PBLI (Supplement Table 2), the previous valid clinical scenario was arranged for residents to teach disorganized clerks how, in a case with high risk for deep vein thrombosis, to initiate and maintain safe, cost-  efficient and effective inpatient and outpatient anticoagulant [1,7]. This PBLI mini-OSTE is characterized by reflecting the resident's daily responsibilities and could be observed in a limited time.
Within every 120 min, 12 qualified faculty raters, who were blinded to the training levels of residents, were randomized and then audited the trainee's PBLI teaching abilities using 12 standardized clerks (Figure 3). At each mini-OSTE station, residents had 8 min to perform the teaching task to the standardized clerks within the PBLI scenario. Faculty raters, the standardized clerks and the PGY 1 /R 1 /R 2 residents themselves (post-training) all independently completed distinct rating forms over 2-min immediately following each encounter. Then, the trainees received 5 min of feedback from faculty and standardized clerks and 20 min of debriefing on the PBLI scenarios. They then filled-in a questionnaire that addressed the case difficulty and the educational value of the auditing (Supplement Table 1).

Auditing the fourth "does" level of hierarchical protocol
Thirty-six clinical teachers that the PGY 1 , R 1 and R 2 residents worked with were asked to rate the pre-training and post-training end-of-rotation PBLI scores. At monthly intervals, individual end-of-rotation PBLI scores were the average of the global "patient care" performance PBLI (Supplement Table 2), mini-CEX and case-based discussion scores using previous valid forms [7,14]. Finally, the average of the pre-training and post-training two monthly end-of-rotation PBLI scores was converted into single pre-training and post-training end-of-rotation PBLI percentage scores for comparison between groups.

Data analysis
We assessed scale reliability by calculating Cronbach's coefficients mean scores for the specific teaching scores [15,16]. We compared the global scores by resident training level (PGY 1 , R 1 and R 2 ) using independent t tests and between raters using c 2 tests. Overall pre-training and post-training QIKAT-R scores among group A, B and C were analyzed using mixed-model ANONAs.

Scale reliability and validity
Cronbach's coefficients for the specific teaching competency scores ranged from 0.78 to 0.92. All mean teaching skills scores significantly were correlated with the global faculty ratings, most with an r > 0.6 and a p < 0.01. Faculty global teaching performance scores correlated significantly with trainees' self-evaluation (r = 0.384; p = 0.002) and score (r = 0.392; p = 0.001), which supports convergent validity.
The first and second "knows and knows how" levels In Figure 4A & B, the pre-training PBLI writing test scores were relatively low for the R 1 and R 2 residents. As shown in Figure 4A, the expert-assessed PBLI writing scores were significantly improved after either the mini-OSTE (R 2 residents) or the workshop + mini-OSTE (R 1 residents) training, respectively. Consequently, similar post-training PBLI scores were noted among the PGY 1 , R 1 and R 2 residents. Notably, the pre-training QIKAT-R writing scores were quite low across PGY 1 , R 1 and R 2 residents. However, the post-training QIKAT-R writing scores were improved after three different training protocols as shown in Figure 4B. Notably, the degree of improvement in QIKAT-R writing scores was larger than for the PBLI writing scores among PGY 1 , R 1 and R 2 residents. These results suggested that our three protocols had been appropriately design based on the baseline evaluation and adequately filled the insufficiency in PBLI/QI proficiency of the various different training levels of residents.

The third "show how" level
In Table 2, the PBLI teaching skills assessed by faculty indicated that PGY 1 residents are still not very sophisticated in the "needs assessment" and "instructural skills" compared with the R 2 residents. However, the self-assessed PBLI mini-OSTE scores were significantly lower in PGY 1 compared with the R 2 residents. These results indicated that the PGY 1 residents had lack of confidence for their PBLI teaching skills than senior residents (R 1 and R 2 residents).

The fourth "does" level
Significantly, the pre-training end-of-rotation PBLI scores were higher for the PGY 1 residents than for the R 2 and R 1 residents ( Figure 4C). After training, the percentage of improvement in the end-of-rotation PBLI scores was significantly higher among the R 1 residents (receiving lecture-based workshop + mini-OSTE) than among the PGY 1 residents (receiving role-playing workshop + mini-OSTE). Notably, a similar percentage of improvement in end-of-rotation PBLI score was noted for the R 2 residents (not receiving workshop + mini-OSTE) and R 1 residents, who have a different number of years of clinical experience ( Figure 4C).

Self-assessed PBLI proficiency
In Table 1, the baseline PBLI proficiency was significantly low for the R 2 residents compared with the PGY 1 residents. The most significantly different PBLI item between PGY 1 and R 2 residents was "using evidence from scientific studies and information technology". Effectively, these items were improved after the PGY 1 and R 2 residents underwent adequate training. Consequently, the final overall PBLI proficiency scores were similar among PGY 1 , R 1 and R 2 residents groups even though each of these groups had received different training and auditing protocols.

Improving trend for the multisources-assessed PBLI/ QIKART-R scores
It is obvious that the PGY 1 residents benefited the most from the PBLI training/auditing protocol due to their participation in the workshop and formative mini-OSTE compared with the R 2 residents who received their training in real clinical environment ( Figure 4D). In addition, the magnitudes of the improvements in QIKAT-R writing scores were markedly higher than the PBLI score for all three different training levels of trainees receiving three different protocols ( Figure 4E). These results suggest that the general improvement in QIKAT-R writing scores among our trainees was the consequence of the simultaneous augmentation of their PBLI proficiency levels.
Trainee's perception for the case difficulty and educational value of the assessment In general, all trainees (standardized clerks, PGY 1 , R 1 and R 2 residents) reported that the PBLI clinical scenario was difficult for them ( Figure 4F). In terms of the PBLI clinical scenario, the R 1 residents reported that they benefited the most from this scenario compared with others.

Discussion
Interestingly, there were significant correlations among PBLI competency and performance ratings from all three perspectives, namely the clinical teachers, the experts and the residents themselves. PBLI proficiency among the R 2 , R 1 and PGY 1 residents was negatively related to their clinical experience based on our baseline assessment. Therefore, the knowledge, skills and ability of teaching and practicing PBLI competency should be re-emphasized between senior residents.
Available methods for assessing teaching skills include self-evaluation, learner evaluations, video analysis by peers, co-teaching and OSTE [17][18][19]. An OSTE is a multi-station exercise using standardized learners to assess teaching skills of the candidate instructors. The OSTE stations can be designed to provide feedback, to teach teaching skills, to create classroom, to allow group discussions, etc. [19]. The main advantages of an OSTE include a facilitation of outcomes-based educational research. Based on the premise "to really know a subject, teach it", we designed teaching skills OSTE stations for the "show how" level training of all trained residents. For the R 1 and PGY 1 residents, the mini-OSTE teaching and application simulations facilitated their ability to transfer workshop-trained PBLI knowledge and skills into the clerks in stressful clinical setting and involved patients.
Immediate collection of data by a paper-and-pencil writing test is able to reduce lost and incomplete data [17,19,20]. In our study, the immediate post-training self-evaluated and expert-evaluated written test helped the curriculum designer to audit PBLI knowledge acquisition and retention of all residents across two hierarchical levels "know or know how" [21] Our trainees reported that the post-training assessment and feedback provide self-reflection opportunities related to their readiness for practicing and teaching PBLI/QI knowledge and skills [22]. The post-training end-of-rotation clinical teacher evaluations provide an opportunity to check the longterm effects of PBLI/QI knowledge acquisition, transfer and retention on the daily practice of our trainees [23][24][25].
Self-reported confidence may not indicate actual competence or performance. Nonetheless, trainees who reported not feeling confident are hardly ready to meet the needs of patients and society. Multi-sources assessment is able to avoid self-reported and single observer bias [16]. Nonetheless, there will be some gaps between the self-reported confidence, and  faculty/expert-assessed competence and the individual's performance [26,27]. In our study, self-reported confidence, formally assessed competence (what trainees can do in a controlled situation) and performance (what trainees really do in actual practice) were simultaneously evaluated by trainees themselves, by the faculty raters/experts and by the clinical teachers. Intriguingly, all of our trainees rated the PBLI workshop and participation in the formative mini-OSTE as being the most helpful dimensions that delivered knowledge to the clerks and allowed interaction with the faculty raters, who are giving constructive feedback. In comparison with our largely lecture-based approach (group B, R 1 residents) and the small group role-playing and videotape watching approach (group C, PGY 1 residents), the formative mini-OSTE simulated the residents' internal desire to apply their learnt skills at their hospital [28]. In other words, this project will encourage more young residents to share their experience of being taught to teach and this will develop their skills overtime. In our study, the scoring rubric and valid clinical scenario for resident's PBLI/QIKAT-R writing test, and PBLI teaching skills mini-OSTE are according to previous studies [7,9,12,13]. Good validity of the items of PBLI mini-OSTE was noted in our study. Additionally, the residents' end-ofrotation assessment by clinical teachers was based on the previous valid checklist-based forms [7,29]. In general, all cases and scoring systems used in the current study had undergone pilot and external testing with good validity previously [2,7,9,12,15,29]. Nonetheless, specific PBLI/QI-based assessment tools should still be validated in future large-scale study design before wide application.
Indeed, PBLI is the strategy to achieve the QI of patient care in the healthcare system. The effectiveness of our PBLI training program was audited using a simulated teaching and clinical environment that included a formative mini-OSTE and a QI writing test. Notably, this four-level hierarchical PBLI training/auditing protocol had additional beneficial effects on the enhancement of QI competency and performance of all trainees. Overall, this study was characterized by its use of a multi-faceted approach to the training and auditing of the PBLI/QI competency across different levels of residents. This was a single-center study that had a narrow focus and this may limit the external validity of our results. Based on our initial positive results, a large-scale study can be undertaken in the future to validate the efficiency of this four hierarchical levels-based training and auditing approach. Different to the faculty raters involved in the formative mini-OSTEs, the end-of-rotation clinical teachers were not blinded to the residents training level in our study. Nonetheless, we avoid the possible confounding effects by consensus building calibration, and standardization of the faculty rating system using a checklist of videotapes examples of appropriate PBLI/QI performance during clinical practice. Another limitation of this study is the lack of assessment as to how the resident's learnt PBLI/QI knowledge and skills affected patient outcomes [18,29]. Finally, all our trainees were motivated learners and their baseline knowledge of and skills of PBLI/QI might be superior to others residents in a similar situation. In other words, a general survey needs to be undertaken in the future to understand the baseline knowledge of whole population of young residents. In contrast to web-based programs, the teaching during clinical practice of the PBLI domains make the QI experience more long-standing and beneficial for residents and clerks.

Conclusion
Both junior and senior residents can be effectively trained to teach the elements of PBLI and QI using a comprehensive four hierarchical levels-based protocol.