Defining new roles and competencies for administrative staff and faculty in the age of competency-based medical education

Abstract Purpose These authors sought to define the new roles and competencies required of administrative staff and faculty in the age of CBME. Method A modified Delphi process was used to define the new CBME roles and competencies needed by faculty and administrative staff. We invited international experts in CBME (volunteers from the ICBME Collaborative email list), as well as faculty members and trainees identified via social media to help us determine the new competencies required of faculty and administrative staff in the CBME era. Results Thirteen new roles were identified. The faculty-specific roles were: National Leader/Facilitator in CBME; Institutional/University lead for CBME; Assessment Process & Systems Designer; Local CBME Leads; CBME-specific Faculty Developers or Trainers; Competence Committee Chair; Competence Committee Faculty Member; Faculty Academic Coach/Advisor or Support Person; Frontline Assessor; Frontline Coach. The staff-specific roles were: Information Technology Lead; CBME Analytics/Data Support; Competence Committee Administrative Assistant. Conclusions The authors present a new set of faculty and staff roles that are relevant to the CBME context. While some of these new roles may be incorporated into existing roles, it may be prudent to examine how best to ensure that all of them are supported within all CBME contexts in some manner.


Introduction
The competency-based medical education (CBME) movement is requiring new approaches and practices of medical learners, teachers and educational leaders around the world Frank et al. 2017). Curricula have been reconfigured , assessment systems have been modernized Lockyer et al. 2017;Chan et al. 2021;Ross et al. 2021), and a new culture around workplace-based learning is emerging in locales where successful implementation has occurred Richardson et al. 2021;Thoma, Caretta-Weyer, et al. 2021). In most jurisdictions, CBME programs have newly defined functions compared to previous medical education systems (Leung 2002).
Many transitions and reforms in medical education to date have been aided by implementation of faculty development programs (Steinert et al. 2006;Cook and Steinert 2013;Steinert et al. 2016). During the early implementation of CBME, faculty development initiatives played a key role in helping with change management and energizing the teaching faculty to engage in new approaches to curriculum development and assessment system changes. However, within this early implementation period, the gaps of what faculty need to know and do, to allow CBME to flourish, have also been highlighted (Dath et al. 2010;Fraser et al. 2016;Nousiainen et al. 2017).
Many authors have called for increased faculty development to address the shifts required for CBME (Dath et al. 2010). Indeed, with large curricular changes in medical education, it would be foolhardy to believe that one could change culture and implement new learning approaches and programs of assessment without faculty development (Carraccio et al. 2016;Lupi et al. 2018). Moreover, with the implementation of CBME, there has been an advent of new assessment methods and curricular systems-requiring new expertise and competencies. CBME has ushered in the increased need for direct observation (Holmboe et al. 2010;Harris et al. 2017;Gauthier et al. 2018;Acai et al. 2019;Gottlieb et al. 2020;Young et al. 2020;George et al. 2021), feedback (Li et al. 2017;Shrivastava and Shrivastava 2019;Branfield Day et al. 2020;Mann et al. 2020;Hall et al. 2021) and coaching, workplace-based assessment (Carraccio et al. 2016;Harris et al. 2017;Sebok-Syer et al. 2018;Weller et al. 2020;Li et al. 2021), data aggregation (McConnell et al. 2016;Chan et al. 2017) and management, and the creation of learning analytics (Ellaway et al. 2014;Chan et al. 2018;Thoma, Ellaway, et al. 2021). As a result, new expertise and competencies are required of faculty.
And yet, there continues to be a paucity of empirical work to determine the roles and competencies that faculty members or administrative staff must fulfill in order to ensure the success of CBME systems. Indeed, many have pondered or highlighted the administrative burdens of CBME (Hawkins et al. 2015;Ali et al. 2017;Wagner et al. 2017;Cheung et al. 2022), but few have studied the requirements for supporting administrators to define or complete the new CBME-related tasks. To date, there is no consensus on what emerging skills and competencies are needed for successful implementation of CBME. Thus, it is important to engage stakeholders (e.g. trainees, frontline teachers, educational leaders) and CBME experts to define the new roles and competencies, with a mind to how it might improve and focus faculty/staff development going forward in CBME.
Through an international, modified Delphi study, this research seeks to determine and define the new roles and competencies required of administrative staff and faculty in the age of CBME.

Materials and methods
We conducted a five-round, modified Delphi study involving three stakeholder groups (international CBME expert members, CBME engaged/experienced faculty members, CBME engaged/experienced learners).

Conceptual framework
Aligned with our constructivist epistemology, our research team engaged in a multi-stakeholder modified Delphi process, similar to one previously described by Gottlieb et al. (2018) to engage both novices and experts simultaneously. Our rationale for this type of engagement was to ensure that wide-reaching perspectives and differing expertise related to CBME could be concurrently explored. While we wished to engage experts, it was critical to acknowledge and incorporate the experiential expertise of frontline educators and CBME learners who were engaging in these new systems and changes first-hand.

Study setting
This internationally distributed modified Delphi survey was associated with the International Competency-Based Medical Education (ICBME) collaborative. For context, this study was conducted during a global pandemic (from September, 2020 to January, 2022). During this time, CBME had been implemented in a number of jurisdictions to variable extents within Canada, United Kingdom, Netherlands, Taiwan, with several ongoing pilots in the United States, but had not yet been universally adopted within these jurisdictions. Of note, although we use a singular term for CBME, it should be acknowledged that the participants within the study all experienced a variety of approaches to CBME implementation and had varying degrees of firsthand or frontline experience with these approaches.

Participants
Prior to Round 1, we initiated contact via the listserv of the International Competency-Based Medical Education (ICBME) collaborative, rostering members of this listserv to participate. The listserv is an email list in which ICBME collaborators use to communicate. After rostering this group, we asked these experts to engage in a snowball nomination process where they were each asked to nominate faculty members and trainees, who had experience (e.g. >2 years of exposure) within a CBME context. This strategy allowed us to find participants in the following two intentionally sampled groups: (1) international CBME leaders; (2) frontline users with lived experience and practical expertise. We invited participants who completed at least two rounds to be listed as a collaborator on subsequent publication.

Design
There were five steps in our modified Delphi process (see Table 1 for details). Supplement Appendix 1 present the questions in each round.
In round 1, we gathered participant demographics, and asked them to reflect upon the new roles that they had noticed emerging within their local contexts or within the literature. These entries were then collated and grouped thematically initially by two of our team members (YY, TC) and subsequently, by the rest of our team.
In round 2 and 3 of the modified Delphi process, we sought to refine the list of roles via two rounds of successive voting. Round 2 required participants to endorse whether the roles identified in the first round were either: (a) novel roles that did not exist prior to the initiation of CBME; or (b) required a substantial shift in the traditional perception of the role. Participants were given the ability to rate each of the items with a Likert scale (1 ¼ Strongly Disagree to 5 ¼ Strongly agree) for inclusion of that role as required since the launch of CBME. Subsequently, in the third round, after providing participants with the report of endorsements from the second round, we asked the participants to re-vote. Round 3 of the Delphi was used as a confirmatory round to ask participants to simply agree or disagree with each item, similar to previous group consensus techniques Thoma et al. 2015;Chan et al. 2016;Gottlieb et al. 2017;Thoma et al. 2017;Stefan et al. 2019;Yilmaz et al. 2022). This process was also used for new items from round 3 which were voted on in Round 4. Items that did not reach at least a 75% endorsement by the participants were not advanced to the next round for further in-depth clarification .
In the final two rounds, participants were asked to elaborate (round 4) and then refine (round 5) the definitions and competencies within each of the new roles. The answers from round 4 were thematically analyzed, first by dyads within our team followed by refinements by consensus by the entire team through multiple rounds of iterative revision. Round 5 consisted of a member check whereby participants as experts of CBME who completed round 1 were sent our final analysis and asked to provide feedback, which was incorporated into the final list of new CBMErelated roles. Member check is a process to use validate the final results of qualitative analysis for credibility of the final analysis (Birt et al. 2016).

Development of the data collection tools
Each of the survey tools was co-constructed by the lead author (YY) and the senior author (TC). The tools for each phase were pilot tested by members of the team for usability and revised accordingly. Please see Table 1 for details of the collection tools and procedures.

Data collection and management procedures
The data were collected using the McMaster University Survey System (i.e. based on LimeSurvey which is an opensource software) at the following url: https://surveys. mcmaster.ca. This online survey tool is hosted by McMaster University ensuring that data are collected and stored in a secure database.
All participants were notified in advance that the survey would run over a series of months. Each round of the survey was sent to participants up to 3 times, and we tracked their completion to ensure that we only reminded those who had not yet completed the survey.
Based on each round's results, we adjusted the subsequent rounds' survey design, using the participants' voting patterns to provide feedback for the next round. Therefore, participants were able to review previous round's results with graphical representations of other votes, allowing them to reconsider and possibly make new choices when responding in the later round.

Quantitative
The McMaster Online Survey system has its own statistical procedures for descriptive statistics. We conducted simple descriptive statistical analyses using Microsoft Excel (Redmond, WA, USA).

Qualitative
We analyzed the answers to the open-ended questions using a generic qualitative approach (Kahlke 2014). We aggregated all de-identified answers from each participant into one dataset and parsed the participant's ideas into codes and then assembled these into themes. After that, we ensured trustworthiness for our analysis through conducting internal audits of our data analysis and conducted a member check to ensure that our participants felt our analysis represented their contributions.

Analytic team
Our team is a diverse group of investigators that all have varying degrees of interest and expertise in CBME. Some are researchers-scholars in the area (YY, MKC, DR, AA, LS, TC), some are frontline teachers in CBME systems (DR, MKC, AA, LS, TC, EB), and all had experience creating or implementing systems of CBME in various contexts.

Ethics
The Hamilton Integrated Research Ethics Board (HiREB) approved this study (protocol number #10613).

Results
The initial study announcement took place within the ICBME Collaborators group and members were invited to participate in the study through a registration survey. The registration survey requested information to identify Refinement Survey Contained bar graphs generated from the % endorsements from Voting Survey. After considering the report of the endorsements from the previous round, participants were asked to reconsider, 'Is this a new or novel role in the age of CBME?' Required a 75% endorsement for inclusion for the final round. 4 Free-text survey For each role which met the final a priori cut point of 75% endorsement in the prior round, we solicited the participants' thoughts via free-text open-ended questions about definitions of the roles and competencies required of these roles. 5 Free-text survey All participants were sent a copy of our final analysis and asked to provide feedback about where the analysis may have missed or misinterpreted their ideas.
others to participate in the study. In the first study announcement through the ICBME Collaborators listserv, 20 participants registered. These 20 individuals then nominated additional participants, who were then encouraged to nominate more themselves. Those nominated were invited to register for the study and to nominate others. We invited the nominees in an iterative fashion. A total of 210 potential participants accessed the registration survey and of those 80 registered for the study through the intake survey. All 80 registrants were invited for round 1 of the study with 68 completing round 1. The average age of the participants was 42.2 (SD ¼ 11.5) years. The majority of the participants were male (58.8%, n ¼ 40), and physicians (92.6%, n ¼ 63). There were 23 ICBME members/ experts, 30 frontline faculty members with CBME experience, and 18 learners with CBME experience, where participants were able to select more than one role. Their exposure to CBME ranged from 3.9 (SD ¼ 3.3) to 7.1 (SD ¼ 4.3) years.
Current practice jurisdictions of participants were diverse, with Canada (45.6%, n ¼ 31) and the USA (26.5%, n ¼ 18) being the most represented groups. Participants described themselves as mainly a national or international expert in CBME (39.7%, n ¼ 27), a frontline faculty member who rates learners (47.1%, n ¼ 32), and a frontline implementer of CBME who is leading (or has led) the implementation of CBME (44.1%, n ¼ 30). Table 2 shows the demographic details of participants. The supplemental digital Supplement Appendix 2 provides more details about the previous CBMErelated experiences of our participants in CBME.

Participation in the process
Free-text analysis of round 1 resulted in 24 CBME roles. After round 2, we added seven existing roles regarding CBME roles, which were suggested from the further freetext analysis of Round 2. Therefore, the participants rated 31 items. Based on the 75% threshold, seven existing roles were excluded from the study while the free-text answers from round 2 generated another five new roles which were included in the next round. We had high participation rates in all five rounds ranging from 78% to 97% (see Supplemental Digital Content for Appendix 3 which shows participation rates and the summary of items in all rounds).

The results of the modified Delphi process
A total of 36 roles appeared over the multiple rounds of study, however 12 of these roles were deemed to have already existed prior to CBME implementation in most regions (e.g. Program Director). See Supplement Appendix 4 for details of pre-existing roles. These 24 roles were subdivided into subsections within our survey: CBME Transition Roles; Learner Development; Assessors; Assessment System Facilitators; Competency Committee Roles; Other New Roles. For each role within each section, each item was rated on a 5-point Likert scale and their mean score was presented in Table 3. Only 13 of the 24 roles listed in Table  3 met the threshold for final inclusion.
Although well-established faculty and/or leadership roles (e.g. program director) originally were added to our list, subsequent voting eliminated these previously existing roles. The additional roles identified in round 2 were included in round 3, first rated on 5-point Likert scale and the next round asked either to be included or not. Overall endorsement rates are also reported in Table 3.
In round 4, the roles were re-presented to the participants, and we solicited descriptions of the competencies needed for each role that reached the threshold (75% endorsement in the final round). Figure 1 contains the complete list of new roles for faculty members and administrative staff in the age of CBME.
Finally, we sent the roles and their descriptions to all participants who completed at least 1 round (n ¼ 68) for the member check process. We incorporated the feedback from the participants for our final reporting within this manuscript. The infographic in Figure 1 depicts the 13 new roles and Supplement Appendix 5 contains the details of each role and their competencies, which was used to conduct a member check with our participants to ensure the trustworthiness of our analysis.

Discussion
Our modified Delphi process identified 13 new roles relevant to CBME implementation. Of these roles, the majority (10 roles) were related to roles best suited to faculty members and a minority (3 roles) were administrative staff-oriented roles.
Our findings intersect quite well with other recent literature around the core components of CBME. Van Melle et al. To fulfill the potential of implementing any CBME system, therefore, faculty development is needed to assist faculty in achieving these components. Among our 13 newly identified CBME-related roles for faculty and administrative staff, all of our new roles map to the core components (See Table 4).
With the introduction of increasingly complex systems such as CBME, the diversity of skills required for faculty members and administrative staff to support learners is substantial. New systems inevitably usher in substantive changes in the structures and processes, and with these new structures and processes come the need for new skill sets. To support systems change, faculty and staff development are required to support organizational growth. Interestingly, there were several roles that our participants endorsed that could be either faculty or staff positions, with many favouring administrative staff roles. This noteworthy finding suggests that moving forward continuing professional development in CBME should include both faculty and staff. This aligns with recent work suggesting that CBME may be ushering in an era of creating deliberately developmental organizations (Kegan and Lahey 2016), which would eliminate siloed development and ask all participating in academic health centres to collaborate and foster each other's development, regardless of rank or role (Thoma, Caretta-Weyer, et al. 2021).
While there is ample literature that affirms the need to develop faculty members to engage within academic settings for supervision and assessment Bearman and Ajjawi 2018), there are far fewer studies specifically aimed at illuminating the requirements for faculty development within a CBME context (Sirianni et al. 2020). What literature suggests is that faculty development is a core requirement to change prior systems into CBME-ready systems (Dath et al. 2010;Walsh et al. 2018;Stefan et al. 2019), but the how and the what to develop has yet to be clearly articulated. Some have highlighted a need for faculty to develop coaching skills, other literature has suggested that the missing link is faculty development in assessment principles (Holmboe et al. 2011), while other researchers have called for increased faculty development to assist with larger mandates like culture change (Griffiths et al. 2019;Hall et al. 2019). Indeed, a recent perspectives paper about jobs for educators in 2025 foretold that there would be a need for: (1) diagnostic assessors, (2) content curators; (3) technology adopters; (4) learner-centered navigators and professional coaches; (5) clinician role models; and 6) learning environment designers, engineers, architects, and implementers (Simpson et al. 2018). Interestingly, with the exception of the content curator (which is a very important role, especially for those who are acting as teachers), most of these proposed 'future jobs of medical educators' overlap with the roles within our final list.
Prior literature has shown that with changing mandates towards more competency-based frameworks, there is role confusion for faculty members (Li et al. 2021). This modified Delphi process elucidates competencies of new and evolving roles that are required within our CBME-based systems to ensure success. We hope the results of our study will bring increased clarity for frontline faculty members and leaders attempting to design or provide faculty/staff development. By describing several of the new roles expected within CBME-based systems, we anticipate clarity for those seeking to enhance CBME skills of their faculty and administrative staff should perceive decreased frustration and resistance to the implementation.

Limitations
Our study has several limitations that must be considered. First, our modified Delphi sought to involve an international cross section of both experts and frontline users of CBME, and therefore was biased towards physicians/learners who had engaged in CBME systems. Caution is required when generalizing our results towards other fields (e.g. veterinarian medicine, dentistry).
Secondly, our study only sought to identify new roles that have been precipitated by the implementation of CBME and did not seek to further clarifying changes to pre-existing or 'traditional' roles (e.g. program director). We foresee that this exploration would be an interesting follow-up study and would be interesting to also elucidate if any of our newly proposed roles have been incorporated into previously existing roles (e.g. program directors assuming a coaching role, administrative assistants taking on an IT lead role). We anticipate that there may be high variation based on the availability of resources both locally, regionally, nationally and globally.
Finally, we did not include administrative staff (e.g. program coordinators and other staff roles) in our consultatory process. As such, we may have been missing key insights from these individuals. Futures studies to clarify their perspectives on CBME and other expertise is required.

Future directions
While we do not expect that all CBME implementation efforts will require all 13 roles that have been identified, each CBME implementation will have its own resource restrictions and limitations, as well as varying central resources (e.g. if your IT team is strong and well-funded within an institution, they may not need to have a dedicated CBME IT lead). We do believe that in most jurisdictions, some variation of all these roles will be required to support the successful implementation of CBME. As such, we must ask ourselves: How might we best support the faculty and administrative staff development within systems to enable teachers, learners, and administrators to flourish within CBME contexts? Should we use these competencies as a means to shift towards competency-based faculty and staff development? Do these roles manifest similarly across all contexts or will they appear different in various versions of CBME? Have some of the present 'new' roles been amalgamated into pre-existing roles-and how could they be better amalgamated? Will these new roles overly burden faculty members and staff and impact patient care? These questions remain to be answered and we would encourage the field to take up the mantle to investigate them.

Conclusions
Along with our collaborators within this modified Delphi study, we have articulated 13 new roles (and some associated competencies for the roles) that we feel will support successful CBME implementation and execution. While these roles or competencies may ultimately be combined with other existing roles, we feel that highlighting them will be useful for those seeking to develop faculty members and administrative staff during the initial implementation and sustaining phases of CBME systems.