Effects of candidates’ demographics and evaluation of the virtual Multiple Mini Interview (vMMI) as a tool for selection into paediatric training in Queensland

Abstract Introduction The Queensland Basic Paediatric Training Network (QBPTN) is responsible for the selection of candidates into paediatric training in Queensland. The COVID-19 pandemic necessitated interviews to be conducted ‘virtually’ as virtual Multiple-Mini-Interviews (vMMI). The study aimed to describe the demographic characteristics of candidates applying for selection into paediatric training in Queensland, and to explore their perspectives and experiences with the vMMI selection tool. Methodology The demographic characteristics of candidates and their vMMI outcomes were collected and analysed with a mixed methods approach. The qualitative component was comprised of seven semi-structured interviews with consenting candidates. Results Seventy-one shortlisted candidates took part in vMMI and 41 were offered training positions. The demographic characteristics of candidates at various stages of selection were similar. The mean vMMI scores were not statistically different between candidates from the Modified Monash Model 1 (MMM1) location and others [mean (SD): 43.5 (5.1) versus 41.7 (6.7), respectively, p = 0.26]. However, there was a statistically significant difference (p value 0.03) between being offered and not offered a training position for candidates from MMM2 and above. The analysis of the semi-structured interviews suggested that candidate experiences of the vMMI were influenced by the quality of the management of the technology used. Flexibility, convenience, and reduced stress were the main factors that influenced candidates’ acceptance of vMMI. Perceptions of the vMMI process focused on the need to build rapport and facilitate communication with the interviewers. Discussion vMMI is a viable alternative to face-to-face (FTF) MMI. The vMMI experience can be improved by facilitating enhanced interviewer training, by making provision for adequate candidate preparation and by having contingency plans in place for unexpected technical challenges. Given government priorities in Australia, the impact of candidates’ geographical location on the vMMI outcome for candidates from MMM >1 location needs to be further explored.


Introduction
Postgraduate medical trainees are simultaneously workers and learners in health care organisations.In Australia and New Zealand, The Royal Australasian College of Physicians (RACP) sets the curriculum, teaching and learning program and the assessment program for paediatric and physician training (Royal Australasian College of Physicians 2020a).The RACP provides recommendations on selection methods for selecting trainees to programs at the state level (Royal Australasian College of Physicians 2020b).The Queensland Basic Paediatric Training Network (QBPTN) is responsible for the coordination of selection and recruitment procedures into paediatric training in Queensland.
Selecting the optimal candidates for health professional programs is challenging, and various tools have been employed in the past.Selection methods need to identify potential candidates based not only on academic abilities, but also non-academic, profession-specific attributes (Patterson et al. 2016;Roberts et al. 2018).The selection tools utilised should facilitate the selection of candidates most likely to complete the training program and practise as competent specialists in the future (Patterson et al. 2013).
One of the selection tools utilised is the face-to-face Multiple Mini Interview (FTF MMI).It was first developed at McMaster University in Canada (Eva et al. 2004).Since 2004, FTF MMIs have been adopted by many educational organisations in order to help select students into undergraduate and postgraduate medical training programs (Dowell et al. 2012;Pau et al. 2013;Patterson et al. 2016;Rees et al. 2016).The RACP also recommends the use of MMIs in preference to traditional panel interviews for the selection of candidates into paediatric training (Royal Australasian College of Physicians 2020b).Since its introduction, there have been several systematic reviews of the utility, feasibility, reliability, and validity of MMI in medical school and postgraduate specialist candidate selection settings (Pau et al. 2016;Rees et al. 2016;Roberts et al. 2018).Generally it is accepted as a reliable and valid selection tool (Ali et al. 2019).
The COVID-19 pandemic has significantly affected selection processes, medical education, and post graduate training (Cleland et al. 2020).Many institutions have moved to virtual formats to optimise participant safety.A scoping review of virtual MMIs (Sabesan 2022) concluded that vMMI was a feasible, acceptable, and reliable selection method for selecting candidates into medical school, residency, and fellowship programs.However, the validity of vMMI needed to be further explored (Sabesan 2022).
The success of rural training experiences for medical students has been well documented, with studies showing the success of rural training leading to recruitment to medical careers based in regional and rural locations (Worley et al. 2008;Sen Gupta et al. 2013;Woolley et al. 2019).Similarly, recruitment of medical students from rural locations has shown that both rural origin and rural clinical school experience encourage subsequent rural recruitment (Seal et al. 2022).However, recruitment of postgraduate trainees has not been investigated from demographic, medical school, or origin perspectives.The demographic characteristics of candidates selected by the vMMI format had not been evaluated.
The aim of this study was to describe the demographic characteristics and the impact on vMMI outcome of candidates applying for selection into paediatric training in Queensland and to explore their perspectives, experiences, and acceptability of the vMMI model of selection.

Methodology
A convergent mixed methods approach was used for this study (Creswell and Plano Clark 2011), emphasising a student-centred approach, and featuring the use of technology for assessment and evaluation of impact on candidates' experiences (Henderson et al. 2017).Quantitative and qualitative data were collected and analysed independently, and the results were triangulated and interpreted (Busetto et al. 2020).De-identified and routinely collected recruitment data for all eligible candidates who applied to the program was utilised.The demographic data of eligible, shortlisted for interview and of candidates offered a training position were compared at various stages of selection.The internship year was considered as Post Graduate Year 1 (PGY1).The degree of remoteness of a candidate's current location was determined using the Modified Monash Model (MMM), and was graded from MMM1 (major city) to MMM 7 (very remote) (Australia.Department of Health and Aged Care 2021).SPSS Version 28.0.was used to analysis the data.A chi-square test was employed to determine the associations between categorical variables.The T test (for two groups) or ANOVA analysis (three or more groups) were carried out to determine the differences between groups for scores.A p-value of <0.05 was considered significant.
All candidates selected for the vMMI were invited to participate in the qualitative component of the study.The Microsoft TeamsV R platform was used to conduct the interviews with consenting candidates.Semi-structured interview questions were based on a previous study (Sabesan et al. 2022), and related to students' experiences with preinterview instructions, processes, technical issues, perceptions of vMMI and areas for improvement.Each interview was video recorded then transcribed verbatim.Data from the interviews were de-identified prior to tabulation.
In order to reflect candidates' lived experience of vMMI, the qualitative data was analysed using an inductive approach to semantic thematic analysis (Braun and Clarke 2006).NVivo software was used for coding, categorising, and analysing the data from the semi-structured interviews.To ensure integrity, the coding of identified themes was confirmed using shared coding sessions between the researchers.

Eligibility
Eligibility for the vMMI required candidates to achieve general registration with the Australian Health Practitioner Regulation Agency (Australia.Department of Health and Aged Care 2022) and be in Post Graduate Year Level (PGY) 3 or above in 2022.There were 131 eligible applicants, and, after a structured review of their resumes, references, and a short statement by one of the four Directors of Paediatric Education, a total of 71 candidates were

Practice points
With the exception of MMM status (geographical location), demographic characteristics did not influence the selection outcome; vMMI mean scores for candidates from MMM1 and others were not statistically different; Candidates' vMMI experiences were related to the management of technology during the vMMI process and upon building rapport with the interviewers; The improved work flexibility, plus the cost and time savings it allows, position vMMI as a more appropriate alternative to FTF MMI overall, and particularly in the current circumstances; and Implementation of improvements to the operation of vMMI such as the development of interviewer training and the establishment of contingency plans for anticipated technical issues may show vMMI to be a viable and cost-effective alternative to FTF MMI beyond the COVID-19 pandemic.

Interview: virtual MMI (vMMI)
In the vMMI, applicants rotated every eight minutes around the virtual stations.The process comprised two minutes to read the scenario, five minutes for the interview and one minute to be moved to the next station by 'the host'.The vMMI contents of each station were created to reflect the domains of the RACP professional practice framework (Royal Australasian College of Physicians 2020c).In RMO 2022, communication, quality and safety, teaching and learning, ethics and professional behaviour, leadership, management and teamwork, and medical expertise were assessed.These were chosen from the QBPTN's peer reviewed FTF MMI question bank (without alterations), in order to maintain construct and content validity.Questions were either 'past behaviour' or 'situational' questions, or a mixture of both.Each station had two to three prompt questions.All six stations were marked by an interviewer who had been trained and calibrated to the process and who was familiar with the Zoom V R platform.

Results of quantitative data
Of the 131 eligible candidates, 71 underwent vMMI and 41 were offered a training position (Table 1).Offers were made based on total shortlisting and vMMI scores on merit (Supplementary Appendix 1).
The demographic variables did not differ between those shortlisted when compared to those not shortlisted (Table 2).Of those shortlisted, a higher proportion (65 vs 35%; p ¼ 0.03) of candidates from MMM1 location were offered a position (Table 3).

Results of qualitative data
After completion of the selection and notification processes, of the 71 candidates seven took part in a semi-structured interview on their experiences with vMMI.This group included candidates who were not offered a training position (n ¼ 3), male gender (n ¼ 2), PGY Level 3 (n ¼ 5), PGY 4 (n ¼ 2), MMM1 (n ¼ 5) and MMM2 (n ¼ 2).The qualitative results were presented across three major themes comprising candidates' experiences, acceptability of the vMMI process, and perceptions of vMMI.

Experiences of vMMI
Management of technology included pre-interview preparation and improved technical processes on the day to facilitate seamless interview movement and experiences for participants.Six candidates reported positive experiences of vMMI regardless of whether they had been offered a training position or not.The pre-interview information included a description of the vMMI process and technical instructions for using Zoom.Candidates found this information very useful and felt they knew what to expect on the day.Candidates were offered pre-interview testing of their devices, cameras, audio, and internet on two different occasions.It was not possible for all the candidates to attend, but the candidates who did attend found it very useful.
The procedural and technological aspects of vMMI were also important factors in determining perceptions and experiences on the day.Candidates were moved between the virtual interview rooms by a staff member-'the host'.This was a positive experience for the candidates, with one commenting 'I didn't have to physically change myself between each of the stations.Someone had done that for me.' (099).The questions were shared by the interviewer during the two-minutes pre-reading time and remained on the screen.Candidates found it useful to have the questions on the screen to refer to while answering questions.
Most of the candidates (n ¼ 6) did not experience technical issues, regardless of their interview location.For example, 'I think the whole process went really smoothly and there are no lags.'(055), 'I didn't have any issues with my internet connection.On the day I just did it [the interview] from home.' (061).Some candidates reported negative experiences with the technology, including missing the pre-warning and being moved abruptly.
When reflecting on their experiences of vMMI and communication, candidates reported different experiences with communication and developing rapport with the interviewer on Zoom.Some commented that it would have been easier to communicate in person, however with modern high-resolution camera and fast internet, they did not consider this a major barrier.One candidate said, 'If you are speaking to a person face-to-face you get the posture and body language.But I feel like with the high-quality cameras it doesn't make much of a difference.'(058).
Another candidate who was apprehensive about the vMMI stated, 'I would have said that building the rapport would have been my main concern, but now that I've obviously gone through it, I don't see that as a big issue at all.' (099).Rapport building was an issue for one candidate who had internet connection difficulties, resulting in her doing the interview from her iPhone.She stated, ' … hard to know via zoom whether I was heading down the right path, purely connectivity issues and unable to kind of read the body language.'(040).Candidates also commented on loss of rapport when the interviewers were looking down and writing notes or when they disappeared from the screen into the virtual background.

Acceptability of vMMI
Candidates noted that stress related to the vMMI process was lessened due to their ability to do the interview from home or from another familiar environment, not having to travel to a central location and not having to physically move between rooms as in FTF MMI.Candidates who underwent FTF MMI for medical school selection felt that the vMMI experience was like FTF MMI.A candidate who participated in FTF MMI for medical school selection stated, 'It wasn't far off the experience of in person.' (038).Candidates commented that the time saved by not having to travel to a central point and consequent reduction in time off work were the main factors that improved flexibility.The following quote supports acceptability of vMMI due to flexibility related to travelling to a central point, 'It saves time for candidates to actually travel to the location for one-to-one interview' (058).Furthermore, during the COVID-19 pandemic, vMMI enabled candidates from other states and regional towns to participate in the selection process.

Candidates' perceptions of vMMI and improvements
Overall, perceptions of vMMI were positive, with candidates saying that they would choose vMMI in the future.One candidate felt vMMI may be better than FTF MMI, 'In terms of time and effort, and the whole experience, I feel like virtual is probably a little bit better than face-to-face' (058).Regardless of whether the candidates had attended the pre-testing or not, they recommended that pre-testing should continue to be offered in the future.
Candidates expressed some ideas on how the process and experiences may be improved in the future.The main suggestions concerned how to overcome difficulties by pre-testing and management of technological aspects.Technological improvements were suggested by all interviewed candidates, including those who had excellent experiences on the day.Those suggestions included a virtual orientation to the vMMI process from the perspective of candidates, conduct of vMMI and troubleshooting processes on the day.Some candidates remarked that on occasion they were moved abruptly before they had a chance to say thank you and goodbye to the interviewer, and it was mainly due to not noticing the 1-minute pre-warning.To improve this, one candidate suggested having an audible pre-warning to draw participants' attention.Having a rest station (in-between moving to the next station) was repeatedly suggested to avoid two candidates being in the same virtual room.

Discussion
Using data from the 2021 QBPTN selection process, we have shown that majority of trainees selected were females (78%), had obtained their primary medical degree in Australia (93%) and were living in an MMM1 location (85%).To the best of our knowledge, this is the first study to report the demographic characteristics of candidates selected into postgraduate paediatric training in Australia, and to explore their perspectives, experiences, and acceptability of vMMI as a selection method.Analysis of the demographic characteristics of candidates for gender, Australian/New Zealand citizenship, English as primary language, primary medical degree from Australia and post graduate year level recorded no difference in the outcome between the groups.Once shortlisted for a vMMI, a lower proportion of candidates from rural or remote areas were finally selected.Major factors that influenced candidates' experiences and perspectives of the vMMI selection process were related to the management of technology and communication, and rapport building with the interviewer over the online platform.
The demographic characteristics of candidates in the current study were proportionally represented at various stages of the selection process.Overall, studies of FTF MMI found that age, gender, and economic status did not appear to influence the outcome (Reiter et al. 2012;Rees et al. 2016).This corresponds with the vMMI outcome of the current study.The current literature suggests that FTF MMI may disadvantage rural and Aboriginal candidates in medical school interviews (Reiter et al. 2012;Raghavan et al. 2013;Rees et al. 2016).It was reassuring to note that the shortlisting and vMMI mean scores for candidates in MMM1 and others were very similar (p value 0.77 and 0.26 respectively).Although the mean scores for the shortlisting [MMM1 25.3 (2.8) versus others 25.0 (3.8)] and vMMI [MMM1 43.5 (5.1) versus others 41.7 (6.7)] were not statistically different, there is a slight positive difference favouring the MMM1 group in both section which has a practical significance.The higher mean for MMM1 relative to the > MMM1 group has resulted in a greater number of MMM1 candidates being offered a training position.This is due to the closeness of the scores and a small advantage will improve the chance of meeting the selection cut off.
There is a decline in health outcomes when rural, regional, and remote patients in Australia are compared to their more urban counterparts (Australian Institute of Health and Welfare 2022).The rate of specialists per head of population also declines with increasing remoteness, from 143 per 100,000 population in major cities to 22 per 100,000 population in very remote areas (Australian Institute of Health and Welfare 2022).Previous studies have shown medical graduates are more likely to work in rural areas if they have a rural upbringing (McGrail et al. 2011) and/or early exposure to rural and remote clinical experience (Ray et al. 2015).A snapshot of the Australian population in 2021 shows that 72% of Australian population live in MMM1, 9% in MMM2 and 19% in MMM3 and above (National Rural Health Alliance 2021).However, 85% of candidates who were offered a training position were working and living in an MMM1 location at the time of application.Current shortlisting criteria assign extra weight to rural upbringing and regional experience, despite which higher proportion of applicants continue to be from an MMM1 location.This study did not specifically look at rural upbringing and previous regional experience of the group who were selected to the program.
Unsatisfactory experiences were related to technological issues and were not necessarily limited to those candidates living outside capital cities.Management of technology and the ability to build rapport with the interviewers on Zoom were important factors contributing to the overall experience.Experiences were unsatisfactory if disruptions occurred due to a poor internet connection or due to other factors such as the interviewer not being able to maintain 'eye contact' with the candidate.Inability to build rapport with interviewer was identified as one of the challenges in previous studies about virtual panel interviews (Asaad et al. 2021;Chandratre and Soman 2020;Day et al. 2020;Vining et al. 2020).However, another study found that the vast majority of surveyed candidates of vMMI were able to communicate adequately with the interviewers (Domes et al. 2021).Therefore, ongoing interviewer training has emerged as a critical component in improving the quality of the vMMI experience for candidates.
The current study suggested that vMMI was a better alternative to FTF MMI.The acceptability of vMMI appears to be a consequence of the flexibility and reduced stress related to the vMMI process.This decreased stress is due to being able to do vMMI from a familiar environment, not having to travel to a central place and not having to physically move between rooms.Several studies have reported high levels of candidate satisfaction with the vMMI process for similar reasons (Tiller et al. 2013;Domes et al. 2021;Inzana et al. 2022;Lund et al. 2021;Sabesan et al. 2022;Singh et al. 2021).In the current study, candidates were moved by 'the host' centrally between virtual rooms and candidates did not have to press or click any buttons.This may have contributed to less cognitive load and, therefore, reduced stress for the candidates.

Limitations
The number of candidates who participated in the semi structured interviews was small (seven) and did not include any candidates from MMM3 locations, with English as a second language, who were overseas graduates and were PGY 5 or above, so the current authors have no knowledge about their potential experiences of vMMI.vMMI was conducted in a resource-rich country and the results may not be applicable to countries with limited resources, particularly places with poor internet connectivity.

Conclusion
Based on our experiences in Queensland, we concluded that demographic characteristics, except for location at the time of application, did not influence the selection outcome.
Candidates' experiences, acceptability, and perceptions were influenced by the management of technology, the flexibility of vMMI and the reduced stress experienced during the process.vMMI is likely to be a viable alternative to FTF MMI beyond the COVID-19 pandemic.Future studies need to specifically correlate rural upbringing and early exposure to regional and remote clinical experience with the selection outcomes and assess predictive validity of vMMI.

Table 1 .
Demographic characteristics of candidates, eligible, shortlisted and offered a training position.

Table 2 .
Comparison between short-listed and not short-listed candidates.

Table 3 .
Comparison between demographic details of candidates who were offered and not offered a training position (out of 71 who underwent vMMI).

Table 4 .
Mean shortlisting scores for various groups.

Table 5 .
Demographic characteristics and vMMI mean scores.