Development and validation of a tool to assess the readiness of a clinical teaching site for interprofessional education (InSITE)

ABSTRACT Interprofessional education within clinical teaching sites is a key part of training for pre-professional students. However, the necessary characteristics of these interprofessional clinical teaching sites is unclear. We developed a tool, the Interprofessional Education Site Readiness, or InSITE, tool, for individuals at a site to use as a self-assessment of the site’s current readiness for providing interprofessional education. The tool progressed through six stages of development, collecting evidence for validity and reliability, resulting in a final tool with 23 questions distributed across five domains. Data from 94 respondents from a variety of national sites were used for the item analysis showing acceptable item-to-total correlations. Internal reliability testing gave a Cronbach’s coefficient alpha of more than 0.70 for each group level comparison. Known groups validity testing provides strong evidence for its responsiveness in detecting differences in sites where IPE is implemented. The results of the testing lead us to conclude that the InSITE tool has acceptable psychometric properties. Additionally, we discovered that the process in which the InSITE tool was used demonstrated that it can facilitate learning in practice for the health professionals and can help make implicit, informal workplace learning and the hidden curriculum explicit.


Introduction
A key component of interprofessional education (IPE) for pre-professional studentsis experiential learning within clinical teaching sites (CTSs), defined as any site in which students are placed to learn from practicing health professionals (D'eon, 2005;Interprofessional Education Collaborative Expert Panel, 2011). Intentionally designing IPE, in which students from two or more professions learn with, about and from each other, is one strategy to enable effective collaboration leading to improved health outcomes (World Health Organization, 2010). In their programs, students learn interprofessional knowledge, skills and attitudes in the didactic portion of the curriculum before entering CTSs that may or may not role model teamwork in practice. To complement early classroom and simulated IPE, CTSs should optimally model teamwork and interprofessional collaboration for students. Today, the necessary characteristics of optimal interprofessional CTSs are unclear. Therefore, a need exists for tools to assess the quality of the learning environment at CTSs and their readiness to accept students for IPE. In response, the authors embarked upon the development and testing of the Interprofessional Education Site Readiness (InSITE) Tool. The InSITE tool has two primary purposes. The first is to enable users at ambulatory CTSs to assess the current state of important characteristics related to IPE in practice. Second, based upon site-specific assessment of these identified characteristics, health professionals can use the InSITE tool to design and establish goals for optimal IPE in practice for all learners (e.g., students, residents, health workers, and current clinicians).
While health professionals at a CTS may adopt and restructure around a collaborative practice model, every site may in reality not be an optimal environment for IPE. What is learned in a CTS may be part of a formal educational program, but more likely than not, much of what is learned within the workplace is informal and implicit. Furthermore, when health professionals do not role model teamwork, the "hidden curriculum" in the CTSs can have a strong negative influence on students, contradicting the collaboration skills learned in their early IPE curriculum (Hafler et al., 2011). Helping health professionals become aware of espoused collaboration and the actual learning environment through intentional workplace learning may offer a strategy to leverage experiences for both formal and informal learning in practice.
IPE within a CTS should be designed purposefully with an appreciation that IPE does not emerge spontaneously within a busy clinical practice (Billet & Choy, 2014). To capitalize on the full educational potential, the first step for a CTS is to reflect upon everyday interprofessional practice and make the experience and what is learned explicit (Nisbet, Lincoln, & Dunn, 2013). One example for doing this includes focusing on existing activities, such as team care conferences (i.e., meetings where health professionals from multiple professions discuss the care plans for patients at the CTS), as an opportunity to explicitly articulate and learn about the role and responsibility of each team members. How a student perceives actual practice may differ from the beliefs, stereotypes, and biases they held before their experience at the CTS. Furthermore, the practice site may not match student perception of what interprofessional practice is based upon what they have learned in didactic, classroom IPE. Helping learners recognize the difference between their preconceived perceptions and actual experiences creates "double loop learning", or thinking more deeply about their own assumptions and beliefs (Argyris & Schon, 1978).
Because workplace learning, especially informal learning, may be confused with normal day-to-day activities, regularlyoccurring, structured activities with discussion and feedback are important to ensure that learning goals of both gaining knowledge and acculturation into the profession are met. This kind of intentional pedagogical intervention allows the learner to develop common professional skills as well as gain better understanding of how to transfer professional competence to site-specific contexts and cultures. (Billet & Choy, 2014;Brandt, Farmer, & Buckmaster, 1993;Eraut, 2012). The authors designed the InSITE tool to make the characteristics and context of interprofessional informal and formal learning explicit and to potentially expose the hidden interprofessional curriculum. Awareness of, and perhaps even modulation of, the hidden interprofessional curriculum by the health professionals at a CTS has the potential to strengthen their effectiveness as team members and impact as role models. In IPE, these strategies move beyond traditional student participation in care (e.g., shadowing, observation and supervised patient care) to the inclusion of intentional debriefs, reflections and other designed components for learners to call attention to the teachable moments about the aspects of interprofessional collaboration that occur and do not occur in practice (Eraut, 2012;Nisbet et al., 2013).
By identifying the current state of the IPE characteristics, the health professionals at a CTS can self-assess their readiness for serving as role models for interprofessional behaviors. Attention can then be focused on areas that need improvement of individual and team competencies leading to further development. The health professionals at the sites know themselves and the practice culture best. Yet, they may be prone to unconscious bias about their own collaborative behaviors. The InSITE tool is designed as a way for health professionals to assess their site's readiness for IPE and set a framework for team discussions to identify potential needed reconfigurations of the practice environment to improve teamwork behaviors. These discussions may require facilitation and guidance from others external to the practice to make necessary changes identified in the InSITE process.
Few papers describe tools to assess readiness for serving as a CTS, which role models interprofessional behaviors for students, and then can be used for site improvement. Rather, systematic reviews focus on tools measuring aspects of teamwork and team performance (Hayver et al., 2014;Marlow, Lacerenza, Iwig, & Salas, n.d.;Shoemaker et al., 2016;Valentine, Nembhard, & Edmondson, 2015) and on assessing student attainment of interprofessional collaboration competency (Hayver et al., 2016). One tool, the Assessment for Collaborative Environments, or the ACE-15, measures perceptions of interprofessional "teamness" in teaching environments; however, it focuses on the ideal teamwork qualities rather than IPE at the site (Tilden, Eckstrom, & Dieckmann, 2016). Another tool, the IP-Compass, created by Parker and Oandasan (2012), measures interprofessional environment, specifically in hospital units, focused on IPE and collaborative practice and does not detail the needs for creating an interprofessional environment.
This article describes the process used to: 1) identify the characteristics of interprofessional ambulatory CTSs; 2) develop and psychometrically test a self-assessment tool for use by individuals at an ambulatory site to assess the site's readiness for serving as an interprofessional CTS; and 3) recommend how systematic assessment can be used in the development of interprofessional ambulatory CTSs.

Methods
InSITE tool development proceeded from conceptualization through empirical testing in six stages as follows: (1) content development, (2) concept testing, (3) cognitive interviews and face validity, (4) scaling and item analysis, (5) reliability testing, and (6) known groups validity testing (see Figure 1). The following briefly describes details for these activities.

Content development
Preliminary content for the creation of the InSITE tool emerged from a group brainstorming exercise. Based upon their IPE expertise and practice background, a group of nine individuals from multiple professions including dentistry, medicine, nursing, pharmacy and public health affiliated with the University of Minnesota Academic Health Center brainstormed what they believed to be the characteristics of interprofessional CTSs. While all of the participants in this process currently practice in urban ambulatory practice settings, their past practice experiences included urban and rural, hospital and ambulatory care, academic and clinical leadership, and educational instruction and design. From this process, the health professionals identified multiple characteristics. The comments allowed preliminary grouping into four broad domains: (1) the type and quality of activities available to learners at the site (referred to as the Learner Experiences at the Site (LE) Domain), (2) the quality of preceptor interprofessional role modeling (referred to as the Preceptor or Supervisor Qualities (PQ) Domain), (3) the structure and function of the interprofessional teams at the site (referred to as the Site Infrastructure (SI) Domain) and (4) the parent organizational support IPE (referred to as the Organizational Support (OS) Domain). To create a preliminary version of the InSITE tool for further testing, the characteristics were turned into questions.

Concept testing and question refinement
This early version of the InSITE tool, composed of questions grouped into four domains, was tested in four ambulatory CTSs identified by the authors to have implemented formal IPE activities. Three sites provide primary care to underserved populations, and one provides primary and specialty care for women. The purpose of the concept exploration was to determine if the group brainstorming had appropriately identified the characteristics of an interprofessional CTS. The InSITE tool was given to a broad variety of health professionals at each site to provide a comprehensive self-assessment of the CTS. One of the authors (BS or AP) conducted a semistructured, focus group at each of the four sites of those individuals who had completed the InSITE tool. Content analysis of the focus group transcript utilized the Classic Analysis Strategy, a constant comparison-like approach (Krueger & Casey, 2009). After each focus group session, participant comments were grouped into themes and used to guide revisions. Revisions made to the tool were then retested with the next CTS until concept saturation with no new information and no new tool revisions was reached. The process of concept exploration revealed more characteristics which grouped under a fifth domain of Site Culture (SC).

Cognitive interviews and face validity testing
The preliminary tool with five domains was evaluated for clarity and face validity in two 1.5 hour cognitive interviews using a Think-Aloud approach (Dillman, Smyth, & Christian, 2014). Interviewees, a psychologist and a physician, were selected for their general knowledge of IPE, decades of clinical experience, and leadership and scholarship on IPE and teamwork. While both work in CTSs in academic health centers, the physician also has experience working with rural, underserved areas. For each question in the tool, the interviewer asked the interviewees to verbalize their thoughts as they went through the survey to describe their thought processes for arriving at their responses. The purpose was to learn their understanding of the questions to clarify the question wording, providing feedback on the intention behind asking the question, as well as whether each question was accurately interpreted by the person completing the tool (Willis, 2005). Revisions after this process included removing jargon, introducing skip logic for questions for customized pathways based upon a respondents answers, and addressing identified double-barrel questions that asked about two concepts.
Using the tool refined after the cognitive interviews, one author (BS) met in-person with seven IPE experts in two separate panel discussions to review the tool content for face validity. The expert teams included a group internal to the University of Minnesota and another group composed of three individuals from three separate international institutions. The local group, different than those from the content development brainstorming, included four faculty representing dentistry, medicine, nursing and pharmacy who all work in ambulatory CTSs and were also members of the institutional team responsible for IPE curricular development at the University of Minnesota. The external group was composed of three international IPE leaders who have published extensively in this field. Before coming together for a panel discussion, reviewers were provided with a copy of the tool and an accompanying survey asking for comments on the necessity of each question ("necessary" vs "not necessary") and appropriateness of the scale of each question ("appropriate" vs "not appropriate"). The facilitator (BS) met with the two groups separately and systematically discussed their comments about each question and at the end asked if there were critical characteristics omitted from the tool. The wording of the questions changed (e.g., "are all relevant parties included" was changed to "are all relevant team members included") and some scales were recommended to change (e.g., "tend to agree" changed to "somewhat agree"); however, at this point no additional characteristics were added.

Scaling and item analysis
At this point in testing the InSITE tool included 5 domains and 46 questions. For item analysis, individual items were evaluated for difficulty and discrimination (Crocker & Algina, 1987). Our goal was for 100 individuals to complete the tool. Difficulty was evaluated by the following three questions: • Were response choices dispersed across the respondents? • How relevant was the item to the respondent's CTS?
(i.e., did they answer "Not Applicable") • Did the respondent have knowledge to respond to the item? (i.e., did they answer "I don't know") To accommodate for missing values, scoring used a mean substitution procedure. In this procedure, a mean value for valid responses was imputed for any missing items completed for at least one-half of the items in the scale score. Classification of missing responses was based on any of the following: an "I don't know" response, a "Not applicable" response, and a skipped question. Reponses to these questions were summarized as means, standard deviations and percentages. Discrimination was assessed using the item-to-total correlation. This was expressed as the Pearson product moment correlation coefficient for the individual item to the total score without the item.
A Likert-type scale was used for scaling responses. Five summative scales were created for each of the domains. Since the number of response choices varied within some of the domains and the investigators wanted to give equal weight to all items, scale scores were transformed to a 0 to 100 value. The resultant scale was based on the mean value of the transformed value for each response for that domain.

Reliability testing
Reliability was measured using Cronbach's coefficient alpha for each of the five scale scores (Nunnally & Bernstein, 1994). Following the recommendations of Nunnally and Bernstein (1994), values greater than 0.70 were judged as acceptable for making group level comparisons, in this case the unit of analysis was assumed as the CTS. Along with coefficient alpha, the intercorrelations of scale scores were examined in the form of a domain-specific inter-correlation matrix. In interpreting the p values, a Bonferroni correction, or a similar adjustment, was made to minimize the threat of "fishing and error rate" problems (Shadish, Cook, & Campbell, 2001).

Known groups validity testing
For the four site-specific scale scores (i.e., excluding Organizational Structure), known-groups testing was used for validation. These sites were medical clinics representing a diversity of professions, roles, locations, and structures. The three sites used were two family medicine clinics in two different urban, underserved communities and a women's health center at an academic health center. In this analysis, 17 tools were completed by clinicians and staff including 3 people from the women's health center, which was treated as a High IPE site, and 14 people from the two family medicine clinics, which were treated as Low IPE sites. A High IPE site was one which had incorporated intentional activities, either experiential or didactic, where students from two or more professions were learning with, from and about each other to improve health outcomes. Low IPE sites were not conducting these activities. The judgement between high and low IPE was made by two of the authors (BS and AP). Staff represented at these sites included: pharmacists, physicians, clinic directors, education coordinators, family medicine residents, nurse midwives and registered nurses. In the analysis, differences in the two conditions were expressed as effect sizes (i.e., difference divided by the standard deviation of the difference). Bootstrap (resampling) methods were used to estimate variation of the differences. Five hundred replications were used for the bootstrap results. Discriminant validity for these scale scores were supported if the effect sizes were higher for the women's health center.

Statistical analysis
The data analysis for this paper was generated using the ninth version of SAS/STAT software (SAS/STAT (Version 9), 2017). Bootstrap resampling used version 14 of Stata (StataCorp, College Station, TX, 2015). Graphs were produced using GraphPad Prism (GraphPad Prism, n.d.).

Ethics
This study (all study materials, assessments, and procedures) was deemed exempt from review by the University of Minnesota Institutional Review Board on January 3, 2013 (Study Number: 1301E26190).

Consent
Each individual who was part of the concept testing at the four ambulatory CTSs was informed that the conversations would be recorded in writing and on a digital audio recording device. Written consent from each individual in this phase was obtained. The risks and benefits of participating and the voluntary nature of the study were clearly explained in writing to all individuals in each of the stages.

Scaling and item analysis
A convenience sample of 94 respondents from sites across the United States affiliated with the National Center of Interprofessional Practice and Education (www.nexusipe.org) completed a long-form (46-question) version of the InSITE tool, which was the length of the tool at the end of the cognitive interviews and face validity testing. The sites the tool was sent to included rural and urban, academic and community, and pri- mary care and specialist practices and included dozens of different types of professionals. The identity, including the profession and the practice site, of the individuals completing the tool was not captured.
After questions with a high percentage of missing responses (>20% missing) and questions with poor discrimination (>90% of response choices at either low-end or highend of the response distribution) were eliminated, the final version of the InSITE tool was comprised of 23 questions grouped into 5 domains critical to readiness for interprofessional education (distribution of 46 questions in parentheses): • Learner Experiences at the Site (LE) -6 Questions (13) • Preceptor or Supervisor Qualities (PQ) -2 Questions (5) • Site Infrastructure (SI) -5 Questions (9) • Organizational Structure (OS) -5 Questions (10) • Site Culture (SC) -5 Questions (9) The first 4 domains are specific to the CTS, within which the respondent practices. The latter, Organizational Structure pertains to the wider organization, or parent organization (if one exists). See Figure 2 for a pictorial representation. Table 1 summarizes the item analysis for the remaining 23items. All scale scores had a range between 0 and 100. The mean for scale scores was higher for Preceptor or Supervisor Qualities and lower for Organizational Structure. The distribution of response choices for individual questions trended toward the lower values for simulations and debriefings, the training of preceptors, the use of health records to create care plans, organizational incentives for IPE, organization holding people accountable and engagement of leaders in IPE.
Respondents to the tool had knowledge of most of the characteristics assessed. This was apparent from lower percentages of respondents lacking knowledge in the site-specific domains or Learner Experiences at the Site, Preceptor or Supervisor Qualities, Site Infrastructure, and Site Culture. Relative to the other domains, more of the respondents lacked knowledge about the Organizational Structure. Very few of the respondents found the items "not relevant" to their CTS.  The item-to-total correlations in Table 1 show that individual items in each of the five scale scales were moderately correlated with the total scale score without the item. These results suggest acceptable discrimination for the questions included in the InSITE tool. Exceptions might be the use of simulations and debriefing (LE) and the importance of conflict to effective teamwork (SC). These items were kept in the corresponding scale scores because they were judged to have high face validity for the scale among our panel of experts.
Reliability Table 2 gives the domain-specific inter-correlation matrix for the InSITE tool. The intercorrelations are for the multi-item scale scores, and the average intercorrelation of items in the scale are expressed as Cronbach's reliability coefficients. These are shown boxed along the diagonal of the matrix. The results support the reliability of the 5 domain scale scores with each scale score exceeding the 0.70 recommended for group level comparisons. The values for the inter-correlations suggest that the subscales measure distinct domains and are weakly or not associated with one another. Reliability as measured by Cronbach's coefficient alpha exceeded the 0.70 level recommended by Nunnally and Bernstein (1994) for group level comparisons. The correlations and associated p values are given below and to the left of the reliability coefficients.
Values of less than 0.005 were acceptable as statistically significant using a Bonferroni correction based on 10 comparisons. The p values reported in Table 2 were before the Bonferroni correction. Learner Experiences at the Site was moderately associated with Site Infrastructure (p value < 0.001) and Site Culture (p value < 0.001). The scale score for Preceptor or Supervisor Qualities was moderately associated with Site Infrastructure (p value < 0.001). Site Infrastructure was moderately associated with Site Culture (p value = 0.001).

Known-groups validity testing
The results for the known-group validity testing are shown in Table 3. A comparison of the High IPE site with the Low IPE site for site-specific scale score provides strong evidence for the responsiveness of the InSITE tool in detecting differences in CTSs where IPE is implemented. Except for Preceptor and Supervisor Qualities, which had an effect size or 0.56, the effect sizes observed were strong, that is, greater than 0.80. This result supports the use of InSITE in comparing relatively small sites. The effect sizes are desirable for planning future IPE research and conducting statistical power analysis (Cohen, 1988). Discriminant validity for these scale scores supported greater effect sizes for the women's health center. In Figures 3-7, the distribution of scale scores are summarized graphically with scores sorted from low-to-high. Mean scores for respondents were compared against the grand mean and are represented as a Mean Scale Score on each Figure. For site-specific scale scores, one effect size below the mean was provided as a suggested Low IPE cut-point represented as a Lower Bound on each Figure.

Discussion
Through the development of the InSITE tool, the authors set out to identify the characteristics of interprofessional education that a CTS could use to both self-assess their readiness to provide IPE and improve the quality of the learning environment. The authors used a combination of brainstorming, "think-aloud", expert panel opinion and real-world testing to arrive at a tool with 23 questions grouped into five domains -Learner Experiences at the Site, Preceptor or Supervisor Qualities, Site Infrastructure, Site Culture and Organizational Structure. This InSITE tool was tested for psychometric properties in a sample of 94 respondents from a variety of practice environments and tested for known-group differences in 3 practice environments.
The psychometric testing of the InSITE tool confirmed the reliability of the five multi-item domain scale scores. In addition to having acceptable face validity, the InSITE tool was shown to have high discriminant validity for distinguishing sites that have implemented IPE from those that have not. Along with the reported effects, this information will be useful in the design of future IPE studies. A particular strength of the current study is that it provides the responsiveness (i.e., mean difference in groups/standard deviation of the difference) of characteristics of a new tool. This is critical to future outcomes research in IPE and should prove useful to investigators of interprofessional community-based CTSs.
It is particularly noteworthy that the lowest mean scores of the 5 domains were for Organizational Structure. This deserves further study particularly since "organization provides incentives for IPE" and "organization holds people accountable for IPE" were scored the lowest among the 23 questions despite the importance of organizational support for IPE found in a systematic review of the literature (Reeves et al., 2016). This may reflect the relative lack of knowledge that individual respondents had about their parent organizations compared to their own site infrastructure. This apparent lack of awareness could represent a concern in communicating future IPE efforts from the organization to the individual CTSs. To get a more accurate assessment of the readiness of a CTS for IPE, respondents with a broader knowledge of their parent organization will need to be selected.
During the concept testing of the preliminary InSITE tool, the process in which the tool was used was found to be important. It was assumed that the health professionals at these sites would have a shared understanding of their collaborative practice model and the formal IPE offered to learners while at the CTS. Instead it was discovered that most individuals had a view of the CTS that was limited to their specific role at the site. To get a comprehensive view of a site's readiness for interprofessional education, the authors found it was beneficial for the individuals at the CTSs to bring their completed InSITE tools to a facilitated group session for the purpose of completing one tool for the site thereby coming to a shared understanding. The authors postulate that, using the results of the tool completed by the group, the health professionals could plan appropriate next steps for individual professional and overall site improvement. When appropriate the site could repeat the process and gauge their improvement. This process will require further testing across settings and professions to determine the most effective and efficient implementation.
This process of site assessment by consensus, showed that the InSITE tool can facilitate learning in practice for the health professionals at the site. While the InSITE tool does emphasize the need for formal, planned educational activities for learners, it also calls out the informal learning opportunities that constitute much of workplace learning (Nisbet et al., 2013). Nisbet et al. (2013) argue that informal learning is an underutilized opportunity within the health workplace and that there is a need to make this learning more explicit. For example, the two questions within the Preceptor or Supervisor Qualities domain were the frequency of interaction of the professionals regarding patient care and frequency of participation in shared clinical decision making. These are activities where health professionals informally learn about other professions roles, scopes of practice, and approaches to patient problems through care discussions about specific patients. Within the Site Infrastructure domain, the inclusion, or lack of inclusion, of relevant professionals in interprofessional patient care activities sends an implicit message to learners and other professionals about the value of each profession's contribution and can encourage changes towards more inclusivity. The entire domain of Site Culture emphasizes the underlying hidden curriculum including the way the professionals at the site address conflict, demonstrate trust and respect and hold each other accountable. The InSITE tool asks the health professionals at the CTS completing the tool to both recognize these informal learning elements as important and assess their implementation of these elements along a continuum. By making these implicit activities explicit through reflective assessment and discussion, the InSITE tool can increase awareness of the need for change among the site's health professionals and encourage more attention to, and hopefully more purposeful incorporation of both informal learning and the hidden curriculum.
The InSITE tool also serves as a tool to drive change within a CTS. It can serve as a way to gain consensus on the clinical site's interprofessional education strengths and weaknesses and collaboratively identify actions for improvement. The individuals at the site can use this information to sustain those domains where there is already progress and improve those domains where there are gaps. Through successive repetitions of the evaluation process, the site can have ongoing measures of team and interprofessional education learning environment development.
A limitation of this work is the anonymity of the 94 respondents to the InSITE tool. Detailed information about the respondent's profession, site, and characteristics of the organization would be valuable to evaluating the construct validity of the InSITE tool. Also, as pointed out in the Introduction, having the individuals self-assess their readiness could lead to bias. However, since this is a tool designed to enable sites to gauge their current readiness for IPE and to establish goals for optimal IPE, it is to the benefit of the individuals to be honest with their assessment. Furthermore, the facilitated group process can mitigate against any individual biased tools.
We strongly support continued testing of InSITE. We recommend that the testing be replicated within other sites because the known groups testing is from a narrow population within a single parent organization. With further testing and confirmation of its ability to discriminate between High and Low IPE sites, the InSITE tool could be used to designate sites as optimal IPE CTSs. Also, the facilitated group process used at the sites will need to be refined with further testing.

Conclusion
The InSITE tool provides a way for CTSs to self-assess their readiness to provide interprofessional education and improve the quality of the practice environment for all learners. Additionally we found that the tool can help make implicit, informal workplace learning and the hidden curriculum more explicit to the professionals at the site and potentially drive a positive change in collaborative behaviors.