Principles to guide spinal cord injury research partnerships: a Delphi consensus study

Abstract Purpose To establish consensus regarding principles that should be used to guide spinal cord injury (SCI) research partnerships between researchers and research users. Materials and methods A three-round Delphi consensus exercise was carried out with researchers and/or research users involved in one or more SCI research partnerships. Participants considered a list of 125 partnership principles. In rounds 1 and 2, participants rated their agreement that a principle should guide SCI research partnerships on an 11-point Likert scale. After each round, principles that received a mean score of ≥8.0 or 70% of participants rated the principle ≥8.0 were retained. In round 3, participants categorized principles as essential, desirable, irrelevant, or unsure. Results At least 20 individuals participated in each round. In round 1, 103 principles met consensus criteria and eight principles were added. In round 2, 93 principles met the criteria. In round 3, 29 principles were categorized as essential and eight as desirable. Recommended principles focused on the interpersonal, relational, and logistical aspects of partnerships. Principles that did not reach consensus related to social justice and actionable impact. Conclusions Findings provide insight into 37 principles that could be used to combat tokenism and inform future guidance to meaningfully engage partners in SCI research. Implications for Rehabilitation Consensus-based research partnership principles (i.e., norms or beliefs) were identified and could be prioritized to help support spinal cord injury (SCI) researchers and research users combat tokenism and meaningfully engage research users as partners in the co-creation of knowledge. The resulting list of recommended research partnership principles was used to inform the development of guidance to support quality partnerships between SCI researchers and research users within and outside the rehabilitation context (www.IKTprinciples.com). Guidance supporting meaningful research partnerships may accelerate the time between discovery and use of research in practice.


Introduction
Spinal cord injury (SCI) research is multidisciplinary, with research spanning from basic science to applied rehabilitation research to social sciences and humanities. Although SCI research has the potential to enhance SCI rehabilitation programs as well as the lives of many people with SCI worldwide, SCI research is often not translated into practice [1,2]. People with SCI often lack access to research discoveries and their priorities are often not reflected in the research agenda [3][4][5][6].
To address the gap between discovery and use of research, people with lived experience of SCI, SCI organizations, and funding agencies are increasingly encouraging, and in some cases, requiring researchers to adopt partnered approaches to research. Specifically, partnership approaches in which researchers partner with relevant research users (e.g., individuals with SCI, families, friends, organizations, and clinicians) throughout the research process [7-9]. These partnered approaches aim to ensure research user engagement, are considered a promising means of producing research that is relevant, useful, used, and align with calls by people with lived experience of disabilities for there to be "nothing about us, without us" [6, [10][11][12][13][14][15]. By engaging research users throughout the research process, these approaches support efforts across disciplinary domains to enhance democratic involvement in science, amplify disenfranchised voices, and engage in real-world problem solving to improve research relevance and impact [16][17][18][19][20][21].
Despite these calls and the potential advantages of partnered approaches, it is still relatively novel to plan, disseminate, and implement SCI research in partnership ([e.g., 1,9]). SCI researchers are often accused of using a tokenistic approach in which SCI research users are only asked to endorse research to legitimize research programs over which they have little real control [6]. To combat tokenism and support SCI research that is relevant, useful, and used, SCI research stakeholders have expressed a need for guidelines that can be used to support research partnerships that meaningfully engage SCI research users throughout the research process [8,9]. This paper is part of a larger project that aims to address this desire for guidance. Further details are provided in the "Methods" section.
Reviews of the research partnership literature are an important resource for developing partnership guidance. A recent review of 86 reviews of research partnership literature identified a variety of approaches and principles used in partnered research across different research areas (e.g., population health, health/rehabilitation services, and biomedical research) [21]. Principles were defined as "fundamental norms, rules, or values that represent what is desirable and positive for a person, group, organization, or community and help it in determining the rightfulness or wrongfulness of its actions. Principles are more basic than policy and objectives and are meant to govern both". Two approaches that have been used to guide SCI research partnerships include community-based participatory research (CBPR) and integrated knowledge translation (IKT). Although a commitment to including research users as partners in research and the co-creation of knowledge is implicit to both CBPR and IKT, these partnership approaches have different histories and traditions, motivations, and social locations. These differences lead to points of divergence and convergence, with the purpose of CBPR being social change and/or justice, and IKT being the application of knowledge (differences and similarities outlined by Jull et al. [22]). Across these different approaches, the researchers also identified 98 principles reported in different reviews of research partnerships. This review is informative in that it provides a long list of partnership approaches and principles that could be used to develop high-quality, evidence-based partnership recommendations. However, further research is needed to identify approaches and principles that are relevant for the SCI research system (i.e., SCI researchers, research users, and funders).
There is a need to establish a shared understanding among SCI researchers and research users on research partnership approaches and the underlying principles that should be used to guide SCI research partnerships. Delphi methodology is a consensus method that has been commonly used in medical, nursing, and health services research that has the potential to harness the insights of the SCI research community [23][24][25]. The Delphi methodology is a formal, systematic, and reproducible method of arriving at a consensus in which experts are individually and anonymously surveyed in multiple rounds [26][27][28]. In each round, participants rate their level of agreement with survey items. Ratings are then statistically summarized and included in subsequent, repeat versions of the survey. Participants use these summarized statistics in subsequent rounds to re-rate their level of agreement to each survey item. This method is advantageous for informing research partnership guidance in that it allows for the opinion of each expert to be equally valued and inform decisionmaking in large, heterogeneous, and geographically dispersed groups such as the SCI research community [26][27][28].
To inform the development of partnership guidance for the SCI research system, the primary objective of this study was to use the Delphi methodology to establish consensus among SCI researchers, research users, and/or people with SCI regarding principles that should be used to guide SCI research partnerships. The secondary objectives of this study were to examine (1) differences in ratings and categorizations of the principles between participant groups (i.e., researchers, research users, and people with SCI) and (2) if principles that reach consensus aligned with specific research partnership approaches. Ultimately, the findings of this study were one data source used to inform the co-development of the first IKT Guiding Principles for conducting and disseminating SCI research in partnership (see www.IKTprinciples.com) [29].

Design
The study used a three-round Delphi consensus exercise. The study was designed and conducted using an IKT partnership approach [30]. In September 2017, a multidisciplinary consensus panel of SCI researchers, clinicians, organizations, research users, and funders with partnership experience was established. The ultimate goal of the panel was to co-develop IKT Guiding Principles for conducting SCI research in partnership (www. IKTprinciples.com) [29]. With the goal of developing IKT Guiding Principles that were informed by different partnership approaches, the panel defined IKT as: "the meaningful engagement of the right research users throughout the SCI research process at the right time" and meaningful engagement as "contributing and influencing a personal or socially meaningful research or dissemination goal and feeling a sense of responsibility to others (e.g., the research team, group of stakeholders, etc.)". The panel decided that the development of the principles needed to be informed by several partnership approaches and data sources, including consensus methods that involved SCI researchers and research users. Together, the panel applied for and received funding, refined the research questions, developed the Delphi consensus methods, interpreted the data, and disseminated the findings. To support co-ownership, a researcher (HG) and a research user (JM) acted as co-leads for this project and made initial decisions. These decisions were discussed and refined throughout the research process through conference calls and emails with the larger panel. Further information about the partnership's research is available on Open Science Framework (OSF) [31]. All methods were approved by the Research Services Behavioural Research Ethics Board of The University of British Columbia Okanagan (H19-00606).

Participants
It is recommended that participants in a Delphi have related backgrounds and experiences with the target issue, are able to contribute constructive insight, and are willing to revise judgments to achieve consensus [32]. Potential participants were recruited through the multidisciplinary consensus panel's North American network using convenience and snowball sampling. To be eligible to participate and considered an expert for this study, participants needed to have identified as (1) a person with SCI, an SCI researcher, and/or an SCI research user and (2) as someone who has completed or is currently involved in one or more SCI research projects that involve a research partnership. A researcher was defined as "a person that produces research and is affiliated with an academic institution". A research user was defined as an "individual or representative of a group that will use, benefit from or apply the research". These groups included but were not limited to researchers who were using findings, persons with lived experience of SCI, policy-makers, service providers, professional organizations, funders, and industry partners. Delphi methodologies are considered a reliable approach for reaching consensus when a minimum of six expert participants are included [28]. Therefore, the goal was to recruit a minimum of six SCI researchers and six SCI research users.

Procedures
Informed consent was obtained from all participants. Participants completed up to three rounds of online Delphi questionnaires and provided demographic information. All questionnaires are provided on OSF [31].
In round 1, participants were presented with a list of 125 principles. The panel wanted the IKT Guiding Principles to be informed by principles used in different research partnership approaches. To ensure this original list of principles was rigorously and systematically developed and included principles used in different research partnership approaches (e.g., CBPR, participatory research, IKT), the team extracted principles from five key data sources: (1) the review of reviews of principles identified in research partnership approaches (n ¼ 86 reviews), (2) a scoping review of principles identified in SCI research partnerships (n ¼ 14 papers), (3) interviews with champions of IKT working in SCI research (n ¼ 5 researchers, five research users), (4) a search of the grey literature, and (5) a purposive literature search. The steps taken to develop the list of principles are provided on OSF [31]. Participants were also instructed to add any additional principles to the list. The 125 principles were organized into overarching principles (n ¼ 18) and corresponding sub-principles (n ¼ 107) that informed the development of the overarching principle, aligned with the format of the review of reviews [21] (see Supplementary File 1). Participants were asked to indicate the extent to which they agreed that a principle should be used to guide SCI research partnerships on an 11-point Likert scale (0 ¼ strongly disagree to 10 ¼ strongly agree). An "unsure" option was also provided. To facilitate scoring and organize the large number of initial principles, the principles were presented in seven sections that aligned with the categories identified by Hoekstra et al. [21]: (1) relationship between researchers and stakeholders, (2) co-production of knowledge, (3) meaningful stakeholder engagement, (4) capacity building, support, and resources, (5) communication between researchers and stakeholders, (6) ethical and practical issues, and (7) overarching principles. The order in which principles were presented in these sections was randomized. Of note, participants could represent multiple roles. Their collective experiences and roles contributed to their expertise and were considered a strength. Therefore, participants were asked to share their collective expertise and not instructed to respond on behalf of one particular role or perspective.
In round 2, participants were provided with the seven randomized lists of principles that were retained after round 1. Each principle was presented with its corresponding round 1 consensus statistics (i.e., average score, lowest score, highest score, and % of unsure responses). Participants were asked to consider these consensus statistics when re-rating each principle using the 11-point Likert scale. The overall format for the questionnaire was identical to round 1.
In round 3, participants were provided with the seven randomized lists of principles that were retained after round 2. Each principle was again presented with its corresponding round 2 consensus statistics (i.e., average score, lowest score, highest score, and % of unsure responses). Participants were asked to consider these consensus statistics and categorize principles as either (1) essential (i.e., necessary to guide an SCI research partnership), (2) desirable (i.e., not essential to guide an SCI research partnership but would be "nice to have", (3) irrelevant (i.e., neither essential or desirable to guide an SCI research partnership, or (4) unsure (i.e., wording of the principle is unclear and/or do not understand the meaning).

Analysis
After round 1 and round 2, the mean score, highest score, and lowest score for each principle rated by participants were calculated. Consistent with previous Delphi research, principles that received a mean score of greater than or equal to 8.0 or 70% of participants rated the principle as 8.0 or higher were retained in the list. All other principles were removed from the list. To examine the consistency of rounds 1 and 2 ratings, intraclass correlation coefficients (ICCs) were calculated [33]. For the final round, percentages for each category (i.e., essential, desirable, irrelevant, and unsure) were calculated. Principles that were rated by at least 70% of participants as "essential" or "desirable" were retained in the list. To examine differences in the categorization of the overarching principles between participant groups (i.e., researchers, research users, and people with SCI), a chi-square test of independence was performed. To understand how the included and excluded principles aligned with and may be relevant to specific partnership approaches (i.e., CBPR, IKT, and both), two independent raters coded all principles for their partnership approach using Jull et al. [20] definitions of IKT and CBPR (see Supplementary File 2 for coding manual). Inter-rater reliability was assessed using Cohen's Kappa statistic. For ICCs and Kappa statistics values of 0.00-0.50 corresponds to poor reliability, 0.50-0.75 to moderate reliability, 0.75-0.90 to good reliability, 0.90 and above to excellent reliability [34].

Participants
Twenty participants completed the Delphi questionnaire in round 1 (n ¼ 13 researchers, 12 research users). 1 In round 2, 21 participants completed the questionnaire (n ¼ 13 researchers, 11 research users). In round 3, 23 participants completed the questionnaire (n ¼ 14 researchers, 13 research users). All participants resided in Canada and the majority of these participants had been involved in more than five research partnerships. Researchers were primarily independent research professors, whereas research users represented people with lived experience of SCI, policy-makers, clinicians, research funders, and industry partners. The majority of people with SCI had tetraplegia, had a complete injury, and used a manual wheelchair as their primary mode of mobility. A detailed list of participant demographics in each round is provided in Table 1.

Consensus findings
In round 1, 103 of the 125 principles presented to participants (82%) reached consensus and moved forward to round 2, including 17 overarching principles and 86 sub-principles. One overarching and 21 sub-principles were removed. Eight additional principles provided by participants in round 1 were included in round 2 (n ¼ 111 principles). In round 2, 93 of the 111 principles (84%) reached consensus and moved forward to round 3, including 14 overarching principles, 72 sub-principles, and seven additional principles. Three overarching principles, 14 sub-principles, and one additional principle were removed. The ICCs calculated for rounds 1 and 2 indicated excellent reliability of ratings (ICC round 1 ¼ 0.97; ICC round 2 ¼ 0.97) across all participants, for researchers (ICC round 1 ¼ 0.97; ICC round 2 ¼ 0.99), and for research-users (ICC round 1 ¼ 0.98; ICC round 2 ¼ 0.97). In round 3, 35 of the 93 principles retained from round 2 were categorized as essential or desirable. Six were overarching principles that reached consensus as essential and none as desirable. Twentyone sub-principles reached consensus as essential, and eight as desirable. Two additional principles added by participants reached consensus for being categorized as essential and none as desirable. None of the principles met the consensus criteria for being categorized as irrelevant or unsure. In round 3, associations were not found between participant groups and the selection of principle categories for the overarching principles (ps > 0.05). The final list of recommended principles categorized as essential or desirable is presented in Table 2. Ratings and recommendations associated with all principles are presented in Supplementary File 1.

Partnership approach: IKT vs. CBPR
Two researchers coded all principles as representing a CBPR partnership approach, an IKT partnership approach, or both IKT and CBPR. The agreement between the researchers was good (Kappa ¼ 0.89). In total, 113 principles were coded as representing both IKT and CBPR partnership approaches, 15 principles were coded as representing only CBPR, one principle was coded as representing only IKT, and four principles were deemed to be uncodeable

Discussion
Consensus was reached on 37 principles that are considered essential or desirable for guiding SCI research partnerships. Our analyses revealed that principles were rated and categorized similarly by researchers and research users, suggesting that these principles represent a shared understanding among SCI researchers and research users on principles that should be used to guide SCI research partnerships. Principles that met consensus criteria reflected both IKT and CBPR partnership approaches. As a whole, these findings have important implications for SCI research partnerships and funding models as they informed the co-development of the first IKT Guiding Principles for conducting and disseminating SCI research in partnership (www.IKTprinciples. com) [29].
Ninety-five percent of 37 recommended principles aligned with both IKT and CBPR partnership approaches. The recommended principles primarily focused on the interpersonal, relational, and logistical aspects of establishing a partnership that conducts research that is meaningful and relevant to both researchers and research users. The three principles with the greatest consensus as being categorized as "essential" were "The partnership avoids tokenism and ensures that many voices are represented", "communicate openly and honestly", and "treat each other with respect". These recommended principles align with several partnership approaches that focus on ensuring that research users are valued, respected, representative, and meaningfully included throughout the research process (e.g., CBPR, participatory research, patient and public involvement, IKT) as well as the most frequently identified principles identified in the previous review of reviews [21].
Except for two recommended principles, the principles specific to only CBPR or only IKT partnership approaches did not reach consensus. These principles that did not reach consensus focused on social justice, building capacity, creating actionable knowledge, and achieving real-world impact. The wide range of ratings that these principles received in early rounds of the Delphi may reflect differing opinions of the role of partnership in SCI research and/or the feasibility of some research disciplines adhering to these principles. While some SCI research disciplines may lend themselves to empowering research users, building capacity, and creating actionable knowledge, participants may have viewed these principles as less feasible or relevant for other SCI research disciplines, such as basic science, to adhere to these principles suggesting that having principles specific to the SCI research system may be necessary.
Overall, the findings from this Delphi consensus exercise provide initial insight into principles that hold potential for informing and supporting the development of guidance for SCI researchers, research users, and funders interested in establishing meaningful research partnerships. The recommended principles provide the first list of partnership principles that were specifically selected for the SCI research context. Nevertheless, these findings should be interpreted with caution as they do not provide sufficient information to establish standards of practice or formal guidelines or recommendations for conducting and disseminating research in partnership. This study did not provide an understanding of why participants rated the principles lower or higher. A principle could have received a high or low rating for several reasons (e.g., participant did not agree with the wording of the principle vs. did not agree with the underlying philosophy of the principle). Furthermore, the study is limited in that participants were recruited through convenience and snowball sampling. Most of the participants were white, resided in Canada, and all of the participants with SCI were wheelchair users. Continued commitment and efforts are needed to increase the engagement of a diverse group of voices in the practice and science of research partnerships. While the list of principles aligns with the principles identified in the review of reviews of research partnerships, research partnership approaches may be different in different countries or different research domains [21]. Therefore, these findings should not be used as a "final" list of principles that should be mandated and implemented. Rather these findings should be and were used as one form of evidence to guide discussion and decision-making around establishing guidance to support meaningful (SCI) research partnerships (see www.IKTprinciples.com). Further research is needed to understand barriers and facilitators to using principles to guide partnerships and to evaluate the effectiveness of these principles within and beyond the SCI research system.

Conclusions
SCI researchers and research users reached consensus on 37 principles that should be used to guide SCI research partnerships. These findings provide initial insight into principles that are tailored for the SCI research system. The principles could be prioritized to help support researchers, research users, and funders Overarching principles are indicated in bold text. Those presented in italics did not reach consensus criteria. Sub-principles associated with the overarching principle are provided below each overarching principle. Consensus % refers to the percentage of the participants who rated the principle as 8.0 or higher. Supplementary File 1 provides the detailed results including findings related to all principles.
combat tokenism and meaningfully engage research users as partners in research and the co-creation of knowledge. Note 1. Note. Participants could represent multiple roles.