Challenges encountered in the conduct of Optimal Health: A patient-centered comparative effectiveness study of interventions for adults with serious mental illness

Background: The aim of patient-centered comparative effectiveness research is to conduct stakeholder-driven investigations that identify which interventions are most effective for which patients under specific circumstances. Conducting this research in real-world settings comes with unique experiences and challenges. We provide the study design, challenges confronted, and the solutions we devised for Optimal Health, a stakeholder-informed patient-centered comparative effectiveness study focused on the needs of seriously mentally ill individuals receiving case management services in community mental health centers across Pennsylvania. Methods: Optimal Health, supported by the Patient-Centered Outcomes Research Institute, is a cluster-randomized trial of two evidence-based interventions for improving health and wellness across 11 provider sites. Participants were followed for 18–24 months, with repeated measurements of self-reported health status and activation in care and administrative measurements of primary and specialty health service utilization. Health-related quality of life, engagement in care, and service utilization are to be compared via random effects mixed models. Stakeholders were, and continue to be, engaged via focus groups, interviews, and stakeholder advisory board meetings. A learning collaborative model was used to support shared learning and implementation fidelity across provider sites. Results: From 1 November 2013 through 15 July 2014, we recruited 1229 adults with serious mental illness, representing 85.1% of those eligible for study participation. Of these, 713 are in the Provider-Supported arm of the study and 516 in Patient Self-Directed Care. Across five data collection time points, we retained 86% and 83% of the participants in the Provider-Supported and Self-Directed arms, respectively. Lessons learned: Lessons learned relate to estimation of the size of our study population, the value of multiple data sources, and intervention training and implementation. The use of historical claims data can lead to an overestimation of eligible participants and, subsequently, a reduced study sample and an imbalance between intervention arms. Disruptions in continuity of care in real-world settings can pose challenges to on-site self-report data collection, although the inclusion of multiple data sources in study design can improve data completeness. Geographic dispersion of rural provider sites and staff turnover can lead to training and intervention fidelity challenges that can be overcome with the use of a “train-the-trainer” model, “wellness champions,” and the use of a Learning Collaborative approach. Stakeholder engagement in mitigating these challenges proved to be critical to study progress. Conclusion: Conducting real-world patient-centered comparative effectiveness research in healthcare systems that care for seriously mentally ill persons is an important yet challenging undertaking, one which requires flexibility in identifying potential adaptations within all major study phases. Advice from a wide range of stakeholders is critical in development of successful strategies.


Background
The Patient-Centered Outcomes Research Institute (PCORI) was established in 2010 by the Affordable Care Act to fund and disseminate patient-centered comparative effectiveness research. The purpose of PCORI research is to inform patients, providers, decision makers, and other stakeholders about which interventions among those shown to be effective works best for whom under which circumstances. A paramount component of PCORI-sponsored research is patient and stakeholder engagement in every step of the research process to ensure relevance, acceptance, uptake, transparency, and validity of research and results. To achieve this, PCORI established stakeholder-driven priorities and methodology standards 1 to ensure that resulting information is actionable. Improving healthcare systems is one of PCORI's five national priorities. 2 Patient-centered comparative effectiveness research provides an avenue to conduct studies that produce information for use in health system transformation but at same time supporting engagement of populations that remain understudied and underserved.
One such population includes adults with serious mental illness, those having a behavioral or emotional disorder of persistent duration resulting in serious impairment and interference with or limits to major life activities. 3,4 The combination of high medical need with difficulty in engaging in effective medical care makes adults with serious mental illness one of the most medically vulnerable populations in America. These adults have high rates of premature death, on average dying as much as 15-25 years earlier than the general population, a trend that has accelerated in recent decades. 5,6 The prevalence of cardiovascular disease and metabolic syndrome among this population is at least twice that of the general population. 7 Key contributors to excess morbidity and mortality among this population include modifiable lifestyle choices and behaviors; 8,9 negative metabolic effects of atypical antipsychotic medications; 10,11 higher rates of undiagnosed, untreated, or poorly treated medical illnesses; 12,13 and limited use of routine preventive and primary care. 14,15 For the 75% of the individuals with serious mental illness who are insured through Medicaid, community mental health centers typically are their first, and often only, points of contact with the healthcare system. 16 Personnel at community health centers have long recognized the important role these centers play in addressing the unmet medical needs of the patients with serious mental illness but have been deterred by a lack of evidence regarding effective ways of integrating primary medical preventive and treatment services into their operations. 17 To address the research knowledge gap around health care for adults with serious mental illness, we forged a multi-stakeholder collaboration in rural Pennsylvania that has involved patients, families, community providers, a large, nonprofit behavioral health managed care organization (Community Care, part of an insurance division of a provider, UPMC), countylevel decision makers, and academic researchers at the University of Pittsburgh who have been working to transform community mental health centers into optimally performing behavioral health homes. We describe herein the design and implementation of a study, Optimal Health, funded by PCORI in 2013, to evaluate and compare the impact of real-world interventions on improving the health, wellness, and recovery of adults with serious mental illness who receive care in community mental health settings. The two study interventions being compared have a strong evidence base to support their effectiveness, and each has the potential to serve as best practice for improving outcomes for these individuals who have or who are at risk for chronic medical conditions. [18][19][20][21][22][23][24] In particular, we describe our experiences with stakeholder engagement and roles, incorporation of PCORI methodological standards, nuances of our system-level study design, and study implementation. The challenges encountered and solutions implemented may be useful to other researchers, especially those who wish to enroll adults with serious mental illness in geographically dispersed locations. 25,26 Methods For 5 years prior to the start of this study, Community Care, the behavioral healthcare payer who holds contracts with counties across Pennsylvania to manage care for their Medicaid beneficiaries, had been working together with beneficiary representatives and community providers to develop and deliver system-level interventions focused on the overall health of mental health service users. The Behavioral Health Home model, which serves as the basis for study interventions was the result of this 5-year development process. Among populations in which integration and coordination of behavioral and physical health care may be inadequate, a Behavioral Health Home model can improve the health and wellness of service users by utilizing elements from the Chronic Care Model, Substance Abuse and Mental Health Services Administration's Eight Dimensions of Wellness, and the health home model described in the Affordable Care Act and defined by the Centers for Medicare and Medicaid Services. [27][28][29] Components of the Behavioral Health Home model include (a) enhanced capacity of behavioral health providers to serve as health homes, (b) comprehensive care management, (c) care coordination and health promotion, and (d) linkage of service users to available community resources to address their psychosocial needs. 29 Implementation of this model and its components have demonstrated improvements in outcomes related to hospital and emergency department admissions, homelessness, quality of life, disease-related symptoms, healthcare costs, and satisfaction with care. [30][31][32][33] Optimal Health, which was approved by the University of Pittsburgh Institutional Review Board, involves ''layering'' of rigorous research onto a realworld, large-scale implementation of the Behavioral Health Home through a payer-provider system initiative with existing partners/stakeholders. The participation of stakeholders at all stages of the research is a critically important and novel aspect of Optimal Health. Figure 1 depicts the organizational structure, and Table 1 summarizes stakeholder roles/responsibilities and examples of contributions in each step of the research. The study has three principal investigators (PIs), one from Community Care/payer (Payer-PI), one from academia (Research-PI), and one who is a behavioral health service user and community mental health staff person (Peer/Patient-PI) who has served a pivotal role in defining both the new models of care and study details. During the pre-proposal phase, stakeholders identified the importance of enhancing health and wellness of individuals with serious mental illness, aided in the design of the study interventions, and provided input on the Letter of Intent (Table 1). Our Peer/ Patient-PI, patients who volunteered to participate in focus groups, county human and mental health service administrators, and providers identified outcomes that matter most to those living with serious mental illness and advised the research team on specific aims, research questions, logic model, proposal drafts, and implementation. Additionally, ongoing feedback throughout the duration of the study led to changes in consenting procedures, self-report measures, intervention delivery, and dissemination planning.

Study aims and PCORI questions
To ensure study focus on questions and outcomes in which patients with serious mental illness are interested and personally invested, 34 we conducted focus groups and interviews with over 40 patients who volunteered to share their perspectives as service users at community providers where the proposed interventions were piloted. While patients endorsed the study questions, they recommended making them more specific to the unique circumstances of the study population, including selection of gender as a moderator given stakeholder endorsement that men and women engage differently in their individual health and wellness plans. Moreover, studies suggest that gender plays a significant role in individuals' engagement and outcomes in health and wellness programs. [35][36][37] Clinical implications of gender differences support further investigation of gender as a moderator.
Study aims are as follows: Primary Aim 1. To compare the effectiveness of the interventions on three primary patient-centered outcomes (i.e. patient activation in care, health status, and engagement in primary/specialty care). Primary Aim 2. To examine the moderating role of gender on the three primary patient-centered outcomes. Secondary Aim 1. To explore the mediating role of patient engagement in the interventions on primary and secondary outcomes. Secondary outcomes include hopefulness, quality of life, patient satisfaction, medication adherence, functional status, emergent care use, and laboratory monitoring. Secondary Aim 2. To explore the impact of the interventions on secondary outcomes.

Study design
Study interventions. The two Optimal Health interventions support the recovery model of behavioral health service delivery, which initially emerged from within the patient and consumer advocacy communities but has since become a mainstay of behavioral health care and other recovery and wellness supports. 38,39 The recovery model recognizes that mental health treatment alone cannot fully address the overall health and wellness of those with serious mental illness and that these patients can live fulfilling lives, even in the face of disabling illness, when they are provided with opportunities to receive adequate psychiatric and medical care. It also endorses the central role of patients in managing their own health and health care through self-determination and shared decision making. 40   improvements, they expressed a desire for opportunities to improve their health and wellness in the context of overall recovery, to be active participants in their care, and to build relationships and make connections with peers or others like them. Many of the components of the Optimal Health interventions were identified by these patients as being of particular interest and/or value to them, such as incorporation of technology into care, increased access to medical care, availability of tools to monitor and achieve wellness goals, and increased connection with peers. Individuals with serious mental illness who participated in pre-study implementation focus groups and interviews identified the ability to manage their chronic medical conditions as fundamental to their overall health and well-being. New methods of intervention that utilize technology are valued by individuals receiving services who wish to become more active in their care because they provide additional support and motivation in a convenient way. Providing multiple services in a familiar environment also adds to their convenience in accessing medical care. Peer support programs, whereby those who have achieved significant improvement in their behavioral health assist others in their recovery, are viewed by patients and their providers as a useful intervention and an important opportunity to improve social connectedness. Because peers share a common experience of illness and recovery with patients, they are able to provide practical and empathetic support. 41,42 Table 2 provides an overview of our two study interventions, Provider-Supported Care and Patient Self-Directed Care. Training materials were developed based on core components of the Substance Abuse and Mental Health Services Administration's 10 3 10 program. 43 Patient-centered outcomes. Specific primary and secondary study outcomes were determined through the engagement of patients and other stakeholders, together with review of the relevant literature. A total of 32 individuals with serious mental illness were engaged to complete the initial set of self-report study measures and comment on content and ease and burden of completion to assist the investigative team to create the final set of study measures (Table 3).
Target population. Our target population included Medicaid-enrolled adults aged 21 and older with serious mental illness as documented through claims data using diagnostic codes associated with schizophrenia, bipolar disorder, or major depression who receive services at community mental health centers with Community Care's network of providers. Patients who are unable to read and speak in English were deemed ineligible.
Sample size. Our original sample size and power calculations assumed 2400 eligible adults recruited from eight community mental health centers, with an 80% enrollment rate and 75% of enrollees with complete data to provide a total analysis sample of 1440 participants. As the study progressed, we realized that our original assumptions were incorrect, as described below. Recalculation of power was based on a smaller sample size, a larger number of clinics, heterogeneity in clinic population sizes, and unequal enrollment across study arms. We assumed 1500 eligible adults across 11 centers with 80% enrollment and 75% of the participants with complete data to yield an analysis sample of 900 participants. We assumed a two-sided test at the 5% significance level, and an intra-cluster correlation of 0.01. This sample size provides 80% power to detect or rule out an effect size of 0.33, which is a ''small'' effect size according to Cohen. 50 That is, we have 80% power to detect or rule out a difference in means of between interventions of 33% of a standard deviation for primary outcomes that are continuous variables such as the Patient Activation Measure and the Health Survey Short Form-12v2.
Randomization. We randomized 11 community mental health centers to one of the two interventions. The choice of a cluster-randomized design at the provider site level was dictated by stakeholder request and scientific consensus to avoid the risk of contamination or bleeding effects and implement only one new model of care in each service delivery site. 51 Our randomization scheme was based on a minimization algorithm 52 that balanced the provider sites on factors likely to impact outcomes including volume of individuals served in and location (rural or not) of community mental health center and service user characteristics including age and psychiatric diagnosis. A raw data matrix including these factors, organized by site, was created. Then, sites were randomly ordered, and the first two sites were assigned to separate arms. Remaining sites were then Study-specific registered nurses employed as ''wellness nurses'' to work with individuals on coordinating care, enhancing communication between providers, offering wellness support and education to study participants, as well as other individuals seeking behavioral health services, and providing consultation to health navigators.
Case managers/service coordinators and peer specialists are trained in wellness coaching to serve as ''health navigators'' promoting care coordination and improved physical health, wellness, and recovery. Use of member registry to identify and stratify eligible members to tier levels based on clinical physical and behavioral health needs for targeted interventions and outreach by clinic staff.
Patient Self-Directed Care Service delivery focuses on providing tools, education, and resources that activate individuals to be more informed and effective managers of their health and health care. Individuals have access to self-management toolkits and web-based physical health and wellness tools and resources to assist them with meeting wellness goals. assigned to the arm which would minimize the imbalance score based on those characteristics. Table 3, Optimal Health utilized three types of data. Participant self-report data are gathered at five time points (i.e. baseline and every 6 months) using the Community Care's secure web portal, a well-tested platform for efficient and effective capture of self-reported information.

Data collection procedures. As outlined in
Financial incentives (i.e. US$20 debit card) were provided to participants for completion of measures at each time point; the value was determined by consulting patients. In addition, we are using existing health service claims data from Community Care and the Pennsylvania Department of Human Services. Finally, trained interviewers were responsible for qualitative data collection via interviews with patients and provider staff. A total of four or five patient participants and two provider participants, selected from case managers, wellness nurses/lead navigators, from each participating provider were chosen to be interviewed at baseline, mid-implementation, and end of the study.

Analysis plan
We are using generalized linear mixed models and generalized estimating equations 53 to analyze patient selfreport, process, and claims data regarding changes over time and assess moderating and mediating variables. We include intervention and covariates of interest as fixed effects in our models and community mental health centers and service users within centers as random effects. We will utilize appropriate bias-correction methods for cluster-robust inference with a small number of clinics. Our analysis will address heterogeneity of treatment effects by assessing the moderating role of gender and will conduct an exploratory analysis of the impact of other patient characteristics. Patient and provider codebooks have been developed for qualitative analysis of both participant and provider interviews following the editing method outlined by Crabtree and Miller. 54 The codebooks are sensitive to comparisons between interventions, capturing themes related to barriers, facilitators, and levels of satisfaction. Transcripts from each of the three qualitative data collection points will be compared across time to understand changes in staff knowledge, skills, and attitudes over the course of Optimal Health implementation. Information about provider and patient experiences will also be used in a formative evaluation to characterize the implementation process and ensure future intervention improvements. Using the codebooks developed, two trained independent analysts will code interviews; inter-coder reliability will be assessed using kappa statistics.

Results: enrollment and retention
From 1 November 2013 through 15 July 2014, 1229 adults with serious mental illness enrolled after consenting to participate in 2 years of intervention, complete self-report questionnaires every 6 months, allow their service use or claims data to be extracted and used in the research, and complete qualitative interviews if selected. The sample of 1229 participants represented 85.1% of those eligible for study participation at the 11 sites. The total number of participants enrolled in each intervention arm is 713 for Provider-Supported Care and 516 for Patient Self-Directed Care. An abbreviated version of our study CONSORT diagram is presented in Figure 2. Across the study data collection time points, we retained 86% and 83% of the participants in the Provider-Supported and Self-Directed arms, respectively.

Lessons learned: challenges and solutions
We have confronted important challenges while conducting Optimal Health with underserved participants and geographically diverse provider partners. Some of these are not unique to those experienced by other comparative effectiveness and clinical trial researchers, including lower than expected enrollment and issues around implementation and participant data collection. We devised solutions in order to meet the goals of Optimal Health.

Recruitment and randomization challenges
Potential participants were identified using Community Care administrative data which we provided to each community mental health center to support recruitment. Center staff were also provided with inclusion and exclusion criteria to use to make referrals to the study. In the original study design, we estimated recruitment at approximately 2000 individuals across eight provider sites, which proved challenging because the number of eligible participants at each site was fewer than originally estimated. We relied on historical claims data as a source for expected volume of eligible participants; however, after funding was obtained, we learned from our center stakeholders that we had overestimated the number of potential enrollees. Other reasons for our initial overestimation included failure to account for patients who move or lose and then regain Medicaid eligibility and individuals who leave treatment. Thus, the total number of individuals served in the system fluctuates. Despite adding three additional provider sites to mitigate the lower than expected number of participants, enrollment was less than planned. Furthermore, in compliance with the planned study enrollment period, participant follow-up period, and fixed duration of the PCORI contract, we ceased enrollment after 10 months. However, given cluster randomization at the provider site level, a decrease in the number of participants within clusters did not create a major scientific issue or compromise power significantly for Primary Aim 1.
In addition, after 6 months of recruitment, allocations to study arms using the minimization algorithm resulted in an enrollment imbalance between intervention arms, with the Provider-Supported and Self-Directed arms achieving an 87.9% and 62.2% enrollment rate, respectively. To achieve a greater degree of sample size balance between the two arms, we extended enrollment through the second data collection time point for the Self-Directed arm, yielding 120 participants for whom we will have four time points of data instead of five. The assumptions that we made based on available administrative data during randomization may have led to this imbalance which will be accounted for in the final analysis.
A final recruitment challenge, identified and resolved with direct stakeholder feedback, was that some participants had difficulty in reading the original consent forms. The study team created a web-based, audiorecorded, consent form that allowed participants to listen to and follow along as it was read aloud. This new approach helped to ensure consent comprehension for potential participants with low literacy and/or visual impairment, while concurrently enhancing enrollment efficiency.

Data collection challenges
We employed numerous strategies to ensure completeness and continuity in data collection, including close monitoring of participant tracking at each site, weekly contact with providers during data collection periods, weekly status reports, and availability of research staff to assist with data collection. The secure web portal system has mitigated challenges of collecting self-report data from our geographically diverse study sample. Our experience with this procedure illustrates the value of using technology to support studies of persons living in difficult to reach settings who ordinarily would have limited access to efficacy or effectiveness studies.
Maximizing the continuity of self-report data has proved to be challenging. Some study participants may have moved, lost/regained Medicaid eligibility, or were discharged from services for a variety of reasons, that is, achievement of treatment goals, changing providers, and engaged/disengaged/re-engaged with partnering *Data collection reflects self-report questionnaire completion and available claims data for the sample for that specific time point. It does not reflect data completion or data availability for participants across all time points. **Primary outcomes are analyzed using both self-report and claims data. As such, loss to follow-up for a participant is calculated at the point in which they no longer complete their self-report questionnaires for a given time point and when they are no longer Medicaid eligible as determined by the proxy of 80% Medicaid coverage in the 12-month period prior to the data collection time point. Whichever of these comes last is the point at which a participant is lost to follow-up.
providers. Furthermore, participant completion of selfreport data through the web portal required participants to sign on during a visit to the community mental health center or during a home visit by a case manager. If neither of these opportunities occurred during a data collection time period, individual participants were unable to complete self-report measures. Provider sites tracked reasons for missing self-report data. Completeness of secondary claims data was higher than self-report/primary data and remain available for each data collection time period, although variable Medicaid eligibility of a participant sometimes affected the availability and continuity of claims data. Overall, however, inclusion of secondary claims data allows for more continuous participant data despite some missing selfreport data and provides vital information about the use of services.

Intervention training and implementation challenges
The geographical dispersion of rural provider sites and partnering organizations posed a challenge during training and implementation. The rural location of many participating provider sites often does not allow for reliable Internet access. Given this geographic dispersion, the use of air cards and jet packs, which provide improved wireless Internet access for iPads and laptops, allowed for increased usage of Internet-based communication sources such as email and webinars and access to online intervention tools and self-report data collection.
Implementing our evidence-based interventions in real-world settings also presented numerous challenges. To enhance spread and sustainability, each organization has a leadership team and several staff to serve as ''wellness champions'' and trainers using a ''train-thetrainer'' model. Moreover, a Learning Collaborative approach was implemented to provide additional support and mutual learning among provider staff and the study team to monitor implementation, ensure intervention fidelity, and provide a forum for quality improvement and implementation support. This model, developed by the Institute for Healthcare Improvement, 55 provides a structured process to implement changes by adopting best practices and sustaining interventions over time utilizing adult learning principles, interactive training methods, and skill-focused learning. 55 Each study arm had a Learning Collaborative comprising agency leadership, clinical program leadership, and persons in recovery who met weekly within their respective agencies and monthly with the entire learning collaborative team.
Another challenge confronting intervention training and implementation was site staff turnover. To overcome this problem and ensure fidelity to the interventions, additional training and implementation support were created. The study team worked directly with provider sites to integrate intervention training into the staff curriculum and also developed an abridged, web-based training curriculum to support new hires.

Data analysis challenges
Optimal Health was originally designed to achieve a sample size of 2000 for 80% power to detect or rule out a difference between interventions with respect to the two primary aims and document the effects of several moderating variables. The calculations accounted for the expected intra-cluster correlation of outcomes among participants attending the same clinic, 56 as well as imbalances in cluster sizes and distribution of demographic measures between the two arms. We adjusted for imbalance using a coefficient of variation approach. 57 Although the final sample size was 1229, reassessment of power revealed that we had 80% power to address the two primary aims; the only moderator that could be examined in a confirmatory manner was gender. We have less power to assess the potentially mediating role of patient engagement and the impact of interventions on secondary outcomes.

Conclusion
Conducting patient-centered comparative effectiveness research in a real-world healthcare system is a challenging undertaking, requiring flexibility and adaptation within all study phases. Despite challenges faced during start-up and implementation, existing partnerships between Community Care and statewide community mental health providers, along with dedicated stakeholders, greatly facilitated implementation of this research. At each stage of Optimal Health, commitment of and engagement with collaborating stakeholders has been critical to identifying and overcoming challenges related to this work.
Experiences we have described are most relevant to researchers embarking on similarly complex studies in real-world settings, comparing the outcomes of effective approaches to select among available interventions for use with specific patients under well-defined circumstances. This goal of PCORI depends on patient and provider stakeholder participation in all phases of the research, from initial concept and design, including selection of interventions and outcomes, and advice regarding implementation. We will continue to prioritize stakeholder participation during interpretation of our findings.
Optimal Health addresses one of the Institute of Medicine's national priorities for comparative effectiveness research, that is, to avoid or minimize early mortality and comorbidity among people with serious mental illness. By adhering to the tenets of patient-centered comparative effectiveness research 58 and dissemination science, 59 we aim to achieve high levels of relevance and clinical meaningfulness required to promote rapid dissemination and adoption of findings into practice.