The STOP-CRC trial investigators deployed many strategies to ensure the inclusion of a diverse population in their trial, including piloting the trial in the most diverse clinic, refining the materials based on patient feedback, assembling an advisory board, and conducting plan-do-study-act cycles at the sites. This video describes their experiences with building partnerships in order to plan for diversity.
In a new article from the NIH Pragmatic Trials Collaboratory Biostatistics and Study Design Core, the authors share analytic considerations for cluster randomized trials with hierarchical nesting of participants within clusters. The authors illustrate the problem using theoretical derivations, a simulation study, and data from the STOP CRC NIH Collaboratory Trial as an example.
“We conclude that an analysis using both an exchangeable working correlation matrix and weighting by inverse cluster size, which may be considered the natural analytic approach, can lead to incorrect results. That is, two weights make a wrong. The bias is minimal when there is homogeneity of treatment effects according to cluster size but unacceptable when there is heterogeneity of treatment effects according to cluster size. In addition, we show that only an analysis with an independence working correlation matrix and weighting by inverse cluster size always provides valid results for the UATE [unit average treatment effect] estimand.”
BeatPain Utah Julie Fritz, PhD, PT Associate Dean for Research, College of Health Adjunct Professor, Orthopaedic Surgery Distinguished Professor, Physical Therapy & Athletic Training Adjunct Professor, Orthopaedic Surgery Operations University of Utah
David Wetter, PhD, MS Professor, Population Health Sciences Adjunct Professor, Psychology University of Utah
STOP CRC Gloria Coronado, PhD Distinguished Investigator Mitch Greenlick Endowed Scientist for Health Disparities Kaiser Permanente Center for Health Research
Amanda Petrik, MS Sr. Research Associate Center for Health Research Kaiser Permanente Northwest
Guest Moderator:
Wendy Weber, ND, PhD, MPH Chief, Clinical Research in Complementary and Integrative Health Branch Division of Extramural Research National Center for Complementary and Integrative Health (NCCIH) National Institutes of Health (NIH)
Topic
Inclusion of Diverse Participants in Pragmatic Clinical Trials: Planning for Diversity – Stakeholder Engagement and Site Selection to Maximize Diversity
Keywords
Participant diversity; Stakeholder engagement strategies; STOP CRC; BeatPain Utah; Health equity; Community health centers; Plan-Do-Study-Act
Key Points
Disparities in pain prevalence and pain management are well established. In particular, odds of receiving opioids for pain management are greater, and odds of nonpharmacologic care lower, in rural, low income, and Latinx communities.
It is essential that communities and researchers come together to create long-term solutions to prevent cancer, chronic and infectious disease, and improve health among underserved populations. Research projects should be driven by the priorities of community partners.
Study advisory boards can include health center leaders, patient advocates, legislators, and community organization leaders. These boards provide local context and, in the case of STOP CRC, they identified policy changes that were needed around access to colorectal cancer screening.
Discussion Themes
Sharing information and resources with community partners—and responding to their needs and building their capacity—helps to build trust around medicine and research.
To what extent is it important to show evidence of interest in and respect for the community beyond the specific focus of your initiative? For example, participating in important community activities and developing a deeper understanding of the culture.
We must design and plan for sustainability at the outset, and we must make targeted, specific efforts to ensure the inclusion of diverse populations in clinical trials.
Another way to increase diversity is to prepare and train more scientists and investigators from diverse populations.
Guest Moderator: Wendy Weber, ND, PhD, MPH
Chief, Clinical Research in Complementary and Integrative Health Branch
Division of Extramural Research
National Center for Complementary and Integrative Health (NCCIH)
National Institutes of Health (NIH)
Panel: BeatPain Utah
Julie Fritz, PhD, PT
Associate Dean for Research, College of Health
Adjunct Professor, Orthopaedic Surgery
Distinguished Professor, Physical Therapy & Athletic Training
Adjunct Professor, Orthopaedic Surgery Operations
University of Utah
David Wetter, PhD, MS
Professor, Population Health Sciences
Adjunct Professor, Psychology
University of Utah
STOP CRC
Gloria Coronado, PhD
Distinguished Investigator
Mitch Greenlick Endowed Scientist for Health Disparities
Kaiser Permanente Center for Health Research
Amanda Petrik, MS
Sr. Research Associate
Center for Health Research
Kaiser Permanente Northwest
Topic: Inclusion of Diverse Participants in Pragmatic Clinical Trials: Planning for Diversity – Stakeholder Engagement and Site Selection to Maximize Diversity
Date: Friday, June 4, 2021, 1:00-2:00 p.m. ET
You must log in to the URL first.
Click ‘Audio Conference’
Choose ‘I will call in’, select the Toll number.
Dial in using the information from the dialog box that appears.
Be certain to use the Access Code AND the Attendee ID.
Troubleshooting:
If the URL above does not work, go to dukemed.webex.com and enter:
Meeting Number: 120 909 9614
Meeting Password: 1234
For Audio ONLY:
Call-in toll number (US/Canada): 1-650-479-3207
Access code: 120 909 9614
NOTE: For Toll-free users, the call-back (call me) services are also available.
NIH Collaboratory researchers in 2020 reported study results, generated new knowledge, and developed innovative research methods in pragmatic clinical trials. Their work included insights from the Coordinating Center and Core Working Groups, analyses from the NIH Collaboratory Distributed Research Network, and results and methodological approaches from the NIH Collaboratory Trials.
So far this year, the NIH Collaboratory has produced more than 3 dozen articles in the peer-reviewed literature, including the primary results of the PROVEN and LIRE trials, the study design of ACP PEACE, insights into the COVID-19 pandemic from TSOS and EMBED, and more:
In 2012, the NIH Common Fund established the NIH Health Care Systems Research Collaboratory. The goal of the program is to improve the way clinical trials are conducted by creating an infrastructure for collaborative research with healthcare systems. The NIH Collaboratory launched with a Coordinating Center, Core Working Groups, and NIH Collaboratory Trials to conduct embedded pragmatic clinical trials (ePCTs) in partnership with healthcare system leaders and to work collaboratively with the NIH to solve problems as they arise, develop best practices, and share lessons and resources to with others conducting ePCTs.
Collaboratory Mission: Strengthen the national capacity to implement cost-effective large-scale research studies that engage healthcare delivery organizations as research partners.
With the first round of NIH Collaboratory Trials nearing completion, the project teams are beginning to publish results and share lessons with other researchers. We asked the principal investigators of the most recently completed projects to share insights about the important contributions of their studies.
Congratulations on finishing your NIH Collaboratory Trial: What do you think is the most important contribution of your study?
ABATE was conducted to determine whether routine bathing and showering with chlorhexidine soap would reduce multidrug-resistant organisms and bloodstream infections compared with usual care. The trial was conducted in 53 HCA Healthcare hospitals (194 non–critical care units)and included 340,000 patients in the intervention period.
“We found that there was no overall benefit to universal antiseptic bathing in non–intensive care units (ICUs). This is in stark contrast to the huge benefit demonstrated in ICUs in the REDUCE-MRSA trial, and may reflect the fact that non–critical care patients stay only a few days in the hospital and are less likely to develop infection. Nevertheless, we did find that antiseptic bathing and nasal decolonization reduced bloodstream infections and antibiotic-resistant organisms by over 30% in patients with devices outside of the ICU. This is important because they are 10% of the non-ICU population, but responsible for over half of bloodstream infections. They provide a valuable targeted population who appear to benefit from this intervention.”
LIRE was conducted to test the effectiveness of a simple and inexpensive intervention: inserting epidemiologic benchmarks into lumbar spine imaging reports. The goal of the trial was to reduce subsequent tests and treatments, including cross-sectional imaging (such as magnetic resonance imaging and computed tomography), opioid prescriptions, spinal injections, or surgery.
“I think that one of the most important contributions of the LIRE trial was demonstrating the feasibility of randomizing hundreds of thousands of patients to receive or not receive an intervention that we inserted into the radiology report. Before our trial began, there was a fair amount of skepticism about whether radiologists would accept routinely inserting prevalence information into their reports on a wide scale. We showed without a doubt that it was feasible.”
PPACT was designed to assess the potential benefit of helping patients adopt self-management skills for chronic pain, limit use of opioid medications, and identify factors amenable to treatment in the primary care setting in three Kaiser Permanente (Northwest, Georgia, and Hawaii) involving approximately 800 patients.
“We started a trial when everybody was still uncertain about what the trade-offs between external validity (and real-world issues that are important for implementation) and the rigor of internal validity. I don’t know if we got that right. There was an assumption that the trial needed to be cluster randomized, and I think it’s informative that only 1 of the 11 NIH-DOD-VA Pain Management Collaboratory trials was cluster randomized. We needed to be able to incubate, have embedded teams stay over time, and really shift the culture. Patients needed to get used to the idea of non-pharmacotherapy over several months, and we may have had more success if we had individually randomized our cohort. I learned a lot in this process.”
PROVEN was designed to evaluate the effectiveness of advance care planning video tools in the nursing home setting by partnering with 2 large healthcare systems that operate 492 nursing homes nationwide.
“PROVEN found an ACP Video Program did not significantly impact hospital transfers, burdensome treatments, or hospice enrollment among nursing home residents with advanced illness, however intervention fidelity was low. Nonetheless, PROVEN was one of the first large pragmatic trials conducted in US nursing homes. Thus, I feel its greatest contribution was setting a foundation of knowledge for the field in terms of methodologies that enable pragmatic trials in this setting and challenges to overcome.”
STOP CRC was conducted to determine whether EHR-embedded tools and clinic staff training in how to implement a mailed fecal immunochemical test (FIT) outreach program could increase colorectal cancer screening uptake among patients with historically lower CRC screening rates and worse CRC outcomes, such as those with low income, or who are on Medicaid or underinsured. STOP CRC was conducted in 26 Federally Qualified Health Centers (FQHCs) in Oregon and California and involved approximately 41,000 patients.
“The ability to work with FQHCs and their new electronic data systems was an important contribution. FQHC settings are not organized healthcare systems, such as Kaiser Permanente, where research is more routine. I think we contributed to the success of this type of research and enabled the FHQCs’ ability to do more of it.” — Dr. Beverly Green
“Our study designed real-time electronic health record tools to allow clinics to mail cancer screening tests to adults who were overdue. We learned a lot about the challenges clinics faced in implementing the program. We shared our learnings with hundreds of additional community clinics in Washington, Oregon, California to help them anticipate and overcome these challenges.”— Dr. Gloria Coronado
TiME was conducted to determine whether treatment with hemodialysis sessions that are longer than many patients in the United States currently receive reduces the high rate of mortality among people being treated with thrice-weekly maintenance hemodialysis. The trial was conducted in 2 large US dialysis provider organizations, DaVita, Inc. and Fresenius Medical Care – North America, and included 266 outpatient dialysis facilities with 7035 patients.
“TiME established a model for conducting real-world research for a group of patients for whom there is very little clinical trial data. Many of the approaches and lessons from TiME are now being applied to a new set of pragmatic trials in dialysis that are being conducted in the US and internationally. In my view, TiME’s greatest contribution was to create a foundation for ongoing efficient and rigorous evidence generation in dialysis.”
Data and resources from the NIH Collaboratory Trials are posted on the NIH Collaboratory’s Data and Resource Sharing page in the coming months. As part of the program’s commitment to sharing, all NIH Collaboratory Trials are expected to share data and resources, such as protocols, consent documents, public use datasets, computable phenotypes, and analytic code.
NIH Collaboratory researchers in 2019 continued to generate new knowledge and research methods in pragmatic clinical trials. Their work included insights from the Coordinating Center and Core Working Groups, large-scale analyses of data from the NIH Collaboratory Distributed Research Network, and results and innovative methodological approaches from the NIH Collaboratory Trials.
So far this year, the NIH Collaboratory has produced nearly 3 dozen articles in the peer-reviewed literature, including the primary results of the ABATE Infection trial, confirmation by the TiME trial of the feasibility of embedding large pragmatic trials in clinical care, and more:
A cost-effectiveness analysis of the Strategies and Opportunities to Stop Colorectal Cancer in Priority Populations (STOP CRC) trial, an NIH Collaboratory Trial, revealed wide variation across participating health centers. The study’s findings reflect the complexity of implementing an intervention in pragmatic research involving community health clinics with diverse patient populations, clinic structures, and resources.
The STOP CRC trial tested a program to improve colorectal cancer screening rates in 26 clinics within 8 federal qualified health centers in California and Oregon. Intervention clinics embedded a tool in the electronic health record to identify patients who were overdue for screening and mailed a fecal immunochemical test (FIT) kit to these patients. Screening rates were higher overall in intervention clinics than in clinics that practiced usual care, despite low and highly variable rates of implementation of the program among participating clinics.
In the subsequent cost-effectiveness study, variability in program implementation was likewise a key factor in the results. Intervention delivery costs were highly variable across health centers, and the incremental cost-effectiveness of the intervention was diminished somewhat because usual care clinics generated more colonoscopies than intervention clinics after abnormal FIT results.
The study has implications for the design of implementation strategies in pragmatic trials that are embedded in diverse community health centers.
The STOP CRC trial was supported within the NIH Collaboratory by a cooperative agreement from the National Cancer Institute and received logistical and technical support from the NIH Collaboratory Coordinating Center. Download a study snapshot of the STOP CRC trial, and learn more about the NIH Collaboratory Trials.
The primary results of the Strategies and Opportunities to Stop Colorectal Cancer in Priority Populations (STOP CRC) trial, an NIH Collaboratory Trial, were published online this week in JAMA Internal Medicine. The analysis found that colorectal cancer screening rates were higher in community clinics that implemented a mailed fecal immunochemical test (FIT) outreach program than in clinics that practiced usual care. The improved screening rates occurred despite low and highly variable rates of implementation of the program among participating clinics.
Almost half of eligible adults in the United States are not up to date with recommended screening for colorectal cancer, the second leading cause of cancer-related deaths. Screening rates are especially low among racial/ethnic minority and low-income populations, including those served at federal qualified health center clinics.
The STOP CRC trial tested a program to improve colorectal cancer screening rates in 26 clinics within 8 federally qualified health centers. The intervention involved embedding a tool in the electronic health record to identify patients who were overdue for colorectal cancer screening, mailing a FIT kit and reminder letter to eligible patients, and implementing a practice improvement process at participating clinics. Of the 26 clinics in the study, 13 received the intervention and 13 practiced usual care.
Compared with clinics that practiced usual care, intervention clinics had a significantly higher proportion of participants who completed a FIT (3.4 percentage points) and any colorectal cancer screening (3.8 percentage points). The higher screening rates occurred despite another important finding of the study, that low rates of implementation of the intervention were common. Higher rates of implementation were correlated with higher rates of FIT completion.
The STOP CRC experience offers lessons on how to use electronic health records to improve guideline-based screening. In a recent NIH Collaboratory Grand Rounds, investigators Dr. Gloria Coronado and Dr. Beverly Green presented findings from the trial and lessons from the implementation of the intervention. Download a study snapshot about the STOP CRC trial.
A new article published in the journal Trials provides a look at how the Pragmatic–Explanatory Continuum Indicator Summary, or PRECIS, rating system can be applied to clinical trials designs in order to examine where a given study sits on the spectrum of explanatory versus pragmatic clinical trials.
The PRECIS-2 criteria are used to rate study designs as more or less “pragmatic” according to multiple domains that include participant eligibility, recruitment methods, setting, organization, analysis methods, primary outcomes, and more. In this context, “pragmatic” refers to trials that are designed to study a therapy or intervention in a “real world” setting similar or identical to the one in which the therapy will actually be used. Pragmatic trials stand in contrast to explanatory trials, which are typically designed to demonstrate the safety and efficacy of an intervention under highly controlled conditions and in carefully selected groups of participants, but which may also be difficult to generalize to larger or more varied populations.
PRECIS-2 Wheel. Kirsty Loudon et al. BMJ 2015;350:bmj.h2147. Copyright 2015 by British Medical Journal Publishing Group. Used by permission.
Clinical trials are almost never wholly “explanatory” or wholly “pragmatic.” Instead, many studies exist somewhere on a spectrum between these two categories. However, understanding how these different attributes apply to trials can help researchers design studies that are optimally fit for purpose, whether that purpose is to describe a biological mechanism (as in an explanatory trial) or to show how effective an intervention is when used across a broad population of patients (as in a pragmatic trial).
In their article in Trials, authors Karin Johnson, Gila Neta, and colleagues applied PRECIS-2 criteria to 5 pragmatic clinical trials (PCTs) being conducted through the NIH Collaboratory. Each trial was found to rate as “highly pragmatic” across the multiple PRECIS-2 domains, highlighting the tool’s potential usefulness in guiding decisions about study design, but also revealing a number of challenges in applying it and interpreting the results.
Study authors Johnson and Neta will be discussing their findings during the NIH Collaboratory’s Grand Rounds on Friday, January 22, 2016 (an archived version of the presentation will be available the following week).
Johnson KE, Neta G, Dember LM, Coronado GD, Suls J, Chambers DA, Rundell S, Smith DH, Liu B, Taplin S, Stoney CM, Farrell MM, Glasgow RE. Use of PRECIS ratings in the National Institutes of Health (NIH) Health Care Systems Research Collaboratory. Trials. 2016;17(1):32. doi: 10.1186/s13063-016-1158-y. PMID: 26772801. PMCID: PMC4715340.
You can read more about the NIH Collaboratory PCTs featured as part of this project at the following links:ABATE (Active Bathing to Eliminate Infection)
LIRE (A pragmatic trial of Lumbar Image Reporting with Epidemiology)
PPACT (Collaborative Care for Chronic Pain in Primary Care)
STOP-CRC (Strategies & Opportunities to Stop Colon Cancer in Priority Populations)
TIME (Time to Reduce Mortality in End-Stage Renal Disease)
Additional Resources
An introductory slide set on PCTs (by study author Karin Johnson) is available from the Living Textbook:
Introduction to Pragmatic Clinical Trials
The University of Colorado Denver - Anschutz Medical Campus publishes an electronic textbook on pragmatic trials:
Pragmatic Trials: A workshop Handbook
In a 
NIH Collaboratory researchers in 2020 reported study results, generated new knowledge, and developed innovative research methods in pragmatic clinical trials. Their work included insights from the Coordinating Center and 
