Impact of a Multi-Level, Multi-Component, System Intervention on HPV Vaccination in a Federally Qualified Health Center

High uptake of the human papillomavirus (HPV) vaccine has the potential to nearly eliminate cervical cancer and significantly reduce the burden of vaginal, vulvar, penile, anal, and oropharyngeal cancers in the population (1, 2). Research has documented significant declines in high-risk HPV infections in 15 countries and in precancerous cervical abnormalities in nine countries since the vaccine's introduction (3). However, low uptake in the United States threatens the realization of these population-level benefits. The HPV vaccine is available for free or at low cost to the vast majority of adolescents in the United States through private or public insurance or the Vaccines for Children Program, a federal program that provides no cost vaccine to uninsured, underinsured and Native American/Alaskan Native children (4). However, in 2020 only 40% of adolescents with public insurance and health care access had received the recommended HPV vaccine doses based on age of initiation per national recommendations (5, 6).

Suboptimal utilization of the HPV vaccine underscores the need for scalable and sustainable interventions to improve adolescent HPV vaccine uptake, particularly in safety net settings that serve a high proportion of publicly insured adolescents. Systematic reviews have documented the effectiveness of patient reminders, provider reminders, and system-based interventions in improving HPV vaccine initiation and completion rates in clinical settings (7–9). Many HPV vaccine promotion interventions based in clinical settings have targeted patients (reminders, recall) and providers [electronic health record (EHR) prompts, audit and feedback], revealing positive effects on HPV vaccination, although the magnitude of the effects has varied substantially across studies (10–19). Few prior studies have included a major focus on standardizing clinic workflows and processes to minimize missed opportunities for vaccination. Furthermore, a limited number of published studies have targeted both girls and boys, focused on low-income immigrants or Latinos, or been conducted in safety net settings such as Federally Qualified Health Centers (FQHC). FQHCs are community-based health centers that receive federal funding to provide comprehensive primary care services in medically underserved areas in the US FQHCs serve nearly 29 million patients each year, equivalent to 1 in 9 children and 1 in 3 people living in poverty (20, 21).

The study involved a collaboration between the University of California, Los Angeles (UCLA; Los Angeles, CA) and one of the largest FQHCs in the nation, which serves a primarily low income, Latino population in Los Angeles and Orange counties in California. Our prior research indicated that only 45% of adolescents in this FQHC system had received even one dose of the HPV vaccine, and only 52% of adolescents who initiated the series completed it (22). Qualitative research has documented a number of team- and clinic-level factors associated with higher HPV vaccination rates, including use of recommended provider communication strategies and designation of immunization champions (22). Based on this prior research and the literature, we designed and implemented a multi-level, multi-component, system intervention to improve HPV vaccine initiation and completion among adolescents served by the FQHC. We utilized a cluster randomized design to evaluate the effect of the intervention on HPV vaccination in adolescents and report trial results here.

Our goal was to utilize a cluster-randomized design to evaluate the effect of the multi-component, system-level intervention on HPV vaccination among adolescents, ages 11 to 17 served by the FQHC. We planned to match clinics on the size of the adolescent patient population and baseline HPV initiation rates and then randomized one clinic within each pair to usual care or the intervention. Administrative data showed that 90% of adolescent patients received primary care from 11 of the FQHC's 38 sites. We decided to exclude three of the 11 clinics from the trial; one was beginning implementation of another organizational practice change, one had a substantially larger adolescent patient population (n = 8,500), and the third was an outlier with a baseline HPV vaccine initiation rate of over 50%. As a result, the study was conducted in the remaining eight clinic sites, which served 53% of the FQHC's adolescent patients. Sites were successfully matched on adolescent population size and baseline HPV vaccination rates and one clinic from each pair was randomly assigned to the intervention (n = 4 clinics) and one to the usual care (n = 4 clinics) condition. Between January 2016 and March 2017, we implemented a multi-level system-level intervention consisting of provider and staff training, clinic-level audit and feedback, and workflow modifications to reduce missed opportunities at the clinics assigned to the intervention condition. Parents/caregivers of adolescent patients at intervention clinics received a reminder card for scheduling subsequent vaccine doses. The primary study outcome was clinic-level HPV vaccine initiation rate and the secondary outcome was HPV vaccine completion rate assessed via EHR data. We assessed clinic-level HPV vaccination rates among adolescent patients, ages 11 to 17 at quarterly intervals during a 12-month preintervention period (January 2015–December 2015) and a 15-month intervention implementation period (January 2016–March 2017). Change over time was compared between the intervention and control conditions.

The multi-level intervention targeted primary care providers, nurses/medical assistants, and parents/patients. Each component was developed in close collaboration with administrative leadership and providers, who provided input at each stage of the design, implementation, and evaluation process. The intervention was intentionally designed to be practical and scalable, and although strategies were implemented at multiple levels, they were relatively low-intensity. We also utilized a pragmatic approach to intervention delivery, monitoring, and follow-up (23). Two authors independently rated the study on PRECIS-2 dimensions consistent with published guidelines for its application (19). Their ratings were in exact agreement for 8 of the 9 dimensions and they came to a consensus the remaining dimension (i.e., setting) after discussion. One author had rated “setting” as “very pragmatic” (i.e., 5) and the other author as “rather pragmatic” (i.e., 4). The decision was made to categorize “setting” as “rather pragmatic” (i.e., 4). This process resulted in 6 of the dimensions rated “very pragmatic” and 3 dimensions (setting, organization, flexibility of delivery) rated “rather pragmatic.”

A major focus of the intervention was on implementing modifications to clinical workflow to reduce missed opportunities. The nature of workflow modifications was determined through a collaborative process between nurse managers and the investigator team. The study team met with provider leadership and nurse managers approximately monthly through in-person and telephone encounters. During initial meetings, nurse managers mapped usual care workflow procedures at each clinic and identified opportunities to enhance, reinforce, and monitor workflow procedures. Clinics were asked to design a systematic protocol for HPV vaccination procedures tailored to their workflow that included a number of key elements. All clinics were asked to check vaccination history and offer the HPV vaccine at every clinic visit, including sick visits. Another key element focused on improving team communication to ensure the primary care provider gave a strong vaccine recommendation to all eligible patients. To facilitate this process, medical assistants at intervention clinics placed a visual prompt on the exam room door (a “bee wise, immunize” sign) to notify providers that a patient was due for a vaccine. Finally, clinics were asked to implement a check out procedure to ensure that all eligible patients had been offered the vaccine before departing.

Primary care providers (pediatricians, family medicine physicians, physician assistants) participated in a 1 hour training and discussion session led by a physician vaccine champion and attended by at least one other investigator. The training was held during the noon hour with lunch served as this was time set aside for staff and provider meetings and educational sessions. The majority of primary care providers within intervention clinics attended the training, as it was required and feasibly given the training was completed during the lunch hour. The training focused on strengthening HPV vaccine recommendations, including the most appropriate communication techniques, bundling the HPV vaccine with other adolescent vaccinations, and framing the vaccine as cancer prevention (24). Providers were also reminded of the system-wide standing order for HPV vaccine delivery and were provided with feedback regarding vaccine initiation and completion rates for their home clinic in relation to other clinic sites within the organization.

Using a train-the-trainer approach, the study team provided an initial training and materials to nurse managers at each intervention clinic, who subsequently trained their team of nurses/medical assistants. The training addressed recommendations for HPV vaccination and improvements to clinical workflow procedures for HPV vaccine delivery. Each clinic site also implemented paper logs to track patients due for follow-up doses. The organization determined that they did not have the resources to automatically track patients through their population health management system. The study team met monthly with nurse managers at intervention clinics during the intervention period to assess fidelity of intervention implementation and to troubleshoot any challenges in implementation and clinic workflow. Nurse managers also monitored implementation through direct staff observation and quality reports and retrained staff as needed.

For parents of adolescents who received their first dose of the HPV vaccine, intervention clinic sites provided a reminder card indicating when the child was due for a second and third dose. The reminder card was a refrigerator magnet designed to signal the importance of the HPV vaccine and serve as a reminder/cue for when to schedule a follow-up appointment.

Usual care practice consisted of offering and providing the HPV vaccine at the discretion of the primary care provider. Shortly before study initiation, the organization implemented a system-wide standing order for the first dose of the HPV vaccine, which authorized clinical staff (licensed vocational nurses or medical assistants) to administer the HPV vaccine without a primary care provider order. Although the standing order was in place, clinics did not routinely offer the HPV vaccine outside of visits with the primary care provider. Around the time of the standing order implementation, nurses and medical assistants in both usual care and intervention clinics participated in a one-time web-based training containing basic information on the HPV vaccine and implementation of the standing order.

Data were extracted from the FQHC system's EHR system and administrative records. EHR data provided information on patient characteristics, medical encounters, and medical and vaccination history. Administrative data were used to obtain provider and clinic characteristics.

The study outcomes were HPV vaccine initiation (primary) and HPV vaccine completion (secondary). We defined HPV vaccine initiation as receipt of at least one HPV vaccine dose. Following the most recent 2016 Advisory Committee on Immunization Practices (ACIP) recommendations (5), we defined HPV vaccine completion according to the age at which the adolescent started the vaccination series. For those who initiated the series before age 15 (two-dose schedule), we defined completion as receipt of at least two doses administered at least 6 months apart. For those who initiated the series at or after age 15 (three-dose schedule), we defined completion as receipt of three doses, with at least 1 month between the first and second doses and 3 months between the second and third doses.

Patient demographic assessed included sex (male, female), race/ethnicity (Latino, White, American Indian/Alaskan Native, Asian, Black/African American, Native Hawaiian or Other Pacific Islander, Multiple, Unreported/Unknown), insurance status (insured, uninsured), insurance type (Medi-Cal, Medicare, other public insurance, or self-pay), and health care utilization (number of visits in past 2 years). We also measured receipt of other adolescent vaccinations, including vaccinations for influenza, tetanus, diphtheria, and pertussis (Tdap/DTap), meningococcal conjugate vaccine (MCV4), and measles, mumps, and rubella (MMR).

Patients were considered eligible for inclusion in the study if they were age eligible for HPV vaccine during the study period and were included in the analysis for all quarters in which they were determined to be an “active patient.” Active patients were defined as those who had at least one visit to a participating clinic site within the 2 years prior to the beginning of the quarter. The intervention and usual care groups were compared on baseline characteristics and initiation and completion rates using t tests and χ² tests. A difference-in-difference approach was utilized to assess the effectiveness of the intervention in increasing HPV vaccine rates (initiation and completion). First, quarterly change in clinic-level vaccination rates were calculated over the 12-month preintervention period and 15-month intervention period. Second, within each study condition, difference in rate of change in the preintervention period compared with the postintervention period was computed. The intervention effect was estimated by computing the difference-in-difference in the pre–post rate of change in vaccination between the intervention and usual care clinics. This approach was selected to accommodate a secular trend of increasing vaccination rates over time in all clinics. The effects were estimated using linear mixed models fit to clinic-level data, in which the outcome variable was clinic-level initiation or completion rates at quarterly intervals. The models used a piecewise linear spline that allowed different mean slopes for usual care and intervention clinics both before and after intervention implementation in January 2016. Difference-in-differences in mean quarterly change in initiation and completion was computed as follows: (quarterly change in vaccination rate at intervention clinics minus quarterly change at usual care clinics during the intervention period) minus (quarterly change in vaccination rate at intervention clinics minus quarterly change at usual care clinics during preintervention period). The models include random intercepts for clinics and allowed for heteroskedastic errors (error variances differing by clinic).

We estimated the following model:

where | ${Y}_{ij}$ | is the outcome, either initiation or completion, for cluster i = 1, …, I in quarter j = 1, …, J, where time is measured quarterly on the first day of January, April, July, and October. | ${X}_{tx}$ | is a {0,1} indicator for whether the cluster receives intervention ( | ${X}_{tx}$ | = 1) or remains in usual care ( | ${X}_{tx}$ | = 0) at any time. The term | ${X}_{quarter}$ | is a continuous predictor for the number of quarters elapsed since the beginning of the study. The term | ${X}_{postint}$ | is the number of quarters observed after the implementation of the intervention. The term u_i ∼ N(0,σ_c²) is a clinic-level random intercept to account for repeated measures of each clinic. Model fit including the specification of a linear slope was confirmed using residuals diagnostics. Analyses were conducted in R 3.6.1. (R Core Team, 2017; ref. 25). Estimates for males and females were obtained by restricting the analyses to these population subsets.

Data analyzed in this study were collected by the FQHC as part of routine patient care and extracted and transferred to the UCLA team through a data use agreement. Given patient confidentiality concerns, the data are not publicly available.

Table 1 provides characteristics of the baseline (January 1, 2015) sample. Of the 14,738 patients included in the baseline sample, 51% were female, 88% identified as Latino, and the mean age was 14 (SD = 2). The majority were insured by Medicaid (66%) and received their primary care from a provider specializing in pediatrics (52%). At baseline, vaccine initiation and completion rates were 58% and 42% respectively. No significant differences were observed between intervention versus control clinics in patient demographic characteristics. The numbers of active, eligible patients ranged from 14,773 to 15,571 per quarter over the course of the study.

Figure 1 displays mean quarterly HPV vaccine initiation and completion rates during the observation period (January 1, 2015–April 1, 2017) by study condition. At study onset, initiation rates ranged from 52.5% to 67.3% in usual care clinics and 41.0% to 69.3% in intervention clinics (P = 0.92, two-sample t test). Completion rates ranged from 20.4% to 42.4% in usual care and 18.2% to 41.6% in intervention clinics (P = 0.77, two-sample t test). There was an upward trend in in both conditions in initiation and completion over the study period.

Table 2 reports the results of the difference-in-differences analysis (i.e., pre—post difference between intervention and control clinics). During the preintervention period, HPV vaccine initiation rates increased by 1.0 percentage point on average per quarter in intervention clinics and 0.84 percentage points per quarter in usual care clinics (P = 0.33). During the intervention period, vaccine initiation rates increased by 1.94 percentage points per quarter on average at intervention clinics and 1.02 percentage points per quarter at usual care clinics (P < 0.001). The difference-in–differences in quarterly rates was 0.75 percentage points (P < 0.001). This finding translates into approximately 114 additional adolescents per quarter who initiated vaccination in the intervention compared with usual care clinics.

During the preintervention period, HPV vaccine completion rates increased by 0.49 percentage points on average per quarter among intervention clinics and 0.22 percentage points per quarter at usual care clinics (P = 0.07; Table 2). During the intervention period, vaccine completion rates increased by 1.89 percentage points per quarter among intervention clinics and 1.44 percentage points per quarter among usual care clinics (P = 0.001). The difference-in-differences in quarterly rates was 0.17 percentage points, which was not statistically significantly different from 0 (P = 0.21).

Among female adolescents, the intervention was associated with a significant improvement in HPV vaccine initiation (0.53 greater percentage point increase per quarter relative to usual care clinics, P < 0.001), but not vaccine completion (0.09 greater percentage point increase per quarter relative to usual care clinics, P = 0.43). Among male adolescents, the intervention had a significant effect on both HPV vaccine initiation (0.84 greater percentage point increase per quarter relative to usual care clinics, P < 0.001) and completion (0.65 greater percentage point increase relative to usual care clinics, P < 0.001).

The goal of this study was to design, implement, and evaluate the effect of a multi-level, multi-component system-based intervention to improve HPV vaccine uptake among adolescents in a large FQHC. Results revealed that the intervention produced a modest but significant effect on HPV vaccine initiation rates overall. Stratified analyses indicated a significant intervention effect on initiation among both girls and boys, and a significant effect on HPV vaccine series completion among boys. Given the large study population, we were able to estimate stable quarterly HPV vaccine initiation and completion rates and examine trends in these rates before and after the intervention was implemented. Though modest, this effect translated into a clinically meaningful number of additional adolescents vaccinated each quarter. Furthermore, our analysis suggests that the relative advantage experienced by the intervention group for both initiation and completion rates appeared to continue trending upwards, which could suggest additional impact of the intervention beyond our observation period.

Previous studies have reported a wide range of effect sizes for provider and system-level interventions. At the lower end of this range, Gilkey and colleagues (2014) documented a small but significant effect of a low-intensity application of Center for Disease Control and Prevention's (CDC) Assessment, Feedback, Incentives, and eXchange (AFIX) program on HPV vaccine update, which included a single session consultation with nurse vaccine coordinators at each clinic (13). The study showed a 1.5 percentage point greater increase in HPV vaccine initiation rates among 11- to 12-year–olds in the intervention compared with the control group at 5-month follow-up. However, this advantage was not retained by the time of 12-month follow-up. Perkins and colleagues (2015) observed a large effect of a more intensive provider-focused intervention on HPV vaccine initiation rates (OR girls = 1.6; OR boys = 11; ref. 16). The intervention included vaccine education, individualized feedback regarding HPV vaccination rates, and continuing education credits delivered by physician educators to providers through a series of six to eight sessions over a 12-month period. Consistent with this range, we observed a 3–percentage-point greater increase in HPV vaccine initiation rates and a 2–percentage-point greater increase in completion in intervention clinics compared with usual care clinics between pre- and post- periods.

Our intervention produced a modest effect, on the lower end of the range observed in the published literature, likely due to its low intensity and pragmatic nature. Our goal was to create an intervention, shaped by stakeholder input, that was feasible for implementation within a busy safety net clinic system. We utilized a train-the-trainer model that intentionally relied on clinic leadership and staff to implement the intervention. We hypothesized that this approach would lead to more buy-in from providers and staff within the organization and be more feasible to sustain long term. There were many changes in organizational leadership over the study period, including departure of the pediatrics director and vaccine champion, which may have reduced the consistency of intervention implementation. We opted not to conduct outreach to parents not presenting for a visit, because the FQHC had made a system-wide decision to focus their quality improvement efforts on reducing missed opportunities among patients presenting for care. We acknowledge that adding outreach to our intervention may have increased the intervention effect size.

The study has a number of limitations. First, the primary goal of the intervention was to increase HPV vaccine uptake. Although the FQHC had established procedures to check eligibility at every visit for any needed vaccine doses, our provider and staff training and feedback focused primarily on the HPV vaccine. It is not clear if focusing more broadly on all adolescent vaccinations may have impacted the effect of our intervention. Intervention materials did emphasize the importance of bundling the HPV vaccine with TdaP and meningococcal when appropriate, but this opportunity may have already passed for many older adolescents who received other adolescent vaccines on time. We acknowledge that the process collaborating with FQHC leadership and nurse managers to develop the intervention could have served as a prompt to address HPV vaccination prior to the official intervention start date. However, we expect that any effect would be minimal given how challenging it is to change clinical practice. In addition, this bias, if present, would reduce our chances of finding an intervention effect. Unfortunately, we were not able to collect detailed implementation data. Although our meetings with physicians, nurses, and other FQHC staff across the intervention suggested substantial variation in the degree to which strategies were implemented, these data are not fine grained enough to be considered in relation to the intervention's effectiveness. Future studies would benefit from more systematic assessment of implementation and intervention fidelity, such as consistency of staff workflow procedures and provider recommendations, as it is vital to determine which intervention elements are most feasible and potentially effective in enhancing HPV vaccination.

In addition to assessing implementation, future research should focus on long-term sustainability of observed intervention effects and the cost of these interventions. It will be important to identify interventions that not only produce a meaningful effect on vaccination, but also are sustainable and scalable.

C.M. Crespi reports grants from NIH during the conduct of the study. L.C. Chang reports grants from NCI during the conduct of the study. No disclosures were reported by the other authors.

B.A. Glenn: Conceptualization, resources, funding acquisition, methodology, writing–original draft, project administration, writing–review and editing. N.J. Nonzee: Conceptualization, writing–original draft, project administration, writing–review and editing. A.K. Herrmann: Conceptualization, methodology, project administration, writing–review and editing. C.M. Crespi: Conceptualization, data curation, software, formal analysis, funding acquisition, methodology, writing–original draft, writing–review and editing. G.G. Haroutunian: Conceptualization, resources, project administration, writing–review and editing. P. Sundin: Formal analysis, writing–review and editing. L.C. Chang: Data curation, formal analysis, writing–review and editing. R. Singhal: Conceptualization, funding acquisition, writing–review and editing. V.M. Taylor: Conceptualization, writing–review and editing. R. Bastani: Conceptualization, resources, data curation, funding acquisition, methodology, project administration, writing–review and editing.

This research was funded by NIH/NCI [grant no. R01 CA154549 to principal investigator (PI): R. Bastani; and grant no. P30 CA16042 (PI: M. Teitell/J. Gasson)]. N.J. Nonzee was also supported by NIH/National Center for Advancing Translational Science [NCATS; grant no. UCLA CTSI TL1 TR000121 (S. Dubinett); and grant no. TL1 TR001883 (PI: S. Ettner/M. Inkelas)] and Agency for Healthcare Research and Quality [AHRQ; grant no. T32 HS000046 (PI: T. Rice)]. We would like to acknowledge the vital contributions of the FQHC staff and leadership without whom the study would not have been possible.

The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.