Validation of BD Onclarity HPV Assay on Vaginal Self-Samples versus Cervical Samples Using the VALHUDES Protocol

Women who do not participate in cervical cancer screening remain at highest risk of developing invasive cervical cancer (ICC; refs. 1, 2). During the last decades, vaginal and urine self-sampling devices have been developed and received international attention of the scientific community and public health authorities. These self-sampling devices can be offered to underscreened women to increase the participation rate and prevent development of ICC (3, 4).

Many countries worldwide have introduced or have been piloting HPV-based cervical cancer screening, which showed high effectiveness in long-term protection against cervical cancer (5–10). In the current outlook of HPV-based cervical cancer screening programs, self-sampling plays an important role in reaching underscreened women (4). Offering self-samples has shown to be more effective in triggering a response than an invitation to visit a physician or nurse for collection of a cervical specimen (4).

More than 200 commercial HPV tests are currently available on the market, of which only few have been validated and approved for cervical cancer screening on clinician collected cervical specimens (11, 12). None of these tests have been approved or validated on self-samples, nevertheless several countries have implemented HPV testing on self-samples within HPV based screening programmes (8–10). Because of lack of data and international consensus on noncervical specimens it is necessary to further study, standardize, optimize, and validate HPV testing using self-sampling devices (13). Therefore, a protocol for validation of human papillomavirus assays and collection devices for HPV testing on self-samples and urine samples (VALHUDES) was established to provide scientific community and stakeholders with comparative accuracy data of HPV tests on noncervical and cervical specimens (14). The first evidence from VALHUDES suggested similar accuracy of the RealTime High-Risk HPV assay (Abbott GmbH) on first-void urine and vaginal self-samples compared with cervical specimens (15).

In this report, we evaluate the clinical performance of BD Onclarity HPV assay (Onclarity HPV; BD Diagnostics) on vaginal self-samples collected either with Evalyn Brush (Rovers Medical Devices) or Qvintip (Aprovix AB) among women referred to colposcopy. Because two self-sampling devices were used in the study, clinical accuracy was demonstrated for both devices separately and combined. In addition, we evaluated analytical performance of Onclarity HPV and compared cycle threshold (Ct) values across the specimens.

The VALHUDES framework (NCT03064087) was established to evaluate clinical performance of HPV assays on self-collected vaginal samples following a diagnostic test accuracy design (14). The first VALHUDES instalment recruited 523 women [median age = 40; interquartile range (IQR), 31–49 years] referred because of previous HPV infection or cervical abnormality to one of five Belgian colposcopy centers [University Hospitals of Antwerp (UZA), Brussels (UZ Brussels), Ghent (UZ Ghent), Liège (CHU de Liège), and the General Regional Hospital Heilig Hart Tienen (RZ Tienen)] between December 2017 and January 2020 (14, 15). Women were excluded from the study if they were pregnant, had hysterectomy or were incapable to understand and sign inform consent or refused to participate. Women were contacted by phone by the nurse inviting them to participate in the study. Upon arrival at the colposcopy center all study participants confirmed and signed the informed consent form.

At the colposcopy center women were instructed to collect a vaginal self-sample with Evalyn Brush (Rovers Medical Devices) or Qvintip (Aprovix AB) followed by the collection of cervical specimens by a gynecologist using the Cervex-Brush (Rovers Medical Devices) in agreement with European guidelines (16). Colposcopy centers in Antwerp and Ghent started to offer Evalyn Brush devices to women, whereas colposcopy centers in Brussels, Liege, and Tienen started using Qvintip. When about half of the required sample size was reached, sampling devices were switched across the study centers.

Cervical samples were suspended in 20 mL PreservCyt medium (Hologic, Inc.) by the gynecologist after collection. Cervical samples and dry vaginal brushes were stored at the colposcopy clinics at room temperature. Both samples were sent to Algemeen Medisch Laboratorium (AML; Antwerp) for storage and further processing within maximum of 6 days after collection. In the laboratory, the dry vaginal specimens were suspended into 20 mL PreservCyt medium. All cervical and vaginal samples were stored at 4°C for a maximum up to 3 months. Thereafter, samples were vortexed for 15 to 20 seconds, aliquoted into 1 mL aliquots and frozen at  −80°C (BioBank, BB190002).

VALHUDES trial (NCT03064087) was approved by the central Ethics Committee of the University Hospital of Antwerp/University of Antwerp (B300201733869) and the local Ethics Committees of all the other involved centers. The study was conducted in accordance of Declaration of Helsinki. Written informed consent was obtained from all study participants prior to enrolment.

HPV testing was performed using real-time PCR-based BD Onclarity HPV assay, which targets E6 and E7 oncogenes of 14 high-risk HPV (hrHPV) genotypes and human β-globin as an internal control. Onclarity HPV has been clinically validated for cervical cancer screening using clinician-collected cervical specimens transferred in PreservCyt or SurePath (BD Diagnostics) solution (11). The assay consists of three wells with four channels each. Each channel targets three groups of HPV types and β-globin. The assay identifies six individual genotypes (HPV16, 18, 31, 45, 51, and 52) and eight genotypes in three groups (HPV33/58, 35/39/68, and 56/59/66). HPV testing was performed in batches of 30 tests per run on the BD Viper LT system (17). Briefly, 0.5 mL of sample was transferred into an BD Viper LT diluent tube coded with the same unique lab sample code. The sample tube was placed in the BD Viper LT system for fully automated DNA extraction and amplification. Real-time PCR and result interpretation are automatically performed by the BD Viper LT software (17, 18). HPV testing on cervical specimens was considered as the comparator test and on vaginal self-samples as the index test. We used colposcopy and histology as the reference standard. Samples from women without abnormal colposcopy impression and without biopsy outcome were classified as <CIN2. If a particular patient had multiple biopsy outcomes, only the most severe outcome was considered for the analysis. Samples were considered HPV positive if cycle threshold Ct value was ≤38.3 for HPV16 and ≤34.2 for all other types, as defined by manufacturer.

Absolute and relative accuracy were estimated for the whole study population and for women of 30 years and older. Sensitivity analysis was performed to evaluate absolute and relative accuracy for women younger than 30. The McNemar test and paired 95% confidence intervals were computed to assess differences in sensitivity and specificity between vaginal and cervical specimens. Differences are not statistically significant when confidence intervals include unity (P_McN > 0.05). When the relative sensitivity and specificity is <1 and confidence intervals exclude unity (P_McN < 0.05), we conclude that Onclarity HPV is less sensitive or specific on vaginal self-samples than on the clinician-taken comparator sample. When the relative sensitivity and specificity is >1 and confidence intervals exclude unity (P_McN < 0.05), we conclude that Onclarity HPV is more sensitive or specific on vaginal self-samples than on clinician-taken sample. The power analysis is described in details in VALHUDES protocol (14).

Cohen κ was used to assess agreement between cervical and vaginal samples (combined and separately for each self-sampling device) as follows: 0.00–0.19 as poor, 0.20–0.39 as fair, 0.40–0.59 as moderate, 0.60–0.79 as good, and 0.80–1.00 as excellent concordance. Agreement and Cohen κ were estimated for overall hrHPV positivity, individual HPV genotypes and groups as follows: HPV16, HPV18, HPV31, HPV45, HPV51, HPV52, HPV33/58, HPV35/39/68, HPV56/59/66, and other hrHPV (other than 16/18). We used the Mann–Whitney test to compare the differences in viral and β-globin median Ct values between matched cervical and vaginal specimens (combined and separately for each self-sampling device). The Wilcoxon signed-rank test was used to assess the difference in viral and β-globin Ct values between Evalyn Brush and Qvintip samples for independent comparisons. We used boxplots indicating median Ct values, interquartile ranges, and extreme values (whiskers) for graphical demonstration of hrHPV and β-globin Ct values. To compare overall hrHPV signal strengths in case of multiple HPV infections, only the lowest Ct value was considered. Similarly, as three channels report β-globin Ct value, only the lowest signal was considered for comparison. Statistical analyses were performed using Stata 14.2 (College Station).

VALHUDES trial (NCT03064087) was approved by the central Ethics Committee of the University Hospital of Antwerp/University of Antwerp (B300201733869) and the local Ethics Committees of all the other involved centers. The study was conducted in accordance of Declaration of Helsinki. Written informed consent was obtained from all study participants prior to enrolment.

Overall, of 523 recruited women (median age = 40; IQR, 31–49 years), 24 samples were excluded due to major protocol violations (15, 19). HPV testing was performed on 499 duplets of samples, of which 485 (median age = 40 years; IQR, 31–49) were included in the main analysis. Six cervical and eight vaginal samples were excluded due to a retest failure on the respective specimens (Fig. 1). Median age in women with <CIN2 (41 years, IQR, 32–50) was higher than in women with CIN2+ (35 years, IQR, 29–44; P-value Mann–Whitney test = 0.002). Forty-seven percent (230/485) of samples were collected with Evalyn Brush (median age = 40; IQR, 31–47) and 52% (255/485) with Qvintip (median age = 40; IQR, 30–50). Characteristics of the study population are reported in Table 1.

Onclarity HPV assay on cervical samples was positive in 78 of 86 CIN2+ and 42 of 45 CIN3 cases with a corresponding absolute sensitivity of 91% (95% CI, 83%–96%) and 93% (95% CI, 82%–99%), respectively. Although, on vaginal self-samples 75 of 86 CIN2+ and 40 of 45 CIN3 cases were HPV positive corresponding to 87% (95% CI, 78%–93%) and 89% (95% CI, 76%–96%) absolute sensitivity values, respectively. Of 399 <CIN2 cases, 199 were HPV negative on cervical (specificity 50%; 95% CI, 45%–55%) and 216 were negative on vaginal samples (specificity 54%; 95% CI, 49%–59%; Table 2).

Clinical sensitivity for CIN2+ on vaginal samples collected with Evalyn Brush (0.92; 95% CI, 0.80–0.98) was higher than on samples collected with Qvintip (0.81; 95% CI, 0.65–0.92; ratio = 0.91; 95% CI, 0.77–1.08; Table 2).

Clinical sensitivity for CIN2+ on vaginal samples combined was not different from the sensitivity on cervical samples (ratio = 0.96; 95% CI, 0.90–1.03), whereas specificity was significantly higher (ratio = 1.09; 95% CI, 1.02–1.16; Table 3). For Evalyn Brush samples, similar sensitivity for CIN2+ was observed with 95% CI including unity (ratio = 1.00; 95% CI, 0.94–1.06) and higher specificity compared with cervical samples (ratio = 1.15; 95% CI, 1.05–1.25). Although sensitivity for CIN2+ on Qvintip samples versus cervical was 9% lower (ratio = 0.91; 95% CI, 0.79–1.04) with similar specificity (ratio = 1.03; 95% CI, 0.95–1.13), however including unity in the 95% CI (Table 3). Similar accuracy estimates were observed when restricting the analysis to women of 30 years and older compared with the total population (Tables 2 and 3). Absolute and relative accuracy estimates for women below 30 years are reported in Supplementary Tables S1 and S2.

Agreement between cervical and vaginal specimens was high with moderate to excellent Kappa values for hrHPV positivity and for individual HPV genotypes or groups of types (Table 4). Fifty percent (242/485) of samples were concordantly hrHPV positive on both specimen types, 7% (36/485) were positive only on cervical and 3% (16/485) only on vaginal specimens (Table 4). HPV test concordance between vaginal Evalyn Brush tended to be higher than the concordance between vaginal Qvintip and cervical samples (Supplementary Table S3).

Median Ct values for HPV and β-globin were lower in cervical compared with vaginal samples overall (Evalyn Brush + Qvintip), indicating higher concentrations of viral and human DNA in cervical specimens (Fig. 2; Supplementary Table S4A). Genotype-specific Ct values were lower in cervical samples for HPV16, HPV31, and HPV52 than in the vaginal samples (Supplementary Table S4). No difference in HPV and β-globin median Ct values was found between Evalyn Brush and Qvintip samples (Supplementary Table S4D). The median viral Ct was lower among women with CIN2+ than in women with <CIN2 in cervical samples, but not in vaginal (P values: 0.007 and 0.179, respectively; Supplementary Fig. S1).

In this study, we evaluated BD Onclarity HPV assay on vaginal-self samples collected with either Evalyn Brush or Qvintip device. Our findings demonstrated similar sensitivity and higher specificity of Onclarity HPV on vaginal versus cervical samples. Interestingly, accuracy of HPV testing on samples collected with Evalyn Brush was better than on Qvintip samples. These findings are in line with our previous report and studies by Cadman and colleagues and Jentschke and colleagues (20, 21). Our previous VALHUDES report evaluated accuracy of Abbott RealTime High-Risk HPV on vaginal samples. Sensitivity of the Abbott HPV assay on vaginal samples to detect CIN2+ was lower than on cervical, although a posterior cut-off optimization resulted in accuracy improvements. We also observed slightly better sensitivity of Abbott HPV assay on samples collected with Evalyn Brush, compared with Qvintip (19). In the Predictors 5.1 trial, four vaginal devices (dry flocked swab, Dacron swab, HerSwab, and Qvintip) were evaluated using Onclarity HPV assay in 600 women (20). Authors reported lower accuracy of Qvintip and HerSwab compared with dry flocked and Dacron swab. Jentschke and colleagues performed a direct comparison of Qvintip and Evalyn Brushes in a small study setting suggesting also slightly better sensitivity with the latter brush. Ørnskov and colleagues evaluated the performance of cobas 4800 HPV assay on vaginal samples collected with Evalyn Brush in a larger referral population with over 300 participants. In agreement with VALHUDES, sensitivity on vaginal samples was not different compared with cervical samples (22). A large randomized-non-inferiority trial also demonstrated a similar accuracy in vaginal versus cervical samples where the Evalyn brush was resuspended in 1.5 mL of PresevCyt media versus 20 mL for the physician-collected sample (23). A recent meta-analysis including 81 studies concluded that PCR-based HPV DNA assays on self-samples are non-inferior to clinician samples to detect CIN2+ or CIN3+ (4). Another meta-analysis showed high agreement of PCR-based HPV assays on self-collected vaginal versus clinician-collected cervical samples suggesting new extension for validation of HPV assays based on the agreement between the specimens (24). This may facilitate validation of HPV assays and devices by evaluating agreement between already validated assay/device with a new one.

Evidence that HPV testing provides better and longer protection against cervical cancer triggered many countries to revise the screening recommendations and switch from cytology to HPV-based cervical cancer screening (5, 6, 8, 9). Nevertheless, coverage of the screening programs remains lower than 70% recommended by WHO (9, 25). A considerable fraction of women does not participate in the screening programs and is at highest risk of developing cervical cancer; therefore, self-sampling strategies could be implemented to reach nonresponders and increase the participation rate. Self-sampling could also be used in resource poor countries, although, before implementation within the screening programmes, it is necessary to generate sufficient scientific evidence of HPV test accuracy on self-samples. Validation of HPV tests on cervical samples are well established. First, in 2009 Meijer guidelines were established outlining requirements of HPV tests to be suitable for primary HPV-based cervical cancer screening. Following criteria were established: sensitivity (≥90%) and specificity (≥98%) for CIN2+ compared with already clinically validated tests, and high inter- and intra-reproducibility (26). In 2012, extended guidelines were formulated for validation of HPV genotyping tests (VALGENT; ref. 27). Four VALGENT frameworks have been designed and more than 50 studies have been published since, to further optimize and provide evidence of HPV tests suitable for cervical cancer screening on cervical samples, but not on self-samples (28). Therefore, new international consensus is necessary to provide evidence and accuracy data of HPV test and devices on self-samples.

In the Netherlands, self-samples are offered to women since 2017, and in 2022 will change from an opt-in to opt-out program. The Dutch HPV-based cervical cancer screening program has published first data including 30,808 vaginal samples collected with Evalyn Brush and 456,207 cervical samples. Dutch colleagues reported slightly lower modelled sensitivity for CIN3+ of HPV testing with cobas 4800 HPV assay on samples collected with Evalyn Brush compared with the clinician-taken samples (9). In line with the Dutch report, the median viral Ct values in both VALHUDES studies were lower in cervical samples, reflecting higher viral load concentration in cervical specimens compared with vaginal (19). In addition, present VALHUDES demonstrated lower β-globin Ct value in cervical versus the vaginal samples, which may have been due to the 20 mL resuspension volume used for both sample types. A Danish self-sampling study showed the opposite trend, β-globin Ct value in vaginal samples taken with the Evalyn brush were approximately 2 Cts lower than corresponding cervical samples, where the self-sample was resuspended in 3 mL versus 10 mL for the physician sample (SurePath preservative media; ref. 29). Interpretation of Ct values data and resuspension volumes of vaginal samples may require further research.

Interestingly, the Onclarity HPV testing on vaginal samples demonstrated better specificity compared with Onclarity testing on cervical samples. Vaginal samples were resuspended in 20 mL PreservCyt solution, which is the standard transport volume for cervical specimens but not for vaginal. In addition, cervical samples were collected by a gynecologist targeting the transformation zone where virus replication occurs (30), whereas vaginal samples were taken by women from the vaginal tract. It is known that the hrHPV viral load is highest in scraped cervical cells and decreases progressively when samples are taken more distally: upper vagina > lower vagina > perineum (31). Accordingly, our data demonstrated higher viral Ct values in vaginal samples indicating lower viral load. The dilution in a quite large volume of transport medium and lower amount of virus in the vagina both may explain lower analytical sensitivity and higher analytical and clinical specificity in vaginal specimens (nondetection of small amounts of virus in women without high-grade lesions).

According to the STARD guidelines the evaluation of an assay should be conducted in the environment where the assay will be used (32). HPV testing on self-samples will have its main application in primary cervical cancer screening. However, VALHUDES is based on the findings of our previous meta-analyses, which demonstrated that the relative sensitivity and relative specificity of HPV testing on self-collected versus clinician-collected samples were similar in primary screening and in follow-up settings (3, 4). There are several advantages of running the study in a colposcopy setting including: efficiency (∼500 patients are sufficient to reach required power), shorter duration of enrolment, and lack of verification bias which is a major issue in primary screening studies. Nevertheless, besides the strengths of the VALHUDES design we recognize that the enrolment in a colposcopy setting as a weakness with respect to the spectrum of disease. Another limitation is that age range of our study population was 19 to 70 years, whereas recommended HPV based screening age is above 29 years. Yet, the clinical performance in screening recommended age group was similar to the total population. Finally, women with normal colposcopy impressions and without reference biopsy were classified as non CIN2+. This classification could have impacted absolute accuracy estimates, as small fraction of women could have been misdiagnosed by colposcopists, yielding a possible overestimation of absolute sensitivity and underestimation of absolute specificity for both samples types. However, such a misclassification bias, if present, occurs in both types of samples (since colposcopists are not aware of HPV test results on either type) and, therefore, influence on relative accuracy would be limited with a possible tendency of relative accuracy estimates towards unity (33).

Our study demonstrated similar accuracy of hrHPV testing with Onclarity HPV assay on self-collected vaginal samples as on clinician-collected cervical specimens, in particular when the self-samples were collected with Evalyn Brush.

A. Latsuzbaia reports grants from Horizon 2020 Framework Programme for Research and Innovation of the EU (grant no. 847845; RISCC Network) and grants and nonfinancial support from VALHUDES during the conduct of the study. S. Van Keer reports grants from University of Antwerp and Research Foundation – Flanders (FWO), and other support from Becton Dickinson during the conduct of the study; grants from Research Foundation – Flanders (FWO), University of Antwerp, and Eurostars (H2020) outside the submitted work; and The University of Antwerp received payment for participation of S Van Keer in an Advisory Board of Novosanis (Subsidiary of OraSure Technologies Inc.). All funds are handled and managed by the University of Antwerp. E. Peeters reports that E. Peeters was supported by the RISCC Network funded by the Horizon 2020 Framework of DGResearch and Innovation, European Commission, Brussels, Belgium (grant no. 847845) at the time of data analysis. A. Vorsters reports other support from Becton Dickinson and grants from University of Antwerp during the conduct of the study; grants from Merck, GSK, Novosanis, Hologic, Abbott outside of the submitted work; also has a patent for liquid collection device, liquid sampler, kit of parts, and method for assembly issued and licensed; and reports employment with Uantwerp spinoff company Novosanis now fully owned by Orasure in which does not have financial interest. M. Arbyn reports grants from Horizon 2020 Framework Programme for Research and Innovation of the EU (grant no. 847845; RISCC Network) and grants and nonfinancial support from VALHUDES during the conduct of the study. No disclosures were reported by the other authors. The VALHUDES project is a researcher-induced study, designed by Sciensano (Principal Investigator; Brussels, Belgium), CEV (University of Antwerp, Antwerp, Belgium), and AML (Antwerp, Belgium). Manufacturers of HPV assays and devices can participate in the VALHUDES framework contributing financial support and equipment for laboratory testing and statistical analysis under the condition of accepting independent publication of results. This research was supported by BD (BD Diagnostics, Sparks, MD, USA), Novosanis NV (Wijnegem, Belgium), University of Antwerp (Antwerp, Belgium). The study group received sample collection devices from Rovers Medical Devices B.V. (Oss, The Netherlands) and Aprovix AB (Uppsala, Sweden).

A. Latsuzbaia: Data curation, formal analysis, visualization, writing–original draft, writing–review and editing. D. Vanden Broeck: Conceptualization, resources, data curation, funding acquisition, validation, methodology, project administration, writing–review and editing. S. Van Keer: Conceptualization, resources, data curation, funding acquisition, validation, methodology, project administration, writing–review and editing. S. Weyers: Investigation, writing–review and editing. G. Donders: Investigation, writing–review and editing. J. Doyen: Investigation, writing–review and editing. W. Tjalma: Investigation, writing–review and editing. P. De Sutter: Investigation, writing–review and editing. E. Peeters: Conceptualization, resources, data curation, funding acquisition, validation, methodology, project administration, writing–review and editing. A. Vorsters: Conceptualization, resources, data curation, funding acquisition, validation, investigation, methodology, writing–review and editing, project administration. M. Arbyn: Conceptualization, resources, data curation, formal analysis, supervision, funding acquisition, validation, investigation, methodology, project administration, writing–review and editing.

We thank the team members of the Gynaecology Departments of Antwerp University Hospital, Ghent University Hospital, General Regional Hospital Heilig Hart Tienen – Femicare vzw, UZ Brussel – VUB and University Hospital Liège for the study recruitment. We thank L De Baere (AML) and D Maes (AML) for the laboratory work. M. Arbyn and A. Latsuzbaia were supported by the RISCC Network (grant no. 8478459) funded by the Horizon 2020 Program for Research and Innovation of the European Commission (Brussels, Belgium). S. Van Keer was supported by a junior postdoctoral fellowship of the Research Foundation – Flanders (1240220N).

The publication costs of this article were defrayed in part by the payment of publication fees. Therefore, and solely to indicate this fact, this article is hereby marked “advertisement” in accordance with 18 USC section 1734.