External Validation of Diagnostic Models to Estimate the Risk of Malignancy in Adnexal Masses

The most important role of the sonologist examining patients with an adnexal mass is to distinguish between benign and malignant masses to optimally triage patients. Benign masses can be managed conservatively or—if technically feasible—be removed by laparoscopy. However, patients with masses that are presumed to be malignant should be referred to gynecologic oncologists for proper staging and debulking (1–3). A mistake in either direction could worsen the patient's prognosis or create unnecessary anxiety, cost, and extensive surgery.

The sonologist can use different strategies to assess the risk of malignancy but there is still no consensus as to the optimal approach.

Subjective assessment of the sonographic characteristics of the mass is an excellent method to discriminate between benign and malignant but only in the hands of experienced ultrasound examiners (4, 5).

The American College of Obstetricians and Gynecologists (ACOG) uses guidelines based on demographic, sonographic, and biochemical parameters for the referral of pre- and postmenopausal patients with an adnexal mass to a gynecologic oncology center. These guidelines reach a high sensitivity with regard to malignancy for postmenopausal patients (93%) but a low sensitivity for premenopausal patients (79%) and poor specificity for both groups (70% and 60% for pre-and postmenopausal patients, respectively; ref. 6).

Several mathematical models or scoring systems have been developed to be used for discrimination between benign and malignant adnexal masses with encouraging results (7–14). Unfortunately, when tested prospectively, most models did worse than in the studies where they had been created, and they did not achieve the performance of experienced sonologists using subjective assessment of the ultrasound image (4, 15–18). The Royal College of Obstetricians and Gynecologists (19) and a recent review article by Geomini and colleagues (20) recommend the risk of malignancy index (RMI; ref. 7) as the method of choice for characterizing ovarian pathology as benign or malignant. The RMI is a scoring system that is derived from a logistic regression formula that combines menopausal status with serum CA125 and ultrasound variables (multilocular cysts, solid areas, metastases, ascites, bilaterality; ref. 7).

Recently, the International Ovarian Tumor Analysis (IOTA) study group developed several mathematical models to calculate the individual risk of malignancy in an adnexal mass. The details of the IOTA studies and the ultrasound protocol have been published previously (16, 21). The IOTA studies are international multicenter studies where women with adnexal masses are examined with ultrasound following a standardized research protocol with the aim of developing and validating mathematical models to estimate the risk of malignancy in adnexal masses. Patients with an adnexal mass were examined by an experienced ultrasound examiner following a standardized protocol. The ultrasound protocol required the assessment of more than 40 clinical and ultrasound variables. It was not mandatory to look for metastases, nor was it required to take a blood sample for the measurement of CA125 but this was encouraged. All patients had to undergo surgery within 4 months after the ultrasound examination to have the final histologic outcome and stage (in case of malignancy). Using this approach, the IOTA group aimed to overcome the problems of generalizability, sample size, and lack of standardized ultrasound criteria associated with previous studies. During the IOTA study phase 1, 1,066 patients were included in 9 centers from 5 countries (Belgium, Italy, Sweden, France, United Kingdom), and 11 mathematical models to estimate the risk of malignancy in adnexal masses were developed using different modeling techniques. Four different types of mathematical models were developed to evaluate which type of model did best (scoring system, logistic regression models, artificial neural networks (ANN), and vector machine models). On internal and temporal validation in the centers that previously developed the models, they had excellent diagnostic performance (16, 22–25). After temporal validation, all models had similar performance (AUCs between 0.945 and 0.950). Because logistic regression models are less complex and easier to use, the logistic regression model LR1 with 12 predictors and the simpler model LR2 with 6 predictors were from then on selected as the main IOTA models.

A crucial step before introducing a model in clinical practice involves external validation of performance in centers unrelated to where the models were developed (26, 27). Therefore, the aims of this study (IOTA phase 2) were to externally validate the ability of the IOTA models to discriminate between benign and malignant adnexal masses and to compare their performance with that of the RMI and other non-IOTA models developed to assess the risk of malignancy in adnexal masses before surgery.

This is a prospective cross-sectional diagnostic accuracy study of patients presenting with at least one persistent adnexal (ovarian, para-ovarian, or tubal) mass for which they were operated on. All patients underwent an ultrasound examination by one of the principal investigators who were all experienced in gynecologic ultrasound. In the case of bilateral adnexal masses, the mass with the most complex ultrasound morphology was included in our statistical analysis. If both masses had similar ultrasound morphology, the largest one or the one most easily accessible by ultrasound was included. Patients who were pregnant, refused transvaginal ultrasonography, or did not undergo surgical removal of the mass within 120 days after the ultrasound examination were excluded from analysis.

Between November 2005 and October 2007, 1,970 patients were examined according to the IOTA phase 2 protocol. Thirty-two of these were excluded because of the following reasons: 15 because surgical removal of the mass occurred more than 120 days after the ultrasound examination, 12 because they were pregnant at the time of the examination, 4 because of errors in data entry, and one because of protocol violation. Of the remaining 1,938 patients, 941 were new patients collected in 7 centers that had also participated in the IOTA phase 1 study and so were not available for external validation. Nine hundred and ninety-seven patients were collected in 12 new centers that had not participated in any IOTA study before, and these patients were used for the external validation of the models.

Seven of the new centers were oncological referral centers (465 patients, 47.6% of all patients), one was a referral center for ultrasonography (135 patients, 13.5%), and 4 were regional hospitals (387 patients, 38.8%). Ethical approval was obtained by the local committees of all centers.

From a literature search, we identified 12 models to estimate the risk of malignancy in adnexal masses that contained variables for which information had been prospectively collected in the IOTA studies: the original RMI from the study of Jacobs and colleagues (7) and 3 variations of the RMI (8, 28, 29), 6 logistic regression models (9–14), and 2 ANNs (more specifically, multilayer perceptrons; ref. 12). The variables used in these models are listed in Table 1.

During IOTA phase 1, we developed one subgroup scoring system (30), 2 logistic regression models (LR1 and LR2; ref. 16), 3 least-square support vector machines (LS-SVM; ref. 23), 3 relevance vector machines (RVM; ref. 23), 2 Bayesian multilayer perceptrons (BMLP; ref. 24), and 1 Bayesian perceptron model (BPER; ref. 24). The performance of the subgroup scoring system on external validation will be analyzed and reported separately and is not included in the current study. The variables used in the IOTA models are listed in Table 1.

After completing the ultrasound examination, the ultrasound examiner classified each mass as benign or malignant according to his/her subjective assessment of the ultrasound images.

Because sensitivity and specificity are highly dependent on the cutoff point used, we regarded the area under the receiver operating characteristic (ROC) curve (AUC) as the most important measure to evaluate the discriminatory performance of the models. Confidence intervals (CI) for the AUC were computed using the logit-transform method (31, 32). CIs for the difference in AUCs were computed using the method of DeLong and colleagues (33). In addition, sensitivity and specificity as well as the positive and negative likelihood ratios (LR⁺ and LR⁻) were calculated for the cutoff points proposed in the original articles. These results were summarized into a single measure of test performance, the diagnostic odds ratio (DOR; refs. 34, 35). This equals LR⁺ divided by LR⁻ and shows how much higher the odds of classifying the tumor as malignant are for patients with a malignant tumor than for patients with a benign tumor. For example, a DOR of 10 means that the odds of classifying the tumor as malignant is 10 times higher for patients with a malignant tumor than for patients with a benign tumor.

In this study, we evaluated 23 models, such that we specified a priori which comparisons between models were of interest to avoid excessive output. We decided to focus on comparing LR1 with LR2 and on comparing LR1, LR2, and the best IOTA model (if different from LR1 or LR2) with RMI and the best non-IOTA model (if different from the RMI). The main comparison between models involved the AUC on all data, and this was used to define the best IOTA and non-IOTA models. Secondary comparisons involved AUCs by menopausal status and the AUC for discriminating stage I ovarian cancer from benign tumors.

In the RMI, one of the variables is “presence of metastases.” This was not a mandatory variable for the IOTA studies, and therefore information on metastases was sometimes missing (16%). To overcome this problem, we analyzed the differences in performance of the RMI when (i) the variable metastases were not used, (ii) the variable was imputed by 0 in case of missing information, and (iii) multiple imputation was used. The ROC curves were virtually identical for all 3 approaches. We decided to impute the variable by 0 in case of missing information because it seemed sensible to believe that in most cases where information on metastases was missing, there were no metastases.

Information on CA125 was missing in 27% of cases (272 of 997). Whenever it was missing, it was imputed using multiple imputation (36, 37) with predictive mean matching regression (38). This allowed us to use the whole database and to overcome selection bias. Missing values for CA125 were imputed 100 times, resulting in 100 complete data sets. Models using CA125 were applied on each complete data set, and results were combined using standard multiple imputation methodology to obtain final results that properly take into account the uncertainty of missing data (37). The 10 resulting ROC curves were averaged to plot a single curve. Additional details on our imputation procedure are described elsewhere (39).

The data set included 742 (74%) benign and 255 (26%) malignant masses. The rate of malignancy was 40% (192 of 475) for the oncology centers, 8% (11 of 135) for the ultrasound referral centers, and 13% (52 of 387) for the regional centers. The final histologic outcome for all patients is listed in Table 2.

The performance of the models for the whole data set is presented in Table 3. LR1, the main IOTA logistic regression model using 12 variables, had the best performance with an AUC of 0.96 (95% CI, 0.94–0.97). All IOTA models had roughly similar performance with AUCs ranging between 0.94 and 0.96 (LR2, 0.95). The best non-IOTA model was a logistic regression model (11) with an AUC of 0.93 (95% CI, 0.91–0.94). The original RMI outperformed its variations with an AUC of 0.91 (95% CI, 0.88–0.93). LR1 and LR2 did significantly better than the RMI and the best non-IOTA model (Table 4). Figure 1 shows the ROC curves for LR1, LR2, the original RMI, and the best non-IOTA model.

The IOTA models were also superior to the non-IOTA models for discriminating between benign masses and stage I invasive ovarian cancers (n = 48), see Tables 3 and 4. The IOTA models had AUCs ranging from 0.92 to 0.94, whereas the AUCs for the non-IOTA models were at most 0.91. LR1 had an AUC of 0.94 (95% CI, 0.88–0.97), LR2 of 0.94 (95% CI, 0.87–0.97), and RMI of 0.85 (95% CI, 0.78–0.90). Table 4 shows that the advantage of LR1 and LR2 over the RMI and the best non-IOTA model is maintained when focusing on discrimination between stage I primary invasive ovarian malignancies and benign tumors.

The diagnostic performance of the models when using the cutoff values suggested in the original articles is presented in Table 5. The sensitivities and specificities of the IOTA models were all around 90%, whereas among the non-IOTA models they differed substantially. The IOTA models had DORs between 46 and 77 (DOR for LR1, 75; sensitivity, 92%; specificity, 87%), whereas the DORs of the non-IOTA models varied between 12 and 37. LR2 had a DOR of 66 (sensitivity, 92%; specificity, 86%), the RMI of 37 (sensitivity, 67%; specificity, 95%), and the best non-IOTA model of 29 (sensitivity, 92%; specificity, 86%). The best trade-off between sensitivity and specificity was achieved by the IOTA models. The detection rate of primary invasive ovarian cancer stage I (n = 48) of the IOTA models varied between 88% and 92% (LR1 and LR2, 92%). Detection rates of stage I cancer were lower for all non-IOTA models except for ANN 2 (94%). However, the very low specificity of ANN2 (41%) indicates that this model classified many more tumors as malignant compared with any other model. The detection rate of primary invasive ovarian cancer stage I was 51% for the RMI and 63% for the best non-IOTA model (Table 4).

Table 6 shows the performance of LR1, the RMI, and the best non-IOTA model in pre- and postmenopausal patients. LR1 and LR2 had better values for all performance measures in premenopausal and postmenopausal patients, although the advantage of LR1/LR2 relative to the RMI and the best non-IOTA model was larger in premenopausal patients.

Table 7 shows the number and types of malignancies that were missed by the 2 IOTA logistic regression models, the RMI of Jacobs and colleagues and subjective assessment by an experienced ultrasound examiner. The RMI missed more malignant tumors of any kind than the IOTA models and subjective assessment.

This is the first study to externally validate and compare the diagnostic performance of IOTA models and non-IOTA models to discriminate between benign and malignant adnexal masses. We have shown that the IOTA models are robust and maintain their performance when tested in new centers in different countries with different population characteristics. Moreover, the IOTA models had highly similar performance and performed clearly better than the non-IOTA models, including the RMI. In particular, the detection rate of stage I primary invasive ovarian cancer was much higher for the IOTA models than for the very commonly used RMI. The differences in performance between the IOTA models and the non-IOTA models were smaller in postmenopausal than premenopausal patients (Table 6 and Supplementary Table). This is likely to be explained by CA125—included in the RMI and many non-IOTA models—performing better as a discriminator between benign and malignant adnexal lesions in postmenopausal women because of lower prevalence of benign lesions associated with increased CA125 levels in postmenopausal women (5, 40, 41). This is in agreement with IOTA models including CA125 not performing any better than models without CA125 (42).

A strength of our study is that it was conducted on a large number of masses. The geographic spread and the different referral profiles of the centers also suggest that the conclusions will have general applicability. Another strength is that we specifically determined the performance of the diagnostic methods to detect stage I ovarian cancer, even though there were only 48 such cancers. We think it is clinically important to achieve a high detection rate of stage I cancers, as explained below. It might be considered a weakness that in all centers, the scans were carried out by clinicians with a specific interest in gynecologic ultrasound. We need to evaluate how well the models perform in less experienced hands. However, obtaining information on the ultrasound variables required for the IOTA models should be possible for any qualified ultrasound practitioner.

Recently, Geomini and colleagues conducted a systematic review of all mathematical models that had been created for discrimination between benign and malignant adnexal masses and undergone external validation (20). They concluded that the RMI given by Jacobs and colleagues (7) is the method of choice for the preoperative assessment of an adnexal mass. However, the review of Geomini and colleagues did not include an evaluation of the IOTA models because at the time of writing that review, the results of the external validation of the IOTA models had not been published. When a cutoff level of 200 for the RMI was used, the pooled estimate for sensitivity was 78% (95% CI, 71%–85%) and that of specificity was 87% (95% CI, 83%–91%). In our prospective validation, the sensitivity of RMI was lower (68%; 95% CI, 62%–73%) but the specificity was higher (95%; 95% CI, 93%–96%). Differences in tumor populations, for example, in stage distribution, might explain this discrepancy.

A clear advantage of the IOTA models over the non-IOTA models is that they seem to have higher detection rate of stage I ovarian cancers than the other models and the RMI. Because stage I ovarian cancer is associated with a survival rate of more than 90% (43), early detection and optimal treatment might be crucial. A false-negative diagnosis of ovarian cancer will often lead to the patient undergoing laparoscopic surgery. This increases the risk of spilling cyst fluid containing malignant cells into the peritoneal cavity (44). Previous studies clearly showed that rupture of a malignant ovarian cyst worsens the prognosis for disease-free survival in stage I invasive ovarian cancers (2, 3). Spilling should also probably be avoided in borderline tumors, even though there are no prospective studies to prove this (44). Spilling and iatrogenic upstaging of the patient is not a serious problem in advanced cancer because spread throughout the peritoneal cavity is already present. Furthermore, higher stage ovarian cancers are often diagnosed clinically, making mathematical models to calculate the risk of malignancy less important. When we examined the types of malignancies that were missed, LR1 and LR2 missed the smallest number of stage I invasive cancers and borderline tumors. These models missed fewer malignancies than subjective assessment, but the latter had higher specificity (22). The RMI missed the highest number of stage I ovarian cancers and missed 73% of borderline tumors.

In conclusion, our results contradict the conclusion of Geomini and colleagues that the RMI is the method of choice to distinguish benign from malignant adnexal masses. External validation shows that the IOTA models outperform other models—including the current reference test RMI—for discriminating between benign and malignant adnexal masses.

No potential conflicts of interest were disclosed.

The authors thank all participating centers, the principal investigators, and the study participants for their contribution. Data from the IOTA 2 study were presented at the 18th World Congress of Ultrasound in Obstetrics and Gynecology, organized by the International Society of Ultrasound in Obstetrics and Gynecology (ISUOG), Chicago, 2008.

Recruitment Centers: The recruitment centers include University Hospitals Leuven (Belgium); Ospedale S. Gerardo, Università di Milano Bicocca, Monza (Italy); Ziekenhuis Oost-Limburg (ZOL), Genk (Belgium); Medical University in Lublin (Poland); University of Cagliari, Ospedale San Giovanni di Dio, Cagliari (Italy); Malmö University Hospital, Lund University (Sweden), University of Bologna (Italy); Università Cattolica del Sacro Cuore Rome (Italy); DCS Sacco University of Milan (Milan A, Italy); General Faculty Hospital of Charles University, Prague (Czech Republic); Chinese PLA General Hospital, Beijing (PR China); King's College Hospital London (UK); Universita degli Studi di Napoli, Napoli (Naples A, Italy); IEO, Milano (Milan B, Italy); Lund University Hospital, Lund (Sweden); Macedonio Melloni Hospital, University of Milan (Milan C, Italy); Università degli Studi di Udine (Italy); McMaster University, St. Joseph's Hospital, Hamilton, Ontario (Canada); and Instituto Nationale dei Tumori, Fondazione Pascale, Napoli (Naples B, Italy).

IOTA Steering Committee: The members of IOTA steering committee are D. Timmerman, Leuven, Belgium; L. Valentin, Malmö, Sweden; T. Bourne, London, UK; A.C. Testa, Rome, Italy; S. Van Huffel, Leuven, Belgium; Ignace Vergote, Leuven, Belgium; and B. Van Calster, Leuven, Belgium.

IOTA Principal Investigators: IOTA principal investigators (in alphabetical order) are A. Czekierdowski, Lublin, Poland; Elisabeth Epstein, Lund, Sweden; Daniela Fischerová, Prague, Czech Republic; Dorella Franchi, Milano, Italy; Robert Fruscio, Monza, Italy; Stefano Greggi, Napoli, Italy; S. Guerriero, Cagliari, Italy; Jingzhang, Beijing, PR China; Davor Jurkovic, London, UK; Francesco P.G. Leone, Milano, Italy; A.A. Lissoni, Monza, Italy; Henry Muggah, Hamilton, Ontario, Canada; Dario Paladini, Napoli, Italy; Alberto Rossi, Udine, Italy; L. Savelli, Bologna, Italy; A.C. Testa, Roma, Italy; D. Timmerman, Leuven, Belgium; Diego Trio, Milano, Italy; L. Valentin, Malmö, Sweden; and C. Van Holsbeke, Genk, Belgium.

B. Van Calster is a postdoctoral researcher funded by the Research Foundation–Flanders (FWO), Belgium. Research supported by Research Council KUL: GOA MaNet, CoE EF/05/006 Optimization in Engineering (OPTEC); Research Foundation–Flanders (FWO): projects G.0302.07 (SVM), G.0341.07 (Data fusion); IWT: TBM070706-IOTA3; Belgian Federal Science Policy Office: IUAP P6/04 (DYSCO, ‘Dynamical systems, control and optimization', 2007–2011); IBBT (Flemish Government); Swedish Medical Research Council: grants nos. K2001-72X 11605-06A, K2002-72X-11605-07B, K2004-73X-11605-09A and K2006-73X-11605-11-3; funds administered by Malmö University Hospital; and two Swedish governmental grants (ALF-medel and Landstingsfinansierad Regional Forskning).

The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.