Ovarian Cancer Early Detection Claims Are Biased

To the Editors: Visintin et al. (1) claim the ability to detect ovarian cancer early and with 95.3% sensitivity and 99.4% specificity. Several serious methodologic issues lead us to conclude that these figures are greatly exaggerated.

In their article, six proteins are used to compare the plasma from ovarian cancer cases and healthy controls. Specimens were divided nonrandomly into two distinct sets. The training set specimens, derived from one cohort, were used to fit several classifiers, and the test set specimens, derived from a different cohort, were used to estimate their performance. The accuracy reported in their conclusion, however, was determined from the combined data and from the classifier that did best in the test set. This violates fundamental principles of statistical analysis, which requires that only the training set be used to select a classifier and only the test set be used to evaluate the classifier. Had they properly used this generally accepted criterion, they would have had to report a lower sensitivity of either 84% or 88% at lower 95% specificity. The analysis they chose to highlight is inappropriate and misleading.

Even this reduced performance to detect cancer may be inflated by confounding factors. Evidence of this may be seen in their own data (see Fig. 1 in ref. 1), which shows that in cases, but not in controls, three markers (Prolactin, MIF II, and Leptin) vary substantially between the training and test cohorts. This bias could be because their cases and controls were derived from two critically different sources. All case samples were collected from women reporting to a gynecologic oncology clinic with a suspicious pelvic mass, whereas the controls were attending a regular gynecologic exam and without prior suspicion of cancer. There are innumerable potential differences between these groups, many known to affect blood chemistry but that are not disease related, including but not limited to psychological stress, time of blood draw, time since last meal, or uncontrolled differences in specimen handling. At least one of their markers (2) has been shown to be highly sensitive to confounding in this way. Thus, one or both of their sets might contain biases that further inflate the performance of their panel.

Finally, distinguishing women with symptomatic ovarian cancer, even early stage symptomatic cancer, from healthy women is far from sufficient to suggest utility for early detection. The latter requires detecting cancer before symptoms develop. The interpretation offered by the investigators is premature and unduly optimistic.