Risk Analysis of Prostate Cancer in PRACTICAL Consortium—Response

We thank Martens and colleagues (1) for their comments on our article. We agree that calibration of the model in the extremes of the risk distribution is important. We observed (2), however, that the estimated ORs for men in the top and bottom 1% of the risk distribution (4.2 and 0.14, respectively) did not in fact differ from those predicted under a model with a continuous (log-additive) effect of the polygenic risk score (PRS) on risk (3.8 and 0.20, respectively). In addition, formal calibration tests showed no evidence of departures from the log-additive model (P = 0.12 by the Hosmer and Lemeshow test).

In our data, the AUC was 0.63 (95% CI, 0.62–0.64) for a model incorporating age and family history, and this increased to 0.69 (95% CI, 0.68–0.70) when the PRS was incorporated. The AUC cannot, however, be compared directly with other models such as the Prostate Cancer Prevention Trial (PCPT) model, which is based on measurement PSA and digital rectal examination and predicts the risk of prostate cancer detection on biopsy. Our model is based solely on genetic data and predicts the subsequent risk of prostate cancer. Clearly, if the PSA level were also incorporated into the model, the AUC would likely to be higher. We also note that the AUC is likely to increase as additional SNPs are identified. For example, a recent study using data on 65 SNPs estimated an AUC of 0.68 (3). Finally, it is worth noting that the AUC is not necessarily a good measure of the predictive value of a model, and that other measures such as the net reclassification index may be more useful.

We agree that the discrimination of risk prediction models may be overestimated in the dataset in which the model was developed, due to overfitting. To address this, we re-estimated the model parameters in a random sample of the dataset that included 90% of the cases and controls. The PRS from this model was then tested on the remaining 10% of the dataset. The OR per 1 SD of the PRS was 1.74 (95% CI, 1.60–1.79) in the training set and 1.60 (1.49–1.73) in the validation set. Thus, the PRS remains highly predictive of risk in the validation dataset, but there is some suggestion of overfitting. Further validation of PRSs based on all available SNPs, in independent datasets, will be important before such scores can be used routinely.

See the original Letter to the Editor, p. 222