Performance of Prediction Models for BRCA Mutation Carriage in Three Racial/Ethnic Groups: Findings from the Northern California Breast Cancer Family Registry

Purpose: Patients with early-onset breast and/or ovarian cancer frequently wish to know if they inherited a mutation in one of the cancer susceptibility genes, BRCA1 or BRCA2. Accurate carrier prediction models are needed to target costly testing. Two widely used models, BRCAPRO and BOADICEA, were developed using data from non-Hispanic Whites (NHW), but their accuracies have not been evaluated in other racial/ethnic populations.

Methods: We evaluated the BRCAPRO and BOADICEA models in a population-based series of African American, Hispanic, and NHW breast cancer patients tested for BRCA1 and BRCA2 mutations. We assessed model calibration by evaluating observed versus predicted mutations and attribute diagrams, and model discrimination using areas under the receiver operating characteristic curves.

Results: Both models were well-calibrated within each racial/ethnic group, with some exceptions. BOADICEA overpredicted mutations in African Americans and older NHWs, and BRCAPRO underpredicted in Hispanics. In all racial/ethnic groups, the models overpredicted in cases whose personal and family histories indicated >80% probability of carriage. The two models showed similar discrimination in each racial/ethnic group, discriminating least well in Hispanics. For example, BRCAPRO's areas under the receiver operating characteristic curves were 83% (95% confidence interval, 63-93%) for NHWs, compared with 74% (59-85%) for African Americans and 58% (45-70%) for Hispanics.

Conclusions: The poor performance of the model for Hispanics may be due to model misspecification in this racial/ethnic group. However, it may also reflect racial/ethnic differences in the distributions of personal and family histories among breast cancer cases in the Northern California population. (Cancer Epidemiol Biomarkers Prev 2009;18(4):1084–91)

Since the identification of the BRCA1 and BRCA2 cancer susceptibility genes, predictive models have been developed to identify individuals likely to carry inherited deleterious BRCA mutations. These models assign a patient a probability of mutation carriage using the cancer histories of the patient and her first-degree, second-degree, and in some cases, more distant relatives, and estimates of BRCA mutation prevalence and penetrance. The BRCAPRO model assumes that all genetic susceptibility to breast cancer is due to BRCA mutations (1-3). The BOADICEA model considers the simultaneous effects of BRCA1, BRCA2, and the residual familial clustering of breast cancer not explained by these genes, which is assumed to be polygenic (4, 5). Multiple studies have validated the performance of BRCAPRO and most have found it to discriminate as well as or more accurately than other models (2, 3, 6-18), although some studies suggest that BOADICEA performs slightly better in certain high-risk populations (1, 6, 19). Most evaluations of BRCAPRO and BOADICEA have included primarily non-Hispanic White (NHW) breast cancer patients, with a few exceptions (13, 17, 18, 20-22); consequently, little is known about the performance of these models across different racial/ethnic groups.

The BRCAPRO model was developed and has been validated with data from patients presenting for clinical genetics evaluation because of strong family cancer history; the BOADICEA model was developed using population-based data. As we and others have observed (23), a substantial portion of population-sampled young breast cancer patients with BRCA mutations do not have a family history of breast or ovarian cancer. Many women with early-onset breast cancer, but no family history, wish to know their mutation status because results may guide subsequent management of breast and ovarian cancer risk, and family planning; such patients increasingly present for evaluation at cancer genetics clinics. An accurate and precise risk prediction algorithm based on personal and family history could identify individuals who lack a known deleterious mutation but who, if tested, would have a 6% to 13% chance of receiving ambiguous and anxiety-producing results (24). We report an analysis of the performance of the BRCAPRO and BOADICEA mutation carriage prediction models in African American, Hispanic, and NHW female breast cancer patients from a population-based study in the San Francisco Bay Area.

To increase the precision of our inferences while maintaining their population-based properties, we identified breast cancer patients diagnosed at age <65 y through the population-based Greater San Francisco Bay Area Cancer Registry, which ascertains all incident cancers as part of the Surveillance, Epidemiology and End Results Program and the California Cancer Registry, and invited them to participate in the Northern California Breast Cancer Family Registry (25-27). We recruited patients using a two-stage sampling design, with oversampling of patients having characteristics that suggest an inherited basis for their cancers (25-27). In stage one of sampling, we administered a brief telephone interview to all patients and assessed self-identified race/ethnicity and family history of breast and ovarian cancer. Based on age at diagnosis and personal and family history, patients were classified into either category A (patients whose cancers are likely to be hereditary) or category B (all other patients with cancers less likely to be hereditary). Category A patients were those who met at least one of the following criteria: (a) breast cancer diagnosis before age 35 y; (b) bilateral breast cancer, with first diagnosis before age 50 y; (c) prior ovarian or childhood cancer; or (d) at least one first-degree relative with breast or ovarian cancer. Categories A and B were designed with the aim of reducing the variance of overall prevalence estimates in the entire population-based series of patients. In stage two, we invited all patients in category A and a random sample of patients in category B to enroll in the family registry. Participants completed questionnaires on family history of cancer and breast cancer risk factors and provided a biospecimen sample. This two-stage sampling design provides unbiased estimates of population-based mutation carrier prediction performance having greater precision than those obtained from a simple random sample of the same size.

In telephone interviews, participants provided information on date of birth, vital status, date of death or last observation, and diagnosis dates and types of all site-specific cancers for themselves and all first-degree relatives. The questionnaire also elicited the occurrence and age at diagnosis of breast or ovarian cancer in second-degree relatives. When possible, reports of breast or ovarian cancer in relatives were verified by interviewing the relatives themselves, by obtaining medical records, or both. Informed consent was obtained from each study subject; the institutional review boards of the Northern California Cancer Center, Stanford University, and the Dana-Farber Cancer Institute approved the study, in accordance with an assurance filed with and approved by the Department of Health and Human Services. The present analysis was restricted to patients ages <65 y diagnosed with invasive breast cancer between January 1, 1995 and April 30, 2003, who identified themselves as African American, Hispanic, or NHW. The number of BRCA mutation carriers among Asian-American patients was too small for reliable analysis. We restricted NHW patients to those without self-identified Ashkenazi Jewish ancestry given the known elevated prevalence of BRCA mutations among Ashkenazim.

Biospecimens from the Northern California Breast Cancer Family Registry were processed by the Coriell Cell Repositories (Coriell Institute for Medical Research, Camden, NJ). Testing was performed using Exon Grouping Analysis, full sequencing performed by Myriad Genetic or two-dimensional gene scanning. The numbers of patients tested by these three methods were 28, 691, and 646, respectively, for BRCA1 and 363, 1,002 and 0, respectively, for BRCA2. All coding exons and surrounding intronic sequences were amplified with 34 primer pairs and analyzed on ABI-377 instruments. PCR fragments with aberrant mobility were sequenced. For two-dimensional gene scanning, the entire coding exon and surrounding intronic sequences were amplified in a two-step PCR process involving six individual multiplex reactions (31, 32); these methods permit the detection of variants in coding regions and splice site mutations. Mutations were classified according to the Breast Cancer Information Core database⁵

and considered pathogenic as described by Couch et al. (33). Regulatory mutations outside of the coding region and splice junctions and large genomic rearrangements were not detected by the methods used here.

We calculated BRCAPRO and BOADICEA probabilities of mutation carriage (hereafter called prediction scores) in patients tested for BRCA1 and BRCA2 mutations. BRCAPRO probabilities were obtained using the CancerGene 4b program (University of Texas Southwestern, Dallas, TX), and BOADICEA probabilities were obtained using the program BOADICEA V3 provided by Antonis Antoniou University of Cambridge, Cambridge, United Kingdom (19). All available demographic data on all patients' first-degree and second-degree relatives ages ≥20 y, whether affected by cancer or not, were included in the analyses. Information on all cases of breast cancer, ovarian cancer, male breast cancer, pancreatic cancer, and prostate cancer among patients and their first-degree and second-degree relatives was included in the prediction models.

Our goal was to obtain precise, population-based evaluation of the calibration and discrimination of each of the two models applied to each of the three racial/ethnic groups. Here, calibration refers to the agreement between mean prediction scores and observed carrier prevalences within subgroups of a population. Discrimination refers to the extent of separation between the prediction scores of carriers and noncarriers. We evaluated model calibration by comparing the observed prevalences with the mean prediction scores visually using attribute diagrams (34). We evaluated model discrimination using the areas under the receiver operating characteristic curves (AUC). Horvitz-Thompson estimating equations (26, 27) were used to adjust all analyses for the two-stage sampling design of the study.

Calibration. We estimated the prevalence of BRCA mutation carriers in a given racial/ethnic group as a weighted average of the two category-specific prevalences π_A and π_B. Here, π_A is the number of carriers identified in category A divided by the total number of tested patients in category A. The overall prevalence estimate (categories A and B combined) was π = wπ_A + (1 − w) π_B, where the weight w is the proportion of all screened patients who were classified in category A. We compared the prevalence π to similarly weighted averages of the category-specific mean prediction scores. In these calculations, we multiplied each prediction score by 90%, assuming that laboratory testing methods were 90% sensitive (35). The overall predicted prevalence was then computed as a weighted average s̄ = ws̄_A + (1 − w) s̄_B of the two category-specific mean scores s̄_A and s̄_B. We evaluated the statistical significance of the observed/predicted differences via the statistic S = (π − s̄)²/V, where V is the Horvitz-Thompson variance estimate (26, 27). This statistic has an approximately χ² distribution on 1 df, under the null hypothesis that the prediction score for each breast cancer patient in a given racial/ethnic population equals her actual probability of mutation carriage.

We also used attribute diagrams to compare observed carrier prevalence to mean prediction scores within subgroups of patients whose scores lie in subintervals of the unit interval. To obtain an even distribution of patients within the subintervals, we grouped the patients according to quantiles of the logits of their scores. Within each subinterval, we computed weighted estimates and Horvitz-Thompson–based confidence intervals (CI) for the proportion of patients with observed mutations. We then plotted these points in attribute diagrams, which are plots of BRCA mutation carrier prevalences against median scores within each score subinterval (1). Results are provided for subgroups according to age and family cancer history, and for the group as a whole.

Discrimination. To evaluate the ability of the models to discriminate between carriers and noncarriers in each racial/ethnic group, we constructed receiver operating characteristic curves and evaluated the AUC (36). We estimated these AUCs as weighted sums of the category-specific AUCs, using weights as described above. We used bootstrap variance estimates to obtain CIs for the AUCs. Results are provided for subgroups according to age and family cancer history, and for the group as a whole.

Figure 1 shows the distribution of African American, Hispanic, and NHW patients that were screened according to sampling categories. They include 1,050 (21%) African Americans, 1,487 (30%) Hispanics, and 2,478 (49%) NHWs; 1,466 (29%) were classified into category A, and the remaining 3,549 (71%) into category B. All category A patients (n = 1,466) were invited to participate, and of these, 1,002 (68%) provided a blood or buccal sample and were tested for BRCA1 and BRCA2 mutations. Of the 3,549 category B patients, 671 were randomly selected for participation, and of these, 435 (65%) provided a biospecimen sample and were tested. Of the 1,437 patients tested for BRCA1 and BRCA2 mutations, 72 were excluded from analysis: 23 because a family member had already been enrolled in the study through ascertainment of a first-degree relative with breast cancer (7 African Americans, 14 Hispanics, and 2 NHWs), 1 NHW because no family history information was available, and 48 NHWs because of Ashkenazi Jewish ancestry. A total of 1,365 tested patients were included in the analysis.

Table 1 shows the distribution of tested patients according to BRCA mutation status, by sampling category (A and B) and race/ethnicity (African American, Hispanic, and non–Ashkenazi Jewish NHWs). Of 1,365 patients tested for BRCA mutations, 44 tested positive for BRCA1 mutations (40 in category A and 4 in category B), and 41 tested positive for BRCA2 mutations (32 in category A and 9 in category B). Estimates of BRCA1 and BRCA2 mutation prevalence were highest among Hispanic patients (prevalence, 3.2% for each gene) and lowest among African American patients (1.1% and 1.8% for BRCA1 and BRCA2, respectively), as we previously reported (23).

Table 2 shows prediction scores and observed prevalences of BRCA1 and BRCA2 mutations, specific for age and racial/ethnic group. Comparison of prediction scores to observed prevalence generally indicates similar levels of calibration for the two models, with notable differences as follows. The BRCAPRO model underpredicted mutation carriage in Hispanics, with the observed prevalence estimate of 6.4% exceeding the prediction of 3.8% (P = 0.04); this underprediction was statistically significant in the subset of Hispanic patients with no family history of breast cancer (P = 0.01), but not in those with a family history of breast cancer (P = 0.14). BRCAPRO overpredicted in the subset of patients with a family history among African Americans (P < 0.01) and NHWs (P < 0.01), but did not overpredict significantly in African Americans and NHWs without a family history of breast cancer. The BOADICEA model overpredicted in African Americans (3.0% observed versus 5.0% predicted; P = 0.02) and older non–Ashkenazi Jewish NHWs (2.8% observed versus 4.0% predicted; P = 0.03). The overprediction of BOADICEA was statistically significant in the subset of patients with a family history of breast cancer among African Americans (P < 0.01) and NHWs (P < 0.001); it underpredicted significantly among Hispanics without a family history (P = 0.04), but not among those with a family history of breast cancer.

Attribute diagrams for each racial/ethnic group are presented in Fig. 2 as a measure of model resolution and reliability, with optimal performance represented by data points on the 45-degree line (1). For both models, data points are clustered on the 45-degree line, with the exception of the BRCAPRO model for Hispanics, in which most points were above the 45-degree line, consistent with the observed underprediction of BRCA mutation carriage as reported in Table 2. In all racial/ethnic groups, BRCAPRO shows overprediction in patients with 80% or greater predicted probability of mutation carriage; BOADICEA shows overprediction for non–Ashkenazi Jewish NHWs with 80% or greater predicted probability of mutation carriage (Fig. 2).

Accuracy in discriminating between carriers and noncarriers, as measured by AUC, was similar for both models within each racial/ethnic group (Table 3). The highest AUC values were observed in non–Ashkenazi Jewish NHWs (BRCAPRO, 83%; 95% CI, 63-93%; BOADICEA, 83%; CI, 63-93%), followed by African Americans (BRCAPRO, 74%; CI, 59-85%; BOADICEA, 75%; CI, 60-85%) and Hispanics (BRCAPRO, 58%; CI, 45-70%; BOADICEA, 56%; CI, 43-68%). Within subsets defined by age and family history of breast cancer, there was a trend toward worse performance of both models in older (BRCAPRO, 49%; CI, 22-76%; BOADICEA, 44%; CI, 17-74%) compared with younger (BRCAPRO, 81%; CI, 67-90%; BOADICEA, 85%; CI 74-92%) African Americans. There was also a trend toward worse discrimination by both models in Hispanics without a family history (BRCAPRO, 52%; CI, 38-65%; BOADICEA, 50%; CI 34-66%), compared with Hispanics with a family history (BRCAPRO, 69%; CI, 56-79%; BOADICEA, 69%; CI, 55-80%) of breast cancer.

We evaluated the performance of the BRCAPRO and BOADICEA BRCA mutation carrier prediction models in three racial/ethnic groups, consisting of African American, Hispanic, and non–Ashkenazi Jewish NHW breast cancer patients from the San Francisco Bay Area. To our knowledge, this is the first study to compare these BRCA mutation prediction models across population samples of such racial/ethnic diversity. In general, the models showed similar discrimination within each racial/ethnic group, but differences in calibration: BRCAPRO underpredicted mutation carriage in Hispanics, whereas BOADICEA overpredicted mutations in African Americans and in older NHWs.

The strength of this study is its focus on population-based samples of racial/ethnic minorities, in contrast to most prior evaluations of BRCA mutation prediction models. Some prior studies have found similar accuracies of BRCAPRO in racial/ethnic minorities as in NHWs (13, 17), but we and others reported underprediction by BRCAPRO and other models among clinic-based minorities including Asian Americans and Hispanics (18, 20). Models may perform less well in racial/ethnic minorities because the prevalence of carriers among breast cancer cases may differ by race/ethnicity. In non–Ashkenazi Jewish NHW cases, our prevalence estimates for BRCA1 (2.1%) and BRCA2 (2.3%) are similar to those used by the BRCAPRO and BOADICEA models (3, 4); by contrast, we found that African American cases had lower carrier prevalence (BRCA1, 1.1%; BRCA2, 1.8%), whereas Hispanic cases had higher carrier prevalence (BRCA1, 3.2%; BRCA2, 3.2%). Notably, the exception to the general overprediction of BOADICEA occurred in Hispanics, as did a significant underprediction by BRCAPRO, both consistent with Hispanics' higher mutation prevalence than that observed in non–Ashkenazi Jewish NHWs. Within subsets specific for family history and age, calibration worsened with increasing divergence from the mutation prevalence expected from the models; for example, both models overpredicted significantly only in African Americans with a family history of breast cancer, whereas underpredicting in Hispanics lacking such family history. Recent publications have reported a higher prevalence of the 185delAG BRCA1 founder mutation in Hispanics than was initially appreciated (18, 37, 38), leading some to suggest a common origin for this mutation in Sephardic and Ashkenazi Jewish populations (18, 37, 38). The present results contrast with those from a recent single-center clinic-based study of BRCAPRO in Hispanics, which reported better model performance than we found (17); variations in the use of BRCAPRO between studies may explain some of this difference. Our finding of lesser BRCAPRO model accuracy in Hispanics also prompts questions as to whether BRCA mutation penetrance, or associated cancer risk, might differ between Hispanics and NHWs. Given the growing size of the Hispanic population in the United States, further study of this issue has important implications for health policy and resource allocation.

In contrast to prior studies of the calibration of BRCAPRO and similar models in clinic-based settings (1, 2, 6-22), this analysis considered populations having lower BRCA mutation prevalence, with 85 (6%) carriers identified among 1,365 patients tested. This study sample reflects the reality of current clinical BRCA mutation testing, given patient preference and practice guidelines that support more inclusive testing than previously advised.⁶

Our finding that BRCAPRO and BOADICEA overpredicted in a substantial proportion of patients, particularly in patients with a family history of breast cancer or with >80% predicted probability of mutation carriage, likely results from lower mutation frequency, and perhaps higher sporadic breast cancer incidence (39, 40) in these groups than the model parameters assume; we anticipate that population-specific corrections may improve model calibration, as others have shown (7).

Prior studies of the discrimination of the BRCAPRO model have reported AUCs in the range of 60% to 88%. Comparisons of BRCAPRO to other BRCA mutation prediction models, including BOADICEA, Couch, Finnish, National Cancer Institute, Frank/Myriad II, the Manchester Scoring System, the Family History Assessment Tool, and Shattuck-Eidens/Myriad I, have revealed relatively few differences in terms of discrimination (1, 2, 6-16, 18, 19, 41, 42). Exceptions include the slightly superior performance of BOADICEA and the Manchester Scoring System in the United Kingdom, of the Italian IC software modification of BRCAPRO among Italians, and of the LAMBDA model among Ashkenazi Jewish probands; some of these models use population-specific mutation prevalence estimates, which tailored them to the groups under study (7, 9, 19, 43). In the present study, we evaluated model discrimination in three separate racial/ethnic populations, which may differ in their prevalence of BRCA mutations, and in the variance of their carriage probabilities. Variation in the probability of mutation carriage within a racial/ethnic population affects the AUC of a model. For example, if all breast cancer patients in a racial/ethnic group had the same mutation carrier probability, its AUC (which is the likelihood that the carriage probability for a randomly selected carrier exceeds that of a randomly selected noncarrier) would equal its minimum of 50%, indicating that the model is no better at discriminating between carriers and noncarriers than random chance. Given such intragroup variability, it is difficult to compare the discrimination of a model across different racial/ethnic groups. Comparing the discriminative abilities of BRCAPRO and BOADICEA within a single population is more straightforward, and we found no difference between models in any of the three racial/ethnic groups under study. Within subsets defined by age and family history of breast cancer, we observed some trends in model performance that did not reach statistical significance (for example, both models discriminated better in younger, compared with older, African Americans). Future research should evaluate race/ethnicity-specific modifications to mutation prevalence assumptions of BRCAPRO and BOADICEA, and compare the discrimination of each model to that of other prediction tools, within the racial/ethnic groups we studied. As understanding of the spectrum of BRCA mutations across race/ethnicity matures, it may prove optimal to develop models specific to each racial/ethnic population.

Although BRCA mutation testing was completed for only 67% of those invited to enroll in the Northern California Breast Cancer Family Registry, the testing rate was similar for patients in category A (68%) and category B (65%). This similarity suggests that family history was not related to patients' willingness to participate in the registry and provide biospecimens for research. We assumed that the combination of BRCA testing methods used was 90% sensitive for the detection of deleterious mutations (35); however, if testing sensitivity was actually lower than 90%, then the models may overpredict less, and underpredict more, than we report here.

In conclusion, the BRCAPRO and BOADICEA models showed differences in performance across racial/ethnic and age groups in a large population-based series of breast cancer patients. This finding emphasizes the need for further study of BRCA mutations in specific racial/ethnic and age groups, and for the development of more accurate mutation prediction methods, with customization for the populations to which they are applied.

No potential conflicts of interest were disclosed.

The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.

We thank Antonis Antoniou for providing BOADICEA software, and Frederick Li for his contributions to the laboratory work for this analysis.