Evaluation of Algorithms Using Automated Health Plan Data to Identify Breast Cancer Recurrences

This article is featured in Selected Articles from This Issue, p. 345

Breast cancer recurrence is an important outcome in cancer treatment and survivorship research. Most population-based tumor registries do not capture recurrences. Recurrences are not documented in a systematic and clearly structured manner in electronic health records (EHR); thus, accurate identification usually requires resource-intensive chart abstraction (1). Several studies have developed and validated algorithms to identify breast cancer recurrences from EHR data, potentially reducing costs and time required for data collection (2–10). One meta-analysis reported a median sensitivity of 87% (range: 44%–97%) and median positive predictive value (PPV) of 73% (range: 14%–94%) across 14 studies with algorithms for breast cancer recurrence (11).

When using previously developed recurrence algorithms, researchers must select an algorithm, adapt it to their data, and address potential false positives (FP) and false negatives (FN). In addition, algorithms may need to be updated (e.g., to reflect new billing and coding practices). Validating an existing algorithm against chart abstraction requires decisions about which patients to include and extent of manual abstraction, particularly in a setting with constrained resources. Only a few studies have validated or described how they adapted a previously developed algorithm to a new population (6, 8). As use of recurrence algorithms becomes more common in epidemiologic research, following standard practices to evaluate recurrence algorithms will help increase comparability and accuracy of data across studies.

We describe our evaluation of previously developed algorithms for breast cancer recurrence (3, 6) in a population of people diagnosed with incident stage I–IIIA breast cancer. The size of our population prohibited manually collecting gold-standard chart abstraction recurrence data on all people. Thus, we sought to identify an algorithm-based approach that would yield high sensitivity and specificity for recurrence in the absence of chart abstraction. In this study, we describe algorithm selection, adaptation, and analytic considerations to provide scientific direction for future studies.

We used data from the Optimal Breast cancer Chemotherapy Dosing (OBCD) study. The OBCD study included adults ≥18 years of age identified as female diagnosed with primary incident stage I–IIIA breast cancer and enrolled at Kaiser Permanente Washington (KPWA) or Kaiser Permanente Northern California (KPNC). For this validation study, we only included participants from KPWA that had definitive breast surgery for their primary breast cancer diagnosis. We obtained Institutional Review Board approval from KPWA, KPNC, and collaborating sites [Memorial Sloan Kettering (New York, NY) and Rutgers University (New Brunswick, NJ)] with a waiver of consent to collect and analyze data in accordance with the U.S. Federal Policy for the Protection of Human Subjects (Common Rule).

We identified algorithms that had been developed and validated using data from KP or similar health systems to increase implementation feasibility and potential accuracy in the same settings (3, 6, 8). We selected a set of recurrence algorithms previously developed and published by Chubak and colleagues (3) among people with stage I–II breast cancer, and later extended by Kroenke and colleagues (6), who evaluated the performance of algorithm pairs with select medical record review to resolve discordant results (triangulation), and included patients with stage I–IIIA breast cancer. We chose one algorithm with high sensitivity (“Algorithm 7” in Kroenke and colleagues) and one algorithm with high specificity and PPV (“Algorithm 9” in Kroenke and colleagues) to identify recurrences (6).

We used the selected algorithms to identify breast cancer recurrences that occurred before each participant's death, health plan disenrollment, or end of follow-up (December 31, 2019)—whichever came first. As described previously, the algorithms used procedure codes [Current Procedural Terminology (CPT) and Healthcare Common Procedure Coding System (HCPCS) codes] and National Drug Codes (NDC; ref. 3). Specifically, Algorithm 7 first identifies recurrence based on codes for a secondary non-breast malignant neoplasm >180 days after primary breast cancer, then mastectomy >180 days after primary breast cancer, and then radiotherapy >365 days after primary breast cancer. Algorithm 9 first identifies recurrence based on two visits with a code for a secondary malignant neoplasm within 60 days of each other and >365 days after primary breast cancer, and then a non-breast cancer record in the Surveillance Epidemiology and End Results (SEER) registry after the primary breast cancer. Because our study included cases diagnosed after the implementation of ICD-10 codes in 2015, we adapted the Chubak and Kroenke algorithms to include ICD-10 diagnosis codes that mapped to the ICD-9 codes used in the original algorithms (available upon request). In addition, we extended the algorithm code to identify recurrence dates. When a recurrence was identified, the recurrence date was defined as the first date associated with an algorithm-identified recurrence (e.g., an ICD code), following previous methods (12). Different algorithms used different indicators; therefore, the first associated date varied by algorithm.

We used three methods to combine algorithms based on how we treated algorithm-discordant pairs between Algorithm 7 and Algorithm 9. First, we categorized all discordant pairs as no recurrence in the “High Specificity and PPV Combination.” Second, we categorized all discordant pairs as recurrence in the “High Sensitivity Combination.” Third, we used triangulation to categorize discordant pairs as recurrence if a confirmed recurrence was identified in chart abstraction and no recurrence if a recurrence was not identified in chart abstraction. Triangulation aligns with Kroenke's prior work and may be option for studies that have additional chart abstraction resources (6).

Abstraction was undertaken among a sample of the 2,859 people with incident breast cancer diagnosed at KPWA between 2004 and 2015 with EHRs available. We abstracted the following: all discordant records in which one algorithm indicated a recurrence while the other did not (N = 404); a random sample of concordant records where both algorithms indicated no recurrence (N = 100); a random sample of concordant records where both algorithms indicated recurrence (N = 100); and all additional concordant records where both algorithms indicated a recurrence, but corresponding recurrence dates differed by |$ \ge $|6 months (N = 24). We included concordant records because we did not want to assume that concordance between the two algorithms represented a true positive (TP) or true negative (TN). None of the people included in the original validation studies by Chubak and Kroenke were included in this validation sample. Out of 628 records abstracted, we excluded 28 from the final analysis. Four were ineligible for this study (upon further review they were not stage I–IIIA breast cancer or had incomplete EHRs), six had recurrence dates beyond the end of follow-up (December 31, 2019), and 18 did not have definitive surgery for the primary breast cancer diagnosis. We included 600 remaining records in this validation analysis, which were weighted back to the overall population using inverse sampling weights based on the probability of being selected for medical record abstraction after accounting for exclusions. The inverse sampling weights were based entirely on the algorithm results (e.g., all discordant or random samples of concordant records) and were not based on any population characteristics.

We abstracted information on breast cancer recurrences from pathology and imaging reports. We provided chart abstractors with information on the primary breast cancer diagnosis (including date) but blinded them to recurrence algorithm results. We defined recurrence as a tumor in the ipsilateral breast, lymph nodes, or any distant site >120 days after the definitive surgery date with the same histologic type as the primary breast tumor (e.g., ductal, lobular, tubular, mucinous, etc.), following methods described previously (13). Rather than relying on ICD diagnosis codes, the abstractor read through text of oncology notes, pathology reports, and imaging reports to identify recurrences. We included metastatic diagnoses found on clinical imaging with unknown histology. We abstracted information on recurrence date, histology, method of diagnosis (pathology or imaging reports), and tumor markers.

We collected demographic and clinical characteristics [e.g., age, race, ethnicity, body mass index (BMI), menopausal status, Charlson comorbidity index] at the time of initial breast cancer diagnosis from EHR data. We categorized people as postmenopausal if they were 55 years or older, or younger than 55 with evidence of a bilateral oophorectomy. Charlson comorbidity index was calculated using administrative data from the 12 months prior to diagnosis (14). We collected information on initial breast cancer tumor characteristics [stage, grade, estrogen receptor (ER), progesterone receptor (PR), HER2 status], treatment (surgery type, chemotherapy, radiotherapy, endocrine therapy), and second primary breast cancers by linking to the Seattle-Puget Sound SEER tumor registry.

We described the chart-abstracted study population, stratified by abstracted recurrence status. We then described the characteristics of the overall study population using inverse sampling weights. We calculated performance characteristics for each algorithm individually and combined, compared with gold-standard chart abstraction. We calculated the number of TP, FP, FN, and TN in the chart-abstracted study population and weighted to the overall study population. For each individual algorithm and combination, we calculated weighted sensitivity, specificity, PPV, and negative predictive value (NPV) using inverse sampling probabilities.

We sought to understand how algorithm performance might change based on study design decisions. Therefore, we evaluated weighted performance measures among subpopulations: people without a second primary breast cancer diagnosis because these diagnoses may be incorrectly identified as recurrences in an algorithm (remaining N = 523); people with follow-up data truncated at the end of 2014 to mimic an algorithm that uses ICD-9 codes only (N = 556); and people with recurrences identified by both algorithms excluding any with recurrence dates ≥6 months apart because these recurrences may have been incorrectly identified by one or both algorithms (remaining N = 549).

We were interested in how different recurrence algorithm approaches might influence associations between risk factors and recurrence. Therefore, we estimated associations between demographic, clinical, tumor, and treatment characteristics and recurrence. Using Cox proportional hazards models, we modeled recurrence using each definition (from chart abstraction, each algorithm alone, and each combined algorithm) in a separate model. We calculated HRs for recurrence with 95% confidence intervals (CI) using linearized variance estimators to compute SEs, adjusted for age and stage of the incident breast cancer, weighting results using inverse sampling probabilities. For these analyses, we additionally censored people at the date of a second primary breast cancer diagnosis. People with missing data for specific risk factors were excluded from analyses for that risk factor. For cases with different recurrence dates in each algorithm, we used the earliest date for analysis. All analyses were conducted using StataMP.

Data contain potentially identifiable information (e.g., dates of diagnoses and exams) that cannot be shared openly without appropriate human subjects approval and data use agreements, but may be available upon reasonable request from the corresponding author.

Among 600 people with primary breast cancer whose charts were abstracted, 118 (20%) had an abstracted recurrence, which, when weighted to the overall population (N = 2,756), corresponded to 224 (8%) estimated recurrences (Table 1). The mean age at primary breast cancer diagnosis was 61 years among the chart abstracted sample and 62 years among the overall population and did not meaningfully vary by recurrence status. Figure 1 shows the number of recurrences identified by Algorithm 7 and Algorithm 9, stratified by chart abstraction recurrence results. Shading indicates whether the recurrences were treated as TP, FN, FP, or TN. Results for Algorithm 7 alone ignore recurrence results from Algorithm 9, and vice versa. The results for the combined algorithms and triangulation show how discordant pairs were classified as FN or TN for the High Specificity and PPV Combination, FP or TP for the High Sensitivity Combination, and TP or TN depending on the chart abstraction results for the Triangulation Combination.

Among chart abstracted records, we identified 388 (66% of 600 records) recurrences using Algorithm 7 alone and 234 (39%) recurrences using Algorithm 9 alone (Table 2). The High Specificity and PPV Combination identified 117 (20%) recurrences and had the highest weighted specificity (98%, 95% CI: 97–99) and PPV (72%, 95% CI: 63–80), with a sensitivity of 64% (95% CI: 44–80). The High Sensitivity Combination identified 505 (84%) recurrences and had the highest weighted sensitivity (80%, 95% CI: 49–94), with a specificity of 83% (95% CI: 80–86) and PPV of 29% (95% CI: 25–34). Algorithms had similar NPVs whether used individually or in combination.

When we excluded people diagnosed with a second primary breast cancer, none of the weighted performance measures changed substantially (Table 2). When we censored follow-up at the end of 2014 (before ICD-10 codes were implemented), the sensitivity increased to 100% (95% CI: 96–100) for the High Sensitivity Combination, and to 78% (95% CI: 67–86) for the High Specificity and PPV Combination. Specificity also increased slightly while the PPV decreased compared with overall results. Sensitivity decreased slightly and specificity increased across all algorithms when we excluded people with recurrence algorithm dates ≥6 months apart.

There were very few FNs from Algorithm 7 alone (N = 5, 1%) and the High Sensitivity Combination (N = 2, 0%; Supplementary Table S1). In weighted analyses, we estimated 48 FNs from Algorithm 7 alone (2%), and 45 FNs from the High Sensitivity Combination (2%) among the overall population. The High Specificity and PPV Combination had the fewest FPs (N = 34, 6%) compared with other algorithms (Supplementary Table S1). In weighted analyses, this equated to an estimated 55 FPs among the overall population. Using triangulation, we identified 150 recurrences (25% of abstracted cases; Supplementary Table S2). Triangulation resulted in a higher sensitivity (80%, 95% CI: 49–94), specificity (98%, 95% CI: 97–99), and PPV (77%, 95% CI: 68–83) than either algorithm individually or combined.

Abstracted recurrence rates declined as age increased, resulting in a HR as low as 0.3 (95% CI: 0.1–1.0) for people 80 years or older versus women ages 50–59 (Table 3). We noted similar inverse associations between age and recurrence rates based on individual and combined algorithms. Higher BMI was associated with increased abstracted recurrence rates (HR = 2.3, 95% CI: 1.0–4.9 for BMI ≥30 vs. BMI <25). When using algorithms, HRs for the associations between BMI ≥30 and recurrence ranged from 1.0 to 1.2 and none reached statistical significance. People with stage IIB/C or IIIA primary breast cancer had a higher recurrence rate than people with stage I breast cancer based on abstracted results (HR = 5.5, 95% CI: 2.0–14.9 for stage IIB/C, and HR = 3.3, 95% CI: 1.2–9.2 for stage IIIA). These HRs were also elevated when using individual algorithms to define recurrence but varied widely ranging from 4.3 to 12.1 for stage IIB/C and 3.9–13.5 for stage IIIA. There was less variation in these results among the combined algorithms (HR range 6.5–7.1 for stage IIB/C and 5.7–7.6 for stage IIIA).

We extended previous approaches (3, 6) to identify breast cancer recurrences in people with early-stage breast cancer using administrative data. We evaluated several options for identifying recurrences: abstraction, individual algorithms, combined algorithms without chart review to categorize discordant results, and combined algorithms with triangulation to categorize discordant results. We noted wide variation in the number of recurrences identified from each method and performance depending on how we categorized discordant algorithm results. In addition, Algorithms 7 and 9 used different codes to identify recurrences, which may have led to different people being classified as FP and FN. For example, Algorithm 7 included ICD codes for secondary non-breast malignant neoplasms, which may or may not have accurately represented a breast cancer recurrence and could have been coded as a FP (3). Alternatively, Algorithm 9 did not require any treatment codes, which may have missed some recurrences and potentially explains why this algorithm resulted in more FN. These differences underscore the tradeoffs made when choosing a method to ascertain recurrence and highlight the importance of understanding how algorithm choice affects recurrence risk estimations (15).

Overall, our performance results for the combined algorithms showed improvements in specificity and PPV over the individual algorithms, particularly for the High Specificity and PPV combination (98% specificity and 72% PPV). These results were well within the range results noted from prior studies (specificity range 78%–99% and PPV range 14%–94%; ref. 11). Studies that may benefit from using the High Specificity and PPV combination are those that have limited to no chart review resources and wish to minimize potential FP recurrences, which could alter any associations between risk factors and recurrence. Reviewing all records with a positive result would help reduce FP. However, using the High Specificity and PPV combination would miss more recurrences than either algorithm alone or combination. Triangulation, which uses combined algorithms followed by chart abstraction to reclassify discordant algorithm results, resulted in higher sensitivity, specificity, and PPV than either algorithm individually or combined. The sensitivity and PPV for triangulation (80% and 98%, respectively) were similar to those previously published by Kroenke and colleagues (81% and 98%, respectively in the Life After Cancer Epidemiology (LACE) cohort and 73% and 92%, respectively in the Women's Health Initiative (WHI) cohort; ref. 6). However, triangulation would require additional chart abstraction (15% of records in our study), which may not be feasible in a larger study.

Ideally, the true association between risk factors and recurrence in our overall study would be in the same direction as the results from chart review and combined algorithms. The fact that the most of HRs for recurrence were similar in magnitude between the two algorithm combinations suggests our study results will be robust, whichever combination of algorithms we choose. One exception was the association between BMI and recurrence, which was elevated when using the abstraction-based definition for recurrence but not the algorithm-based definitions. This difference may reflect unique characteristics of the chart abstraction sample and/or reduced precision in our estimates. Several prior studies demonstrated small increased recurrence risks associated with increasing BMI with HRs between 1.10–1.30—similar to the HRs we noted for BMI with the combined algorithms (16–19). In addition, while all models indicated an association between increasing cancer stage and recurrence, the HRs differed slightly between models with Algorithm 9 producing the highest HRs. This may be related to FP records that were identified as a recurrence in Algorithm 9 but not in chart review; if they were diagnosed at higher stages, this might alter the association between stage and recurrence for Algorithm 9. Future studies may consider using additional statistical methods to account for potential misclassification in both predictors and outcomes (20).

Researchers should consider potential impacts of study design decisions on algorithm results. For example, specificity and PPV improved slightly after removing people diagnosed with second primary breast cancers from the analysis, suggesting some second primary cancers resulted in algorithm FP. Though this improvement was small, it may be worth censoring records at the time of a second primary breast cancer diagnosis when using a recurrence algorithm if they can be easily identified from a tumor registry. In addition, censoring follow-up in 2014 improved sensitivity while decreasing PPV because truncating follow-up reduced the total number of positives (both TP and FP) by recoding any recurrences diagnosed after 2014 from positive to negative. Thus, ICD-10 codes, which had not been included in previous validation studies for these algorithms, did not substantially alter algorithm performance in this population, but instead reduced the overall number of recurrences. One previous study that evaluated expansion from ICD-9 to ICD-10 codes noted similar results (2). Algorithm development should continue to incorporate and evaluate new codes as they are implemented and used in clinical practice.

When combining algorithms, we had to decide how to handle different recurrence dates; 51 records in our study had recurrences in both algorithms with dates ≥6 months apart. Excluding these records reduced sensitivity because many of them were TP recurrences despite the difference in dates. When evaluating associations between risk factors and recurrences based on combined algorithm results, we selected the earlier of the two algorithm recurrence dates for our outcome. However, one could alternatively use the later date, as this might indicate recurrence completeness, or a midpoint between the two. Understanding potential biases associated with selecting an earlier versus later recurrence date from algorithms is important work for future study. Focusing chart abstraction on records with different recurrence dates from each algorithm could be another approach to improving accuracy with limited resources. While validating recurrence date was beyond the scope of this analysis, this is worth examining in future research.

A strength of our study was our ability to validate two algorithms for breast cancer recurrence and calculate performance measures using different combinations of these algorithms. Although the algorithms we validated were based on work previously done at KPWA (3) and KPNC (6), we conducted this validation in a sample of breast cancer records that were not previously included in either publication. Nonetheless, we acknowledge these algorithms may perform better in our population than in a non-KP population. We can also be reasonably confident that we had access to all diagnosis, procedure, and drug codes for each person (including codes from external claims) up to the date of disenrollment because the KPWA health plan relies on accurate coding for billing (21).

One important limitation of our study is the limited racial and ethnic diversity in our sample. The demographics, while reflective of the KP populations used for the analysis, resulted in wide CIs when analyzing associations between race or ethnicity and risk of recurrence. We were unable to make inferences about how these variables may be associated with algorithm performance. We recognize the importance of conducting future analyses using delivery system data with more proportionate racial/ethnic representation to ensure that we understand the impact of using recurrence algorithms for populations who have historically been excluded from or mistreated by research and subjected to disparities in health care delivery. In addition, we were not able to account for death as a competing event in our analyses while also accounting for sampling weights. Our performance analyses also excluded 28 cases (4%) of the medical record review based on study eligibility criteria. We recognize that a study that did not have resources for medical review would not know to exclude these cases, which may affect algorithm performance. Finally, we acknowledge that the improved accuracy we saw from combining algorithms only applies to the two algorithms we included in this study. Several breast cancer recurrence algorithms exist, and performance of different algorithm combinations will likely differ from our results.

In conclusion, our study advances previous approaches to identifying recurrences using two previously developed algorithms without additional chart abstraction, and highlights issues to consider when applying algorithms to EHR data to identify breast cancer recurrences. We suggest potential analyses to evaluate algorithm performance and describe tradeoffs in deciding which approach to use. Accurate recurrence identification via algorithms could be incredibly useful in large, observational studies to further research on risk factors and treatments that affect recurrence rates. Continued testing, refinement, and standardization of recurrence algorithms across different populations should help improve performance, increase generalizability, and further reduce the need for resource-intensive medical record abstraction to identify breast cancer recurrences (1, 11).

E.J. Aiello Bowles reports grants from NCI during the conduct of the study. C.H. Kroenke reports grants from NCI during the conduct of the study. J. Chubak reports grants from NIH during the conduct of the study. J. Bhimani reports grants from Memorial Sloan Kettering Cancer Center during the conduct of the study. K. O'Connell reports grants from NCI of the National Institutes of Health and Geoffrey Beene Cancer Research Center during the conduct of the study. S. Brandzel reports grants from NCI during the conduct of the study. M.K. Theis reports grants from NCI during the conduct of the study. E.V. Bandera reports grants from NIH during the conduct of the study; personal fees from Pfizer, Inc outside the submitted work. L.H. Kushi reports grants from NCI during the conduct of the study. E.D. Kantor reports grants from NCI and other support from the Geoffrey Beene Cancer Research Center during the conduct of the study. No disclosures were reported by the other authors.

The funder did not play a role in the design of the study; the collection, analysis, and interpretation of the data; the writing of the article; and the decision to submit the article for publication.

E.J. Aiello Bowles: Conceptualization, resources, data curation, formal analysis, supervision, funding acquisition, validation, investigation, visualization, methodology, writing–original draft, writing–review and editing. C.H. Kroenke: Conceptualization, validation, investigation, visualization, methodology, writing–original draft, writing–review and editing. J. Chubak: Conceptualization, validation, investigation, visualization, methodology, writing–original draft, writing–review and editing. J. Bhimani: Investigation, methodology, writing–review and editing. K. O'Connell: Data curation, formal analysis, validation, methodology, writing–review and editing. S. Brandzel: Supervision, investigation, project administration, writing–review and editing. E. Valice: Data curation, software, writing–review and editing. R. Doud: Data curation, software, writing–review and editing. M.K. Theis: Data curation, software, writing–review and editing. J.M. Roh: Project administration, writing–review and editing. N. Heon: Project administration, writing–review and editing. S. Persaud: Project administration, writing–review and editing. J.J. Griggs: Validation, investigation, writing–review and editing. E.V. Bandera: Conceptualization, funding acquisition, validation, investigation, methodology, project administration, writing–review and editing. L.H. Kushi: Conceptualization, funding acquisition, validation, investigation, methodology, project administration, writing–review and editing. E.D. Kantor: Conceptualization, resources, funding acquisition, validation, investigation, visualization, methodology, writing–original draft, project administration, writing–review and editing.

This work was supported by grants from the NCI [R37CA222793 (all authors received), P30CA008748, P01CA154292, R50CA211115 (E.J. Aiello Bowles), and R01CA230440, R01CA253028 (C.H. Kroenke)] as well as the Geoffrey Beene Cancer Research Center at Memorial Sloan Kettering Cancer Research. This work was also supported by the Cancer Surveillance System of the Fred Hutchinson Cancer Research Center, which is funded by contract nos. N01-PC-35142, N01-PC-2010-00029, HHSN261201300012I, and N01 PC-2013-00012 from the SEER Program of the NCI with additional support from the Fred Hutchinson Cancer Research Center and the State of Washington.