Validity of Women’s Self-Reports of Cancer Screening Test Utilization in a Managed Care Population

To gain information on the prevalence of screening in a particular population, most studies rely on self-reports rather than medical record review because of the time and cost constraints of medical record abstraction as well as the difficulties in accessing medical records (1). However, the quality of these estimates is only as good as the ability of people to recall and report the use of screening services. It is always a concern that self-reported data might be inaccurate or biased. A number of studies have validated self-reports of mammography and/or Papanicolaou tests with medical records (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13). In most cases, the validity of self-reported mammography data was demonstrated, and the women were found to be highly accurate in their self-reports (1, 2, 3, 4, 6, 7). In one study, medical record review found women to be 97% accurate in reporting whether or not they had a mammogram within the past year (4). In general, studies have found that women were accurate in reporting whether or not they had a mammogram but often underestimated the time since their last mammogram (1, 2, 3, 7, 12). In studies that looked at self-reports of both mammography and Pap tests, most demonstrated greater accuracy for self-reports of mammography than for self-reports of Pap test (1, 9, 12). In one of these studies, women reported mammograms 3 months later and Pap tests 23 months later than they actually had them (1). The studies that reported exclusively on Pap tests suggested that self-reports are not particularly accurate (10, 11, 13). The one study that also looked at CBE⁵ found that concordance between self-report and medical record was lower for CBE (69.8%) than Pap test (78.4%) or mammogram (83.7%) (12).

Population-based surveys are an important tool for estimating the use of health care services among the population. The BRFSS is administered jointly by the CDC and state health departments to monitor the prevalence of major risk factors for chronic diseases and the utilization of screening services by each state’s adult population (14). Although several studies have examined the validity of self-reported BRFSS data (15, 16, 17), only one study has dealt specifically with mammography, CBE, and Pap tests (18). This report describes the results of a validation of the self-reported use of mammography, CBE, and Pap smear using the BRFSS in a managed care population.

The Colorado Department of Public Health and Environment has been conducting the BRFSS on an ongoing basis since January, 1990. In 1993, it collaborated with KPC to conduct a special BRFSS among a sample of the KPC managed care population to ascertain the prevalence of risk behaviors in this population and to assess the validity of selected questions on the BRFSS. Toward this end, an age- and sex-stratified sample of adult members of KPC was interviewed, and their medical records were audited. The methods and results of this investigation are described in detail elsewhere (19).

A second sample consisting of women aged 40–74 years who were enrolled in KPC at the time of the study and who had been enrolled for at least 5 continuous years before the study was also selected to address the validity of responses to questions included in the Woman’s Health Section of the BRFSS. The questions evaluated address whether a woman has ever had a mammogram, CBE, or Pap smear; her reasons for having one of these procedures; and the length of time since her last procedure. This sample was restricted to women aged 40–74 years to account for consensus guidelines in 1993 regarding age of initiation of mammography screening and ongoing discussions on the utility of mammography among women aged 75 years and older (20, 21). Whereas the recommendation for cervical cancer screening is that Pap tests be done beginning at age 18 years (or after the initiation of sexual activity), data from the BRFSS have indicated that Pap smear utilization tends to decrease after the reproductive years (22). Therefore, from a prevention perspective, Pap test reporting by older women was of particular interest.

In 1993, approximately 280,000 individuals were enrolled in KPC in the Denver metropolitan area. Of these, about 22% were women aged 40 years or older, and about a third of these women had been enrolled in KPC for at least 5 years. Within this sampling frame, approximately equal numbers of women who were currently enrolled in KPC either as the primary insurance carrier or a dependent for at least 5 continuous years were randomly selected from three age strata (40–49, 50–64, and 65–74 years) for inclusion in the validation study. Potential study subjects were contacted by telephone and asked to participate. At least 8 attempts during different calling periods (evenings, week-ends, and daytime) were made before it was determined that a respondent could not be reached. In all, 592 women were sampled, and 498 completed interviews were obtained for a response rate of 84%. Among the 94 women who did not respond, 24 women refused to participate, 40 could not be located or contacted with the information in Kaiser files, 26 were no longer Kaiser members, and 4 could not communicate over the phone due to physical or mental impairments. Study participants responded to questions from the Demographic and Women’s Health Sections of the 1993 BRFSS. Additional questions were added to the survey instrument to ascertain whether the most recent mammogram, CBE, and/or Pap test had occurred at KPC or some other service delivery site. The medical records of respondents were abstracted by an employee of KPC who was unaware of the results of the telephone survey. Data from the medical chart review were directly entered into a computer through a structured screen entry program. Quality of the data was maintained through reabstraction audits and use of computerized edits. An additional 18 respondents were excluded from further analysis because they did not have medical record information. This left a total sample of 480 women for analysis.

Survey responses were compared with information obtained from the medical records. Medical records where used as the “gold standard.” Sensitivity, specificity, PPV, and NPV as defined by Morrison (23) were calculated for the overall sample and for demographic subgroups. In assessing the validity of “ever having had a procedure,” respondents who responded positively to having had a procedure but reported having it someplace other than KPC were excluded from analyses. Only medical records from KPC were included in the study, and therefore, reports of “never” having had a procedure cannot be confirmed. It is possible that a respondent received a mammogram, CBE, or Pap test from some other health care provider. Therefore, in this report the “never” category used in analyses should be interpreted as “never having had the procedure at KPC.”

Comparisons between self-reports and medical record data were made for “ever having had a screening exam” and for “had a mammogram within the past 2 years,” “had a CBE within the past 2 years,” and “had a Pap smear within the past 3 years.” For comparison purposes, time since last screening exam ascertained from the medical record was defined as the difference between the date of the most recent procedure found in the medical record and the date that the study subject’s questionnaire was completed.

The percentage of overall agreement between the survey responses and the medical records was also calculated. Logistic modeling was used to determine whether any of the demographic variables were associated with higher overall agreement regarding whether women had a mammogram or CBE within the previous 2 years or a Pap test within the previous 3 years (24). Cohen’s κ statistic was also calculated to provide a measure of agreement beyond chance (25). Values of κ range from 1.0 (complete agreement) to −1.0 (complete disagreement), with a score of 0 indicating expected agreement by chance alone.

The 480 women were approximately equally distributed over the three age strata (40–49, 50–64, and 65–74 years). The women were predominantly white (91%), non-Hispanic (92%), married (69.7%), and high school graduates (93%). More than 50% of the women were employed. Nearly 97% of the women reported ever having a mammogram and a CBE, whereas 99.2% reported ever having a Pap test. About 94% of the women claimed that their last mammogram was at KPC, with a similar result for CBE (96.6%) and Pap test (94.1%).

The sensitivity of the BRFSS for determining whether a woman ever had one of these screening tests at KPC was 100% for mammography, 97.3% for CBE, and 99.8% for Pap tests. Sensitivity was consistently >91% across demographic subgroups. In contrast, the specificity of the BRFSS for determining whether a woman ever had one of these screening tests at KPC was low overall and for most demographic subgroups. For mammography, the overall specificity was only 53.1%. It was highest in women who were retired (75%) and in women 40–49 years of age (60%). For CBE and Pap tests, overall specificity was calculated as 12% and 5%, respectively, and was consistently low across demographic subgroups.

Table 1 presents a comparison of BRFSS responses for having had a mammogram within the previous 2 years with information obtained from KPC medical records for the 453 women for whom it was possible to determine from both sources whether or not they had a mammogram within the past 2 years. As seen in this table, the sensitivity for determining whether a woman had a mammogram within the past 2 years at KPC was 98.6% and was over 95% for each of the demographic subgroups examined. Specificity was 54.2% overall, whereas PPV and NPV were about 90% overall. Across all four measures, values were lower among nonwhites than whites and among women who did not graduate high school as compared with those who did. The percentage of overall agreement was 88.4% and was found to be significantly higher in women 50–64 years old than in women 40–49 years old when modeled using logistic regression. To correct for chance agreement, the κ statistic was calculated and found to be 0.61, with the lowest values (<50%) seen in women aged 50–64 years, nonwhites, Hispanics, women with less than a high school education, and women who were unemployed.

As seen in Table 2, the sensitivity for determining whether a woman had a CBE within the past 2 years at KPC was 95.1% and was >90% for each of the subgroups tested. Specificity was 45.5% overall and was lowest in women aged 50–64 years (31.6%) and in retirees (20.0%). The PPV was >86% in all subgroups, whereas the NPV was highest in employed women (79.2%). The percentage of overall agreement was 87.9% and was found to be significantly higher in retired women than in employed women when modeled. The overall κ was 0.45 and ranged between 0.13 in retired women and 0.57 in women 40–49 years.

The sensitivity for determining whether a woman had a Pap test within the past 3 years at KPC was 99.1% and was >96% for each of the subgroups (Table 3). PPV was also high (86.4%). As with mammography and with CBE, the specificity of the BRFSS was quite low (44.9% overall and 50% or less across the subgroups). The percentage of overall agreement was 87.2% but was not found to be associated with any of the demographic variables when modeled. κ for all women was 0.54, with some variation among the subgroups of the demographic variables.

Tables 4,5,6 show concordance in the time interval since the last mammogram, CBE, and Pap between a woman’s self-report and the medical record for those women for whom it was possible to determine the actual time interval from both sources. The Pearson correlations between self-report and chart abstraction for most recent mammogram, CBE, and Pap were 0.72, 0.58, and 0.65, respectively. Concordance increased substantially when the time interval was allowed to vary by ±1 year. In most cases of disagreement, the self-report underestimated the time since the most recent exam. A total of 101 of 433 women underestimated the time since their last mammogram, and nearly half of them did so by more than 1 year, whereas only 20 women overestimated the time interval (one of whom did so by >1 year). A total of 71 of 416 women underestimated the time since their last CBE exam (nearly one-fourth did so by >1 year), whereas only 25 women overestimated it (8 did so by >1 year). Ninety-four of 387 women underestimated the time since their last Pap test, nearly one-third of whom did so by >1 year, whereas only 27 women overestimated this time interval (4 women did so by >1 year).

The results suggest that self-reported BRFSS data are highly sensitive for assessing the prevalence of breast and cervical cancer screening in this managed care population but not very specific. However, it is important to keep in mind that this study used a relatively homogenous insured managed care population composed of mainly white women, aged 40–75 years, with at least a high school education, who were either currently employed or retired. Although the results cannot be generalized to the United States population, they provide credible insight regarding the utility of the BRFSS in an important segment of the population.

Previous studies of the utilization of screening tests for female cancers in managed care populations have shown high sensitivities and low specificities (4, 12). In our study, specificity tended to be low for mammography, CBE, and Pap test. This finding may be due to an underestimation of the time since the last exam. When asked how long it had been since their last mammogram, CBE, or Pap test, women in the study often underestimated the actual time interval. This phenomenon, known as telescoping, is one of the most consistent findings among studies that compare self-reports with medical records (3, 5, 7, 8, 9, 10, 11, 12). Reporting the date of the screening examination has generally been shown to be more problematic for individuals than reporting on whether or not a screening examination was performed. As a result, self-reports often overestimate the use of screening.

Before 1989, there were no published studies in which the validity of mammography self-reports was evaluated. Since that time, a number of studies have been published (2, 3, 4, 5, 8, 9, 12). Our results are similar to those seen in most of the other studies, with sensitivities tending to be >90% and percentage of overall agreement tending to be >80%.

Our results for Pap test tended to be higher than those seen in a number of other studies (1, 9, 11, 12, 13). It is difficult to speculate on the reasons for this variation in results. Some of the variation might be a function of the medical record documentation in the medical facilities serving the women in the studies. Medical records for women using public health services with multiple providers may not be as complete as medical records in the managed care setting. In addition, it is possible that the medical record abstractors in these studies were not able to identify all of the sites in which these women might have obtained a Pap test, thus preventing them from obtaining a complete history. KPC maintains an integrated medical record, or one record for each patient. Therefore, even if a woman went to different sites within KPC in different years, the abstractor only had to review one medical record to locate all of her laboratory and imaging information. This made it possible for the abstractor to get the actual chart 100% of the time and to find all of the information arranged in a similar manner. In addition, some variation seen between studies might be attributable to the interval selected. The overall level of agreement was 87.2% for Pap tests within 3 years. In another study, an overall agreement of only 66% was seen between self-report and medical records for a Pap test within 1 year, but an overall agreement of 78% was seen when the interval was increased to 2 years (12).

In our study, κ values of 0.62, 0.54, and 0.45 were calculated for agreement between the self-report and the medical record for mammograms, Pap tests, and CBEs, respectively. This ranking was similar to that reported by Gordon et al. (12), who reported κ values of 0.61 for mammograms, 0.38 for Pap tests, and 0.23 for CBEs. One might expect this, given the fact that mammograms and Pap tests generate special documentation in the medical record, whereas CBE does not and is merely recorded by the provider who performed the examination. In fact, because physicians do not necessarily document in the medical record everything they do during an office visit, it is quite reasonable to suspect that relying on the medical record as a gold standard for CBE performance might lead to lower levels of concordance.

An important consideration when analyzing data such as these is whether to consider this a validity study or a reliability study. The basic question is whether or not the medical record is always or virtually always correct and therefore has the status of a gold standard. In this study, the medical record was considered to be the gold standard because members of KPC receive the majority of their care through the health management organization (HMO), all of which is recorded in one integrated medical record, thus increasing the likelihood of the data being complete. However, given the fact that some women do receive care outside of KPC, which would be absent from the medical record, as well as the fact that errors are sometimes made in the notation and entry of data, we included measures of the reliability between the patient self-reports and the medical record (percentage of overall agreement and κ), in addition to the validity measures.

There are several limitations to this study. In addition to the lack of generalizability already discussed, validation of mammography, CBE, and Pap test was restricted to women who reported that their last tests were done at KPC, and no attempt was made to verify self-reports of tests done elsewhere. Despite the limitations, our study indicates that self-reported BRFSS data are highly sensitive for assessing women’s utilization of screening tests for breast and cervical cancer in this managed care population. One advantage of our study is that it limited itself to screening tests. By asking the women what was the purpose of their screening test, our study was able to determine which tests were for screening purposes and which were for further diagnostic purposes. As a result, from our study we are able to make observations regarding self-reported utilization of screening tests. The level of agreement above chance as measured by the κ statistic indicates fair to good agreement of self-report with medical records overall, although poorer agreement was noted for some demographic subgroups such as those with less than a high school education. Telephone surveys are a relatively inexpensive, time-efficient method of gathering information about the use of health services. The BRFSS has been widely used by a number of state health departments and the CDC to evaluate the impact of programs aimed at increasing breast and cervical cancer screening. Our study results suggest that self-reported data ascertained using the BRFSS provide an accurate estimate of the prevalence of screening for breast and cervical cancers in KPC and possibly other similar managed care populations with similar enrollees. Therefore, it would seem reasonable for the BRFSS to continue to use self-reporting as the means of obtaining its data. In addition, it is reasonable and appropriate for the BRFSS to continue to use its current wording to obtain its data regarding women’s usage of mammography, CBE, and Pap tests.

Accuracy of self-reported health behaviors has been recently brought into question as a result of a critical review of the literature on the topic conducted by Newell et al. (26). They found that self-reported data more often than not overestimated the proportions screened determined from gold standards. Although our own findings lead us to suggest that self-reported data on breast and cervical cancer screening exams indeed have value, we acknowledge that investigators relying on self-reported data should seriously consider the suggestions of Newell et al. (26) for improving the accuracy of such data.