Validating Self-Reported Mammography Use in Vulnerable Communities: Findings and Recommendations

Most health surveys ask women whether they have had a mammogram and when they had their last one. Virtually all such surveys show that Black and White women are obtaining mammograms at about the same rate. For example, according to the Web-based prevalence and trend data of the 2012 U.S. Behavioral Risk Factor Surveillance System (BRFSS), 74% of non-Hispanic White women, 78% of non-Hispanic Black (NHB) women, and 69% of Hispanic women reported receiving a mammogram in the past 2 years (1). A community-based program called the Racial and Ethnic Approaches to Community Health (REACH) found that in 2009, 80% of Black women and 77% of Latina women reported recent mammography use in Chicago neighborhoods (2). These reports, and others like them, suggest that a substantial majority of women who are getting mammograms routinely have been doing so for several years, and that there is no notable racial disparity.

Other studies have found, however, that self-reported health behaviors may be problematic because people tend to overreport desirable behaviors (e.g., exercise) and underreport undesirable behaviors (e.g., smoking; refs. 3–6). Many studies compared self-reported mammography histories with those documented from chart reviews (7–26) or other records [e.g., Medicare databases (27) or national mammography registries (28)]. These studies, including a recent meta analysis, found a general tendency to overreport rather than underreport mammography use, and suggest that overreporting might be greater in racial/ethnic minorities (29).

As part of the Helping Her Live intervention, a unique outreach and community navigation program aimed at increasing routine mammography use in two poor communities on the west side of Chicago, we conducted a baseline survey that asked about, among other things, prior mammography history (30). The purpose of this survey was to not only find out how women report mammography usage, but also how well mammography and breast cancer is understood by women in these vulnerable communities as well as to verify self-reported mammography usage. The findings of the survey were to be used to inform the educational component of the intervention and not aimed to recruit women into the intervention. During the survey, we also requested permission to examine medical records of the surveyed women, which allowed us to compare self-report with medical records information. Although such comparisons have been made before, very few have pursued this issue in such vulnerable communities where one might expect low mammography rates and substantial overreporting (8–9, 12).

The purpose of this analysis was to examine the potential extent of overreporting and possibly underreporting of mammography use in these vulnerable populations, and to determine how self-report inaccuracies might bias estimated associations between patient characteristics and mammography use.

We interviewed women ages 40 and older who lived in the Chicago communities of Humboldt Park (HP) and North Lawndale (NL). HP is half Black, a quarter Mexican, and a quarter Puerto Rican, whereas NL is almost entirely Black (31). Both communities are very poor (31, 32). On the basis of the 2010 American Community Survey 5-year estimates, the Median Household Income (MHI) for Chicago residents was about $56,000. However, in HP, a community that is in the process of gentrification, the MHI was about $33,000; in NL, the MHI was $27,000 (32). We targeted 500 women from each of the following four sectors of the two Chicago community areas: NHB women in NL; NHB women in HP; Puerto Rican women in HP; and Mexican women in HP.

Thirteen female interviewers (6 Spanish speaking) who resided in the targeted communities were hired and trained to administer the survey. The 12-hour training involved discussions on how to conduct research studies, privacy and Health Insurance Portability and Accountability Act (HIPAA) rules, interviewing skills, and the general protocol for interviewing women in the community. In addition, each question, including the response options, and its meaning were thoroughly reviewed, and interviewers practiced their skills through role-playing with trainers as well as piloting their skills with caregivers in the cafeteria of the hospital in which this program was housed. In each step, all interviewers were provided feedback to assure quality and systematic data collection. The survey was approved by the Institutional Review Board of the Sinai Health System.

We produced a comprehensive list of venues in the targeted areas that served women ages 40 years and older, and these venues served as the sampling frame. Examples of venues included local pharmacies, laundromats, and grocery stores. Between March and June 2008, 3,199 women were screened for eligibility based on their age (40 years and over) and residence. Our interviewers were culturally matched to the sector in which the interviews were conducted. Interviewers were deployed at the various venues in each community area every day and there were very few instances where interviewers were shared across sectors. Adult women were screened for eligibility if they were at the venue at the same time as an interviewer. Interviewers completed 2,200 surveys and, among them, 144 respondents did not reside in either NL or HP based on the self-reported address provided and were thus excluded (Fig. 1).

Face-to-face interviews were conducted, and responses were recorded on a paper-based instrument; each survey took about 20 minutes to complete. As an incentive, each respondent was given a $20 gift card for a local business, including Walgreens, CVS, supermarkets, and Wal-Mart. An additional 147 surveys were excluded due to missing data on race/ethnicity or reporting a race/ethnicity other than NHB, Mexican, or Puerto Rican (final sample, n = 1,909 women; Fig. 1).

The survey included the following questions relevant to this analysis:

1. “Have you ever had a mammogram or breast X-ray?” (33).

2. “How long has it been since you had your last mammogram?” (33).

3. “In what month and year was your last mammogram?” (33). For those who could not recall the month we also asked, “In what season was your last mammogram: winter, spring, summer, or fall?” (new question).

In addition, women reported where they received their most recent mammogram, as well as any other facilities they used for primary or preventive care. We used this information to document prior mammogram histories during the past 5 years.

Each woman was asked to sign a HIPAA authorization form giving us permission to access her medical records at the specific facilities she reported; 1,566 of the 1,909 surveyed provided authorization (82%). Many of the 128 facilities listed on the authorizations were located out of state or had been reported by less than 10 women. We focused our abstraction efforts at the 18 facilities with more than 10 corresponding medical record authorizations, together representing 78% of the sample who gave us authorization. We requested medical records on the 1,221 women who provided a valid HIPAA authorization form and had at least one facility listed on the form (Fig. 1).

Data describing the facilities listed on the medical record authorization forms were entered into a Microsoft Access database that was linked to the survey responses. Before sending a request to the facility, each entry was cleaned by hand to ensure accuracy of the entries. Each facility received a request package, which was either hand delivered by a staff member of the project or mailed to a specific contact in the facility's Medical Records Department. Communication between the research staff and the mammography facility was ongoing as long as we were abstracting data from the site. Thus, some sites required several requests before completion of the data collection.

For each of the 18 facilities from which we abstracted data, we created a protocol outlining the unique methods for that particular site. Abstractors were blinded to self-reported mammography status. Three sites allowed our staff to abstract data on site using an electronic medical record, while the remaining sites returned photocopies of the following (where available): breast imaging records for those who received a mammogram between 2003 and present, mammography referrals, lists of appointments attended, pathology reports for breast tissue specimens, and physician notes. For those who had no mammograms during this time frame, but who were past or current patients at the facility, we received the patient's medical record number and any breast imaging referral data (if available) or a memo from the facility stating that there were no breast images during the time period. If a patient we requested had never been seen at the facility listed on the HIPAA, a memo from the facility was provided stating that the records for the patient could not be located.

The abstractors collected information on any mention of a breast image documented in the medical record during the past 5 years that included the image date and the procedure type (mammogram, ultrasound, etc.). Every record was later reviewed by a senior abstractor. A random sample of completed abstractions was also re-reviewed to ensure further accuracy of data collection. Each record was thus reviewed at least twice and some were reviewed 3 times. All data entry elements were independently reviewed by a second member of our team for accuracy, and a senior abstractor randomly performed a quality check on 10% of the records.

Data were analyzed using SAS statistical software v 9.1.3 and Stata version 12. We compared self-reported mammography in the 24 months before the date of the survey with corresponding documented history in the medical record (treated as the gold standard). For each self-reported versus documented screening history, we identified the number of positive reports that were documented as positive (true positives); the number of positive reports that were documented as negative (false positives); the number of negative reports that were documented as positive (false negatives); and the number of negative reports that were documented as negative (true negatives).

To assess the robustness of our results to potential selection bias due to incomplete record abstraction, we conducted multiple imputation analyses as follows. We started with the complete dataset of 1,909 eligible women, of whom 1,221 were included in our main analysis. Using the method of chained equations as implemented by the mi command in Stata, we conducted multiple imputations to account for the missing values for self-reported mammography use (N = 29), documented mammography use (N = 688), education (N = 25), and income (N = 213). We set the three women with missing data on insurance status to the mode (uninsured). This method enables the analyst to choose an imputation model suitable for the distribution of each variable with missing data (e.g., logistic regression for binary variables) and to tailor the choice of predictor variables to each variable being imputed (34). We created 20 imputed datasets each with a size of 1,909. We then reestimated our analysis models using Rubin rules for combined estimation results across multiply-imputed datasets (35).

We calculated measures to gauge the extent to which reliance on self-report might over- or underestimate population prevalence of screening mammography. Reports to records ratio (RRR) was defined as the percentage of positive mammogram self-reports divided by the percentage of positive documented mammogram use. A value over 1 suggested overreporting and a value below 1 indicated underreporting. Reports to records difference (RRD) was defined as the difference between the percentage of positive mammogram self-reports and the percentage of positive documented mammogram use. A value greater than 0 suggested overreporting and a value below 0 indicated underreporting.

We also calculated standard measures of self-report accuracy, including concordance, κ, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV; 36). Extent of overreporting was also assessed by calculating false positive rates (FPR) equal to one minus the specificity, with FPR = 0 meaning no overreporting, and FPR = 0.5 meaning that half of all truly negative histories were reported as positive.

We were able to abstract medical records from 1,221 (64%) of our eligible sample of 1,909 women; 343 women refused consent to medical record abstraction, and another 345 had documentation at a facility at which we did not abstract. Women who reported a mammogram in the prior 2 years were considerably more likely to have their medical records abstracted than women not reporting a prior history (72% vs. 50%, P < 0.0001; Table 1). Women in their 40s were less likely than older women to have records abstracted, and Mexican American women were less likely than others to have their records abstracted. Uninsured women were the least likely, and Medicaid patients the most likely to have their records abstracted for this study. Women with less income or less education were more likely to have their records abstracted (Table 1). The data in Supplementary Table S1 suggest that those who had data abstracted were less likely to report having a mammogram in the last 2 years, more likely to be younger, more likely to be Mexican American, and more likely to be uninsured compared with those who did not have their data abstracted regardless of the reason for nonabstraction (Supplementary Table S1).

For the final sample of 1,221 women with interview and complete medical record data, three fourths (73%) reported a mammogram in the past 24 months, but only 44% of these had a documented prior mammogram within 2 years (Table 2). Half the sample was NHB, and a fourth (23%) lacked any type of health insurance. More than half (55%) had annual incomes at or below $10,000 (Table 2).

Table 3 presents the measures of accuracy by various demographic factors. Prevalence of mammography use via self-reports and documentation is presented before and after attempting to account for selection bias via multiple imputation methods; both sets of analyses yielded very similar results. Mexican and Puerto Rican women reported mammography use at higher rates than NHB women (P < 0.0001). Reported use was lowest for women in their 40s compared with older women, and highest among uninsured and Medicare-insured women, and lowest for privately insured women. Reported use was also higher for women with more education and with more income. Associations were similar with respect to documented use, although no longer statistically significant with respect to education and income (Table 3). Across all categories of all sociodeomographic variables examined, mammography use estimates based on self-reports were considerably larger than the corresponding estimates based on medical record documentation (RRR ranged from 1.53 to 1.91; RRD ranged from 25 to 33 percentage points; Table 4).

Accuracy estimates for Mexican and Puerto Rican women were virtually indistinguishable from one another and so these two ethnic groups are combined in Table 4. Most of the inaccuracy in self-report seemed to be in the form of overreporting, as seen by the low specificity and low PPVs overall and within racial/ethnic groups (Table 4). Latinas had a higher FPR, indicating greater overreporting among Latina compared with NHB women. The higher apparent FPR remained even after adjusting for venue type, age, education, income, and health insurance type (Table 5).

Our study demonstrated a considerable amount of overreporting of mammography use in this sample of vulnerable populations, which was evident across levels of other sociodeomgraphic variables and was accompanied by very little apparent underreporting. We estimated that about 4 of every 10 self-reported prior mammogram histories are overreported (PPV = 58%), and that more than half of all documented negative histories corresponded to a positive self-report (FPR = 56%). The extent of overreporting in our study seems to be greater than that estimated in most prior studies that have estimated overreporting in terms of PPV and/or FPR (29).

The level of overreporting varied by race such that Mexican and Puerto Rican women, although culturally very distinct, were both more likely to overreport compared with NHB. FPRs were also high in all three ethnic groups, but were nearly 20 percentage points higher in Puerto Rican women and nearly 30 percentage points higher for Mexican women than for NHB women. Despite the disproportionate extent of overreporting in Latina versus NHB women, at the population level, racial/ethnic-specific estimates of mammography use were equally overly optimistic in all the ethnic minority groups: self-reports overestimated documented use by 27 to 30 percentage points across the three groups.

Very few prior studies have recruited people from community-based venues and with a similar demographic makeup to our study (15–17). One such study oversampled African Americans and Puerto Ricans at a community health center and reported similar concordance rates to our study (65% overall, 71% for NHB, and 60% for Puerto Ricans; ref. 17). These women were already linked to a medical home, unlike many of the participants in our study.

Other studies that reported lower levels of overreporting drew their samples from patients with access to care (e.g., members of insurance plans). For example, Caplan and colleagues reported data on samples drawn from two different insurance plans and reported much less overreporting than our study (PPV = 0.88 and 0.84, respectively, and specificity = 0.54 for both studies; ref. 21, 22). These findings suggest that there may be a difference in the amount of overreporting based not only on race/ethnicity but also on access to care more generally.

Mammography is particularly vulnerable to forward telescoping of the dates of last mammogram (8, 9, 19, 21, 29). Women may recall a mammogram occurring sooner than it actually occurred based on the medical record. Depending on the extent of telescoping, this may cause an overestimation of measures of accuracy. Some prior studies have included a grace period to allow a woman to “forward-telescope” the date of her most recent mammogram by a few months and still be considered an accurate reporter. We chose not to include such a grace period. As a result, our estimates of overreporting may be somewhat higher than from some other studies in part due to this difference in study design.

We estimated that about one of every 20 documented mammograms went unreported (sensitivity = 0.94). Consistent with most prior studies, the level of underreporting of mammography use in our sample was far outweighed by the corresponding level of overreporting (12, 21, 29). The general interpretation is that there is little underreporting regardless of the study methodology.

There are three overarching differences between the studies reviewed above and our study, which lead to important strengths in our study. First, the sample size for our study (n = 1,221) is more than 3 times that of most other similar validation studies. Second, the sample in our study includes women found at various community-based venues rather than in clinics, health care insurance plans, or even one type of community-based venue. As such, our sample represents a more disenfranchised group of women, many of whom lack a medical home. Finally, unlike many studies, we gathered mammography histories on women regardless of how they reported mammography use. Because of the sample size and our strict adherence to our data collection methods, gathering this data was a substantial undertaking involving 18 hospitals in and around Chicago. Collecting data on negative self reports allows us to look at the whole picture of mammography reporting accuracy and reliability, including NPV, sensitivity, and specificity.

In addition to these strengths, there are some limitations worth noting. Women were asked to list all of the sites where they had received a mammogram in the last 5 years. Some women may not accurately recall where their most recent mammogram took place and could therefore be misclassified as a false positive when they were in fact a true positive, resulting in an overestimate of overreporting. This is a potential concern for all validation studies, because one can never validate a woman's report of her last mammogram location. We tried to overcome this limitation by asking for all of her facilities for primary and preventive care, and requesting records from all of these facilities, including facilities that are contracted or subcontracted with the State Early Detection Program, Illinois Breast and Cervical Cancer Program. Because of the issue of telescoping, extending the period beyond 5 years may have been warranted to have more documentation from medical records, which would have allowed a more complete assessment of the accuracy of self-report. However, we focused on the presence or absence of a mammogram within 2 years of the interview, and allowed a 5-year window for documentation in the medical record.

It is possible that the environment of specific venues where women were interviewed, as well as the associated interview context, may have affected the quality of responses when compared with other studies where women were interviewed in a more controlled setting, though this is speculative. In addition, the face-to-face methodology might have introduced some bias. There is some evidence to support that there are very small, and not statistically significant, differences in responses about various colorectal cancer screening behaviors when a survey is done face-to-face compared with other, more private, modes of survey data collection such as by telephone or by mail (37). Unfortunately, a similar study has not been conducted for other screening behaviors such as mammography; we can assume that the results would be similar.

Finally, women for whom we abstracted medical records were more likely to have reported a recent prior mammogram, to be older, be NHB or Puerto Rican, have health insurance, and have less income and education, than women for whom we did not abstract records. Some bias might have been introduced by excluding approximately 10% eligible respondents because the facility listed on the authorization was invalid according to our protocol (n = 198, see Fig. 1). Although we attempted to account for potential selection bias in our analysis via multiple imputation, unmeasured selection factors may nonetheless have affected estimated results as with any study design. Of particular concern is the potential for underascertainment of documented mammography histories among recent Mexican immigrants, some of whom may have had a mammogram at a facility from which we did not abstract data. The FPR was 66% for the 1,221 women who provided authorization and whose records were abstracted. If the true (unobserved) FPR was similar for women who provided authorization, but whose records were not abstracted (e.g., 66%), and was higher for women who refused HIPAA authorization (e.g., 80%), then the corrected FPR would be 68.5% instead of the observed estimate of 66%. Therefore, very little change in our outcomes and conclusions would be revealed under this scenario.

It has long been established that reliance on self-reports for estimating mammography use produces an overestimate. Our results suggest that this problem is exacerbated in vulnerable ethnic minority populations. For many years, researchers and clinicians have been misled by self-reported data. National surveys frequently publish self-reported data without adjusting for overreporting of actual mammography usage (31, 38–40). One of the Healthy People 2020 goals is to increase the proportion of women who have had a mammogram in the last 2 years to 84% (41). According to numbers from the BRFSS, which was used as a baseline estimate, 73.7% had already reported a recent mammogram (1). Stratifying these data by race reveals small disparities in reporting for the BRFSS survey. However, these small estimated disparities in self-reported mammography use are not consistent with the larger disparities in stage at diagnosis and mortality by race (42–44). Thus, although self-reported data from surveys suggest that use is similar between racial/ethnic groups, albeit slightly lower among some minority populations, overreporting among certain groups of women is masking an even lower rate of use. Research that relies on self-reported mammography usage data might be misleading due to these reporting differences. Therefore, studies of mammography behavior should not rely on self-reported mammography use data.

No potential conflicts of interest were disclosed.

Conception and design: K.L. Allgood, S. Whitman, A.M. Shah

Development of methodology: K.L. Allgood, S. Whitman, G. Vasquez-Jones, A.M. Shah

Acquisition of data (provided animals, acquired and managed patients, provided facilities, etc.): K.L. Allgood, G. Vasquez-Jones, A.M. Shah

Analysis and interpretation of data (e.g., statistical analysis, biostatistics, computational analysis): K.L. Allgood, G.H. Rauscher, S. Whitman

Writing, review, and/or revision of the manuscript: K.L. Allgood, G.H. Rauscher, S. Whitman, G. Vasquez-Jones, A.M. Shah

Administrative, technical, or material support (i.e., reporting or organizing data, constructing databases): K.L. Allgood, G. Vasquez-Jones

Study supervision: K.L. Allgood, S. Whitman, G. Vasquez-Jones, A.M. Shah

Other (overall project director during survey design, data collection, and preliminary data analysis phases of the study; reviewed two versions of the manuscript and provided comments): A.M. Shah

The authors thank the Avon Foundation for Women Breast Cancer Crusade for the support and generous funding; Marc Hulbert, PhD, Executive Director of the Avon Breast Cancer Crusade, for encouraging their team to gather these data. The authors thank Ana Rosa Garcia, Wanda Rodriguez, and Celevia Taylor (Helping Her Live Community Health Workers) for their immeasurable assistance in organizing the packets, entering, and cleaning the data for this project. Finally, they also thank the participants in the survey and especially those who agreed to allow the authors to view their medical record.

This work was supported by the Avon Foundation for Women Breast Cancer Crusade (grant numbers 05-2011-034 and 05-2007-004; to K.L. Allgood, S. Whitman, A.M. Shah, and G. Vasquez-Jones) and the Agency for Health Research and Quality (grant number 1 R01 HS018366-01A1; to G.H. Rauscher).

The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.