The Effect of Multiple Mailings on Recruitment: The Multiethnic Cohort

Mail survey methodology is appealing in epidemiologic research for reasons of cost and convenience. However, sampling error, adequacy of the sampling frame, measurement inaccuracies, and nonresponse must be considered for a well-designed postal study to be generalizable to the population of interest (1). Several recent epidemiology reports have focused on the issue of nonresponse error, which occurs when sampled persons who do not respond to a survey differ from those who do. Randomized trials and surveillance studies have investigated recruitment method strategies to increase response rates, a standard approach to reducing nonresponse error via reducing the number of all persons who do not respond. These strategies have included providing a pen/pencil, financial incentives, use of a culturally sensitive letter, and varying survey length (2-7). In contrast, reports of post hoc analyses compared respondents to nonrespondents and those who responded early or later to recruitment efforts (7-21), some of which make inferences about bias and the value of extensive recruitment methods. There have also been simulations to show how assumptions about nonrespondents or late respondents can potentially affect prevalence estimates for disease or exposure (22, 23).

In the current study, we investigated differences between early and late respondents to a self-administered mail survey of lifestyle exposures that may increase the risk of cancer and other disease in the Multiethnic Cohort (MEC) using all measures available from the survey. Five specific ethnic groups were targeted for recruitment, which provides a unique opportunity to learn more about ethnic differences in recruitment efforts. The main findings presented here concern early versus late response based on eventual participants. Although it would be ideal to make comparisons with nonrespondents, presumed ethnicity is the only variable available for comparison. In addition, we examined differences between younger and retirement-aged adults and show how response completeness may be affected by repeatedly mailing questionnaires to increase the response rate.

The MEC is a prospective study that was initiated to investigate lifestyle exposures in relation to disease outcomes, especially cancer. Recruitment took place from 1993 to 1996 in Hawaii and Los Angeles, California, and targeted five ethnic/racial groups: African-American, Native Hawaiian, Japanese-American, Latino, and White. The sampling frame was the driver's license files of the state of Hawaii and of the county of Los Angeles, supplemented by the voter's registration lists in Hawaii, to ensure that older Japanese who do not drive were adequately covered and the Health Care Financing Administration member list in California to increase the number of older African-American men in the sampling frame. Sex is available from these sources, but detailed race/ethnicity is not. Only African-Americans were recruited from the Health Care Financing Administration list based on their racial designation of white, black, and other. Some Native Hawaiians were identified based on voter's registration information that listed eligibility to vote in elections only open to Native Hawaiians. Others in the sampling frame were assigned a presumed race/ethnic group based on names and residence, as follows. Last name lists were compiled from data from past epidemiology studies and public sources for the following groups: Japanese, Chinese, Korean, Filipino, Hawaiian, Samoan, and Latino; first name lists were compiled from public sources for Japanese and Hawaiians. The last names of individuals in the sampling frame were compared against the name lists and assigned race/ethnicity if there was a match. The first names of individuals not assigned Japanese or Native Hawaiian were compared against the first name lists; this step was to further identify Japanese and Native Hawaiian women with other married names. Remaining individuals were assigned a presumed race of white or African-American. In Hawaii, these individuals were assigned white, and in Los Angeles, they were assigned African-American when they lived in a census tract that was >50% black in the 1990 census, or otherwise white. Based on these steps, the mailings were targeted to the five ethnic/racial groups. A tracking database, including estimated ethnicity, was used to record recruitment efforts and final status for all persons in the sample.

The MEC design and recruitment methods have been described in detail elsewhere (24). In brief, >215,000 men and women ages 45 to 75 years became participants by completing a 26-page, self-administered survey received through the U.S. mail, with a cover letter describing the study. This survey is referred to as the baseline questionnaire. Both Spanish and English versions of the survey were sent to persons in Los Angeles with Latino surnames. The surveys were sent in seven batches to ∼100,000 persons each, spaced ∼6 months apart. Nonrespondents were sent the same questionnaire up to two additional times over a period of 3 to 7 months. The mailings to the nonrespondents occurred when the number of incoming completed surveys dropped substantially but were sometimes delayed to avoid sending questionnaires during holidays. The repeated postings within batch are referred to in this article as mailings, and the corresponding response to the mailing is referred to as the wave of response. In the present analysis of early versus late response, we excluded 6% of participants who were from batches that were mailed near the end of the initial cohort recruitment, as they were given less opportunity to respond, being sent a maximum of two surveys. These batches targeted Native Hawaiians and Latinos, which suggests their overall response rates would have been somewhat higher had they been given the same opportunity to respond. An additional <1% of participants were excluded because their surveys were received after an extended delay of >5 months after the third survey was mailed, suggesting extenuating circumstances in locating the participants or in their readiness to respond. The present analysis of response wave is based on a total of 201,461 cohort members. University of Hawaii and University of Southern California Institutional Review Boards reviewed and approved all MEC study protocols.

Follow-up Survey. A four-page follow-up survey was mailed to the cohort participants ∼5 years later in years 1999 to 2000. The follow-up survey was mailed up to four times (i.e., initial survey and three repeats) and phone contact was also used to maximize the response rate.

We wish to explore differences between individuals who required more or less effort to recruit. Therefore, the key variable of interest in this study is the wave number in which the baseline questionnaire was returned. The baseline questionnaires were not tracked with a mailing number, and precise mailing dates were difficult to assess due to the bulk mailing procedures used by the U.S. postal service. Therefore, we chose to define wave number by the distribution of surveys returned over time, within batches and location. Figure 1 shows the baseline questionnaire receipt date for one batch of Hawaii participants and illustrates the substantial increases that suggest response to a follow-up mailing. The cutoff dates used in determining wave number were independently determined by two of the authors with 100% agreement. These cutoff dates were also consistent with approximate dates that nonrespondents were mailed a duplicate survey. This procedure was followed for each batch by location.

For the follow-up survey, we defined wave number based on return dates in a similar manner to the baseline questionnaire, but we also had the number of surveys mailed in a tracking database. In addition, for the follow-up questionnaire, the questionnaire was administered by phone, when possible, to individuals who did not respond to a third mailing, and a fourth mailing was sent to individuals who had not responded and could not be contacted by phone. For some analyses, these individuals were assigned a wave of 3. We validated our approach for defining wave number by comparing the wave number assigned for the follow-up survey with the number of questionnaires mailed and found excellent agreement (κ = 0.93).

Information on demographics, history of prior medical conditions, reproductive history for women, lifestyle behaviors (e.g., diet, smoking, and physical activity), and medication and vitamin use was asked on the initial survey, and these items were considered in this analysis as potential determinants of response behavior. The specific variables included in this analysis are listed in Table 2. Family history of cancer asked about parents' or siblings' incidence of breast, ovarian, prostate, colon, and “other” cancers. Individuals were asked if they had ever been informed by their doctors that they had any of a list of medical conditions. Dietary information was assessed via a validated food frequency questionnaire, including >180 food items, developed for this multiethnic population. Daily intake of dietary components was computed as described previously (24, 25). Body mass index (BMI) was calculated as self-reported weight in kilograms divided by self-reported height in meters, squared. Medication history asked for regular use, defined as two times per week for 1 month or longer, currently or in the past. Vitamin and mineral use was recorded if taken at least once per week during the last year; use was asked for multivitamin/minerals and selected single supplements. Women were asked if they currently use or ever used estrogen or progesterone for menopause or other reasons.

All analyses were conducted using Statistical Analysis System 9.1 (26). Logistic regression modeling wave number was the primary analysis tool. Initially, polytomous logistic models were used on an outcome variable with three levels for wave, but the findings for the second and third waves were generally consistent. Therefore, the final outcome variable collapsed the wave numbers 2 and 3 and was modeled using a dichotomous logistic model. Odds ratios (OR) and 95% confidence intervals were computed. Independent variables were tested for a bivariate relation to wave number using logistic regression. Two variables, current regular use of pain reliever other than aspirin and hypertension medication use, had no bivariate association with wave and were excluded from further analyses. All remaining variables were entered into a logistic regression model predicting response in the later waves, with separate models stratified by sex, sex-age group (under 65 years, 65+ years), and sex-ethnic group.

The pattern of response to the baseline questionnaire, defined by wave number, was compared with the wave number for the follow-up survey to test the notion that past behavior predicts future behavior. The χ² statistic for association was used for statistical comparison. In addition, the sex-specific logistic models were repeated for the follow-up survey using covariates from the baseline questionnaire.

Finally, we determined whether the extent of missing data on select variables increased with wave of response. Diet was considered missing if the calories computed from responses were implausible as described elsewhere (27). The Mantel-Haenszel χ² statistic (26) was used to test the trend in the percent missing over wave; in addition, logistic models were done regressing the missing status on wave of response (initial or later), controlling for age, sex, ethnicity, and language version of the questionnaire.

The response rate for the eligible sample of the five targeted ethnic groups was 25.6%, including in the denominator 15% of the sample for whom mailings were undeliverable. Table 1 shows the response and undeliverable rate by group based on the estimated ethnicity/race, which ranged from 17% for Latinos to 45% for Japanese, including undeliverable surveys. The undeliverable rate, which means the address was not valid and a forwarding address was not available, varied by ethnic group as well. These calculations exclude persons found to be ineligible for various reasons, including age, place of residence, and being deceased, the prevalences of which range from 0.4% to 3%. Undelivered surveys may have included some ineligible persons but this could not be determined. The accuracy of the estimated ethnic/racial group was assessed based on cohort participants' self-reported ethnicity. In general, there was reasonable agreement (80%) between the presumed and self-reported categories in the targeted age group, especially for African-Americans (86%), Native Hawaiians (86%), and Japanese-Americans (94%); as expected, agreement was better for men than for women, likely due to their married names not matching their ethnicity. The brief follow-up survey sent to cohort members 5 years later was completed by 78% of cohort members; 9% refused, 7% were deceased, and 6% were unreachable.

Sample characteristics for participants in this analysis are summarized in Table 2. Approximately 65% of the total participant sample had responded in the first wave, 23% in the second, and 12% in the third; males were ∼9% more likely than females to respond later (OR, 1.09; 95% confidence interval, 1.07-1.11). Nineteen percent of participants were born outside of the United States, and 36% of Latinos completed a Spanish version questionnaire. Although the mean age at questionnaire completion was 60 (SD, 8.9) years, we divided the sample into age groups of under 65 (64%) and 65+ years (36%) to investigate whether older adults, who are less likely to be working, would have a different pattern of response.

Table 3 shows model results predicting later response (i.e., in the second or third wave) for females and males, respectively. The results were generally consistent across age groups within sex (data not shown). Ethnicity was among the strongest effects, based on the magnitude of the ORs, with all nonwhite groups more likely to respond later compared with whites. Although Japanese were most likely to participate (response rate, 45%), they were 11% to 16% more likely to respond in a later wave compared with whites. Latinos, who were least likely to participate (response rate, 17%), had a likelihood of responding later, if they did participate, comparable with Japanese (female OR, 1.18; male OR, 1.06). However, persons responding to Spanish language surveys or who were foreign born were more likely to respond later, thus attenuating the effect of ethnicity for Latinos (female OR, 1.29; male OR, 1.15, when place of birth and language of survey were excluded from the models). In fact, Latinos and African-Americans had nearly identical patterns of response (later wave: Latino, 42%; African-American, 41%). African-Americans (response rate, 21%) had the highest ORs for the ethnicity effect (female OR, 1.57; male OR, 1.38), followed by Native Hawaiians (female OR, 1.32; male OR, 1.27; response rate, 34%), when compared with whites.

Persons with more years of education were more likely to respond in the initial wave, with a decrease in risk of a later response of 4% to 5% per year of education. Hawaii participants were more likely to have responded in the initial wave than participants from the Los Angeles area; however, there was little overlap in ethnic groups across the two locations. Almost all Latinos and African-Americans were recruited from Los Angeles. Sex differences were noted in the effects for marital status. Married men were somewhat more likely to respond later to the questionnaire, especially for working age men. Marital status was not related to later response for women.

Among health history variables, personal and family history of cancer were associated with response in the initial wave. Other medical conditions showed small but significant associations to questionnaire response time. Having asthma or other allergies was associated with response in the initial wave. Other chronic diseases, such as cardiovascular, digestive disease, and hypertension, were associated with response in select subgroups only (data not shown). BMI was not related to later response.

Among behavioral factors, energy intake was not related to wave of response, but getting more exercise per week was related to response in the first wave. However, this was a small difference, with initial responders reporting a mean of 1.2 h per week of moderate and vigorous activity and later responders reporting 1.1 h. For all males and working age females, former smokers were more likely to respond in the initial wave than those who never smoked, whereas current smokers were more likely to respond in a later wave for all males and for older females (data not shown).

Medication and supplement use showed some small associations, most in the direction of increasing the likelihood of response in the initial wave. Regular current use of aspirin was significant for all males as well as older females (data not shown). Antihistamines, antacids, and laxative use combined into a category of over-the-counter remedies was significant for males, particularly in the older group. All females showed a small effect for hormone replacement (estrogen and/or progesterone).

The use of up to 26 variables in the multivariate models resulted in a not insignificant number of individuals with missing values (12.2%). To determine if the missing data resulted in a biased sample, we compared two models containing only the demographic independent variables: one including participants with no missing data on the eight demographic variables (96.6% of the sample) to one excluding participants having missing data on any of the 26 model variables (87.8%). The models, analyzed by sex, showed consistent findings, suggesting that no bias was introduced due to missing data.

Subgroup analyses were conducted using all of the same variables (except ethnicity) within each of the five targeted ethnic groups by sex. Few differences were found compared with the overall models.

Comparison of these results with those from the follow-up questionnaire showed a significant relation between baseline and follow-up wave of response [χ²(10) = 6809.9; P < 0.0001; Table 4]. For each wave of baseline response, the highest percentage of persons responded in the initial wave to the brief follow-up survey. However, there were proportionally more wave 1 baseline respondents who also responded in the first wave of follow-up. Similarly, wave 3 baseline respondents were proportionally more likely to respond in wave 3 follow-up, as well as be unreachable or refuse to participate. The logistic models for later response to the follow-up questionnaire showed similar results to the baseline analysis. Most differences were in the same direction, although the level of significance varied.

Table 5 shows the percentage of missing data across mailing wave for seven variables used in baseline modeling and shows that the percent missing generally increased with each wave of response. Across the seven variables, 12.8% of wave 3 respondents had at least one missing variable. Each variable was also tested controlling for ethnicity, Spanish language survey, age, and sex in a logistic model; those who responded later compared with those responding to the initial mailing were at 27% to 68% higher risk of having missing data. Within ethnic groups, differences were generally in the same direction but sometimes attenuated (data not shown).

We have shown that there are substantial associations between ethnicity and wave of response to a 26-page self-administered mail questionnaire, with nonwhite ethnic/racial groups more likely to respond to a later mailing of the questionnaire compared with whites. Japanese respondents had a higher response rate than whites but were more likely to respond later. Native Hawaiians were also likely to respond later, although their response rate was better than average for this sample. African-Americans and Latinos had lower response rates and were substantially more likely to respond in a later wave. Relatedly, use of the Spanish language questionnaire and being foreign born showed similar associations. Not surprisingly, more education was related to earlier response. In addition, a personal or family history of cancer, having asthma, regular use of aspirin, and engaging in more physical activity were associated with early response, whereas being a current smoker was associated with later response. We also show that most associations are present within ethnic groups and also apply to a follow-up questionnaire. Although the three mailings of questionnaires were costly, it was necessary for recruiting ethnic minorities and those with less healthy lifestyles. However, each wave of response was associated with more missing values, which effectively reduces the number of cases that could be used for multivariate analyses.

These findings are generally similar to findings reported in the literature. Certainly, people have shown education and/or other proxies for socioeconomic level were inversely related to timing of response (9, 10, 13, 17, 18, 28). The association of current smoking with late response has also been reported (9, 10, 15, 17, 29). Several studies noted more effort was needed to recruit Hispanic and African-American participants (20, 30-32). Some studies (12, 16, 20) reported findings that suggest personal salience of the topic was related to early response. Our early responders showed more personal experience with cancer for a family member and/or self, whereas other diseases had mixed associations with response wave. Among several studies in the literature, there are notable differences in the timing of mailings, use of incentives, survey topic, and the burden of completing the questionnaire. The populations have been from several different countries, in specific age groups and sexes, and even included participants that were recruited previously for other research. Given these differences, common findings may be considered robust.

As important as these common findings is the context in which they are presented. Some have stated that repeated mailings or extra recruitment efforts did not affect prevalence estimates (e.g., chronic pain) or remove differences between responders and nonresponders (12, 16, 21, 29). Our findings suggest a meaningful difference exists for repeated mailings for nonwhite ethnic groups. For example, the third wave had 12.3% of respondents overall, yet 15.8% of Latinos and 14.7% of African-Americans. In contrast to ethnicity, wave had less practical importance for the recruitment of current smokers; 13.2% were recruited in wave 3. Similarly, the physical activity differences were nominal, and energy intake and BMI did not differ. The purpose of the MEC study is to investigate associations between lifestyle and cancer and other diseases among different ethnic groups. The definitive test will be if there are differences in the lifestyle-disease associations between early and late responders, which we will examine when more follow-up data are available.

Some researchers conducting similar analyses have failed to support a “continuum of resistance” model (16, 33), which posits that initial responders are on a continuum with later responders and nonresponders, thus providing the basis for using late responders to assess potential nonresponse bias. It is certain that persons who respond later after repeated solicitations would be nonresponders if recruitment efforts were less rigorous, for any given cross-sectional sample. Our results for the follow-up questionnaire partially support this concept because those persons most difficult to recruit at baseline were more likely to refuse, be unreachable, or respond later on follow-up. However, 31% of late responders at baseline responded to the initial survey at follow-up.

As a practical matter, identifying potential participants for the four nonwhite ethnic groups was challenging, and it was necessary to recruit as many as possible within our lists given the constraints of money and time. Studies that have advocated for less rigorous recruitment efforts (12, 21, 29) had better initial (41-53%) and final (58-83%) mail response rates than we obtained. One of these studies sampled a more homogeneous population of retirement community residents (29) and another sampled previous research volunteers (12). Our response rate was 25.6%, including undeliverable addresses in the denominator (30.2% excluding them), to a study that required participants to complete a 26-page self-administered survey and asked for them to join a longitudinal cohort study. Our response rate for African-Americans (21%) was consistent with another cancer prevention surveillance study that reported a 17.5% response for African-Americans (32). If potential participants are abundant for an ethnic group of interest, and depending on the research question, it may be advantageous to attempt recruitment with new individuals rather than resending questionnaires. In our sample, initial responders did not differ from later responders on energy intake and BMI and differed little on physical activity and smoking. For all ethnic groups, the first mailing yielded the highest response rate and fewer resources were needed for follow-up.

One recent simulation study (23) suggested that late respondents were likely to have error in their responses. Using missing data, our results support that claim. We found that response waves had progressively more missing data even after adjusting for age, sex, language of questionnaire, and ethnicity. The percent missing never became unacceptable in any of the three mailings. However, for multivariate analyses, cases excluded due to missing data may affect generalizability.

A limitation of the current study was that we had little data for comparing respondents with nonrespondents, necessary to truly assess nonresponse bias. We had nonresponse data for the follow-up questionnaire, but that comparison occurred in a self-selected sample of previous participants. However, our cohort participants were similar to the U.S. Census data for Hawaii and Los Angeles on education and marital status, suggesting that the sample was representative (24). Our results are unlikely to apply to all studies, as our response wave definition was data driven and our mailing dates were not at predetermined time intervals as many in other studies were.

In conclusion, our recommendation is to pursue the difficult-to-recruit participants through repeated contacts, especially when sample availability is limited. The increased response rate may serve to lessen nonresponse bias; however, missing data may be more problematic when less motivated participants are finally recruited. To compensate, we suggest the use of adjunct measures to calibrate and/or validate measurements on subsamples using more intensive data collection protocols as was previously done for dietary data (25).