Reliability and Validity of a Self-Administered Food Frequency Questionnaire in a Chemoprevention Trial of Adenoma Recurrence¹

WBF³ has been prominently mentioned among the agents proposed for chemoprevention of colorectal cancer (1, 2, 3). However, it is plausible that the consumption of even a modest WBF supplement could, over the 3–4-year duration of a chemoprevention trial, alter a number of dietary practices and that these alterations could obscure the protective effects of WBF supplementation. Clearly, understanding of any impact of WBF on overall dietary practice depends on adequate dietary assessment.

Several dietary intervention trials have attempted to evaluate dietary intake by means of food frequency questionnaires. Because it is important to consider the degree to which such questionnaires can measure diet, it cannot simply be assumed that the food frequency instrument will adequately capture dietary practice. The interpretation of any study of diet and specific end points can be substantially enhanced by quantitative information on the validity and reproducibility of the method used to measure exposure in the target population. In general, diet records, with a small number of replicate measures per participant, with statistical correction for within-person variation, are used as a standard for this evaluation (4).

Given that individuals in chemoprevention trials remain active participants for 3 years or longer, reliability and validity studies of food frequency questionnaires are feasible in most of these settings. We conducted reproducibility and validity analyses of our AFFQ in our recently completed, randomized, Phase III chemoprevention trial testing the effects of WBF intervention on adenoma recurrence.

Details of the WBF trial have been described previously (5). Briefly, the WBF study is a double-blind, high versus low fiber, Phase III trial designed to measure the effects of WBF supplementation for 3 years on adenoma recurrence. Men and women ages 40–80 years, who had removal of one or more colorectal adenoma(s) of 3 mm or larger at colonoscopy within 3 months before study entry, were recruited from three clinical sites in the Phoenix metropolitan area. The AFFQ assessed dietary intake in the previous year and was administered at baseline and annually thereafter.

A total of 190 male and female participants eligible for inclusion in the WBF trial were invited to participate in the reproducibility and validity study by providing a baseline AFFQ and by collecting dietary records for 4 randomly selected days. A second AFFQ was mailed to the trial participants for completion before their annual visit as part of the WBF trial design. A total of 139 individuals had complete data for the 4-day diet records, baseline AFFQ, and year 1 AFFQ and were included in the final analyses.

Details of the development of the AFFQ have been described elsewhere (6); however, the validation of the current version of the questionnaire has not been reported. We used a modified version of the National Cancer Institute’s Health Habits and History Questionnaire food frequency component and the corresponding database (7) to develop the AFFQ. The modifications included the addition of commonly consumed Southwestern foods and increased detail in fiber-containing foods reflecting the focus of ongoing studies at our institution. The current version is a computer scannable, 113-item, semiquantitative food frequency questionnaire that asks about usual eating habits over the previous year. Frequency of consumption is grouped into seven categories ranging from rarely/never to >3 times/day. Participants were asked to report portion size as small, medium, or large. The questionnaire includes open-ended questions to identify specific types of cold cereals, fruits and vegetables not listed, and specific brands of multiple vitamins and individual supplements. Nutrient intakes were calculated from the AFFQ by multiplying the frequency of use by the nutrient composition for the portion size specified for each item. Age- and sex-specific portion size estimates are used in these calculations (8). Nutrient calculations for modifications and updates to the AFFQ are based on data from the United States Department of Agriculture’s Continuing Survey of Food Intake of Individuals, 1985–1986 (CSFII-86), and the United States Department of Agriculture’s National Food Consumption Survey, 1987–1988 (9). Participants were instructed to record portion and frequency of consumption of their WBF supplement in the AFFQ food item corresponding to 100% bran cereal. Therefore, because nutrient data at year 1 were analyzed in a blinded fashion, all participants are assumed to have consumed the high WBF cereal product.

Dietary records were collected during the run-in phase of the study, 4–6 weeks after the baseline AFFQ. These were collected on four nonconsecutive, randomly assigned days (2 weekdays and 2 weekend days). The project nutritionist trained the participants on the method of recording food and beverage consumption and portion size estimates. Additional points covered by the nutritionist included the complete description of foods and the recording of recipes or foods prepared by others. Once the records were turned in, they were reviewed by the nutrition staff members, and any clarifications were made via telephone. Recipes were obtained from participants, and the appropriate amounts of the component foods were entered for food items that were not in the database. The records were entered and analyzed using the Nutritionist III nutrient software package (version 6; N² Computing, Salem, OR). Additional constituents such as carotenoids, fatty acids, and specific fiber components were not available at the time of this study because they were not part of the nutrient database.

Means and SDs were calculated for total nutrient intakes using the AFFQs and the 4-day diet records. Because all nutrients, as measured by the AFFQ, were correlated with energy intake, variation due to energy intake and its associated measurement error was minimized by energy adjustment of the nutrients using the regression method (10). This method of energy adjustment is computed from the residuals of the regression model with total energy intake as the independent variable and the nutrient as the dependent variable. Because residuals do not provide an intuitive sense of nutrient intake, the mean energy intake of the study population was added to this value. Energy adjustment was conducted separately for each diet method and provides a value that is uncorrelated with total energy intake. Most nutrient distributions were skewed toward higher values; therefore, all variables were log (natural)-transformed before analysis. Intraclass correlations between intakes assessed by the two AFFQs were computed for unadjusted and energy-adjusted nutrients and alcohol. Pearson correlation coefficients were used to compare data from the AFFQ and diet records for both unadjusted and energy-adjusted nutrients. Spearman correlation coefficients did not differ appreciably from the parametric statistics presented in this study. Disattenuated correlation coefficients were also generated using diet record data to calculate within- and between-person variation (11, 12).

The participants in the present analysis included 97 males and 42 females with a mean age of 66.4 years. Ninety-seven percent of the participants were white, and 84% had 12 or more years of education. There were no significant differences in age, gender, education, or treatment group distribution between this study population and that of the total randomized participants in the WBF trial (5).

Table 1 presents the means and SDs for the average of the unadjusted and energy-adjusted nutrient intakes based on the diet records and the baseline and year 1 AFFQs for the 139 participants. When compared with the diet record data, a lower intake of macronutrients was observed based on the AFFQs, regardless of energy adjustment (except for carbohydrates at year 1). Using the diet record data as standard, there tended to be more overestimates when using the year 1 AFFQ nutrient intakes than when the baseline AFFQ data were used. Consumption of alcohol was underestimated 20–25% by the AFFQ when compared with the diet records. Of particular interest, dietary fiber (the active agent of the intervention) was slightly lower than the diet record estimate when using the baseline AFFQ. Not surprisingly, however, intake of dietary fiber was substantially higher in the year 1 AFFQ as compared with either the baseline AFFQ or diet records. As noted previously, these blinded analyses assume all participants were consuming the high-fiber supplement.

Table 2 presents the intraclass correlations for unadjusted and energy-adjusted nutrient intakes and alcohol intake between the average of the baseline and year 1 AFFQs. These comparisons indicate that the reproducibility of the baseline and year 1 AFFQs was moderately high, with a mean intraclass correlation for unadjusted nutrients of 0.54 (range, 0.33 for calcium to 0.64% of energy from fat). Energy adjustment did not improve the reliability estimates. For the energy-adjusted nutrients, the mean intraclass correlation was 0.48 (range, 0.28–0.66). A modest degree of reproducibility was shown for dietary fiber (r = 0.56 for unadjusted intake and 0.52 for energy-adjusted intake).

To assess the relative validity of the questionnaire, we compared unadjusted and energy-adjusted nutrient intakes estimated from each of the AFFQs with the average of the 4-day diet records (Table 3). Correlations between the AFFQ at baseline and the 4-day diet records were lower than those comparing year 1 AFFQ with the diet records using unadjusted nutrient data. The mean correlation coefficient using the baseline AFFQ was 0.28 for unadjusted nutrients (plus alcohol) and 0.38 for energy-adjusted nutrients. The corresponding mean values using the year 1 AFFQ were 0.36 and 0.36, respectively. The deattenuated correlations, reflecting the correction for the effects of within-person variation of the energy-adjusted nutrients (except energy and alcohol; Ref. 12 ), were the same for baseline and year 1 AFFQs. The deattenuated correlation coefficients ranged from 0.22 for energy to 0.65 for alcohol using the baseline questionnaire and from 0.25 for vitamin E to 0.67 for total fat and cholesterol using the year 1 questionnaire; the means of the two groups were identical. The corresponding results for dietary fiber indicate that the relative validity of this nutrient was similar whether using the baseline (r = 0.45) or the year 1 (r = 0.44) AFFQ.

We were also interested in whether combining data from two food frequency questionnaires would improve their relative validities when compared with the average of the 4-day records. As shown in Table 4, using the average of the two AFFQs, no appreciable improvements were made in the correlations using the unadjusted nutrients as compared with those of year 1 AFFQ and the diet records. However, improvements were made for the energy-adjusted data as well as the deattenuated correlations for the majority of the nutrients, including dietary fiber. A mean increase of 19% was observed for the correlations using the energy-adjusted nutrients, and a 14% increase was observed for the deattenuated correlations.

We compared the intakes of energy, 16 macro- and micronutrients, and alcohol estimated from a 113-item food frequency questionnaire, with intake calculated from the average of a set of 4-day dietary records collected over a 1-month period. After correcting for within-person variance assessed by the dietary records, correlations averaged 0.49 whether using the baseline or year 1 AFFQ. Of particular interest, we were able to show a modest degree of reproducibility and validity for dietary fiber, the target intervention in the chemoprevention trial. The nutrient intakes based on the baseline AFFQ were not substantially different from those of the randomized study participants (5). Therefore, it is unlikely that these results are biased, because reasons for nonparticipation in the validation study do not appear to be associated with major differences in diet or with the ability to complete a questionnaire.

The reproducibility of food frequency questionnaires similar to ours has been reported in various settings (13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23). The correlation coefficients from our reproducibility study of the AFFQ administered 1 year apart ranged from 0.33–0.64 for unadjusted nutrients; the correlation coefficient for alcohol was 0.81. These reproducibility coefficients, including that of fiber, are in accordance with the results of studies of food frequency questionnaires administered over a period of 1–10 years.

Validation studies of food frequency questionnaires that have used diet records as the comparison method have also been published previously (8, 13, 14, 15, 18, 20, 23, 24, 25, 26, 27, 29, 30, 31), but only a few have used food frequency questionnaires before and after diet record collection (13, 15). Correlation coefficients in these studies ranged from 0.00–0.24 for the study by Stuff et al. (20) to 0.69–0.94 for the study by Balogh et al. (31). Given the wide variation in design and methodology among the numerous validation studies published to date, it is difficult to directly compare the results of these studies with those of ours. Although the correlations we observed were on the lower end, they were within the overall range of those reported in the literature (see Ref. 4 for a complete review). In one of these studies (23), the reliability and validity of a food frequency questionnaire was assessed against separate sets of dietary records collected at baseline and at the 6-month follow-up among participants in the Women’s Health Trial. The results of this study showed increases in validity from baseline to 6 months, and this increase tended to be larger among women in the intervention group as compared to the control group. A challenge was presented in our design because the diet record collection period occurred before randomization. Although the year 1 AFFQ technically covered the period of time reflected by the diet records, its administration was approximately 11 months after the diet record collection period. Because of this, correlations similar to those of the baseline AFFQ were observed for the comparison of the latter AFFQ and the diet records. This is in contrast to what has been reported in the literature for validation studies not involving an intervention, with stronger correlations observed for the year 1 food frequency questionnaire (13, 15). Based on the mean correlation coefficients, energy adjustment of nutrients improved the overall correlations of the baseline AFFQ and diet records but had no overall effect when the year 1 AFFQ was used. Recent publications of cohort studies where food frequency questionnaires are collected throughout the follow-up period (32, 33) have used data from two to three questionnaires to assess consistency in dietary intake. The results of our analyses suggest that some improvements in the validity estimates are possible when combining data from two food frequency questionnaires.

While it is generally agreed that there is no perfect standard method with which to assess the validity of a food frequency questionnaire, validation studies compare this method with one that is judged to be superior: a series of diet records. Given that the diet records were collected close in time to the baseline AFFQ, it is possible that the process of filling out the AFFQ heightened the participant’s awareness and altered the recording of the diet records. This lack of independence would result in high correlated errors and an overestimate of validity. However, because the source of errors for the AFFQ is minimally shared with that of diet records (4), there is reason to believe that there is independence between the two methods. In addition, the collection of the year 1 AFFQ, which took place much later in time, provides an additional comparison. It is generally assumed that errors from diet records are uncorrelated with those of the food frequency questionnaire, although recent work by Spiegelman et al. (34) has raised questions about this assumption. Our data show that the disattenuated correlation of nutrient intake by food frequency questionnaire methodology with a gold standard is 0.49. If the frequency and record-based methods are both seen as imperfect indicators of diet, this correlation can be interpreted as indicating that approximately one-half of the observed variance in either indicator is due to error (35). If the true correlation of nutrient intake with an outcome was 0.5, and the nutrient was measured with half the variance as error, the observed correlation would be between 0.3 and 0.4 (36). Thus, the association of any nutrient intake with outcome would in all likelihood be underestimated. As well, the ability to look at change in diet is limited due to measurement error. Variance around the mean at baseline and follow-up is greatly increased by measurement error; this excess variance at these time points results in an underestimate of the significance of change in mean consumption.

An important consideration in a trial such as the WBF is whether the intervention caused a change in nutrient intake other than that targeted by the intervention. This change can be assessed by data from the year 1 AFFQ. Based on validation studies in which food frequency questionnaire data were collected at baseline and 1 year, change in nutrient intake occurs even in the absence of an intervention (13, 15). Results of these studies indicate that participants tended to report a lower intake of nutrients in the second administration of the food frequency questionnaire compared to the first administration of the food frequency questionnaire. Data from our AFFQs show lower intakes in only 5 of the 17 nutrients when comparing the year 1 AFFQ with the baseline AFFQ. Because the nutrient analyses are blinded (i.e., it was assumed that everyone consumed the high-fiber cereal product), the differences could reflect the overall impact of the WBF intervention. However, because we have no diet record data close to the year 1 AFFQ administration for this set of individuals, we cannot assess the relative validity of the reported changes. In studies such as the Women’s Health Initiative (31), where the intervention comprises a dietary pattern, monitoring of dietary change is an important component. However, in a trial such as ours, a WBF intervention has little effect on the intake of fiber from other sources (37). Furthermore, for sources of total fat, the WBF intervention resulted only in an increase in milk consumption and a decrease in table fat (i.e., butter, margarine, and salad dressing). Collection of dietary records should be conducted in dietary intervention studies where assessment of dietary change is important due to the intervention. Dietary intake based on these records could then be compared with the food frequency questionnaires to assess the relative validity of dietary change (31).

Reproducibility and validity analysis of food frequency questionnaires is quite feasible in the setting of a chemoprevention trial. Due to the make-up of the WBF trial participants (i.e., Caucasian, high socioeconomic status), the results of this study can be generalized to study populations such as ours. The degree of variation in nutrient intakes and the reproducibility and validity of our questionnaire indicate that important associations between diet and various end points can be reasonably quantified by food frequency instruments, subject to the strictures of measurement error, in chemoprevention trials.