Validity of Short Food Frequency Questionnaires Used in Cancer Chemoprevention Trials: Results from the Prostate Cancer Prevention Trial¹

Assessing diet in large, randomized chemoprevention trials is a significant challenge. Most dietary assessment instruments, including self-administered tools such as FFQs,³ are sufficiently burdensome to participants and clinic staff to potentially affect compliance with key parts of study protocol. However, in many trials, it is desirable to measure at least some aspects of the diet because nutrient intake may be a covariate in evaluating the study’s main outcome or it may be part of secondary hypotheses. Two large and important chemoprevention trials, the Breast Cancer Prevention Trial, evaluating tamoxifen (1), and the PCPT, evaluating finasteride (2), have used a short, 13-item dietary screener as a surrogate measure of percentage energy from fat (3). Although there remains considerable controversy about whether dietary fat is associated with either breast or prostate cancer, in both chemoprevention trials, the protective mechanism of the agents tested and the hypothesized risk from dietary fat are through hormone regulation. Thus, an important scientific question is whether there are differences in the effects of the chemopreventive agents between persons eating low- and high-fat diets. The ability of these large trials to address interactions of diet with treatment effect is, in part, dependent upon the validity of this short dietary screener.

The 13-item dietary screener was developed to identify groups of individuals with high (or low) fat intake by estimating intakes of fat and saturated fat in the few foods believed to contribute most to total fat intake (3). The screener has been used for population-level nutritional surveillance (4) and for epidemiological studies of specific risk factors for chronic diseases (5). The intent of the screener in these studies is as a surrogate for a more comprehensive dietary assessment, in which the relevant exposure measure is either percentage energy from fat or fat intake controlled for energy. In a 1989 study describing the use of the 13-item screener in a sample of well-educated, healthy women 45 years of age and older, the correlation coefficient between the screener and percentage energy from fat calculated from the mean of three 4-day diet records was 0.58 (3). In subsequent studies, correlations between the screener and more comprehensive dietary measures ranged from 0.02 to 0.60 (6, 7). These studies did not address whether the 13-item screener is a reasonable alternative for a comprehensive FFQ, the standard dietary assessment instrument for large epidemiological studies. This study examines the validity of the 13-item dietary screener as used in the PCPT. Results are based on nearly 11,000 men, ages 55–74, from 204 clinic sites throughout the United States. We use the results of these analyses to address a broader question of whether it scientifically reasonable to use this and similar short dietary instruments in large and important chemoprevention trials.

Data are from the PCPT, a multicenter, double-blinded, placebo-controlled trial of the drug finasteride (Proscar) for the primary prevention of prostate cancer. Eligibility criteria included: age of 55–70 years, serum prostate-specific antigen concentration of <3 mg/ml, normal digital rectal exam, no history of prostate cancer; no history of other cancer within the previous 5 years, and no history of medical problems that would interfere with study procedures. Recruitment began in 1993 and ended in 1997, after the randomization of 18,882 participants. All participants completed questionnaires on demographic characteristics, medical history, and health-related behaviors and provided blood and anthropometric measurements at study enrollment (3 months prerandomization).

In 1995, the PCPT Diet Ancillary Study was added to assess the association of diet, particularly the role of dietary fat, with the incidence of prostate cancer. The Diet Ancillary Study included a nested validity study to assess the validity and reliability of the FFQ in the study sample. Data for these analyses are from 10,913 men who had both baseline data and FFQs available for analysis as of November 1, 1997, as well as 146 men in the nested dietary validity study. Details of each dietary assessment instrument are given below.

At study enrollment, all participants completed the 13-item screener (3). In the original version of the screener, participants indicated frequency of consumption of the foods as “per day/week/month/year” or “rarely/never.” These response options were modified for PCPT and included nine categories ranging from “never or less than once a month” to “two or more times per day.” Participants recorded the frequency of consumption and portion size (small, medium, or large compared with the stated medium portion size) of: hamburgers, cheeseburgers, and meat loaf; beef steaks and roasts; pork, including chops and roast; hot dogs; ham and lunch meats; whole milk; cheese; eggs; cookies and pastries; white bread and rolls; margarine or butter; salad dressing and mayonnaise; and French fries. This instrument yields measures of total fat (g) and saturated fat (g), which were calculated using algorithms given in the original manuscript (3).

At the first annual visit, all participants completed a 109-item FFQ that asked about usual consumption of foods eaten over the past year. This FFQ, based on the instrument used in the Women’s Health Initiative, was designed to be sensitive to sources of fat and antioxidant micronutrients (8). Similar FFQs have been used in diverse populations, including racial and ethnic minorities and persons of lower socioeconomic status (9). The FFQ nutrient database is derived from the University of Minnesota Nutrition Coordinating Center nutrient database (10), and the algorithms for analysis are described elsewhere (11). These analyses excluded 276 (2%) FFQs with energy intakes of <800 kcal and >5000 kcal because we considered these unreliable. There were 10,913 participants with complete FFQ data available for analysis. Measures used from the FFQ were total fat (g), saturated fat (g), and percentage energy from fat and saturated fat.

As part of the nested validation study, a random sample of 155 participants completed unannounced, telephone-administered 24-h dietary recalls in the year following completion of the FFQ. Recalls were collected using the University of Minnesota Nutrition Data System (NDS, Version 2.9; University of Minnesota, Minneapolis, MN) software. Two of the recalls were completed on Monday to capture weekend eating patterns (12). These analyses include 146 participants who completed at least four of the six dietary recalls [6 (4%) completed four and 11 (7%) completed five]. Measures used from the 24-h recalls were total fat (g), saturated fat (g), and percentage energy from fat and saturated fat.

We used two approaches to evaluate the validity of the screener. In the first set of analyses, we used the repeat 24-h dietary recalls as the criterion measure of fat intake. We defined validity as the Pearson correlation of the criterion measures of fat with the corresponding measures from the screener and the FFQ. Total energy, fat (g), and saturated fat (g) were log-transformed for analysis and then back-transformed for ease of interpretation. Statistical tests comparing the validity of the screener with the FFQ used the method described by Steiger (13), which allows comparison of two nonindependent samples. In the second set of analyses, we examined associations of demographic characteristics (age and education) and health-related measures (smoking and BMI) with fat intake and compared results based on the screener to those based on the FFQ.

The mean age of the study sample was 63.5 years, 95% were white, and 53.1% had a college degree. Seven % of participants smoked, 25% were overweight (BMI = 27.8–31.0), and 16.1% were obese (BMI ≥ 31.1; Ref. 14).

Table 1 gives the mean intakes of energy, total and saturated fat, and percentage energy from total and saturated fat for all PCPT participants and for the 146 participants in the nested validity study. The mean total and saturated fat estimated from the screener were 31–37% of the corresponding measures from the dietary recalls and FFQs. Among the 146 participants in the validity study, the energy and fat intakes estimated by the FFQ were within 5% of the estimates from the dietary recalls.

Correlation coefficients between the screener and FFQ measures of fat in the entire sample (n = 10,913) were moderate. The correlations of total and saturated fat from the screener with percentage energy from total and saturated fat from the FFQ were 0.50 and 0.52, respectively (data not shown).

Table 2 gives correlation coefficients among measures of fat from the three dietary assessment instruments used in the validity study (n = 146). The criterion measures in these analyses are the mean nutrient estimates of the 24-h dietary recalls. Correlations between the criterion measures of total and saturated fat (g) and the screener were nearly identical to those between the criterion and the FFQ. In contrast, correlation coefficients between criterion measures of percentage energy from total and saturated fat were statistically significantly larger for the FFQ compared to those for the screener.

Table 3 gives associations of participant characteristics with fat intake, contrasting associations based on the screener to those based on the FFQ. The differences in the strengths of these associations are an additional assessment of the validity of the screener and FFQ measures of fat. As expected, results from both instruments show that sociodemographic and lifestyle characteristics are associated with fat consumption. For both the screener and the FFQ, increasing age and education were associated with lower fat intake, whereas smoking and increasing BMI were associated with higher fat intake. For education, smoking status and BMI, these associations were stronger with percentage energy from fat from the FFQ compared to the screener and reached statistical significance for education and BMI. Contrasts between the screener and the FFQ measure of total grams of fat were inconsistent. Correlations for education and smoking were significantly smaller for the FFQ than for the screener, whereas there were no statistically significant differences for age or BMI.

The purpose of this study was to investigate the measurement characteristics of a 13-item dietary screener used in large chemoprevention trials. This is an important issue because many of the null or modest findings in studies of diet and disease may be attributed in part to measurement error (15). The principal finding from this study is that the 13-item dietary screener is a significantly less valid measure of percentage energy from fat than is a comprehensive FFQ. Because many epidemiological investigations of diet and disease use percentage energy from fat to rank individuals or to select individuals for intervention (8), use of the less precise screener could misclassify participants and, thus, seriously alter interpretation of study results. A second important finding relates to the influence of demographic characteristics, such as age and education, on measures of fat intake. In many studies, these demographic variables are correlated with diet (9, 16, 17) and are treated as confounders in analyses of diet and disease. We found statistically significant differences in the magnitude of associations of demographic characteristics with fat intake between the FFQ and screener, suggesting that the screener does not perform as well as the FFQ across the demographic groups. Taken together, these findings suggest that use of the dietary screener as the sole dietary assessment instrument is not adequate for studies where diet may be an important exposure variable.

The original intention of the screener was to identify groups of individuals with a high mean percentage energy from fat (3). Although there were modest correlations of ≈0.50 between the criterion measures (repeat 24-h recalls) and the screener, these correlations were statistically significantly larger with the FFQ. The correlations between the FFQ and the dietary recalls were 0.71 and 0.72 for percentage energy from fat and saturated fat, respectively. The potential effects of these differences in validity between the screener and the FFQ are substantial and may have important implications for study outcome. Assuming nondifferential exposure measurement error, the observable relative risk or odds ratio is attenuated toward the null as a function of the square of the validity or correlation coefficient (18). For example, in the PCPT, if the true association of fat intake with prostate cancer was a relative risk of 2.5, the observed relative risk using an FFQ to measure fat intake would be 1.6, which would be further attenuated to 1.3 using the screener.

We found significant associations of participant characteristics with fat intake that we interpret as evidence for differences in food intake patterns across sociodemographic groups. However, the magnitudes and even the direction of some of these associations differed substantially between the screener and the FFQ. The likely reason for this observation is that the 13 items on the screener do not include all of the high-fat foods that contribute to differences in fat intake across some demographic groups. This is consistent with the findings of Coates et al. (4), who found that, among certain population subgroups, such as Hispanics, the screener did not perform well, as the limited number of items did not represent the largest contributors of the group’s fat intake (4). Thus, using the screener to control for fat intake in the analysis of a chemoprevention trial could introduce a bias because the measure would be differentially valid across demographic groups.

A larger question is whether abbreviated dietary assessment instruments such as the 13-item screener should be used in large randomized trials where diet may be an important covariate. In addition to their lower validity and potential bias (as discussed above), there are reasons that these short questionnaires are not appropriate measures for use in either population-level nutritional surveillance or in epidemiological studies, including randomized trials. (a) Short instruments such as the screener cannot be adjusted for total energy intake. Most scientists believe these adjustments are necessary because consumption of many nutrients, such as fat, are strongly correlated with total energy intake. Failure to adjust for total energy intake can confound study findings if energy itself is a risk factor for disease or if the variability in nutrient consumption across energy intake levels introduces a bias in analysis (19). The use of energy-adjusted nutrients is particularly important when etiological hypotheses are tested (19). (b) The screener is a list of only 13 foods chosen because of their high contribution to total fat intake of the United States population (3), yet in our study, these foods captured <40% of total fat in the diet. This method of selecting a food list may, therefore, yield low between-person variation, and thus, results generated from such an instrument would be limited in the ability to rank or discriminate individuals (19, 20, 21). For example, Caan et al. (7) found that the 13-item screener performed poorly when used to rank individuals by percentage energy from fat. On the other hand, Martin et al. (22) used a full FFQ very similar to the one used in this study and found that the comprehensive instrument was able to accurately discriminate participants with low and high fat intakes. (c) The screener contains none of the newly available fat-modified foods, and failure to capture this information results in a bias in estimates of energy and fat intake (23). (d) Analyses based on screener results are limited to one nutrient only, effectively eliminating potentially important analyses of either groups of nutrients, such as carotenoids, or total diet with disease.

A limitation to our study is the fact that participants were male, older, mostly white, and well educated. This restricted sample reduces our ability to generalize our study findings to other populations.

The use of short dietary screeners as a sole assessment instrument will result in significant loss of information regarding the primary exposure (e.g., fat intake) and loss of precious scientific opportunity to enhance our knowledge regarding other dietary factors and cancer risk. We conclude that every effort should be made to use comprehensive measures of diet in large cancer prevention trials.