A nine-membered panel of experts was asked to determine expert opinions of mortality risks associated with use of low-nitrosamine smokeless tobacco (LN-SLT) marketed for oral use. A modified Delphi approach was employed. For total mortality, the estimated median relative risks for individual users of LN-SLT were 9% and 5% of the risk associated with smoking for those ages 35 to 49 and ≥50 years, respectively. Median mortality risks relative to smoking were estimated to be 2% to 3% for lung cancer, 10% for heart disease, and 15% to 30% for oral cancer. Although individual estimates often varied between 0% and 50%, most panel members were confident or very confident of their estimates by the last round of consultation. In comparison with smoking, experts perceive at least a 90% reduction in the relative risk of LN-SLT use. The risks of using LN-SLT products therefore should not be portrayed as comparable with those of smoking cigarettes as has been the practice of some governmental and public health authorities in the past. Importantly, the overall public health impact of LN-SLT will reflect use patterns, its marketing, and governmental regulation of tobacco products.

In 2001, a group of experts impaneled by the Institute of Medicine laid the foundation for tobacco-related research assessing products that might reduce the risks associated with using tobacco (1). The Institute of Medicine committee termed these sorts of products as potentially reduced exposure products. One example of these products is low-nitrosamine smokeless tobacco (LN-SLT) marketed for oral use, such as Swedish snus or Ariva Cigalets (2).

One might expect that the health risks of a noncombustible, LN-SLT product would be substantially lower than that of a combustible, higher nitrosamine product. Whereas much professional debate focuses on the circumstances under which consumers use LN-SLT and how it is to be regulated, controversy regarding the health issues is still indicated by (1) recent commentaries in the scientific literature that have presented different views on the merits of marketing smokeless tobacco products as less harmful alternatives to cigarette smoking (3-10) and (2) conflicting views from public health agencies and officials about the health risks posed by smokeless tobacco compared with cigarettes (11-14). Interestingly, a recent survey of smokers in the United States found that 90% held the belief that the cancer risk from using chewing tobacco was equal to that from cigarettes (15). Some countries have banned smokeless tobacco products, with the perception that, like smoking cigarettes, they pose great danger to health (16). Meanwhile, smokeless tobacco manufacturers have recently pushed government regulators to officially endorse claims about the lower relative health risks compared with conventional cigarettes (17).

The objective of this study was to examine the available health literature on the risks of LN-SLT products and to determine if, by exposing a group of experienced tobacco epidemiologists (our Health Panel) to this information, a consensus could be reached regarding the central questions: whether LN-SLT products are less hazardous compared with conventional cigarettes and, if so, by how much. The Health Panel was asked to estimate the relative risks (RR) associated with the use of LN-SLT compared with cigarette smoking. This article describes the results.

Specification of the Problem

In evaluating the health risks of LN-SLT, we recognized that the data provided by individual studies were open to interpretation. In particular, questions are raised regarding the applicability of the studies to different environments [e.g., translation of the Swedish snus experience (8) to the other countries], the range of appropriate studies to consider (e.g., the use of smokeless tobacco in the United States), and the relevant alternatives (how the risks compare with no tobacco/nicotine use and with smoking traditional cigarettes). Given the degree of uncertainty surrounding the RR estimates and their application, we selected the Delphi method for risk estimation (18-23). The Delphi process has been applied in numerous fields including health care and health policy since its development in the 1950s (24-27). It involves eliciting informed estimates from a group of experts, sharing those estimates with the group, and in an iterative process trying to reach consensus.

The limited availability of information defined the scope of our study; with few studies on morbidity available, the Health Panel's task was limited to examining mortality risks. To help define the elements of risk, we narrowed the health risks of interest to four: (1) premature total mortality and mortality attributable to (2) lung cancer, (3) heart disease, and (4) oral cancer. Each of these outcomes would be associated with a distinct risk for ages 35 to 49 and ≥50 years to distinguish possible differences by age. The Health Panel was not asked to distinguish risks by gender or to evaluate potential reductions in mortality for persons other than the individual tobacco user (e.g., reduced exposure to secondhand smoke).

The Health Panel would be charged with assessing the mortality risk associated with long-term use of LN-SLT relative to two alternatives: no use of tobacco and long-term smoking of conventional cigarettes. Lacking data, we were unable to examine the risks associated with switching tobacco products (e.g., from cigarettes to LN-SLT) nor the concurrent use of cigarettes and a smokeless tobacco product.

Specification of the Methodology

The Delphi process was conducted through e-mail. Drawing on the collective expertise of the authors, a list of 16 potential panel members was developed. The experts were selected first based on their knowledge of the health risks associated with both smokeless tobacco and cigarette use. From that pool of scientists, we purposefully excluded those who, while also experts, might be too strongly associated with a particular viewpoint (in terms of either their support or rejection of LN-SLT as an alternative to smoking). Based on financial constraints and the requirements for expertise, the research team decided to recruit nine experts. Initially, letters were sent via e-mail to nine candidates prioritized by their expertise in the field and the need to maintain a balanced set of views. When a potential candidate declined to participate or did not respond in a timely fashion, he/she were replaced by a candidate chosen to maintain a balanced set of views. A total of 14 letters were sent, successfully obtaining the agreement of nine experts (three from outside the United States) to serve on the panel (Table 1).

During the research team's initial contacts with prospective Health Panel members, we told them that the purpose of the project was to draw on expert opinion to estimate the relative mortality risks of use of LN-SLT compared with cigarette smoking. Further, we informed the prospective panelists that these estimates would be used by the authors in a simulation analysis of the effects of a policy encouraging inveterate smokers to switch to LN-SLT. That study is ongoing as of this writing. Health Panel members were also aware of the research team's intent to submit a paper for publication describing the Delphi process and its results.

Health Panel members were informed that they would be identified as a member of the panel, but personal identifiers regarding estimates and comments would be removed from all transmission of estimates to fellow panel members and from all output of the study. Panel members committed to participate through several rounds of a Delphi process to reach a general consensus and were paid $500 on completion of the process. A copy of the initial contact letter is attached as Appendix A.

We distributed a list of published articles to Health Panel members to provide some common base of knowledge. The original list was based on a review of the literature developed from a search of Medline and Centers of Disease Control and Prevention's Office of Smoking and Health database and reviews of the reference lists of published articles and unpublished manuscripts. We included articles regarding the health risks of smokeless tobacco use in the United States as well as snus use in Sweden. Studies of smokeless tobacco use in other countries were considered less relevant. Once committed to participating, panel members were asked to review the list of references and to suggest additional articles; we added to the reference list those suggested articles for which the study focus was on health risks (rather than primarily behavioral aspects) associated with smokeless tobacco use. A package containing hard copies of each article on the final list was mailed to all panel members for ease of reference. The final list of articles is provided in Appendix B. The full list of collaborating Health Panel members was also provided to other members of the panel at the outset.

In round 1, Health Panel members were provided with the following background assumptions (see Appendix C):

  • Assume an open-ended scale, with the absence of riskequal to 0 and the risk of premature mortality associated with lifelong conventional cigarette smoking equal to 100. A risk for smokeless products greater than cigarettes thus would imply a risk equal to >100.

  • Assume that the smokeless product user uses only a LN-SLT product (e.g., Ariva and Swedish snus).

  • Assume that the dose each individual consumes represents “typical use” for that product.

  • Assume that the prototypical current smoker of conventional cigarettes and the user of smokeless tobacco do not switch to the other product nor do they quit use.

  • Assume the current smoker of conventional cigarettes and the user of LN-SLT products started regular or established use at age 17 years.

We recognized from the outset that, given the incomplete data available and the inability to predict behavioral patterns with precision, it would be useful for each panel member to provide some indication of relative confidence in his or her own estimate. Panel members were directed to indicate with each estimate whether they were very confident, confident, or not very confident of their response. These qualitative descriptors contributed to panel members' substantive debate of estimates in subsequent rounds.

In subsequent rounds, the research team provided a summary of panel members' responses to the full panel for review and reassessment (with personal identifiers removed). In addition, the research team submitted questions to the panel members to help determine areas of agreement and disagreement. For each round, panel members were asked to respond within 1 week, although in practice each round actually took 2 to 3 weeks.

All nine panelists participated in each of the rounds. Here, we focus on the final estimates produced in round 3 as well as the changes that took place to reach the final estimates (results from rounds 1 and 2 are reported in Appendix D). Except for one panel member's responses, the RR estimates of the Health Panel members in round 3 changed little from round 2. Likewise, although the levels of confidence attributed to their estimates increased in round 2 compared with round 1, reported confidence changed little between rounds 2 and 3. Consequently, we decided to conclude the process with round 3, the results of which are reported in Table 2.

For total mortality, the mean (median, lowest estimate to highest estimate) RRs for individuals ages 35 to 49 and ≥50 years were 18.1% (9%, 0-75%) and 10.1% (5%, 1-25%), respectively. One panel member began with and continued to submit estimates higher than all others, albeit with large changes by round 3. This panel member provided no comments to justify his/her response, even after panel co-members had criticized the estimates. Under these circumstances, we would argue that the “extreme” estimate from that one panel member could be considered an outlier and could be dropped. In Table 2 (column 9), we present adjusted means that exclude the highest estimate for each category. The adjusted means are generally lower than the original means. In the ≥50 years age group, the adjusted mean moves closer to the median for total mortality. Because of the highly skewed nature of the distributions and the weight given to extreme observations by the mean, we view the median estimate as the better summary measure.

For lung cancer, the median RRs were 2.0% and 3.4% for individuals ages 35 to 49 and ≥50 years, respectively. For heart disease, the median RR for both age groups was 10%. For oral cancer, the median RRs were 15% and 30% for individuals ages 35 to 49 and ≥50 years, respectively. Thus, estimated risks were highest for oral cancer and lowest for lung cancer, with heart disease risks slightly lower than total risks. The estimated risks for lung cancer, heart disease, and oral cancer generally ranged from 0% or 1% to 50%, with narrower ranges for lung cancer and heart disease risks for those ages ≥50 years. When the highest estimate was omitted, lung cancer risks varied between 0% and 10% for those ages 35 to 49 years and between 0% and 6% for those ages ≥50 years.

After rounds 1 and 2, we (as moderators) prodded panel members to justify their responses on age-related differences in health risks. However, the panel members did not reach a consensus on the relationship of age and RR. The majority submitted higher estimates for the younger age group, citing the strong impact of smoking duration on older smokers' risk. Panel members who estimated higher risks for older LN-SLT users provided few comments, although one panel member commented that the American Cancer Society's Cancer Prevention Study II data did not bear out the assertion that RRs associated with smokeless tobacco should drop with age. The lack of consensus is reflected in a comparison of mean responses between rounds 3 and 1: estimates for total mortality and heart disease declined more for ages ≥50 years than for the younger age group (in fact, the estimate increased for heart disease for younger individuals over the three rounds), but there were similar declines in both age groups for oral cancer and a greater decline in lung cancer estimates for ages 35 to 49 years (compared with a small increase for lung cancer at older ages).

Compared with round 1, the mean response from panel members by round 3 decreased for all categories, except heart disease for ages 35 to 49 years and lung cancer for ages ≥50 years. Medians changed little, although they increased in three instances (total mortality for ages 35-49 years, lung cancer for ages ≥50 years, and oral cancer for ages ≥50 years). Compared with round 2, the round 3 estimated mean RR decreased for each cause of premature mortality in each age group. This change is primarily due to a considerable decrease in estimates for each category by the panel member who was highest in round 2. This panel member commented that he/she revised his/her estimates to correct “for earlier confusion and did not mean to suggest a particular biological mechanism as of yet unbroached.”

By round 3, the majority of panel members placed themselves in the “confident” or “very confident” category for all estimates, except oral cancer for individuals ages 35 to 49 years. Panel members tended to be more confident for estimates pertaining to total premature mortality and lung cancer risks than for estimates for heart disease and oral cancer. Panel members submitting the higher estimates tended to be less confident. Confidence tended to be slightly lower regarding estimates for the younger age group.

The results show that, within the selected Health Panel of experts, there is consensus that LN-SLT products are less hazardous than conventional cigarette smoking by a wide margin. Using an iterative Delphi process of estimation and discussion, the panel's consensus estimate (median total mortality RR) associated with LN-SLT fell between 5% and 10% of the risk of smoking. Estimated risk was highest for oral cancer, with the median risk of LN-SLT between 15% and 30% of the risk associated with smoking. As might be expected, given the smokeless nature of the product, the median risk estimates relative to smoking were substantially lower for lung cancer. The estimates were intermediate for heart disease, perhaps reflecting the differing conclusions reached by the studies conducted to date (28).

The median total mortality risks for LN-SLT relative to smoking were greater for younger tobacco users. Higher LN-SLT risks relative to smoking at younger ages, as explained by some panel members, reflect the strong impact of smoking duration on older smokers' risk, implying that risks for LN-SLT users do not increase with duration of use and thus age as much as smokers' risks. However, for lung and oral cancer, the median risks for older users of LN-SLT relative to smoking were greater than for younger users, without much justification. The general lack of comments and consensus on the issue may reflect the lack of corroborating research.

Certain limitations of the process as applied in this exercise bear mention. The construction of the questionnaire may have created differences in panel members' interpretation and responses. Some panel members expressed confusion about calculating the index anchored between 0 for nonusers and 100 for smokers, particularly regarding the role of background risks; the moderator made attempts to clarify. Questions were asked about the different types of health risks that enter into total mortality risks of LN-SLT users. Outcomes would perhaps have varied if the moderator had played a greater or lesser role in describing and evaluating the results, pressing on issues of differences, and providing greater guidance in the interpretation of assumptions. Some questions were raised by panel members on whether low-nitrosamine standards would be established in the United States. Little was said about the regulatory environment, leaving room for panel members to interpret how the Swedish experience with snus might apply to the United States. There may also be differences in panel members' knowledge of the literature on the risks of smokeless tobacco. We provided a common base of articles to participants, and some participants referred to specific articles in their responses.

A final limitation is that a relatively small number of panel members participated compared with prior Delphi analyses (18-23). We limited our original pool of potential participants by the number of scientists who are expert in health risks associated with both smoking and smokeless tobacco use and who would represent a balanced set of views. Financial limitations also constrained the size of the panel. Although we attempted to recruit a balanced Health Panel in terms of the range of views, the results unavoidably reflect the group of panel members who agreed to participate.

The results from this study should not be interpreted to mean that there is consensus that smokeless tobacco products are an acceptable harm reduction alternative to conventional cigarettes. In addition to its toxicity, an evaluation of the harm reduction potential of LN-SLT should consider who uses the product and how much they use it. Attention should be given to whether it substitutes for smoking, is used in conjunction with or as a gateway to smoking, or substitutes for complete nonuse of tobacco products. For example, LN-SLT might be used by a former smoker who would have remained off tobacco but was enticed back by the message that LN-SLT is relatively “safe.” However, for some individuals, LN-SLT use might also become a gateway out of smoking by acting as a substitute for cigarettes or as a mechanism for quitting all tobacco use. In addition, use patterns and toxicity will depend on how the product is marketed and the type of governmental regulations that surround its use. The results from this study also should not be interpreted to mean that all smokeless tobacco products are less hazardous or less risky by the same margin than conventional cigarettes because our panel members only considered a handful of unique LN-SLT products. Thus, smokeless tobacco manufacturers cannot use these results to justify blanket health claims in marketing smokeless tobacco products.

On the narrow question of the relative health risk of LN-SLT products, these results clearly indicate that experts perceive these products to be far less dangerous than conventional cigarettes. Based on the available published scientific literature as of 2003, there seems to be consensus that LN-SLT products pose a substantially lower risk to the user than do conventional cigarettes. This finding raises ethical questions concerning whether it is inappropriate and misleading for government officials or public health experts to characterize smokeless tobacco products as comparably dangerous with cigarette smoking (29).

Dear Dr. XXX,

I am contacting you to request your participation as a recognized expert in tobacco-related health risks in an exciting new research project intended to better understand issues related to tobacco harm reduction. A group of colleagues (M. Cummings, B. Gilpin, G. Giovino, A. Hyland, D. Levy, D. Sweanor, and K. Warner) and I are investigating the impact of a potentially reduced exposure product on U.S. tobacco consumption and health.

Initially, we will focus on the effect of low-nitrosamine smokeless tobacco (e.g., Ariva and snus) on mortality risk under a policy regimen of appropriate government warnings. Smokeless products were selected for two reasons: First, it is area of substantial contemporary discussion and controversy. Second, although the necessary data will certainly be incomplete, more is known about the use and health effects of this category of product than other tobacco potentially reduced exposure products, such as Omni. In the future, we hope to consider the effect of other potentially reduced exposure products, possibly assessing the impact of other policy regimens (e.g., differential taxation according to the perceived relative danger of a product).

We seek your input to assess the RRs for selected health outcomes associated with low-nitrosamine smokeless tobacco products. Developments within both tobacco and pharmaceutical industries require timely research and response from the public health community. We are eager to develop this research to advance the debate on the public health viability of harm reduction. The first stage of our work is the estimation of RRs using a Delphi approach. (See the attached Project Description for more details.) A behavioral panel of experts will be convened separately to identify and estimate probabilities associated with relevant behavioral pathways. Our group intends to submit a paper for publication describing the research process, crediting the individual Health Panelists' contributions, and discussing findings. Your participation as an expert on this Health Panel (nine members planned) would be greatly appreciated as an important contribution to the analysis of a pressing public health question.

The attached Project Description specifies the study goals and Health Panelists' anticipated contribution, project timeframe, and compensation. I have also attached a bibliography of the seminal studies that we expect would be helpful in forming your expert opinion. When you confirm your participation, please provide any additional citations you believe are essential to this collection of literature. We will provide the full package of background literature via express mail before the Delphi process is formally initiated.

Please let me know by July 3, 2003 if you will be able to participate and, if so, send your full contact information (address, telephone, and fax) and social security number. In addition, if you have vacation or extended leave scheduled during this summer in which you will not have time or access to a computer/e-mail, please specify those periods so that we can schedule the Delphi rounds accordingly. Our PIRE project administrators will send you a consulting agreement once we hear from you.

We hope you will be able to contribute to what we believe holds the potential to be truly important public health research.

Sincerely,

David T. Levy

Attachments: Project Description and Preliminary Bibliography

  1. Accortt NA, et al. Chronic disease mortality in a cohort of smokeless tobacco users. Am J Epidemiol 2002;156:730–7.

  2. Asplund K. Smokeless tobacco and cardiovascular disease. Prog Cardiovasc Dis 2003;45:383–94.

  3. Asplund K, Nasic S, Janlert U, Stegmayr B. Smokeless tobacco as a possible risk factor for stroke in men: a nested case-control study. Stroke 2003;34.

  4. Blot WJ, McLaughlin JK, Winn DM, et al. Smoking and drinking in relation to oral and pharyngeal cancer. Cancer Res 1988;48:3282–7. (Presents odds ratio for smokeless unconfounded by smoking)

  5. Bolinder G, et al. Smokeless tobacco use and increased cardiovascular mortality among Swedish construction workers. Am J Public Health 1994;84: 399–404.

  6. Critchley JA, Unal B. Health effects associated with smokeless tobacco: a systematic review. Thorax 2003 May;58:435–43. (Also includes appendices, listing all studies)

  7. Hecht SS. Biochemistry, biology, and carcinogenicity of tobacco-specific N-nitrosamines. Chem Res Toxicol 1998;11:559–603.

  8. Hoffman D, Harley NH, Fisenne I, Adams JD, Brunneman KD. Carcinogenic agents in snuff. J Natl Cancer Inst 1986 Mar;76.

  9. Hoffmann D, Djordjevic MV, Fan J, Zang E, Glynn T, Connolly GN. Five leading U.S. commercial brands of moist snuff in 1994: assessment of carcinogenic N-nitrosamines. J Natl Cancer Inst 1995;87:1862–9.

  10. Huhtasaari F, et al. Tobacco and myocardial infarction: is snuff less dangerous than cigarettes? BMJ 1992;305:1252–6.

  11. Huhtasaari F, et al. Smokeless tobacco as a possible risk factor for myocardial infarction: a population-based study in middle-aged men. J Am Coll Cardiol 1999;34:1784–90.

  12. Lewin F, et al. Smoking tobacco, oral snuff, and alcohol in the etiology of squamous cell carcinoma of the head and neck: a population-based case-referent study in Sweden. Cancer 1998;82:1367–75.

  13. Martinez I. Factors associated with cancer of the esophagus, mouth, and pharynx in Puerto Rico. J Natl Cancer Inst 1969;42:1069–94.

  14. Mashberg A, et al. Tobacco smoking, alcohol drinking, and cancer of the oral cavity and oropharynx among U.S. veterans. Cancer 1993;72:1369–75.

  15. Rodu B, Cole P. Smokeless tobacco use and cancer of the upper respiratory tract. Oral Surg 2002;93: 511–5.

  16. Schildt EB, et al. Oral snuff, smoking habits and alcohol consumption in relation to oral cancer in a Swedish case-control study. Int J Cancer 1998;77: 341–6.

  17. Stockwell HG, Lyman GH. Impact of smoking and smokeless tobacco on the risk of cancer of the head and neck. Head Neck Surg 1986;9:104–10.

  18. Wallenfeldt K, Hulthe J, Bokemark L, Wikstrand J, Fagerberg B. Carotid and femoral atherosclerosis, cardiovascular risk factors and C-reactive protein in relation to smokeless tobacco use or smoking in 58-year-old men. J Intern Med 2001 Dec;250:492–501.

  19. Williams RR, Horm JW. Association of cancer sites with tobacco and alcohol consumption and socioeconomic status of patients: interview study from the Third National Cancer Survey. J Natl Cancer Inst 1977;58:525–47.

  20. Winn DM, et al. Snuff dipping and oral cancer among women in the southern United States. N Engl J Med 1981;304:745–9.

  • ASSUMPTIONS

    Assume an open-ended scale, with the absence of risk equal to 0 and the risk of premature mortality associated with lifelong conventional cigarette smoking equal to 100. A risk for smokeless products greater than cigarettes thus would imply a risk equal to >100.

  • Assume that the smokeless product user uses only a LN-SLT product (e.g., Ariva and Swedish snus).

  • Assume that the dose each individual consumes represents “typical use” for that product.

  • Assume that the prototypical current smoker of conventional cigarettes and the user of smokeless tobacco do not switch to the other product nor do they quit use.

  • Assume the current smoker of conventional cigarettes and the user of LN-SLT products started regular or established use at age 17 years.

The following questions do not address RRs associated with switching tobacco products nor with concurrent use of cigarettes and a smokeless product. We do not believe data are available to address these two areas, and we do not mean to press you to develop estimates in this exercise. However, your comments on health risks associated with these potential behavioral pathways would be appreciated at any point.

QUESTIONS

  • Relative to the risk to a lifelong smoker of conventional cigarettes (e.g., Marlboro Red and Newport), what is the RR of premature total mortality to a lifelong user of LN-SLT products (e.g., Ariva and Swedish snus)?

  • Relative to the risk to a lifelong smoker of conventional cigarettes (e.g., Marlboro Red and Newport), what is the RR of lung cancer mortality to a lifelong user of LN-SLT products (e.g., Ariva and Swedish snus)?

  • Relative to the risk to a lifelong smoker of conventional cigarettes (e.g., Marlboro Red and Newport), what is the RR of heart disease mortality to a lifelong user of LN-SLT products (e.g., Ariva and Swedish snus)?

  • Relative to the risk to a lifelong smoker of conventional cigarettes (e.g., Marlboro Red and Newport), what is the RR of oral cancer mortality to a lifelong user of LN-SLT products (e.g., Ariva and Swedish snus)?

Grant support: National Cancer Institute Tobacco Control Research Branch, Division of Cancer Control and Population Sciences and National Cancer Institute CISNET Program grant UO1-CA-097450 (D.T. Levy and E.A. Mumford).

The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement inaccordance with 18 U.S.C. Section 1734 solely to indicate this fact.

We thank Scott Leischow for his encouragement in developing this project and Hamed Rezaishiraz for his literature search and article summary.

1
Stratton K, Shetty P, Wallace R, Bondurant S. Clearing the smoke: assessing the science base for tobacco harm reduction. Washington (DC); Institute of Medicine, National Academy Press; 2001.
2
Shiffman S, Gitchell JG, Warner KE, et al. Tobacco harm reduction: conceptual structure and nomenclature for analysis and research. Nicotine Tob Res 2002;4 Suppl 2:S113–29.
3
Gilljam H, Galanti MR. Role of snus (oral moist snuff) in smoking cessation and smoking reduction in Sweden. Addiction 2003; 98:1183–9.
4
Fagerstrom KO, Schildt EB. Should the European Union lift the ban on snus? Evidence from the Swedish experience. Addiction 2003; 98;1191–5.
5
Ramstrom L. Snus: part of the problem or part of the solution? Addiction 2003;98:1198–9; discussion 1204–7.
6
Sweanor D. Reducing toxicity exposure for the nicotine-dependent. Addiction 2003;98:1197–8; discussion 1204–7.
7
Bolinder G. Swedish snuff: a hazardous experiment when interpreting scientific data into public health ethics. Addiction 2003;98:1201–4; discussion 1204–7.
8
Foulds J, Ramstrom L, Burke M, Fagerstrom K. Effect of smokeless tobacco (snus) on smoking and public health in Sweden.
Tob Control
2003
;
12
:
349
–59.
9
Tomar SL, Connolly GN, Wilkenfeld J, Henningfield JE. Declining smoking in Sweden: is Swedish Match getting the credit for Swedish tobacco control's efforts? Tob Control 2003;12:368–71.
10
Bates C, Fagerstrom K, Jarvis MJ, et al. European Union policy on smokeless tobacco: a statement in favour of evidence based regulation for public health.
Tob Control
2003
;
12
:
360
–7.
11
Can tobacco cure smoking: review of tobacco harm reduction. In: Subcommittee on Commerce, Trade and Consumer Protection of the Committee on Energy and Commerce. Washington (DC): Government Printing Office; 2003.
12
Warner KE, Martin EG. The US tobacco control community's view of the future of tobacco harm reduction. Tob Control 2003;12: 383–90.
13
Joseph AM, Hennrikus D, Thoele MJ, Krueger R, Hatsukami D. Community tobacco control leaders' perceptions of harm reduction.
Tob Control
2004
;
13
:
108
–13.
14
Martin EG, Warner KE, Lantz PM.
Tobacco harm reduction: what do the experts think? Tob Control
2004
;
13
:
123
–8.
15
Cummings KM. Can capitalism advance the goals of tobacco control? Addiction 2002;97:957–8; discussion 961–2.
16
Bates C, Fagerstrom K, Jarvis M, et al. European Union policy on smokeless tobacco: a statement in favour of evidence-based regulation for public health. 2003.
17
U.S. Smokeless Tobacco Company. Federal Trade Commission petition. Greenwich (CT): U.S. Smokeless Tobacco Company; 2003.
18
Dalkey N, Helmer O. An experimental application of the Delphi method to the use of experts.
Manage Sci
1963
;
9
:
458
–67.
19
Dunn W. Public policy analysis. Englewood Cliffs (NJ): Prentice-Hall; 1981.
20
Crisp J, Pelletier D, Duffield C, Nagy S, Adams A. It's all in a name.
When is a “Delphi study” not a Delphi study? Aust J Adv Nurs
1999
;
16
:
32
–7.
21
Everett A. Piercing the veil of the future. A review of the Delphi method of research.
Prof Nurse
1993
;
9
:
181
–5.
22
Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique.
J Adv Nurs
2000
;
32
:
1008
–15.
23
Powell C. The Delphi technique: myths and realities.
J Adv Nurs
2003
;
41
:
376
–82.
24
Hahn EJ, Tourney CP, Rayens MK, McCoy CA. Kentucky legislators' views on tobacco policy.
Am J Prev Med
1999
;
16
:
81
–8.
25
Rainhorn JD, Brudon-Jakobowicz P, Reich MR. Priorities for pharmaceutical policies in developing countries: results of a Delphi survey.
Bull World Health Organ
1994
;
72
:
257
–64.
26
Burnette D, Morrow-Howell N, Chen LM. Setting priorities for gerontological social work research: a national Delphi study.
Gerontologist
2003
;
43
:
828
–38.
27
Kirigia JM. Economic evaluation in schistosomiasis: valuation of health states preferences. A research note. Health Econ 1998;7: 551–6.
28
Critchley JA, Unal B. Health effects associated with smokeless tobacco: a systematic review.
Thorax
2003
;
58
:
435
–43.
29
Kozlowski LT. Harm reduction, public health, and human rights: smokers have a right to be informed of significant harm reduction options. Nicotine Tob Res 2002;4 Suppl 2;S55–60.