Review of the quality of studies on the economic effects of smoke-free policies on the hospitality industry
- 1VicHealth Centre for Tobacco Control, The Cancer Council, Victoria, Melbourne, Victoria, Australia
- 2Roswell Park Cancer Institute, Department of Cancer Prevention, Epidemiology, and Biostatistics, Buffalo, New York, USA
- 3Institute of Health Policy Studies and Cardiovascular Research Institute, University of California, San Francisco, California, USA
- Correspondence to: Michelle Scollo, VicHealth Centre for Tobacco Control, The Cancer Council, Victoria, 100 Drummond Street, Carlton, 3053, Australia;
- Received 1 October 2002
- Accepted 20 October 2002
Objective: To compare the quality and funding source of studies concluding a negative economic impact of smoke-free policies in the hospitality industry to studies concluding no such negative impact.
Data sources: Researchers sought all studies produced before 31 August 2002. Articles published in scientific journals were located with Medline, Science Citation Index, Social Sciences Citation Index, Current Contents, PsychInfo, Econlit, and Healthstar. Unpublished studies were located from tobacco company websites and through internet searches.
Study selection: 97 studies that made statements about economic impact were included. 93% of the studies located met the selection criteria as determined by consensus between multiple reviewers.
Data extraction: Findings and characteristics of studies (apart from funding source) were classified independently by two researchers. A third assessor blind to both the objective of the present study and to funding source also classified each study.
Data synthesis: In studies concluding a negative impact, the odds of using a subjective outcome measure was 4.0 times (95% confidence interval (CI) 1.4 to 9.6; p = 0.007) and the odds of not being peer reviewed was 20 times (95% CI 2.6 to 166.7; p = 0.004) that of studies concluding no such negative impact. All of the studies concluding a negative impact were supported by the tobacco industry. 94% of the tobacco industry supported studies concluded a negative economic impact compared to none of the non-industry supported studies.
Conclusion: All of the best designed studies report no impact or a positive impact of smoke-free restaurant and bar laws on sales or employment. Policymakers can act to protect workers and patrons from the toxins in secondhand smoke confident in rejecting industry claims that there will be an adverse economic impact.
Smoke-free workplace policies reduce both exposure to secondhand tobacco smoke and cigarette consumption.1–3 Smoke-free restaurants and bars similarly reduce exposure to tobacco smoke toxins among hospitality workers and patrons but also represent a serious business threat to the tobacco industry.
In California in 1987, a 100% smoke-free restaurant ordinance in Beverly Hills was rolled back, partly in response to claims that the ordinance was responsible for reducing restaurant revenues by 30%, claims which later turned out to be unsubstantiated.4 Since then, tobacco companies and allied groups have routinely predicted that enactment of such legislation would severely impact restaurant and bar sales and employment.5–9 Health advocates, by contrast, have presented studies indicating that no such adverse effects actually occurred.10
Policymakers are typically presented with a large amount of conflicting material, with evidence ranging from anecdotes about individual businesses11 to scientific studies analysing objective information collected independently across an entire hospitality sector.12 Such data are often confusing to interpret and it is difficult for policymakers to reach an evidence based conclusion. In their case study of deliberations by the Maryland Occupational Safety and Health Advisory Board, Montini et al demonstrate that those opposing proposed smoke-free workplace regulations lodged twice the number of submissions as those supporting it, but that evidence from opponents was substantially less scientifically rigorous than evidence provided by supporters of workplace smoking regulations.13 Similar findings were observed in relation to the Californian Environmental Protection Agency’s risk assessment of secondhand smoke,14 and in Maryland and Washington hearings on proposed clean indoor air regulations.15 Bero and her colleagues have repeatedly called on advocates to more forcefully draw to legislators’ attention the superior scientific quality of the evidence base relied upon by public health groups in calling for clean air legislation.13–15
This paper compares the quality of evidence and conclusions about the economic impact of smoke-free laws on the hospitality industry based on the type of data used, how the studies are designed, analysed and interpreted, and the funding source.
Studies included in this analysis are listed in a comprehensive summary produced by the VicHealth Centre for Tobacco Control.16 Centre researchers attempted to locate all studies produced in English before 31 August 2002 that purported to assess the economic impact of smoke-free policies in the hospitality industry. Peer reviewed articles were located with Medline, Science Citation Index, Social Sciences Citation Index, Current Contents, PsychInfo, Econlit, and Healthstar using the terms smok* and restaurants, bars, hospitality, economic, regulation and law. Unpublished studies were also included in the analysis. These studies were located from a compilation by the Alberta Tobacco Control Centre,10 by a request to members of the International Union Against Cancer’s International Tobacco Control Network (GLOBALink), and an examination of hospitality industry websites and the websites of tobacco companies based in major English speaking countries, including the Philip Morris “Options” website, www.pmoptions.com. The researchers also conducted an internet search with the Google search engine www.google.com, using the terms “smok* bans” and “restaurants” or “bars”, limited by the terms “economic impact” or “study”.
Studies included measured changes in sales, employment, numbers of establishments, bankruptcy data, public reports of intentions about or recent changes in patronage, spending or time spent dining, proprietors predictions or perceptions of sales changes and costs and estimated numbers of tourists. Studies were excluded where these made no explicit or implicit attempt to quantify the economic impact of smoking restrictions. Studies assessing opinions about smoke-free policy were included where the study included a question asking specifically whether people would attend venues more or less frequently were such policies to be introduced.
Ninety three per cent of the studies located (97/104) met the selection criteria as determined by consensus between multiple reviewers.
Findings and characteristics of studies (apart from funding source) were classified independently by two researchers in most cases several months before the start of the study (MS and AL). Both assessors had tertiary qualifications in behavioural science and economics. A third assessor, (LH) a postgraduate psychology student blind to both the objective of the present study and to the funding source, also classified each study.
We used Siegel’s criteria12 to judge study quality: use of objective data (for example, tax receipts or employment statistics); inclusion of all data points after the law was implemented and several years before; use of regression or other statistical methods that control for secular trends and random fluctuation in the data; and appropriate control for overall economic trend. The more criteria a particular study met, the more certain one can be about the validity of results.
An outcome measure was deemed “objective” if it was based on data collected routinely by an independent agency covering the periods both before and after the smoke-free policy was in force. Objective measures included: sales figures provided for the purposes of taxation assessment; employment figures provided to government agencies generally for insurance purposes; and numbers of new or existing establishments based on business permit applications or registrations to the government agency that issues such permits, and bankruptcy data.
Unverifiable predictions of future changes or estimates of recent changes in patronage or spending were deemed “subjective”. Subjective measures included anecdotal reports and self report data collected in polls of, or interviews with, patrons or owners of restaurants, bars or similar businesses, conducted either before or after the policy was put in place.
Another indicator of the quality of a study is whether it has been subject to peer review. The scientific quality of original research on secondhand smoke published in peer reviewed journals is superior to that in non-peer reviewed publications in terms of study design, reporting, and evaluation.17 A study was deemed to have been peer reviewed if it was an article published in an academic journal.
Studies were classified as indicating or not indicating a negative economic effect based on their stated conclusions about the impact or potential impact on employment or profitability of the various sections of the hospitality industry at issue. All three raters agreed on the conclusions in all but one of the 97 studies. The Masotti study18 was classified as negative, as per the conclusion of two of the three raters. Two studies19,20 did not draw conclusions; we included them in the analysis based on their face-value findings.
Funding sources for each paper were noted after completion of all the other classification tasks. Funding was determined from acknowledgments in the studies (obscured on copies of papers classified by the third rater). When the source was not clearly disclosed, authors were contacted where possible and attempts were made to determine whether authors or sponsoring agencies had ever received financial support from a tobacco company or affiliated group. Searches were undertaken of previously secret tobacco industry documents made available as part of settlement agreements between tobacco companies the US attorneys general21,22 (accessible through www.tobaccoarchives.com).
We used contingency tables and odds ratios to compare studies concluding a negative impact with those not concluding a negative impact.
A total of 97 reports were located. 5,18–20,23–115. Studies covered numerous local jurisdictions in 31 state or provincial areas, in eight countries. Thirty four of the studies examined the impact of smoke-free policies for drinking establishments, and 90 the impact of smoke-free policies for restaurants. Two studies examined the impact for recreational venues. One examined the impact of smoke-free restaurants on hotels and another on overall tourism.
Of the 97 studies, 38% (37/97) used objective outcome measures and 25% (24/97) were peer reviewed. Twenty one studies met Siegel’s four criteria for methodological quality, representing 57% (21/37) of the studies that used objective measures. By contrast, 98% (59/60) of the studies using only subjective outcome measures met none of the criteria. The odds of peer reviewed articles meeting all four criteria was 5.33 times that of non-peer reviewed studies (95% confidence interval (CI) 1.9 to 15.1; p = 0.002).
Of the 97 studies, 32% (31) were funded either by the tobacco industry or a group known to have received funding from a tobacco company or tobacco industry ally.19,20,23,57,59–61,80–83,84,86–97,103,107,108,114 Four studies did not disclose funding source, but were conducted by consultants or organisations known to have a connection with the tobacco industry.18,31,58,85 For three of these studies, there was strong evidence of collaboration between tobacco companies and the study authors.31,58,85
The two studies by consultants where close ties with the tobacco industry had been established and those funded by organisations known to have received funding from the tobacco industry were included with the tobacco industry funded studies, yielding 31 tobacco industry supported studies. The funding source was treated as missing for six studies18,27,99,104–106 because no data on funding source could be located.
All the studies used in this analysis is summarised in tables 1 and 2.
Study quality and funding
The methodological quality of the industry sponsored studies was significantly lower (p < 0.001) than the non-industry studies (table 3). Only one of the 31 tobacco industry supported studies (3%) has been published in a peer reviewed journal compared to 38% (23/60) of the non-industry funded studies. None (0/31) of the tobacco industry supported studies met all of Siegel’s12 four methodological quality criteria. Indeed, 84% (26/31) of the industry supported studies met none of the criteria. By contrast, 35% (21/60) of the studies not supported by the tobacco industry met all of Siegel’s12 criteria5,25,30,33–36,42–46,48,49,51–56,98 (p < 0.001). Eleven of these non-industry funded studies have been published in peer reviewed journals.
Study quality and conclusion
Table 4 sets out the findings of those studies meeting each of various indicators of high quality: using objective outcome measures; meeting this and Siegel’s other three criteria for quality; being funded by a source clearly independent of the tobacco industry; and being peer reviewed.
None of the 21 studies that met all four of Siegel’s12 quality criteria reported a negative impact (table 4). In fact, four of the studies report a positive impact on taxable sales receipts of restaurants, bars, hotels, or tourism.5,36,43,48
Only a handful of studies based on objective data conclude a negative impact. None of these meets more than one of Siegel’s other three criteria for methodological quality. Only one peer reviewed study concluded a negative impact.84 This study relied on subjective data and was funded by a tobacco company.
Table 5, conversely, shows, for those studies concluding negative impact, whether each of the various quality criteria was met. Once again, studies concluding a negative impact rarely included an objective measure and were almost never peer reviewed.
In studies concluding a negative impact, the odds of using only a subjective measure was 4.0 times (95% CI 1.4 to 9.9; p = 0.007) and the odds of being peer reviewed was 20 times (95% CI 2.6 to 166.7; p = 0.004) that of studies concluding no such negative impact (table 6)
Funding source and conclusion
There was a significant association of tobacco industry support with negative conclusions of the study (p < 0.001) (table 6). Ninety four per cent (29/31) of the tobacco industry supported studies concluded that there was or would be a negative economic impact of implementing a smoke-free policy. The odds ratio for a negative conclusion associated with tobacco industry support was infinite because none of the 60 non-industry funded studies concluded a negative economic impact.
Lower quality studies were much more likely to conclude smoke-free regulations adversely impact the hospitality industry, and weaker studies were much more likely to be funded by the tobacco industry. In addition, the industry studies were less likely to be published in the peer reviewed literature. Almost all (94%) of industry supported studies, compared to none of the studies funded by sources other than the tobacco industry, claimed a negative economic impact.
These results are consistent with a similar linkage between tobacco industry funding and conclusions in reviews of the effects of secondhand smoke.116 Barnes and Bero116 identified 106 reviews of the relation between secondhand smoke and disease. Thirty seven per cent of these reviews concluded that passive smoking was not harmful to health; 74% of these reviews were written by authors with tobacco industry affiliations. Among reviews written by individuals with no industry affiliations, only 13% (10/75) reached the conclusion that passive smoking is not harmful to health. In logistic regression analyses controlling for article quality, peer review status, article topic, and publication year, the only factor associated with concluding that passive smoking is not harmful was whether an author was affiliated with the tobacco industry (odds ratio 88.4; p < 0.001).
The possibility of publication bias always exists. It is possible that studies by those sympathetic to public health goals that detect a negative impact would be less likely to be submitted for publication. On the other hand those funded by the tobacco industry would be similarly unlikely to release studies detecting no negative impact. We have made every effort to identify all studies done on the effects of smoke-free laws and regulations on the hospitality industry. The fact that the tobacco industry has a strong motivation to publicise all negative studies adds to our confidence that we have not missed a substantial number of studies concluding a negative economic impact.
Siegel’s criteria are a valuable tool for assessing the quality of studies on the economic impact of smoke-free policies in the hospitality industry. Our findings suggest that policymakers can make a quick preliminary assessment of study quality by asking three questions:
Was the study funded by a source clearly independent of the tobacco industry?
Did the study objectively measure what actually happened, or was it based on subjective predictions or assessments?
Was it published in a peer reviewed journal?
Of the 35 studies on this topic published that concluded a negative impact, none have been funded by a source clearly independent of the tobacco industry, and none have both used an objective measure and been peer reviewed. In fact, 80% of these studies passed none of these basic tests of quality. With all 21 of the well designed studies finding that smoke-free restaurant and bar laws had no negative impact on revenue or jobs, policymakers can act to protect workers and patrons from the toxins in secondhand smoke confident in rejecting predictions that there will an adverse economic impact.
Thanks to Dr Mohammad Siahpush from the VicHealth Centre for Tobacco Control for advice on statistical analysis, and Louisa Hoey for assistance with classification of studies.
The VicHealth Centre for Tobacco Control is funded by the Victorian Health Promotion Foundation to conduct economic, legal, and social research in tobacco control. Dr Hyland’s work was supported by the Roswell Park Cancer Institute NCI-funded Cancer Center Support Grant, CA16056-26 as a member of the Biomathematics/Biostatistics Core Resource. Dr Glantz’s work was supported by US National Cancer Institute grant CA-61021.