Electronic nicotine delivery systems and/or electronic non-nicotine delivery systems for tobacco smoking cessation or reduction: a systematic review and meta-analysis

Eligibility criteria

• Study designs: RCTs and prospective cohort studies.

• Participants: cigarette smokers, regardless of whether the users were using them as part of a quit attempt.

• Interventions: ENDS or ENNDS.

• Comparators:

  • no smoking cessation aid;
  • alternative non-electronic smoking cessation aid, including NRT, behavioural and/or pharmacological cessation aids (eg, bupropion and varenicline) and
  • alternative electronic smoking cessation aid (ENDS or ENNDS).
• Outcomes:

  • tobacco smoking cessation, with preference to biochemically validated outcomes (eg, carbon monoxide (CO)) measured at 6 months or longer follow-up;
  • reduction in cigarette use of at least 50% and
  • serious (eg, pneumonia, myocardial infarction) and non-serious (eg, nausea, vomiting) adverse events measured at 1 week or longer follow-up.

Data source and searches

A previous Cochrane review with similar eligibility criteria ran a comprehensive search strategy up to July 2014.⁸ Using medical subject headings (MeSH) based on the terms ‘electronic nicotine,’ ‘smoking-cessation,’ ‘tobacco-use-disorder,’ ‘tobacco-smoking’ and ‘quit’, we replicated the search strategy of that review⁸ in Medline, EMBASE, PsycInfo, CINAHL, Cochrane Central Register of Controlled Trials (CENTRAL), ISI Web of Science and the trial registry (clinicaltrials.gov). The appendix table 1 shows the search strategy for Ovid MEDLINE. This strategy was adapted for the other databases and run from 1 April 2014 to 29 December 2015. We did not impose any language restrictions.

In addition, we established a literature surveillance strategy based on the weekly search alerts by Centers for Disease Control and prevention (CDC)'s Smoking and Health Resource Library of published articles (http://nccd.cdc.gov/shrl/NewCitationsSearch.aspx) as well as the Gene Borio's daily news items (http://www.tobacco.org). The surveillance strategy started from the time of running the comprehensive literature search up to the time of the submission of this manuscript.

Selection of studies

Three pairs of reviewers underwent calibration exercises and used standardised pilot tested screening forms. They worked in teams of two and independently screened all titles and abstracts identified by the literature search, obtained full-text articles of all potentially eligible studies and evaluated them for eligibility. Reviewers resolved disagreement by discussion or, if necessary, with third party adjudication. We also considered studies reported only as conference abstracts. For each study, we cite all articles that used data from that study.

Data extraction

Reviewers underwent calibration exercises, and worked in pairs to independently extract data from included studies. They resolved disagreement by discussion or, if necessary, with third party adjudication. They abstracted the following data using a pretested data extraction form: study design; participants; interventions; comparators; outcome assessed and relevant statistical data. When available, we prioritised CO measurements as evidence of quitting. When CO measurement was unavailable, we used self-report measures of quitting.

Risk of bias assessment

Reviewers, working in pairs, independently assessed the risk of bias of included RCTs using a modified version of the Cochrane Collaboration's instrument¹⁵ (http:/distillercer.com/resources/).¹⁶ That version includes nine domains: adequacy of sequence generation, allocation sequence concealment, blinding of participants and caregivers, blinding of data collectors, blinding for outcome assessment, blinding of data analysts, incomplete outcome data, selective outcome reporting and the presence of other potential sources of bias not accounted for in the previously cited domains.¹⁶

For cohort studies, reviewers independently assessed risk of bias with a modified version of the Ottawa–Newcastle instrument¹⁷ that includes confidence in assessment of exposure and outcome, adjusted analysis for differences between groups in prognostic characteristics and missing data.¹⁷ For incomplete outcome data in individual studies (RCTs and prospective cohort studies), we stipulated as low risk of bias for loss to follow-up of <10% and a difference of <5% in missing data between intervention/exposure and control groups.

When information regarding risk of bias or other aspects of methods or results was unavailable, we attempted to contact study authors for additional information.

Certainty of evidence

We summarised the evidence and assessed its certainty separately for bodies of evidence from RCTs and cohort studies. We used the GRADE methodology to rate certainty of the evidence for each outcome as high, moderate, low or very low.¹⁸ In the GRADE approach, RCTs begin as high certainty and cohort studies as low certainty. Detailed GRADE guidance was used to assess overall risk of bias,¹⁹ imprecision,²⁰ inconsistency,²¹ indirectness²² and publication bias²³ and to summarise results in an evidence profile. We planned to assess publication bias through visual inspection of funnel plots for each outcome in which we identified 10 or more eligible studies; however, we were not able to because there were an insufficient number of studies to allow for this assessment. Cohort studies can be rated up for a large effect size, evidence of dose–response gradient or if all plausible confounding would reduce an apparent effect.²⁴

Data synthesis and statistical analysis

We analysed all outcomes as dichotomous variables. In three-arm studies, we combined results from arms judged to be sufficiently similar (eg, Caponnetto 2013,²⁵ two arms with similar ENDS regimens: 7.2 mg ENDS and 7.2 mg ENDS plus 5.4 mg ENDS). When studies reported results for daily or intensive use of ENDS separately from non-daily or less intensive use, we included only the daily/intensive use in the primary pooled analysis (eg, Brose 2015,^26–28 we excluded patients with non-daily users; and Biener and Hargraves,²⁹ we excluded patients with intermittent defined use). We conducted a sensitivity analysis in which we included all ENDS users, daily/intensive and intermittent/less intensive use. For this analysis, when necessary, we assumed a correlation of 0.5 between the effects in the daily/intensive and intermittent/less intensive groups.

We synthesised the evidence separately for bodies of evidence from RCTs and cohort studies. For RCTs, we calculated pooled Mantel-Haenszel risk ratios (RRs) and associated 95% CIs using random effects models. For observational studies, we pooled adjusted ORs using random effects models.

After calculating pooled relative effects, we also calculated absolute effects and 95% CI. For each outcome, we multiplied the pooled RR and its 95% CI by the median probability of that outcome. We obtained the median probability from the control groups of the available randomised trials. We planned to perform separate analyses for comparisons with interventions consisting of ENDS and/or ENNDS and each type of control interventions with known different effects (no smoking cessation aid; alternative non-electronic smoking cessation aid including NRT and alternative electronic smoking cessation aid (ENDS or ENNDS)). For meta-analyses, we used 6 months data or the nearest follow-up to 6 months available.

For dealing with missing data, we used complete case as our primary analysis; that is, we excluded participants with missing data. If results of the primary analysis achieved or approached statistical significance, we conducted sensitivity analyses to test the robustness of those results. Specifically, we conducted a plausible worst-case sensitivity analysis in which all participants with missing data from the arm of the study with the lower quit rates were assumed to have three times the quit rate as those with complete data, and those with missing data from the other arm were assumed to have the same quit rate as participants with complete data.^{30 31}

We assessed variability in results across studies by using the I² statistic and the p-value for the χ² test of heterogeneity provided by Review Manager. We used Review Manager (RevMan) (V.5.3; Nordic Cochrane Centre, Cochrane) for all analyses.³²

Study selection

Figure 1 presents the process of identifying eligible studies, including publications in the last systematic review,⁸ citations identified through search in electronic databases and studies identified through contact with experts in the field. Based on title and abstract screening, we assessed 69 full texts of which we included 19 publications describing three RCTs involving 1007 participants^{25 33–39} and nine cohort studies with a total of 13 115 participants.^{26–29 40–46} The interobserver agreement for the full-text screening was substantial (kappa 0.73).

We contacted the authors of the 12 included studies, 9 of whom^{26–29 33 41 43 44 46} supplied us with all requested data; authors of further 3 studies^{25 42 46} did not supply the requested information (see online supplementary appendix table S2).

Study characteristics

Table 1 describes study characteristics related to design of study, setting, number of participants, mean age, gender, inclusion and exclusion criteria and follow-up.

Five studies^{25–28 33 42 46} were conducted largely in Europe, six in the USA,^{29 40 41 43–45} and one in New Zealand.^34–39 Randomised trials sample size ranged from 50³³ to 657,^34–39 and observational studies from 100⁴⁶ to 3891.^26–28 Typical participants were women in their 40 s and 50 s. Studies followed participants from 4 weeks⁴⁶ to 36 months.²⁹

Table 2 describes study characteristics related to population, intervention or exposure groups, comparator and assessed outcomes.

Of the three RCTs, one compared ENDS with NRT and ENNDS,^34–39 another compared different concentrations of ENDS with ENNDS²⁵ and the third compared different types of ENDS.³³ Only the Borderud study⁴¹ included participants who were also currently receiving other behavioural and pharmacologic treatment. The participants from Vickerman 2013⁴⁴ study were all enrolled in a state quitline programme that provided behavioural treatment and in some cases NRT. All nine cohort studies^{26–29 34–46} compared ENDS with no use of ENDS^{26–29 40 41} or tobacco cigarettes only;⁴² in one,⁴¹ exposure and non-exposure groups received behavioural and other pharmacologic treatment.

Table 3 describes the mean number of conventional cigarettes used per day at baseline and the end of study.

The mean number at baseline ranged from 11.9 in the no ENDS group⁴⁵ to 20.6 in the ENDS group.³³ In only two studies^{26–28 45} the mean number of conventional cigarettes used per day presented a reduction from the baseline to the end of study in the ENDS group compared with the no ENDS groups, mainly in the daily users.^26–28 No included study addressed users of combustible tobacco products other than cigarettes.

Online supplementary appendix table S3 presents the types of e-cigarettes used in the included studies. The three RCTs^{25 33 36–39} evaluated only ENDS-type cigalikes. A total of 23.7% of the participants from Brose 2015^26–28 study used tank and in the Hajek 2015⁴⁶ study participants used either cigalike or tank. The remaining studies did not report the type of ENDS used.

Risk of bias

Figures 2 and 3, and table 4, describe the risk of bias assessment for the RCTs.

The major issue regarding risk of bias in the RCTs of ENDS versus ENNDS was the extent of missing outcome data.^{25 34–39} RCTs comparing ENDS with other nicotine replacement therapies had additional problems of concealment of randomisation³³ and blinding.^33–39

Figure 4 and table 5 describe the risk of bias assessment of the cohort studies.

Seven^{26–29 40–42 44 45} of nine cohort studies were rated as high risk of bias for limitations in matching exposed and unexposed groups or adjusting analysis for prognosis variables; confidence in the assessment of the presence or absence of prognostic factors; confidence in the assessment of outcome and similarity of cointerventions between groups; all studies suffered from high risk of bias for missing outcome data.

Outcomes

The mean number of conventional cigarettes/tobacco products used per day at the end of the studies ranged from 0.7^34–39 in ENDS and ENNDS groups to 13.9^26–28 among non-daily users of ENDS (table 3). The three RCTs^{25 33–39} and one cohort study⁴² biochemically confirmed nicotine abstinence while the others presented only self-reported data^{26–29 40 41 43–45} (table 3).

Main findings

Based on pooled data from two randomised trials with 481 participants, we found evidence for a possible increase in tobacco smoking cessation with ENDS in comparison with ENNDS (figure 5). The evidence is, however, of low certainty: the 95% CI of the relative risk crossed 1.0 and a plausible worse case sensitivity analysis to assess the risks of bias associated with missing participant data yielded results that were inconsistent with the primary complete case analysis (see online supplementary appendix figure 1). Furthermore, in all these RCTs, the ENDS tested were of earlier generation; it is unknown whether providing later generation of e-cigarettes or a realistic scenario of allowing users to choose e-cigarettes based on self-preference would have greater benefit. There was no robust evidence of side effects associated with ENDS in the RCTs.

Cohort studies provide very low-certainty evidence suggesting a possible reduction in quit rates with use of ENDS compared with no use of ENDS (figure 6). These studies had a number of limitations: an unknown number of these participants were not using ENDS as a cessation device; some were not using ENDS during a quit attempt; many did not have immediate plans to quit smoking and some may have already failed attempts to stop smoking. In our risk of bias assessment, we judged that seven of nine studies did not have optimal adjustment for prognostic variables. Further, as any cohort study, the results are vulnerable to residual confounding. In particular, use of ENDS may reflect the degree of commitment to smoking cessation, and it may be the degree of commitment, rather than use of ENDS, that is responsible for the change in quit rates. For instance, the finding in two studies that daily use of ENDS, but not intermittent use, increased quit/reduction rates could be interpreted as evidence of the effectiveness of daily use. An alternative interpretation, however, is that those that used ENDS daily were more motivated to stop smoking, and the increased motivation, rather than daily use of ENDS, was responsible for their degree of success. It is worth to mention that motivation to quit smoking is a major determinant of success regardless of the aid used.

In terms of bias against ENDS, cohort studies sometimes enrol smokers already using ENDS and still smoking. Such individuals may already be failing in their attempts to stop smoking. If so, enrolling these participants will underestimate ENDS beneficial effects. Additional concerns with cohort studies include their failure to provide optimal adjustment for prognostic variables or provide data regarding use of alternative smoking reduction aids.

Strengths and limitations

Strengths of our review include a comprehensive search; assessment of eligibility, risk of bias and data abstraction independently and in duplicate; assessment of risk of bias that included a sensitivity analysis addressing loss to follow-up and use of the GRADE approach in rating the certainty of evidence for each outcome.

The primary limitation of our review is the low certainty consequent on study limitations. We identified only a small number of RCTs with a modest number of participants resulting wide CIs. Moreover, loss to follow-up was substantial, and our sensitivity analysis demonstrated the vulnerability of borderline effects to missing data. The limitations of the cohort studies led us to a rating of very low-certainty evidence from which no credible inferences can be drawn.

Another limitation of this review is the fact that we could not address our hypothesis about increased rates in smoking cessation in those who used e-cigarettes with higher concentrations of nicotine compared with those using less nicotine, or daily e-cigarette users compared with non-daily e-cigarette users, or those who use newer forms of ENDS compared with users of first generation devices due to lack of evidence. However, although these assumptions seems logical, nicotine delivery from ENDS depends on other factors such as the efficiency of the device in aerosolising the liquid and user experience, apart from the concentration of nicotine in the ENDS liquid.

Furthermore, whether or not ENDS are an effective aid in the cessation of smoking may depend on whether the users were using ENDS as part of a quit attempt or not, and this may play an important role also as a possible confounder. Data is not yet available to conduct a subgroup analysis addressing this hypothesis. Subsequent trials should help provide information regarding whether their impact on cessation of smoking depends on whether users were intended to quit smoking, as well as the other unresolved issues.

Other limitations of this review were the fact of having insufficient number of included studies to allow the complete statistical analysis that we had planned. We were not able to assess publication bias because there were <10 eligible studies addressing the same outcome in a meta-analysis. We also planned to perform subgroup analyses according to the characteristics of:

• participants (commitment to stopping smoking, use of e-cigarettes at baseline);

• interventions (dose of nicotine delivered by the e-cigarette, frequency of use of the e-cigarette and type of e-cigarettes) and

• concomitant interventions in e-cigarettes and control groups.

However, we also were not able to conduct these analyses because they did not meet our minimal criteria, which were at least five studies available, with at least two in each subgroup. A final statistical limitation is that we calculated differences from 6 to 12 months of follow-up. Absolute differences may differ across this time frame and constitute a source of variability. Moreover, there are three schools of thought with respect to use of fixed and random effect models: those who prefer always to use fixed effects, those who prefer (almost) always random effects and those who would choose fixed and random depending on the degree of heterogeneity. Each argument has its proponents within the statistical community. The argument in favour of the second rather than the third is 1) there is always some heterogeneity, so any threshold of switching models is arbitrary and 2) when there is little heterogeneity, fixed and random yield similar or identical results, so one might as well commit oneself to random from the start. We find these two arguments compelling; thus, our choice.

Finally, another limitation of the observational studies in this review is the potential for selection bias as the populations compared differ in terms of intention to quit. Furthermore, in all these RCTs, the ENDS tested were earlier generation; it is possible that later generation of e-cigarettes would have greater benefit.

Although this review presents several limitations, the issue is whether one should dismiss these results entirely or consider them bearing in mind the limitations. The latter represent our view of the matter.

Relation to prior work

The previous Cochrane review⁸ concluded that due to low event rates and wide CIs, only low-certainty evidence was available from studies comparing ENDS with ENNDS. We excluded some studies included in that Cochrane review as they were either case series, cross-sectional or did not include one arm with ENDS/ENNDS compared with alternative strategies. We also included one additional RCT,³³ and nine new cohort studies,^{26–29 40–44 46} not included in the Cochrane review. The rationale for including the prospective cohort studies in our review was that it was anticipated that the search would return few RCTs. The authors of the Cochrane review found that ENDS is a useful aid to stop smoking long term compared with ENNDS.

Another review¹⁰ including two of our three RCTs,^{25 34–39} and further two case series and two cross-sectional studies, assessed the impact of e-cigarettes in achieving smoking abstinence or reduction in cigarette consumption among current smokers who had used the devices for 6 months or more. The authors concluded that e-cigarette use is associated with smoking cessation; these results are similar to our meta-analysis comparing ENDS with ENNDS (figure 5). Khoudigian's 2016 review¹¹ reported a non-statistically significant trend toward smoking cessation in adults using nicotine e-cigarettes compared with other therapies or placebo. However, the review by Kalkhoran and Glatz 2016⁹ concluded that e-cigarettes are associated with significantly less quitting among smokers.

Implications

Existing smoking reduction aids such as NRT are effective, but their impact is limited: the proportion of those who quit when using these aids remains small. The available evidence, of low or very low quality, can neither verify nor exclude the hypothesis that, because they address nicotine addiction and potentially deal with behavioural and sensory aspects of cigarette use, ENDS may be more effective than other nicotine replacement strategies. This is an important finding, and raises questions regarding how effective it may be addressing the behavioural and sensory aspects of cigarette use in their addictive potential. Thus, the focus of subsequent work should perhaps be on the dose and delivery of nicotine, though teasing out the nicotine effects from sensory aspects is likely to be challenging. It is possible that type of ENDS or dose of exposure may influence quit rates, and that newer models may be more effective, but there is insufficient data to provide insight into these issues. Lack of usefulness with regard to address the question of e- cigarettes' efficacy on smoking reduction and cessation was largely due to poor reporting.

Therefore, owing to the limitations of the studies included in this analysis it is impossible to make strong inferences regarding whether e-cigarette use promotes, has no effect or hinders smoking cessation. This review underlines the need to conduct well-designed trials in this field measuring biochemically validated outcomes and adverse effects.