International Expert Consensus on Relevant Health and Functioning Concepts to Assess in Users of Tobacco and Nicotine Products: Delphi Study

Introduction

Reducing exposure to harmful and potentially harmful constituents in cigarette smoke by cessation of smoking or switching to reduced risk tobacco and/or nicotine products (TNPs) is a major public health focus worldwide [1]. Although nicotine-replacement therapy (NRT) and targeted counseling services can help individuals achieve their quitting goals, there can be hurdles to becoming and/or staying smoking free [2].

Smoke-free tobacco and/or nicotine products (sf-TNPs) refer to TNPs that do not undergo combustion and therefore do not produce the harmful smoke generated by cigarettes, which has been established as the main risk factor of tobacco-related diseases [3]. These sf-TNPs can include heated tobacco products, e-cigarettes (vapes), other e-vapor products (e-pipes, e-cigars, and so on), nicotine pouches, and smokeless tobacco products (snuff, snus, and chewing tobacco) [3]. Regulatory bodies, like the US Food and Drug Administration (FDA), require robust data to evaluate the health impacts of sf-TNPs and guide regulatory decisions on product authorization. For instance, the FDA’s modified risk tobacco product pathway mandates comprehensive scientific evidence to support claims that a tobacco product reduces harm or the risk of tobacco-related disease in individual tobacco users and benefits the health of the population as a whole [4]. Some sf-TNPs have been recognized as modified-risk tobacco products under this pathway [4,5] and may help reduce the negative health effects associated with commercially marketed combustible tobacco products such as cigarettes [6-8].

The adverse effects of smoking on health have been well documented, whereas stopping smoking can improve health and reduce the risk of disease [6,9,10]. Evidence also suggests that switching away from cigarettes to using sf-TNPs can help reduce cigarette consumption and may lead to cessation in some cases [8]. While studies have shown that sf-TNPs reduce exposure to many of the harmful and potentially harmful constituents found in cigarette smoke, they may still pose health risks. For instance, e-cigarettes, heated tobacco products, and smokeless tobacco have been associated with some cardiovascular, respiratory, and oral health risks, albeit to a lesser extent than combustible cigarettes [11-14]. The long-term health effects of sf-TNPs are not yet fully understood, and further research is necessary to comprehensively assess their safety profile. In addition, little is known about the self-reported health impact of switching from cigarettes to sf-TNPs [6]. In this context, measuring the self-reported experience of health and functioning (including health status, functional status, and other health-related quality of life constructs) is crucial to understanding the impact of tobacco harm reduction strategies [15].

Generic health status measures, such as the Short Form Survey–12 [16] and Short Form Survey–36 instruments [17], have shown that those who smoke tend to report lower health status compared with those who never smoked, although the impact of cessation on health status seems to be more complex. Some smoking-specific measures have been developed but not yet widely standardized; therefore, they may not be directly applicable to sf-TNPs [9,18,19]. Crucially, current measures often lack the necessary sensitivity to detect longitudinal health changes in healthy populations due to high ceiling effects at baseline [20], hindering the assessment of changes in health and functioning following a switch from smoking cigarettes to using sf-TNPs.

To help address these challenges, the development of a new self-reported measure (ABOUT – Health and Functioning) was undertaken to assess the health and functioning status of individuals using sf-TNPs. This new measure is part of the portfolio of the ABOUT Toolbox [21], consisting of self-report measures to assess perceptions and behavioral outcomes related to the use of sf-TNPs. The initial preparatory phase of the instrument development was based on several research activities (systematic literature review, reanalysis of qualitative data, and expert insights [22]) and resulted in the identification of 69 health and functioning concepts relevant to TNP use. The qualitative research phase that followed consisted of (1) concept elicitation interviews of users to understand their perceptions of health and functioning after switching to sf-TNPs [23,24] and (2) the Delphi panel study reported in this paper. The aim of the Delphi study was to identify the health and functioning concepts considered most important by clinical and health care experts (ie, health care professionals, researchers, policy makers, and those involved in smoking cessation or tobacco harm reduction) when assessing the health and functioning status of individuals who stop smoking cigarettes or switch to using sf-TNPs or NRTs. Delphi panel methodology is a well-established process for determining consensus among relevant groups of individuals to aid issue prioritization, ranking, development, and obtaining agreement on guidelines, concept-framework development, and development of outcome measures [25-27], including issues relevant to smoking-related behaviors and sf-TNP use [25,28,29]. Delphi panels have previously been used to forecast trends and changes over time in health-related matters [30-32].

Methods

Objectives and Summary of Approach

This Delphi panel was organized for experts to select and rank health and functioning concepts considered important to assess when individuals stop smoking combustible TNPs (eg, cigarettes) or switch to using sf-TNPs (eg, e-cigarettes [vapes], heated tobacco, or smokeless tobacco products) or NRTs. The study was conducted and reported based on available guidance for conducting and reporting Delphi studies in health care research [33].

A preliminary set of 69 concepts was identified from earlier research activities including a scoping literature review, results from the reanalysis of previous TNP consumer qualitative research results, and expert opinion from a small group of key opinion leaders [22]. This list was presented for evaluation in this Delphi panel according to the process described in Figure 1. An adapted approach was used for the Delphi panel, whereby the results of round I were reviewed and refined in light of the ongoing qualitative concept elicitation interview findings from consumers of TNPs and ongoing review by key opinion leaders (Supplementary Information and Table S1 in Multimedia Appendix 1). The concept list to be evaluated in round II was adjusted accordingly and refined considering the round I findings. Round III was a detailed review of the final set of health and functioning concepts. This step also included an evaluation of likelihood of change that experts considered important for each concept for inclusion in a self-reported measure of clinical relevance.

‎

Participants and Recruitment

For this adapted Delphi panel, we recruited experts who routinely treat, communicate with, or advise individuals who smoke or wish to stop smoking or switch to sf-TNPs. The recruitment aim was to ensure that a minimum of 100 experts globally completed all 3 rounds of the Delphi panel. To anticipate attrition between rounds, the initial recruitment target was set at 120 experts.

Recruitment was undertaken by QualWorld, who identified experts through their survey panels, existing contacts, publications, word of mouth, and canvassing through appropriate organizations in individual countries. The process and route of contact to identify potential respondents are outlined in Figure 2.

To anticipate and account for potential sensitivities among health care practitioners about undertaking a project sponsored by a tobacco company, the recruitment company sent out an initial survey asking which industries experts would be interested in working on health-related topics. Tobacco companies were listed among other industries such as mining, pharmaceuticals, and oil and gas. Only experts stating unprompted that they would consider participating in studies by tobacco companies were chosen to receive further correspondence linked to the Delphi panel. Subsequent recruitment steps proceeded on an individual basis according to predefined inclusion and exclusion criteria (Textbox 1).

To ensure consistency in responses across different regions, the Delphi panel was conducted in English where possible and in local languages for experts in Japan, South Korea, Russia, Ukraine, Czech Republic, and Bulgaria. A professional translation service translated the survey materials and responses, and back-translation techniques were used to verify accuracy. The local bilingual recruitment team members also performed proofreading and quality checks from a language and contextual perspective to ensure the translated content accurately reflected the original meaning. For each step of the Delphi panel, the English and local language surveys were programmed and uploaded for a user acceptability test by the project team before launch.

‎

Ethical Considerations

The New England Independent Review Board reviewed the study protocol and all study materials and any amendments and granted approval of the study (reference 1‐9184‐1).

Before participation, all participants were required to review and sign an informed consent form, which provided a detailed description of the study, outlined the procedures, and specified participant expectations. Participants had the opportunity to ask questions and receive answers before signing the consent form.

All study data were deidentified and participants were identified only by unique ID numbers throughout the research rounds.

All experts were compensated and received a cash honorarium according to local professional rates in their respective countries, ranging from £115 to £450 (approximately US $147 to US $576 at the time of the research), for their time after each round.

Procedure and Analytical Methods

MaxDiff Analysis

This study incorporated the use of MaxDiff—a type of best-worst (maximum difference) scaling analytical method [34-36]. This methodology was primarily chosen due to the large number of initial concepts to be evaluated. Ranking and rating exercises with numerous concepts can lead to a cognitive burden, so a best-worst scaling approach overcomes fatigue while generating additional information by making respondents choose the most and least important concepts. The result is a list of concepts in order of relative importance [36-38].

For each round, the MaxDiff Sawtooth software algorithm [39] generated a list of scaled scores from most to least important, as well as the relative importance of one concept compared with another. This was based on the number of times that concept was rated as the most or least important. In addition, cumulative scores were used to differentiate the number of concepts accounting for 20%, 50%, and 70% of the most important concepts; the 50th percentile was used as an initial guide for which concepts to retain.

Rounds I and II also featured an “anchor” value, which was a score marking the boundary between concepts considered important (to be retained in the next round) or unimportant (to be excluded from the next round). This anchor value was derived from an anchor question representing a threshold of “important or not important” included for each concept to be assessed as above (ie, important) or below (ie, unimportant) the anchor. The anchor question in this survey required one of the following 3 responses: “none of these [items] are important,” “some of these [items] are important,” or “all of these [items] are important.” The anchor value was statistically derived using the Sawtooth software [38]. An illustration of a typical MaxDiff question is shown in Figure 3.

‎

Latent Class Analysis

Latent class analysis [40] was also performed to determine whether some concepts were more important to some groups of experts over others, according to experts’ demographic or other characteristics. This ensured that the final set of items was not biased toward one subset of health care professionals at the expense of others’ preferences. Where there were concepts that were only important to subsets of the study sample but unimportant overall, these concepts could be retained.

Semiqualitative Assessment

Semiqualitative questions were incorporated into each Delphi panel round to provide an increased understanding of participants’ responses to the results of the ranking and scoring exercise. These questions were asked to ensure no concepts were missing or needed rewording, and to understand the reasons behind experts’ selections of importance or unimportance. They also sought to gauge agreement or disagreement with the overall ranking results and to gain insights into the reasons for these viewpoints. After each round, the semiqualitative comments were reviewed alongside the MaxDiff scores. This process involved coding and analyzing the qualitative feedback to identify common themes and insights that could provide context to the quantitative rankings. This feedback was then integrated with the quantitative MaxDiff results to refine and inform decisions about the list of concepts to be included for the next rounds. It helped explain the rationale behind the experts’ choices and highlighted any discrepancies or areas of consensus, ensuring that both qualitative and quantitative perspectives were considered in the decision-making process.

Results

Descriptive Participant Demographics

Approximately 3100 experts were contacted, with a final sample of 105 experts recruited across 26 countries. Table 1 shows the demographics and participant characteristics, and Table 2 shows the participant quotas per region and per specialty. Of the initial 105 experts, all participated in exploratory round I and round II, and 101 (96%) completed round III (Table 2). Nearly all participants (90/105, 87%) reported frequently addressing smoking cessation with clients or patients. A total of 64% (67/105) of experts in the study identified as male, and most were located in the Americas, Europe, or Asia. Regarding place of work, three-quarters (79/105, 75%) of experts were based in hospitals or other clinical settings, mostly as medical specialists or general care practitioners.

MaxDiff Analysis and Semiqualitative Assessment for Reduction or Refinement of Health and Functioning Concepts

Round I

A cumulative score was calculated by adding all the scaled scores (in this case, the total score was 14,900.17) and dividing by the scaled score (concept 1: 425/14,900=3%) and adding to the scaled score of the next highest concept (concept 2: 422/14,900=3%+3% concept 1 and so on; Figure 4). Regarding the relative importance of concepts in round I, when switching from combustible to sf-TNPs, experts placed importance on withdrawal symptoms, general health impacts, and emotion-related impacts. Conversely, concepts that ranked low in priority were those that referred to aesthetic concerns that would have less of a health impact or were not considered exclusive to those who smoke. Experts considered withdrawal symptoms such as anxiety and irritability as potential barriers to switching to sf-TNPs and highlighted there were levels of satisfaction (eg, enjoyment and craving relief) that those who smoke cigarettes would not want to lose when switching to sf-TNPs

‎

When determining which concepts to include in round II of the Delphi panel evaluation, the relative score (ie, importance) for each concept was considered, resulting in 36 out of 69 concepts (52% of all concepts in ranked order) accounting for 69% (10,285/14,900) of the total importance score. Based on the results and alignment with other activities ongoing within development of the health and functioning measurement instrument, the health and functioning concepts were further reviewed for relevance and clarity, and a different, refined, reduced, and synthesized list of 36 concepts was taken forward for evaluation in round II (refer to Supplementary Information and Table S1 in Multimedia Appendix 1 for further details).

Round II

Of the 36 concepts in this round, 14 made up 50% of the total scaled score (Figure 5). In general, physical health symptoms, and worries about the impact on health concepts remained the highest-ranked concepts and were considered by experts as more important than aesthetic considerations. Experts commented that concepts rated as unimportant and ranked lower in the logit analysis tended to lack specificity to smoking (ie, could be due to other causes), did not directly affect health, or were not reported frequently by TNP users. Of the 36 concepts evaluated in round II, a final selection of 20 highest-ranked concepts were identified for validation and discussion in round III.

‎

Round III

Of the final 20 concepts ranked in round III (Figure 6), the sum of the top 4 concept scores accounted for more 50% of the total score: cardiovascular symptoms, shortness of breath or dyspnea, chest pain heaviness, and worry about smoking-related diseases and impact on health in general. Physical health concepts remained the most important experts across all 3 rounds

‎

Latent Class Analysis

Latent class analysis of the results in round III revealed 2 identifiable subgroups, each with its own ranking and ordering of the final 20 concepts (Textbox 2). In both cases, concepts related to physical health, particularly cardiovascular and respiratory symptoms, ranked highest. In subgroup 1, the lowest-ranked concepts were related to psychosocial or general well-being and social interactions, and all scored equally low. For subgroup 2, there was minimal differentiation between concepts beyond the highest-ranking group.

In stage 2 of round III, following the MaxDiff analysis, the ranked concept lists for each of the 2 latent class subgroups were shown to the experts, who were asked to identify which group best reflected their own thinking and provide a rationale for their choice. Just over half of the experts (58/100, 58%) reported that subgroup 1 best reflected their thinking compared with (42/100, 42%) for subgroup 2. When asked about their rationale for aligning with subgroup 1, the main reason was a focus on the most important physical health symptoms (33/58, 57%). The main rationale for choosing subgroup 2 was a greater emphasis on the importance of the consumer and their perspective (10/42, 24%), including psychological elements, the well-being of consumers, and impact of smoking on the health of others (14/42, 33%).

Regarding the influence of professional specialty in self-alignment with subgroups (Table 3), preference for subgroup 1 tended to be among oncologists (14/19, 74%), dentists (11/15, 73%), counselors (4/6, 67%), and general practitioners or internal medicine specialists (17/27, 63%). Conversely, all policy advisors and charity workers (5/5) felt that subgroup 2 best reflected their views. When geographical location was considered (Table 3), there was a strong preference for subgroup 2 in participants from Asia (12/17, 71%) and a slight preference among those in North America (9/17, 53%). Experts from all other regions tended mostly to identify with subgroup 1 (Middle East: 5/6, 83%; Africa: 4/5, 80%; Central Europe: 4/5, 80%; Eastern Europe: 7/9, 78%; Western Europe: 9/13, 69%; Southern Europe: 8/13, 62%; and Central or South America: 8/15, 53%). Regarding the final selection of 20 concepts, both subgroups considered all concepts to be important and relevant for inclusion in any self-reported outcome measure.

Additional Semiqualitative Assessment—Likelihood to Perceive Change

In round III (in response to the question: “Now that you have seen the list of concepts collated from Rounds I & II, how likely do you think it is to see a change in this concept when a TNP user switches from a combustible TNP to an sf-TNP?”), most experts reported they would expect to find measurable levels of change in 13 out of 20 individual concepts. Furthermore, over half of the respondents stated it was “likely” or “extremely likely” that a measurable change would occur in these concepts when a TNP user switched from a combustible TNP to a sf-TNP (Figure 7). For example, majority of experts felt that it was “likely” or “extremely likely” to observe a change in gum problems (74/101, 73%), phlegm or mucus while coughing or not coughing (72/101, 71%), general perception of well-being (72/101, 71%), and throat irritation/sore throat (72/101, 71%). In addition, between 5% and 20% considered that switching was “unlikely” or “very unlikely” to result in a change in the severity of any single concept, including cardiovascular symptoms, withdrawal symptoms associated with quitting, shortness of breath, and worry about smoking-related diseases. Reasons given by experts for considering a change in a concept to be unlikely are listed in Table 4 and were mostly related to lack of perceived substantial differences between combustible TNPs and sf-TNPs.

‎

Discussion

Principal Findings

This adapted Delphi panel study was conducted to identify the health and functioning concepts that experts consider to be most relevant and useful for incorporation into a new TNP-specific health and functioning measurement instrument to assess the self-reported impact of switching from combustible TNPs to sf-TNPs [21]. We recruited a geographically and professionally representative panel of experts who routinely treated those who smoke or communicated on smoking and TNP use. They were asked to reduce and refine a preliminary list of health and functioning concepts and to rank the final list in order of importance. After 3 rounds of investigation, the initial list of 69 concepts was refined to the most important 20 they considered could be included in a new self-reported measure (Table 5). The range of final concepts reflected health and psychosocial issues associated with smoking, including respiratory and cardiovascular health, withdrawal symptoms, and worries about the impact on health of self and others.

Overall, the final 20 concepts were considered both clinically relevant and important to the experts in their evaluation of the possible impact of switching to sf-TNPs or stopping smoking cigarettes. We observed that the concepts consistently ranked highly among experts focused on objectively measurable health consequences such as respiratory and cardiovascular symptoms, as well as overall physical functioning. In contrast, concepts such as the ability to cope with stress, risk of getting colds, oral health, physical appearance, social functioning, and sensory impacts (taste, sense of smell, and odor) were ranked as less important. Those concepts ranking highest reflect current evidence regarding smoking-related health outcomes and potential improvements attainable by ceasing cigarette consumption or switching to sf-TNPs [6,41-46]. The experts’ knowledge of evidence for the effect of cessation or switching on pre-existing respiratory conditions [6,43,44,47,48] may also be reflected in the ranking.

It is essential to note that the relative ranking of these concepts is likely to be different for TNP consumers compared with health care professionals. Individuals trying to stop smoking or switch to sf-TNPs recognize the respiratory and other physical health and functional benefits and understand the potential for improved quality of life by reducing cigarette consumption or stopping smoking [49,50], as well as reduced exposure of others to smoking-related harm [6,51]. They also tend to report benefits and preferences for general hygiene and smell, better oral health, and sensory improvements [6,41,43]. However, no concepts related to general physical appearance or hygiene were in the final top-ranking concepts of the current study. These results explicitly reflect the participants’ own experiences and opinions, which may differ from those of TNP users and be dependent on TNP user characteristics.

Although objective health outcomes such as cardiovascular and respiratory function were rated highly by experts throughout the study, some individuals rated psychosocial outcomes and effects on families and others (through secondhand smoke and so on) as equally important as objective physical measures. Latent class analysis in the final round of the study suggested that geographical location, cultural milieu, and professional specialism may each play a major role in these observations. Specifically, we identified a subgroup that focused mostly on physical, objectively measurable concepts (eg, cardiovascular symptoms, shortness of breath, chest pain, neurological symptoms, and cough), compared with the other subgroup, which considered both objective and subjective or emotional concepts (eg, worry about smoking-related diseases and impact on health both of self and others) to be of equal importance. These results may correspond with established patterns of perceptions and considerations among Western cultures, which may be broadly defined as individualistic (ie, driven by personal goals), whereas those from Asia and South America can be defined as collectivist (reflecting the primacy of mutual obligations among members of society) [51-53]. Furthermore, qualitative investigation of alignment with the subgroups revealed that for certain job roles (eg, counselors and policy advisors), the consumer-focused concepts were more important than for other health care professionals (eg, specialist clinicians), who were primarily concerned with objective, physical concepts. These findings have important implications for the development of a self-report measure that can be widely used and is adaptable to individual or local requirements [54]. When disseminating results based on the newly developed self-reported measure, it may be essential to tailor communications on sf-TNPs for specific audiences with different considerations of what is clinically meaningful to the experts themselves, as well as to the individual TNP user [26,33,54]. To validate the latent class findings in round III, we asked respondents to self-identify with the most relevant of the 2 subgroups. Overall, participants were able to do so readily, indicating that the 2 classes were culturally and professionally relevant, and that the new self-report measure will maximize content validity by including the range of concepts identified and ranked in round III to accommodate cultural and professional requirements and differences.

There are 3 main strengths of this study. First, we used the MaxDiff ranking to address the challenge of ranking an initial high number of concepts. Second, the incorporation of an anchor was essential in objectively identifying concepts for elimination. Third, a large panel of experts representing a broad range of professional expertise and geographies was recruited for this study. This allowed us to confirm validity of the concepts identified in the preparatory phase of the development of the new measure. It also enabled us to parse out cultural and professional subtleties that are important to consider when disseminating outcomes from the new measure. Development of an outcome measure involving both the target population and clinicians is a key component of creating a high-quality instrument [33,54]. Building this manner of collaboration and bridging between TNP users and relevant experts in this Delphi panel into the development of this new instrument would serve to enhance the potential quality and validity of the final measure by feeding into subsequent qualitative and psychometric evaluations [26].

There are also some limitations to our study. First, we did not use a strict Delphi panel process, and round I may be seen as a preliminary evaluation rather than a pure ranking and scoring exercise. Delphi panel methodology typically follows a highly structured process. In this particular case, it would have been ideal to perform additional preliminary qualitative insight work to review all the elements of a study, formally review and refine concept terminology, and test with external audiences. And, as previously mentioned, this Delphi panel focused on the experience and perceptions of experts and professionals who interact with TNP users. Consequently, the findings reported here do not have the vital context of TNP consumers opinions and perceptions. Unfortunately, it would have been impractical to address both angles in a single study with an initial list of 69 concepts. Instead, further research should consider evaluating TNP users’ perceptions and rankings in a separate study using similar methodology. In addition, there is a potential for selection bias in the study due to the exclusion of experts who specified they would not consider participating in studies by tobacco companies. This exclusion could result in a sample that is not fully representative of the broader expert community, potentially impacting the generalizability of the findings. To mitigate this, we used a diverse recruitment strategy to include experts from various regions and specialties. However, the views of nonparticipating experts might differ from those who participated, and this limitation should be considered when interpreting the results. Finally, the survey was conducted in various languages, and despite the measures taken to ensure the consistency and validity of the translated surveys, linguistic and cultural differences may have led to variations in interpretation of the survey. These variations could potentially affect the study’s findings and should be considered when interpreting the results.

Conclusion

In conclusion, this 3-round, adapted Delphi panel identified a ranked list of 20 concepts to be considered when assessing the health and functioning status of individuals who stop smoking cigarettes or switch to using sf-TNPs or NRTs. It is to be expected that the scale of importance of each concept will vary based on the health status and concerns of an individual, whereas the ranking presented here represents a global, generalized view provided by the participating experts. In addition, the sensitivity of the concepts to accurately reflect changes in TNP use behavior will need to be determined. This would support the evaluation of the self-reported experience and impact of switching from conventional cigarettes to sf-TNPs on health risks and contribute to the regulatory and scientific evidence base for understanding both the individual and public health impacts of sf-TNPs.