The questionnaire was drafted in accordance with principles outlined by prominent research organizations (eg, ISPOR) and United States Food & Drug Administration guidance. Content for the initial RSES was generated based on the review of relevant literature, including existing respiratory symptom or cough questionnaires and national or international surveys. Feedback from subject matter experts, including 5 pulmonologists, was used to refine the draft questionnaire.

Next, the draft questionnaire was evaluated through individual semistructured cognitive debriefing interviews with a total of 49 smokers and switchers. Participants were diverse with respect to demographics and included individuals with low health literacy. Participant feedback was used to rectify potential sources of confusion or response error in the questionnaire, to modify item content to enhance relevance and meaningfulness to participants, and to provide evidence of content validity.

The RSES generated from cognitive testing is presented in Textbox 1. The RSES is written below an eighth grade reading level (Flesch-Kincaid Grade Level=7.7).

Smokers, switchers, and former smokers were recruited from varied locations around the United States from opt-in consumer research panels maintained by Kantar Profiles. Individuals sign up for such panels, agreeing to be invited to studies as they come up and to be compensated according to the panel’s practices. Soft target quotas were used to ensure a diverse sample of participants.

The participant inclusion criteria were as follows: must be >31 years of age (to allow for 10 years of smoking as an adult), reside in the United States, and have access to a device with internet access to complete the survey. The exclusion criteria were as follows: have a first-degree relative who is a current or former employee of the tobacco or e-cigarette industry, have a household member in litigation with a tobacco or e-cigarette company, and have participated in marketing research pertaining to tobacco or e-cigarettes in the past month.

The study was approved by the Sterling institutional review board (protocol 8613, approved January 7, 2021). All participants signed a web-based informed consent document that accurately described the study procedures and their risks and benefits. Participants in the initial qualitative testing were compensated with US $65. Participants in the quantitative phase were compensated with “panel points” redeemable for goods in accordance with the standards of the research panel they had signed up for; these usually vary with the length of the survey. For the initial survey, this varied between US $1.00 and $4.50 for different panels. For the retest assessment, this varied from US $0.80 to $4.00. Study data were provided to the sponsor in a deidentified form and held securely and confidentially.

A GRM [17] was fit in Mplus using the weighted least-squares estimator and θ parameterization. The GRM’s assumption of the ordinality of response categories was empirically evaluated by fitting a generalized partial credit model [18,19]. To evaluate the GRM’s assumption of unidimensionality, a Monte Carlo simulation (parallel analysis) with 10,000 randomly generated data sets was conducted using a principal components analysis extraction method. Test information and reliability were evaluated within the context of the GRM. The stability of the RSES was evaluated by calculating an ICC between RSES scores from the time 1 and time 2 surveys. Convergent validity was evaluated by calculating a Spearman correlation between RSES scores and self-reported global health status and by evaluating RSES scores by diagnostic status via independent-samples t tests.

As an indirect indication of the RSES’s ability to detect change, differences between smokers’ and former smokers’ scores were evaluated via an independent-samples t test. Similarly, using independent-samples t tests, exploratory analyses evaluated differences between switchers’ RSES scores and those of smokers and former smokers. To estimate the minimally important difference, the difference in adjusted (least-squares) mean RSES scores between those with and without respiratory symptom–relevant diagnoses was evaluated using a linear regression model controlling for tobacco use status (ie, study group membership). Finally, a receiver operating characteristic (ROC) curve was generated to determine the optimal RSES score cutoff in predicting self-reported respiratory symptom–relevant diagnosis.

In total, 55,572 panelists were invited to participate. Of the 14,158 who started the recruitment screener, 1907 passed. The first 610 of these were enrolled in the study. All 610 participants started and completed the survey and comprise the analytic sample of 202 smokers, 208 switchers, and 200 former smokers.

Participant characteristics are presented in Table 1. Approximately half (285/610, 46.7%) of participants reported 1 or more diagnoses; 17% (104/610) reported a diagnosis of COPD. Per study requirements, all participants reported having smoked for at least 10 years, with a mean smoking history of 32.9 (SD 14.4) years. On average, smokers reported having smoked for 41.9 (SD 11.7) years, whereas switchers and former smokers smoked for approximately 28 years (mean 28.8, SD 13.8 years and mean 28.0, SD 13.2 years, respectively). Not surprisingly, switchers were on average younger than smokers and former smokers (mean 54.6 years vs mean 65.0 and 67.6 years, respectively), as the use of ENDS is more common in younger cohorts [20,21]. Nearly all smokers reported daily use of cigarettes (188/202, 93.1%), and switchers reported daily use of ENDS (193/208, 92.8%), respectively. Most switchers reported having used ENDS for more than 1 year (187/208, 89.9%). Switchers were using a variety of ENDS brands.

In total, 145 participants who completed the time 1 survey completed the time 2 survey. Of these, 128 participants who reported not being sick with a cold- or flu-like symptoms at either time point comprised the retest sample.

Observed category averages and category thresholds derived from the generalized partial credit model were ordered as expected, providing empirical evidence that it requires a higher frequency of respiratory symptoms to endorse a more severe response option (eg, “Every day” vs “Most days”).

Results from the parallel analysis revealed a single significant factor (eigenvalue=3.14), providing support for unidimensionality of the RSES. This factor accounted for 62.7% of the variance.

Fit statistics for the initial GRM indicated a poor fit of the data to the model (χ²₅=285.086, P<.001; comparative fit index=0.963; root mean square error of approximation=0.303; standardized root mean square residual=0.085). Because of the conceptual similarity between items, in conjunction with large observed modification indices, the model was rerun allowing for 2 correlated errors (“Morning Cough” and “Cough Frequently” and “Shortness of Breath” and “Easily Winded”). This model exhibited an acceptable fit (χ²₃=3.056, P=.38; comparative fit index=1.000; root mean square error of approximation=0.006; standardized root mean square residual=0.005).

The RSES items’ discrimination parameters were approximately 1 or higher, suggesting that the items were effectively differentiating between respondents with different levels of respiratory symptoms. See Table 2 for the RSES items’ discrimination and difficulty parameters.

The test information function produced from the GRM (Figure 1) indicates that the RSES most precisely estimates respiratory symptoms from θ of 0.2 to 2.4. The reliability function (derived from the test information function by dividing information by information + 1) [22] indicates that the RSES exhibits a reliability of 0.80 or higher from θ of −0.40 to 3.00.

Given the high correlation between the GRM-derived scoring and the raw scores (r=0.94) and the complexities associated with using scoring from a 2-parameter item response theory model, it is recommended that researchers use the raw RSES item scores to calculate a composite (mean). Therefore, raw scores were used for the remaining analyses.

Test-retest reliability was good (absolute ICC=0.89).

Smokers’ RSES scores were significantly higher than Former Smokers’ scores (t₄₀₀=3.87, P<.001; d=0.39, a small to medium effect size [23]), providing initial support for the RSES’s ability to detect change over time (Table 3). Given the observed differences in years smoked between study groups, these analyses were replicated using linear regression controlling for years smoked. The adjusted (least-squares) means were similar to the unadjusted means (smoker mean 2.00, former smoker mean 1.79), and the groups’ RSES scores remained significantly different (P=.02). Sensitivity testing using the nonparametric Mann-Whitney U test yielded the same statistical conclusion.

Switchers’ RSES scores were significantly lower than smokers’ (t_374.84=4.59, P<.001; d=0.46, a small to medium effect size [23]; Table 3) and did not differ from former smokers’ (t_382.44=0.34, P=.74; d=0.03). These conclusions remained when regression models controlled for the number of years smoked: adjusted (least-squares) means were similar to the unadjusted means (switcher mean 1.76), and the RSES scores of smokers and switchers (P=.005) remained significantly different, whereas the scores of switchers and former smokers did not differ (P=.68).

As anticipated, higher RSES scores were related to poorer self-reported health status (r_s=0.38, P<.001). With respect to known-groups validity, RSES scores were significantly higher among participants who reported 1 or more respiratory symptom–relevant diagnoses (mean 2.16, SD 0.92) compared with those who did not (mean 1.58, SD 0.64; t_496.07=8.86, P<.001, d=0.74, a medium to large effect size). RSES scores were also significantly higher among participants with COPD (mean 2.76, SD 0.92) than those without COPD (mean 1.67, SD 0.68; t_126.94=11.55, P<.001, d=1.52, large effect size). Nonparametric testing (Mann-Whitney U test) yielded the same conclusions.

In linear regression with self-reported COPD and non-COPD diagnoses (ie, asthma, allergies, congestive heart failure, and obesity) as predictors, these diagnoses accounted for 29.2% of the variance in RSES scores. COPD diagnosis accounted for the majority of this variance (20.6%; P<.001) in models adjusting for non-COPD diagnoses. Controlling for age and years smoked did not have a material impact on results, with a small increase in the overall variance accounted for by the model (31.1%), and COPD diagnosis was still significantly related to RSES scores (P<.001) and explained 17.0% of the variance.

Finally, to directly quantify the relationship of RSES scores to self-reported COPD diagnosis, a logistic regression was run with COPD diagnosis as the outcome and RSES scores as univariate predictors. The analysis showed that with every 1 unit increase in mean RSES scores, the odds of COPD were 4.72 times greater (95% CI 3.47-6.40; P<.001).

The difference in adjusted means for those with (least-squares mean 2.16, SE 0.05) and without (least-squares mean 1.58, SE 0.04) respiratory symptom–relevant diagnoses was 0.57 (P<.001). These results suggest that changes of about half a point or more in RSES scores reflect meaningful differences or changes in frequency of respiratory symptoms.

The ROC analysis yielded an area under the curve of 0.69 and identified a score of 2.0 as the optimal cut-point for identifying diagnosed individuals (specificity=0.76, sensitivity=0.57).

The RSES was developed and validated in electronic form, and electronic administration is recommended. RSES instructions are presented on a screen before item 1 (Textbox 1). To facilitate administration on a small-screen device, each RSES item is administered on a separate screen with the following instructions on each screen to remind participants of the recall period: “Over the past 30 days, how often did you experience the following?” RSES items are administered in a fixed order (Textbox 1), and the response option order is also fixed. Given the high correlation between the GRM-derived scoring and the raw scores observed here (r=0.94) and the complexities associated with using scoring from a 2-parameter item response theory model, it is recommended that researchers use the raw RSES item scores to calculate a composite. A composite score is calculated by taking the average of the 5 RSES items. If an item is missing, a composite should not be calculated.

The RSES fills an important gap in the existing toolkit of respiratory symptom patient-reported outcome measures, as it is a validated questionnaire appropriate for use with adult current and former smokers without a formal diagnosis of COPD.

The RSES is likely to be useful in many contexts, including assessing the health consequences of tobacco product use and the relatively immediate health benefits of quitting smoking. Demonstrating the improvement of respiratory symptomology associated with quitting smoking has implications for public health, as the frequency of respiratory symptoms has been shown to be associated with increased motivation to quit smoking [24-27]. Accordingly, ameliorating respiratory symptoms may be a particularly motivating driver for smokers to stop smoking, as respiratory symptoms may be experienced as tangible and relatable health effects that reflect more immediate consequences of quitting smoking.

RSES development and validation was a rigorous multiphase process conducted in accordance with guidance and best practices. The RSES is understood and correctly interpreted by diverse samples of respondents with respect to age, ethnicity, and health literacy status. Results from the psychometric evaluation provide evidence of unidimensionality, high levels of measurement precision across a range of relevant symptom levels, convergent validity, and stability over a 1-week interval.

Although this study’s cross-sectional design did not permit evaluation of intraindividual change in respiratory symptoms with smoking abstention, observed differences between smokers’ and former smokers’ RSES scores suggest that the RSES may be sensitive to changes that occur when a smoker stops smoking. Former smokers who switched to ENDS reported significantly lower respiratory symptoms than current smokers, even after accounting for years of smoking. Moreover, switchers’ level of self-reported respiratory symptoms did not significantly differ from former smokers who no longer using any tobacco products. These results suggest that switching from cigarettes to ENDS may improve smokers’ respiratory symptoms.

Our study included smokers who switched completely to ENDS, finding that their level of respiratory symptoms differed from that of smokers and was close to that of former smokers who were not using ENDS. The scale would also likely be useful for testing respiratory symptoms in smokers who have switched to other noncombusted sources of nicotine, including heated tobacco products [28], which involve inhalation of an aerosol derived from heating—but not burning—tobacco. It would also be of interest to assess respiratory symptoms in smokers who have not switched completely away from cigarette smoking but are engaged in dual use of both cigarettes and ENDS (or heated tobacco products). It is likely that in such dual-user populations, the degree of recovery from smoking-related respiratory symptoms may depend on how much they are still smoking. Observational studies (eg, [29]) suggest that dual users often substantially reduce their cigarette consumption, which could result in reduced respiratory symptoms, though likely to a lesser degree than switching completely away from smoking. Future studies might also fruitfully include a group of never smokers as a reference point for comparison with groups with a history of smoking.

The RSES significantly differentiated those with and without relevant respiratory conditions, even after controlling for smoking status, and an average score of 2.0 (at least monthly experience of symptoms) can help identify individuals with meaningful symptoms.

COPD diagnosis was significantly related to RSES scores above and beyond other respiratory symptom–related diagnoses, suggesting that the symptoms assessed by the RSES may have some specificity for COPD. Further, each 1 unit increase in RSES scores was associated with nearly 5 times greater odds of COPD. Although the RSES is not intended to function as a screener or diagnostic tool for COPD, results from these analyses suggest that respiratory symptoms assessed by the RSES may be related to eventual pulmonary disease.

The RSES was developed in a way that facilitates electronic administration on devices of different size (eg, a single item was administered on each screen to prevent the need for scrolling, the item stem was administered on each screen along with the item to remind participants of the timeframe of reference, etc). This feature may be useful, as self-report questionnaires may be increasingly administered across various electronic platforms.

Study groups differed in age and years smoked. Although this finding is not particularly surprising (eg, switchers were expected to be younger than smokers and former smokers), statistically controlling for these variables may not completely account for intergroup differences, and there could be other unmeasured differences not included in the analysis.

As this study was not longitudinal, it was not possible to draw causal inferences of the relationships between changes in tobacco use and a subsequent change in respiratory symptoms. Future longitudinal research should evaluate an intraindividual change in smokers’ respiratory symptoms, as assessed by the RSES, after stopping smoking and complete switching to ENDS.

The study relied exclusively on self-report; tobacco use was not biochemically verified, and diagnostic status was self-reported. Additionally, the survey was fielded during the COVID-19 pandemic, so it is possible that COVID-related symptoms added unaccounted variation in symptom reporting.

Research evaluating respiratory symptoms among current and former smokers without diagnosed clinical pulmonary disease has been hindered due to the lack of a self-report questionnaire valid for assessing respiratory symptoms among nondiseased populations. The RSES fills an important gap in the existing toolkit of respiratory symptom patient-reported outcome measures as it is psychometrically validated and appropriate for use among adult current and former smokers without a formal diagnosis of COPD.