A total of 20 wheelchair users and 20 people without a disability were included in the study. Both groups consisted of 11 male participants and 9 female participants and had similar demographic characteristics (Table 1). Participants were included if they were aged between 18 and 60 years and without injury or other health issues that could be aggravated by physical exertion. Included in the wheelchair user group were individuals that used a manual wheelchair as a main form of transport or were ambulatory wheelchair users. The wheelchair user group was comprised of individuals with SCI (n=11), spina bifida (n=2), and cerebral palsy (n=2). A total of 5 participants had other neurological, musculoskeletal, or joint impairments. Participants were recruited from sports associations and organizations for people with disabilities in Norway and social media.

Three test days of wheelchair propulsion with different treadmill incline-speed combinations were conducted within 2 consecutive weeks. A minimum of 24 hours separated each test day, and sessions occurred at approximately the same time of day to account for diurnal variations. All test days started with a 5-minute warmup at a 0.5% incline at a self-chosen speed that corresponded to a rating of perceived exertion of 7-9 on the Borg scale [25]. Then, 3 standardized 4-minute stages were performed at a predetermined incline for the day (either 0.5%, 2.5%, or 5%) with increasing speed across the stages. The order of the test days was counterbalanced. The speed at each incline was established through pilot testing and determined to be manageable for the participants (Table 2). Anthropometric data (age and sex) were collected before testing, and body mass and height were collected on the first test day. Participants were instructed to avoid high intensity training and alcohol consumption 24 hours before testing, avoid caffeine on the day of testing, and fast for at least 2 hours before testing.

Participants’ body mass was measured using a Kistler force plate (Kistler 9286BA; Kistler Instruments AG) before the first test day. Body mass was determined for participants in the wheelchair user group while seated in their own wheelchair and obtained by subtracting the mass of the individual wheelchair (range 6.5-18.2 kg). All people without a disability were weighed while standing without any equipment. Participants wore a facemask (7450 V2 Series; Hans Rudolph Inc), which was connected to a Vyntus CPX ergospirometer with a mixing chamber (Vyaire, Medical GmbH) to measure gas exchange as 10-second averages, from which the criterion device EE was calculated. The Vyntus CPX was calibrated against a known gas mixture of 15% O₂ and 5% CO₂ before every test. Participants were fitted with a Polar HR monitor (version M400; Polar Electro Oy) and a Polar chest strap (version H10; Polar Electro Oy), which served as the criterion device for HR.

Participants wore 2 SWs on their nondominant wrist: an AW Series 4 software version OS 7.3.3 and a Fitbit Versa (2017) software version OS 5.0. The SWs tracked HR using photoplethysmography, and both SWs had a built-in accelerometer and gyroscope. The SW placement (closest to the wrist) was counterbalanced. Participant characteristics were entered in the devices, and the activity settings of “outdoor push walking pace” (AW) and “treadmill running” (Fitbit) were used throughout testing. “Treadmill running” was chosen for the Fitbit in the absence of a wheelchair-specific setting. wheelchair users used their personal wheelchair, and people without a disability used a standardized wheelchair (Küschall K-Series Attract; Invacare; mass 11.7kg). The wheelchairs were secured on a motorized 5×3 m treadmill (Forcelink Technology) with a mobile traverse bar attached to side rails (Figure 1). The side rails were equipped with safety stoppers to prevent participants from rolling off the back of the treadmill.

Blood lactate concentrations (mmol/L) were measured in a rested state and after every stage from a 20-μl blood sample obtained from the participants earlobe [26]. Concentrations were analyzed using the Biosen C-Line Sport lactate measurement system (EKF Industrial Electronics). Rating of perceived exertion was obtained for muscular, respiratory, and total effort on the 6-20 Borg scale after each stage [25].

The criterion EE was calculated for every 10-second average VO₂ and VCO₂ values using the Weir formula [27]:

An average over the entire 4-minute period (as opposed to a steady-state average) was calculated for EE and HR for each stage of the criterion device. This was done since the SWs displayed an estimated average EE and HR for each entire stage. It was not possible to retrieve data with a higher time resolution from the SWs. The EE values provided by the criterion device and SWs were converted to kcal/min for comparison. Lastly, incomplete 4-minute stages were excluded from the analyses.

In total, 20 of the 360 performed stages were incomplete and excluded from the analyses. Incomplete stages mostly occurred at the highest speeds and often at the 5% incline, which was due to a combination of an age- or disability-related lack of physical fitness or upper-body strength. Additionally, equipment failure or human error contributed to the following missing data: incorrect activity setting (AW, n=2), no HR recorded after activity (AW, n=32), and a lost HR signal (criterion, n=1).

Statistical analyses were conducted, and visualizations created in R Studio (version 4.2.1; R Core Team) [28]. Descriptive statistics were calculated for EE and HR for criterion devices, AW, and Fitbit and visualized with box plots using the R Studio package ggplot2 (Multimedia Appendix 1).

The data collection and processing were approved by the Norwegian Centre for Research Data (216680) and conducted in accordance with the Declaration of Helsinki. All participants were informed of the study purpose, design, potential risks, and the possibility to withdraw without penalty before signing the consent form. Participation was voluntary and without financial compensation beyond insight into the individual’s collected data. The collected data were deidentified.

For the EE reported by the AW Series 4, the MAPE (with all inclines and stages combined) was 27.4% (SD 16.7%) in wheelchair users and 32.1% (SD 14.4%) in people without a disability. The EE provided by the Fitbit Versa had a MAPE of 73.9% (SD 57.2%) in wheelchair users and 44.7% (SD 37.8%) in people without a disability. Absolute agreement based on the MAPE for each incline-stage combination was mostly poor (Figure 2).

For HR, the MAPE with all stages and inclines combined was 8.5 (SD 10.4%) in wheelchair users and 8.1 (SD 13.6%) in people without a disability for the AW, and 17.4 (SD 12.4%) in wheelchair users and 14.3 (SD 10.7%) in people without a disability for the Fitbit. The absolute agreement for HR in each incline-stage combination was moderate-good for the AW and poor-moderate for the Fitbit (Figure 2).

The mixed model analyses indicated that EE was underestimated by the AW and overestimated by the Fitbit, while both SWs underestimated HR (Figure 3). Additionally, the differences between criterion and comparison devices increased negatively with higher exercise intensity for EE and HR (all comparisons, P<.001; Figure 3 and Multimedia Appendix 2). This led to reduced accuracy in the AW (larger underestimation) and improved accuracy in the Fitbit (lower overestimation) with increased intensity. EE and HR differences between SWs and criterion devices were not significantly different in wheelchair users compared to people without a disability in most comparisons (P>.06), with the exception of the HR reported by the AW (–5.27 beats/min; P=.02). Furthermore, for EE, the differences were significantly larger in female participants for the AW (–0.69 kcal/min; P<.001) and in male participants for the Fitbit (–2.08 kcal/min; P<.001). For HR, the only sex difference was found for the AW, with smaller differences between the AW and criterion device in female participants (5.23 beats/min; P=.02).

Apart from 1 EE (Figure 4; a 5% incline for stage 3) and 1 HR ICC (Figure 4; a 0.5% incline for stage 2), all remaining ICCs indicate poor to moderate relative agreement between the criterion devices and SWs (Figure 4). The ICCs for each incline-stage combination for the AW had a range from 0.12 to 0.57 for EE and from 0.11 to 0.86 for HR. For the Fitbit, the corresponding ranges were from 0.06 to 0.85 for EE and from 0.03 to 0.29 for HR.

The aim of this study was to assess the accuracy of the AW Series 4 and the Fitbit Versa for estimating EE and HR during wheelchair propulsion at different intensities. The AW underestimated EE and the Fitbit overestimated EE, suggesting that neither of the SWs are accurate enough for estimating EE in wheelchair users. Furthermore, both the AW and Fitbit underestimated HR. Lastly, the differences in HR and EE between SWs and criterion devices increased with increasing intensity, and they mostly did not differ between groups.

The MAPE of approximately 30% and poor to moderate ICCs of the AW Series 4 are similar to the ones reported by Moreno et al [24] for the AW Series 1. Since the participant characteristics were similar those in this study, there seems to be no improvement in the AW’s EE estimation algorithms. The SW algorithms are proprietary technology, and it cannot therefore be determined why the AW underestimates EE for all participants. A plausible explanation is that Apple intentionally chose to report lower values for the sake of obesity prevention. Another possible explanation is that the data for developing the AW algorithms were collected from wheelchair users with lower training status or higher levels of SCI, who have a lower EE compared to the wheelchair users tested in this study. If the latter is the case, injury and fitness levels are important factors to consider when estimating the EE of wheelchair propulsion.

In this study, the AW was found to be equally inaccurate when estimating EE for wheelchair users and people without a disability, a finding that also aligns with Moreno et al [24]. Since the AW does not request user information on, for example, impairment levels, this finding indicates that the watch’s software is not capable of identifying these types of individual characteristics from other factors such as movement patterns or HR responses. In contrast, sex could be inputted into the AW settings, and we found a significantly better estimated EE for male participants. Preliminary evidence [32] indicates that algorithms for wearable devices are developed mostly based on reference data from male participants and may therefore be less accurate for female participants.

The AW’s underestimation of EE increased with intensity, which contradicts findings from previous AW studies [24,33]. While Pope et al [33] reported lower accuracy at moderate compared to low and high intensity during running, Moreno et al [24] reported consistent accuracy across wheelchair propulsion stages with an increasing stroke rate. The test protocol with the standardized stroke rate increases in Moreno et al [24] might more closely resemble the way the AW estimation algorithm works. Possibly, the AW uses the accelerometry data to determine how much exercise intensity, and thereby EE, has increased. In contrast, in this study, intensity increased with higher speeds at a given incline, with the steeper incline days being more physiologically taxing with a larger anaerobic contribution than the flatter incline days. Furthermore, it is possible that the AW EE estimation algorithms were developed mostly based on low-intensity data or without a physiological intensity measure (eg, HR) since wheelchair users spend most of their day at or below low-intensity exercise levels. In line with this, preliminary findings of our research group [34] indicate that estimation algorithms developed for wheelchair users perform less well on high-intensity data if they are only developed based on low- to moderate-intensity data.

The Fitbit did, in contrast to the AW and other SW studies on EE [24,33], show a systematic decrease in the error with higher intensities for both groups, which resulted in lower overestimations. This finding was most likely related to using the “treadmill running mode” in the absence of a wheelchair-specific setting. This setting leads the estimation algorithm to expect a weight-bearing and leg-dependent activity with higher muscle activation. However, wheelchair propulsion is a non–weight-bearing activity that allows longer rest between cycles, especially at lower speeds and inclines. The reduction of the Fitbit error at higher intensities might therefore be a result of more active muscle mass, faster cycle rates, and longer cycle lengths with increased incline or speed. As such, the physiological effort of wheelchair propulsion may be more similar to running at higher intensities. Although the Fitbit displayed improved accuracy at higher intensities, the MAPEs were far greater than the ±3% acceptable accuracy cutoff, and the Fitbit should therefore not be used to estimate EE in wheelchair users.

Both SWs showed better accuracy for HR compared to EE. However, only the AW had a MAPE below the arbitrary cutoff of ±10% that is commonly used for acceptable accuracy in free-living activities, with no values below the ±3% cutoff for standardized settings [18,21]. Both SWs additionally showed reduced accuracy at the highest intensities, which is in agreement with previous findings of HR measured from wrist-worn devices during running [22]. Furthermore, wheelchair users were found to have a larger underestimation of HR (ie, –5 beats/min) compared to people without a disability. While the reasons for this are somewhat unclear, it seems like this is due to more negative outliers in wheelchair users. Overall, the high MAPE variance and mostly poor to moderate (AW) and poor (Fitbit) ICCs indicate a high risk of individual inaccuracy when monitoring HR from these wrist-worn SWs during wheelchair propulsion.

Two main factors need to be addressed for better EE and HR estimation algorithms in wrist-worn SWs: (1) the sensor hardware and (2) the sensor software (estimation algorithm). The sensor hardware of current wrist-worn technology is not capable of reporting precise or consistent HR signals during activity, which was partly highlighted by the missing data in this study. With regard to further improving the estimation algorithms, an assessment is needed on the impact and relative importance of factors such as personal characteristics (sex, age, body mass, training status, etc) or more wheelchair users–specific aspects such as type and level of impairment. These additions may increase estimation accuracy and reduce variation within the highly heterogeneous wheelchair user group, even without hardware improvements.

Furthermore, investigation into appropriate cutoffs for acceptable accuracy of EE and HR provided by SWs is needed. The commonly used cutoffs (±3% in standardized and ±10% in free-living settings) are based on previous research assessing step count during walking [35-38] or pushes during wheelchair propulsion [39]. It may be plausible to establish higher cutoffs for EE or HR, especially when testing a heterogeneous wheelchair user group during an upper-body activity. However, a standardized low cutoff is essential to avoid differentiating acceptable accuracy between parameters in both controlled and free-living settings. Accurate estimates are also crucial for various populations, for example, to address an imbalance between energy intake and expenditure in athletes trying to regulate nutrition for performance purposes or in wheelchair users attempting to prevent or counteract obesity. Therefore, it is likely the ±10% error cutoff for free-living activities is too high.

Lastly, the effect of filtering done by the AW and Fitbit on the EE or HR data deserves mentioning. The AW removed many average HR values, which is likely attributed to a low number of data points. Comparatively, Fitbit reported data for all activities, although with lower accuracy. For now, we advise the use of HR belts for increased accuracy of the parameters investigated.

Neither the wrist-worn AW nor Fitbit were sufficiently accurate for estimating EE or HR during wheelchair propulsion. The AW underestimated EE while the Fitbit overestimated the EE across all incline-stage combinations. The underestimation of the AW increased and the overestimation of the Fitbit decreased with higher intensities, suggesting that neither watch sufficiently adjusts for the change in intensity. Additionally, both SWs underestimated HR. High MAPEs were found for both SWs and parameters (ie, EE and HR), in addition to the poor relative agreement indicated by low ICCs. Furthermore, neither the wheelchair-specific algorithm for estimating EE nor its ability to differentiate between wheelchair users and people without a disability have been improved for the AW Series 4 as compared to the previously investigated AW Series 1. Overall, our findings suggest that caution is required when using SWs as a tool for training intensity regulation and energy balance or imbalance in wheelchair users.