The inclusion criteria were active-duty ambulance nurses within the EMS organization with a variation in sex and age to capture a broad range of perspectives. The recommendations based on ISPOR guidelines determined the number of participants that would constitute the sample (5‐8 respondents) [29].

The inclusion criteria were EMS personnel employed by the participating organizations. A census sampling approach was used, where the final sample reflects those who chose to respond. Demographic information collected included gender, age, years of experience in ambulance care, professional occupation, and previous experience with video consultations in health care. Participants in the cognitive interviews were ineligible for the main survey and were therefore excluded from data collection. The sample size was evaluated against published recommendations for Rasch analysis. Previous methodological studies suggest that samples of 50 to 100 participants can provide stable and informative item calibrations in exploratory and pilot Rasch studies, depending on targeting and desired precision. Therefore, the achieved sample of 91 participants was considered adequate for the psychometric evaluation and Rasch analyses performed in this pilot study [40-42].

Data collection was conducted in 2 rounds. The first round took place between November 7 and November 20, 2024, and the second round between December 30, 2024, and January 14, 2025. Potential participants were identified by the EMS directors of the 2 ambulance organizations at the request of the research team, who subsequently contacted the participants by email and invited them to participate in the study. After the participants responded that they wanted to be part of the study, their names were provided to the research team. Each potential participant was contacted individually by email and invited to participate in a cognitive interview. Consent was given when participants agreed to participate in the interview. Written information about the study was provided during questionnaire distribution. Interviews were conducted online using Microsoft Teams, recorded as video and audio, and transcribed verbatim. All recordings were downloaded and securely stored on a university research server and deleted from the Teams cloud to ensure data protection. The research team had no control over the selection of these individuals. A semistructured interview guide based on a combination of “think-aloud” and verbal probing questions was used [27,28]. Cognitive interviews were conducted in 2 iterative rounds to assess the conceptual equivalence and comprehensibility of the culturally adapted UTAUT questionnaire. These can be found in the Multimedia Appendix 1. Round 1 aimed at identifying ambiguities, interpretation, and translation issues. The second round aimed to evaluate the effectiveness of the revision by resolving issues identified in round 1. The same participants were interviewed in both cognitive interview rounds, allowing them to provide feedback on revisions and whether previously identified issues had been addressed. The participants were asked to complete the questionnaire online and to comment on each item with a focus on understanding, clarity, their interpretation of terms, and relevance. Each item was presented separately, with no possibility for the participants to see the entire questionnaire. This allowed participants to describe their understanding of each item and to identify wording that was unclear, ambiguous, or inappropriate for the prehospital context. Probe questions focused on comprehension, judgment, and suitability of response options.

The data collection was conducted from February 13, 2025, to May 1, 2025. The participants in the pilot study were recruited from the 2 different ambulance organizations. A total of 357 employees were invited to participate in the study. The participation was voluntary. Questionnaire data were collected using an online survey tool hosted on a secure platform by Dalarna University. To facilitate questionnaire distribution, a survey link was distributed by the EMS directors to the entire ambulance organization. During data collection, the research team sent 2 reminder emails, and one of the research team members visited every ambulance station of both ambulance organizations to remind participants to complete the questionnaire and to answer questions raised during the data collection period. During the visits, a flyer with a QR code was distributed to remind participants to complete the questionnaire. Participants in the pilot study had the opportunity to provide free-text comments on each questionnaire item.

Interview transcripts were analyzed using charting to enable structural item-by-item synthesis. Each participant’s response was entered in a predefined Microsoft Excel spreadsheet organized by questionnaire item. The responses were then summarized item-by-item and across participants to identify issues or ambiguities. Ambiguities were not treated uniformly; issues reported by multiple participants or impacting the conceptual equivalence were prioritized for revision. Minor linguistic issues were addressed later in the refinement process. Items identified as problematic and suggested for revision by participants were reviewed by the research team, and discussions were held to reach consensus about adjustments and modifications to improve clarity and conceptual equivalence. The revised version was then tested in a second round of cognitive interviews with the same participants from the first round of cognitive interviews. The purpose of the second round was to assess if previously identified issues had been resolved and to confirm contextual relevance and clarity of the revised items.

The psychometric evaluation of the translated and culturally adapted questionnaire was conducted using Rasch analysis based on the partial credit model to evaluate the measurement properties of each UTAUT subscale. Rasch analysis was performed using the easyRasch package [43] in the R statistical computing environment (version 4.5.2; R Core Team), including its dependency packages, which for the Rasch analysis are eRm [44], iarm [45], mirt [46], and psychotree [47]. Differential item functioning (DIF) was assessed using easyRasch to test for statistical differences and RUMM2030 software (RUMM Laboratory Pty LTD) [48] to obtain numerical DIF estimates and graphical displays to support interpretation and verification of DIF, particularly if statistical tests in easyRasch indicated no significant DIF. The easyRasch package required a minimum of 3 responses to generate stable threshold estimates [43]. All questionnaire items were mandatory, and all participants (n=91) provided complete response data and were included in the analysis with no missing data.

Unidimensionality was assessed using fit-statistics, assessment of local independence, and principal component analysis of residuals. The interpretation item fit was primarily based on simulation-based thresholds provided by the easyRasch package and derived from the actual sample size and scale characteristics [49]. Local dependency was assessed by evaluating the simulation-based cutoffs derived under the Rasch model provided by easyRasch [43].

When assessing principal component analysis of residuals, Eigenvalues below 1.5 are commonly considered supportive of unidimensionality [50]. However, given the relatively small sample size of 91 and limited items (3-4) per subscale, this rule of thumb might be overly restrictive. In this study, an Eigenvalue of below 2.0 was considered an acceptable indication of unidimensionality. Ordered response category properties were assessed using item characteristic curve plots and threshold maps. DIF in this study was used only for exploratory purposes, due to the limited sample size (n=91), which could reduce the statistical power to detect meaningful, reliable group-related differences [51]. Mean item location values within ±0.5 were considered well targeted [22,40]. Reliability was primarily assessed by reporting the person separation index (PSI) and Cronbach α, and secondarily by weighting the likelihood estimate by relative measurement uncertainty (WLE-RMU) [52].

A total of 6 active-duty ambulance nurses, 3 from each EMS organization, participated in the cognitive interviews. Most were men (n=5), and the ages ranged from 34 to 63 (mean 47, SD 10.6) years. The interviews in round 1 lasted between 36 minutes and 81 minutes, and those in round 2 lasted between 30 and 51 minutes.

Round 2 cognitive interviews were conducted 6 weeks after the first round of cognitive interviews. Based on the results from round 1, revisions were made to items with interpretation problems. This revision aimed to clarify and align wording more closely to the theoretical model and constructs of UTAUT. Findings indicated an overall improved clarity and conceptual alignment across all constructs. Issues identified in round 1 were mostly resolved, and no new systematic comprehension problems emerged during round 2 (Table 2).

Findings in round 2 indicated that the revisions made after round 1 clarified the technological context by using the term “video consultation” throughout the questionnaire. The term “productivity” in item PE3 was replaced by “availability of ambulance care”; participants could relate the term to available ambulance resources, logistics, and geographical location. No further ambiguities were identified, and the other items were considered satisfactory.

Following the revision of item EE1, the item wording changed from “my commitment” to “clear and understandable.” Participants interpreted this as self-managed, smooth use of technology, and clear guidelines. This change created a minor conceptual nuance shift but retained the main concept of perceived ease of use. The item was therefore retained for further evaluation. Items EE2-EE4 were perceived as clear and understandable, and no additional issues were identified. To enhance readability and linguistic fluency, the research team made minor grammatical adjustments.

Items investigating SI underwent a shift in interpretation after revision. In round 1, participants related the items to coworkers and workplace-related colleagues. Based on the participants’ feedback, items SI1 and SI2 were reformulated to enhance contextual collegial relationship (“People I work with”). This adaptation aimed to reflect the workplace orientation described by the participants during cognitive interviews in round 1, focusing only on professional relationships. Item SI4’s wording was revised and aligned more with the original UTAUT item by specifying the context of prehospital care. Probing questions revealed that the participants interpreted “organization” as the ambulance service and justified a contextual clarification of the item.

In round 2, participants demonstrated a better understanding of the FC items after revision. For item FC1, the word “resource” was replaced by “conditions.” Responses to the probe questions revealed that participants interpreted the new word as both individual and organizational conditions for using video consultations. After the removal of the negative wording in FC3, this item was considered clear and understandable by the participants. Think-aloud and probing data showed that participants understood the item as referring to technical compatibility and integration of the system with existing health care systems.

Findings in round 2 indicated improved clarity and better differentiation regarding future intentions to use video consultations. Based on the responses from round 1, the wording “in the longer term” in items BI1-BI3 was replaced by a specific time frame (“within a year”) and the assumption that video consultation would be implemented by the organization. This clarification shifted the focus among the participants to the intention-related wordings “I intend” (BI1), “I predict” (BI2), and “I plan” (BI3). “I intend” was interpreted as a general intention to use video consultations, whereas “I predict” was interpreted as an expectation to use video consultations, and “I plan” was understood as a more concrete plan to use the technology. Some participants reported some overlap between “I intend” and “I predict.”

A total of 357 employees were invited to participate in the study, of whom 94 completed the questionnaire. Of these, 91 met the eligibility criteria and were included in the analysis, resulting in an inclusion rate of 25.5%.

A total of 91 ambulance nurses participated in this study and were included in the final analysis. The 6 ambulance nurses involved in the cognitive interviews did not complete the questionnaire. The demographic characteristics of the participants are presented in Table 3. The majority of participants were female (50/91, 55%), and a large share of participants had no previous experience with video consultation, although a few reported limited or extensive experience with video consultations.

The initial Rasch analysis of the instrument, with all items, revealed several disordered response categories and did not function as intended. Some response categories had very few or incomplete data for several items. To address this issue, those response categories were collapsed, and all the items were reanalyzed. Response categories for the items EE2-EE4, FC2, BI1, and BI2 were collapsed into 3 categories, while response categories for the items PE1-PE3, EE1, SI1-SI4, and FC1-FC3 were collapsed into 4 categories. The only remaining items with 5 response categories were PE2 and FC4. Results presented in this section refer to the dataset with reordered and collapsed items, where each construct of the UTAUT was analyzed separately. Table 4 presents an overview of Rasch analysis results for each UTAUT subscale after collapsing response categories, whereas Figure 1 provides a person-item map for distributions of person locations relative to item locations on the logit scale for each UTAUT subscale. Item-level fit statistics are presented in Table 5, and a summary of the cognitive interviews and Rasch analyses findings is presented in Table 6. Detailed information on response category curves, item thresholds, and targeting for each subscale is provided in the Multimedia Appendix 1.

The PE subscale showed acceptable fit to the Rasch model (Table 4 and Multimedia Appendix 1). Unidimensionality was supported, with Eigenvalues within acceptable limits and residual correlations showed no substantial local dependency with a cutoff of 0.116. One item (PE1) showed a disordered threshold despite collapsing, which suggests that participants had a limited ability to distinguish between response categories 1 and 2. Item locations covered a reasonable range of the latent trait with a mean person location of 0.71 (SD 1.85), indicating acceptable targeting. No statistically significant DIF was found. The WLE-RMU was 0.811 (95% highest density credible interval [HDCI] 0.756-0.864) with 1000 iterations.

The EE subscale demonstrated an acceptable fit to the Rasch model. Unidimensionality was supported, with no indication of local dependency (Table 4 and Multimedia Appendix 1). Thresholds were ordered, and targeting was acceptable, with a mean person location of 0.74 (SD 1.83). No DIF across sex (female vs male) was reported, and the WLE-RMU was 0.775 (95% HDCI 0.709-0.837).

SI showed an acceptable fit to the Rasch model. Unidimensionality was supported, and residual correlation did not indicate substantial local dependency (Table 4 and Multimedia Appendix 1). Thresholds were ordered, suggesting the appropriate functioning of response categories. Targeting was acceptable with a mean person location of 0.71 (SD 1.56). No DIF across sex was detected, and the WLE-RMU was 0.734 (95% HDCI 0.659-0.806).

The FC subscale demonstrated an overall acceptable fit to the Rasch model (Table 4 and Multimedia Appendix 1). One item (FC1) demonstrated overfit, but this was considered minor and did not affect the overall measurement properties of this subscale. The unidimensionality was supported, and no indication of local dependency was observed. Good targeting was reported with a mean person location of 0.29 (SD 1.70). The WLE-RMU was 0.775 (95% HDCI 0.708-0.836).

The BI demonstrated an acceptable overall fit to the Rasch model (Table 4 and Multimedia Appendix 1) and only 1 item (BI 2) showed overfit. However, this was isolated, did not affect unidimensionality, and no local dependency was reported, indicating no practical impact on the overall measurement properties of this subscale. Targeting was poor, and the mean person location exceeded the threshold (mean 1.24, SD 1.94). The WLE-RMU was 0.752 (95% HDCI 0.678-0.822).

Overall, the findings from the cognitive interviews and Rasch analyses indicated that the item revisions generally improved clarity and understanding. The UTAUT constructs were associated with acceptable Rasch model requirements. However, disordered response category thresholds and item misfit were observed, particularly in the construct’s PE and BI constructs. These findings suggest that cognitive interviews identified interpretation and wording issues, while Rasch analyses provided additional measurement properties information of the adapted questionnaire.

Cross-cultural adaptation for PE goes beyond mere linguistic translation, as cultural variation might influence interpretation. Item revision improved comprehension but shifted the focus of 1 item from individual productivity to organizational resource availability. The term “productivity” is seldom used in everyday EMS clinical practice and might suggest a performance indicator, not acknowledged as individual performance, but rather as system performance [54]. Although the 2 concepts are related, aspects like compliance with structured patient assessment and management, adherence to treatment protocols, and response times could be associated with productivity, especially if the individual actions of an ambulance nurse directly impact those factors [55]. However, these aspects reflect a broader performance dimension that includes teamwork, system efficiency, and resource availability [56]. This could suggest that the availability of an EMS system might capture individual performance and productivity in a broader sense. Nonetheless, an accurate definition of productivity in the health care context remains elusive [57]. The need for response category collapsing, due to sparse response distribution, may indicate that participants experienced difficulties in distinguishing between different response options or a broader conceptual interpretation of the construct PE. This suggests that professional terminology and the operational context of EMS influence how questionnaire items are perceived and interpreted [58-60].

Item revision for the construct EE enhanced comprehension and preserved the core concept of ease of use. Nevertheless, a subtle nuance shift occurred in 1 item (EE1), which might alter the interpretation of the ease of technology use. The interpretation shifted from perceived individual user effort to individual and system usability, with clear guidelines on clinical use. In the context of EMS, this suggests that clear and understandable interaction with video consultation is associated with effortless use within clear clinical application guidelines, which is supported by Candefjord et al [11], emphasizing clear clinical protocols and system design for effortless use in clinical settings. However, both the original and the revised version emphasized the anticipatory expectation of ease and usability [18].

For SI, the contextual adaptations increased the specificity of “important others” to professional peers and members of the organization. In the Swedish version, the wording “persons with whom I collaborate” was used in both items SI1 and SI2 to capture important members of the social group at work, which is in line with the concept of the UTAUT. Although not explicitly defined in the original UTAUT questionnaire, underlying theories of the UTAUT define important people and individuals who influence others as workplace-related reference groups [18,26,61]. Narrowing this group to professional peers might have contributed to acceptable measurement properties supported by Rasch analysis, suggesting the adequacy of contextual adaptation. However, the small residual correlation between items SI1 and SI2 may be related to the shared wording in these items, rather than to a violation of the unidimensionality of the construct.

The FC items were affected by linguistic adjustments, which expanded the interpretation to individual and organizational conditions, and away from the initial interpretation related to logistics. Item FC1 showed an overfit in infit statistics, which may indicate a more predictable response pattern than expected by the model. This might reflect the linguistic adaptation after 2 rounds of cognitive interviews. The removal of negative wording in item FC3 reduced the cognitive load and might have contributed to response category functioning. Negatively worded items can introduce response bias, additional variance, and violate unidimensionality because negative statements tend to be easier to reject than positive ones [62]. However, the revision to a positively worded item was based on feedback in the cognitive interviews and was not guided by Rasch analysis.

The BI item revisions focused on timeliness. In the original UTAUT instrument, BI items include a specific time frame (“I plan to use the system in the <n> month”). As video consultations have not yet been widely implemented in Swedish EMS, the phrasing “in the future” was used to maintain conceptual equivalence. However, this replacement was perceived by respondents as too ambiguous and as redundant temporal wording. Even carefully translated UTAUT statements may not accurately reflect participants’ perception of literally translated items [63]. Instead, and in line with published UTAUT adaptations [64], where a target technology is not yet implemented, alternative formulations were developed to maintain conceptual equivalence while ensuring contextual relevance for EMS. These adaptations were framed for the intention to operationalize video consultations without requiring immediate availability [65-67]. Cognitive interviews guided by the ISPOR principles proved to be essential to ensure semantic and conceptual equivalence, enhancing the content validity of the adapted UTAUT questionnaire for this specific context [29]. The poor targeting may reflect a ceiling effect and that the intention to use video consultation may already be high in this sample of medical professionals, as it was reported earlier in a Swedish primary care context [68]. Item misfit may be related to semantic overlap between items and may lead to a redundant response pattern.

An important methodological insight from this study is that conceptual translation might enhance linguistic clarity and contextual appropriateness but may not necessarily preserve psychometric construct properties. The kind of potential discrepancy between conceptual equivalence and construct validity seen in this study has been reported earlier in cross-cultural adaptation in prehospital care [69], highlighting the complexity of cross-cultural adaptation in health care, and that methodological choices made during the translation process can significantly impact construct validity [70].

The approach to combine cognitive interviews with Rasch analysis in a translation process demonstrates how qualitative and quantitative methods can complement each other. Cognitive interviews describe how individuals interpret items, and Rasch analysis evaluates measurement properties, which can strengthen the validity of a questionnaire. This approach is encouraged by the development of measurement instruments [71]. Previous psychometric validation studies of the UTAUT have typically relied on classical test theory approaches using structural equation modeling as a preferred method [26]. However, a growing body of literature is applying Rasch analysis for psychometric validation of instruments in health care. This is encouraged in health care science [72] as Rasch models are particularly useful for their ability to assess whether items of a scale fit a single underlying construct, thereby confirming unidimensionality [22]. While psychometric evaluations using Rasch analysis in emergency care, compared with general health care, remain limited [16], this study reflects an early stage of implementation and may therefore influence how constructs are interpreted compared with studies already using technology.

Due to the preliminary implementation stage of video consultations in Swedish EMS, this translated questionnaire contained hypothetical statements. To reduce bias introduced by hypothetical statements, several methodological strategies were used. First, cognitive interviews, as described in the ISPOR guidelines, were conducted with ambulance nurses during the adaptation of the questionnaire, allowing for probing of their understanding of hypothetical items, such as phrases like “in the future,” and refining of wordings for conceptual clarity and to reduce the risk of misinterpretation [29]. This step, combined with research panel review, reduced ambiguities that could have increased hypothetical bias. Second, a pilot study was conducted to test the questionnaire with the possibility for respondents who did not participate in the cognitive interviews to further comment on each item individually. Third, Rasch analysis was used to perform a psychometric evaluation of the questionnaire, assessing measurement properties for prospective intention items on acceptance and use of the technology. The collapsing and reordering of response categories facilitated the psychometric evaluation of the instrument. Although some items lacked data in response categories, this might also indicate that respondents experienced difficulties in distinguishing between response categories. Disordered thresholds are recognized as a psychometric issue when rating scales do not function appropriately [22]. Collapsing response categories is a methodologically appropriate practice to improve scale functioning and ensure that each category provides distinct and ordered information about the latent trait of the construct [73]. This approach has also been reported in other Rasch-based evaluations in health care and is not unique to this study [74]. It is an indication that refinement is warranted when instruments are adapted to a new population or a new context. However, this study cannot determine whether response category disorder reflects the participants’ difficulties in distinguishing between adjacent response categories or the small sample size. This should be investigated further and confirmed with larger samples.

The cognitive interviews and results were interpreted in combination rather than focusing on a single source of analysis (Table 6). This integrated approach strengthened the overall evaluation of the adapted Swedish version questionnaire by enabling the identification of how revisions improved contextual relevance and clarity in relation to participants’ interpretations, while simultaneously revealing psychometric challenges such as disordered thresholds or item misfit. This demonstrates the value of combining qualitative and quantitative methods, as neither approach alone would have identified interpretation and measurement-related issues.

Our findings indicate that each UTAUT construct (PE, EE, SI, FC, and BI) demonstrated unidimensional properties and are in line with the general concept of the UTAUT, where each construct is a single latent factor [18]. This consistency suggests that conceptual integrity was largely maintained during the translation process into the Swedish prehospital care context. However, the study raises questions about the transferability of a generic translated UTAUT questionnaire, focusing on individual perspective and not taking the specific clinical context into account [63]. Studies using UTAUT in the prehospital context have been published earlier. However, these studies did not provide item-level comparisons [75,76], and there is a significant gap in the literature in prehospital care regarding psychometric evaluations using Rasch analysis on translated versions of prehospital diagnostic or performance scales. Previous UTAUT research within health care identified PE as a main driver for intention to use technology [26] and highlighted organizational and contextual factors for successful implementation [24]. The findings in this study may suggest that determinants relevant for other health care settings may also be relevant for the prehospital context.

This translated questionnaire used hypothetical statements due to the preimplementation stage of video consultation in Swedish EMS. Investigating acceptance of technology relying on hypothetical scenarios, such as asking participants to predict behavior for a system not yet experienced, creates further challenges, such as social desirability bias, and yields a challenge in demonstrating any connection between an attitude and subsequent observed behavior [77]. However, this is a legitimate, valuable, and necessary step for strategic planning before implementation [12], and studies suggest that hypothetical questions can indeed shape future judgment and behavior by increasing the availability of specific knowledge related to the context [78].

Taken together, the UTAUT retained largely structural validity in the context of Swedish EMS, while the interpretation of some constructs appears to shift toward more context-dependent perspectives. This implies that acceptance of technology in EMS cannot be fully understood from an individual perspective alone, but must account for operational conditions, teamwork, and resource availability. Further research is required to refine the UTAUT questionnaire in Swedish EMS and to determine how organizational factors influence individual acceptance of technology.