This study was approved by the institutional review board of the Nara Medical University, Japan, and Nara Institute of Science and Technology, Japan (reference number: 2019-I-27‐2). We explained the experiment’s procedure and purpose to the participants and obtained informed consent. All data were anonymized. Participants received monetary rewards of up to 16,000 Japanese yen.

The items in the English version of the SPRS are provided in Table 1, where each of them is rated on a Likert scale of 1 to 5 [25].

We developed a Japanese version of the SPRS. As suggested by an earlier work [26], in its guide to good practices in translation and cultural adaptation, we first translated the scale’s contents from English to Japanese and then from Japanese to English. This forward and backward translation method provides quality control to verify consistency between the translation and the original version. Inter Group Corp (Osaka, Japan), a company specializing in language and cross-cultural adaptation, translated the SPRS at our request. This Japanese version of the scale is available upon request to the authors.

We used a Japanese SST dataset collected in a previous study [14]. We collected SST data at Nara Institute of Science and Technology for the control group and at Nara Medical University for the clinical group. The dataset includes data from 50 adults with the following characteristics: 16 in the ASD group, 15 in the schizophrenia group, and 19 in the control group. We collected these psychiatric or neurodevelopmental clinical groups to investigate their social communication difficulties. We included these groups because the main SST clients in clinical facilities are those with ASD and schizophrenia. The control group participants were hired by a recruitment agency. Two psychiatrists with SST experience joined this study as trainers (roleplay partners). All participants performed SST using roleplay and feedback from the trainer. Four basic skills were performed as follows: asking for requests, declining requests, telling about positive feelings, and listening to others (ie, asking, declining, telling, and listening). Each roleplay lasts 1 to 3 minutes. Some participants performed SST multiple times, although our analysis was done on the initial roleplay data.

We excluded from the control group participants who had undergone eye surgery or had a history of psychiatric hospitalization. For the clinical group, we excluded participants who scored less than 70 on the third edition of the Wechsler Adult Intelligence Scale [27]. We excluded participants with drug or alcohol addictions, personality disorders, or organic mental disorders. The data collection period was from January 2020 to January 2021. Due to COVID-19 concerns, a transparent partition was placed between the participants and trainers. Two video cameras were placed behind each conversationalist to record the other individual at chest level from the participants’ faces. We also recorded a video and used 2 Kinect (Microsoft Corporation) sensors installed at the side of each conversationalist.

We assessed each participant using the following questionnaires for correlation analysis with the SPRS and action units: the Facial Emotion Identification Test (FEIT) adopted in previous work [28], Kikuchi’s scale of social skills with 18 items (Kiss-18) [29], Singelis independent-interdependent Self-Construal Scale (Self-Construal Scale) [30], the second edition of the Social Responsiveness Scale (SRS-2) [31] (Japanese version [32]), and the Japanese version of the Brief Assessment of Cognition in Schizophrenia (BACS-J) [33]. In addition, we used the Autism Diagnostic Observation Schedule, Second Edition (ADOS-2), for the ASD group and the Positive and Negative Syndrome Scale (PANSS) for the schizophrenia group [2,34]. Where possible, we obtained the total score and subscales of each questionnaire. Excluding the FEIT, the questionnaires were collected before the SST data collection.

The FEIT, which assesses the emotional perception of facial emotions, includes facial images of 19 different people classified into 1 of 6 emotions: happiness, sadness, anger, surprise, fear, or disgust. We included this assessment because people with ASD and schizophrenia struggle with social cognition in facial images [34]. Kiss-18, which measures social skill levels, is comprised of 18 questions based on 6 social skill categories. This metric comprehensively measures social skills. Self-Construal Scale, which consists of questions on a 7-point rating scale, was developed to measure how people view themselves in relation to others. The SRS-2, composed of 65 questions, is an evaluation metric of the severity of social impairment. Although the SRS-2 was initially designed to assess people with ASD, it can also differentiate among various social communication difficulties. Its effectiveness has been investigated with clinical subjects and members of the general population [31].

To evaluate the participants during SST roleplay, we also adapted a modified roleplay test [15]. The scale is available only in Japanese. It includes items more closely related to psychopathology than the SPRS because the main target of this test is adults with schizophrenia. For this test, third-party psychiatrists watch recorded videos from both oblique and side views. Our 2 raters used a Likert scale from 1 to 5 for eye contact, body direction and distance, facial expression, voice variation, clarity, fluency, and social appropriateness. Since the required skills are situation-dependent, the social appropriateness differs depending on each SST task. Let us explain examples of social appropriateness for Bellack’s basic tasks. Listening to others, which determines whether the participants paid attention to the interlocutor, includes nodding, back-channels, and other nonverbal behaviors (eg, eye contact, smiling). In expressing positive feelings, social appropriateness involves expressing attention to the interlocutor’s responses and the suitability of the participant’s speech content. For making requests, social appropriateness assesses whether they explained the details of their request, including what kind of help they needed. It also includes whether they listened to the interlocutor. For declining, social appropriateness concerns whether they expressed remorse and appropriate reasons for their refusal. It also includes whether they proposed alternatives to the request (eg, “I’m sorry, but I propose doing it another way next time”), which is essential to this situation.

Two third-party raters performed the annotations; all are nonexperts in psychiatry, psychology, or SST. The raters watched the recorded videos from oblique and side views for this evaluation. They initially evaluated all participants’ roleplay in the first of the 4 basic tasks during the SST. Here, they watched the first 3 videos and discussed the SPRS content for evaluation practice, and then they evaluated the scores of the other videos without discussion. They were not informed directly about the nature of the groups (clinical outpatients or control), but it is possible that they may have noticed the existence of these groups by the background in the video, since each group conducted SST in a different location. These raters were not given information about the other questionnaire scales, and they watched the videos in a shuffled order regardless of the group.

The intensity of action units on OpenFace toolkit (version 2.0) [35] is measured in terms of mean and SD during individual roleplaying. From the action units extracted by OpenFace, we selected specific action units: AU01 (inner brow raiser), AU02 (outer brow raiser), AU04 (brow lower), AU06 (cheek raiser), and AU12 (lip corner puller). These action units represent various facial muscle activations. For our analysis, we referred to a previous study that focused on action units less directly associated with lip movement [36]. The intensity of each action unit ranges from 0 to 5, indicating a spectrum from no activation to high activation. First, the intensity of frames with less than 0.7 confidence in the data was excluded as not a number. We then applied a moving average filter on 3-second frames. We then calculated the mean and SD of the cleaned intensity data for each action unit to assess the intensity and variability, respectively.

The analyses were performed to investigate whether the SPRS could be used to evaluate social communication skills during SST. We analyzed the interrater agreement, items’ characteristics, construct validity, and internal consistency. To check the interrater agreement, we calculated the intraclass correlation coefficient (ICC, 2 k) [37]. We averaged the SPRS scores for the 2 raters for our analysis. We calculated the Spearman correlation coefficients to examine the homogeneity of the SPRS and the revised roleplay test. We assessed the factor structure of the SPRS by exploratory factor analysis with oblimin rotation to confirm the construct validity [38]. We examined the Bartlett test of sphericity (P<.001) and the Kaiser–Meyer–Olkin test for a sampling adequacy of 0.881 to apply the exploratory factorial analyses [39]. The internal consistency was assessed by calculating the Cronbach α coefficient [40] and McDonald ω [41].

We also analyzed the total of SPRS items in relation to the 3 groups, 4 tasks of SST, and the questionnaires. In comparing the control, ASD, and schizophrenia groups, we used the nonparametric Kruskal–Wallis test. Post-hoc analysis was conducted using the Mann-Whitney U test with Bonferroni correction. In comparing the asking, declining, telling, and listening tasks, we used the Wilcoxon signed rank test with Bonferroni correction. For action units, we applied the Mann-Whitney U test with Bonferroni correction to compare the control, ASD, and schizophrenia groups. Spearman correlation coefficients were calculated between the total SPRS items, action units, and questionnaire scores for all, control, ASD, and schizophrenia groups, although some questionnaires were not obtained from all participants. We used the JASP (Jeffreys’s Amazing Statistics program) for the statistical analysis [42].

We measured the interrater agreement by calculating the ICC (2 k) for validation of the SPRS annotated by the 2 raters. We opted for the random 2-way ICC method with the mean as the unit of evaluation, as shown in Table 2. The ICC ranged from 0 to 1, and all items had reliability values between 0.509 and 0.804. The ICCs were above 0.5, which indicates more than moderate reliability [22]. The score for each SPRS item was averaged for the 2 raters’ annotations in subsequent analyses. The Spearman correlation performed between the SPRS items and the modified roleplay test items shows significant results (all P<.001) and positive relationships. The correlation between all of them was greater than 0.6. Exploratory factor analyses were conducted on the 5 items of the SPRS using oblique rotation to investigate underlying constructs and assess dimensionality. Following a previous study [22], a 1-factor solution was adopted. Table 3 provides the factor loadings after rotation, illustrating how each of the 5 items aligns with the selected factor. The results show that the first factor explains 82.1% of the variance, indicating strong explanatory power. Factor loadings ranged from 0.874 for the “Discomfort” item to 0.932 for the “Conversation Flow” item, reflecting high values across all items. Additionally, Table 3 displays the uniqueness values for each variable, where uniqueness represents the proportion of variance specific to each variable that the factor does not explain. Higher uniqueness suggests lower relevance of that variable within the factorial model. For example, 23.6% of the variance in the “Discomfort” item remains unexplained by the factor, whereas the “Conversation Flow” item has a smaller unexplained variance (13.2%), showing greater alignment with the factor model. In sum, the factor analysis of the SPRS identifies a single factor composed of 5 items, capturing the verbal and nonverbal social skills essential for effective communication with others.

We created an overall performance measure by summing the scores of the 5 SPRS items. This measure revealed that Cronbach α was 0.957 and McDonald ω was 0.958. These results indicate that the SPRS shows excellent internal consistency.

Box plots of the 5 items of the SPRS and the total of the items for the control, ASD, and schizophrenia groups are shown in Figure 1. There were significant differences in all items and in the total of the SPRS items (P<.001). Post-hoc analysis was conducted using the Mann-Whitney U test with Bonferroni correction. We found significant differences in gaze, voice quality, length, discomfort, conversation flow, and the total of SPRS items for control-ASD and control-schizophrenia (P<.001). In ASD-schizophrenia, we found significantly greater gaze scores in the ASD group than in the schizophrenia group (P=.005).

SST task differences and a comparison are shown in Table 4. In the control group, we found significant differences in all items except gaze in asking-declining tasks and telling-declining tasks, and there were significant differences in the length of the asking-listening and telling-listening tasks. The declining task had lower scores than the asking and telling tasks for the control group. The length item of the listening task also had a low score compared with asking and telling tasks. We did not find significant differences between the 4 tasks in the ASD group. In the schizophrenia group, we found significant differences in the length of telling-listening tasks. For length, the listening task had a lower score than the telling task.

Participant group differences and comparisons, as well as mean and SD, are shown in Figure 2. We present the cases where significant differences were observed. We found that AU02 is significantly activated in people with schizophrenia compared with other groups. For mean values of the action units, this implies that AU02 is significantly less for the control group than the ASD group. For AU06, schizophrenia is significantly less than the control group. For AU12, it is more evident that schizophrenia is significantly less than either the control or ASD group. Regarding SD, people with ASD have significantly activated AU02 compared with the control group. For AU12, schizophrenia has significantly less variance than the ASD group.

Spearman correlation coefficients were calculated between the total SPRS items and questionnaire scores for all the following groups: control, ASD, and schizophrenia. The correlation coefficients are displayed in Figure 3. Across all groups, significant correlations were found between the total SPRS items and various measures, including the FEIT, SRS-2, Self-Construal Scale, Kiss-18, ADOS-2, PANSS, as well as the mean and SD values of AU06 and AU12. These findings suggest a strong relationship between social performance and various action unit statistics, as well as questionnaire scores.

In the control group, significant correlations with the total SPRS items were noted for Kiss-18 and the Self-Construal Scale, reflecting the cultural emphasis on harmony over assertiveness, which may influence social communication skills in Japan. In the ASD group, significant correlations were found with Kiss-18, the Self-Construal Scale, SRS-2, BACS-J, and ADOS-2, suggesting a relationship between these questionnaires and the social performance of individuals with ASD. Notably, significant negative correlations were observed in the SRS-2, BACS-J, and ADOS-2, highlighting the complexities of social interactions within this group.

In the schizophrenia group, significant correlations with the total SPRS items were found for the FEIT, BACS-J, and PANSS. These results suggest that emotional awareness and social performance are interconnected.

This is the first study applying the SPRS and analysis of action units for SST roleplay data. In this paper, we propose evaluating roleplay performance using the SPRS. We calculated the interrater reliability between 2 raters’ evaluated SPRS scores and found that the values were above 0.509 for any SPRS item. Our results show that the reliability was moderate to good [43]. Hamet Bagnou et al [22] calculated reliability among SPRS raters during collaborative tasks and showed that the reliability of gaze was the highest, but our results show that it has the lowest values. One reason could be the difference in conversation content. In our study, we applied 4 SST tasks, and the annotators needed to evaluate the SPRS based on the content of the tasks. The evaluation of SST tasks may have been more complex than that of collaborative tasks. Although there are slight differences between their study and ours, the results are nearly identical, with a reliability of 0.5 or higher in both studies.

The Spearman correlation between the SPRS items and revised roleplay test items was above 0.6. The factor analysis for the SPRS showed that the first factor explained 82.1% of the variance. The Cronbach α was 0.957, and the McDonald ω was 0.958. These results indicate that the SPRS shows excellent internal consistency and generally correlates to the modified roleplay test but is not identical.

We compared the SPRS scores by group and task. The control group differed significantly from the ASD and schizophrenia groups on every item of the SPRS. Several studies have shown that a control group shows higher social communication skills than an ASD group and a schizophrenia group, and we showed similar results.

The ASD group had a higher gaze score than the schizophrenia group. We consider it relevant that the eye movement abnormalities were severe in the schizophrenia group compared with the ASD group [44].

We conducted analyses of the differences in the SPRS scores between tasks for each group. In the control group, the declining task had lower scores than the asking and telling tasks, except for gaze. Declining is the most difficult task of the basic SST tasks, which may have contributed to the lower scores [11]. The listening task had lower scores than the asking and telling tasks in length. In the schizophrenia group, the listening task also scored lower than the telling task in length. The control group met the trainer for the first time in the data recording. It is possible that some control participants were not interested in the topics provided by the trainer in the listening task. The ASD and schizophrenia groups had met with the trainer several times. However, the schizophrenia group had low scores. Some participants with schizophrenia might have seemed disinterested in the conversation [11], and the raters might have annotated the low scores.

The control group showed that the SPRS had significant positive correlations with Kiss-18 and with the Self-Construal Scale.

The independence of the Self-Construal Scale measures is characterized by autonomy and distinction from others [45], and it is embedded within the culture [46]. In cultures with strong interdependence, harmony is an important concern, while assertiveness tends to be discouraged. This is predominant in the East, including Japan, and it may be related to social communication skills in Japan.

In the ASD group, there were significant positive or negative correlations between the SPRS and Kiss-18 or the Self-Construal Scale; this was also the case for the control group. We found significant negative correlations in the SRS-2, BACS-J, and ADOS-2. Our results suggest a relationship between the questionnaire for ASD and the social communication skills of the ASD group. The schizophrenia group had significant positive or negative correlations between the SPRS and the FEIT, BACS-J, and PANSS. Emotional awareness and communication skills are also related [47], and we surmise that our results show a significant difference in the FEIT. Another study showed that negative symptoms related to social motivation may be primary determinants of social outcomes in schizophrenia, potentially having a greater impact than social competence [48].

The results on action units also resemble the previous findings, in which an ASD group showed less smiling as represented in AU06 and AU12 [15,17], but there was no significant difference in terms of AU01 [18]. We found that AU02 was significantly activated in people with ASD compared with other groups. Moreover, this was the first study to conduct a detailed analysis of action units for schizophrenia. We also found that the AU06 and AU12 action units were significantly deactivated in people with schizophrenia compared with the control group. Accordingly, our results demonstrated that action units could work as social behavioral markers of people with ASD or schizophrenia.

This study is limited in terms of the population used and its generalizability. In particular, the SPRS was measured by 2 raters. Although we found high interrater correlations, a greater number of raters is needed to enhance the consistency and validity of the scale. In fact, previous work used an even larger number of raters, but a high interrater agreement was still maintained [21]. Our SST dataset was collected using outpatients with mild cases who were able to conduct SST verbally. Further study is needed to test whether the SPRS and action unit analysis could be applied to patients with more severe cases.

We applied and validated a Japanese version of the SPRS to SST roleplay data to measure individual social communication skills. We analyzed the SST data of 4 tasks carried out by adults with ASD or schizophrenia as well as control participants. The results found significant differences between the control and ASD as well as schizophrenia in all SPRS items. We found significantly greater gaze scores in the ASD group than in the schizophrenia group. There were also task-based differences in the schizophrenia and control groups. Our results suggest that the SPRS can be a useful tool for assessing social communication skills in different cultures and different pathologies and, in addition, can evaluate SST effectiveness as well as the modified roleplay test. We also found that the action units numbered AU06 and AU12 were significantly deactivated in people with schizophrenia compared with the control group. Possible future directions include using the SPRS for assessing social behavior during interaction with a digital agent.