This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on http://formative.jmir.org, as well as this copyright and license information must be included.
Auditory verbal hallucinations (AVH) are the most frequent symptom in early-onset psychosis (EOP) and a risk factor for increased suicide attempts in adolescents. Increased knowledge of AVH characteristics can lead to better prediction of risk and precision of diagnosis and help identify individuals with AVH who need care. As 98% of Norwegian adolescents aged 12 to 16 years own a mobile phone, the use of mobile phone apps in symptom assessment and patient communication is a promising new tool. However, when introducing new technology to patients, their subjective experiences are crucial in identifying risks, further development, and potential integration into clinical care.
The objective was to explore the feasibility and acceptability of a newly developed mobile phone app in adolescents with EOP by examining compliance with the app and user experiences. Indication of validity was explored by examining associations between AVH dimensions, which were correlated and analyzed.
Three adolescents with EOP and active AVH were enrolled. Real-time AVH were logged on an iPod touch using the experience sampling method (ESM), for seven or more consecutive days. The app included five dimensions of AVH characteristics and was programmed with five daily notifications. Feasibility and acceptability were examined using the mean response rate of data sampling and by interviewing the participants. Validity was assessed by examining associations between the AVH dimensions using nonparametric correlation analysis and by visual inspection of temporal fluctuations of the AVH dimensions.
One participant was excluded from the statistical analyses but completed the interview and was included in the examination of acceptability. The sampling period of the two participants was mean 12 (SD 6) days with overall completed sampling rate of 74% (SD 30%), indicating adequate to high compliance with the procedure. The user experiences from the interviews clustered into four categories: (1) increased awareness, (2) personal privacy, (3) design and procedure, and (4) usefulness and clinical care. One participant experienced more commenting voices during the sampling period, and all three participants had concerns regarding personal privacy when using electronic devices in symptom assessment. The AVH dimensions of content, control, and influence showed moderate to strong significant correlations with all dimensions (
This study demonstrates the value of including patients’ experiences in the development and pilot-testing of new technology. Based on the small sample size, the use of mobile phones with ESM seems feasible for patients with EOP, but the acceptability of using apps should be considered. Further investigation with larger samples is warranted before definitive conclusions are made.
Early-onset psychosis (EOP) is psychotic disorders with age of onset before 18 years of age. EOP is often characterized by a chronic illness course with impaired social and daily functioning [
Assessing AVH using traditional retrospective measures, such as the Positive and Negative Syndrome Scale (PANSS) [
In Norway, 98% of adolescents aged 12 to 16 years own a mobile phone [
We piloted a newly developed mobile phone app designed to monitor AVH characteristics at the moment they occur in adolescents with EOP. It is of particular interest to investigate these phenomena in adolescents because this life period is a challenging maturation process for neurological and psychological development [
The study was approved by the South East Regional Committee for Medical and Health Research Ethics, Oslo, Norway, and conducted in accordance with the Helsinki Declaration. Four adolescents with EOP and active AVH, already included in the larger ongoing Thematically Organized Psychosis Study for Youth (Youth-TOP) from 2016 to 2018, were invited to participate in this pilot study. One patient refused to participate for unknown reasons, resulting in a total of three enrolled participants. Of the three, one used the app in a psychiatric unit and two used it in their home environment.
The participants were recruited from adolescent psychiatric inpatient units and outpatient clinics in the Oslo region. The inclusion criteria were (1) nonaffective EOP (schizophrenia, schizoaffective disorder, schizophreniform disorder, psychotic disorder not otherwise specified, brief psychotic disorder) according to the
Diagnoses were confirmed using the Schedule for Affective Disorders and Schizophrenia for School-Age Children-Present and Lifetime version [
The mobile phone app was originally developed by coauthors KH and JB at the University of Bergen, Norway (see [
How the app is used by the participants. In the first screen the participant chooses yes or no to whether they hear voices right now. If they answer “yes,” the five following screens with the visual analogue scales for the five auditory verbal hallucinations (AVH) dimensions appear. Each screen represents the following AVH characteristic: content (emotional), localization (perceptual), control (cognitive), intensity (intensity), and influence (influence). Previously presented on a poster at the International Congress on Schizophrenia Research, March 24-28, 2017; San Diego, CA, with the abstract by Bless et al [
The participants sampled real-time AVH experiences on an iPod touch (Apple Inc, Cupertino, CA, US) provided by us, with a preinstalled version of the app. We chose to use iPods instead of mobile phones in the pilot study to guarantee the personal privacy of the participants. The iPods were not connected to the internet. The app was programmed with five daily notifications at pseudorandom time points from 10:00 to 22:00 (within five time intervals of 1.5 hours), in accordance with recommendations from Palmier-Claus and colleagues [
Statistical analyses were performed using IBM SPSS Statistics for Windows version 25. Feasibility was examined using the mean response rate of the data sampling, calculated by dividing the total number of responses by days using the app multiplied by five (as five daily registrations indicated 100% compliance). Acceptability was examined by interviewing the participants about their subjective experiences of using the app. The participant experiences were written down. No voice or video recordings were used. Thus, the transcriptions of the participants’ subjective experiences are not verbatim quotes, but reproductions based on the written material. Indication of validity was assessed by examining associations between the AVH dimensions using two-tailed Spearman rho (ρ) nonparametric correlation analysis and visual investigation of the fluctuations of the AVH dimensions. This was done by making a graph with the mean symptom scores and days of sampling, defined as the number of data entries. Different time intervals between each data entry were not taken into consideration (ie, the time period between the first and second data entry might have differed from the second and third entry).
The demographics of the three participants were mean age 17.7 (SD 1.6) years, mean age of onset for psychosis 14.9 (SD 0.7) years, mean duration of untreated psychosis 32 (SD 22) weeks, mean illness length 2.8 (SD 1.8) years, mean IQ score 93.7 (SD 8.0), and mean CGAS score 36 (SD 4). Two of the three participants were taking atypical antipsychotic medications. One of the three was excluded from the statistical analyses due to noncompliance with the procedure. This participant did not sample real-time data for seven consecutive days and had some aggregated registrations (ie, five samplings at the same time during the afternoon/evening). According to the participant, the reason for noncompliance was partly due to having to bring two devices to daily activities. The participant did not differ from the remaining two regarding age, sex, global functioning, or symptom severity. Thus, the participant’s subjective experiences were included in the examination of acceptability. The average sampling period of the two remaining participants was mean 12 (SD 6) days with a mean response rate of 74% (SD 30%) indicating adequate to high compliance with the sampling procedure.
The participants had stable PANSS scores before and after the sampling period with an average positive sum score of 14 and a P3 score of 4. The malevolence scale on the BAVQ-R was reduced by 3 points, from 11.0 to 8.0. The omnipotence scale was similarly reduced by 1.6 points, from 13.3 to 11.7. The resistance scale was also reduced by 0.7 points, from 16.0 to 15.3, and the engagement scale was reduced by 2 points, from 6.0 to 4.0. The BAVQ scores thus showed a general improvement in AVH characteristics regarding malevolence and omnipotence during the sampling period.
Mixed experiences of the app were reported when exploring the acceptability, clustered into four main categories: (1) increased awareness, (2) personal privacy, (3) design and procedure, and (4) usefulness and clinical care. Examples of the participant subjective reports are presented in
Illustrations of participant experiences after the sampling period (N=3).
Experience | Sample statementsa |
Increased awareness | I became aware at every registration. It was okay to become more aware. Nothing was negative. |
I was reminded of the voices. I usually try to avoid that. I got more commenting voices, saying ‘now he/she’s doing it again, why is he/she registering us?’ That’s why I haven’t used it so much. | |
Personal privacy | I was a bit worried of being watched when I used the app. I saw the camera in front of the iPod. I don’t like cameras. |
I have felt monitored when using the app, particularly in the beginning, because of the notifications. I liked that it was offline. I’m afraid of leaving traces on the internet. | |
It’s not secure, you may get hacked. | |
Design and procedure | The app was short and easy to understand. It was a bit much of the same questions. I was watching the iPod all the time, not knowing when the next notification would come. I thought a lot about when the next notification would come. |
It was okay with five notifications per day. If the notifications came at set time points it could help reduce thoughts about being monitored. I liked that I could write notes at the end. | |
The layout was nice; the visual scales and the order of the statements were nice. Maybe three registrations in one day instead. | |
Usefulness and clinical care | It wasn’t very useful. Might be helpful to others, I don’t know. |
Nice to register symptoms. It would be very nice to have in clinical care. When I’m feeling down it’s hard to answer questions. It’s okay to use the app if the voices are not in control, that they don’t have opinions about what you should answer. If the voices are in control you might give the wrong answer. | |
It was useful to learn about the voices. It was difficult to remember to sample data toward the end. I forgot the iPod at home or in my backpack. I don’t want it as part of my treatment, I prefer to talk. |
aThe statements are illustrations of the participants’ subjective experiences, not quotes.
Two participants reported becoming more aware of their AVH when using the app. One participant reported that the increased awareness was positive, whereas the other reported it as negative, resulting in more commenting voices. This was reported as the main reason for stopping using the app by the latter participant.
The participants had concerns regarding personal privacy when using apps and technology as part of their symptom assessment. Although they knew that the iPod was offline during the sampling period, they reported having fears of being monitored through the camera or the notifications from the app. In general, they reported worries about leaving electronic traces on the internet and being hacked. All participants considered it positive that the iPod was not connected to the internet during the pilot testing. This may be an argument to use iPod-like devices rather than real mobile phones to reduce these kinds of worries. However, it should be noted that such worries and concerns may also be part of the illness, as increased suspiciousness of being “monitored” from the outside is a not uncommon symptom.
The participants liked the design of the app, that it was short (completed the self-assessments in about 1 minute), easy to understand, and it had the option of writing notes at the end. One participant would have preferred having numbers on the VAS scales to make scoring easier. Two participants reported that five daily notifications were adequate, whereas one thought this was too much and suggested three as an alternative. Two participants reported discomfort from the random notifications, reporting thoughts of being monitored and concerns and preoccupations regarding the next notification. They suggested having notifications at set times. Furthermore, one participant reported that the repetitiveness of the self-assessments was a bit much when sampling data five times per day for one week, and suggested adding more varied self-assessments to the app. The reports were mixed, which points to the importance of considering more technical issues such as design when developing apps.
Two participants reported that it was nice to register symptoms and learn more about the voices. However, the same participants reported decreased motivation toward the end of the sampling period and when their mood was low. One participant reported that using technology in treatment could be a useful alternative to answer verbal questions, but that self-reports could be biased if the voices were in control and had opinions about what to report. This is not an argument against the use of apps because the same could apply using traditional self-reports. However, the statement indicates that personal contact is an important part of clinical care and that apps and technology potentially can be used to monitor remission and relapse as part of measurement-based care, as suggested by Insel [
Indications of validity were assessed by examining associations between the AVH dimensions and by examining the fluctuations of the AVH dimensions during 1 week of sampling.
Spearman rho (ρ) correlations for the participants completing the sampling period (N=2). Influence: not at all (0) to very disturbing (1); content: negative (0) to positive (1); intensity: yelling (0) to whispering (1); control: no control (0) to full control (1); localization: outside head (0) to inside head (1). All correlations were two-tailed.
AVHa dimension | Content | Control | Influence | Intensity | Localization | Day | ||||||
ρ | ρ | ρ | ρ | ρ | ρ | |||||||
Content | 1 | ―b | .73 | <.001 | –.80 | <.001 | .43 | <.001 | .50 | <.001 | .22 | .06 |
Control | .73 | <.001 | 1 | ― | –.77 | <.001 | .48 | <.001 | .41 | <.001 | .21 | .07 |
Influence | –.80 | <.001 | –.77 | <.001 | 1 | ― | –.44 | <.001 | –.56 | <.001 | –.25 | .03 |
Intensity | .43 | <.001 | .48 | <.001 | –.44 | <.001 | 1 | ― | .12 | .30 | .05 | .64 |
Localization | .50 | <.001 | .41 | <.001 | –.56 | <.001 | .12 | .30 | 1 | ― | .51 | <.001 |
Day | .22 | .06 | .21 | .07 | –.25 | .03 | .05 | .64 | .51 | <.001 | 1 | ― |
aAVH: auditory verbal hallucinations.
bNot applicable.
Mean scores of two participants for the five auditory verbal hallucination dimensions within and across 7 days of sampling.
The objective of this exploratory study was to examine the feasibility, acceptability, and indication of validity of a newly developed mobile phone app for auditory verbal hallucination characteristics in adolescents with EOP.
Although the statistical analyses of this study only included two participants, the correlation analyses and the graph containing within and across day fluctuations indicate that the app is valid for use in adolescents with EOP. The content, control, and influence dimensions were significantly correlated with all the AVH dimensions, indicating that these characteristics are important among these patients. This finding is in accordance with a study in adults with AVH, showing that experiences of mostly negative, uncontrollable, and distressing voices were important characteristics in separating nonpsychotic individuals from individuals with psychotic disorders [
The user-experience interviews revealed that two participants experienced increased awareness of the AVH when using the app, highlighting the need for researchers to consider that the assessment process itself may exert an influence on the individual through reactivity effects [
The participants reported concerns about personal privacy when using electronic devices for symptom assessment and considered it positive that the iPod was offline during the pilot study. This is in accordance with a study showing that one-third of a sample of young adults with psychotic disorders experienced discomfort in online settings [
Previous mobile phone studies including adolescents and adults with psychotic disorders showed enrollment and dropout rates of approximately 50% and 5%, respectively [
The number and randomness of notifications and the repetitiveness of the statements were reported as too much and associated with discomfort. Studies have found that too many notifications and repetitive self-assessments may result in higher dropout rates in ESM studies and a biased representation of participants completing the sampling period [
One participant considered integrating technology into clinical care as a potentially useful alternative to verbal communication if the voices were not in control. However, the interpersonal aspect of existing care was emphasized as an important factor. This finding is in accordance with a previous study in adults with psychotic disorders, showing that technology may be perceived as a threat to existing care and that self-reports may be biased [
This study shows that including patients’ subjective experiences in the development and pilot-testing of health care technology may provide useful information in addition to statistical analyses. Based on the mean response rate of the two participants included in the statistical analyses, the use of mobile phones with ESM in adolescents with EOP seems feasible, although this warrants further investigation with larger samples. Regarding acceptability, there are limitations with the use of apps, and care should be taken such that the app is perceived as meaningful, comfortable, and safe. As mentioned previously, limitations may be that apps produce unexpected distressing thoughts and unwanted focus on negative feelings such as anxiety. Therefore, it may help to have a plan if distress is experienced. As an alternative to online apps, offline versions could be considered to avoid the risk of inducing or reinforcing unwanted persecutory delusions. Our conclusion is that the use of app technology for real-time monitoring of psychotic symptoms can provide new knowledge of daily and even hourly fluctuations and severity of hallucinatory episodes, which can increase diagnostic resolution. Moreover, such information can have therapeutic effects if it corresponds with similar fluctuations in cognitive control and experienced distress, for example, which can be suitable targets for psychosocial interventions, such as cognitive therapy.
auditory verbal hallucinations
Beliefs About Voices Questionnaire-revised version
Children’s Global Assessment Scale
early-onset psychosis
experience sampling method
intelligence quotient
Positive and Negative Syndrome Scale
visual analog scale
The authors would like to thank the research participants and their families who provided the data. This work was supported by the South-Eastern Norway Regional Health Authority (2016-118), and the Research Council of Norway (213700, 250358) to Ingrid Agartz, and the European Research Council (ERC) (249516), and the Western Norway Regional Health Authority (912045) to Kenneth Hugdahl.
RES recruited and interviewed the participants, drafted the manuscript, and performed the statistical analyses. KH and JB designed and developed the mobile phone app. IA and KH obtained research funding and contributed to the design of the study protocol. JB, KH, and IA gave feedback on the manuscript draft. All authors have read and approved the final version of the manuscript.
None declared.