This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.
Young people are increasingly seeking out web-based support for their mental health and well-being. Peer support forums are popular with this age group, with young individuals valuing the fact that the forums are available 24/7, providing a safe and anonymous space for exploration. Currently, little systematic evaluation of the helpfulness of such forums in providing support has been conducted.
This study examined the helpfulness of the support offered within web-based mental health and well-being peer support forums for young people. It specifically investigated the self-reported user ratings of helpfulness reported through the completion of a developing experience measure. The ratings will be used to consider further development of the measure and reflect upon the overall helpfulness of the forums as indicated by the reported scores.
The study used routinely collected practice-based outcome data from web-based mental health forums for young people. These forums are hosted by the UK-based web-based therapy and support service, Kooth. A cross-sectional design was used to explore—using a range of inferential statistical measures—the outcomes reported by those accessing the forums using a Peer Online Community Experience Measure (POCEM). To consider the helpfulness in general, 23,443 POCEMs completed in 2020 were used. A second data set of 17,137 completed POCEMs from the same year was used to consider whether various engagement indicators had an impact upon the helpfulness rating.
Female users aged between 11 and 16 years predominantly completed the POCEM. This is in keeping with the majority of those using the service. In total, 74.6% (8240/11,045) of the scores on the POCEM indicated that the individuals found the posts helpful. An ANOVA indicated that male users were more likely to report obtaining intrapersonal support, whereas female users obtained interpersonal support. Furthermore, the POCEM scores reflected the internal consistency of the measure and provided an insight into the way that young people made use of the peer support resource; for instance, posts that were rated more helpful were correlated with spending longer time reading them, and the topics discussed varied throughout the day with more mental health issues being discussed later at night.
The results seem to demonstrate that, overall, the young people involved in this study found web-based peer support helpful. They indicate that peer support can provide an important strand of care within a supportive mental health ecosystem, particularly during time periods when in-person support is typically closed. However, limitations were noted, suggesting that caution is needed when interpreting the results of this study. Although such services are incredibly well used, they have received little research attention to date. As such, further investigation into what constitutes helpful and unhelpful peer support is needed.
Several definitions of the term
Even when forums are professionally led and moderated, web-based peer support is not intended to replace services provided by qualified therapists [
There are a number of challenges associated with the use of web-based forums for CYP to discuss mental health, such as ensuring user safety, building feelings of trust [
Within forums, support can be either directive or nondirective in nature. A directive approach comprises receipt or provision of explicit advice on how to overcome an issue, and a nondirective approach involves supportively sharing experiences [
It is clear from the outlined research that there are numerous positive outcomes of web-based mental health forums and communities for CYP [
This study aimed to explore user-rated measurement of the helpfulness of web-based peer support forum content by investigating the following research questions:
What is represented in the outcomes of an experience measure (POCEM) that claims to rate the helpfulness of web-based community content?
What trends and commonalities exist within posts that are deemed either helpful or less helpful by CYP?
This study used an inferential cross-sectional design [
The data for this study came from Kooth, which is a UK-based digital therapy and support service for CYP aged 10 to 25 years. The service provides a predominantly humanistic and idiographic approach to counseling, where users are actively involved in every step of their therapeutic journey [
No referral or joining a waiting list is needed to use Kooth, which is a free-of-charge service. However, service access is, at present, contract limited. This means that access is restricted to certain geographical regions as governed by whole-population contracts with local National Health Service trusts. Kooth can also be commissioned by other organizations, such as local authorities, for use with certain subsets of the population for whom a need for provision is recognized. Although the web-based community forums are accessible 24/7, live counselors and moderators are only active until 10 PM.
All data used in this study were collected from service users who had consented to their data being used for research purposes. As this was routinely collected information, the research process did not influence the data [
The POCEM is a 3-step web-based experience measure designed for users to rate content within the Kooth community. It aims to measure satisfaction with, and quality of, web-based community resources in relation to how well they met the expectations of the service user completing the measure. Any time a service user engages with a community item such as a forum post or an activity, they are, as a first step, asked how helpful they found it by rating it on a Likert scale (-2: No; -1: Not really; 0: Not sure; 1: A bit; 2: Loads). The second and third steps are explained in the Data Set 1 section.
Two data sets were used to investigate the research aims. For the first aim—to examine the user-rated measurement of helpfulness and experiences gained from community forum interaction—a cross-sectional extraction of every POCEM completed in 2020 was performed. This data set of 23,443 POCEMs was completed by 11,045 unique users. Each user completed a mean of 2.12 (SD 2.64) POCEMs. This data extraction captured the second and third steps of the 3-step measure. In the second step, after assigning a numerical score to a post, the service user was prompted to indicate a reason for their rating, choosing from 4 options to reflect the nature of their experience. This was optional, and their Likert rating choice was recorded even if they chose not to engage further. In addition, if the user selected
The selectable domains and their corresponding statements that are available in the Peer Online Community Experience Measure and the high-level support outcomes that they represent.
Service user–experience domains | Additional selectable statements under each domain | High-level support represented by each domain |
“I want information about something important to me” (Important to me) |
I got information that helped me learn about myself The information I received today was helpful to my problem I now know what I need to do to feel better I learned something new today I now know that others have the same experiences as me |
Informational intrapersonal support |
“I want to learn some skills to try with other people” (Learn skills) |
I have learned how to express myself I have learned enough to make a positive change I have developed skills to open up more I now have knowledge and skills to help others I have learned how to support others |
Informational interpersonal support |
“I want to explore more about how I relate to other people” (Relate to others) |
I feel safe in the Kooth community I felt connected to someone I feel that I’m just as valuable as others I know who to ask for help I feel motivated to give advice to others It feels good not to be judged |
Emotional interpersonal support |
“I want to understand myself more” (Understand myself) |
I felt accepted I now feel more hopeful I now feel able to ask for support outside of Kooth My problems now feel more manageable I am now able to find solutions to my problems I now want to make changes in my life I no longer feel alone |
Emotional intrapersonal support |
The second data set, a subset of data set 1, comprised 17,137 POCEMs completed by 10,612 unique users. In this data set, an average of 1.62 (SD 1.89) POCEMs were completed per user. It was used to explore the nuances and commonalities that exist among posts that were rated a certain way, as per the second research aim. This data set was a cross-sectional data extraction of those POCEMs completed within 1 month of their corresponding community forum post being submitted. As a clarifying example, a POCEM that was completed on April 21 that related to a forum post submitted on March 22 would be included in this extraction; however, a POCEM completed on April 23 would not be included. This limit was set because engagement with posts tends to become sparse when they are >1 month old. As in the first data set, a measurement score as well as the category topic of the community forum post were captured. The number of comments on, and views of, a post at the time of each POCEM completion were ascertained too. Owing to the interface of Kooth community forums, these features related to what a user can see when they enter the forum environment. The amount of time each user spent interacting with the post (in minutes) before they submitted the measurement scores, the average score of already submitted POCEMs, and the age of each post (in days) when each POCEM was completed were also recorded. In the Kooth web-based community, forum posts do not move to the top of the page when a new comment is submitted. If the age of posts is linked to POCEM completion likelihood and score, it could provide useful insight into how users search for, and engage with, posts.
Frequency tables were produced to explore the demographics of those who completed POCEMs, the distribution of scores, and the topics to which the content related. The split of gender and ages across the 4 experience domains (
Heat maps were produced to explore which pairs of corresponding statements (refer to the
A simple correlation matrix explored the potential linear relationships among the various qualities of a community post at the time of POCEM completion. Next, a new variable was computed to split the data based on the time of day that each POCEM was completed to investigate how CYP use their free time outside of education or work to take care of their mental health and what patterns of completion followed after 10 PM when live counselors are offline. Using this variable, 1-way Welch ANOVAs investigated whether POCEM scores, time spent reading a post within the web-based community forum, and the post’s age at POCEM completion varied depending on the time of day. Frequency tables examined trends relating to the topics interacted with at different times of day, with further ANOVAs used to explore points of interest by isolating topic categories through the computation of dummy variables.
In the following sections, we detail the findings of the investigations outlined in the Methods section. Please note that where dummy variables were inputted into significance tests, means are not provided because they would represent transformations rather than meaningful reflections of the original data.
In total, 75.3% (17,653/23,443) of the POCEMs in this data set were completed by female users and 18.9% (4431/23,443) by male users, whereas 3.7% (867/23,443) and 2.1% (492/23,443) were completed by respondents who identified as gender fluid and agender, respectively. In total, 91.2% (21,380/23,443) of the measures were completed by users aged between 11 and 16 years, which corresponds with secondary school age in the United Kingdom. Of the 11,045 users, 4628 (41.9%) were aged 12 or 13 years, and only 376 (3.4%) were aged >18 years. In terms of ethnicity, 79.79% (8813/11,045) of the sample identified as White (White British, White Irish, or “other White background”), which falls below the 86% reported in the 2011 England and Wales census [
In terms of helpfulness scores, 74.6% (8240/11,045) of the users selected
Significant gender differences existed in the likelihood of choosing the domains
Heat maps showing the concentration of pairs of selectable statements. Those selected together with the highest frequency are dark green, with red indicating the lowest frequency. N/A: not applicable.
Negligible yet significant correlations were found between the POCEM score and age of the post at time of measure completion (
Significant correlations were also found between the age of the post and the following variables: time spent on the post (
The amount of time in minutes a user spent reading or otherwise interacting with a post before completing a POCEM was significantly correlated with the number of views that the post had at the time of POCEM completion (
When the POCEMs were broken down per the time of day in which they were completed, 48.9% (8380/17,137) were completed between 4 PM and 10 PM (
The POCEM scores did not vary depending on time of day (
When examining the topics of forum posts for which POCEMs were completed at different times of day, the most popular topics were checked for apparent patterns. In the
Exploring these phenomena, measure completion for mental health–related posts significantly varied depending on time of day (
POCEM completion for hobbies- or interests-related posts also significantly varied depending on time of day (
Games-Howell pairwise comparisons among times of day in terms of forum post age (in days) at the time of Peer Online Community Experience Measure completion.
Time of day (mean post age in days) and pairwise comparison | |||
|
|||
|
Morning | .003a | |
|
Afternoon | .01a | |
|
Evening | .12 | |
|
Night | .003a | |
|
|||
|
Late night | .003a | |
|
Afternoon | .99 | |
|
Evening | <.001a | |
|
Night | <.001a | |
|
|||
|
Late night | .01a | |
|
Morning | .99 | |
|
Evening | <.001a | |
|
Night | <.001a | |
|
|||
|
Late night | .12 | |
|
Morning | <.001a | |
|
Afternoon | <.001a | |
|
Night | .32 | |
|
|||
|
Late night | .003a | |
|
Morning | <.001a | |
|
Afternoon | <.001a | |
|
Evening | .32 |
aValues that met the significance threshold (
Games-Howell pairwise comparisons among times of day in terms of time spent engaging with a post (in minutes) before Peer Online Community Experience Measure completion.
Time of day (mean time spent engaging with post in minutes) and pairwise comparison | |||
|
|||
|
Morning | .02a | |
|
Afternoon | .34 | |
|
Evening | .03a | |
|
Night | .89 | |
|
|||
|
Late night | .02a | |
|
Afternoon | .62 | |
|
Evening | .99 | |
|
Night | .14 | |
|
|||
|
Late night | .34 | |
|
Morning | .62 | |
|
Evening | .78 | |
|
Night | .87 | |
|
|||
|
Late night | .03a | |
|
Morning | .99 | |
|
Afternoon | .78 | |
|
Night | .23 | |
|
|||
|
Late night | .89 | |
|
Morning | .14 | |
|
Afternoon | .87 | |
|
Evening | .23 |
aValues that met the significance threshold (
Percentage of total Peer Online Community Experience Measures completed that related to a mental health–themed post at different times of day.
Percentage of total Peer Online Community Experience Measures completed that related to a hobbies- or interests-themed post at different times of day.
Web-based peer support, especially that pertaining to CYP mental health, has received very little direct research attention thus far despite its extensive use by this demographic. To close this gap, this study builds on a recent measure development paper (Mindel, C, unpublished data, November 2021) by providing novel insight into how CYP interact with a web-based community measure designed to rate the helpfulness of peer support forum content. From the information provided in the POCEMs, we were able to investigate the commonalities underpinning forum posts and web-based community content that were rated in certain ways.
In the first data set, user-rated measurements of helpfulness were examined. In total, 41.9% (9823/23,443) of the POCEMs within this extraction were completed by users aged 12 or 13 years. These ages are characterized by the transition from childhood into adolescence. Many social, educational, and biological changes affect a young person’s life at this time [
We also found that considerably more POCEMs were completed by female users (17,653/23,443, 75.3%) than male users (4431/23,443, 18.9%). Data from the Millennium Cohort Study found that emotional mental health symptoms increased from 12% to 18% for girls aged between 11 and 14 years, although prevalence for boys stayed the same [
More male users than female users chose the domains
In the second data set, significant correlational relationships of varying strengths were found among the qualities of a post at each instance of measure completion. Posts with higher POCEM scores were older, and users spent longer time reading them. Although these correlational relationships were negligible, post age and time spent interacting were key factors of interest, and more POCEMs were also completed on older and longer-read posts. As mentioned earlier, within Kooth’s forums, posts do not move to the top of a page when a new comment is posted. In addition, the only way to attempt to locate relevant posts is by selecting a topic category. Accessing and completing the measure for an older post therefore implies a targeted search for that content: the user has likely scrolled through several pages and selected a topic to find information pertaining to their issue. They have likely logged in and spent time engaging with the community because of a specific want or need. Once they find the relevant information, it is likely to be viewed more favorably because of this specific need resolution. We also found that older posts were engaged with for longer before POCEM completion, which further strengthens this idea of targeted searches being more useful—more time is spent reading and absorbing the information within, leading to increased likelihood of POCEM completion and higher rating. Posts interacted with for longer before POCEM completion also had more comments and views. Comments and views are the only 2 engagement figures visible to forum users; therefore, high numbers may elicit a curiosity that results in longer reading time. These relationships all suggest that a more user-friendly search function would be beneficial. Older posts are evidently scrolled through, interacted with, and deemed useful, and the findings suggest that CYP are using the forums to search for information and to seek help in a targeted fashion, in addition to general browsing and interacting. A search-term function, in addition to a facility to sort posts by factors such as
Looking at POCEM completion across times of day in the web-based community proved insightful, particularly in terms of how CYP choose to spend their time outside of school or work—their
Arguably, the most interesting findings concerning time of day appeared when we examined the topics of posts that were rated at different times of day. As shown in
Although several useful insights were gained from this research, they must be viewed in the context of the following limitations. The first of these relates to the study’s cross-sectional design. The large data set of 23,443 POCEM completions was an asset of the study, given the statistical power this offered our analyses. However, examining trends over time by means of a longitudinal design would allow us to explore how forum use and user experiences fluctuate over time. This would be of particular interest during times of widespread crisis, such as the COVID-19 pandemic. This would also allow examination of how long-term forum users engage. Linked to this idea, we mentioned in the Methods section that the 23,443 completed POCEMs came from 11,045 unique users. Given that the study’s aims primarily focused on post helpfulness rather than repeated patterns by users, our analyses did not account for the fact that many users completed the measure multiple times. We suggest that each POCEM represents a valid and distinct user experience. However, future longitudinal research could investigate user intensity and patterns of response in 1-time users compared with repeat users. This could be to identify whether helpfulness ratings differ and whether users tend to report the same or different experience domains whenever they interact with the POCEM.
It is also wise to acknowledge the limitations that exist with the finding that 74.6% (8240/11,045) of the POCEM scores were positive. Ratings of satisfaction tend to be completed more frequently by those at opposite ends of the experience spectrum. This extremity bias [
There are also several methodological constraints relating to the use of heat maps (
To summarize this research, this study used a large data set to explore the use of a user-rated helpfulness measure in a CYP mental health peer support forum context. The results seem to indicate that the young service users involved in this study found web-based peer support helpful and that the POCEM itself was engaged with well. This suggests that peer support can provide an important strand of care within a supportive mental health ecosystem, particularly during time periods when in-person support is typically closed. This latter finding stresses the importance of vigilant support provision at these times. Measures were most often completed by secondary school–aged CYP and by many more female users than male users. The age of a post and the time spent engaging with a post before POCEM completion were found to be factors of interest, providing insight into the search habits of users. These factors denote key areas where peer support forums can be made more intuitive and user friendly. Caution is advised when interpreting the results of this study. Although such services are popular, and indeed useful, they have received little research attention to date. As such, further investigation into the nature of helpful and unhelpful peer support is warranted, as well as increased focus on how it should be incorporated into comprehensive systems of digital mental health support.
children and young people
Peer Online Community Experience Measure
The time spent on this paper by EB, SDOG, CM, TK, and AS was funded by Kooth Digital Health. Special thanks to Holly Brick and Katherine Jackson who helped to develop and conceptualize the measure; to the product team at Kooth for implementing the measure in the digital platform; to Dr Lynne Green and Dr Hannah Wilson, among other members of the clinical team, for safeguarding and overseeing the research project at Kooth; and to Lex Young as well as the content team and moderators for helping to run the forum community.
SDOG, CM, TK, and AS are employed by Kooth Digital Health where the data for this study were collected. EB was externally commissioned and funded by Kooth to work on the project but is not an employee of the company.