Published on 26.05.20 in Vol 4, No 5 (2020): May
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/13170, first published Dec 17, 2018.
Toward Gamified Pain Management Apps: Mobile Application Rating Scale–Based Quality Assessment of Pain-Mentor’s First Prototype Through an Expert Study
Background: The use of health apps to support the treatment of chronic pain is gaining importance. Most available pain management apps are still lacking in content quality and quantity as their developers neither involve health experts to ensure target group suitability nor use gamification to engage and motivate the user. To close this gap, we aimed to develop a gamified pain management app, Pain-Mentor.
Objective: To determine whether medical professionals would approve of Pain-Mentor’s concept and content, this study aimed to evaluate the quality of the app’s first prototype with experts from the field of chronic pain management and to discover necessary improvements.
Methods: A total of 11 health professionals with a background in chronic pain treatment and 2 mobile health experts participated in this study. Each expert first received a detailed presentation of the app. Afterward, they tested Pain-Mentor and then rated its quality using the mobile application rating scale (MARS) in a semistructured interview.
Results: The experts found the app to be of excellent general (mean 4.54, SD 0.55) and subjective quality (mean 4.57, SD 0.43). The app-specific section was rated as good (mean 4.38, SD 0.75). Overall, the experts approved of the app’s content, namely, pain and stress management techniques, behavior change techniques, and gamification. They believed that the use of gamification in Pain-Mentor positively influences the patients’ motivation and engagement and thus has the potential to promote the learning of pain management techniques. Moreover, applying the MARS in a semistructured interview provided in-depth insight into the ratings and concrete suggestions for improvement.
Conclusions: The experts rated Pain-Mentor to be of excellent quality. It can be concluded that experts perceived the use of gamification in this pain management app in a positive manner. This showed that combining pain management with gamification did not negatively affect the app’s integrity. This study was therefore a promising first step in the development of Pain-Mentor.
JMIR Form Res 2020;4(5):e13170
Approximately one-third of the American and European population suffers from chronic pain [, ]. This makes chronic pain a major health care problem that needs to be taken more seriously [ ]. With profound negative consequences on psychological, social, physical, and economic aspects for those affected, chronic pain can have a serious negative impact on a person’s overall quality of life [ - ].
Next to medical treatments (eg, medication, surgical rehabilitation, and physical therapy), psychological treatments are an important aspect of pain management .
In fact, the combination of 5 theory-based functionalities, namely, pain-related education, self-monitoring, goal setting, social support, and the training of self-care strategies, has been suggested to promote the self-management of chronic pain [- ]. As patients need to understand and manage the thoughts, emotions, and behaviors that often accompany chronic pain [ ], the mediated self-care strategies should include stress-coping skills such as relaxation techniques, problem solving, and communication skills training [ - ]. Multimodal approaches that integrate these aspects can improve the overall quality of life in patients with chronic pain, as compared with treatments that are strictly medication focused [ - ].
However, the integration of multimodal approaches into routine primary and tertiary care has been slow . Major barriers such as poor accessibility because of geographical reasons, limited availability of trained professionals, and high therapy-related costs keep patients from accessing pain-specific education and psychological treatment [ - ]. As a result, most patients never receive the required education or skills training to promote self-management of pain [ - ].
However, the care for chronic pain is no longer strictly limited to medical environments and clinician-guided telehealth because of the rising number of mobile health (mHealth) products . mHealth describes the use of mobile technology to improve health [ ] by affecting the user’s education, motivation, and adherence [ , ]. It has already been applied to support mental as well as physical health programs [ ]. As such, mHealth can enhance the self-management of chronic conditions [ ]. Indeed, preliminary evidence suggests that pain management apps have great potential to support chronic pain treatment and are well received by patients [ , ] and health care professionals [ ]. In fact, a majority of studies reviewed by Thurnheer et al [ ] showed beneficial effects of the use of pain management apps on pain severity. As such apps are available anywhere, anytime [ ], they can reduce the frequency and cost of face-to-face interventions [ ]. Moreover, they can combine a variety of features (eg, educational content, a diary, personalized recommendations, and communication with health care professionals) within one app [ ]. As a result, they have the potential to make health care systems more effective [ ].
To ensure their effectiveness, apps for chronic pain management must be based on evidence-based content (ie, pain-related education, self-monitoring, goal setting, social support, and the training of self-care strategies including stress management) [, , ]. Even though these aspects are easy enough to implement, app reviews show that existing pain management apps have limited content. Rather than providing evidence-based behavior change programs, the reviewed pain management apps reveal a lack of combination of evidence-based functionalities [ - ]. Indeed, most apps only include 1 of the 5 suggested components [ , ]. Apps mostly focus on supplying information [ - ]. They seldom help to achieve social support and often lack evidence-based self-care skills and tracking of the multidimensional experience of pain. Most apps only allow to track pain intensity (eg, FitBack [ ]) [ , , , ]. Some apps also allow the user to track other pain-related aspects such as pain location, medication, and pain source (eg, Pain Squat [ ]). However, only a small number of apps also allow the assessment of emotional and cognitive aspects. In addition, the educational content that is included is often of poor quality. An exception is the pain diary app, PainTracker, which includes 3 of the 5 suggested functionalities. It allows the tracking of a number of pain-related aspects, allows goal setting, and provides informational content [ ]. Notwithstanding this exception, the overall lack of content in available pain management apps leads to a distrust in their effectiveness [ ]. As a result, comprehensive, evidence-based, and clinically informed smartphone apps for pain self-management are highly needed [ ].
Although important, the use of evidence-based content alone has been considered as insufficient to ensure adequate user engagement and motivation , two aspects that have great influence on the usage of an intervention program [ ]. In fact, further improvement is needed to make pain management apps more engaging and entertaining [ ].
One way to increase user engagement and motivation is through gamification. Gamification, the use of game elements in nongame contexts , aims to make interventions, such as health apps, more enjoyable, motivating, and engaging [ ]. However, the use of gamification in health apps has been critically discussed as its effects depend on the context and the goal of the app [ ]. Although users do not always want the implementation of gamification in health apps [ ], it could be shown that its use can have positive effects on both health and wellness [ ].
Gamification has already been shown to positively influence user self-management , lifestyle [ ], health and behavioral outcomes [ ], and the retention of desired user behaviors [ ]. This confirms that the implementation of gamification can be effective in promoting behavior change through apps [ ].
Nevertheless, few pain management apps make use of this concept. Two pain management apps for adolescent patients with cancer have included gamification in the form of a virtual rewards system and ranks that are linked to the users’ pain diaries and adherence [, ]. However, to our knowledge, there is no chronic pain management app with an extensive gamification framework for adults.
As current pain management apps lack in their use of both evidence-based self-management skills  and gamification [ ], we aimed to develop Pain-Mentor to close this gap in research. Pain management apps have great potential to promote patient self-care in out-clinic settings [ ]. Therefore, Pain-Mentor supports the therapy of patients with chronic pain by teaching evidence-based techniques from multi-modal pain therapy and how they can be applied in everyday life. The app is based on the concept of Stress-Mentor [ ], a gamified stress management app. Stress-Mentor includes all 5 suggested theory-based functionalities. For example, it realizes self-monitoring through a diary that allows the tracking of up to 14 diary categories (ie, sleep duration and quality, sport duration and intensity, daily uplifts and daily hassles, stress level, mood, digestion, consumption of water, fruits and vegetables, coffee and alcohol, and step count). In addition, the app teaches different self-help skills through daily and weekly tasks. In these tasks, the user can choose 1 of 3 suggested skills that he or she wants to practice. The techniques offered by the app depend on the user’s entries into a stress checklist. In the stress checklist, the user can enter stress-related aspects (ie, fears and worries, sadness, anger, stress at work, stress in private life, muscle tension, and head, neck, and back grievances caused by tension, digestive problems, and sleep problems) on a scale of 0 to 10 on a weekly basis. This concept of personalized tasks encourages the user to set daily and weekly goals and supports the repetition of exercises. The mediated skills include relaxation exercises (ie, abdominal breathing, meditation, mindfulness, progressive muscle relaxation, guided imagery, stretching exercises, and self-massage), problem solving (ie, time management, goal setting, planning of social support and social change, and barrier identification), cognitive aspects (ie, assertiveness training, refuting irrational ideas, appraisal of stress and stressful situations, and avoiding perfectionism), and the transfer of educational information about stress. Moreover, the app provides tips on stress based on the user’s documented stress level.
In addition to self-monitoring, stress management skills, and educational information, Stress-Mentor includes several other behavior change techniques (see the study by Christmann et al  for a detailed list) to support long-term behavior change. The included behavior change techniques are linked to an extensive gamification concept aimed at motivating and engaging the user [ ]. As such, the app includes an avatar (a bird-like cartoon animal) that provides feedback by reflecting both the user’s diary entries (vicarious reinforcement) [ , ] and progress. Another aspect is the app’s agent (a wise owl), who is a mentor that entrusts the care of the avatar to the user via a behavioral contract and provides instructions on app functions, general encouragement, and educational tips on stress. The user can collect woodland coins that can later be exchanged for items for the user’s avatar. Moreover, the app provides feedback on the user’s performance through progress bars, a diary overview diagram [ ], badges, and the visual development of the avatar and its surroundings. A detailed description of the implemented behavior change techniques and how they were linked with gamification was previously published [ ].
As stress management and cognitive and behavioral aspects play an important role in the treatment of chronic pain , Stress-Mentor’s concept was adopted for the first prototype of the pain management app (Pain-Mentor) evaluated in this study. Although all stress-related content remains, additions were made to the existing diary, tips, symptoms checklist, and daily tasks to further adapt Pain-Mentor to the context of chronic pain treatment. We made the following adjustments to better suit Pain-Mentor to its designated usage context. First, the stress checklist was renamed to symptoms checklist. The symptoms checklist was then extended with a numerical rating scale for pain that is commonly used in therapy. It allows the user to enter his or her pain level on a scale of 0 (no pain) to 10 (worst possible pain) [ ]. The diary was also extended by this scale. This provides users with the opportunity to track the trend of their pain on a daily basis [ ]. A total of 8 pain-specific daily tasks were added to the task pool: 1 each to develop a plan for setbacks, planning social support for pain management, and planning a dropped activity and 5 physical exercises for muscle strengthening and stretching. Moreover, the tips given by the app’s mentor were extended with additional information on chronic pain and pain management. A screenshot of the app is displayed in .
All in all, Pain-Mentor differs from other pain management apps regarding one important property: it includes all 5 suggested self-management functionalities (ie, educational information on pain and stress, a total of 87 pain-specific and stress-specific self-help skills, goal setting through tasks of the day and tasks of the week, multidimensional self-monitoring, and social support) and combines this content with gamification to motivate and engage the user. This poses great potential for supporting in-person therapy and reducing therapy costs .
In contrast to the general recommendation, most pain management apps are not based on scientific evidence and have not been thoroughly tested . This also means that designers have neither included experts from pain management in the development of their apps [ ] nor used expert reviews to assess their quality [ ]. Contrary to this, Pain-Mentor’s contents were extended in consultation with a physician who specialized in chronic pain treatment. The involvement of health care professionals in the development of health apps, as was done for Pain-Mentor, is important for health apps to contribute value to the delivery of health care and chronic disease management [ ]. With many apps promising effective pain treatment, patients face a large array of possible apps to choose from, with little guidance regarding their quality [ ]. To ensure quality, functionality, and relevance of the content, health apps need to be tested in scientific trials and must involve health care professionals not only during their development but also in the evaluation process [ ].
In general, health apps must be acceptable to both the user who must decide whether the program is usable and can provide benefit in an operational environment and the health professionals who determine whether the app does what it is supposed to do . In contrast to the users’ goal, the experts’ primary goal is to assess the quality of a health app to identify apps that can be recommended to their patients [ ]. They focus on different aspects and provide different feedback than users and developers [ ]. Therefore, even though experts have scientifically evaluated few pain management apps [ ], testing the quality of health apps through expert evaluations is essential to assess the quality of key app features [ ].
This study, therefore, conducted an expert evaluation of the first prototype of the newly developed pain management app, Pain-Mentor, that combines a multimodal approach to pain self-management with an extensive gamification framework. The aim of this study was to gather information on how to further improve Pain-Mentor to create an app that has high value, the potential to positively influence chronic pain patients, and is accepted and recommended by medical professionals. For this purpose, the app’s general quality was evaluated from the perspective of health professionals. This approach enabled us to identify areas that need improvement and helped to further adjust Pain-Mentor for the purpose of its application.
To assess the app’s quality per the standards of health professionals, experts with a background in chronic pain management and mHealth development were recruited (). For this purpose, based on an internet search, physicians specializing in pain treatment and general psychotherapists in a 100-km radius of the Technische Universität Kaiserslautern (Germany) were contacted via email. Among the 94 experts contacted, 8 were willing to participate in this study. An additional 5 experts learned about the study from one of their colleagues and volunteered to participate as a result. In the end, 13 experts (5 physicians, 1 nurse with a background in pain management, 5 psychotherapists, and 2 mHealth developers) participated in this study. Previous research suggested the use of at least 2 to 4 experts [ - ], although a larger sample size increases the percentage of identified problems in the apps. As about 95% of all problems can be identified with as many as 9 participants [ ], it can be concluded that the sample size of this study provided good insight into the app’s quality and enabled to identify most of the concerns arising from experts from chronic pain treatment. All participants had specific experience in the field of pain and pain management.
The whole procedure was approved by the local ethics committee from the Department of Social Sciences, Technische Universität Kaiserslautern.
To ensure a standardized approach, the following study procedure was predefined. At the beginning, the health professional was informed about the procedure, aim, and data collected in the study. Each participant gave written consent according to the Declaration of Helsinki. Afterward, Pain-Mentor and its contents were presented to the professional in detail in a slide presentation that explained the pain and stress management methods, behavior change techniques, and gamification aspects that were included and how they were interconnected. After the presentation, the expert tested the app () on a tablet (Lenovo TB-4706F). For this purpose, the app was set to a specific prerequisite to ensure all participants were exposed to the same content and features.
Testing also followed a predefined process that took 15 to 25 min for each participant (seefor a detailed description).
After testing Pain-Mentor, all experts rated the quality of the app and were asked for feedback. For this purpose, the mobile application rating scale (MARS)  was applied. MARS was specifically designed to assess the quality of health apps with the help of experts from health and information technology [ ]. MARS comprises 6 subscales. Four of those scales (ie, engagement, functionality, esthetics, and information quality) assess the general app quality; the subjective quality section evaluates the user’s overall satisfaction; whereas the app-specific section assesses the perceived impact of the app on the user’s knowledge, attitudes, intentions to change, and the likelihood of actual change in the target health behavior. The complete structure of MARS [ ] is listed in .
|Section and subsection||Definition||Items|
|A: Engagement||Fun, interesting, customizable, interactive (eg, sends alerts, messages, reminders, and feedback and enables sharing), and well targeted to audience|
|B: Functionality||App functioning, easy to learn, navigation, flow logic, and gestural design of app|
|C: Aesthetics||Graphic design, overall visual appeal, color scheme, and stylistic consistency|
|D: Information||Contains high quality information (eg, text, feedback, measures, and references) from a credible source|
|App’s subjective quality||Contains subjective, personal opinion of the app, including recommendation to others, estimated usage, willingness to pay, and overall star rating|
|App-specific||Perceived impact of the app on the user’s knowledge, attitudes, and intentions to change as well as the likelihood of actual change in the target health behavior|
Participants rated each of the 23 MARS items on a 5-point Likert scale (from 1=inadequate to 5=excellent). To allow for differentiated user feedback, MARS was applied as a semistructured interview. This means, after each rating, participants had the opportunity to explain their answer and give suggestions regarding further improvement of the app (open-response format). Presenting MARS and other questionnaires as semistructured interviews has been done in previous studies and promises deeper insights into the raters’ reasoning and possible improvements (eg, the study by Anderson et al ). As the experts spent limited time trying out the app (approximately 20-30 min), an additional answer option was added to each question, namely, “I cannot assess this.” This ensured that the experts were not forced to answer, if they felt that they did not have enough time with the app to assess an aspect.
Overall, 3 questions from the subscale, subjective quality, were removed from the questionnaire: (1) question 18, “Does the app come from a credible source?” because the source of the app was explained to the participants in detail; (2) question 19, “Has the app been tested?” because an evaluation regarding the app’s effectiveness has not been conducted so far; and (3) question 22, “Would you pay for this app?” because health experts are not the target user audience of this app. In addition, question 20 was adapted to the context and changed into “Would you recommend this app to patients who might benefit from it?” The questions from the app-specific section were adapted to the context, ie, the term health behavior was replaced with stress and pain management.
Additional App-Specific Questions
In addition to MARS, the participants answered 7 additional questions on a 5-point Likert scale (from 1=inadequate to 5=excellent) and 1 open-response question with regard to the expected app’s appeal for patients and specific app features. As with MARS, participants received the opportunity to explain their answers. Seefor a list of all additional questions.
Mobile Application Rating Scale–Based Outcomes
Overall, experts rated Pain-Mentor to be of excellent quality (mean 4.54, SD 0.55). Subjective app quality was appraised as excellent (mean 4.57, SD 0.43). The app-specific questions were rated as good, with a mean of 4.38 (SD 0.75).
The MARS quality subsection, engagement, was rated as good, and functionality, esthetics, and information were rated as excellent (seefor mean values and standard deviations).
Evaluating each question of these subsections in detail, the results showed a mean value ranging from of 4.07 (SD 0.76) for customization to 4.71 (SD 0.47) for interest for the subsection engagement. Functionality showed mean rating between 4.43 (SD 0.65) for ease of use and 4.77 (SD 0.44) for navigation. The app’s esthetics were assessed as excellent, with mean values of 4.43 (SD 0.51) for visual appeal, 4.50 (SD 0.65) for layout, and 4.64 (SD 0.50) for graphics. The information communicated in Pain-Mentor also showed good to excellent ratings, with means ranging from 4.21 (SD 0.67) for the quality of information and 4.92 (SD 0.28) for information quantity. The subjective quality of the app showed ratings of 4.36 (SD 0.50) with regard to the recommendation of the app to patients and the overall star rating and 4.50 (SD 0.52) for usage duration. The app-specific questions of MARS showed values between 3.86 (SD 0.95) with regard to encouraging patients to seek help outside the app and 4.75 (SD 0.46) regarding the app’s potential to promote behavior change. A detailed visualization of the results is included in.
Occasionally, experts were unable to answer a question (seefor details). Participants gave several reasons for being unable to assess these questions, which will be discussed in the Limitations.
In addition to the quantitative ratings, applying MARS as a semistructured interview provided differentiated expert feedback. The experts’ comments are discussed in the Principal Findings section of the Discussion.
Outcomes From Additional Questions
Good to excellent mean ratings could be observed for all additional questions (see). Experts thought that it was highly likely that Pain-Mentor would appeal to patients (mean 4.50, SD 0.52). The app’s gamification concept was rated as good (mean 4.29, SD 0.61). The experts’ expectations of Pain-Mentor were mostly fulfilled (mean 4.35, SD 0.63). They rated the implemented diary (mean 4.71, SD 0.61), daily tasks (mean 4.64. SD 0.63), and symptoms checklist (mean 4.79, SD 0.43) as sensible and useful. Moreover, they believed that using Pain-Mentor to support in-person therapy would be useful (mean 4.57, SD 0.51). All experts were able to assess all additional questions.
|Question||Rating, mean (SD)|
|Do you think the app would appeal to patients?||4.50 (0.52)|
|Did you like the gamification concept?||4.29 (0.61)|
|Did the app meet your expectations?||4.35 (0.63)|
|How useful is the diary?||4.71 (0.61)|
|How useful is the concept of daily exercises?||4.64 (0.63)|
|How useful is the symptoms checklist?||4.79 (0.43)|
|How useful is it to apply the app in addition to therapy?||4.57 (0.51)|
Overall, experts rated the app to be of excellent overall and subjective quality. The app-specific section of MARS was rated as good. These positive results reflect that the app and its contents (ie, diary, daily tasks, information, symptoms checklist, and gamification concept) met the experts’ expectations. As expected, these results show that combining the 5 suggested self-management functionalities (ie, educational information, self-care skills, self-monitoring, goal setting, and social support) [- ] in an app targeting chronic pain management is approved by health experts.
In addition, the experts thought that the app’s gamification concept, especially the avatar, made the program engaging and interesting to use. This was mirrored by the experts’ comments, eg, E4 said, “I think the app motivates those patients who want to be proactive and do something to manage their pain.” This implicates that the use of gamification could pose a solution to the lack of engagement and entertainment in pain management apps as mentioned in the study by Jamison et al . It also agrees with the results from a user study with the stress management app, Stress-Mentor, which showed that the app’s gamified concept led to increased usage compared with a nongamified control group (Hoffman et al, unpublished data). Though this is a promising indication of the effectiveness of the concept of Pain-Mentor, a user study with patients with chronic pain is needed to show whether these results remain.
Two experts expressed concerns that the existing gamification concept, especially the choice in avatar, might be too childlike and not suitable for the elderly. Although most users prefer human avatars that match their own gender , it is very subjective as to which avatar appeals to a user. To solve this problem, the participants suggested providing the user a choice among different avatars and including more options to adapt the avatar to the user’s preferences. Despite some experts’ skepticism regarding the avatar’s suitability for the elderly, other experts thought that Pain-Mentor was well suited for the target group (ie, adults with chronic pain) and that it was very likely that the app would appeal to patients. This further supports the combination of evidence-based content and gamification [ , - ] and indicates that such approaches should not be limited to adolescents. In addition, gamification most likely had a positive impact on the app’s esthetics.
Experts especially liked that they could choose the avatar’s color and that the avatar’s appearance was linked to both diary entries and progress (eg, E9 stated, “It’s nice that the user can pick the avatar’s color. Especially the visual elements invite you to explore and play around a little”), and they liked that the app’s simple visual design allowed them to use it intuitively. This backs both personalization  and the use of vicarious reinforcement through avatars in health apps [ ].
Although the experts were mostly satisfied with Pain-Mentor’s customizability, they also suggested including more reminders to help the user remember to practice throughout the day. This shows that one reminder is the minimum, whereas the inclusion of more appears to be preferable. Nonetheless, the reminder function was generally perceived positively, and the experts thought that this feature was likely to promote the user’s self-commitment. Overall, 7 experts also suggested the addition of new diary aspects. However, there was no consensus among the experts on the aspects that should be added (they suggested, eg, weight, additional dietary aspects, pain location, and notation of additional exercises). Thus, it cannot be concluded which categories would make the most sense to be added based on the feedback obtained from this study. Moreover, the addition of aspects could easily overwhelm the users and lead to a decline in the app’s usability and usage . Furthermore, 2 experts also commented that not all aspects are equally important for every patient. To solve this issue, the developers could add a notes section that leaves room for further patient-specific entries, as has been implemented in other pain management apps (eg, Chronic Pain Tracker and FitBack) [ , ]. Another approach could be to allow the addition of individual scales in the diary [ ]. Both solutions would leave the choice of adding further diary categories to the user and his or her treating health professional.
Regardless of the suggested extensions, the diary was perceived as very useful by the experts. One participant expressly mentioned that she especially liked that the diary was not overly focused on pain, but rather allowed for tracking the patient’s overall well-being, including emotional (stress level and mood) and cognitive aspects (daily hassles and daily uplifts). This goes in line with previous studies [, ] that criticized that most pain management apps focused only on tracking medication and pain levels. Another aspect that was mentioned was the advantage for patients in keeping a digital diary instead of applying a pen and paper approach [ ].
Though the app’s interactivity was assessed as good, one expert commented that it could be further improved by giving advice based on the user’s diary entries. Consequently, developers should think about further personalizing their apps by linking the users’ entries (eg, from a diary) to suitable health information and tasks. For example, the app, MyBehavior, automatically provides personalized suggestions based on a health diary .
Participants thought that the information imparted in the app was generally well formulated and of high quality. Moreover, experts would recommend the app to many of their patients based on whether he or she would profit from using the app. This shows that the experts thought the app to be a good therapy supplement. However, their recommendation largely depends on the patients’ age, disease pattern, and current state. Nevertheless, age does not necessarily impact compliance or satisfaction with a pain management app . Moreover, the combination of visual features (because of the gamification concept) and content [ ] was perceived very positively (eg, E9 said, “It [the app] has a good balance of simple visual design and good content-related information”). This further supports the use of gamification in the context of pain management.
There was general approval of the self-management skills that are imparted in the app. Nevertheless, 3 experts suggested including additional tasks, such as more stretching exercises, more tasks specifically aimed at dealing with pain, and exercises aimed at distracting patients from their pain. This emphasized that experts see the potential of using apps to teach a large number of stress-related self-help skills to the user, including relaxation, problem solving, and cognitive aspects . However, it should be supplemented by more pain-specific aspects to provide maximal suitability.
The experts also mentioned that they would like to be able to review the app’s data with their patients on a computer to monitor and discuss their patients’ progress. However, automatic data transmission to the treating health professionals was seen as problematic because of the experts’ limited availability of time. To avoid this problem, an optional sharing function could be added that allows patients to share their data with the health professionals on a voluntary basis . Such functions provide health professionals the opportunity to gather data on a patient’s behavior [ ]. My Pain Diary, eg, offers the export of data to a computer [ ].
The experts’ positive ratings of the app’s ability to positively influence patients’ awareness, knowledge, attitude, intention to change, help-seeking, and behavior change are in line with the fact that health apps can enhance users’ self-management of chronic conditions . However, it was emphasized that the individual played an important role regarding these aspects. Nonetheless, the experts thought that it would be very useful to employ the app to supplement in-person therapy. This highlights the potential of health apps to support regular treatment [ ].
In addition to using the app for therapy support, one expert suggested to use the app to bridge the time until patients can receive in-person therapy. As waiting times for therapy are often long, using health apps in this manner could further increase their potential to improve health care  and diminish the number of patients who do not receive adequate care [ ]. This further supports the conclusion that pain management apps could be especially beneficial in out-clinic settings [ ]. Therefore, this area of application should be the focus of future research.
When asked which aspects they thought would keep patients from using the app, experts mostly mentioned a lack of motivation for people to change. However, they also mentioned that it is often difficult for patients to deal with the subject matter. In addition, they mentioned a lack of familiarity with mobile devices, apps and data security. This is in line with previous studies that have shown that technical affinity  and data privacy and security [ ] are important aspects for choosing and using health technologies. Therefore, developers should make sure they pay special attention to data security when developing health apps [ ]. Pain-Mentor, for instance, only saves the user’s data locally on his or her smartphone in an encrypted form.
Overall, Pain-Mentor’s MARS ratings are similar to those of the best-rated pain management apps reviewed by Salazar et al . Averaged across all reviewed apps, the mean functionality was assessed as good, whereas Pain-Mentor received an excellent score. Similar observations can be made for esthetics (average for Salazar et al [ ] and excellent for Pain-Mentor), engagement (average for Salazar et al [ ] and good for Pain-Mentor), information (average for Salazar et al [ ] and excellent for Pain-Mentor), subjective quality (average for Salazar et al [ ] and excellent for Pain-Mentor), and app-specific scores (poor for Salazar et al [ ] and excellent for Pain-Mentor).
Although the experts’ comments showed that some adjustments, such as adding more pain-specific exercises, diary categories, and reminders, could further improve the app, they still rated Pain-Mentor to be of excellent overall quality (mean rating of average for Salazar et al ). This shows that minor adjustments of suitable health apps (eg, from stress management) can make them useful tools that can be applied in different contexts (eg, chronic pain management). Nonetheless, whether an app will be assessed as useful or not depends on both the app’s content and the suggested context of use [ ]. Not every health app should be applied in or adjusted for other contexts. Underlining the importance of expert evaluations [ ], this study showed that involving health experts from the target context helps to determine an app’s suitability and to identify necessary adjustments.
The experts approved of the app’s gamification concept and thought that using gamification in this manner could improve patients’ motivation and engagement. As experts and patients focus on different aspects , a next step should be to get the opinion of patients with chronic pain to further adjust the app to their needs. Though user studies showed that the app’s gamified concept is accepted by users in the context of stress management (Hoffman et al, unpublished data and [ ]), it remains unclear as to what extent the results of this study hold true in the context of pain management. In addition, further research is needed to determine the effects of gamification on users’ behavior. Moreover, although experts approved of the idea to use the app to support therapy, randomized controlled trials are needed to actually determine the effectiveness of Pain-Mentor as a therapy support tool.
The inclusion of 9 experts has been suggested to be sufficient to reveal most problems within an app . Thus, although only 13 experts participated, in this study, we should have identified the most important improvements that are required. Nonetheless, for future studies, it should be noted that it can be hard to obtain experts to participate. Out of 94 contacts, only a few were willing to participate. A major reason for this was the lack of time to accommodate the study within their busy work schedule.
Other aspects that future studies might encompass include the participating experts’ affinity to technology and their knowledge in testing mobile apps. Though the participants’ affinity to technology was not systematically collected in this study, 3 participants mentioned that they were not proficient in using smartphones and tablets. This means, not all participating experts had high technological knowledge.
As all experts were given a detailed introduction of Pain-Mentor and spent approximately 20 to 30 min using the app, they received detailed insights into how the app worked. Nonetheless, experts had trouble assessing the app’s gestural design and the app’s potential to positively affect behavior change. Future studies could avoid this problem through longer trial periods.
This study provided a first affirmation of Pain-Mentor’s concept. The participating health experts approved of the app’s gamification aspects and described this approach as a good way to enhance user motivation and engagement. Moreover, the app received positive ratings with regard to general and subjective quality as well as app-specific aspects (MARS). This showed that the use of gamification did not have a negative impact on the app’s credibility and integrity and that the combination of gamification with the 5 recommended self-management functionalities (ie, pain-related education, self-monitoring, goal setting, social support, and the training of self-care strategies) led to an overall positive evaluation of Pain-Mentor. This indicated that the app’s development is on the right track.
The study also showed that applying MARS in combination with additional, more app-specific questions in a semistructured interview can provide insights into the participants’ ratings and disclose possible areas for improvement. In fact, this approach revealed areas where adjustments need to be made to further tailor Pain-Mentor for its application as a support tool for chronic pain therapy. The applied approach therefore helps to adjust health apps for a specific target audience and to identify further scenarios for application.
The authors thank Dr Katja Regenspurger for her input. The authors also thank the health experts who volunteered to participate in this study. The junior research group, wearHEALTH, is funded by the Federal Ministry of Education and Research (Bundesministerium für Bildung und Forschung, reference number: 16SV7115).
Conflicts of Interest
In accordance with our ethical obligation as researchers, we report that the authors of this paper developed Pain-Mentor, which is the focus of the presented research.
List of additional questions for the semistructured expert interview.DOCX File , 14 KB
Means and standard deviations of the experts’ ratings for each category of the mobile application rating scale questionnaire.DOCX File , 12 KB
List of questions that at least one expert felt unable to assess and the number of experts that were unable to assess each of these questions.DOCX File , 13 KB
- American Psychological Association. 2013. Managing Chronic Pain: How Psychologists Help With Pain Management URL: http://www.apa.org/helpcenter/pain-management.pdf [accessed 2018-07-10] [WebCite Cache]
- Breivik H, Collett B, Ventafridda V, Cohen R, Gallacher D. Survey of chronic pain in Europe: prevalence, impact on daily life, and treatment. Eur J Pain 2006 May;10(4):287-333. [CrossRef] [Medline]
- Becker N, Thomsen AB, Olsen AK, Sjøgren P, Bech P, Eriksen J. Pain epidemiology and health related quality of life in chronic non-malignant pain patients referred to a Danish multidisciplinary pain center. Pain 1997 Dec;73(3):393-400. [CrossRef] [Medline]
- Carmona L, Ballina J, Gabriel R, Laffon A, EPISER Study Group. The burden of musculoskeletal diseases in the general population of Spain: results from a national survey. Ann Rheum Dis 2001 Nov;60(11):1040-1045 [FREE Full text] [CrossRef] [Medline]
- Molander P, Dong HJ, Äng B, Enthoven P, Gerdle B. The role of pain in chronic pain patients' perception of health-related quality of life: a cross-sectional SQRP study of 40,000 patients. Scand J Pain 2018 Jul 26;18(3):417-429. [CrossRef] [Medline]
- Pérez C, Margarit C, Sánchez-Magro I, de Antonio A, Villoria J. Chronic pain features relate to quality of life more than physiopathology: a cross-sectional evaluation in pain clinics. Pain Pract 2017 Sep;17(7):866-878. [CrossRef] [Medline]
- Murray E, Burns J, See TS, Lai R, Nazareth I. Interactive Health Communication Applications for people with chronic disease. Cochrane Database Syst Rev 2005 Oct 19(4):CD004274. [CrossRef] [Medline]
- Wantland DJ, Portillo CJ, Holzemer WL, Slaughter R, McGhee EM. The effectiveness of web-based vs non-web-based interventions: a meta-analysis of behavioral change outcomes. J Med Internet Res 2004 Nov 10;6(4):e40 [FREE Full text] [CrossRef] [Medline]
- Alexander JC, Joshi GP. Smartphone applications for chronic pain management: a critical appraisal. J Pain Res 2016;9:731-734 [FREE Full text] [CrossRef] [Medline]
- Stinson J, White M, Isaac L, Campbell F, Brown S, Ruskin D, et al. Understanding the information and service needs of young adults with chronic pain: perspectives of young adults and their providers. Clin J Pain 2013 Jul;29(7):600-612. [CrossRef] [Medline]
- Stinson JN, Lalloo C, Harris L, Isaac L, Campbell F, Brown S, et al. iCanCope with Pain: User-centred design of a web- and mobile-based self-management program for youth with chronic pain based on identified health care needs. Pain Res Manag 2014;19(5):257-265 [FREE Full text] [CrossRef] [Medline]
- Irvine AB, Russell H, Manocchia M, Mino DE, Glassen TC, Morgan R, et al. Mobile-Web app to self-manage low back pain: randomized controlled trial. J Med Internet Res 2015 Jan 2;17(1):e1 [FREE Full text] [CrossRef] [Medline]
- Macea DD, Gajos K, Calil YA, Fregni F. The efficacy of web-based cognitive behavioral interventions for chronic pain: a systematic review and meta-analysis. J Pain 2010 Oct;11(10):917-929. [CrossRef] [Medline]
- Jensen MP, Nielson WR, Kerns RD. Toward the development of a motivational model of pain self-management. J Pain 2003 Nov;4(9):477-492. [CrossRef] [Medline]
- Flor H, Fydrich T, Turk DC. Efficacy of multidisciplinary pain treatment centers: a meta-analytic review. Pain 1992 May;49(2):221-230. [CrossRef] [Medline]
- Elgar FJ, McGrath PJ. Self-administered psychosocial treatments for children and families. J Clin Psychol 2003 Mar;59(3):321-339. [CrossRef] [Medline]
- Barlow JH, Ellard DR. The psychosocial well-being of children with chronic disease, their parents and siblings: an overview of the research evidence base. Child Care Health Dev 2006 Jan;32(1):19-31. [CrossRef] [Medline]
- Stinson JN, Jibb LA, Nguyen C, Nathan PC, Maloney AM, Dupuis LL, et al. Development and testing of a multidimensional iPhone pain assessment application for adolescents with cancer. J Med Internet Res 2013 Mar 8;15(3):e51 [FREE Full text] [CrossRef] [Medline]
- Peng P, Stinson JN, Choiniere M, Dion D, Intrater H, Lefort S, STOPPAIN Investigators Group. Dedicated multidisciplinary pain management centres for children in Canada: the current status. Can J Anaesth 2007 Dec;54(12):985-991. [CrossRef] [Medline]
- Peng P, Choiniere M, Dion D, Intrater H, Lefort S, Lynch M, STOPPAIN Investigators Group. Challenges in accessing multidisciplinary pain treatment facilities in Canada. Can J Anaesth 2007 Dec;54(12):977-984. [CrossRef] [Medline]
- Lynch ME, Campbell F, Clark AJ, Dunbar MJ, Goldstein D, Peng P, et al. A systematic review of the effect of waiting for treatment for chronic pain. Pain 2008 May;136(1-2):97-116. [CrossRef] [Medline]
- Demiris G, Afrin LB, Speedie S, Courtney KL, Sondhi M, Vimarlund V, et al. Patient-centered applications: use of information technology to promote disease management and wellness. A white paper by the AMIA knowledge in motion working group. J Am Med Inform Assoc 2008;15(1):8-13 [FREE Full text] [CrossRef] [Medline]
- Harrison V, Proudfoot J, Wee PP, Parker G, Pavlovic DH, Manicavasagar V. Mobile mental health: review of the emerging field and proof of concept study. J Ment Health 2011 Dec;20(6):509-524. [CrossRef] [Medline]
- Handel MJ. mHealth (mobile health)-using Apps for health and wellness. Explore (NY) 2011;7(4):256-261. [CrossRef] [Medline]
- Ahtinen A, Mattila E, Välkkynen P, Kaipainen K, Vanhala T, Ermes M, et al. Mobile mental wellness training for stress management: feasibility and design implications based on a one-month field study. JMIR Mhealth Uhealth 2013 Jul 10;1(2):e11. [CrossRef] [Medline]
- WHO. mHealth: New Horizons for Health Through Mobile Technologies: Second Global Survey on eHealth. Geneva: World Health Organization; 2011.
- Anderson K, Burford O, Emmerton L. Mobile health apps to facilitate self-care: a qualitative study of user experiences. PLoS One 2016;11(5):e0156164 [FREE Full text] [CrossRef] [Medline]
- Jamison RN, Mei A, Ross EL. Longitudinal trial of a smartphone pain application for chronic pain patients: predictors of compliance and satisfaction. J Telemed Telecare 2018 Feb;24(2):93-100. [CrossRef] [Medline]
- Thurnheer SE, Gravestock I, Pichierri G, Steurer J, Burgstaller JM. Benefits of mobile apps in pain management: systematic review. JMIR Mhealth Uhealth 2018 Oct 22;6(10):e11231 [FREE Full text] [CrossRef] [Medline]
- Blackburne T, Rodriguez A, Johnstone SJ. A serious game to increase healthy food consumption in overweight or obese adults: randomized controlled trial. JMIR Serious Games 2016 Jul 13;4(2):e10 [FREE Full text] [CrossRef] [Medline]
- IMS Institute for Healthcare Informatics. Moodle - Université Droit et Santé. 2013 Oct. Patient Apps for Improved Healthcare: From Novelty to Mainstream URL: https://www.webcitation.org/76G4msQuC [accessed 2020-02-15]
- Morris ME, Kathawala Q, Leen TK, Gorenstein EE, Guilak F, Labhard M, et al. Mobile therapy: case study evaluations of a cell phone application for emotional self-awareness. J Med Internet Res 2010 Apr 30;12(2):e10 [FREE Full text] [CrossRef] [Medline]
- Chittaro L, Sioni R. Evaluating mobile apps for breathing training: The effectiveness of visualization. Comput Hum Behav 2014 Nov;40:56-63. [CrossRef]
- Rosser BA, Eccleston C. Smartphone applications for pain management. J Telemed Telecare 2011;17(6):308-312. [CrossRef] [Medline]
- Lalloo C, Jibb LA, Rivera J, Agarwal A, Stinson JN. 'There's a pain app for that': review of patient-targeted smartphone applications for pain management. Clin J Pain 2015 Jun;31(6):557-563. [CrossRef] [Medline]
- Wallace LS, Dhingra LK. A systematic review of smartphone applications for chronic pain available for download in the United States. J Opioid Manag 2014;10(1):63-68. [CrossRef] [Medline]
- de la Vega R, Miró J. mHealth: a strategic field without a solid scientific soul. a systematic review of pain-related apps. PLoS One 2014;9(7):e101312 [FREE Full text] [CrossRef] [Medline]
- Portelli P, Eldred C. A quality review of smartphone applications for the management of pain. Br J Pain 2016 Aug;10(3):135-140 [FREE Full text] [CrossRef] [Medline]
- Salazar A, de Sola H, Failde I, Moral-Munoz JA. Measuring the quality of mobile apps for the management of pain: systematic search and evaluation using the mobile app rating scale. JMIR Mhealth Uhealth 2018 Oct 25;6(10):e10718 [FREE Full text] [CrossRef] [Medline]
- Vandelanotte C, Spathonis KM, Eakin EG, Owen N. Website-delivered physical activity interventions a review of the literature. Am J Prev Med 2007 Jul;33(1):54-64. [CrossRef] [Medline]
- Webber KH, Tate DF, Ward DS, Bowling JM. Motivation and its relationship to adherence to self-monitoring and weight loss in a 16-week Internet behavioral weight loss intervention. J Nutr Educ Behav 2010;42(3):161-167. [CrossRef] [Medline]
- Deterding S, Khaled R, Nacke LE, Dixon D. Gamification: Toward a Definition. In: Proceedings of The ACM CHI Conference on Human Factors in Computing Systems. 2011 Presented at: CHI'11; May 7–12, 2011; Vancouver, BC, Canada URL: http://gamification-research.org/wp-content/uploads/2011/04/02-Deterding-Khaled-Nacke-Dixon.pdf
- AlMarshedi A, Wills G, Ranchhod A. Gamifying self-management of chronic illnesses: a mixed-methods study. JMIR Serious Games 2016 Sep 9;4(2):e14 [FREE Full text] [CrossRef] [Medline]
- Johnson D, Deterding S, Kuhn KA, Staneva A, Stoyanov S, Hides L. Gamification for health and wellbeing: a systematic review of the literature. Internet Interv 2016 Nov;6:89-106 [FREE Full text] [CrossRef] [Medline]
- Boulos MN, Gammon S, Dixon MC, MacRury SM, Fergusson MJ, Rodrigues FM, et al. Digital games for type 1 and type 2 diabetes: underpinning theory with three illustrative examples. JMIR Serious Games 2015 Mar 18;3(1):e3 [FREE Full text] [CrossRef] [Medline]
- Allam A, Kostova Z, Nakamoto K, Schulz PJ. The effect of social support features and gamification on a web-based intervention for rheumatoid arthritis patients: randomized controlled trial. J Med Internet Res 2015 Jan 9;17(1):e14 [FREE Full text] [CrossRef] [Medline]
- Kuo M, Chuang T. How gamification motivates visits and engagement for online academic dissemination – An empirical study. Comput Hum Behav 2016 Feb;55:16-27. [CrossRef]
- González CS, Gómez N, Navarro V, Cairós M, Quirce C, Toledo P, et al. Learning healthy lifestyles through active videogames, motor games and the gamification of educational activities. Comput Hum Behav 2016 Feb;55:529-551. [CrossRef]
- Jibb LA, Stevens BJ, Nathan PC, Seto E, Cafazzo JA, Johnston DL, et al. Perceptions of adolescents with cancer related to a pain management app and its evaluation: qualitative study nested within a multicenter pilot feasibility study. JMIR Mhealth Uhealth 2018 Apr 6;6(4):e80 [FREE Full text] [CrossRef] [Medline]
- Christmann CA, Hoffmann A, Zolynski G, Bleser G. Stress-Mentor: Linking Gamification and Behavior Change Theory in a Stress Management Application. In: Proceedings of the 2018 International Conference on Human-Computer Interaction. 2018 Presented at: HCI'18; July 15-20, 2018; Las Vegas, NV, USA p. 387-393.
- Christmann CA, Zolysnki G, Hoffmann A, Bleser G. Towards More Interactive Stress-Related Self-monitoring Tools to Improve Quality of Life. In: Proceedings of the 20th Congress of the International Ergonomics Association.: Springer, Cham; 2018 Presented at: IEA'18; August 31-September 1, 2018; Florence, Italy. [CrossRef]
- Bandura A, Ross D, Ross SA. Vicarious reinforcement and imitative learning. J Abnorm Psychol 1963 Dec;67:601-607. [CrossRef] [Medline]
- Christmann CA, Zolynski G, Hoffmann A, Bleser G. Effective Visualization of Long Term Health Data to Support Behavior Change. In: Proceedings of the 8th International Conference on Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management.: Springer, Cham; 2017 Presented at: DHM'17; July 9-14, 2017; Vancouver, BC, Canada p. 237-247. [CrossRef]
- Farrar JT, Young JP, LaMoreaux L, Werth JL, Poole MR. Clinical importance of changes in chronic pain intensity measured on an 11-point numerical pain rating scale. Pain 2001 Nov;94(2):149-158. [CrossRef] [Medline]
- Hawker GA, Mian S, Kendzerska T, French M. Measures of adult pain: Visual Analog Scale for Pain (VAS Pain), Numeric Rating Scale for Pain (NRS Pain), McGill Pain Questionnaire (MPQ), Short-Form McGill Pain Questionnaire (SF-MPQ), Chronic Pain Grade Scale (CPGS), Short Form-36 Bodily Pain Scale (SF-36 BPS), and Measure of Intermittent and Constant Osteoarthritis Pain (ICOAP). Arthritis Care Res (Hoboken) 2011 Nov;63(Suppl 11):S240-S252 [FREE Full text] [CrossRef] [Medline]
- Bender JL, Radhakrishnan A, Diorio C, Englesakis M, Jadad AR. Can pain be managed through the Internet? A systematic review of randomized controlled trials. Pain 2011 Aug;152(8):1740-1750. [CrossRef] [Medline]
- Kao C, Liebovitz DM. Consumer mobile health apps: current state, barriers, and future directions. Phys Med Rehabil 2017 May;9(5S):S106-S115. [CrossRef] [Medline]
- Rivera J, McPherson A, Hamilton J, Birken C, Coons M, Iyer S, et al. Mobile apps for weight management: a scoping review. JMIR Mhealth Uhealth 2016 Jul 26;4(3):e87 [FREE Full text] [CrossRef] [Medline]
- Stead WW, Haynes RB, Fuller S, Friedman CP, Travis LE, Beck JR, et al. Designing medical informatics research and library--resource projects to increase what is learned. J Am Med Inform Assoc 1994;1(1):28-33 [FREE Full text] [CrossRef] [Medline]
- Baptista S, Oldenburg B, O'Neil A. Response to 'Development and validation of the user version of the mobile application rating scale (uMARS)'. JMIR Mhealth Uhealth 2017 Jun 9;5(6):e16 [FREE Full text] [CrossRef] [Medline]
- Wang J, Yao NA, Liu Y, Geng Z, Wang Y, Shen N, et al. Development of a smartphone application to monitor pediatric patient-reported outcomes. Comput Inform Nurs 2017 Nov;35(11):590-598. [CrossRef] [Medline]
- Gibbons P, Boudreaux ED, Haskins BL. Using practical search strategies to identify health apps: A case study with smoking cessation. JournalMTM 2018;7(1):16-23. [CrossRef]
- Nielsen J. Usability Engineering. Burlington, Massachusetts, USA: Morgan Kaufmann Publishers; 1994.
- Barnum CM. Usability Testing Essentials: Ready, Set...Test!. Amsterdam: Elsevier; 2011.
- Creber RM, Maurer MS, Reading M, Hiraldo G, Hickey KT, Iribarren S. Review and analysis of existing mobile phone apps to support heart failure symptom monitoring and self-care management using the mobile application rating scale (MARS). JMIR Mhealth Uhealth 2016 Jun 14;4(2):e74 [FREE Full text] [CrossRef] [Medline]
- Virzi RA. Refining the test phase of usability evaluation: how many subjects is enough? Hum Factors 1992;34(4):457-468. [CrossRef]
- Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015 Mar 11;3(1):e27 [FREE Full text] [CrossRef] [Medline]
- Anderson K, Burford O, Emmerton L. App chronic disease checklist: protocol to evaluate mobile apps for chronic disease self-management. JMIR Res Protoc 2016 Nov 4;5(4):e204 [FREE Full text] [CrossRef] [Medline]
- Nowak KL, Rauh C. The influence of the avatar on online perceptions of anthropomorphism, androgyny, credibility, homophily, and attraction. J Comp Mediated Comm 2005 Nov;11(1):153-178. [CrossRef]
- Gay V, Leijdekkers P. Personalised mobile health and fitness apps: lessons learned from myFitnessCompanion. Stud Health Technol Inform 2012;177:248-253. [Medline]
- Thompson DV, Hamilton RW, Rust RT. Feature fatigue: when product capabilities become too much of a good thing. J Mark Res 2005;42(4):431-442. [CrossRef]
- Koskinen E, Salminen J. A customizable mobile tool for supporting health behavior interventions. Conf Proc IEEE Eng Med Biol Soc 2007;2007:5908-5911. [CrossRef] [Medline]
- Wharton CM, Johnston CS, Cunningham BK, Sterner D. Dietary self-monitoring, but not dietary quality, improves with use of smartphone app technology in an 8-week weight loss trial. J Nutr Educ Behav 2014;46(5):440-444. [CrossRef] [Medline]
- Rabbi M, Pfammatter A, Zhang M, Spring B, Choudhury T. Automated personalized feedback for physical activity and dietary behavior change with mobile phones: a randomized controlled trial on adults. JMIR Mhealth Uhealth 2015 May 14;3(2):e42 [FREE Full text] [CrossRef] [Medline]
- Luxton DD, McCann RA, Bush NE, Mishkind MC, Reger GM. mHealth for mental health: integrating smartphone technology in behavioral healthcare. Prof Psychol Res Pr 2011 Dec;42(6):505-512. [CrossRef]
- Czaja SJ, Charness N, Fisk AD, Hertzog C, Nair SN, Rogers WA, et al. Factors predicting the use of technology: findings from the Center for Research and Education on Aging and Technology Enhancement (CREATE). Psychol Aging 2006 Jun;21(2):333-352 [FREE Full text] [CrossRef] [Medline]
- Wilkowska W, Ziefle M. Privacy and data security in E-health: requirements from the user's perspective. Health Informatics J 2012 Sep;18(3):191-201. [CrossRef] [Medline]
- Lewis TL, Wyatt JC. mHealth and mobile medical Apps: a framework to assess risk and promote safer use. J Med Internet Res 2014 Sep 15;16(9):e210 [FREE Full text] [CrossRef] [Medline]
- Hoffmann A, Christmann CA, Bleser G. Gamification in stress management apps: a critical app review. JMIR Serious Games 2017 Jun 7;5(2):e13 [FREE Full text] [CrossRef] [Medline]
|MARS: mobile application rating scale|
|mHealth: mobile health|
Edited by G Eysenbach; submitted 17.12.18; peer-reviewed by T Janmohamed, J Moral-Munoz, K Usop; comments to author 27.08.19; revised version received 22.01.20; accepted 28.01.20; published 26.05.20
©Alexandra Hoffmann, Corinna A Faust-Christmann, Gregor Zolynski, Gabriele Bleser. Originally published in JMIR Formative Research (http://formative.jmir.org), 26.05.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on http://formative.jmir.org, as well as this copyright and license information must be included.