This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on http://formative.jmir.org, as well as this copyright and license information must be included.
In the past years, a mobile health (mHealth) app called the Dutch Talking Touch Screen Questionnaire (DTTSQ) was developed in The Netherlands. The aim of development was to enable Dutch physical therapy patients to autonomously complete a health-related questionnaire regardless of their level of literacy and digital skills.
The aim of this study was to evaluate the usability (defined as the effectiveness, efficiency, and satisfaction) of the prototype of the DTTSQ for Dutch physical therapy patients with diverse levels of experience in using mobile technology.
The qualitative Three-Step Test-Interview method, including both think-aloud and retrospective probing techniques, was used to gain insight into the usability of the DTTSQ. A total of 24 physical therapy patients were included. The interview data were analyzed using a thematic content analysis approach aimed at analyzing the accuracy and completeness with which participants completed the questionnaire (effectiveness), the time it took the participants to complete the questionnaire (efficiency), and the extent to which the participants were satisfied with the ease of use of the questionnaire (satisfaction). The problems encountered by the participants in this study were given a severity rating that was used to provide a rough estimate of the need for additional usability efforts.
All participants within this study were very satisfied with the ease of use of the DTTSQ. Overall, 9 participants stated that the usability of the app exceeded their expectations. The group of 4 average-/high-experienced participants encountered only 1 problem in total, whereas the 11 little-experienced participants encountered an average of 2 problems per person and the 9 inexperienced participants an average of 3 problems per person. A total of 13 different kind of problems were found during this study. Of these problems, 4 need to be addressed before the DTTSQ will be released because they have the potential to negatively influence future usage of the tool. The other 9 problems were less likely to influence future usage of the tool substantially.
The usability of the DTTSQ needs to be improved before it can be released. No problems were found with satisfaction or efficiency during the usability test. The effectiveness needs to be improved by (1) making it easier to navigate through screens without the possibility of accidentally skipping one, (2) enabling the possibility to insert an answer by tapping on the text underneath a photograph instead of just touching the photograph itself, and (3) making it easier to correct wrong answers. This study shows the importance of including less skilled participants in a usability study when striving for inclusive design and the importance of measuring not just satisfaction but also efficiency and effectiveness during such studies.
Electronic health (eHealth) is developing rapidly [
The development of a specific form of eHealth technology, called mobile health (mHealth) technology, seems especially promising when it comes to reducing health inequalities [
Still, having access to the internet and digital technology does not automatically mean that people are able and willing to use it effectively to increase their health or that different people use it in the same way [
In the past years, a prototype of an mHealth app, called the Dutch Talking Touch Screen Questionnaire (DTTSQ), was developed in The Netherlands. The idea of developing a talking touch screen was inspired by the work of Hahn and Cella [
The aim of this study was to test the prototype of the DTTSQ within the physical therapy context to see what parts of the prototype needed adjustment for it to be user-friendly for physical therapy patients regardless of their level of experience with operating mobile technology.
The research question underlying this study was:
What is the usability of the prototype of the DTTSQ for physical therapy patients with different levels of experience in using mobile technology?
A qualitative descriptive study was carried out. Observational data on the way participants operated the DTTSQ were collected through the Three-Step Test-Interview (TSTI) method [
Data were collected in the same study population and at the same time as the data reported in a paper earlier published by Welbie et al [
Characteristics of study population (N=24).
Characteristics | Study population | |
Age (years), mean (range) | 56 (18-79) | |
|
||
|
Male | 9 (38) |
|
Female | 15 (62) |
|
||
|
Lowa | 6 (25) |
|
Moderateb | 13 (54) |
|
Highc | 5 (21) |
|
||
|
None | 9 (37) |
|
Little | 11 (46) |
|
Average/high | 4 (17) |
aLow: no or at most primary education finished.
bModerate: lower secondary education, (upper) secondary education, or postsecondary nontertiary education (including vocational education).
cHigh: tertiary education (bachelor’s degree or higher).
Characteristics per participant.
Pseudonym | Experience with mobile technology | Age (years) | Level of education |
Ida | None | 66 | Lowa |
Bill | None | 72 | Moderateb |
Mia | None | 73 | Moderate |
Dora | None | 77 | Low |
Ilene | None | 79 | Low |
Bob | None | 68 | Moderate |
Jerome | None | 47 | Low |
Helga | None | 54 | Highc |
Michelle | None | 56 | Low |
Roger | Little | 70 | Moderate |
Peter | Little | 18 | Moderate |
Christine | Little | 39 | Moderate |
Jill | Little | 55 | High |
Lydia | Little | 56 | Moderate |
Rose | Little | 60 | Moderate |
Francine | Little | 61 | Moderate |
Harald | Little | 63 | High |
Henry | Little | 64 | Moderate |
Ronald | Little | 70 | Low |
Bernie | Little | 76 | High |
Jude | Average/high | 18 | Moderate |
Joline | Average/high | 19 | Moderate |
Ellen | Average/high | 32 | High |
Sandra | Average/high | 39 | Moderate |
aLow: no or at most primary education finished.
bModerate: lower secondary education, (upper) secondary education, or postsecondary nontertiary education (including vocational education).
cHigh: tertiary education (bachelor’s degree or higher).
The prototype of the DTTSQ was a digital app on a tablet computer. It was developed during a co-design process [
Instructions were given in the form of 3 video clips:
An introduction clip in which the purpose of the questionnaire and all functions of the questionnaire were explained (see
An instruction clip in which the purpose of question 4 and a newly added navigation function were explained (see screenshot 6 in
A closing clip in which the participant is thanked, explained what the physical therapist would do next, and told that the questionnaire would close down automatically (see screenshot 16 in
Introduction movie.
It is a navigation function to go to the next screen. It is not activated unless a response item is selected (except for question 4; see
It activates the help function: the text on the screen is read aloud, the purpose of the question is explained, and operating instructions for the particular screen are provided.
Tapping a second time on a response item, deselects the item.
It is an escape function: it shuts down the questionnaire. All previous given answers are saved.
To help participants keep track of their answers, overviews of previous given answers were provided regularly during completion of the questionnaire (see
Question 4: “Select the activities in which you are limited”.
Overview answers total questionnaire: “On the screen you see an overview of all your answers you provided until now.”.
Data collection took place at the physical therapy practice or the participant’s home, depending on the preference of the participant. Researchers IT and JS were present. Researcher IT was in the lead during the interviews. Researcher JS asked complementary questions if she missed information.
The following steps were taken according to the TSTI method [
Each participant was observed by researchers IT and JS while they were completing the DTTSQ thinking out loud. This step was aimed at collecting observational data regarding the usability of the DTTSQ. The data collected consisted of 2 kinds: (1) observations of participant’s behavior and (2) think-aloud data. The data were recorded on videotapes as well as audiotapes. In addition, the researchers took real-time notes for use during the following steps of the interviews as well as for later analysis. The researchers wrote their notes down on hardcopies of print screens of the DTTSQ. Researchers IT and JS noted problems with operating the tablet computer, including using the touch screen, navigating through the questionnaire, understanding the task given in each screen, selecting response items, and using the correction function. They also wrote down when the stop button was used. The researchers did not interfere in the completion process by asking any questions or providing help.
Researcher IT conducted an in-depth interview after the participant finished completing the DTTSQ. Data collection during this step was exclusively focused on filling possible gaps and checking the observational data collected during step 1.
During step 3 of the TSTI, researcher IT conducted a semistructured interview aimed at eliciting experiences and opinions of the participant. During the interview, each screen was operated in the same way the participant did during step 1 and the same answers were entered. This was done to help the participant to clearly remember all his thoughts and actions during the completion of the questionnaire. Participants were stimulated to report feelings and express opinions, preferences, and recommendations. If they encountered problems in operating the DTTSQ, they were asked what they thought the exact nature and possible cause of each type of problem was and how they tried to overcome the problem. Then, the participants were questioned about their satisfaction regarding the ease of use of the user interface, technical operation, layout and content, and overall usability of the DTTSQ. Researcher JS was allowed to ask complementary questions, if she felt it was necessary, to get complete and enriched data. Researcher IT finished the interview by collecting demographic data and data on self-reported experience with mobile technology (see
Data were analyzed using a thematic content analysis approach [
To get more familiar with the data and to create an overview, researcher MW made a descriptive summary of each case on the basis of all 4 types of generated data. Each summary contained information on whether or not the questionnaire was fully completed, if, when, and why the stop function was used, the kind of problems that occurred with the operation, the completion time, and all emerging themes regarding satisfaction or dissatisfaction with the ease of use of the questionnaire. The summaries were supplemented with information regarding educational level, age, gender, and experience in using mobile technology.
Subsequently, researcher MW derived the observed problems from the summaries. She clustered the problems. For every new problem, a new category was made. MW analyzed the video recordings to see how many times each problem was made in total, per participant and per question/screen of the questionnaire. After a full overview of problems had emerged, she scored the level of severity of each problem, as described by Nielsen and Loranger [
Frequency: Do a substantial number of users encounter the problem? Within this study, this question was answered with yes if one-third or more participants had encountered the problem.
Impact: Does the problem cause much trouble to those users who encounter it? Within this study, this question was answered with yes if the problem had led at least one participant to stop completing the questionnaire.
Persistence: Does the problem cause trouble repeatedly? Within this study, this question was answered with yes if the problem had occurred with an average of more than one time per participant.
This resulted in a 0- to 3-point score per problem. Each score was related to a level of severity: 0=low, 1=medium, 2=serious, and 3=critical.
These severity ratings give an indication of which problems lead to disastrous usability problems and which problems are more cosmetic in nature [
As a next step, researcher MW started open coding of all fragments in the transcripts of the semistructured interviews that were related to (dis)satisfaction about the ease of use of the questionnaire using MAXQDA 10 (VERBI Software). After she finished open coding, she organized and structured the codes until a coding scheme emerged on the basis of which the part of the research question that was related to satisfaction of the participants could be answered sufficiently.
As a last step, researcher MW ordered the analyzed data into 3 groups: data of participants who had (1) no, (2) little, and (3) average/high experience in using mobile technology. This was done to see whether or not data differed within and between these groups.
During the whole course of the study procedures, coding, analysis steps, and interpretation decisions were discussed with researchers HW, MJW, and WD.
No external funding was received by the Utrecht University of Applied Sciences to conduct this study. This study was submitted to the medical ethics committee of the Academic Medical Centre of Amsterdam which declared that it does not fall under the scope of the
Overall, 9 out of the 24 participants in this study did not complete the DTTSQ fully (see
Experience with mobile technology and completion of the Dutch Talking Touch Screen Questionnaire.
(Sub)Population | Not fully completed | Fully completed |
No experience using mobile technology (n=9) | 7 | 2 |
Little experience using mobile technology (n=11) | 2 | 9 |
Average/high experience using mobile technology (n=4) | — | 4 |
Total population (N=24) | 9 | 15 |
Inexperienced Michelle (56 years), Ida (66 years), Ilene (79 years), Dora (77 years), and Mia (73 years) and little-experienced Peter (18 years) and Rose (60 years) failed to fully complete the DTTSQ because they failed to select answering options and/or unintentionally skipped questions by double-tapping on the next button (see problems 1-5 in
When inexperienced Michelle (56 years) noticed most of her answers were missing from the summary in question 6, she got confused. In question 6, she was asked to choose the 3 most important activities in which she was limited. The screen contained only 1 activity photo whereas, in her mind, she had selected a lot of photo’s earlier. Except for the 1 photo that she had managed to select, she had tapped on the text beneath the photos, in which case, the item was not activated (see problem 5 in
Inexperienced Bill (72 years) had a lot of trouble operating the questionnaire. He commented on the introduction clip:
I do not think that what she is saying is difficult, but I just am not able to remember it. I have no experience with these kind of devices. So I forgot what she said right away.
Bill managed to get to question 4 by activating the help function on each screen he entered. When he touched the navigation button to see all the activity photos in question 4, the photo gallery moved in a different direction then he had presumed. This startled him somewhat and made him forget that he had to push the next button to go to the next screen (see problem 7 in
Inexperienced Helga (54 years) operated the digital questionnaire fluently until she had to choose the 3 activities that were most important to her in question 5. She did not use the navigation function of the photo gallery and as a result she did not see all her earlier selected activities (see problem 4 in
A complete overview of frequency and severity of all problems encountered can be found in
Frequency and severity of encountered problems during the completion processes of all participants.
Problem | Number of participants | Frequency | Severity rating |
1. Accidently skipping a screen by double tapping on the next button | 8 | 16 | Serious |
2. Double-tap on answering option causing activation and deactivation of the answer of choice | 1 | 1 | Low |
3. Skipping a screen by accidently touching the next button with the palm of the hand | 1 | 1 | Low |
4. Not using the navigation function of the photo gallery in question 4 causing the participant not seeing all presented response items | 2 | 2 | Medium |
5. Touching the text underneath a photo in question 4 to select an activity instead of touching the photo itself causing the activity not to be selected | 3 | 30 | Serious |
6. Not able to see whether or not a selected answer is activated (not accentuated enough) | 8 | 8 | Medium |
7. Not knowing how to get to the next screen | 1 | 1 | Medium |
8. Pushing too hard or tapping too soft on the touch screen causing the touch screen not to respond | 11 | 40 | Serious |
9. Not able to correct a wrong answer | 8 | 13 | Serious |
10. Not reading the text above the photos of question 5 causing the participant to keep on performing the task given with question 4 | 4 | 8 | Medium |
11. Not noticing that the multiple numerical rating scale-effort scores in question 8 are related to different activities, which by mistake results in identical scores for different activities | 1 | 1 | Low |
12. Scoring the body chart in question 2 mirrored | 2 | 2 | Low |
13. Scoring (serial) questions that do not apply to the participants’ situation (forced by the software) | 1 | 4 | Medium |
Average-/high-experienced Ellen (32 years), Sandra (39 years), and Joline (19 years) and little-experienced Jill (55 years), Lydia (56 years), and Christine (39 years) were able to complete the questionnaire without any problems. The other 18 participants were not able to operate the questionnaire fluently. In an absolute as well as relative sense, more participants with no experience in using mobile technology encountered problems during the completion of the DTTSQ than little-experienced participants did (see
Number of participants encountering each problem per level of experience with using mobile technology (N=24).
Problem | No experience (n=9) | Little experience (n=11) | Average/high experience (n=4) | Total population |
1 | 5 | 3 | —a | 8 |
2 | 1 | — | — | 1 |
3 | — | 1 | — | 1 |
4 | 1 | 1 | — | 2 |
5 | 2 | 1 | — | 3 |
6 | 4 | 4 | — | 8 |
7 | 1 | — | — | 1 |
8 | 6 | 5 | — | 11 |
9 | 3 | 4 | 1 | 8 |
10 | 2 | 2 | — | 4 |
11 | 1 | — | — | 1 |
12 | 1 | 1 | — | 2 |
13 | — | 1 | — | 1 |
aNot applicable.
The 21 participants who got to the end of the questionnaire had an average completion time of 10 min and 25 seconds. Inexperienced participants needed more time than little-experienced participants did, who in their turn needed more time than average-/high-experienced participants did (see
Completion time of all participants who did not end the questionnaire prematurely.
(Sub)Population | Mean completion time (min) | Median completion time (min) | Range of completion times (min) |
No experience with mobile technology (n=6) | 11.38 | 9.38 | 8.2 to 22.10 |
Little experience with mobile technology (n=11) | 10.41 | 9.57 | 6.54 to 18.10 |
Average/high experience with mobile technology (n=4) | 7. 55 | 7.42 | 5.50 to 10.26 |
Total population (n=21) | 10.25 | 9.43 | 5.50 to 22.10 |
All participants were satisfied with the ease of use of the questionnaire. The use of plain language, the way ICT was used, and the way the user interface was designed were greatly appreciated by the participants:
Everything was well described. I am not always able to understand everything, but this went well. I understood what was asked of me.
I have trouble operating my mobile phone and I own a notebook but don’t you ask me how that thing works! I am capable of a lot but I am not technical in that way. [...] This was the first time for me to use a tablet computer. I only had to follow the instructions. I did not have to start it up or open something, it just started working and it shut down by itself. I thought it was easy to work with. Better than when you have to write things down.
I am a very visual person. And this thing is very visual. [...] Like green is ‘no pain’ and red is ‘a lot of pain'.
All participants were satisfied with the completion time of the DTTSQ.
Operation problems, regardless of the amount and severity of the problems encountered by each individual participant, did not influence satisfaction about the ease of use of the questionnaire. Little-experienced Francine (61 years), for instance, was asked how she felt about the fact that the app did not always respond to her touch right away (see problem 8 in
When inexperienced Bill (72 years), who used the stop button, was asked if he would have preferred a paper-based questionnaire he said the following:
A total of 9 participants explicitly stated that operating the questionnaire was easier than they had expected beforehand. When inexperienced Ida (66 years) was confronted with the questionnaire she agitatedly said the following:
Noticeably reluctant and nervous, she started to complete the questionnaire. When she finished, she seemed surprised and relieved. She smiled and said the following:
And then she started laughing out loud and cheerfully asked if anyone would like to have some coffee.
Little-experienced Christine (39 years) was positively surprised too:
The most mentioned recommendations for improvement of the usability of the DTTSQ by participants were: shorten the length of the instructions, accentuate the activated response items, and improve the user interface of question 4 by giving participants a complete overview of activities to choose from in one screen, without having to use complicated navigation functions.
All participants within this study were very satisfied with the ease of use of the DTTSQ. Overall, 9 participants stated that the usability of the app exceeded their expectations. The participants who had no experience with using mobile technology completed the prototype of the DTTSQ less effectively and efficiently than the little- and average-/high-experienced participants did. In the group of average-/high-experienced participants, only 1 problem was encountered in total, whereas the inexperienced participants encountered an average of 3 and the little-experienced participants an average of 2 problems per person. Overall, 13 different kind of problems were encountered during this study. From a cost-benefit perspective, 4 of these problems will need to be addressed during future development of the DTTSQ because they have the potential to influence the future usage of the tool negatively [
In earlier studies, talking touch screens were found to be easy to use for people with different levels of education, literacy, or digital skills. These conclusions were based on study participants’ level of satisfaction with the ease of use of the tool [
The results of this study show how difficult it is to strive for an
According to Frokjaer et al, relations between the 3 aspects of usability depend in complex ways on the app domain, use context, and user’s experience [
It is a strength of this study that all 3 aspects of usability, instead of just satisfaction and efficiency, were thoroughly tested and that all of the results of the tests were differentiated for inexperienced and little- and average-/high-experienced users (which was not the case in the reports of the comparable studies [
It is a strength in itself that inexperienced as well as little and average-/high-experienced users of mobile technology were included in this study. Although recommended in the literature [
The qualitative TSTI method [
A limitation of this study was that participants were encouraged by the interviewer to try touching the screen again when they looked startled because it did not react to their initial touch. This may have influenced the results on effectiveness because it is unknown what would have happened if the interviewer would not have interfered. This may vary from no effect because the participant would have tried it again anyway, to a higher frequency of occurrence of problem 8, to more participants prematurely stopping to complete the DTTSQ because of being under the impression that the app had stopped working. Any kind of interference in the process of usability testing has a direct influence on the effectiveness results and possibly also on the efficiency and satisfaction results and should therefore be avoided.
The usability of the DTTSQ needs to be improved before it can be released. No problems were found with satisfaction or efficiency during the usability test. Effectiveness needs to be enhanced by (1) making it easier to navigate through screens without the possibility of accidently skipping one, (2) enabling the possibility to insert an answer by tapping on the text underneath a photograph instead of just touching the photograph itself, and (3) making it easier to correct wrong answers. Participants additionally recommended to minimize the length of the instructions and present all the answering options of question 4 in one screen.
During further development of the DTTSQ, both the results of this study and the study on response process of the DTTSQ [
Considering the difference in the number of problems encountered by inexperienced and little-experienced participants versus average-/high-experienced participants within this study, it can be concluded that in striving for an inclusive design, it is vital to involve potential users at risk of exclusion during further development and testing of the DTTSQ. Selecting quantitative methods for this purpose may be quite challenging because the researchers will have to develop a quantitative study design that will enable people with low literacy skills and low educational levels to participate. Research designs that include reading and writing tasks for participants are ineligible because these tasks may lead to exclusion of these vulnerable and hard-to-reach populations [
Researchers who want to investigate the usability of mHealth tools in populations that include little-experienced or inexperienced participants should take into account that the expectations of these participants may easily be exceeded resulting in high participant satisfaction outcomes regardless of the effectiveness and efficiency with which the tool is used. Satisfaction outcomes are influenced by the expectations that participants have before the test. It could be interesting to measure and further investigate computer anxiety and self-efficacy toward the use of the tested tool before and after usability testing to be able to put satisfaction outcomes into perspective.
Further research is necessary to gain more insight into the needs, preferences, capacities, values, and goals in relation to mHealth technology of people with low literacy skills, low educational levels, and no or little experience with using mobile technology. Insight is also needed into what effects meeting these user requirements will have on actual future use of these tools by these specific populations.
An overview of the screenshots of the Dutch Talking Touch Screen Questionnaire.
Dutch Talking Touch Screen Questionnaire
electronic health
information and communication technology
mobile health
Three-Step Test-Interview
The authors would like to thank Anita Cremers, PhD, for her contribution to reflecting on the data from a designer perspective and Kees van der Veer, PhD, and Jan Pool, PhD, for helping them to think about the application of the TSTI method in low-educated populations.
None declared.