This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.
Hay fever is a highly prevalent, heterogenous, and multifactorial disease. Patients may benefit from longitudinal assessments using mobile health (mHealth) principles. We have previously attempted to establish an effective mHealth platform for patients with hay fever through AllerSearch, our in-house smartphone app that assesses electronic patient-reported outcomes through a questionnaire on hay fever and provides evidence-based advice. To be used by the public, an investigation on its reliability and validity is necessary.
The aim of this paper is to assess the reliability and validity of subjective symptom data on hay fever collected through our app, AllerSearch.
This study used a prospective observational design. The participants were patients aged ≥20 years recruited from a single university hospital between June 2, 2021, and January 26, 2022. We excluded patients who could not use smartphones as well as those with incomplete data records and outlier data. All participants answered the Japanese Allergic Conjunctival Disease Standard Quality of Life Questionnaire (JACQLQ), first in the paper-and-pencil format and subsequently on AllerSearch on the same day. The JACQLQ comprises the following three domains: Domain I, with 9 items on ocular or nasal symptoms; Domain II, with 17 items on daily activity and psychological well-being; and Domain III, with 3 items on overall condition by face score. The concordance rate of each domain between the 2 platforms was calculated. The internal consistency of Domains I and II of the 2 platforms was assessed using Cronbach alpha coefficients, the concurrent validity of Domains I and II was assessed by calculating Pearson correlation coefficients, and the mean differences between the 2 platforms were assessed using Bland-Altman analysis.
In total, 22 participants were recruited; the data of 20 (91%) participants were analyzed. The average age was 65.4 (SD 12.8) years, and 80% (16/20) of the participants were women. The concordance rate of Domains I, II, and III between the paper-based and app-based JACQLQ was 0.78, 0.85, and 0.90, respectively. The internal consistency of Domains I and II between the 2 platforms was satisfactory (Cronbach alpha of .964 and .919, respectively). Pearson correlation analysis yielded a significant positive correlation between Domains I and II across the 2 platforms (
Our findings indicate that AllerSearch is a valid and reliable tool for the collection of electronic patient-reported outcomes to assess hay fever, contributing to the advantages of the mHealth platform.
Hay fever is currently believed to be the most common immunologic and allergic disease worldwide, with reports of nearly 30 million cases in the United States and Japan [
Recent findings have increasingly confirmed the advantages of adopting patient-reported outcomes (PROs), which are clinical data grounded in patients’ own subjective experiences that are not readily captured by routine medical evaluations [
We have previously taken advantage of the novel mHealth platform and conducted studies through our in-house hay fever smartphone app, AllerSearch, released in February 2018 [
Hence, we evaluated the reliability and validity of the subjective symptom data collected through our mHealth app by conducting a comparative study between paper-based and app-based versions of the JACQLQ to evaluate the applicability of AllerSearch as a novel clinical tool for assessing hay fever.
AllerSearch was initially developed in Japan using Apple Inc’s open-source framework, ResearchKit [
This study employed a prospective observational design based on previously published validation studies of medical instruments [
Written informed consent was obtained from all participants prior to the commencement of the study. The study was approved by the Independent Ethics Committee of Juntendo University Graduate School of Medicine (approval number H20-0242-H01, November 6, 2020) and adheres to the tenets of the Declaration of Helsinki.
The participants were patients aged ≥20 years, recruited between June 2, 2021, and January 26, 2022, from the Department of Ophthalmology, Juntendo University Hospital, Tokyo, Japan. We excluded patients who could not use smartphones as well as those with incomplete data records and outlier data.
All participants answered the paper-based JACQLQ at the outpatient service in the Department of Ophthalmology, Juntendo University Hospital. They subsequently answered the same questionnaire on an iOS version of AllerSearch (app-based JACQLQ) on the same day. AllerSearch had been preinstalled on the mobile phones provided for the purpose of this study. Our previous study contained the description of survey items in AllerSearch [
The JACQLQ is a well-established metric that enables clinicians to comprehensively assess QOL among patients in the Japanese-speaking population who are affected by allergic conjunctival diseases [
The sample size for the Cronbach alpha test was predetermined based on the formula by Bonett [
The median scores for the paper-based and app-based JACQLQ were compared using Wilcoxon matched-pairs signed-rank tests [
Statistical analyses were performed using Stata/MP version 16.1 (Stata Corp) and GraphPad Prism version 9.1.2 (GraphPad Software). Statistical significance was set at
Input on the AllerSearch survey questionnaire was obtained to produce a version that was agreed upon by a committee comprising allergy specialists, ophthalmologists, otolaryngologists, epidemiologists, and the patient and public involvement members [
In total, 22 participants were recruited for this study. Following the exclusion of an individual with incomplete data records and another with outlier data, the data of 20 (91%) participants were analyzed.
Participants’ characteristics (N=20).
Characteristics | Values | ||
Age (year), mean (SD) | 65.4 (12.8) | ||
|
|||
|
Female | 16 (80) | |
|
Male | 4 (20) | |
BCVAa, logMAR (SD) | –0.04 (0.07) | ||
IOPb, mmHg (SD) | 13.9 (2.6) |
aBCVA: best-corrected visual acuity.
bIOP: intraocular pressure.
The median total score for Domains I and II was 6.5 (range: 1.75-13.25) for the paper-based JACQLQ and 4.5 (range: 1–8) for the app-based JACQLQ (
JACQLQa item scores and concordance rate between paper-based and app-based JACQLQ.
JACQLQ items | Paper-based JACQLQ, median (IQR) | App-based JACQLQ, median (IQR) | Concordance rate (%) | |||||||
|
3.5 (1.75-6) | 2 (1-4.25) | .004 | 78 | ||||||
|
|
2 (1-4.25) | 1.5 (0-3) | .01 | 82 | |||||
|
|
1. Itchy eyes | 1 (0-1) | 1 (0-1) | >.99 | 95 | ||||
|
|
2. Foreign body sensation | 1 (0-1) | 0 (0-1) | .12 | 65 | ||||
|
|
3. Red eyes | 0 (0-1) | 0 (0-0.25) | .13 | 80 | ||||
|
|
4. Watery eyes | 0 (0-0) | 0 (0-0) | .50 | 90 | ||||
|
|
5. Eye discharge | 0 (0-1) | 0 (0-0.25) | .13 | 80 | ||||
|
|
1 (0-2.25) | 0.5 (0-1) | .01 | 74 | |||||
|
|
6. Runny nose | 0 (0-1) | 0 (0-0.25) | .06 | 75 | ||||
|
|
7. Sneezing | 0.5 (0-1) | 0 (0-0.25) | .03 | 70 | ||||
|
|
8. Stuffy nose | 0 (0-1) | 0 (0-0) | .38 | 75 | ||||
|
|
9. Itchy nose | 0 (0-0.25) | 0 (0-0) | .06 | 75 | ||||
|
2 (0-5.5) | 1.5 (0-5.5) | .04 | 85 | ||||||
|
|
2 (0-4.25) | 0.5 (0-2) | .002 | 85 | |||||
|
|
1. Obstacles to studying, working, and housework | 0 (0-1) | 0 (0-0) | .13 | 80 | ||||
|
|
2. Poor mental concentration | 0 (0-1) | 0 (0-0) | .50 | 90 | ||||
|
|
3. Decreased thinking ability | 0 (0-0) | 0 (0-0) | >.99 | 85 | ||||
|
|
4. Impaired reading newspapers and other materials | 0 (0-1) | 0 (0-1) | .13 | 80 | ||||
|
|
5. Poor memory | 0 (0-1) | 0 (0-0) | .50 | 90 | ||||
|
|
6. Limitation of outdoor life such as sports and picnics | 0 (0-0) | 0 (0-0) | >.99 | 85 | ||||
|
|
7. Limitation of going out | 0 (0-0) | 0 (0-0) | .38 | 80 | ||||
|
|
8. Obstacles to socializing with people | 0 (0-0) | 0 (0-0) | .50 | 90 | ||||
|
|
9. Interfering with conversations and telephone calls with others | 0 (0-0) | 0 (0-0) | >.99 | 90 | ||||
|
|
10. Anxiety about people around you | 0 (0-0) | 0 (0-0) | >.99 | 95 | ||||
|
|
11. Sleeping disorder | 0.5 (0-1) | 0 (0-1) | .03 | 70 | ||||
|
|
0 (0-1.25) | 0 (0-2) | >.99 | 86 | |||||
|
|
12. Dullness | 0 (0-0.25) | 0 (0-0.25) | >.99 | 80 | ||||
|
|
13. Fatigue | 0 (0-0) | 0 (0-1) | .75 | 85 | ||||
|
|
14. Frustrated | 0 (0-0) | 0 (0-0) | .25 | 85 | ||||
|
|
15. Irritable | 0 (0-0) | 0 (0-0) | >.99 | 95 | ||||
|
|
16. Depressed | 0 (0-0.25) | 0 (0-0) | .63 | 80 | ||||
|
|
17. Dissatisfaction with life | 0 (0-0) | 0 (0-0) | >.99 | 90 | ||||
Domain III, 0-4 | 1 (1-2) | 1 (1-2) | >.99 | 90 |
aJACQLQ: Japanese Allergic Conjunctival Disease Standard Quality of Life Questionnaire.
The internal consistency of the total, subscale, and domain scores between the paper-based and app-based JACQLQ is indicated in
Reliability between the paper-based and app-based JACQLQa.
JACQLQ | No. of items | Cronbach alpha | ||
|
Paper-based | App-based | ||
Domains I and II | 26 | .964 | .919 | |
|
9 | .897 | .746 | |
|
Eye symptoms | 5 | .856 | .788 |
|
Nasal symptoms | 4 | .776 | .331 |
|
17 | .953 | .896 | |
|
Daily activity | 11 | .914 | .915 |
|
Psychological well-being | 6 | .937 | .731 |
aJACQLQ: Japanese Allergic Conjunctival Disease Standard Quality of Life Questionnaire.
Correlation between the app-based and paper-based Japanese Allergic Conjunctival Disease Standard Quality of Life Questionnaires (JACQLQs). The correlation between the paper-based and app-based JACQLQs of Domains I and II, Domain I, and Domain II.
Bland-Altman plot for the paper-based and app-based Japanese Allergic Conjunctival Disease Standard Quality of Life Questionnaires (JACQLQs). The x-axis indicates the average of the 2 methods’ scores, and the y-axis indicates the difference between the 2 methods’ scores. The central line indicates the mean difference (bias) between the scores from the 2 methods, whereas the superior and inferior lines depict the intervals, which include the 95% limits of agreement. Differences between the JACQLQ of Domains I and II, Domain I, and Domain II.
Hay fever, a highly heterogenous and multifactorial disease, requires personalized assessments to develop effective preventive measures and management strategies. In this study, we examined the reliability and validity of our smartphone app, AllerSearch, regarding collecting data on hay fever symptoms. Our results indicate that the digital administration of the JACQLQ through AllerSearch shows satisfactory reliability and validity metrics; AllerSearch may therefore be an accessible tool for hay fever management. Its accessibility may prove advantageous for screening the undiagnosed population and promoting early, personalized interventions. The COVID-19 pandemic accelerated the breakthrough and subsequent growth of telemedicine and effective self-management. AllerSearch’s ability to assist in the self-management of hay fever, with its extensive reach, aligns well with the aforementioned changing medical paradigm.
Our results show satisfactory internal consistency for most questionnaire items (Cronbach alpha>.70), except for nasal symptoms in the app-based version (Cronbach alpha=.331). Further, there were significant positive correlations between the 2 measurements (Domains I and II:
Our results also indicate that the app-based collection of nasal symptoms showed low internal consistency, which may lead to a discrepancy between nasal and nonnasal symptom assessments. However, nonnasal symptoms, as well as overall symptoms, maintained a high internal consistency, and the low internal consistency observed for nasal symptoms may be attributed to the small sample size in this study. Future efforts to increase power should be pursued to verify or improve on the observed low internal consistency for nasal symptoms.
Traditionally, in-person assessments have not proved very effective in comprehensive evaluations, mostly owing to the low frequency and time constraints of typical outpatient visits [
Another explanation for the 3-point mean difference between the 2 platforms could be the length and order of the questionnaire items in the app-based version. Demographic and medical history questions preceded the JACQLQ, which might have led to response fatigue [
This study has several limitations. First, there may be a degree of selection bias stemming from the participants’ demographics, including age and gender. This was also a single-center study, making the selection process prone to selection bias. Further, while there has not been a study, to the best of our knowledge, comparing paper-based and app-based questionnaires, this study had a smaller sample size in comparison to previous studies that investigated discrepancies between digital and paper questionnaires [
Our findings indicate that the data collected through the AllerSearch app had good internal consistency, with a Cronbach alpha of >.70 and significant positive correlations between the paper-based and app-based JACQLQ (Domains I and II:
electronic patient-related outcome
Japanese Allergic Conjunctival Disease Standard Quality of Life Questionnaire
mobile health
patient-reported outcome
We thank Drs Fujisawa K, Muto K, Nojiri S, Nagao M, and Okano M, and the members of our patient panel for advising us on how to improve our research project, and Medical Logue, Inc (Tokyo, Japan) for developing and releasing the Android and updated versions of AllerSearch.
All data generated or analyzed during this study are included in this published article.
Conceptualization was implemented by, T Inomata; methodology by YA and T Inomata; software by YA; validation by YA and T Inomata; formal analysis by YA; investigation by YA, T Inomata, YO, KF, MM, and KH; resources by T Inomata; data curation by YA; writing—original draft preparation by YA, T Inomata, and JS; writing—review and editing by YO, KF, MM, KH, MI, MM, NE, MN, T Ide, KN, and AM; visualization by YA; supervision by AM; project administration by T Inomata; and funding acquisition by T Inomata. All authors have read and agreed to the published version of the manuscript.
This research was funded by the Japan Agency for Medical Research and Development, grant number JP21ek0410063, the Institute for Environmental and Gender-Specific Medicine, Juntendo University, and the OTC Self-Medication Promotion Foundation. The funding sources had no role in the study design; collection, analysis, and interpretation of data; writing of the report; and in the decision to submit the article for publication.
AllerSearch was created using Apple’s ResearchKit. T Inomata and YO are the owners of InnoJin, Inc, which developed AllerSearch. YA, JS, KF, MM, KH, MI, NM, NE, MN, T Ide, KN, and AM declare no conflicts of interest.