Our study reports a secondary data analysis using the data set collected on a prior research project [21].

A convenience (nonrandom) sampling method was used to invite participants (parents and caregivers) to participate in the study within the network of the complex care clinic at a large pediatric hospital in the midwest. We sent email invitations and announced the research participation opportunity over social media and digital boards at the hospital. The recruitment occurred between October-December 2020. Inclusion criteria for the study were being a parent of a child with one or more complex medical conditions and having an iPhone with iOS 13 or above (or an iPhone 8 or above) during the study period. A total of 41 participants met these criteria and consented to participate. Of these, 24 participants completed the full study, which included a 2-week period of app use.

Data were collected from 24 parents of children with special health care needs (CSHCN) between October 2020 and January 2021. Participants were onboarded to the study via a web-based screening and survey tool. The eligible participants were guided through a web-based tutorial to install and use the research app. The app has functions to record, transcribe, and store notes [21]. Participants were instructed to use the app for a 2-week period for medical note-taking. During the study period, participants received periodic (every 2-3 days) reminders and tips about how to use the app features. They had the option to create medical note entries through voice or text while at home. Voice entries were transcribed by using Amazon Web Services Transcribe services [22], and the transcriptions were used as patient notes for analysis (Figure 1).

Building upon an earlier NLP pipeline proposal with simulated notes [19], we created a pipeline that leverages NER to identify and map terms to existing ontologies, particularly the RXNorm and SNOMED CT ontologies for medication and symptoms, respectively. After that, our algorithm searches over the sentence-level dependency parse trees alongside part-of-speech tags to extract further entity information based on the syntactic properties of the note [19]. Additionally, the relative date of the note (the date reported in the notes), if different from the authorship date, is derived with the assistance of the dateparser python library [23]. The NER and dependency parsing tools are retrieved from the open-source SciSpaCy Python package, which consists of PLM and ML models [24]. More specifically, our pipeline uses the SpaCy en_core_sci_lg model, which is a pipeline for biomedical data leveraging word embedding with more than 780,000 vocabulary terms and 600,000 word vectors [24], and a set of tools that use deep learning models to process biomedical and clinical text [25,26]. Researchers (SAH, ES) compared the model output against the original sentence and assessed it. They marked the information that the model incorrectly extracted (false positive), incorrectly left behind (false negative), and correctly extracted (true positive and true negative) for the categories of medication instance and symptom. We then calculated precision, recall, and combined F₁ score. This pipeline uniquely seeks to extract information about medication and symptoms using a zero-shot approach, which requires no training data and is adaptive to low-resource settings [27]. Likewise, our study does not have standardized tasks; therefore, no other baseline pipelines are available to compare against it. For this reason, evaluation metrics are presented solitarily to enable the evaluation and feasibility of the approach.

Figure 1 illustrates the data collection, processing, and output flow. In summary, a patient or caregiver creates the notes through voice interaction or text entries. Amazon Web Services transcribes voice entries and stores. The entries, unstructured PGHD, are processed using the proposed NLP model in this study to identify medication amount, dose, medication name (dependency parsing is shown with arrows), time and date, and symptoms. The extracted information can be integrated into a chart that can be used to inform a patient, caregiver, and provider through local logs on a device, cloud services, or medical records assisted by data visualization tools [19]. The integration component is not in the scope of this study.

The study involves human participants and was reviewed and approved by the institutional review board at Nationwide Children’s Hospital (#00000231). The participants signed written informed consent to participate in this study, allowing the use of the data set for the data analyses described here. All data reported in this study are deidentified. Participants were compensated for their time with a gift card (US $30).

The participants were aged between 26 and 59 years (mean 39, median 38), mostly White (n=22, 92%), had more than one child (n=16, 67%), lived in Ohio (n=22, 92%), had mid or upper-mid household income (n=15, 62.5%), and received higher-level education (n=24, 58%). Participants had CSHCN with multiple chronic conditions. Frequently reported conditions included developmental delay; speech, vision, and physical problems; and genetic and neurological disorders. The majority of parents were “always” or “often” tracking their child’s symptoms, medications, or conditions (n=17, 71%) using an app (n=13, 54%) or patient portal (n=16, 67%).

In total, 87 patient notes were included (voice entry transcriptions n=78 and text entries n=9). Thirty of the notes are drug and medication-related, and 46 of the notes are symptom-related, but there are overlapping notes having both or none of the symptom and medication information. More specifically, 24 notes have no symptom or medication information; 33 notes have only symptom information but no medication information; 17 notes have only medication information and no symptom information; 13 notes have both medication and symptom information. Each note is structured as 1 to 3 sentences, briefly recording the state of the patient, medication given, and symptoms (see Textbox 1, [21]). Content-wise, a patient note entry may have multiple components (symptom and medication detail), such as a summary of the day instead of multiple notes created throughout a day. Please see Multimedia Appendix 1 for the medication and symptoms captured through the pipeline and their frequencies.

Table 1 presents precision, recall, and F₁ scores. In the table, notes refer to the number of individual notes considered for each data component (eg, medication or symptom-related term). Instances refer to the number of times a data component is considered with allowance for multiple instances per note. A medication instance refers to all subcomponents (medication name, unit of measurement, quantity, and date information) of the medication when present. An instance is correct if all subcomponents are correctly extracted when present and is wrong if any one of the presented subcomponents is wrong. Precision-recall scores of medication information subcomponents provide a granular breakdown per extracted instance. Some of the extracted symptoms could be ambiguous to be classified as a symptom, such as emotional states (eg, “happiness”). We categorized them as symptoms in our research, as mood can affect health conditions.

At an instance level, which is considered correct if all relevant subcomponents are correctly extracted, the medication-instance extraction pipeline has moderately good performance, with a precision and recall both near 0.8. At the subcomponent level, medication name recognition has the highest performance with high precision, while unit and quantity extraction have low recalls. In medication-related false-positive cases, patient names, or common terms (eg, water) overlap with medication names or ingredients (eg, water irrigation solution) within the RXNorm ontology. Unit and quantity extraction errors are often caused by irregular sentence structures or information split between multiple sentences. Date extraction performs well in both precision and recall (except in cases with phrasal referents, such as “A few days ago”). Both brand names and generic names are identified with errors caused by overlap between patient names and RXNORM. For symptoms, we find recall scores higher than precision, implying that often entities were extracted when they were not proper symptoms (eg, “sitting” and “sign”).

We present findings of our proposed NLP pipeline with real-world PGHD to demonstrate the feasibility of its implementation. The results demonstrate that the NLP pipeline performance matches contemporary works with a zero-shot approach for information extraction [28], specifically for most of our targeted information categories of medications and symptoms (with an F₁ score above 0.6). These results cannot be compared directly since the other studies in the literature do not focus on PGHD or the same combination of medication, dosage, and symptom extraction. However, the results indicate acceptable performance when using NER and dependency parsing through open-source and hybrid NLP models. The performance of the pipeline may increase over time with improvements in automatic speech recognition and text prediction and suggestion methods (methods that also use NLP models that are not covered within the scope of this study) [29-31]. However, in this study, the pipeline performance was potentially affected by the transcription errors or typing errors existing in the data set (n=16, 18% of 87 notes had at least one error; errors have not been corrected to contain real-world data features).

This study extends the existing literature [15], presenting the capability of current models to extract key information from patient notes. This approach can inform patients and caregivers out of the clinic toward enabling self-care (eg, improving medical adherence, symptom tracking) and remote monitoring (eg, detection, intervention, and communication) [11]. In clinical practice, the use of such artificial intelligence and machine learning approaches potentially facilitates the inclusion of personal health records into medical records, which can allow the identification of health condition changes and build early detection mechanisms [3,32]. Integrating unstructured PGHD via an NLP pipeline within electronic medical records can also contribute to improving patient-reported outcomes and shared decision-making at the clinic, allowing health care providers to remotely observe health conditions and intervene in a timely manner [33,34].

Considering the patient population with special health care needs and their caregivers that receive care from multiple providers and clinics, there is a continuous need for documentation, medication, and symptom tracking during home care. Timely communication of patient conditions with multiple health care providers is needed but creates additional burden and stress for patients and caregivers given their daily life and workload [35,36]. The literature shows that currently available digital health technologies including mobile apps, SMS text messages, and web portals have been used for patient care management [37-41] and communication of patient medical conditions remotely [41-44]. As digital health technology eases the process of documentation and the tracking of symptoms and medications, NLP approaches can improve the process by enabling the use of natural and preferred language. Furthermore, an NLP pipeline integrated with preferred technologies (text-based and voice-interactive apps, patient portals) can reduce the burden and complexity of accessing personal notes, summarizing and searching patient notes, and reducing the need to learn a technology to complete tasks or take notes, and reduce required attention on a device or modality and time spent on documentation [12,21,45]. In addition, the use of the zero-shot approach demonstrates the ability to use artificial intelligence and machine learning in data-scarce environments (eg, data on rare diseases, data from rural hospitals), which increases the equitable and accessible use of artificial intelligence and machine learning in health care.

Our study presents the feasibility of PLM use within zero-shot biomedical settings. Whereas other works require specialized pretraining of NLP models [28,46] or are limited to handling the formalized writing style of the biomedical literature, our approach makes use of more task-general biomedical PLMs to better generalize over the various syntactic forms found in PGHD. Namely, we gather our PLMs from the SciSpacy model suite, which shows high performance in its various capabilities when evaluated on PubMed and clinical notes [25]. We use SciSpacy’s entity extraction capabilities before linking to various ontologies. It is followed by the extraction of additional information related to these entities, such as medication dosage information by leveraging sentence-level dependency parse trees, providing insight into the capabilities and using automatic dependency parsing. This is a novel approach to our study, which has not previously been implemented in biomedical research [47-49]. Furthermore, the performance of our pipeline can be improved through the use of cohort- or patient-specific vocabularies to augment the NER subcomponent, allowing for a human-in-the-loop component to our hybrid model where domain experts can define model parameters and integrate human knowledge [50]. Human-in-the-loop methods span a variety of directions with the general consensus being that such methods allow for compensation of model weaknesses with domain expertise and vice versa, alongside a high ratio of model performance against model creation cost, causing it to be an increasingly important component of applied machine learning [18,50]. Using publicly available models and ontologies improve the dissemination of the model as well as access and customization specific to conditions and patient populations. Since our NLP models are pretrained on clinical data and use publicly available ontologies, replication and scalability of a pipeline have low costs in terms of the requirement for training data, computational power requirements, and expertise.

Our study has several limitations. We did not use fine-tuned models or custom vocabularies, which might improve the performance (eg, missing condition-specific treatments and therapies). In addition, we do not implement and evaluate negation. We are unable to compare the feasibility of our pipeline against other NLP models. This is, in part, due to the lack of comparable and available pipelines and tasks to those presented in this study. However, future work is suggested to compare our pipeline against other PLMs as well as performance comparison against a model fine-tuned on general PGHD and clinical notes. These additional approaches may inform how our pipeline can be improved in the future to improve task completion or generalizability.

Our evaluation process was toward feasibility, rather than only performance assessment. In that regard, we used a nonstandard evaluation process as we annotated the examples coming after we ran the pipeline over these examples. As it was necessary to capture the false positives that the pipeline extracted, certain extracted values were considered partially valid only after the model was seen to have captured them, such as moods being considered symptoms. In these cases, the capture of such entities was not considered a false positive, while omittance was not necessarily considered a false negative. Additionally, for medication and dosage entities, it was unlikely for bias to be introduced due to the definite nature of these values (eg, if “Tylenol” is referenced, it is clearly a medication that should be captured).

Due to the limited text-based entries against voice entries, we were not able to measure the discrepancy of the model performance of extracting entities from written notes versus voice transcriptions.

The first key contribution is the use of pretrained language and deep learning models and the extraction of information using syntax parsing and entity-to-ontology linking in the proposed methodology. To our knowledge, these individual components have yet to be combined into a clinical information extraction pipeline. Second, we focus on PGHD for chronic conditions which has been understudied, and, to our knowledge, no examples exist in the literature regarding the use of a text-processing pipeline with the real-world PGHD from this specific population (CSHCN). While previous studies explore similar NLP pipelines with NER, only a few studies use the automatic dependency parser for further relation and entity extraction [47], layer multiple rounds of dependency parsing to extract general relationships from within scientific literature [49], and use dependency parsing to identify SNOMED CT expressions form clinical notes and [48]. These studies show the promise of using syntactic dependency trees to improve generalizability but do not fit the unstructured format of PGHD and are not focused on extracting values related to chronic care, such as medications and dosages.

To accomplish the task of extracting key care-related information from PGHD (medications and symptoms), we created a pipeline that applies dependency parsing to key terms and relationships for chronic care management. Instead of expanding generalizability through training of a deep learning model, which can often be costly in time and labor, the pipeline can be improved and expanded with a system to allow for user input to inform NER and dependency-based relation extraction systems. This inclusivity of users (eg, clinician, patient, or caregiver) allows for flexibility of modification of our system depending on patient condition or clinical needs [19]. Furthermore, by using publicly available ontologies and models, the proposed pipeline can be replicated, customized, and improved for different cohorts with chronic conditions.

Future work will focus on building and fine-tuning condition-specific models, ontologies, and vocabularies, prototyping PGHD integration to clinical workflow, and improving clinical decision support mechanisms through PGHD-informed visuals.

In addition, we plan to analyze voice and audio characteristics and extract-related features (such as pause rates, pitch, loudness, acoustic and spectral features, and multiple speakers, such as parent and child) [51,52]. Voice analytics will add a new dimension to PGHD analytics by investigating vocal and environmental audio features (markers) with patient notes and building a multimodal pipeline, such as improving transcription quality, improving sentiment analysis, identifying the environmental factors [53,54], and guiding future data collection protocols. Textbox 2 provides a glimpse at the data for the proposed future work with voice analytics.

We present the feasibility of an NLP pipeline with real-world data in a low-resource setting, focusing on medication and symptom information extraction. Unstructured PGHD can inform decision-making and support remote monitoring and self-care. With customizable information extraction methods using NER and medical ontologies, NLP models can feasibly extract a broad range of clinical information from unstructured PGHD in low-resource settings. We suggest future work to build medical condition-specific models, test the integration to clinical workflow, and use audio features in the analysis.