This was a cross-sectional study. We developed 2 web-based surveys using Qualtrics. One survey was for individuals (ie, individual survey) who indicated a self-reported positive test for COVID-19, and another survey was for individuals whose relatives (ie, family survey), living in the same house, tested positive for COVID-19. We hosted both surveys on MTurk between August and December 2020. To improve the quality of data collection through MTurk and to make our study sample more representative of the target population, we followed the best practices suggested by Young et al [30].

A few restrictions were implemented to exclude certain survey responses from the final data analysis. First, only those participants who provided an existing COVID-19 test type (nasal/throat/blood/sputum) answer in response to our screening question could continue with the survey. Second, participants could fill the survey only once for themselves (individual survey) and for 1 family member (family survey). Third, a quality control question was included during the questionnaire, which stated, “Do not answer this question (Please click NEXT to go to the next question).” If the question was answered, the survey responses were excluded from the data analyses. Fourth, in the family survey, we asked the participants about their confidence level in their responses regarding their family member. Responses with low reported confidence were excluded from the final analyses.

This study was judged as imposing only minimal risks on participants and was determined to be exempt research by the Johns Hopkins University (IRB00248053) Institutional Review Board.

The web-based surveys had 5 blocks as follows (Multimedia Appendix 1 and Multimedia Appendix 2):

Survey measures were from the Johns Hopkins University COVID-19 community response survey guidance toolkit that draws from multiple sources [32]. An additional data source we used beyond the toolkit to compile COVID-19 symptoms was Twitter [18,32,33].

To validate our findings, we performed a comparison with existing systematic review or meta-analysis papers that assessed symptoms as risk factors for COVID-19 adverse outcomes. Articles for which the analyses occurred prior to our data collection were selected for comparison.

For each article and this study, individual symptoms were checked for being reported as (1) a significant risk factor for an adverse outcome (“yes”) and (2) a nonsignificant risk factor for an adverse outcome (“no”). We also noted if a symptom was not assessed (“NA”). When synthesizing findings across studies, if we found a statistically significant association between an adverse outcome and a symptom that was not studied by others, we labeled it “New.” If there was agreement between this study and at least one other study in identifying a symptom as a risk factor (significant or nonsignificant), we labeled it “1.” Symptoms we did not assess were labeled “NA.”

Pilot survey data were collected from 259 respondents who passed both the qualification test and the screening questions, and of these, 147 (56.8%) were considered to have “high quality” responses. For the experience groups 100-499, 500-999, and 1000+ HITs, the proportions of high-quality responses were 58% (48/83), 43% (41/95), and 72% (58/81), respectively (Table 1). There was no significant difference between the experience groups for obtaining high-quality responses (P=.14). There was, however, a significant difference between the groups for nonduplicate responses (P<.001). Comparisons of demographic characteristics across all experience groups among MTurk workers are shown in Multimedia Appendix 3.

Two modifications were made to our crowdsourcing pipeline following the pilot. First, we included only workers with 500+ prior HITs in our final filtering criteria. Given the differences in nonduplicate responses between groups, we reasoned that for tasks requiring a higher cognitive ability, workers with 500+ HITs may provide more high-quality responses than those with 100-499 HITs. Second, we added an attention check question to the Qualtrics survey (ie, “don’t answer this question”).

After implementing our final crowdsourcing pipeline, we collected data from 930 individual surveys and 1243 family surveys; however, data from 410 individual surveys and 496 family surveys were excluded (late August to December 31, 2020). The reasons for exclusion were completion of the survey previously, noncompletion of the survey, initial screening failure for age or comprehension check, and attention check failure (Figure 1). Thus, we finally collected data from 1267 eligible COVID-19–positive participants, and of these, 520 were from individual surveys and 747 were from family surveys. Thirteen participants were further excluded as they were either only slightly confident (n=12) or not confident at all (n=1) regarding their responses in the family survey. Thus, data from 1254 surveys were analyzed. The average time required to complete the general survey was 5.5 minutes.

Regarding family survey respondents, 68.3% (501/734) provided answers about a first-degree family member, 25.7% (189/734) provided answers about a second-degree family member, and only 6.0% (44/734) provided answers about a third-degree relative. There were no statistically significant differences in characteristics or outcomes between the individual respondents and the persons the respondents completed the family survey for, except for age (Multimedia Appendix 4). Therefore, the analysis presented here combined data from both surveys.

Over 90% (1159/1254, 92.4%) of the participants were up to 65 years old, and only 1.2% (15/1254) were less than 18 years old. Moreover, 52.0% (652/1254) were male, 81.2% (1018/1254) were white, 79.5% (997/1254) were not Hispanic or Latino, 68.4% (858/1254) had a bachelor’s degree or any postgraduate degree, 14.4% (180/1254) had yearly income of US $75,000 or more, 39.6% (496/1254) were smokers, and 46.8% (587/1254) had an influenza vaccine in the last season (Multimedia Appendix 5). Eight responders mentioned “passed away” in the family survey. As reflected in our sample, the MTurk worker population tended to be younger than the overall US population, with household incomes below that of the average US population [27].

We identified the following 6 symptomology groups using hierarchical cluster analysis (Figure 2): Group 1, abdominal and bladder pain; Group 2, flu-like symptoms (loss of smell/taste/appetite); Group 3, hoarseness and sputum production; Group 4, joint aches and stomach cramps; Group 5, skin or eye dryness and vomiting; and Group 6, no symptoms. We found sociodemographic and clinical differences between the symptomology groups (Table 2).

The flu-like symptoms group (Group 2) mostly represented the general study population. The abdominal and bladder pain group (Group 1) and the skin or eye dryness group (Group 5) had the highest hospitalization frequencies (153/227, 67.4% and 134/196, 68.4%, respectively). Both groups were characterized by a lower chance of high income (19/227, 8.4% and 21/196, 10.7%, respectively), more smoking (121/227, 53.3% and 102/196, 52.0%, respectively), and more influenza vaccinations (144/227, 63.4% and 102/196, 52.0%, respectively). The group with abdominal and bladder pain symptoms (Group 1) had higher proportions of Hispanic participants (82/227, 36.1%), asthma patients (64/227, 28.2%), alcohol disorder patients (64/227, 28.2%), and anemia patients (54/227, 23.8%). The group with skin or eye dryness (Group 5) had higher proportions of patients with depression (64/196, 32.7%), diabetes (28/196, 14.3%), weight loss (27/196, 13.8%), and ulcers (25/196, 12.8%). The group with joint aches and stomach cramps (Group 4) had lower proportions of hospitalization (65/158, 41.1%) and mechanical ventilation (31/158, 47.7%). Compared with the general study population, the asymptomatic group (Group 6) was younger (age 18-44 years; 70/85, 82.4%), had more males (54/85, 63.5%), had less white participants (60/85, 70.6%), had less Hispanic participants (7/85, 8.2%), had more participants with a high income (18/85, 21.2%), had less smokers (26/85, 30.6%), had less influenza vaccinations reported (30/85, 35.3%), had a higher proportion of participants with no chronic conditions (45/85, 52.9%), and had a very low risk for hospitalization (12/85, 14.1%).

A comparison of our findings with those of other studies can be found in Multimedia Appendix 9. At the time of our analysis, we found 3 systematic review or meta-analysis studies mapping the association of symptoms with the risk of adverse outcomes of COVID-19 [19-21].

We found agreement between this study and previous studies for 18 symptoms, 6 of which were associated with adverse outcomes (abdominal pain, cough, dyspnea/shortness of breath, fever, fatigue, and vomiting). In addition, we assessed 14 symptoms that were not previously studied by others, 6 of which were associated with adverse outcomes (bladder pain, dry eyes, dry skin, loss of appetite, seizure, and skin rash).

Our results identified individual symptoms and behaviors associated with COVID-19 adverse outcomes. Among these, some were well-known and some were new. We also identified 6 symptomology groups, with 3 groups showing statistically significant associations with COVID-19 outcomes. Furthermore, the findings of this work increase our understanding of the MTurk population and show that with precautionary measures, high-quality data can be obtained.

Well-known single COVID-19 symptoms identified (ie, abdominal pain, cough, fever, and shortness of breath) were associated with hospitalization [5,6]. Less common symptoms identified, such as bladder pain, eye dryness, and skin dryness were also associated with adverse COVID-19 outcomes. We provided additional validation of our findings by comparing the results with the findings of systematic review and meta-analysis studies. The individual symptoms we identified as being associated with adverse COVID-19 outcomes were consistent with the symptoms in those studies.

Our analysis of chronic conditions and associations with COVID-19 adverse outcomes showed that patients with preexisting asthma, diabetes, depression, and bladder problems were at high risk for hospitalization, similar to the findings in previous studies. Although previous studies have shown an increased risk of severe COVID-19 among people with obesity [34], our study did not find a significant increase in the risk of hospitalization among obese people. This result may be due to the participants in our sample being younger than those in other studies, resulting in a weaker link between obesity and chronic diseases that are the actual drivers of COVID-19 severity.

When studying behaviors influencing adverse COVID-19 outcomes, like previous studies, we found that smoking increased the risk of severe COVID‐19 outcomes [35-37]. Current smokers and past smokers who quit less than a year ago had a higher risk of hospitalization, and every day smokers also had a higher risk for mechanical ventilation. Our finding showing an effect of influenza vaccination on adverse outcomes contradicts the findings in some other studies. For example, it has been previously reported that influenza vaccination could be considered a protective factor again severe cases of COVID-19 infection [38,39]. Our data, however, suggested that COVID-19–positive respondents who were vaccinated against influenza in Autumn 2019 had higher odds of hospitalization and mechanical ventilation after adjusting for demographic factors, chronic conditions, and COVID-19 symptoms, as the influenza vaccination status might be associated with preexisting comorbidities and a person’s demographics. This is not an isolated finding as others have reported that there is a positive association between influenza vaccination rates and COVID-19 death rates [40], that influenza vaccination coverage in a country is a risk factor associated with higher infection rates of COVID-19 [41], and that there is a need to investigate the potential impact of influenza vaccination on COVID-19 risk and severity [42].

In addition to studying individual symptoms and behaviors, this study identified 6 COVID-19 symptomology groups by cluster analysis and assessed their associations with adverse outcomes of the disease. Three symptomology groups (flu-like symptoms, abdominal and bladder pain symptoms, and eye and skin dryness symptoms) were highly associated with a high risk for hospitalization. While the characteristics of respondents in the flu-like symptoms group were similar to the characteristics of the general population, the abdominal and bladder pain group included survey respondents who had lower income, and were more likely to have smoked and to be influenza vaccinated. They also tended to have chronic conditions, such as asthma and anemia, and alcohol disorder. The survey respondents in the eye and skin dryness group were generally older and had a greater possibility of being white. They were also more likely to have smoked and to be influenza vaccinated. This group also had a very high percentage of survey respondents with depression, diabetes, and ulcers.

Characterizing patients according to clusters using artificial intelligence devices and machine learning is a pioneering method in a variety of infectious and noninfectious diseases. The use of scientific methods to identify clusters of patients with similar characteristics and specific disease risks might improve awareness of heterogeneity in symptomology, and may enable targeted interventions to reduce disease severity. Other studies of COVID-19 disease trajectories have been able to identify vulnerable population clusters that could benefit from specific health resources, and have provided insights for public health targets for managing the pandemic [43,44]. One previous study identified 3 symptomatic groups and 1 asymptomatic group among COVID-19 patients [43]. However, that study did not analyze the associations between the symptomology groups and COVID-19 outcomes. Our analysis of 6 symptomatic groups found that the risk factors for COVID-19 adverse outcomes differed between participants from the different symptomology groups. For the asymptomatic COVID-19 group, other studies have shown that asymptomatic carriers account for 15% to 60% of the infected population and play a key role in disease transmission [45]. Adding to our understanding of asymptomatic carriers, our findings indicated that the asymptomatic symptomology group had a low percentage of hospitalization; a high percentage of young non-Hispanic men with high income; and a low percentage of people with chronic conditions, smoking, and influenza vaccination. These characteristics add to those described in a review study of asymptomatic COVID-19 carriers’ characteristics that found young age alone to be a significant factor for having no symptoms [46-48]. Another study of Mexican outpatients found a lower frequency of smokers and influenza vaccination among asymptomatic responders [43].

The percentage of those connected to a mechanical ventilator among hospitalized patients may seem high in our study (61.8%); however, the management of patients hospitalized with COVID-19 has changed considerably over the course of the pandemic. More than half of the study population had been hospitalized, and two-thirds of them were on ventilators. Since the survey was conducted in the first months of the COVID-19 pandemic, many people who got sick with COVID-19 were hospitalized and then connected to a mechanical ventilator. Over time, fewer people with COVID-19 were hospitalized, and among those who were hospitalized, only patients with more severe disease were put on ventilators. Other studies have also shown a high percentage (68%) of ventilator use among hospitalized COVID patients [49].

This work also showed that with precautionary measures to ensure high-quality data collection, a crowdsourcing model can be used to collect data to characterize symptomology for COVID-19 diagnosis and prognosis. There are many studies assessing health data on MTurk as a source of high-quality and rapidly collected data, and it has demonstrated good reliability [30,50,51]. However, to improve data quality on MTurk, there are recommendations to include workers with an “approval rate” above 95% and keep the “number of HITs approved” to at least 100 [30,31,51]. Prior studies have not investigated data quality from workers by comparing survey responses of 3 experience levels (according to the number of HITs approved) in a pilot study. By launching a pilot study, we found no difference in the approval rate of workers from different experience groups; thus, all could provide adequate data to satisfy the basic approval criterion. For specific tasks requiring higher cognitive ability, however, workers with more experience may provide higher quality data. In our case, we found that those with 500+ HITs submitted fewer duplicate responses than those with 100-499 HITs. While this may exacerbate the superworker issue, the tradeoff of quality data for the use of more experienced workers may be necessary depending on the task. To provide additional validation of our findings, we compared the findings of individual symptoms associated with COVID-19 to the findings of other researchers and identified many concordant findings.

A major limitation of this study was the self-reported data, which can be less reliable than physiological assessments. Our crowdsourced approach, however, allowed for reaching many participants, which helped mitigate the noise, and the fast data collection process was helpful during this pandemic. In addition, during this pandemic, many risk factors of COVID-19 were discovered through social media and other self-reported surveys [52-56]. To use those data sources, crowdsourced practices are emerging in research fields such as infodemiology (defined as collecting and analyzing data in real time through an electronic medium with the aim to inform public health decision makers) [57-59]. Another growing field is digital epidemiology, in which researchers are using internet data for epidemiological purposes [60-62]. The techniques of capturing relevant real-world data are promising but need to be further developed to meet the possible public health challenges in the future. Second, some of our findings warrant further validation. The risk factors first reported in our study, such as bladder pain symptoms and eye or skin dryness symptoms, need to be more extensively studied so that they can be used in clinical assessments. Furthermore, the influence of influenza vaccination on COVID-19 adverse outcomes should be further investigated as it appears now that humans will have to co-exist with both diseases for a long time even after this pandemic.

Our work demonstrated that a crowdsourced approach was effective for collecting data to assess the symptomology associated with COVID-19. Conducting a pilot study to assess data quality and population representation facilitated refining the filtering criteria for our final data collection strategy. We validated our approach by comparing the findings from assessing individual symptoms associated with COVID-19 to those identified by others and found highly concordant results. In our assessment of symptomology groups, we discovered that the bladder pain and skin or eye dryness groups had a high risk of COVID-19 hospitalization. Given these findings, we believe that a crowdsourcing strategy, such as the one proposed here, should be considered by others for quick and cost-effective assessments in a rapidly changing spectrum of infectious diseases, and societal and environmental factors.