The validity of the findings of any study hinges on the integrity of the data collected. Common concerns among psychology researchers using self-report measures are data quality and participant compliance. Participants may not fully read or process the self-report measure instructions or items, adding noise to data rather than reflecting the constructs being assessed, or they may not complete the study protocol, reducing the number of assessments used for analyses. Both reduce the study’s power [1] and may hinder the generalizability of its findings [2,3]. Given that clinical trials of psychological treatments are chronically underpowered [4], reduced power due to poor data quality can exacerbate the lack of trust in the study findings. To prevent these negative effects on statistical power and external validity, researchers may aim to recruit compliant participants, better incentivize compliance, and detect and remove noncompliant participants from data sets. Given that college students comprise most samples in psychological studies [5-7], it is necessary to understand the associations between recruitment sources targeting college students and participant compliance. For this paper, we defined participant compliance as providing high quality data (ie, putting in reasonable effort and fully reading each item before responding) and completing follow-up assessments (ie, retention; only applicable for longitudinal studies).

One approach to identifying participant noncompliance that affects data quality is to use attention check items with instructions to select a particular answer (eg, “Select ‘slightly agree’ for this item”) or that have factual answers (eg, “Which number is largest?”); these are also called instructional manipulation checks [1], bogus items [8], infrequency scales [9], or random-responding indicators [10]. These items can identify participants who are satisficing (ie, putting in minimal effort and potentially not fully reading or comprehending each item), which is sometimes called careless responding. Removing these participants may increase statistical power such that correlations are stronger among relevant study variables and experimental effects across conditions are larger [1,10-12] or may otherwise reduce “noise” among study variable associations [13]. Identifying poor quality data where responses may not reflect the true study construct via attention checks and removing these satisficing cases may reduce random error and increase statistical power, but it also may result in removing a meaningful group and introducing bias, as certain demographics (eg, gender, age, race, education, and intrinsic motivation) can be associated with satisficing [11,14]. It is possible that some recruitment sources may yield participants who are not only less inclined to satisfice but are also more demographically diverse, allowing for the removal of satisficing participants without limiting the diversity and generalizability of the sample.

In a model explaining how individuals formulate and respond to survey questions, Tourangeau et al [15] proposed that poor data quality comes from failing to engage in at least one of the four stages of the cognitive processing: (1) understanding the meaning of the item, (2) finding information in memory relevant to the item, (3) summarizing the information found, and (4) using that information to choose a response given the options. Satisficing could result from failing to engage in any one of these stages of cognitive processing. However, the attention checks used in this examination as an indicator of data quality are designed to detect the most egregious forms of satisficing (stage 1: not fully reading and understanding the item) rather than the later forms of cognitive processing (eg, taking the time to process exactly how anxious they remember being or whether this is best reflected as an endorsement of 4 vs 5 for a given item). The study by Krosnick [16] gave an overview of a variety of response strategies that respondents may choose when engaging in satisficing to conserve their mental energy and suggested that some individuals may choose to engage in satisficing to conserve their mental energy when faced with great task difficulty (eg, many items in a survey) and that the likelihood of satisficing increases as the burden of the survey increases, and participants become less motivated to perform well as they become more fatigued. Moreover, it suggested that respondents may first be less diligent about the later stages of cognitive processing (eg, taking the time to decide between 4 and 5) before fully omitting stages (eg, not fully reading the item). A systematic review of 141 studies that included various indicators of satisficing revealed that 74% of the studies found that task difficulty was significantly associated with satisficing, and 68% of the studies found that respondent motivation was significantly associated with satisficing [17], suggesting that satisficing is a result of the qualities of both the survey task (highly burdensome) and the participant (low motivation).

Another issue of participant compliance particularly important for longitudinal research such as examinations of psychological health is study retention rates. Many studies require multiple assessments, such as observing natural developmental trajectories or following changes in behaviors, symptoms, or attitudes after the administration of an intervention. Although interventions offer additional benefits to participants compared with nonintervention research (eg, potential improvements in mental or physical health), these benefits are often obtained immediately and do not extend to incentivizing retention for follow-up surveys. Moreover, follow-up assessments are typically administered remotely and after substantial time has passed (eg, weeks or months). Therefore, retention rates typically drop with each additional follow-up. For example, challenges with retention have been noted in a meta-analysis of cohort studies on mental and behavioral health [18], a meta-analysis of dissonance-based interventions for health behavior change [19], and a meta-analysis of digital interventions for the treatment and prevention of eating disorders [20]. This is particularly challenging for studies involving college students. An integrative analysis of 24 studies of brief interventions for college drinking found retention rates as low as 46% for the 6-month follow-ups and 51% for the 9- to 12-month follow-ups [21]. The authors of these studies have noted how challenging it can be to retain college students, particularly when administering interventions such as those for college drinking [22,23] or web-based programs for students with depression, anxiety, or stress [24]. Identifying recruitment methods that are associated with better data quality and higher compliance (eg, passing attention checks and completing study protocols including follow-up assessments) may reduce the costs associated with longitudinal research, increase the benefits and practicality of the study designs, and strengthen the trust in the study findings.

Undergraduate college students serve as study samples in most psychology studies, a consistent trend over time. An investigation of 6 top journals in multiple fields of psychology over 20 years (1975, 1985, and 1995) included 1559 articles with human participants [7]. Researchers found that most studies (68%) exclusively used undergraduate college student samples and that this finding was consistent over time (69.8% in 1975, 66.7% in 1985, and 68.2% in 1995). An investigation into 1 specific premier journal, the Journal of Personality and Social Psychology, revealed that 67% of American study samples were specifically undergraduate students enrolled in psychology courses, and this rose to 80% of samples for non-American studies [5]; however, this number has dropped to 42% overall (39% of American-based studies and 54% of non-American studies) in more recent years [6]. This makes it imperative to examine study compliance among college students and, in particular, whether the use of psychology student pools has an impact.

Student participant pools have received both praise and criticism from the psychology community [25,26]. They provide researchers with a low-cost and efficient recruitment source, which may be particularly important for student researchers who do not have funding [26]. Although there are concerns that student participant pools mainly comprise female, White, and young psychology majors [26,27] and that this can result in samples that do not generalize beyond Western, educated, industrialized, rich, and democratic societies [28], student participation pools are becoming more demographically diverse, mirroring the increasing diversity among those attending college [26]. Moreover, some research questions focus specifically on student populations (eg, studies that focus on unique college stressors and their links to mental health or interventions targeting college drinking), creating a need for student participant sources.

Student participation pools may potentially reflect the true population of interest in addition to being convenient; however, these pools can also be associated with lower enrollment and study compliance. Sharpe and Poets [26] found that as many as 56.7% of students in 2 large introductory courses chose not to participate in research or earn any research credits. Motivational issues and time commitment are the 2 primary factors linked to student nonparticipation in research pools [29,30]. It is possible that factors contributing to low motivation to participate may also impact study compliance among those who choose to participate.

Antoun et al [31] suggested that the type of recruitment method may explain the differences in data quality, differentiating between pull-in and push-out recruitment. They defined pull-in recruitment methods as those that post studies to participant pools already opting into research to some degree, such as Amazon Mechanical Turk (MTurk) workers or individuals looking for paid research opportunities on Craigslist (similar to student participant pools at academic institutions), whereas push-out recruitment uses methods in which advertisements are posted in venues not already focused on research, such as advertisements on websites not dedicated to the purpose of recruiting study participants (eg, Facebook advertisements), flyers, and email blasts. In a study comparing pull-in versus push-out approaches to recruit iPhone users for a cross-sectional web-based survey, Antoun et al [31] found that pull-in methods (using Craigslist and MTurk) were more efficient in recruiting participants, in that the rate of enrollment was faster and the cost per participant who enrolled in the study was lower, than push-out methods (using paid advertisements on Google and Facebook). Although no attention checks were included in the data collection, the authors concluded that the participants recruited through pull-in methods provided better data (ie, fewer “don’t know” responses and fewer skipped or incomplete responses), possibly indicating less satisficing. Multiple studies have extended these findings by recruiting samples across both pull-in and push-out approaches and including attention checks as indicators of data quality. One such study recruited participants from MTurk (pull-in), Facebook (push-out), and Qualtrics panels (pull-in) and included 1 attention check [32]. They found that the rate of passing the attention check question was highest among the participants recruited through MTurk (93%) compared with those recruited through Facebook advertisements (66%) and Qualtrics panels (40%). The participants recruited through MTurk endorsed “don’t know” responses only 0.4% of the time compared with those recruited through Facebook (4%) and Qualtrics panels (5%). These findings suggested that the push-out and pull-in distinction may be less important than the sources, given that both the highest (MTurk) and lowest (Qualtrics panels) rates of attention check failure were associated with pull-in sources. A similar study found that MTurk samples were more likely to pass the attention check (97.5%) than panel respondents via Dynata (91.6%), which were both pull-in sources [33]. These findings suggested that further research into the pull-in versus push-out distinction is necessary; differences in participant compliance have not been explored using the pull-in and push-out recruitment methods more commonly used on college campuses (ie, pull-in: a psychology student participation pool; push-out: email announcements to the general undergraduate student body). Moreover, no study to date has explored study compliance across these 2 sources among college students.

This examination explored study compliance (ie, data quality and retention) by recruitment source across 2 studies of college students in the United States with varying design protocols. These were follow-up analyses of published studies with different primary research goals. Study 1 focused on unique college stressors and links to mental health [34] and involved remotely distributing a web-based survey (fully remote and cross-sectional design). Study 2 examined an intervention targeting college drinking (ClinicalTrials.gov NCT03440463) [35] and involved an in-person baseline procedure with a computerized survey and remote web-based follow-up surveys 1 month and 3 months later (in-person component and longitudinal design). For both studies, participants were recruited from (1) a psychology student participation pool, receiving research credit in psychology courses as compensation and (2) the general student body via emailed announcements, receiving either a raffle entry (study 1) or monetary compensation (study 2).

This examination aimed to examine the following: (1) whether data quality varied across recruitment sources, (2) whether study retention varied across recruitment sources, (3) the impact of data quality on study variable associations, (4) the impact of data quality on measures of internal consistency, and (5) whether the demographic qualities of participants significantly varied across those who failed attention checks versus those who did not. Data quality was examined with attention checks that were used in both study 1 and study 2, and retention was operationalized as follow-up completion rates in study 2 only. Given the limited research on study compliance by recruitment methods with college samples, the analyses for aims 1, 2, and 5 were exploratory in nature. For aims 3 and 4, consistent with previous findings that satisficing can add noise to the assessment and reduce the strength of effects [1,11], we hypothesized that both internal consistency indicators and study variable associations would be stronger after eliminating those who failed attention checks. In particular, satisficing participants tend to endorse midpoints across multiple measures [10], potentially reducing the strength of association among variables, and the inclusion of satisficing participants can mask the strong effects that are revealed after their removal [1], supporting our hypothesis for aim 3. In addition to tendencies to endorse scale midpoints, satisficing participants also fail to notice scale reversals (ie, reverse-scored items) [11], potentially reducing indicators of internal consistency, supporting our hypothesis for aim 4.

Study 1 was a cross-sectional examination of worry as a mediator between psychosocial stressors and anxiety, stress, and depression [34]. The main outcomes of interest to the original study included worry, stress, depression, and anxiety.

Study 2 was a longitudinal (ClinicalTrials.gov NCT03440463) randomized control trial designed to examine the effects on drinking outcomes of personalized normative feedback booster emails sent after completing a web-based alcohol intervention administered in person [35]. The main outcomes of interest to the original study included alcohol consumption, alcohol-related problems, and descriptive normative perceptions (ie, how much one thinks relevant others drink).

An examination of whether data quality varies across recruitment sources (aim 1) revealed that a greater proportion of college student participants recruited through the psychology pool failed attention checks than that of those recruited through general emailed announcements, suggesting poorer data quality through satisficing. An examination of whether retention varies across recruitment sources (aim 2) revealed that the psychology pool was also associated with worse compliance via lower retention rates for the web-based follow-up surveys at 1 month and 3 months after baseline (study 2 only). For the examination of the impact of data quality on study variable associations (aim 3), there was no clear pattern of differences when comparing the strength of correlations between participants who did not fail any attention checks and those who did. The direction of the significant effect was consistent with our hypothesis for study 2 (ie, a stronger correlation was found among those who did not fail any attention checks) but was contrary to what was hypothesized for the 2 significant findings of study 1 (ie, stronger correlations were found among those who failed at least one attention check). Study 1 had 2 additional findings that were consistent with our hypothesis but did not reach significance. As for the impact of data quality on measures of internal consistency (aim 4), the impact of omitting those who failed attention checks was negligible on measures of internal consistency. Finally, when examining whether the demographic qualities of participants significantly varied across those who failed attention checks versus those who did not (aim 5), attention check failure was significantly greater among those who identified as Black or African American (both studies) and significantly lower among those who identified as White (study 1 only). It was not significantly associated with other racial identities, ethnicity, gender, age, year in school, or employment status.

Studies 1 and 2 were consistent in their findings that attention check failure rates were lower among the students recruited via general emailed announcements than among the psychology pool participants, suggesting better data quality (aim 1). This was true for both a remote, web-based, cross-sectional survey focused on college stressors and mental health (study 1) and an in-person longitudinal design examining an intervention for college drinking (study 2). However, the difference in rates was greater for the completely remote web-based study protocol (study 1: 19.3% vs 10.1% for failing any attention checks) than for the in-person baseline protocol (study 2: 13.5% vs 7.1%) for failing any attention checks, and rates were also generally higher for the web-based protocol. This finding became nonsignificant for study 2, as the sample was split into a smaller, more unbalanced proportion for examinations of failing ≥2 attention checks. Ward and Pond [44] found that having a researcher present via virtual meeting reduced careless responses by 2.13%, so having a researcher present for the in-person protocol at baseline may have reduced satisficing among participants who would otherwise have satisficed on the web (ie, changing the behavior of those enrolled in the study). In addition, the on-site protocol required that students sign up for a specific time slot and show up at a particular location on campus, requiring greater commitment. This may reflect greater motivation to participate (ie, impacting those who enrolled in the study), which has been linked to reduced satisficing among college students in prior work [14] and is consistent with the suppositions by Krosnick [16]. Thus, in-person protocols with specific sessions and researchers present may result in higher data quality through both who enrolls (self-selection of only those with greater motivation to participate) and through the protocol impacting the behavior of those enrolled (increasing motivation).

Antoun et al [31] noted that pull-in recruitment sources were more efficient (ie, faster rate of enrollment and lower cost) than push-out recruitment sources. This was true for study 2, a longitudinal study with an in-person baseline session. Enrollment was much higher using the psychology pool (n=401) than the general student body contacted via emailed announcements (n=127). Similarly, the psychology pool cost is lower (using research credits as compensation rather than monetary payments). However, the findings were contradictory for study 1, which yielded lower enrollment using the psychology pool (n=127) than the emailed announcements to the general student body (n=257). Both recruitment methods were relatively low cost, with participants from the general student body compensated only with entry into one of a handful of raffles for relatively low-cost gift cards.

The finding that satisficing was greater in the pull-in recruitment source (the psychology pool) than the push-out recruitment source (emailed general announcements) was contrary to the findings of Antoun et al [31]. In total, 3 studies comparing pull-in versus push-out recruitment focused on recruitment not specific to the college population, where participant presence in the pool or panel was completely through self-selection (eg, MTurk and Qualtrics or Dynata panels) [31-33]. These individuals joined the panel specifically to participate in research and earn money. By contrast, the pull-in source for this study included students enrolled in psychology courses who could participate in research studies for course credit (either as extra credit or as part of the requirements for the class). Although they could participate in research to earn a reward, and this is the sole purpose of the panel, their existence in the pool or panel was determined through course enrollment. This could suggest that their presence in the panel was less voluntary. However, equivalent credit could be earned through article critiques rather than study participation, making study participation fully voluntary. This could suggest that which recruitment method is best depends on whether the source is college specific or general sources. One study we are aware of has compared satisficing across a general pull-in source (MTurk for US $0.50) versus a college-specific pull-in source (a psychology pool for course credit), where they operationalized satisficing using nondifferentiation (ie, selecting the same response option for all items within a scale) [45]. They found the MTurk sample engaged in more satisficing than the college psychology pool. Given that both are considered pull-in methods, it may be that compensation structure (money vs course credit) was driving this difference, with the participants completing the survey for course credit providing better-quality data. Conversely, in our study comparing 2 college sources, we found that financial compensation was associated with lower rates of satisficing, whether these payments were larger and guaranteed (US $20 for baseline in study 1) or based on chance (raffles for study 2). In particular, the pull-in approach used in this study (a psychology student participation pool at a single institution) is widely used, and results to other psychology pools are highly generalizable. However, many pull-in approaches (eg, Amazon MTurk) contain participant panels of individuals from across the country and often the globe. This makes the findings of this study less generalizable to pull-in approaches more broadly. With no robust findings across studies regarding push-in versus pull-out methods or financial compensation versus course credit, it appears that there is no guaranteed method to minimize satisficing, making its detection critically important. Attention is a prerequisite for receiving the treatment in most survey experiments, and attention checks effectively reveal who receives the treatment and who does not, such as when Berinsky et al [11] found large condition effects among those who passed the attention screener and no condition effects among those who failed. Detecting and eliminating satisficing is critical for researchers conducting studies examining treatments for psychological health.

The same recruitment source (announcements emailed to the general student body) provided both greater study retention (aim 2) and higher data quality (aim 1); thus, longitudinal researchers can choose a recruitment method that optimizes study compliance for both minimizing satisficing and promoting retention. It is worth noting that retention may have been better for the students who participated through student announcements because their compensation for follow-up surveys was consistent with their compensation for baseline (financial), unlike the psychology pool (course credit). Attrition for longitudinal psychological treatment studies is particularly critical, as meta-analyses have shown dropout rates of 24% to 35% for smartphone-delivered mental health interventions [46], 26% for cognitive behavioral therapy [47], 21% for eating disorder e-treatments [20], and 25% for individual college drinking interventions [48], among others. Researchers striving to minimize satisficing in their clinical trials must still try to optimize retention, and choosing an appropriate recruitment method may help with both concerns.

Contrary to our expectations, correlations did not show consistent strengthening of study variable associations or effects across the 2 studies (aim 3). Select correlations did change significantly in both studies, but the effects went in both directions (sometimes stronger and sometimes weaker). The strengthening of some correlations is consistent with multiple prior studies finding stronger effects after screening out participants who satisficed [1,11,12]. However, falsely inflated values in the full sample were similar to those reported by Huang et al [13], still pointing to a bias introduced by including these participants. Moreover, Credé [10] noted that whether random responding inflates or deflates the true value of correlations may be influenced by whether the measures examined naturally peak around the lower end of the response option continuum (such as with suicide ideation, psychopathy, and depression) versus around the higher end of the continuum (such as with self-esteem and altruistic behavior) as well as by whether the correlation among those not satisficing is positive or negative, suggesting that both inflated and deflated correlations can be expected with satisficing. Moreover, how participants are carelessly responding may influence the direction of bias. King et al [49] found that when data are not skewed, uniformly responding (ie, each response option has an equal chance of being selected) falsely deflates estimates, whereas long-string responding (ie, selecting the same response option for many items in a row) falsely inflates estimates. Thus, our findings demonstrating correlations changing in both directions support the notion that screening out participants who are satisficing does impact study findings, potentially reducing bias.

Also contrary to our expectations, measures of internal consistency were not stronger after dropping participants who failed attention checks (aim 4). The differences were negligible across the 2 studies. If satisficing by participants adds noise to the data set, researchers might expect it to add measurement error as well. Oppenheimer et al [1] found that internal consistency was reduced among those failing an instructional manipulation for a measure containing reverse-scored items, but these findings were not replicated in this study. However, only 1 measure in this study (the Penn State Worry Questionnaire) [36] contained reverse-scored items, which might be more sensitive to satisficing.

In this study, examinations were repeated for multiple cutoffs for satisficing (ie, failing any attention checks vs failing a larger number of attention checks such as 2 or 3). A zero-tolerance approach for identifying satisficing, excluding participants who had ≥1 incorrect responses to attention checks, is consistent with what is most commonly reported by researchers [9]. Although a recent examination revealed that the zero-tolerance approach can result in excluding more participants, in particular those who do not demonstrate satisficing on other indices, it is the most common way of screening participants for data quality [9]. This study did not reveal major differences in the pattern of findings across zero tolerance versus basing the cutoff on a larger number, suggesting that there may be some flexibility in which approach a researcher might choose.

One concern raised by prior researchers is whether screening out satisficing participants could introduce a different source of bias, namely reducing the demographic diversity of the sample [11,14]. This study found that attention check failure was not significantly associated with ethnicity, gender, age, year in school, or employment status, indicating that bias is not introduced for these dimensions. However, the participants who identified as Black or African American were more likely to fail at least one attention check, suggesting that screening out participants could introduce a concern relating to reducing the diversity of the sample. Researchers should be thoughtful regarding recruitment strategies to access larger numbers of participants who could be lost to this screening process so that the final sample still has a substantial representation of this group, as in this study.

In addition to screening out those who fail attention checks, another possible approach that would allow researchers to retain everyone in the sample is to use live feedback to inform participants that the researchers have noticed that they are not paying attention and ask them to read the items carefully. Prompting respondents who completed survey items very quickly to note that this was likely too fast to respond accurately and asking whether they want to reconsider their answers led to reduced satisficing and more accurate responses [50]. Similarly, providing feedback when someone fails an attention check can increase measurement quality [51]. King et al [49] noted that almost no published research in the addiction literature reports screened their data for satisficing. It may be that for research that heavily invests resources in obtaining each data point (such as with longitudinal research and clinical trials, common in the addiction field), throwing out cases results in heavy resource loss, and researchers may be more motivated to try to detect and eliminate satisficing as it occurs. Berinsky et al [11] used different strategies to improve attention during data collection, including warning participants that their data would be monitored before beginning the survey, pairing this warning with a message thanking the participants for their time and careful attention, and providing live feedback (ie, “There was a problem with your response. Please try again”). All 3 approaches resulted in higher rates of passing the attention check items. However, these approaches did not result in reduced noise or bias in associations among study variables or larger treatment effects. It may be that the framing of these messages matter. A systematic review of studies examining prompts in health promotion or health behavior interventions found that messages were more effective if they were tailored with a personal touch [52]. Similarly, a review of retention in panel studies emphasized that explaining the importance of the project and the contributions of the participants is key to engaging participants and promoting good study retention [53]. The same approaches can be used to promote good data quality. Accentuating the purpose and importance of the study, how the participants are helping, and that their responses are of great value to the researchers may have an effect on not just passing attention checks but also actually increasing attention and minimizing noise. Pairing this warm introduction to the fact that the responses will be monitored with a similarly framed live feedback message when attention checks are failed (eg, “Your answer for this question is not correct. Your contributions to our research are extremely valuable. Please be sure to read questions thoroughly and answer carefully”) may have more of an impact on data quality.

To promote data quality and minimize bias, we have several recommendations for researchers. (1) Use attention checks to detect satisficing. Failure of attention checks was prevalent in both studies across both recruitment sources, suggesting that satisficing is occurring among college students regardless of the study design or recruitment method. Moreover, findings changed after screening out those who failed at least one attention check, suggesting that ignoring this phenomenon could introduce bias into study conclusions. Attention checks can help researchers identify who is providing higher quality data. (2) Carefully consider the recruitment source. Although using psychology pools can cost less and be more efficient (as in study 2, a longitudinal study with an in-person baseline session) and potentially be more convenient, recruiting using broader methods to reach students may result in a better-quality sample (ie, lower satisficing and greater retention). Moreover, the broader recruitment source was more efficient in study 1 (remote and cross-sectional), suggesting that researchers may want to consider their study design in making this determination. When possible, researchers might use multiple recruitment sources to diversify their samples. (3) Weigh the benefits of screening out the participants who fail attention checks (demonstrated to reduced bias in study findings) versus including live feedback (very limited research on this approach). Related to this, (4) consider whether screening out participants could reduce demographic diversity. It could be problematic to increase internal validity to the detriment of external validity. If researchers intend to use attention checks for screening purposes, then they might oversample from populations more likely to be screened out (if possible). Alternatively, letting participants know that their responses will be monitored for data quality and providing live feedback could minimize attention check failure. For treatment studies or other longitudinal studies where tossing cases is problematic, live feedback may be a better option. Finally, (5) researchers interested in minimizing satisficing rather than detecting and removing the data from these participants might consider holding time-specific sessions with a researcher present (in person or on the web).

This investigation was a 2-study examination using different study designs (cross-sectional web-based survey vs in-person baseline for a longitudinal randomized controlled trial) and different domains of inquiry (mental health vs drinking behaviors) to maximize the external validity and relevance for other psychological health researchers. However, several limitations should be noted. First, attention checks were the only indicators of data quality used. More robust approaches have included additional indicators, such as psychometric antonyms or synonyms (ie, within-person correlations of similar items), LongStrings (ie, length of response patterns with the same value), Mahalanobis distance values (ie, multivariate outliers for similar items), and self-report items of attention and effort [8,34]. The main advantages of this approach are ease of use, nonrequirement of specialized data management skills, and speed of the screening process. However, researchers may consider using an error-balancing approach that takes multiple indices into account, particularly if working with smaller data sets of specialized populations that are harder to access, where keeping more cases is much more critical.

Another limitation of this examination was that the recruitment sources were limited to single-site data collection methods at 1 institution in the United States. Amazon MTurk is another approach that researchers can use to access the student population more broadly, potentially increasing the demographic diversity of the study samples while maintaining high data quality, including lower rates of satisficing [54,55]. Other web-based approaches, not limited to a single site, could include advertisements on Facebook, Craigslist, etc.

This may increase the demographic and geographic diversity of the sample [54], although the confirmation of student status may be harder. In addition, these pull-in methods may result in a sample with lower income that engages in greater risky behaviors [56] if such qualities are relevant to the research questions being examined. Moreover, although the samples used in this study had a strong representation of Black or African American and White racial identities, other identities were not as well represented. In particular, aim 5 had low sample sizes for some examinations. Although study 2 used a protocol that allowed us to identify all the participants and prohibited repeat sign-ups, study 1’s fully web-based protocol did not. The psychology pool participants likely also saw the survey via university-wide announcements, although the recruitment materials requested that students complete the survey only once. Unfortunately, the nature of the system used for the psychology pool uses only anonymous identifiers to issue research credits in the system, so we could not verify for ourselves that the psychology pool participants did not also complete the survey via university announcements.

Finally, although we focused on recruitment sources to label the differences between these 2 groups, compensation was also different. Students in the psychology pool were compensated with research credits that could be applied to their course grades. Students in the emailed announcement group were compensated monetarily (with a raffle entry in study 1 and direct payments in study 2). We believe that this is consistent with most studies using these recruitment sources and feel that compensation is part of these approaches. What is notable is that the pattern of reduced satisficing in the emailed announcement group was true even when the compensation was weak (raffle entry) rather than strong (direct monetary payments), suggesting that the strength of compensation is not driving the effect.

This investigation examined participant compliance (ie, data quality and retention) by recruitment source across 2 studies of college students with varying design protocols (study 1: a fully remote, cross-sectional design examining college stressors and psychological health; study 2: a longitudinal design with an in-person baseline session that examined an intervention targeting college drinking). For both studies, the participants were recruited from (1) a psychology student participation pool, receiving research credit in psychology courses as compensation, and (2) the general student body via emailed announcements, receiving either a raffle entry (study 1) or monetary compensation (study 2). The examination revealed that a greater proportion of college student participants recruited through the psychology pool failed attention checks than that of those recruited through general emailed announcements, suggesting poorer data quality through satisficing in both studies. Moreover, the psychology pool was also associated with worse compliance via lower retention rates in the web-based follow-up surveys at 1 month and 3 months after baseline (study 2 only). After screening out those who failed at least one attention check, some correlations among the study variables were strengthened (potentially due to reducing noise), some were weakened, and some were fairly similar; this mixed pattern potentially points to a bias introduced by including these participants. Finally, attention check failure was not significantly associated with most demographic characteristics (ethnicity, gender, age, year in school, employment status, and select racial identities) but was greater among those who identified as Black or African American (both studies) and significantly lower among those who identified as White (study 1 only). Investigators focused on student research should carefully consider recruitment in their study design and include attention checks or other means of detecting poor quality data. Satisficing was detected across both sources, although it was worse in the psychology pool than in the general student body. Researchers should carefully consider how the study design could promote engagement (eg, live sessions with a researcher), weigh screening participants versus providing live feedback, and consider oversampling demographics that are more likely to be screened out, if possible.