This study was based exclusively on computer-generated simulated data and did not involve human participants, identifiable human material, or identifiable human data. Therefore, research ethics committee review and informed consent were not applicable.

This simulation study investigated single-choice Type A items with 1 correct answer option and 4 distractors. The focus on 5 answer options reflects common practice in medical education; however, the analytical model can be generalized to other numbers of options with adjusted guessing probabilities. The following 3 scoring methods most frequently described in the literature were selected [4]:

For each scoring method, the expected chance scores from random guessing [11] and cutoff scores corresponding to 50% knowledge-based response probability were derived analytically and are summarized in Table 1.

Examination data were generated using a Monte Carlo simulation framework designed to isolate the mathematical effects of scoring transformations. Monte Carlo simulation is a well-established approach for evaluating statistical properties of measurement procedures [12]. Each examinee was characterized by a knowledge-based response probability k∈[0,1], representing the assumed probability of producing a correct response based on knowledge rather than guessing. This definition is consistent with the probabilistic framework used in previous work on scoring methods [13]. The parameter k is a model parameter and should not be interpreted as an observed percentage of knowledge, a direct measure of competence, or a validated latent ability measure.

For a given value of k, the probability of answering an item correctly was modeled as p(k)=k+(1–k)/5, representing a combination of knowledge-based responding (with probability k) and uniform random guessing (with probability 1−k across 5 response options). The modeling approach serves as a simplified baseline that does not include informed guessing or partial knowledge. It is conceptually related to probabilistic frameworks used in item response theory, in which observed responses are linked to underlying examinee characteristics [14]. The formulation assumes that lack of knowledge results in random guessing across all answer options, excluding informed guessing or distractor elimination strategies, consistent with established models of multiple-choice responding that include a guessing component [15]. Responses to individual items were assumed to be conditionally independent given k and were generated as Bernoulli trials with success probability p(k) [16]. The model is intended to isolate the structural effects of scoring transformations rather than to represent the full complexity of response behavior in real examinations.

To represent different levels of knowledge-based response probability, 3 groups of examinees were selected for illustrative graphical representation: examinees with (1) low k (k=0.25; expected to fail), (2) intermediate k (k=0.55; expected to pass), and (3) high k (k=0.85; expected to pass). These values were chosen as illustrative model scenarios.

For each simulated examination, response data were scored according to the 3 scoring methods. In a second approach, total scores were converted into proportions of the maximum possible score to ensure comparability among varying test lengths. For each scoring method, a pass mark (ie, the expected score of an examinee with k=0.5) was defined (Table 1). This calibration is to ensure comparability across scoring methods and to isolate the effect of score transformation. Examinees with observed scores at or above the pass mark were classified as passing, whereas those below the pass mark were classified as failing. Because the number of correct responses is discrete, noninteger pass thresholds were rounded up to the next attainable score. A small numerical tolerance was applied to avoid floating-point artifacts.

The primary outcomes were classification accuracy and misclassification rates, analyzed in relation to item number, scoring method, and distance from the model-defined cutoff. Classification accuracy was defined as the proportion of examinees correctly classified as passing or failing relative to the model-defined cutoff at k=0.5.

For each simulation, 4 classification outcomes were identified:

Overall accuracy was then computed as (TP+TN)/(TP+FP+TN+FN).

All analyses were conducted using R (version 4.5.2; R Foundation for Statistical Computing) [17]. Simulations were implemented using custom scripts. Detailed model specifications, analytical equations, selected implementation code, and the fixed random-number settings used for stochastic analysis component are provided in Multimedia Appendix 1. Machine-readable study-specific outputs underlying the reported tables and figures and the software session information are provided separately in Multimedia Appendices 2 and 3.

To illustrate variation in examination scores with increasing test length, cumulative score trajectories were generated for randomly selected examinees from each of the 3 predefined knowledge-based response probabilities (k=0.25, 0.55, and 0.85) and plotted as functions of the number of administered items.

To evaluate classification performance, simulations with 10,000 replications were conducted using cohorts of 20 to 1000 examinees to ensure numerical stability of the results. Importantly, cohort size does not influence the classification mechanism itself but only the precision of estimated accuracy metrics. Corresponding simulation-based variability bands were graphically displayed for 50, 100, and 1000 examinees. Two different distributions of knowledge-based response probability were considered: (1) a uniform distribution over the interval [0,1], representing a heterogeneous population with uniformly distributed k; and (2) a truncated normal distribution centered at the cutoff k=0.5, with approximately 95% of values falling between k=0.05 and k=0.95, representing a population concentrated around the cutoff threshold.

For each cohort, examination responses were generated and scored according to the 3 scoring methods. Classification accuracy, sensitivity, and specificity were computed as functions of test length. In addition to the simulation-based estimates, theoretical classification accuracy was derived analytically from the underlying stochastic response model using binomial probabilities and normal approximations. The analytical results were compared with the corresponding simulation estimates to assess numerical consistency and convergence [18].

Figures 1 and 2 illustrate variation in examination scores with increasing item numbers. At low item counts, substantial fluctuations in trajectories of cumulative scores and the percentage of the maximum possible score were observed. With increasing examination length, interindividual variability decreased and trajectories converged toward their expected values. Beyond a certain number of items, relative examination results changed only marginally with the addition of further items. Across all scoring methods, convergence toward the expected score occurred earlier for higher k (k=0.85) than for intermediate (k=0.55) and low k (k=0.25). This pattern is consistent with a reduced contribution of guessing-related random variation at higher k.

As shown in Figure 2, expressing results as a percentage of the maximum possible score yields a linear rescaling of the score scale across different scoring methods. Although the scale is transformed, the ordering of examinees and the position of the pass mark remain unchanged when thresholds are aligned. Accordingly, the magnitude of observed fluctuations is stretched or compressed, whereas the shape of score trajectories with increasing item numbers remains unchanged across scoring methods. If pass marks are defined consistently with the respective scoring scale (ie, using the score threshold corresponding to the same model-defined cutoff), the resulting pass or fail classifications are identical across all tested scoring methods. This finding also held across all modeled test lengths and distributions of k.

At the cohort level, overall classification accuracy (ie, correct pass or fail classification relative to the model-defined cutoff) increased with the number of items (Figure 3). Accuracy estimates became more stable with increasing cohort size: 95% simulation-based variability bands were wide for small cohorts (n=50), narrower for moderate cohorts (n=100), and narrow for large cohorts (n=1000), whereas the underlying classification accuracy itself remained unchanged. This pattern was consistent across both underlying distributions of k (uniform over [0,1] and truncated normal). The mean classification accuracy, however, differed by distribution. Under a uniform distribution, k is uniformly distributed, whereas the truncated normal distribution results in a higher concentration of examinees in the vicinity of the cutoff region, where discrimination between passing and failing is at its lowest and misclassification risk is at its highest. Consequently, cohort-level accuracy increased more slowly under the truncated normal distribution. For the uniform distribution, mean accuracy reached 0.90 after 24 items and 0.95 after 94 items (Figure 3A). In contrast, under a normal distribution truncated to [0,1] and centered at k=0.5 (SD 0.45/1.96; approximately 95% of k values fell between 0.05 and 0.95), mean accuracy reached 0.90 after 71 items and approached 0.95 only marginally at 300 items (Figure 3B). These item numbers are model- and distribution-dependent and should not be interpreted as recommended examination lengths. Similar plots for sensitivity and specificity, as well as the underlying data for accuracy, sensitivity, and specificity are shown in Multimedia Appendices 4-6.

Misclassification rates were estimated using repeated simulations at fixed k levels (Table 2). For each k level, cohorts of 1000 examinees were simulated, with 10,000 repetitions per level. Misclassifications were strongly concentrated around the cutoff and declined rapidly as knowledge-based response probability moved away from the decision boundary. For very low k (eg, k≤0.2), incorrect classifications fell below 0.1% by 50 items and approached zero beyond 100 items. For high k (eg, k≥0.85), incorrect classifications were already rare at 20 items and approached zero thereafter. In contrast, misclassification risk remained substantially elevated near the cutoff even at large item counts. At k=0.45 (ie, fail), the mean misclassification rate was 9% at 300 items. At k=0.55 (ie, pass), it remained 6.7% at 300 items. At the exact cutoff (k=0.5), the expected observed score coincides with the classification boundary. Consequently, increasing test length does not lead to near-certain classification. At 300 items, approximately 47.5% of simulated examinees were classified as failing despite the model-defined reference classification of passing (Table 2).

If a fixed 60% pass mark is applied across all scoring methods (ie, without calibrating pass marks to the same underlying knowledge-based response probability threshold), misclassification rates follow a similar pattern. For formula scoring and right-minus-wrong scoring, examinees are required to show a higher level of knowledge-based response probability to achieve scores at or above the 60% pass mark (ie, 60% and 75%, respectively). Therefore, misclassifications cluster around these values of k, as shown in Multimedia Appendices 7 and 8.

For individual examinees, the number of items required to achieve a given probability of correct classification was derived for different levels of k (Figure 4; Table 3). The required number of items depended nonlinearly on k and increased sharply near the cutoff (k=0.5). In this region, classification was highly sensitive to random score variation. Substantially fewer items were needed further away from the cutoff. For example, at k=0.25, 17 items were required to reach P value (correct classification) ≥.95, whereas at k=0.85, 3 items were sufficient (Table 3). Near the cutoff, the required number of items increased markedly (eg, k=0.45: 402 items; k=0.55: 370 items for P≥.95; Table 3).

This study demonstrates a fundamental invariance property of pass or fail classification decisions under linear score transformations, supported by analytical methods and simulations: under calibrated pass marks and assuming behavioral invariance, the investigated scoring methods for single-choice multiple-choice examinations yield identical classification outcomes. This equivalence follows mathematically from the linear relationship between the scoring rules; the simulations illustrate the effects of test length, distance from the cutoff, and the distribution of k on classification accuracy.

When decision thresholds are properly calibrated and examinees’ response behavior is not influenced by the scoring method, modifying scoring methods alone does not alter decision quality. These findings directly challenge the common assumption that penalty-based scoring methods improve the accuracy of pass or fail decisions.

The present findings distinguish between two conceptually different mechanisms by which scoring methods may influence examination outcomes: (1) structural effects arising from mathematical score transformations and (2) behavioral effects arising from changes in examinees’ response behavior. By isolating the structural component, the present study shows that scoring methods alone do not change pass or fail classification outcomes when pass marks are aligned to the same model-defined cutoff. The absence of an effect under these conditions implies that scoring transformations do not improve classification accuracy independently of behavioral mechanisms. However, in real-world settings, scoring rules may influence examinees’ behavior, including omission strategies, guessing behavior, and risk-taking. Such behavioral responses may significantly affect observed scores independently of examinees’ knowledge-based response probability.

Beyond scoring, classification accuracy increased with the number of items across all conditions. However, the rate of improvement depended strongly on the distribution of examinees’ knowledge-based response probability. Accuracy increased rapidly under a uniform distribution but substantially more slowly when examinees were concentrated around the cutoff. Misclassification risk was highest near the decision threshold and decreased as knowledge-based response probability moved away from this boundary. These patterns were identical across all scoring methods, indicating that classification accuracy is primarily determined by examinees’ k relative to the cutoff rather than by the scoring methods.

In practice, scoring rules may influence examinee behavior and thereby affect observed performance. Within the conditions simulated here, classification accuracy increased with test length. Penalty-based scoring methods merely rescale observed scores without changing the underlying classification when pass marks are calibrated, and response behavior remains unchanged. Additional concerns, such as construct-irrelevant variance related to risk-taking, increased test anxiety, fairness issues, and potential legal constraints [5-8], may further limit their practical justification in summative assessments where classification accuracy is the primary objective.

More broadly, these findings should be viewed in the context of longstanding efforts to correct multiple-choice scores for guessing [4,10,11]. From an educational perspective, the results indicate that classification accuracy in summative multiple-choice examinations is primarily determined by test length and the distribution of examinees’ knowledge-based response probability rather than by the choice of scoring method. Although scoring transformations modify the score scale, they do not alter the underlying decision problem when pass marks are aligned to the same model-defined cutoff. However, real-world assessment policy decisions may still need to consider behavioral responses of examinees and institutional or regulatory constraints.

Regarding pass or fail classifications in summative assessments, standard-setting procedures define minimum competence levels and are inherently affected by measurement error near the cutoff. Classification accuracy is lowest close to the decision threshold, where small variations in observed scores can lead to different classification outcomes [19-21]. In populations where many examinees are located near the pass or fail threshold, classification accuracy improves only slowly, even with large numbers of items. This pattern is consistent with established findings on decision accuracy and classification consistency in educational measurement [20,21] and emphasizes a fundamental limitation of dichotomous decision-making.

A major strength of this study is the combination of analytical methods and large-scale Monte Carlo simulations. This allows for both exact and empirical evaluation of classification behavior under controlled conditions, and the calibrated pass marks ensured comparability across different scoring methods by focusing on the mathematical effects of score transformations. In addition, the systematic variation of item numbers, cohort sizes, and knowledge distributions provides a comprehensive characterization of classification accuracy across the prespecified range of testing scenarios.

The most important limitation of this study is the assumption of behavioral invariance. This assumption is intentionally restrictive and serves to isolate the purely mathematical contribution of scoring systems. In real-world examination settings, scoring rules can systematically influence examinees’ response behavior, including omission strategies, guessing behavior, and risk-taking [5-7]. Such behavioral adaptations may dominate observed differences between scoring methods and are not captured in the present model. The present findings should be interpreted as isolating the structural properties of scoring methods and not as a complete account of their impact in applied assessment contexts.

Beyond this central assumption, several additional modeling simplifications should be considered. First, item difficulty was not varied, implying a homogeneous item set and excluding potential effects of a realistic mix of easy and difficult questions or varying values of k across different topics on score convergence and classification performance. This simplification is intentional, as it isolates the effect of scoring transformations independently of item parameter heterogeneity, which would otherwise obscure the underlying invariance property. Second, informed guessing was not modeled, and responses were generated without incorporating partial knowledge or test-taking strategies such as systematic distractor elimination (ie, test-wiseness). Third, no item omissions were modeled, and all examinees were assumed to respond to all items. Fourth, the model assumes conditional independence between items, which may not hold in real examinations. Finally, although cohort size did not influence the classification mechanism itself, it affected the precision of estimated accuracy. In addition, the distribution of knowledge-based response probability in the target population is typically unknown but directly affects cohort-level classification accuracy.

Despite these simplifications, the model serves its intended purpose of isolating fundamental statistical relationships between scoring methods, test length, and classification accuracy.

Future research should extend the present model to more realistic testing conditions. Incorporating heterogeneous item characteristics within an item response theory framework would allow the modeling of varying item difficulty and discrimination and provide a closer approximation to real examinations. In addition, future models should explicitly account for behavioral responses to different scoring methods, including omission strategies and individual differences in risk-taking, to assess the combined psychometric and behavioral effects of penalty-based scoring.

Further work also needs to model examinees’ partial knowledge and informed guessing, which may alter the relationship between k and observed response probabilities. Finally, empirical validation studies using real-world examination data are essential to determine the extent to which the theoretical equivalence of scoring methods observed under calibrated conditions holds in practice.

When pass marks are calibrated to a common model-defined cutoff, the investigated scoring methods yield equivalent classification outcomes under complete responding and unchanged response behavior. Classification accuracy increased with test length and depended on the distribution of k, while uncertainty remained greatest near the cutoff. These findings describe the structural effects of the scoring transformations examined here and do not address behavioral effects or establish an optimal examination length.