A Mutation-Based Framework for Assessing Code Understanding Through Behavioral Change
Manish Kumar · Aug 15, 2026
A learner who can correctly predict what a piece of code will output has shown one skill; a learner who can also predict how that output changes after a small edit, and explain why, has shown something closer to real understanding. This paper builds an assessment around that second skill. Each item pairs an original program with a small, behavior-changing modification of it, evaluates both on the same fixed input, and then asks four questions in sequence: what the original program outputs, what the modified program outputs, whether the two outputs differ, and why. Conventional mutation testing uses the same kind of small, artificial change but points it at a test suite rather than a person — a mutant is "killed" if some test detects it, and the exercise measures test-suite adequacy. Here the mutant is aimed at the participant instead: it becomes the question, and the participant's response is what gets studied. The resulting instrument has seven items and a maximum score of 35 points, backed by a Python pipeline that generates the assessment, validates incoming responses, scores both the objective and free-text answers, checks the dataset for integrity problems, and produces summary statistics. No human participants have been recruited yet, so the pipeline was instead run against a clearly labeled synthetic pilot — 20 simulated respondent profiles across three ability bands, yielding 140 item-level responses. That run produced a mean score of 25.55/35 (73.0%), band means of 20.43/35, 23.29/35, and 34.17/35 for the Beginner, Intermediate, and Strong profiles, and objective-item accuracies of 80.7%, 74.3%, and 80.0% on the first three questions. None of these numbers describe real learners; they describe whether the pipeline itself behaves correctly, which it did. What this paper contributes, then, is not a set of findings but a validated instrument and pipeline, ready to be pointed at real participants. Keywords: code understanding; program comprehension; mutation analysis; mutation testing; behavioral change; programming assessment; reproducible pipeline; synthetic pilot