A Mutation-Based Framework for Assessing Code Understanding Through Behavioral Change

1. Introduction

Understanding a piece of code and being able to write one are not the same skill. The first requires tracing control flow, following how data moves and transforms, and anticipating the consequences of each line — capabilities that classroom exercises rarely test directly, since it is easier to grade a learner on whether their own program runs than on whether they can reason about someone else's. Yet comprehension is not a byproduct of writing ability. Prior work has tied it to factors such as prior exposure to a codebase and general programming experience [4], which is one reason to measure it on its own terms rather than infer it from output correctness on a fresh problem.

Mutation testing supplies a useful mechanism here, because it introduces a small, well-defined change into an otherwise correct program. The technique has traditionally served test suites, not people: a mutant counts as killed once some test detects it, and the resulting score measures how adequate the suite is [1], [2]. Computing-education research has since redirected the idea toward students, showing that a carefully chosen mutant can expose a specific misconception, provided it represents something a learner might plausibly get wrong rather than an arbitrary syntactic accident [3].

That second use of mutation is the starting point here. Rather than asking a learner to write a program, or to read one program on its own, this instrument shows two programs — an original and a modified version of it — and asks the learner to reason about both. Four questions carry that reasoning forward in stages: predict the original behavior, predict the modified behavior, decide whether the two differ, and explain why. Keeping the questions in this order matters, since it stops a participant from writing an explanation before they have actually committed to a prediction.

Section 2 motivates the approach against existing mutation-testing work. Section 3 lists the research questions. Section 4 describes the instrument. Sections 5 and 6 cover the implementation, its quality-assurance checks, and the evaluation protocol used for the pilot. Section 7 reports pilot results, Section 8 discusses them, and Sections 9 through 12 address validity, reproducibility, limitations, and conclusions.

1.1 Contributions

• A seven-item, four-question assessment instrument (35 points total) that keeps output prediction and causal explanation as separate, separately scored skills.
• A Python pipeline that handles the whole path from assessment generation through response validation, scoring, and aggregate analysis without manual steps in between.
• A data-integrity protocol that keeps synthetic and real participant data in physically separate files, checked automatically rather than tracked by hand.
• A completed synthetic pilot — 20 profiles, 140 responses — that exercises every stage of the pipeline before a single real participant is involved.

2. Motivation and Research Gap

Mutation testing and comprehension assessment sound similar but ask different questions. One asks whether a test suite is sensitive enough to notice a change; the other asks whether a person can notice it, predict its effect, and explain the mechanism behind it. Killing a mutant proves nothing about whether a human being could describe what the mutant actually did.

Conceptual mutation testing has already shown that a mutant can stand in for a specific, realistic misconception rather than an arbitrary fault [3]. What this framework adds is making the participant's response to that mutant the measurement itself, not a side observation. Participants predict two outputs and then justify the gap between them, so the explanation is scored on its own rather than treated as commentary.

The claim stays deliberately modest. Mutation-based questions are not offered here as a complete theory of code understanding — only as a controlled setting, one where the cause of a behavioral difference is known in advance, in which a participant's reasoning about that cause can actually be observed.

3. Research Questions

• RQ1. Can a small set of controlled mutations be turned into assessment items whose original and modified versions produce clearly distinguishable outputs on a fixed input?
• RQ2. Does splitting the assessment into four questions actually separate distinct skills — predicting baseline behavior, predicting modified behavior, recognizing that behavior changed, and explaining why — rather than measuring the same thing four times?
• RQ3. Can a single software pipeline validate, score, and summarize response data automatically, while keeping synthetic and real participant data from ever mixing?
• RQ4. What does running the pipeline on a synthetic pilot show about its technical correctness, and which questions does it leave open until real participants are involved?

4. Framework Design

4.1 Assessment unit

Five parts make up every item: an original program, a modified version of it, one fixed input, four questions, and a scoring rule. Fixing the input, rather than letting participants supply their own, keeps every response checkable against a single known pair of outputs.

4.2 Mutation operators

Table 1 lists the four operators used, each swapping a comparison operator or a numeric constant. A change this small is enough to flip a branch or shift a computed value, yet the modified program still reads like a close relative of the original — which keeps the reasoning required within reach of a short assessment session.

Operator	Change	Typical effect
Equality to inequality	== → !=	Inverts a conditional branch
Addition to subtraction	+ → −	Alters an arithmetic computation
One to zero	1 → 0	Corrupts an initialization value
Zero to one	0 → 1	Corrupts a counter or threshold value

4.3 Assessment items

Seven programs make up the instrument: a linear search, a counting routine, two factorial implementations, and three classifiers that count even, positive, and negative numbers. Between them they cover distinct control-flow and arithmetic patterns without requiring any specialized domain knowledge.

4.4 Participant questions

Every item asks the same four questions in the same order — original output, modified output, whether the two differ, and why. The ordering is not incidental: it keeps a participant from explaining a change before they have committed to what that change actually produced.

4.5 Scoring

Each item carries five points, thirty-five in total across the seven. Q1 through Q3 are graded against a known correct answer. Q4 follows a three-level rubric instead: two points for an explanation that names both the mechanism and its effect, one point for correctly noticing that behavior changed without correctly explaining why, and zero for anything incorrect, off-topic, or blank.

5. Implementation and Quality Assurance

The pipeline is a set of Python scripts working over CSV files rather than a database — a choice that keeps every artifact readable and auditable without special tooling. It generates the participant-facing assessment document, assigns participant identifiers, imports response files, checks that each response points to a real assessment item, rejects duplicate participant records, flags missing answers, scores both the objective and free-text responses, and produces the summary statistics reported in Section 7.

Checks run before scoring ever starts. The assessment package is verified for all seven items, for the presence of original and modified code with its input and questions, and for the absence of any answer key or scoring detail that could leak into what a participant sees. Response files go through a separate pass checking for duplicate identifiers, references to nonexistent items, and unanswered questions.

Before packaging, the active research dataset — the one meant to eventually hold real responses — was reset to empty on purpose. Synthetic pilot data lives in its own file, labeled SYNTHETIC throughout. A final integrity pass confirmed zero records in the real dataset and exactly 140 labeled records in the synthetic one.

6. Evaluation Protocol

What follows is a technical pilot, not a study of real participants. Twenty synthetic profiles were built to push data through every stage of the pipeline: seven Beginner, seven Intermediate, six Strong. Each answered all seven items, for 140 response records in total.

Validation ran before scoring, the same way it eventually will on real data: record count, 20 distinct participant identifiers, all seven items represented, seven responses per profile with none duplicated, no missing answers, and a valid SYNTHETIC label and profile category on every record.

Scoring followed validation — per-profile totals, per-question accuracy, Q4 scores, band summaries, the score distribution, and a handful of pilot-quality indicators. None of it speaks to how real learners would do. It speaks to whether the scoring and analysis code is correct, which is the only thing this stage is designed to check.

7. Results

7.1 Overall synthetic-pilot performance

The 20 synthetic profiles produced a mean score of 25.55 out of 35 (73.0%), a median of 22.50/35, a standard deviation of 6.14, and a range running from 16/35 up to a perfect 35/35 — summarized in Table 2 and Figure 1.(Distribution of synthetic participant scores)

7.2 Profile-level performance

By band, the separation is exactly what the construction intended: Beginner profiles average 20.43/35 (n = 7), Intermediate 23.29/35 (n = 7), Strong 34.17/35 (n = 6), plotted in Figure 2 (Mean synthetic pi;lot score by profile). That separation confirms the scoring logic responds correctly to its input; it says nothing about how real learners at different skill levels would actually score, since the bands were built to differ rather than observed to.

7.3 Objective questions

Q1 was answered correctly on 113 of 140 responses (80.7%), Q2 on 104 of 140 (74.3%), and Q3 on 112 of 140 (80.0%) — see Figure 3 (objective question accuracy (Q1-Q3) in the synthetic pilot)

7.4 Explanation question

Q4 carries up to 14 points per profile. The overall mean landed at 9.10/14 (65.0%): Beginner and Intermediate profiles both averaged 7.00/14, and Strong profiles hit the ceiling at 14.00/14. At the level of individual responses, 98 of 140 scored one point and 42 of 140 scored two.

Table 1. Final assessment coverage across the seven items.

Item	Primary construct	Example mutation family
A001	Linear search / condition reasoning	== → !=
A002	Counting / equality reasoning	== → !=
A003	Factorial / arithmetic reasoning	+ → −
A004	Factorial / initialization reasoning	1 → 0
A005	Even-number classification	0 → 1
A006	Positive-number classification	0 → 1
A007	Negative-number classification	1 → 0

Table 2. Synthetic pilot summary.

Metric	Value
Synthetic profiles	20
Response records	140
Beginner / Intermediate / Strong	7 / 7 / 6
Mean score	25.55 / 35 (73.0%)
Median score	22.50 / 35
Standard deviation	6.14
Minimum / maximum	16 / 35
Q1 / Q2 / Q3 accuracy	80.7% / 74.3% / 80.0%
Mean Q4 score	9.10 / 14 (65.0%)

8. Discussion

What the pilot mainly demonstrates is that the pipeline runs end to end — item generation through summary statistics — without a human touching it in between. Pairing an original program with a modified one gives every item a known cause for its behavioral difference, and the four-question structure turns that single cause into several distinct, gradually more interpretive measurements.

Q2's lower accuracy relative to Q1 is worth pointing out for a specific reason: it argues against collapsing the two output-prediction questions into a single score. Here the gap comes from how the synthetic profiles were built, not from anything about real participants, but it shows the pipeline is capable of catching a difference like this if one shows up in real data.

The Q4 rubric exists to keep two things apart that are easy to blur together — noticing that behavior changed, and correctly explaining why it changed. A participant can answer Q3 correctly while still missing the actual mechanism, and the two-point scale is built to catch exactly that gap rather than reward a vague sense that "something is different."

None of the individual numbers above are really the point. What matters is that generation, validation, scoring, and analysis all ran correctly against a labeled synthetic dataset, with synthetic and real data kept apart the entire time.

9. Threats to Validity

Construct validity. Output prediction captures only part of what it means to understand code. Behavior-recognition and explanation questions widen that net, but seven items built around a handful of control-flow and arithmetic patterns still fall well short of the full range of comprehension a working developer needs.

Internal validity. A fixed input and a controlled mutation give tight control over what changes between the two programs in an item. That control does not extend to the pilot data itself: the synthetic profiles were generated, not observed, so any differences between them describe the generator, not real cognition.

External validity. Every result in Section 7 comes from a synthetic dataset, so none of it generalizes to students, professional developers, or any other population — there is no human sample behind these numbers to generalize from in the first place.

Conclusion validity. The averages and percentages in Section 7 are descriptive statistics about the synthetic pilot, full stop. Inferential claims, reliability estimates, or anything about a wider population have to wait for real participant data.

Data-integrity validity. Automated checks catch duplicate records, missing answers, cross-contaminated participant data, invalid item references, and mislabeled synthetic records. They guarantee the data-handling is clean; they say nothing about whether the assessment itself is a good measure of code understanding, which remains open.

10. Reproducibility and Research Transparency

Two datasets sit behind this project, kept apart on purpose: an active research file, empty as of this writing, and a synthetic pilot file holding the 140 records discussed above. The full package — assessment items, scoring rubric, processing scripts, protocol document, synthetic outputs, and validation scripts — travels together, so reproducing the pipeline does not require piecing it back together from scratch.

The separation exists for a simple reason: it lets someone else inspect the pipeline and the synthetic pilot without any chance of mistaking a generated record for something a real person actually said.

11. Limitations and Future Work

No human has taken this assessment yet, and that is the limitation that matters most. Everything in this paper describes a framework and a technical pilot — not proof that the instrument measures code understanding well once real people are involved.

The obvious next step is recruiting participants under whatever institutional and consent process applies, running the finished instrument on them, and analyzing item difficulty, discrimination, reliability, and how Q1 through Q4 relate to each other. The operator set and program list used here are small enough to extend later without much additional design work.

A comparison worth running eventually is this instrument against a conventional output-only quiz, to see whether the added behavior and explanation questions actually catch comprehension gaps that a simple output check would miss.

12. Conclusion

This paper set out a mutation-based framework for assessing code understanding through controlled changes in behavior. It departs from ordinary mutation testing in where the mutation is aimed: at a person rather than a test suite, with that person's reasoning about the mutant as the thing being measured.

The instrument itself: seven items, four questions each, a 35-point scale, an explicit rubric for the explanation question, and a pipeline that validates, scores, and summarizes automatically. A synthetic pilot of 20 profiles and 140 responses passed every structural check it was given and moved cleanly through the full pipeline.

Read the pilot's numbers as a description of the pilot, not of real learners. The contribution here is the instrument and the pipeline behind it — both ready now for a study with actual participants, after which reliability, validity, and practical usefulness can finally be tested properly.

Research transparency note. At the time of writing, the active research dataset holds zero participant records. All 20 profiles and 140 responses discussed in Section 7 are synthetic, generated only to validate the pipeline; none of them is presented as, or should be read as, a real participant's response.

References

[1] Y. Jia and M. Harman, "An Analysis and Survey of the Development of Mutation Testing," IEEE Transactions on Software Engineering, vol. 37, no. 5, pp. 649–678, 2011, doi: 10.1109/TSE.2010.62.

[2] M. Papadakis, M. Kintis, J. Zhang, Y. Jia, Y. Le Traon, and M. Harman, "Mutation Testing Advances: An Analysis and Survey," Advances in Computers, vol. 112, pp. 275–378, 2019, doi: 10.1016/bs.adcom.2018.03.015.

[3] S. Prasad, B. Greenman, T. Nelson, and S. Krishnamurthi, "Conceptual Mutation Testing for Student Programming Misconceptions," The Art, Science, and Engineering of Programming, vol. 8, no. 2, Article 7, 2024, doi: 10.22152/programming-journal.org/2024/8/7.

[4] C. S. Yu, C. Treude, and M. Aniche, "Comprehending Test Code: An Empirical Study," in Proc. 35th IEEE International Conference on Software Maintenance and Evolution (ICSME), 2019, pp. 501–512, doi: 10.1109/ICSME.2019.00084.