The Necromancer: An Adversarial Audit Swarm for Vibe-Coded Web Applications

The Necromancer: An Adversarial Audit Swarm for Vibe-Coded Web Applications

Author: Vijay License: CC-BY 4.0

---

Abstract

A large share of web applications built today are assembled quickly with the help of AI coding assistants — a practice now commonly called "vibe coding." These applications tend to work under normal conditions but are rarely stress-tested against malformed input, dependency risk, or unfair treatment of different user groups, because the person who "wrote" the code often never read most of it. This paper introduces Necromancer, a multi-agent adversarial audit tool built to catch exactly these blind spots in Flask and FastAPI applications. Necromancer runs three independent agents — a fuzzer that hunts for unhandled crashes, a dependency auditor that flags risky or typosquatted packages, and a bias auditor that measures disparate treatment across demographic proxies using Disparate Impact Ratio and Demographic Parity Difference — and merges their output into a single report with a blame map tying each finding back to the offending endpoint or file. We validate Necromancer against three targets of increasing unfamiliarity, including a live third-party FastAPI + SQLite Todo API it had never seen during development, where it surfaced two reproducible high-severity crash paths. That same validation run also exposed a precision bug in our own dependency auditor — it was flagging the well-known black formatter as a typosquat of flask — which we diagnosed, fixed by switching to a Damerau-Levenshtein edit-distance check with a stricter threshold, and confirmed with a regression test. We treat that as a feature of the methodology, not an embarrassment: real external validation is supposed to break your assumptions, and a tool that only works on the examples you built it against isn't actually validated.

---

1. Introduction

Vibe coding — building software largely by prompting an LLM and accepting what it produces — has gone from a niche joke to a real way software gets shipped, especially for small internal tools, hackathon projects, and MVPs. The upside is obvious: a working CRUD API or dashboard that used to take a day now takes twenty minutes. The downside is less discussed but arguably more dangerous. When a human writes code line by line, mistakes tend to cluster around genuinely hard problems. When an LLM writes code and a human accepts it with a quick skim, mistakes cluster around things nobody looked at closely at all — malformed input handling, dependency hygiene, and whether the app quietly treats certain users worse than others.

None of these failure modes are new. What's new is the scale at which they're being introduced, and the fact that the person shipping the code frequently isn't equipped to catch them, because they didn't write the logic they're reviewing. Traditional QA and security tooling assumes a developer who understands their own codebase well enough to know where to look. Vibe-coded apps break that assumption by design.

Necromancer is our attempt at a tool built specifically for this gap. Rather than one general-purpose scanner, it runs three specialized agents in parallel, each targeting a distinct failure class, and each producing findings that trace back to a specific route, file, or field. The name is a bit of a joke and a bit literal — it digs through code nobody wants to look at closely and drags the bugs out into the light.

This paper covers the system's design, the three agents and how they reach a verdict, the configuration-generation layer that lets Necromancer point itself at an arbitrary Flask or FastAPI target without manual setup, and a validation pass against real third-party repositories that neither the tool nor its author had built.

---

2. Related Work

Fuzzing tools like AFL and its many descendants have long been used to find crashes in compiled software by mutating input and watching for failures; applying the same instinct to a web API's request bodies is a natural extension, though most existing API fuzzers (e.g. RESTler) are built for large-scale enterprise APIs with formal OpenAPI specs already in hand, not for the kind of loosely structured, spec-optional endpoints a vibe-coded app tends to produce.

Dependency auditing has its own established lineage — tools like pip-audit and GitHub's Dependabot check installed packages against known vulnerability databases. Typosquatting detection specifically is a smaller and less mature area; most guidance is folded into broader supply-chain security advice rather than shipped as a standalone, tunable heuristic. Our own experience building this piece (detailed in Section 6) suggests that's for a good reason — naive edit-distance thresholds produce false positives against completely legitimate packages, and getting the threshold right matters more than it first appears.

On the typosquatting side specifically, academic treatment is thinner than the security-blog coverage might suggest, but it exists. A 2020 study of typosquatting and combosquatting attacks on the Python ecosystem examined real historical attacks and found that the large majority of misspelling-based attacks (as opposed to more sophisticated "confusion" attacks using unrelated but similar-sounding names) sat at an edit distance of one or two from the package they were impersonating. That finding is a useful anchor for tool design: it means a conservative, low-threshold edit-distance check — the direction we ultimately moved Paranoid toward, described in Section 6 — is defensible as a design choice on its own merits, not just as a bug patch.

Bias auditing in machine learning has a much deeper academic footing, with metrics like Disparate Impact Ratio and Demographic Parity Difference well established in the fairness literature (e.g. the AIF360 toolkit from IBM). What's less common is applying those metrics not to a trained model's predictions, but to a web application's behavior directly — treating the app itself as a black box and perturbing demographic-adjacent fields in requests to see whether outcomes shift. That's the angle our Bias Auditor takes.

To our knowledge, no existing tool combines all three of these concerns — crash robustness, dependency risk, and behavioral fairness — into a single audit pass aimed specifically at rapidly-assembled, LLM-generated web applications.

---

3. System Architecture

Necromancer is a command-line tool. You point it at a target directory and it runs:

python -m necromancer audit <target>

Under the hood this triggers three agents that run more or less independently and report back into a shared aggregator, followed by a full walkthrough of what each agent actually does and how its verdicts are formed.

3.1 Saboteur (the fuzzer)

Saboteur works off the route and field map that ConfigGen produces (Section 3.4). For every field it identifies on every route, it doesn't just throw one bad value at it — it works through a set of adversarial categories meant to mirror the kinds of input a real user, or a real attacker, eventually sends by accident or on purpose:

• Type confusion — sending a string where an integer is expected, a nested object where a scalar is expected, and so on.
• Boundary and overflow values — empty strings, extremely long strings, negative numbers where only positive ones make sense, zero where division or indexing might occur downstream.
• Missing and null fields — omitting a required field entirely, or sending null explicitly, since these are handled by very different code paths in most frameworks and often only one of the two gets tested by hand.
• Malformed structure — broken JSON, mismatched content types, and fields that are present but structurally wrong (an array where an object was expected).

Each payload is fired at the live endpoint and the response is inspected. A clean 4xx response with a sensible error body is treated as correct behavior and produces no finding — Saboteur isn't trying to punish an app for rejecting bad input, only for failing to handle it gracefully. An unhandled exception, a 500-level response, or a stack trace leaking into the response body is what generates a finding. Severity is assigned based on how "natural" the payload is: a missing field or a wrong type is something a real client could plausibly send by accident, so failures there get tagged high; failures that only appear under deliberately hostile structural corruption get tagged lower, since they represent a real bug but a less immediately exploitable one.

3.2 Paranoid (the dependency auditor)

Paranoid parses whatever dependency manifest the target uses — requirements.txt, pyproject.toml, or equivalent — and extracts the declared package names. Each name is normalized (lowercased, underscores and hyphens treated as equivalent) and compared against a curated list of common, high-traffic Python packages. If a declared dependency is suspiciously close to one of these well-known names but isn't an exact match, Paranoid flags it as a potential typosquat — the kind of attack where a malicious actor publishes a package with a name one keystroke away from something popular, hoping someone installs it by mistake. The design assumption, discussed further in Section 6, is that the closeness check needs to be conservative: too loose a threshold and the tool starts flagging entirely legitimate packages that simply happen to share letters with something popular.

3.3 Bias Auditor

Bias Auditor works differently from the other two agents because it isn't looking for a crash, it's looking for a pattern across many requests. It identifies fields in a route's input schema that plausibly act as demographic proxies — names that suggest ethnicity or gender, addresses that correlate with income or region, and similar — and constructs matched pairs of requests that are identical except for that one field. Each pair is sent through the same endpoint, and the outcomes are compared in aggregate across many such pairs.

From those aggregated outcomes, Bias Auditor computes two standard fairness metrics: Disparate Impact Ratio, the ratio of favorable-outcome rates between the two groups being compared, and Demographic Parity Difference, the raw difference between those rates. A DIR far from 1.0 or a DPD far from 0 across enough samples indicates the endpoint is behaving differently for the two groups in a way that isn't explained by anything else in the request — which is flagged as a finding. Just as important, when an endpoint has no field that plausibly proxies for a protected attribute, Bias Auditor correctly produces nothing, rather than forcing a result. Section 6 covers a case where that null result was itself informative.

3.4 Aggregation and the blame map

All three agents report into a shared schema — regardless of which agent produced a finding, it carries the same severity levels, the same evidence structure, and a route or file reference. The aggregator then builds a blame map: a dictionary keyed by route or filename, with every relevant finding ID attached underneath it. This is a small piece of the system but it's the one that makes the report actually actionable — instead of a flat list of twelve findings a developer has to cross-reference by hand, the report says, explicitly, "these two findings both belong to PUT /todos/{todo_id}," so the fix effort is scoped correctly from the start.

Reports are saved as timestamped JSON for reproducibility, with a latest_report.json convenience copy kept alongside for quick access.

3.5 ConfigGen

One practical obstacle to auditing an arbitrary third-party repo is that Necromancer needs to know what routes exist and what shape their inputs take before it can generate meaningful adversarial payloads. Manually writing that configuration for every new target defeats the point of the tool. configgen.py solves this by inspecting the target itself: for Flask apps it walks the AST of the route files directly, and for FastAPI apps — which expose a live OpenAPI schema — it queries that schema at runtime instead. Both paths converge on the same internal config format the other three agents consume.

Getting the Flask AST path right took some iteration. Flask route handlers commonly call request.get_json() and immediately use the result in a boolean context — if request.get_json(): — which the AST walker was initially misreading, because Python wraps that pattern in a BoolOp node that isn't a simple function call. Naively walking the tree missed these calls entirely, meaning some routes were audited with an incomplete picture of their inputs. Once we recognized the pattern needed to be unwrapped from its BoolOp context specifically, the extraction became reliable.

---

4. Threat Model and Scope

It's worth being explicit about what Necromancer is not trying to do, because "adversarial audit tool" is a broad enough label to invite the wrong expectations. Necromancer is not a general-purpose static application security testing (SAST) tool — it doesn't do full data-flow analysis, doesn't catch injection vulnerabilities that depend on how a value flows through several functions before reaching a database call, and doesn't replace a proper security review for anything handling sensitive data at scale. It also doesn't test authentication or session handling; every request Saboteur and Bias Auditor send assumes whatever access the target already grants, rather than trying to bypass access controls.

What Necromancer does target is narrower and, we'd argue, more directly matched to how vibe-coded apps actually fail: input handling that was never stress-tested because the person accepting the LLM's code never thought to try it, dependency lists that were assembled by copy-pasting an LLM's suggested requirements.txt without a second look, and fairness properties that nobody checked because nobody was thinking about fairness at all while prompting for a CRUD endpoint. It's meant to run in the gap between "no testing at all" and "a full professional security and fairness audit," which is exactly the gap most vibe-coded projects currently sit in.

5. Methodology

We built Necromancer through six phases. The first several focused on getting each agent correct in isolation against a controlled target — an app we wrote ourselves, called QuickApprove, where we knew exactly what bugs, dependency risks, and fairness issues we'd planted, so we had ground truth to check our agents' output against. Later phases built out the reporting layer and, eventually, configgen.py so the tool could point at code it had never seen.

The real test of any audit tool, though, isn't how it performs against a target you designed around it — it's how it performs against code you didn't write and didn't tune the tool for. That's what Phase 6 is: pointing Necromancer at genuine third-party repositories and seeing what happens without any hand-holding.

We selected targets of increasing unfamiliarity:

1. QuickApprove — our own controlled target, used as a sanity check that the full pipeline still behaves as expected after all the Phase 1-5 changes.
2. Flask Calculator API — an external, simple Flask app, used to check that ConfigGen's AST-based extraction generalizes past our own codebase.
3. FastAPI Todo API — an external FastAPI + SQLite app with a live OpenAPI schema, used to check the FastAPI extraction path and to see whether the full three-agent audit produces sensible, actionable findings on code with no relationship to anything we'd built before.

We ran the full python -m necromancer audit <target> pipeline against each with no manual configuration beyond pointing ConfigGen at the target directory, and reviewed every finding by hand to check whether it was a genuine issue or a false positive.

QuickApprove deserves a bit more explanation, since it's the baseline the other two targets are compared against. It's an application we built ourselves with a known set of issues seeded across all three failure categories — routes that crash on malformed input, a dependency list containing at least one deliberately suspicious package name, and endpoints with demographic-proxy fields where outcomes were made to differ across groups. Running Necromancer against QuickApprove isn't a test of generalization the way the other two targets are; it's a regression check that all three agents are still doing their job correctly after every round of changes to the codebase, and that the aggregation and blame-map logic haven't silently broken. The 12 findings recorded in Table 1 reflect that full pipeline run at the point we moved on to external validation.

---

6. Results

Target	Purpose	Result
QuickApprove	Controlled ground-truth check	12 findings across all three agents
Flask Calculator API	External robustness check (AST path)	4 Saboteur findings
FastAPI Todo API	External ConfigGen + full audit (OpenAPI path)	2 Saboteur findings, 0 Paranoid, 0 Bias Auditor (post-fix)

The FastAPI Todo API run is worth walking through in detail because it's the cleanest external validation we have, and because it's where we found a real bug in our own tooling.

The first run completed in just over 11 seconds and returned three findings: two high-severity Saboteur findings, both tracing back to PUT /todos/{todo_id} — the endpoint crashes with an unhandled exception when the title or completed fields are malformed rather than returning a clean 4xx response. Both are reproducible, and both are the kind of bug that's easy to miss in a quick manual test because nobody usually tries to break their own PUT endpoint with garbage input before shipping it.

The Bias Auditor correctly returned zero findings, which matters as much as a true positive would. The Todo API has no fields that plausibly proxy for a protected attribute, so the correct behavior is silence, not a forced finding. A bias auditor that always finds something isn't measuring anything.

Paranoid, on the other hand, returned one finding that turned out to be wrong: it flagged the black package (a widely used, entirely legitimate Python code formatter) as a likely typosquat, apparently because its edit-distance heuristic was comparing it against flask and finding them close enough to trip the threshold. This is exactly the kind of failure external validation is supposed to surface — our own test suite never would have caught it, because we never tested against a requirements.txt that happened to include black.

We fixed it by replacing the plain Levenshtein distance calculation with a Damerau-Levenshtein variant, which additionally treats adjacent-character transpositions (like reqeusts → requests) as a single edit rather than two, and by tightening the threshold so only genuine one-edit typos get flagged. We added a regression test asserting that a requirements.txt containing black==24.0.0 produces zero Paranoid findings, and confirmed the full test suite — 36 tests — passed. Re-running the audit against the same Todo API target afterward produced the expected result: the two genuine Saboteur crashes remained, and the Paranoid false positive was gone.

We consider this the strongest piece of evidence in the whole validation pass, not despite the fact that it started as a bug, but because of it. A tool that performs flawlessly the first time it meets code it wasn't built around is a tool that probably wasn't tested against anything hard enough to matter.

6.1 A worked example: the post-fix report

To make the report format concrete rather than describing it only in the abstract, here is the terminal summary Necromancer produced on the second, post-fix run against the FastAPI Todo API:

NECROMANCER REPORT — target: targets\fastapi_todo\...\todo-api
run_id: ec847a63  |  duration: 10.796s
[saboteur] status: ok — 2 findings — {'low': 0, 'medium': 0, 'high': 2, 'critical': 0}
(high) Unhandled crash when 'title' is malformed (/todos/{todo_id})
(high) Unhandled crash when 'completed' is malformed (/todos/{todo_id})
[paranoid] status: ok — 0 findings — {'low': 0, 'medium': 0, 'high': 0, 'critical': 0}
[bias_auditor] status: ok — 0 findings — {'low': 0, 'medium': 0, 'high': 0, 'critical': 0}
Total findings: 2
Blame map: {
  "PUT /todos/{todo_id}": ["sab-001", "sab-002"]
}

Two things stand out here beyond the raw numbers. First, the blame map collapses both Saboteur findings under a single route rather than leaving a developer to notice by hand that sab-001 and sab-002 are related — a small piece of aggregation logic, but one that turns "here are two findings" into "here is one endpoint you need to fix." Second, the report is silent where silence is correct: Paranoid and Bias Auditor both report zero findings with an explicit status: ok, rather than omitting themselves from the output, so a reader can distinguish "this agent ran and found nothing" from "this agent didn't run." That distinction matters for trusting a zero-finding result — an agent that might have silently failed and an agent that genuinely found nothing look identical unless the report says otherwise.

---

7. Discussion and Limitations

A few honest caveats. First, our sample of external targets is small — three repositories, not thirty. The results are consistent and the bug we caught is real, but we wouldn't claim statistical confidence about false-positive or false-negative rates at this scale. A larger validation pass across a broader set of third-party repos, ideally spanning different frameworks and coding styles, would be the natural next step.

Second, the Bias Auditor's zero-finding result on the Todo API is correct but also somewhat unsurprising — a todo list app genuinely has almost no attack surface for demographic bias. A more interesting future test would be running the Bias Auditor against a target that plausibly does have sensitive proxy fields (a hiring app, a loan-approval mock, a healthcare intake form) to see whether it catches something real rather than confirming an absence.

Third, ConfigGen's two extraction paths — AST-based for Flask, OpenAPI-based for FastAPI — cover the two most common Python web frameworks but leave out others (Django, for instance) entirely. Extending ConfigGen to more frameworks is mostly an engineering problem rather than a conceptual one, but it's real work that hasn't been done yet.

Finally, we deliberately chose to stay CLI-first and skip building a web dashboard, on the reasoning that a local report viewer changes the presentation of the tool but not its actual audit capability. That was the right call for keeping the project focused during this build, but a well-designed report browser would probably make Necromancer easier to adopt for teams auditing many targets over time.

---

8. Conclusion

Vibe coding is not going away, and the gap it creates — code shipped by people who didn't closely read what they shipped — is a real and growing surface for exactly the kind of failures Necromancer is built to catch: crashes on malformed input, risky dependencies, and unfair treatment baked into request handling nobody scrutinized. A three-agent adversarial swarm, run automatically against a target with no manual setup required, is a workable way to close some of that gap.

More importantly, we think the process matters as much as the result. Necromancer wasn't validated by running it against code we wrote for it to succeed against — it was validated by pointing it at code we had no hand in, and letting it fail in a way that taught us something. It found real crashes in an app we'd never seen. It also found a bug in itself, which we fixed and verified. Both outcomes are what a genuine audit pass is supposed to produce.

Future work includes broadening the third-party validation set, extending ConfigGen to additional frameworks, and testing the Bias Auditor against targets where fairness concerns are more directly present in the data model.

---

References

1. Zalewski, M. American Fuzzy Lop (AFL). https://lcamtuf.coredump.cx/afl/
2. Atlidakis, V., Godefroid, P., Polishchuk, M. RESTler: Stateful REST API Fuzzing. ICSE 2019.
3. Python Packaging Authority. pip-audit. https://github.com/pypa/pip-audit
4. Bellamy, R. K. E. et al. AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias. IBM Journal of Research and Development, 2019.
5. Vu, D.-L., Pashchenko, I., Massacci, F., Plate, H., Sabetta, A. Typosquatting and Combosquatting Attacks on the Python Ecosystem. IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), 2020, pp. 509–514.
6. Nakamoto, S. Bitcoin: A Peer-to-Peer Electronic Cash System. 2008. https://bitcoin.org/bitcoin.pdf