SQS: A Multi-Dimensional Framework for Evaluating Software Specifications in AI-Assisted Development

1. Introduction

Modern AI-assisted software development does not move directly from specification to code. A more accurate model of the pipeline is:

Specification → clarification → planning/task decomposition → context retrieval → implementation → testing → verification → iteration.

An AI coding agent in this paper is an AI-powered software-development agent, which can interpret specifications and make implementation actions by code generation and development tools. In theory, a competent AI coding assistant can ask clarifying questions at the initial stage, instead of assuming. In practice, this can be accomplished with the agentic code-producing tools, depending on the product and the workflow: Interactive clarification is possible with some agentic coding tools, and low-interaction completion is possible with others, where pausing to ask is either not possible or is not encouraged by the design of the workflow. In workflows of the second sort, where the specification does not contain a specific answer for an issue that the agent must address, the agent substitutes his own "inferred" answer for the missing one. We define an implementation decision about behavior, constraints, interfaces or acceptance conditions that is not specified but is required to implement the specification as a specification-induced assumption. We would suggest the following decision rule to differentiate it from an ordinary implementation choice, which is a decision that a specification intentionally leaves open because several different choices would equally satisfy what the specification specifies. A decision is an assumption that is made in the specification. If a change in the decision would cause the implementation to fail to meet at least one explicit requirement, acceptance criterion, interface constraint, or stated behavioral expectation in the specification, then the decision is an assumption induced by the specification. Otherwise, it is a decision of the implementer. According to this rule, the language of the frontend (React or Vue) is usually up to the implementer since no specification restricts the tech stack and both technologies exhibit the same specified behavior. In contrast, if the specification requires file uploads to "must succeed" but does not explicitly describe the agent's action if the file is invalid, then the two options provide materially different, and likely requirement-violating, behavior (either the agent rejects the file outright, or the agent silently accepts it and discards it). Under this rule, we mark one case as truly ambiguous: If the specification does not specify anything that makes size relevant, e.g., a statement that it must be stored in some specified time or delayed for some time or that it must be read from a given tape, or that the file must be in a certain size, the maximum size is more like an implementation detail. This does not make the rule less operational than an intuitive judgment, but it is indicative of the fact that it is still not completely mechanical and must be applied carefully. It is important to distinguish these two classes because a future quantitative study that counts "things that the agent decided that weren't in the spec" would lump "real" specification failures with engineering latitude. This distinction is the part of the upstream side of which this paper is interested in measuring.

The hypothesis is not just "better specs = better code": it's more specific, and stated in hypothesized, rather than established, language: we hypothesize that the quality of the specification may decrease the number of assumptions that the agent must make about it, which in turn may decrease the number of deviations in the implementation from what the specification says. This paper suggests a model to measure the first of those links, namely the quality of the specification. It does not test the chain, and nothing in this paper should be read to assert the chain is established. Let's make this clear: SQS is a measure of the quality of a specification as an input artifact for an AI coding agent's implementation process. It doesn't measure the quality of the software created, the effectiveness of the overall software development process, or the quality of the agent's reasoning — it should not be interpreted to mean "longer the better" (meaning that every line of information in a specification is sufficient to fully constrain the behavior it covers) or "more the more" (meaning that the information density in a specification is identical to the quality of the software created). A good requirement, when given to a poor pipeline, can result in bad software, but a good pipeline with a clarification step can partially offset the shortcomings of a mediocre requirement. SQS is used to represent a specification's profile (strengths and weaknesses) and not as a binary readiness gate in a larger system. This model is summarised in Figure 1. This paper proposes and defines solid boxes and dashed lines, and "hypothesized" labels are used to illustrate the relationships that this paper does not test and proposes as future work (Section 7).

Figure 1. SQS conceptual model. This paper proposes solid boxes; the dashed arrow and hypothesis labels (H1–H3) mark relationships this paper does not test.

This paper makes three contributions:

• Adaptation of the 5-dimensional, rubric-anchored operationalization of specification quality (Section 4) to develop explicit scoring criteria for each band of each dimension of specification quality that an AI coding agent needs to receive from the specification.
• An illustrated example of the framework on excerpts from the specification of varying quality should include an explicit explanation of the importance of the sub-scores (not just the overall score) (Section 5).
• An honest description of what this framework sets up and clearly and falsifiably states is absent, which certainly includes a concrete experimental validation design, about which the present paper does not provide any in itself (Section 7).

2. Problem Statement

How can quality be measured in a software specification on attributes that are plausible and can reasonably be used to predict the number of specification-induced assumptions that an AI coding agent might need to make, rather than attributes based on human-readability alone?

Human-readable and machine-readable is not a property of the same thing. An experienced engineer reading a specification fills gaps in the document automatically, based on his knowledge, which is tacit. While some tools enable an AI coding agent to access its repository, retrieval capabilities, or even the agent's own memory, an AI agent deploying in an environment may lack the same amount of “tacit organizational knowledge, stakeholders context, or institutional assumptions” that a human designer might have, even with access to the repository. In cases where that mitigation can't be accomplished or isn't sufficient, the same lack of meaning to a human reader may be a significant source of drift to an agent.

3. Related Work

Ambiguity detection: The specific form of ambiguity identified here, referred to as "cross-domain term ambiguity" because the same term may have different definitions in the documents used by different groups, can be found using a computational method that compares the use of words in different language models created for the groups' respective documents, as demonstrated in Ferrari and Esuli (2019). The first kind of ambiguity is not pragmatic ambiguity, which takes place when a requirement is linguistically unambiguous in one domain, yet can be interpreted a different way in another, context-dependent domain, and Nair et al. (2026) propose a retrieval-augmented framework that focuses exclusively on this pragmatic kind of ambiguity. In the first, ambiguity is something which needs to be detected independently, in the second, it is treated as part of its own detection pipeline, and neither contends with completeness, consistency, or document structure – it is anomaly detection these two views target.

Duplication and detection of plagiarism:In their study, Saleem et al. (2024) test 760 various pipelines for detecting duplicate or conflicting requirements from six public benchmark datasets to find the pipelines that can perform best: "Hybrid pipelines, which utilize similarity-based knowledge in conjunction with an LLM's context, are consistently superior to either standalone LLM-based or similarity-knowledge-based approaches. Fantechi et al. (2023) perform an initial assessment about directly applying ChatGPT to detect inconsistency in natural-language requirements. More recent efforts have begun to reconcile the notion of conflict detection as having a specific focus on multi-agent LLM systems, in particular "semantic intent divergence" — the case where AI agents working together interpret the same goal differently because they have different context knowledge, which was identified by Acharya (2026) as the resulting of a novel failure mode that was formalized and for which there are yet no known solutions for in enterprise multi-agent applications. Each of these, while a kind of detection method, is not a scored dimension that is part of a composite specification-quality method, unlike what a future extension of SQS's Internal Consistency dimension into a full conflict-detection method would require to build upon.

Quick and accurate code generation: To measure the influence of the specific prompt ordering, they design a partial order between the prompts in a code generation test suite, ranging from minimal to maximally detailed, and try it out on ParEval, HumanEval, and both serial and OpenMP subsets of this test suite; they report overall trends of increasing pass@1 scores with increasing prompt specificity, while pointing out that it was explicit I/O specifications, handling edge cases, and doing it in steps, that the pure increase was determined to be. A question that ties in with this one, in a slightly different way, is how much generated code is influenced by changes in the prompt encoding, specifically how sensitive the generated code is to changes in the prompt wording, which Paleyes et al. (2025) study with a synthetic evaluation pipeline as well as with a persona-based evaluation approach, finding that code similarity drops quickly when the coding is created by perturbing the prompt wording in typo style but is more stable when the prompt wording is paraphrased and synonyms substituted. Taken together, these two papers provide the best available evidence that the quality of specifications (definitionally, prompt specificity and robustness) has an impact on code-generation results. Both, however, deal with one-shot coding-benchmark questions, and neither suggests a general, reusable rulebook for the quality of a coding-benchmark specification.

Specification-driven development: While there might not be a single specific name for such a methodology, Piskala (2026) refers to it as specification-driven development (SDD). Specifications, and not code, are the main artifact; code is a generated and/or verified secondary artifact; and it shows several use cases for SDD for API development, enterprise systems, and embedded software programs, including a mapping between SDD and tooling like GitHub's Spec Kit. If there is a proper reason why the quality of specifications is more important after the implementation is automated by an AI agent, then there isn't a way to tell whether a given specification is actually up to that task. SQS is an argument for the downstream part of the pipeline: With SDD we have identified the specification as the new primary artefact of the pipeline; SQS is one suggestion for doing the checks before relying on the specification.

Positioning: The concept of composite (multi-dimensional) requirements-quality frameworks is not new in the general field of requirements engineering research. What is relatively unaddressed, however, is an operationalization of those dimensions that has a narrow focus on what an AI coding agent needs, with anchors tailored for each band and not open-ended, human judgement-based at the extremes. We are not suggesting that the concepts of completeness, ambiguity, consistency, and traceability are new; none of them are new, of course: completeness, ambiguity, consistency, and traceability have long and established histories in requirements engineering as cited above. We go another way narrower than Opalka's opener, and we add another dimension, "Machine-Actionability," not found in the traditional literature of the art of requirements quality in a human-agent-oriented setting, into one AI-agent-oriented specification-quality profile that has explicit scoring criteria tailored for the decisions an AI coding agent must make when implementing a specification. It is the combination and operationalization of this, not the general concept of multi-dimensional quality scoring, that is the contribution of this paper.

3.1 Why these five dimensions

SQS did not pick its 5 dimensions at random, but rather the choice is a design decision – it is not necessarily the best choice – and the table below is not intended to be the only correct choice for 5 dimensions. Each dimension is inserted to align with the corresponding familiar requirements-quality issue among the concerns being different in the case of human readers as opposed to AI coding agents:

Established RE concern	SQS dimension	Why it is adapted, specifically, for an AI coding agent
Completeness	Completeness	A human engineer might go to a (unrecorded) project or organization knowledge, and fill a gap; a human agent working with bounded context is more apt to fill in a gap with an unconstrained guess.
Ambiguity	Unambiguity	When faced with ambiguity, a human engineer might simply need to stroll over to his fellow and ask, an autonomous or low-interaction agent is more likely to resolve its fuzzy state, at least, by silent inference (Section 1).
Consistency	Internal Consistency	A human reader can usually be relied on to detect contradictions with a broader understanding of the system, whereas a system reasoning mainly based on the text will be much more susceptible to cases in which there is an unresolved contradiction.
Traceability	Intent Traceability	We include rationale because it plausibly contributes to a reasonable agent decision making in the many places where a specification is insufficient to completely specify the trade-offs; this link is weaker than the other four, but note that we are explicit in Section 4.7 that this is so.
(No direct RE equivalent — closest is document structure/formality conventions)	Machine-Actionability	Whilst a human reader can easily navigate text and mentally reorganize the structure if needed, the extent to which an agent can successfully extract a constraint from the text is greater if the structure is explicit and regular.

Other dimensions were considered, but were ruled out as being either so closely related to the ones above as we defined them (specificity overlaps strongly with Unambiguity) or as belonging to the question of whether the underlying engineering problem is likely to be solvable by an AI coding agent (verifiability, feasibility, etc. are both properties of the requirement's content independent of how well it can be served by an AI coding agent). This exclusion is made on a case-by-case basis, as stated here, and a different set of dimensions is not ruled out by this paper.

4. The SQS Framework

4.1 Scope and positioning

SQS is assessed on a specification by itself. It checks internal consistency, that the requirements in the document are compatible; it does not check external consistency with other documents (such as architecture documents), with existing code, API contracts or organisational standards for which access is required outside the document. It also has a defined specification scope; it is not compared against a domain-different, absolute checklist of completeness, but to what the specification says it covers. Also, a front end UI specification is not required to be complete, nor is a medical-device specification required to describe the same set of fields. SQS also isn't designed to compare two different scope types without accounting for the differences: for example, for a medical device we might be required to describe its endpoints, request/repfield formats, mechanisms for authentication, and error states – fields that the frontend UI specification does not necessarily need to, but this paper does not formally define and consequently suggests that it would be useful to define a different schema for each scope type for these types of comparisons though.

The five dimensions below are considered in isolation, each measured on a separate rubric, and statistically independent of each other, although in reality a single underlying problem (e.g. some requirement that is poorly-scoped) can cause all three of the dimensions to decline. The reader is not vexed by the legitimate confusion that can arise with additive scoring, as described in Section 7. SQS thinks best of well-formulated constraints and recoverable intent, it does not care about the length of documents or the amount of information. It is possible to have a short specification that narrowly and clearly defines everything it is supposed to define that scores as high as the longer version with all the re-stated or redundant detail that does not necessarily score higher on any of the other considerations below.

4.2 Formal definition

$$SQS = C + U + IC + IT + MA$$

where C, U, IC, IT, MA ∈ [0, 20], so SQS ∈ [0, 100]

The five metrics used for the measures described above are the following: where C = Completeness, U = Unambiguity, IC = Internal Consistency, IT = Intent Traceability, and MA = Machine-Actionability. It is still a sum where each dimension is given the same weight (1), which is a provisional weighting not a validated weighting and is not explained yet; this is explained in Section 4.9; and it cannot be explained from the total alone (no sub-score profile) which is illustrated in Section 5.4

4.3 General scoring anchors

The scores range from 0 to 20 for each dimension against the general dimensions bands outlined below, and dimension-specific criteria are provided for each general dimension bands shown below, in Sections 4.4–4.7. The bands and their number ranges are, first of all, an ordinal rubric, not empirically determined thresholds: a 17 is in a different rank than a 16, but that difference has no meaning downstream of us, only consistency we could describe. It was not the intent of this version to declare or assert that the scoring is psychometrically certain - it is just a system to operationalise the criteria at all bands with no claim to precision at this point; a score of 73 does not plan to "mean" much better than a score of 72, it simply means that it falls into this band. A scorer inside the band should use the bottom end of the band for a case that just meets the criteria of the band, and the top end of the band for one that clearly exceeds the criteria of the band (we don't expect anyone to be able to replicate this finer-grained placement without a significant effort of inter-rater work, outlined in Section 7).

Band	Range	General meaning
Severely deficient	0–4	The dimension is effectively lost: solely through inference an agent would be working for this aspect.
Major deficiencies	5–8	The dimension is dealt with truly only at isolated locations and most of the time it needs to be inferred.
Partially adequate	9–12	It focuses primarily or statistically on size as the dimension for common (or principal) cases, but leaves room for meaning.
Strong with minor gaps	13–16	The dimension is explored adequately, and isolated and minor gaps, which would be easily handled by a careful agent with low risk, are addressed.
Highly robust	17–20	This aspect is dealt with in full and is made explicit; the agent should have to make very little inference about this aspect.

4.4 Completeness (0–20)

Context: The degree of scope of the specification sufficiency for, and unreliability on, making inferences: functional behavior, actors, applicable non-functional constraints, error/edge-case handling, and acceptability criteria suitable to that scope.

Score	Criteria
0–4	Core behavior and scope is very under-defined; the specification doesn't really define anything more than a feature.
5–8	Core behaviour is communicated but there is no reference to key relevant categories (actors, constraints, or acceptance criteria) that are needed for implementation.
9–12	At least one additional behavior relevant to implementation is listed and some significant edge case(s) or secondary requirements aren't covered.
13–16	Most behaviour relevant to the scope is captured as is most constraint and actors and acceptance criteria, the remaining gaps are minor and low impact.
17–20	Coverage is wide compared to the scope that is stated; edge and contra-cases are covered, etc. which would be expected by a reasonable implementer.

4.5 Unambiguity (0–20)

Definition: To sidestep the extremes of unconstrained qualitative language — requirements without defined interpretation and requirements without a statement of tolerance for open requirements, and to not penalise intentionally qualitative requirements with an overly defined, widely-agreed interpretation. Rule 2, that the system should use a human-readable error message, is qualitative but reasonably actionable because the meaning of human-readable error message is widely shared in software practice even if it is not explicitly defined; rule 3, the system should be fast, is unconstrained and does not have a clear shared interpretation of fast without a threshold. Defining what is "broadly shared" is one of the subjectivity elements which depends on the subject-specific expertise of the evaluator. To turn this concept into a real test rather than a mere intuition, the following can be suggested: can two competent developers reasonably program two distinct behaviors and both come to believe that they were complying with this requirement? If it is, the claim is taken as being unconstrained; if it is not, it is considered sufficiently defined either with or without being quantified. It is more reproducible than "broadly shared interpretation" alone, but requires evaluator judgment and relies on the assumption that the test is applied uniformly by different evaluators; such uniformity could be accomplished by using a shared reference lexicon or by measuring the agreement amongst multiple raters and, as this paper does not attempt to do, such agreement would be considered validational (Section 7).

Score	Criteria
0–4	Almost all the material claims contain unqualified, qualitative statements that are not interpreted, and which are not shared in interpretation.
5–8	A great majority of the substantive claims are unrestricted; a few are indexed, or otherwise precisely defined.
9–12	About half of substantive claims are numerical, or there is a definite common understanding of what they're about; otherwise they're unconstrained.
13–16	Most substantive claims are quantified or expressed in a conscious and defensible qualitative manner; a few keep them in a free state.
17–20	All substantive claims are qualified and/or clearly subjectively interpreted, or are expressed deliberately as an unquantifiable claim.

4.6 Internal Consistency (0–20)

Definition: The extent to which none of the requirements of the specification explicitly says the opposite of anything else in the specification. It isn't penalized for having no information here, and if a specification has a very limited amount of information, it's not considered contradictory by default; that will be scored under Completeness. A specification is high on Internal Consistency if everything it says does not contradict anything else in it, even though it does not say much, even though it’s a little or a lot vague (that gets scored under Unambiguity, and not here).

Score	Criteria
0–4	There are a number of explicit conflicts containing requirements.
5–8	There is at least one statement in which the requirements contradict each other.
9–12	Although there is no direct contradiction, there is at least one semantic tension that arises between two stated requirements, which would have to be addressed in order to implement it.
13–16	No direct contradiction and no obvious apparent semantic conflict; perhaps, a redeployment threat can lurk in the background that in the future, a more specific requirement will clash with a currently more general one.
17–20	No tensions were expressed or implied in contradiction to any other expressions.

4.7 Intent Traceability (0–20)

Definition: How well each requirement can be traced back to the originating objective, and/or verifiable acceptance criterion. It also focuses on a narrower than full lifecycle requirements traceability, since it only asks if the need to do something or the reason to do something is recoverable from a specification (the latter does not trace through the design and implementation artifacts, etc. to the requirement for testing itself). We add it to SQS because it is a plausible means of an agent solving under-specified trade-offs, and of understanding (or even not needing to understand) competing requirements; it supports clarification and requirement maintenance over the course of the iterative requirement regeneration process; but, unlike the other four dimensions, its relationship with implementation correctness on first-pass correctness is not as direct, in that an agent can implement one of the requirements correctly without knowing why they were saying they wanted it to be that way. Noting that it remains assessable from the specification but is still plausibly relevant to agent action, however, we flag this weaker and more indirect link honestly as a core dimension, but not as an auxiliary dimension, because all five dimensions are equally referable to code correctness as it is stated in the specification.

Score	Criteria
0–4	There is no statement of the objective or statement of the acceptance criterion that is associated with any requirement in the specification..
5–8	Occasionally there are a few requirements that are associated with an identified objective or acceptance criterion.
9–12	About half of the requirements are associated with a known objective or an observable fulfillment criterion.
13–16	The most of the requirements are also associated with a stated objective or acceptance criterion, while a minority of requirements are not.
17–20	Each requirement will reference a written objective and/or an accepted criterion that is measurable.

4.8 Machine-Actionability (0–20)

Definition: Structured, explicit and predictable information in the specification, revealing information relevant to implementation, on which an AI coding agent can reliably access, consistent naming of fields, explicit data types/format/thresholds, predictable document structure. This is not the question whether an LLM can 'read' the prose - that should be an "easy" question because LLM's can generally do that, but rather the question of whether the LLM can extract implementation-relevant constraints without using its knowledge of the document's structure to make inferences the document has never explicitly made. It is also different from agent capability: capability of implementing the integrity constraint or adapting it to a given instance and situation; whether a machine-actionable capability does exist in fact in the agent, but is not defined in the specification is out of scope for SQS.

There is also an overlap with, but a distinction from, Unambiguity: Unambiguity asks "Is the meaning of this claim semantically clear?"; Machine-Actionability asks "Is the meaning of this claim explicit and to-be-extracted? The requirement can be semantically clear but not easily machine-interpretable, such as "the API must return an error within 2 seconds of requests and must be offered in both English and Dutch". On the other hand, a requirement can have a clear structure and be ambiguous — a field that is outputted as error_timeout: fast can be identified and found but doesn't answer the question of what fast means. A specification which includes explicit thresholds in structured fields will normally be high on both dimensions at the same time and will therefore usually travel together in reality, but they are distinct and it is possible for them to move in opposing directions, as these two illustrate.

Score	Criteria
0–4	No consistent field format, more of a free format free for interpretation by an agent, nearly everything would be assumed.
5–8	Section headings are almost nonexistent, and there is no consistency in the field names and format of the field constraints.
9–12	Some requirements have structure, but it is not applied consistently throughout the document (e.g., table or labelled fields).
13–16	Most constraints have explicit thresholds and formats, and clean structure with consistent naming throughout most of document.
17–20	Consistently apply structure, field naming, thresholds and formats; make sure that implementation-relevant constraints are identified, but don't assume structure to get thresholds/formats.

4.9 Aggregation and its limits

The current equal weighted sum is a preliminary design preference, and not an empirical choice of weighting, as there is no evidence currently which suggests all five dimensions equally contribute to implementation outcomes in the downstream. The bottom line is that a completely additive total can make for a confusing profile: 20/20, 0/20, 0/20, 0/20 means there's a total of 80/100; that's impossible to use as you have a spec breakdown that's totally unusable by an automated pipeline that can't parse it. The total should be understood as a five independent profile (or sub-scores), not as a single readiness indicator for the use of AI. In the next section (the appended 5.4) two specifications are provided as concrete examples that have the same totals but very different profiles. The future should look into learned weights based on the outcome data in the downstream, rather than the average of the two, or threshold based aggregation (for example, total was >= some threshold X AND each dimension was >= some threshold Y). Both are mentioned in further detail in Section 7.

4.10 From construct to measure

The table below makes explicit the bridge between each dimension as a concept and how this paper currently measures it, together with an honest statement of validation status — none of the five is currently validated in the psychometric sense (inter-rater reliability, correlation with outcomes), and this paper does not claim otherwise:

Construct	Operational measure (this paper)	Current validation status
Completeness	Judging of the band elements against the band criteria (Section 4.4) in relation to the statement of scope in the specification	Not validated; there are no schema specific to this yet defined (Section 7)
Unambiguity	Creating two reasonably different kinds of behavior based on the same description while each person thought they fulfilled the description—“could two competent developers reasonably implement materially different behavior while each person thought they fulfilled the description?” judgement against the band criteria in Section 4.5	Not validation: lab of potential use has been proposed but not validated by more than one independent lab.
Internal Consistency	The performance of all the players must be judged in relation to the band criteria in 4.6.	Based on automated contradiction-detection work, but the manual scoring used in SQS here based on some prior work, as described in Section 3.
Intent Traceability	Part of the Band Section 4.7—Scorer judgement.	Not validated; least theoretical of immediate correctness out of the five (Section 4.7)
Machine-Actionability	Scorer Judgement with reference to the band criteria in Section 4.8	Unvalidated, some computing proxies have reasonable plausibility (schema/structure conformance checker – not implemented here)

Most of the rows above the operational measure are not yet automated judgment tests; they refer specifically to the judgment being made and the criteria being applied to the judgment and, as such, constitute a "operationalized" judgment test.

5. Illustrative Case Studies

The case studies below illustrate SQS, but not its predictive power. Case Study 1 is based on actual project work, save for an artefact on document-intelligence service, which was written as part of the ongoing capstone project of the author, and identified and corrected during the scoring process (Section 5.1 note). It is displayed here as a representation of the upper end of the rubric and is not intended to be a criterion of what specifications are expected to achieve quantitative data detail to be scored as high quality, as a shorter specification, that could indicate a much narrower spectrum, would yield comparable scores. Case Studies 2 and 3 are not parts of a real corpus of industry requirements documents but have been "built by hands and fingers" by the author to show patterns.

5.1 Case Study 1 — High Quality (Structured Specification Excerpt)

Module: OCR Engine. Read text from PDF/PNG/JPG/TIFF. Output: Markdown, JSON. Keep layout, tables and headers. Max latency: 15s per page (measured end to end, 4-vCPU reference instance single-page equivalent throughput).

Non-Functional: Performance. Single image < 10s; 10 page pdfs < 60s (assumes uniform pdf page complexity; larger pdfs are processed asynchronously with webhook call back).

Acceptance Criterion 1. If a PDF that was scanned with more than 300 DPI (and had been scanned using a font from the Latin-1 character set), and the OCR is requested, then full text is provided with > 95% character accuracy (matched characters / total ground-truth characters) over a held-out benchmark set that had been annotated.

Scoring:

Dimension	Score	Justification
Completeness	17/20	Input format, output format, latency (time), and definite acceptance criterion covered; still not clear enough on unsupported input formats.
Unambiguity	19/20	All performances and all shows of accuracy are stated to varying extents with a precisely stated, but not ambiguous, method, as "standard fonts" is not the same as "any fonts".
Internal Consistency	19/20	All the latency numbers explicitly mention what they measure, so there are no more contradictions.
Intent Traceability	14/20	The criteria for acceptance are based on the OCR Module well and clearly; it is only a small portion of the source document that is not a restatement of the business goal; the full document does.
Machine-Actionability	19/20	Formatted in a standardized, labeled structure with clear thresholds and measurement techniques that can be readily retrieved.
Total SQS	88/100

Note on revision: The original source had identical latency and accuracy values, but it did not state what was measured nor did it define "standard fonts. That gap had been a genuine Unambiguity and Machine-Actionability problem, rather than a Consistency problem, and correcting it toughened the definitions contained in the specification, not just padded its cost now after it was found.

Worked example at the requirement level.  The criteria for each dimension is used in two separate statements in this excerpt instead of in the excerpt as a single statement so that the scoring process becomes more transparent than it otherwise would be, as shown below:

Statement	Completeness	Unambiguity	Internal Consistency	Intent Traceability	Machine-Actionability
"Max latency: 15s per page (measured end-to-end... 4-vCPU reference instance)"	Present, with measurement basis stated	Assessed using carefully established procedure — Passes	No difference for other specified latencies	Links with OCR module, but not to a specific business objective.	Labeled field, and explicit threshold and unit.
When OCR is requested, then more than 95% of characters are accurate, computed as "... full text is extracted with > 95% character accuracy".	State with phrase of formula	Occurs when the computation is defined numerically — passes.	No conflict with other statements	Treat the following as an acceptance criterion (passes)	Structured Given/When/Then with explicit formula

Both of these statements individually support a high sub score, as the 'Completeness' score of the excerpt (17 instead of 20) demonstrates a deficiency in some aspect of the module, which neither of these statements demonstrates. This is intended to show that the overall scores (shown in the table above) can be broken down to specific text-based evidence; this level of detail has not been used for each requirement in each case study, due to length constraints within this paper.

5.2 Case Study 2 — Medium Quality (Representative PRD-Style Paragraph)

The dashboard needs to be fast to load and give an overview of what's going on in the account. It must accept most of the widely-used file extensions for import and must be robust enough to handle errors without complaint. Its visuals should be modern and intuitive exactly like something users would assume in contemporary SaaS items.

Scoring:

Dimension	Score	Justification
Completeness	8/20	There are no actors, no acceptance criteria, and "account activity" does not have the typical characteristic of a dashboard feature specification quantifying what it means.
Unambiguity	4/20	These are all unconstrained, qualitative claims, without specific interpretation for the word, with the help of the words quickly, good overview, most common file types, gracefully and clean and intuitive.
Internal Consistency	20/20	All statements are mutually consistent with each other. A weakness that is being discussed, and is considered an Unambiguity, but not a Consistency weakness, is the lack of specificity of "most common file types": not scored twice.
Intent Traceability	6/20	The word for motivation is not used as a business objective; it is only implied.
Machine-Actionability	5/20	Prosied paragraphs that do not have a consistent field structure, and the agent would need to infer if it's a category or not.
Total SQS	43/100

5.3 Case Study 3 — Low Quality (Vague Feature Request)

Integrate AI into the search box for smarter search functionality and user-focused results.

Scoring:

Dimension	Score	Justification
Completeness	2/20	No scope, no actors, no acceptance criteria and no constraint of any sort, related with any feature request what may be considered the minimum requirements needed.
Unambiguity	1/20	The terms “Smarter” and “what people actually want” are unchecked, and lack any workable common understanding and usage as written.
Internal Consistency	18/20	Any statement does not preclude any other statement. This is a Completeness issue, not a Consistency issue, and double-penalized here since there is a near total lack of content.
Intent Traceability	2/20	No specific goal other than the request's goal.
Machine-Actionability	2/20	No structure in the free-form sentence to be extracted.
Total SQS	25/100

5.4 Why the profile matters more than the total

The three worked examples above (88, 43, 25) have been categorized so as to capture high, medium and low quality examples – not surprisingly, since the examples were specifically created to represent these extremes. A more useful example of how the lack of a single number description is useful is the set of two hypothetical specifications, each of which has the same aggregate price but has a different “bad” attribute. Let's look at two specifications - neither of the ones listed above - that we have just created to illustrate that point:

Dimension	Hypothetical Spec A	Hypothetical Spec B
Completeness	18	12
Unambiguity	18	12
Internal Consistency	18	20
Intent Traceability	8	20
Machine-Actionability	18	16
Total	80	80

Both total 80/100. This is characteristics of Spec A — it is quite explicit and clear but does not tightly map requirements back to objectives and an agent building it would likely do the right thing but without understanding why it might be important, or understanding the trade-offs that the specification was not aware of, will be at risk when maintaining or assessing it in the future. Implementing an agent for Spec B will be very well informed but much less complete and less precise: the agent can be guided easily by the “why” of the requirements, but is left with more to make inferences about what to build. Each is a different risk profile, which a single overall score would mask. The total should thus be read descriptively, along with the sub-scores, not as a single readiness indicator – this is why Section 4.9 of the document states this.

6. Discussion

For each SQS dimension, possible computational proxies exist in techniques already developed in the literature reviewed in Section 3 - ambiguity detection using domain-comparison or LLM classification, internal consistency using embedding-based contradiction detection, machine-actionability using structural/schema conformance checking. We should emphasize that we are not asserting the feasibility of "automating," but only point to the possibility of an automatic tool, in that we deliberately use the word "plausible" instead of "automatable" because none of the above is currently implemented and not one has been tested against the manual "scoring" utilized in our case studies, and because automating is an evaluation task yet to be performed rather than one performed. What the paper provides is not a new detection method in one dimension; instead, it is composite framing that revolves around an assumption of specification, and rubric anchors set in each band, which would serve to create by the rubric a more consistent and repeatable process for manual scoring than a reluctant, extreme judgment call.

The framework—bear in mind that SQS is not any tool that calculates SQS. This framework arose out of the applied context of the applied-required design of a prototype specification-engineering platform, dubbed SpecForge AI, that the author designed; during design, it became apparent that an ingested specification needed to be good enough to drive the implementation of AI and that it was a concrete, unmet requirement of the platform. We present and defend the measurement framework – the definitions, rubric, and scoring criteria in Section 4 – apart from any specific software which might calculate it; the implementation of the framework – evidence, status, architecture, and evaluation of implementation of this framework – is beyond the scope of this paper and does not affect the validity of the measurement framework.

7. Limitations and Future Work

The following explicit limitations are in this paper:

• No predictive validation. The accuracy of code generation is not demonstrated in this paper using SQS. That is a correlational claim and the controlled experiment described below is needed.

• Small, non-random illustrative sample. Three case studies (two of which were built by the author, one of which is based on actual project work) illustrate how the framework is used and do not form an empirical sample, nor were they intended to.

• Single-scorer subjectivity. Each score was given by one scorer using the rubric. Inter-rater reliability is untested.

• Non-independence of dimensions. In practice, as stated in Section 4.1, the five dimensions are correlated: one underlying specification fault can potentially negatively impact multiple scores. This is not computed at this time in Additive scoring.

• Unweighted, un-thresholded aggregation. Equal weighting is tentative (Section 4.9) and has not been validated using outcome measures.

• Completeness has no formal scope schema. Completeness is not yet scored against a scope-specific schema (e.g., a specific checklist of what the specification's scope would expect for an API specification, vs. UI specification); it is currently scored against the scorer's impression of what the specification's scope requires. Creation of these schemas per specification type is future work.

• Unambiguity's "broadly shared interpretation" test is not yet operationalized. The current interpretation of whether a qualitative claim has a sufficiently shared interpretation would be determined by the scorer's judgment, which requires either a reference ambiguity lexicon or agreement between raters.

• Reproducibility of scoring is untested. The band criteria in section 4 are not quite as prescriptive as a simple 'guesstimate', but we haven't demonstrated that a consistent result can be obtained from two independent scorers, that use the same specification, and that obtain similar scores.

• Numerical precision within bands is not validated. A distinction of 17-18 is not indicated to be a meaningful difference as mentioned in the previous section 4.3, but the band placement is only defensibly meaningful.

• Proposed empirical extension. We make the distinction from a causal experiment on purpose, but a controlled correlational validation study may contain the defect-injection design described below, whereupon it may be considered a causal experiment. The study is guided by a causal model that is specified as hypotheses that are also testable and refutable: Causal model (SQS → specification-induced assumptions → implementation correctness) with hypotheses tested at various places:

1. H1 (mechanism, first link): SQS is inversely related to the number of specification-driven assumptions that an AI coding agent makes for the implementation of a given specification.

2. H2 (mechanism, second link): The more assumptions needed for the specification, the less correct the implementations are about the specified criteria of functional test pass rate.

3. H3 (total association): SQS has a positive correlation with implementation correctness according to the specification's own acceptance criteria, under the name "total association". It is important to note that H3 is not tested alone, but is rather the net effect of H1 and H2, so testing H1 and H2 is testing, in essence, of the overall association.

H2 and H3 are not framed against "actual" intent (which would need to be elicited using another data collection process (interviews with product owners, or some held-out gold-standard requirements document) separate from the process of writing the specification); instead, they are against some specified intent which the specification itself states. We have another circularity issue: If SQS's own Intent Traceability dimension rewards well-specified acceptance criteria, and H2/H3 evaluate correctness based on the same acceptance criteria, part of what the study measures might be “specifications with more testable requirements produce code that passes more tests” — a very real, but less surprising relationship than general quality vs. outcome. If possible, the report should include in addition to the specification-defined validations, added tests written by others or tests conducted by human experts to differentiate between SQS's descriptive validity (is it measuring what it says it is measuring) and its predictive validity (is it predicting what it says it is predicting).

Design:

• Independent variable: is the total and sub dimension scores of the SQS across a range of scores as captured in a specification corpus..

• Dependent variables: Functional test pass-rate according to the specified acceptance criteria, number of assumptions resulting from the specification, that are not satisfied by the test-function and are added to the specification by "human correction effort", identified by the procedure of annotations presented below.

• Controls: ixed model and model version (or fixed toolchain), fixed task type and repository context (or fixed generation budget, important: task complexity, a naturally simple task can generate high SQS (small decision problem) and high correctness at the same time, but this does not depend on the causality between SQS and correctness. A variety of factors can be used to measure the complexity of candidates; number of requirements, number of different entities and/or interfaces, number of acceptance criteria are some examples.

• Corpus construction: Rather than gathering some naturally occurring, imperfect specifications (which might risk finding the items vary with quality, but also with the task, or the domain), or simply sticking to writing dolled down versions of the same specification (which could risk the artificial-defect confound), the best designs make many controlled versions of the same design class with systematically and individually varied defects — e.g., two high-SQS baseline versions in the design, and variations where just one of the dimensions (or a subset) is degraded. This can also help in conducting causal comparison (same task, manipulated specification quality) as opposed to observational correlation..

• Assumption-annotation procedure: For each implementation produced by the procedure, (1) identify implementation decisions for which there is no explicit statement in the requirement, using implementation artifact and not agent's reported reasoning trace, since they may be absent or variable or not reflect actual reasoning; (2) classify each of those decisions as explicitly-specified, an implementation choice, a specification-induced decision (according to the decision rule above), or an unrelated implementation detail; (3) get multiple human annotators to independently classify the decisions, then (4) measure interrater agreement (e.g., Cohen's κ for the categorical classifications) before accepting the counts as reliable. Log generated by Agent(s) or explanation(s) (if available) may be evidence, but NOT part of the classification.

• Research questions: Some potential research questions that one might ask are (a) Do H1 – H3 hold, and does this effect still emerge when task complexity is controlled? (b) Which factor seems the most predictive of implementation correctness and do this answers warrant departure from the precedent of equal weighting? (c) Is a threshold-based pass/fail criterion (See Section 4.9) more effective at predicting outcomes than the additive total? (d) Are the scores from independent scorers who are using the Section 4 rubric to the same specification corpus sufficiently inter-rater reliable (e.g., weighted κ or ICC) to make SQS a reliable instrument and not merely the structured opinion of one person?

In addition to this study, Internal Consistency is a natural extension to a more complete conflict-detection method that will include external consistency with architecture and existing code, while Intent Traceability is a natural extension to an Intent Mapping taxonomy of sections of specification to agent roles in multi-agent development pipelines. These are both areas this framework can service – not be assuming of commitments in their place.

8. Conclusion

It is not just about how a specification is written, but about how much work an agent must infer from the specification. Between what a specification says and what an agent must infer is not just a matter of writing, but a measurable and meaningful property of the specification. The key contribution of this paper is presenting the specification induced assumption as a named mechanism that relates the quality of the specification to implementation outcomes and SQS as a rubric that attempts to operationalize specification quality based on five dimensions previously determined and concluded to be specifically relevant to an AI coding agent, rather than borrowed from requirement quality traditions directly from humans. The open questions raised by this paper for the next ones are not if the five dimensions are interesting; the worked case studies and the construct-measure table in Section 4.10 suggest that they are, but whether they are reliable from one scorer to another and whether they predict what the results from which they are expected to predict are.  Sections 4.10 and 7 name that gap directly rather than leaving it implicit, and the controlled study proposed there is the next step in this research program.

References

1.      Zhao, L., Alhoshan, W., Ferrari, A., Letsholo, K. J., Ajagbe, M. A., Chioasca, E.-V., & Batista-Navarro, R. T. (2020). Natural language processing (NLP) for requirements engineering: A systematic mapping study. arXiv:2004.01099.

2.      Wang, Z., Li, J., Li, G., & Jin, Z. (2023). ChatCoder: Chat-based refine requirement improves LLMs' code generation. arXiv:2311.00272.

3.      Fantechi, A., Gnesi, S., Passaro, L., & Semini, L. (2023). Inconsistency detection in natural language requirements using ChatGPT: A preliminary evaluation. In 2023 IEEE 31st International Requirements Engineering Conference (RE).

4.      Vogelsang, A., & Fischbach, J. (2024). Using large language models for natural language processing tasks in requirements engineering: A systematic guideline. arXiv:2402.13823.

5.      Krishna, M., Gaur, B., Verma, A., & Jalote, P. (2024). Using LLMs in software requirements specifications: An empirical evaluation. arXiv:2404.17842.

6.      Frattini, J., Montgomery, L., Fischbach, J., & Unterkalmsteiner, M. (2024). NLP4RE tools: Classification, overview, and management. arXiv:2403.06685.

7.      Saleem, S., Asim, M. N., & Dengel, A. (2024). PassionNet: An innovative framework for duplicate and conflicting requirements identification. arXiv:2412.01657.

8.      "Multi-Label Requirements Classification with Large Taxonomies." (2024). arXiv:2406.04797.

9.      "Replication in Requirements Engineering: The NLP for RE Case." (2024). arXiv:2304.10265.

10.   "Practical Guidelines for the Selection and Evaluation of NLP Techniques in Requirements Engineering." (2024). arXiv:2401.01508.

11.   "Automated Repair of Ambiguous Natural Language Requirements." (2025). arXiv:2505.07270.

12.   Huang, K., Wang, F., Huang, Y., & Arora, C. (2025). Prompt engineering for requirements engineering: A literature review and roadmap. arXiv:2507.07682.

13.   Paleyes, A., Sendyka, R., Robinson, D., Cabrera, C., & Lawrence, N. D. (2025). Prompt variability effects on LLM code generation. arXiv:2506.10204.

14.   Zi, Y., Menon, H., & Guha, A. (2025). More than a score: Probing the impact of prompt specificity on LLM code generation. In Proceedings of IJCNLP-AACL 2025, pp. 2380–2402. Also arXiv:2508.03678.

15.   "Do Prompt Patterns Affect Code Quality? A First Empirical Assessment of ChatGPT-Generated Code." (2025). arXiv:2504.13656.

16.   "When Prompt Under-Specification Improves Code Correctness." (2026). arXiv:2604.24712.

17.   Acharya, V. (2026). Semantic consensus: Process-aware conflict detection and resolution for enterprise multi-agent LLM systems. arXiv:2604.16339.

18.   Nair, P. P. M., et al. (2026). A retrieval-augmented framework for detecting and resolving pragmatic ambiguities in natural language requirements. arXiv:2607.04436.

19.   Piskala, D. B. (2026). Spec-driven development: From code to contract in the age of AI coding assistants. arXiv:2602.00180.

20.   "Specification-Driven Development as the Foundation of AI-Native Enterprise Software Engineering." (2026). arXiv:2607.16680.