nw-at-completeness-check

AT Completeness Check — Canonical Taxonomy

Mechanical gate for acceptance-test completeness. Runs against any candidate AT set. Each unchecked item = potential gap. Verdict deterministic by count, not judgment.

Provenance: research-anchored 7-category taxonomy, paradigm-neutral. See docs/research/at-edge-case-taxonomy-2026-05-19.md for full literature review.

Anchors [1]-[14] reference research doc bibliography.

Domain extensions

Project-specific AT-class specializations live as YAML overlays in domain-extensions/ (sibling dir to this SKILL.md). The canonical 7-category taxonomy stays paradigm-neutral; domain overlays add extra_checks that the reviewer appends to the 15-item checklist for features opting in.

When to add an extension: project surfaces an AT-class that is a specialization of one or more canonical categories (typically C5 + C6) but not a new general category. Example: nWave IP/Privacy boundary = public:false mode flag (C5) + leak-in-output failure-contract (C6) — lives in domain-extensions/nwave-installer.yaml, NOT in canonical taxonomy.

Overlay schema (domain-extensions/<kebab-case-domain-id>.yaml):

name: <kebab-case-domain-id>
version: <semver>
applies_to: <project / package / feature-pattern>
extends_canonical: [C5, C6]      # which canonical categories this specializes
extra_checks:
  - id: <DomainID>a
    description: <what to verify>
    maps_to_canonical: C6
    mandatory: <bool>

Opt-in per feature — in docs/feature/{id}/distill/at-completeness-extensions.yaml:

extensions: [nwave-installer]

Reviewer adds overlay's extra_checks to the canonical 15-item checklist for that feature only. Verdict thresholds scale with total item count.

1. Canonical 7-category taxonomy (paradigm-neutral)

ID	Category	Anchor	One-line definition
C1	Equivalence & Boundary	ISTQB §4.2 [1] / Beizer ch.5 [2]	Partition input domain + test at/adjacent to each boundary
C2	State & Transition	ISTQB §4.2.4 [1] / Hendrickson [3] / Hypothesis stateful [4]	Every legal transition + illegal-event-from-each-state + self-loops/terminal exits
C3	Count Cardinality (0/1/N)	Hendrickson "Count" [3] / Adzic key examples [5]	Empty / singleton / many for every collection input or output
C4	CRUD-Lifecycle & Idempotency	Hendrickson "CRUD" [3] / Hillel Wayne PBT+Contracts [6]	Repeat / replay / out-of-order ops preserve invariants; f(f(x)) == f(x) for idempotent ops
C5	Mode-Flag / Decision-Table	ISTQB §4.2.3 [1] / Adzic "key examples" [7]	Every materially-distinct Cartesian combination of mode flags exercised
C6	Negative & Robustness (Postel)	RFC 760 §1.2.10 [8] / FEW HICCUPPS [9] / Kaner LLST [10] / RIMGEA [11]	Hostile/degenerate input → explicit typed-error contract, never silent coercion
C7	Configuration / Environment / Interruption	Bach HTSM SFDIPOT [12] / Hendrickson "Configurations+Interruptions+Starvation" [3] / Marick Q4 [13]	Resource starvation + interruption mid-flow + concurrent actors

C1. Equivalence & Boundary

Partition the input domain into equivalence classes; test at least one representative per class plus values immediately on/adjacent to each partition boundary. Failures cluster at edges, not interiors. Citation: ISTQB Foundation v4.0 §4.2 [1]; Beizer 1990 ch.5 "Domain Testing" [2].

C2. State & Transition

Model the SUT as states + events + guards + transitions. Cover (a) every legal transition, (b) ≥1 illegal-event-from-each-state (rejected gracefully), (c) self-loops and terminal-state exits. Citation: ISTQB §4.2.4 [1]; Hendrickson "State Analysis" [3]; Hypothesis RuleBasedStateMachine [4].

C3. Count Cardinality (0/1/N)

For every collection-shaped input or output, exercise zero, one, and many. Zero is the canonical bug magnet (null-deref, divide-by-zero, "no items" UI). Citation: Hendrickson/Lyndsay/Emery cheat sheet — "Count: Zero, One, Many" [3]; Adzic [5].

C4. CRUD-Lifecycle & Idempotency

Full Create/Read/Update/Delete lifecycle + verify repeat/replay/out-of-order operations preserve invariants. Idempotency = f(f(x)) == f(x). Citation: Hendrickson "CRUD" [3]; Hillel Wayne PBT+Contracts [6]; ISTQB decision-table [1].

C5. Mode-Flag / Decision-Table Coverage

When SUT exposes mode flags (dry_run, force, verbose, public), every Cartesian combination materially-different in behavior is a distinct AT. Citation: ISTQB §4.2.3 decision-table [1]; Adzic "Focus on key examples" [7]; Adzic SbE [5].

C6. Negative & Robustness (Postel)

Every input channel accepts hostile/degenerate input; SUT must respond with explicit, asserted failure contract (typed error, exit code, empty-valid output) — never crash, never silently accept. Citation: RFC 760 §1.2.10 [8]; FEW HICCUPPS "Standards"+"Claims" [9]; Kaner/Bach/Pettichord LLST [10]; RIMGEA [11].

C7. Configuration / Environment / Interruption

SUT runs under varying resource availability and may be interrupted mid-flow. Cover (a) resource starvation, (b) interruption mid-transaction, (c) concurrent actors. Citation: Bach HTSM SFDIPOT [12]; Hendrickson "Configurations"/"Interruptions"/"Starvation"/"Multi-user"/"Flood" [3]; Marick Q4 [13].

---

2. Mechanical 15-item application checklist

Run against any candidate AT set. Unchecked = potential completeness gap.

C1a — ≥1 AT exercises empty/zero/minimum-size input
C1b — ≥1 AT on each partition boundary (max-1, max, max+1)
C2a — SUT state machine documented in AT module docstring
C2b — For each state, ≥1 AT for illegal-event-from-that-state
C3  — parametrize/PBT covering n ∈ {0, 1, many} for each collection input
C4a — Each mutating op has "apply twice" AT (idempotency or correct non-idempotency)
C4b — ≥1 AT for inverse op without prerequisite (uninstall-without-install)
C5a — Each mode flag: every materially-distinct combination exercised
C5b — ≥1 AT asserting flag orthogonality (verbose toggles output only)
C6a — Each input param: ≥1 AT with malformed value (wrong type, malformed encoding)
C6b — Each declared error in contract: ≥1 AT triggers exactly that error
C6c — ≥1 AT asserts closed error set (no other error escapes)
C7a — ≥1 AT under degraded-resource condition (read-only FS / no network / low disk)
C7b — ≥1 AT for interruption mid-operation (SIGINT / timeout / partial commit)
C7c — If concurrent-safe by claim: ≥1 multi-actor AT (two parallel invocations)

Machine-readable form: checklist-15-item.yaml (alongside this file).

Verdict thresholds (deterministic)

Count passing	Verdict
< 10 / 15	INCOMPLETE — reject; route per §4
10–12 / 15	ACCEPTABLE_WITH_DOCUMENTED_GAPS — pass with explicit listed gaps
≥ 13 / 15	COMPLETE — pass

The reviewer agent computes the count mechanically, not subjectively. Items not applicable (e.g. C7c for non-concurrent SUTs) count as passing — document the rationale in verdict output.

---

3. PBT / parametrize signatures per category

One-line code template per category. Crafter/acceptance-designer copy-adapt.

# C1 — Equivalence & Boundary
@given(st.lists(elt, min_size=0, max_size=N+1))
@example([]) @example([single])

# C2 — State & Transition
class M(RuleBasedStateMachine):
    @rule(...)
    @invariant()
    @precondition(...)

# C3 — Count Cardinality
@pytest.mark.parametrize("n", [0, 1, 2, 100])
# or st.integers(min_value=0, max_value=...)

# C4 — CRUD-Lifecycle & Idempotency
# Property: f(f(x)) == f(x)
# Or: RuleBasedStateMachine with @rule chains over CRUD sequence

# C5 — Mode-Flag / Decision-Table
from itertools import product
@pytest.mark.parametrize("flags", list(product([True, False], repeat=k)))
# filter degenerate combinations

# C6 — Negative & Robustness
@given(st.one_of(st.text(), st.integers(), st.binary()))
# assert raises typed error from closed set:
# with pytest.raises(DeclaredError): ...

# C7 — Configuration / Environment / Interruption
class M(RuleBasedStateMachine):
    @rule(...)  # includes interruption events
# + parametrize over resource-degradation fixtures

Language-equivalent frameworks: Hypothesis (Python), fast-check (TS/JS), QuickCheck (Haskell), quickcheck (Rust), jqwik (Java), FsCheck (C#).

---

4. Reviewer output schema

Reviewer emits a typed verdict. Two kinds of finding only.

@dataclass(frozen=True)
class ATGap:
    scenario_class: str            # e.g. "C5a:dry_run-and-force-combo"
    current_at_count: int          # 0 if missing entirely
    reason: str
    kind: ATGapKind
    severity: Severity             # BLOCKER | HIGH | MEDIUM | LOW

class ATGapKind(str, Enum):
    AT_GAP_IN_DELIVERY_SCOPE = "at_gap_in_delivery_scope"
    SPECIFICATION_AMBIGUITY  = "specification_ambiguity"

ARCHITECTURE_SCOPE_MISS is NOT a reviewer-authored kind. Phase D router derives it via second-order rule (≥2 gaps sharing a scenario_class mapping to component absent from DESIGN output).

---

5. Upstream-wave routing rule

Categories C2, C5, C6, C7 require upstream-wave specification. If absent → SPECIFICATION_AMBIGUITY, NOT AT_GAP_IN_DELIVERY_SCOPE. Phase D routes back to upstream wave, not back to DISTILL.

Category	Upstream owner	Required artifact
C2 (state machine)	DISCUSS	State diagram in user-stories Elevator Pitch + DoD
C5 (mode-flag inventory)	DESIGN	Flag enumeration in component manifest
C6 (error contract)	DESIGN + DISCUSS	Typed error set + invariants per port
C7 (env / interruption matrix)	DEVOPS	Env matrix + concurrency/interruption contract

Routing decision (mechanical):

if upstream artifact for category X missing
    → ATGap.kind = SPECIFICATION_AMBIGUITY → re-enter the upstream wave
else
    → ATGap.kind = AT_GAP_IN_DELIVERY_SCOPE → address within the delivery cycle

This closes the Mandate-12 SSOT loop: domain types in DISCUSS/DESIGN/DEVOPS drive taxonomy population.

---

6. Domain extensions

Canonical taxonomy is paradigm-neutral. Project-specific instantiations live in domain-extensions/*.yaml.

nWave/skills/nw-at-completeness-check/
├── SKILL.md                          # this file — canonical 7-category (GENERIC)
├── checklist-15-item.yaml            # machine-readable mechanical gate
└── domain-extensions/                # per-project overlays (kebab-case)
    ├── README.md                     # how to add a domain extension
    ├── nwave-installer.yaml          # IP/Privacy + filesystem-shape (nWave-specific) [SLOT]
    └── nwave-des.yaml                # DES marker-specific extensions [SLOT]

Per-feature opt-in: docs/feature/{id}/distill/at-completeness-extensions.yaml lists which overlays apply (e.g. extensions: [nwave-installer]).

Example mapping: IP/Privacy boundary (nWave domain) → instance of C5 (public:false mode flag) + C6 (leak-in-output as failure-contract assertion). Lives in domain-extensions/nwave-installer.yaml, NOT in canonical taxonomy.

---

7. Falsifier-gate (taxonomy self-pruning)

Telemetry per gate run: (feature_id, category_id, finding_count, severity_max) → 3-month rolling window.

Signal	Decision
3 consecutive zero-findings on category X across pilot features	PRUNE X from default checklist (cost ≤ benefit)
1 BLOCKER found via category X	ESCALATE X to MANDATORY (cannot be skipped)

This makes the taxonomy itself empirically-falsifiable. Default state: all 7 categories active.

---

8. Empirical class → research category mapping (proves generality)

Empirical class (spike-3 2026-05-19)	Research category	Notes
IP/Privacy boundary	C5 + C6 (instantiation)	Domain-specific overlay, NOT general
Negative paths (missing file, malformed JSON)	C6 direct	Canonical robustness/Postel
Idempotency (uninstall w/o install)	C4 direct	CRUD lifecycle + idempotency property
Mode flags (dry_run / force / verbose)	C5 direct	Decision-table coverage
Failure contract on degenerate state	C6 direct	FEW HICCUPPS "Claims" consistency
Type-domain (bool/int where str expected)	C6 direct	Type-level robustness; PBT st.one_of natural fit

Empirical classes 2–6 generalize via C4+C5+C6. Class 1 (IP/Privacy) → domain extension. C1, C2, C3, C7 = categories spikes did NOT surface — predictable next adversarial-reviewer hits.

---

References (research doc bibliography)

[1] ISTQB Foundation v4.0 §4.2 Black-Box Test Techniques (2023). [2] Beizer, Software Testing Techniques 2nd ed., 1990, ch.5 Domain Testing. [3] Hendrickson/Lyndsay/Emery, Test Heuristics Cheat Sheet, testobsessed.com / Ministry of Testing. [4] Hypothesis stateful tests docs. [5] Adzic, Specification by Example, Manning 2011. [6] Wayne, "Property Tests + Contracts = Integration Tests", hillelwayne.com 2019. [7] Adzic, "Focus on key examples", 2014. [8] RFC 760 §1.2.10 Robustness Principle (Postel, IETF 1980). [9] Bolton/Bach, "FEW HICCUPPS", DevelopSense. [10] Kaner/Bach/Pettichord, Lessons Learned in Software Testing, Wiley 2001. [11] Kaner, RIMGEA/RIMGEN bug-reporting mnemonic. [12] Bach, Heuristic Test Strategy Model v6.3 (SFDIPOT), Satisfice. [13] Crispin/Gregory, Agile Testing, Addison-Wesley 2009 (Marick quadrants synthesis). [14] Thomson/Nottingham, "The Robustness Principle Reconsidered", CACM 2011.

Full research doc: docs/research/at-edge-case-taxonomy-2026-05-19.md.