rea/docs/testing.md

Testing REA

Use this guide when choosing coverage, pruning tests, or running a verification lane. Start with the commands below; use real-toolchain lanes for engine prerequisites and client verification for installed-client and agent runs.

Historical verification records may refer to experiments/analysis-efficiency. Those experiment scripts are available in the archived source tree; they are not included in current checkouts. The retained test support contains deterministic checkers and fixtures. The allocation report and its original receipt JSON remain documentation, checked by the report producer.

Developer commands

Use source feedback while editing, explicit boundary checks for the changed behavior, and complete CI evidence before merging.

Command	Scope
npm run test:local	Dirty source tests via the import graph, without build; explicit source paths run regardless of Git status
npm run test:focused -- PATH...	Exact existing test files; compiled boundaries build first, and unmatched paths fail
npm run test:changed	Source tests affected since the merge base with origin/main, including committed and dirty changes
npm run test:fast	All domain, service, adapter, composition, conformance, and evaluation tests without build
npm run test:boundary	Boundary, CLI boundary, MCP boundary, process boundary, and process-global projects
npm run test:mcp	MCP boundary project
npm run test:acceptance	Complete compiled CLI and MCP acceptance workflows
npm run test:python	LLDB and mitmproxy producer contracts plus analysis-efficiency checks; no real debugger or proxy required
npm run test:watch	Dirty source tests in watch mode, without build
npm run test:watch:all	Changed tests from every project; builds at startup, so rebuild after production edits before relying on compiled tests
npm run check:changed	Cached typecheck/lint and branch-related source feedback
npm run check:pr	Opt-in complete local deterministic gate and generated-file checks
npm run docs:check	Generated-document validation and fresh onboarding example consistency checks
npm run docs:build	Render VitePress documentation and apply its page-link checks; no live external-link or GitHub fragment verification

The general source suite above runs on Linux in CI, and Windows CI runs a curated lane (Native platform baseline in CI). A Windows source checkout can fail source tests for reasons unrelated to the change, such as POSIX fixture paths, #!/bin/sh tool stubs, or POSIX rename semantics. Compare failures against the base revision, or run the suite on Linux or WSL, before attributing them to a change. See CONTRIBUTING.md → Windows source checkouts.

The CI documentation lane runs both docs:check and docs:build when the existing change classifier selects documentation or its producing inputs. Generated facts and rendered page links have separate checks. VitePress does not establish that every external URL or GitHub-rendered localized heading exists; verify those destinations when editing them.

For example:

npm run test:local -- src/config.test.ts
npm run test:focused -- tests/acceptance/applications/runtime.test.ts
npm run test:changed -- --base origin/main
npm run test:changed -- --dry-run

test:focused accepts exact repository-relative test file paths. Source-only paths do not build; boundary, acceptance, process-global, or unfamiliar tests/ paths build conservatively. Tests use Vitest concurrency and isolated workspaces; commands do not hold a broad test lock. Build and documentation writers retain checkout-local locks for their shared output files. Explicit selections do not use --changed or permit zero-test success. The dry-run option reports the chosen merge base, scope and build prerequisite without executing tests or building. A missing Git base reports how to fetch it or select another revision.

Source typechecking needs no compiled runtime or generated test metadata. Typecheck, runtime compilation, and Frida binding verification share scripts/run-typescript.mjs. The pinned tsrs package supplies Linux x64/arm64 (GNU and musl) and macOS arm64 compilers. Windows and other unsupported hosts retain the existing stable official TypeScript compiler. Its upstream version can differ from tsrs, so both compiler tasks hash the selected package and version through scripts/run-turbo.mjs before cache lookup. Outputs from different compilers occupy different cache entries. Repository npm scripts and focused-test preparation use that entrypoint; a raw Turbo compiler task without matching identity fails instead of publishing an ambiguous successful cache entry. A missing binary on a supported host also fails without silently falling back.

Development verifiers load their shared command, terminal and file owners from source using Node's in-process TypeScript transformation. A single process-lifetime loader handles this checkout's src/ and scripts/lib/ graphs, resolving missing relative .js/.mjs imports to their .ts/.mts source. Existing JavaScript and modules outside those roots keep native resolution and failures. Node's module cache shares source identities within the verifier process; transformed output is not cached across Node versions. Restart the verifier after source edits. This avoids a cold-cache compiler subprocess surviving command cleanup; the live process-lineage checks remain required.

The Rust compiler defaults to at most two checker threads. For --noEmit checks, --maxMemory 3G also sets a soft memory target; emit builds do not use that target. Explicit compiler arguments can override the defaults. These controls do not impose a process-family memory limit. Termux's explicit build, Stryker's compiler API dependency, and independent fixture compiler oracles keep their existing TypeScript versions.

npm run build:cached compiles the CLI/MCP runtime and its packaged skill. The build also records dist/mcp-catalog.json (compact inputs) and dist/mcp-catalog-full.json from the canonical SDK registrations and binds each to the runtime with its own SHA-256 digest. These are runtime artifacts, separate from the test catalog below. Default target-free MCP startup serves its schemas without constructing analysis validators; the first tool or guided-workflow request materializes the canonical runtime on that same SDK server. Explicit compact/full input selections preserve their advertised schemas throughout materialization. Initial-target, HTTP and output-schema-off startup keep the eager path; source runs without recorded artifacts also use eager composition. Compare memory both after discovery and after a completed request; report diagnostic GC separately from normal resident memory.

Turbo's strict task environment preserves caller-selected GOMAXPROCS, GOMEMLIMIT, and UV_THREADPOOL_SIZE for resource-constrained builds and checks. These execution controls do not change artifact content, so changing their values retains cache reuse. They control individual runtimes; monitor the complete process family separately when enforcing a total memory budget. npm run test:prepare also generates the MCP contract test catalog, product catalog, and portable managed evidence used by the complete suite. Focused tests prepare those extra outputs only when their selected files consume them. The MCP test catalog is JSON in .cache/mcp-tool-catalog.json; the tracked test loader supplies its types without making generation a source-check prerequisite. Run npm run mcp-catalog:generate to refresh it independently. Documentation generation and managed conformance run through their separate docs:generate and evidence:generate task graphs.

Changed selection can miss runtime registration, generated data, shell entrypoints, bridges, or other relationships absent from the import graph. An empty changed selection means no tests were selected, not verified correctness. Select relevant boundary files and real-provider lanes explicitly. Source projects also contain large capacity regressions; test:fast promises no compiled-runtime prerequisite, not a fixed time budget.

npm run lint prints errors and a warning summary; npm run jscpd prints duplication statistics. Complete reports remain in .cache/static-checks/. The fast gate uploads them as static-check-reports. Full qualification uploads duplication diagnostics from the deep job as static-deep-check-reports, including after that gate fails. Task-graph recording runs check:ci:fast:tasks and check:ci:deep:tasks without the artifacts lock, so it can record hashes while check:pr or check:test already holds that lock. The CI jobs still run the locked check:ci:fast and check:ci:deep commands. Use npm run lint:details or npm run jscpd:details for all console diagnostics. Both duplicate commands scan production and test code using the limit in .jscpd.json. The summary commands preserve checker failures and restore reports from cached static tasks. Both lint caches include the website, documentation configuration, and experiment code scanned by the checker; a regression compares the actual lint inventory with Turbo's resolved inputs.

Static checks locate candidates; they do not establish semantic ownership. jscpd finds token clones, and Knip counts test references as uses. Module boundaries and architecture guards enforce their declared import and source rules. Passing them does not prove that two parsers agree, a wrapper is still needed at runtime, or cleanup has one owner.

The lint gate treats no-async-promise-executor, no-unreachable, and no-unsafe-optional-chaining as errors. These catch lost async executor failures, dead code, and unsafe uses of optional-chain results. no-unsafe-finally remains a warning because cleanup paths can deliberately throw aggregated failures; review whether they preserve the primary failure. Two local rules cover narrower observed failures: rea/no-buffer-deep-equality flags Vitest deep comparisons of Buffers from recognized constructors and file reads, including simple const aliases; rea/cancellation-cause requires an explicit cause when a directly checked caller AbortSignal is converted to an AnalysisCancelledError. The rules do not infer arbitrary Buffer types, follow indirect cancellation helpers, or prove that an explicit cause equals the caller's reason. Keep focused boundary and lifecycle tests for those facts.

rea/no-evidence-json-preadmission rejects direct generic JSON parses in createEvidence payload expressions; it does not analyze intermediate values. rea/no-positional-tool-contract rejects catalog indices used to select operation names, schemas or types; display-only selection remains allowed. rea/no-fixture-root-cleanup follows const fixture-root aliases into teardown deletion; mutable bindings and mixed root arrays remain outside its scope.

rea/no-shared-module-mock resolves Vitest import bindings before rejecting module mocks in projects without per-file isolation. The test-project definitions own isolation for Vitest and lint; the executable gate checks the resolved policy and runs the same collector. Aliases and namespace imports count; comments, shadowed locals, ordinary methods and spyOn do not. Negative CLI probes use an external temporary project so concurrent repository checks cannot discover them.

Focused .only declarations, broken assertion promise chains, and unused disable directives are errors. vitest/valid-expect warns (maxArgs: 2) because valid stored and collected assertions can be awaited later. import/no-duplicates warns with preferInline: false.

no-useless-escape keeps intentional \- regex escapes because advertised patterns must also compile in v mode; the catalog regression checks every advertised pattern.

Fast lint stays syntax-based. npm run lint:typed runs serially with pinned oxlint-tsgolint 7.0.2005 and records native plus typed diagnostics in .cache/static-checks/typed-oxlint.json. Production src treats no-misused-promises as an error; tests/fixtures warn. no-floating-promises is an error throughout the lint inventory. return-await is an error in error-handling scopes so asynchronous completion stays inside its catch or cleanup boundary. Exhaustiveness is an error when a union member has neither a case nor a default. Intentional defaults count as handling the remaining union; new members covered by those defaults therefore require behavioral review.

Two HTTP/3 source-test files allow the runner-owned node:test registration API; ordinary unowned promises in those files still fail. The authored code-split recovery input is exempt from floating-promise policy because it is analysis input, rather than application orchestration. Preserve framework-owned callbacks such as an awaited Promise.finally; inspect the actual completion owner before changing a callback solely to satisfy its declared void return type.

npm run knip:production writes advisory candidates to .cache/static-checks/production-reachability.json; execution/report errors fail. Ordinary Knip remains. check, check:pr, and check:ci:deep run reachability then typed lint sequentially after cached work. Typed lint builds the project type graph and stays separate from fast feedback. Its default Go settings are GOMAXPROCS=1 and GOMEMLIMIT=1024MiB; caller-selected settings take precedence. One Go worker reuses checker caches across the file queue without changing the file inventory or TypeScript project scope. The Go memory target is soft and does not cap the complete process family's memory; measure the full invocation when enforcing a host budget.

Routine iterations and rebases need focused regressions and relevant checks. Before handing off a PR, run npm run check and generated-document checks when applicable; CI owns the full suite and coverage. Use the full local gate for broad changes or diagnosing CI, rather than after every edit. Package/install changes additionally need package verification; provider changes need actual provider evidence.

Vitest projects use up to two workers, bounded by host parallelism. process-boundary runs later with serial files because process-tree sampling shares host resources. Acceptance and process-global cases use isolated forks without serial scheduling. Pure domain/service, composition, conformance, evaluation, CLI boundary, and MCP boundary projects share module graphs. Their tests avoid process-global state and keep resource cleanup test-scoped. Each MCP session still owns its resources, and CLI cases retain fresh command instances or subprocesses. Fixtures using vi.mock or vi.doMock need an isolated project. CLI/MCP fixtures normally live in tests/process-global/. When a path must stay, assign it explicitly in scripts/lib/test-projects.mjs, exclude it from its shared owner, and preserve its source/built group. Product catalog verification stays with the isolated filesystem boundary tests because it loads both source and compiled module graphs. Adapter, other boundary, acceptance, and process-global projects retain per-file isolation. See vitest.config.ts for current settings.

Inject producer faults through scoped dependencies in shared workers: a module mock cannot replace a consumer that was imported earlier. Check these tests with the consumer preloaded as well as alone. For simulated protocol traces, use an acknowledged event barrier before asserting complete observations; a zero-duration capture does not guarantee queued events have arrived.

Place faults at the semantic boundary under test: the selected read, EOF, hash completion, or cleanup attempt. Global call ordinals can shift during setup or refactoring without reaching that phase. For controlled asynchronous gates, use awaitAcknowledgement from tests/fixtures/lifecycleAcknowledgement.ts to race the acknowledgement against triggering-operation settlement and a bounded deadline. Register release gates before launch, then release and join launched work in finally before restoring mocks, even when the barrier fails.

Focused contract tests should import the named contract family they exercise; reserve the full tool catalog for catalog-wide assertions. Resolve contracts by operation name rather than array position.

Check advertised schemas after SDK conversion for every supported schema profile. Meta-schema validity does not establish client acceptance or useful rendered declarations. Keep canonical rejection checks separate from the intentionally reduced compact advertisement, and probe actual client renderers when changing schema representations.

The reused CLI boundary workers cap V8 old space at 512 MiB, with or without coverage. Ordinary CI also exhausted the former 384 MiB budget after earlier files loaded the shared source/contract graph, while later command registration and schema/help rendering still needed transient allocation space. This is a measured capacity control, not a suite-wide speedup claim. The worker count remains bounded at two, and spawned REA processes retain their own heap settings. Coverage enablement still controls instrumentation, test deadlines and coverage thresholds; explicitly disabling it or selecting only a provider does not enable instrumentation.

Vitest worker limits do not bound subprocesses launched within a test. Bound those batches separately so repeated CLI validation cannot oversubscribe the runner. Preserve simultaneous launches where concurrency is itself under test.

Coordinate expensive checks across agents sharing a host. If a test unexpectedly consumes heavy CPU, stop its owned process family, retain its log and revision, and profile a bounded representative case before retrying. Separate production work from fixture generation, schema compilation, subprocess startup and cleanup. Choose elapsed, CPU and process-family memory limits before the run; a worker count or test timeout alone does not enforce those limits.

Keep capacity fixtures intact when diagnosing a test failure. For byte equality of large buffers, use Buffer.equals or Buffer.compare; a generic structural diff can allocate far more than the production operation.

Remove demonstrated repeated work while preserving distinct assertions and resource ownership. Share immutable fixture preparation where safe; keep fresh sessions when lifecycle or concurrency is the behavior under test. Compare the same completed workload before and after, report cache state and measurement limits, and record interrupted runs separately from passes. Increasing a timeout does not explain a CPU regression.

Build/documentation writers hold checkout-local output locks. Tests do not hold a broad command lock; check:pr finishes tests before document validation. Turbo caches deterministic preparation and static checks, but test:run always executes. Repository files do not fully describe installed tool contents or host state. A previous test pass therefore cannot replace current validation, even when the source tree is unchanged. check:pr also runs the fast Python producer suite; CI runs it in the static job for code changes. These producer tests do not establish real LLDB engine coverage.

Vitest's persistent compile cache remains disabled. The cliTest fixture shares a temporary Node compile cache among its fresh CLI subprocesses. The worker removes it after test-scoped processes finish. Explicit child environment settings take precedence; children requesting NODE_V8_COVERAGE do not receive the fixture cache. These subprocesses are outside aggregate Vitest coverage; subprocess coverage attachment remains disabled. Cold-start import checks launch independently without this fixture cache.

analyzeJavaScriptDirectory stdio MCP cases in one cliTest file share one server and that compile cache. The first case pays startup; later cases reuse the process. File teardown waits for startup still in flight, closes a server registered during that wait, and does not open a server after close. A failed handshake stays the caller error when its cleanup also fails, including a rejection without a reason. That cleanup stays owned until a later close or call releases it. Another server does not open while the cleanup remains, and a cleanup that already succeeded is not run again. Pass mcpCall when a case needs its own server, environment, or lifecycle. CLI cases for that helper share one compiled CLI process per child environment. The first case pays process startup; later cases reuse the loaded runtime. The process restarts when the environment changes, after 32 commands, or when the worker reports a resident set of at least 1 GiB. A fatal worker failure or a broken session frame retires that process and waits for it to exit before the next command starts. cli.run still starts a fresh process, including stdin and explicit deadlines. runFileCli uses the file session for commands that have neither. Go binary, IL2CPP metadata, and dylib-resolution commands use that session. File teardown stops the shared CLI process, including startup still in flight, and does not open another after close. exportedFunctionBindingsCli.test.ts uses the shared compiled CLI session for its file and compares its JSON command results with the MCP session. It selects 20-second startup and command deadlines; other callers retain their existing deadline behavior. Retired processes stay owned until exit, block replacement, and remain available for a later close to retry after failed termination. Shared session cleanup closes stdin so the worker can release JavaScript resources, waits the bounded grace, then escalates to SIGTERM and SIGKILL only if needed. Confirmed exit succeeds; an unconfirmed exit reports cleanup failure. When startup or a command fails and cleanup also fails, the reported aggregate retains both failures and the original failure as its cause. applicationWorkflowCli.test.ts keeps one compiled CLI process for its JSON commands, including a non-zero status whose stdout is still a JSON document, and closes that process after the file.

The Apple artifact verifier step shares a job-temporary bytecode cache among its fresh CLI/MCP children via REA_ARTIFACT_NODE_COMPILE_CACHE. The artifact helper maps this setting to child NODE_COMPILE_CACHE; npm, Turbo, and nested Vitest runs do not inherit that setting. Explicit NODE_COMPILE_CACHE takes precedence, and coverage collection or NODE_DISABLE_COMPILE_CACHE=1 disables the helper's cache. The cache is discarded with the CI job.

To evaluate persistent compile caching for repeated local runs, opt in for both cold and warm measurements with an isolated cache:

NODE_COMPILE_CACHE=.cache/node-compile npm test

Do not report the warm result as a cold-suite improvement, and do not enable the cache in coverage or benchmark CI without first showing that its instrumentation remains equivalent.

Choose verification for the change

test:local is fast source-level feedback. For the strongest practical regression, prefer a real public CLI/MCP workflow, then production-boundary integration, then a producer golden; use the table to select quick feedback and claim-specific additional evidence. Provider-behavior claims require the matching real-provider lane.

Change or claim	Fast feedback	Claim-specific coverage
Source semantics	npm run test:local -- PATH...	Relevant boundary cases when the behavior crosses a port
CLI/MCP workflow	npm run test:focused -- PATH...	Actual public calls; an injected provider has limited scope
Documentation or generated metadata	npm run docs:check	npm run docs:build for rendered pages and links
Engine adapter	Its real-provider lane below	Host, version, target identity, and owned cleanup
Client configuration or schema conversion	Native client lane	Distinguish loopback models from live services
Agent investigation quality	Agent evaluation	Review transcript and scenario-specific evidence

Test design and pruning

Prefer real public CLI/MCP workflows, then production-boundary integration, then goldens captured from real producers. Keep module tests for distinct failures or semantics that stronger workflows cannot reliably reproduce. Paths, compiled imports, and suite names do not establish test depth.

Prune tests that mirror implementation details or duplicate a stronger claim. Before deleting a distinct regression, identify the replacement scenario and assertions. Preserve malformed formats, cancellation, cleanup, permissions, capacity, target identity, and combinations with distinct interactions. Trivial helper assertions need no replacement. Remove abandoned production scaffolding only after checking runtime and verifier consumers, including compiled imports.

Assert error codes, structured limit facts, source provenance, and observed recovery behavior instead of freezing explanatory sentences or banning obsolete phrases. Keep exact checks for protocol values, setting and parameter names, caller data, and diagnostic provenance. To verify guidance delivery, compare the consumer with the owning formatter or constant, alongside independent assertions for the behavior and facts; shared text alone is not a correctness oracle. Documentation wording belongs in review, not substring tests of source files. When replacing prose assertions, verify that a harmless wording change passes and an incorrect budget or lost fact still fails.

After an owner changes, move fault injection and event spies to the path the production consumer actually uses, and assert that the intended boundary was reached. Keep independent expectations from authored input, producer output, or an external oracle; comparing two paths through the same replacement algorithm does not preserve the old oracle. A negative probe inside a conditional or loop must assert that it ran.

Use representative scenarios rather than Cartesian matrices of equivalent cases. Share setup without hiding inputs or expected evidence. For successful Result values, throw the typed error before asserting the value; a preceding .ok assertion adds no coverage. Protocol fixtures must reject unmodeled commands and preserve actual producer reply shapes.

For comparison normalization, include a legitimate equivalent pair and a distinct pair that would collapse if representation tags or scope were lost. For learned identifiers, cover conflicting pairs and reused values in different collections; mappings must remain one-to-one within their declared scope.

For coverage propagation, compare the same returned rows under complete and incomplete examination. If a row exists on the left but is missing from a fully examined compatible right inventory, removal can be established. If the right side examined only eight of ten applicable records, or returns only one page, the same missing row remains unknown unless separate evidence proves absence. Assert the structured coverage and comparison status as well as the explanation.

For format compatibility, identify the producer revision that establishes each rule. Include an accepted boundary case beside the rejected case so a stricter parser cannot pass by refusing both. Keep source-derived behavior, synthetic fixtures, and executed producer or runtime oracles distinct in validation reports. A fixture accepted by REA proves its reader behavior; it does not prove that a real loader accepts the artifact. If multiple producer revisions share a format identifier but differ in behavior, record that ambiguity rather than using one installed version as an oracle for all of them.

Boundary regressions

For parser, inventory, snapshot, or projection fixes, select the applicable cases below. Assert the producing boundary's facts and their meaning in affected consumers. Add lifecycle regressions when resource ownership or finalization changes; a single malformed fixture cannot establish the whole contract.

Failure class	Regression that catches it
Format assumptions	Pair valid padding, offsets, or names containing delimiters with malformed input. Identify the producer revision when behavior varies by version; an unrelated runtime API's range is not an on-disk format limit.
Raw identity and normalization	Exercise allowed empty or unusual source names/metadata keys and values that normalize to one lookup or display value. Preserve admitted raw spelling under the existing redactions for transport credentials and explicitly marked sensitive values. Distinguish local paths, file URLs and HTTP paths; assert source identity through consumers.
Missing, empty, and unknown values	Distinguish omitted, null, empty, zero, and false where the format admits them. Pair a malformed optional field with usable sibling facts; assert uncertainty without inventing absence or discarding the row.
Completed work followed by failure	Complete one input, then fail or cancel a later input. Assert earlier digests, metadata, and parsed facts survive with the later failure reason and accurate processed/unprocessed coverage. For cancellation, assert the exact caller abort reason; where a primary failure was already produced, assert it survives.
Independent limits and failures	Trigger a sample/output limit alongside a read or parse failure. Assert both dimensions survive; reaching a limit does not erase sampling failures or prove complete coverage.
Representation expansion and capacity	Keep a valid set near the limit and a large malformed input followed by a valid input. Cover JSON escaping, expanded paths or decoded rows as applicable. Assert serialized bounds, retained identities, later valid results, and unknown omissions without charging discarded interpretations as retained output.
Snapshots, caches, and comparison	Vary metadata that affects the result while keeping payload bytes equal. Exercise historical records with newly introduced fields omitted, current caller inputs on replay, and reuse across differing selections. Assert correct invalidation or unknown coverage, current request provenance, and unchanged saved Evidence and IDs.
Public success and error delivery	Exercise affected public CLI/MCP paths, including partial Evidence on failure. Assert retained facts and named contracts; retrieve partial Evidence when MCP returns a reference. If SDK cancellation suppresses the reply, check server settlement and any retained Evidence. An error-code assertion alone misses lost observations.

For partial Evidence delivery, verify that same-session readback preserves identity and provenance without rerunning the provider. Include a no-writer control that keeps completed Evidence inline and preserves the original failure.

When producer order is unspecified, synchronize fixtures on the observed request or completion event. A filename, generated key, or fixed delay does not prove which item completed before cancellation. Preserve real producer ordering and use a controlled seam for the phase that needs deterministic failure.

Packed bridge validation uses Python's standard-library AST and tar readers with -I -S: caller-local modules, PYTHONPATH, and site startup hooks must not replace those readers or execute during validation. Controlled ast.py and sitecustomize.py fixtures must leave their markers absent and still reject malformed packed Python with SyntaxError. This checks static package verification, not real Hopper or Ghidra behavior.

Stateless source-analysis matrices may share a file-owned stdio MCP connection and retain representative compiled CLI parity checks. Every distinct source scenario still runs through the public workflow; session-lifecycle scenarios need their own sessions and cleanup assertions.

When a matrix already holds a named tool definition from live tools/list, pass that definition through the SDK's toolDefinition call option. This keeps the selected output-schema validation without eagerly compiling unrelated catalog schemas. Re-read definitions if the server advertises a changed list.

For exact nested input constraints, select the full profile explicitly or use the full-SDK conformance suite that owns the field. Compact workflow tests should assert canonical failure categories and provider-call controls where rejection before provider work matters. Do not make today's omission of a constraint a compact-profile invariant.

When a file repeatedly validates the same MCP schemas, use createMcpSchemaValidation for file-owned preparation and clear it in afterAll. Its content keys include the complete schema, and the SDK provider keeps dialect and format validation. Keep sessions and connections scoped to each test; share immutable schema preparation through the fixture's validator dependency.

Artifact verification uses artifactMcpCall for both lifecycle and analysis calls. A fresh SDK client has no held tool listing, so a bare callTool can skip output-schema validation. The helper discovers actual advertised definitions, rejects missing or ambiguous schemas, and validates the requested tool without compiling unrelated catalog schemas. REA registers process-static tool contracts; bindings share one discovery per negotiated connection and are discarded on reconnect or failed discovery. Keep raw artifact calls on this shared path and retain explicit request options such as timeouts. Caller-supplied tool definitions are rejected: the helper owns selection from the advertised catalog so a stale or permissive schema cannot bypass it.

The regression fixture changes only canonical_path in a real stdio response to a number. The old helper accepts that response; the named schema must reject it. This client-boundary fault injection does not establish native Apple behavior, which remains covered by the macOS verification lane.

Report unavailable authority as a named skip. Await owned cleanup before a verifier reports success. Native process inspectors are released at Vitest worker teardown; fork termination does not run Node exit hooks, and per-file teardown would retire inspectors still needed by other files.

Measure slow cases before pruning capacity coverage. Retain inputs beyond the formerly failing size/depth and distinguish cold runs, warm builds, coverage, and toolchain conditions when comparing timings.

Lifecycle regressions

For changes to resource acquisition, readiness, cancellation, or cleanup, select the affected owners and applicable scenarios below. Include temporary snapshots and one-shot commands. Use public CLI/MCP workflows where practical and a controlled production seam for interleavings that cannot be reproduced reliably.

Scenario	Assertion and control
Target admitted; startup still pending	Status preserves the readiness distinction and observed startup activity; readiness requires the provider's documented proof.
First handshake is late, absent, invalid, or recovered during close	Initial and recovery paths retain the same authenticated identity facts. Foreign run/token controls cannot authorize cleanup; authenticated idle/native work survives the unauthenticated listener deadline.
Launch returns an error with a valid process handle	Cover already-exited and still-live owned processes. Verify observed exit separately from creation; failed termination or unavailable identity remains unresolved.
Cancellation arrives after workload exit, during result reads or temporary-root removal	Return typed cancellation with available partial observations and actual cleanup status. The non-aborted control succeeds; existing primary/cleanup failures keep their precedence.
Children have mixed health or cleanup outcomes	Missing peer health does not erase an observed failure or activity. Idle plus unknown stays unknown. Retry a failed child without closing a successful peer again, including after a close-option change.
Cleanup fails after success or primary failure	Public output retains the completed observation or primary diagnostic, remaining resources, and owner recovery action. Restore the denied resource and verify the owning cleanup retry.
Cleanup owners recover after an error was returned	Retain the original child and aggregate errors and project them after each owner releases. A successful peer stays released; the aggregate remains incomplete until all owners release. Preserve the first cause even when rejection is undefined.
Target exits without an acknowledgement, or transport disappears while it stays alive	Positive owned exit proof resolves target lifetime; other resources still need cleanup. Lost transport alone does not prove exit. Verify unrelated/attached targets survive and same-connection recovery works when supported.

Shared process-supervision controls separate leader exit from output closure. Cover output during held or failed cleanup, bounded retention and drain timeout, retry, and borrowed stream authority. The POSIX inherited-stderr control uses a separate pipe as a release barrier after leader exit; it does not establish native Windows Job Object behavior.

Synchronize cancellation with the finalization boundary being tested; a fixed delay after dispatch does not establish that phase. Use markers, barriers, or owned cleanup seams while preserving real cleanup where feasible. Check server settlement and retained resources even if the MCP SDK rejects the client promise before the handler finishes. A cancellation during active execution does not cover cancellation during final cleanup. For ZIP extraction, replace the pathname after inventory reader close while the admitted root descriptor remains owned. Verify that identical member bytes in a different carrier, a directory, or a missing pathname cannot switch the materialized source. Exercise descriptor close failure and retry before output publication. Use explicit abort reasons and assert their identity at admission and after provider completion, alongside retained observations and cleanup state.

Electron fuse regressions cover actual descriptor close denial and retry through the built CLI, same-connection MCP recovery, mixed descriptor outcomes, and server shutdown. Controlled seams cover cancellation during final close and after reader finalization, plus active-provider shutdown during preflight. They preserve real filesystem cleanup where possible; mocked launch boundaries do not establish real Electron teardown or naturally occurring OS failures.

Active Electron cleanup controls retain independent transport and process-family owners, block replacement captures until recovery, and retry only unresolved owners. Cover late close acknowledgement and rejection after a wait deadline, opaque rejection reasons, cancellation during failed finalization, and missing lineage followed by a confirmed passive family census. Windows controls require a fresh lineage observation and do not signal cached descendant PIDs after root termination. Simulated Windows controls do not establish native Windows behavior.

Native process-ownership controls distinguish start identity, zombie lifecycle, current parent/group relation, and token-read evidence. Retain generation-bound candidates through smaller lineage observations, table omissions, reparenting, group changes, and failed cleanup retries. Identityless exit-refresh controls cover a replacement row that is not linked to the prior descendant set: a matching token on its own group is signaled, a foreign or absent token is not, an unreadable live token stays unresolved, a replacement that appears only in the confirmation table is still token-checked, a group discovered during revalidation is signaled before cleanup succeeds, a member that changes groups during confirmation is signaled in its new group, a PID first seen with a foreign token is read again on the confirmation table, a PID whose token was already read is read again even when its parent and group return to the pre-exit values, a pre-existing PID is read again when its parent or group changes, an unchanged pre-existing row that was never token-read stays unread, an unbounded confirmation chain stays unresolved, lineage reads a row again when its launcher-tree membership changes, a cached lineage token is read again on the next refresh, an initial descendant whose token already matched is read again on a later refresh, the launcher is read again on that refresh and a missing, unowned, or relation-changed launcher PID stays unavailable, a descendant absent from the latest refresh is omitted, a group that reappears after it was processed is revalidated once, a further reappearance stays unresolved, lineage publishes one row per PID, and lineage does not publish a verified empty descendant list while that row remains unexamined. A readable token with unavailable identity stays unresolved. Native UID facts, rather than an earlier process-table row, determine whether a token denial concerns another user. Darwin tests compile the helper and exercise snapshot growth, absence, lifecycle, and UID observations; controlled Linux procfs tests do not establish real Linux exit or PID-reuse behavior.

For atomic writes, exercise cancellation on both sides of the documented commit point and preserve any publication receipt. A late interruption does not prove that no output was published.

For successful provider reports, test cancellation immediately before the report reaches the application workflow as well as during provider cleanup. Validate the report before classifying completion: a valid report must retain its Evidence on cancellation, and an invalid report must keep its output error. Exercise both the provider boundary and the shared CLI/MCP workflow; a downstream guard does not repair the provider's own completion contract. When callers share startup, cancel each waiter independently and verify that the last departure awaits owned cleanup while interested peers can still reach readiness. Repeat the same shutdown event after a cleanup failure to verify that its listener can retry. Exercise non-default lifecycle deadlines through queued requests and their consumers; default-only controls cannot detect dropped request options.

For descriptor-backed reads, inject a close failure after a successful read and after a primary read or cancellation failure. Verify that the exact descriptor remains owned, both failures survive, and a later close retries it. Repeat admission attempts while close still fails: no additional descriptor or native command should start. Restore close and verify that the exact owner resolves the originating cleanup marker and allows later work, while the earlier failure remains available as a diagnostic. Retain errors returned by multiple failed retries, including admission and shutdown aggregates, and assert that each becomes complete after recovery. For an aggregate with multiple owners, release one owner while another still fails and assert that it remains incomplete. Cover recovery before the caller-specific error is bound to its retained owner, as another request can retry cleanup concurrently. Confirm that an already released owner resolves a later binding too. Include admission failure after opening the descriptor. If caller cancellation returns before an inspection settles, gate its final read or close and verify that session shutdown waits for the owned work and reports any retained resources. Gate a later read in a multi-read inspection, not only its first read. Overlap operations so one retains a failed descriptor while another reaches its next read; both the operation and shutdown must settle without nested-admission deadlock. Deny child close and verify its containing root survives until retry.

Exercise publication on filesystems without hard links using an unsupported-link seam. Check exclusive destination creation, stat/write/close failures, completed output receipts, and cancellation at the documented cutoff. A streamed fallback has different visibility from an atomic hard-link commit and must report it accurately. For Windows command supervision, keep a descendant writing after its leader exits, deny cleanup observation, and verify a competing writer remains excluded until the exact Job Object closes. Portable doubles cover retry logic; the native Windows lane must establish actual process-tree behavior.

Supervisor tests must observe process exit, output-pipe closure, and IPC disconnection separately. In the pinned Node runtime, a parent-initiated IPC disconnect can omit the aggregate ChildProcess close event. Join the actual resource observations before reporting completion; process exit alone does not establish pipe or process-tree release.

When changing an internal helper contract, check its JavaScript verifier and subprocess-script consumers alongside TypeScript references. Source typecheck does not check those calls; run their orchestration tests against the built modules, including the verifier's failure and cleanup paths.

Representative regressions include process cleanup-time SIGINT, late authenticated cleanup, LLDB launch and finalization, composed child ownership, and public CLI cleanup denial. These controlled probes do not replace the matching real-provider lane. Record the base failure, passing control, exact host/provider, verified cleanup, and unavailable coverage; a missing engine is not a pass.

Behavioral depths

Depth	Location	What it proves
Module	src/**/*.test.ts	One domain, service, or adapter through its narrow public surface
Composition	tests/composition/**	Provider-neutral session and registry wiring; filesystem access is limited to fixture materialization
Boundary	tests/boundary/**	Exactly one production filesystem, process, network, browser, CLI, or provider boundary
MCP boundary	tests/boundary/mcp/**	MCP transport and tool-contract boundaries with isolated sessions and explicit cleanup
Acceptance	tests/acceptance/**	A compiled CLI or MCP journey; injected providers still make it integration
Conformance	tests/conformance/**	Shared provider contracts, parameterized by declared capabilities and explicit opt-outs
Evaluation	tests/evaluation/**	Deterministic evaluator parsing, scoring, and report generation

Colocated domain tests may import only their owning domain and inward dependencies. They do not construct sessions or servers and do not start subprocesses or network listeners. Colocated application tests exercise one service through explicit ports and recording adapters, never a CLI or MCP entrypoint. Composition tests may assemble provider-neutral sessions and registries but do not cross production filesystem, process, socket, or browser boundaries. Boundary tests cross one production boundary. Only acceptance tests assemble the complete runtime or invoke the compiled product surface.

Focused immutable builders and recording ports shared by one test family live beside their production owner as src/**/*.fixture.ts. They are typechecked with the suite and excluded from package builds; broader runtime and provider fixtures remain under tests/fixtures/**.

tests/process-global/** is reserved for cases with a demonstrated dependency on process-global state. Those tests use isolated forks so environment and exit-status changes cannot leak between files. Serialize a case only when it demonstrably shares an external resource that cannot be isolated. Reusable, test-scoped fixtures live under tests/support/**; immutable source artifacts remain under tests/fixtures/**. The process-global Vitest configuration contract rejects new direct temporary-root creation outside the workspace seam and its narrowly documented boundary/package exceptions.

The shared temporary-directory and workspace helpers preserve caller labels under an owned rea- prefix when needed. Their cleanup ownership differs: createTestTempDirectory registers onTestFinished deletion, while createTestWorkspace requires caller cleanup. Close resources before removing their directory. npm run verify:test-temp-hygiene uses the same ownership rule to find retained fixtures under a private temporary root. Third-party caches remain outside that ownership rule.

Real Hopper, Ghidra, IDA, browser, package, and managed-code claims belong to their explicit npm run verify:* lanes. The reconstruction-readiness lane separates observed native CLI inspection and JavaScript/Electron static CLI/MCP analysis from synthetic report-evaluator conformance. Its determinism, tamper, and stale-source checks concern the synthetic evaluator; they do not execute fixture applications or verify runtime authority, cleanup, comparison, or reconstruction closure. The evaluator and its semantic tests belong to tests/conformance/readiness/report/, outside the shipped runtime. When application runtime behavior is needed, exercise the actual target through browser, Electron, or process capture. Real model trials are manual; Vitest covers deterministic evaluator logic.

verify:managed runs the portable PE byte-fixture conformance entrypoint under scripts/verify/managed/, with its byte builder under scripts/fixtures/managed/. It checks static classification, members, reconstruction, native-boundary relationships and application graphs without executing fixture PE files. Operator-local manifests and actual ILSpy oracles remain optional, separately reported checks; the real Ghidra NativeAOT lane has its own toolchain prerequisites. See the managed guide for those configurations. Generated completion-ledger checks use the same owning entrypoint and include its verifier/fixture files in their cache inputs.

End-to-end, integration and golden evidence

Prefer the public CLI or MCP workflow when it can observe the required behavior. Use production-boundary integration for contracts the complete workflow cannot reliably exercise, and producer goldens for format-specific facts. An acceptance case with an injected provider verifies that integration boundary; it does not establish real-provider behavior.

For installed-client versions, executable selectors, disposable profiles, model fixtures, and detailed receipts, see client verification. Use real-toolchain lanes for engines and agent evaluation for investigation quality. Report unavailable tools or authority explicitly and verify owned cleanup before claiming success.

Coverage and timing

CI owns coverage; aggregate and domain/contract thresholds are maintained in vitest.config.ts. They are glob-specific and never updated automatically. Coverage does not replace named boundary or real-provider scenarios.

The reverse-tools commands run the focused tests and report coverage and mutation results without an additional percentage requirement. Test failures, invalid mutation configuration, and runner failures still fail the check. Mutation targets cover reviewed production behavior, including signal-terminated commands, Cutter descriptor identity, debugger cleanup ownership, GDB command exit and cancellation, and Rizin startup, timeout Evidence, and process exit. A second exhaustive pass runs the focused Cutter and Rizin timeout semantics tests against every compilable selected mutant to avoid relying on per-test mutation coverage associations; Stryker reports compile-error mutants separately because those cannot execute tests. Both passes resolve named syntax owners, scoped declarations, guards, callbacks, and reviewed expressions from the current TypeScript source; no hand-maintained line ranges select mutations. Missing, ambiguous, or malformed source targets stop configuration before mutation generation. The selections retain their review scope rather than expanding to whole source files. The focused suite also exercises affected MCP routes and the Python Cutter bridge. Contract, integration, and real-provider lanes complement these focused tests; simulated providers do not establish real-provider behavior.

Required CI retains fresh reverse-tools coverage and preflights both canonical Stryker configurations with --dryRunOnly. These preflights resolve the owned targets, instrument the source, and execute the real initial tests. Invalid configuration, target resolution, and initial test failures still fail the job. Preflights execute no mutants and produce no mutation-outcome report. Both fast feedback and full qualification, including exact-revision release qualification, require coverage and both preflights. A failed, canceled, skipped or missing selected quality job still fails the required gate.

The separate Mutation sweep workflow runs both complete configurations daily and on manual dispatch, in two isolated jobs. It is a maintenance check outside the PR and release publication gates. ci:full requests full regression, coverage and provider qualification; it does not request a complete mutation sweep. Use the Mutation sweep workflow to request that result. Targets, checkers, concurrency, deadlines and the existing non-blocking mutation-score threshold are unchanged. npm run test:mutation:reverse-tools also retains both complete passes for local callers.

The two passes write separate JSON and HTML reports under reports/mutation/: reverse-tools and reverse-tools-semantics. Required CI retains focused coverage; the sweep retains each configuration's JSON/HTML reports separately. Both retain available diagnostics for seven days after failure and skip uploads on cancellation. Artifact names identify the tier or configuration and run attempt. A successful sweep requires both jobs and their nonempty reports; partial diagnostics and preflights do not establish complete mutation outcomes or qualify the release revision for mutation coverage.

Fast CI starts the source-only projects after classification, independently of build and static checks, and runs eight built-dependent shards after the build. Those nine test jobs still select every portable test. Full qualification keeps six complete Vitest shards with coverage. Neither tier retries tests. Canonical project ownership lives in scripts/lib/test-projects.mjs, shared by Vitest, CI group execution, and development-test preparation. A test importing compiled verifier code belongs to the built group even if its directory is conformance. Source and built blobs have separate labels to prevent merge collisions. Each test job emits a blob report; the merge job produces JUnit and JSON diagnostics, a module timing report, plus aggregate coverage during full qualification. It retains available reports even when a shard fails, uploads available diagnostics, and writes the slowest files to the workflow summary. Partial reports are marked diagnostic only. Failed, cancelled, or missing selected shards still fail the required gate, and aggregate coverage thresholds remain enforced in full qualification. Only a successful complete merge on the default branch publishes shard history, and only from a push, schedule, or manual run. Fast runs publish test-timings-fast. Full runs publish test-timings-full. Pull requests do not publish it: the lookup reads one unpaginated artifact page, and pull-request artifacts would crowd out the default branch. The build job downloads the artifact that matches its tier, so fast shards use durations measured without coverage and full shards keep coverage-weighted durations. A tier with no qualifying artifact leaves history unset. The download is that small checkout-relative timing file, not the coverage/JUnit bundle. Timing weights come from Vitest module diagnostics and include tests and hooks, worker preparation, environment, imports, and setup. Import timings already include transform waits, which are not added again. The weights describe file costs under the measured worker and coverage settings; their sum is not workflow elapsed time. The diagnostic artifact retains the phase breakdown, while shard history keeps only each file's total. Legacy test-body-only history and missing history fall back to native sharding. Static checks, documentation, build, package verification, and real-system lanes remain separate jobs so one kind of evidence cannot stand in for another. Workflow syntax and expression validation runs inside the required static-check job when workflow changes select it, avoiding a separate runner. Native, provider, fixture, SQLite compatibility, and website lanes wait for the fast static checks (typecheck, built lint, and format) to succeed before allocating their runners. Knip, duplication, architecture guards, test discovery, and the Python producer run in the separate full-tier deep static job and do not gate provider admission. Portable test shards and documentation checks remain independent of static results, so they can still provide diagnostic feedback. The required gate rejects skipped selected lanes when static checks fail; on the full tier it also requires the deep static job. Passing revisions retain all selected coverage. This ordering reduces peak demand on invalid revisions but can add static-check waiting to otherwise passing runs; it is not a measured reduction in end-to-end latency.

Core jobs have execution limits of five minutes for classification, ten minutes for static checks, builds, docs, and report merging, and thirty minutes per portable test shard. These limits leave headroom for cold setup and full coverage while preventing a hung job from holding the active main run for hours. They do not bound runner queue waiting. Timeouts still fail the required gate.

The separate CI timings workflow also diagnoses missing, startup-failed, queued/running, approval-required, canceled, and failed PR CI through bounded, read-only API scans. Incomplete pagination or changing identity stays unknown. It never closes/reopens PRs, reruns workflows, posts comments, or changes gates. The timing manifest retains each run's event and head branch, including explicit unknowns, so PR runs cannot consume the eligible-main comparison budget. Its static failure report compares supported lint/duplication diagnostics with failed main runs using exact source, ancestry, task hashes, and toolchain proof. A match describes those diagnostics only; it does not establish that every failure in a job came from main. Duplication reports retain the actual checker exit result, so passing duplication checks preserve lint failures and caller thresholds remain owned by the producer. Missing checker outcomes leave those duplication diagnostics unclassified. Missing historical receipts remain unknown. The reporter executes only trusted default-branch code and parses whitelisted JSON artifact members without executing artifact content. Reports and limits are retained in ci-health-diagnostics; its ten-minute job limit covers the existing four-minute timing collection plus the bounded diagnostic stages.

Fast built shards target under ten minutes of uncontended shard-job execution. That figure is the test-shard job. The job waits for classification and the build, and the portable-test merge runs after the slowest shard. On 2026-10-20, run 39451548945 measured those surrounding jobs with runner queue excluded: classification 15 seconds, build 49 seconds, and the portable-test merge 29 seconds. Shard 1/5 of that run spent 22 seconds before built Vitest and 3 seconds after it, including npm ci at 9 seconds. node_modules is about 674 MiB, so publishing it as a shared artifact would transfer more data than that install saves. The npm download cache stays the setup cache.

File timings from that run put the built suite at 7944 seconds of module time. The sequencer keeps each file's full duration on one worker. Extra workers overlap other files in the same sequence group, and process-boundary runs after that group. Scheduling the unsplit inventory onto five shards peaks at 820 seconds of tests. That run's slowest built shard job lasted 925 seconds (shard 4/5), and shard 1/5 spent 874 seconds in built Vitest, so hosted job time sits above the file schedule. Six shards of the same inventory peak at 684 seconds of tests. Eight shards, after the 578-second loader file is split by case count (54 cases stay in that file and 46 move into each of three files), peak at 513 seconds of tests. With the measured 25-second shard setup, that shard job is about 538 seconds. Adding the measured classification, build, and merge makes an uncontended portable critical path of about 631 seconds, still before the gap between this file schedule and hosted Vitest. Runner queue is separate and grows when more shard jobs compete for runners. process-boundary stays serial because process-tree sampling shares the host. Raising the worker cap from two to four on this four-core host cut a six-file acceptance sample from 96–101 seconds to 71 seconds and did not remove the single-file floor, so the cap stays two. Full qualification keeps six covered shards; its longer instrumentation is outside this fast budget.

The separate CI timings workflow batches recent completed CI and Release runs hourly or on manual dispatch. It uploads ci-job-timings-batch with raw API pages and a normalized point-in-time report for each collected workflow attempt, plus a batch manifest describing examined coverage, failures, and limits. Overlapping lookback windows may collect an attempt again; identify records by run ID and attempt. Each batch examines at most 100 completed runs per workflow created in the preceding 24 hours, with a four-minute API collection budget. It collects the latest attempt of each selected run and does not claim an exhaustive archive. It runs the reporter from the trusted default branch with read-only permissions, treating workflow runs as API data. It does not allocate a new reporter job for every completed, cancelled, or skipped workflow. Reports separate queue waiting (created_at to started_at) from completed execution and report percentiles plus matched npm ci and artifact-step durations. Queue excludes dependency waiting before job creation and does not establish whether account concurrency or runner availability caused the wait. Queued jobs with placeholder start times, skipped jobs, and missing or invalid timestamps remain unknown. The reporter targets each selected run and attempt explicitly. Summed job elapsed time is parallel work, not workflow wall time, CPU time, memory usage, or billing. Step names identify samples, not complete transfer costs. Its queue, execution, and failures are independent of the required gate and publication. CPU time and peak process-family memory are not measured by this API.

The build job saves portable Turbo outputs across runs. Eight Linux real-system lanes reuse its current-run artifact; standalone dispatches restore the cross-run cache. The shared action derives its key from test:prepare --dry=json task hashes, runner OS and architecture, .npmrc, .nvmrc, and the lockfile; test-only changes can reuse compiled outputs. Every build and verification command still runs, so Turbo checks task inputs and rebuilds misses. A restore prefix with the same platform and toolchain permits partial task reuse when one generator's inputs change; Turbo still invalidates that generator's entry. Native outputs are not stored in this cache.

Native package jobs, the Windows curated lane, and the Apple artifact lane reuse the same workflow's Turbo cache for portable compiled JavaScript, skills, and catalogs. The key parser retains the validated test:prepare task identities in .cache/ci-build-plan.json; staging reads this plan without starting another Turbo run that could evict cache entries. The producer stages matching archives, metadata, and available manifests into .cache/ci-build-cache/. portable-build-cache.json records availability per task. Evicted or partially missing entries are omitted as whole entries, and consumers rebuild cache misses, including when every entry is unavailable. Malformed plans, non-file entries, and I/O errors other than disappearance still fail publication. After a trusted static run, superseded cross-run Turbo snapshots for that OS and architecture are deleted down to the two newest generations. The Windows lane also runs the official compiler's complete source typecheck with two checker threads, independently of the accelerated Linux checker. Both lanes retain the normal build and prepack commands, which restore matching outputs or rebuild on a cache miss. Native artifacts, dependency installs, package archives, and runtime verification remain host-local. Keep platform-dependent outputs out of this shared cache.

The Windows native verifier's private-runtime ACL observer reads the owner and access rules only in SID form, checks that every identity it compares is a SecurityIdentifier, and never uses Get-Acl or the convenience Access property, which resolves account names. Its Windows PowerShell observers call .NET members directly and build their JSON without cmdlets, pipelines, module autoloading, or type-table updates. The private-runtime-acl stderr markers report the powershell.exe process uptime and each completed phase; inspect them on a timeout before attributing it to startup, ACL reads, or identity lookup. The 60-second observer deadline bounds a hung observer while covering a cold powershell.exe start on a starved hosted runner. All ownership, DACL, and readonly assertions remain in force. A passing retry alone does not explain a previous timeout.

Record cold and warm preparation separately from fresh test execution when benchmarking the optional npm run check:pr gate. Compare multiple runs on the same host, including subprocess and queue costs; a cached pass is not a test runtime measurement. A PR that touches packaging or real-system behavior requires the applicable verify:* lanes; packaging and installation changes also require npm run verify:package. The full-gate benchmark measures that explicit lane, not the routine iteration requirement.

The package-update registry fixture retains its first proxy/body failure and aborts the owned updater command. It joins the helper and installer descendants before workspace removal. Incomplete process cleanup retains the exact owner and workspace rather than reporting successful removal. The controlled packageUpdateRegistryFailure regression needs built process modules, uses no public registry, and checks both the primary failure and live descendant absence.

Real-toolchain verification lanes

Linux Rizin debugger ownership changes require a separate real-provider gate on Rizin 0.7.4. Run npm run test:focused -- tests/boundary/providers/RizinInferiorTty.test.ts. This test verifies the caller-selected TTY against the actual inferior's file descriptors, confirms dbg.exitkills=false, detaches through the olj/o- protocol, and checks that the target survives debugger close and writes to the selected TTY. The suite is included in the reverse-tools Vitest and mutation typecheck selections; its real-provider assertions skip when Rizin 0.7.4 is unavailable. Run the focused command above on a host with that provider for the explicit real-provider verification gate.

Each real-toolchain command must require only the host tools needed to prove its stated claim. Use a host-native fixture for host/provider acceptance, and place optional cross-target formats or platform-specific runners in separate commands. Check prerequisites before starting expensive work and name the missing command, target, and lane in any failure message. A lane must not imply that a host or target is covered when it was skipped.

Provider admission accepts Ghidra 12.x (>=12.1) and the JDK range declared by that installation (application.java.min through application.java.max). Current 12.1 releases require JDK 21 or newer and set no maximum. The lanes below still prove behavior on the verified Ghidra 12.1.4 and JDK 21 build.

Ghidra lane	Supported runner/target	Additional local tools
npm run verify:ghidra	Linux x64/arm64 ELF or macOS x64/arm64 Mach-O	Host C compiler, Ghidra 12.1.4, and full JDK 21
npm run verify:ghidra:seeds	POSIX; raw x86 fixture; seed ordering, BSS labels, analyzer readback, not-applicable and unknown analyzer names	Ghidra 12.1.4 and full JDK 21
npm run verify:ghidra:swift	macOS x64/arm64 Mach-O	Host Swift compiler, Ghidra 12.1.4, and full JDK 21
npm run verify:ghidra:riscv	Linux/macOS x64/arm64 host; source-owned RV32/RV64 ELF through CLI and MCP	RISC-V Clang, LLD, LLVM readelf, Ghidra 12.1.4, and full JDK 21
npm run verify:ghidra:switch	Linux x64 ELF; GCC/Clang optimized and stripped switch fixtures	GCC, Clang, GNU nm/objdump/strip, Ghidra 12.1.4, and full JDK 21
npm run verify:ghidra:aarch64-jump-table	Any supported Ghidra host; AArch64 ELF; signed/unsigned byte/halfword tables; host ARM64 Mach-O	Clang with AArch64 target support, Ghidra 12.1.4, and full JDK 21
npm run verify:ghidra:cross-format	Any supported Ghidra host; also analyzes AArch64 ELF, x86-64 PE, and x86-64 Mach-O	clang, LLD, and lld-link in addition to host-lane prerequisites
npm run verify:ghidra:windows	Controlled Windows x64 with native x86-64 PE; -- --x86 selects native x86 PE	Ghidra 12.1.4, full JDK 21, and the matching native artifact
npm run verify:ghidra:dol	Any supported Ghidra host; source-owned PowerPC GameCube DOL fixture built from a linker script (never executed)	Clang with PowerPC target, LLD, GNU readelf, Ghidra 12.1.4, and full JDK 21

The optional Swift lane compiles a source-owned fixture and checks modern and provider-demangled procedure names through real stdio MCP, with and without the dSYM companion. It checks unfiltered, structs, and literal-name filters, same-address alias evidence, deduplication, and explicit unresolved categories. It does not add a Swift prerequisite to the host C fixture lane.

The host lane also verifies namespaced C++ ABI symbols and annotation identity. That facet requires Ghidra's matching demangler_gnu_v2_41 native component in GPL/DemanglerGnu/os/<platform>/ or build/os/<platform>/, in addition to the native decompiler. The namespace verifier reports missing or non-executable components before testing the fixture. REA does not build or install them.

Windows native conformance runs with npm run verify:windows-native and does not require Ghidra or Java. It includes Job Object commit-limit controls (a limited child that reaches the limit, a control that does not, and invalid limit rejection). An optional independently compiled Windows fixture adds in-place reparse, breakaway, environment, and token observations. The separate scripts/verify-windows-native-inspection.mjs lane uses the .cache/windows-native-inspection-fixture artifact built on Linux with node scripts/build-windows-native.mjs --inspection-fixture. Its controlled first deletion failure checks retained NTFS ownership across a same-volume rename, later-query handle-based recovery, successful-admission caching, and environment-teardown cleanup. It neither establishes the cause of the intermittent #1701 host failure nor replaces verification with the shipped production artifact. npm run verify:ghidra:windows:package additionally packs and installs REA into an isolated prefix and runs ordinary-user CLI/MCP operations against their canonical schemas. Run the controlled fixture generator before that lane, or supply an installed package root and an explicit fixture as arguments. The host-native Ghidra lane also verifies native value tracing through the production CLI and a separate stdio MCP process. It compares complete dependency graphs, validates Evidence and upstream/workflow profiles, checks capability discovery, and closes the MCP session. No provider or transport is mocked. It also validates every advertised input/output JSON Schema and the exercised MCP outputs, probes address spelling and name/address ambiguity, and checks direct versus targetless calls, byte-read completeness, invalid input diagnostics, CLI/MCP parity, atomic annotation rollback, refreshed inventories, unchanged executable bytes, and discarded edits after reopen. A deliberately long temporary path exercises private Unix socket allocation and cleanup, including cancellation after a real headless process launches. Native annotation probes reject NUL and unpaired Unicode surrogates without partial edits or a broken bridge, preserve supported Unicode and control text, and check lossless malformed-text diagnostics. Memory-to-file mapping is checked against original artifact bytes. The fixture also stores a pointer one byte past a function entry; exact xrefs, raw procedure references, and CLI/MCP dossiers must retain that data edge. A valid legacy snapshot reconstructs the former omitted edge under its older profile; CLI and MCP must reject that binding with a mismatch reason and recovery advice. The rejected open must preserve the active live session. Exact external entries must resolve while retaining an empty body; unknown external addresses remain unresolved and external annotations are rejected. An adversarial regex over a full 12 KB literal must report stack exhaustion as a resource constraint, preserve live annotations, and allow complete literal searches afterward; CLI and MCP must agree on both results and recovery advice. Real snapshot lifecycle checks retain edited API results as Evidence while rejecting immutable snapshot saves and imports before and after a repeated open of the same target. They verify unchanged live annotations and run identity, absent rejected output files, an unchanged source snapshot, and successful snapshot import/save after closing and recreating the database. The pristine snapshot is written by an independent real CLI session. They also start a real annotation and snapshot close concurrently: the edit must succeed, the snapshot must be rejected without creating a file, and the edited session must remain usable until explicitly closed. Source-admission probes change a caller-owned fixture after open_binary but before the first Ghidra query. They require an actionable artifact_changed error preserving both digests and the selected path, unchanged provider availability, failed-copy cleanup, and successful recovery after reopening. After import, deleting that source must preserve the captured database identity. Instruction inspection and containing-function lookup also agree across hexadecimal case, leading zeros, and encoded default address-space spellings. The same source-acquisition workflow exercises missing and directory-replaced inputs, plus real read-permission denial on a host that enforces it. Permission fixtures must probe the relevant read, write or search operation before relying on POSIX mode bits: a non-root Linux process may bypass DAC through capabilities. requirePermissionDenial reports an explicit skip when the host permits the probe; it restores the probe's permissions before skipping. Such a skip leaves real permission coverage unverified. Filesystem integration separately checks selected-platform routing and exclusive creation. Namespace annotation probes compile a separate host C fixture with C++ ABI symbols, avoiding a C++ runtime prerequisite. Real Ghidra demangling supplies duplicate leaf names in two top-level namespaces and a nested namespace. The workflow verifies leaf and qualified renames, repeated reuse of fully qualified readback, lookup by the returned name, literal namespace-like leaf names, rejection of empty qualified leaf names without changing comments, CLI behavior, and independent CLI/MCP database ownership. Large-result probes compile initialized host-native data sized from the pinned MCP SDK receive budget. Real byte reads, annotation edits and function dossiers exceed that budget while preserving the connection and active analysis run. Successful-result and oversized-error delivery constraints must identify their successfully retained Evidence records; export must recover every source byte, complete annotation and original error diagnostic, with CLI parity and an unchanged executable. Formatter integration separately checks unacknowledged Evidence recording. Long ordinary procedure names and encoded address-space selectors must produce normal validation errors without exhausting Java's regex stack or losing the private bridge connection. Each rejection is followed by a real provider lookup; address-like literal names still resolve exactly after annotation. Malformed annotation readback, memory completeness, and inventory data remain separate SDK/provider integration cases; success from a real provider cannot establish rejection of a contradictory provider response.

The Linux switch lane checks dense, sparse-with-holes, shared-body, nonzero, negative, and nonexact JSON integer labels plus a comparison-only control. Independent source labels, ELF file bytes, table slots, and bounds branches define expected case/default destinations. Production CLI and MCP must agree; debug labels must retain their signed values. For stripped negative fixtures, the independently checked 32-bit dispatch permits equivalent unsigned labels only alongside the reported low-confidence undefined4 parameter; original source signedness remains unknown and the ABI residual must remain visible. unsafe labels remain unresolved and every recovered destination is retained. The compiler oracle also injects malformed records to check that its assertions reject missing/default-confused labels, wrong destinations, and numeric guesses. It also invokes the actual bridge methods on detached Ghidra model objects to check ambiguous dispatches, signed literals, precision bounds, shared targets, and conflicting labels. This reflection fixture depends on the pinned Ghidra model and does not claim a compiler produced those synthetic graph shapes. Pass --entrypoint /path/to/installed/rea-agents/scripts/rea.mjs directly to scripts/verify-real-ghidra-switch.mjs to verify an installed package through the same compiler oracles and CLI/MCP checks.

The cross-format Ghidra lane also analyzes an optimized AArch64 ELF switch fixture. It checks the recovered case values against the source cases and requires unresolved table bounds or case mappings to remain visible as residual unknowns.

npm run verify:inspector requires the supported Node.js runtime and installed REA dependencies. CI runs it on Linux and macOS x64/arm64 and Windows x64. It starts owned loopback Node Inspector fixtures and verifies discovery and passive observation through the CLI and stdio MCP, including special filenames, unresolved discovery locations, and independently resolved loaded scripts. Double-quote filenames are tested on POSIX only because Windows does not support them.

Node Inspector verification does not establish Electron GUI or other engine behavior. Browser and Inspector share CDP transport/value and file-location helpers; their producer workflows remain distinct.

Android APK analysis

The real Android lane also runs on Windows x64 with its matching bundled native controls. It verifies bidirectional protocol input, CLI/MCP result parity, real MCP cancellation/disconnect and Java exit after abrupt CLI owner termination. The forced Windows exit does not exercise the POSIX SIGTERM handler and can leave temporary workspace files. verify:windows-native separately checks binary input, backpressure, EOF and pending-write job closure.

npm run verify:android requires an existing Java 17+ and an explicit REA_JADX_MCP_JAR for jadx-headless-mcp 0.7.1. Set REA_ANDROID_TEST_APK to the fixed public ApiDemos v6.0.18 fixture. Obtain both with the explicit npm run fixtures:android command; files are SHA-256 verified and kept under ignored _reference/. No Gradle build, Android SDK, emulator or application execution is required. The lane compares real CLI/MCP package, class search, class inventory, method decompilation and incoming references. See Android analysis for boundaries and resource budgets.

The Real Android metadata bridge workflow compiles the Java bridge and runs its metadata regressions against the same checksum-pinned engine and public APK. It runs on branch pushes affecting its bridge, fixture or engine inputs, and can also be dispatched manually or called by another workflow. It uses the runner's JDK with two active processors and a 512 MiB heap; it needs no npm dependency installation. This lane checks exact raw/display class selectors, raw member names and field DEX types used by mapping lookups, and the existing cold-metadata regressions. It does not establish CLI/MCP parity or application execution behavior. The broader verify:android lane checks CLI/MCP parity; neither lane runs the APK.

verify:jeb requires a caller-started JEB client serving MCP at REA_JEB_MCP_URL (default http://127.0.0.1:8425/mcp) with a project already open, and exercises shared application/provider client inspection, unit listing coverage, and method decompilation. The script records the engine's exact response to open_jeb_project; the earlier licensed 5.48.0 headless verification did not advertise that tool. REA does not install or launch JEB; see JEB analysis for the bring-your-own boundary. The earlier licensed build was verified against JEB 5.48.0 serving jeb-mcp-server 1.3.0 at four scales: a compiled Java class fixture, a locally built signed probe APK with a launcher activity (manifest, v1/v2/v3 certificates, dex bytecode, filtered and paginated listing), the published Signal 8.30.3 universal release APK (109 MB, R8-processed Kotlin/Compose, four signature schemes, native arm64 ELF units) with MainActivity method decompilation through both CLI and MCP, and the official Flutter Gallery 2.9.2 release APK (112 MB): per-ABI libapp.so Dart AOT snapshot images analyzed as native code (75k methods on arm64) and de