rea/docs/testing.md

Testing REA

Use this guide when choosing coverage, pruning tests, or running a verification lane. Start with the commands below; use real-toolchain lanes for engine prerequisites and client verification for installed-client and agent runs.

Developer commands

Use source feedback while editing, explicit boundary checks for the changed behavior, and complete CI evidence before merging.

Command	Scope
npm run test:local	Dirty source tests via the import graph, without build; explicit source paths run regardless of Git status
npm run test:focused -- PATH...	Exact existing test files; compiled boundaries build first, and unmatched paths fail
npm run test:changed	Source tests affected since the merge base with origin/main, including committed and dirty changes
npm run test:fast	All domain, service, adapter, composition, conformance, and evaluation tests without build
npm run test:boundary	Boundary, CLI boundary, MCP boundary, process boundary, and process-global projects
npm run test:mcp	MCP boundary project
npm run test:acceptance	Complete compiled CLI and MCP acceptance workflows
npm run test:python	LLDB and mitmproxy producer contracts plus analysis-efficiency checks; no real debugger or proxy required
npm run test:watch	Dirty source tests in watch mode, without build
npm run test:watch:all	Changed tests from every project; builds at startup, so rebuild after production edits before relying on compiled tests
npm run check:changed	Cached typecheck/lint and branch-related source feedback
npm run check:pr	Opt-in complete local deterministic gate and generated-file checks
npm run docs:check	Generated-document validation and fresh onboarding example consistency checks

The general source suite above runs on Linux in CI, and Windows CI runs a curated lane (Native platform baseline in CI). A Windows source checkout can fail source tests for reasons unrelated to the change, such as POSIX fixture paths, #!/bin/sh tool stubs, or POSIX rename semantics. Compare failures against the base revision, or run the suite on Linux or WSL, before attributing them to a change. See CONTRIBUTING.md → Windows source checkouts.

For example:

npm run test:local -- src/config.test.ts
npm run test:focused -- tests/acceptance/applications/runtime.test.ts
npm run test:changed -- --base origin/main
npm run test:changed -- --dry-run

test:focused accepts exact repository-relative test file paths. Source-only paths do not build; boundary, acceptance, process-global, or unfamiliar tests/ paths build conservatively. Tests use Vitest concurrency and isolated workspaces; commands do not hold a broad test lock. Build and documentation writers retain checkout-local locks for their shared output files. Explicit selections do not use --changed or permit zero-test success. The dry-run option reports the chosen merge base, scope and build prerequisite without executing tests or building. A missing Git base reports how to fetch it or select another revision.

Source typechecking needs no compiled runtime or generated test metadata. Typecheck, runtime compilation, and Frida binding verification share scripts/run-typescript.mjs. The pinned tsrs package supplies Linux x64/arm64 (GNU and musl) and macOS arm64 compilers. Windows and other unsupported hosts retain the existing stable official TypeScript compiler. Its upstream version can differ from tsrs, so both compiler tasks hash the selected package and version through scripts/run-turbo.mjs before cache lookup. Outputs from different compilers occupy different cache entries. Repository npm scripts and focused-test preparation use that entrypoint; a raw Turbo compiler task without matching identity fails instead of publishing an ambiguous successful cache entry. A missing binary on a supported host also fails without silently falling back.

The Rust compiler defaults to at most two checker threads. For --noEmit checks, --maxMemory 3G also sets a soft memory target; emit builds do not use that target. Explicit compiler arguments can override the defaults. These controls do not impose a process-family memory limit. Termux's explicit build, Stryker's compiler API dependency, and independent fixture compiler oracles keep their existing TypeScript versions.

npm run build:cached compiles the CLI/MCP runtime and its packaged skill. The build also records dist/mcp-catalog.json (compact inputs) and dist/mcp-catalog-full.json from the canonical SDK registrations and binds each to the runtime with its own SHA-256 digest. These are runtime artifacts, separate from the test catalog below. Default target-free MCP startup serves its schemas without constructing analysis validators; the first tool or guided-workflow request materializes the canonical runtime on that same SDK server. Explicit compact/full input selections preserve their advertised schemas throughout materialization. Initial-target, HTTP and output-schema-off startup keep the eager path; source runs without recorded artifacts also use eager composition. Compare memory both after discovery and after a completed request; report diagnostic GC separately from normal resident memory.

Turbo's strict task environment preserves caller-selected GOMAXPROCS, GOMEMLIMIT, and UV_THREADPOOL_SIZE for resource-constrained builds and checks. These execution controls do not change artifact content, so changing their values retains cache reuse. They control individual runtimes; monitor the complete process family separately when enforcing a total memory budget. npm run test:prepare also generates the MCP contract test catalog, product catalog, and portable managed evidence used by the complete suite. Focused tests prepare those extra outputs only when their selected files consume them. The MCP test catalog is JSON in .cache/mcp-tool-catalog.json; the tracked test loader supplies its types without making generation a source-check prerequisite. Run npm run mcp-catalog:generate to refresh it independently. Documentation generation and managed conformance run through their separate docs:generate and evidence:generate task graphs.

Changed selection can miss runtime registration, generated data, shell entrypoints, bridges, or other relationships absent from the import graph. An empty changed selection means no tests were selected, not verified correctness. Select relevant boundary files and real-provider lanes explicitly. Source projects also contain large capacity regressions; test:fast promises no compiled-runtime prerequisite, not a fixed time budget.

npm run lint prints errors and a warning summary; npm run jscpd prints duplication statistics. Complete reports remain in .cache/static-checks/. The fast gate uploads them as static-check-reports. Full qualification uploads duplication diagnostics from the deep job as static-deep-check-reports, including after that gate fails. Task-graph recording runs check:ci:fast:tasks and check:ci:deep:tasks without the artifacts lock, so it can record hashes while check:pr or check:test already holds that lock. The CI jobs still run the locked check:ci:fast and check:ci:deep commands. Use npm run lint:details or npm run jscpd:details for all console diagnostics. Both duplicate commands scan production and test code using the limit in .jscpd.json. The summary commands preserve checker failures and restore reports from cached static tasks. Both lint caches include the website, documentation configuration, and experiment code scanned by the checker; a regression compares the actual lint inventory with Turbo's resolved inputs.

Routine iterations and rebases need focused regressions and relevant checks. Before handing off a PR, run npm run check and generated-document checks when applicable; CI owns the full suite and coverage. Use the full local gate for broad changes or diagnosing CI, rather than after every edit. Package/install changes additionally need package verification; provider changes need actual provider evidence.

Vitest projects use up to two workers, bounded by host parallelism. process-boundary runs later with serial files because process-tree sampling shares host resources. Acceptance and process-global cases use isolated forks without serial scheduling. Pure domain/service, composition, conformance, evaluation, CLI boundary, and MCP boundary projects share module graphs. Their tests avoid process-global state and keep resource cleanup test-scoped. Each MCP session still owns its resources, and CLI cases retain fresh command instances or subprocesses. CLI and MCP module mocks belong in tests/process-global/ so they keep per-file isolation. Product catalog verification stays with the isolated filesystem boundary tests because it loads both source and compiled module graphs. Adapter, other boundary, acceptance, and process-global projects retain per-file isolation. See vitest.config.ts for current settings.

Inject producer faults through scoped dependencies in shared workers: a module mock cannot replace a consumer that was imported earlier. Check these tests with the consumer preloaded as well as alone. For simulated protocol traces, use an acknowledged event barrier before asserting complete observations; a zero-duration capture does not guarantee queued events have arrived.

Check advertised schemas after SDK conversion for every supported schema profile. Meta-schema validity does not establish client acceptance or useful rendered declarations. Keep canonical rejection checks separate from the intentionally reduced compact advertisement, and probe actual client renderers when changing schema representations.

The reused CLI boundary workers cap V8 old space at 384 MiB for ordinary runs and 512 MiB with coverage enabled. Covered ADB fixtures exhaust the ordinary 384 MiB budget; those fixtures also complete with the shared CLI graph preloaded at 512 MiB. This is a measured capacity control, not a suite-wide speedup claim. Both profiles keep the same bounded worker count, and spawned REA processes retain their own heap settings. Explicitly disabling coverage retains the ordinary profile; coverage provider options alone do not enable instrumentation.

Vitest worker limits do not bound subprocesses launched within a test. Bound those batches separately so repeated CLI validation cannot oversubscribe the runner. Preserve simultaneous launches where concurrency is itself under test.

Coordinate expensive checks across agents sharing a host. If a test unexpectedly consumes heavy CPU, stop its owned process family, retain its log and revision, and profile a bounded representative case before retrying. Separate production work from fixture generation, schema compilation, subprocess startup and cleanup. Choose elapsed, CPU and process-family memory limits before the run; a worker count or test timeout alone does not enforce those limits.

Remove demonstrated repeated work while preserving distinct assertions and resource ownership. Share immutable fixture preparation where safe; keep fresh sessions when lifecycle or concurrency is the behavior under test. Compare the same completed workload before and after, report cache state and measurement limits, and record interrupted runs separately from passes. Increasing a timeout does not explain a CPU regression.

Build/documentation writers hold checkout-local output locks. Tests do not hold a broad command lock; check:pr finishes tests before document validation. Turbo caches deterministic preparation and static checks, but test:run always executes. Repository files do not fully describe installed tool contents or host state. A previous test pass therefore cannot replace current validation, even when the source tree is unchanged. check:pr also runs the fast Python producer suite; CI runs it in the static job for code changes. These producer tests do not establish real LLDB engine coverage.

Vitest's persistent compile cache remains disabled. The cliTest fixture shares a temporary Node compile cache among its fresh CLI subprocesses. The worker removes it after test-scoped processes finish. Explicit child environment settings take precedence; children requesting NODE_V8_COVERAGE do not receive the fixture cache. These subprocesses are outside aggregate Vitest coverage; subprocess coverage attachment remains disabled. Cold-start import checks launch independently without this fixture cache.

analyzeJavaScriptDirectory stdio MCP cases in one cliTest file share one server and that compile cache. The first case pays startup; later cases reuse the process. File teardown waits for startup still in flight, closes a server registered during that wait, and does not open a server after close. A failed handshake stays the caller error when its cleanup also fails, including a rejection without a reason. That cleanup stays owned until a later close or call releases it. Another server does not open while the cleanup remains, and a cleanup that already succeeded is not run again. Pass mcpCall when a case needs its own server, environment, or lifecycle. CLI cases for that helper share one compiled CLI process per child environment. The first case pays process startup; later cases reuse the loaded runtime. The process restarts when the environment changes, after 32 commands, or when the worker reports a resident set of at least 1 GiB. A fatal worker failure or a broken session frame retires that process and waits for it to exit before the next command starts. cli.run still starts a fresh process, including stdin and explicit deadlines. runFileCli uses the file session for commands that have neither. Go binary, IL2CPP metadata, and dylib-resolution commands use that session. File teardown stops the shared CLI process, including startup still in flight, and does not open another after close. exportedFunctionBindingsCli.test.ts keeps its own compiled CLI parity process for the file and still compares that process's JSON command results with the MCP session. A command that does not respond is rejected. That process stays owned until it exits, and another process does not start while it remains. A process that stays running after SIGKILL remains owned so a later close can retry. Closing the session reports a failure when the process does not exit after stdin closes, once the child has been stopped. applicationWorkflowCli.test.ts keeps one compiled CLI process for its JSON commands, including a non-zero status whose stdout is still a JSON document, and closes that process after the file.

The Apple artifact verifier step shares a job-temporary bytecode cache among its fresh CLI/MCP children via REA_ARTIFACT_NODE_COMPILE_CACHE. The artifact helper maps this setting to child NODE_COMPILE_CACHE; npm, Turbo, and nested Vitest runs do not inherit that setting. Explicit NODE_COMPILE_CACHE takes precedence, and coverage collection or NODE_DISABLE_COMPILE_CACHE=1 disables the helper's cache. The cache is discarded with the CI job.

To evaluate persistent compile caching for repeated local runs, opt in for both cold and warm measurements with an isolated cache:

NODE_COMPILE_CACHE=.cache/node-compile npm test

Do not report the warm result as a cold-suite improvement, and do not enable the cache in coverage or benchmark CI without first showing that its instrumentation remains equivalent.

Choose verification for the change

test:local is fast source-level feedback. For the strongest practical regression, prefer a real public CLI/MCP workflow, then production-boundary integration, then a producer golden; use the table to select quick feedback and claim-specific additional evidence. Provider-behavior claims require the matching real-provider lane.

Change or claim	Fast feedback	Claim-specific coverage
Source semantics	npm run test:local -- PATH...	Relevant boundary cases when the behavior crosses a port
CLI/MCP workflow	npm run test:focused -- PATH...	Actual public calls; an injected provider has limited scope
Documentation or generated metadata	npm run docs:check	npm run docs:build for rendered pages and links
Engine adapter	Its real-provider lane below	Host, version, target identity, and owned cleanup
Client configuration or schema conversion	Native client lane	Distinguish loopback models from live services
Agent investigation quality	Agent evaluation	Review transcript and scenario-specific evidence

Test design and pruning

Prefer real public CLI/MCP workflows, then production-boundary integration, then goldens captured from real producers. Keep module tests for distinct failures or semantics that stronger workflows cannot reliably reproduce. Paths, compiled imports, and suite names do not establish test depth.

Prune tests that mirror implementation details or duplicate a stronger claim. Before deleting a distinct regression, identify the replacement scenario and assertions. Preserve malformed formats, cancellation, cleanup, permissions, capacity, target identity, and combinations with distinct interactions. Trivial helper assertions need no replacement. Remove abandoned production scaffolding only after checking runtime and verifier consumers, including compiled imports.

Use representative scenarios rather than Cartesian matrices of equivalent cases. Share setup without hiding inputs or expected evidence. For successful Result values, throw the typed error before asserting the value; a preceding .ok assertion adds no coverage. Protocol fixtures must reject unmodeled commands and preserve actual producer reply shapes.

For comparison normalization, include a legitimate equivalent pair and a distinct pair that would collapse if representation tags or scope were lost. For learned identifiers, cover conflicting pairs and reused values in different collections; mappings must remain one-to-one within their declared scope.

For coverage propagation, compare the same returned rows under complete and incomplete examination. If a row exists on the left but is missing from a fully examined compatible right inventory, removal can be established. If the right side examined only eight of ten applicable records, or returns only one page, the same missing row remains unknown unless separate evidence proves absence. Assert the structured coverage and comparison status as well as the explanation.

For format compatibility, identify the producer revision that establishes each rule. Include an accepted boundary case beside the rejected case so a stricter parser cannot pass by refusing both. Keep source-derived behavior, synthetic fixtures, and executed producer or runtime oracles distinct in validation reports. A fixture accepted by REA proves its reader behavior; it does not prove that a real loader accepts the artifact. If multiple producer revisions share a format identifier but differ in behavior, record that ambiguity rather than using one installed version as an oracle for all of them.

Packed bridge validation uses Python's standard-library AST and tar readers with -I -S: caller-local modules, PYTHONPATH, and site startup hooks must not replace those readers or execute during validation. Controlled ast.py and sitecustomize.py fixtures must leave their markers absent and still reject malformed packed Python with SyntaxError. This checks static package verification, not real Hopper or Ghidra behavior.

Stateless source-analysis matrices may share a file-owned stdio MCP connection and retain representative compiled CLI parity checks. Every distinct source scenario still runs through the public workflow; session-lifecycle scenarios need their own sessions and cleanup assertions.

When a matrix already holds a named tool definition from live tools/list, pass that definition through the SDK's toolDefinition call option. This keeps the selected output-schema validation without eagerly compiling unrelated catalog schemas. Re-read definitions if the server advertises a changed list.

Artifact verification uses artifactMcpCall for both lifecycle and analysis calls. A fresh SDK client has no held tool listing, so a bare callTool can skip output-schema validation. The helper discovers actual advertised definitions, rejects missing or ambiguous schemas, and validates the requested tool without compiling unrelated catalog schemas. REA registers process-static tool contracts; bindings share one discovery per negotiated connection and are discarded on reconnect or failed discovery. Keep raw artifact calls on this shared path and retain explicit request options such as timeouts. Caller-supplied tool definitions are rejected: the helper owns selection from the advertised catalog so a stale or permissive schema cannot bypass it.

The regression fixture changes only canonical_path in a real stdio response to a number. The old helper accepts that response; the named schema must reject it. This client-boundary fault injection does not establish native Apple behavior, which remains covered by the macOS verification lane.

Report unavailable authority as a named skip. Await owned cleanup before a verifier reports success. Native process inspectors are released at Vitest worker teardown; fork termination does not run Node exit hooks, and per-file teardown would retire inspectors still needed by other files.

Measure slow cases before pruning capacity coverage. Retain inputs beyond the formerly failing size/depth and distinguish cold runs, warm builds, coverage, and toolchain conditions when comparing timings.

Lifecycle regressions

For changes to resource acquisition, readiness, cancellation, or cleanup, select the affected owners and applicable scenarios below. Include temporary snapshots and one-shot commands. Use public CLI/MCP workflows where practical and a controlled production seam for interleavings that cannot be reproduced reliably.

Scenario	Assertion and control
Target admitted; startup still pending	Status preserves the readiness distinction and observed startup activity; readiness requires the provider's documented proof.
First handshake is late, absent, invalid, or recovered during close	Initial and recovery paths retain the same authenticated identity facts. Foreign run/token controls cannot authorize cleanup; authenticated idle/native work survives the unauthenticated listener deadline.
Launch returns an error with a valid process handle	Cover already-exited and still-live owned processes. Verify observed exit separately from creation; failed termination or unavailable identity remains unresolved.
Cancellation arrives after workload exit, during result reads or temporary-root removal	Return typed cancellation with available partial observations and actual cleanup status. The non-aborted control succeeds; existing primary/cleanup failures keep their precedence.
Children have mixed health or cleanup outcomes	Missing peer health does not erase an observed failure or activity. Idle plus unknown stays unknown. Retry a failed child without closing a successful peer again, including after a close-option change.
Cleanup fails after success or primary failure	Public output retains the completed observation or primary diagnostic, remaining resources, and owner recovery action. Restore the denied resource and verify the owning cleanup retry.
Target exits without an acknowledgement, or transport disappears while it stays alive	Positive owned exit proof resolves target lifetime; other resources still need cleanup. Lost transport alone does not prove exit. Verify unrelated/attached targets survive and same-connection recovery works when supported.

Shared process-supervision controls separate leader exit from output closure. Cover output during held or failed cleanup, bounded retention and drain timeout, retry, and borrowed stream authority. The POSIX inherited-stderr control uses a separate pipe as a release barrier after leader exit; it does not establish native Windows Job Object behavior.

Synchronize cancellation with the finalization boundary being tested; a fixed delay after dispatch does not establish that phase. Use markers, barriers, or owned cleanup seams while preserving real cleanup where feasible. Check server settlement and retained resources even if the MCP SDK rejects the client promise before the handler finishes. A cancellation during active execution does not cover cancellation during final cleanup. Use explicit abort reasons and assert their identity at admission and after provider completion, alongside retained observations and cleanup state.

Electron fuse regressions cover actual descriptor close denial and retry through the built CLI, same-connection MCP recovery, mixed descriptor outcomes, and server shutdown. Controlled seams cover cancellation during final close and after reader finalization, plus active-provider shutdown during preflight. They preserve real filesystem cleanup where possible; mocked launch boundaries do not establish real Electron teardown or naturally occurring OS failures.

Active Electron cleanup controls retain independent transport and process-family owners, block replacement captures until recovery, and retry only unresolved owners. Cover late close acknowledgement and rejection after a wait deadline, opaque rejection reasons, cancellation during failed finalization, and missing lineage followed by a confirmed passive family census. Windows controls require a fresh lineage observation and do not signal cached descendant PIDs after root termination. Simulated Windows controls do not establish native Windows behavior.

Native process-ownership controls distinguish start identity, zombie lifecycle, current parent/group relation, and token-read evidence. Retain generation-bound candidates through smaller lineage observations, table omissions, reparenting, group changes, and failed cleanup retries. Identityless exit-refresh controls cover a replacement row that is not linked to the prior descendant set: a matching token on its own group is signaled, a foreign or absent token is not, an unreadable live token stays unresolved, a replacement that appears only in the confirmation table is still token-checked, a group discovered during revalidation is signaled before cleanup succeeds, a member that changes groups during confirmation is signaled in its new group, a PID first seen with a foreign token is read again on the confirmation table, a PID whose token was already read is read again even when its parent and group return to the pre-exit values, a pre-existing PID is read again when its parent or group changes, an unchanged pre-existing row that was never token-read stays unread, an unbounded confirmation chain stays unresolved, lineage reads a row again when its launcher-tree membership changes, a cached lineage token is read again on the next refresh, an initial descendant whose token already matched is read again on a later refresh, the launcher is read again on that refresh and a missing, unowned, or relation-changed launcher PID stays unavailable, a descendant absent from the latest refresh is omitted, a group that reappears after it was processed is revalidated once, a further reappearance stays unresolved, lineage publishes one row per PID, and lineage does not publish a verified empty descendant list while that row remains unexamined. A readable token with unavailable identity stays unresolved. Native UID facts, rather than an earlier process-table row, determine whether a token denial concerns another user. Darwin tests compile the helper and exercise snapshot growth, absence, lifecycle, and UID observations; controlled Linux procfs tests do not establish real Linux exit or PID-reuse behavior.

For atomic writes, exercise cancellation on both sides of the documented commit point and preserve any publication receipt. A late interruption does not prove that no output was published.

For successful provider reports, test cancellation immediately before the report reaches the application workflow as well as during provider cleanup. Validate the report before classifying completion: a valid report must retain its Evidence on cancellation, and an invalid report must keep its output error. Exercise both the provider boundary and the shared CLI/MCP workflow; a downstream guard does not repair the provider's own completion contract. When callers share startup, cancel each waiter independently and verify that the last departure awaits owned cleanup while interested peers can still reach readiness. Repeat the same shutdown event after a cleanup failure to verify that its listener can retry. Exercise non-default lifecycle deadlines through queued requests and their consumers; default-only controls cannot detect dropped request options.

For descriptor-backed reads, inject a close failure after a successful read and after a primary read or cancellation failure. Verify that the exact descriptor remains owned, both failures survive, and a later close retries it. Repeat admission attempts while close still fails: no additional descriptor or native command should start. Restore close and verify that the exact owner resolves the originating cleanup marker and allows later work, while the earlier failure remains available as a diagnostic. Retain errors returned by multiple failed retries, including admission and shutdown aggregates, and assert that each becomes complete after recovery. For an aggregate with multiple owners, release one owner while another still fails and assert that it remains incomplete. Cover recovery before the caller-specific error is bound to its retained owner, as another request can retry cleanup concurrently. Confirm that an already released owner resolves a later binding too. Include admission failure after opening the descriptor. If caller cancellation returns before an inspection settles, gate its final read or close and verify that session shutdown waits for the owned work and reports any retained resources. Gate a later read in a multi-read inspection, not only its first read. Overlap operations so one retains a failed descriptor while another reaches its next read; both the operation and shutdown must settle without nested-admission deadlock. Deny child close and verify its containing root survives until retry.

Exercise publication on filesystems without hard links using an unsupported-link seam. Check exclusive destination creation, stat/write/close failures, completed output receipts, and cancellation at the documented cutoff. A streamed fallback has different visibility from an atomic hard-link commit and must report it accurately. For Windows command supervision, keep a descendant writing after its leader exits, deny cleanup observation, and verify a competing writer remains excluded until the exact Job Object closes. Portable doubles cover retry logic; the native Windows lane must establish actual process-tree behavior.

Supervisor tests must observe process exit, output-pipe closure, and IPC disconnection separately. In the pinned Node runtime, a parent-initiated IPC disconnect can omit the aggregate ChildProcess close event. Join the actual resource observations