Black-box test harness for hotblocks + first conformance tests

[mo4islona]

The service layer has no automated tests. Storage and query are covered; ingestion, the dataset controller, the API and liveness are not — which is why the last few production incidents were found in production. This adds the harness that makes them testable, plus the first two tests, and fixes the bug the harness found on its first run.

The bug it found (first commit)

LineStream::take_final_line empties its buffer but left unchecked_pos pointing past the end of it. A source response whose JSONL body does not end with a newline therefore yields its last record and then panics on the next poll, out of bounds.

The panic kills the ingest task: the buffered batch is lost, the dataset parks for a minute, and the source serves the same body on retry — so it crash-loops and that dataset never ingests again. Silent, apart from ingest task panicked in the log.

One line to fix; pinned by unit tests in data-client and end to end by ct9_source_faults.

The harness (crates/hotblocks-harness)

The system under test is the real binary, spawned as a child process and driven over HTTP. Nothing links against its internals: an assertion that cannot be made through the HTTP surface is an assertion a client could not rely on either — and owning the process is what will make the crash/restart and shutdown classes expressible later.

  script ──┬──▶ SourceSim ──HTTP──▶ [ sqd-hotblocks ] ──HTTP──▶ Client ──▶ validators
           └──▶ Model  ◀──────── compare at quiescence ────────────┘

Module	What it is
sim	source simulator: scripted chain, fork signals, finality headers, fault injection
model	the reference model — the oracle. Block-exact; well-formedness asserted after every transition
driver	client: the read binding, the structural validators, the anchored follower and backfill scanner
compare	quiescence comparator: diffs every observable and reports all divergences, not the first
sut	process supervisor: config, spawn, readiness, SIGTERM, SIGKILL, restart on the same database
chain	kind-parametric payloads (hyperliquid-fills — the cheapest kind to synthesize)

A test reads like this:

let mut h = Harness::start(HarnessConfig::from_block(
    env!("CARGO_BIN_EXE_sqd-hotblocks"), Arc::new(HlFills), 1_000
)).await?;

h.produce(50)?;              // the source mints 50 blocks; the model extends by the same run
h.finalize_with_lag(5)?;     // finality trails the tip, as on a real chain
h.settle().await?;           // wait for the service to converge on the script
h.assert_conforms().await?;  // HEAD, FINALIZED-HEAD, STATUS, METADATA, the chain gauges,
                             // and a full-window scan diffed against the model, block for block

A failing test prints the service's own log tail (with the HTTP access log filtered out — it is never the interesting part).

The tests

• ct1_happy_path (~10 s) — cold ingest of a 50-block window with finality trailing the tip, then an anchored client tailing 20 more. Checks the structural chain, watermark bounds, provenance (what the source served is what is read back, payload included), append and finality monotonicity, response shape and completeness, anchored ancestry across responses, progress, range honesty, the bounded long-poll wait, and the per-dataset gauges.
• ct9_source_faults — a source whose last record is unterminated must not stall ingestion. Fails without the first commit.

Notes for review

• Tests live in crates/hotblocks/tests/ because only a test inside that package gets env!("CARGO_BIN_EXE_sqd-hotblocks"). Everything reusable is in the harness crate, so a soak or benchmark runner can use it outside cargo test.
• Two facts about the service shape the simulator, and undoing either silently breaks everything: the service only commits a batch when the source's response ends (a simulator holding one long-lived stream open leaves every block invisible), and it re-requests instantly (so a request with nothing to serve must be held, or the ingest loop spins at full CPU).
• Scans use includeAllBlocks because coverage is not otherwise recoverable by a client from a filter-sparse response.
• The comparator compares the window's first block exactly. That is right today and wrong the moment a test trims: the service trims whole chunks, so its window may legally be larger than the model's. Whoever writes the retention tests must give the comparator that tolerance first.
• The harness is written against the hotblocks behavioral specification; the INV-* / RP-* / CT-* identifiers cited on the assertions are its vocabulary. Those documents land separately.

🤖 Generated with Claude Code