Add hash → block number index + lookup API

.github/workflows/tests.yaml

@@ -11,6 +11,12 @@ on:
 
 env:
   CARGO_TERM_COLOR: always
+  # Cargo's default for dev/test is debug = 2 -- full DWARF, which is most of a ~280MB test
+  # binary and makes linking the slowest part of the job. Line tables keep file:line in panic
+  # backtraces, which is all CI reads. Set here rather than in [profile.*] so local debuggers
+  # keep full info.
+  CARGO_PROFILE_DEV_DEBUG: line-tables-only
+  CARGO_PROFILE_TEST_DEBUG: line-tables-only
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.head_ref || github.ref }}
@@ -65,7 +71,7 @@ jobs:
         with:
           shared-key: rust-ci-test
           cache-on-failure: true
-      - name: Fetch LFS files
-        run: git lfs fetch --all && git lfs checkout
+      # 8 cores, shared with the second runner -- which picks up this workflow's clippy job.
+      # cc(1) inherits NUM_JOBS, so this caps the RocksDB C++ build too.
       - name: Run tests
-        run: cargo test --all-features --jobs 2
+        run: cargo test --all-features --jobs 4

crates/hotblocks-harness/src/compare.rs

@@ -96,6 +96,28 @@ pub async fn await_quiescence(client: &Client, model: &Model, q: &Quiescence) ->
     }
 }
 
+/// Every block on the model's canonical branch must resolve to the same block reference through
+/// the public hash-index endpoint. This is opt-in because the production index itself is opt-in.
+pub async fn assert_hash_index_conforms(client: &Client, model: &Model) -> Result<()> {
+    let mut errs = Vec::new();
+    for block in &model.seg {
+        let expected = Some(block.as_ref());
+        let observed = client.block_by_hash(&block.hash).await?;
+        if observed != expected {
+            errs.push(format!("hash {}: expected {expected:?}, got {observed:?}", block.hash));
+        }
+    }
+
+    if errs.is_empty() {
+        Ok(())
+    } else {
+        bail!(
+            "the block-hash index diverged from the model:\n  - {}",
+            errs.join("\n  - ")
+        )
+    }
+}
+
 /// Diff every observable against the model, reporting *all* violations, not the first.
 pub async fn assert_conforms(client: &Client, model: &Model, chain: &dyn Chain, dataset: &str) -> Result<()> {
     let mut errs: Vec<String> = Vec::new();

crates/hotblocks-harness/src/driver.rs

@@ -148,6 +148,23 @@ impl Client {
             .await?)
     }
 
+    /// Resolves a block hash through the public hash-index endpoint. A missing hash is a
+    /// normal `None`; every other non-success response is a binding failure.
+    pub async fn block_by_hash(&self, hash: &str) -> Result<Option<BlockRef>> {
+        let res = self.http.get(self.url(&format!("hashes/{hash}/block"))).send().await?;
+        let status = res.status();
+
+        match status.as_u16() {
+            200 => Ok(Some(res.json().await.context("malformed block-hash lookup payload")?)),
+            404 => Ok(None),
+            _ => bail!(
+                "block-hash lookup failed with {}: {}",
+                status.as_u16(),
+                res.text().await.unwrap_or_default()
+            )
+        }
+    }
+
     /// SET-RETENTION (DEF-9). Returns the status code: `403` unless the dataset is API-controlled.
     pub async fn set_retention(&self, policy: &Value) -> Result<u16> {
         let res = self.http.post(self.url("retention")).json(policy).send().await?;

crates/hotblocks-harness/src/harness.rs

@@ -165,6 +165,11 @@ impl Harness {
         compare::assert_conforms(&self.client, &self.model, &*self.chain, &self.dataset).await
     }
 
+    /// Diff the opt-in block-hash lookup endpoint against every canonical block in the model.
+    pub async fn assert_hash_index_conforms(&self) -> Result<()> {
+        compare::assert_hash_index_conforms(&self.client, &self.model).await
+    }
+
     /// An anchored follower positioned at the bottom of the window (04 §7).
     pub fn follower(&self) -> Follower {
         Follower::new(

crates/hotblocks/spec/01-overview.md

@@ -11,6 +11,12 @@ It is the "hot" complement to an archival store: archives hold the deep, immutab
 Hotblocks holds the volatile tip where blocks arrive continuously, forks and rollbacks
 happen, and freshness is measured in fractions of a second.
 
+Alongside range queries it offers **point lookups by hash** — resolving a block hash, or a
+transaction hash, to a position in the window — for consumers that hold a bare hash from a
+log, an event, or a third party and need a block number to query with. These lookups are
+served from optional derived indexes ([DEF-17](02-data-model.md)) and come with a sharp
+caveat spelled out in NG7 below.
+
 The system is a single service instance hosting many datasets concurrently (tens of
 datasets is the normal operating point). Datasets are independent chains: different
 networks, different chain families ("kinds"), different retention policies.
@@ -59,6 +65,13 @@ networks, different chain families ("kinds"), different retention policies.
   the server ([RP-10](04-read-path.md)).
 - **NG6 — No exactly-once source consumption.** Source delivery is at-least-once;
   ingestion is idempotent with respect to redelivery ([WP-16](03-write-path.md)).
+- **NG7 — Hash lookups are a convenience, not an oracle.** The hash indexes are optional,
+  derived, and deliberately incomplete: never backfilled, enabled per deployment, defined
+  only for kinds that expose the hash. A lookup **hit** is authoritative; a lookup **miss**
+  is not evidence that the block or transaction is absent from the window
+  ([RP-19](04-read-path.md)). Making misses meaningful would require backfilling history
+  the hot store has already accepted without an index — work whose cost scales with the
+  window, for a guarantee a range query already provides.
 
 ## 5. Trust model
 

crates/hotblocks/spec/02-data-model.md

@@ -173,6 +173,27 @@ progress ([RP-9](04-read-path.md)).
 **DEF-15 (Watermark reads).** Point reads returning the current committed `head`, `fin`,
 retention policy, and dataset status. Defined in [04-read-path.md §6](04-read-path.md).
 
+**DEF-17 (Hash indexes).** Two optional per-dataset **partial** maps over the current state:
+
+```
+bidx(D) : Hash ⇀ ℕ            — block hash → block number
+tidx(D) : Hash ⇀ ℕ × ℕ        — transaction hash → ⟨block number, transaction index⟩
+```
+
+They are **derived**, never primary: everything they name already lives in `seg`, no
+transition reads them, and dropping them loses no information. They are **partial by
+design** — a hash may be absent from an index while its block sits in `seg`. Absence
+therefore says nothing about `seg`; only presence does ([RP-19](04-read-path.md)). The
+sources of partiality are structural, not incidental: an index is enabled per deployment
+(`P-BLOCK-INDEX` / `P-TX-INDEX`), it is never backfilled, so blocks ingested while it was
+off stay unnamed for as long as they remain in the window, and it is defined only for
+kinds whose schema exposes the hash (EVM today).
+
+Within one dataset each index is a function: a block hash names at most one block, a
+transaction hash at most one transaction *of the current chain*. Across branches the two
+differ — a block hash belongs to exactly one branch forever, while a transaction hash is
+routinely re-included at a different position after a reorg ([INV-47](06-invariants.md)).
+
 ## 6. Transitions (summary)
 
 The complete write-side vocabulary; semantics in [03-write-path.md](03-write-path.md):
@@ -203,4 +224,5 @@ advance arriving together); invariants are evaluated at commit points only.
 | preceding block / `hash_at(p)` | DEF-16 |
 | coverage | DEF-14 |
 | fork hints | `ForkSignal.hints` (DEF-12), also the payload of the CONFLICT error (RP-11) |
+| hash index / `bidx`, `tidx` | DEF-17 |
 | batch | the unit of one `EXTEND`/`REPLACE` commit (bounded by P-BATCH-ROWS / P-BATCH-BYTES) |

crates/hotblocks/spec/03-write-path.md

@@ -250,6 +250,16 @@ history cannot be re-acquired through RETAIN.
   concurrent controller of the same dataset is detected (state changed under the writer's
   feet), the writer MUST stop mutating that dataset and raise an alarm (FM-OP-3) — both
   writers continuing is forbidden.
+- **WP-20 (Derived index maintenance).** A derived index (DEF-17) is maintained *inside*
+  the transition that changes what it names, never by a follow-up pass: entries appear and
+  disappear in the same commit as their blocks (INV-46). A reader therefore can never
+  observe an index that disagrees with the window it was read from, and a crash can never
+  land between a block and its entry. Enabling or disabling an index is a deployment
+  change, not a transition: it decides whether *newly* ingested blocks are named and never
+  rewrites committed history — which is precisely why the indexes are partial (DEF-17).
+  Removal is unconditional: entries are dropped with their blocks whether or not the index
+  is currently enabled, so a disabled index drains within one retention period rather than
+  rotting into stale entries.
 
 ## 4. Ingestion error handling
 

crates/hotblocks/spec/04-read-path.md

@@ -13,6 +13,8 @@ mapping lives in [13-interface-binding.md](13-interface-binding.md).
 | `HEAD` | dataset | `head(D)` Ref or "none" |
 | `FINALIZED-HEAD` | dataset | `fin(D)` Ref or "none" |
 | `STATUS` | dataset | kind, retention policy, `first`, `head` (+hash, time), `fin` |
+| `BLOCK-BY-HASH` | dataset, hash | the Ref of the block with that hash, or "none" (§6.1) |
+| `TX-BY-HASH` | dataset, hash | `⟨block number, transaction index⟩` + the hash, or "none" (§6.1) |
 | `GET-RETENTION` / `SET-RETENTION` | dataset (+policy) | current policy / acceptance |
 
 ## 2. Query admission
@@ -130,6 +132,31 @@ For coverage `[from, L]` against snapshot `S`:
 - **RP-14 (Status).** STATUS reports a consistent view: kind, retention policy, `first`,
   `head` (number, hash, time when known), `fin` — all from one snapshot.
 
+### 6.1 Hash lookups
+
+`BLOCK-BY-HASH` and `TX-BY-HASH` answer from `bidx`/`tidx` (DEF-17) of one snapshot `S`.
+
+- **RP-19 (Sound, not complete).** The two directions of a lookup carry very different
+  weight, and clients MUST treat them differently:
+  - a **hit** is true of `S` and as trustworthy as a range query: the named block is in
+    `seg(S)` carrying exactly that hash, and the named transaction sits in that block at
+    that index (INV-45). A hit is never stale, never orphaned, never from a replaced
+    branch;
+  - a **miss** proves *nothing*. The block or transaction MAY be sitting in the window
+    unindexed — the index was disabled when it was ingested, the dataset predates it, or
+    the kind is not covered (DEF-17). Clients MUST NOT read "none" as "not in the window";
+    the authoritative absence test remains a range query.
+
+  A lookup against a dataset whose index is disabled behaves as a lookup against an empty
+  index — "none", not an error. This is what makes the miss direction uninformative *by
+  construction* rather than by accident, and it is the whole price of not backfilling
+  (NG7).
+- **RP-20 (Bounded and non-competing).** A hash argument longer than `P-HASH-MAXLEN` (or
+  empty) MUST be rejected as `MALFORMED_REQUEST` before the store is touched. Lookups are
+  point reads: their cost MUST NOT scale with the window, and they MUST NOT consume the
+  query execution or waiter budgets (`P-EXEC-SLOTS`, `P-WAITERS`) — a lookup flood may not
+  starve range queries, and a query flood may not stall lookups (PF-4).
+
 ## 7. Fork handling — the CONFLICT protocol
 
 - **RP-11 (Anchored queries).** When `expected_parent ≠ ⊥`, it asserts the hash of
@@ -184,6 +211,7 @@ stable API surface. An error response carries no block data.
 | `RANGE_UNAVAILABLE` | `from < first` (window moved past it) | no — re-anchor upward |
 | `ITEM_UNAVAILABLE` | selection touches an absent item collection (RP-8) | no |
 | `NO_DATA` | `from` above watermark after bounded wait (RP-5/6) | yes — poll |
+| `NOT_FOUND` | hash lookup with no index entry (RP-19 — **not** proof of absence) | no |
 | `CONFLICT` | anchored-ancestry mismatch (RP-11) | yes — after re-anchoring |
 | `OVERLOADED` | admission control (RP-3) / waiter cap | yes — backoff |
 | `FORBIDDEN` | SET-RETENTION on a non-External dataset | no |

crates/hotblocks/spec/06-invariants.md

@@ -256,6 +256,50 @@ eat committed blocks; a system crash may only rewind to a committed state within
 
 ---
 
+## G. Derived indexes
+
+The hash indexes (DEF-17) are optional and deliberately incomplete, so there is exactly one
+thing they must never do: lie. The catalog below is what "never lies" decomposes into.
+
+**INV-45 — Index soundness.** [state]
+Every entry of `bidx(D)` / `tidx(D)` is true of the state it is read from: `bidx(h) = n`
+implies `seg` holds a block numbered `n` whose hash is `h`; `tidx(h) = ⟨n, i⟩` implies that
+block holds a transaction at index `i` with hash `h`. No entry names a block outside the
+window, a replaced branch, or nothing at all.
+*Why:* an index that may lie is worse than no index — a client cannot cheaply re-verify a
+hit, so a false positive propagates silently. Partiality is a documented cost (RP-19);
+unsoundness is a defect.
+*Check:* CT-1 — after every commit, every hash of every block (and transaction) removed by
+a fork or a trim MUST resolve to "none", and every hash still in the window MUST resolve to
+its true position or to "none", never to a wrong one. CT-2 repeats it across restarts.
+
+**INV-46 — Index maintenance is part of the transition.** [transition]
+Every transition that removes a block from `seg` (REPLACE, RETAIN, RESET, DROP) removes
+that block's entry — and the entries of its transactions — in the *same* commit (WP-20).
+Transitions that preserve `seg` while reorganizing storage (INV-17) preserve the indexes
+exactly. No committed state, and no recovered state, contains an entry for a block outside
+its window.
+*Why:* this is what makes INV-45 survive churn, and it is the cheapest thing in the system
+to get wrong — one delete path that forgets its entries leaves an index that lies forever,
+and the lie is invisible until a client trusts it.
+*Check:* CT-1 fork + trim churn; CT-7 soak (entry count tracks window size with no upward
+drift); CT-2 crash during a trim.
+
+**INV-47 — Fork re-inclusion names the new position.** [transition]
+If a REPLACE removes a transaction and the new branch re-includes it at a different block
+or index, then after the commit `tidx` resolves its hash to the **new** position — never to
+the old one, and never to "none".
+*Why:* block hashes and transaction hashes behave differently across a fork, and the
+difference is a trap. A block hash belongs to one branch forever, so a fork can remove and
+add entries in any order. A transaction hash is routinely re-included on the winning branch
+— so a fork that removes the old entries *after* adding the new ones erases exactly the
+entry a client is about to ask for, and the resulting "none" is indistinguishable from the
+legitimate partiality of RP-19. Removal must precede insertion within the commit.
+*Check:* CT-4 — reorg script that re-includes a transaction at a new block/index; assert
+the lookup names the new position (not "none", not the old position).
+
+---
+
 ## Reading the catalog in tests
 
 A minimal harness assertion set that touches most of the catalog on every step:
@@ -265,4 +309,5 @@ A minimal harness assertion set that touches most of the catalog on every step:
 2. Every response through validators ⇒ INV-20..27.
 3. Model diff at quiescence ⇒ INV-7, 10, 11, 16, 17, 35, 44.
 4. Kill/restart cycles ⇒ INV-40, 42, 43.
-5. Fork/finality corpora ⇒ INV-12, 13, 14, 23, 24, 31, 36.
+5. Fork/finality corpora ⇒ INV-12, 13, 14, 23, 24, 31, 36, 47.
+6. Hash lookups of everything the window gained and lost, after every commit ⇒ INV-45, 46.

crates/hotblocks/spec/09-retention-and-space.md

@@ -76,6 +76,21 @@ Requirements:
   not O(bytes); physical reclamation runs at bounded amortized cost without violating
   LIV-2 (deletion-induced maintenance debt counts inside the stall budget). Deleting a
   large dataset MUST have bounded peak memory (not proportional to the dataset's size).
+- **RS-12 (Derived index space).** Index bytes (DEF-17) count toward `live_bytes` and fall
+  under RS-6 like any other bytes: enabling an index raises the denominator, so the
+  amplification bound neither loosens nor tightens. Entries are removed in the same commit
+  as their blocks (INV-46), so their space becomes ordinary `debt_bytes` and converges per
+  RS-5/LIV-7 — **an index is never a leak path**, in either flag direction: enabling one
+  does not backfill existing blocks, disabling one does not eagerly erase entries, and both
+  states converge within one retention period as the window turns over.
+
+  Sizing is where the two indexes part company, and operators MUST budget them separately.
+  `bidx` costs one entry per *block*. `tidx` costs one entry per *transaction* — on a busy
+  EVM chain roughly two orders of magnitude more, so its footprint tracks the transaction
+  rate × retention, not the block rate. A retention window that makes `bidx` a rounding
+  error can make `tidx` the largest single consumer in the store. This asymmetry is why the
+  two are independently enabled (`P-BLOCK-INDEX`, `P-TX-INDEX`) rather than sharing a
+  switch.
 
 ## 3. Interactions
 
@@ -88,5 +103,10 @@ Requirements:
   the *next* query below the new `first` gets `RANGE_UNAVAILABLE` (RP-4).
 - **Retention × recovery:** recovered state reflects committed trims exactly (INV-40);
   a trim's anchor carry-over (INV-18) survives restarts.
+- **Retention × hash indexes:** retention is what bounds an index (RS-12) *and* what makes
+  it forget — a hash resolvable today stops resolving once its block leaves the window, and
+  a client cannot distinguish that from a hash that was never indexed (RP-19). Retention is
+  also the only mechanism that repairs an index: history missed while the flag was off
+  drains out on its own.
 - **Space × liveness:** reclamation lag is bounded (LIV-7); maintenance debt feeds back
   into the write path only within the stall budget (LIV-2, HZ-2/HZ-5).

crates/hotblocks/spec/11-observability.md

@@ -50,6 +50,13 @@ surface of the binding (13 §5) with bounded cardinality.
   engine state) sufficient to attribute the stall post-hoc. Rationale: the observed
   production freezes could not be root-caused from standard metrics; capture-on-stall
   turns the next occurrence into data (GAP-1).
+- **OB-12 (Hash index state).** Per dataset × index (DEF-17): whether it is enabled, its
+  entry count and estimated bytes (feeding OB-6's live/debt accounting — RS-12), and lookup
+  counts split by outcome (hit / miss) with latency. Rationale: because a miss is
+  indistinguishable from a genuine absence by design (RP-19), the miss *rate* is the only
+  signal an operator has. A rate near 1 is the signature of an index that is empty for a
+  structural reason — flag switched on after the window had filled, unsupported kind — and
+  without this signal it looks exactly like a chain on which nobody queries real hashes.
 
 ## 2. Property → observable mapping