Reclaim hotblocks disk: point-delete cleanup + startup DeleteFilesInRange

[mo4islona]

What & why

Fixes the NET-819 mainnet full-disk incident and NET-798 (blocking disk cleanup on startup).

During the incident the hotblocks volume hit 100% and compaction could not reclaim space: at a full disk compaction deadlocks (it needs scratch space to write merged output before deleting its inputs), and write-based deletion only grows the DB until compaction runs. The one operation that frees space without writing is DeleteFilesInRange, which unlinks whole SST files at the filesystem level.

Approach

Two independent mechanisms.

Phase 1 — routine logical cleanup (logical_cleanup, every 10s). Point-deletes a deleted table's keys and drops its bookkeeping entry, one write batch per table. Snapshot-safe: the deletes are MVCC-versioned, so an in-flight query holding an older snapshot is unaffected and no grace period is needed. Frees no space directly; it lets compaction drop the data. This is the normal-operation reclaim path, and it is always on.

Point deletes rather than a single delete_range tombstone per table, because range deletions are not officially supported on the transactional OptimisticTransactionDB we run.

Compaction tuning (now load-bearing, since routine reclaim relies entirely on it):

• compact-on-deletion collector → picks up the SSTs a table purge turned tombstone-dense;
• 3-day periodic compaction → backstops files that never cross the density threshold. Not 24h: each pass rewrites every SST older than the period, live data included, so a large stable dataset would be rewritten daily for nothing;
• max_background_jobs=8, max_subcompactions=4, level_compaction_dynamic_level_bytes → the default 2 background jobs could not keep up during the incident.

Physical reclaim — startup only, off by default (reclaim_disk_space). Unlinks whole CF_TABLES SST files below the min-live-TableId watermark via DeleteFilesInRange. No writes, no scratch space → frees disk even at 100%, where compaction deadlocks. This is the "reclaim before ingest" step (NET-798). Crash-orphaned DIRTY_TABLES markers are purged at startup first, so they don't pin the watermark.

Startup order matters at a full disk: unlink (write-free) → bookkeeping writes that lift the watermark → unlink again. Every step is best-effort and never blocks startup.

Rollout: measure first, then enable

Physical reclaim is gated behind --startup-disk-reclaim, default off, so this can ship to prod inertly and be switched on deliberately.

The orphan purge is gated together with the unlink rather than left on: the purge is what lifts the watermark, and the measurement tool below reports "below wm now" vs "below wm after purge". Purging by default would collapse that comparison to zero on the first restart and destroy the measurement.

crates/reclaim-measure is a read-only probe, now built into the hotblocks image. It opens the live DB as a RocksDB secondary instance — no lock, no writes, no SST opens (sizes come from live_files() metadata) — and reports how much --startup-disk-reclaim would actually free. Its DB path defaults to /run/db, so it can be exec'd straight into a running pod:

kubectl exec -it <hotblocks-pod> -- /app/reclaim-measure

It prints CF_TABLES split into live / dirty / dead (reclaimable) / boundary bytes, plus how much a startup unlink would free now and after the orphan purge. Deploy → run it → if the number justifies it, add --startup-disk-reclaim and restart.

Deleting unconfigured datasets at startup is not gated; it predates this branch and still runs unconditionally.

Trade-offs & deferred work

The file unlink ignores snapshots, so it can break an in-flight query whose snapshot predates a deletion. We therefore run it only at startup, where there are no live readers — it never corrupts the DB, only risks an in-flight read, and only at a moment when there are none.

The watermark is a global minimum over all datasets, so one old live table (or one undecodable chunk) blocks the unlink everywhere until it is gone. Above-watermark and boundary garbage is left to compaction.

A runtime disk-pressure trigger (free-space threshold via statvfs, or on a write ENOSPC) — the emergency reclaim compaction cannot deliver at a full disk — is documented in code as a future improvement, not built here.

Changed from the earlier iteration

This supersedes the grace-period design. Removed: per-table deletion timestamps, the wall-clock grace gate (--reclaim-grace-secs), the runtime orphan reaper (--max-build-secs), and the two-phase bookkeeping. That eliminates a class of wall-clock data-loss bugs (clock jumps, slow clients outliving the grace, malformed-value coercion) in exchange for running the unlink only at startup. reclaim_disk_space and Phase 1 are now fully decoupled (watermark vs. work-queue).

Testing

crates/storage/tests/cleanup_reclaim.rs (8 tests): snapshot-safe logical delete, file unlink below the watermark, live-table watermark pinning, idempotency, startup orphan purge, reclaim/Phase-1 decoupling, and the documented "unlink breaks a live pre-deletion reader" trade-off. Full sqd-storage suite green (cleanup 8, compaction 5, database_ops 10, write_read 15, lib 1).

The flag was exercised end-to-end against a real DB: with it off, startup logs startup disk reclaim is off; enable with --startup-disk-reclaim and neither reclaim step runs; with it on, both run clean. reclaim-measure was run against the same DB. The docker image build itself was not verified locally (no daemon available) — the Dockerfile change is a second -p on the existing cargo build plus one extra COPY.

[eldargab]

Range deletions are not officially supported for transactional databases - facebook/rocksdb#4812.

According to rust-rocksdb/rust-rocksdb#839 , it seems support for this feature was added to the Rust wrapper simply because someone was able to chase all the required pointers, despite lacking a full understanding of all potential edge cases.

[mo4islona]

Thanks. Reverted the old behaviour for now

[elina-chertova]

Fresh recurrence of the full-disk condition this PR targets, in case it helps prioritize review.

On 2026-07-04 ~00:37 UTC the network-hotblocks-mainnet hotblocks-db volume (local local-storage PV, 768Gi, on-prem node) hit 100% and fired a critical node-disk alert. The deployed image is subsquid/data-hotblocks:b5b8886 (2026-05-20), which predates this PR, so none of the reclaim mechanisms here are present.

The recovery matched this PR's premise exactly: the pod was restarted ~00:30 UTC and free space jumped from 0 → ~674 GiB in a single step (768Gi volume went from 100% to ~13% used). That is precisely the "unlink whole SST files below the live-TableId watermark, reclaim even at 100% where compaction deadlocks" behavior described here — today it only happened because of a manual restart, not the routine logical_cleanup / startup reclaim_disk_space this PR adds.

Evidence during the incident: retention was advancing logically (per-dataset hotblocks_first_block was moving forward for all datasets), yet on-disk usage had grown monotonically and sat >96% full for the preceding ~30 days with no sawtooth — consistent with tombstone/compaction not reclaiming space until a restart.

Until this ships and the deployed image is bumped, the operational mitigation is a manual hotblocks-db restart to trigger reclaim (and/or capping the heaviest datasets via a retention: <N> Head strategy).