Skip to content

feat(metrics): RocksDB write-stall gauges#83

Closed
mo4islona wants to merge 1 commit into
masterfrom
feat/rocksdb-stall-metrics
Closed

feat(metrics): RocksDB write-stall gauges#83
mo4islona wants to merge 1 commit into
masterfrom
feat/rocksdb-stall-metrics

Conversation

@mo4islona

@mo4islona mo4islona commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Why

The post-deploy ingest freeze (all datasets freeze ~6 min, then resume together) looks like a global RocksDB write-stall on the shared Arc<Database>: RocksDB keeps one WriteController per DB, so a stall triggered by any single column family throttles writes DB-wide → every dataset freezes at once, releasing when compaction catches up. Retrospective telemetry can't show it — RocksDB's own stall state was never exported.

What

hotblocks_rocksdb_* gauges via a pull-collector (RocksDbCollector). Reads intrinsic RocksDB properties on scrape only — no write-path cost, no enable_statistics() needed:

  • DB-wide: write_stopped (decisive), actual_delayed_write_rate, running_compactions, running_flushes
  • per-cf (CHUNKS, TABLES): num_files_at_level0, estimate_pending_compaction_bytes, num_immutable_mem_table, mem_table_flush_pending

Plus Database::get_int_property + a test pinning the property names (a typo → silent None → missing gauge).

Verifying the freeze

On the next deploy/freeze, hotblocks_rocksdb_write_stopped == 1 plus a spike in num_files_at_level0 / estimate_pending_compaction_bytes confirms a write-stall and names the trigger. The existing --rocksdb-max-background-jobs flag is the lever to raise background jobs if the autodetected 2..=8 default is too low.

🤖 Generated with Claude Code

@mo4islona mo4islona marked this pull request as draft July 11, 2026 08:03
Pull-collector exposing RocksDB write-controller and LSM-backlog properties as hotblocks_rocksdb_* gauges (write_stopped, actual_delayed_write_rate, running_compactions/flushes; per-cf L0 files, pending-compaction bytes, immutable memtables, flush-pending) so the post-deploy ingest freeze -- a global write-stall on the shared DB -- shows up in metrics. Reads free intrinsic properties on scrape, no write-path cost.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mo4islona mo4islona force-pushed the feat/rocksdb-stall-metrics branch from fdb7a26 to 71986f4 Compare July 11, 2026 10:52
@mo4islona mo4islona changed the title feat(metrics): RocksDB write-stall gauges + background-jobs env override feat(metrics): RocksDB write-stall gauges Jul 11, 2026
@mo4islona

Copy link
Copy Markdown
Contributor Author

Closing — the --rocksdb-max-background-jobs flag already ships on master (#79), so bumping background jobs needs no code: just run the pod with --rocksdb-max-background-jobs 8. The RocksDB write-stall metrics from this branch aren't needed right now; can revisit separately.

@mo4islona mo4islona closed this Jul 11, 2026
@mo4islona mo4islona deleted the feat/rocksdb-stall-metrics branch July 11, 2026 11:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant