feat(metrics): RocksDB write-stall gauges#83
Closed
mo4islona wants to merge 1 commit into
Closed
Conversation
Pull-collector exposing RocksDB write-controller and LSM-backlog properties as hotblocks_rocksdb_* gauges (write_stopped, actual_delayed_write_rate, running_compactions/flushes; per-cf L0 files, pending-compaction bytes, immutable memtables, flush-pending) so the post-deploy ingest freeze -- a global write-stall on the shared DB -- shows up in metrics. Reads free intrinsic properties on scrape, no write-path cost. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fdb7a26 to
71986f4
Compare
Contributor
Author
|
Closing — the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The post-deploy ingest freeze (all datasets freeze ~6 min, then resume together) looks like a global RocksDB write-stall on the shared
Arc<Database>: RocksDB keeps oneWriteControllerper DB, so a stall triggered by any single column family throttles writes DB-wide → every dataset freezes at once, releasing when compaction catches up. Retrospective telemetry can't show it — RocksDB's own stall state was never exported.What
hotblocks_rocksdb_*gauges via a pull-collector (RocksDbCollector). Reads intrinsic RocksDB properties on scrape only — no write-path cost, noenable_statistics()needed:write_stopped(decisive),actual_delayed_write_rate,running_compactions,running_flushesCHUNKS,TABLES):num_files_at_level0,estimate_pending_compaction_bytes,num_immutable_mem_table,mem_table_flush_pendingPlus
Database::get_int_property+ a test pinning the property names (a typo → silentNone→ missing gauge).Verifying the freeze
On the next deploy/freeze,
hotblocks_rocksdb_write_stopped == 1plus a spike innum_files_at_level0/estimate_pending_compaction_bytesconfirms a write-stall and names the trigger. The existing--rocksdb-max-background-jobsflag is the lever to raise background jobs if the autodetected2..=8default is too low.🤖 Generated with Claude Code