[DO NOT MERGE] Run-only: gb200 dsr1 measured power+temp (canonical NVIDIA)#1791
[DO NOT MERGE] Run-only: gb200 dsr1 measured power+temp (canonical NVIDIA)#1791arygupt wants to merge 3 commits into
Conversation
|
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers. If additional help is needed, PR authors can reach out to core maintainers over Slack. |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| total_system_energy_j = avg_power_w * num_gpus * duration | ||
| joules_per_output_token = total_system_energy_j / total_output | ||
| total_tokens = total_output + total_input | ||
| joules_per_output_token = total_system_energy_j / total_output # cluster fallback |
There was a problem hiding this comment.
Desync samples inflate joules
Medium Severity
Multinode energy uses avg_power_w * num_gpus * duration while num_gpus counts every GPU seen across all timestamp buckets, but per-bucket averages can come from only a subset of nodes when perfmon polls don’t align. That pairing can overstate joules_per_* and disagg stage energy even when per-GPU wattage looks correct.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit c0b79f4. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27578362429 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27578777750 |
arygupt
force-pushed
the
feat/dsr1-gb200-powercheck-run
branch
from
c0b79f4 to
742665d
Compare
|
|
||
|
|
||
| # Back-compat shim — some external callers may have imported _parse_power. | ||
| _parse_power = _parse_numeric_cell |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
There are 3 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 742665d. Configure here.
| git clone https://github.com/SemiAnalysisAI/srt-slurm.git "$SRT_REPO_DIR" | ||
| cd "$SRT_REPO_DIR" | ||
| git checkout feat/inferencex-perfmon | ||
| export PERFMON_ENABLED=1 |
There was a problem hiding this comment.
Gb200 dsr1 lacks fp4 guard
Medium Severity
The new perfmon fork branch matches all dynamo-sglang/dsr1 runs, not only the fp4 powercheck. launch_b200-dgxc.sh gates the same campaign on PRECISION == "fp4", but gb200 also switches fp8 configs (e.g. dsr1-fp8-gb200-dynamo-sglang) to SemiAnalysisAI/srt-slurm@feat/inferencex-perfmon and sets PERFMON_ENABLED=1, replacing the validated cquil11/srt-slurm-nv path for unrelated jobs on this commit.
Reviewed by Cursor Bugbot for commit 742665d. Configure here.
| if ! grep -q '^health_check:' "$CONFIG_PATH"; then | ||
| printf '\nhealth_check:\n max_attempts: 540\n interval_seconds: 10\n' >> "$CONFIG_PATH" | ||
| echo "[perfmon] injected health_check (90min ceiling) into $CONFIG_PATH" | ||
| fi |
There was a problem hiding this comment.
Existing health_check not bumped
Medium Severity
On the perfmon path, gb200 only appends a 90-minute health_check when the recipe has no top-level health_check: key. Typical dsr1 recipes already define health_check with max_attempts: 360 (60 minutes), so the launcher leaves that limit unchanged. The b200 path forcibly sets four-space max_attempts to 540 for the same warmup rationale, so gb200 powercheck may run with a shorter ready ceiling than intended.
Reviewed by Cursor Bugbot for commit 742665d. Configure here.
…en pattern) Mirrors the gb300 wiring: the dynamo-sglang + dsr1 path now clones the SemiAnalysisAI perfmon fork (which ships the gb200 dsr1 recipes at recipes/gb200-fp?/<seq>/*.yaml — the exact CONFIG_FILE the dsr1-fp?-gb200-dynamo-sglang configs reference — PLUS the perfmon machinery) instead of cquil11/srt-slurm-nv. Injects monitoring: + health_check (90min ceiling for the ~35min dsr1 warmup), and stages perf_samples_*.csv + GPU_METRICS_CSV_GLOB for Process-result. Targets canonical NVIDIA measured power+temp on the HEALTHY gb200 runner, sidestepping the wedged gb300-nv fleet. Run is gated. Known risk to validate on first run: the Oracle/watchtower gb200 cluster may not cross-mount compute->head, so per-node perf_samples written on compute may need the shared-FS treatment the minimax path already uses. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nical NVIDIA) Path-B-style run-only branch off main to get CANONICAL NVIDIA dsr1 power+temp on the HEALTHY gb200 runner, sidestepping the wedged gb300-nv fleet. Carries #1574's self-contained consumer code (aggregate_power.py/process_result.py + tests) + a 1-job changelog entry (dsr1-fp4-gb200-dynamo-sglang-powercheck). gb200 launcher perfmon wiring is the prior commit. Matrix verified = exactly 1 gb200 job. DO NOT MERGE (duplicates #1574). Run-only to harvest measured data; close after it lands. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ical NVIDIA)
Wire perfmon into launch_b200-dgxc.sh's dsr1-fp4 dynamo-sglang branch:
clone SemiAnalysisAI/srt-slurm@feat/inferencex-perfmon (ships recipes/
b200-fp4/1k1k.yaml + nvidia-smi perfmon) instead of NVIDIA/srt-slurm@main,
insert monitoring: INTO the recipe base: block (survives the zip_override),
bump base.health_check.max_attempts to 540 (90min), and stage
perf_samples_*.csv + GPU_METRICS_CSV_GLOB before the outputs cleanup.
Add dsr1-fp4-b200-dynamo-sglang-powercheck (1k1k stp_lowlat[0], 1P TP4 +
5D TP8, conc[16]) -> matrix expands to ONE b200 job. The b200-multinode
pool (dgxc-slurm_{7,8,9}) health is unverified (idle 60+ sweeps) but is
the best of the wedged/idle NVIDIA multinode pools. Run-only; do not merge.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
arygupt
force-pushed
the
feat/dsr1-gb200-powercheck-run
branch
from
742665d to
06f8f37
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27649212722 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27650031789 |
1 participant
Purpose: get canonical NVIDIA measured dsr1 power+temp on the healthy gb200 runner, sidestepping the wedged gb300-nv fleet.
Carries #1574's self-contained consumer code (
aggregate_power.py/process_result.py+ tests) on top ofmain, plus the gb200 launcher perfmon wiring and a single 1-job changelog entry (dsr1-fp4-gb200-dynamo-sglang-powercheck). Matrix = exactly one gb200 job (1k/1k, conc 8, 1×prefill TP4 + 2×decode TP4) — verified withgenerate_sweep_configs.Same proven pipeline as the gb300 CoreWeave validation (run
27170607165):launch_gb200-nv.shclones the perfmon fork (gb200 dsr1 recipes + perfmon machinery), injectsmonitoring:+ a 90-minhealth_check, and stagesperf_samples_*.csv+GPU_METRICS_CSV_GLOB→ Process result →aggregate_power.pypatches the agg JSON withavg_power_w+ prefill/decode + per-worker temp/util/mem.Why not merge: duplicates #1574's
aggregate_power.py. Closed once the data lands.Known risk to watch on first run: the Oracle/watchtower gb200 cluster may not cross-mount compute→head, so per-node
perf_samples(written on compute) may need the shared-FS treatment the minimax path already uses (flagged inline in the launcher).🤖 Generated with Claude Code
Note
Medium Risk
Changes how benchmark agg JSON is patched and multinode jobs clone external srt-slurm forks; wrong CSV selection or aggregation could publish incorrect power metrics, though behavior is guarded and best-effort.
Overview
Adds run-only sweep configs
dsr1-fp4-gb200-dynamo-sglang-powercheckanddsr1-fp4-b200-dynamo-sglang-powercheck(single 1k/1k jobs) plus changelog entries to validate measured energy on healthy NVIDIA multinode pools.GB200/B200 launchers for
dynamo-sglang+dsr1now cloneSemiAnalysisAI/srt-slurm@feat/inferencex-perfmon, enablePERFMON_ENABLED, inject recipemonitoring:and longerhealth_check, then stageperf_samples_*.csvand setGPU_METRICS_CSV_GLOBfor downstream processing.aggregate_power.pygrows from power-only to multinode-aware telemetry:--csv-glob, temp/util/mem columns, per-worker rollups fromperf_samples_<role>_w<idx>_<host>.csv, disagg prefill/decode power and per-stage joules, and bench-window fallbacks when multinode JSON lacks Unix timestamps.process_result.pyprefersGPU_METRICS_CSV_GLOBover single-node CSVs (no stale fallback) and passesdisagginto aggregation.CI: multinode workflow Slurm pre/post cleanup now
sudo rm -rfroot-ownedbenchmark_logsso cancelled container runs do not breakactions/checkouton shared runners.Reviewed by Cursor Bugbot for commit 06f8f37. Bugbot is set up for automated code reviews on this repo. Configure here.