fix(disp): div 0 error when using disp_avg under multitask#5809
fix(disp): div 0 error when using disp_avg under multitask#5809OutisLi wants to merge 3 commits into
Conversation
📝 WalkthroughWalkthroughMultitask training-loss logging now handles unsampled intervals by emitting NaN values for affected loss keys. Display averaging initializes missing task accumulators, computes per-task validation results, and a regression test verifies the resulting learning-curve output. ChangesMultitask loss logging
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@deepmd/pt/train/training.py`:
- Around line 1619-1632: Update the multitask display-step handling in run()
around the disp_avg branch: perform a dummy forward pass for each unsampled
model key to populate self.train_loss_accu with its loss keys before generating
train_results, excluding l2_ fields as in the proposed flow. Move the validation
and rank-zero console logging loop outside the disp_avg conditional so
log_loss_valid and progress messages execute for both averaged and non-averaged
modes.
In `@source/tests/pt/test_multitask.py`:
- Around line 273-277: Update the lcurve.out assertions in the multitask test to
read and tokenize the header row, then assert every data row has exactly the
header’s column count. Preserve the existing displayed_steps and “nan” checks
while ensuring the unsampled task does not produce a shorter row.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 5ac6ebae-2ba6-49ed-b897-c162751bd612
📒 Files selected for processing (2)
deepmd/pt/train/training.pysource/tests/pt/test_multitask.py
njzjz-bot
left a comment
njzjz-bot
left a comment
There was a problem hiding this comment.
Requesting changes because the filtered-batch path can still leave an unsampled task without the metric schema required by learning-curve output.
Coding agent: Codex
Codex version: codex-cli 0.144.4
Model: gpt-5.6-sol
Reasoning effort: xhigh
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #5809 +/- ##
==========================================
- Coverage 78.67% 78.44% -0.24%
==========================================
Files 1053 1053
Lines 120889 120896 +7
Branches 4381 4383 +2
==========================================
- Hits 95107 94833 -274
- Misses 24234 24510 +276
- Partials 1548 1553 +5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@deepmd/pt/train/training.py`:
- Around line 1715-1716: Update the unsampled-task initialization around the `if
not task_input` early return so filtered `{}` batches cannot leave
`train_results[_key]` without its metric keys. Populate the expected metric
schema independently of a consumable training batch, or continue fetching until
`get_data()` returns a valid batch, while preserving the existing `disp_avg` and
validation display behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 28d64a76-12e4-4629-83a0-3c8a68c14e06
📒 Files selected for processing (2)
deepmd/pt/train/training.pysource/tests/pt/test_multitask.py
🚧 Files skipped from review as they are similar to previous changes (1)
- source/tests/pt/test_multitask.py
| if not task_input: | ||
| return |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Handle filtered batches when initializing unsampled metrics.
When disp_avg is enabled and a task was not sampled in the interval, get_data() can return {} if min_pair_dist rejects the fetched batch. Returning early leaves train_results[_key] without metric keys. With validation configured, print_on_training() then indexes those missing training metrics and raises a KeyError at the first display; without validation, the header is incomplete and later rows become misaligned.
Please populate the metric schema without relying on one consumable training batch, or loop until a valid batch is returned.
🛠️ Proposed fix
- if not task_input:
- return
+ while not task_input:
+ task_input, task_label, _ = self.get_data(
+ is_train=True, task_key=_task_key
+ )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@deepmd/pt/train/training.py` around lines 1715 - 1716, Update the
unsampled-task initialization around the `if not task_input` early return so
filtered `{}` batches cannot leave `train_results[_key]` without its metric
keys. Populate the expected metric schema independently of a consumable training
batch, or continue fetching until `get_data()` returns a valid batch, while
preserving the existing `disp_avg` and validation display behavior.
njzjz-bot
left a comment
njzjz-bot
left a comment
There was a problem hiding this comment.
Request changes: the unsampled-task schema is still not guaranteed when a training batch is filtered out by min_pair_dist.
initialize_task_loss_accumulator() returns immediately on an empty task_input. That leaves train_results[_task_key] empty. At the first display, print_on_training() iterates the validation metric keys and indexes the corresponding missing training keys, causing a KeyError; without validation, the generated lcurve.out header is incomplete and later rows can become misaligned.
Please populate the placeholder metric schema without depending on a single consumable batch (or keep fetching until one is usable), and add a regression test covering an unsampled task whose initialization batch is fully filtered.
All CI checks are otherwise passing.
— OpenClaw 2026.6.11
Summary by CodeRabbit
NaNvalues instead of missing/empty results.disp_avglogging to keep displayed indices and columns consistent, includingNaNentries when averages can’t be computed.disp_avgcorrectly handles unsampled intervals and thatlcurve.outshowsNaNin the expected row/columns.