Skip to content

[Task Clean-up][Camera] Dexterous Part 6/11: Add renderer presets to the Direct camera task#6416

Open
hujc7 wants to merge 2 commits into
isaac-sim:developfrom
hujc7:jichuanh/task-cleanup-dex-part06
Open

[Task Clean-up][Camera] Dexterous Part 6/11: Add renderer presets to the Direct camera task#6416
hujc7 wants to merge 2 commits into
isaac-sim:developfrom
hujc7:jichuanh/task-cleanup-dex-part06

Conversation

@hujc7

@hujc7 hujc7 commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Adds an RGB-depth camera preset so the Shadow Hand camera environment can train with the Newton Warp renderer in addition to RTX (the only preset that is both Warp-renderable and CNN-trainable), with preset acceptance tests.
  • Deprecates the Isaac-Reorient-Cube-Shadow-Camera-Benchmark-Direct registration in favor of the env.feature_extractor.enabled=false override (no consumer exists; no other task family registers benchmark variants).
  • Scope note: the camera manager enablement (shared validation helper, keypoint relocation, manager observation terms) lives in Part 10 per review.
  • Validated by full camera training runs; split out of the lumped validation branch [DO-NOT-MERGE][Task Clean-up] Dexterous: lumped validation branch (split into Parts 1-11) #6324 (Part 6 of 11).

Dependencies

The Shadow Hand camera environment gains an RGB-depth camera preset so
it can train with the Newton Warp renderer in addition to RTX, and the
feature extractor exposes compute_cube_keypoints (the module-level
compute_keypoints shim is deprecated in its favor).

Validated by full camera training runs and replay video generation.
@github-actions github-actions Bot added the isaac-lab Related to Isaac Lab team label Jul 8, 2026
@greptile-apps

greptile-apps Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds an rgb_depth camera preset for the Shadow Hand camera environment, enabling training with the Newton Warp renderer using RGB + depth (4 channels). It also moves the cube-keypoint computation into a shared compute_cube_keypoints TorchScript function in feature_extractor.py and deprecates the old module-level compute_keypoints in shadow_hand_camera_env.py.

  • New preset: ShadowHandTiledCameraCfg.rgb_depth sets data_types=[\"rgb\", \"depth\"]; warp-validation tests updated to include it.
  • Refactor: Validation logic extracted to validate_shadow_hand_camera_settings; compute_keypoints becomes a warnings.warn shim delegating to the new function.
  • New manager term infrastructure: ShadowHandCameraFeatures, shadow_hand_camera_cached_features, and shadow_hand_goal_keypoints are added to feature_extractor.py for future Manager-based observation use, but are not yet wired into any environment config in this PR.

Confidence Score: 4/5

The direct-env path (ShadowHandCameraEnv) is clean, and the preset/validation refactor is safe. The new ShadowHandCameraFeatures manager term has an AttributeError waiting to trigger when it is wired in.

The new ShadowHandCameraFeatures.call unconditionally calls .torch on physics-state tensors (root_pos_w, root_quat_w), which are plain torch.Tensor objects in Isaac Lab. torch.Tensor has no .torch attribute, so this will raise AttributeError the first time the class is exercised. Camera output is guarded correctly with isinstance, making the inconsistency clear. The class is not yet wired into any env config in this PR, so CI will not catch it now, but it will fail on first real use.

feature_extractor.py — specifically the ShadowHandCameraFeatures.call method at the object-state data access lines.

Important Files Changed

Filename Overview
source/isaaclab_tasks/isaaclab_tasks/core/reorient/config/shadow_hand/feature_extractor.py Adds compute_cube_keypoints (TorchScript), ShadowHandCameraFeatures manager term, and two helper functions. The new manager term has a latent AttributeError on physics-state .torch access.
source/isaaclab_tasks/isaaclab_tasks/core/reorient/config/shadow_hand/shadow_hand_camera_env.py Swaps direct compute_keypoints calls to compute_cube_keypoints; demotes old function to a deprecated shim with warnings.warn. Straightforward cleanup.
source/isaaclab_tasks/isaaclab_tasks/core/reorient/config/shadow_hand/shadow_hand_camera_env_cfg.py Extracts validation logic into validate_shadow_hand_camera_settings and adds the rgb_depth preset. Behavior-equivalent refactor with a clean new preset.
source/isaaclab_tasks/test/core/test_shadow_hand_camera_presets.py Adds rgb_depth to the valid-preset list and warp-valid-preset list; tests correctly cover the new preset.
source/isaaclab_tasks/changelog.d/task-cleanup-dex-part06.minor.rst Changelog entry correctly documents the new rgb_depth preset and deprecation of compute_keypoints.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[ShadowHandCameraEnvCfg.validate_config] --> B[validate_shadow_hand_camera_settings]
    B --> C{tiled_camera is PresetCfg?}
    C -- Yes --> D[resolve to .default]
    D --> C
    C -- No --> E{renderer_cfg is PresetCfg?}
    E -- Yes --> F[resolve to .default]
    F --> E
    E -- No --> G{renderer_type == newton_warp?}
    G -- Yes --> H{unsupported data_types?}
    H -- Yes --> I[raise ValueError: unsupported types]
    H -- No --> J{depth-only + CNN enabled?}
    G -- No --> J
    J -- Yes --> K[raise ValueError: depth-only]
    J -- No --> L[valid config]

    subgraph New Presets
        M[ShadowHandTiledCameraCfg]
        M --> N[rgb: RGB only 3ch]
        M --> O[rgb_depth: RGB+depth 4ch NEW]
        M --> P[depth: depth-only benchmark]
        M --> Q[albedo/shading variants]
        M --> R[default/full: RGB+depth+seg 7ch]
    end
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[ShadowHandCameraEnvCfg.validate_config] --> B[validate_shadow_hand_camera_settings]
    B --> C{tiled_camera is PresetCfg?}
    C -- Yes --> D[resolve to .default]
    D --> C
    C -- No --> E{renderer_cfg is PresetCfg?}
    E -- Yes --> F[resolve to .default]
    F --> E
    E -- No --> G{renderer_type == newton_warp?}
    G -- Yes --> H{unsupported data_types?}
    H -- Yes --> I[raise ValueError: unsupported types]
    H -- No --> J{depth-only + CNN enabled?}
    G -- No --> J
    J -- Yes --> K[raise ValueError: depth-only]
    J -- No --> L[valid config]

    subgraph New Presets
        M[ShadowHandTiledCameraCfg]
        M --> N[rgb: RGB only 3ch]
        M --> O[rgb_depth: RGB+depth 4ch NEW]
        M --> P[depth: depth-only benchmark]
        M --> Q[albedo/shading variants]
        M --> R[default/full: RGB+depth+seg 7ch]
    end
Loading

Reviews (1): Last reviewed commit: "Add renderer presets to the Direct camer..." | Re-trigger Greptile

Comment on lines +429 to +430
object_pos = object_asset.data.root_pos_w.torch - env.scene.env_origins
object_pose = torch.cat((object_pos, object_asset.data.root_quat_w.torch), dim=-1)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Unconditional .torch on physics-state tensors will raise AttributeError

object_asset.data.root_pos_w and root_quat_w return plain torch.Tensor objects — they come from PhysX, not from the Warp renderer. PyTorch tensors do not have a .torch attribute, so these two lines will raise AttributeError: 'Tensor' object has no attribute 'torch' the first time this class is exercised.

The camera output (a few lines below) handles this correctly with an isinstance guard, but the object-state path does not. The same defensive pattern is needed here.

Strict review scope: the camera manager enablement (validation helper,
keypoint relocation, manager observation terms) moves to the camera
manager part; this part keeps the rgb_depth preset, its tests, and the
Benchmark registration deprecation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant