Skip to content

Nest cloudlab/local config under a providers: section in config.yaml #440

Description

@milindsrivastava1997

Context

#424 added local-machine execution via a new top-level local: config section, injected at the CLI with Hydra's +local.home_dir=... syntax (since local isn't a structural key in config/config.yaml by default — providers/factory.py's create_provider() and detect_provider_type() use hasattr(cfg, "local") to decide which provider to build, and that only works because the key is absent unless explicitly added).

Proposal

Restructure config.yaml so cloudlab: and local: both live under a providers: namespace:

providers:
  cloudlab:
    num_nodes: ???
    node_offset: 0
    username: ???
    hostname_suffix: ???
  local:
    home_dir: null

This is more symmetric (no more "one provider gets a real CLI flag, the other needs Hydra's + add-key syntax") and lets detection switch from presence of a section to the value of providers.local.home_dir (set vs null) — actually a cleaner signal than hasattr, and avoids the + prefix entirely (providers.local.home_dir=... works with normal Hydra override syntax once the key exists with a default).

Why this is a separate issue, not part of #424

config/config.yaml is shared by every Hydra-based entrypoint in this directory, not just the two #424 touched (experiment_run_e2e.py, experiment_run_clickhouse.py). In particular:

  • The YAML interpolation streaming.remote_write.ip: "${remote_write_ip:${cloudlab.node_offset}}" lives in config.yaml itself and would need to become ${providers.cloudlab.node_offset} — this affects every script that loads config_name="config", including ones Support running experiments fully locally (no CloudLab dependency) #424 explicitly left untouched (experiment_teardown_everything.py, experiment_run_sketchdboffline.py, experiment_run_exporters_and_prometheus.py, experiment_run_grafana_demo.py, experiment_only_ingest_path.py, grafana_config.py).
  • grafana_config.py and experiment_run_clickhouse.py have CLI usage examples in docstrings/help text (cloudlab.num_nodes=2, etc.) that would go stale.
  • HYDRA_CONFIG_USAGE.md and CONFIG_PARAMETERS_REFERENCE.md document the current flat cloudlab.* paths extensively.

Scope of the change (when picked up)

  • config/config.yaml — nest both sections under providers:, update the remote_write_ip resolver interpolation
  • experiment_utils/config.pyget_node_params(), required_cloudlab_params(), Args.__init__ (all currently read cfg.cloudlab.*)
  • experiment_utils/providers/factory.pycreate_provider()/detect_provider_type() (currently hasattr(cfg, "local") / hasattr(cfg, "cloudlab"))
  • experiment_run_clickhouse.py, grafana_config.py — update CLI usage docstrings
  • HYDRA_CONFIG_USAGE.md, CONFIG_PARAMETERS_REFERENCE.md — update documented config paths
  • experiment_utils/test_local_provider.py — update the self-check's assertions

Not urgent; current local:/+local.home_dir=... setup (#424) works and is tested. This is a cosmetic/ergonomic follow-up.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions