You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#424 added local-machine execution via a new top-level local: config section, injected at the CLI with Hydra's +local.home_dir=... syntax (since local isn't a structural key in config/config.yaml by default — providers/factory.py's create_provider() and detect_provider_type() use hasattr(cfg, "local") to decide which provider to build, and that only works because the key is absent unless explicitly added).
Proposal
Restructure config.yaml so cloudlab: and local: both live under a providers: namespace:
This is more symmetric (no more "one provider gets a real CLI flag, the other needs Hydra's + add-key syntax") and lets detection switch from presence of a section to the value of providers.local.home_dir (set vs null) — actually a cleaner signal than hasattr, and avoids the + prefix entirely (providers.local.home_dir=... works with normal Hydra override syntax once the key exists with a default).
config/config.yaml is shared by every Hydra-based entrypoint in this directory, not just the two #424 touched (experiment_run_e2e.py, experiment_run_clickhouse.py). In particular:
The YAML interpolation streaming.remote_write.ip: "${remote_write_ip:${cloudlab.node_offset}}" lives in config.yaml itself and would need to become ${providers.cloudlab.node_offset} — this affects every script that loads config_name="config", including ones Support running experiments fully locally (no CloudLab dependency) #424 explicitly left untouched (experiment_teardown_everything.py, experiment_run_sketchdboffline.py, experiment_run_exporters_and_prometheus.py, experiment_run_grafana_demo.py, experiment_only_ingest_path.py, grafana_config.py).
grafana_config.py and experiment_run_clickhouse.py have CLI usage examples in docstrings/help text (cloudlab.num_nodes=2, etc.) that would go stale.
HYDRA_CONFIG_USAGE.md and CONFIG_PARAMETERS_REFERENCE.md document the current flat cloudlab.* paths extensively.
Scope of the change (when picked up)
config/config.yaml — nest both sections under providers:, update the remote_write_ip resolver interpolation
experiment_utils/config.py — get_node_params(), required_cloudlab_params(), Args.__init__ (all currently read cfg.cloudlab.*)
Context
#424 added local-machine execution via a new top-level
local:config section, injected at the CLI with Hydra's+local.home_dir=...syntax (sincelocalisn't a structural key inconfig/config.yamlby default —providers/factory.py'screate_provider()anddetect_provider_type()usehasattr(cfg, "local")to decide which provider to build, and that only works because the key is absent unless explicitly added).Proposal
Restructure
config.yamlsocloudlab:andlocal:both live under aproviders:namespace:This is more symmetric (no more "one provider gets a real CLI flag, the other needs Hydra's
+add-key syntax") and lets detection switch from presence of a section to the value ofproviders.local.home_dir(set vsnull) — actually a cleaner signal thanhasattr, and avoids the+prefix entirely (providers.local.home_dir=...works with normal Hydra override syntax once the key exists with a default).Why this is a separate issue, not part of #424
config/config.yamlis shared by every Hydra-based entrypoint in this directory, not just the two #424 touched (experiment_run_e2e.py,experiment_run_clickhouse.py). In particular:streaming.remote_write.ip: "${remote_write_ip:${cloudlab.node_offset}}"lives inconfig.yamlitself and would need to become${providers.cloudlab.node_offset}— this affects every script that loadsconfig_name="config", including ones Support running experiments fully locally (no CloudLab dependency) #424 explicitly left untouched (experiment_teardown_everything.py,experiment_run_sketchdboffline.py,experiment_run_exporters_and_prometheus.py,experiment_run_grafana_demo.py,experiment_only_ingest_path.py,grafana_config.py).grafana_config.pyandexperiment_run_clickhouse.pyhave CLI usage examples in docstrings/help text (cloudlab.num_nodes=2, etc.) that would go stale.HYDRA_CONFIG_USAGE.mdandCONFIG_PARAMETERS_REFERENCE.mddocument the current flatcloudlab.*paths extensively.Scope of the change (when picked up)
config/config.yaml— nest both sections underproviders:, update theremote_write_ipresolver interpolationexperiment_utils/config.py—get_node_params(),required_cloudlab_params(),Args.__init__(all currently readcfg.cloudlab.*)experiment_utils/providers/factory.py—create_provider()/detect_provider_type()(currentlyhasattr(cfg, "local")/hasattr(cfg, "cloudlab"))experiment_run_clickhouse.py,grafana_config.py— update CLI usage docstringsHYDRA_CONFIG_USAGE.md,CONFIG_PARAMETERS_REFERENCE.md— update documented config pathsexperiment_utils/test_local_provider.py— update the self-check's assertionsNot urgent; current
local:/+local.home_dir=...setup (#424) works and is tested. This is a cosmetic/ergonomic follow-up.