feat(asap-tools): add remote_monitor (CPU/memory) support to experiment_run_clickhouse.py

[akanksha-akkihal]

Changes

experiment_run_clickhouse.py

• New flow.remote_monitor flag gates monitoring.
• _run_query_workload() wraps the prometheus-client invocation in RemoteMonitorService when enabled; otherwise unchanged _run_query_client().
• _clickhouse_monitor_keywords() selects ps-matchable process keywords: clickhouse (baseline) and query_engine_rust (sketchdb).
• Waits for the timed monitor to finish before teardown.

config/config.yaml

• Added flow.remote_monitor: false (default off).

RemoteMonitorService.start()

• New backend_type and monitor_keywords parameters.
• When monitor_keywords is provided, use it directly (skip Prometheus-config keyword discovery).
• Invoke python3.11 (deps live there on the node) instead of python3.
• Pass --backend_type through to remote_monitor.py.
• arroyo_service made optional.

remote_monitor.py

• New --backend_type arg, forwarded to PrometheusClientService (was hardcoded "prometheus"); clickhouse skips PromQL query-time alignment.
• Missing keyword → warn and skip instead of raising, so monitoring tolerates absent processes.

Issues fixed along the way

• python3 on the node lacked loguru/experiment deps → switched to python3.11.
• clickhouse-server container name didn't match ps (host-network bare-metal ClickHouse) → use clickhouse keyword.
• A defaulted parameter preceded required params in start() → reordered.

Usage

python3 experiment_run_clickhouse.py \
  experiment_type=clickhouse experiment.name=netflow_2mrps \
  providers.cloudlab.num_nodes=1 providers.cloudlab.username=<user> \
  providers.cloudlab.hostname_suffix=<suffix> \
  use_container.prometheus_client=false use_container.query_engine=false \
  use_container.controller=false \
  flow.remote_monitor=true

Then:

python3 post_experiment/analyze_monitor_output.py \
  experiment_outputs/<name>/<mode>/remote_monitor_output/monitor_output.json \
  --print --plot --save --save-to-experiment-dir

Output

experiment_outputs/<name>/<baseline|sketchdb>/remote_monitor_output/monitor_output.json

[milindsrivastava1997]

Follow-up cleanup on the variable names introduced here:

• monitoring_tool renamed to backend_tool: it describes which concrete tool is running on the node for monitoring ("prometheus" or "victoriametrics"), which controls which process keywords remote_monitor_service searches for.
• backend_type renamed to backend_protocol: it describes which query protocol the system under test speaks ("prometheus" = PromQL, "clickhouse" = SQL), which controls whether the prometheus-client applies --align-query-time.

The two params are related (both about the backend stack) but not the same axis — the shared backend_ prefix signals the relationship, while _tool vs _protocol signals the distinction. On the ClickHouse path backend_tool is now Optional (defaulting to None) since no TSDB runs there and the value was previously a dead dummy "clickhouse".