Skip to content

feat(tools): updated experiment infra to run experiemnts against Clickhouse using different strategies such as using materialized views, or using sketches#495

Draft
milindsrivastava1997 wants to merge 3 commits into
mainfrom
491-featasap-tools-benchmark-asapquery-against-clickhouse-precompute-strategies-sketches-materialized-views-and-both
Draft

feat(tools): updated experiment infra to run experiemnts against Clickhouse using different strategies such as using materialized views, or using sketches#495
milindsrivastava1997 wants to merge 3 commits into
mainfrom
491-featasap-tools-benchmark-asapquery-against-clickhouse-precompute-strategies-sketches-materialized-views-and-both

Conversation

@milindsrivastava1997

@milindsrivastava1997 milindsrivastava1997 commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Issue #491 needs ClickHouse benchmarked using its own precompute mechanisms (sketch functions, materialized views, materialized views + sketches) as additional comparison arms against ASAPQuery, alongside the existing exact baseline and sketchdb arms. Each new arm needs different SQL (and in some cases a different schema) than baseline, not just a different protocol/server.

This PR lays the config plumbing for that:

  • Adds mode constants for the 3 new arms: baseline_sketch, baseline_mv, baseline_mv_sketch (asap-tools/experiments/constants.py).
  • generate_clickhouse_client_configs now resolves each query group's SQL per mode, inside the mode loop, instead of building one shared query list once and reusing it for every mode (which is why baseline and sketchdb always got identical queries today).
  • sketchdb always resolves its SQL through the baseline entry — ASAP's SQL-mode planner needs to see the exact same SQL text as baseline to pattern-match it against available sketches; it can't understand ClickHouse-specific functions or MV-backed FROM clauses.

sql_file format change (breaking)

query_groups[i].sql_file is now a dict keyed by mode name instead of a single shared path, since each precompute arm may query different SQL/schema (e.g. baseline_mv reads from a materialized view instead of the raw table).

Old — one path shared by every mode:

query_groups:
  - name: default_queries
    sql_file: /path/to/baseline.sql

New — dict keyed by mode name:

query_groups:
  - name: default_queries
    sql_file:
      baseline: /path/to/baseline.sql
      baseline_sketch: /path/to/baseline_sketch.sql
      baseline_mv: /path/to/baseline_mv.sql
      baseline_mv_sketch: /path/to/baseline_mv_sketch.sql
      # no "sketchdb" key — it always resolves through "baseline"

For a baseline + sketchdb-only experiment (today's common case), the new format only needs a baseline entry:

query_groups:
  - name: default_queries
    sql_file:
      baseline: /path/to/baseline.sql

Existing local configs need to be updated to the dict format to keep working.

Out of scope (left for follow-up)

  • The actual DDL (materialized views + target tables) and per-arm SQL query files

…khouse using different strategies such as using materialized views, or using sketches
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(asap-tools): benchmark ASAPQuery against ClickHouse precompute strategies (sketches, materialized views, and both)

1 participant