Skip to content

feat(asap-tools): benchmark ASAPQuery against ClickHouse precompute strategies (sketches, materialized views, and both) #491

Description

@akanksha-akkihal

Summary

Today the netflow benchmark compares ASAPQuery (sketchdb) against a single ClickHouse arm: exact aggregation over the raw MergeTree table (baseline). This is a useful correctness reference, but it is not a fair performance comparison — ClickHouse is doing full-scan exact work while ASAP serves results from pre-computed sketches.

To make a credible apples-to-apples claim, we need to compare ASAPQuery against ClickHouse configured to also precompute, using the mechanisms ClickHouse actually offers. There are three distinct ClickHouse strategies worth measuring, and we currently support none of them.

Comparisons to support

Comparison ClickHouse strategy Question it answers
A. ASAP vs ClickHouse with sketches Approximate sketch functions at query time over raw data Does ASAP's ingest-time precompute beat ClickHouse's query-time sketches?
B. ASAP vs ClickHouse with materialized views Pre-aggregated (exact) state maintained at ingest Does ASAP beat ClickHouse's classic incremental pre-aggregation?
C. ASAP vs ClickHouse with materialized views + sketches Pre-aggregated sketch state maintained at ingest The fairest match: both sides precompute approximate state — who wins on latency/resources at equal accuracy?

The exact baseline arm remains as the shared ground truth for accuracy across all three.

Goals / what this should achieve

  • Run all comparisons over the same netflow workload and queries
  • Produce, for each comparison, the metrics we already track: query latency, speedup vs ASAP, and CPU/memory of the monitored backend process.
  • Make the comparisons reproducible from a single experiment run where practical (one data load, multiple backend strategies), so results are directly comparable and not skewed by separate loads.
  • Slot cleanly into the existing post-experiment analysis and plotting so the three ClickHouse strategies appear as additional series alongside ASAP.

Acceptance criteria

  • Each of the three comparisons (A, B, C) can be run against the netflow workload.
  • Latency, speedup, and resource results are generated for each, comparable to current baseline/sketchdb outputs.
  • Per-query accuracy vs exact baseline is reported for every approximate arm.
  • Results are visualized alongside the existing ASAP-vs-baseline plots.
  • Known per-query semantic caveats are documented with the results.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions