Summary
Today the netflow benchmark compares ASAPQuery (sketchdb) against a single ClickHouse arm: exact aggregation over the raw MergeTree table (baseline). This is a useful correctness reference, but it is not a fair performance comparison — ClickHouse is doing full-scan exact work while ASAP serves results from pre-computed sketches.
To make a credible apples-to-apples claim, we need to compare ASAPQuery against ClickHouse configured to also precompute, using the mechanisms ClickHouse actually offers. There are three distinct ClickHouse strategies worth measuring, and we currently support none of them.
Comparisons to support
| Comparison |
ClickHouse strategy |
Question it answers |
| A. ASAP vs ClickHouse with sketches |
Approximate sketch functions at query time over raw data |
Does ASAP's ingest-time precompute beat ClickHouse's query-time sketches? |
| B. ASAP vs ClickHouse with materialized views |
Pre-aggregated (exact) state maintained at ingest |
Does ASAP beat ClickHouse's classic incremental pre-aggregation? |
| C. ASAP vs ClickHouse with materialized views + sketches |
Pre-aggregated sketch state maintained at ingest |
The fairest match: both sides precompute approximate state — who wins on latency/resources at equal accuracy? |
The exact baseline arm remains as the shared ground truth for accuracy across all three.
Goals / what this should achieve
- Run all comparisons over the same netflow workload and queries
- Produce, for each comparison, the metrics we already track: query latency, speedup vs ASAP, and CPU/memory of the monitored backend process.
- Make the comparisons reproducible from a single experiment run where practical (one data load, multiple backend strategies), so results are directly comparable and not skewed by separate loads.
- Slot cleanly into the existing post-experiment analysis and plotting so the three ClickHouse strategies appear as additional series alongside ASAP.
Acceptance criteria
Summary
Today the netflow benchmark compares ASAPQuery (sketchdb) against a single ClickHouse arm: exact aggregation over the raw
MergeTreetable (baseline). This is a useful correctness reference, but it is not a fair performance comparison — ClickHouse is doing full-scan exact work while ASAP serves results from pre-computed sketches.To make a credible apples-to-apples claim, we need to compare ASAPQuery against ClickHouse configured to also precompute, using the mechanisms ClickHouse actually offers. There are three distinct ClickHouse strategies worth measuring, and we currently support none of them.
Comparisons to support
The exact
baselinearm remains as the shared ground truth for accuracy across all three.Goals / what this should achieve
Acceptance criteria
baseline/sketchdboutputs.baselineis reported for every approximate arm.