Skip to content

feat(experimentation): add experiment exposures model, warehouse query and payload mapper#7740

Open
gagantrivedi wants to merge 12 commits into
mainfrom
feat/experiment-exposure-snapshots
Open

feat(experimentation): add experiment exposures model, warehouse query and payload mapper#7740
gagantrivedi wants to merge 12 commits into
mainfrom
feat/experiment-exposure-snapshots

Conversation

@gagantrivedi

@gagantrivedi gagantrivedi commented Jun 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Thanks for submitting a PR! Please check the boxes below:

  • I have read the Contributing Guide.
  • I have added information to docs/ if required so people know about the feature.
  • I have filled in the "Changes" section below.
  • I have filled in the "How did you test this code" section below.

Changes

Contributes to the experimentation results layer (v0.1: experiment exposures panel).

First backend slice — no tasks, API or UI yet (those follow in stacked PRs):

  • ExperimentExposures model: one row per experiment (OneToOne), updated in place by a future task. A failed refresh records last_error/last_error_at without clobbering the last good payload/as_of, and there is no append-only history to prune. Each refresh recomputes the full window (started_at → as_of) — identity dedup, multi-variant quarantine and late event delivery make incremental windows unsound.
  • get_exposure_buckets ClickHouse query: per-variant, per-time-bucket counts of first-exposed identities from $flag_exposure events — identities count once, in the bucket of their first exposure (immune to at-least-once duplicate delivery); identities seen in more than one variant are quarantined under a $multiple sentinel; buckets are hourly for windows up to 72 hours, daily beyond.
  • Pure payload mapper: totals, observed shares per variant, and the cumulative time series, shaped as the JSON the exposures API will serve.

No docs changes: the feature is flag-gated and not user-visible yet; docs land with the UI.

How did you test this code?

  • Unit tests for the mapper (totals/shares, control-first ordering, $multiple exclusion, carry-forward cumulative series, day rounding, UTC serialisation) and the query service (bucket granularity selection, row mapping).
  • pytest tests/unit/experimentation/ — 249 passed; ruff and mypy strict clean.

@gagantrivedi gagantrivedi requested a review from a team as a code owner June 10, 2026 09:19
@gagantrivedi gagantrivedi requested review from khvn26 and removed request for a team June 10, 2026 09:19
@vercel

vercel Bot commented Jun 10, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview, Comment Jun 10, 2026 11:49am
2 Skipped Deployments
Project Deployment Actions Updated (UTC)
flagsmith-frontend-preview Ignored Ignored Preview Jun 10, 2026 11:49am
flagsmith-frontend-staging Ignored Ignored Preview Jun 10, 2026 11:49am

Request Review

@github-actions github-actions Bot added api Issue related to the REST API feature New feature or request labels Jun 10, 2026
@github-actions

github-actions Bot commented Jun 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Docker builds report

Image Build Status Security report
ghcr.io/flagsmith/flagsmith-e2e:pr-7740 Finished ✅ Skipped
ghcr.io/flagsmith/flagsmith-frontend:pr-7740 Finished ✅ Results ✅
ghcr.io/flagsmith/flagsmith-api-test:pr-7740 Finished ✅ Skipped
ghcr.io/flagsmith/flagsmith-api:pr-7740 Finished ✅ Results ✅
ghcr.io/flagsmith/flagsmith:pr-7740 Finished ✅ Results ✅
ghcr.io/flagsmith/flagsmith-private-cloud:pr-7740 Finished ✅ Results ✅

@gagantrivedi

gagantrivedi commented Jun 10, 2026

Copy link
Copy Markdown
Member Author

/gemini review

@github-actions

github-actions Bot commented Jun 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  33 seconds
commit  f41c6e6
info  🔄 Run: #17372 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  34 seconds
commit  f41c6e6
info  🔄 Run: #17372 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  43.7 seconds
commit  f41c6e6
info  🔄 Run: #17372 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  35.3 seconds
commit  f41c6e6
info  🔄 Run: #17372 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  32.8 seconds
commit  7648791
info  🔄 Run: #17375 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  38.1 seconds
commit  7648791
info  🔄 Run: #17375 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  3 passed

Details

stats  3 tests across 3 suites
duration  32.8 seconds
commit  7648791
info  🔄 Run: #17375 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  3 passed

Details

stats  3 tests across 3 suites
duration  1 minute, 2 seconds
commit  7648791
info  🔄 Run: #17375 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  33.1 seconds
commit  dfd2b72
info  🔄 Run: #17383 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  43.5 seconds
commit  dfd2b72
info  🔄 Run: #17383 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  3 passed

Details

stats  3 tests across 3 suites
duration  34.1 seconds
commit  dfd2b72
info  🔄 Run: #17383 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  42.9 seconds
commit  dfd2b72
info  🔄 Run: #17383 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  19 passed

Details

stats  19 tests across 15 suites
duration  1 minute, 5 seconds
commit  7f49a1f
info  🔄 Run: #17384 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  39.6 seconds
commit  7f49a1f
info  🔄 Run: #17384 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  45.3 seconds
commit  7f49a1f
info  🔄 Run: #17384 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  31.4 seconds
commit  7f49a1f
info  🔄 Run: #17384 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  34.1 seconds
commit  6d45a2a
info  🔄 Run: #17385 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  43.3 seconds
commit  6d45a2a
info  🔄 Run: #17385 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  32.4 seconds
commit  6d45a2a
info  🔄 Run: #17385 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  3 passed

Details

stats  3 tests across 3 suites
duration  49 seconds
commit  6d45a2a
info  🔄 Run: #17385 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  40.8 seconds
commit  edb4f2c
info  🔄 Run: #17386 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  43.7 seconds
commit  edb4f2c
info  🔄 Run: #17386 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  41.2 seconds
commit  edb4f2c
info  🔄 Run: #17386 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  1 minute, 1 second
commit  edb4f2c
info  🔄 Run: #17386 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  40.9 seconds
commit  2645353
info  🔄 Run: #17387 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  44.4 seconds
commit  2645353
info  🔄 Run: #17387 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  32.4 seconds
commit  2645353
info  🔄 Run: #17387 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  57.1 seconds
commit  2645353
info  🔄 Run: #17387 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  19 passed

Details

stats  19 tests across 15 suites
duration  1 minute, 11 seconds
commit  a1598cb
info  🔄 Run: #17388 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  45.1 seconds
commit  a1598cb
info  🔄 Run: #17388 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  39.5 seconds
commit  a1598cb
info  🔄 Run: #17388 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  4 passed

Details

stats  4 tests across 4 suites
duration  33.1 seconds
commit  a1598cb
info  🔄 Run: #17388 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  32.9 seconds
commit  7d32960
info  🔄 Run: #17389 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  43.5 seconds
commit  7d32960
info  🔄 Run: #17389 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  3 passed

Details

stats  3 tests across 3 suites
duration  34 seconds
commit  7d32960
info  🔄 Run: #17389 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  3 passed

Details

stats  3 tests across 3 suites
duration  40.2 seconds
commit  7d32960
info  🔄 Run: #17389 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  40.7 seconds
commit  5e7665f
info  🔄 Run: #17390 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  45.6 seconds
commit  5e7665f
info  🔄 Run: #17390 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  3 passed

Details

stats  3 tests across 3 suites
duration  40.9 seconds
commit  5e7665f
info  🔄 Run: #17390 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  3 passed

Details

stats  3 tests across 3 suites
duration  39.6 seconds
commit  5e7665f
info  🔄 Run: #17390 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  40.8 seconds
commit  f63c59d
info  🔄 Run: #17391 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  44 seconds
commit  f63c59d
info  🔄 Run: #17391 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  3 passed

Details

stats  3 tests across 3 suites
duration  48.6 seconds
commit  f63c59d
info  🔄 Run: #17391 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  3 passed

Details

stats  3 tests across 3 suites
duration  43.3 seconds
commit  f63c59d
info  🔄 Run: #17391 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  40.6 seconds
commit  c832b05
info  🔄 Run: #17392 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  45.7 seconds
commit  c832b05
info  🔄 Run: #17392 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  32.6 seconds
commit  c832b05
info  🔄 Run: #17392 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  57 seconds
commit  c832b05
info  🔄 Run: #17392 (attempt 1)

gemini-code-assist[bot]
gemini-code-assist Bot reviewed Jun 10, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements the backend infrastructure for tracking and snapshotting experiment exposures from the data warehouse. It introduces the ExperimentExposureSnapshot Django model, ClickHouse queries to bucket exposures by hour or day, and mapper functions to build the exposure payload. The review feedback highlights a performance optimization opportunity in _build_timeseries_points within api/experimentation/mappers.py to reduce its time complexity from O(N^2) to O(N log N) by pre-grouping buckets.

Comment thread api/experimentation/mappers.py Outdated
Comment on lines +91 to +107
def _build_timeseries_points(
buckets: Sequence[ExposureBucket],
) -> list[ExposureTimeseriesPoint]:
variant_keys = {b.variant for b in buckets}
running = dict.fromkeys(variant_keys, 0)
points: list[ExposureTimeseriesPoint] = []
for bucket_start in sorted({b.bucket for b in buckets}):
for b in buckets:
if b.bucket == bucket_start:
running[b.variant] += b.new_units
points.append(
ExposureTimeseriesPoint(
bucket=_isoformat_utc(bucket_start),
cumulative_units=dict(running),
)
)
return points

@gemini-code-assist gemini-code-assist Bot Jun 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The current implementation of _build_timeseries_points has an $O(U \times N)$ time complexity (where $N$ is the number of buckets and $U$ is the number of unique bucket timestamps). In the worst case, this is $O(N^2)$. For long-running experiments with many variants or hourly granularities, $N$ can grow to several thousands, leading to millions of iterations and potential API timeouts or high CPU utilization.

We can optimize this to $O(N \log N)$ by grouping the buckets by their start time using a dictionary before iterating over the sorted unique timestamps.

def _build_timeseries_points(
    buckets: Sequence[ExposureBucket],
) -> list[ExposureTimeseriesPoint]:
    variant_keys = {b.variant for b in buckets}
    running = dict.fromkeys(variant_keys, 0)
    
    buckets_by_start: dict[datetime, list[ExposureBucket]] = {}
    for b in buckets:
        buckets_by_start.setdefault(b.bucket, []).append(b)
        
    points: list[ExposureTimeseriesPoint] = []
    for bucket_start in sorted(buckets_by_start.keys()):
        for b in buckets_by_start[bucket_start]:
            running[b.variant] += b.new_units
        points.append(
            ExposureTimeseriesPoint(
                bucket=_isoformat_utc(bucket_start),
                cumulative_units=dict(running),
            )
        )
    return points

@gagantrivedi gagantrivedi Jun 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in dfd2b72 (both here and in _build_variants, which had the same shape). Worth noting the practical bounds, though: granularity selection caps buckets at ~72 hourly points (windows ≤72h) or ~90 daily points (90-day events TTL), and this mapper runs in the task processor when a snapshot is computed — never in the request path — so the quadratic scan couldn't reach API latency. Taking the grouped version anyway since it's no less readable and stays safe if the granularity rules ever loosen.

@github-actions

github-actions Bot commented Jun 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Visual Regression

19 screenshots compared. See report for details.
View full report

@gagantrivedi gagantrivedi requested a review from a team as a code owner June 10, 2026 09:27
@github-actions github-actions Bot added the docs Documentation updates label Jun 10, 2026
@github-actions github-actions Bot added feature New feature or request and removed feature New feature or request docs Documentation updates labels Jun 10, 2026
@codecov

codecov Bot commented Jun 10, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.55%. Comparing base (a8a7758) to head (c832b05).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files 
@@           Coverage Diff            @@
##             main    #7740    +/-   ##
========================================
  Coverage   98.54%   98.55%            
========================================
  Files        1452     1453     +1     
  Lines       55821    56001   +180     
========================================
+ Hits        55011    55191   +180     
  Misses        810      810

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@gagantrivedi
refactor(experimentation): rename exposure units to first_exposed_ide…
dfd2b72 
…ntities

Adopt Flagsmith's product vocabulary (identities) over experimentation
jargon (units) across the dataclass, payload and SQL alias, and group
buckets before iterating in the mapper helpers.
@github-actions github-actions Bot added the docs Documentation updates label Jun 10, 2026
@github-actions github-actions Bot added feature New feature or request and removed feature New feature or request docs Documentation updates labels Jun 10, 2026
@github-actions github-actions Bot added docs Documentation updates and removed feature New feature or request labels Jun 10, 2026
@gagantrivedi
refactor(experimentation): address design review findings
a1598cb 
- compute_exposures_payload orchestrates granularity selection, query
  and mapping behind one entry point; granularity can no longer diverge
  between the query and the payload, and the selection policy moves out
  of the mapper.
- Bucket datetimes are normalised to aware UTC at the service boundary,
  removing the driver workaround from the pure mapper.
- ExperimentExposures gains record_refresh/record_failure so the
  failure-preserves-payload invariant lives on the model; last_error is
  dropped (structured logs cover operators, last_error_at covers the UI).
@github-actions github-actions Bot added docs Documentation updates feature New feature or request and removed feature New feature or request docs Documentation updates labels Jun 10, 2026
@gagantrivedi
refactor(experimentation): pin exposure bucket boundaries to UTC in t…
7d32960 
…he query

Passing the timezone to the bucket function makes the result column
UTC-typed, so the driver returns aware datetimes natively — and day
boundaries no longer depend on the ClickHouse server timezone.
@github-actions github-actions Bot added docs Documentation updates feature New feature or request and removed feature New feature or request docs Documentation updates labels Jun 10, 2026
@gagantrivedi
refactor(experimentation): drop derivable share from the exposures pa…
5e7665f 
…yload

Consumers receive identities per variant and the total; the share is
one division away and the field couldn't name its own denominator.
@github-actions github-actions Bot added docs Documentation updates feature New feature or request and removed feature New feature or request docs Documentation updates labels Jun 10, 2026
@gagantrivedi
refactor(experimentation): replace mapper TypedDicts with summary dat…
f63c59d 
…aclasses

The exposures computation now constructs frozen dataclasses
(ExposuresSummary and parts) instead of hand-knitting TypedDict
payloads; the JSON shape is defined once, by asdict() at the model
boundary. mappers.py folds into services as the pure half of the
single compute_exposures_summary entry point.
@github-actions github-actions Bot added the docs Documentation updates label Jun 10, 2026
@github-actions github-actions Bot added feature New feature or request and removed feature New feature or request docs Documentation updates labels Jun 10, 2026
@gagantrivedi
refactor(experimentation): collapse variant summaries to an identitie…
c832b05 
…s-by-variant mapping

is_control is derivable from the reserved "control" key and the
control-first ordering is presentation — both move to the UI. The
variants list becomes a plain mapping, symmetric with the timeseries
points.
@github-actions github-actions Bot added docs Documentation updates feature New feature or request and removed feature New feature or request docs Documentation updates labels Jun 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@khvn26 khvn26 Awaiting requested review from khvn26 khvn26 is a code owner automatically assigned from Flagsmith/flagsmith-back-end

1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

api Issue related to the REST API feature New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@gagantrivedi @khvn26