Darling Web: Custom Views v2 — compose-your-own metrics & dashboards (#1563)#1579
Merged
Conversation
…#1563) The B1 backend for Custom Views v2 — a real metric/dimension/unit/filter/group/ chart composer over the collector store, alongside v1's fixed-read picker. - MeasureCatalog: hand-authored slice (wait/cpu/procedure/query/file_io/long-query) proving all four archetypes (cumulative/delta/gauge/per-event) + same-table ratios + the #1568 derived object_name; pinned to the real collector PayloadColumns by test. - ComposeSpec.TryParsePanel: the single write-time + compile authority — cross-checks every identifier-bearing field vs the catalog + declared view variables, with the ship-blocking DoS caps (window/bucket/topN/filter/groupBy). - ComposeCompiler: catalog-only identifiers, schema-qualified collect.*, every value bound (= ANY / LIKE / LIMIT / naive-UTC window), archetype-gated aggs, percentile per-event only, same-table ratios, unit conversion, window-bounded #1568 module join. - ValidateDefinition rewritten to dispatch v1 read panels vs v2 composed panels and to validate view-level variables + range; the v1 read-panel contract is preserved. - /api/catalog gains a v2 "compose" section; new POST /api/compose/run compiles + runs one panel spec on the least-privilege viewer pool, returning rows + the compiled SQL. - statement_timeout DoS backstop on the viewer + mcp roles (managed provisioning + tools/provision-roles.sql). Delivered as role provisioning, NOT a versioned migration: a role setting has no probeable schema footprint, so a V32 bump would break the viewer's connect-time schema-version gate. - Scrubbed the stale "loopback-only" comments (DarlingManagedRoles + provision-roles.sql) to match the shipped any-authenticated-seat networked-edit behavior. - DarlingComposeTests: catalog pins, spec validator, compiler safety (config-exclusion, bound params, #1568 bounding, archetype gating, unit conversion, DoS ceiling), ValidateDefinition v1/v2 dispatch, /api/catalog v2, and the provisioning backstop. Build + full Darling.Tests green (2580 passed, 141 live-gated skipped; Installer.Tests and DB-touching tests deliberately not run). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the two design-locked gaps on the B1 compose backend (Erik Decision 1 / §2b + acceptance Flow B; design §4), keeping every safety invariant. - server is now a universal virtual dimension (server_name, present on every source) — filterable and group-by-able on every measure (MeasureCatalog.Dimension resolves it for any source; ComposeSpec allows it regardless of a measure's AllowedDimensions). - ComposeRunContext takes an OPTIONAL/multi server scope instead of a single ServerId: null/empty $server = the whole fleet (no server predicate); one or many names => a bound server_name = ANY($n), never interpolated. So "top-N databases by CPU across the fleet" and "top servers by CPU" (the grain-trap alternative) both compile. - The #1568 module CTE is now fleet-aware: partitioned by (server_name, sql_handle), scoped to the same server set, still window-bounded; the join matches server + handle. - /api/compose/run resolves the $server variable (a name, an array, or "All"=fleet) to that scope — no server-id resolution, no hardcoded single server. - v2 viz vocabulary (line/area/bar/stacked/pie/table/stat) with viz<->mode coherence (line/area/stacked = time series; bar/pie = ranked; stat = scalar; table = any mode), so a stored def can never be un-renderable. v1 read panels keep their own KnownViz. - /api/catalog compose section serves the v2 viz vocab + a universalDimensions:[server]. - Tests extended: fleet (no-scope / multi-server bound array / group-by-server), the fleet-aware #1568 CTE, server allowed on a server-grain measure, viz<->mode coherence. Build + full Darling.Tests green (2593 passed, 141 live-gated skipped). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two end-to-end tests proving the acceptance flows validate AND compile on the B1 backend, per the build brief: - Flow A: avg procedure elapsed LIKE 'dbo.usp_Payment%' on $server, hourly -> line (schema-qualified, time-bucketed, LIKE + server bound, pattern never interpolated). - Flow B: top-10 databases by CPU across the whole fleet -> bar (grouped by database, ranked with a bound LIMIT, no server predicate = fleet-wide). Build + full Darling.Tests green (2595 passed, 141 live-gated skipped). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Author the remaining collector tables into the compose measure catalog against the same archetype/unit/dimension model B1 proved, every (table,column) pinned to the real collector PayloadColumns by the existing parameterized pins. Added measures + dimensions for 23 tables: - Cumulative+delta: latch_stats, spinlock_stats. - Gauges: cpu_scheduler_stats, plan_cache_stats, memory_stats, memory_clerks, memory_grant_stats, tempdb_stats, database_size_stats, index_object_stats (sizes + cumulative usage exposed as current-snapshot gauges since no delta column exists), session_stats, session_summary_stats, waiting_tasks, query_snapshots. - Per-event: blocked_process_reports + dmv_blocking_snapshots (count + block-duration percentile), deadlocks + system_health_events (count-only), default_trace_events, job_history. - running_jobs (gauge duration); perfmon_stats (EAV counter value/delta; unit is a 'count' placeholder because cntr_type is not stored — the user picks the counter). - query_store_stats: the correct-composable subset only (executions summable; peak duration/CPU as MAX gauges). The pre-aggregated AVG durations are deliberately omitted (a plain avg-of-avgs is wrong per design; a correct execution-weighted average needs a compiler primitive Compose lacks) — deferred as a documented follow-up, not shipped wrong. Deliberately EXCLUDED (not composable numeric metrics): the config snapshots (server_config, database_config, database_scoped_config, trace_flags, server_properties - point-in-time config, on-load only), agent_status (a boolean status snapshot), and memory_pressure_events (ordinal 0/2/3+ indicators). server stays the universal fleet dimension on every source. Tests: explicit B3 coverage (source list, count-only event compile, server-grain gauge group-by-server, per-event count + percentile) on top of the parameterized pins. Build + full Darling.Tests green (2599 passed, 141 live-gated skipped). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The bread-and-butter derived metrics the B3 fill was missing (design §2c), each a correct execution-weighted / rate ratio over same-source cumulative measures — SUM(a)/NULLIF(SUM(b),0), never an avg-of-avgs: - query/procedure avg duration + avg CPU PER EXECUTION (the correct execution-weighted form of "average duration") — adds query_executions as the divisor scalar (procedure_stats already had proc_executions). - latch avg wait per request (ms/request); file avg read latency (ms/read) — adds file_reads scalar. The existing pins already validate these (ratios reference real same-source scalars); added a ratio compile test proving the SUM/NULLIF(SUM) shape. Build + full Darling.Tests green (2600 passed, 141 live-gated skipped). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Build the B2 frontend composer on top of the v1 read-panel skeleton; both panel kinds coexist (a stored panel is composed when it names a `source`, read when it names a `read`). Composer (editor.js): - Per-panel kind toggle (Composed metric / Read); new panels default composed. - Data-driven off /api/catalog's `compose` node (never a hardcoded measure): measure/ratio picker grouped by category + self-describing caption, aggregate from the measure's validAggregates, a Shape control (Over time / Ranked / Single value => timeBucket|topN), group-by chips (allowed dims + fleet), typeahead-free filter rows (dimension + op + literal/multi/$var; LIKE only on likeable dims), and a chart picker that greys out viz types incoherent with the shape (with a reason). - View scope: a default time range + template variables ($database one-click); stored as the definition's `range` + `variables`. - Grain-trap guidance: a measure with no per-database dimension points to same-category measures that have one. - Live WYSIWYG preview via /api/compose/run; unit/percentile/per-panel-time and the compiled SQL fold away under Advanced. Render path (compose.js, new): runs a composed panel against the view scope and shapes the compiler's three row shapes into charts.js — pivots grouped time series into multi-series, ranks into bar/pie categories, formats every value in the panel's chosen unit. Empty rows are a clean no-data state; a group-by-server panel notes it only lists collecting servers. Charts (charts.js): area + stacked modes on the line chart, plus horizontal bar and donut renderers; a wider categorical palette. All SVG stays here (one SVG_NS occurrence for the air-gap scan). Saved views (pages/views.js): a view-level server (single/multi/All) + time + variable control bar that re-scopes every panel at once; composed panels dispatch through /api/compose/run, read panels through v1 renderPanel. views-api.js surfaces the compose catalog and extends the client-side import validator to composed panels. compose.js is pinned in DarlingWebAssetsTests. Build succeeded; DarlingWebAssets + air-gap self-containment tests green (26); compose + web-endpoint contract tests green (111). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…1563) Add the [Unreleased] CHANGELOG entry for the v2 metric composer (#1579) and correct a stale ComposeCompiler param-order doc comment found in the integration review (the window is $1/$2 with an optional bound server ANY, not a leading serverId). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Custom Views v2 — compose-your-own metrics & dashboards
Closes the flexibility gap in #1563. Rebuilds #1576's custom views (a picker over ~69 fixed reads, 4 viz types) into a real compose-your-own metric/dashboard model: choose the metric, filter it (incl. name patterns), group it (server/database/object), pick the aggregation, units, and chart type, and re-scope a whole dashboard at once. Reuses the entire #1574 web host + security stack and the #1576 stored-view CRUD; both panel kinds coexist (a panel is composed when it names a
source, a v1 read when it names aread).Scope was designed, hard-reviewed (a model/UX review + a security/feasibility review, both grounded in the code), and locked (custom dashboards + same-source ratios IN v1; full core pass; rollup tier deferred). Built by three subagents (backend machinery + vertical slice; full catalog fill; frontend composer); this PR is the integrated result after an orchestrator full-diff + security pass.
Backend
Compose/MeasureCatalog.cs) — ~40 measures across the queryable collector tables; unit/archetype/dimension are a hand-authored test-pinned layer overCollectorCatalog.PayloadColumns. Four archetypes (cumulative / delta / gauge / per-event) gate legal aggregations; same-source ratios give the correct execution-weighted avg-per-execution, signal-wait %, latch avg wait, file avg latency.Compose/ComposeCompiler.cs) — spec → parameterized SQL; identifiers come only from the catalog, emitted schema-qualifiedcollect.<table>; every value is bound (server_name = ANY,= ANY/LIKE, the naive-UTC window,LIMIT);date_truncbucketing; the Queries pane: attribute Top Queries rows to a named module (procedure/function) or ad hoc — Lite + Darling viewer #1568 module join is window-bounded + server-scoped.ComposeSpec.TryParsePanel— the single write-time + compile-time authority; rejects any off-catalog identifier, so a stored def always compiles and vice-versa.serveris a universal dimension; view-level$server/$database+ a global time range drive every panel; All / omitted = the whole fleet.statement_timeouton the viewer+mcp roles via role provisioning (NOT a migration — a role GUC has no probeable schema footprint, so a version bump would break the connect-time gate); 90-day window ceiling; window×bucket-count cap; forced bucketing; per-panel caps.GET /api/cataloggains acomposenode (data-driven composer);POST /api/compose/runcompiles+runs one panel on the least-privilege viewer pool →{sql, rows}.Frontend (vanilla ES modules, air-gapped)
editor.js): category-grouped measure/ratio picker, aggregate from the measure's valid set, a Shape control (over-time / ranked / single), group-by chips, filter rows (dimension + op + literal/multi/$var;LIKEonly on likeable dims), a chart picker that greys viz types incoherent with the shape, grain-trap guidance, and a live WYSIWYG preview.compose.js): pivots the compiler's three row shapes into line/area/stacked/bar/pie/stat/table; honest no-data / partial-fleet / error states. All untrusted text reaches the DOM viatextContent.Safety
Network-reachable surface: no arbitrary SQL (allowlisted catalog identifiers only, schema-qualified
collect.*, never rides search_path); the config control-plane is excluded by the allowlist (verified by test); CSRF via theapplication/jsonrequirement; window clamp +statement_timeoutas the runaway backstop.Testing
Independently verified by the orchestrator: build 0 errors; full
Darling.Testsgreen — 2601 passed / 0 failed / 141 gated-live skipped. Includes catalog (table,column)/time-column/gauge-non-summable/ratio-scalar pins, spec validation (off-catalog + percentile-on-non-per-event + LIKE-on-non-likeable + undeclared-variable + caps), compiler safety (schema-qualified, all-bound, config-excluded, #1568-bounded), viz coherence, and explicit Flow A (proc-nameLIKEon$server→ line) + Flow B (top-10 DBs by CPU across the fleet → bar). Installer.Tests + DB-touching tests deliberately not run.Deferred (documented follow-ups — honest limits, not silent holes)
Gauge-based ratios (grant used/ideal, single-use-plan %) + a Query Store execution-weighted AVG need an avg/avg per-row compiler primitive Compose doesn't have yet (QS ships executions + peak duration/CPU, which are correct); per-server measure-availability greying needs an
appliesTocatalog field (no-data + partial-fleet states cover honest degradation); stacked-bar (stacked-area today), thresholds/annotations, per-panel drill-down, and notebook mode are phase-2.🤖 Generated with Claude Code