Skip to content

Add is_first_party label to CLI command metrics#7758

Open
SimonSiju wants to merge 2 commits into
mainfrom
simon/1p-org-migration-cli-metrics-22933
Open

Add is_first_party label to CLI command metrics#7758
SimonSiju wants to merge 2 commits into
mainfrom
simon/1p-org-migration-cli-metrics-22933

Conversation

@SimonSiju

Copy link
Copy Markdown

WHY are these changes introduced?

closes: https://github.com/shop/issues-develop/issues/22933

Part of the 1P org migration observability work (GSD #50881). "1P CI/CD pipelines function without breakage after cutover" is a named guardrail on the [Dev Platform] 1P Orgs Migration dashboard (Observe si2kjvt), but today it can't be measured: cli_commands_total (and the companion duration/elapsed metrics) carry only cli_version, exit, and job labels — no org / 1P dimension — so the CLI guardrail panels blend 1P + 3P + all external developers.

WHAT is this pull request doing?

Adds an is_first_party label to the CLI command metrics emitted from cli-kit:

  • cli_commands_total
  • cli_commands_duration_ms
  • cli_commands_wall_clock_elapsed_ms

The label is keyed off the 1P dev path via firstPartyDev() (the SHOPIFY_CLI_1P_DEV env var), which is the same signal that sets the X-Shopify-Cli-Employee header. Values are the strings "true" / "false", keeping cardinality low.

This lets the dashboard's CLI guardrail panels filter to 1P, so we get a 1P-scoped success/failure signal for commands like app dev, app dev clean, and generate extension after cutover.

Files changed:

  • packages/cli-kit/src/private/node/otel-metrics.ts — added is_first_party to the Labels type and populated it from firstPartyDev() in recordMetrics; imported firstPartyDev from context/local.
  • packages/cli-kit/src/private/node/otel-metrics.test.ts — added a test asserting all recorded metrics carry is_first_party: "true" when SHOPIFY_CLI_1P_DEV is set; added afterEach env cleanup.
  • packages/cli-kit/src/private/node/__snapshots__/otel-metrics.test.ts.snap — regenerated to include the new label ("false" when the 1P dev path is off).
  • .changeset/first-party-cli-metrics-label.md@shopify/cli-kit patch changeset.

How to test your changes?

  1. From the repo root, run the unit tests for the metrics module:
    pnpm --filter @shopify/cli-kit vitest run src/private/node/otel-metrics.test.ts
    
    All three tests should pass, including labels metrics as first-party when the 1P dev path is enabled.
  2. Manually verify the label is emitted. With metrics debug output enabled, run a CLI command as a third-party developer and confirm cli_commands_total records is_first_party: "false".
  3. Re-run the same command with the 1P dev path enabled (SHOPIFY_CLI_1P_DEV=1 shopify app dev) and confirm the metric records is_first_party: "true".

Checklist

  • I've considered possible cross-platform impacts (Mac, Linux, Windows)
  • I've considered possible documentation changes
  • I've considered analytics changes to measure impact
  • The change is user-facing — I've identified the correct bump type (patch for bug fixes · minor for new features · major for breaking changes) and added a changeset with pnpm changeset add

Adds an is_first_party label to cli_commands_total (and the duration /
wall-clock-elapsed metrics) so first-party (1P) Shopify developer usage
can be distinguished from third-party usage in observability dashboards.

The label is keyed off the 1P dev path (SHOPIFY_CLI_1P_DEV) via
firstPartyDev(), the same signal that sets the X-Shopify-Cli-Employee
header. This unblocks the 1P CLI guardrail panels on the [Dev Platform]
1P Orgs Migration dashboard, which currently blend 1P + 3P traffic.

Closes: shop/issues-develop#22933
@SimonSiju SimonSiju requested review from a team as code owners June 8, 2026 19:20
@github-actions github-actions Bot added the Area: @shopify/cli @shopify/cli package issues label Jun 8, 2026
@github-actions github-actions Bot added no-changelog This PR doesn't include a changeset entry. Is an internal only change not relevant to end users. and removed Area: @shopify/cli @shopify/cli package issues labels Jun 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-changelog This PR doesn't include a changeset entry. Is an internal only change not relevant to end users.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant