Skip to content

chore(tests): add tests#12

Merged
nimarb merged 7 commits into
mainfrom
nimar/add-tests
Jul 17, 2026
Merged

chore(tests): add tests#12
nimarb merged 7 commits into
mainfrom
nimar/add-tests

Conversation

@nimarb

@nimarb nimarb commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

@nimarb
nimarb requested a review from a team as a code owner July 16, 2026 16:37

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e19a96ff1f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/ci.yml Outdated
Comment thread package.json
@nimarb

nimarb commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator Author

@claude review

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds a large new conformance-testing subsystem (OpenAPI-driven CLI conformance harness, ~19 source/spec files). A real bug was found in flattenBody() that causes required-but-empty nested body fields to be silently dropped, producing false conformance failures — this and the related cycle-detection nit are covered in the inline comments. Given the size and correctness-sensitivity of this new tooling, human review is warranted.

Extended reasoning...

This PR introduces a full conformance-testing subsystem for the langfuse CLI: a policy/catalog config, 8 versioned OpenAPI spec snapshots, and ~12 new TypeScript modules (schema sampling, vector generation, request capture, adapters, runner, serialization) plus their test suites. It replaces the earlier incomplete version of this PR (which was missing conformance/src/cli.ts entirely, breaking CI — that issue is now resolved, as conformance/src/cli.ts and the rest of conformance/src are present in this revision).\n\n### Security risks\nNone of significance. This is internal developer/CI tooling that generates synthetic API requests against sample OpenAPI specs and diffs CLI output — it does not touch production auth, data storage, or user-facing surfaces. The auth/secret values in conformance/policy.json are fixture placeholders ("conformance-public-key"), not real credentials.\n\n### Level of scrutiny\nThis warrants a moderate-to-high level of scrutiny despite being test/dev tooling, because its entire purpose is to produce a trustworthy pass/fail signal about CLI correctness — a bug in the harness itself (as found) silently produces false failures that look like real CLI defects, which could mislead future contributors or block merges for the wrong reason. It is not production-critical code, but correctness bugs in it have real downstream cost.\n\n### Other factors\nA bug hunting pass found a concrete, reproducible defect in flattenBody() (conformance/src/adapters.ts): it silently omits any required body field whose sampled value is an empty nested object, causing spurious conformance failures for operations like scim_createUser. A second, lower-severity nit was found in the cycle-detection trail in sampleFromSchema (schema.ts), which is currently latent (no shipped spec triggers it) but would break on a future recursive schema pulled in via conformance:sync. Both are already surfaced as inline comments. Given the size of this diff and that a real, reproducible bug was found in core harness logic, this should get a human look rather than an automatic approval.

Comment thread conformance/src/adapters.ts
Comment thread conformance/src/schema.ts
@nimarb
nimarb merged commit 65f0b53 into main Jul 17, 2026
4 checks passed
@nimarb
nimarb deleted the nimar/add-tests branch July 17, 2026 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant