chore(tests): add tests#12
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e19a96ff1f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@claude review |
There was a problem hiding this comment.
This PR adds a large new conformance-testing subsystem (OpenAPI-driven CLI conformance harness, ~19 source/spec files). A real bug was found in flattenBody() that causes required-but-empty nested body fields to be silently dropped, producing false conformance failures — this and the related cycle-detection nit are covered in the inline comments. Given the size and correctness-sensitivity of this new tooling, human review is warranted.
Extended reasoning...
This PR introduces a full conformance-testing subsystem for the langfuse CLI: a policy/catalog config, 8 versioned OpenAPI spec snapshots, and ~12 new TypeScript modules (schema sampling, vector generation, request capture, adapters, runner, serialization) plus their test suites. It replaces the earlier incomplete version of this PR (which was missing conformance/src/cli.ts entirely, breaking CI — that issue is now resolved, as conformance/src/cli.ts and the rest of conformance/src are present in this revision).\n\n### Security risks\nNone of significance. This is internal developer/CI tooling that generates synthetic API requests against sample OpenAPI specs and diffs CLI output — it does not touch production auth, data storage, or user-facing surfaces. The auth/secret values in conformance/policy.json are fixture placeholders ("conformance-public-key"), not real credentials.\n\n### Level of scrutiny\nThis warrants a moderate-to-high level of scrutiny despite being test/dev tooling, because its entire purpose is to produce a trustworthy pass/fail signal about CLI correctness — a bug in the harness itself (as found) silently produces false failures that look like real CLI defects, which could mislead future contributors or block merges for the wrong reason. It is not production-critical code, but correctness bugs in it have real downstream cost.\n\n### Other factors\nA bug hunting pass found a concrete, reproducible defect in flattenBody() (conformance/src/adapters.ts): it silently omits any required body field whose sampled value is an empty nested object, causing spurious conformance failures for operations like scim_createUser. A second, lower-severity nit was found in the cycle-detection trail in sampleFromSchema (schema.ts), which is currently latent (no shipped spec triggers it) but would break on a future recursive schema pulled in via conformance:sync. Both are already surfaced as inline comments. Given the size of this diff and that a real, reproducible bug was found in core harness logic, this should get a human look rather than an automatic approval.
No description provided.