diff --git a/ROADMAP.md b/ROADMAP.md index 889bc2b..f2ec057 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -44,7 +44,8 @@ Strawberry, Quart, RQ, APScheduler, Jobify, Flet, ag2. - **Dependency-graph export** (Mermaid / Graphviz) for debugging and docs. ### Trust & observability -- **Public, reproducible benchmark suite** with neutral methodology. +- **Public, reproducible benchmark suite** with neutral methodology — + shipped; see [Performance](https://modern-di.modern-python.org/introduction/performance/). - **Optional OpenTelemetry instrumentation** of resolution and finalization. ### Docs & ecosystem diff --git a/docs/introduction/comparison.md b/docs/introduction/comparison.md index 8b0cf8d..6346e4f 100644 --- a/docs/introduction/comparison.md +++ b/docs/introduction/comparison.md @@ -145,3 +145,5 @@ lifetime dialect, so here is how the same six concepts translate: - [Design decisions](design-decisions.md) — the reasoning behind sync-only resolution, no global state, a conservative core, and the deliberate [non-goals](design-decisions.md#non-goals) that keep it that way. +- [Performance](performance.md) — comparative benchmarks: how fast resolution is + versus other DI frameworks, and the method behind the numbers. diff --git a/docs/introduction/performance.md b/docs/introduction/performance.md new file mode 100644 index 0000000..dcad95b --- /dev/null +++ b/docs/introduction/performance.md @@ -0,0 +1,95 @@ +# Performance + +This page compares modern-di's resolution performance against four other Python +DI frameworks, states the method, and gives a command to reproduce the numbers. +modern-di has no runtime dependencies and generates no code. The comparison set +includes two frameworks that use `exec` codegen (dishka, wireup), one with a +Cython-compiled core (dependency-injector), and one pure-Python framework +(that-depends). + +> Absolute timings depend on the machine and CPython build and will differ on +> yours. The ratios between frameworks are more portable across machines, so the +> table below is expressed as ratios. + +## What is measured + +Four scenarios, each the smallest graph that isolates one cost, run with +[`pytest-benchmark`](https://pytest-benchmark.readthedocs.io/) in an isolated +environment with pinned rival versions: + +| ID | Scenario | Isolates | +|----|----------|----------| +| C1 | Transient resolve, single dependency | pure wiring cost | +| C2 | Singleton resolve, warm cache | cache-hit lookup | +| C3 | Deep chain, depth 6 | per-edge wiring | +| C4 | Request lifecycle: enter scope → resolve → async-finalize on exit | whole per-request cost | + +Each framework uses its own idiomatic request-scope and resource-teardown +spelling, not modern-di's names forced onto it. Full per-framework mapping and +rules: [`benchmarks/README.md`](https://github.com/modern-python/modern-di/blob/main/benchmarks/README.md). + +## Results + +Measured 2026-07-17 on an Apple M2 (macOS), CPython 3.14.4, median-of-medians +over 5 runs (run-to-run variation small). Rival versions: dishka 1.10.1, +dependency-injector 4.49.1, that-depends 4.0.2, wireup 2.12.0. Reproduce with +`just bench-compare`. + +Each cell is modern-di ÷ rival: below 1.0 (bold) means modern-di is faster, +above 1.0 means slower. + +| Scenario | modern-di | vs dishka | vs dependency-injector | vs that-depends | vs wireup | +|----------|-----------|-----------|------------------------|-----------------|-----------| +| C1 transient | 541 ns | 1.30 | **0.81** | 1.08 | 1.77 | +| C2 warm singleton | 282 ns | 1.17 | 4.58 | 3.37 | 2.95 | +| C3 deep chain (6) | 1250 ns | 1.87 | **0.57** | **0.81** | 1.32 | +| C4 request lifecycle | 31.0 µs | 1.02 | **0.18** | **0.75** | **0.69** | + +## What the numbers show + +- Against `dependency-injector`, modern-di is faster on C1, C3, and C4, and + slower on C2. dependency-injector's warm-singleton hit (C2) is a C-level slot + read; modern-di's is a Python dict lookup behind an override guard. +- Against `dishka` and `wireup` on the construction-heavy scenarios (C1 + transient, C3 deep chain), modern-di is roughly 1.3–1.9x slower. Both inline + dependency calls into `exec`-generated source, which removes the per-node + function-call frame that modern-di keeps. modern-di does not generate code (a + [documented non-goal](design-decisions.md#non-goals)), so this difference is + expected rather than a regression. +- On C4 (request lifecycle) modern-di is within run-to-run noise of `dishka` + (1.02) and faster than the other three. See the caveat below before reading + the C4 column as a resolve-speed comparison. + +**C4 is not a like-for-like resolve.** modern-di resolves the connection +synchronously while finalizing it asynchronously; the other four force an awaited +resolve once the finalizer is async. C4 therefore measures the whole request +lifecycle (enter scope → resolve → async-finalize) under a shared event loop, not +an isolated resolve. C1–C3 are synchronous resolves for every framework. + +## Why the results look this way + +Since 2.29.0, modern-di compiles one specialized closure per provider on first +resolve, memoized on the providers registry, replacing a generic per-call +interpreted resolver. Each compiled resolver hoists its scope navigation, +override check, and cache lookup out of the per-call path and calls its +dependencies' resolvers directly. The remaining gap to the codegen frameworks on +C1 and C3 is the per-node call frame that `exec`-inlined source removes and +modern-di keeps. + +## Reproduce it yourself + +```bash +git clone https://github.com/modern-python/modern-di +cd modern-di +just bench-compare # isolated env; first run resolves the pinned rival deps +``` + +The comparative environment is isolated and its result files are not committed, +so absolute numbers will differ from those above. The ratios are more comparable +across machines than the absolute times. + +## See also + +- [Comparison](comparison.md) — how modern-di compares on features. +- [Design decisions](design-decisions.md) — why resolution is sync-only and why + `exec` codegen is a non-goal. diff --git a/mkdocs.yml b/mkdocs.yml index cfa1500..3594b95 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -10,6 +10,7 @@ nav: - Resolving: introduction/resolving.md - Design decisions: introduction/design-decisions.md - Comparison: introduction/comparison.md + - Performance: introduction/performance.md - modern-di for FastAPI users: introduction/for-fastapi-users.md - Providers: - Scopes: providers/scopes.md @@ -159,6 +160,7 @@ plugins: - introduction/resolving.md: Resolving dependencies by type or by provider reference - introduction/design-decisions.md: The deliberate API choices behind modern-di - introduction/comparison.md: How modern-di compares to other DI approaches, including that-depends + - introduction/performance.md: Comparative benchmarks vs other DI frameworks, and how to reproduce them - introduction/for-fastapi-users.md: Translating FastAPI's Depends idioms to modern-di Providers: - providers/scopes.md: The five built-in scopes and the resolution rule diff --git a/planning/changes/2026-07-17.06-benchmarks-page.md b/planning/changes/2026-07-17.06-benchmarks-page.md new file mode 100644 index 0000000..766bc23 --- /dev/null +++ b/planning/changes/2026-07-17.06-benchmarks-page.md @@ -0,0 +1,92 @@ +--- +summary: Publish a public "Performance" page under Introduction (after Comparison) that evidences the fast + zero-dependency story with ratios-first comparative numbers, a stated method, honest wins/losses, and a reproduce command — closing deferred B3. +--- + +# Design: Public performance page + +## Summary + +Add `docs/introduction/performance.md`, nav'd directly after Comparison, that +turns the existing (private, dev-only) comparative benchmark suite into public +evidence for modern-di's "fast **and** zero-dependency" claim. It leads with the +durable facts (cross-framework **ratios**), shows absolute ns once under an +explicit machine/version header, and is honest about where modern-di loses. No +code changes; docs and planning bookkeeping only. + +## Motivation + +The `benchmarks/` suite (guard + comparative tiers) and its methodology exist, +but there is **no public page** — the story is currently unevidenced in public +(deferred B3). The 2.29.0 compiled-closure resolver just produced fresh numbers +placing modern-di mid-pack and the **only zero-dependency pure-Python framework +holding its own**: it beats the Cython `dependency-injector` on 3 of 4 scenarios +and ties `dishka` on the async lifecycle. The ROADMAP already lists "Public, +reproducible benchmark suite with neutral methodology" under *Trust & +observability*; this delivers it. The research that produced these numbers +explicitly caught rivals shipping unverified vendor-self-reported claims — so the +bar is a page that any reader can reproduce, not a marketing table. + +## Design + +New page `docs/introduction/performance.md`, six short sections: + +1. **The claim** — only zero-dep pure-Python framework holding mid-pack; speed + was never the pitch (one typed wiring is) but it is not a tax either. +2. **Method** — the four C1-C4 scenarios (one line each), `pytest-benchmark`, + isolated pinned env, median-of-medians over 5 runs, the machine + CPython + version, pinned rival versions (dishka 1.10.1, dependency-injector 4.49.1, + that-depends 4.0.2, wireup 2.12.0). Links `benchmarks/README.md` and the + `just bench-compare` command. +3. **Results** — **ratios-first** table (modern-di ÷ each rival per scenario; + `>1` = modern-di slower), bold where modern-di wins. Absolute ns shown once + under a dated machine/version header, framed "your numbers will differ; the + ratios are the durable fact." +4. **Honest reading** — wins named, and losses named: 1.3-1.9x behind the + `exec`-codegen frameworks on transient/deep-chain, the accepted cost of + staying `exec`-free. States plainly that **C4 is not sync-vs-sync** — modern-di + resolves synchronously under an async finalizer while the rivals force an + awaited resolve, so C4 measures the whole request lifecycle, not an isolated + resolve. +5. **Why it's fast** — the 2.29.0 compiled-closure resolver, specialized once per + provider; zero-dep stance rules out `exec` codegen, which *is* the + construction-heavy ceiling. Links to Design-decisions / non-goals. +6. **Reproduce** — exact commands; restate that ratios travel across machines, + absolutes do not. + +Wiring: `mkdocs.yml` nav gains the page after Comparison, and the `llmstxt` +Introduction section list gains a one-line description. Comparison's "is it +fast?" gap gets a pointer to Performance; Performance links back. + +**Numbers.** At implementation I run `just bench-compare` fresh, capturing +CPython + rival versions + date, and compute ratios from the medians. If the +isolated env (Cython `dependency-injector` build, etc.) will not build cleanly +here, I **pause and check with the maintainer** rather than publish the older +snapshot unverified (maintainer ruling, 2026-07-17). No number ships without its +reproduce path. + +## Non-goals + +- **Guard tier on the public page** — G1-G7 stays an internal dev artifact in + `benchmarks/README.md`; the public page is comparative-only. +- **CI-gated numbers** — the comparative env is local-only and machine-relative; + the page is a dated, pinned-version snapshot, not a CI check. +- **Committing result files** — unchanged; artifacts stay git-ignored. +- **Code / benchmark-suite changes** — the suite already exists. + +## Testing + +- `just docs-build` — `mkdocs build --strict` clean (nav, links, anchors); the + new page and both cross-links resolve. +- `just lint-ci` — clean, including planning-bundle validation. +- Every published number reproduced by an actual `just bench-compare` run on the + documented machine (or the pause-and-ask path above). + +## Risk + +- **Numbers go stale as rivals release** (likely × low). Mitigation: dated + header + pinned rival versions shown inline; ratios framed as the durable fact. +- **Reads as marketing / cherry-picked** (medium × high to credibility). + Mitigation: losses stated as prominently as wins, method + reproduce command up + front, C4 caveat explicit. +- **Env won't build on this machine** (medium × low). Mitigation: pause-and-ask; + do not publish unverified numbers. diff --git a/planning/deferred.md b/planning/deferred.md index c43fbc7..9f55d8b 100644 --- a/planning/deferred.md +++ b/planning/deferred.md @@ -78,21 +78,6 @@ on the hot path plus 5-8 log statements through resolution code. **Revisit trigger:** the first user issue that a resolution trace would have answered. See [2026-07-05 3.0 UX research, ERR-8](audits/2026-07-05-v3-ux-research-report.md). -## Eager warm-up of cached providers (API-8 / INT-3) — from 2026-07-05 3.0 UX research - -A `container.init_cache()`-style method (name open: warm_up/prebuild/init_cached — one decision for -both research items) resolving every provider with cache settings at the container's scope, in -declaration order, respecting overrides; finalizers still run LIFO at close. Closes the gap that -`validate()` checks wiring but never calls creators, so a bad DB URL in a `Factory(cache=True)` -creator surfaces on the first request instead of at boot. Field precedent: Spring eager singletons, -Koin `createdAtStart`, that-depends/dependency-injector `init_resources()`. Purely additive, -sync-only by construction. Note: the from-that-depends guide currently editorializes "no equivalent -needed" — shipping this reverses that documented stance. - -**Revisit trigger:** the first user/integration request for startup fail-fast (or when a sibling -integration wants a warm-up hook next to `open()`/`validate()` in its lifespan setup). -See [2026-07-05 3.0 UX research, items 13-14](audits/2026-07-05-v3-ux-research-report.md). - ## Shared conformance test suite for integration repos (INT-6) — from 2026-07-05 3.0 UX research A reusable pytest contract suite, parametrized over each integration's app factory + setup function, @@ -153,20 +138,6 @@ Drafted as §6 of the org launch playbook; never executed. **Revisit trigger:** the launch window. See [2026-06-18 adoption research, §5](audits/2026-06-18-adoption-strategy-report.md). -## Publish the benchmarks page — from 2026-06-18 adoption research (B3) - -The `benchmarks/` suite (guard + comparative tiers) and its `benchmarks/README.md` methodology now -exist, but there is **no public benchmarks page in the mkdocs nav**. The "fast + zero-dependency" -story is currently unevidenced in public. - -**Verify every number before publishing.** The research caught rivals shipping unverified -vendor-self-reported perf claims; modern-di must not join them. Anything published needs a -reproducible method stated alongside it. - -**Revisit trigger:** when the benchmark suite is stable enough that a published number won't need -retracting. -See [2026-06-18 adoption research](audits/2026-06-18-adoption-strategy-report.md). - ## Size the DI market with real download data — from 2026-06-18 adoption research (evidence gap) The research verified **no PyPI download figures for any framework** — its own single biggest gap, and