Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/demos/circuit-breaker.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ and probing for recovery.
<script>
document.addEventListener('DOMContentLoaded', function () {
HttpwareDemo.mount('#cb-demo', {
ring: 'reset',
scenarios: [
{ id: 'down', label: 'Backend goes down', dur: 12.5,
fault: (now) => (now >= 2.0 && now < 8.0)
Expand Down
32 changes: 32 additions & 0 deletions docs/demos/demos.css
Original file line number Diff line number Diff line change
Expand Up @@ -132,3 +132,35 @@
@media (prefers-reduced-motion: reduce) {
.hw-demo .playhead, .hw-demo .ring, .hw-demo .box, .hw-demo .lane, .hw-demo .dot { transition: none; }
}

/* thundering-herd rate strips */
.hw-demo .herd-wrap { margin-top: 26px; padding-top: 22px; border-top: 1px solid var(--hw-line); }
.hw-demo .herd-lane { border: 2px solid var(--hw-line); border-radius: 12px; padding: 14px 16px; margin-bottom: 14px; }
.hw-demo .herd-lane h3 { margin: 0 0 10px; font-size: .95rem; display: flex; align-items: center; gap: 8px; }
.hw-demo .herd-strip { height: 96px; }
.hw-demo .herd-svg { width: 100%; height: 96px; display: block; }
.hw-demo .herd-band { fill: rgba(229, 72, 77, .12); }
.hw-demo .herd-baseline { stroke: var(--hw-muted); stroke-width: .6; stroke-dasharray: 2 2; opacity: .7; }
.hw-demo .herd-bar-naive { fill: var(--hw-bad); }
.hw-demo .herd-bar-hw { fill: var(--hw-ok); }

/* countdown ring overlaid on a lane element */
.hw-demo .counting { position: relative; }
.hw-demo .counting::after {
content: ""; position: absolute; right: -18px; top: 50%; width: 13px; height: 13px;
margin-top: -6px; border-radius: 50%;
background: conic-gradient(var(--hw-warn) var(--hw-ring, 0deg), var(--hw-line) 0);
}

/* jagged failure dot: readable without color (star/spikes vs the round success dot) */
.hw-demo .dot.bad {
border-radius: 0;
clip-path: polygon(50% 0, 61% 35%, 98% 35%, 68% 57%, 79% 91%, 50% 70%, 21% 91%, 32% 57%, 2% 35%, 39% 35%);
width: 11px; height: 11px; top: 4px;
}

/* full-stack macro strip (reuses .herd-strip / .herd-bar-hw) */
.hw-demo .macro { margin-top: 6px; }
.hw-demo .macro-head { font-family: var(--hw-mono); font-size: .72rem; color: var(--hw-muted); margin-bottom: 4px; }
.hw-demo .macro .herd-strip { height: 54px; }
.hw-demo .macro .herd-svg { height: 54px; }
411 changes: 345 additions & 66 deletions docs/demos/engine.js

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions docs/demos/full-stack.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,11 @@ document.addEventListener('DOMContentLoaded', function () {
retry: { maxAttempts: 3, baseDelay: 0.1, maxDelay: 5.0 },
budget: { ttl: 10.0, minRetriesPerSec: 10.0, percentCanRetry: 0.2 } } },
],
macroStrip: true,
stageLabel: (now) => now < 2 ? 'healthy'
: now < 5 ? 'phase 1 — latency spike'
: now < 8 ? 'phase 2 — brownout'
: now < 12 ? 'phase 3 — hard down' : 'recovered',
buildStops: () => [
{ when: (s) => s.now >= 2.4, spot: ['poolB', 'elapsedB'], title: 'Phase 1 — latency spike',
body: 'Bulkhead caps concurrency so the slow phase can’t exhaust the client; timeout bounds each operation at 2s.' },
Expand Down
30 changes: 30 additions & 0 deletions docs/demos/retry.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ a dead backend can't be amplified.

<div class="hw-demo" id="retry-demo"></div>

<p>That was <b>one</b> client. The real danger of blind retries shows up at scale — when <b>many</b> clients retry the same outage at once.</p>

<div class="hw-demo" id="retry-herd"></div>

<script>
document.addEventListener('DOMContentLoaded', function () {
HttpwareDemo.mount('#retry-demo', {
Expand Down Expand Up @@ -40,5 +44,31 @@ document.addEventListener('DOMContentLoaded', function () {
body: 'The backend healed. Compare the ✗ failed counts: the plain client surfaced far more failures than httpware. That gap is exactly what retry buys you on a transient blip.' },
],
});

HttpwareDemo.mountHerd('#retry-herd', {
clients: 20,
intro: 'A real backend rarely dies cleanly — it <b>flaps</b>: fails, recovers, fails again. These strips show <b>backend call-rate over time</b> for twenty clients through three dips. Press play and watch the shape.',
scenario: { id: 'storm', dur: 12.5,
fault: (now) => {
const down = (now >= 2.0 && now < 4.0) || (now >= 5.5 && now < 7.5) || (now >= 9.0 && now < 11.0);
return down ? { ok: false, ms: 0.05, label: 'DOWN' } : { ok: true, ms: 0.05 };
} },
retry: { maxAttempts: 3, baseDelay: 0.1, maxDelay: 5.0 },
budget: { ttl: 10.0, minRetriesPerSec: 10.0, percentCanRetry: 0.2 },
buildStops: (sim) => [
{ when: (s) => s.revealed >= Math.round(2.2 / sim.dt), spot: ['naiveStrip', 'hwStrip'],
title: 'A flapping backend',
body: 'The backend drops, recovers, drops again — three dips (shaded). Every failed request wants to retry. Watch what each herd does to the backend call-rate through the dips.' },
{ when: (s) => s.revealed >= Math.round(4.6 / sim.dt), spot: ['naiveStrip'],
title: 'Naive: a surge on every dip',
body: 'On each dip, twenty clients retry unbounded — the load piles up into a surge that keeps climbing until the backend recovers, then clears. Three dips, three spikes, with recovery gaps between: the retry storm hitting a backend every time it tries to come back.' },
{ when: (s) => s.revealed >= Math.round(8.2 / sim.dt), spot: ['hwStrip'],
title: 'httpware: flat through every dip',
body: 'Full jitter spreads each client’s retries out, and each client’s own max_attempts=3 cap (with its per-client budget as the guarantee at higher volume) limits how much it can add — so httpware holds a low, steady few-times-baseline through every dip instead of spiking.' },
{ when: (s) => s.revealed >= sim.buckets - 1, spot: ['naiveMult', 'hwMult'],
title: 'Peak load: the whole point',
body: 'On its worst dip the naive herd spiked to about 18× the healthy load; httpware never exceeded about 3×, capped by each client’s max_attempts. That flat rate is exactly what lets the backend recover in the gaps — instead of being knocked back down by a retry surge every time it heals.' },
],
});
});
</script>
1 change: 1 addition & 0 deletions docs/demos/timeout.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ including every retry and every backoff sleep, so one call can't blow your laten
<script>
document.addEventListener('DOMContentLoaded', function () {
HttpwareDemo.mount('#to-demo', {
ring: 'deadline',
scenarios: [
{ id: 'brownout', label: 'Slow brownout under retry', dur: 12.5,
fault: (now, rnd) => (now >= 2.0 && now < 9.0)
Expand Down
136 changes: 136 additions & 0 deletions planning/changes/2026-07-19.02-resilience-demo-herd-and-enrich.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
---
summary: Added a many-client thundering-herd macro view to the retry page (naive ~18x vs httpware ~3x peak backend load through a flapping outage, httpware bounded by max_attempts at demo scale with a per-client budget) and folded three enrich techniques across the demo suite — a backoff countdown ring on timeout + circuit-breaker only (retry's sub-tick backoff makes a faithful ring there impossible, so it's dropped, not faked), a color-independent jagged failure shape everywhere, and a live backend call-rate macro strip with active-phase label on the full-stack page.
---

# Design: Resilience demo herd view + enrich pass

## Summary

Improve the resilience demo suite (shipped in `2026-07-18.02`) using techniques
proven by the best existing visualizations. Two parts: (1) a **thundering-herd
macro view** stacked below the single-client retry/budget demo — 20 naive clients
vs 20 httpware clients, rate-into-the-backend over a flapping outage, so the
retry storm is *shown* not just described; (2) an **enrich pass** folding three
smaller techniques into the existing pages — a backoff/wait **countdown ring**, a
color-independent **jagged failure shape**, and a small **macro strip** on the
full-stack page. Still a faithful model, still pure static assets, still no
sandbox sliders. `architecture/` unchanged — docs only.

## Motivation

Surveying the strongest existing work — [Encore's interactive retries
study](https://encore.dev/blog/retries) and the [AWS exponential-backoff-and-jitter
post](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/) —
exposes one real gap and a few cheap wins:

- **The thundering herd is invisible.** Our retry/budget demo is single-client, so
the one visual that makes "retry storm" and "why jitter" click — a flapping backend
driving unbounded retries into spikes-with-gaps, jitter smoothing them into a flat
rate — we only assert in prose. AWS's canonical framing is a call-rate-over-time
chart: without jitter, "clusters of calls" with dead gaps; with jitter, an
approximately constant rate. We don't draw it.
- **The backoff wait isn't shown.** A dot just eventually resolves; Encore renders a
countdown timer *while a client waits between attempts*, making the delay tangible.
- **Failures lean on color.** A failed request is a flat square distinguished mainly
by red; Encore's "spiked edges" read without color.

## Design

**Herd view — stacked section on the retry/budget page.** Below the current
single-client side-by-side, a distinct "Now scale it to 20 clients" section with its
own play button and mini-tour, preserving a micro → macro arc down the page (one
client's retries, then the herd). A new engine **rate-strip render mode** (isolated
from the existing lane renderer, selected by mount config) draws two stacked strips
plotting calls-into-the-backend per tick over a **flapping** outage — the backend
goes down, recovers, goes down again, three dips with real recovery gaps between:

- **Naive lane:** 20 clients, fixed backoff, no jitter, unbounded retries → on each
dip a client retries until the backend recovers, so retries accumulate into a
surge that clears the instant the dip ends — a spike per dip with a genuine
recovery gap between (AWS's finding; the gaps come from the backend healing, not
from client synchronization — arrivals are phase-staggered, never lockstep).
- **httpware lane:** 20 clients, full-jitter backoff + per-client budget → jitter
decorrelates retry timing into a near-constant rate through every dip; at this
client count each client's `max_attempts=3` cap is what bounds its amplification
(the per-client budget's floor doesn't bind until far higher per-client volume —
it's the guarantee that holds as the herd grows).

Both lanes reuse the *real* retry + budget models from the engine's source-of-truth
constants block, one model instance per simulated client. **Fidelity framing that
must hold:** the budget is **per-client**, not shared across the herd — at demo
traffic levels the *visible* flattening is jitter decorrelating timing, and
httpware's peak is bounded near its `max_attempts` cap (~3×), with the per-client
budget the ceiling that holds as volume grows. Copy says exactly that; it must not
imply a shared/global budget or that the budget (rather than `max_attempts`) does the
capping at this scale.

**Payoff metric.** The mini-tour spotlights the **peak load multiplier** — "backend
saw N× baseline". Measured against the fixed seed on the flapping scenario: **naive
~18×** on its worst dip (unbounded retries surge each time the backend drops) **vs
httpware ~3×** (bounded by `max_attempts` at this client count) — the number reads
directly off the tallest spike the strip already draws (spotlight number == visible
fact). **Total calls** during the outage is a secondary, un-spotlighted counter
(volume story). "The flat rate leaves the backend room to recover" is the qualitative
closing beat, grounded in the scenario itself (the backend genuinely recovers between
dips) rather than an invented recovery number. ~4 beats: the flapping backend →
naive's first dip-spike → httpware stays flat (jitter + the `max_attempts` cap) →
peak-multiplier payoff.

**Enrich pass.**

| Technique | Pages | What it shows |
|---|---|---|
| Countdown ring | circuit-breaker, timeout | CB: `reset_timeout` until OPEN→HALF_OPEN probe · timeout: the deadline ticking down |
| Jagged failure shape | all five (CSS only) | failure readable without color |
| Macro strip | full-stack only | small rate + which-stage-active sparkline |

The countdown ring is a CSS-conic overlay on a fixed lane-B element, showing
elapsed-wait as a fraction; each page's mount config names which wait it visualizes.
Skipped on bulkhead (the `acquire_timeout` wait either grants a slot instantly or
fast-rejects — no visible dwell), on full-stack (a ring per stage is noise), and — a
fidelity call taken during planning — on **retry**: at the demo's time compression a
full-jitter `base_delay=0.1s` backoff rounds to 0–1 engine ticks (sub-tick), so a
faithful ring there would never render and inflating it would misstate the modeled
wait. Retry teaches backoff timing through the herd view's fixed-vs-jittered contrast
instead. The full-stack macro strip reuses the herd's rate-series renderer at small
size. Jagged failure shape is one `demos.css` change.

## Non-goals

- No shared/global budget across the herd — would misrepresent httpware; per-client only.
- No backend-recovery number — not modeled; recovery stays qualitative.
- No sandbox sliders — the herd is seeded and guided, like every other scenario.
- No per-page bespoke sparklines (CB failure-count, bulkhead slots, timeout latency)
— those signals are already live as counters/flow-box state; a macro strip goes
only where *rate* is the story (full-stack; herd covers retry).
- No JS test runner, no new Python dependency — unchanged from `2026-07-18.02`.
- `architecture/` untouched — docs only; no capability contract moves.

## Testing

`just docs-build --strict` clean (pages resolve, nav valid, no broken links).
`node --check docs/demos/engine.js`. Scratchpad jsdom harnesses extended: the
existing `verify-demos.js` / `verify-real-page.js` drive every stop including the new
herd mini-tour to completion without stalling; a herd trace asserts naive peak
multiplier ≫ httpware peak (dramatic gap, ~18× vs ~3×) with the spikes-with-gaps
shape itself (multiple separated naive spikes, one per flapping dip, with
near-baseline recovery gaps between), httpware bounded near the `max_attempts` cap,
and a fine bucket count for a smooth strip. Manual: retry page
reads micro → macro top-to-bottom under mkdocs-material light and dark; countdown
ring animates and empties on CB / timeout; failure shape distinguishable in
grayscale; `prefers-reduced-motion` degrades to stepped updates.

## Risk

- **Herd fidelity overclaim (medium × medium).** Easy to imply lockstep
synchronization or a shared budget. Mitigation: spikes-with-gaps come from the
flapping backend's dip/recovery cycle (not lockstep), per-client budget instances,
and copy reviewed specifically against the fidelity framing above; reuse the real
models rather than a bespoke herd approximation.
- **Rate-strip renderer as new surface (low × medium).** A second render mode adds
engine branching and bug surface. Mitigation: isolate it from the lane renderer
behind mount config; the full-stack macro strip reuses it, so one renderer serves
both.
- **Retry page grows long/busy (low × low).** Two stacked demos plus a tour.
Mitigation: the herd section is visually distinct with its own play control; the
single-client demo is unchanged above it.