Skip to content

Bulk-scan single-line string bodies#491

Merged
frostming merged 1 commit into
python-poetry:masterfrom
AstekGroup:perf/string-scan
Jun 10, 2026
Merged

Bulk-scan single-line string bodies#491
frostming merged 1 commit into
python-poetry:masterfrom
AstekGroup:perf/string-scan

Conversation

@tfoutrein

Copy link
Copy Markdown
Contributor

Stacked on #489 and #490 — the first two commits here are those PRs (index-based Source + bulk run scanning). Best reviewed/merged after them; GitHub will collapse this to just the string-scan commit once they land. Happy to rebase.

What

Parsing a single-line string appended its body one character at a time (value += current; inc()). For long string values (the bulk of most config / lockfile content) this dominates.

This scans the run of ordinary characters up to the next delimiter, backslash or control character in a single pass (Source.advance_until) and appends the whole slice at once; the stop character is then handled by the existing branch on the next iteration. Multiline strings keep the per-character loop (CRLF handling).

The stop-set is exactly the control characters the per-character loop rejects, so InvalidControlChar / escape / delimiter handling is unchanged, and a mid-string EOF raises UnexpectedEofError just as the per-char inc(exception=...) did.

Benchmarks

Parsing speedup across document shapes (median, interleaved A/B vs master, includes #489+#490):

document speedup
large flat, single-line string values (~90 KB) 4.9×
pyproject.toml 2.1×
poetry.lock-like (~64 KB) 2.1×
typical mixed (~4 KB) 1.5×

No regression on any shape (multiline-heavy and nested docs are unchanged).

Tests

Full suite passes (972 tests, incl. the toml-test conformance submodule). On top of that, a 4135-input adversarial differential (random escapes valid+invalid, every control byte 0x00–0x1F+DEL in basic & literal, unicode/astral/combining, the other-quote char inside strings, truncated/malformed inputs for error parity) is byte-identical in output and exception type to the per-character loop. No public API or behaviour change.

@frostming

Copy link
Copy Markdown
Contributor

@tfoutrein please resolve the conflicts

Parsing a single-line string appended its body one character at a time
(`value += current; inc()`). For long string values this dominates.

Scan the run of ordinary characters up to the next delimiter, backslash
or control character in a single pass (`Source.advance_until`) and append
the whole slice at once; the stop character is then handled by the
existing branch on the next iteration. Multiline strings keep the
per-character loop (CRLF handling). The stop-set is exactly the control
characters the per-character loop rejects, so InvalidControlChar / escape
/ delimiter handling is unchanged.

No behaviour change (972 tests incl. the toml-test conformance submodule;
plus a 4135-input adversarial differential — output and error-type
byte-identical to the per-char loop). Up to ~5x faster parsing on
string-heavy single-line documents.
@tfoutrein

Copy link
Copy Markdown
Contributor Author

Rebased onto master now that #490 has merged — this is a single commit on top of current master, no conflicts. CI is green. Ready for squash-merge whenever you'd like. Thanks!

@frostming frostming merged commit 4623b45 into python-poetry:master Jun 10, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants