Skip to content

[TASK] Optimize RST parsing with pattern and instance caching - #1288

Open
CybotTM wants to merge 3 commits into
phpDocumentor:mainfrom
CybotTM:perf/rst-parsing-optimizations
Open

[TASK] Optimize RST parsing with pattern and instance caching#1288
CybotTM wants to merge 3 commits into
phpDocumentor:mainfrom
CybotTM:perf/rst-parsing-optimizations

Conversation

@CybotTM

@CybotTM CybotTM commented Jan 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Optimizes RST parsing with instance reuse and O(1) hash set lookups for hyperlink validation.

Changes

  • ExternalReferenceResolver: Add SUPPORTED_SCHEMAS_LIST and isSupportedScheme() for O(1) hash set lookup
  • InlineParser: Reuse InlineLexer instance instead of creating new one per parse
  • InlineLexer: Use ExternalReferenceResolver::isSupportedScheme() for URI scheme validation (~6x faster)
  • LineChecker: Cache compiled regex patterns
  • Buffer: Cache unindent calculations

Performance Impact

See Performance Analysis Report for detailed benchmarks.

The hash set optimization for URI schemes provides approximately 6x speedup compared to the previous regex-based approach for the 371 IANA-registered schemes.

Merge Note

Both this PR and #1287 add the same isSupportedScheme() method to ExternalReferenceResolver. When the second PR merges, the conflict is trivially resolved by keeping the existing code.


Related PRs

PR Description Status
#1287 Rendering caching layer Open, reworked after review (trivial merge conflict on ExternalReferenceResolver)
#1288 This PR - RST parsing optimizations
#1289 CLI container caching Closed — the cache was never written
#1291 Symfony 8 compatibility ✅ Merged
#1293 ProjectNode O(1) document lookup Independent

All PRs can be merged independently in any order.

Update: three corrections after review

The three caches in LineChecker are static and keyed by the line itself, with nothing clearing or bounding them. DirectiveRule::applies() asks about every line of every document, so they retained a copy of each distinct line for the lifetime of the process — across documents, and across projects in a long running renderer. They are bounded now: a full table is dropped, which keeps the working set cached, since documents are parsed line by line, and puts a ceiling on what is held. Behaviour is unchanged.

InlineParser::parse() reused one lexer instance, but a rule may parse content of its own while it applies — a text role parsing its argument, for example. The nested call handed the same lexer new input, discarding the token stream the outer parse was reading, and a later rollback() then indexed positions that no longer existed: a TypeError, not a wrong result. No caller in this repository nests today, so no test could have caught it; for a third party text role it was a silent contract change. The shared instance is now only handed out while it is free, and a nested parse allocates its own. The docblock warned about "race conditions" — there are no threads here, the hazard is re-entrancy, and it now says so.

Buffer::trimLines() changed the lines without resetting the unindented memo that every other mutator resets. Harmless today, because trim() leaves no indentation for unIndent() to remove, but the invariant did not hold.

Two tests come with it: one for the nested parse, which fails against the previous state of this branch, and one pinning the two scheme tables — the deprecated regex the lexer still builds from, and the list behind isSupportedScheme() — to the same content, so a scheme added to one only cannot make the lexer find a link the resolver then refuses.

Assisted by claude-code:claude-opus-5 — Session

@CybotTM
CybotTM force-pushed the perf/rst-parsing-optimizations branch from edb847a to 6d2e211 Compare January 22, 2026 00:26
@CybotTM CybotTM changed the title perf: Optimize RST parsing with regex and instance caching perf: Optimize RST parsing with pattern and instance caching Jan 22, 2026
@CybotTM
CybotTM force-pushed the perf/rst-parsing-optimizations branch 2 times, most recently from bcc53c1 to b642af7 Compare January 22, 2026 01:41
@CybotTM
CybotTM force-pushed the perf/rst-parsing-optimizations branch from b642af7 to 6a14fda Compare January 23, 2026 13:00
@CybotTM
CybotTM marked this pull request as ready for review January 23, 2026 13:06
@CybotTM
CybotTM force-pushed the perf/rst-parsing-optimizations branch from 6a14fda to a7f8348 Compare March 9, 2026 00:21
@CybotTM
CybotTM force-pushed the perf/rst-parsing-optimizations branch from a7f8348 to 270201e Compare April 16, 2026 22:00
@CybotTM
CybotTM force-pushed the perf/rst-parsing-optimizations branch from 270201e to a7886c6 Compare June 2, 2026 14:56
@CybotTM
CybotTM force-pushed the perf/rst-parsing-optimizations branch 2 times, most recently from eae4a9b to 396c7d8 Compare June 24, 2026 15:26
CybotTM added 2 commits July 1, 2026 14:02
Add caching optimizations for hot paths in RST parsing:

- InlineParser: reuse single InlineLexer instance instead of creating
  new one per parse call (lexer state fully reset via setInput())
- InlineLexer: cache expensive hyperlink pattern built from
  SUPPORTED_SCHEMAS (5600+ chars) as static variable
- LineChecker: add static caches for isDirective(), isLink(), and
  isAnnotation() regex results with proper cache key handling
- Buffer: ensure unindented flag is reset in all mutators (set, pop,
  clear) for consistent cache invalidation
- CachableInlineRule: simplify type annotations

Note: Lexer reuse assumes single-threaded parsing. Concurrent parsing
would require separate instances.

See https://cybottm.github.io/render-guides/ for benchmark data.
Add SUPPORTED_SCHEMAS_LIST and isSupportedScheme() to ExternalReferenceResolver
for O(1) hash set lookup instead of regex matching against 371 IANA schemes.
This is ~6x faster than the 5600+ character regex pattern.

InlineLexer now uses ExternalReferenceResolver::isSupportedScheme() to
validate URI schemes during tokenization.

Note: This change is also in PR phpDocumentor#1287 - when both PRs merge, the conflict
is trivially resolved by keeping one version.
@CybotTM
CybotTM force-pushed the perf/rst-parsing-optimizations branch from 396c7d8 to 9158070 Compare July 1, 2026 12:02
Three corrections to the previous commits of this branch, all found by review.

The three caches in `LineChecker` are static and keyed by the line itself, with
nothing clearing or bounding them. `DirectiveRule::applies()` asks about every
line of every document, so they retained a copy of each distinct line for the
lifetime of the process — across documents, across projects in a long running
renderer. Bound them: when a table is full it is dropped, which keeps the
working set cached, since documents are parsed line by line, and puts a ceiling
on what is held.

`InlineParser::parse()` reused one lexer instance, but a rule may parse content
of its own while it applies — a text role parsing its argument, for example. The
nested call then handed the same lexer new input, discarding the token stream the
outer parse was reading, and a later rollback indexed positions that no longer
existed: a `TypeError`, not a wrong result. The shared instance is now only
handed out while it is free, and a nested parse allocates its own. The docblock
also warned about "race conditions"; there are no threads here, the hazard is
re-entrancy, and it now says so.

`Buffer::trimLines()` changed the lines without resetting the `unindented` memo
that every other mutator resets. Harmless today because `trim()` leaves no
indentation for `unIndent()` to remove, but the invariant did not hold.

Add the test that pins the two scheme tables — the deprecated regex the lexer
still builds from, and the list behind `isSupportedScheme()` — to the same
content, so a scheme added to one only cannot make the lexer find a link the
resolver refuses. Add a test for the nested parse; it fails against the previous
state.

Assisted-by: claude-code:claude-opus-5
Agent-Session: https://claude.ai/code/session_015QXXkquh2eQNBiTYA39Wss
Signed-off-by: Sebastian Mendel <sebastian.mendel@netresearch.de>
@CybotTM CybotTM changed the title perf: Optimize RST parsing with pattern and instance caching [TASK] Optimize RST parsing with pattern and instance caching Aug 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant