Skip to content

Build the well-known-string table at compile time - #13559

Open
moonchen wants to merge 2 commits into
apache:masterfrom
moonchen:wks-compile-time-table
Open

Build the well-known-string table at compile time#13559
moonchen wants to merge 2 commits into
apache:masterfrom
moonchen:wks-compile-time-table

Conversation

@moonchen

@moonchen moonchen commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

hdrtoken_init() builds the well-known-string (WKS) table at startup. This PR does the work at compile-time, which enables more robust checking, and removes an old restriction regarding prefixes.

1. Modernize the well-known-string initialization

Make use of constexpr to initialize WKS at compile time. Remove the unnecessary name field from the WKS table at runtime. Trim the hdrtoken_hash_table size to just the reachable number of slots, build it at compile time as well, and shrink its buckets from 16 to 8 bytes by storing an index instead of a pointer.

before after
WKS table ~15 KB on the heap 10,800 B in .rodata
hdrtoken_hash_table 1,052,904 B 262,144 B in .rodata
sizeof(HdrTokenHeapPrefix) 56 48

2. Add compile-time checks for WKS correctness

Checks are now done at compile-time, and the new checks are more strict than the old release asserts.

An append-only ledger (_hdrtoken_strs_frozen) fingerprints the frozen entries: inserting, reordering, or editing them fails to compile, and appending strings requires appending a matching ledger row.

3. Remove the restriction that prevents a WKS from being added after its prefix

In order to maintain cache compatibility, new strings needed to be appended to the list. Previously, it was not legal to append a string to WKS if its prefix already exists.

This PR removes the prefix restriction. We can now append strings like Server-Timing, Accept-CH, and Expect-CT.

@moonchen
moonchen force-pushed the wks-compile-time-table branch 5 times, most recently from 634f1de to bf93e5e Compare August 18, 2026 19:39
@moonchen moonchen self-assigned this Aug 18, 2026
@moonchen moonchen added the HTTP label Aug 18, 2026
@moonchen

Copy link
Copy Markdown
Contributor Author

[approve ci Autest 1]

@moonchen

Copy link
Copy Markdown
Contributor Author

[approve ci autest 1]

@moonchen
moonchen requested a balanced review from Copilot August 19, 2026 17:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Moves well-known-string (WKS) construction from runtime to compile time using constexpr, enabling stronger correctness checks and reducing startup work / memory overhead.

Changes:

  • Build the WKS table at compile time and populate runtime “hot” parallel arrays from it during hdrtoken_init().
  • Add compile-time validation (initializer/name resolution, hash-slot uniqueness) and shrink the hash table to the reachable slot space.
  • Relocate Cache-Control cooked-mask bindings into WKS initialization and remove the MIME startup hook.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
src/proxy/hdrs/MIME.cc Removes runtime Cache-Control mask initialization; updates WKS/prefix access to const.
src/proxy/hdrs/HdrToken.cc Introduces compile-time WKS table build + checks; updates hashing/slot sizing; keeps runtime init lightweight.
include/proxy/hdrs/MIME.h Removes declaration of the deleted Cache-Control init hook.
include/proxy/hdrs/HdrToken.h Updates field initializer types and makes WKS prefix access read-only (const).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/proxy/hdrs/HdrToken.cc
Comment thread src/proxy/hdrs/HdrToken.cc
Comment thread include/proxy/hdrs/HdrToken.h Outdated
hdrtoken_init() built the well-known-string table at startup in an
ats_calloc heap, and resolved initializer names through a PCRE2 DFA.
Those patterns are start-anchored, so a name matched the first pattern
it prefixed. That forced the array to keep the longer entry of every
case-insensitive prefix pair at the lower index, which contradicts the
rule that new strings must be appended because their indexes are
stored on disk for cached objects. An appended name that an existing
entry prefixes resolves to that entry and silently overwrites its slot
id, presence mask and flags.

Build the whole table during translation from a constexpr array of
string_view, and resolve each initializer name with an exact
case-insensitive comparison, so a prefix pair works in either order.
static_assert now enforces what the startup checks did: every
initializer name matches an entry, no two rows of a table claim the
same entry, and no two strings share a hash slot. Both
ink_release_assert range tests are gone, along with the collision scan
and its abort().

The table is read-only, so the prefix accessors return const and the
cooked Cache-Control masks move in from MIME.cc. The name field held a
pointer into the table, which a constant may not do, and nothing read
it, so it now lives only in the initializer row type.
hdrtoken_hash_table halves as well: hash_to_slot() masks a hash to 15
bits, but the table held 65536 buckets, so half was unreachable.
@moonchen
moonchen force-pushed the wks-compile-time-table branch from bf93e5e to 9184ae1 Compare August 19, 2026 22:02
@moonchen
moonchen marked this pull request as ready for review August 20, 2026 18:14
Copilot AI review requested due to automatic review settings August 20, 2026 18:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Comment thread include/proxy/hdrs/HdrToken.h Outdated
Comment thread src/proxy/hdrs/HdrToken.cc
Comment thread src/proxy/hdrs/HdrToken.cc
Comment thread src/proxy/hdrs/HdrToken.cc Outdated
Copilot AI review requested due to automatic review settings August 20, 2026 22:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/proxy/hdrs/HdrToken.cc:354

  • This fingerprint uses the case-folding token hash, so it does not actually detect every edit that the frozen ledger promises to reject. For example, changing the frozen method GET to get leaves this fingerprint unchanged, but then hdrtoken_method_tokenize() rejects GET and accepts get. Hash the original bytes (without hdrtoken_ascii_toupper) for the ledger; token lookup can remain case-insensitive.
    for (char const c : _hdrtoken_strs[i]) {
      hval = hdrtoken_hash_step(hval, static_cast<unsigned char>(c));
    }
    hval = hdrtoken_hash_step(hval, '\0'); // fold in a terminator so entry boundaries matter

src/proxy/hdrs/HdrToken.cc:624

  • These range endpoints no longer belong to one character array: every HdrTokenWksEntry::str is a distinct array subobject. hdrtoken_is_wks() still classifies pointers with built-in >=/<=, whose ordering is unspecified across these subarrays, so a valid WKS pointer from an intermediate entry is not portably guaranteed to fall between these endpoints. Represent and compare the bounds as uintptr_t values (consistent with hdrtoken_wks_to_prefix()), or otherwise use a defined total pointer ordering.
const char *_hdrtoken_strs_heap_f = &hdrtoken_wks_table[0].str[0]; // storage first byte
const char *_hdrtoken_strs_heap_l = &hdrtoken_wks_table[std::size(_hdrtoken_strs) - 1].str[HDRTOKEN_WKS_STORAGE - 1];

Computing "wks - sizeof(HdrTokenHeapPrefix)" formed a pointer outside
the entry's str array member, which is undefined behavior even though
the prefix is physically adjacent. Recover the entry by index through
integer arithmetic instead.

Also enforce the append-only rule with a fingerprint of the frozen
entries, validate the slot, presence-mask, and Cache-Control
invariants of the initializer rows, and build the hash table at
compile time, replacing pointer buckets with index buckets at half
the size and removing hdrtoken_hash_init().
Copilot AI review requested due to automatic review settings August 20, 2026 23:12
@moonchen
moonchen force-pushed the wks-compile-time-table branch from aa8a773 to e3179c1 Compare August 20, 2026 23:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants