feat(readState): add NIP-RS override layer format and parse (slice 1) - #3966
Open
wpfleger96 wants to merge 4 commits into
Open
feat(readState): add NIP-RS override layer format and parse (slice 1)#3966wpfleger96 wants to merge 4 commits into
wpfleger96 wants to merge 4 commits into
Conversation
Implements the manual-unread override register algebra from the merged NIP-RS spec (docs/nips/NIP-RS.md, merge commit 209536a) as pure functions in the format and snapshot layers. No manager or UI wiring — slice 1 only; slices 2 and 3 depend on the types exported here. Changes in readStateFormat.ts: - escapeFrontierKey / unescapeFrontierKey: bijection for raw context IDs that begin with ov_ or esc: (Reserved Namespace spec rule). - isOverrideKey: classifies ov_s:, ov_c:, ov_b: wire keys. - OverrideRegister interface: the (S, C, B) triple for one context. - OverrideLiveness interface: evaluation result for slice 3 consumers. - isOverrideActive: liveness predicate S>0 AND F<=B AND S>C (clear-wins, mandatory per spec Tie Policy section). - mergeOverrideRegisters: componentwise max() — commutative, associative, idempotent; matches model.py::merge_reg_b semantics exactly. - encodeOverrideGroup: canonical wire encoding (live three-key / tombstone floor / virgin-omit rules per Mandatory Canonical Publication section). - partitionOverrideGroups: group-first override validation required by Content Validation — collects complete logical groups before any per-entry processing; partial groups are dropped, frontier retained. - ValidatedOverrideGroup discriminated union: live | floor. - sanitizeContexts: extended to run group-first validation (step 1), then per-entry sanitization on non-override keys (step 2), then fold validated groups back as integers (step 3). Existing callers unaffected. Changes in readStateSnapshot.ts: - extractOverrideRegisters: reconstructs register map from a sanitized flat contexts map (read-side counterpart of encodeOverrideGroup). - mergeOverrideRegisterMaps: merges register maps from multiple blobs via componentwise max() (register-level analogue of frontier merge). New file readStateOverride.test.mjs (58 tests): - escape/unescape bijection including double-escape and round-trip cases. - isOverrideKey coverage for all three prefixes and negative cases. - isOverrideActive: live, virgin, frontier-dominated, tie (clear-wins), clear-exceeds-set, and tombstone-floor cases. - mergeOverrideRegisters: componentwise max, commutativity, idempotence, associativity. - encodeOverrideGroup: all canonical shapes (live, dead, tie, frontier- dominated, pre-compacted floor, virgin). - partitionOverrideGroups: live group, tombstone floor, partial group (s-only, s+c only), invalid value, negative value, invalid floor c, multi-context with mixed validity. - sanitizeContexts: live group, tombstone floor, partial-group dropped with frontier retained, normal entries, escaped frontier key pass-through. - extractOverrideRegisters: live group, floor, no overrides, plain record. - mergeOverrideRegisterMaps: two-map, disjoint, empty, single, three-way. - Round-trip: live register, tombstone floor, and virgin register. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Replace the flat-map decode path with a parseContexts() function that
returns {frontiers, overrides} — separate namespaces keyed by raw
context ID. This fixes three correctness defects in the previous
implementation:
- Frontier keys were stored with their esc: prefix intact, so
isOverrideActive evaluated against the wrong frontier for any raw
context ID starting with ov_ or esc: (e.g. raw ov_s:evil had its
frontier stored as esc:ov_s:evil while the override register was
keyed by ov_s:evil).
- The 256-byte UTF-8 key-length check was applied only to frontier
entries; ov_* sibling keys were written back unchecked, allowing
wire keys up to 5 + suffix bytes through.
- extractOverrideRegisters accepted an arbitrary flat map and zero-filled
missing components, reconstructing partial groups that group-first
validation had already rejected.
Changes:
- readStateFormat.ts: add ParsedContexts interface and parseContexts();
make partitionOverrideGroups internal; add 256-byte enforcement to
every wire key (frontier and override siblings alike); fix the
sanitizeContexts doc comment.
- readStateSnapshot.ts: add contexts: ParsedContexts to
ParsedReadStateEvent; parseReadStateEvent populates both blob.contexts
(flat, for readStateManager backward compat) and contexts (structured);
mergeReadStateEvents iterates parsed.contexts.frontiers so callers
receive unescaped raw ctx IDs; remove extractOverrideRegisters; update
mergeOverrideRegisterMaps to accept ReadonlyMap inputs.
- readStateOverride.test.mjs: rewrite around parseContexts(); add the
full partial-group rejection matrix (S-only, B-only, S+B, C+B, S+C),
invalid sibling at each position, uint32 boundary values, 256/257-byte
key-length cases for live and floor shapes, multibyte UTF-8 boundary,
and Thufir's exact collision witness (raw ctx ov_s:evil alongside ctx
evil with its own ov_s:evil counter). 3968/3968 pass.
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
… and liveness
Addresses all three Thufir pass-2 IMPORTANTs and the MINOR:
decodeContexts() — single internal 3-pass validator returning
{ wire, frontiers, overrides }. sanitizeContexts() and parseContexts()
are now thin projections of that one result; partitionOverrideGroups is
deleted. Validation logic exists exactly once.
mergeReadStateEventsStructured() — primary public merge function returning
MergedReadState { frontiers, overrides } using mergeOverrideRegisterMaps
internally. computeOverrideLiveness() evaluates OverrideLiveness from a
merged result, giving OverrideLiveness its first production path.
mergeReadStateEvents() becomes a thin frontier-only wrapper over
mergeReadStateEventsStructured, byte-identical for existing callers.
ParsedContexts doc comment updated: wire escaping is removed at decode;
raw IDs may themselves begin with reserved prefixes such as ov_* or esc:.
Tests: 3978/3978 pass (+10 tests). Added event-level integration suite
exercising parseReadStateEvent + mergeReadStateEventsStructured directly:
two-blob merge asserting both frontier and register components, liveness
after join (active and inactive cases), unknown-pubkey empty result, and
the full collision witness (ov_s:evil with its own ov_s:ov_s:evil register
alongside evil's plain register and both frontiers through two events).
Added exact multibyte 256/257-byte boundary cases for both live and floor
shapes (é × 125 + a/aa = 251/252 bytes suffix → wire key 256/257 bytes).
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…/Snapshot
Finding 1 — effective frontier for liveness:
computeOverrideLiveness now takes effectiveFrontier as a required third
argument (was reading merged.frontiers.get(rawCtxId) internally). NIP-RS
:169-196/:510-514 requires the hierarchical effective frontier, not the
context's own stored value. Callers supply max(own, parent) before calling.
Finding 2 — d-tag slot regex:
isValidReadStateDTag now validates against /^[0-9a-f]{32}$/ (was any
1-64 ASCII characters). Nonconforming coordinates are now correctly
rejected per NIP-RS :55/:68. Updated all test fixtures (slot1/slot2
→ valid 32-hex constants; read-state:test in communityUnreadObserver
→ valid slot). Removed the now-dead isAscii helper.
Finding 3 — null-prototype wire projection:
decodeContexts builds wire with Object.create(null) instead of {}, so a
legitimate __proto__ context ID is stored as an own property in both the
wire and frontiers projections rather than silently dropped from wire.
Finding 4 — decode once in parseReadStateEvent:
decodeContexts is now exported. parseReadStateEvent calls it once and
derives both blob.contexts (wire) and contexts (frontiers/overrides) from
the single result, eliminating the double traversal of up to 10k entries.
Deleted the unused exported ValidatedOverrideGroup type.
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Implements the format/parse layer for NIP-RS manual-unread override (slice 1 of 3): pure functions in
readStateFormat.ts,readStateSnapshot.ts, and the test suite.Architecture
Single decode core
decodeContexts(raw)— internal function; the only place the 3-pass validation algorithm lives.Returns three projections of the same validated data:
wire— flatRecord<string, number>withov_*keys in wire form (backward compat forReadStateBlob.contexts)frontiers—Map<rawCtx, number>with frontier keys unescapedoverrides—Map<rawCtx, OverrideRegister>keyed by the same raw context IDssanitizeContexts()andparseContexts()are thin projections ofdecodeContexts— validation logic exists exactly once.Validation rules (spec NIP-RS §Content Validation)
ov_*entries by suffixesc:…→ raw ctx) for the structured maps.Structured event merge
mergeReadStateEventsStructured(events, pubkey, decrypt?)— primary public entry point. ReturnsMergedReadState { frontiers, overrides }usingmergeOverrideRegisterMapsinternally.computeOverrideLiveness(merged, rawCtxId)— evaluatesOverrideLivenessfrom a merged result. This is the production path forOverrideLiveness.mergeReadStateEvents()— legacy wrapper returning only.frontiers; byte-identical forcommunityUnreadObserverand other existing callers.Exported surface (slices 2–3 contract)
ParsedContextsreadStateFormat.ts{frontiers, overrides}from one blobparseContexts()readStateFormat.tsMergedReadStatereadStateSnapshot.ts{frontiers, overrides}across all eventsmergeReadStateEventsStructured()readStateSnapshot.tscomputeOverrideLiveness()readStateSnapshot.tsmergeReadStateEvents()readStateSnapshot.tsmergeOverrideRegisterMaps()readStateSnapshot.tsOverrideRegisterreadStateFormat.tsOverrideLivenessreadStateFormat.ts{active, frontier}isOverrideActive()readStateFormat.tsmergeOverrideRegisters()readStateFormat.tsencodeOverrideGroup()readStateFormat.tsescapeFrontierKey()/unescapeFrontierKey()readStateFormat.tsTest coverage (3978/3978 pass)
isOverrideKey,isOverrideActive(liveness predicate + uint32 boundaries)mergeOverrideRegisters: commutative, associative, idempotent, uint32 maxencodeOverrideGroup: all 6 canonical shapes (live, 4 dead forms, virgin-omit)parseContexts: every illegal partial-group subset (S-only, B-only, S+C, S+B, C+B), invalid sibling at each position, uint32 boundaries, 256/257-byte ASCII and multibyte exact boundary cases for both live and floor shapesov_s:evilwith its own register +evilwith its own register — distinct namespaces post-decodeparseReadStateEvent+mergeReadStateEventsStructured: two-blob merge asserting frontier AND register components, liveness evaluation (active/inactive), unknown-pubkey empty result, full collision witness across two eventsStack
Stack: this PR → #4002 → #4005