Conversation
jwils
added this pull request to stack #1381
September 11, 2026 20:58
jwils
changed the title
jwils
removed this pull request from stack #1381
September 13, 2026 15:10
jwils
force-pushed
the
joshuaw/buf-breaking-validation
branch
from
September 13, 2026 15:24
cd9bb00 to
4fd140d
Compare
jwils
added a commit
that referenced
this pull request
Sep 13, 2026
) ## Summary Closes #1360. `schema.proto` only had messages for the indexed types and the types they reference, so a publisher could not send an event for a pure-source type: one that feeds a `sourced_from` field but has no index of its own. #1346 widened the JSON schema event envelope to accept these events, but proto ingestion never got the same treatment. Nothing in an indexed type's proto representation refers to a pure-source type, so the traversal never reached it. Both JSON and proto now use `Results#ingestible_types_by_name` to select indexed document types and `sourced_from` source types, excluding derived indexing types. `Results` resolves the completed schema once and passes the shared map to both builders. Proto traverses referenced types and emits abstract wrappers; JSON selects concrete event names. The messages themselves needed no change, as #1360 predicted. `proto_fields` builds from `indexing_fields_by_name_in_index`, which does not depend on the type having an index: ```protobuf message ComponentDesign { string id = 1; string component_id = 2; string designer_name = 3; // Next field number: 4 } ``` ## Derived indexing types (a Buf-breaking change) Derived indexing types are now excluded from the seed. ElasticGraph builds their documents from the events of other types, so a publisher never sends one, and the JSON schema artifact has never described them either. Proto ingestion described them only because a derived type has its own index, which put it in the old seed. Removing a message is a breaking change under the Buf check that #1378 adds. No published schema depends on these messages yet, so this is the moment to align the two formats. ## Abstract source types A `sourced_from` source type can be an interface or a union. Such a type keeps the `oneof` wrapper message it would get as an indexed abstract type, and its subtypes get messages too. Note the JSON event envelope does not accept an abstract type by name, and it does not accept the concrete subtypes either, because they are neither indexed nor source types. So an abstract source type is not publishable over JSON today. That gap is tracked in #1386 and is not addressed here. ## Notes - No schema artifact churn. The root `Rakefile` does not register the proto extension, and this repo commits no `schema.proto`. - Proto schema generation now runs the `sourced_from` resolver, so it can raise relationship and `sourced_from` validation errors that it previously skipped. No existing proto spec defines a relationship, so no fixture needed a repair.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Stable field numbers prevent protobuf data from being reinterpreted, but they do not catch incompatible field-type, syntax, package, or file-option changes. Maintaining those compatibility rules ourselves would duplicate Buf and leave room for subtle gaps.
What
proto_schema_version, defaulting to 1, and record it in the generatedschema.proto.FILEbreaking rules before dumping a changed artifact at the same version.enforce_proto_schema_version falseescape hatch.Risk
This adds a development-time dependency on the Buf CLI whenever an existing
schema.protochanges without a version bump. First-time generation and explicitly versioned breaking changes do not invoke Buf. Projects can temporarily disable enforcement while prototyping.Test plan
script/run_gem_specs elasticgraph-proto_ingestion: 101 examples, 100% line and branch coverage.script/lint,script/type_check,script/spellcheck, documentation coverage, workflow/config validation, README snippet validation, andsite:validate.script/quick_build: 5,291 of 5,292 examples passed; one unrelated datastore acceptance example observed a record from another test environment and passed immediately when rerun in isolation.References