feat(sync-service): tail-drop empty shape-GET response spans by alco · Pull Request #4682 · electric-sql/electric

alco · 2026-07-02T16:21:17Z

What

Tail-drop the OpenTelemetry spans of empty / up-to-date shape-GET responses at the origin, to cut exported trace volume. Empty responses are ~95% of Plug_shape_get root spans in production, yet carry no useful trace detail. Part of the ongoing effort to reduce Honeycomb trace volume.

How

Electric.Telemetry.EmptyResponseSampler — a pure, unit-tested decision function for the root span's SampleRate weight. In order:
1. status >= 500 → SampleRate = 1, never dropped (keep-on-error wins).
2. empty, non-SSE 2xx + drop enabled → SampleRate = 0 (the drop sentinel).
3. otherwise → unchanged (whatever the upstream rate hint produced).
Electric.Telemetry.OpenTelemetry.EmptyResponseDropProcessor — an :otel_span_processor whose on_end/2 returns :dropped for spans stamped SampleRate = 0. Registered ahead of the exporting processors so that a dropped span is never queued for export. Only empty-response spans carry the sentinel; everything else passes through.

Config

ELECTRIC_DROP_EMPTY_RESPONSE_SPANS (boolean, default false) — opt-in; set it to true to enable the drop. Error (5xx) and SSE responses are never dropped.
ELECTRIC_LONG_POLL_TIMEOUT (integer, ms) — exposes the existing long-poll timeout as a config knob (used by the integration test to trigger an empty long-poll response quickly).

Fixes #4664

🤖 Generated with Claude Code

Pure function deciding the SampleRate weight for a shape-GET root span so the spans of empty/up-to-date responses can be tail-dropped. Keep-on-error wins first (5xx -> SampleRate=1), empty non-SSE 2xx responses get the SampleRate=0 drop sentinel when enabled, everything else is left unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add an OTel span processor whose on_end/2 returns :dropped for spans carrying SampleRate=0, and register it ahead of otel_batch_processor. The SDK folds on_end with a short-circuiting andalso, so a non-true return before the batch processor prevents the span from ever being queued for export. Only empty-response spans carry the sentinel; all others pass through unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Wire the empty-response drop into the request-end SampleRate stamp: empty, non-SSE 2xx shape-GET responses are stamped with the SampleRate=0 sentinel so the drop processor skips them at export time. Gated by the new ELECTRIC_DROP_EMPTY_RESPONSE_SPANS env var (default true); set it to false to export those spans normally. Empty/up-to-date responses are ~95% of shape-GET root spans in production, so dropping them substantially cuts exported trace volume. Error (5xx) and SSE responses are never dropped. Ratio-based keep-sampling is deferred to the cloud worker, which will propagate its rate to the origin via tracestate. Fixes #4664 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

codecov · 2026-07-02T16:23:25Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.20%. Comparing base (1b611ca) to head (b497d62).
⚠️ Report is 5 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4682   +/-   ##
=======================================
  Coverage   60.20%   60.20%           
=======================================
  Files         410      410           
  Lines       44378    44393   +15     
  Branches    12581    12587    +6     
=======================================
+ Hits        26716    26727   +11     
- Misses      17584    17585    +1     
- Partials       78       81    +3

Flag	Coverage Δ
electric-telemetry	`73.80% <ø> (+0.43%)`	⬆️
elixir	`73.80% <ø> (+0.43%)`	⬆️
packages/agents	`72.64% <ø> (ø)`
packages/agents-mcp	`77.70% <ø> (ø)`
packages/agents-mobile	`80.67% <ø> (ø)`
packages/agents-runtime	`83.72% <ø> (-0.03%)`	⬇️
packages/agents-server	`75.45% <ø> (-0.03%)`	⬇️
packages/agents-server-ui	`8.32% <ø> (ø)`
packages/electric-ax	`51.06% <ø> (ø)`
packages/experimental	`87.73% <ø> (ø)`
packages/react-hooks	`86.48% <ø> (ø)`
packages/start	`82.83% <ø> (ø)`
packages/typescript-client	`91.86% <ø> (+0.03%)`	⬆️
packages/y-electric	`56.05% <ø> (ø)`
typescript	`60.00% <ø> (-0.01%)`	⬇️
unit-tests	`60.20% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

claude · 2026-07-02T16:32:09Z

Claude Code Review

Summary

Tail-drops the OTel spans of empty/up-to-date shape-GET responses via a SampleRate = 0 sentinel stamped on the root Plug_shape_get span and a first-in-line :otel_span_processor that returns :dropped. Iteration 7 adds a single test-only commit since my last pass (Disable connection manager ping in the slot-self-conflict Lux test). No production code changed; no new correctness issues.

What's Working Well

The new commit is a well-formed, contained test fix. replication-slot-self-conflict.lux now starts Electric via setup_electric_with_env "DO_NOT_START_CONN_MAN_PING=1" instead of the bare setup_electric. This suppresses the periodic ConnectionManagerPing (started conditionally at _macros.luxinc:186 only when DO_NOT_START_CONN_MAN_PING is unset), whose log output is noise for a test that deliberately matches against idle-warning patterns and uses -...|$fail_pattern failure matches over a 12s window. Reducing that noise makes the slot-conflict assertions more robust, exactly as the commit message states.
Consistent with established conventions. DO_NOT_START_CONN_MAN_PING=1 mirrors existing usage in otel-export.lux (=1) and db-connection-scaledown.lux (=true); the macro gate is is_nil/1, so any non-nil value disables the ping. Correct and idiomatic.
Everything from prior iterations still holds. Drop ships default-off (drop_empty_response_spans?: false); keep-on-error (status >= 500 -> SampleRate = 1) is checked first; SSE responses are excluded; sample_rate_attrs/1 returns base_attrs verbatim when the decision is :unchanged; the :empty metric tag keeps empty vs non-empty exactly countable independent of span sampling; the changeset covers both publishable packages (@core/electric-telemetry, @core/sync-service); and the end-to-end lux assertions verify the dropped-but-counted invariant. All of @alco's inline comments remain addressed.

Issues Found

Critical (Must Fix)

None identified.

Important (Should Fix)

None identified.

Suggestions (Nice to Have)

Both carried over from prior iterations, both still optional:

1. Orphaned-child robustness comment. The processor drops only the single span carrying SampleRate = 0; the empty path is flat by construction (no_change_response leaves chunked: false, so no stream_chunk child), so nothing is orphaned today. A one-line note near the sentinel recording that flat-trace invariant would protect a future child-span addition — the issue's "verify children are not orphaned" ask.

2. Positional booleans in sample_rate/4. Three consecutive boolean() args are transposition-prone at future call sites; the single current call site is correct.

(Minor, non-blocking: the new lux fix is a slot-conflict test noise-reduction unrelated to the empty-response-span feature this PR ships. Bundling it here is harmless but slightly widens the PR's scope beyond #4664.)

Issue Conformance

Fixes #4664. Ships the all-or-nothing boolean (ELECTRIC_DROP_EMPTY_RESPONSE_SPANS, default false) rather than the issue's floated ELECTRIC_EMPTY_SPAN_SAMPLE_RATIO, deferring ratio-based keep-sampling to the cloud worker via the tracestate rate hint — a refinement the issue explicitly invited. Count-visibility (consequence (a)) stays resolved by the :empty metric tag. Ratio-based keep-sampling of empties (consequence (b)) remains deferred by design; it only bites once the flag is enabled and a keep-rate hint is propagated, at which point the origin's unconditional SampleRate = 0 for empties will need revisiting.

Previous Review Status

New this iteration: one test-only commit disabling the connection-manager ping in replication-slot-self-conflict.lux to reduce log noise. Verified correct against the _macros.luxinc gate and consistent with existing usage. Nothing regressed.
Carried over (both optional): orphaned-child invariant comment and positional booleans in sample_rate/4.
Still deferred (by design): ratio-based keep-sampling of empties (consequence (b)).

Review iteration: 7 | 2026-07-07

Extend the otel-export lux suite: an already-up-to-date request returns an empty no-change response, and the suite asserts a data Plug_shape_get root span is exported (shape_req.is_empty_response=false) while the empty one is never exported (no Plug_shape_get span with is_empty_response=true reaches the collector). Exercises the drop end-to-end through the real OTel export pipeline. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Extract request_and_response/1 so open_telemetry_attrs/1 and response_trace_attrs/1 share a single resolver for the request/response structs instead of each re-deriving them from the conn assigns, keeping the two in sync. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The empty-response assertion was exercising the wrong path: ot_is_empty_response is set only by no_change_response, which is reached only when a live long-poll times out with no new data. A plain non-live request at the current offset returns is_empty_response=false, so the drop was never triggered and the assertion passed vacuously. Use a live request that long-polls until timeout, and assert its body is the up-to-date control only (no data element) so the drop check is meaningful. Also replace the bare "0" count match, which flaked on lux prompt-gluing (SH-PROMPT:0), with the suite's `test ... && echo TOKEN` / `??TOKEN` idiom. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Flip ELECTRIC_DROP_EMPTY_RESPONSE_SPANS to default false so the drop ships dormant and is opt-in per environment: merging the code changes nothing until an operator sets it true. The otel-export integration test now enables it explicitly to exercise the drop. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Expose the existing long_poll_timeout config value via ELECTRIC_LONG_POLL_TIMEOUT so it can be tuned per environment, e.g. shortened in integration tests to trigger an empty long-poll response quickly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Address review feedback: remove over-explanatory module/inline comments, drop references to the private worker repo and to SDK-internal fold mechanics, and stop restating default values in comments. Simplify the otel-export lux test to trigger the empty long-poll response via a short ELECTRIC_LONG_POLL_TIMEOUT instead of waiting out the default, and tighten its comments. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add `empty` to the [:electric, :plug, :serve_shape] event metadata and to the electric.plug.serve_shape.requests.count metric's tags. This keeps a precise count of empty vs non-empty responses via metrics even when their spans are dropped (ELECTRIC_DROP_EMPTY_RESPONSE_SPANS=true), since the metric path is independent of span sampling. The otel-export test asserts the dropped empty response is still counted with empty=true in the exported metric. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

A bit too noisy log output

github-actions · 2026-07-08T04:42:47Z

This PR has been released! 🚀

The following packages include changes from this PR:

@core/sync-service@1.7.7

Thanks for contributing to Electric!

alco and others added 3 commits July 2, 2026 18:15

alco added the claude label Jul 2, 2026

alco and others added 4 commits July 2, 2026 18:37

alco commented Jul 3, 2026

View reviewed changes

alco and others added 3 commits July 3, 2026 17:34

alco self-assigned this Jul 3, 2026

alco added 2 commits July 7, 2026 10:30

Add electric-telemetry to the changeset

85be33a

Disable connection manager ping in the slot-self-conflict Lux test

b497d62

A bit too noisy log output

icehaunter approved these changes Jul 7, 2026

View reviewed changes

alco merged commit c259e7b into main Jul 7, 2026
72 checks passed

alco deleted the alco--origin-drop-empty-response-spans branch July 7, 2026 10:52

github-actions Bot mentioned this pull request Jul 8, 2026

Tail-drop empty/up-to-date shape-GET spans at the origin to cut OTel event volume #4664

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(sync-service): tail-drop empty shape-GET response spans#4682

feat(sync-service): tail-drop empty shape-GET response spans#4682
alco merged 12 commits into
mainfrom
alco--origin-drop-empty-response-spans

alco commented Jul 2, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 2, 2026 •

edited

Loading

Uh oh!

claude Bot commented Jul 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

alco commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Config

Uh oh!

codecov Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

claude Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Claude Code Review

Summary

What's Working Well

Issues Found

Critical (Must Fix)

Important (Should Fix)

Suggestions (Nice to Have)

Issue Conformance

Previous Review Status

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

alco commented Jul 2, 2026 •

edited

Loading

codecov Bot commented Jul 2, 2026 •

edited

Loading

claude Bot commented Jul 2, 2026 •

edited

Loading