Skip to content

fix(aggregation): one addressing model for an aggregating entity's resources - #626

Open
bburda wants to merge 17 commits into
mainfrom
fix/aggregator-peer-fanout-and-timeouts
Open

fix(aggregation): one addressing model for an aggregating entity's resources#626
bburda wants to merge 17 commits into
mainfrom
fix/aggregator-peer-fanout-and-timeouts

Conversation

@bburda

@bburda bburda commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

An aggregating gateway lists resources that belong to members on other gateways, but it served every request for them locally. A data read looked for the peer's topic on its own ROS graph. An execution called the peer's service there. A configuration resolved against nodes only this gateway can see. All three failed while the member and its gateway were healthy.

This branch gives every entity and every resource collection one addressing model:

  • An item id carries its member only when more than one member provides it. Ambiguity is decided from the declared tree, not from one response, so a listing and an execution cannot disagree about the same id.
  • A member-qualified request is served by the member's own route on the gateway that owns it. Data, operations, executions and configurations share one dispatch point. A member this gateway owns is still served here.
  • Availability is a separate question. It is carried by the entities a request can be addressed to, which are Apps and Components. An Area or a Function groups members and has no availability of its own.
  • An entity a peer declared in its manifest stays in the tree when that peer goes quiet, keeps the items it last reported, and answers 504 not-responding. An entity the peer only discovered at runtime disappears.
  • Where one provider exposes the same short name at two ROS paths, those operations are addressed by their path. Every id that resolves today is unchanged.

Defects found while building this and fixed here:

  • A partial read of a peer was published as a complete picture. A Function whose detail request failed was merged with no members, and that memberless copy replaced the retained declaration, so it outlived the peer.
  • A 504 not-responding from a nested collection was read as a failed request, so one unreachable member behind a peer froze that peer's whole view until it was restarted.
  • X-Client-Id was not forwarded, so a lock on a peer-owned entity could not even be taken through an aggregating gateway.
  • The Location header of a forwarded execution was removed by the response header allowlist, so clients were not pointed at the gateway that owns the goal.
  • The orphan sweep deleted a just-restored persistent trigger before discovery had reported its entity. That also deletes the row from the store, so a persistent trigger did not survive a gateway restart under load and nothing said so.
  • A cyclic subscription accepted a resource path for a collection that is streamed whole, then discarded it and streamed everything.

Issue

Not covered by this PR and still open: #528 and #612 (timeout handling), #611 (fault stream on an aggregating gateway).

Type

  • Bug fix
  • New feature or tests
  • Breaking change
  • Documentation only

Testing

New end-to-end coverage drives two and three gateway topologies over HTTP, kills a peer mid-run, and asserts on payloads and on which gateway answered, not on status codes alone.

Measured on Jazzy:

  • gateway unit suite: 124 tests, 0 failures (ctest -j2 -LE "linter|integration")
  • gateway linters: 9 tests, 0 failures
  • integration package: 102 tests, 0 failures
  • full workspace build: 0 compiler warnings

Aggregation feature specs grew from 0 to 49 cases in test_grouping_entity_aggregation, plus new specs for aggregator-only configurations, lock identity across a hop, and persistent trigger restore before discovery.

Two integration tests in ros2_medkit_graph_provider failed once each under parallel load during verification (test_graph_provider_greenwave, test_graph_provider_stale). Both are timing assertions, both pass repeatedly in isolation, and this branch changes no production code in that package.

Checklist

  • Breaking changes are clearly described (and announced in docs / changelog if needed)
  • Tests were added or updated if needed
  • Docs were updated if behavior or public API changed

bburda added 15 commits August 20, 2026 11:42
…model

A grouping entity - a merged Area, a merged Function, a Component with
members - has no owner of its own, and its resources come from members that do.
Nothing resolved those members, so its collections were assembled from whatever
the local walk happened to find.

Members are now resolved through one walk of the declared tree, and every listed
item names the members that provide it. An item more than one member provides is
addressed `<member>:<item>`; the bare form is refused rather than run against
whichever member was walked first. An item a single member provides keeps its
bare id, which is what every current client sends.

The specification drives two gateways over HTTP rather than reasoning about one.
When a peer stops answering, what it declared in its manifest does not stop
being true. Those entities stay listed, keep the items they last reported, and
answer `504 not-responding`. What the peer only discovered at runtime disappears,
because nothing can observe that graph any more.

Reachability is reported separately from ambiguity. It belongs to the entities a
request can be addressed to, which are Apps and Components; an Area or a Function
groups members and has none of its own. A retained item still counts towards
ambiguity, so an id two members provide stays qualified whether or not either of
them is answering.
…re of it

A Function's members, a Component's relationships and an Area's subareas are
carried only by the routes that describe them one at a time. Those requests were
allowed to fail quietly, so a fetch that lost one still reported success and
published an entity stripped of the part it could not read. Retention made that
permanent, because the declaration a peer is retained from is replaced whole on
every successful fetch.

An unreadable sub-response now fails the fetch, so the last complete declaration
stands. A 404 on a nested collection is a peer too old to offer that route, and a
peer whose health check still passes is not called unreachable for missing one
read.
The ambiguity check ran only on the bare form and counted members. A qualified id
whose member exposes the same short name at two ROS paths skipped it entirely and
ran whichever copy was walked first. On an entity that exposes its own operations
the count came back as empty strings, so the refusal named no member and offered
a form that entity cannot parse.

The check now counts what the id actually resolves to, using the predicate the
resolver walks. When the copies belong to different members the remedy is to name
one; when they belong to a single member the answer says so and reports the ROS
paths that collided.
…at owns the member

An aggregating entity listed resources belonging to members on other gateways and
then served every request for them locally, looking for a peer's topic or service
on a graph where it does not exist. A read answered "not provided by member" and
an execution answered "service not available" while the member and its gateway
were healthy.

Ownership is settled once, after the id is resolved. An unreachable member is
answered before anything is forwarded, and a member this gateway owns is served
here without a hop. Data, operations and configurations share the one dispatch
point. The forward addresses the member's own route rather than replaying the
incoming path, so the SSRF guard applies to the path actually sent.
…ges the lock

A lock is held against a client id and every later request is judged against it.
The id was not forwarded, so a request that crossed a gateway boundary arrived
anonymous and the peer refused to record a lock naming no client. Aggregation
therefore had no working locking at all on the entities only a peer owns.

The id now travels with a forwarded request. It is not a credential and is not
governed by forward_auth: it names the caller rather than granting it anything.
The rule this makes true was written down and never verified, so it has a
specification now, with a second gateway holding the lock.
…der shares the name

An operation's wire id is the last segment of its ROS path, so one provider
exposing left/calibrate and right/calibrate offers the same id twice. The member
half cannot separate those: it separates copies belonging to different members,
and these belong to one. The collection listed the duplicate and execution
refused it, which leaves the resource unreachable.

Those operations now carry the ROS path as the item half, and both forms resolve.
A short name never contains a slash, so neither form can be read as the other, and
an id that identifies one operation today is untouched. One rule decides this, and
the listing, the capability document and the resolver all read it.
…ode count

A configuration id was split into member and parameter only when the entity
counted as aggregating, and that count came from the nodes this gateway can
resolve on its own ROS graph. A peer never reports a ROS binding for its apps, so
an entity whose members all live on peers looked like it had none, and one with a
single local node beside several peer ones looked unaggregated.

The split now happens when the prefix names a member of the entity. No id that
resolves today changes. Reset on an aggregating entity also stops reporting plain
success while leaving peer-owned members untouched.
Listing executions resolved nothing. It joined a Component's namespace to the id
to guess a ROS path, and for an App it scanned actions alone and only for a bare
name, so a member-qualified or path-shaped id never matched and a service answered
"entity not found" for an entity that plainly exists. The route also skipped the
entity validation its siblings perform.

Reading an operation returned whichever of several matched first, while executing
the same id was refused. Both now go through one construction, and the answer
names the ids that do address them. A service answers with an empty execution
collection: it completes inside its own request and creates no execution resource.
A peer's statement that an entity cannot be reached was never read back, so a
gateway two hops from a dead leaf presented it as reachable. An App leaked the
news through is_online; a Component, which has no such field, said nothing.

A nested collection answering 504 not-responding was also read as a failed
request rather than as a statement about one member, so a single unreachable
member behind a peer aborted that peer's whole refresh and the aggregator replayed
its last pre-failure picture indefinitely. A 504 is now carried where it names an
entity and still fails the fetch where it does not.

fetch_all_peer_entities is deleted. It dropped a failed peer fetch silently and
had no caller outside its own tests.
Reading, updating and cancelling an execution resolved only goals tracked here, so
a member's execution on another gateway answered not-found through the aggregate
while the same request against the peer's own route answered normally. The
operation id is in the route, so the owning member can be resolved exactly as
listing them now does.

The forwarded Location that should have pointed clients at the owning gateway was
being removed by the response header allowlist, so this was a broken promise
rather than an inconsistency.
A restart restores what the persistent store holds at the moment the gateway is
constructed. The test started its second gateway on a fixed timer, so a slow
setup left the store empty when that gateway came up and the later cases failed
on a trigger that had never been restored.

The gateway now starts because the trigger is in the store, not because a clock
said so.
A subscription URI may name a single resource after the collection, and only data
required one. For faults, configurations and logs the path was parsed, recorded on
the subscription and then discarded by the sampler, which streams those
collections whole. A client asking for one parameter was told the subscription
existed and then handed everything, every tick.

A sampler now declares whether it honours a resource path, and a path given to one
that does not is refused with the collection named. Not declaring means not
honouring, so refusing is the direction that cannot silently answer a different
question.
… at least once

The orphan sweep treated "entity not in the discovery cache" as "entity gone", and
removing a persistent trigger deletes its row from the shared store. Restore runs
once, while the gateway is constructing, so a trigger restored on startup was
raced by the first sweep tick against a cache that had not yet heard from nodes
which were already running. Measured on a restarted gateway: restored, then
deleted 109 ms later, with nothing to bring it back and nothing said about it.

A trigger now records whether its entity has ever been observed. Only one that was
seen and is now missing is an orphan. Restore and the sweep both name what they
discarded and why.
Whether a sampler narrows its payload to one named resource is something the
gateway's own registry needs to know, and nothing else. Putting it on the plugin
context made it part of an interface plugins call through, and because the
ROS-facing context derives from that interface, any change to the base shifts the
derived slots too - so every plugin binary would have needed rebuilding for a
capability no plugin in the tree asks for.

A sampler a plugin registers is recorded as streaming its collection whole, so a
subscription naming a single resource on it is refused exactly as before.
Copilot AI lite review requested due to automatic review settings August 20, 2026 09:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates ros2_medkit_gateway’s aggregation behavior so that resource IDs and request routing are consistent across listing and execution/read/write flows in multi-gateway topologies, including retained (manifest-declared) entities when a peer becomes unreachable. It also adds/updates integration + unit tests and documentation to validate the new addressing model and related edge cases.

Changes:

  • Introduces member-qualified addressing + centralized “dispatch-to-member” forwarding so requests for member-owned resources are served by the owning gateway, with 504 not-responding for retained-but-unreachable entities.
  • Extends entity/resource metadata to carry contributor ownership (member_ids) and reachability (x-medkit.available emitted only when false), and unifies member traversal in the entity cache.
  • Adds significant test coverage (unit + integration) for trigger restore/sweep behavior, resource-path validation for cyclic subscriptions, lock identity forwarding, and multi-hop aggregation reachability; updates relevant docs.

Reviewed changes

Copilot reviewed 57 out of 58 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/ros2_medkit_integration_tests/test/features/test_triggers_restore_before_discovery.test.py New e2e test ensuring restored persistent triggers are not swept before discovery repopulates entities.
src/ros2_medkit_integration_tests/test/features/test_triggers_persistent.test.py Replaces timer-based “restart” sequencing with a gate-file handshake; improves SQLite WAL cleanup.
src/ros2_medkit_integration_tests/test/features/test_multi_collection_subscriptions.test.py Adds SSE collection assertions and validates resource-path rejection for whole-collection streams.
src/ros2_medkit_integration_tests/test/features/test_daisy_chain_aggregation.test.py Adds 2-hop “tail killed” reachability assertion via x-medkit.available: false.
src/ros2_medkit_integration_tests/test/features/test_aggregate_lock_identity.test.py New e2e test that X-Client-Id survives forwarding and lock enforcement works across a hop.
src/ros2_medkit_integration_tests/ros2_medkit_test_utils/launch_helpers.py Adds dual_calibration demo-node entry for operation-id collision scenarios.
src/ros2_medkit_integration_tests/README.md Documents the dual_calibration demo node and its purpose.
src/ros2_medkit_integration_tests/demo_nodes/dual_calibration_service.cpp New demo node providing same short-name operations at different ROS paths (services + actions).
src/ros2_medkit_integration_tests/demo_nodes/calibration_service.cpp Adds a writable parameter to exercise configurations addressing.
src/ros2_medkit_integration_tests/CMakeLists.txt Builds/installs new demo node; adjusts domain allocations for multi-gateway tests.
src/ros2_medkit_gateway/test/test_trigger_manager.cpp Adds unit tests pinning restored-trigger sweep semantics (before/after discovery).
src/ros2_medkit_gateway/test/test_resource_sampler_registry.cpp Adds tests for honours_resource_path declarations on samplers.
src/ros2_medkit_gateway/test/test_plugin_manager.cpp Adds test ensuring plugin samplers registered via context refuse per-item resource paths.
src/ros2_medkit_gateway/test/test_peer_client.cpp Expands peer-fetch tests: detail-route structure, availability read-back, and 504 not-responding handling.
src/ros2_medkit_gateway/test/test_operation_handlers.cpp Adds fixture coverage for qualified operation IDs, per-member resolution, and plugin-owned entity routing.
src/ros2_medkit_gateway/test/test_entity_resource_model.cpp Adds tests for unified member traversal across Areas/Functions/Components, including cycles.
src/ros2_medkit_gateway/test/test_cyclic_subscription_handlers.cpp Adds unit tests for resource-path support validation on subscriptions.
src/ros2_medkit_gateway/src/plugins/plugin_context.cpp Registers plugin samplers as whole-collection streams (honours_resource_path=false).
src/ros2_medkit_gateway/src/openapi/capability_generator.cpp Updates OpenAPI generation for member-qualified/ROS-path operation IDs (see review comment).
src/ros2_medkit_gateway/src/http/rest_server.cpp Reorders operation-item route registration to avoid greedy {operation_id} matching sub-resources.
src/ros2_medkit_gateway/src/http/handlers/handler_context.cpp Adds dispatch_to_member and local 504 not-responding response for retained remote entities.
src/ros2_medkit_gateway/src/http/handlers/discovery_handlers.cpp Emits x-medkit.available only when false for Apps/Components.
src/ros2_medkit_gateway/src/http/handlers/data_handlers.cpp Adds member-qualified data addressing/dispatch and contributor attribution (member_ids) on list.
src/ros2_medkit_gateway/src/http/handlers/cyclic_subscription_handlers.cpp Refuses per-item resource paths for samplers that don’t honour them.
src/ros2_medkit_gateway/src/http/handlers/config_handlers.cpp Routes member-qualified parameter requests to owning gateway; improves reset-all reporting for peer-owned members.
src/ros2_medkit_gateway/src/gateway_node.cpp Logs restored trigger count; updates built-in sampler registrations with honours_resource_path metadata.
src/ros2_medkit_gateway/src/core/resource_sampler.cpp Extends sampler registry entries with honours_resource_path + query method.
src/ros2_medkit_gateway/src/core/openapi/route_registry.cpp Updates regex handling for {operation_id} and improves path-param descriptions (see review comment).
src/ros2_medkit_gateway/src/core/models/thread_safe_entity_cache.cpp Adds relationship indexes + unified get_members() traversal; records per-item owners for topics/operations.
src/ros2_medkit_gateway/src/core/managers/trigger_manager.cpp Prevents sweeping restored triggers before entity discovery; returns restored count; improves restore warnings.
src/ros2_medkit_gateway/src/aggregation/aggregation_manager.cpp Retains declared entities across peer failures; replays declarations with reachability-aware availability marking; adds targeted forwarding path overload.
src/ros2_medkit_gateway/README.md Documents the addressing model, dispatch behavior, availability semantics, and refresh completeness rules.
src/ros2_medkit_gateway/include/ros2_medkit_gateway/http/handlers/handler_context.hpp Declares MemberDispatch, dispatch_to_member(), and is_entity_available().
src/ros2_medkit_gateway/include/ros2_medkit_gateway/http/handlers/cyclic_subscription_handlers.hpp Declares validate_resource_path_support().
src/ros2_medkit_gateway/include/ros2_medkit_gateway/dto/x_medkit.hpp Adds available to App/Component x-medkit DTOs (emitted only when false).
src/ros2_medkit_gateway/include/ros2_medkit_gateway/dto/operations.hpp Adds member_ids + available to operation item x-medkit payload.
src/ros2_medkit_gateway/include/ros2_medkit_gateway/dto/data.hpp Adds member_ids to data item x-medkit payload.
src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/resource_sampler.hpp Adds honours_resource_path to sampler registration API and query method.
src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/models/thread_safe_entity_cache.hpp Adds per-item owner maps and get_members() API; adds new relationship indexes.
src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/managers/trigger_manager.hpp Makes load_persistent_triggers() return restored count; adds entity_seen tracking.
src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/http/operation_item_id.hpp New helpers to decide when operation IDs use short-name vs ROS-path item-half.
src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/http/member_qualified_id.hpp New helpers for member-qualified IDs and ambiguity qualification.
src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/discovery/models/function.hpp Adds declared_source for retention decisions.
src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/discovery/models/component.hpp Adds declared_source + available for retention/reachability.
src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/discovery/models/area.hpp Adds declared_source for retention decisions.
src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/discovery/models/app.hpp Adds declared_source + available for retention/reachability.
src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/aggregation/peer_client.hpp Documents multi-request peer fetch completeness + absent-routes and not-responding semantics.
src/ros2_medkit_gateway/include/ros2_medkit_gateway/aggregation/aggregation_manager.hpp Adds reachability-aware retained declarations and forwarding overload with explicit target path.
src/ros2_medkit_gateway/design/aggregation.rst Updates design doc for member dispatch, refresh completeness, and availability propagation.
docs/tutorials/plugin-system.rst Documents that plugin-context samplers stream whole-collection and per-item URIs are refused.
docs/tutorials/graph-provider.rst Clarifies that graph subscription streams whole document; per-item paths are refused.
docs/config/aggregation.rst Documents member dispatch, X-Client-Id forwarding semantics, and refresh completeness/availability read-back.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +284 to +299
std::unordered_map<std::string, size_t> short_name_counts;
for (const auto & svc : ops.services) {
++short_name_counts[svc.name];
}
for (const auto & action : ops.actions) {
++short_name_counts[action.name];
}
const auto owner_of = [&ops](const std::string & full_path) -> std::string {
auto owner = ops.owner_by_path.find(full_path);
return owner != ops.owner_by_path.end() ? owner->second : std::string{};
};
const auto addressed_by_path = http::operation_paths_addressed_by_path(ops);
const auto operation_id_of = [&](const std::string & name, const std::string & full_path) {
return qualified_item_id(http::operation_item_half(name, full_path, addressed_by_path), short_name_counts[name],
owner_of(full_path));
};
Comment on lines +503 to +505
{"operation_id",
"The operation identifier (service or action short name), or 'member_id:operation' when more than "
"one member of the entity exposes that name"},
@bburda bburda self-assigned this Aug 20, 2026
Comment thread docs/api/rest.rst

What this means for a request:

- A bare id that names one item works, on every route. Every client that sends

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This claim doesn't hold for /data on a peer-aggregated entity: the collection fan-out lists a peer-only topic bare, but a bare id is always served locally (address_data_item leaves member_id empty, dispatch_to_member returns kServeLocally), and since peer_client never pulls topics into App, there is no owner to dispatch to - the local sample then 404s with x-medkit-ros2-topic-unavailable. Operations solve exactly this via ops.owner_by_path lookup before dispatch; data has no equivalent, so either scope this bullet to routes where it's true or give /data the same owner-based dispatch. Same sentence repeated in src/ros2_medkit_gateway/README.md:301.

// topic disappeared between shutdown and restart, or rclcpp threw
// inside TriggerTopicSubscriber). Queue the trigger for retry on
// the next refresh tick instead of leaving it active-but-silent.
unresolved_data_triggers_.push_back({trigger_id, entity_id, resource_path, std::chrono::steady_clock::now()});

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The entity_seen shield keeps a restored data trigger alive for as long as discovery takes, but both subscription-recovery paths still give up after ~60s: the subscriber's pending queue (kPendingTimeoutSec=60 in trigger_topic_subscriber.hpp, and that is the branch actually taken here - subscribe() defers instead of failing when the topic is absent, so this unresolved_data_triggers_ path only fires when rclcpp throws) and unresolved_timeout_ on this queue. If the entity shows up later than that - the exact scenario this PR fixes - the trigger survives the sweep but no subscription is ever re-created, so it reports active and never fires. Consider re-arming resolution when entity_seen flips true, or holding both retry budgets open while entity_seen is false.

paths_from_peers.insert(std::move(path));
}
}
qualify_from_declared_tree(item);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When both gateways declare an app with the same id, the merge renames the peer's copy to <peer>__<id>, but fan-out items still carry member_ids in the peer's own naming - so qualify_from_declared_tree here stamps the peer's operation with the local app's name, and the collection ends up with two items both called shared_sensor:calibrate while resolve_operation (owner_by_path match) always picks the local copy. The correctly-named retained item (<peer>__shared_sensor:calibrate) is exactly the one dropped at line 727 when the peer answers, so the healthy path shows the wrong id and the peer's copy becomes unaddressable. The peer's member_ids need translating through the collision rename (routing table) before qualifying - same pattern applies to the qualify_ambiguous_ids call in data_handlers.cpp:445.

if (!path.empty() && paths_from_peers.count(path) > 0u) {
continue; // the owner answered for itself, which is the better copy
}
item.x_medkit->available = false;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the mixed shape (function declared only on this gateway with a peer-owned member, exactly what MIXED_FUNCTION in test_aggregator_only_configurations builds), get_peer_contributors returns empty for the function, fan_out_collection early-returns, and this loop then stamps every retained peer-owned operation available:false even though the owning peer is healthy - the dto contract says false means the provider is not answering, and dispatch_to_member happily executes the same id. The fallback needs to distinguish "no peer contributes this entity, so fan-out never ran" from "fan-out ran and the owner did not answer", e.g. by checking the member's reachability instead of inferring it from paths_from_peers; a /operations assertion on MIXED_FUNCTION would pin this down, since the existing healthy-path tests only cover the both-sides-declared function.

test_peer_aggregation
test_cross_ecu_fanout
test_daisy_chain_aggregation
test_grouping_entity_aggregation

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test_grouping_entity_aggregation is missing from MEDKIT_TEST_TIMEOUT_OVERRIDES, so its 49 cases run under the feature glob's 120 s ctest kill - yet its own budgets exceed that: setUpClass alone may poll 3x60 s, z1 polls up to 60 s after killing the peer, and five cases park 15 s action goals (order=30 at 2 Hz) that later cases drain through 30-60 s waits. A slow runner gets the whole file SIGKILLed with no test name in the output, which is exactly why test_graph_provider* (smaller budgets) got 300 here - give this suite the same.

# Everything above runs against a live pair. These run after it, because
# unittest orders methods alphabetically within a class and these are the
# only ones prefixed `test_z`. The peer is killed exactly once, by the
# first of them, and the rest read the aggregator afterwards.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The z-block proves a silent peer's declared entities are retained and flipped unavailable, but nothing anywhere (this suite, daisy chain, or the unit tests) ever brings a peer back: mark_unreachable is one-way, so recovery depends entirely on a later successful fetch superseding the retained declaration, and that path is unobserved - a regression that kept replaying the retained copy would permanently poison the tree after one transient outage. Since aggregation.rst explicitly promises "when a peer recovers ... it is automatically re-included", a kill-then-restart case (the gate-started-process pattern from test_triggers_restore_before_discovery would work) asserting available flips back to true and runtime entities re-merge would pin the other half of this contract.

self.assertEqual(response.status_code, 200)
ids = [item.get('id') for item in response.json().get('items', [])]

matching = [i for i in ids if i == COLLIDING_LEAF or i.endswith(f'__{COLLIDING_LEAF}')]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

COLLIDING_LEAF never appears in the member half of any compound id: this R1 test checks /apps listing and detail 200, but nothing asserts the merged Function's member_ids carry secondary_gateway__shared_sensor, and no read ever sends shared_sensor: or the renamed form, so the __-strip in forward_request is only unit-tested, never driven E2E. That hides a real inconsistency: merge_apps renames the colliding App, but merge_functions dedups the peer's hosts list verbatim (entity_merger.cpp:113-119), so the renamed copy is never a member of the merged Function - contradicting the new comment in the configurations-reset handler that claims the merge makes this impossible. A member-qualified read through MERGED_FUNCTION on both shared_sensor copies right after this test would pin it.

bburda added 2 commits August 20, 2026 13:37
A test server was started on a thread and its port returned immediately. stop()
only interrupts a server that is already listening, so a teardown reaching it
first left the listen running with nothing to end it, and the join never
returned. The binary then hung until ctest killed it, naming whichever case
happened to be running rather than the one that lost the race.

Five files start a server this way; the rest of the suite already waits for
readiness. One of them slept instead, which is the same race with a number
attached to it.
… inside

The suite polls for up to a minute several times over: once per gateway while it
comes up, and again after a peer is killed. Those add to more than the feature
glob's default kill, so a slow runner takes the whole file down at once and
reports no test name at all.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] The aggregator refuses or 404s entity work its peers can serve

3 participants