feat(system-tests): give local-backend testnets a non-ULA IPv6 prefix - #11234
Conversation
There was a problem hiding this comment.
Pull request overview
This PR removes a long-standing divergence between Farm and the local backend by changing the local backend’s per-group IPv6 addressing scheme from ULA (fd00::/8) to the documentation range (2001:db8::/32). This prevents the orchestrator’s self-registration path from treating local-backend nodes as “cloud” instances and stalling on metadata-server discovery, enabling previously Farm-pinned system tests to run on the local backend.
Changes:
- Switch local-backend group IPv6 allocation to
2001:db8:<group>::/64and rename the exported range constant toLocalBackend::GROUP_PREFIX. - Update firewall/system-test logic to reference the centralized
GROUP_PREFIXinstead of duplicating the old ULA literal. - Drop
backend = "farm"pinning for node self-registration-related system tests now that local backend addresses no longer trigger cloud discovery.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| rs/tests/node/BUILD.bazel | Unpins ipv4_integration_test from Farm so it can run on the local backend. |
| rs/tests/consensus/orchestrator/BUILD.bazel | Unpins node registration tests from Farm. |
| rs/tests/driver/src/driver/local_backend.rs | Changes local-backend group IPv6 addressing to 2001:db8::/32 and renames the exported group range constant. |
| rs/tests/driver/src/driver/ic.rs | Updates group-wide firewall whitelisting helper to use LocalBackend::GROUP_PREFIX. |
| rs/tests/driver/src/driver/bootstrap.rs | Updates comments to reflect the new local-backend IPv6 range. |
| rs/tests/networking/firewall/firewall_priority_test.rs | Uses LocalBackend::GROUP_PREFIX instead of hardcoding the old ULA range. |
| rs/tests/networking/canister_http_socks_test.rs | Updates an explanatory comment example IPv6 address to match the new prefix family. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
Suppressed comments (1)
rs/tests/driver/src/driver/local_backend.rs:441
debug_assert!(subnet_id < 4, ...)is compiled out in release builds, so an invalidsubnet_idcould silently produce an address outside the intended bit layout. Since this is a cheap invariant check and wrong values would be logic bugs, preferassert!so it’s enforced in all builds.
debug_assert!(subnet_id < 4, "subnet-id must fit in 2 bits");
ea5de3e to
45bfc59
Compare
`ipv4_integration_test` wipes an unassigned node's keys and restarts its services so the node registers *itself* again through the orchestrator. That never completed on the local backend, which is why the test was pinned to Farm along with `node_registration_with_node_allowance_test` and `node_registration_with_max_rewardable_nodes_test`. The local backend addressed its groups out of the ULA range `fd00::/8`, and `assemble_add_node_message` (rs/orchestrator/src/registration.rs) reads a ULA or link-local address as a sign that it is running in a cloud: it goes off to `CloudType::discover()` to learn its public address before it can build the `add_node` payload. There is no metadata server at 169.254.169.254 here, so that call spun through its 120 retries and failed, the payload was never assembled, `add_node` was never called and the registry never advanced. Initial node setup was unaffected because those records are written by ic-prep on the driver side, bypassing self-registration entirely. Address the groups out of `2001:db8::/32` instead. `is_unique_local()` is then false and the cloud branch is skipped, exactly as it is on Farm, whose nodes are globally routable — so this removes a divergence rather than working around one. RFC 3849 reserves the range for documentation, so nothing can legitimately route it, which is what the backend wants: its addresses never leave its network namespace, yet they must not look local to the node software. The backend already relies on being able to squat on global addresses in that namespace — it puts the Cloudflare and Google resolver addresses on the group bridge so nodes get a resolver without any node-side configuration. The layout is unchanged otherwise: nodes on `2001:db8:<group>::/64` and the driver's management, journald-streaming and file-server addresses on subnet-ids 1, 2 and 3, so the driver keeps addressing the nodes from outside their `/64` and the firewall's accept-own-prefix rule still cannot shadow registry deny rules. The group digest shrinks from 32 to 16 bits, since a `/32` base leaves one 16-bit group before those subnet-ids. That is unobservable: each test owns a private network namespace, and bridge and TAP names keep their own 40-bit digests. `GROUP_ULA_PREFIX` becomes `GROUP_PREFIX`, and `firewall_priority_test` — which had the range written out a second time — now takes it from there. Verified on the local backend: the three tests above, plus the ones that reason about addresses rather than just using them — `firewall_correctness_test`, `firewall_priority_test`, `firewall_max_connections_test`, `canister_http_socks_test` and `rosetta_test`, the one caller of `with_group_wide_firewall_whitelist`. The rest of the `_local` suite is left to the nightly `local-system-tests` job: the machine I verified on has 31 GiB of RAM against 4 GiB per node VM, so the tests only pass run one at a time and a full sweep does not fit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…db8::/32 The documentation prefix broke `btc_get_balance_test`, the only failure in 960 tests on RBE. Bitcoin Core marks RFC 3849 invalid outright — `CNetAddr::IsValid()` returns false for `IsRFC3849()`, and `CSubNet::Match()` refuses any address that is not valid — so `-rpcallowip='::/0'` can never match the driver and every RPC comes back "Client network is not allowed RPC access". 272k rejections, and `get_balance` times out. Two more conflicts with the same range turned up while looking into it, so this was the wrong choice rather than bad luck: the registry canister's `IPV6_STRICT_CHECKS` names `2001:db8::/32` invalid (dormant only because the caller passes `strict=false`), and `canister_http_correctness_test` asserts that an outcall to `https://[2001:db8::1]` fails. The mistake was reasoning that a range reserved for documentation is safe *because* nothing routes it. That is backwards: reserved ranges are exactly the ones software special-cases. What the backend actually needs is a prefix no classifier has any reason to look twice at, so use `2a00:fb01:400::/56` — zh1, DFINITY's Zurich DC. Nothing is routed there either, since the addresses never leave the group's network namespace; the difference is that an ordinary global unicast prefix is classified as ordinary. The layout keeps its shape — nodes on subnet-id 0, the driver's management, journald and file-server addresses on 1, 2 and 3 — but a `/56` leaves one byte before the `/64` boundary, so it is now 6 bits of group digest and 2 bits of subnet-id. Collisions stay unobservable: each test owns a private network namespace, and bridge and TAP names keep their own 40-bit digests. The four `format!`s become one packing helper, since the bit layout has to agree in all four places. Reusing a real DC prefix means the group's addresses are no longer distinguishable from production ones in an allow-list. `firewall_priority_test` builds its deny list from the template's `default_rules` and then appends `GROUP_PREFIX`, which is now one of them; deduplicate, because these render into an anonymous nftables set and `nft` drops the whole ruleset if an element repeats. Verified on the local backend, one target per `bazel test` run: `btc_get_balance_test` (the regression: 0 rejections, was 272k, and `get_balance` 26s against a 600s timeout), the three tests this branch un-pins, and the ones that reason about addresses rather than just using them — `firewall_priority_test`, `firewall_correctness_test`, `firewall_max_connections_test`, `canister_http_socks_test` and `rosetta_test`. The rest of the `_local` suite is left to the nightly `local-system-tests` job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
45bfc59 to
4d4ab2c
Compare
What
Support the
ipv4_integration_test,node_registration_with_node_allowance_testandnode_registration_with_max_rewardable_nodes_teston the local backend.Why
ipv4_integration_testwipes an unassigned node's keys and restarts its services so the node registers itself again through the orchestrator. That never completed on the local backend, which is why the test was pinned to Farm along withnode_registration_with_node_allowance_testandnode_registration_with_max_rewardable_nodes_test.Why it could not run locally
The local backend addressed its groups out of the ULA range
fd00::/8, andassemble_add_node_message(rs/orchestrator/src/registration.rs) reads a ULA or link-local address as a sign that it is running in a cloud: it goes off toCloudType::discover()to learn its public address before it can build theadd_nodepayload. There is no metadata server at 169.254.169.254 in the local backend, so that call spun through its 120 retries and failed, the payload was never assembled,add_nodewas never called and the registry never advanced. Initial node setup was unaffected because those records are written by ic-prep on the driver side, bypassing self-registration entirely.How
Address the groups out of a regular IPv6 global unicast address. We choose
2a00:fb01:400::/56— DFINITY's zh1 DC.is_unique_local()is then false and the cloud branch is skipped, exactly as it is on Farm, whose nodes are globally routable. This removes a divergence rather than working around one.The layout keeps its shape — nodes on subnet-id 0, the driver's management, journald and file-server addresses on 1, 2 and 3, so the driver still reaches the nodes from outside their
/64and the firewall's accept-own-prefix rule cannot shadow registry deny rules. A/56leaves one byte before the/64boundary, so it is 6 bits of group digest and 2 bits of subnet-id. Collisions stay unobservable: each test owns a private network namespace, and bridge and TAP names keep their own 40-bit digests. The fourformat!s become one packing helper, since the bit layout has to agree in all four places.GROUP_ULA_PREFIXbecomesGROUP_PREFIX, andfirewall_priority_test— which had the range written out a second time — now takes it from there.Why not a reserved range
The first attempt used
2001:db8::/32(RFC 3849), on the reasoning that a range reserved for documentation is safe because nothing routes it. That is backwards, and it brokebtc_get_balance_test— the only failure in 960 tests on RBE. Bitcoin Core marks the documentation range invalid outright:CNetAddr::IsValid()returns false forIsRFC3849(), andCSubNet::Match()refuses any address that is not valid, so-rpcallowip='::/0'can never match the driver. 272k rejections of "Client network is not allowed RPC access", andget_balancetimed out.Two more conflicts with the same range turned up while looking into it, so it was simply the wrong choice: the registry canister's
IPV6_STRICT_CHECKSnames2001:db8::/32invalid (dormant only because the caller passesstrict=false), andcanister_http_correctness_testasserts that an outcall tohttps://[2001:db8::1]fails.Reserved ranges are exactly the ones software special-cases. What the backend needs is a prefix no classifier has reason to look twice at. Nothing is routed to zh1 from here either — the addresses never leave the group's network namespace — the difference is that an ordinary global unicast prefix is classified as ordinary. The backend already relies on being able to squat inside that namespace: it puts the Cloudflare and Google resolver addresses on the group bridge so nodes get a resolver without any node side configuration.