Skip to content

feat(system-tests): give local-backend testnets a non-ULA IPv6 prefix - #11234

Merged
basvandijk merged 2 commits into
bas/local-backend-firewall-correctnessfrom
bas/local-backend-non-ula-prefix
Aug 21, 2026
Merged

feat(system-tests): give local-backend testnets a non-ULA IPv6 prefix#11234
basvandijk merged 2 commits into
bas/local-backend-firewall-correctnessfrom
bas/local-backend-non-ula-prefix

Conversation

@basvandijk

@basvandijk basvandijk commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

What

Support the ipv4_integration_test, node_registration_with_node_allowance_test and node_registration_with_max_rewardable_nodes_test on the local backend.

Why

ipv4_integration_test wipes an unassigned node's keys and restarts its services so the node registers itself again through the orchestrator. That never completed on the local backend, which is why the test was pinned to Farm along with node_registration_with_node_allowance_test and node_registration_with_max_rewardable_nodes_test.

Why it could not run locally

The local backend addressed its groups out of the ULA range fd00::/8, and assemble_add_node_message (rs/orchestrator/src/registration.rs) reads a ULA or link-local address as a sign that it is running in a cloud: it goes off to CloudType::discover() to learn its public address before it can build the add_node payload. There is no metadata server at 169.254.169.254 in the local backend, so that call spun through its 120 retries and failed, the payload was never assembled, add_node was never called and the registry never advanced. Initial node setup was unaffected because those records are written by ic-prep on the driver side, bypassing self-registration entirely.

How

Address the groups out of a regular IPv6 global unicast address. We choose 2a00:fb01:400::/56 — DFINITY's zh1 DC. is_unique_local() is then false and the cloud branch is skipped, exactly as it is on Farm, whose nodes are globally routable. This removes a divergence rather than working around one.

The layout keeps its shape — nodes on subnet-id 0, the driver's management, journald and file-server addresses on 1, 2 and 3, so the driver still reaches the nodes from outside their /64 and the firewall's accept-own-prefix rule cannot shadow registry deny rules. A /56 leaves one byte before the /64 boundary, so it is 6 bits of group digest and 2 bits of subnet-id. Collisions stay unobservable: each test owns a private network namespace, and bridge and TAP names keep their own 40-bit digests. The four format!s become one packing helper, since the bit layout has to agree in all four places.

GROUP_ULA_PREFIX becomes GROUP_PREFIX, and firewall_priority_test — which had the range written out a second time — now takes it from there.

Why not a reserved range

The first attempt used 2001:db8::/32 (RFC 3849), on the reasoning that a range reserved for documentation is safe because nothing routes it. That is backwards, and it broke btc_get_balance_test — the only failure in 960 tests on RBE. Bitcoin Core marks the documentation range invalid outright: CNetAddr::IsValid() returns false for IsRFC3849(), and CSubNet::Match() refuses any address that is not valid, so -rpcallowip='::/0' can never match the driver. 272k rejections of "Client network is not allowed RPC access", and get_balance timed out.

Two more conflicts with the same range turned up while looking into it, so it was simply the wrong choice: the registry canister's IPV6_STRICT_CHECKS names 2001:db8::/32 invalid (dormant only because the caller passes strict=false), and canister_http_correctness_test asserts that an outcall to https://[2001:db8::1] fails.

Reserved ranges are exactly the ones software special-cases. What the backend needs is a prefix no classifier has reason to look twice at. Nothing is routed to zh1 from here either — the addresses never leave the group's network namespace — the difference is that an ordinary global unicast prefix is classified as ordinary. The backend already relies on being able to squat inside that namespace: it puts the Cloudflare and Google resolver addresses on the group bridge so nodes get a resolver without any node side configuration.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR removes a long-standing divergence between Farm and the local backend by changing the local backend’s per-group IPv6 addressing scheme from ULA (fd00::/8) to the documentation range (2001:db8::/32). This prevents the orchestrator’s self-registration path from treating local-backend nodes as “cloud” instances and stalling on metadata-server discovery, enabling previously Farm-pinned system tests to run on the local backend.

Changes:

  • Switch local-backend group IPv6 allocation to 2001:db8:<group>::/64 and rename the exported range constant to LocalBackend::GROUP_PREFIX.
  • Update firewall/system-test logic to reference the centralized GROUP_PREFIX instead of duplicating the old ULA literal.
  • Drop backend = "farm" pinning for node self-registration-related system tests now that local backend addresses no longer trigger cloud discovery.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file
File Description
rs/tests/node/BUILD.bazel Unpins ipv4_integration_test from Farm so it can run on the local backend.
rs/tests/consensus/orchestrator/BUILD.bazel Unpins node registration tests from Farm.
rs/tests/driver/src/driver/local_backend.rs Changes local-backend group IPv6 addressing to 2001:db8::/32 and renames the exported group range constant.
rs/tests/driver/src/driver/ic.rs Updates group-wide firewall whitelisting helper to use LocalBackend::GROUP_PREFIX.
rs/tests/driver/src/driver/bootstrap.rs Updates comments to reflect the new local-backend IPv6 range.
rs/tests/networking/firewall/firewall_priority_test.rs Uses LocalBackend::GROUP_PREFIX instead of hardcoding the old ULA range.
rs/tests/networking/canister_http_socks_test.rs Updates an explanatory comment example IPv6 address to match the new prefix family.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@basvandijk basvandijk changed the title feat(system-tests): give local-backend groups a non-ULA IPv6 prefix feat(system-tests): give local-backend testnets a non-ULA IPv6 prefix Aug 20, 2026
@basvandijk basvandijk added the CI_ALL_BAZEL_TARGETS Runs all bazel targets label Aug 20, 2026
@basvandijk
basvandijk requested a lite review from Copilot August 20, 2026 14:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (1)

rs/tests/driver/src/driver/local_backend.rs:441

  • debug_assert!(subnet_id < 4, ...) is compiled out in release builds, so an invalid subnet_id could silently produce an address outside the intended bit layout. Since this is a cheap invariant check and wrong values would be logic bugs, prefer assert! so it’s enforced in all builds.
        debug_assert!(subnet_id < 4, "subnet-id must fit in 2 bits");

@basvandijk
basvandijk force-pushed the bas/local-backend-non-ula-prefix branch from ea5de3e to 45bfc59 Compare August 21, 2026 10:39
basvandijk and others added 2 commits August 21, 2026 14:32
`ipv4_integration_test` wipes an unassigned node's keys and restarts its
services so the node registers *itself* again through the orchestrator. That
never completed on the local backend, which is why the test was pinned to Farm
along with `node_registration_with_node_allowance_test` and
`node_registration_with_max_rewardable_nodes_test`.

The local backend addressed its groups out of the ULA range `fd00::/8`, and
`assemble_add_node_message` (rs/orchestrator/src/registration.rs) reads a ULA or
link-local address as a sign that it is running in a cloud: it goes off to
`CloudType::discover()` to learn its public address before it can build the
`add_node` payload. There is no metadata server at 169.254.169.254 here, so
that call spun through its 120 retries and failed, the payload was never
assembled, `add_node` was never called and the registry never advanced. Initial
node setup was unaffected because those records are written by ic-prep on the
driver side, bypassing self-registration entirely.

Address the groups out of `2001:db8::/32` instead. `is_unique_local()` is then
false and the cloud branch is skipped, exactly as it is on Farm, whose nodes are
globally routable — so this removes a divergence rather than working around one.
RFC 3849 reserves the range for documentation, so nothing can legitimately route
it, which is what the backend wants: its addresses never leave its network
namespace, yet they must not look local to the node software. The backend
already relies on being able to squat on global addresses in that namespace —
it puts the Cloudflare and Google resolver addresses on the group bridge so
nodes get a resolver without any node-side configuration.

The layout is unchanged otherwise: nodes on `2001:db8:<group>::/64` and the
driver's management, journald-streaming and file-server addresses on subnet-ids
1, 2 and 3, so the driver keeps addressing the nodes from outside their `/64`
and the firewall's accept-own-prefix rule still cannot shadow registry deny
rules. The group digest shrinks from 32 to 16 bits, since a `/32` base leaves
one 16-bit group before those subnet-ids. That is unobservable: each test owns
a private network namespace, and bridge and TAP names keep their own 40-bit
digests.

`GROUP_ULA_PREFIX` becomes `GROUP_PREFIX`, and `firewall_priority_test` — which
had the range written out a second time — now takes it from there.

Verified on the local backend: the three tests above, plus the ones that reason
about addresses rather than just using them — `firewall_correctness_test`,
`firewall_priority_test`, `firewall_max_connections_test`,
`canister_http_socks_test` and `rosetta_test`, the one caller of
`with_group_wide_firewall_whitelist`. The rest of the `_local` suite is left to
the nightly `local-system-tests` job: the machine I verified on has 31 GiB of
RAM against 4 GiB per node VM, so the tests only pass run one at a time and a
full sweep does not fit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…db8::/32

The documentation prefix broke `btc_get_balance_test`, the only failure in 960
tests on RBE. Bitcoin Core marks RFC 3849 invalid outright — `CNetAddr::IsValid()`
returns false for `IsRFC3849()`, and `CSubNet::Match()` refuses any address that
is not valid — so `-rpcallowip='::/0'` can never match the driver and every RPC
comes back "Client network is not allowed RPC access". 272k rejections, and
`get_balance` times out.

Two more conflicts with the same range turned up while looking into it, so this
was the wrong choice rather than bad luck: the registry canister's
`IPV6_STRICT_CHECKS` names `2001:db8::/32` invalid (dormant only because the
caller passes `strict=false`), and `canister_http_correctness_test` asserts that
an outcall to `https://[2001:db8::1]` fails.

The mistake was reasoning that a range reserved for documentation is safe
*because* nothing routes it. That is backwards: reserved ranges are exactly the
ones software special-cases. What the backend actually needs is a prefix no
classifier has any reason to look twice at, so use `2a00:fb01:400::/56` — zh1,
DFINITY's Zurich DC. Nothing is routed there either, since the addresses never
leave the group's network namespace; the difference is that an ordinary global
unicast prefix is classified as ordinary.

The layout keeps its shape — nodes on subnet-id 0, the driver's management,
journald and file-server addresses on 1, 2 and 3 — but a `/56` leaves one byte
before the `/64` boundary, so it is now 6 bits of group digest and 2 bits of
subnet-id. Collisions stay unobservable: each test owns a private network
namespace, and bridge and TAP names keep their own 40-bit digests. The four
`format!`s become one packing helper, since the bit layout has to agree in all
four places.

Reusing a real DC prefix means the group's addresses are no longer distinguishable
from production ones in an allow-list. `firewall_priority_test` builds its deny
list from the template's `default_rules` and then appends `GROUP_PREFIX`, which is
now one of them; deduplicate, because these render into an anonymous nftables set
and `nft` drops the whole ruleset if an element repeats.

Verified on the local backend, one target per `bazel test` run:
`btc_get_balance_test` (the regression: 0 rejections, was 272k, and `get_balance`
26s against a 600s timeout), the three tests this branch un-pins, and the ones
that reason about addresses rather than just using them — `firewall_priority_test`,
`firewall_correctness_test`, `firewall_max_connections_test`,
`canister_http_socks_test` and `rosetta_test`. The rest of the `_local` suite is
left to the nightly `local-system-tests` job.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@basvandijk
basvandijk force-pushed the bas/local-backend-non-ula-prefix branch from 45bfc59 to 4d4ab2c Compare August 21, 2026 12:32
@basvandijk basvandijk removed the CI_ALL_BAZEL_TARGETS Runs all bazel targets label Aug 21, 2026
@basvandijk
basvandijk added this pull request to the merge queue Aug 21, 2026
Merged via the queue into master with commit 31e0cfa Aug 21, 2026
49 of 77 checks passed
@basvandijk
basvandijk deleted the bas/local-backend-non-ula-prefix branch August 21, 2026 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants