Skip to content

Client IP resolves to the fronting edge node, not the reader, when TS runs behind another Fastly service #1040

Description

@jevansnyc

Summary

On the Fastly adapter, the client IP is taken only from req.get_client_ip_addr(). Fastly defines that as the immediate downstream peer. When a Trusted Server Compute service runs behind another Fastly service (service chaining), that peer is the fronting service's edge node, so every request presents the POP's IP instead of the reader's.

Geo lookup and EC ID generation both read that same single value, so both are wrong together. It fails silently: the site works, ads serve, and nothing errors.

Evidence

Measured against a live TS deployment (x-ts-version: 404) from one residential client IP. Same service, same passphrase, same source IP, same request. The only difference is whether the request went straight to the Compute service or through the fronting Fastly delivery service. Publisher identity redacted.

Route x-geo-city x-geo-coordinates x-geo-region
ts.prospect-a.com (direct, 1 hop) southampton 40.89,-72.4 NY
www.prospect-a.com (chained, 2 hops) newark 40.77,-74.17 NJ

Three runs each, no variance. x-served-by names an EWR node on both, and 40.77,-74.17 is the EWR POP itself. The one-hop row is the reader's ISP geo; the two-hop row is the datacenter.

Note that the chained response carries two x-served-by entries and the direct response one, which is the extra hop showing up in the response.

Why this reaches EC and not just geo

Geo and EC consume the same value:

  • client_info_from_request sets client_ip from req.get_client_ip_addr() (crates/trusted-server-adapter-fastly/src/platform.rs:708)
  • FastlyPlatformGeo::lookup geolocates that value (crates/trusted-server-adapter-fastly/src/platform.rs:690-694)
  • EcContext reads the same services.client_info().client_ip, normalizes it, and HMACs it (crates/trusted-server-core/src/ec/mod.rs:210-212 and 275-281, crates/trusted-server-core/src/ec/generation.rs:91)

So the geo divergence above is direct evidence that the EC HMAC input diverged. Behind a fronting service, the EC hash prefix is derived from the edge node, and collapses to roughly one value per POP.

Impact

  • Cluster classification. evaluate_cluster counts KV keys sharing the 64-hex prefix to detect NAT and office networks (crates/trusted-server-core/src/ec/kv.rs:696-697). One prefix per POP means cluster_size grows toward every EC ID ever minted there, against a default cluster_trust_threshold of 10 (crates/trusted-server-core/src/settings.rs:472). Readers progressively misclassify as shared networks rather than individuals. The mechanism is working correctly on a constant input.
  • Pull-sync rate limiting. pull_rate_limit_key is pull:{source_domain}:{ec_hash} (crates/trusted-server-core/src/ec/pull_sync.rs:286), so a per-user limit becomes a single bucket shared by every reader through that POP.
  • Consent jurisdiction. Consent context takes geo as an input (crates/trusted-server-core/src/ec/mod.rs:214-222), so jurisdiction is derived from the POP location rather than the reader's. Benign for a single-country deployment, wrong for anything EU-facing.

What is not affected

This is not an identity merge. generate_ec_id appends a random 6-character suffix, so each minted EC ID is still unique, KV keys stay distinct, and returning readers are recognized from the ts-ec cookie regardless of IP. The damage is confined to anything derived from the HMAC prefix or from geo.

Root cause

Per Fastly's documentation, client.ip is always the immediate peer, and across a shield or a chain the second service sees the first. The original client IP survives in the Fastly-Client-IP header, which "doesn't change when being forwarded from one Fastly server to another." The Fastly adapter never reads that header.

Scope

Fastly adapter only. The Cloudflare adapter reads cf-connecting-ip (crates/trusted-server-adapter-cloudflare/src/platform.rs:727) and the Spin adapter has an equivalent. Fastly is the one adapter with no header path.

This is not limited to Fastly-behind-Fastly. Any deployment where TS sits behind a CDN has the same defect, which covers the publisher-fronted topologies TS documents as supported.

Fix

Deliberately not "read Fastly-Client-IP instead." That header is not protected at the Fastly edge (per Fastly: "if a client sets this header themselves, we will use it"), and it is not in SPOOFABLE_FORWARDED_HEADERS (crates/trusted-server-core/src/http_util.rs:40-45). Preferring it unconditionally would let a caller choose their own EC identity by setting a request header, which is worse than the current bug.

The fix needs a trust boundary. Tracked separately.

Reproducing

With TS deployed behind a fronting Fastly service, request the same path directly against the Compute service and through the front door from the same client, and compare x-geo-city / x-geo-coordinates. Note that responses with status 401 suppress geo headers by design (crates/trusted-server-adapter-fastly/src/middleware.rs:162-175), so the request has to reach a 200.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions