iocore/net: add RFC 7250 raw public key (RPK) support for TLS hops

[maskit]

Summary

Layered-cache deployments have TLS connections between nodes under the same operator's control, where the CA/hostname-verification machinery that X.509 exists for isn't needed -- a raw public key, pinned per hop, is sufficient. RPK is negotiated alongside X.509 rather than replacing it, so a hop between nodes at different points in a rolling upgrade falls back to a normal certificate exchange rather than failing.

Adds build-time detection of OpenSSL 3.2+'s SSL_CTX_set1_server_cert_type and BoringSSL's SSL_CREDENTIAL_new_raw_public_key, and a library-agnostic SSLRPKUtils helper for loading and pinning trusted keys.

New settings:

• ssl_multicert.yaml: ssl_rpk_enabled, ssl_client_rpk_ca_name
• sni.yaml: client_rpk_enabled, server_rpk_ca

On BoringSSL, accepting an RPK client cert requires switching to SSL_CTX_set_custom_verify, which disables automatic X.509 chain verification for the whole connection, so the X.509 fallback path is reimplemented manually there on both client and server sides.

Test plan

• New unit tests (test_SSLRPKUtils.cc, test_YamlSNIConfig.cc) covering key loading/pinning and sni.yaml parsing
• New autest (tls_rpk_hop.test.py) covering RPK negotiation, X.509 fallback during a rolling upgrade, pin mismatch under ENFORCED/PERMISSIVE, and mTLS pinning via ssl_client_rpk_ca_name
• Verified on both OpenSSL 3.2+ and BoringSSL builds

[Copilot]

Pull request overview

This PR adds RFC 7250 Raw Public Key (RPK) negotiation/pinning support to ATS’s TLS hop logic (layered-cache / ATS-to-ATS use cases), with build-time capability detection for OpenSSL vs. BoringSSL and accompanying unit + AuTest coverage.

Changes:

• Add library-agnostic SSLRPKUtils for loading/pinning trusted raw public keys and integrate it into inbound mTLS and outbound next-hop verification paths.
• Extend ssl_multicert.yaml and sni.yaml parsing/config plumbing to enable RPK offering and key pinning.
• Add unit tests and a new gold test for RPK negotiation, fallback to X.509, and pin mismatch behaviors.

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/gold_tests/tls/tls_rpk_hop.test.py	New gold test exercising RPK hop negotiation, fallback, and pin mismatch behaviors.
tests/gold_tests/tls/ssl/server.pubkey.pem	Test fixture: pinned public key for RPK hop.
tests/gold_tests/tls/ssl/server.wrongpubkey.pem	Test fixture: mismatching public key for negative pin tests.
src/traffic_layout/info.cc	Expose TS_USE_RPK in traffic_layout feature output for runtime feature gating.
src/iocore/net/YamlSNIConfig.cc	Parse new client_rpk_enabled / server_rpk_ca SNI YAML keys (with no-RPK build warning).
src/iocore/net/unit_tests/test_YamlSNIConfig.cc	Unit test coverage for new SNI YAML fields.
src/iocore/net/unit_tests/test_SSLRPKUtils.cc	New unit tests for key loading and pin matching logic.
src/iocore/net/unit_tests/sni_conf_test.yaml	Add SNI YAML fixture entry for RPK parsing test.
src/iocore/net/unit_tests/rpk_single.pem	Unit test fixture: single trusted key.
src/iocore/net/unit_tests/rpk_other.pem	Unit test fixture: alternate key.
src/iocore/net/unit_tests/rpk_multi.pem	Unit test fixture: multi-key rotation set.
src/iocore/net/unit_tests/rpk_malformed.pem	Unit test fixture: malformed key file.
src/iocore/net/SSLUtils.cc	Inbound-side RPK client-auth handling; BoringSSL custom-verify path with X.509 fallback rebuild.
src/iocore/net/SSLSNIConfig.cc	Plumb outbound RPK config from parsed SNI items; probe-load pinned key files during config load.
src/iocore/net/SSLRPKUtils.h	New helper API for loading/pinning raw public keys.
src/iocore/net/SSLRPKUtils.cc	New implementation for PEM key loading and DER pin comparisons.
src/iocore/net/SSLNetVConnection.cc	Apply outbound RPK setup during TLS handshake when next hop is configured for RPK.
src/iocore/net/SSLClientUtils.cc	Outbound-side RPK verification + BoringSSL custom-verify callback for RPK/X.509 fallback.
src/iocore/net/P_SSLClientUtils.h	Declare ssl_client_setup_rpk() helper for outbound connections.
src/iocore/net/P_SSLCertLookup.h	Extend multicert settings with rpk_enabled and client_rpk_ca wiring.
src/iocore/net/CMakeLists.txt	Build/link SSLRPKUtils and its unit test.
src/config/ssl_multicert.cc	Parse/emit new ssl_rpk_enabled and ssl_client_rpk_ca_name fields (YAML + legacy).
include/tscore/ink_config.h.cmake.in	Add TS_USE_RPK and feature-detection macros for RPK-related APIs.
include/iocore/net/YamlSNIConfig.h	Add client_rpk_enabled / server_rpk_ca to the parsed config model.
include/iocore/net/SSLSNIConfig.h	Add next-hop properties for outbound RPK offering and pin file path.
include/config/ssl_multicert.h	Add multicert entry fields for RPK enabling and client RPK CA file name.
doc/admin-guide/files/ssl_multicert.yaml.en.rst	Document ssl_rpk_enabled and ssl_client_rpk_ca_name.
doc/admin-guide/files/sni.yaml.en.rst	Document client_rpk_enabled and server_rpk_ca semantics and fallback behavior.
CMakeLists.txt	Detect RPK-capable TLS APIs and define TS_USE_RPK.

Suppressed comments (1)

tests/gold_tests/tls/tls_rpk_hop.test.py:199

• This TestRun also relies on parent_rpk but doesn't start it; if parent_rpk was stopped after the prior run, this run will fail to connect upstream.

tr.Processes.Default.StartBefore(edge_badpin_permissive)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

src/iocore/net/SSLSNIConfig.cc:207

• server_rpk_ca key probing is performed unconditionally. On builds without TS_USE_RPK, this can make sni.yaml loading fail (returning false) if the pinned key file is unreadable, even though the parser explicitly warns that these fields “do not apply” and should be ignored. Gate this probe-load behind #if TS_USE_RPK so non-RPK builds only warn and continue.

  if (!item.server_rpk_ca.empty()) {
    SSLConfig::scoped_config params;
    nps.prop.server_rpk_ca_file = Layout::get()->relative_to(params->clientCACertPath, item.server_rpk_ca.data());
    // Fail the config load now rather than at handshake time if the pinned keys are unreadable.
    SSLRPKUtils::TrustedKeySet probe;

src/iocore/net/SSLClientUtils.cc:507

• ssl_client_setup_rpk() loads and parses the trusted RPK PEM file on every outbound TLS connection handshake. In layered-cache deployments with frequent connects, this introduces repeated disk I/O and PEM parsing overhead that could be avoided by caching the parsed TrustedKeySet by path (e.g., in SNIConfigParams/NextHopProperty as a shared_ptr, and attaching a non-owning pointer per-SSL).

  if (!trusted_key_file.empty()) {
    auto *trusted = new SSLRPKUtils::TrustedKeySet();
    if (!SSLRPKUtils::loadTrustedKeys(trusted_key_file.c_str(), *trusted)) {
      delete trusted;
      return false;
    }

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/iocore/net/SSLUtils.cc:2621

• SSLMultiCertConfigLoader::load_certs allocates a TrustedKeySet and attaches it via SSL_CTX_set_ex_data(), but the return value is not checked. If SSL_CTX_set_ex_data fails, the TrustedKeySet leaks and later verification will behave as if no trusted keys were configured.

    // ssl_client_rpk_ca_ex_free() releases `trusted` when ctx is freed.
    SSL_CTX_set_ex_data(ctx, ssl_client_rpk_ca_index, trusted);

tests/gold_tests/tls/tls_rpk_hop.test.py:179

• In the second TestRun, parent_rpk is not included in StillRunningAfter. AuTest will typically stop processes that are not listed, so later runs that depend on parent_rpk (pin-mismatch cases) may fail because parent_rpk was stopped after this run.

tr.StillRunningAfter = server
tr.StillRunningAfter += parent_x509
tr.StillRunningAfter += edge_fallback

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated no new comments.

Suppressed comments (3)

src/iocore/net/SSLRPKUtils.cc:99

• loadTrustedKeys() accepts a PUBLIC KEY PEM even if the DER payload has trailing bytes (d2i_PUBKEY parses a prefix and ignores the rest). In that case the stored trusted key includes the trailing bytes and will never match the peer key DER produced by i2d_PUBKEY(), causing confusing pin mismatches. Treat trailing data as a parse failure (and free the partially-parsed key).

      const unsigned char *p = data;
      // The PEM payload for a PUBLIC KEY block is already a DER SubjectPublicKeyInfo, which is
      // exactly what gets pinned -- but decode it anyway so a corrupt key is rejected at config
      // load rather than silently pinned as opaque bytes.
      pkey = d2i_PUBKEY(nullptr, &p, len);
      if (pkey == nullptr) {
        SSLError("SSLRPKUtils: failed to parse a raw public key from %s", path);

include/iocore/net/SSLSNIConfig.h:56

• The new NextHopProperty member has inconsistent spacing (std::string server_rpk_ca_file;) compared to the surrounding declarations; this is likely to trip clang-format/formatting checks.

  bool        client_rpk_enabled = false;     // offer a RFC 7250 raw public key (derived from the configured client
                                              // cert/key) alongside X.509 when connecting to this next hop
  std::string             server_rpk_ca_file; // full path to the PEM of trusted next-hop raw public keys to pin against

src/iocore/net/P_SSLCertLookup.h:73

• The new members in SSLMultiCertConfigParams have inconsistent alignment/spacing (notably ats_scoped_str client_rpk_ca;) and the trailing comment line is overly long; aligning/wrapping here will match the style used throughout this header and avoid format-check churn.

  bool           rpk_enabled = false;    ///< Offer RFC 7250 raw public keys (using this entry's existing cert/key as the
                                         ///< identity) alongside X.509, negotiated per connection
  ats_scoped_str       client_rpk_ca; ///< Trusted client raw public keys (PEM, may contain more than one) for inbound mTLS pinning
  SSLCertContextOption opt;           ///< SSLCertContext special handling option

[bryancall]

I read the full diff across all 29 files and traced the inbound and outbound paths through the surrounding code. This is careful work, and several parts are better than they had to be:

• RPK is negotiated alongside X.509 rather than replacing it, so a hop mid-rolling-upgrade degrades to a certificate exchange instead of failing, and the autest covers that fallback explicitly.
• The comments explain the non-obvious library behavior rather than restating the code: why signature_ok/preverify_ok are always 0 on the OpenSSL RPK path, why PEM_read_bio is used instead of PEM_read_bio_PUBKEY for unambiguous end-of-file detection, and why the outbound pin set is SSL-level ex_data rather than SSL_CTX-level, since getCTX caches contexts by certificate, key and CA and a context-level pin would bleed across next hops.
• Fail-closed defaults throughout, with a unit test asserting each: an empty trust set matches nothing, and a malformed pin file fails config load rather than yielding an accept-nothing set.
• Both new verify paths preserve the existing contract that the TS_EVENT_SSL_VERIFY_* hook always runs and can only add rejection.

I am requesting changes on one item.

Blocking: inbound RPK settings are inert on any non-default multicert entry

src/iocore/net/SSLUtils.cc:2583-2645

ssl_rpk_enabled and ssl_client_rpk_ca_name are applied to the per-entry SSL_CTX in load_certs(). ATS does not hand that context to the connection. SSLNetVConnection.cc:1178 unconditionally creates every inbound SSL from lookup->defaultContext(), which is the dest_ip: "*" entry per SSLUtils.cc:1734 and 1793, and TLSCertSwitchSupport.cc:96 only later calls SSL_set_SSL_CTX() for the matched entry.

I checked this against the OpenSSL 3.2 source rather than inferring it. In ssl/ssl_lib.c, ossl_ssl_connection_reset() (the SSL_new path, lines 892-905) copies ctx->client_cert_type and ctx->server_cert_type into the SSL_CONNECTION, and SSL_set_SSL_CTX() contains no cert_type handling at all; it re-duplicates only ctx->cert and the session id context. So SSL_CTX_set1_server_cert_type and SSL_CTX_set1_client_cert_type applied to a non-default entry never reach the connection, and the cert_type extension response is computed from the default context's empty list before the SNI callback even runs. The same applies to the BoringSSL custom verify callback, which is copied into ssl->config at SSL_new.

The practical result: an operator who scopes RPK to specific hops, which is the natural layered-cache configuration, silently gets plain X.509 with none of the pinning they configured, and no warning that the setting did nothing. Conversely, setting these keys on the "*" entry applies RPK to every inbound connection. The autest cannot catch this because make_parent() emits a single dest_ip: "*" entry, which is also the default context.

That the outbound path in this same PR correctly uses the per-connection SSL_set1_server_cert_type is, I think, the tell that this is an oversight rather than a design decision.

Two ways out: apply the settings per-SSL in the certificate-selection callback, or document explicitly that these keys only work on the "*" entry. Either way a multi-entry autest would lock it in.

Also worth addressing

src/iocore/net/SSLUtils.cc:1626 On BoringSSL the custom verify callback is installed only inside if (params->clientCertLevel != 0). The entire body of _setup_client_cert_verification() (lines 1419-1449), including the new SSL_CTX_set_custom_verify() branch, sits under that guard, so at the default level 0 the function returns without touching the context. load_certs() has already called SSL_CTX_set1_accepted_peer_cert_types({rpk, x509}) independently, so the server advertises RPK acceptance. A deployment doing per-SNI mTLS with verify_client: STRICT gets the classic ssl_verify_client_callback installed via setClientCertLevel(), and BoringSSL's own header is explicit that without a custom verification callback raw public keys are rejected by default. So an RPK-offering client on that connection fails the handshake rather than being pinned. No CI job builds BoringSSL, so nothing here catches it.

src/iocore/net/SSLClientUtils.cc:506 ssl_client_setup_rpk() opens and PEM-parses the next hop's pin file on every outbound handshake, on the net thread. That is a blocking open and read plus a DER decode plus a heap allocation in the connection-setup path, and SSLSNIConfig.cc:199-205 already loads and validates the same file at config load and throws the result away. There is a robustness half too: if the file is later removed or made unreadable, loadTrustedKeys fails, ssl_client_setup_rpk returns false, and _sslStartHandShake returns EVENT_ERROR, so every outbound connection to that hop fails with no config reload having occurred. Keeping the parsed set on NextHopProperty as a shared_ptr and attaching a non-owning pointer per SSL removes both problems.

Smaller items

• src/iocore/net/SSLRPKUtils.cc:98 loadTrustedKeys() decodes each block with d2i_PUBKEY to validate it, then pins the original PEM payload bytes rather than the re-encoded SubjectPublicKeyInfo. d2i_PUBKEY does not reject trailing bytes, so a payload with an appended encoding artifact loads cleanly and can never match, and the only diagnostic points at the wrong thing. Storing i2d_PUBKEY(pkey) is a smaller change than the current code and makes the stored form canonical by construction. Only reachable with a non-canonically encoded pin file, so low severity.
• doc/admin-guide/files/ssl_multicert.yaml.en.rst:152 The documentation says the client certificate level's fail-closed-or-log choice applies to inbound RPK pin failures. ssl_verify_client_callback()'s RPK branch returns pin_ok unconditionally without consulting clientCertLevel, and your own autest comment on scenario 6 says the opposite of the documentation. An operator reading this expects to roll out in log-only mode first.
• src/iocore/net/SSLSNIConfig.cc:199 On a build with TS_USE_RPK off, YamlSNIConfig.cc:474 warns that the key is ignored, but load_certs_if_client_cert_specified still probe-loads the path and returns false if it is missing, failing the whole sni.yaml. A key the parser said it was ignoring can take down every unrelated SNI action.
• src/iocore/net/SSLUtils.cc:2625 The SSL_CTX_set_ex_data() return value is not checked when attaching the inbound trusted-key set, unlike the outbound counterpart which does check. On failure the set leaks and RPK acceptance is still enabled, so the failure presents as a pin mismatch for a configuration that is actually fine.
• src/iocore/net/SSLClientUtils.cc:411 SSL_SESSION_get0_peer_rpk() is guarded by the union macro TS_USE_RPK while every other RPK call is behind a library-specific guard. I confirmed the symbol in the OpenSSL headers but could not confirm it in BoringSSL. If it is not there, that build breaks and no CI job would notice. You state both builds were verified, so a one-line confirmation is enough for me.
• The inbound BoringSSL X.509 fallback in ssl_custom_verify_client_callback() (SSLUtils.cc:249-380), the hand-rolled chain reconstruction and manual X509_verify_cert, is the riskiest code in the PR and is exercised by no test and compiled by no CI job. I am not asking you to solve the BoringSSL coverage problem in this PR, but it is worth stating in the description that this path ships unverified.