Skip to content

Make the metrics endpoint move discoverable - #6369

Merged
jhrozek merged 2 commits into
mainfrom
metrics-move-discoverability
Aug 19, 2026
Merged

Make the metrics endpoint move discoverable#6369
jhrozek merged 2 commits into
mainfrom
metrics-move-discoverability

Conversation

@amirejaz

Copy link
Copy Markdown
Contributor

Summary

#6296 moved /metrics off the transport port. For a deployment that had metrics
enabled, that upgrade is close to silent: Prometheus marks the target down with a
404, which says nothing about why, and the only server-side clue was an INFO line
among many.

This gives the move two signals, so diagnosing it is a lookup rather than an
investigation:

  • The dead endpoint explains itself. GET /metrics on the transport port still
    returns 404, but with a body naming the diagnostics port and pointing at the docs.
    A bare 404 is indistinguishable from a typo.
  • The startup line becomes a WARN stating the endpoint is not on the
    application port, with the resolved address. It fires only when metrics are
    enabled — exactly the deployments whose scrape configuration has to change, and
    silent for everyone else.

Neither prevents the break. They address the part of it that can be fixed without
shipping the shared-port exposure for another release. This has not shipped yet —
v0.44.0 was cut before #6296 merged — so it can land in the same release as the
move itself.

Part of #6271

Type of change

  • Bug fix
  • New feature
  • Refactoring (no behavior change)
  • Dependency update
  • Documentation
  • Other (describe): operability — makes an already-merged breaking change diagnosable

Test plan

  • Unit tests (task test)
  • E2E tests (task test-e2e)
  • Linting (task lint-fix)

TestNotServedHereHandler asserts the status, content type, nosniff, and that the
body names the path, the port, and the docs. The existing
TestMetricsNotServedWithoutHandler now also asserts the body carries the port, so
the explanatory text cannot silently regress to a bare 404.

Pre-existing failures on main in untouched files: TestValidateOCIRegistryHost,
TestParseGitReference_*, and lint findings in cmd/thv/app/upgrade.go and
pkg/vmcp/config/crd_cli_roundtrip_test.go.

Does this introduce a user-facing change?

No behaviour change — the status code and routing are unchanged. The 404 response
body is new, and the startup log moves from INFO to WARN with clearer wording.

Special notes for reviewers

Why 404 and not 410 Gone. 410 is semantically tempting — "intentionally removed"
— but this handler is also mounted on deployments that never served metrics on that
port, where claiming the resource was removed would be untrue. Happy to switch if you
disagree.

Why WARN. The house rule reserves WARN for non-fatal issues, which a migration
notice fits. The alternative is leaving it at INFO, where it competes with startup
noise and the operator misses it — the failure mode this PR exists to prevent. It is
gated on metrics being enabled, so it is not chatty.

What this deliberately does not do. It does not keep the old endpoint working, so
scrapes still break on upgrade. That is the open cutover question from #6296 and
#6368: hard move with loud signalling, versus a legacy path behind a default-on flag.
This PR is the part that is useful under either answer. @aponcedeleonch @ChrisJBurns
your call on #5956 is the precedent here, and this is my attempt to address the
"silent" half of it without shipping default-insecure.

Generated with Claude Code

#6296 moved /metrics off the transport port. For a deployment that had
enabled metrics, that upgrade is close to silent: Prometheus marks the
target down with a 404, which says nothing about why, and the only clue
on the server side was an INFO line among many.

Give the move two signals. The 404 on the transport port now carries a
body naming the diagnostics port and pointing at the docs, so curl
answers the question directly; a bare 404 is indistinguishable from a
typo. The startup line becomes a WARN stating the endpoint is not on the
application port, which fires only when metrics are enabled -- exactly
the deployments whose scrape configuration has to change.

Neither prevents the break. They shorten the diagnosis from an
investigation to a lookup, which is the part that can be fixed without
shipping the shared-port exposure for another release.

The status stays 404 rather than 410 Gone: the handler is also mounted on
deployments that never served metrics there, where claiming the resource
was removed would be untrue.

Part of #6271

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added the size/XS Extra small PR: < 100 lines changed label Aug 19, 2026
@codecov

codecov Bot commented Aug 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.06%. Comparing base (7b72d4b) to head (50be5d7).
⚠️ Report is 9 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6369      +/-   ##
==========================================
+ Coverage   73.04%   73.06%   +0.01%     
==========================================
  Files         744      745       +1     
  Lines       78611    78816     +205     
==========================================
+ Hits        57424    57583     +159     
- Misses      17169    17220      +51     
+ Partials     4018     4013       -5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jhrozek

jhrozek commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

suggestion: The new 404 response always names DefaultPort (9464), but the runner accepts telemetry.prometheusPort and Server.bind can fall back when that port is occupied. In either case, following the response sends the scraper to the wrong endpoint, so the discovery path fails for a supported configuration. Could this handler receive the resolved diagnostics address/port, or avoid naming a port and direct callers to the startup log? Please add coverage for an explicit port and the fallback path.

The 404 body hardcoded DefaultPort. The diagnostics listener honours a
configured prometheusPort and falls back to an available port when that
one is taken, so the body was wrong for both of those supported
configurations -- sending someone who is already lost to a second dead
endpoint, which is worse than saying nothing.

Name no port. Point at the startup log instead, which carries the
resolved address and is the one source that is always correct, and share
the log message as a constant so the text the body tells you to grep for
cannot drift from the text that is logged.

Passing the resolved address to the handler would be better still, but it
is not available where the handler is registered: the proxies receive
their /metrics handler as a positional constructor argument, so it would
mean a new field on the shared transport config plus a fourth parameter
on three public constructors and their call sites, for a message that is
removed once the migration finishes.

Cover the two configurations the body cannot name -- an explicit port and
the occupied-port fallback -- by asserting the reported address is the
one actually serving, and guard against a port creeping back into the
body.

Part of #6271

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added size/S Small PR: 100-299 lines changed and removed size/XS Extra small PR: < 100 lines changed labels Aug 19, 2026
@jhrozek
jhrozek merged commit ccabad6 into main Aug 19, 2026
48 checks passed
@jhrozek
jhrozek deleted the metrics-move-discoverability branch August 19, 2026 19:58
@github-actions github-actions Bot mentioned this pull request Aug 26, 2026
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/S Small PR: 100-299 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants