Make the metrics endpoint move discoverable - #6369
Conversation
#6296 moved /metrics off the transport port. For a deployment that had enabled metrics, that upgrade is close to silent: Prometheus marks the target down with a 404, which says nothing about why, and the only clue on the server side was an INFO line among many. Give the move two signals. The 404 on the transport port now carries a body naming the diagnostics port and pointing at the docs, so curl answers the question directly; a bare 404 is indistinguishable from a typo. The startup line becomes a WARN stating the endpoint is not on the application port, which fires only when metrics are enabled -- exactly the deployments whose scrape configuration has to change. Neither prevents the break. They shorten the diagnosis from an investigation to a lookup, which is the part that can be fixed without shipping the shared-port exposure for another release. The status stays 404 rather than 410 Gone: the handler is also mounted on deployments that never served metrics there, where claiming the resource was removed would be untrue. Part of #6271 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6369 +/- ##
==========================================
+ Coverage 73.04% 73.06% +0.01%
==========================================
Files 744 745 +1
Lines 78611 78816 +205
==========================================
+ Hits 57424 57583 +159
- Misses 17169 17220 +51
+ Partials 4018 4013 -5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
suggestion: The new 404 response always names |
The 404 body hardcoded DefaultPort. The diagnostics listener honours a configured prometheusPort and falls back to an available port when that one is taken, so the body was wrong for both of those supported configurations -- sending someone who is already lost to a second dead endpoint, which is worse than saying nothing. Name no port. Point at the startup log instead, which carries the resolved address and is the one source that is always correct, and share the log message as a constant so the text the body tells you to grep for cannot drift from the text that is logged. Passing the resolved address to the handler would be better still, but it is not available where the handler is registered: the proxies receive their /metrics handler as a positional constructor argument, so it would mean a new field on the shared transport config plus a fourth parameter on three public constructors and their call sites, for a message that is removed once the migration finishes. Cover the two configurations the body cannot name -- an explicit port and the occupied-port fallback -- by asserting the reported address is the one actually serving, and guard against a port creeping back into the body. Part of #6271 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Summary
#6296 moved
/metricsoff the transport port. For a deployment that had metricsenabled, that upgrade is close to silent: Prometheus marks the target down with a
404, which says nothing about why, and the only server-side clue was an INFO line
among many.
This gives the move two signals, so diagnosing it is a lookup rather than an
investigation:
GET /metricson the transport port stillreturns 404, but with a body naming the diagnostics port and pointing at the docs.
A bare 404 is indistinguishable from a typo.
WARNstating the endpoint is not on theapplication port, with the resolved address. It fires only when metrics are
enabled — exactly the deployments whose scrape configuration has to change, and
silent for everyone else.
Neither prevents the break. They address the part of it that can be fixed without
shipping the shared-port exposure for another release. This has not shipped yet —
v0.44.0was cut before #6296 merged — so it can land in the same release as themove itself.
Part of #6271
Type of change
Test plan
task test)task test-e2e)task lint-fix)TestNotServedHereHandlerasserts the status, content type,nosniff, and that thebody names the path, the port, and the docs. The existing
TestMetricsNotServedWithoutHandlernow also asserts the body carries the port, sothe explanatory text cannot silently regress to a bare 404.
Pre-existing failures on
mainin untouched files:TestValidateOCIRegistryHost,TestParseGitReference_*, and lint findings incmd/thv/app/upgrade.goandpkg/vmcp/config/crd_cli_roundtrip_test.go.Does this introduce a user-facing change?
No behaviour change — the status code and routing are unchanged. The 404 response
body is new, and the startup log moves from INFO to WARN with clearer wording.
Special notes for reviewers
Why 404 and not 410 Gone. 410 is semantically tempting — "intentionally removed"
— but this handler is also mounted on deployments that never served metrics on that
port, where claiming the resource was removed would be untrue. Happy to switch if you
disagree.
Why WARN. The house rule reserves WARN for non-fatal issues, which a migration
notice fits. The alternative is leaving it at INFO, where it competes with startup
noise and the operator misses it — the failure mode this PR exists to prevent. It is
gated on metrics being enabled, so it is not chatty.
What this deliberately does not do. It does not keep the old endpoint working, so
scrapes still break on upgrade. That is the open cutover question from #6296 and
#6368: hard move with loud signalling, versus a legacy path behind a default-on flag.
This PR is the part that is useful under either answer. @aponcedeleonch @ChrisJBurns —
your call on #5956 is the precedent here, and this is my attempt to address the
"silent" half of it without shipping default-insecure.
Generated with Claude Code