Release v0.45.0 - #6436
Merged
Merged
Conversation
Release-Triggered-By: reyortiz3
toolhive-release-app
Bot
requested review from
ChrisJBurns,
JAORMX,
blkt,
jerm-dro,
jhrozek,
rdimitrov,
reyortiz3 and
tgrunnagle
as code owners
August 26, 2026 12:55
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6436 +/- ##
==========================================
+ Coverage 77.85% 77.92% +0.06%
==========================================
Files 762 762
Lines 73490 73490
==========================================
+ Hits 57216 57264 +48
+ Misses 16269 16221 -48
Partials 5 5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
reyortiz3
approved these changes
Aug 26, 2026
Contributor
📝 Generated release notes for
|
| Package | Change | PR |
|---|---|---|
pkg/plugins |
MaterializationAdapter gains required EnsureRegistered(ctx, DematerializeRequest) error and Health(ctx, DematerializeRequest) error |
#6314 |
pkg/groups |
RemovePluginFromAllGroups removed (use RemovePluginFromGroup per group); AddPluginToGroup and AddSkillToGroup now return (added bool, err error) |
#6314, #6352 |
pkg/state |
Writers must implement Aborter; Close() now publishes and can fail |
#6350 |
pkg/skills |
InstallOptions.Visited removed, replaced by ExpectedCanonicalName string |
#6352 |
pkg/authserver/storage |
UpstreamTokenStorage (and transitively Storage) gains required ResolveUpstreamTokenRowID |
#6361 |
pkg/authserver/server/registration |
LoopbackClient, NewLoopbackClient, MatchRedirectURI, GetMatchingRedirectURI removed; use the free function RegisteredLoopbackRedirectURI |
#6215 |
pkg/authserver/server/registration |
ValidateDCRRequest gains an allowPrivateKeyJWT bool parameter |
#6427 |
pkg/authserver/server/tokenexchange |
ValidateTrustedIssuers and NewMultiIssuerTokenValidator gain an allowedAudiences []string parameter |
#6391 |
pkg/api/v1 |
WorkloadService.BuildFullRunConfig gains a fourth parameter |
#6214 |
cmd/thv-operator/pkg/validation |
ValidateRemoteURL(rawURL string) → ValidateRemoteURL(rawURL string, opts ValidateRemoteURLOptions) |
#6195 |
Two of these deserve concrete code:
pkg/state — writers must abort, and Close() publishes
LocalStore writers now write to a temp file and publish atomically on Close (os.Rename for GetWriter, os.Link for CreateExclusive). Three consequences: the target name does not exist until Close; Close returns real errors that must be handled; and CreateExclusive conflicts surface from Close rather than from the call itself.
Aborter is documented as required but enforced only at runtime — a Store whose writer lacks Abort() still compiles, and every abandon path then returns "state writer does not support abort" and leaks the file handle. Add a compile-time assertion.
// Before
writer, err := store.GetWriter(ctx, name)
if err != nil { return err }
defer func() {
if err := writer.Close(); err != nil { slog.Warn("failed to close writer", "error", err) }
}()
if _, err := writer.Write(data); err != nil { return err }
return nil
// After
var _ state.Aborter = (*myWriter)(nil) // catch a missing Abort() at compile time
writer, err := store.GetWriter(ctx, name)
if err != nil { return err }
closed := false
defer func() {
if !closed {
if err := state.AbortWriter(writer); err != nil {
slog.Warn("failed to abort writer", "name", name, "error", err)
}
}
}()
if _, err := writer.Write(data); err != nil { return err }
if err := writer.Close(); err != nil { // this is the publish — must be returned
closed = true
return fmt.Errorf("failed to close writer: %w", err)
}
closed = true
return nilpkg/plugins — two new adapter methods
// EnsureRegistered re-applies only the client-config registration, without
// re-extracting files. Must be idempotent.
func (a *MyAdapter) EnsureRegistered(ctx context.Context, req plugins.DematerializeRequest) error {
dir, err := a.paths(req)
if err != nil { return err }
return a.writeRegistration(req.Name, dir)
}
// Health is a presence check only — do not hash file contents into a digest.
func (a *MyAdapter) Health(ctx context.Context, req plugins.DematerializeRequest) error {
dir, err := a.paths(req)
if err != nil { return err }
if _, err := os.Stat(dir); err != nil {
return fmt.Errorf("plugin directory missing: %w", err)
}
return a.registrationPresent(req.Name, dir)
}Migration steps
- Regenerate mocks with
task genafter updating any implementation. - For
UpstreamTokenStorage, return a deterministic, non-empty, side-effect-free ID derived from your key scheme, and do no I/O — resolution happens before the singleflight joins, so a round-trip defeats the dedup. Never alias rows that are not physically the same row. - For
RegisteredLoopbackRedirectURI, note the changed return semantics: the old method returned the requested URI (dynamic port preserved); the new function returns the registered URI. Keep your own requested value if you need the port. - Move any
httperr.Code(err) == http.StatusConflictcheck from theCreateExclusivecall site to theClose()call site.
🔄 Deprecations
/metricson the transport port is deprecated in favour of the dedicated diagnostics listener (default port9464) — the transport-port copy still serves by default in v0.45.0 viametricsOnTransportPort, but that default will flip in a future release (#6296, #6370, #6371, #6368)
Deprecation detail: moving Prometheus metrics to the diagnostics port
Existing scrape configurations keep working in v0.45.0. DefaultMetricsOnTransportPort is true and the field is a *bool with no CRD default, so an unset value inherits the release default and is not pinned into stored config. Metrics are simply served in two places during the migration window.
Why the move: /metrics shared the port that serves MCP traffic, which the operator binds to 0.0.0.0 and the Service maps. Kubernetes NetworkPolicy matches on pods, ports and protocols and cannot filter on HTTP path, so while the endpoint shares the transport port there is no way to express "allow MCP traffic, deny metrics scraping". Note the move adds no authentication — the diagnostics listener carries no middleware by design, and restricting who can reach the port is what protects it.
Cutting over
# CLI
thv run --otel-metrics-on-transport-port=false …# Operator — MCPTelemetryConfig
spec:
prometheus:
metricsOnTransportPort: false
# Operator — VirtualMCPServer (inline)
spec:
config:
telemetry:
metricsOnTransportPort: false
prometheusPort: 9464 # vMCP only; MCPServer/MCPRemoteProxy are fixed at 9464- Point your scraper at the diagnostics port (
9464unless overridden) and confirm metrics arrive. - Set
metricsOnTransportPort: falseand confirm nothing else was still scraping the transport port. - Leave it unset if you want to inherit the new default automatically when the window closes; set it explicitly only to opt out of that change.
- Restrict the diagnostics port with a
NetworkPolicy— seedocs/observability.md. It binds0.0.0.0under the operator, so any pod in the cluster can reach it by pod IP until you do. Do not add it to a Service or Ingress. - Because no
containerPortor Service port is declared,ServiceMonitor/named-portPodMonitordiscovery will not find it — scrape withkubernetes_sd_configsrole: podand an explicit__address__relabel to:9464. - Expect a new startup WARN on every metrics-enabled workload naming the diagnostics address, and a
404on the transport port once you opt out (the body explains itself and names the log line to grep for).
One genuinely breaking side effect, still present: when ToolHive metrics are not served on the transport port, /metrics now returns 404 on the application listener instead of falling through to the backend. Under the transparent proxy — remote servers via thv run <url> / MCPRemoteProxy, and container sse/streamable-http workloads — a backend that exposed its own /metrics through the ToolHive proxy is no longer reachable there. Scrape such backends directly instead.
📋 Upgrade Notes
kubectl applyof the rawvirtualmcpserversCRD exceeds the 262144-byte annotation limit. This is pre-existing (it was already over at v0.44.0) rather than introduced here, but #6183 grew themcpserversandmcpremoteproxiesCRDs ~2.5× by expanding thecorev1.Affinityschema, so it is worth stating plainly.helm install/upgradeand Flux are unaffected; Argo CD with the default client-side apply is not. Usekubectl apply --server-side --force-conflicts -f <crd-dir>, or addServerSideApply=trueto the Argo CDApplication'ssyncOptions.- #6379 makes
MCPServerreadiness honest. A server whose workload StatefulSet was deleted out-of-band previously reportedReady=Truewhile clients hit a dead backend; it now reportsPending/Ready=Falseand is auto-healed by bouncing the proxy (2-minute cooldown). Only already-broken servers are affected, butkubectl wait --for=condition=Readyand Argo/Flux health checks will now correctly show them as not ready. The operator also adopts a controller owner-ref on that StatefulSet — adoption is metadata-only and causes no pod churn, but deleting anMCPServernow garbage-collects its StatefulSet even if the finalizer does not run. - #6426 turns a previously silent misconfiguration into a reconcile error — confidential or delegate clients with a plain-HTTP non-loopback issuer now fail reconciliation instead of reconciling green and then crashlooping.
- If your auth-server replicas share Redis, finish the rolling upgrade before enabling
allowPrivateKeyJwtRegistration— a v0.44.0 replica silently drops the newjwksfield when reading a row a v0.45.0 replica wrote. - All CRD changes in this release are additive or relaxing — no field was removed, renamed or retyped in any of the 14 CRDs across both served versions. Apply the updated CRDs as part of the normal operator upgrade.
🆕 New Features
- Trusted external workloads can obtain MCP access tokens with a signed RFC 7523 assertion, without registering a ToolHive OAuth client — per-issuer policy, replay-safe memory and Redis storage, and audience/subject/resource binding (#6391)
- Delegate clients can authenticate with
private_key_jwt(RFC 7523 §2.2) instead of a shared secret, generating their own keypair and registering only the public half (#6427) - Trusted issuers can authorize external-actor delegation with a CEL expression over the token's full verified claims, so role- or group-based trust no longer needs an operator to edit an allowlist for every new value (#6364)
- The MCPServer proxy Deployment can be steered onto specific nodes with
nodeSelector,tolerationsandaffinityunderresourceOverrides.proxyDeployment, so the proxy lands on the same pre-warmed pool as its server (#6183) MCPServerEntryandMCPRemoteProxygainspec.allowPrivateEndpoint, letting a Virtual MCP reach a co-located in-cluster backend in-mesh so the backend's workload-identity authorization still applies — loopback, link-local, cloud-metadata andkubernetes.default*stay blocked regardless (#6195)- Prometheus metrics are served on a dedicated diagnostics listener (default
9464) for both the proxy and Virtual MCP, so access can be governed by port with aNetworkPolicy(#6296, #6368), reachable from the CLI via--otel-metrics-on-transport-portand from the operator viaprometheus.metricsOnTransportPort(#6371) - Operators can now distinguish a rate-limit dependency failure from an enforcement outcome — the new
toolhive_rate_limit_fail_open_totalcounter and arate_limit.fail_openspan attribute record when a check failed open after a Redis error (#6282) thv skill pushsigns keylessly by default: the CLI acquires an OIDC identity token (GitHub Actions ambient token in CI, browser sign-in on a terminal) and the server exchanges it with Fulcio and records a Rekor entry (#6385, #6390); release pushes in CI are signed rather than carrying the old--no-signstopgap, and a new staging job verifies the result with stockcosign(#6402)- A skill's very first install is no longer trust-on-first-use — when it resolves through the catalog and the entry declares a
provenance, that becomes the expected signer identity (#6420) - AI plugins gain a project lock file and Sigstore verification, behind
TOOLHIVE_PLUGINS_LOCK_ENABLED=trueand inert by default: project installs pin intotoolhive.lock.yaml(#6314),thv ai-plugin syncrestores and drift-checks them (#6316),thv ai-plugin upgradeadvances a pin under review (#6317), bundles and git signatures are persisted (#6396), signatures are verified at install (#6397), and stored signatures are re-verified offline on every sync (#6399) thv client register qoderconfigures Qoder IDE for MCP server integration and skill installation (#5870)kubectl get mcpgroupshows a Proxies column fromstatus.remoteProxyCount, so a group made entirely ofMCPRemoteProxymembers no longer looks empty (#6376)
🐛 Bug Fixes
- Virtual MCP client sessions now receive
notifications/tools/list_changedand an updatedtools/listwhen a backend recovers or fails health checks, instead of serving the registration-time snapshot until reconnect — note that tools can now also disappear mid-session, since resync uses replace semantics (#6196) - Virtual MCP reuses tool embeddings across sessions instead of re-embedding the whole catalogue on every connect — measured on 140 aggregated tools, warm sessions drop from 16–19 s to sub-second with zero embedding calls (#5996)
- Virtual MCP honours the configured
operational.timeoutsfor backend calls, and no longer tears down a slow POST at the 30 s server write deadline (#6411) - Native MCP clients registered through DCR (VS Code, Claude Code) can complete the authorization flow against the embedded auth server — a portless
http://localhost/callbackregistration listening on an ephemeral port was rejected as aredirect_urimismatch, and OAuth errors now reach the client's real listener (#6215) - A refresh token can no longer be redeemed twice by callers that resolve to the same storage row under different session IDs, which could trigger IdP replay detection and revoke the credential family (#6361)
- A refresh token the IdP has rejected now surfaces as an actionable "log in again" error naming
thv llm setup, instead of an opaqueinvalid_grantthat every consumer read as a transient provider fault and retried forever (#6389) - Delegate clients can use a loopback HTTP issuer when
insecureAllowConfidentialOverLoopbackHTTPis explicitly enabled, unblocking local development; non-loopback HTTP issuers remain rejected (#6426) - Local state writes are atomic — a crash or error mid-write no longer leaves a truncated state file, and
CreateExclusive's exists-check and creation are no longer racy (#6350) kubectl rollout restarton ToolHive proxy Deployments and MCPServer workload StatefulSets is honoured instead of being reverted on the next reconcile (#6378)- A deleted MCPServer workload StatefulSet is recreated, and
Readyis no longer claimed on a proxy-only stack serving a dead backend (#6379) unix://socket URLs round-trip correctly on Windows — POSIX paths no longer gain a fourth slash, and drive-letter paths parse instead of being rejected as not absolute (#6416)thv skill syncandthv skill upgradere-read each skill under its lock before classifying or mutating, so a concurrent uninstall is not resurrected and a newer install is not overwritten (#6352)- A UTF-8 BOM on a list response no longer bypasses authz and tool-filter list filtering (#6304)
- A non-2xx list response is no longer rewritten to HTTP 200 with an unfiltered body (#6335)
- The workload REST API honours all four
runtime_configfields instead of silently droppingbuild_withandruntime_env(#6214) - Plugin uninstall now fails retryably on a group-cleanup error with the install intact, instead of succeeding with leaked group memberships (#6314)
- The
/metricsendpoint move is diagnosable: the dead endpoint returns an explanatory 404 body and the startup line is aWARNnaming the resolved diagnostics address (#6369) - Transport-port metrics are restored behind
metricsOnTransportPort, defaulting to on, so no existing scrape configuration breaks on upgrade (#6370)
🧹 Misc
- Skill artifact signing switched to
toolhive-core'scontainer/signer, deleting the local duplicate that only ever supported key-pair signing (#6383) - Fixed a flaky
close of closed channelpanic in the vMCP backend session tests that aborted the whole test binary and surfaced as unrelated failures (#6363) - Local
task test-e2eruns sweep workloads leaked by a Ginkgo timeout-kill, which had been exhausting the Docker network address pool (#6367) - The vMCP dual-era e2e specs run under the spec-required
Accept: application/json, text/event-streamheader (#6123) - Pinned
golangci-lintto v2.12.2 to avoid an upstreamnilnessanalyzer panic that was failing CI onmain(#6393) - The
GO-2026-5932openpgp suppression now names a checkable removal trigger and records why the dependency cannot be fixed locally (#6286) - Five documented paths now point at the files they were renamed to (#6388)
📦 Dependencies
| Module | Version |
|---|---|
github.com/moby/go-archive |
v0.3.0 |
github.com/stacklok/toolhive-catalog |
v0.20260824.0 |
anthropics/claude-code-action |
v1.0.205 |
Also bumped as part of feature work: github.com/stacklok/toolhive-core to v0.0.41 (#6383) and v0.0.42 (#6420) — the latter migrated cel to the renamed cel.dev/cel-go module.
👋 Welcome to our newest contributors: @TANTIOPE, @haaaashimi, @RaviTharuma, @premctl, @melbinjp, @christensenjairus, @talshechanovitz 🎉
Full commit log
What's Changed
- fix(authz): commit recorded status before flushing in ResponseFilteri… by @Yanhaoxi in fix(authz): commit recorded status before flushing in ResponseFilteri… #6335
- Strip leading UTF-8 BOM before filtering list responses by @Yanhaoxi in Strip leading UTF-8 BOM before filtering list responses #6304
- Guard test backend ready channel against double close by @jhrozek in Guard test backend ready channel against double close #6363
- Serve Prometheus metrics on a separate diagnostics listener by @amirejaz in Serve Prometheus metrics on a separate diagnostics listener #6296
- Bump github.com/moby/go-archive from 0.2.0 to 0.3.0 by @dependabot[bot] in Bump github.com/moby/go-archive from 0.2.0 to 0.3.0 #6372
- Sweep leaked workloads after local e2e runs by @jhrozek in Sweep leaked workloads after local e2e runs #6367
- Update anthropics/claude-code-action action to v1.0.195 by @renovate[bot] in Update anthropics/claude-code-action action to v1.0.195 #6338
- Update module github.com/stacklok/toolhive-catalog to v0.20260817.0 by @renovate[bot] in Update module github.com/stacklok/toolhive-catalog to v0.20260817.0 #6375
- Fix atomic local state writes by @jhrozek in Fix atomic local state writes #6350
- Add CEL actor matching for trusted issuers by @jhrozek in Add CEL actor matching for trusted issuers #6364
- Deduplicate refreshes by storage row by @jhrozek in Deduplicate refreshes by storage row #6361
- Honor all runtime_config fields over the workload API by @jhrozek in Honor all runtime_config fields over the workload API #6214
- Accept localhost dynamic-port loopback redirect_uris by @jhrozek in Accept localhost dynamic-port loopback redirect_uris #6215
- Reuse tool embeddings across sessions by @TANTIOPE in Reuse tool embeddings across sessions #5996
- Record plugin installs in the project lock file by @samuv in Record plugin installs in the project lock file #6314
- Switch skill artifact signing to toolhive-core's signer by @samuv in Switch skill artifact signing to toolhive-core's signer #6383
- Name the concrete removal trigger for the openpgp exclusion by @samuv in Name the concrete removal trigger for the openpgp exclusion #6286
- Rate limiting observability (metrics and tracing) PR C by @Sanskarzz in Rate limiting observability (metrics and tracing) PR C #6282
- Add plugin lock-file sync by @samuv in Add plugin lock-file sync #6316
- Make the metrics endpoint move discoverable by @amirejaz in Make the metrics endpoint move discoverable #6369
- Add Qoder IDE as a supported MCP client by @haaaashimi in Add Qoder IDE as a supported MCP client #5870
- Serialize skill sync and upgrade under the lock by @samuv in Serialize skill sync and upgrade under the lock #6352
- Pin golangci-lint to avoid nilness panic on main by @samuv in Pin golangci-lint to avoid nilness panic on main #6393
- Add plugin lock-file upgrade by @samuv in Add plugin lock-file upgrade #6317
- Add JWT-bearer assertion grant by @jhrozek in Add JWT-bearer assertion grant #6391
- Restore transport-port metrics behind a migration switch by @amirejaz in Restore transport-port metrics behind a migration switch #6370
- Plumb an identity token through skill push for keyless signing by @samuv in Plumb an identity token through skill push for keyless signing #6385
- fix(operator): add MCPGroup Proxies printer column by @RaviTharuma in fix(operator): add MCPGroup Proxies printer column #6376
- Allow opt-in private endpoints for remote URLs by @premctl in Allow opt-in private endpoints for remote URLs #6195
- fix(operator): honor kubectl rollout restart on ToolHive workloads by @RaviTharuma in fix(operator): honor kubectl rollout restart on ToolHive workloads #6378
- fix(operator): recreate deleted MCPServer StatefulSet by @RaviTharuma in fix(operator): recreate deleted MCPServer StatefulSet #6379
- Add CLI identity-token acquisition for keyless skill push by @samuv in Add CLI identity-token acquisition for keyless skill push #6390
- Enable keyless signing for CI skill pushes by @samuv in Enable keyless signing for CI skill pushes #6402
- Report a rejected stored credential as re-login required by @aponcedeleonch in Report a rejected stored credential as re-login required #6389
- Resync session tools when backend health changes by @premctl in Resync session tools when backend health changes #6196
- Point five stale doc paths at the moved files by @melbinjp in Point five stale doc paths at the moved files #6388
- Check first skill install against catalog-declared provenance by @samuv in Check first skill install against catalog-declared provenance #6420
- Update module github.com/stacklok/toolhive-catalog to v0.20260824.0 by @renovate[bot] in Update module github.com/stacklok/toolhive-catalog to v0.20260824.0 #6422
- Fix Windows unix socket URL round-trip by @stantheman0128 in Fix Windows unix socket URL round-trip #6416
- Update anthropics/claude-code-action action to v1.0.205 by @renovate[bot] in Update anthropics/claude-code-action action to v1.0.205 #6412
- Store plugin sigstore bundles, carry git signature by @samuv in Store plugin sigstore bundles, carry git signature #6396
- Allow loopback delegate clients by @jhrozek in Allow loopback delegate clients #6426
- Honor configured vMCP backend timeouts by @christensenjairus in Honor configured vMCP backend timeouts #6411
- Serve vMCP metrics on a separate diagnostics listener by @amirejaz in Serve vMCP metrics on a separate diagnostics listener #6368
- Add private_key_jwt DCR client authentication by @jhrozek in Add private_key_jwt DCR client authentication #6427
- Verify plugin signatures at install time by @samuv in Verify plugin signatures at install time #6397
- Expose the metrics migration switch to CLI and operator by @amirejaz in Expose the metrics migration switch to CLI and operator #6371
- feat(operator): allow pod scheduling on the proxy Deployment via resourceOverrides by @talshechanovitz in feat(operator): allow pod scheduling on the proxy Deployment via resourceOverrides #6183
- Use conformant Accept in dual-era e2e by @kocaemre in Use conformant Accept in dual-era e2e #6123
- Re-verify stored plugin signatures during sync by @samuv in Re-verify stored plugin signatures during sync #6399
- Release v0.45.0 by @toolhive-release-app[bot] in Release v0.45.0 #6436
New Contributors
- @TANTIOPE made their first contribution in Reuse tool embeddings across sessions #5996
- @haaaashimi made their first contribution in Add Qoder IDE as a supported MCP client #5870
- @RaviTharuma made their first contribution in fix(operator): add MCPGroup Proxies printer column #6376
- @premctl made their first contribution in Allow opt-in private endpoints for remote URLs #6195
- @melbinjp made their first contribution in Point five stale doc paths at the moved files #6388
- @christensenjairus made their first contribution in Honor configured vMCP backend timeouts #6411
- @talshechanovitz made their first contribution in feat(operator): allow pod scheduling on the proxy Deployment via resourceOverrides #6183
Full Changelog: v0.44.0...v0.45.0
🔗 Full changelog: v0.44.0...v0.45.0
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release v0.45.0
Version Bump
minor release
Files Updated
VERSIONdeploy/charts/operator-crds/Chart.yaml(path:version)deploy/charts/operator-crds/Chart.yaml(path:appVersion)deploy/charts/operator/Chart.yaml(path:version)deploy/charts/operator/Chart.yaml(path:appVersion)deploy/charts/operator/values.yaml(path:operator.image)deploy/charts/operator/values.yaml(path:operator.toolhiveRunnerImage)deploy/charts/operator/values.yaml(path:operator.vmcpImage)Next Steps
Checklist