Update stacklok/toolhive to v0.45.0

.github/upstream-projects.yaml

@@ -44,7 +44,7 @@ projects:
 
   - id: toolhive
     repo: stacklok/toolhive
-    version: v0.44.0
+    version: v0.45.0
     # toolhive is a monorepo covering the CLI, the Kubernetes
     # operator, and the vMCP gateway. It also introduces cross-
     # cutting features that land in concepts/, integrations/,

docs/toolhive/concepts/observability.mdx

@@ -231,8 +231,8 @@ For details, see the
 
 When [rate limiting](../guides-k8s/rate-limiting.mdx) is configured on an
 MCPServer or VirtualMCPServer, ToolHive emits metrics for bucket decisions,
-Redis errors, and Redis Lua check latency so you can watch for rejections and
-detect fail-open periods when Redis is unreachable. See
+Redis errors, fail-open events, and Redis Lua check latency so you can watch for
+rejections and detect periods when enforcement had to fail open. See
 [Observe rate limit activity](../guides-k8s/rate-limiting.mdx#observe-rate-limit-activity)
 for metric names, attributes, and example queries.
 
@@ -273,6 +273,16 @@ ToolHive can expose Prometheus-style metrics at a `/metrics` endpoint, enabling:
 - **Service discovery** in Kubernetes environments
 - **Integration** with existing Prometheus-based monitoring stacks
 
+The Prometheus endpoint is served on a dedicated diagnostics listener (port
+`9464` by default) so scraper access can be governed by port with a firewall or
+Kubernetes `NetworkPolicy`. For compatibility, `/metrics` is also served on the
+MCP transport port; that duplicate can be disabled per workload with
+`--otel-metrics-on-transport-port=false` on the CLI or
+`spec.prometheus.metricsOnTransportPort: false` on the `MCPTelemetryConfig`, and
+the default is expected to flip to off in a future release. The diagnostics
+listener carries no authentication middleware by design; restrict access at the
+network layer.
+
 ### Dual export
 
 Both OTLP and Prometheus can be enabled simultaneously, allowing you to:

docs/toolhive/guides-cli/api-server.mdx

@@ -26,6 +26,20 @@ instances.
 
 :::
 
+## Send JSON request bodies
+
+For `POST`, `PUT`, `PATCH`, and `DELETE` requests with a body, set
+`Content-Type: application/json`. Otherwise, the API returns
+`415 Unsupported Media Type`.
+
+For example, add the header when using `curl -d`:
+
+```bash
+curl -X POST http://127.0.0.1:8080/api/v1beta/workloads \
+  -H 'Content-Type: application/json' \
+  -d '{"name":"fetch","image":"ghcr.io/example/fetch:latest"}'
+```
+
 ## Start the API server
 
 To start the API server, use the following command:

docs/toolhive/guides-cli/skills-management.mdx

@@ -244,6 +244,19 @@ Missing or drifted skills are reinstalled at their pinned digest. Sync prompts
 for confirmation before installing; pass `--yes` in non-interactive environments
 such as CI.
 
+By default, `thv skill sync` targets **every** skill-supporting client installed
+on the machine, so a lock entry is only current when the skill is present in
+each client's directory. Pass `--clients` to constrain sync to a specific set:
+
+```bash
+thv skill sync --project-root . --clients claude-code,cursor
+```
+
+This is also the safest option in CI, where an unexpected client (for example, a
+newly supported IDE that appears in a `thv` upgrade) would otherwise be treated
+as drift and materialize the skill into a directory the project doesn't check
+in.
+
 Use `--check` to report drift without installing or writing anything. This is
 useful as a CI gate:
 
@@ -382,19 +395,46 @@ After building, push the artifact to a remote OCI registry:
 thv skill push ghcr.io/my-org/skills/my-skill:v1.0.0
 ```
 
-`thv skill push` signs the pushed artifact by default. Pass `--key` to use a
-cosign private key on disk, or `--no-sign` to push without signing:
+`thv skill push` signs the pushed artifact. Pick one of three signing modes:
+
+- **Keyless (default when no flag is set)**: ToolHive acquires a short-lived
+  Sigstore identity token and records the signature in the public transparency
+  log. In GitHub Actions with `id-token: write` permission, the ambient OIDC
+  token is used automatically; on an interactive terminal, ToolHive prompts for
+  browser sign-in; in any other environment, the push fails with an actionable
+  error before publishing anything.
+- **Key-pair with `--key`**: sign with a cosign private key on disk. Set
+  `COSIGN_PASSWORD` in the `thv serve` environment to decrypt an encrypted key.
+- **Explicitly unsigned with `--no-sign`**: publish without any signature.
+
+The three signing inputs are mutually exclusive. Combining `--key`,
+`--identity-token`, and `--no-sign` returns a `400` from the API.
 
 ```bash
-# Sign with a cosign key on disk. Set COSIGN_PASSWORD in the thv serve
-# environment to decrypt an encrypted key.
+# Keyless (recommended in CI with id-token: write)
+thv skill push ghcr.io/my-org/skills/my-skill:v1.0.0
+
+# Sign with a cosign key on disk.
 thv skill push ghcr.io/my-org/skills/my-skill:v1.0.0 \
   --key cosign.key
 
-# Push without a signature
+# Supply a pre-acquired OIDC identity token (advanced).
+thv skill push ghcr.io/my-org/skills/my-skill:v1.0.0 \
+  --identity-token "$IDENTITY_TOKEN"
+
+# Push without a signature.
 thv skill push ghcr.io/my-org/skills/my-skill:v1.0.0 --no-sign
 ```
 
+:::tip[In CI, pass a signing flag explicitly]
+
+A bare `thv skill push` in CI without `id-token: write` fails before the push
+completes. Pass one of `--key`, `--identity-token`, or `--no-sign` explicitly,
+or ensure the workflow grants `id-token: write` at the job level so keyless
+signing succeeds.
+
+:::
+
 Signatures let consumers verify who published a skill before installing it. When
 a consumer installs an unsigned skill project-scoped, they must pass
 `--allow-unsigned` to `thv skill install`. User-scoped installs do not enforce

docs/toolhive/guides-cli/telemetry-and-metrics.mdx

@@ -163,8 +163,19 @@ compatibility with observability tools.
 
 ### Enable Prometheus metrics
 
-You can expose Prometheus-style metrics at `/metrics` on the main transport port
-for local scraping using the `--otel-enable-prometheus-metrics-path` flag.
+You can expose Prometheus-style metrics at `/metrics` using the
+`--otel-enable-prometheus-metrics-path` flag. Metrics are served in two places:
+
+- The **diagnostics listener** on port `9464`, which is dedicated to Prometheus
+  scraping and carries no other traffic.
+- The **transport port** (the same port MCP traffic uses), retained for
+  compatibility with existing scrape configurations.
+
+Governing scraper access by port with a firewall rule or Kubernetes
+`NetworkPolicy` is only possible on the diagnostics listener, since the
+transport port also serves MCP traffic. The diagnostics listener carries no
+authentication middleware by design; restricting who can reach the port is what
+protects it.
 
 This example runs the Fetch MCP server and enables the Prometheus metrics
 endpoint:
@@ -173,18 +184,44 @@ endpoint:
 thv run --otel-enable-prometheus-metrics-path fetch
 ```
 
-To access the metrics, you can use `curl` or any Prometheus-compatible scraper.
-The metrics are available at `http://127.0.0.1:<PORT>/metrics`, where `<PORT>`
-is the port assigned to the MCP server.
+To access the metrics on the diagnostics port:
 
 ```bash
-# Get the port number assigned to the MCP server
-thv list
+curl http://127.0.0.1:9464/metrics
+```
+
+The same metrics are available on the MCP server's transport port at
+`http://127.0.0.1:<PORT>/metrics`, where `<PORT>` is the port assigned to the
+MCP server (visible in `thv list`).
 
-# Replace <PORT> with the actual port number from the output of `thv list`
-curl http://127.0.0.1:<PORT>/metrics
+#### Move metrics off the transport port
+
+To stop serving `/metrics` on the transport port and only expose it on the
+dedicated diagnostics listener, pass `--otel-metrics-on-transport-port=false`:
+
+```bash
+thv run \
+  --otel-enable-prometheus-metrics-path \
+  --otel-metrics-on-transport-port=false \
+  fetch
 ```
 
+With this flag set to `false`, the transport port returns `404` for `/metrics`
+with a body explaining where the endpoint moved to. Under the transparent proxy
+(used for `sse` and `streamable-http` container workloads and for remote-URL
+runs), a backend that exposed its own `/metrics` through ToolHive is no longer
+reachable at that path when the flag is off; scrape such backends directly
+instead.
+
+:::info[Upcoming default change]
+
+`--otel-metrics-on-transport-port` currently defaults to `true` so existing
+scrape configurations keep working. The default is planned to flip to `false` in
+a future release. To inherit the new default automatically, leave the flag
+unset. To opt out of the change, set it explicitly to `true`.
+
+:::
+
 ### Dual export
 
 You can export to both an OTLP endpoint and expose Prometheus metrics
@@ -234,6 +271,7 @@ thv run [--otel-endpoint <URL>] [--otel-service-name <NAME>] \
 | `--otel-env-vars`                       | List of environment variables to include in telemetry spans         | None           |
 | `--otel-insecure`                       | Connect using HTTP instead of HTTPS                                 | `false`        |
 | `--otel-enable-prometheus-metrics-path` | Enable `/metrics` endpoint                                          | `false`        |
+| `--otel-metrics-on-transport-port`      | Also serve `/metrics` on the MCP transport port                     | `true`         |
 | `--otel-use-legacy-attributes`          | Emit legacy attribute names alongside new OTel semantic conventions | `true`         |
 
 ### Global configuration
@@ -311,13 +349,14 @@ Prometheus configuration:
 scrape_configs:
   - job_name: 'toolhive-mcp-proxy'
     static_configs:
-      - targets: ['localhost:<MCP_SERVER_PORT>']
+      - targets: ['localhost:9464']
     scrape_interval: 15s
     metrics_path: /metrics
 ```
 
-You can add multiple MCP servers to the `targets` list. Replace
-`<MCP_SERVER_PORT>` with the port number assigned to each MCP server.
+`9464` is the dedicated diagnostics port that serves only `/metrics`. To scrape
+the transport port instead, replace `9464` with the port number assigned to each
+MCP server (from `thv list`).
 
 ### Jaeger
 
@@ -463,17 +502,19 @@ If traces aren't showing up in your backend:
 If the `/metrics` endpoint isn't reachable:
 
 1. Confirm the server was started with `--otel-enable-prometheus-metrics-path`.
-   You can verify by re-checking `thv list` and curling the URL it shows with
-   `/metrics` appended:
+   Metrics are served on the diagnostics port `9464` by default:
 
    ```bash
-   thv list
-   curl http://127.0.0.1:<PORT>/metrics
+   curl http://127.0.0.1:9464/metrics
    ```
 
-2. The `/metrics` endpoint is served on the same proxy port as the MCP server.
-   If `curl` returns connection refused, the server isn't running on that port
-   at all - check `thv list` again and look for the actual port.
+2. If you also expect `/metrics` on the transport port, confirm
+   `--otel-metrics-on-transport-port` was not set to `false`. When it is
+   `false`, the transport port returns `404` for `/metrics` with a body that
+   points at the diagnostics port.
+
+3. Look for a startup log line reporting the diagnostics address; if it did not
+   appear, the diagnostics listener never started.
 
 </details>
 

docs/toolhive/guides-k8s/mcp-server-entry.mdx

@@ -332,11 +332,39 @@ patterns. The operator rejects URLs that target:
 - **Loopback addresses**: `127.0.0.0/8`, `::1`
 - **Link-local addresses**: `169.254.0.0/16`, `fe80::/10`
 - **Cloud metadata endpoints**: `169.254.169.254` (AWS, GCP, Azure)
-- **Private network ranges**: `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`
+- **`kubernetes.default*`** hostnames
+- **Private network ranges**: `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`,
+  and IPv6 unique-local addresses
 
 If a URL fails SSRF validation, the entry's phase is set to `Failed` with a
 condition describing the rejection reason.
 
+### Allow in-cluster backends
+
+Set `spec.allowPrivateEndpoint: true` to permit private network ranges, IPv6
+unique-local addresses, and `*.cluster.local` hostnames. Use this when vMCP
+needs to reach a co-located backend inside the cluster so the backend's own
+workload-identity authorization still applies.
+
+```yaml title="in-cluster-entry.yaml"
+apiVersion: toolhive.stacklok.dev/v1beta1
+kind: MCPServerEntry
+metadata:
+  name: internal-tool
+  namespace: toolhive-system
+spec:
+  groupRef:
+    name: my-group
+  remoteUrl: https://analytics-mcp.analytics.svc.cluster.local/mcp
+  transport: streamable-http
+  # highlight-next-line
+  allowPrivateEndpoint: true
+```
+
+Loopback, link-local, cloud-metadata, and `kubernetes.default*` targets remain
+blocked regardless of `allowPrivateEndpoint`. Bare `.svc` hostnames are never
+blocked because the operator does not resolve DNS during validation.
+
 ## Next steps
 
 - [Configure a VirtualMCPServer](../guides-vmcp/configuration.mdx) to aggregate
@@ -393,8 +421,10 @@ kubectl get mcpserverentry <NAME> -n toolhive-system \
 Common causes:
 
 - **SSRF validation failure**: The `remoteUrl` targets a blocked address range
-  (loopback, link-local, private network, or cloud metadata). Use an externally
-  routable URL
+  (loopback, link-local, cloud metadata, or `kubernetes.default*`). Use an
+  externally routable URL, or, for an in-cluster backend, set
+  `spec.allowPrivateEndpoint: true` to allow private network ranges and
+  `*.cluster.local` hostnames.
 - **Missing MCPGroup**: The group referenced in `groupRef` doesn't exist. Create
   the MCPGroup first
 - **Missing MCPExternalAuthConfig**: The auth config referenced in

docs/toolhive/guides-k8s/rate-limiting.mdx

@@ -255,6 +255,7 @@ latency histogram is exported with the `_seconds` unit suffix and the standard
 | ----------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------- |
 | `toolhive_rate_limit_decisions`     | Counter   | `decision` (`allowed` or `rejected`), `scope` (`shared` or `per_user`), `operation_type` (`server` or `tool`) |
 | `toolhive_rate_limit_redis_errors`  | Counter   | `error_type` (`timeout`, `connection`, `auth`, or `other`)                                                    |
+| `toolhive_rate_limit_fail_open`     | Counter   | `error_type` (matches the `redis_errors` classification)                                                      |
 | `toolhive_rate_limit_check_latency` | Histogram | (none)                                                                                                        |
 
 Counting semantics for `toolhive_rate_limit_decisions`:
@@ -272,9 +273,11 @@ Counting semantics for `toolhive_rate_limit_decisions`:
 
 The `toolhive_rate_limit_check_latency` histogram records the duration of each
 Redis Lua call, whether the call succeeds or fails. Rate limit enforcement
-[fails open](#how-rate-limiting-works) when Redis is unreachable, so watch
-`toolhive_rate_limit_redis_errors` alongside decisions to detect fail-open
-periods.
+[fails open](#how-rate-limiting-works) when Redis is unreachable.
+`toolhive_rate_limit_fail_open` increments each time enforcement chose to allow
+a request after a Redis error, so alerting on it directly measures the
+security-visible impact rather than every Redis blip. Traces for those requests
+carry a `rate_limit.fail_open` span attribute set to `true`.
 
 Example PromQL queries against a Prometheus scrape of the server's `/metrics`
 endpoint: