feat(plugins): add livekit-plugins-floe (metered inference + budget guard)

[achris7]

What

Adds livekit-plugins-floe — a plugin that routes an agent's LLM through Floe so spend is metered and can be guarded against a budget, in two modes:

• Keyless gateway — Floe holds the upstream provider keys and bills your Floe balance (FLOE_API_KEY only).
• BYOK — bring your own upstream provider key, forwarded via X-Floe-Provider-Key; Floe meters spend against your budget (FLOE_PROVIDER_KEY).

floe.LLM subclasses the openai plugin's LLM (Floe's endpoint is OpenAI-compatible), so it drops into an AgentSession unchanged:

from livekit.plugins import floe
session = AgentSession(llm=floe.LLM(model="openai/gpt-4o"), stt=..., tts=...)

Usage reconciliation

The plugin also ships FloeUsageReconciler, which subscribes to session_usage_updated and reconciles LiveKit's per-model token accounting against Floe's cost map — LiveKit metrics on one side, Floe pricing on the other, per served model. A divergence between the local estimate and Floe's billed amount is the signal worth acting on. The README documents an OpenTelemetry export path for the same numbers (cost observability, not enforcement — the budget guard stays in floe-guard).

Scope / notes

• STT/TTS are intentionally out of scope for now — this is LLM-only.
• Depends on floe-guard (PyPI) for the cost map + pricing. It ships type hints but no py.typed marker yet, so I added a [[tool.mypy.overrides]] for floe_guard.* mirroring the existing boto3/mcp overrides. Happy to switch to requiring a py.typed-shipping floe-guard release instead if you'd prefer no root-config change.
• Follows the livekit-plugins-groq layout (PEP 420 namespace, hatchling, Plugin.register_plugin); auto-included by the livekit-plugins/* uv workspace glob.

Testing

• ruff check + ruff format --check: clean.
• mypy --strict (root config): Success: no issues found.
• pytest: exports + plugin registration pass; reconciler pricing verified against a synthetic AgentSessionUsage (gpt-4o 1000in/500out → $0.0075; unpriceable models fail closed).

Draft while I complete the CLA. Feedback on the floe-guard typing approach and on whether a metering/guard plugin fits the plugin taxonomy is very welcome.

[achris7]

@CodeRabbit review

[achris7]

Thanks for the reviews — addressed in the latest push:

• BYOK client lifecycle + timeout/retries (Devin, Copilot): dropped the hand-built openai.AsyncClient and now inject X-Floe-Provider-Key via the parent LLM's extra_headers. The parent owns the client (so it's closed on aclose, no leaked connection) and applies timeout/max_retries in BYOK mode too. Good catch.

• Missing test category marker (Copilot, Devin): added pytestmark = pytest.mark.unit so the module is collected under the project's test rules.

• Confusing provider/model in the reconciliation report (Copilot): the report now splits the served provider/model id for display; pricing still resolves against the full id.

• Provider key forwarded to a user-supplied base_url (Devin): I left this as documented behavior rather than adding a host allowlist. Overriding base_url is the supported way to point the plugin at your own Floe instance — including a self-hosted deployment on a custom domain — so the X-Floe-Provider-Key is meant to travel there. A hostname allowlist would break self-hosting, and this matches how every OpenAI-compatible client sends its key to the configured base_url. I added a docstring note making that explicit; happy to revisit if you'd prefer a scheme (https-only) guard.

Also open to your read on the bigger question: does a metering/budget-guard-oriented plugin fit the plugin taxonomy here, or would you rather see this as an OpenTelemetry integration (there's a fallback exporter documented in the README)?

[CLAassistant]

All committers have signed the CLA.

[achris7]

Both addressed in the latest push:

• Plugin wiring / installability (Devin): registered livekit-plugins-floe in the root [tool.uv.sources] and in livekit-agents's [project.optional-dependencies] (floe extra), and set the plugin version to 1.6.10 to match livekit-agents and the other plugins (lockstep). uv.lock refreshed; uv lock --check is consistent, diff is floe-only. Small note: the type-check task wasn't actually failing beforehand — the livekit-plugins/* workspace-members glob already synced the package (CI type-check was green) — but the source/extras registration is the right convention regardless, so it's in.

• Provider secret over cleartext (Devin): added a fail-closed guard — in BYOK mode the resolved base_url must be https before X-Floe-Provider-Key is attached (loopback http allowed for local dev), else it raises. So the key is never sent over a non-TLS connection. I kept this scheme-based rather than a host allowlist so self-hosted Floe on any https domain still works; keyless mode is unaffected. Added tests for the reject/allow/keyless-unaffected paths.

[achris7]

Good catch — you're right, and it's fixed (not just the resolved base_url now). The parent ignores base_url when a caller passes their own client, but extra_headers rides every request, so a custom http:// client + FLOE_PROVIDER_KEY could still leak the key. The guard now validates the effective endpoint — client.base_url when a client is supplied, else the resolved base_url — so BYOK fails closed in both paths. Also corrected the client docstring (it wrongly implied the header isn't applied to a caller's client). Added tests for the custom-client reject (http) and allow (https) cases.

[achris7]

Both valid — fixed:

• Floe key over cleartext in keyless mode: the TLS guard now runs unconditionally (keyless and BYOK), validating the effective endpoint (a caller-supplied client's address included). The Floe API key is a bearer credential on every request, so keyless refuses a non-TLS target the same way BYOK does. Helper renamed _require_secure_url.
• Inflated cost from non-Floe usage: floe.LLM now tags its usage provider="floe" (a stable tag that works for self-hosted Floe on any domain — the parent's request formatting uses _provider_fmt, so this only affects the usage label), and the reconciler counts only provider="floe" entries. A session that mixes Floe with other providers or realtime no longer inflates the Floe-estimated total, keeping it comparable to Floe's bill.

Tests added for keyless plaintext rejection and Floe-only reconciliation (9 total).

[achris7]

Deferring this rather than patching, for a few reasons:

• The parent LLM builds its httpx client internally with follow_redirects=True and exposes no follow_redirects/http_client passthrough (only timeout). Disabling redirects would require supplying a fully caller-owned client — which reintroduces the connection-lifecycle and dropped timeout/max_retries issues flagged earlier in this same review. So there's no per-plugin fix that doesn't undo those.
• It's not floe-specific: every OpenAI-compatible plugin here shares this client behavior and sends Authorization, so redirect-downgrade is a base-plugin concern rather than something to solve once per plugin.
• httpx strips Authorization on cross-origin redirects (a scheme downgrade is cross-origin) by default, so the Floe bearer key isn't replayed over http. The residual is the custom X-Floe-Provider-Key header on a same-host https→http downgrade, which requires pointing base_url at a hostile server.

Better addressed upstream in the base openai plugin (a follow_redirects / http_client knob) than per-plugin — happy to raise that separately if it'd be useful.

[achris7]

Both fixed — thanks, the blocking one was a real bug:

• Event-loop stall (🔴): the ~10s hosted_remaining_usd() urllib call no longer runs on the handler thread. Since AgentSession.emit doesn't schedule async callbacks, the sync handler now fires the budget read as a background task (asyncio.to_thread, throttled to once per 30s, task ref stored), so audio/speech never stall. Each receipt uses the last cached value (a one-turn lag), and the very first turn simply shows no budget until the first refresh completes.
• Stale budget on failure (🟡): a failed read now sets remaining_usd = None (drops the budget, matching the docstring) instead of keeping the previous figure. Failures are throttled by the same TTL so a down endpoint isn't retried every turn.

Tests: an off-load assertion (the read is not called synchronously; runs once off-loop; cached value applies next turn) and a drop-on-failure test. 12 pass, mypy --strict clean.

[achris7]

Fixed — good edge case. fetched_at now starts at float("-inf") instead of 0.0, so the first budget lookup always fires regardless of the monotonic clock's (boot-relative) origin; no more missing left $… for the first ~30s on a freshly booted worker. Mirrored the same fix in the pipecat twin.

[achris7]

Fixed — real bug, good catch. enable_cost_receipts now takes an api_key and reads the budget with hosted_remaining_usd(api_key), so the left $… balance is for the account being billed. If you pass your key in code to floe.LLM(api_key=...), pass the same key here; if it's in FLOE_API_KEY, omit it (zero-config). Also widened the enable gate so an in-code key isn't blocked by the env-only hosted_enforcement_available() check. Test + README updated; mypy --strict clean, 13 tests pass.

[devin-ai-integration]

Devin Review found 1 new potential issue.

View 5 additional findings in Devin Review.

[achris7]

Valid — prompt-cached input tokens are billed at a discount, and pricing every input token at the full rate overstates the estimate when caching is active. I'm going to fix this properly rather than patch it partially, because the correct fix is provider-aware:

• The discount differs by provider (e.g. Anthropic cache-read ≈ 0.1×, OpenAI cache ≈ 0.5×), and it applies to both the per-turn receipt (turn_cost) and the reconciler (price_tokens) — so the right place is floe-guard, exposing the cached-token counts through turn_cost/price_tokens with per-provider multipliers, then having both surfaces subtract cached tokens from the full-rate input and price them at the cache rate.

Doing it half-way here (single hardcoded multiplier) would be right for Anthropic and wrong for OpenAI, which is worse than the current honest-but-coarse est. Tracking it as a floe-guard accuracy follow-up. In the meantime the figure is explicitly labelled est (a local estimate; Floe's bill is authoritative), so it doesn't claim invoice parity.