[tinker] Survive completion bursts at the socket layer (accept backlog, keep-alive)

[avigyabb]

Stack 4/7.

uvicorn.run: backlog=SKYRL_HTTP_CONNECTION_LIMIT (50k, as on Chuck's branch; the effective value is capped by net.core.somaxconn, raise it to match) and timeout_keep_alive=75.

With 131072 outstanding samples and 212k-token results, each 2048-result completion burst kept the event loop busy ~16 s; uvicorn's 5 s keep-alive then closed every idle client connection, all clients reconnected at once and the 2048-entry accept backlog overflowed, refusing 109k of 131072 requests. With these settings the same run completed 130917 of 131072 with zero forwarding errors and zero reconnects (earlier runs on the 5 s keep-alive showed hundreds to thousands).

Neither setting is needed for correctness: the SDK retries refused or dropped connections. They avoid the reconnect storm rather than fix a failure, and with the SDK's per-client in-flight cap the burst that overflowed the backlog does not occur. An earlier revision of this PR also exposed sample_max_concurrent_requests from EngineConfig via /client/config; that was dropped as unnecessary (its default equalled the SDK default, and no measured run depended on it).

Stack (each PR retargets to main as the one below merges)

1. [tinker] Route uvicorn's access log to a plain handler instead of Rich #2160
2. [tinker] Forward samples with aiohttp instead of httpx #2161
3. [tinker] Keep an undelivered sample result alive for the SDK's retry #2162
4. [tinker] Survive completion bursts at the socket layer (accept backlog, keep-alive) #2163
5. [tinker] Encode forwarded sample results to proto once and serve them as-is #2164
6. [tinker] Decode vLLM completion bodies straight into numpy with pysimdjson #2165
7. [tinker] Load harness for the API server's sampling path at 131k concurrency #2166

🤖 Generated with Claude Code

---

Note

Low Risk
Server-only uvicorn listen/keep-alive defaults; no API or auth behavior change, with SDK retries as a fallback.

Overview
The Tinker API server now passes uvicorn socket tuning so large completion bursts do not trigger mass client reconnects and accept-queue overflows.

timeout_keep_alive is set to 75s (via HTTP_KEEP_ALIVE_TIMEOUT_SECONDS) instead of uvicorn’s 5s default, so idle SDK connections stay open while the event loop is busy for many seconds during bursts.

backlog is set to SKYRL_HTTP_CONNECTION_LIMIT (default 50k, overridable by env), so pending connections queue in the kernel instead of being refused when the loop cannot accept fast enough (effective cap is net.core.somaxconn).

These are reliability/performance knobs, not correctness fixes—the SDK already retries refused or dropped connections.

^{Reviewed by Cursor Bugbot for commit 27c838e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[gemini-code-assist]

Code Review

This pull request configures Uvicorn's connection backlog and keep-alive timeout to better handle connection bursts and prevent clients from reconnecting simultaneously. The review feedback suggests improving deployment flexibility by allowing both the keep-alive timeout and the backlog size to be configured independently via environment variables rather than being hardcoded or coupled to existing limits.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge with no concrete blocking or independently actionable non-blocking issue identified.

The configured backlog uses the documented integer connection limit already applied to another Uvicorn server, while the longer keep-alive directly addresses the described reconnect storms without changing request correctness.

Important Files Changed

Filename	Overview
skyrl/tinker/api.py	Adds documented Uvicorn backlog and keep-alive settings using an existing repository-wide connection-limit configuration.

_{Reviews (1): Last reviewed commit: "[tinker] Survive completion bursts at th..." | Re-trigger Greptile}