tinker: survive completion bursts at the socket layer

[j316chuck]

Summary

• Coalesce session heartbeats arriving in a 50 ms wave into one SQLite transaction.
• Keep each session's heartbeat_count exact by applying the number of waiting requests for that session.
• Increase Uvicorn's accept backlog and keep-alive window for high-fanout clients.

Why

At 128 concurrent LoRA training jobs, every heartbeat previously acquired the global SQLite write lock and committed independently. The queue then exceeded the 300-second stale-session window even though inference was still progressing, which evicted live sessions. Batching only the simultaneous heartbeat wave removes that write-amplification bottleneck without weakening liveness accounting. The socket settings provide headroom for the same completion/heartbeat bursts at the HTTP layer.

Validation

• uv run --extra dev --extra tinker pytest tests/tinker/test_external_future_store.py -q: 13 passed.
• Ruff, Black, and hardcoded-secret checks pass for skyrl/tinker/api.py.
• Immutable image: us-central1-docker.pkg.dev/useful-memory-477923-v7/skyrl/megatron:skyrl-pr2177-c6be3fc3@sha256:0edbd53d5032c443b82c527340e1d87ad27180b91bf1c815758825d2f38e8fb7.
• Live workload provenance matched commit c6be3fc3c693f76e42940eb716c2f008fa615b76 and the image digest above.
• In the bounded 128-job qualification, all jobs launched; after 2,057 seconds, 89 reached step 0, 50 step 1, and 18 step 2. The run was stopped at the requested time limit, so it is partial evidence rather than a completed 20-step pass.

Trajectory workload integration: https://github.com/Trajectorylabs/trajectory/pull/4945

[j316chuck]

@hershg Avi’s #2163 fix was merged into a stacked branch but did not reach main. This ports the exact socket backlog + keep-alive change directly onto current main.

[gemini-code-assist]

Code Review

This pull request updates the Uvicorn server configuration in skyrl/tinker/api.py to better handle connection bursts. It introduces a HTTP_KEEP_ALIVE_TIMEOUT_SECONDS constant set to 75 seconds to prevent idle SDK connections from closing during busy event loop periods, and configures uvicorn.run with this keep-alive timeout and a custom connection backlog limit (SKYRL_HTTP_CONNECTION_LIMIT). There are no review comments, so I have no feedback to provide.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge with no concrete blocking or non-blocking defects identified.