fix(cli): report a killed edge runtime container instead of a lost log stream - #6615
Conversation
There was a problem hiding this comment.
🤖 AI Review
Both independent reviews completed. Six distinct findings were adjudicated: five confirmed and one refuted. No critical or major defects were found.
Findings
| Severity | Location | Category | Sources | Claim |
|---|---|---|---|---|
| 🟡 MINOR | apps/cli/src/shared/functions/serve.ts:1424 |
behavior-change |
claude | Removing the non-OOM exit-137 retry introduces a race where supabase stop can force-kill the container and functions serve fails before the subsequent prune removes it. |
| 🟡 MINOR | apps/cli/src/command-internal/docker-suggest.ts:13 |
correctness |
codex | The remediation overstates OOMKilled as proof that the container exceeded its own configured memory limit. |
| 🟡 MINOR | apps/cli/src/shared/functions/serve.ts:1464 |
error-handling |
claude | When docker logs -f exits non-zero and inspection finds a stopped OOM-killed container, the code reports a generic log-stream error instead of the container crash and memory remediation. |
| ⚪ NIT | apps/cli/src/commands/functions/serve/serve.integration.test.ts:2366 |
test-coverage |
claude | The five-second timeout does not provide the stated retry-regression detection and unnecessarily includes potentially slow command bring-up. |
| ⚪ NIT | apps/cli/src/commands/functions/serve/SIDE_EFFECTS.md:75 |
documentation |
claude | The exit-code documentation incorrectly restricts the memory-limit classification to exit 137 even though the implementation classifies any OOMKilled: true non-zero exit as a resource limit. |
Findings outside the diff
- 🟡 MINOR
apps/cli/src/shared/functions/serve.ts:1464— Whendocker logs -fexits non-zero and inspection finds a stopped OOM-killed container, the code reports a generic log-stream error instead of the container crash and memory remediation.
Refuted findings (kept for transparency, not posted as review comments)
apps/cli/src/shared/telemetry/error-actionability.ts:365(telemetry): Usingsuggestion_type: update_configfor runtime memory allocation misleadingly categorizes the remediation as a CLI configuration edit.
Refuted: The closed vocabulary definesUpdateConfiggenerically, not specifically asconfig.tomlor CLI-owned configuration. Raising the container runtime's memory allocation is a configuration change, and the separateresource_limiterror category preserves the cause.
Stats
Claude findings: 5 · Codex findings: 1 · Confirmed: 5 · Refuted: 1 · Uncertain: 0
Models: claude-opus-5 + gpt-5.6-sol · Trigger: auto · Workflow run
This review runs once per PR. A maintainer can request another with a /ai-review comment.
Adds a resource_limit category under user_actionable and its out_of_memory / container_killed fingerprint suffixes, so an edge runtime container killed for exceeding its memory limit can be classified separately from an internal bug. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Exit 137 re-attached to the log stream until the re-attach cap gave up, reporting an out-of-memory kill as a lost log stream against a container the message claimed was still running. `State.OOMKilled` separates a memory-limit kill, which carries a memory-allocation remediation, from a kill the CLI cannot attribute to either side. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
d322ccd to
e0a6ea2
Compare
|
Working on the third filed minor issue |
Coly010
left a comment
There was a problem hiding this comment.
Request changes on one user-facing string.
SUGGEST_CONTAINER_MEMORY_LIMIT tells the user to "raise its memory limit", but functions serve never sets one. The docker create argv at apps/cli/src/shared/functions/serve.ts:1902 passes --ulimit nofile and no memory flag, and there is no --memory argument or memory key anywhere in apps/cli/src or packages/config/src.
So State.OOMKilled: true here can only mean the container's cgroup inherited its parent's limit and exhausted it — the host's RAM on Linux, the Docker Desktop / Colima / OrbStack VM's allocation on macOS and Windows. There is no per-container limit for the user to raise: config.toml has no key for it, serve has no flag for it, and docker update --memory evaporates on the next restart since the container is recreated each time.
That matters more than usual in this PR. The whole point of the new user_actionable / resource_limit classification is to replace a remediation that sent users somewhere useless (external_service / network plus "rerun with --debug"). As written, the replacement sends them hunting for a config key that does not exist, and the remedy that actually fits most machines — raising the container runtime's own memory allocation — is not mentioned at all.
Suggestion inline. I have left the constant's name alone: SUGGEST_CONTAINER_OUT_OF_MEMORY would describe the event rather than a configured limit, but that rename also touches serve.errors.ts and both test files, so a single-file suggestion cannot carry it. Your call whether it is worth doing here.
Scoped to this one ask.
Co-authored-by: Colum Ferry <cferry09@gmail.com>
…errors (supabase#6714) ## CLI-2476: `functions serve` skips OOM classification when the `docker logs` stream itself errors `streamContainerLogs` ends in one of two places depending on how `docker logs -f` itself exited. When it exits `0`, the follow-up inspect's `State.OOMKilled` (added in supabase#6615 / CLI-2427) classifies a memory-limit kill as `user_actionable` / `resource_limit` with a remediation. When `docker logs -f` itself errors, the same follow-up inspect runs and produces the same `oomKilled` value — but `DockerLogsStreamError` had no field to carry it, so the kill fell into `unknown` with no suggestion. This gives `DockerLogsStreamError` an `oomKilled` field, sourced from the same inspect that already produces `daemonDown`, and branches its actionability/`suggestion` getters OOM-first, mirroring `EdgeRuntimeContainerCrashedError`. **Stacks on supabase#6615** (base branch `fix/serve-container-killed`) — this diff is additive on top of that PR and should be reviewed/merged after it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Colum Ferry <cferry09@gmail.com>
TL;DR
An edge runtime container that gets killed during
supabase functions serve— most often by the out-of-memory killer — is retried until the re-attach cap gives up, and then reported as a lost log stream against a container the message claims is still running. This reports the kill instead, and separates a memory-limit kill from one the CLI cannot attribute.Refs CLI-2427 —
functions servemisreports a killed edge runtime container as a lost log stream (exit 137). This is the137gap #6594 left open.What was wrong
The retry can never succeed.
streamContainerLogstreats container exit137as retriable, but the container is created with no--restartpolicy, so once it is killed it stays dead.docker logs -fis respawned, exits, the container is inspected,137comes back.The resulting message says the opposite of what happened. After the cap the command fails with
lost the Edge Runtime log stream 5 times; container … is still running. The container is not running — that is the one fact the precedingdocker container inspectestablished.The classification sends the user the wrong way.
EdgeRuntimeLogStreamLostErrordeclaresexternal_service/networkwith a "rerun with--debug" remediation. An out-of-memory kill is not a network problem, and that remediation leads nowhere.The retry also contradicts both of its neighbours:
apps/cli-go/internal/utils/docker.gomaps137toErrContainerKilledwith no retry, anddb difftreats137as a genuine failure.What changed
137falls through toEdgeRuntimeContainerCrashedError, which now carriesState.OOMKilledfrom thedocker container inspectthe code already makes. That call uses--format "{{json .State}}", so this is one more field read inparseContainerState— no format string changes.error_kind/error_category137,OOMKilled: trueuser_actionable/resource_limit137,OOMKilled: false, container still presentunknown/unknown137,OOMKilled: false, container gone on re-inspect129/130/131/143internal_bug/runtime_crashresource_limitis a newuser_actionablecategory. Both failing137rows get their own fingerprint suffix, so they stay separable from each other and from the untagged bulk of their categories.A non-OOM
137gets one re-inspect before failing:supabase stopforce-kills a runtime that ignoresSIGTERMand prunes it right after, and that prune racing the CLI's own inspect is what the deleted retry backed into fixing by accident. The re-inspect makes the fix deliberate instead — still failing fast for a kill from outside that leaves the container in place.Two calls worth reviewing
An out-of-memory kill is not
internal_bug/runtime_crash. It is Docker's memory allocation or the user's own function, and the user can act on it. Filing it against the internal-bug counter would be wrong in the direction this work exists to correct.A non-OOM
137isunknown, notuser_cancelled. Docker setsOOMKilledreliably only for container-limit OOMs; a host or VM out-of-memory kill on macOS lands here withOOMKilled: false. Calling that "the user cancelled" asserts something we do not know.unknownwith a named fingerprint keeps it separable so it can be reclassified once there is data on which cause dominates — at the cost of knowingly adding to theunknownbucket.Behaviour change to be aware of
A container killed for a reason the CLI cannot attribute still ends the session successfully if a supervisor (e.g.
supabase stop) prunes it before the CLI's one re-inspect runs — that is unchanged. Otherwise, a killed container now fails after that single re-inspect rather than after five re-attaches, and fails with a different tagged error than before. Afunctions serve &CI step relying on the old re-attach window to survive a kill will now fail sooner, at the kill.🤖 Generated with Claude Code