Skip to content

fix(cli): report a killed edge runtime container instead of a lost log stream - #6615

Merged
Prashansa-K merged 4 commits into
developfrom
fix/serve-container-killed
Sep 23, 2026
Merged

Prashansa-K merged 4 commits into
developfrom
fix/serve-container-killed

Conversation

@Prashansa-K

@Prashansa-K Prashansa-K commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR

An edge runtime container that gets killed during supabase functions serve — most often by the out-of-memory killer — is retried until the re-attach cap gives up, and then reported as a lost log stream against a container the message claims is still running. This reports the kill instead, and separates a memory-limit kill from one the CLI cannot attribute.

Refs CLI-2427 — functions serve misreports a killed edge runtime container as a lost log stream (exit 137). This is the 137 gap #6594 left open.

What was wrong

The retry can never succeed. streamContainerLogs treats container exit 137 as retriable, but the container is created with no --restart policy, so once it is killed it stays dead. docker logs -f is respawned, exits, the container is inspected, 137 comes back.

The resulting message says the opposite of what happened. After the cap the command fails with lost the Edge Runtime log stream 5 times; container … is still running. The container is not running — that is the one fact the preceding docker container inspect established.

The classification sends the user the wrong way. EdgeRuntimeLogStreamLostError declares external_service / network with a "rerun with --debug" remediation. An out-of-memory kill is not a network problem, and that remediation leads nowhere.

The retry also contradicts both of its neighbours: apps/cli-go/internal/utils/docker.go maps 137 to ErrContainerKilled with no retry, and db diff treats 137 as a genuine failure.

What changed

137 falls through to EdgeRuntimeContainerCrashedError, which now carries State.OOMKilled from the docker container inspect the code already makes. That call uses --format "{{json .State}}", so this is one more field read in parseContainerState — no format string changes.

Container state error_kind / error_category Suggestion
137, OOMKilled: true user_actionable / resource_limit raise the container's memory limit, or free up memory on the host
137, OOMKilled: false, container still present unknown / unknown none
137, OOMKilled: false, container gone on re-inspect unchanged — the session ends successfully —
129/130/131/143 unchanged — the session ends successfully —
any other non-zero unchanged — internal_bug / runtime_crash —

resource_limit is a new user_actionable category. Both failing 137 rows get their own fingerprint suffix, so they stay separable from each other and from the untagged bulk of their categories.

A non-OOM 137 gets one re-inspect before failing: supabase stop force-kills a runtime that ignores SIGTERM and prunes it right after, and that prune racing the CLI's own inspect is what the deleted retry backed into fixing by accident. The re-inspect makes the fix deliberate instead — still failing fast for a kill from outside that leaves the container in place.

Two calls worth reviewing

An out-of-memory kill is not internal_bug / runtime_crash. It is Docker's memory allocation or the user's own function, and the user can act on it. Filing it against the internal-bug counter would be wrong in the direction this work exists to correct.

A non-OOM 137 is unknown, not user_cancelled. Docker sets OOMKilled reliably only for container-limit OOMs; a host or VM out-of-memory kill on macOS lands here with OOMKilled: false. Calling that "the user cancelled" asserts something we do not know. unknown with a named fingerprint keeps it separable so it can be reclassified once there is data on which cause dominates — at the cost of knowingly adding to the unknown bucket.

Behaviour change to be aware of

A container killed for a reason the CLI cannot attribute still ends the session successfully if a supervisor (e.g. supabase stop) prunes it before the CLI's one re-inspect runs — that is unchanged. Otherwise, a killed container now fails after that single re-inspect rather than after five re-attaches, and fails with a different tagged error than before. A functions serve & CI step relying on the old re-attach window to survive a kill will now fail sooner, at the kill.

🤖 Generated with Claude Code

@Prashansa-K
Prashansa-K requested a review from a team as a code owner September 15, 2026 05:26
@Prashansa-K Prashansa-K changed the title fix(cli): report a killed edge runtime container instead of a lost log stream (CLI-2427) fix(cli): report a killed edge runtime container instead of a lost log stream Sep 15, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI Review

Both independent reviews completed. Six distinct findings were adjudicated: five confirmed and one refuted. No critical or major defects were found.

Findings

Severity Location Category Sources Claim
🟡 MINOR apps/cli/src/shared/functions/serve.ts:1424 behavior-change claude Removing the non-OOM exit-137 retry introduces a race where supabase stop can force-kill the container and functions serve fails before the subsequent prune removes it.
🟡 MINOR apps/cli/src/command-internal/docker-suggest.ts:13 correctness codex The remediation overstates OOMKilled as proof that the container exceeded its own configured memory limit.
🟡 MINOR apps/cli/src/shared/functions/serve.ts:1464 error-handling claude When docker logs -f exits non-zero and inspection finds a stopped OOM-killed container, the code reports a generic log-stream error instead of the container crash and memory remediation.
⚪ NIT apps/cli/src/commands/functions/serve/serve.integration.test.ts:2366 test-coverage claude The five-second timeout does not provide the stated retry-regression detection and unnecessarily includes potentially slow command bring-up.
⚪ NIT apps/cli/src/commands/functions/serve/SIDE_EFFECTS.md:75 documentation claude The exit-code documentation incorrectly restricts the memory-limit classification to exit 137 even though the implementation classifies any OOMKilled: true non-zero exit as a resource limit.

Findings outside the diff

  • 🟡 MINOR apps/cli/src/shared/functions/serve.ts:1464 — When docker logs -f exits non-zero and inspection finds a stopped OOM-killed container, the code reports a generic log-stream error instead of the container crash and memory remediation.
Refuted findings (kept for transparency, not posted as review comments)
  • apps/cli/src/shared/telemetry/error-actionability.ts:365 (telemetry): Using suggestion_type: update_config for runtime memory allocation misleadingly categorizes the remediation as a CLI configuration edit.
    Refuted: The closed vocabulary defines UpdateConfig generically, not specifically as config.toml or CLI-owned configuration. Raising the container runtime's memory allocation is a configuration change, and the separate resource_limit error category preserves the cause.

Stats

Claude findings: 5 · Codex findings: 1 · Confirmed: 5 · Refuted: 1 · Uncertain: 0


Models: claude-opus-5 + gpt-5.6-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

Comment thread apps/cli/src/shared/functions/serve.ts Outdated
Comment thread apps/cli/src/commands/functions/serve/serve.integration.test.ts Outdated
Comment thread apps/cli/src/commands/functions/serve/SIDE_EFFECTS.md Outdated
Comment thread apps/cli/src/command-internal/docker-suggest.ts Outdated
Prashansa-K and others added 2 commits September 23, 2026 10:58
Adds a resource_limit category under user_actionable and its
out_of_memory / container_killed fingerprint suffixes, so an edge
runtime container killed for exceeding its memory limit can be
classified separately from an internal bug.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Exit 137 re-attached to the log stream until the re-attach cap gave up,
reporting an out-of-memory kill as a lost log stream against a container
the message claimed was still running. `State.OOMKilled` separates a
memory-limit kill, which carries a memory-allocation remediation, from a
kill the CLI cannot attribute to either side.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Prashansa-K
Prashansa-K force-pushed the fix/serve-container-killed branch from d322ccd to e0a6ea2 Compare September 23, 2026 05:42
@Prashansa-K

Copy link
Copy Markdown
Contributor Author

Working on the third filed minor issue When docker logs -f exits non-zero and inspection finds a stopped OOM-killed container, the code reports a generic log-stream error instead of the container crash and memory remediation. in a separate PR, so as to not make this one too broad.

@Coly010 Coly010 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes on one user-facing string.

SUGGEST_CONTAINER_MEMORY_LIMIT tells the user to "raise its memory limit", but functions serve never sets one. The docker create argv at apps/cli/src/shared/functions/serve.ts:1902 passes --ulimit nofile and no memory flag, and there is no --memory argument or memory key anywhere in apps/cli/src or packages/config/src.

So State.OOMKilled: true here can only mean the container's cgroup inherited its parent's limit and exhausted it — the host's RAM on Linux, the Docker Desktop / Colima / OrbStack VM's allocation on macOS and Windows. There is no per-container limit for the user to raise: config.toml has no key for it, serve has no flag for it, and docker update --memory evaporates on the next restart since the container is recreated each time.

That matters more than usual in this PR. The whole point of the new user_actionable / resource_limit classification is to replace a remediation that sent users somewhere useless (external_service / network plus "rerun with --debug"). As written, the replacement sends them hunting for a config key that does not exist, and the remedy that actually fits most machines — raising the container runtime's own memory allocation — is not mentioned at all.

Suggestion inline. I have left the constant's name alone: SUGGEST_CONTAINER_OUT_OF_MEMORY would describe the event rather than a configured limit, but that rename also touches serve.errors.ts and both test files, so a single-file suggestion cannot carry it. Your call whether it is worth doing here.

Scoped to this one ask.

Comment thread apps/cli/src/command-internal/docker-suggest.ts Outdated
@Prashansa-K
Prashansa-K added this pull request to the merge queue Sep 23, 2026
@Prashansa-K
Prashansa-K removed this pull request from the merge queue due to a manual request Sep 23, 2026
@Prashansa-K
Prashansa-K added this pull request to the merge queue Sep 23, 2026
Merged via the queue into develop with commit 6530571 Sep 23, 2026
40 of 42 checks passed
@Prashansa-K
Prashansa-K deleted the fix/serve-container-killed branch September 23, 2026 11:08
pull Bot pushed a commit to chizee/cli that referenced this pull request Sep 23, 2026
…errors (supabase#6714)

## CLI-2476: `functions serve` skips OOM classification when the `docker
logs` stream itself errors

`streamContainerLogs` ends in one of two places depending on how `docker
logs -f` itself exited. When it exits `0`, the follow-up inspect's
`State.OOMKilled` (added in supabase#6615 / CLI-2427) classifies a memory-limit
kill as `user_actionable` / `resource_limit` with a remediation. When
`docker logs -f` itself errors, the same follow-up inspect runs and
produces the same `oomKilled` value — but `DockerLogsStreamError` had no
field to carry it, so the kill fell into `unknown` with no suggestion.

This gives `DockerLogsStreamError` an `oomKilled` field, sourced from
the same inspect that already produces `daemonDown`, and branches its
actionability/`suggestion` getters OOM-first, mirroring
`EdgeRuntimeContainerCrashedError`.

**Stacks on supabase#6615** (base branch `fix/serve-container-killed`) — this
diff is additive on top of that PR and should be reviewed/merged after
it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Colum Ferry <cferry09@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants