fix(bridge): add 20s timeout guard to core.shutdown() to prevent orphaned processes#1799
fix(bridge): add 20s timeout guard to core.shutdown() to prevent orphaned processes#1799chiefmojo wants to merge 6 commits into
Conversation
core.shutdown() drains the L2/L3/skill flush pipeline which can block on hanging LLM calls. Without a deadline the bridge never exits after stdin EOF when the Python parent is already gone, re-creating the process-leak condition. Race all three shutdown sites (daemon SIGTERM, non-daemon SIGTERM, headless stdin-EOF) against a 20s timeout so the process always terminates within a bounded time. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
73cce94 to
71b669d
Compare
Three sites missed by the original patch (56ebe7a3) were calling
core.shutdown() without withShutdownTimeout, leaving the bridge process
able to hang indefinitely if L2/L3/skill LLM calls stalled at shutdown:
• bridge.cts: EADDRINUSE exit (×2) — daemon can't bind viewer port
• bridge.cts: viewer-running keepalive path — stdin closes but viewer
is still serving; core.shutdown fires from the interval callback
All six core.shutdown() call sites now go through withShutdownTimeout,
guaranteeing the bridge exits within 20s regardless of which path is
taken. Adds bridge-shutdown-audit.md documenting the full call chain and
confirming no blocking sync calls prevent the timeout from firing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Automated Test Results: PASSED Cloud test-engine rerun against
Manual code review is still required before merge. |
shinetata
left a comment
There was a problem hiding this comment.
Review: Request Changes
Thanks for tackling this — the root cause in #1798 is real and the approach (a bounded Promise.race timeout followed by process.exit) is the correct shape for preventing orphaned bridges. I verified the six hunks apply cleanly onto current main's bridge.cts with no conflicts. Two blockers plus a few smaller items before this can merge.
Blockers
1. bridge.mts is untouched — the fix doesn't cover the preferred entry.
bridge.cts is the legacy CommonJS entry. bridge.mts is its pure-ESM successor and carries the same six core.shutdown() call sites (currently lines 395 / 401 / 410 / 481 / 502 / 511). The Python launcher resolves entries in this order:
dist/bridge.mjs → dist/bridge.cjs → bridge.mts → bridge.cts
(see adapters/hermes/memos_provider/bridge_client.py and daemon_manager.py, pinned by tests/python/test_bridge_script_resolution.py). Any normally-built deployment runs dist/bridge.mjs (compiled from bridge.mts) and never touches the patched .cts / .cjs. As written, the orphaned-bridge fix does not apply to the default production path. Please mirror the guard into bridge.mts, or factor withShutdownTimeout into a shared module both entries import.
2. bridge-shutdown-audit.md is added at the repository root.
This is a Python monorepo; a plugin-specific investigation doc shouldn't live in the repo root. Please move it under apps/memos-local-plugin/docs/ (where RFC-002 already lives) or drop it from the PR. Its line references (bridge.cts:373–381, 479–494, 517–520, 500–515, 358–370) are from the internal companion-stable branch and don't match main (actual sites: 471 / 546 / 574 / 563 / 456) — please correct or remove them if the doc stays.
Should fix
- No test for the new logic.
withShutdownTimeoutis an easily testable pure function and the repo follows TDD. Please add a unit test (fake timers) asserting the helper resolves after 20s whencore.shutdown()never settles — seetests/unit/bridge/stdio.test.tsfor a pattern. (The automated "33 passed" comment ran againstdev-v2.0.22and does not exercise this code path.) - PR description doesn't match the diff. It states sites #2/#3 live in
bridge/stdio.ts, but there is nobridge/stdio.tschange — all six sites are inbridge.cts(the non-daemon path merely callswaitForShutdown, which is defined in stdio.ts).
Nit
withShutdownTimeout'ssetTimeoutis neitherunref()'d nor cleared. Harmless today because every call site is immediately followed byprocess.exit(), but it will keep the loop alive for 20s if the helper is ever reused without an immediate exit — consider.unref().- The branch merged
dev-20260604-v2.0.19/mainand the commit list includes unrelated fixes; please squash on merge so only the 2-file change lands.
…os-local-plugin/docs/ - Removed companion-stable branch reference from header - Updated bridge.cts line numbers to match main (per shinetata review) - Removed companion-stable commit hash 56ebe7a3 - Moved from repo root to apps/memos-local-plugin/docs/ per review request
🤖 Open Code ReviewTarget: PR #1799 ✅ OpenCodeReview: No supported files changed. Generated by cloud-assistant via Open Code Review. |
✅ Automated Test Results: PASSEDAll tests passed (33/33 executed). memos_local_plugin/unit: 33/33. Duration: 5s [advisory, non-gating] AI-generated tests on branch test/auto-gen-eeab53129f00e68c-20260718033510: 66/70 passed, 4 failed — these do NOT affect the PR verdict; review the branch manually. Branch: |
✅ Automated Test Results: PASSEDAll tests passed (33/33 executed). memos_local_plugin/unit: 33/33. Duration: 5s [advisory, non-gating] AI-generated tests on branch test/auto-gen-c92f89ebf770abfc-20260718215045: 18/18 passed — these do NOT affect the PR verdict; review the branch manually. Branch: |
Problem
When the Hermes gateway dies abnormally (SIGKILL, OOM, crash), the
--no-viewerbridge process callscore.shutdown()which chains throughflush()→ L2/L3 LLM calls that can block indefinitely. With the parent gone, nobody remains to send SIGKILL. Over 36 hours, 19 orphaned bridges accumulated consuming 299% CPU and writing duplicate traces (6,572 copies of a single turn).Fixes #1798.
Changes
Adds a
withShutdownTimeout()helper that racescore.shutdown()against a 20-second deadline. Wraps all sixcore.shutdown()call sites:bridge.cts)bridge/stdio.ts)bridge/stdio.ts)Also adds
bridge-shutdown-audit.mddocumenting the full call chain throughflush()→ L2/L3/skill, confirming all async operations yield the event loop and the 20s timeout is effective on every path.Verification
Related
This PR addresses the bridge-side half of the shutdown problem. The adapter-side complement — moving the synchronous
session.closeHTTP call off the asyncio event loop thread to prevent Discord heartbeat stalls when the bridge is unresponsive — is in #1953 (fix(adapter): fire session.close in daemon thread to unblock event loop).