Stream IMAP message parts instead of buffering whole messages

[ivarsb]

Problem

The mail-mcp pod was OOMKilled every ~10 minutes at a 384Mi limit — Restart Count: 101, exit code 137, ~16h of crashlooping.

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      Mon, 24 Aug 2026 13:41:45 +0300
  Finished:     Mon, 24 Aug 2026 13:51:29 +0300
Restart Count:  101

Root cause

get_message fetched RFC822, which materializes the whole message as one String, parsed it with Mail, then base64-decoded each attachment while the raw message was still reachable.

Measured on an 8.27MB message, peak live strings hit 55.6MB — 6.7× the wire size:

stage	live strings >64KB
wire RFC822	8.27 MB
after Mail.new	24.80 MB
after text/html decode (forces the part tree)	49.60 MB
at the synchronous S3 upload	55.60 MB

net-imap literals are always ASCII-8BIT (documented at net/imap.rb:3119), and Mail.new on a binary source eagerly allocates two more full-size copies on top of the aliased raw_source; materializing the part tree then copies every part's base64 body again.

Why it OOMKilled

config/puma.rb runs a single worker with threads 1, 5, so that peak is per in-flight
request. Five concurrent large messages is ~460MB of peak plus boot, which clears 384Mi on
its own — measured at 614MB for five concurrent 25MB-attachment fetches. The ~10 minute
cadence is how often such a burst lands, not a steady climb.

Verified on glibc in ruby:3.4.9-slim that RSS does not accumulate across fetches — it
plateaus on the first one and stays there, with or without MALLOC_ARENA_MAX=2:

	boot	fetch 1	fetch 8
default arenas	42.8 MB	99.2 MB	99.3 MB
MALLOC_ARENA_MAX=2	42.8 MB	99.2 MB	99.3 MB

(20.7MB message, 15MB attachment; 56.5MB above boot ≈ 2.7× wire, matching the peak above.
The same loop on macOS malloc grows ~34.5MB per fetch without plateauing, so that behaviour
is allocator-specific and not what production hits. Container was aarch64; AKS is x86_64,
and glibc's allocator behaves broadly alike across the two.)

So this is a peak-memory problem, not a leak: raising the limit genuinely helps, and
lowering the per-request peak is what makes the limit hold.

Fix

MessageReader walks BODYSTRUCTURE and fetches each piece on its own: header, inline bodies, then each attachment via partial fetches BODY.PEEK[n]<offset.length> through an incremental base64 decoder into a Tempfile, uploaded and discarded before the next attachment is fetched.

The largest allocation in the path is now a 4MiB chunk regardless of message size, so the per-request peak no longer scales with the message — which is what makes a fixed memory limit hold. That holds by construction, independently of the numbers below.

AttachmentStore gains a streaming upload_io and memoizes its client and presigner, which were being rebuilt twice per attachment (measured: ~110ms and ~19MB for the first, ~0.8ms after).

Verification

A differential harness runs old vs new over 11 fixtures — single-part, multipart/alternative, nested related>alternative, non-UTF-8 charset, quoted-printable, attachment with no filename, inline image with a Content-ID, message/rfc822, and a multi-chunk 9MB attachment. Attachment bytes are identical in every case; no difference is unexplained. The fake IMAP server derives part addressing from Mail's own tree, so the walker is not tested against itself.

• Five concurrent 25MB-attachment fetches peak at 221MB, against 614MB before — a 2.8× cut in the number that has to fit under the limit.
• RSS flat across 12 sequential fetches, as before the change on glibc.
• put_object with a Tempfile body genuinely streams: +18.6MB for a 40MB body.
• 114 specs pass; rubocop clean.

Concurrency and peak numbers are macOS malloc; the glibc figures in Root cause come from ruby:3.4.9-slim. The peak multiplier agrees across both (~2.7–3.4× wire). The construction argument above is what does not depend on the allocator.

Behaviour changes

• text_body on a non-multipart message. Mail#text_part searches all_parts, which is empty unless the message is multipart — so the old code returned nil for the body of every plain-text email. It is now returned. Pre-existing bug, fixed incidentally.
• Attachments with an RFC 2231-split or RFC 2047-encoded filename now appear at all. A filename is what marks a part as an attachment, and the old lookup returned nil for both forms, so those attachments vanished from the response entirely.
• content_type is the bare MIME type rather than the full Content-Type header; filename is already its own field.
• A 7bit/8bit attachment keeps the line endings the server sent, where Mail#decoded normalized CRLF to LF. base64 attachments are byte-identical.
• A missing section mid-stream raises MessageReader::IncompletePart instead of uploading a zero-byte attachment under a valid-looking presigned URL.

Body sections are requested with BODY.PEEK, so reading a message cannot set \Seen.

Deploying

/tmp in the running container is the overlay filesystem, not tmpfs — verified — so attachment spooling lands on disk and no volume change is needed. It is also on the container's writable layer, which is recreated per restart, so a spool file leaked by a SIGKILL does not accumulate.

Worth pairing with this:

• Keep the memory limit at 1Gi initially as margin, and watch container_memory_working_set_bytes for an hour before tightening. Baseline to beat is 101 restarts / ~10 min lifetime. Since this is a peak problem rather than a leak, the curve should be sawtooth with a stable floor — a rising floor would mean something here is wrong.
• Add ephemeral-storage requests/limits — the spool is currently untracked and the node is at 79%. Worst case is threads × largest attachment ≈ 250MB.
• Raise timeoutSeconds/failureThreshold on both probes. The Unhealthy events seen alongside the OOMKills are a separate mechanism: put_object is synchronous and holds a Puma thread, so /health can queue behind five busy threads. This PR does not address thread occupancy. Do not lower the thread count as a memory measure — it makes probe starvation more likely.

Deliberately not in this PR

• Streaming quoted-printable. Mail's QP decoder normalizes line endings and repairs hard breaks mis-encoded as hex across the whole string, and is empirically not decomposable across chunk boundaries — it mismatches at every slice size tested, including when only the trailing 8 bytes are split off. A hand-rolled substitute risks silently corrupting attachment content, which is worse than the bounded-but-larger memory use of an encoding only chosen for text. The one-pass path now logs when taken.
• Capping inline body size. Inline parts are still fetched whole, so a very large inline HTML body is unbounded. Pre-existing, and truncating a body changes what the tool returns.
• Eager S3 client init. @s3 ||= is not atomic, so a first concurrent burst can build up to five clients and discard four. Benign, but moving the ~110ms out of the first request touches config.ru.
• search_messages uses @imap.search, which returns sequence numbers, while every other method is UID-based and the tool labels the result uids. Separate bug, cheap fix.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Streams IMAP message parts to reduce memory usage during attachment processing.

Changes:

• Adds BODYSTRUCTURE-based message reading and chunked Base64 decoding.
• Streams attachments through temporary files to memoized S3 clients.
• Adds coverage for MIME structures, encoding, and streaming behavior.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
lib/mail_mcp.rb	Loads new streaming components.
lib/mail_mcp/attachment_store.rb	Streams IO uploads and memoizes AWS clients.
lib/mail_mcp/base64_stream.rb	Adds incremental Base64 decoding.
lib/mail_mcp/imap_client.rb	Delegates message retrieval to MessageReader.
lib/mail_mcp/message_reader.rb	Fetches and processes individual IMAP sections.
lib/mail_mcp/message_structure.rb	Flattens BODYSTRUCTURE into fetchable parts.
spec/mail_mcp/attachment_store_spec.rb	Tests streaming S3 uploads.
spec/mail_mcp/base64_stream_spec.rb	Tests incremental decoding.
spec/mail_mcp/imap_client_spec.rb	Tests per-part message retrieval.
spec/mail_mcp/message_structure_spec.rb	Tests MIME structure handling.
spec/spec_helper.rb	Loads the fake IMAP server.
spec/support/fake_imap_server.rb	Simulates BODYSTRUCTURE and section fetches.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[ivarsb]

Checked this against the old whole-message path with a multipart/related carrying Content-Disposition: attachment; filename="page.mht". The new output matches the old one exactly:

field	old (RFC822 + Mail)	new (per-part)
attachments	[]	[]
text_body	"real message body"	"real message body"
html_body	"<p>archived page</p>"	"<p>archived page</p>"

So there's no regression here — and the reason is that this is Mail's own behaviour rather than an accident of the rewrite. Mail::AttachmentsList (attachments_list.rb:8-19) only ever treats leaf parts as attachments:

elsif p.parts.empty?
  p if p.attachment?     # only leaves are candidates
else
  p.attachments          # a node with children recurses; itself never included
end

The multipart node's attachment? really is true and its filename really is "page.mht", but AttachmentsList still skips it. message/rfc822 is the one special case in that method, and it already arrives here as a leaf (BodyTypeMessage#multipart? is false), so forwarded-message attachments are picked up as before — there's a fixture for that.

Emitting the multipart node as an attachment would therefore diverge from Mail and add an attachment to responses the old path never returned. Since this PR's verification is a differential against that old output, I've left the semantics as-is and added a spec so they're deliberate rather than incidental (7d3d12e).

The second half of the observation is fair on its own terms: html_body can come from inside an attached archive. That predates this branch, matches Mail#html_part, and changing it changes what the tool returns — so it belongs in its own change rather than this one.

[ivarsb]

Correction to the original description

I've edited the description above. The measurement it originally leaned on does not hold on the platform this actually runs on, so flagging the change rather than quietly rewriting history.

What I originally claimed: RSS grew by one message size per fetch, linearly and without plateau, so no memory limit was safe and raising it only bought time.

What is actually true: that growth is specific to macOS malloc, where I first measured it. Re-run in ruby:3.4.9-slim, RSS plateaus on the first fetch and stays flat:

	boot	fetch 1	fetch 8
default arenas	42.8 MB	99.2 MB	99.3 MB
MALLOC_ARENA_MAX=2	42.8 MB	99.2 MB	99.3 MB

So this is a peak problem, not accumulation. Two consequences:

• Raising the memory limit does genuinely help, contrary to what I first wrote.
• MALLOC_ARENA_MAX=2, which I had recommended deploying alongside this, measured no effect on glibc here. Dropped from the checklist.

What is unchanged: the peak multiplier, which is the reason for the PR. The per-request peak scaled with message size (~2.7× wire on glibc, ~3.4× on macOS, and 6.7× counting Ruby-level live strings), and with threads 1, 5 that peak is per in-flight request — five concurrent 25MB-attachment fetches measured 614MB, which clears 384Mi by itself. After this change the same five peak at 221MB, and the largest single allocation is a 4MiB chunk regardless of message size, so the peak no longer tracks the message at all. That is what makes a fixed limit hold, and it is the part that does not depend on the allocator.

The ~10 minute restart cadence is therefore how often a burst of concurrent large fetches lands, not a steady climb.