Skip to content

Chunk vectored socket I/O at IOV_MAX instead of rejecting large iovec spans - #885

Open
majunze2001 wants to merge 1 commit into
google:mainfrom
majunze2001:jeff/iov-max-chunked-writev
Open

majunze2001 wants to merge 1 commit into
google:mainfrom
majunze2001:jeff/iov-max-chunked-writev

Conversation

@majunze2001

Copy link
Copy Markdown

Summary

peregrine::WriteVExact / ReadVExact returned
InvalidArgumentError("#iovs=N") for any span longer than IOV_MAX (1024 on
Linux). Sharded / hybrid KV cache resharding now resolves a single block into
thousands of small, non-contiguous byte spans, and the block transport hands
that whole list to these helpers in one call, so such transfers failed
outright.

This change makes the helpers loop over the span in IOV_MAX-sized pieces.
IOV_MAX bounds one writev(2) / readv(2) call, not the logical byte
stream. The test is rewritten so it builds in the OSS tree and gains cases at,
above, and far above the limit.

Root cause

  • The helpers in tpu_sync/transport/peregrine/src/api/socket_util.h forward
    to TcpSocketUtil::SendV / RecvV only when 1 <= n <= IOV_MAX, otherwise
    they error without touching the socket.
  • KVCacheManagerBase::GetBlockChunks (plan-driven path) emits one chunk per
    strided unit of every matching schedule entry. PR Route byte spans per destination unit for sharded KV cache resharding. #829 / 1eb2414 ("Route
    byte spans per destination unit for sharded KV cache resharding") made the
    planner produce per-destination-unit spans, so one block can resolve to
    many thousands of chunks. The plan-less path does the same for sparse pool
    regions via ComputePoolBlockCopyExtents.
  • block_transport.cc and socket_transport_adapter.cc convert those chunks
    to one iovec each and call the helpers once per block with no cap.
    (raw_buffer_transport.cc caps its batches at IOV_MAX itself and is not
    affected.)
  • Failure mode: sender writes the size header, then the payload call returns
    InvalidArgument; it logs "Write payload failed", shuts the socket and drops
    the send. Receiver side returns the same error and drops the connection.
    Deterministic hard failure, no corruption.

Fix

socket_util.h: when n > IOV_MAX, call SendV / RecvV on consecutive
IOV_MAX-sized sub-spans until the span is exhausted, returning the first
non-OK status. Correctness rests on two facts:

  • TcpSocketUtil::SendV / RecvV already loop until every byte of their
    sub-span has moved (they handle partial writev / readv returns).
  • TCP is an ordered byte stream, so sender and receiver batch boundaries need
    not agree.

Each sub-span still satisfies the DCHECK_LE(iovecs.size(), IOV_MAX) inside
the inner helpers. The n == 0 case still returns InvalidArgument.

Tests

socket_util_test.cc is rewritten:

  • The old test depended on //tpu_sync/transport/peregrine/src/util
    (RandomNonZero), which depends on //third_party/xxhash. That target does
    not exist in this repository, so the test could not build here. The new
    version uses only in-tree APIs (IpAddr::Create, TcpSocket::Create, port 0
    plus SelfAddrPort to discover the listening port) and a deterministic
    1 + i % 251 byte pattern instead of RNG.
  • The read/write check is parameterised by iovec count. New cases:
    ReadWriteAtIovMax (exactly IOV_MAX on both sides),
    ReadWriteAboveIovMax (IOV_MAX + 1 read vs 2 * IOV_MAX + 3 write) and
    ReadWriteManyIovs (16 * IOV_MAX + 3 vs 14 * IOV_MAX + 1, above the
    fragment counts seen in hybrid-state resharding). All run across
    IPv4/IPv6 x plain/vectored read x plain/vectored write.
  • api/BUILD: test deps swap internal/util:test_util and src/util for
    internal/base:ipaddr and internal/socket:socket_util.

Testing performed

With bazel 8.6.0, --config=oss, clang 18:

  • //tpu_sync/transport/peregrine/src/api:socket_util_test: passes, 32/32
    cases across all parameterisations.
  • socket_transport_adapter_test, chunk_serializer_test,
    peregrine_control_service_test: pass.
  • block_transport_test and raw_buffer_transport_test do not build in this
    tree as-is (pre-existing: missing //third_party/xxhash, and google3-only
    ASSERT_OK-family macros). Built with local, uncommitted shims,
    block_transport_test passes; raw_buffer_transport_test has one flaky
    case (PushBuffersTriggersOnDataReceivedWhenChunksExceedOrEqualExpected)
    that fails at the same rate without this change and never reaches the new
    code path, since that transport caps batches at IOV_MAX.

… spans.

WriteVExact/ReadVExact returned InvalidArgument for spans longer than
IOV_MAX (1024). Per-destination-unit resharding (1eb2414) resolves a
single block into thousands of small chunks, and block_transport.cc /
socket_transport_adapter.cc pass the whole list in one call, so those
transfers failed outright. IOV_MAX bounds one writev/readv call, not the
byte stream: loop over IOV_MAX-sized sub-spans, relying on SendV/RecvV
being exact within each sub-span and on TCP ordering across them.

Rewrite socket_util_test.cc so it builds in this tree (it depended on
//tpu_sync/transport/peregrine/src/util, which needs the absent
//third_party/xxhash) and add cases at, above and far above IOV_MAX.

Claude-Session: https://claude.ai/code/session_012cbF1oLqDSJhgcQS3BBhiy
Signed-off-by: Jeff Ma <jeffjma@umich.edu>
@google-cla

google-cla Bot commented Sep 8, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant