Syscall failure injection for backend error-path coverage

[sgerbino]

Problem

Most uncovered lines in the published coverage reports are the
if (r < 0) branches after a system call. Those branches only execute
under resource exhaustion or kernel misconfiguration, which no test can
provoke portably. Because CI merges with --merge-mode-functions separate, every backend's copy of the same error branch counts
separately, so the gap is multiplied by the number of backends.

Proposal

Inject failures from the test executable only. The library is not
touched: no wrapper namespace, no macro, no extra indirection. The unit
test target gains one translation unit per platform that shadows the OS
entry points corosio calls, forwards to the real function by default,
and fails the call when a test has armed a fault.

How the shadowing works

• Linux / FreeBSD. The test TU defines extern "C" functions with
  the libc names (socket, epoll_ctl, kevent, close, …). Static
  build: every call binds to the executable's definition at link time.
  Shared build: libboost_corosio.so calls libc through its PLT, and
  the executable is first in the dynamic linker's lookup scope, so the
  same definition interposes the library's calls — what LD_PRELOAD
  does, without the preload. The real function is fetched once with
  dlsym(RTLD_NEXT, name). Precondition: the library is never linked
  with -Bsymbolic-functions or -fno-plt (it is not today); the
  startup self-check fails loudly if that changes.
• macOS. Static build works like ELF. Shared build does not:
  two-level namespace binds the dylib's _socket to libSystem at
  link time, so an executable-level definition is never consulted. The
  harness rebinds the corosio dylib's __got / __la_symbol_ptr
  entries at startup (fishhook-style, ~150 lines; the Mach-O twin of
  the Windows IAT walk). A __DATA,__interpose section in the
  executable may be enough on current dyld and should be tried first,
  with rebinding as the fallback.
• IOCP completion errors. Unlike io_uring, a failed overlapped
  completion is delivered through GetQueuedCompletionStatus, itself
  an IAT import, so the hook rewrites the returned entry's error before
  corosio sees it. No memory-ring trick needed.
• liburing. Three kinds of io_uring error branch, three levers:
  1. Exported calls returning < 0 (io_uring_queue_init_params,
     io_uring_submit, io_uring_submit_and_wait_timeout,
     io_uring_submit_and_get_events, io_uring_wait_cqe_timeout,
     io_uring_enter) shadow like libc (archive order if liburing.a,
     interposition if .so).
  2. io_uring_get_sqe returning null is static inline arithmetic on
     the user-mapped SQ and has no symbol. The harness reaches it by
     shadowing io_uring_queue_init_params to request entries = 1
     and shadowing io_uring_submit to return 0 without forwarding,
     so the SQ stays full across corosio's submit-and-retry.
  3. cqe->res < 0 is read from the mmap'd CQ ring, not returned by any
     function. The shadows for io_uring_wait_cqe_timeout /
     io_uring_submit_and_get_events forward to the real call, then
     walk the visible CQEs (io_uring_for_each_cqe, inline) and
     overwrite res for the CQE whose user_data matches the armed
     op. The ring is mapped read-write, so this is a plain store and
     corosio dispatches the rewritten result as if the kernel produced
     it. Only io_uring_get_sqe among the inlines can fail; prep_*,
     cqe_get_data, cq_advance never do and need no hook.
• Windows. Win32 imports are called through the IAT
  (call [__imp_WSASocketW]), so a same-named definition does not
  intercept. The test TU patches the import table of every module that
  contains corosio code — the executable and, in shared builds,
  boost_corosio.dll — for the ws2_32 / kernel32 / ntdll entries
  listed below (walk IMAGE_DIRECTORY_ENTRY_IMPORT, VirtualProtect,
  swap the pointer; ~60 lines, no Detours). AcceptEx, ConnectEx and
  NtSetInformationFile are obtained at runtime through WSAIoctl and
  GetProcAddress, so hooking those two lets the harness hand back
  wrappers for the rest.

No production symbol changes, no ABI change, no cost in the library.

One target, either link mode

All backend code is instantiated inside the library's own translation
units (io_context.cpp, tcp_socket.cpp, …), so in a shared build
every syscall corosio makes is issued from a single image. The harness
therefore targets "every image containing corosio code": the
executable in a static build, the executable plus
libboost_corosio.{so,dylib,dll} in a shared build. Nothing about it
depends on BUILD_SHARED_LIBS.

The injection tests still live in their own target,
boost_corosio_fault_tests, linked against corosio in whatever mode
the build selects. Keeping them out of boost_corosio_tests keeps the
close / read / write hooks out of the regular suite, even though
they are transparent when unarmed. The harness is never part of the
library and never installed. All 37 ci.yml legs (28 shared, 9
static) plus FreeBSD (link=shared) run it; the three coverage legs
(GCC 15 Linux, Apple-Clang macOS, MinGW Windows) are static, so the
published numbers move without any matrix change. Note the Windows
coverage leg is MinGW, not MSVC: the IAT walk must work against
MinGW's PE output first.

A startup self-check walks every corosio image and asserts each census
symbol is bound to the hook (symbol readback on ELF, GOT readback on
Mach-O, IAT readback on PE). A library link-flag change that defeats
interposition fails the whole fault suite at once rather than leaving
dead injections.

Arming an injection

// test/unit/fault/fault.hpp
enum class sys { socket, bind, listen, accept, /* ... */ };

struct fault_scope
{
    // fail the nth matching call on this thread with err, then disarm
    fault_scope(sys which, int err, unsigned nth = 1);
    // succeed the nth matching call with a short count instead: the
    // hook forwards the real call with its length truncated to
    // `count`, so the bytes are genuinely moved; 0 on a read forwards
    // nothing and reports EOF
    static fault_scope returning(sys which, std::size_t count,
                                 unsigned nth = 1);
    ~fault_scope();                 // disarms even if never fired
    bool fired() const noexcept;
};

// io_uring only: rewrite the completion for `user_data` to `res`
// (negative errno, or a short non-negative count) before corosio
// sees it
struct cqe_fault_scope
{
    cqe_fault_scope(void const* user_data, int res);
    ~cqe_fault_scope();
    bool fired() const noexcept;
};

• err is errno-style on POSIX, WSA* / ERROR_* on Windows; the
  hook sets errno / WSASetLastError / SetLastError and returns the
  function's documented failure value (-1, INVALID_SOCKET,
  INVALID_HANDLE_VALUE, nullptr, FALSE, or a negative -errno
  for liburing).
• Thread-local, so parallel ctest and multi-threaded tests never see
  each other's faults.
• fired() lets a test assert the injection was reached, so a refactor
  that removes the call site fails the test instead of leaving a dead
  injection.
• One armed fault per thread; nesting asserts.
• EAGAIN / EWOULDBLOCK / EINTR / ERROR_IO_PENDING are ordinary
  faults with those codes; they reach the would-block re-arm and
  retry branches (~80 sites) with no special support.
• returning(...) covers the count-dependent branches: 0 for the
  EOF paths (bytes_transferred == 0, res == 0) and short positive
  counts for the *_some contracts. corosio has no internal
  short-count loops, so this is the whole surface. On completion
  backends a rewritten short count leaves bytes consumed but
  unreported; such tests assert on the branch taken, not on payload.

{
    test::fault_scope f(test::sys::epoll_ctl, EPERM);
    tcp_socket s(ioc);
    auto r = s.open(tcp::v4());
    BOOST_TEST(f.fired());
    BOOST_TEST_EQ(r.error(), make_error_code(std::errc::operation_not_permitted));
    BOOST_TEST(!s.is_open());
}

Inventory to shadow

Census of every OS entry point corosio calls today (include/ +
src/corosio/src/), with call-site counts. The harness must cover the
whole list; anything not on it that appears later is a review item.

POSIX common (posix/, reactor/, src/): socket(11) socketpair(1)
bind(17) listen(10) accept(4) accept4(2) connect(4)
getsockname(38) getpeername(10) getsockopt(18) setsockopt(31)
shutdown(9) close(82) read(5) write(7) readv(4) writev(1)
preadv(2) pwritev(2) recv(2) send(3) recvmsg(6) sendmsg(10)
poll(2) pipe(2) fcntl(21) ioctl(2) open(4) fstat(6)
lseek(1) ftruncate(4) fsync(8) fdatasync(4) posix_fadvise(4)
unlink(1) sigaction(3) signal(4) getaddrinfo(1)
freeaddrinfo(1) getnameinfo(1) gethostname(1)

epoll: epoll_create1(1) epoll_ctl(4) epoll_wait(1) eventfd(2)
timerfd_create(1) timerfd_settime(1)

select: select(1) (interrupter shares pipe/fcntl above)

kqueue: kqueue(1) kevent(5)

io_uring (shadowable): io_uring_queue_init_params(1)
io_uring_queue_exit(4) io_uring_submit(10)
io_uring_submit_and_wait_timeout(1) io_uring_submit_and_get_events(2)
io_uring_wait_cqe_timeout(1) io_uring_enter(1), plus eventfd
above. Inline, reached indirectly: io_uring_get_sqe(12) via the
ring-size clamp + no-op submit. Not a call at all: cqe->res (22
branches in the op files) via CQ rewrite.

IOCP: WSAStartup(1) WSACleanup(1) WSASocketW(10)
WSAConnect(1) WSARecv(5) WSASend(3) WSARecvFrom(1) WSASendTo(1)
WSAPoll(1) WSAIoctl(2) closesocket(38) ioctlsocket(2)
CreateIoCompletionPort(17) GetQueuedCompletionStatus(3)
PostQueuedCompletionStatus(9) CancelIoEx(51) CloseHandle(9)
CreateFileW(2) ReadFile(2) WriteFile(2) SetFilePointerEx(2)
GetFileSizeEx(4) SetEndOfFile(4) FlushFileBuffers(4)
DeleteFileA(1) CreateWaitableTimerW(1) SetWaitableTimer(2)
WaitForSingleObject(1) GetAddrInfoExW(1) GetAddrInfoExCancel(1)
FreeAddrInfoExW(2) GetNameInfoW(1) GetComputerNameExW(2)
GetModuleHandleA/W(3) GetProcAddress(3) MultiByteToWideChar(2)
WideCharToMultiByte(4); via function pointer: AcceptEx, ConnectEx,
NtSetInformationFile.

Not in scope, not syscalls: Interlocked*, *CriticalSection,
htonl, WSAGetLastError/GetLastError/SetLastError (the hooks set
these, never fail them).

Known sharp edges

• _FORTIFY_SOURCE redirects read, recv, poll, pread* to
  __*_chk; the test target either builds without fortification or
  shadows the _chk names too. Verify with nm on the test binary.
• Hooks for close, read, write, fcntl are also hit by the test
  framework, the sanitizer runtime and libstdc++; the thread-local arm
  plus per-symbol matching keeps them transparent, but the hook must
  never allocate or lock before forwarding.
• macOS x86_64 uses $INODE64 / $UNIX2003 symbol variants for
  fstat, select, connect, etc.; CI is arm64 where these are gone,
  but the hook TU should shadow both spellings so a local Intel build
  still works.
• With /GL or LTCG on MSVC the IAT walk is unchanged; only the
  __imp_ data-symbol trick would be fragile, which is why the
  proposal patches the IAT at runtime instead.
• Sanitizer legs (8 of 37) intercept the same libc functions the
  harness shadows. With a statically linked sanitizer runtime (Clang's
  default) the interceptors are non-weak symbols in an always-pulled
  archive member, so the harness's close / read / send collide at
  link time — the same reason a user malloc fails to link under
  ASan. With a dynamic runtime (GCC's default, -shared-libsan on
  Clang) the executable's definition wins and dlsym(RTLD_NEXT)
  forwards through the sanitizer's interceptor, so ASan still sees
  every call. TSan models fd happens-before in those interceptors, so
  bypassing them yields false races; forwarding through RTLD_NEXT
  preserves it only in the dynamic case. Windows ASan patches the IAT
  itself and would fight the harness for the same slots. Rule: build
  boost_corosio_fault_tests on sanitizer legs only with a dynamic
  runtime; expected-skip under Windows ASan and under TSan. None of
  the sanitizer legs is a coverage leg, so the published numbers do
  not depend on this.
• Shared macOS is the one configuration where hooking is not a
  link-time property. If __interpose from the executable proves
  unreliable and GOT rebinding fights chained fixups on some SDK, the
  fallback is to mark that single leg's fault suite as expected-skip
  while every other leg runs it; the macOS coverage leg is static and
  unaffected either way.

Scope

Every backend error branch reachable only through a failing OS call
gets a test that injects the failure and checks (a) the reported
error_code, (b) that the object is in the documented post-failure
state, and (c) that no descriptor/handle leaks (ASan/LSan plus the
existing pending_io_ drain assertions on IOCP).

• Harness: test/unit/fault/fault.hpp, fault_posix.cpp,
  fault_uring.cpp, fault_win.cpp; boost_corosio_fault_tests
  target in CMake + Jamfile, either link mode; self-tests
  (fires on nth, disarms on scope exit, thread-isolated, forwards
  transparently when unarmed, returning truncates and forwards, _chk coverage, every census symbol
  bound to the hook in every corosio image — symbol / GOT / IAT
  readback at startup, in both link modes)
• epoll: create, ctl add/mod/del, eventfd, timerfd, traits socket ops
• select: interrupter pipe/fcntl, select
• kqueue: kqueue, kevent register + wait, pipe
• io_uring: queue_init, submit* < 0, wait_cqe_timeout < 0, SQ-full
  via clamp + no-op submit (wakeup, signal reader, op submit),
  enter, queue_exit paths, every cqe->res < 0 branch not already
  reached by real cancel/reset via cqe_fault_scope
• iocp: WSASocketW, CreateIoCompletionPort, AcceptEx, ConnectEx,
  WSARecv/WSASend/From/To, CreateFileW, CancelIoEx, waitable timer,
  GetAddrInfoExW, NtSetInformationFile dissociation
• posix common: resolver, signal pipe, stream + random-access file
  ops, socket options, local_connect_pair, host_name
• Each backend PR links the coverage delta it produced

Test files live in test/unit/fault/<backend>_faults.cpp and use the
existing context.hpp backend-tag templating so one body runs per
available backend.

Non-goals

• LD_PRELOAD, seccomp, Detours, or any external tooling.
• TLS engine faults (SSL_* / wolfSSL); separate layer, separate
  issue.

Acceptance

• Linux, macOS and Windows coverage badges each rise; closed when no
  backend syscall-guarded branch remains uncovered in the published
  report.
• Zero diff under include/ and src/ from the harness itself.
• boost_corosio_fault_tests builds and passes on every ci.yml leg
  and on FreeBSD, in both static and shared link modes, except the
  documented expected-skips (Windows ASan, TSan, and shared macOS if
  interposition proves unreliable). Every skip is explicit in the
  matrix generator, never a silent no-op.