diff --git a/.agents/sow/current/SOW-0027-20260629-netdata-vendor-memory-safety-update.md b/.agents/sow/current/SOW-0027-20260629-netdata-vendor-memory-safety-update.md index 50c8c157..5fc8fb1f 100644 --- a/.agents/sow/current/SOW-0027-20260629-netdata-vendor-memory-safety-update.md +++ b/.agents/sow/current/SOW-0027-20260629-netdata-vendor-memory-safety-update.md @@ -2,9 +2,12 @@ ## Status -Status: in-progress +Status: paused -Sub-state: Netdata vendoring exposed a missing public L3 cache enumeration API required by the eBPF cgroup consumer; implementation is paused for a user design decision. +Sub-state: paused on 2026-07-12 so SOW-0030 can take the active slot for the +fleet-observed SHM SIGBUS crash. The earlier L3 API decision was resolved by using the +direct L2 snapshot path; resume from final validation, artifact maintenance, review, +and lifecycle closure for the already merged Netdata vendor update. ## Requirements diff --git a/.agents/sow/current/SOW-0030-20260712-shm-sigbus-on-full-backing-store.md b/.agents/sow/current/SOW-0030-20260712-shm-sigbus-on-full-backing-store.md new file mode 100644 index 00000000..7743a063 --- /dev/null +++ b/.agents/sow/current/SOW-0030-20260712-shm-sigbus-on-full-backing-store.md @@ -0,0 +1,531 @@ +# SOW-0030 - SHM region creation crashes with SIGBUS when the backing store is full + +## Status + +Status: in-progress + +Sub-state: active; long-term-best design approved by the user and pre-implementation +gate complete. Linux and native Windows/MSYS validation are complete. Source pull-request +CI/scanner evidence and Netdata re-vendoring remain. + +## Requirements + +### Purpose + +Stop netipc SHM region creation from crashing the host process (SIGBUS) when the +shared-memory backing store (tmpfs `/dev/shm`, or a disk-backed directory) has no free +space. Creation must fail gracefully with an error the caller can handle, in all three +language implementations (C, Rust, Go), preserving cross-language interoperability. + +### User Request + +The user analyzed the crashes of the latest Netdata nightlies reported to the +agent-events pipeline and asked for a SOW in this repository so the fix is implemented +here (upstream) and then re-vendored into netdata/netdata. + +### Assistant Understanding + +Facts: + +- Netdata fleet evidence (agent-events crash pipeline, 48h window, nightlies + v2.10.0-733 / v2.10.0-738, 2026-07-10..12): 765 crash events total; the largest + real-code-bug cluster is ~22 events from ~15 distinct agents, all + `SIGBUS/BUS_ADRERR`, in netipc server threads named `P[cglkupipc]` and + `P[cgroupsipc]` (the Netdata cgroups lookup/cache IPC services). +- Representative decoded stack (from a glibc aarch64 build): + `memset` -> `nipc_shm_server_create` (netipc_shm.c:372) -> + `server_prepare_accept_config` (netipc_service_posix_server.c:74) -> + `nipc_server_run` (netipc_service_posix_server.c:206). + Several more events show only musl frames (`src/string/x86_64/memset.s`) consistent + with the same zero-fill site on static builds. +- Root cause (C): `src/libnetdata/netipc/src/transport/posix/netipc_shm.c` + - `:355` `ftruncate(fd, region_size)` — on tmpfs this creates a **sparse** file and + succeeds regardless of available space; + - `:362` `mmap(PROT_READ|PROT_WRITE, MAP_SHARED)`; + - `:372` `memset(map, 0, region_size)` — the first write to a page that tmpfs cannot + back raises `SIGBUS` (`BUS_ADRERR`). This is documented POSIX/Linux mmap behavior + when the backing object cannot provide the page. +- The same pattern exists in the other two implementations: + - Rust: `src/crates/netipc/src/transport/shm.rs:272` (`libc::ftruncate`) and `:303` + (`ptr::write_bytes(base, 0, region_size)`); + - Go: `src/go/pkg/netipc/transport/posix/shm_linux.go:174` (`syscall.Ftruncate`), + followed by mapped-region writes. +- The crashing binary is the vendored copy in `netdata/netdata @ 5cd588067e` + (`src/libnetdata/netipc/src/transport/posix/netipc_shm.c`, same line numbers as this + repository), so the fix must land here first and then be re-vendored + (`diff-netdata-vendor.sh` exists at the repo root for the sync). +- The POSIX SHM transport is Linux-only in the authoritative contract and in the + Rust and Go build gates. FreeBSD and macOS use baseline UDS. +- Native Windows and Windows under `MSYSTEM=MSYS` use the separate Windows + Named Pipe / file-mapping implementation. They do not use the Linux file-backed + SHM path, but remain mandatory regression-validation targets. + +Inferences: + +- The affected hosts had a full (or nearly full) `/dev/shm` / backing filesystem at + region-creation time. SIGBUS at first-write is the classic sparse-file-on-tmpfs + failure mode; ~15 distinct agents across two nightlies in 48h makes a code-side + memory-corruption explanation unlikely. +- After a successful create (full zero-fill completed), all pages are committed, so + the SIGBUS window is region creation. A client attaching to a region created by a + fixed server cannot SIGBUS on unbacked pages; mixed-version scenarios (old server, + new client) retain the old behavior until the vendor copy is updated. + +Resolved investigation: + +- C, Rust, and Go managed servers already remove SHM profiles and continue over + baseline when SHM preparation fails and baseline is configured. Existing obstruction + tests prove the generic fallback; this SOW must add allocation-specific coverage. +- All known server implementations zero the complete region before publishing a valid + header. An old server that fails during zero-fill cannot complete a successful SHM + negotiation, while a successfully published old region has already backed every page. +- Deterministic validation requires both allocation fault injection and a real + size-limited Linux tmpfs test in child processes. + +### Acceptance Criteria + +- With a full SHM backing store, region creation in C, Rust, and Go returns a normal + error (no SIGBUS, no crash); verified by a test that constrains the backing store + (e.g. a size-limited tmpfs mount in a user/mount namespace on Linux) or an + equivalent deterministic simulation. +- When baseline is configured, the consuming service keeps operating without SHM + acceleration when region creation fails (verified against the C service server used + by Netdata's cgroups IPC). An explicitly SHM-only configuration may fail because it + has no remaining transport. +- All three implementations keep identical region layout and interop behavior + (existing interop tests still pass). +- Error taxonomy stays consistent across languages (new or existing error code used + uniformly; docs under `docs/` updated if the error surface changes). +- Vendor follow-up tracked: re-vendor into netdata/netdata after merge (tracked as a + follow-up item here, executed in the netdata repository). +- Native Windows and `MSYSTEM=MSYS` build/tests remain green; no Linux allocation API + or error is introduced into the Windows SHM implementation. + +## Analysis + +Sources checked: + +- This repo: `src/libnetdata/netipc/src/transport/posix/netipc_shm.c` (:336-:399), + `src/libnetdata/netipc/src/service/netipc_service_posix_server.c` (:74, :206), + `src/crates/netipc/src/transport/shm.rs` (:211-:310), + `src/go/pkg/netipc/transport/posix/shm_linux.go` (:174). +- netdata/netdata @ 5cd588067e — vendored copy and fleet crash correlation (Netdata + agent-events pipeline; evidence summarized above, raw records remain in the Netdata + workstation checkout under its local-only audit directory, not in this repository). + +Current state: + +- All three implementations size the region with `ftruncate` only, then write to the + mapping. On tmpfs, allocation is deferred to first write, converting ENOSPC into an + uncatchable-by-default SIGBUS inside library code, crashing the embedding process + (the Netdata daemon in the fleet evidence). + +Risks: + +- Linux filesystems or sandbox policies that return an ordinary error from native + `fallocate()` disable SHM for that session and use baseline when available. A strict + seccomp policy using `KILL_PROCESS`/`KILL_THREAD` terminates, while `TRAP` delivers + `SIGSYS` before NetIPC can receive an error. NetIPC installs no process-global SIGSYS + emulation handler, and `USER_NOTIF`/`TRACE` supervisor behavior is outside its + contract; strict policies must explicitly allow `fallocate`. This is an intentional + safety boundary: do not substitute a weaker allocation mechanism where the no-SIGBUS + guarantee cannot be proven. +- Behavior parity: the Windows transport (`netipc_win_shm.h` family) uses a different + section-object allocation model. It is not changed, but native Windows and MSYS + regression validation remains mandatory. +- The current CMake `NOT APPLE` SHM gates are broader than the Linux-only public + contract and include FreeBSD. That pre-existing build-contract discrepancy will be + tracked separately rather than expanding this fleet-crash repair. + +## Pre-Implementation Gate + +Status: ready + +Problem / root-cause model: + +- Sparse allocation (`ftruncate`) + lazy tmpfs page allocation + first-write zero-fill + = SIGBUS on full backing store, crashing the host process. Evidence: fleet cluster + (~22 events / ~15 agents / 48h) with stacks pinned to the zero-fill instruction; + identical pattern present in C, Rust, and Go transports. + +Evidence reviewed: + +- See Analysis. External: POSIX `mmap` specification and Linux `mmap(2)` SIGBUS + semantics for writes beyond the backing object's allocated space. +- netdata/netdata @ 5cd588067e + src/libnetdata/netipc/src/transport/posix/netipc_shm.c:355 (vendored copy). +- Microsoft `CreateFileMappingW` documentation: a page-file-backed mapping created with + `PAGE_READWRITE` and no explicit allocation attribute assumes `SEC_COMMIT`; the system + must have enough committable pages for the whole mapping or creation fails normally. +- C, Rust, and Go Windows creators all use `INVALID_HANDLE_VALUE` page-file backing and + return existing create-mapping/map-view errors before touching the view when those APIs + fail. +- Linux kernel seccomp documentation: `ERRNO` directly returns an error, + `KILL_PROCESS`/`KILL_THREAD` terminate, `TRAP` delivers `SIGSYS`, and + `USER_NOTIF`/`TRACE` depend on a supervisor or tracer: + . +- moby/profiles @ f9bc03ec19b2dc4c091449b08e88f85c0caa9f0b + `seccomp/default.json:2,112` uses `SCMP_ACT_ERRNO` by default and explicitly allows + `fallocate`, so the common Docker default profile satisfies the requirement. + +Affected contracts and surfaces: + +- C API error codes of `nipc_shm_server_create()` (and possibly client attach); Rust + `ShmError`; Go transport errors; docs under `docs/` describing transport errors; + interop and unit tests; the netdata vendored copy (follow-up). + +Existing patterns to reuse: + +- Existing stage-based error taxonomy and cleanup paths. Add an allocation-stage error + rather than misreporting a native allocation failure as truncation or encoding only + one OS cause such as ENOSPC. +- Cleanup-on-failure paths already exist at each create step (close/unlink pattern in + `netipc_shm.c:355-:368`); the fix slots in as one more failable step. + +Risk and blast radius: + +- Small, localized change per language; no wire-format or layout change. Main risk is + platform portability of preallocation and CI coverage for the failure path. + +Sensitive data handling plan: + +- Fleet evidence is summarized in aggregate only; no agent identifiers, hostnames, + IPs, or customer-identifying details are recorded in this SOW or in any durable + artifact of this repository. Raw crash records stay in the Netdata workstation's + local-only audit directory. + +Implementation plan: + +1. C, Rust, and Go Linux creators: replace sparse-only sizing with native Linux + `fallocate(fd, 0, 0, region_size)` before `mmap`, retrying EINTR. Do not use + `ftruncate`, libc-emulated `posix_fallocate`, or a write-loop fallback. Unsupported + allocation disables SHM and preserves the stronger safety guarantee. +2. Keep the existing mapped zero-fill after successful allocation to initialize the + complete region and atomics consistently. +3. Add a general allocation-stage error while preserving existing public values: + C `NIPC_SHM_ERR_ALLOCATE` appended to the enum, Rust `ShmError::Allocate(errno)`, + and Go `ErrShmAllocate` with the underlying OS error preserved for `errors.Is`. +4. Do not add client-side preallocation or a process-global SIGBUS handler. The known + mixed-version lifecycle cannot publish an under-backed valid region. +5. Add allocation-specific service tests proving normal fallback to baseline when + baseline is configured, without changing the existing SHM-only semantics. +6. Add deterministic per-language fault injection and a real size-limited Linux tmpfs + subprocess test that proves normal error return and no SIGBUS. +7. Re-run POSIX interop and native Windows/MSYS regression validation. Windows transport + code remains unchanged unless validation reveals an evidence-backed equivalent gap. +8. Update the Linux SHM lifecycle and allocation-error contract under `docs/`. +9. Follow-up: re-vendor into netdata/netdata through the mandatory preflight workflow. + +Validation plan: + +- Per-language allocation-helper fault injection for ENOSPC, EINTR retry, cleanup, + error classification, and proof that mmap/mapped writes are not reached on failure. +- Size-limited tmpfs via a private user/mount namespace or a dedicated privileged Linux + CI job. Run C, Rust, and Go creators as child processes and assert normal failure, + no SIGBUS, and no leaked region file. The real-kernel test may skip only on jobs not + designated to provide namespace/mount coverage; at least one Linux CI path must run it. +- Full interop matrix (C/Rust/Go) unchanged behavior on the success path. +- Native Windows and `MSYSTEM=MSYS` build/tests for the Named Pipe and Windows SHM + transports, proving Linux-only changes did not leak into Windows builds. +- Same-failure search: audit every other mapped-region write path (client attach, + Windows section mapping, any `mremap`/grow paths) for the same lazy-allocation gap. + +Artifact impact plan: + +- AGENTS.md: likely unaffected. +- Runtime project skills: likely unaffected. +- Specs: transport spec under `.agents/sow/specs/` (or `docs/`) — update error + semantics for region creation. +- End-user/operator docs: likely unaffected (library-internal). +- End-user/operator skills: none present. +- SOW lifecycle: follow-up item for the netdata re-vendor; close via `done/` when + merged here. + +Open-source reference evidence: + +- netdata/netdata @ 5cd588067e + src/libnetdata/netipc/src/transport/posix/netipc_shm.c:355,372 (vendored copy that + produced the fleet crashes). + +Open decisions: + +- None. The user delegated all technical choices to the assistant with an explicit + requirement to choose the long-term-best design without shortcuts. + +## Implications And Decisions + +- User decision (2026-07-12): choose the long-term-best option for every technical + fork without shortcuts; keep both native Windows and `MSYSTEM=MSYS` support in scope. +- Decision: Linux uses direct native `fallocate()` with no weaker fallback. On + unsupported filesystems or policies that return errno, SHM preparation fails normally + and managed services use baseline when configured. NetIPC cannot automatically recover + from kill/trap seccomp actions and strict policies must allow `fallocate`. +- Decision: add a general allocation-stage public error across C, Rust, and Go; + preserve the OS cause as C `errno` and in Rust/Go error values, and append the C enum + value to avoid renumbering existing public values. +- Compatibility implication: adding Rust `ShmError::Allocate(i32)` breaks downstream + exhaustive matches over this public enum. The crate is still pre-1.0 (`0.1.0`), the + new stage must be distinguishable, and preserving `ShmError::Truncate` avoids breaking + callers that construct or selectively match the old variant. Making only this enum + `#[non_exhaustive]` would impose the same one-time source break while leaving the + project's other public error enums inconsistent; broader enum-evolution policy is + outside this production crash repair. +- Decision: do not modify client attach or install SIGBUS recovery. Known servers + cannot publish a valid under-backed region, and a process-global signal handler is + incompatible with an embeddable library's blast-radius requirements. +- Decision: validation has two mandatory layers: deterministic allocation fault + injection and real size-limited tmpfs subprocess coverage. +- Decision: the Linux fix must not alter native Windows or MSYS transport behavior; + both remain mandatory regression targets. +- Decision: track the broader CMake Linux-gating inconsistency separately so this + production crash repair stays focused. + +## Plan + +1. Activate this SOW after pausing the previous active SOW. +2. Implement the Linux allocation helper and allocation error in C, Rust, and Go. +3. Add allocation-failure, cleanup, service-fallback, and real tmpfs tests. +4. Update the authoritative Linux SHM contract and related operator/integrator guidance. +5. Run POSIX, cross-language interop, native Windows, and MSYS validation. +6. Run same-failure and reviewer passes, then resolve all findings. +7. Re-vendor through the project vendoring preflight and validate Netdata. + +## Execution Log + +### 2026-07-12 + +- SOW created from Netdata fleet crash analysis (agent-events pipeline, nightlies + v2.10.0-733/738, 48h window): SIGBUS cluster attributed to sparse SHM allocation. + No implementation started. +- Researched official Linux/POSIX allocation and mmap behavior, the complete C/Rust/Go + lifecycle and fallback paths, and mature open-source implementations. +- Reproduced the allocation distinction in a private 64 KiB tmpfs: sparse 128 KiB + `truncate` succeeded with zero allocated blocks while 128 KiB `fallocate` returned + ENOSPC normally. +- Recorded the user's long-term-best authorization and the resolved design above. +- Verified the Windows allocation contract against Microsoft documentation and all three + implementations. Native Windows and MSYS page-file-backed mappings default to + `SEC_COMMIT`; insufficient system commit fails through `CreateFileMappingW` or + `MapViewOfFile`, and the existing language-specific error paths return normally. + No Windows allocation change is justified by this failure class. +- Replaced sparse `ftruncate` creation with direct Linux `fallocate` in C, Rust, and Go. + All three retry only `EINTR`, reject unsupported allocation without a sparse fallback, + clean up before `mmap`, and retain complete mapped-region zero-fill. +- Added the allocation-stage error surface while preserving legacy values/symbols: + appended C `NIPC_SHM_ERR_ALLOCATE`, Rust `ShmError::Allocate(errno)`, and Go + `ErrShmAllocate` wrapping the OS error. +- Added deterministic C/Rust/Go tests for `ENOSPC`, `EINTR`, unsupported allocation, + cleanup, and success after retry. C allocation faults use a test-only linker wrapper; + the production `netipc_shm` library exports no fault-control symbols and does no + test-state work. +- Added the managed C service regression proving allocation failure removes SHM from + negotiation, selects baseline, and completes a typed cgroups snapshot call. +- Added a private size-limited tmpfs regression for all three real creator binaries. + The first reviewer pass found that the initial script accepted unrelated ordinary + failures; strengthening it to require a shared `NIPC_SHM_ALLOCATE_ENOSPC` marker + exposed a real pre-allocation directory-permission false positive. The mount is now + private mode `0700`, and the test requires exit 1, the exact marker, no `READY`, no + signal/timeout/invocation failure, and no leaked file. +- Added a dedicated required Linux CI job for the real tmpfs test. Ordinary local CTest + recognizes exit 77 as skipped when namespaces are unavailable; the designated CI job + runs with `NIPC_REQUIRE_TMPFS_TEST=1` and a privileged mount-only namespace. Both the + unprivileged developer path and exact privileged CI command pass locally. +- Updated the Linux and Windows authoritative SHM documents, integrator guidance, and + coverage exclusions. Added pending SOW-0031 for the pre-existing CMake Linux-gating + discrepancy without mixing that separate scope into this repair. +- Committed and pushed the reviewed implementation as + `dd3c546a90789c96c8eb31bd0b0f97c5a9b70a4c` on + `fix/shm-full-backing-store`. +- Ran the complete native Windows/MSYS validation on `win11` from a dedicated checkout + pinned to that exact commit. The MSYS C build, native Rust and Go binaries, all 12 + functional/stress/interop cases, and 10 consecutive `test_win_shm` runs passed. The + 11-row paired MinGW/MSYS benchmark policy also passed, including C/Rust mixed SHM and + snapshot directions. +- Began the mandatory Netdata vendoring preflight. The last source-to-Netdata baseline is + plugin-ipc `98ec474d103ab2a7ed150f88c37225b74e1a9d27` copied by Netdata commit + `ca9a5426a6b1cdf6e1b2c82d3d5710e9e0a66fed`. Netdata `origin/master` at + `2329145de31cd9f5ed25f12b1816415f02772a3e` has no downstream changes in normalized + vendored sources since that baseline; its only NetIPC-path change is the excluded + Netdata wrapper `netipc_netdata.h`. +- Two-way drift is fully classified. Upstream changed exactly six vendored files since + the baseline: the C SHM header/implementation, Rust SHM implementation/tests, and Go + Linux SHM implementation/tests. `diff-netdata-vendor.sh` against current Netdata + reports exactly those same six differences and no unexplained C, Rust, or Go drift. +- Repository scanners report zero open Dependabot alerts and zero open secret-scanning + alerts. The one open code-scanning alert, 7750, is a stale historical result rather + than a current defect: its last Semgrep analysis is commit `b4dfe405`, while + `d00beebb0538b478a9e8515b1e75439bc1efb208` removed the reported post-bind + `chmod(path)` race and is an ancestor of current `main`. Current source creates the + socket owner-only during `bind()` and contains no reported `chmod(path)` operation. + A current pull-request Semgrep run remains required to confirm the candidate SHA. +- The scanner review exposed a separate process-global `umask()` and cross-language + socket-permission-contract concern. It does not reintroduce the fixed pathname race + or affect SHM allocation; pending SOW-0032 tracks its design and validation rather + than expanding this production crash repair. +- The first pull-request run of `SHM Full Backing Store` failed before executing the + regression: the runner installed CMake/CTest under `/usr/local/bin`, while the job + hard-coded `/usr/bin/ctest`. A plain `PATH` lookup is also unsafe when an environment + contains a stale wrapper. The workflow now derives and verifies `ctest` as the sibling + of the exact `cmake` selected by the job, then passes that absolute paired-tool path + through `sudo`. The rebuilt local CI directory passed that exact privileged command, + and actionlint plus the SOW audit remain clean. The entire candidate matrix must rerun + after this workflow fix. + +## Validation + +Acceptance criteria evidence: + +- Full Linux success-path and failure-path evidence is recorded below. The only pending + acceptance evidence is pull-request CI/scanner status and downstream Netdata + validation after the mandatory preflight passes. + +Tests or equivalent validation: + +- Full configured Linux matrix: 49/49 CTest tests passed in 472.07 seconds, including all + C/Rust/Go units, fuzz targets, stress, nine SHM interop directions, typed service/cache + SHM interop, lookup scale, and `test_shm_full_backing_store`. +- C coverage: all tracked files meet the 90% gate; `netipc_shm.c` 91.1%, total 91.8%. +- Rust: 380/380 library tests passed; `transport/shm.rs` 96.35%, total 92.63% coverage. +- Go: POSIX transport 91.6%, `shm_linux.go` 91.9%, total 90.2% coverage. +- C ASAN/UBSAN: 7/7 selected suites passed, zero findings. +- C TSAN: 6/6 multithreaded suites passed, zero data races. +- Independent musl validation in an ephemeral Alpine 3.24.1 container (musl 1.2.6, + GCC 15.2.0) built the read-only-mounted source, passed C `test_shm` 1/1, and returned + the exact ENOSPC marker/status 1 with no leaked file on a real 64 KiB tmpfs. +- Static/security: `git diff --check`, Rust fmt, Go fmt, ShellCheck, actionlint, Clippy + correctness/suspicious gates, cppcheck, flawfinder, Go vet/staticcheck/gosec, + govulncheck, Cargo audit, and Cargo deny passed. Clang-tidy completed with its existing + non-fatal project warning set and no new allocation finding. +- Windows local compile isolation: Rust library and all binaries pass + `cargo check --target x86_64-pc-windows-gnu`; all Go Windows packages and Windows + interop fixtures cross-compile with `GOOS=windows GOARCH=amd64 CGO_ENABLED=0`. +- Native Windows/MSYS runtime at source commit `dd3c546a90789c96c8eb31bd0b0f97c5a9b70a4c`: + the MSYS C configure/build and native Rust/Go builds passed; Named Pipe, Windows SHM, + service, payload-limit, extra-service, cache, stress, and every checked C/Rust/Go + interoperability case passed. `test_win_shm` passed 10/10 consecutive executions. +- Windows toolchain comparison: all 11 policy rows passed. The rows cover named pipe, + Windows SHM, baseline and SHM snapshots, lookup, fixed 100k rates, and both C/Rust + client/server directions. This is regression evidence only; no performance claim was + changed. + +Real-use evidence: + +- The real C, Rust, and Go creators each ran in a private 64 KiB tmpfs while requesting + an approximately 128 KiB region. Each returned the explicit ENOSPC allocation marker + and status 1, never emitted `READY`, received no signal, and leaked no region file. +- The managed C service path used by Netdata injected ENOSPC at `fallocate`, negotiated + baseline, remained ready, and completed a typed cgroups snapshot call with a valid + response. +- The independent Alpine/musl real-tmpfs run covers the libc family present in the + original static-build fleet crash evidence, not only the workstation's glibc path. + +Reviewer findings: + +- First independent pass: three reviewers agreed the runtime fix was correct but found + two acceptance blockers: the real tmpfs script could false-pass unrelated failures, + and no checked-in Linux CI route required the test. Both were fixed as described in + the execution log. +- One reviewer also found that the initial C fault seam shipped in the production + library. It was replaced with a test-only `--wrap=fallocate` object; `nm` confirms the + production library has no `nipc_shm_test_fault_set/clear` symbols while the test binary + has them. +- Reviewers identified the Rust exhaustive-match source-compatibility implication; it + is explicitly recorded under Implications And Decisions. +- Second full pass found that C did not explicitly preserve the allocation `errno`, the + required CI regex could succeed with no matching test, and one test comment still + named `ftruncate`. C now saves/restores and tests the errno, CI uses + `--no-tests=error`, and the comment describes publication state. +- Third full pass found no code defect. It identified stale README validation counts and + coverage values; README now records 49/49, current totals with measurement scope, and + the real tmpfs regression. +- Fourth full pass found no additional correctness, interoperability, security, CI, + documentation, compatibility, or unexpected-side-effect issue. Review consensus: + the implemented Linux repair is production-grade; only the explicit external + Windows/MSYS, post-push CI/scanner, vendoring, and lifecycle gates remain. +- A later independent portability pass found an overbroad seccomp fallback claim plus + two harness/artifact details. The contract now states that errno-returning denials + fall back while NetIPC cannot automatically fall back from kill/trap actions and strict + policies must allow `fallocate`; the privileged probe trace and mode-specific skip + diagnostic are fixed; the SOW records C errno preservation. +- Three fresh full reviews after the seccomp/harness fixes found no further source, + test, CI, documentation, security, portability, or unexpected-side-effect issue. + Review consensus remained production-grade code with only external process gates. +- A final documentation-precision pass noted that seccomp `TRAP` can be emulated by a + host SIGSYS handler and `USER_NOTIF`/`TRACE` are supervisor-dependent. The contract now + states the exact NetIPC boundary: it installs no such handler and cannot automatically + fall back, while external supervision remains outside scope. The final full review + found no remaining issue and retained the production-grade code verdict. +- Scanner-alert review classified GitHub code-scanning alert 7750 as stale/fixed, not a + false positive. The historical `bind()` then `chmod(path)` sequence was conditionally + raceable when an attacker could write the caller-provided runtime directory; current + commit `d00beebb` eliminated the pathname operation. The review also identified the + separate process-global `umask()`/cross-language permission-contract concern now + tracked by pending SOW-0032. +- CI failure review found that the required full-backing-store test did not execute on + its first pull-request run because the workflow assumed `/usr/bin/ctest`. The failure + was a CI portability defect, not a product-test result; tool resolution now uses the + verified sibling of the selected `cmake` and remains guarded by + `--no-tests=error`. + +Same-failure scan: + +- No production Linux SHM creator retains `ftruncate`/sparse creation. Remaining + `ftruncate` calls are malformed/truncated-region test fixtures. +- C, Rust, and Go client attach only map a region that a server fully allocated, zeroed, + and published; no grow/remap path exists. +- Native Windows/MSYS C, Rust, and Go use separate page-file-backed + `CreateFileMappingW(INVALID_HANDLE_VALUE, PAGE_READWRITE)` paths. Default `SEC_COMMIT` + provides the equivalent reservation, and create/map-view errors return normally. +- No wire-layout, offsets, capacity calculation, or negotiation profile value changed. + +Sensitive data gate: + +- The SOW contains only aggregate fleet counts, sanitized stack/function evidence, + source paths, commands, and public API names. Tests will use synthetic service names, + private mount namespaces, and generated temporary paths. No raw fleet records, + customer identifiers, credentials, private endpoints, or production data are needed. + +Artifact maintenance gate: + +- `AGENTS.md`: unchanged; no project-wide responsibility or workflow rule changed. +- Runtime project skill: unchanged; Netdata vendoring still follows the existing + mandatory `project-netdata-vendoring` workflow. Its read-only preflight and drift + classification are recorded above; no Netdata vendored source has been modified yet. +- Specs: updated authoritative `docs/level1-posix-shm.md` and + `docs/level1-windows-shm.md`; no `.agents/sow/specs/` duplicate is needed because + `docs/` is authoritative for this transport contract. +- End-user/operator docs: public transport lifecycle/error behavior and README current + validation evidence updated; no CLI, configuration, schema, or operator runbook + changed. +- End-user/operator skill: updated `docs/netipc-integrator-skill.md` with the Linux + allocation and Windows committed-mapping behavior. +- SOW lifecycle: SOW-0027 remains paused; SOW-0030 is the sole active implementation; + pending SOW-0031 tracks the separate platform-gating issue and pending SOW-0032 tracks + the UDS permission contract. Completion/move remains pending source CI/scanners and + the required vendor workflow. + +SOW lifecycle and follow-up mapping: + +- Netdata re-vendoring remains mandatory in this SOW after source commit, push, CI, and + scanner preflight. The separate CMake platform-gating discrepancy is represented by + pending SOW-0031; the separate UDS permission-contract issue is represented by pending + SOW-0032. + +## Outcome + +Pending. + +## Lessons Extracted + +Pending. + +## Followup + +- Re-vendor the fix into netdata/netdata after merge (executed in the netdata + repository; the fleet crash volume there is the success metric — SIGBUS cluster in + `P[cglkupipc]` / `P[cgroupsipc]` threads should disappear in nightlies after the + vendor update). + +## Regression Log + +None yet. diff --git a/.agents/sow/pending/SOW-0031-20260712-linux-shm-build-gating.md b/.agents/sow/pending/SOW-0031-20260712-linux-shm-build-gating.md new file mode 100644 index 00000000..868d2b20 --- /dev/null +++ b/.agents/sow/pending/SOW-0031-20260712-linux-shm-build-gating.md @@ -0,0 +1,244 @@ +# SOW-0031 - Align Linux SHM build gating with the platform contract + +## Status + +Status: open + +Sub-state: tracked follow-up discovered by SOW-0030; execution must wait until the +active SOW completes. + +## Requirements + +### Purpose + +Make the build system match NetIPC's public platform contract: Linux and native +Windows provide SHM transports, while FreeBSD and macOS use their baseline transports +without attempting to compile Linux futex code. + +### User Request + +The user delegated all SOW-0030 technical choices to the assistant with a requirement +to choose long-term-best solutions without shortcuts and reminded the project to keep +both native Windows and `MSYSTEM=MSYS` support. The approved plan tracks the broader +build-gating inconsistency separately from the urgent Linux SIGBUS fix. + +### Assistant Understanding + +Facts: + +- `docs/level1-posix-shm.md` defines POSIX SHM as Linux-only and says FreeBSD/macOS + use baseline UDS. +- The C Linux SHM source includes Linux futex headers and syscalls. +- Rust and Go use explicit Linux target/build gates for POSIX SHM. +- CMake commonly uses `NOT APPLE` for C SHM, service, tests, and interop targets; that + condition also includes FreeBSD. +- CMake treats Windows, Cygwin, and `MSYSTEM=MSYS` as the Windows runtime and must + preserve native Windows transport coverage. + +Inferences: + +- The CMake gates encode a broader platform set than the implementation and public + contract support. +- Repair may require separating baseline POSIX service targets from Linux-only SHM + components rather than mechanically replacing every `NOT APPLE` condition. + +Unknowns: + +- The exact smallest coherent CMake target split must be established by a complete + dependency graph and cross-platform configure/build investigation when this SOW is + activated. + +### Acceptance Criteria + +- Linux C/Rust/Go SHM and service targets retain their current behavior and tests. +- Native Windows and `MSYSTEM=MSYS` Named Pipe/Windows SHM builds and tests remain green. +- FreeBSD and macOS configure/build only supported baseline components and never compile + Linux futex sources. +- CMake conditions, public docs, and actual target availability agree. + +## Analysis + +Sources checked: + +- `docs/level1-posix-shm.md` +- `docs/level1-transport.md` +- `CMakeLists.txt` +- `src/libnetdata/netipc/src/transport/posix/netipc_shm.c` +- `src/crates/netipc/src/transport/mod.rs` +- `src/go/pkg/netipc/transport/posix/shm_linux.go` + +Current state: + +- The platform contract and language build gates are Linux-specific, while several + CMake gates use the broader `NOT APPLE` condition. + +Risks: + +- A mechanical condition replacement could remove useful baseline service targets on + FreeBSD/macOS or accidentally affect Windows/MSYS target selection. +- Cross-platform proof requires access to the authorized test systems or equivalent CI. + +## Pre-Implementation Gate + +Status: blocked + +Problem / root-cause model: + +- CMake uses exclusion-based `NOT APPLE` conditions where the contract requires an + explicit Linux capability boundary. The correct fix depends on whether each gated + target is entirely Linux-only or mixes reusable baseline code with SHM dependencies. + +Evidence reviewed: + +- `docs/level1-posix-shm.md:5-11` +- `docs/level1-transport.md:470-500` +- `CMakeLists.txt:31,67-100,290-305,488-508` +- `src/libnetdata/netipc/src/transport/posix/netipc_shm.c:1-25` +- `src/crates/netipc/src/transport/mod.rs:6-10` +- `src/go/pkg/netipc/transport/posix/shm_linux.go:1-6` + +Affected contracts and surfaces: + +- CMake target availability, Linux/FreeBSD/macOS/native-Windows/MSYS builds, POSIX + baseline service composition, Linux and Windows SHM transports, tests, CI, and public + platform documentation. + +Existing patterns to reuse: + +- Rust `cfg(target_os = "linux")`, Go `//go:build linux`, and CMake's existing + `NETIPC_WINDOWS_RUNTIME` capability boundary. + +Risk and blast radius: + +- Medium: build-system changes can silently omit targets or compile unsupported sources. + Keep the work separate from SOW-0030 and validate every supported platform family. + +Sensitive data handling plan: + +- No sensitive data is needed. Durable artifacts will contain only platform names, + source paths, build commands, and sanitized test outcomes. + +Implementation plan: + +1. Map every CMake platform condition to its source dependencies and public contract. +2. Define explicit Linux SHM gates while retaining baseline POSIX and Windows/MSYS + components on their supported platforms. +3. Update tests, CI, and docs where target availability changes. + +Validation plan: + +- Linux configure/build/test and POSIX interop. +- Authorized FreeBSD and macOS configure/build/test for baseline UDS behavior. +- Authorized native Windows and `MSYSTEM=MSYS` configure/build/test for Named Pipe and + Windows SHM behavior. +- Same-pattern search for broad platform exclusions that stand in for Linux capability. + +Artifact impact plan: + +- AGENTS.md: update only if supported validation commands or platform workflow changes. +- Runtime project skills: unaffected unless a reusable cross-platform validation workflow + is discovered. +- Specs: update if investigation changes actual supported target availability. +- End-user/operator docs: update platform/build documentation if commands or targets change. +- End-user/operator skills: update `docs/netipc-integrator-skill.md` if platform guidance changes. +- SOW lifecycle: remain pending until SOW-0030 completes; execute independently. + +Open-source reference evidence: + +- None needed yet; this follow-up tracks an internal build-contract mismatch. External + build-system references may be researched when the SOW is activated. + +Open decisions: + +- None at creation. Any design fork exposed by the dependency graph will be investigated + and presented before implementation. + +## Implications And Decisions + +- User decision (2026-07-12): choose the long-term-best design without shortcuts and + preserve both native Windows and `MSYSTEM=MSYS` support. +- Decision: track this build-contract repair separately so the urgent SOW-0030 crash fix + remains focused. + +## Plan + +1. Wait for SOW-0030 to complete. +2. Complete the target/dependency analysis and pre-implementation gate. +3. Implement and validate the approved cross-platform target split. + +## Execution Log + +### 2026-07-12 + +- Created as the real pending follow-up for the CMake/platform-contract discrepancy + discovered during SOW-0030 research. + +## Validation + +Acceptance criteria evidence: + +- Pending activation. + +Tests or equivalent validation: + +- Pending activation. + +Real-use evidence: + +- Pending activation. + +Reviewer findings: + +- Pending activation. + +Same-failure scan: + +- Pending activation. + +Sensitive data gate: + +- This SOW contains only public platform names, source paths, and build-contract facts. + +Artifact maintenance gate: + +- Pending activation for all artifact classes. + +Specs update: + +- Pending activation. + +Project skills update: + +- Pending activation. + +End-user/operator docs update: + +- Pending activation. + +End-user/operator skills update: + +- Pending activation. + +Lessons: + +- Pending activation. + +Follow-up mapping: + +- This file is the tracked follow-up from SOW-0030. + +## Outcome + +Pending. + +## Lessons Extracted + +Pending. + +## Followup + +None yet. + +## Regression Log + +None yet. diff --git a/.agents/sow/pending/SOW-0032-20260712-uds-permission-contract.md b/.agents/sow/pending/SOW-0032-20260712-uds-permission-contract.md new file mode 100644 index 00000000..5a5d800c --- /dev/null +++ b/.agents/sow/pending/SOW-0032-20260712-uds-permission-contract.md @@ -0,0 +1,178 @@ +# SOW-0032 - Align POSIX UDS permission handling across languages + +## Status + +Status: open + +Sub-state: pending investigation after SOW-0030; no implementation started. + +## Requirements + +### Purpose + +Define and enforce one secure, portable POSIX Unix-domain-socket permission contract for +C, Rust, and Go without process-global permission side effects in embedding programs. + +### User Request + +The user delegated technical choices to the assistant with an explicit requirement to +choose the long-term-best design without shortcuts. SOW-0030 scanner triage discovered +this separate cross-language security and embedding-safety concern. + +### Assistant Understanding + +Facts: + +- C currently changes the process-global `umask()` around `bind()` under a NetIPC-local + mutex to create the socket as owner-only. +- A NetIPC-local mutex cannot serialize unrelated threads in the embedding process that + create files while the temporary mask is active. +- Rust and Go bind their pathname sockets without the same explicit owner-only creation + mechanism, so final modes depend on the embedding process mask. +- The documented transport contract expects a caller-controlled private runtime + directory. + +Inferences: + +- Directory ownership and permissions are the portable primary security boundary for + pathname sockets. Exact socket mode may remain useful as defense in depth, but it must + not silently mutate process-global state in an embeddable library. + +Unknowns: + +- The portable mechanism and compatibility policy that should replace or constrain the + current language-specific behavior require focused research and cross-platform tests. + +### Acceptance Criteria + +- One explicit UDS runtime-directory and socket-mode contract is documented and applied + consistently to C, Rust, and Go. +- The implementation has no uncontrolled process-global permission window and includes + concurrency tests that exercise unrelated file creation during listener setup. +- Existing cross-language UDS interoperability and stale-endpoint recovery remain green. + +## Analysis + +Sources checked: + +- `src/libnetdata/netipc/src/transport/posix/netipc_uds_lifecycle.c:16-44,212` +- `src/crates/netipc/src/transport/posix.rs:637-679,995-1019` +- `src/go/pkg/netipc/transport/posix/uds_listener.go:23-60` +- `docs/level1-transport.md:559-561` +- GitHub code-scanning alert 7750 and its historical/fixed commits, recorded in SOW-0030. + +Current state: + +- The historical post-bind `chmod(path)` race is fixed. The remaining issue is the C + process-global `umask()` window and inconsistent C/Rust/Go permission behavior. + +Risks: + +- An unrelated thread can create a file with unexpectedly restrictive permissions while + C temporarily changes the process mask, causing availability or operational failures. +- Changing modes or rejecting unsafe directories can break existing integrations unless + compatibility and migration behavior are designed explicitly. + +## Pre-Implementation Gate + +Status: blocked + +Problem / root-cause model: + +- POSIX `umask()` is process-global, but the C library uses it inside a local mutex that + cannot protect unrelated embedding threads. Rust and Go rely on ambient process state, + so the three language implementations do not expose one stable security contract. + +Evidence reviewed: + +- Initial evidence is listed under Analysis. Full POSIX/Linux/macOS/FreeBSD behavior, + project patterns, and mature open-source approaches remain to be researched before + implementation. + +Affected contracts and surfaces: + +- POSIX UDS listener creation in C, Rust, and Go; runtime-directory requirements; + embedding-process behavior; UDS tests; public transport docs and integrator guidance. + +Existing patterns to reuse: + +- Safe directory-relative operations already used by POSIX SHM and stale UDS cleanup; + the existing private-runtime-directory contract; cross-language interop fixtures. + +Risk and blast radius: + +- Cross-language behavioral and source compatibility, endpoint availability, filesystem + permissions, macOS/FreeBSD portability, and embedding-process concurrency. + +Sensitive data handling plan: + +- Use synthetic paths and service names only. No credentials, customer data, private + endpoints, personal data, or production logs are required in durable artifacts. + +Implementation plan: + +1. Research portable directory and socket permission primitives and existing project or + mature open-source patterns across Linux, macOS, and FreeBSD. +2. Present any irreducible public-contract choices before implementation. +3. Implement one C/Rust/Go contract with concurrency, permission, stale-path, and interop + coverage, then update public docs and integrator guidance. + +Validation plan: + +- Deterministic concurrent unrelated-file creation tests; mode and ownership assertions; + C/Rust/Go interop; stale endpoint recovery; Linux, macOS, FreeBSD, and sanitizer runs. + +Artifact impact plan: + +- AGENTS.md: likely unaffected unless a reusable security workflow is discovered. +- Runtime project skills: likely unaffected unless a reusable validation workflow is + discovered. +- Specs: POSIX UDS/transport contract likely updated. +- End-user/operator docs: integrator/runtime-directory guidance likely updated. +- End-user/operator skills: `docs/netipc-integrator-skill.md` likely updated. +- SOW lifecycle: remains pending until SOW-0030 completes; must be activated separately. + +Open-source reference evidence: + +- None yet; research is intentionally deferred to this dedicated SOW. + +Open decisions: + +- Exact portable permission mechanism and compatibility/migration behavior require the + pre-implementation investigation. + +## Implications And Decisions + +- Decision: track this independently from SOW-0030 because it affects UDS security and + embedding semantics, not the Linux SHM allocation crash. + +## Plan + +1. Complete the pre-implementation investigation and contract comparison. +2. Obtain any irreducible user decisions, then implement and validate independently. + +## Execution Log + +### 2026-07-12 + +- Created from SOW-0030 scanner triage. No implementation started. + +## Validation + +Pending; this SOW is open and blocked on its required pre-implementation investigation. + +## Outcome + +Pending. + +## Lessons Extracted + +Pending. + +## Followup + +None yet. + +## Regression Log + +None yet. diff --git a/.github/workflows/runtime-safety.yml b/.github/workflows/runtime-safety.yml index 844259e3..04f0281f 100644 --- a/.github/workflows/runtime-safety.yml +++ b/.github/workflows/runtime-safety.yml @@ -97,6 +97,42 @@ jobs: - name: Run Go race detector run: bash tests/run-go-race.sh + shm-full-backing-store: + name: SHM Full Backing Store + runs-on: ubuntu-latest + timeout-minutes: 15 + + steps: + - name: Checkout + uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 + with: + persist-credentials: false + + - name: Set up Go + uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0 + with: + go-version-file: src/go/go.mod + cache: false + + - name: Build C, Rust, and Go SHM creators + run: | + cmake -S . -B build-shm-regression + cmake --build build-shm-regression --parallel 2 \ + --target interop_shm_c interop_shm_rs interop_shm_go + + - name: Require real tmpfs capacity-exhaustion regression + run: | + ctest_bin="$(dirname "$(command -v cmake)")/ctest" + test -x "$ctest_bin" + sudo env \ + PATH="$PATH" \ + NIPC_REQUIRE_TMPFS_TEST=1 \ + "$ctest_bin" \ + --test-dir build-shm-regression \ + --output-on-failure \ + --no-tests=error \ + -R '^test_shm_full_backing_store$' + windows-msys: name: Windows MSYS2 Runtime runs-on: windows-latest diff --git a/CMakeLists.txt b/CMakeLists.txt index c7a42157..992a2229 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -290,7 +290,22 @@ endif() # --- L1 SHM transport tests (Linux only) ----------------------------------- if(NOT NETIPC_WINDOWS_RUNTIME AND NOT APPLE) + add_library(netipc_shm_fallocate_test_wrap OBJECT + tests/fixtures/c/shm_fallocate_wrap.c + ) + target_compile_definitions(netipc_shm_fallocate_test_wrap PRIVATE + NIPC_INTERNAL_TESTING=1 + ) + target_include_directories(netipc_shm_fallocate_test_wrap PRIVATE + src/libnetdata/netipc/include + ) + add_executable(test_shm tests/fixtures/c/test_shm.c) + target_sources(test_shm PRIVATE + $ + ) + target_compile_definitions(test_shm PRIVATE NIPC_INTERNAL_TESTING=1) + target_link_options(test_shm PRIVATE "LINKER:--wrap=fallocate") target_link_libraries(test_shm PRIVATE netipc_shm netipc_uds netipc_protocol Threads::Threads) add_test(NAME test_shm COMMAND test_shm) set_tests_properties(test_shm PROPERTIES TIMEOUT 30) @@ -304,10 +319,14 @@ endif() if(NOT NETIPC_WINDOWS_RUNTIME AND NOT APPLE) add_executable(test_service tests/fixtures/c/test_service.c) + target_sources(test_service PRIVATE + $ + ) target_link_libraries(test_service PRIVATE netipc_service netipc_shm netipc_uds netipc_protocol Threads::Threads ) target_compile_definitions(test_service PRIVATE NIPC_INTERNAL_TESTING=1) + target_link_options(test_service PRIVATE "LINKER:--wrap=fallocate") add_test(NAME test_service COMMAND test_service) set_tests_properties(test_service PROPERTIES TIMEOUT 60) @@ -559,6 +578,19 @@ if(NOT NETIPC_WINDOWS_RUNTIME AND NOT APPLE AND CARGO_EXECUTABLE) TIMEOUT 60 ENVIRONMENT "INTEROP_SHM_C=$;INTEROP_SHM_RS=${INTEROP_SHM_RS};INTEROP_SHM_GO=${INTEROP_SHM_GO}" ) + + if(GO_EXECUTABLE AND CMAKE_SYSTEM_NAME STREQUAL "Linux") + add_test(NAME test_shm_full_backing_store + COMMAND bash "${CMAKE_SOURCE_DIR}/tests/test_shm_full_backing_store.sh" + WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}" + ) + set_tests_properties(test_shm_full_backing_store PROPERTIES + TIMEOUT 60 + RESOURCE_LOCK shm_full_backing_store + SKIP_RETURN_CODE 77 + ENVIRONMENT "INTEROP_SHM_C=$;INTEROP_SHM_RS=${INTEROP_SHM_RS};INTEROP_SHM_GO=${INTEROP_SHM_GO}" + ) + endif() endif() # --- Cross-language L2 service interop tests (Linux only) ------------------ diff --git a/COVERAGE-EXCLUSIONS.md b/COVERAGE-EXCLUSIONS.md index cbca1bb4..80efb5ba 100644 --- a/COVERAGE-EXCLUSIONS.md +++ b/COVERAGE-EXCLUSIONS.md @@ -123,7 +123,7 @@ Verified on `2026-03-28`: - `ShmContext::receive()` waking successfully under a finite timeout budget - remaining Linux Rust misses are now dominated by: - fixed-size encode guards - - raw `socket` / `listen` / `ftruncate` / `mmap` / `fstat` failures + - raw `socket` / `listen` / `mmap` / `fstat` failures - a few send-break / teardown timing edges - one likely unreachable guard in `dispatch_cgroups_snapshot()` where `builder.finish()` would have to return `0` @@ -258,6 +258,11 @@ These require deterministic allocation-failure injection to cover reliably. ### 3. OS / kernel failure branches +Linux SHM backing-store allocation is no longer excluded: C, Rust, and Go inject +`fallocate` failures deterministically, and the cross-language regression test runs the +real creators against a size-limited tmpfs to prove `ENOSPC` returns normally without +`SIGBUS`. + Examples: - socket creation failures diff --git a/README.md b/README.md index 1760f7ae..41b6d00a 100644 --- a/README.md +++ b/README.md @@ -292,6 +292,7 @@ The current validation story includes: - typed service tests - transport tests - shared-memory tests +- a required Linux size-limited-tmpfs regression for graceful SHM allocation failure - coverage scripts for C, Go, and Rust - benchmark generators that reject incomplete matrices @@ -300,10 +301,10 @@ The current validation story includes: Linux / POSIX: - build: passing -- `ctest`: `46/46` passing -- C coverage: `94.1%` -- Go coverage: `95.8%` -- Rust coverage: `98.57%` +- `ctest`: `49/49` passing +- C coverage (tracked C gate files): `91.8%` +- Go coverage (overall source packages): `90.2%` +- Rust coverage (Linux library, Windows-tagged files excluded): `92.63%` - measured with `cargo-llvm-cov` - Linux run now excludes Windows-tagged Rust files from the Linux total - Unix Rust service tests now live in a separate `cgroups_unix_tests.rs` file @@ -441,6 +442,8 @@ This is the honest current state: - coverage thresholds are enforced, but they are **not** at `100%` - Linux and Windows are functionally close, but Windows still has less chaos / hardening / stress breadth +- Linux SHM requires `fallocate` in strict syscall allowlists; seccomp policies that + kill or trap it cannot be converted automatically into NetIPC baseline fallback - some documented exclusions still require special infrastructure such as: - allocation-failure injection - kernel / OS failure injection diff --git a/docs/level1-posix-shm.md b/docs/level1-posix-shm.md index 7d822300..5604ae68 100644 --- a/docs/level1-posix-shm.md +++ b/docs/level1-posix-shm.md @@ -24,7 +24,9 @@ Each session gets its own SHM region. Multiple concurrent clients each have independent regions with independent request/response areas, sequence numbers, and futex signal words. -Created by the server via `open` + `ftruncate` on a filesystem path. +Created by the server via `open` + native Linux `fallocate` on a filesystem path. +The complete range is allocated before it is mapped, so capacity exhaustion is +reported as a normal creation error instead of a later `SIGBUS` during mapped access. The client opens the same path after the handshake negotiates an SHM profile, using the `session_id` received in the hello-ack. @@ -200,14 +202,42 @@ the `session_id` assigned during the handshake. 1. Derive the region path: `{run_dir}/{service_name}-{session_id:016x}.ipcshm`. 2. Create the file via `open(O_RDWR | O_CREAT | O_EXCL, 0600)`. `O_EXCL` ensures no collision with an existing region. -3. `ftruncate` to the required size (header + request area + response - area). +3. Allocate the required size (header + request area + response area) with + native Linux `fallocate(fd, 0, 0, size)`. Retry `EINTR`; any other failure + aborts SHM preparation, closes and unlinks the new region, and reports the + allocation-stage transport error. Do not fall back to sparse `ftruncate` + or an allocation mechanism without the same reservation guarantee. 4. `mmap` the region with `MAP_SHARED`. -5. Write the header: magic, version, header_len, owner_pid, - owner_generation, offsets, capacities. Initialize all atomic fields - to zero. +5. Zero the complete mapped region, then write the header: magic, version, + header_len, owner_pid, owner_generation, offsets, capacities. Initialize + all atomic fields to zero. 6. The region is now ready for the client. +A successful native allocation guarantees that later writes within the region do not +fail because the backing filesystem is out of space. `ENOSPC`, quota exhaustion, +unsupported allocation, and other allocation failures are reported through the +allocation-stage error (`NIPC_SHM_ERR_ALLOCATE`, `ShmError::Allocate`, or +`ErrShmAllocate`). C preserves the allocation `errno`; Rust and Go carry the OS cause +in their error values. Managed services remove SHM from that session's negotiation and +continue over baseline when baseline is configured. An explicitly SHM-only service has +no fallback transport and may reject or retry the session. + +Linux runtime sandboxes must allow the `fallocate` system call or arrange for it to +return an ordinary error such as `EPERM`, `ENOSYS`, or `EOPNOTSUPP`. NetIPC can convert +an error return into SHM disablement and baseline fallback. Seccomp `KILL_PROCESS` and +`KILL_THREAD` terminate instead; `TRAP` delivers `SIGSYS` before NetIPC receives an +error. NetIPC deliberately installs no process-global `SIGSYS` emulation handler, so it +cannot automatically fall back for those actions. `USER_NOTIF` and `TRACE` behavior +depends on an external supervisor or tracer and is outside this transport contract. +Strict custom syscall allowlists must therefore add `fallocate` before enabling Linux +SHM profiles. + +Client attach does not repeat allocation. Every conforming server allocates and zeroes +the complete region before it publishes a valid header or completes SHM negotiation. +An older server that fails while backing or zeroing the region cannot advertise a +usable SHM session; an older server that successfully advertises SHM has already +touched every region page. + The server must track all active per-session SHM regions so they can be cleaned up on session close and server shutdown. diff --git a/docs/level1-windows-shm.md b/docs/level1-windows-shm.md index 512af724..90ade044 100644 --- a/docs/level1-windows-shm.md +++ b/docs/level1-windows-shm.md @@ -237,8 +237,11 @@ handshake negotiates an SHM profile. The kernel object names are derived from the `session_id` assigned during the handshake. 1. Derive kernel object names using `session_id` (see naming section). -2. Create the file mapping via `CreateFileMappingW` using the derived - mapping name. +2. Create a page-file-backed mapping via `CreateFileMappingW` using + `INVALID_HANDLE_VALUE`, `PAGE_READWRITE`, and the derived mapping name. + With no explicit allocation attribute, Windows assumes `SEC_COMMIT`: the + system must have enough committable pages for the complete mapping or + creation fails normally before the view is used. 3. Map the view via `MapViewOfFile`. 4. Write the header: magic, version, header_len, profile, offsets, capacities, spin_tries. Initialize all volatile fields to zero. @@ -249,6 +252,10 @@ derived from the `session_id` assigned during the handshake. direction model. 6. The region is now ready for the client. +`CreateFileMappingW` and `MapViewOfFile` failures are SHM preparation errors. Managed +servers remove Windows SHM profiles and continue over the Named Pipe baseline when it +is configured. Windows does not use the Linux file-backed `fallocate` lifecycle. + The server must track all active per-session SHM regions so they can be cleaned up on session close and server shutdown. diff --git a/docs/netipc-integrator-skill.md b/docs/netipc-integrator-skill.md index 7adc7759..a90e4664 100644 --- a/docs/netipc-integrator-skill.md +++ b/docs/netipc-integrator-skill.md @@ -662,6 +662,27 @@ Implication: - shutdown paths can abort a blocked call, but normal shared-client access still needs external synchronization in C and Go +### SHM preparation failure + +Current required behavior: + +- Linux servers allocate the complete file-backed SHM region before mapping or + publishing it +- if backing-store allocation fails, including `ENOSPC`, SHM preparation returns a + normal allocation-stage error; it must not reach a mapped write or terminate the host + process with `SIGBUS` +- managed servers remove SHM from that session's supported and preferred profiles and + continue over baseline when baseline is configured +- an explicitly SHM-only configuration has no fallback transport and may reject or + retry the session +- Linux sandbox/seccomp policy must allow `fallocate` or arrange an ordinary errno + return; `KILL_PROCESS`/`KILL_THREAD` terminate, and `TRAP` requires host-level + `SIGSYS` emulation that NetIPC does not install; `USER_NOTIF`/`TRACE` supervisor + behavior is outside the transport contract, so strict custom allowlists must include + `fallocate` +- native Windows and `MSYSTEM=MSYS` use committed page-file-backed mappings; mapping + creation or view failures follow the existing Windows SHM preparation error path + ### SHM attach failure Current required behavior: diff --git a/src/crates/netipc/src/transport/shm.rs b/src/crates/netipc/src/transport/shm.rs index 56de5936..561fbc59 100644 --- a/src/crates/netipc/src/transport/shm.rs +++ b/src/crates/netipc/src/transport/shm.rs @@ -44,8 +44,10 @@ pub enum ShmError { PathTooLong, /// open/shm_open failed. Open(i32), - /// ftruncate failed. + /// Legacy compatibility; SHM creation no longer uses ftruncate. Truncate(i32), + /// Backing storage allocation failed. + Allocate(i32), /// mmap failed. Mmap(i32), /// Header magic mismatch. @@ -76,6 +78,7 @@ impl std::fmt::Display for ShmError { ShmError::PathTooLong => write!(f, "SHM path exceeds limit"), ShmError::Open(e) => write!(f, "open failed: errno {e}"), ShmError::Truncate(e) => write!(f, "ftruncate failed: errno {e}"), + ShmError::Allocate(e) => write!(f, "SHM allocation failed: errno {e}"), ShmError::Mmap(e) => write!(f, "mmap failed: errno {e}"), ShmError::BadMagic => write!(f, "SHM header magic mismatch"), ShmError::BadVersion => write!(f, "SHM header version mismatch"), @@ -269,25 +272,15 @@ impl ShmContext { return Err(ShmError::Open(errno())); } - if unsafe { libc::ftruncate(fd, region_size as libc::off_t) } < 0 { - let e = errno(); + if let Err(e) = allocate_region(fd, region_size) { unsafe { libc::close(fd); libc::unlink(c_path.as_ptr()); } - return Err(ShmError::Truncate(e)); + return Err(ShmError::Allocate(e)); } - let base = unsafe { - libc::mmap( - ptr::null_mut(), - region_size, - libc::PROT_READ | libc::PROT_WRITE, - libc::MAP_SHARED, - fd, - 0, - ) - }; + let base = unsafe { map_region(fd, region_size) }; if base == libc::MAP_FAILED { let e = errno(); unsafe { @@ -870,6 +863,109 @@ fn errno() -> i32 { unsafe { *libc::__errno_location() } } +/// Reserve all backing storage before exposing the region through mmap. +/// +/// Linux may interrupt fallocate before it changes the file, so retry EINTR. +/// Every other error is returned to the caller; SHM has no unsafe sparse-file +/// fallback. +fn allocate_region(fd: i32, region_size: usize) -> Result<(), i32> { + loop { + #[cfg(test)] + record_fallocate_call(); + + #[cfg(test)] + let result = match take_fallocate_fault() { + Some(error) => { + unsafe { *libc::__errno_location() = error }; + -1 + } + None => unsafe { libc::fallocate(fd, 0, 0, region_size as libc::off_t) }, + }; + + #[cfg(not(test))] + let result = unsafe { libc::fallocate(fd, 0, 0, region_size as libc::off_t) }; + + if result == 0 { + return Ok(()); + } + + let error = errno(); + if error != libc::EINTR { + return Err(error); + } + } +} + +unsafe fn map_region(fd: i32, region_size: usize) -> *mut libc::c_void { + #[cfg(test)] + record_mmap_call(); + + libc::mmap( + ptr::null_mut(), + region_size, + libc::PROT_READ | libc::PROT_WRITE, + libc::MAP_SHARED, + fd, + 0, + ) +} + +#[cfg(test)] +#[derive(Default)] +struct ShmSyscallTestState { + fallocate_faults: std::collections::VecDeque, + fallocate_calls: usize, + mmap_calls: usize, +} + +#[cfg(test)] +thread_local! { + static SHM_SYSCALL_TEST_STATE: std::cell::RefCell = + std::cell::RefCell::new(ShmSyscallTestState::default()); +} + +#[cfg(test)] +fn install_fallocate_faults(errors: &[i32]) { + SHM_SYSCALL_TEST_STATE.with(|state| { + let mut state = state.borrow_mut(); + assert!( + state.fallocate_faults.is_empty(), + "fallocate fault sequence already installed" + ); + state.fallocate_faults.extend(errors.iter().copied()); + state.fallocate_calls = 0; + state.mmap_calls = 0; + }); +} + +#[cfg(test)] +fn clear_fallocate_faults() { + SHM_SYSCALL_TEST_STATE.with(|state| *state.borrow_mut() = ShmSyscallTestState::default()); +} + +#[cfg(test)] +fn take_fallocate_fault() -> Option { + SHM_SYSCALL_TEST_STATE.with(|state| state.borrow_mut().fallocate_faults.pop_front()) +} + +#[cfg(test)] +fn record_fallocate_call() { + SHM_SYSCALL_TEST_STATE.with(|state| state.borrow_mut().fallocate_calls += 1); +} + +#[cfg(test)] +fn record_mmap_call() { + SHM_SYSCALL_TEST_STATE.with(|state| state.borrow_mut().mmap_calls += 1); +} + +#[cfg(test)] +fn syscall_test_counts() -> (usize, usize) { + SHM_SYSCALL_TEST_STATE.with(|state| { + let state = state.borrow(); + (state.fallocate_calls, state.mmap_calls) + }) +} + fn pid_alive(pid: i32) -> bool { if pid <= 0 { return false; diff --git a/src/crates/netipc/src/transport/shm_tests.rs b/src/crates/netipc/src/transport/shm_tests.rs index e2f00f3d..0119a5c8 100644 --- a/src/crates/netipc/src/transport/shm_tests.rs +++ b/src/crates/netipc/src/transport/shm_tests.rs @@ -19,6 +19,21 @@ fn cleanup_shm(service: &str, session_id: u64) { let _ = std::fs::remove_file(&path); } +struct FallocateFaultGuard; + +impl FallocateFaultGuard { + fn install(errors: &[i32]) -> Self { + install_fallocate_faults(errors); + Self + } +} + +impl Drop for FallocateFaultGuard { + fn drop(&mut self) { + clear_fallocate_faults(); + } +} + fn wait_for_server_ready(rx: &mpsc::Receiver, service: &str) -> u64 { rx.recv_timeout(SERVER_READY_TIMEOUT).unwrap_or_else(|err| { panic!("timed out waiting for server readiness for service={service}: {err}") @@ -523,6 +538,7 @@ fn shm_error_display_all_variants() { (ShmError::PathTooLong, "SHM path exceeds limit"), (ShmError::Open(2), "open failed: errno 2"), (ShmError::Truncate(28), "ftruncate failed: errno 28"), + (ShmError::Allocate(28), "SHM allocation failed: errno 28"), (ShmError::Mmap(12), "mmap failed: errno 12"), (ShmError::BadMagic, "SHM header magic mismatch"), (ShmError::BadVersion, "SHM header version mismatch"), @@ -542,6 +558,86 @@ fn shm_error_display_all_variants() { let _ = format!("{e}"); } +#[test] +fn test_server_create_allocate_enospc_cleans_up_before_mmap() { + ensure_run_dir(); + let svc = "rs_shm_alloc_enospc"; + let sid = 0xa110_c001; + cleanup_shm(svc, sid); + + let fault = FallocateFaultGuard::install(&[libc::ENOSPC]); + let err = match ShmContext::server_create(TEST_RUN_DIR, svc, sid, 1024, 1024) { + Ok(_) => panic!("injected ENOSPC must fail SHM creation"), + Err(err) => err, + }; + + assert_eq!(err, ShmError::Allocate(libc::ENOSPC)); + assert_eq!(syscall_test_counts(), (1, 0), "mmap must not be reached"); + + let path = build_shm_path(TEST_RUN_DIR, svc, sid).expect("SHM path"); + assert!(!path.exists(), "failed allocation must unlink its file"); + + drop(fault); + let mut server = ShmContext::server_create(TEST_RUN_DIR, svc, sid, 1024, 1024) + .expect("cleanup must permit recreation at the same path"); + server.destroy(); +} + +#[test] +fn test_server_create_retries_fallocate_after_eintr() { + ensure_run_dir(); + let svc = "rs_shm_alloc_eintr"; + let sid = 0xa110_c002; + cleanup_shm(svc, sid); + + let _fault = FallocateFaultGuard::install(&[libc::EINTR]); + let mut server = ShmContext::server_create(TEST_RUN_DIR, svc, sid, 1024, 2048) + .expect("fallocate must retry EINTR"); + + assert_eq!( + syscall_test_counts(), + (2, 1), + "one interrupted allocation retry must precede one mmap" + ); + + let request = unsafe { + std::slice::from_raw_parts( + server.base.add(server.request_offset as usize), + server.request_capacity as usize, + ) + }; + let response = unsafe { + std::slice::from_raw_parts( + server.base.add(server.response_offset as usize), + server.response_capacity as usize, + ) + }; + assert!(request.iter().all(|byte| *byte == 0)); + assert!(response.iter().all(|byte| *byte == 0)); + + server.destroy(); +} + +#[test] +fn test_server_create_does_not_fallback_when_fallocate_is_unsupported() { + ensure_run_dir(); + let svc = "rs_shm_alloc_unsupported"; + let sid = 0xa110_c003; + cleanup_shm(svc, sid); + + let _fault = FallocateFaultGuard::install(&[libc::EOPNOTSUPP]); + let err = match ShmContext::server_create(TEST_RUN_DIR, svc, sid, 1024, 1024) { + Ok(_) => panic!("unsupported fallocate must disable SHM creation"), + Err(err) => err, + }; + + assert_eq!(err, ShmError::Allocate(libc::EOPNOTSUPP)); + assert_eq!(syscall_test_counts(), (1, 0), "mmap must not be reached"); + + let path = build_shm_path(TEST_RUN_DIR, svc, sid).expect("SHM path"); + assert!(!path.exists(), "failed allocation must unlink its file"); +} + // ----------------------------------------------------------------------- // ShmContext accessors: role(), fd() (lines 164-165, 169-170) // ----------------------------------------------------------------------- diff --git a/src/go/pkg/netipc/transport/posix/shm_linux.go b/src/go/pkg/netipc/transport/posix/shm_linux.go index e7c651f7..725067a3 100644 --- a/src/go/pkg/netipc/transport/posix/shm_linux.go +++ b/src/go/pkg/netipc/transport/posix/shm_linux.go @@ -61,6 +61,8 @@ const ( var ( ErrShmPathTooLong = errors.New("SHM path exceeds limit") ErrShmOpen = errors.New("SHM open failed") + ErrShmAllocate = errors.New("SHM allocation failed") + // Deprecated: SHM creation no longer uses ftruncate; use ErrShmAllocate. ErrShmTruncate = errors.New("SHM ftruncate failed") ErrShmMmap = errors.New("SHM mmap failed") ErrShmBadMagic = errors.New("SHM header magic mismatch") @@ -140,8 +142,24 @@ func (c *ShmContext) OwnerAlive() bool { // Server API // --------------------------------------------------------------------------- +type shmFallocateFunc func(fd int, mode uint32, off, len int64) error + +func shmPreallocate(fd int, size int64, fallocate shmFallocateFunc) error { + for { + err := fallocate(fd, 0, 0, size) + if err == syscall.EINTR { + continue + } + return err + } +} + // ShmServerCreate creates a SHM region at {runDir}/{serviceName}-{sessionID}.ipcshm. func ShmServerCreate(runDir, serviceName string, sessionID uint64, reqCapacity, respCapacity uint32) (*ShmContext, error) { + return shmServerCreate(runDir, serviceName, sessionID, reqCapacity, respCapacity, syscall.Fallocate) +} + +func shmServerCreate(runDir, serviceName string, sessionID uint64, reqCapacity, respCapacity uint32, fallocate shmFallocateFunc) (*ShmContext, error) { path, err := buildShmPath(runDir, serviceName, sessionID) if err != nil { return nil, err @@ -171,10 +189,10 @@ func ShmServerCreate(runDir, serviceName string, sessionID uint64, reqCapacity, } fd := int(f.Fd()) - if err := syscall.Ftruncate(fd, int64(regionSize)); err != nil { + if err := shmPreallocate(fd, int64(regionSize), fallocate); err != nil { _ = f.Close() _ = os.Remove(path) - return nil, fmt.Errorf("%w: %v", ErrShmTruncate, err) + return nil, fmt.Errorf("%w: %w", ErrShmAllocate, err) } data, err := syscall.Mmap(fd, 0, regionSize, diff --git a/src/go/pkg/netipc/transport/posix/shm_more_edge_test.go b/src/go/pkg/netipc/transport/posix/shm_more_edge_test.go index a0339fdc..d15c895e 100644 --- a/src/go/pkg/netipc/transport/posix/shm_more_edge_test.go +++ b/src/go/pkg/netipc/transport/posix/shm_more_edge_test.go @@ -7,6 +7,7 @@ import ( "errors" "os" "path/filepath" + "syscall" "testing" ) @@ -33,6 +34,99 @@ func fillShmHeader(data []byte, ownerPID int32, ownerGen uint32, reqOff, reqCap, binary.NativeEndian.PutUint32(data[shmHeaderRespCapOff:shmHeaderRespCapOff+4], respCap) } +func TestShmServerCreatePreallocation(t *testing.T) { + t.Run("ENOSPC preserves cause and removes file", func(t *testing.T) { + runDir := t.TempDir() + const ( + svc = "go_shm_allocate_enospc" + sessionID = uint64(41) + ) + + var calls int + var gotMode uint32 + var gotOff, gotLen int64 + ctx, err := shmServerCreate(runDir, svc, sessionID, 1024, 2048, + func(_ int, mode uint32, off, len int64) error { + calls++ + gotMode, gotOff, gotLen = mode, off, len + return syscall.ENOSPC + }) + if ctx != nil { + ctx.ShmDestroy() + t.Fatal("allocation failure returned a context") + } + if !errors.Is(err, ErrShmAllocate) { + t.Fatalf("create error = %v, want ErrShmAllocate", err) + } + if !errors.Is(err, syscall.ENOSPC) { + t.Fatalf("create error = %v, want underlying ENOSPC", err) + } + if calls != 1 { + t.Fatalf("fallocate calls = %d, want 1", calls) + } + wantLen := int64(shmAlign64(uint32(shmHeaderLen)) + shmAlign64(1024) + shmAlign64(2048)) + if gotMode != 0 || gotOff != 0 || gotLen != wantLen { + t.Fatalf("fallocate args = mode %d off %d len %d, want 0, 0, %d", gotMode, gotOff, gotLen, wantLen) + } + + path, pathErr := buildShmPath(runDir, svc, sessionID) + if pathErr != nil { + t.Fatalf("build path: %v", pathErr) + } + if _, statErr := os.Stat(path); !errors.Is(statErr, os.ErrNotExist) { + t.Fatalf("failed allocation file should be removed, stat err = %v", statErr) + } + }) + + t.Run("unsupported allocation has no sparse fallback", func(t *testing.T) { + runDir := t.TempDir() + const ( + svc = "go_shm_allocate_unsupported" + sessionID = uint64(43) + ) + + ctx, err := shmServerCreate(runDir, svc, sessionID, 1024, 1024, + func(_ int, _ uint32, _, _ int64) error { + return syscall.EOPNOTSUPP + }) + if ctx != nil { + ctx.ShmDestroy() + t.Fatal("unsupported allocation returned a context") + } + if !errors.Is(err, ErrShmAllocate) || !errors.Is(err, syscall.EOPNOTSUPP) { + t.Fatalf("create error = %v, want allocation error wrapping EOPNOTSUPP", err) + } + + path, pathErr := buildShmPath(runDir, svc, sessionID) + if pathErr != nil { + t.Fatalf("build path: %v", pathErr) + } + if _, statErr := os.Stat(path); !errors.Is(statErr, os.ErrNotExist) { + t.Fatalf("unsupported allocation file should be removed, stat err = %v", statErr) + } + }) + + t.Run("EINTR retries", func(t *testing.T) { + runDir := t.TempDir() + var calls int + ctx, err := shmServerCreate(runDir, "go_shm_allocate_eintr", 42, 1024, 1024, + func(fd int, mode uint32, off, len int64) error { + calls++ + if calls == 1 { + return syscall.EINTR + } + return syscall.Fallocate(fd, mode, off, len) + }) + if err != nil { + t.Fatalf("create after EINTR: %v", err) + } + defer ctx.ShmDestroy() + if calls != 2 { + t.Fatalf("fallocate calls = %d, want 2", calls) + } + }) +} + func writeRawShmRegionFile(t *testing.T, runDir, service string, sessionID uint64, size int, fill func([]byte)) string { t.Helper() if err := os.MkdirAll(runDir, 0700); err != nil { diff --git a/src/libnetdata/netipc/include/netipc/netipc_shm.h b/src/libnetdata/netipc/include/netipc/netipc_shm.h index 5a434833..51629ad1 100644 --- a/src/libnetdata/netipc/include/netipc/netipc_shm.h +++ b/src/libnetdata/netipc/include/netipc/netipc_shm.h @@ -41,7 +41,7 @@ typedef enum { NIPC_SHM_OK = 0, NIPC_SHM_ERR_PATH_TOO_LONG, /* SHM path exceeds limit */ NIPC_SHM_ERR_OPEN, /* open/shm_open failed */ - NIPC_SHM_ERR_TRUNCATE, /* ftruncate failed */ + NIPC_SHM_ERR_TRUNCATE, /* legacy compatibility; creation no longer truncates */ NIPC_SHM_ERR_MMAP, /* mmap failed */ NIPC_SHM_ERR_BAD_MAGIC, /* header magic mismatch */ NIPC_SHM_ERR_BAD_VERSION, /* header version mismatch */ @@ -53,6 +53,7 @@ typedef enum { NIPC_SHM_ERR_TIMEOUT, /* futex wait timed out */ NIPC_SHM_ERR_BAD_PARAM, /* invalid argument */ NIPC_SHM_ERR_PEER_DEAD, /* owner process has exited */ + NIPC_SHM_ERR_ALLOCATE, /* backing allocation failed; errno preserved */ } nipc_shm_error_t; /* ------------------------------------------------------------------ */ @@ -217,6 +218,17 @@ bool nipc_shm_owner_alive(const nipc_shm_ctx_t *ctx); */ void nipc_shm_cleanup_stale(const char *run_dir, const char *service_name); +#ifdef NIPC_INTERNAL_TESTING +typedef enum { + NIPC_SHM_TEST_FAULT_NONE = 0, + NIPC_SHM_TEST_FAULT_ALLOCATE, +} nipc_shm_test_fault_site_t; + +void nipc_shm_test_fault_set(nipc_shm_test_fault_site_t site, + int error_number); +void nipc_shm_test_fault_clear(void); +#endif + /* Get the file descriptor for external event integration. */ static inline int nipc_shm_fd(const nipc_shm_ctx_t *ctx) { return ctx->fd; diff --git a/src/libnetdata/netipc/src/transport/posix/netipc_shm.c b/src/libnetdata/netipc/src/transport/posix/netipc_shm.c index fa7015a6..f307f5bc 100644 --- a/src/libnetdata/netipc/src/transport/posix/netipc_shm.c +++ b/src/libnetdata/netipc/src/transport/posix/netipc_shm.c @@ -1,3 +1,7 @@ +#ifndef _GNU_SOURCE +#define _GNU_SOURCE +#endif + /* * netipc_shm.c - L1 POSIX SHM transport (Linux only). * @@ -352,11 +356,18 @@ nipc_shm_error_t nipc_shm_server_create(const char *run_dir, return NIPC_SHM_ERR_OPEN; } - if (ftruncate(fd, (off_t)region_size) < 0) { + int allocate_rc; + do { + allocate_rc = fallocate(fd, 0, 0, (off_t)region_size); + } while (allocate_rc < 0 && errno == EINTR); + + if (allocate_rc < 0) { + int allocation_errno = errno; close(fd); unlinkat(dir_fd, shm_name, 0); close(dir_fd); - return NIPC_SHM_ERR_TRUNCATE; + errno = allocation_errno; + return NIPC_SHM_ERR_ALLOCATE; } void *map = mmap(NULL, region_size, PROT_READ | PROT_WRITE, diff --git a/tests/fixtures/c/interop_shm.c b/tests/fixtures/c/interop_shm.c index 1d7d3b50..95fe76ac 100644 --- a/tests/fixtures/c/interop_shm.c +++ b/tests/fixtures/c/interop_shm.c @@ -13,6 +13,7 @@ #include "netipc/netipc_protocol.h" #include "interop_path.h" +#include #include #include #include @@ -25,6 +26,8 @@ static int run_server(const char *run_dir, const char *service) nipc_shm_error_t err = nipc_shm_server_create( run_dir, service, 1, 65536, 65536, &shm); if (err != NIPC_SHM_OK) { + if (err == NIPC_SHM_ERR_ALLOCATE && errno == ENOSPC) + fprintf(stderr, "NIPC_SHM_ALLOCATE_ENOSPC\n"); fprintf(stderr, "server: shm create failed: %d\n", err); return 1; } diff --git a/tests/fixtures/c/shm_fallocate_wrap.c b/tests/fixtures/c/shm_fallocate_wrap.c new file mode 100644 index 00000000..28e96495 --- /dev/null +++ b/tests/fixtures/c/shm_fallocate_wrap.c @@ -0,0 +1,42 @@ +#ifndef _GNU_SOURCE +#define _GNU_SOURCE +#endif + +/* Test-only linker wrapper for deterministic Linux fallocate failures. */ + +#include "netipc/netipc_shm.h" + +#include +#include +#include +#include + +static uint32_t g_fault_site = NIPC_SHM_TEST_FAULT_NONE; +static uint32_t g_fault_error = 0; + +void nipc_shm_test_fault_set(nipc_shm_test_fault_site_t site, int error_number) +{ + __atomic_store_n(&g_fault_site, NIPC_SHM_TEST_FAULT_NONE, __ATOMIC_RELEASE); + __atomic_store_n(&g_fault_error, (uint32_t)error_number, __ATOMIC_RELAXED); + __atomic_store_n(&g_fault_site, (uint32_t)site, __ATOMIC_RELEASE); +} + +void nipc_shm_test_fault_clear(void) +{ + __atomic_store_n(&g_fault_site, NIPC_SHM_TEST_FAULT_NONE, __ATOMIC_RELEASE); +} + +int __real_fallocate(int fd, int mode, off_t offset, off_t length); + +int __wrap_fallocate(int fd, int mode, off_t offset, off_t length) +{ + uint32_t expected = NIPC_SHM_TEST_FAULT_ALLOCATE; + if (__atomic_compare_exchange_n(&g_fault_site, &expected, + NIPC_SHM_TEST_FAULT_NONE, false, + __ATOMIC_ACQ_REL, __ATOMIC_ACQUIRE)) { + errno = (int)__atomic_load_n(&g_fault_error, __ATOMIC_RELAXED); + return -1; + } + + return __real_fallocate(fd, mode, offset, length); +} diff --git a/tests/fixtures/c/test_service.c b/tests/fixtures/c/test_service.c index 3788d47d..7a35684e 100644 --- a/tests/fixtures/c/test_service.c +++ b/tests/fixtures/c/test_service.c @@ -7189,6 +7189,59 @@ static void test_shm_negotiation_falls_back_to_baseline_on_obstructed_region(voi cleanup_all(svc); } +static void test_shm_allocation_failure_falls_back_to_working_baseline(void) +{ + printf("Test: SHM allocation failure falls back to working baseline\n"); + const char *svc = "svc_shm_alloc_fallback"; + cleanup_all(svc); + + shm_server_ctx_t sctx; + memset(&sctx, 0, sizeof(sctx)); + sctx.service = svc; + + pthread_t tid; + pthread_create(&tid, NULL, shm_server_thread_fn, &sctx); + for (int i = 0; i < 2000 + && !__atomic_load_n(&sctx.ready, __ATOMIC_ACQUIRE) + && !__atomic_load_n(&sctx.done, __ATOMIC_ACQUIRE); i++) + usleep(500); + check("allocation-fallback SHM server started", + __atomic_load_n(&sctx.ready, __ATOMIC_ACQUIRE) == 1); + + nipc_client_config_t ccfg = { + .supported_profiles = NIPC_PROFILE_BASELINE | NIPC_PROFILE_SHM_HYBRID, + .preferred_profiles = NIPC_PROFILE_SHM_HYBRID, + .max_request_batch_items = 1, + .max_response_payload_bytes = RESPONSE_BUF_SIZE, + .auth_token = AUTH_TOKEN, + }; + + nipc_client_ctx_t client; + nipc_client_init(&client, TEST_RUN_DIR, svc, &ccfg); + nipc_shm_test_fault_set(NIPC_SHM_TEST_FAULT_ALLOCATE, ENOSPC); + bool changed = nipc_client_refresh(&client); + nipc_shm_test_fault_clear(); + + check("allocation failure refresh reports READY transition", changed); + check("allocation failure leaves client ready", nipc_client_ready(&client)); + check("allocation failure selects baseline", + client.session_valid && client.shm == NULL && + client.session.selected_profile == NIPC_PROFILE_BASELINE); + + nipc_cgroups_resp_view_t view; + nipc_error_t err = nipc_client_call_cgroups_snapshot(&client, &view); + check("baseline call succeeds after allocation failure", err == NIPC_OK); + if (err == NIPC_OK) + check("baseline response is valid after allocation failure", + view.item_count == 3); + + nipc_client_close(&client); + nipc_server_stop(&sctx.server); + pthread_join(tid, NULL); + cleanup_session_shm(svc, 1); + cleanup_all(svc); +} + static void test_client_shm_attach_failure_falls_back_to_baseline(void) { printf("Test: Client-side SHM attach failure falls back to baseline\n"); @@ -7323,6 +7376,7 @@ int main(void) test_server_init_worker_floor_and_long_run_dir(); printf("\n"); test_client_init_defaults_and_truncation(); printf("\n"); test_shm_negotiation_falls_back_to_baseline_on_obstructed_region(); printf("\n"); + test_shm_allocation_failure_falls_back_to_working_baseline(); printf("\n"); test_client_shm_attach_failure_falls_back_to_baseline(); printf("\n"); printf("=== Results: %d passed, %d failed ===\n", g_pass, g_fail); diff --git a/tests/fixtures/c/test_shm.c b/tests/fixtures/c/test_shm.c index 0c996104..ec90953b 100644 --- a/tests/fixtures/c/test_shm.c +++ b/tests/fixtures/c/test_shm.c @@ -369,7 +369,7 @@ static void test_negotiated_shm(void) if (serr == NIPC_SHM_OK) break; /* Retryable: file doesn't exist, header not ready, or - * header not yet written (magic is 0 after ftruncate). */ + * header not yet published (magic is still zero). */ if (serr == NIPC_SHM_ERR_NOT_READY || serr == NIPC_SHM_ERR_OPEN || serr == NIPC_SHM_ERR_BAD_MAGIC) @@ -985,6 +985,60 @@ static void test_server_create_validation(void) &ctx) == NIPC_SHM_ERR_BAD_PARAM); } +static void test_server_create_allocation_failures(void) +{ + printf("Test: Server create allocation failures\n"); + const char *svc = "shm_alloc_failure"; + char path[256]; + snprintf(path, sizeof(path), "%s/%s-%016" PRIx64 ".ipcshm", + TEST_RUN_DIR, svc, (uint64_t)1); + cleanup_shm(svc); + + nipc_shm_ctx_t ctx; + nipc_shm_test_fault_set(NIPC_SHM_TEST_FAULT_ALLOCATE, ENOSPC); + nipc_shm_error_t err = nipc_shm_server_create( + TEST_RUN_DIR, svc, 1, 4096, 4096, &ctx); + int allocation_errno = errno; + nipc_shm_test_fault_clear(); + + check("ENOSPC returns allocation error", err == NIPC_SHM_ERR_ALLOCATE); + check("ENOSPC preserves allocation errno", allocation_errno == ENOSPC); + check("ENOSPC leaves no mapped context", + ctx.fd == -1 && ctx.base == NULL && ctx.region_size == 0); + check("ENOSPC removes incomplete region", access(path, F_OK) != 0); + + nipc_shm_test_fault_set(NIPC_SHM_TEST_FAULT_ALLOCATE, EOPNOTSUPP); + err = nipc_shm_server_create( + TEST_RUN_DIR, svc, 1, 4096, 4096, &ctx); + allocation_errno = errno; + nipc_shm_test_fault_clear(); + + check("unsupported allocation has no sparse fallback", + err == NIPC_SHM_ERR_ALLOCATE); + check("unsupported allocation preserves errno", + allocation_errno == EOPNOTSUPP); + check("unsupported allocation leaves no mapped context", + ctx.fd == -1 && ctx.base == NULL && ctx.region_size == 0); + check("unsupported allocation removes incomplete region", + access(path, F_OK) != 0); + + nipc_shm_test_fault_set(NIPC_SHM_TEST_FAULT_ALLOCATE, EINTR); + err = nipc_shm_server_create(TEST_RUN_DIR, svc, 1, 4096, 4096, &ctx); + nipc_shm_test_fault_clear(); + + check("EINTR retries allocation", err == NIPC_SHM_OK); + if (err == NIPC_SHM_OK) { + nipc_shm_region_header_t *hdr = + (nipc_shm_region_header_t *)ctx.base; + check("EINTR retry initializes region header", + hdr->magic == NIPC_SHM_REGION_MAGIC && + hdr->version == NIPC_SHM_REGION_VERSION); + nipc_shm_destroy(&ctx); + } + check("EINTR retry region is destroyed", access(path, F_OK) != 0); + cleanup_shm(svc); +} + static void test_client_attach_validation(void) { printf("Test: Client attach validation errors\n"); @@ -1798,6 +1852,7 @@ int main(void) test_addr_in_use(); printf("\n"); test_multiple_roundtrips(); printf("\n"); test_server_create_validation(); printf("\n"); + test_server_create_allocation_failures(); printf("\n"); test_client_attach_validation(); printf("\n"); test_shm_close_null(); printf("\n"); test_shm_send_bad_param(); printf("\n"); diff --git a/tests/fixtures/go/cmd/interop_shm/main.go b/tests/fixtures/go/cmd/interop_shm/main.go index 9098d3ac..1db8972a 100644 --- a/tests/fixtures/go/cmd/interop_shm/main.go +++ b/tests/fixtures/go/cmd/interop_shm/main.go @@ -44,6 +44,9 @@ func buildMessage(kind, code uint16, messageID uint64, payload []byte) []byte { func runServer(runDir, service string) int { ctx, err := posix.ShmServerCreate(runDir, service, 1, 65536, 65536) if err != nil { + if errors.Is(err, posix.ErrShmAllocate) && errors.Is(err, syscall.ENOSPC) { + fmt.Fprintln(os.Stderr, "NIPC_SHM_ALLOCATE_ENOSPC") + } fmt.Fprintf(os.Stderr, "server: shm create failed: %v\n", err) return 1 } diff --git a/tests/fixtures/rust/src/bin/interop_shm.rs b/tests/fixtures/rust/src/bin/interop_shm.rs index bc10768a..55a53782 100644 --- a/tests/fixtures/rust/src/bin/interop_shm.rs +++ b/tests/fixtures/rust/src/bin/interop_shm.rs @@ -39,7 +39,14 @@ mod linux_only { } fn run_server(run_dir: &str, service: &str) -> Result<(), Box> { - let mut ctx = ShmContext::server_create(run_dir, service, 1, 65536, 65536)?; + let mut ctx = match ShmContext::server_create(run_dir, service, 1, 65536, 65536) { + Ok(ctx) => ctx, + Err(netipc::transport::shm::ShmError::Allocate(libc::ENOSPC)) => { + eprintln!("NIPC_SHM_ALLOCATE_ENOSPC"); + return Err(netipc::transport::shm::ShmError::Allocate(libc::ENOSPC).into()); + } + Err(error) => return Err(error.into()), + }; // Signal readiness println!("READY"); diff --git a/tests/test_shm_full_backing_store.sh b/tests/test_shm_full_backing_store.sh new file mode 100755 index 00000000..36e78920 --- /dev/null +++ b/tests/test_shm_full_backing_store.sh @@ -0,0 +1,175 @@ +#!/usr/bin/env bash +# +# Verify that every Linux SHM creator returns a normal allocation error instead +# of receiving SIGBUS when its tmpfs backing store cannot hold the region. + +set -euo pipefail + +# shellcheck disable=SC1091 # Resolved relative to this script at runtime. +source "$(dirname "${BASH_SOURCE[0]}")/run-low-priority.sh" +netipc_low_priority_self "$@" + +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +GRAY='\033[0;90m' +NC='\033[0m' + +run() { + printf >&2 '%b%s >%b ' "${GRAY}" "$(pwd)" "${NC}" + printf >&2 '%b' "${YELLOW}" + printf >&2 "%q " "$@" + printf >&2 '%b\n' "${NC}" + + "$@" || { + local exit_code=$? + echo -e >&2 "${RED}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}" + echo -e >&2 "${RED}[ERROR]${NC} Command failed with exit code ${exit_code}: ${YELLOW}$1${NC}" + echo -e >&2 "${RED} Full command:${NC} $*" + echo -e >&2 "${RED} Working dir:${NC} $(pwd)" + echo -e >&2 "${RED}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}" + return "$exit_code" + } +} + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" + +: "${INTEROP_SHM_C:=${ROOT_DIR}/build/bin/interop_shm_c}" +: "${INTEROP_SHM_RS:=${ROOT_DIR}/src/crates/netipc/target/debug/interop_shm}" +: "${INTEROP_SHM_GO:=${ROOT_DIR}/build/bin/interop_shm_go}" +: "${NIPC_REQUIRE_TMPFS_TEST:=0}" + +TMP_ROOT="$(mktemp -d "${TMPDIR:-/tmp}/nipc_shm_full.XXXXXX")" +MOUNT_DIR="${TMP_ROOT}/shm" +LOG_DIR="${TMP_ROOT}/logs" + +cleanup() { + run rm -rf "${TMP_ROOT}" || true +} +trap cleanup EXIT + +skip_or_fail() { + local reason="$1" + if [[ "${NIPC_REQUIRE_TMPFS_TEST}" == "1" ]]; then + echo -e >&2 "${RED}[FAIL]${NC} ${reason}" + exit 1 + fi + echo -e >&2 "${YELLOW}[SKIP]${NC} ${reason}" + exit 77 +} + +for binary in "${INTEROP_SHM_C}" "${INTEROP_SHM_RS}" "${INTEROP_SHM_GO}"; do + if [[ ! -x "${binary}" ]]; then + echo -e >&2 "${RED}[FAIL]${NC} Missing SHM creator binary: ${binary}" + exit 1 + fi +done + +if ! command -v unshare >/dev/null 2>&1; then + skip_or_fail "unshare is required for the private tmpfs regression test" +fi +if ! command -v mount >/dev/null 2>&1; then + skip_or_fail "mount is required for the private tmpfs regression test" +fi +if ! command -v timeout >/dev/null 2>&1; then + skip_or_fail "timeout is required to bound SHM creator subprocesses" +fi + +run mkdir -p "${MOUNT_DIR}" "${LOG_DIR}" + +NAMESPACE_CMD=() + +if [[ ${EUID} -eq 0 ]]; then + printf >&2 "${GRAY}$(pwd) >${NC} ${YELLOW}%q %q %q %q %q %q %q${NC}\n" \ + unshare -m sh -eu -c '' "${MOUNT_DIR}" + # shellcheck disable=SC2016 # The script is evaluated inside the new namespace. + if unshare -m sh -eu -c ' + mount --make-rprivate / + mount -t tmpfs -o size=64k,nr_inodes=128,mode=0700 tmpfs "$1" + umount "$1" + ' sh "${MOUNT_DIR}" >/dev/null 2>&1; then + NAMESPACE_CMD=(unshare -m) + fi +else + printf >&2 "${GRAY}$(pwd) >${NC} ${YELLOW}%q %q %q %q %q %q %q${NC}\n" \ + unshare -Urnm sh -eu -c '' "${MOUNT_DIR}" + # shellcheck disable=SC2016 # The script is evaluated inside the new namespace. + if unshare -Urnm sh -eu -c ' + mount --make-rprivate / + mount -t tmpfs -o size=64k,nr_inodes=128,mode=0700 tmpfs "$1" + umount "$1" + ' sh "${MOUNT_DIR}" >/dev/null 2>&1; then + NAMESPACE_CMD=(unshare -Urnm) + fi +fi + +if [[ ${#NAMESPACE_CMD[@]} -eq 0 ]]; then + if [[ ${EUID} -eq 0 ]]; then + skip_or_fail "a privileged mount namespace cannot mount a private tmpfs" + fi + skip_or_fail "unprivileged user/mount namespaces cannot mount a private tmpfs" +fi + +# shellcheck disable=SC2016 # The script is evaluated inside the new namespace. +run "${NAMESPACE_CMD[@]}" bash -euo pipefail -c ' + mount_dir=$1 + log_dir=$2 + shift 2 + + trace() { + printf >&2 "%s > " "$PWD" + printf >&2 "%q " "$@" + printf >&2 "\n" + } + + trace mount --make-rprivate / + mount --make-rprivate / + trace mount -t tmpfs -o size=64k,nr_inodes=128,mode=0700 tmpfs "$mount_dir" + mount -t tmpfs -o size=64k,nr_inodes=128,mode=0700 tmpfs "$mount_dir" + trap '\''trace umount "$mount_dir"; umount "$mount_dir" 2>/dev/null || true'\'' EXIT + + index=0 + while [[ $# -gt 0 ]]; do + language=$1 + binary=$2 + shift 2 + index=$((index + 1)) + service="shm_full_${language}_${index}" + log="${log_dir}/${language}.log" + + set +e + trace timeout 10 "$binary" server "$mount_dir" "$service" + timeout 10 "$binary" server "$mount_dir" "$service" >"$log" 2>&1 + rc=$? + set -e + + if [[ $rc -ne 1 ]]; then + echo "${language}: creator returned unexpected status ${rc}; expected allocation failure status 1" >&2 + cat "$log" >&2 + exit 1 + fi + + if ! grep -Fxq NIPC_SHM_ALLOCATE_ENOSPC "$log"; then + echo "${language}: creator did not report the expected ENOSPC allocation-stage failure" >&2 + cat "$log" >&2 + exit 1 + fi + if grep -Fxq READY "$log"; then + echo "${language}: creator published readiness after allocation failure" >&2 + cat "$log" >&2 + exit 1 + fi + trace find "$mount_dir" -mindepth 1 -print -quit + if find "$mount_dir" -mindepth 1 -print -quit | grep -q .; then + echo "${language}: failed creation leaked a backing-store entry" >&2 + find "$mount_dir" -mindepth 1 -maxdepth 1 -printf "%f\n" >&2 + exit 1 + fi + done +' bash "${MOUNT_DIR}" "${LOG_DIR}" \ + c "${INTEROP_SHM_C}" \ + rust "${INTEROP_SHM_RS}" \ + go "${INTEROP_SHM_GO}" + +echo -e "${GREEN}[PASS]${NC} C, Rust, and Go returned normal errors on a full tmpfs; no SIGBUS or leaked region"