test: size the daemon-application wait budget for TSan's slowdown tail - #1224
Merged
Conversation
The suite's observation waits shared a 2000 ms budget that is a hang detector, not a latency assertion — every waiter returns the moment its condition holds, so green runs never pay it. Under ThreadSanitizer's 5-20x slowdown on a loaded 3-core macOS CI runner the cancel-delivery wait lost the tail of that distribution once in seven otherwise-green TSan rounds (the cancel itself had succeeded; only the fake worker's counter observation timed out). The budget is now 10000 ms. Verified: nine consecutive daemon_application runs under native macOS TSan with the widened budget, all green, zero ThreadSanitizer warnings. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The dry run's windows-11-arm shard failed its install-flow tests with the source-directory refusal the stamp exists to prevent — while the identical commands, run from the identically dirty state on the ARM64 VM, stamp clean and pass. The difference is invisible because the stamp was fully silenced; a silent load-bearing step cost a full CI round to even learn whether it had run. The stamp now reports each icacls failure with the user and directory, re-inspects the DACL afterwards, and prints one line in either direction: stamped clean, or the surviving cross-account grants. Verified on the VM from a deliberately reset (inherited, Authenticated- Users-writable) build directory through the real --par path: stamp reports clean, suites green. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
On one dry run the two arm shards split: the shard whose serial tail held the install-flow suites failed the source-directory policy minutes after its pre-wave stamp, while its sibling — same runner image, same stamp — passed with no install-flow suites to notice. Wave suites spawn Cygwin-family tooling that can rewrite the build directory's DACL behind the first stamp, so the stamp is now a function invoked both pre-wave and at the tail boundary, where the deadline-sensitive suites that depend on it actually run. Both invocations self-verify out loud. Verified on the VM from a deliberately reset (inherited) build directory through the real --par path: both stamps report clean, mini shard green. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The private-file prepare treated slow CONTENTION like permanent OBSTRUCTION: one 250 ms deadline for both. With many concurrent appenders on a loaded, sanitized runner the last writer can legitimately wait longer than that behind its peers' brief exclusive windows, and the concurrent-append guard correctly failed a dropped conflict event (dry run: one appender of the pack lost). Transient share/lock violations now retry against a 2000 ms contention budget while every other error class keeps the 250 ms fast-fail — a rejected client is still waiting behind this on the hello path. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The suite's observation waits shared a 2000 ms budget that is a hang detector, not a latency assertion — green runs never pay it. Under TSan's 5-20x slowdown on a loaded macOS runner, the cancel-delivery wait lost the distribution tail once in seven otherwise-green TSan rounds (the cancel itself succeeded; only the fake worker's counter observation timed out). Budget widened to 10000 ms.
Verified: nine consecutive
daemon_applicationruns under native macOS TSan with the widened budget — all green, zero ThreadSanitizer warnings.