Skip to content

test: Fix flaky test_lifecycle_on_platform_without_websocket - #1121

Open
vdusek wants to merge 1 commit into
masterfrom
fix/flaky-ws-unreachable-port
Open

test: Fix flaky test_lifecycle_on_platform_without_websocket#1121
vdusek wants to merge 1 commit into
masterfrom
fix/flaky-ws-unreachable-port

Conversation

@vdusek

@vdusek vdusek commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

test_lifecycle_on_platform_without_websocket pointed the events WebSocket URL at a hard-coded ws://localhost:56565 and asserted that the connection is refused. That port sits inside the OS ephemeral range (32768-60999 on Linux, 49152-65535 on Windows), so any bind(host, 0) elsewhere in the parallel suite can be handed exactly that port, and this test file alone starts three such servers. When a sibling xdist worker got 56565, the connection succeeded and the test failed with DID NOT RAISE RuntimeError, as in this Windows run. Attempt 2 of the same run passed on the same commit.

The test now reserves a port instead of guessing one. A new _unreachable_ws_url helper binds ('127.0.0.1', 0) and holds the socket open without calling listen(). Holding it keeps the OS from handing the port to anyone else, and a bound socket that never listens refuses every connect with ECONNREFUSED, which is the OSError the test asserts as the cause. The assertions themselves are unchanged.

Running a real WebSocket server on 127.0.0.1:56565 reproduces the CI failure exactly, which gives a before/after: 20/20 runs failed before the fix, 0/100 after. The fixed test also survived 0/25 whole-file runs under --numprocesses=auto with that port occupied, and 0/60 runs with a background process holding ~400 rotating ephemeral listeners.

✍️ Drafted by Claude Code

@vdusek vdusek added adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team. labels Sep 7, 2026
@vdusek vdusek self-assigned this Sep 7, 2026
@github-actions github-actions Bot added this to the 149th sprint - Tooling team milestone Sep 7, 2026
@github-actions github-actions Bot added the tested Temporary label used only programatically for some analytics. label Sep 7, 2026
@codecov

codecov Bot commented Sep 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.78%. Comparing base (8532389) to head (d43fe2d).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1121      +/-   ##
==========================================
+ Coverage   92.75%   92.78%   +0.02%     
==========================================
  Files          53       53              
  Lines        3519     3519              
==========================================
+ Hits         3264     3265       +1     
+ Misses        255      254       -1     
Flag Coverage Δ
e2e 32.73% <ø> (ø)
integration 53.36% <ø> (+0.02%) ⬆️
unit 89.34% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vdusek
vdusek requested a review from Pijukatel September 7, 2026 13:09
@vdusek
vdusek marked this pull request as ready for review September 7, 2026 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants