TESTING (do not merge): identify the hanging pytest in CI

[ramakrishnap-nv]

Description

Testing / diagnosis only — do not merge.

Opened to get CI to tell us which tests are hanging in the pytest cuopt step. Not a fix.

The step has been hitting its time limit on 13.3.0 amd64 with 140 of 142 tests complete — two tests start and never finish. When the outer timeout fires, pytest is killed before it can report, and the conda path runs without -v, so the output is bare progress dots and the responsible tests cannot be identified. Raising the limit does not help: 30m and 45m both end at 140/142.

Adds three diagnostics to the shared runner so both the conda and wheel paths get them:

• -v names each test as it is dispatched.
• faulthandler_timeout dumps the stack of any test still running after FAULTHANDLER_TIMEOUT seconds (default 600), so a stuck test identifies itself while the step is still alive.
• --durations=25 surfaces tests approaching the limit.

None of these kill a test; they only make it visible.

Verified locally against a deliberately hanging test under xdist:

tests/test_hang_demo.py::test_hangs
Timeout (0:00:08)!
  File ".../tests/test_hang_demo.py", line 9 in test_hangs

The hang is not specific to any one PR — it reproduces on a branch that changes only ci/*.sh.

Once CI names the offending tests here, the real fix belongs in a separate PR against those tests.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3beb9995-929d-4bd5-8750-6705ec74ab5c

📥 Commits

Reviewing files that changed from the base of the PR and between bae1d87 and 9788729.

📒 Files selected for processing (1)

• ci/run_cuopt_pytests.sh

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.

---

📝 Walkthrough

Walkthrough

The CI pytest script adds configurable faulthandler timeouts, verbose dispatch, and reporting for the 25 slowest tests to nightly and non-nightly test runs.

Changes

Pytest diagnostics

Layer / File(s)	Summary
Apply pytest diagnostic flags ci/run_cuopt_pytests.sh	Defines FAULTHANDLER_TIMEOUT with a 600-second default. Applies verbose output, the timeout, and --durations=25 to both pytest invocation paths.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 97887

This change only improves visibility into hanging pytest runs; no actionable merge-blocking risk remains, so it is merge-ready after normal checks and review.

Suggested reviewers: kylefromnvidia

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1 files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly states that the change adds CI testing to identify a hanging pytest.
Description check	✅ Passed	The description accurately explains the pytest diagnostics, affected execution paths, and testing-only purpose.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

CI Test Summary

3 failed · 28 passed · 0 skipped

conda-cpp-tests / 13.3.0, 3.13, arm64, ubuntu26.04, l4, latest-driver, latest-deps — 2 failed tests

• DefaultServerTests.DeleteQueuedJobPreventsRun
• DefaultServerTests.DeleteRunningJobCancelsWorker