Skip to content

refactor(bootstrapper): decompose monolithic bootstrapper into a subpackage#1226

Open
dhellmann wants to merge 19 commits into
python-wheel-build:mainfrom
dhellmann:task-specific-work-items
Open

refactor(bootstrapper): decompose monolithic bootstrapper into a subpackage#1226
dhellmann wants to merge 19 commits into
python-wheel-build:mainfrom
dhellmann:task-specific-work-items

Conversation

@dhellmann

@dhellmann dhellmann commented Jul 2, 2026

Copy link
Copy Markdown
Member

Summary

This branch refactors the monolithic bootstrapper.py (2278 lines) into a well-structured subpackage with focused modules, making the code easier to navigate, test, and extend.

What changed

  • Extracted phase class hierarchy (Phase and its subclasses) out of Bootstrapper into individual modules, one per bootstrap phase
  • Converted bootstrapper.py to a subpackage (bootstrapper/) with 10 focused modules: _phase.py, _resolve.py, _start.py, _prepare_source.py, _prepare_build.py, _build.py, _process_install_deps.py, _complete.py, _work_item.py, _cache.py
  • Moved logic to the right owners: do_build and build helpers → Build, install-dep extraction → module-level function in _process_install_deps, processing_build_requirement state → WorkItem
  • Cleaned up Bootstrapper: inlined _create_build_env, moved exclusive-build pool drain into the loop, made internal members public where phase classes need them, exposed _resolver as a property, wrapped _failed_versions behind has_failed_version()
  • Renamed phase classes to drop the redundant Item suffix (PhaseItemPhase, BuildItemBuild, ResolveItemResolve, etc.) and renamed files to match

Commit sequence

  1. design: phase classes — design doc / plan
  2. refactor: extract PhaseItem class hierarchy — initial extraction
  3. Series of encapsulation commits making Bootstrapper members accessible to phase classes
  4. Logic migration commits (do_build, install deps, WorkItem state)
  5. refactor: convert bootstrapper.py to a subpackage
  6. refactor: rename phase classes and files, drop "Item" suffix

Test plan

  • All existing tests pass (hatch run test:test)
  • mypy clean (hatch run mypy:check)
  • lint clean (hatch run lint:check)

Advice to reviewers

There are no new features here, but there are a lot of code changes. It may be easier to review them by looking at the commit history individually instead of trying to understand the entire changeset as one big diff.

🤖 Generated with Claude Code

@dhellmann dhellmann requested a review from a team as a code owner July 2, 2026 22:51
@coderabbitai

coderabbitai Bot commented Jul 2, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@dhellmann, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 36 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: d40029ec-2272-4304-89e5-5ee3fbfd3387

📥 Commits

Reviewing files that changed from the base of the PR and between 9ece337 and 8d117d2.

📒 Files selected for processing (3)
  • src/fromager/bootstrapper/_bootstrapper.py
  • tests/test_bootstrap.py
  • tests/test_bootstrapper.py
📝 Walkthrough

Walkthrough

Changes

The old src/fromager/bootstrapper.py module is removed and replaced by a src/fromager/bootstrapper package. The package adds shared bootstrap types, a WorkItem model, an abstract Phase base class, seven concrete phases, cache and prebuilt-wheel helpers, and a new Bootstrapper orchestrator. The bootstrap flow now runs through Resolve, Start, PrepareSource, PrepareBuild, Build, ProcessInstallDeps, and Complete, with background work, dependency-graph updates, build-order persistence, git URL handling, and test-mode/multiple-versions failure paths. Tests were rewritten around the new phase API and helpers.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Bootstrapper
  participant Resolve
  participant Start
  participant PrepareSource
  participant PrepareBuild
  participant Build
  participant ProcessInstallDeps
  participant Complete

  Bootstrapper->>Resolve: run(bt)
  Resolve->>Start: returns Start(work_item)
  Start->>PrepareSource: returns PrepareSource(wi)
  PrepareSource->>PrepareBuild: returns PrepareBuild(wi) + deps
  PrepareBuild->>Build: returns Build(wi) + deps
  Build->>ProcessInstallDeps: returns ProcessInstallDeps(wi)
  ProcessInstallDeps->>Complete: returns Complete(wi) + deps
  Complete->>Bootstrapper: returns []
Loading

Related PRs: None specified.

Suggested labels: refactor, bootstrapper, tests

Suggested reviewers: None specified.

🐰 one old module now split in parts,
phases and helpers, graphs and starts,
tests now follow the new road map,
and the loop still lands on its final clap.

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main change: refactoring the monolithic bootstrapper into a subpackage.
Description check ✅ Passed The description is clearly aligned with the refactor and matches the modules, phase extraction, and cleanup shown in the changeset.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@mergify mergify Bot added the ci label Jul 2, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (5)
src/fromager/bootstrapper/_phase.py (1)

17-33: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

phase ClassVar has no default and isn't enforced by abc.

A subclass that forgets to declare phase won't fail at class-definition time — it'll only blow up with AttributeError the first time something accesses self.phase (e.g. in as_json()). Since run() is the only abstractmethod, phase is easy to miss when adding a new concrete subclass.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/fromager/bootstrapper/_phase.py` around lines 17 - 33, The Phase base
class currently relies on subclasses to remember to define the phase ClassVar,
which can slip through until runtime. Make phase mandatory by enforcing it in
the Phase hierarchy, either by turning it into an abstract class attribute
pattern or by validating it when concrete subclasses are created, and ensure any
subclass access points like as_json() can safely assume phase is present. Refer
to the Phase class and its subclass implementations when applying the fix.
src/fromager/bootstrapper/_cache.py (1)

212-227: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Reuse _create_unpack_dir instead of duplicating its logic.

Lines 224-225 duplicate the exact body of _create_unpack_dir (lines 30-31) instead of calling it.

♻️ Proposed fix
     wheel_filename = wheels.download_wheel(req, wheel_url, ctx.wheels_prebuilt)
-    unpack_dir = ctx.work_dir / f"{req.name}-{resolved_version}"
-    unpack_dir.mkdir(parents=True, exist_ok=True)
+    unpack_dir = _create_unpack_dir(ctx.work_dir, req, resolved_version)
     server.update_wheel_mirror(ctx)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/fromager/bootstrapper/_cache.py` around lines 212 - 227, The
`_bg_prepare_prebuilt` function duplicates unpack-directory creation logic that
already exists in `_create_unpack_dir`. Replace the inline `unpack_dir =
ctx.work_dir / f"{req.name}-{resolved_version}"` and `mkdir(..., exist_ok=True)`
sequence with a call to `_create_unpack_dir(req, resolved_version,
ctx.work_dir)` so the unpack path is built in one place and stays consistent
with the existing helper.
src/fromager/bootstrapper/_resolve.py (1)

88-126: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy lift

Synchronous per-version cache-server lookups in run().

The multiple_versions filter loop calls _cache._find_cached_wheel synchronously once per candidate version. _find_cached_wheel can fall through to _download_wheel_from_cache, which does network I/O against the cache server — unlike other phases in this refactor, which push such I/O into background_work/bg_future. For packages with many candidate versions this can serialize several network round trips on the main dispatch path.

Consider moving the cache lookup for each candidate version into background work (or parallelizing across the thread pool) to keep this consistent with the rest of the phase architecture.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/fromager/bootstrapper/_resolve.py` around lines 88 - 126, The
multiple-versions filtering in `_resolve.py` is doing synchronous cache-server
checks one version at a time via `_cache._find_cached_wheel`, which blocks the
main `run()` path on network I/O. Move the per-candidate wheel cache lookup into
background work or equivalent parallel execution, and then apply the same
cached/non-cached filtering logic to the results before building `filtered` in
the `multiple_versions` branch. Keep the existing behavior around retaining the
highest version when everything is cached, but ensure the lookup work is
scheduled consistently with the rest of the background-based phase handling.
src/fromager/bootstrapper/_bootstrapper.py (2)

814-837: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Missing docstrings on public Bootstrapper methods.

add_to_graph (814-837), has_been_seen (874-881), and add_to_build_order (883-918) are public methods without docstrings, unlike their siblings (mark_as_seen, create_unresolved_work_items, etc.).

📝 Suggested docstrings
+    def add_to_graph(
         self,
         req: Requirement,
         req_type: RequirementType,
         req_version: Version,
         download_url: str,
         parent: tuple[Requirement, Version] | None,
     ) -> None:
+        """Add a resolved dependency edge to the dependency graph and persist it."""
         parent_req, parent_version = parent if parent else (None, None)
     def has_been_seen(
         self,
         req: Requirement,
         version: Version,
         sdist_only: bool = False,
     ) -> bool:
+        """Return True if this (req, version) has already been built (sdist or wheel)."""
         typ: typing.Literal["sdist", "wheel"] = "sdist" if sdist_only else "wheel"
     def add_to_build_order(
         self,
         req: Requirement,
         version: Version,
         source_url: str,
         source_type: SourceType,
         prebuilt: bool = False,
         constraint: Requirement | None = None,
     ) -> None:
+        """Record a built package in the build-order file, deduplicating by name+version."""
         # We only care if this version of this package has been built,

Based on learnings, as per coding guidelines: **/*.py: "Add docstrings on all public functions and classes".

Also applies to: 855-918

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/fromager/bootstrapper/_bootstrapper.py` around lines 814 - 837, Public
Bootstrapper methods are missing required docstrings, specifically add
documentation to add_to_graph, has_been_seen, and add_to_build_order so they
match the style of neighboring public methods like mark_as_seen and
create_unresolved_work_items. Add concise docstrings that describe each method’s
purpose, inputs, and side effects, and ensure all public functions/classes in
this area comply with the docstring guideline.

Source: Coding guidelines


304-319: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Exclusive-build log line issued outside req_ctxvar_context.

The "requires exclusive build, draining background pool" log (line 310-313) fires before entering the req_ctxvar_context block below (316-319), so it won't get the automatic per-requirement log prefix used elsewhere in the codebase.

♻️ Suggested fix
             if item.requires_exclusive_run:
-                logger.info(
-                    "%s requires exclusive build, draining background pool",
-                    item.work_item.req,
-                )
-                self._drain_background_pool()
-
-            with (
                 req_ctxvar_context(item.work_item.req, item.work_item.resolved_version),
                 self._track_why(item),
             ):
+                if item.requires_exclusive_run:
+                    logger.info("requires exclusive build, draining background pool")
+                    self._drain_background_pool()
+
                 try:

As per coding guidelines: "Use req_ctxvar_context() for per-requirement logging".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/fromager/bootstrapper/_bootstrapper.py` around lines 304 - 319, The
exclusive-build log in the bootstrapper loop is emitted before entering
req_ctxvar_context, so it misses the per-requirement logging prefix. Move the
logger.info call in the stack-processing path inside the req_ctxvar_context
block in _bootstrapper.py, near the existing _track_why context, so the
"requires exclusive build, draining background pool" message is logged with the
active requirement context.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/fromager/bootstrapper/_bootstrapper.py`:
- Around line 814-837: Public Bootstrapper methods are missing required
docstrings, specifically add documentation to add_to_graph, has_been_seen, and
add_to_build_order so they match the style of neighboring public methods like
mark_as_seen and create_unresolved_work_items. Add concise docstrings that
describe each method’s purpose, inputs, and side effects, and ensure all public
functions/classes in this area comply with the docstring guideline.
- Around line 304-319: The exclusive-build log in the bootstrapper loop is
emitted before entering req_ctxvar_context, so it misses the per-requirement
logging prefix. Move the logger.info call in the stack-processing path inside
the req_ctxvar_context block in _bootstrapper.py, near the existing _track_why
context, so the "requires exclusive build, draining background pool" message is
logged with the active requirement context.

In `@src/fromager/bootstrapper/_cache.py`:
- Around line 212-227: The `_bg_prepare_prebuilt` function duplicates
unpack-directory creation logic that already exists in `_create_unpack_dir`.
Replace the inline `unpack_dir = ctx.work_dir /
f"{req.name}-{resolved_version}"` and `mkdir(..., exist_ok=True)` sequence with
a call to `_create_unpack_dir(req, resolved_version, ctx.work_dir)` so the
unpack path is built in one place and stays consistent with the existing helper.

In `@src/fromager/bootstrapper/_phase.py`:
- Around line 17-33: The Phase base class currently relies on subclasses to
remember to define the phase ClassVar, which can slip through until runtime.
Make phase mandatory by enforcing it in the Phase hierarchy, either by turning
it into an abstract class attribute pattern or by validating it when concrete
subclasses are created, and ensure any subclass access points like as_json() can
safely assume phase is present. Refer to the Phase class and its subclass
implementations when applying the fix.

In `@src/fromager/bootstrapper/_resolve.py`:
- Around line 88-126: The multiple-versions filtering in `_resolve.py` is doing
synchronous cache-server checks one version at a time via
`_cache._find_cached_wheel`, which blocks the main `run()` path on network I/O.
Move the per-candidate wheel cache lookup into background work or equivalent
parallel execution, and then apply the same cached/non-cached filtering logic to
the results before building `filtered` in the `multiple_versions` branch. Keep
the existing behavior around retaining the highest version when everything is
cached, but ensure the lookup work is scheduled consistently with the rest of
the background-based phase handling.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 8d6d2213-e334-4785-96a4-204ebc0ac369

📥 Commits

Reviewing files that changed from the base of the PR and between c9ddc51 and b2d0605.

📒 Files selected for processing (17)
  • docs/proposals/phase-classes-plan.md
  • src/fromager/bootstrapper.py
  • src/fromager/bootstrapper/__init__.py
  • src/fromager/bootstrapper/_bootstrapper.py
  • src/fromager/bootstrapper/_build.py
  • src/fromager/bootstrapper/_cache.py
  • src/fromager/bootstrapper/_complete.py
  • src/fromager/bootstrapper/_phase.py
  • src/fromager/bootstrapper/_prepare_build.py
  • src/fromager/bootstrapper/_prepare_source.py
  • src/fromager/bootstrapper/_process_install_deps.py
  • src/fromager/bootstrapper/_resolve.py
  • src/fromager/bootstrapper/_start.py
  • src/fromager/bootstrapper/_types.py
  • src/fromager/bootstrapper/_work_item.py
  • tests/test_bootstrapper.py
  • tests/test_bootstrapper_iterative.py
💤 Files with no reviewable changes (1)
  • src/fromager/bootstrapper.py

@dhellmann dhellmann requested review from rd4398 and tiran July 6, 2026 13:25

@rd4398 rd4398 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this refactoring. It makes code more readable and cleaner for sure. I have added few comments / sugegstions. Lets see what @tiran and / or @LalatenduMohanty has to say

Comment thread docs/proposals/phase-classes-plan.md Outdated
Comment thread docs/proposals/phase-classes-plan.md Outdated
Comment thread docs/proposals/phase-classes-plan.md Outdated
Comment thread src/fromager/bootstrapper/_cache.py Outdated
Comment thread src/fromager/bootstrapper/_start.py
@dhellmann

Copy link
Copy Markdown
Member Author

I've pushed updates to respond to the coderabbit feedback.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/fromager/bootstrapper/_phase.py (1)

28-48: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add docstrings for consistency with the rest of the class.

__init_subclass__ (lines 28-44) and __init__ (lines 45-48) rely on inline comments only, while every other method here (run, requires_exclusive_run, background_work, __str__, as_json) has a docstring. The __init_subclass__ logic in particular is subtle enough (ABCMeta ordering workaround) to deserve a proper docstring rather than just a trailing comment block.

📝 Suggested docstrings
     def __init_subclass__(cls, **kwargs: typing.Any) -> None:
+        """Validate that concrete subclasses define the `phase` class attribute.
+
+        Raises:
+            TypeError: If a non-abstract subclass omits `phase`.
+        """
         super().__init_subclass__(**kwargs)
     def __init__(self, work_item: WorkItem) -> None:
+        """Wrap `work_item` and initialize the background-future slot."""
         self.work_item = work_item
         self.bg_future: concurrent.futures.Future[typing.Any] | None = None

As per coding guidelines, "Add docstrings on all public functions and classes".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/fromager/bootstrapper/_phase.py` around lines 28 - 48, Add docstrings to
the public methods __init_subclass__ and __init__ in the Bootstrapper base class
for consistency with run, requires_exclusive_run, background_work, __str__, and
as_json. Describe the subclass-phase enforcement behavior in __init_subclass__
(including the ABCMeta ordering workaround) and the initialization of work_item
and bg_future in __init__, replacing the existing inline comment block with the
docstring on __init_subclass__.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/fromager/bootstrapper/_phase.py`:
- Around line 28-48: Add docstrings to the public methods __init_subclass__ and
__init__ in the Bootstrapper base class for consistency with run,
requires_exclusive_run, background_work, __str__, and as_json. Describe the
subclass-phase enforcement behavior in __init_subclass__ (including the ABCMeta
ordering workaround) and the initialization of work_item and bg_future in
__init__, replacing the existing inline comment block with the docstring on
__init_subclass__.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 48feda80-1f47-4ded-9093-26426d03840d

📥 Commits

Reviewing files that changed from the base of the PR and between b2d0605 and c136b16.

📒 Files selected for processing (11)
  • src/fromager/bootstrapper/_bootstrapper.py
  • src/fromager/bootstrapper/_build.py
  • src/fromager/bootstrapper/_cache.py
  • src/fromager/bootstrapper/_complete.py
  • src/fromager/bootstrapper/_phase.py
  • src/fromager/bootstrapper/_prepare_build.py
  • src/fromager/bootstrapper/_prepare_source.py
  • src/fromager/bootstrapper/_process_install_deps.py
  • src/fromager/bootstrapper/_resolve.py
  • src/fromager/bootstrapper/_start.py
  • tests/test_bootstrapper.py
🚧 Files skipped from review as they are similar to previous changes (10)
  • src/fromager/bootstrapper/_complete.py
  • src/fromager/bootstrapper/_prepare_build.py
  • src/fromager/bootstrapper/_process_install_deps.py
  • src/fromager/bootstrapper/_build.py
  • src/fromager/bootstrapper/_resolve.py
  • src/fromager/bootstrapper/_start.py
  • src/fromager/bootstrapper/_prepare_source.py
  • src/fromager/bootstrapper/_cache.py
  • src/fromager/bootstrapper/_bootstrapper.py
  • tests/test_bootstrapper.py

@rd4398

rd4398 commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

I've pushed updates to respond to the coderabbit feedback.

This looks good! Thanks

@dhellmann dhellmann force-pushed the task-specific-work-items branch from c136b16 to 147f8fb Compare July 7, 2026 12:29

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Nitpick comments (1)
src/fromager/bootstrapper/_bootstrapper.py (1)

51-52: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add a class docstring for the public Bootstrapper API.

As per coding guidelines, **/*.py: “Add docstrings on all public functions and classes.”

Suggested change
 class Bootstrapper:
+    """Coordinate requirement resolution, dependency bootstrapping, and build phases."""
     def __init__(
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/fromager/bootstrapper/_bootstrapper.py` around lines 51 - 52, The public
Bootstrapper class is missing a docstring. Add a class-level docstring directly
on Bootstrapper to describe its purpose and high-level role in the bootstrapping
API, following the project’s public API documentation guideline for classes.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/fromager/bootstrapper/_bootstrapper.py`:
- Around line 1283-1285: The context-manager cleanup in the bootstrapper is
shutting down the background pool without waiting, so running tasks can outlive
`__exit__` and race with cleanup. Update the shutdown logic in the
bootstrapper’s background-pool cleanup path to wait for active work to finish
instead of returning immediately, and keep the pool reset logic in the same
`_bg_pool` handling block so cleanup only proceeds after background tasks have
completed.
- Around line 518-520: The install guard in the bootstrapper is too broad: using
build_dependencies.isdisjoint(build_system_dependencies) skips installing every
backend/sdist dependency whenever there is any overlap. Update the logic around
build_dependencies and build_env.install() so it filters out only the
dependencies already satisfied by build_system_dependencies and installs the
remaining ones, rather than skipping the whole set.
- Around line 760-761: Create the destination parent directory before calling
shutil.move in the bootstrapper’s git clone flow. In the code around
logger.info("moving cloned repo to %s", working_src_dir) and
shutil.move(clone_dir, str(working_src_dir)), ensure working_src_dir.parent (or
the relevant nested parent for the first-time versioned path) is created so
first-time git URL resolution does not fail with FileNotFoundError. Keep the fix
in the bootstrapper path handling near the clone/move logic.

In `@tests/test_bootstrapper.py`:
- Around line 1052-1063: The wheel-build test patches
fromager.server.update_wheel_mirror but never verifies the call, so the test
does not actually cover the mirror-update behavior. In the item._build_wheel
test, add an assertion on the mocked update_wheel_mirror alongside the existing
mock_build_sdist checks, and make sure the assertion matches the expected
arguments from _build_wheel(tmp_context) so the test fails if the wiring is
removed.

---

Nitpick comments:
In `@src/fromager/bootstrapper/_bootstrapper.py`:
- Around line 51-52: The public Bootstrapper class is missing a docstring. Add a
class-level docstring directly on Bootstrapper to describe its purpose and
high-level role in the bootstrapping API, following the project’s public API
documentation guideline for classes.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: c79e8c11-99fd-4896-be7d-139abecfb41b

📥 Commits

Reviewing files that changed from the base of the PR and between c136b16 and 147f8fb.

📒 Files selected for processing (16)
  • src/fromager/bootstrapper.py
  • src/fromager/bootstrapper/__init__.py
  • src/fromager/bootstrapper/_bootstrapper.py
  • src/fromager/bootstrapper/_build.py
  • src/fromager/bootstrapper/_cache.py
  • src/fromager/bootstrapper/_complete.py
  • src/fromager/bootstrapper/_phase.py
  • src/fromager/bootstrapper/_prepare_build.py
  • src/fromager/bootstrapper/_prepare_source.py
  • src/fromager/bootstrapper/_process_install_deps.py
  • src/fromager/bootstrapper/_resolve.py
  • src/fromager/bootstrapper/_start.py
  • src/fromager/bootstrapper/_types.py
  • src/fromager/bootstrapper/_work_item.py
  • tests/test_bootstrapper.py
  • tests/test_bootstrapper_iterative.py
💤 Files with no reviewable changes (1)
  • src/fromager/bootstrapper.py
🚧 Files skipped from review as they are similar to previous changes (13)
  • src/fromager/bootstrapper/_work_item.py
  • src/fromager/bootstrapper/_prepare_build.py
  • src/fromager/bootstrapper/_complete.py
  • src/fromager/bootstrapper/init.py
  • src/fromager/bootstrapper/_phase.py
  • src/fromager/bootstrapper/_types.py
  • src/fromager/bootstrapper/_start.py
  • src/fromager/bootstrapper/_prepare_source.py
  • src/fromager/bootstrapper/_build.py
  • src/fromager/bootstrapper/_resolve.py
  • src/fromager/bootstrapper/_process_install_deps.py
  • src/fromager/bootstrapper/_cache.py
  • tests/test_bootstrapper_iterative.py

Comment thread src/fromager/bootstrapper/_bootstrapper.py
Comment thread src/fromager/bootstrapper/_bootstrapper.py
Comment thread src/fromager/bootstrapper/_bootstrapper.py
Comment thread tests/test_bootstrapper.py
dhellmann and others added 17 commits July 7, 2026 11:27
Move phase functions out of the bootstrapper into their own classes.

Signed-off-by: Doug Hellmann <dhellmann@redhat.com>
…rapper

Move the seven `_phase_*` handler methods out of `Bootstrapper` and
into a `PhaseItem` abstract base class hierarchy. Each concrete
subclass (`ResolveItem`, `StartItem`, `PrepareSourceItem`,
`PrepareBuildItem`, `BuildItem`, `ProcessInstallDepsItem`,
`CompleteItem`) encodes one bootstrap phase and implements its own
`run()` method, making each phase self-contained.

Key changes:
- Add `PhaseItem` ABC with `phase`/`tracks_why` ClassVars, `bg_future`,
  `background_work()`, `__str__()`, and `as_json()`
- Add 7 concrete `PhaseItem` subclasses; move `_phase_*` bodies into
  their `run()` methods
- Remove `phase` and `bg_future` fields from `WorkItem`
- Remove `tracks_why` property from `BootstrapPhase` enum
- Remove `_dispatch_phase` (was a one-liner: `item.run(self)`) and
  inline the call in `_run_bootstrap_loop`
- Remove `_get_background_work`; replace with `PhaseItem.background_work()`
- Remove all 7 `_phase_*` methods from `Bootstrapper`
- Simplify `_push_items`, `_track_why`, `_record_stack_state`,
  `_handle_phase_error`, and related methods to use `PhaseItem`
- Update tests to use `PhaseItem` subclass instances and `item.run(bt)`

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Doug Hellmann <dhellmann@redhat.com>
Add a public `resolver` property to the Bootstrapper class to provide
a public API for accessing the version resolver. This allows
ResolveItem.background_work() to access the resolver through the
public property instead of directly touching the private _resolver
attribute, maintaining the boundary between PhaseItem and Bootstrapper.

Co-Authored-By: Claude <claude@anthropic.com>
Signed-off-by: Doug Hellmann <dhellmann@redhat.com>
…rsion()

Replace direct access to _failed_versions dict in ResolveItem.run() with a
public has_failed_version() method. This maintains encapsulation of the private
dict and provides a clean public API for checking if a package version has
previously failed resolution.

Co-Authored-By: Claude <claude@anthropic.com>
Signed-off-by: Doug Hellmann <dhellmann@redhat.com>
Remove leading underscores from Bootstrapper methods that are accessed
from StartItem.run() to fix cross-boundary private-member access:

- _add_to_graph → add_to_graph
- _processing_build_requirement → processing_build_requirement
- _has_been_seen → has_been_seen
- _mark_as_seen → mark_as_seen
- _explain → explain

Also update internal callers within Bootstrapper:
- _filter_deps_satisfied_by_build_system() calls self.add_to_graph()
- _build_wheel() calls self.explain

Update all references in tests/test_bootstrapper.py and
tests/test_bootstrapper_iterative.py.

Co-Authored-By: Claude <claude@anthropic.com>
Signed-off-by: Doug Hellmann <dhellmann@redhat.com>
…ved_work_items public

Inline the thin wrapper _create_build_env() directly in PrepareSourceItem.run()
since its req and resolved_version parameters were unused. This simplifies the
code and removes an unnecessary layer of indirection.

Rename _create_unresolved_work_items() to create_unresolved_work_items() to make
it public, as it is called from three different PhaseItem subclasses
(PrepareSourceItem, PrepareBuildItem, ProcessInstallDepsItem).

Update all call sites in bootstrapper.py and test references in
test_bootstrapper_iterative.py.

Co-Authored-By: Claude <claude@anthropic.com>
Signed-off-by: Doug Hellmann <dhellmann@redhat.com>
Remove leading underscores from `_get_resolved_build_system_versions` and
`_filter_deps_satisfied_by_build_system` to make them public methods. These
methods are accessed from `PrepareBuildItem.run()` across module boundaries
and should be part of the public API.

Co-Authored-By: Claude <claude@anthropic.com>
Signed-off-by: Doug Hellmann <dhellmann@redhat.com>
Rename _drain_background_pool() to drain_background_pool() and
_do_build() to do_build() to indicate these methods are accessed from
BuildItem.run(). Internal methods _build_sdist() and _build_wheel()
remain private as they are only called within Bootstrapper.

Co-Authored-By: Claude <claude@anthropic.com>
Signed-off-by: Doug Hellmann <dhellmann@redhat.com>
…public

Rename three private methods to public to support ProcessInstallDepsItem.run()
accessing them without underscore prefix:
- _record_test_mode_failure() -> record_test_mode_failure()
- _get_install_dependencies() -> get_install_dependencies()
- _add_to_build_order() -> add_to_build_order()

All internal callers and test references have been updated accordingly.

Co-Authored-By: Claude <claude@anthropic.com>
Signed-off-by: Doug Hellmann <dhellmann@redhat.com>
Each PhaseItem now declares whether it requires exclusive execution via
a `requires_exclusive_run` property (default False). `_run_bootstrap_loop`
drains the background pool before dispatching any item that returns True,
rather than having `BuildItem.run()` reach into the Bootstrapper to do it.

`BuildItem` overrides `requires_exclusive_run` to return
`self.work_item.exclusive_build`, which is cached on `WorkItem` during
`StartItem.run()` alongside `pbi_pre_built`.

`drain_background_pool` is now private again since it's only called
from within the Bootstrapper loop.

Co-Authored-By: Claude <claude@anthropic.com>
Signed-off-by: Doug Hellmann <dhellmann@redhat.com>
…vel function

The method only used `self.ctx` from the Bootstrapper; all other
arguments were passed explicitly. It now follows the same pattern as
`_bg_resolve`, `_bg_prepare_source`, and `_find_cached_wheel`: a
module-level function that takes `ctx` directly.

Co-Authored-By: Claude <claude@anthropic.com>
Signed-off-by: Doug Hellmann <dhellmann@redhat.com>
The method only needed `self.req_type` and `self.why` (which equals
`wi.why_snapshot` at call time in the bootstrap loop). It is now
`WorkItem.is_build_requirement_context()`, reading `self.req_type` and
`self.why_snapshot` directly with no parameters.

Co-Authored-By: Claude <claude@anthropic.com>
Signed-off-by: Doug Hellmann <dhellmann@redhat.com>
`do_build()`, `_build_sdist()`, and `_build_wheel()` on `Bootstrapper`
are only called from `BuildItem.run()` and operate exclusively on
`WorkItem` state.  Moving them to `BuildItem` eliminates the need to
pass `req`, `resolved_version`, and other `WorkItem` fields as
parameters: the methods read them from `self.work_item` directly.

`BuildItem.run()` now calls `self.do_build(bt.ctx, bt.explain)` instead
of `bt.do_build(req=wi.req, ...)`.

The only external state needed from `Bootstrapper` is:
- `ctx` (WorkContext) — passed explicitly
- `explain` (formatted why-stack string used in one log line) — passed explicitly

Existing tests that patched `bt.do_build` or `bt._build_wheel` are
updated to patch the corresponding `BuildItem` instance methods instead.
New unit tests cover the three `BuildItem` methods directly.

Co-Authored-By: Claude <claude@anthropic.com>
Signed-off-by: Doug Hellmann <dhellmann@redhat.com>
Split the 2,351-line `bootstrapper.py` into a subpackage
`src/fromager/bootstrapper/` with one module per logical layer:

- `_types.py`: type aliases, dataclasses, enums, and constants
- `_cache.py`: shared wheel-cache and prebuilt-download helpers
- `_work_item.py`: `WorkItem` dataclass
- `_phase_item.py`: `PhaseItem` ABC
- `_resolve_item.py`: `ResolveItem` + `_bg_resolve`
- `_start_item.py`: `StartItem`
- `_prepare_source_item.py`: `PrepareSourceItem` + `_bg_prepare_source`
- `_prepare_build_item.py`: `PrepareBuildItem`
- `_build_item.py`: `BuildItem`
- `_process_install_deps_item.py`: `ProcessInstallDepsItem` + `_get_install_dependencies`
- `_complete_item.py`: `CompleteItem`
- `_bootstrapper.py`: `Bootstrapper` class
- `__init__.py`: exports `Bootstrapper` and `_DEFAULT_BG_THREADS` only

Phase modules that call `_cache` helpers use module-level imports
(`from . import _cache`) so mock patch targets remain unambiguous
(e.g. `fromager.bootstrapper._cache._find_cached_wheel`).

Tests updated to import directly from submodules and use the new
patch paths.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Doug Hellmann <dhellmann@redhat.com>
…uffix

Rename all phase subclasses of `PhaseItem` to drop the redundant "Item"
suffix, and rename the abstract base `PhaseItem` to `Phase`. File names
follow the class names.

Class renames:
- PhaseItem     → Phase
- ResolveItem   → Resolve
- StartItem     → Start
- PrepareSourceItem → PrepareSource
- PrepareBuildItem  → PrepareBuild
- BuildItem     → Build
- ProcessInstallDepsItem → ProcessInstallDeps
- CompleteItem  → Complete

File renames (in src/fromager/bootstrapper/):
- _phase_item.py            → _phase.py
- _resolve_item.py          → _resolve.py
- _start_item.py            → _start.py
- _prepare_source_item.py   → _prepare_source.py
- _prepare_build_item.py    → _prepare_build.py
- _build_item.py            → _build.py
- _process_install_deps_item.py → _process_install_deps.py
- _complete_item.py         → _complete.py

`WorkItem` and `_work_item.py` are unchanged.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Doug Hellmann <dhellmann@redhat.com>
…og context

- Add docstrings to `add_to_graph`, `has_been_seen`, and `add_to_build_order`
  in `Bootstrapper`
- Move the "exclusive build, draining background pool" log and
  `_drain_background_pool()` call inside `req_ctxvar_context` so the message
  carries the per-requirement log prefix
- Use the `_create_unpack_dir` helper in `_bg_prepare_prebuilt` instead of
  duplicating the inline path-and-mkdir logic
- Add `Phase.__init_subclass__` to raise `TypeError` at class-definition time
  when a concrete subclass omits the required `phase` ClassVar; add two tests
- Expand class docstrings on all seven `Phase` subclasses to describe each
  phase's role and its next-phase transitions

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Doug Hellmann <dhellmann@redhat.com>
The doc added to the proposals directory was the agent's plan, not
really a proposal. Clean that up.

Signed-off-by: Doug Hellmann <dhellmann@redhat.com>
@dhellmann dhellmann force-pushed the task-specific-work-items branch from 147f8fb to 9ece337 Compare July 7, 2026 15:27
dhellmann and others added 2 commits July 7, 2026 11:49
…n git clone flow

On first-time git URL resolution, working_src_dir.parent did not exist
yet, causing shutil.move to raise FileNotFoundError. The mkdir call was
inside the `if working_src_dir.exists()` branch, so it was skipped when
the destination was new. Move it unconditionally before shutil.move, and
add an assert to satisfy the type checker.

Adds a regression test that lets the real filesystem run (no shutil.move
mock) to verify the parent directory is created before the move.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Doug Hellmann <dhellmann@redhat.com>
…eel test

The wheel-build test patched `fromager.server.update_wheel_mirror` but
never asserted the call, so removing the wiring would go undetected.
Capture the mock and assert it is called once with the expected context.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Doug Hellmann <dhellmann@redhat.com>
@dhellmann dhellmann requested a review from rd4398 July 7, 2026 15:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants