Skip to content

fix: restore the package import and reconcile the merged backend behaviour - #66

Open
Kartikey1306 wants to merge 2 commits into
embeddedos-org:masterfrom
Kartikey1306:fix/restore-master-build-and-tests
Open

fix: restore the package import and reconcile the merged backend behaviour#66
Kartikey1306 wants to merge 2 commits into
embeddedos-org:masterfrom
Kartikey1306:fix/restore-master-build-and-tests

Conversation

@Kartikey1306

@Kartikey1306 Kartikey1306 commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

The architectural problem

import ebuild.build.dispatch raises SyntaxError on master. The CLI does not start, and five test modules fail at collection.

Same root cause as the breakage in the sibling repos: overlapping PRs squash-merged on stale bases, with master never re-verified afterwards. Every one of the PRs involved was green on its own branch. What landed is a build tool that cannot be imported, and — once it can be — a default build backend that crashes.

What is broken

dispatch.py does not parse

configure() ends with two consecutive else: blocks from an unresolved conflict.

The two blocks also disagree about the exception type, and each has a test suite behind it:

  • tests/ebuild/test_dispatch.py expects ValueError("Unknown build backend '<name>'")
  • tests/unit/test_dispatch.py expects RuntimeError matching "ninja"

Both readings are legitimate. An unrecognized backend name is a bad argument; a "ninja" that reaches the dispatcher is a CLI routing failure, since ebuild's own ninja backend is invoked directly and never dispatched here. Rather than pick a winner and delete someone's regression test, this adds UnknownBackendError(ValueError, RuntimeError) with a message that covers both cases, raised from configure(), build() and clean().

"ninja" is also no longer a silent no-op in configure(). That silence is what let ebuild build report "Build completed successfully" without ever running a compiler — the exact failure tests/unit/test_dispatch.py was written to prevent.

The default build backend crashes

NinjaBackend._object_path() was deleted by a cflags refactor while both of its callers survived, so generate() dies with AttributeError. Restored, with its target-namespaced object paths — the thing that stops two targets sharing a source file from claiming the same output and making ninja reject the graph with multiple rules generate ....

This also un-breaks the depfile tests from #48: header edits currently leave stale objects behind and the build silently reports success.

Two incompatible shared-library designs both merged

One PR added a dedicated link_shared ninja rule; another put the platform's shared-object flag into ldflags on the generic link rule. Both landed, with a test each, and the tests contradict — one asserts : link_shared , the other asserts : link plus no -shared anywhere in the file.

The link_shared rule was dead: nothing emitted a build line using it, and it hardcoded -shared, which is wrong on macOS (-dynamiclib) and skipped the -L/-l wiring. Dropped it, and rewrote the test that asserted the dead rule to cover the surviving, platform-correct behaviour.

The Windows matrix has never run — and the backend has never worked there

The test matrix includes windows-2022, where the default shell is PowerShell. The Run test suite step uses backslash line continuations, which PowerShell rejects:

ParserError: Missing expression after unary operator '--'.
   3 |    --cov=ebuild --cov-report=xml --cov-report=term-missing \

So the Windows jobs failed before pytest started, on master and on every branch. Marked shell: bash.

With those jobs actually running, they exposed a real portability defect: NinjaBackend never escaped paths in build statements.

ninja: error: build.ninja:20: expected build command name
build C:\...\main.o: cc main.c
      ^ near here

Ninja splits build statements on unescaped spaces and colons, so a Windows drive-letter colon lands where Ninja expects the separator between outputs and the rule name. Every generated build.ninja was rejected before a command ran; a POSIX path containing a space fails identically. _ninja_path() escapes $, : and , applied to build-statement paths only — variable values (cflags, ldflags) are read to end of line and are deliberately left alone, since escaping them would hand the compiler mangled flags. Four regression tests added, including one asserting each build statement contains exactly one unescaped colon.

Also on Windows: tests/ebuild/test_integration_initramfs_security.py died with WinError 2 before reaching the injection behaviour it exists to check — _create_initramfs() drives find(1) and cpio(1) directly, neither of which exists there. Building a Linux initramfs is not a Windows operation, so those three now skip when the tools are absent, matching how test_ninja_backend.py already skips without a host C compiler.

Finally, mypy . aborted immediately with Duplicate module named "tests" (layers/eosuite/ vendors its own tests/ package). Because the step is continue-on-error, this went unnoticed and the type check had been checking zero files. Excluding layers/ makes it check 81 source files; it stays continue-on-error, so the 11 pre-existing findings are visible without gating the build.

Verification

Check Result
pytest tests/ 206 passed
pytest tests/performance/ 1 passed
ninja escaping the exact CI error reproduces with the unescaped form and is gone with the escaped one

Before this change: SyntaxError at collection, then 11 failures once patched past it, then 4 more on Windows once that leg could run.

Relationship to open PRs

#65 also fixes the dispatch.py else, and #64 also restores _object_path. Neither reconciles the two contradictory dispatch test suites — they pick one exception type, which leaves the other suite failing — and neither addresses the duplicate shared-library design or the Windows breakage. Happy to rebase onto whichever lands first.

🤖 Generated with Claude Code

@codecov-commenter

codecov-commenter commented Aug 28, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

srpatcha
srpatcha previously approved these changes Aug 29, 2026

@srpatcha srpatcha left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified — this is the fix master needs

Reproduced the breakage on origin/master first, so this is not a stale CI badge:

$ ebuild new hi && cd hi && ebuild build
  File ".../ebuild/build/dispatch.py", line 133
    else:
    ^^^^
SyntaxError: invalid syntax

$ pytest
2 errors during collection

With this branch merged onto current master: 206 passed, and ebuild build reaches the compiler instead of raising a traceback.

I had independently written the same repair (#70) before seeing this. Yours came first and is the one I would take; I will close mine. Checking them against each other, this branch covers three of the four defects I found:

Two consecutive else: in configure() fixed
"ninja" listed among backends needing no configure — so configure("ninja") silently succeeded fixed
NinjaBackend._object_path called twice, defined nowhere fixed
ninja invoked as python -m ninja still present

The one left

ebuild/build/dispatch.py:244   [sys.executable, "-m", "ninja", "-C", ...]
ebuild/cli/commands.py:690     [sys.executable, "-m", "ninja", "-f", ...]

commands.py:690 is the build path, so this is the more interesting of the two. It works today only because pyproject.toml declares ninja>=1.11, which puts the PyPI wheel in the environment. But it means a machine with a perfectly good ninja on PATH is ignored, and the moment that wheel is absent — a system-package install, a distro build, a container that pip-installed with --no-deps — the build fails with No module named ninja rather than anything a developer can act on.

A three-line helper covers it:

def ninja_command() -> list:
    exe = shutil.which("ninja")
    return [exe] if exe else [sys.executable, "-m", "ninja"]

Happy for that to be a follow-up rather than growing this PR — the SyntaxError is the urgent half and I do not want to hold it. Say the word and I will open it against your branch or after it lands.

Verification

Merged onto origin/master locally; pytest 206 passed; ebuild new + ebuild build walked by hand from a clean project.

@srpatcha

Copy link
Copy Markdown
Member

Verified merge order for the ebuild backlog

master cannot collect its own test suite (2 errors in 0.25s), so I applied every open PR in sequence onto origin/master locally and ran pytest after each step.

# PR after merging
1 #66 206 passed — from a suite that could not be collected
2 #67 223
3 #68 234
4 #69 234
5 #73 243
6 #76 247
7 #63 247
8 #71 275
9 #72
10 #74

#71, #72 and #74 stack in that order; the last of them brings the suite to 316 when applied on top of the merged base.

#66 goes first, and supersedes my #70

Three PRs fix the same SyntaxError in ebuild/build/dispatch.py#66, #70 and #75. I wrote #70 before seeing #66; #66 came first and is the better change, so I have marked mine superseded rather than competing with it.

What I rebased

#64, #70, #71, #72 and #74 all conflicted in dispatch.py and ninja_backend.py — not because they disagree with #66, but because they were branched on top of my own repair of the same files.

None of #71, #72 or #74 actually touches dispatch.py. Rebased onto plain master, they are independent of #66 and merge clean.

That rebase also fixed something worth flagging. The footprint commit had normalised 1,814 CRLF line endings in ebuild/cli/commands.py, so its diff read as 1,903 changed lines:

raw:          1903  1814  ebuild/cli/commands.py
ignoring CR:    89     0  ebuild/cli/commands.py

The real change is 89 lines. commands.py has mixed CRLF/LF endings, and a whole-file rewrite silently normalises them — worth knowing before anyone else edits that file with a script.

Still to resolve

#65 and #75 conflict with #66 in dispatch.py. Both are alternate fixes to the same defect; I have commented on each with what is uniquely theirs and worth keeping after a rebase.

#64 conflicts more deeply and needs its own pass.

Verification

Every row above is pytest run on the merged tree, in that order, on 29 Aug 2026.

srpatcha added a commit that referenced this pull request Aug 30, 2026
The rebase resolution spliced test() into the middle of clean(), leaving
clean()'s body orphaned under a second `def clean` and dropping the
module-level TestOutcome dataclass with its two parser helpers entirely.
The result imported far enough to collect and then failed:

    ImportError: cannot import name 'TestOutcome' from 'ebuild.build.dispatch'

and before that, six of master's own clean() tests failed with

    NameError: name 'dry_run' is not defined

because the shadowing duplicate did not take the keyword.

This is the same defect this session has been repairing elsewhere — two
versions of one function spliced by a merge that nothing recompiled — and
it is worth naming rather than quietly fixing, because I introduced it by
resolving a conflict with "keep both sides" without rebuilding afterwards.
That is exactly how eos's sync.c ended up referencing fields no version of
the file ever defined at once.

clean() and test() are now separate again and the dataclass is back.

Verified against the branch's base rather than a stale master:

    base (#66)     205 passed, 1 skipped
    this branch    240 passed, 1 skipped
    new failures   none

    ebuild setup / new / configure / build / test / flash / monitor  all present
    ebuild test on a real CMake project  ->  1 passed, 0 failed  exit 0

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@AshrafAhmed9

Copy link
Copy Markdown

@srpatcha suggested I flag this here. I hit the same -shared / -dynamiclib bug independently in #86 and went the other way, correcting the rule's flag instead of removing the rule. Their review makes the case that deleting it is the better shape, which I agree with, so I've closed mine in favour of this.

Nothing needed from you, just noting it for the changelog mention they offered.

Rebuilt on current master. Master has since fixed the dispatch.py SyntaxError
and restored NinjaBackend._object_path, so embeddedos-org#66's versions of those are dropped
in favour of what landed — including its choice of RuntimeError for an unknown
backend, which is now consistent across both dispatch test suites. What follows
is what master still does not have.

**build.ninja is invalid on Windows.** Ninja splits build statements on
unescaped spaces and colons, so a Windows absolute path puts a drive-letter
colon where Ninja expects the separator between outputs and the rule name:

    ninja: error: build.ninja:20: expected build command name
    build C:\...\main.o: cc main.c
          ^ near here

Every generated file was rejected before a command ran; a POSIX path containing
a space fails identically. _ninja_path() escapes `$`, `:` and ` `, applied to
build-statement paths only — variable values (cflags, ldflags) are read to end
of line and are left alone, since escaping them hands the compiler mangled
flags. Four regression tests, one asserting each build statement contains
exactly one unescaped colon.

**test_shared_library_uses_shared_link_rule fails on macOS.** It asserts the
literal "-shared", but _shared_flag() correctly returns "-dynamiclib" on
darwin. The implementation is right and the test was not; made it
platform-aware. This is the one test failing on master today.

**The Windows leg of the test matrix has never run.** `Run test suite` uses
backslash line continuations, which PowerShell rejects:

    ParserError: Missing expression after unary operator '--'.

Marked `shell: bash`, which GitHub provides on Windows runners.

**macos-13 is a retired runner image**, so those jobs are never assigned a
runner and sit queued until they time out. Every other workflow here already
uses macos-latest.

**`mypy .` checks nothing.** It aborts with `Duplicate module named "tests"`
because layers/eosuite/ vendors its own tests/ package. The step is
continue-on-error, so this went unnoticed. Excluding layers/ makes it check 84
files; it stays continue-on-error, so the 12 pre-existing findings are visible
without gating the build.

**ci.yml has no concurrency group**, alone among this repo's workflows, so
pushes pile up queued runs competing for the same scarce runners.

Also gitignored _build/, which the suite leaves in the repo root.

Verified: pytest 292 passed (287 + the fixed shared-flag test + 4 new).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Kartikey1306

Copy link
Copy Markdown
Contributor Author

@srpatcha — thank you for the merge-order analysis, and for marking #70 superseded rather than competing. Since then master moved and this PR went CONFLICTING, so I have rebuilt it on current master and dropped everything master now does better.

Dropped (master's versions are in, and are the right calls):

  • the dispatch.py SyntaxError repair, and my UnknownBackendError(ValueError, RuntimeError). Master resolved it as RuntimeError and updated tests/ebuild/test_dispatch.py to match, so both suites now agree — the dual-inheritance trick I used to satisfy two contradictory suites is no longer needed.
  • NinjaBackend._object_path, restored on master.
  • my removal of the link_shared rule. Master kept the rule and moved the flag behind _shared_flag(), which is the better shape: -dynamiclib on darwin, -shared elsewhere, with the -L/-l wiring intact. I was wrong to delete it.

What is left is what master still does not have — now 4 files instead of the previous sprawl:

  1. build.ninja is invalid on Windows. Ninja splits build statements on unescaped spaces and colons, so a drive-letter colon lands where the outputs/rule separator belongs: ninja: error: expected build command name. _ninja_path() escapes $, : and in build-statement paths only — variable values are read to end of line and must not be escaped. Four regression tests.
  2. test_shared_library_uses_shared_link_rule fails on master today — on macOS. It asserts the literal "-shared" while _shared_flag() correctly returns -dynamiclib. The implementation is right; the test was not.
  3. The Windows leg of the test matrix has never run — backslash continuations under PowerShell. shell: bash.
  4. macos-13 is a retired image, so those jobs queue until they time out.
  5. mypy . checks nothing — it aborts on Duplicate module named "tests" (layers/eosuite vendors its own), hidden by continue-on-error. Excluding layers/ makes it check 84 files.
  6. No concurrency group, alone among this repo's workflows.

pytest: 292 passed (287 on master + the fixed shared-flag test + 4 new).

Re: the CRLF finding in commands.py — that matches something I hit in eBoot, where a scripted whole-file rewrite turned a 40-line change into 312. Worth a .gitattributes entry rather than relying on everyone remembering.

With `shell: bash` the Windows leg of the matrix runs pytest for the first
time, and it fails four tests. Neither cause is new; both were simply never
executed.

`test_test_target_links_like_an_executable` picks the link edge out of
build.ninja with `l.split(":")[0]`, meaning "the text before the rule
separator". On Windows the first colon is the drive letter, so that expression
returns "build C" for every line, the `.o` filter never matches, and the test
selects the compile edge and asserts `": link "` against it.

Splitting on the first *unescaped* colon is what was meant, and is now
unambiguous: `_ninja_path()` writes the drive colon as `$:` and leaves exactly
one bare colon per statement, the separator.

    build C$:\...\obj\t_smoke\t.c.o: cc t.c      -> outputs end in .o   (compile)
    build C$:\...\t_smoke.exe: link ...          -> outputs do not      (link)

The three `test_integration_initramfs_security.py` cases fail with WinError 2:
`_create_initramfs()` drives find(1) and cpio(1) directly and neither exists on
a stock Windows runner, so they die before reaching the command-injection
behaviour they exist to check. Building a Linux initramfs is not a Windows
operation, so they skip when the tools are absent — the same shape as the
existing skip in test_ninja_backend.py when no host C compiler is present.

Verified: pytest 292 passed locally; the Windows selection logic checked
against an escaped drive-letter path directly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants