feat(cli): add ebuild test and ebuild monitor — the last two golden-path steps by srpatcha · Pull Request #84 · embeddedos-org/ebuild

srpatcha · 2026-08-30T14:08:04Z

Closes #83.

§7 names eBuild's MVP responsibility as "Setup, configure, build, test,
flash, monitor", and §7.1 gives the sequence verbatim. Two of the eight
commands did not exist:

ebuild setup      EXISTS        ebuild build      EXISTS
ebuild new        EXISTS        ebuild test       MISSING
ebuild configure  EXISTS        ebuild flash      EXISTS
                                ebuild monitor    MISSING

Twenty-three commands were present, and neither of the two the MVP definition
names. The golden path stopped at step 5 of 8.

test is the step that makes a first firmware run verified rather than merely
completed — the stated success criterion. monitor is how a developer sees it
run at all. ebuild qemu is adjacent but different: it starts an emulator and
cannot observe the board step 7 just flashed.

`ebuild test`

Dispatches through the existing backend detection and reports what the runner
printed. Counts are parsed from the runner's own summary for ctest, meson and
cargo, and withheld when nothing recognisable was printed — a number derived
from a zero exit status is a guess presented as a measurement. make has no
standard summary, so its counts stay unknown and the exit status carries the
verdict.

The case that drove the design: ctest exits 0 when it finds nothing to run.
Trusting the exit status reports a green suite for a project containing no tests
at all — worse than having no test command, because it actively reassures. An
empty run is therefore treated as a failure.

Verified against all three outcomes on a real CMake project:

zero tests      [error] The runner completed without executing a
                single test.                                   exit=1

1 pass 1 fail   [error] Tests failed (1 passed, 1 failed).      exit=1

1 pass 0 fail   [ok] Tests passed (1 passed, 0 failed).         exit=0

`ebuild monitor`

Opens a serial console, reading the baud rate from board.console.baud or
monitor.baud in build.yaml, falling back to 115200.

It auto-selects a port only when exactly one is present; with several it lists
them and asks, because picking one at random is how a session ends up silently
attached to the wrong board. An explicit --port is always honoured, since
enumeration misses some adapters. --list shows what the host can see. Ctrl-C
ends the session and closes the port — leaving it held makes the next run fail
with a busy device that looks like a hardware fault.

A .gitignore trap, fixed in passing

Hit while adding these. The rule build/ matched ebuild/build/ — the backend
package, not build output:

$ touch ebuild/build/probe.py
$ git status --porcelain ebuild/build/probe.py
(nothing)

A new module added there is silently untracked: it works locally and is simply
absent from the repo. All four current files predate the rule so nothing has
been lost yet, but it would have caught this PR had monitor.py gone under
ebuild/build/ rather than ebuild/firmware/.

Fixed with a negation rather than re-anchoring to /build/, so
examples/*/build/ stays ignored. Both directions verified:

ebuild/build/probe.py       ??  (now visible)
examples/*/build/out.o      still ignored

Verified

tests    99 -> 121 passed, 0 failed

The 22 new tests cover the count parsers for all four backends, the empty-run
detection by both routes (marker phrase and zero totals), and every branch of
port resolution and baud resolution — including that a malformed build.yaml
falls back rather than raising.

README updated with the full golden path and the two commands' behaviour.

…viour `import ebuild.build.dispatch` raises SyntaxError on master, so the CLI and five test modules fail outright. As with the other breakage below, the cause is overlapping PRs squash-merged on stale bases with no re-verification. - dispatch.py: configure() ended with two consecutive `else:` blocks from an unresolved conflict. The two blocks also disagreed about the exception -- tests/ebuild expects ValueError("Unknown build backend '<name>'") while tests/unit expects RuntimeError matching "ninja". Both are legitimate readings: an unrecognized name is a bad argument, and a "ninja" that reaches the dispatcher is a CLI routing failure. Introduced UnknownBackendError(ValueError, RuntimeError) with a message covering both, raised from configure(), build(), and clean(). "ninja" is no longer a silent no-op in configure(), which is what let `ebuild build` report "Build completed successfully" without running a compiler. - ninja_backend.py: NinjaBackend._object_path() was deleted by a cflags refactor while both of its callers survived, so generate() died with AttributeError -- the default build backend did not work at all. Restored with its target-namespaced object paths, which is what stops two targets sharing a source from claiming the same output. - ninja_backend.py: the emitted `link_shared` rule was dead -- nothing used it -- and it hardcoded `-shared`, which is wrong on macOS. Shared libraries already link through the generic `link` rule with the platform's flag (-dynamiclib/-shared) and the -L/-l wiring in ldflags. Dropped the rule and updated tests/ebuild/test_ninja_backend.py, which asserted the dead rule, to cover the surviving behaviour instead. Its removal also unblocks test_static_library_unaffected, which asserts no "-shared" anywhere in the generated file. Verified: pytest tests/ 202 passed (201 passed + 1 skipped without the ninja package installed); tests/performance 1 passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The matrix includes windows-2022, where the default shell is PowerShell. The "Run test suite" step uses backslash line continuations, which PowerShell does not accept: ParserError: Missing expression after unary operator '--'. 3 | --cov=ebuild --cov-report=xml --cov-report=term-missing \ So the Windows jobs failed before pytest started — on master and on every branch. Marked `shell: bash`, which GitHub provides on Windows runners via Git Bash, keeping the command readable and identical across the three platforms. Also: `mypy .` aborted immediately with tests\__init__.py: error: Duplicate module named "tests" (also at ".\layers\eosuite\tests\__init__.py") Found 1 error in 1 file (errors prevented further checking) layers/eosuite/ vendors its own tests/ package. Because the step is continue-on-error, this went unnoticed and the type check has been checking zero files. Excluding layers/ makes it check 81 source files; it stays continue-on-error, so the 11 pre-existing findings it now reports are visible without gating the build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

With the Windows jobs actually running pytest, they surface a backend that has never worked on Windows: ninja: error: build.ninja:20: expected build command name build C:\...\main.o: cc main.c ^ near here Ninja splits build statements on unescaped spaces and colons, so a Windows absolute path puts a drive-letter colon where Ninja expects the separator between outputs and the rule name. Every generated build.ninja was rejected before a single command ran. A POSIX path containing a space fails the same way. _ninja_path() escapes `$`, `:` and ` ` and is applied to the paths in build statements only. Variable values (cflags, ldflags) are read to end of line and are deliberately left alone -- escaping them would hand the compiler mangled flags. Also skipped tests/ebuild/test_integration_initramfs_security.py on hosts without find(1)/cpio(1). _create_initramfs() drives both directly, so on a Windows runner all three died with WinError 2 before reaching the injection behaviour they exist to check. Building a Linux initramfs is not a Windows operation; the skip matches how test_ninja_backend.py already skips when no host C compiler is present. Adds four regression tests for the escaping, including one asserting that each build statement contains exactly one unescaped colon. Verified: pytest tests/ 206 passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ci.yml is the only workflow in this repo without a concurrency group, and it is the heaviest one -- a matrix spanning ubuntu, macos and windows. Every push to a pull request therefore left the previous run queued, and all of them competed for the same scarce windows/macos runners. On this branch three superseded runs sat ahead of the current one for over an hour, testing commits that were no longer HEAD. Uses the same group expression the sibling workflows already use, with cancel-in-progress: true, because a superseded commit's result is not wanted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The macos-13 image is retired, so jobs requesting it are never assigned a runner -- they sit queued until they time out. Across four runs on this branch every ubuntu-22.04 and windows-2022 job started and finished within minutes while all three macos-13 jobs stayed queued for more than two hours, leaving the workflow permanently incomplete. Every other workflow in this repo already targets macos-latest; ci.yml was the one place still pinning macos-13. release.yml also lists macos-13, alongside macos-14, in its wheel-build matrix. Left alone: building wheels against an older macOS deployment target may be deliberate there, and it does not gate pull requests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

+        summary can be parsed the counts stay None and only ok/ran are
+        meaningful — reported honestly as unknown rather than filled in.
+        """
+        config = config or {}


+    finally:
+        try:
+            conn.close()
+        except Exception:


…ath steps §7 names eBuild's MVP responsibility as "Setup, configure, build, test, flash, monitor", and §7.1 gives the sequence verbatim. Two of the eight commands did not exist: ebuild setup EXISTS ebuild build EXISTS ebuild new EXISTS ebuild test MISSING ebuild configure EXISTS ebuild flash EXISTS ebuild monitor MISSING Twenty-three commands were present, and neither of the two the MVP definition names. The golden path stopped at step 5 of 8. test is the step that makes a first firmware run verified rather than merely completed — the stated success criterion — and monitor is how a developer sees it run at all. ebuild qemu is adjacent but different: it starts an emulator and cannot observe the board step 7 just flashed. ebuild test dispatches through the existing backend detection and reports what the runner printed. Counts are parsed from the runner's own summary for ctest, meson and cargo, and withheld when nothing recognisable was printed — a number derived from a zero exit status is a guess presented as a measurement. make has no standard summary, so its counts stay unknown and the exit status carries the verdict. The case that drove the design: ctest exits 0 when it finds nothing to run. Trusting the exit status reports a green suite for a project with no tests at all, which is worse than having no test command, because it actively reassures. An empty run is therefore a failure: zero tests [error] The runner completed without executing a single test. exit=1 1 pass 1 fail [error] Tests failed (1 passed, 1 failed). exit=1 1 pass 0 fail [ok] Tests passed (1 passed, 0 failed). exit=0 ebuild monitor opens a serial console, reading the baud rate from board.console.baud or monitor.baud in build.yaml and falling back to 115200. It auto-selects a port only when exactly one is present; with several it lists them and asks, because picking one at random is how a session ends up silently attached to the wrong board. An explicit --port is always honoured, since enumeration misses some adapters. Ctrl-C ends the session and closes the port — leaving it held makes the next run fail with a busy device that looks like a hardware fault. Also fixes a .gitignore trap hit while adding these. The rule "build/" matched ebuild/build/, the backend package, not just build output: $ touch ebuild/build/probe.py $ git status --porcelain ebuild/build/probe.py (nothing) A new module added there is silently untracked — it works locally and is simply absent from the repo. Fixed with a negation rather than anchoring to /build/, so examples/*/build/ stays ignored. Both verified. ebuild/build/probe.py ?? (now visible) examples/*/build/out.o still ignored Tests: 99 -> 121 passed, 0 failed. Closes #83 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

§8 lists "consistent documentation generated from tested examples" as an MLP requirement. Generating prose from examples is a large change; verifying that the examples already published actually run is the part that stops the docs lying, and it costs almost nothing. The failure this prevents has already happened in this organisation. EoStudio carried this, for the tool it is a front end for: f"ebuild --platform {platform} --config board.yaml {source_dir}" ebuild has no top-level --platform or --config. The string sat in a dict and was never executed, so nothing found out it could not work. A command published in a README decays the same way, except the person who discovers it is a developer following the quick start. Reads only fenced code blocks. Prose says things like "ebuild is a unified build system", and treating "is" as a subcommand would fail for no reason — a plain grep over the README yields "ebuild and", "ebuild can", "ebuild is" alongside the real ones. Three checks. Each documented invocation names a command the CLI registers; the eight §7.1 golden-path steps are asserted directly, so the path cannot be quietly broken by editing the README instead of the CLI; and a guard that the extraction found something at all, since a silently broken matcher would make every other assertion pass vacuously. Verified the check fails rather than merely existing: README: ebuild verify -> assert 'verify' in {'add', 'analyze', 'build', 'clean', ...} FAILED test_a_documented_command_exists[verify-README.md] restored -> 11 passed Tests: 121 -> 132 passed, 0 failed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The rebase resolution spliced test() into the middle of clean(), leaving clean()'s body orphaned under a second `def clean` and dropping the module-level TestOutcome dataclass with its two parser helpers entirely. The result imported far enough to collect and then failed: ImportError: cannot import name 'TestOutcome' from 'ebuild.build.dispatch' and before that, six of master's own clean() tests failed with NameError: name 'dry_run' is not defined because the shadowing duplicate did not take the keyword. This is the same defect this session has been repairing elsewhere — two versions of one function spliced by a merge that nothing recompiled — and it is worth naming rather than quietly fixing, because I introduced it by resolving a conflict with "keep both sides" without rebuilding afterwards. That is exactly how eos's sync.c ended up referencing fields no version of the file ever defined at once. clean() and test() are now separate again and the dataclass is back. Verified against the branch's base rather than a stale master: base (#66) 205 passed, 1 skipped this branch 240 passed, 1 skipped new failures none ebuild setup / new / configure / build / test / flash / monitor all present ebuild test on a real CMake project -> 1 passed, 0 failed exit 0 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

srpatcha · 2026-08-30T21:09:37Z

Correction to this PR, and it is one worth stating plainly: the numbers I
originally posted were measured against the wrong base.

This branch was cut from a stale local master and was 75 commits behind
origin/master. The "121 → 132 passed" figures described a tree that is not what
this would merge into. That is the same "green branch, base has moved, nothing
re-verified the result" failure I have been filing against other PRs here, and I
did it myself.

Rebased and re-measured

Rebased onto #66 rather than origin/master, because origin/master's
ebuild/build/dispatch.py does not parse:

$ python3 -c "import ast; ast.parse(open('dispatch.py').read())"
  line 133
        else:
        ^^^^
SyntaxError: invalid syntax

An orphan else: after a raise block — another merge splice. #66 repairs it
and is already approved, so this now stacks on it. Merge #66 first.

Corrected figures, measured against this branch's actual base:

base (#66)      205 passed, 1 skipped
this branch     240 passed, 1 skipped
new failures    none

The golden path re-verified after the rebase:

ebuild setup / new / configure / build / test / flash / monitor    all present
ebuild test on a real CMake project   ->  1 passed, 0 failed, exit 0

And a defect I introduced during the rebase, since it makes the point

Resolving the dispatch.py conflict by keeping both sides spliced test() into
the middle of clean(), orphaned clean()'s body under a second def clean,
and dropped the module-level TestOutcome dataclass altogether:

ImportError: cannot import name 'TestOutcome' from 'ebuild.build.dispatch'
NameError: name 'dry_run' is not defined        (6 of master's own clean() tests)

Caught by running the suite and diffing the failure list against the base, not by
reading the diff. Fixed in the follow-up commit.

It is the identical shape to eos's sync.c, where #55 and #67 implemented
priority inheritance with incompatible designs and the squash spliced half of
each together. A conflict resolution is a code change and needs the same
verification as one.

srpatcha · 2026-08-31T01:20:08Z

Superseded by #89.

ebuild test and ebuild monitor have both landed on master from another PR,
so the main body of this work is redundant — verified all eight golden-path steps
now resolve on a clean clone:

ebuild setup / new / configure / build / test / flash / monitor    all EXISTS

Two things here are not on master, and #89 carries them:

The empty-run guard. Master's ebuild test reports [ok] All tests passed.
and exits 0 for a project with zero tests — ctest exits 0 on "No tests were
found!!!", and a CMakeLists.txt with enable_testing() and no add_test()
still produces a CTestTestfile.cmake, so the runner is found and the wrapper
trusts the status. Confirmed against current master.

The docs check — that every ebuild command shown in the docs exists, code
blocks only, verified to fail when a fake command is added.

Closing rather than rebasing: the overlap is large enough that a narrow PR
against master reviews better than a conflict resolution against a branch that
has been overtaken.

Kartikey1306 and others added 5 commits August 28, 2026 19:18

srpatcha requested a review from hshanmug12 as a code owner August 30, 2026 14:08

github-code-quality Bot found potential problems Aug 30, 2026

View reviewed changes

Comment thread ebuild/build/dispatch.py

summary can be parsed the counts stay None and only ok/ran are

meaningful — reported honestly as unknown rather than filled in.

"""

config = config or {}

Comment thread ebuild/firmware/monitor.py

finally:

try:

conn.close()

except Exception:

srpatcha and others added 3 commits August 30, 2026 14:07

srpatcha force-pushed the feat/golden-path-test-and-monitor branch from a70b2d1 to 3d3374f Compare August 30, 2026 21:09

srpatcha mentioned this pull request Aug 31, 2026

fix(cli): a test run that executed nothing is not a pass #89

Open

srpatcha closed this Aug 31, 2026

Navigation Menu

Uh oh!

feat(cli): add ebuild test and ebuild monitor — the last two golden-path steps - #84

feat(cli): add ebuild test and ebuild monitor — the last two golden-path steps#84
srpatcha wants to merge 8 commits into
masterfrom
feat/golden-path-test-and-monitor

srpatcha commented Aug 30, 2026

Uh oh!

srpatcha commented Aug 30, 2026

Uh oh!

srpatcha commented Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

srpatcha commented Aug 30, 2026

ebuild test

ebuild monitor

A .gitignore trap, fixed in passing

Verified

Uh oh!

srpatcha commented Aug 30, 2026

Rebased and re-measured

And a defect I introduced during the rebase, since it makes the point

Uh oh!

srpatcha commented Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

`ebuild test`

`ebuild monitor`