Skip to content

fix(integration): build initramfs without host cpio - #90

Merged
srpatcha merged 1 commit into
embeddedos-org:masterfrom
BhavarSingh:fix/portable-initramfs-newc
Aug 31, 2026
Merged

fix(integration): build initramfs without host cpio#90
srpatcha merged 1 commit into
embeddedos-org:masterfrom
BhavarSingh:fix/portable-initramfs-newc

Conversation

@BhavarSingh

Copy link
Copy Markdown
Contributor

Summary

Integration builds currently create initramfs.cpio.gz by launching host
find, cpio, and gzip processes. That toolchain is absent on Windows and
makes a core build artifact depend on Unix host utilities.

This change replaces the subprocess pipeline with a standard-library newc
serializer, so integration builds create the initramfs on every supported host.

Type of Change

  • fix — Bug fix
  • test — Add or fix tests

Changes

  • Serialize gzip-compressed newc records directly, including required
    alignment and the TRAILER!!! marker.
  • Preserve regular files, directories, symbolic links, metadata, and hard-link
    identity without duplicating hard-linked payloads.
  • Infer executable bits for shebang scripts and ELF binaries on Windows, where
    host mode bits do not represent Linux executability.
  • Run initramfs creation on Windows and remove the cpio requirement from the
    QEMU error message.
  • Replace system-cpio test assertions with an internal format parser and add
    coverage for literal metacharacter paths, mode bits, symlinks, and hard links.

The record layout and hard-link behavior follow the Linux kernel's
documented initramfs buffer format.

Testing

  • Focused unit tests: 5 passed, 1 skipped
  • New tests added for new functionality
  • Independent bsdtar extraction validated both hard-link names, shared
    inode identity, and payload content
  • flake8 --select=F passes for both changed Python files

Full Python suite: 288 passed, 2 skipped, 1 failed. The single failure is an
existing Windows-only Ninja path-parsing assertion in
test_test_target_links_like_an_executable; it is unrelated to initramfs code
and is recorded as T-002 in TASKS.md.

Pre-Submission Checklist

  • All existing tests pass (one unrelated Windows failure described above)
  • New tests added for new functionality
  • Documentation updated if API changed (not applicable)
  • Commit messages follow <type>(<scope>): <description> convention
  • Branch is based on latest master

Related Issues

None.

Screenshots / Logs

Not applicable.

Additional Notes

  • A QEMU boot was not run on this Windows host.
  • Device nodes and other special files remain unsupported and produce an
    explicit error; the generated project rootfs currently contains regular
    files, directories, and symlinks.
  • The symlink test is skipped on Windows because symlink creation privileges
    are not reliably available; it runs on POSIX CI hosts.

@srpatcha
srpatcha merged commit d3958f7 into embeddedos-org:master Aug 31, 2026

@srpatcha srpatcha left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. I validated the archive output independently rather than trusting the
unit tests, because a hand-written newc writer is exactly the kind of thing
that passes its own tests and produces something a real tool will not read.

Independent validation

Built an initramfs from a rootfs with nested directories, a shell script, an
ELF-magic binary and a text payload, then extracted it with GNU cpio:

$ gzip -dc initramfs.cpio.gz | cpio -idmv
.
./bin
./bin/prog
./etc
./etc/motd
./init
2 blocks

$ cat etc/motd
hello payload

-rwxrwxr-x  init
-rwxrwxr-x  bin/prog

Every path present, payload byte-identical, execute bits preserved on both the
script and the binary. The four-byte alignment and header fields are right —
cpio is strict about both and would have complained.

Confirmed the host dependency is genuinely gone: _create_initramfs() contains
no subprocess or shutil.which call, only the docstring describing the
find | cpio | gzip pipeline it replaced.

full suite: 290 passed, 1 skipped

On the Windows mode handling

def _newc_mode(path: Path, metadata) -> int:
    """Windows reports regular files as non-executable even when they are
    shell scripts or cross-compiled ELF binaries..."""

Right call, and right to be conservative about it. Sniffing content to restore
execute bits only on Windows, leaving POSIX modes untouched, means the common
case carries no heuristic at all. An initramfs whose /init is not executable
is a kernel panic at boot with a famously unhelpful message, so getting this
wrong is expensive and getting it approximately right on the platform that
cannot express the information is the best available answer.

On the evidence row

QEMU boot was not run on Windows.
1 unrelated failure in the pre-existing Windows Ninja path assertion,
recorded as T-002.

Stating what was not verified, and separating a pre-existing failure from
anything this PR caused, is more useful than an unqualified pass. It let me spend
verification effort on the archive format instead of re-running your suite.

For what it is worth: I could not run the QEMU boot either — QEMU is not
installed in this environment, which is also why nothing in this organisation has
yet been observed to boot. Extraction by an independent implementation is the
strongest check available short of that, and it passes.

One thought, not blocking

_newc_padding() and the record writer are general enough to be worth testing
against a fixture archive produced by real cpio, byte for byte, if this format
grows more cases (device nodes, long paths over 256 bytes). The current tests
cover behaviour; a golden-bytes comparison would pin the wire format the way
eBoot#66 pins the .efw header.

srpatcha pushed a commit that referenced this pull request Sep 1, 2026
`import ebuild.cli.commands` raises NameError on master, so the `ebuild`
command does not start and pytest cannot collect nine test modules --
the suite reports "9 errors during collection" and runs zero tests.

Four PRs landed their tests while their implementation was discarded
during conflict resolution, and the merges that did it were not gated by
a run: f5209ac, current master, has no CI runs at all.

  #64 (6a22e36)  tests/unit/test_golden_path_commands.py landed;
                 _workspace_repo_paths, _cached_repo_libraries,
                 _record_board_selection, _EOS_PROJECT_CONFIG and the
                 `configure --board` option did not.
  #89 (32348f0)  commands.py gained five `re.` uses, no `import re`.
  #90 (d3958f7)  the initramfs test kept its assertions but lost its
                 `_newc_members` helper and its `os` / `stat` imports.
  #92            registry.version_key was renamed version_sort_key
                 without updating its caller in the tests.

Restoring them, taken verbatim from the branches that were merged:

* commands.py: `import re`; `_workspace_repo_paths`,
  `_cached_repo_libraries`, `_record_board_selection`,
  `_EOS_PROJECT_CONFIG`; the `--board` option on `configure` and the
  call that persists it; and the `{**_workspace_repo_paths(), **...}`
  merge at both `_install_packages` call sites.

  Without that last hunk a scaffolded project that `use`s eos or eboot
  gets no include or link paths from ~/.ebuild/repos, which is the
  "fatal error: eos/hal.h: No such file or directory" that #64 fixed.
  Without the option, the documented golden path -- `configure --board`
  then a bare `build` -- fails with "no such option: --board".

* the initramfs test: `os` / `stat` imports and the `_newc_members`
  newc parser the assertions call.

* test_package_registry.py: import the name the module now exports.

Collection goes from 9 errors / 0 tests to 446 passing. The 12 that
still fail are pre-existing and out of scope here: `doctor` (#72) and
`package` (#77) are two more features whose implementations the same
merges dropped and which should be restored by their authors, and the
rest are separate bugs -- see the PR description.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants