Endianness fixes - #102
Conversation
Use explicit 16-bit types and byte-wise little-endian reconstruction for DXT1 color endpoints and DXT5 packed words. Also make the DXT5 selector accessor const-correct. This clarifies that these values are reconstructed from the serialized byte representation rather than relying on host endianness.
Avoid treating color_quad_u8 as a native-endian uint32 when building DXT1 unique-color hashes and selectors. Construct the RGBA value explicitly in byte order instead. Also handle the DXT1 alpha threshold using the source pixel's alpha component directly.
The GCC configuration unconditionally defined CRNLIB_LITTLE_ENDIAN_CPU, which caused big-endian targets such as s390x to be treated as little-endian. Use the compiler-provided byte-order macros to set the appropriate endianness flag instead. This fixes serialization of little-endian formats such as DDS on big-endian platforms.
- Add little-, big-, and native-endian KTX output modes. - Serialize KTX headers and size fields using the selected byte order. - Propagate KTX endianness through compression parameters and file output. - Expose -ktxBigEndian and -ktxNativeEndian in crunch. - Preserve the selected KTX byte order when copying textures.
Read the KTX endianness marker as raw bytes before interpreting the remaining header fields, then configure the serializer to match the file's byte order. Use endian-aware deserialization for key-value sizes and mip image sizes instead of manually swapping them.
Fix CRN DXT1 decompression on big-endian systems.
Fix CRN DXT5 decompression on big-endian systems.
Fix CRN DXN decompression on big-endian systems.
Avoid relying on host byte order when packing ETC1 blocks. The ETC1 bitstream is defined in big-endian byte order, but the packer previously used the host-endian m_uint64 and m_u32 union members directly. Pack the RGB endpoint components explicitly and write the resulting 64-bit ETC1 block using the big-endian helper.
The code was assigning color_quad_u8::m_u32 directly to an endpoint value. m_u32 depends on the host byte order, causing different CRN output on big-endian targets. Use get_rgba_u32() to explicitly define the RGBA byte layout. This is one of several endian-related conversions in the CRN path.
The color endpoint optimization path was relying on m_u32 assignments when unpacking packed endpoints into color_quad_u8 values. This made the conversion depend on host byte order and produced different CRN output on big-endian architectures. Extract endpoint components explicitly to ensure identical output across little- and big-endian systems.
Use color_quad::set_rgba_u32() instead of writing the m_u32 union member directly when unpacking ETC color endpoints. The m_u32 representation depends on host byte order, while color components have a fixed RGBA ordering. Explicit component assignment keeps endpoint reconstruction portable across little and big-endian architectures.
Avoid writing DDS metadata using the host byte order. This produced invalid DDS files on big-endian systems, with swapped signatures, FOURCC values, and header fields. Serialize the DDS header fields explicitly as little endian so the generated files are identical on little and big endian systems.
Avoid writing DDS metadata using the host byte order. This produced invalid DDS files on big-endian systems, with swapped signatures, FOURCC values, and header fields. Serialize the DDS header fields explicitly as little endian so the generated files are identical on little and big endian systems.
5b9e044 to
5536ad1
Compare
79c458a to
fd2cd89
Compare
|
This has been a wild ride! Of course I faced all the things I expected, like colors bytes not being properly ordered (CRN, explaining why the colors were messy), headers not being properly written (DDS) or not properly read (CRN, again). But the most messy thing looked to have been on purpose… It looks like KTX supports two format variants, big endian and little endian. It's probably a remnant from the golden age of PowerPC game consoles, where saving a few instructions and a few cycles not swapping bytes at load time could make sense. But the way Crunch implemented KTX was beyond what I imagined. Not only it generated big endian KTX images on big endian platforms and little endian KTX images on little endian platforms, but it could not load (and generate) images from another endianness. So basically if you made a project integrating Crunch as a KTX producer in your game production pipeline and integrated Crunch in your engine as a KTX reader, your game would only been able to load images on the same platform those images were created. The absolute contrary of cross-platform gaming. All of this is fixed. Crunch now produces LE KTX by default, and one can produce BE KTX on demand, and Crunch can now read any KTX whatever the endianness of the KTX file and the endianness of the host machine. |
|
This identified variants we need to add to the CI tests:
|
|
It also would be useful to add at least one big endian architecture to the CI matrix. The identity of said architecture isn't important as long as it makes sure it tests BE binaries loading and producing LE files properly. On my end for testing this I ended up with an s390x chroot just because of the assumption the usermode qemu support for it would probably be among the best ones, even if that's very likely the platform I expect the least to run Crunch lol. The emulation was performant enough and that's what mattered to me. |
d7b3523 to
ef10501
Compare
5536ad1 to
517a67d
Compare
91651bf to
8ef17d7
Compare
f81efaa to
05402fe
Compare
517a67d to
26eadaf
Compare
05402fe to
b6dc0a4
Compare
26eadaf to
fa752b4
Compare
b6dc0a4 to
30871e6
Compare
- Big Engian Debian platforms: * ppc64 * s390x - Modern Debian platforms: * ppc64el * loong64 * riscv64 - Modern Windows platforms (using MSYS2 on Windows): * arm64 Also upgradee the Linux distributions running MinGW, Wine and QEMU.
a08acf2 to
a1da3b5
Compare
a1da3b5 to
c8c3fb6
Compare
|
Big Endian ppc64 and s390x are now running the 814 tests properly. Including KTX conversions from little to big endian and back, including layouts like DXN. |
1 participant
Endianness fixes, makes it work on Big-Endian. This way Debian can blindly enable the build of the crunch package on all their systems without any specialization, even if when we don't support those architectures ourselves.
And the code is more robust.
I tested with a Debian Trixie s390x chroot.
All the tests from
masterand #98 are passing:That includes all the tests of
crunchand the examples.