Rework external_deps for Saigo and CMake-based NaCl rebuild

[illwieckz]

This introduce a big change: the NaCl loader and bootstrap executables are now suffixed with their architecture, like the IRT: nacl_loader-amd64, nacl_helper_bootstrap-amd4.

To avoid name collision, the armhf loader wrapper for running armhf on arm64 is now named nacl_helper_bootstrap-ldarmhf, it chainloads what is now nacl_helper_bootstrap-armhf.

This change genericize what already existed for loading the amd64 loader on windows-amd64 when running the i686 engine, except now that's all the loaders that have a suffix, not just the non-native one.

This makes things very easy to manage, for example the external_deps script just loops over all the different nexe architecture an engine architecture can support and builds the nacl_loader for it. This already works for the windows-i686 deps build: it builds both the i686 and amd64 loaders using MinGW.

This same mechanism would be usable in the future to build for the linux-arm64 deps both the armhf loader and the amd64 loader (purposed to be running on box64 when armhf support isn't there in hardware).

With that loader suffix thing I deeply rewrote the NaCl startup code in VirtualMachine.cpp, which is then already shaped for the ability to chose between armhf or amd64 on box64 at run time. The integration of box64 (CMake code, etc.) isn't there, but the VirtualMachine.cpp code is ready. That rework is deep and deduplicate and refactor intensively the binary name construction and the command construction.

I started adding the macos-arm64-default deps configuration. For now not everything works, but it already successfully builds the amd64 loader.

The IRT was previously fetched as part of the naclsdk, which is not removed yet, so the IRT is still shipped the old way at this point.

For Saigo it uses the recent release I packaged:

• https://github.com/DaemonEngine/saigo-release-scripts/releases/tag/v21.0-20260625

For the NaCl loader it uses my illwieckz/staging branch that contains all my unmerged code:

• https://github.com/DaemonEngine/native_client/tree/illwieckz/staging
• Merging my work native_client#19

[illwieckz]

The CI fails because I haven't uploaded preview external_deps archives yet (it's too early).

[illwieckz]

Building the NaCl loader with MinGW requires JWasm:

• https://github.com/JWasm/JWasm

I wonder if we can extend the script to build specific host tools.

I actually suspect that some macOS tools are meant to be built for the host as well (nasm, pkg-config…), and that we were lucky to be able to reuse the ones built for the target because we weren't cross-compiling…

[illwieckz]

I'm very proud of this:

build_naclruntime() {
	local nacl_arch_list=()

	case "${PLATFORM}" in
	windows-amd64-*)
		nacl_arch_list+=('amd64')
		;;
	windows-i686-*)
		nacl_arch_list+=('i686')
		nacl_arch_list+=('amd64')
		;;
	linux-amd64-*)
		nacl_arch_list+=('amd64')
		;;
	linux-i686-*)
		nacl_arch_list+=('i686')
		;;
	linux-arm64-*)
		nacl_arch_list+=('armhf')
		;;
	linux-armhf-*)
		nacl_arch_list+=('armhf')
		;;
	macos-amd64-*)
		nacl_arch_list+=('amd64')
		;;
	macos-arm64-*)
		nacl_arch_list+=('amd64')
		;;
	*)
		log ERROR 'Unsupported platform for naclruntime'
		;;
	esac

	local dir_name="DaemonEngine-native_client-${NACLRUNTIME_REVISION:0:7}"
	local archive_name="native_client-${NACLRUNTIME_REVISION}.zip"

	download_extract naclruntime "${archive_name}" \
		"{$NACLRUNTIME_BASEURL}/${NACLRUNTIME_REVISION}"

	"${download_only}" && return

	cd "${dir_name}"

	for nacl_arch in "${nacl_arch_list[@]}"
	do
		(
			setup_platform "${PLATFORM_SYSTEM}-${nacl_arch}-${PLATFORM_COMPILER}"
			cmake_build
		)

		case "${PLATFORM}" in
		linux-*)
			mv "${PREFIX}/bin/nacl_helper_bootstrap" "${PREFIX}/nacl_helper_bootstrap-${nacl_arch}"
			;;
		esac

		mv "${PREFIX}/bin/sel_ldr${EXE_EXT}" "${PREFIX}/nacl_loader-${nacl_arch}${EXE_EXT}"
	done

[illwieckz]

I actually suspect that some macOS tools are meant to be built for the host as well (nasm, pkg-config…), and that we were lucky to be able to reuse the ones built for the target because we weren't cross-compiling…

Yes, pkg-config is a host tool required to build opusfile on macOS. There is already some specific code (but specific to pkg-config) to build it for the host, but we still install it in the packaged prefix.

So we better refactor this and reuse the mechanism for JWasm.

[illwieckz]

Hmm, it looks like the CI is not firing if the latest commit in the branch doesn't touch code… That's unexpected, we only want to skip CI when no commit in the PR touch code…

[illwieckz]

Hmm, it looks like the CI is not firing if the latest commit in the branch doesn't touch code… That's unexpected, we only want to skip CI when no commit in the PR touch code…

That was a known problem with AppVeyor if I'm right. It looks like Azure is just not spawning at all.

[illwieckz]

That was due to a mistake. A mistake in CI file isn't reported as a CI error…

[illwieckz]

The macOS arm64 CI is working! 🎉️

[illwieckz]

This now knows how to bundle and to run the bundled box64.

See:

• Box64 support on Linux Arm64 #1989

For now there is a bug though, the amd64 loader spawns properly on box64 but later the nexe catches a weird bug, after having started its execution successfully.

But all the plumbing is there.