This could be solved https://sd-card-images.johang.se/ style: single kernel+rootfs OS image, with the small board-specific bootloader + dtb, which you concatenate to the OS image.

This is what NixOS does as well, but it still requires the kernel to effectively have support for your specific SoC.

On ARM ACPI/UEFI, ACPI abstracts many parts away reducing the need for this.