After fumbling a lot, I finally managed to get archarm aarch64 running on a CM4. It seems like the default u-boot config that comes with the image doesn't work, and you have to replace it. The official instructions at the archarm website don't work out of the box, so I followed the modified instructions from
https://blog.huenet.net/how-to-install-64-bit-arch-on-a-raspberry-pi-4/.
Important to note, you cannot change the device in
fstab
as suggested by the official instruction, otherwise it gets stuck.