I have two CM4s (4GB RAM/16GB EMMC) which I've installed in my TPi2 and can successfully flash them (into EMMC). I wanted to boot the CM4s from some more robust and faster media. I tried... 1) Using the NVME slots on the base of the TPi2., but found that this doesn't work because the CM4's single PCI lane isn't routed to those NVME slots. 2) I got a couple of Mini PCIE to SATA adapters and connected SATA SSDs. I was able to partition, format and mount the SSDs in the OS, but can't boot from them. I believe that this is because the Pi bootloader doesn't have a SATA overlay to allow the drive to be seen during the boot process. 3) I got a couple of Mini PCIE to M2 adapters and attached NVME drives. Still no success. Booting from EMMC the OS didn't see the drive and couldn't even detect a PCI device using lspci. 4) Then I saw that only specific NVME drives will work. I found some WD Black SN750s which are said to be compatible. These also didn't work and again no PCI device detected by lspci. I've tried this with both CM4s. I've also tried using TPi2 slots 1 and 2, but this makes no difference. I saw a comment that the bootloader won't try loading the NVME overlay if NVME isn't in the boot order, so I used rpiboot to change it to 0xf16 (PCIE NVME first, then EMMC), but that didn't help. I haven't tried using a Mini PCIE to USB adapter with a USB to SATA cable, or an NVME drive in a USB module, because I don't know how the drives would get their power supply (the Mini PCIE doesn't have a supply?) In any case, I believe that connecting via USB will make drive access considerably slower than direct SATA/NVME access to a drive so even if I could get this working it's a compromise. So now I'm staring at the brick wall. I've run out of ideas. Can anyone suggest anything else I can try?

Perhaps not exactly what you were after, but some kind of a workaround - boot from emmc, but mount rootfs already from sata? Or maybe - keep the "mostly read only" filesystems still used from emmc, but the more heavily used/ io-intensive directories (like /var/log, /tmp or whatever else is necessary) move to sata disk, mounting the sata partitions into relevant mount-points in your filesystem? In case of CM4 - don't anyway expect any particularly great performance from SSD - of course, still better than emmc - but I think the main factor for me would be reliability of SSD vs emmc, and ability of replacing broken SSD, when the time comes for it (perhaps not so soon anyway) Having that said - for a light use, without any particularly big amount of writes - emmc should also survive for quite some time. Anyway - let's see - maybe there will be yet some more, better ideas in this thread.

I'm usually suggesting the same - put the OS on the SD card or eMMC (depending on a version of your CM4) and use SATA SSD for storage only (like your home directory, /var/www, logs, whatever - you can symlink). The advantages are you have a separate system disk from the data disk and that means you can easily replace the OS without worrying about the data (like a need to copy them over), if the OS breaks you don't have to do anything to get the data back, you can move the data disk between nodes/modules, etc. it's usually better to separate the OS from the data anyway. You least likely will feel the difference while running the OS from the SD card/eMMC vs USB-SATA SSD.

I've already tried booting from emmc with rootfs on SATA but it doesn't work. With the benefit of 20/20 hindsight it's obvious why it doesn't. The bootloader has to load the main OS from rootfs but the bootloader can't talk to the SATA disk yet because it hasn't got the required overlay. I was wondering about moving just high traffic directories to SATA but I don't know enough about the Linux boot process to know if that would suceed. For example, if I move /var/log to SATA but the OS tries to write to the log before the SATA driver is initialised what happens? It would also need to do something I've never tried...are you allowed to mount a partition over an existing directory that contains files?

I would mount the drive somewhere like

/mnt

and create

log

folder there (so the example path would be

/mnt/sata_ssd/log

), then

cd /var

->

mv log log_old

->

ln -s /mnt/sata_ssd/log

I've tried the WD disk with the latest Raspbian, DietPi and Ubuntu 22.04 and got the same result with each one. The adapter is this one: https://www.amazon.co.uk/Heayzoki-Mini-Adapter-Converter-Support-default/dp/B08M66GK9W/ Another one I saw mentioned (Huygen or similar) wouldn't work with a 2280 NVME because it wouldn't clear the components near the slot 1 socket and would overhang the TPi2 in the slot 2 socket (which doesn't work because my TPi2 is mounted in a Mini ITX case.

This adapter should work, that's the right one. Interesting

PXE boot might be an option but it shifts the goal posts. It would mean that my TPi2 systems would be tied to my NAS for the OS and once I've done that I might as well use the NAS for all other storage which sort of defeats the object of having a nice TPi2 system all in one box. PXE boot for the Pi also looks kinda complicated too!

Where did you find info about WD SN750? I see in the discord no mention of this one, only SN520 and SN530

I've got CM4s and SATA disks doing nothing, so I'll experiement with mounting the high traffic directories and see what happens!

Hmm, ok. Interesting

Anything that might help would be great when you're able to do it. I have a long ToDo list too so I understand 😉

Yeah, so we have the RK1 launch, RK1 32GB prototype testing, TPi2 v2.5 prototype, firmware updates, etc. I'll try to look at this in a few coming days

I've got two RK1s on the way...stuck in an airport warehouse somewhere between China and the UK.

They usually do not stay there for too long

Just checked again and they've arrived at the destination airport...hopefully that means in the UK!

If you check the tracking number in site like https://17track.net/ you might get more detailed information about the localization

17Track still show it waiting to be picked up in China! The link in my TP.com account gives a lot of movement info but no details of where it is at each step of the way.

booting stuff is pretty much on a separate /boot partition so as long as you get to starting the kernel from it (in your case on emmc) - you should be (more or less) good. But, of course, that isn't really all - extending the short story a bit with more details - you may need to ensure that your kernel has SATA drivers compiled into it. This is likely the problem here. And I believe that may be what you were failing on (?) - in standard kernel (may depend on the system- here, based on ubuntu) they are likely compiled as modules, and those, indeed are on ext4 partition, which you hope to mount from SATA, without having SATA support provided, so you can't load those kernel modules - chicken and egg problem 🙂 This, however, can be changed - as above - kernel can be compiled with SATA support built in, instead of loadable modules. I totally understand the inconvenience of having the kernel recompiled, installed etc. - depends on how much determined you are to have your rootfs on SATA SSD. All in all - perhaps the easier option for now will be the other one - starting your system from emmc, just moving io-intensive filesystems to SATA/SSD. As to doubts about what to move - iostat/sar may help you with making decisions - pretty much whichever directory takes some regular writes. For binary/system updates once in a time - things like those in /usr or libraries here and there - they will be still good on emmc imho. Good candidates to move are perhaps things like: - /var/log - /tmp - /var/lib/docker - if you plan to use it - /home Maybe, potentially the whole /var, to make things a bit easier, with less dilemmas? (unless I forget about something, but the important system libs are anyway under /usr)

> you may need to ensure that your kernel has SATA drivers compiled into it The Raspberry Pi OS does, but the Ubuntu image does not (you need to

apt install -y linux-modules-extra-raspi

), DietPi also does iirc

Exactly - as modules are on the SATA drive which you can't use yet 🙂

I am not sure if the modules do not land in the /boot partition. I've never checked that with Ubuntu

This would be the location, I think: /lib/modules/5.15.0-1044-raspi/kernel/drivers/ata - and the module there is ahci for this purpose, if I'm not mistaken.

Yes, yes, I get it, and this is how this may be made

Loaded raspbian onto one of my modules meanwhile, out of curiosity - seems that it's actually also a module:

#
# Controllers with non-SFF native interface
#
CONFIG_SATA_AHCI=m
CONFIG_SATA_MOBILE_LPM_POLICY=0
# CONFIG_SATA_AHCI_PLATFORM is not set
# CONFIG_AHCI_BRCM is not set
# CONFIG_AHCI_DWC is not set
# CONFIG_AHCI_CEVA is not set
# CONFIG_AHCI_QORIQ is not set
# CONFIG_SATA_INIC162X is not set
# CONFIG_SATA_ACARD_AHCI is not set
# CONFIG_SATA_SIL24 is not set
CONFIG_ATA_SFF=y

EDIT: as proved in further messages this doesn't seem to be a problem after all and rootfs mount from SATA disk seems possible with Raspbian bookworm 64bit without any need to recompile the kernel.

Yeah, damn. A docs page about moving the root onto a SATA disk might be useful then, containing all the steps

Let's see actually what google say 🙂

This https://www.jeffgeerling.com/blog/2021/raspberry-pi-os-now-has-sata-support-built#:~:text=After%20months%20of%20testing%20various,support%20out%20of%20the%20box. suggests that SATA support is now compiled into the kernel, which is what I've found...once the OS is up and running the SATA disk is accessible. This https://github.com/raspberrypi/firmware/issues/1653 says that booting from SATA isn't yet possible and probably won't be possible until the EEPROM size is increased. I assume that whilst loading the kernel reads the rootFS to pull drivers in and read config files from /etc, but at that point it's still 'running' under the bootloader and that doesn't have SATA capabilities.

You could also just move the logs

It's probably better to move the whole of /var. It could have mail and a web site in it as well as the logs.

What i've done is to update fstab to mount my drive on /, rsync the old root partition to the drive, change cmdline.txt and then reboot. I'm on my phone at the moment but I'll write it up in more detail when I get off work

Yes, booting from SATA with CM4/RPI4 apparently won't be possible, but that's exactly why we're talking here about booting the system (kernel) from /boot still on emmc - details have been explained in earlier post and also how to check whether you're using modules (most likely) or kernel with drivers built-in to it directly - the procedure to boot with rootfs on SATA disk is actually described quite well in the raspberry pi forum page linked.

SATA SSDs, not USB

Have you considered using the TP2 BMC as PXE server? That way the system would not be tied to the NAS...

Ah yes, I didn't see that you've already linked to an article that describes the procedure. That's exactly what I did minus the rebuild of the kernel. The standard raspbian distro already works

Good to know - however that AHCI being a module also in raspbian made me expect otherwise. Need to test it myself in a free time then, with rootfs from SATA. That's still quite a decent option, even if actual boot from SATA isn't possible.

I did set up a PXE boot system years ago at work, but it's long enough ago that numerous sleep cycles have wiped most of the details from my brain. As you've surmised this is a home network. I hadn't considered the BMC as a host for the PXE boot. I had assumed it wouldn't have had enough spare storage but I don't really know how much is needed, nor how much is free.

The SDCard at the bottom of the board can be mounted into the BMC and there could be plenty of space for images, etc there. To me the bigger challenge is installing a TFTP server (and maybe a DHCP server) into the BMC in a way that does not impact the BMC negatively...

Not sure personally if I would use BMC for that purpose - but may be possible. Space perhaps shouldn't be a problem anyway for rather small kernel and initrd themselves, but there's also a bit of binaries to bring as well to make it work. BMC's own storage is rather limited for its own software as far as I see. It seems possible to run BMC from SD card as well - where the limitation is not a case, but still... I'd leave BMC be a BMC, without acting as additional dhcpd or tftp. Those components might be perhaps built into a router, if offers tftp and configurable pxe. Otherwise - one of running nodes?

OK. My brain hurts! I've just re-done the process of moving rootFS to SATA - a process I've already done many times and failed each time - and of course this time it's worked. I really don't understand what I did differently though. Perhaps a small un-noticed process change and maybe a particular OS image I've not tried before. This was with Raspbian 64-bit Bookworm Lite. It's getting late here so I'm going to sleep on this victory and then try rebuilding my other CM4/SATA pair in slot 1 tomorrow to see if I can replicate the process.

Cool - that's anyway a good news, if it works this way. Glad to see that after all there's no need to recompile kernel to use rootfs on SATA, really good news. Cheers for update.

I've run through the migration process some more times, managing to get it wrong twice more along the way too! However, I've now documented the process and repeated it three times whilst writing the instructions so I'm now feeling confident that the instructions will work.

I did get this working using a CM4-lite (no EMMC) and a USB to SATA adapter, connected to slot 4. The initial imaging was pretty simple as you can directly image the sata disk from the rpi imager, for a regular CM4 it might be possible to disable booting from EMMC, and just have it boot from USB. Performance seems reasonable.

root@r004:~# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: KINGSTON Model:  SKC600256G      Rev: 0
  Type:   Direct-Access                    ANSI  SCSI revision: 06
root@r004:~# lsusb
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 002: ID 14b0:0207 StarTech.com Ltd.  SKC600256G
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 004: ID 1b1c:0c0b Corsair Lighting Node Pro
Bus 001 Device 005: ID 04a9:220e Canon, Inc. CanoScan N1240U/LiDE 30
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

- The adapter model is this one: https://www.amazon.co.uk/gp/product/B00XLAZODE/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1

CM4/RPi can boot off of USB devices for a very long time. The case here is what I'd call native SATA, where you have a PCIe-connected SATA controller.

I don't know if this means anything, but with the mPCIE to M2 adapter installed with the WD Black SN750 attached to it, if I try to flash an OS to the CM4 in that slot (using BMC flashing), it fails with a "USB driver init error". I can't flash the slot until I remove the adapter. This is trying to flash with RPi OS Lite (64-bit Bookworm). I'm wondering whether the adapter is trying to connect the NVME by USB rather than natively? Sort of mPCIE -> USB -> NVME all in one card? There's no electronics on the card though so it's a very simpe device.

This adapter is just a passive adapter. My best guess is it also carries USB signals and this causes signal integrity issues. This will be fixed in the new Turing Pi2 board revision