https://turingpi.com logo
#│forum
Trying to boot CM4 from a 'real' disk
# │forum
p

packman2008

01/23/2024, 10:08 AM
I have two CM4s (4GB RAM/16GB EMMC) which I've installed in my TPi2 and can successfully flash them (into EMMC). I wanted to boot the CM4s from some more robust and faster media. I tried... 1) Using the NVME slots on the base of the TPi2., but found that this doesn't work because the CM4's single PCI lane isn't routed to those NVME slots. 2) I got a couple of Mini PCIE to SATA adapters and connected SATA SSDs. I was able to partition, format and mount the SSDs in the OS, but can't boot from them. I believe that this is because the Pi bootloader doesn't have a SATA overlay to allow the drive to be seen during the boot process. 3) I got a couple of Mini PCIE to M2 adapters and attached NVME drives. Still no success. Booting from EMMC the OS didn't see the drive and couldn't even detect a PCI device using lspci. 4) Then I saw that only specific NVME drives will work. I found some WD Black SN750s which are said to be compatible. These also didn't work and again no PCI device detected by lspci. I've tried this with both CM4s. I've also tried using TPi2 slots 1 and 2, but this makes no difference. I saw a comment that the bootloader won't try loading the NVME overlay if NVME isn't in the boot order, so I used rpiboot to change it to 0xf16 (PCIE NVME first, then EMMC), but that didn't help. I haven't tried using a Mini PCIE to USB adapter with a USB to SATA cable, or an NVME drive in a USB module, because I don't know how the drives would get their power supply (the Mini PCIE doesn't have a supply?) In any case, I believe that connecting via USB will make drive access considerably slower than direct SATA/NVME access to a drive so even if I could get this working it's a compromise. So now I'm staring at the brick wall. I've run out of ideas. Can anyone suggest anything else I can try?
j

jrudowski

01/23/2024, 2:53 PM
Perhaps not exactly what you were after, but some kind of a workaround - boot from emmc, but mount rootfs already from sata? Or maybe - keep the "mostly read only" filesystems still used from emmc, but the more heavily used/ io-intensive directories (like /var/log, /tmp or whatever else is necessary) move to sata disk, mounting the sata partitions into relevant mount-points in your filesystem? In case of CM4 - don't anyway expect any particularly great performance from SSD - of course, still better than emmc - but I think the main factor for me would be reliability of SSD vs emmc, and ability of replacing broken SSD, when the time comes for it (perhaps not so soon anyway) Having that said - for a light use, without any particularly big amount of writes - emmc should also survive for quite some time. Anyway - let's see - maybe there will be yet some more, better ideas in this thread.
Something yet I'm looking to try in the future maybe - PXE boot, with data on SSD
u

_dhanos_

01/23/2024, 2:57 PM
I'm usually suggesting the same - put the OS on the SD card or eMMC (depending on a version of your CM4) and use SATA SSD for storage only (like your home directory, /var/www, logs, whatever - you can symlink). The advantages are you have a separate system disk from the data disk and that means you can easily replace the OS without worrying about the data (like a need to copy them over), if the OS breaks you don't have to do anything to get the data back, you can move the data disk between nodes/modules, etc. it's usually better to separate the OS from the data anyway. You least likely will feel the difference while running the OS from the SD card/eMMC vs USB-SATA SSD.
I also used a trick of moving just the root filesystem to a USB drive (like a HDD or SSD) with RPi4, so I guess it should work about the same with CM4s
The fact that your CM4 does not detect this WD drive is curious, though. Which OS are you using? Have you tried with Raspberry Pi OS?
Also, which exactly Mini PCIE to M2 adapter did you get? Not all of them will work
p

packman2008

01/23/2024, 3:38 PM
I've already tried booting from emmc with rootfs on SATA but it doesn't work. With the benefit of 20/20 hindsight it's obvious why it doesn't. The bootloader has to load the main OS from rootfs but the bootloader can't talk to the SATA disk yet because it hasn't got the required overlay. I was wondering about moving just high traffic directories to SATA but I don't know enough about the Linux boot process to know if that would suceed. For example, if I move /var/log to SATA but the OS tries to write to the log before the SATA driver is initialised what happens? It would also need to do something I've never tried...are you allowed to mount a partition over an existing directory that contains files?
u

_dhanos_

01/23/2024, 3:41 PM
I would mount the drive somewhere like
/mnt
and create
log
folder there (so the example path would be
/mnt/sata_ssd/log
), then
cd /var
->
mv log log_old
->
ln -s /mnt/sata_ssd/log
If you want to mount a partition, the mounting point directory must exist and be empty
p

packman2008

01/23/2024, 3:48 PM
I've tried the WD disk with the latest Raspbian, DietPi and Ubuntu 22.04 and got the same result with each one. The adapter is this one: https://www.amazon.co.uk/Heayzoki-Mini-Adapter-Converter-Support-default/dp/B08M66GK9W/ Another one I saw mentioned (Huygen or similar) wouldn't work with a 2280 NVME because it wouldn't clear the components near the slot 1 socket and would overhang the TPi2 in the slot 2 socket (which doesn't work because my TPi2 is mounted in a Mini ITX case.
u

_dhanos_

01/23/2024, 3:51 PM
This adapter should work, that's the right one. Interesting
I might need to get one of these NVMes myself and test
p

packman2008

01/23/2024, 4:03 PM
PXE boot might be an option but it shifts the goal posts. It would mean that my TPi2 systems would be tied to my NAS for the OS and once I've done that I might as well use the NAS for all other storage which sort of defeats the object of having a nice TPi2 system all in one box. PXE boot for the Pi also looks kinda complicated too!
u

_dhanos_

01/23/2024, 4:04 PM
Where did you find info about WD SN750? I see in the discord no mention of this one, only SN520 and SN530
p

packman2008

01/23/2024, 4:05 PM
I've got CM4s and SATA disks doing nothing, so I'll experiement with mounting the high traffic directories and see what happens!
It's on Jeff Geerlings list of NVMEs that work with Pi PCIE
SN520/530s all seem to be back-ordered...order now get one in a couple of weeks (maybe)
u

_dhanos_

01/23/2024, 4:10 PM
Hmm, ok. Interesting
I see that Samsung 970 Evo Plus is also on the list and I have one as well as the exact same adapter that you have, so I may test this combination
Can't promise when I do this, but I'm adding it to the list
p

packman2008

01/23/2024, 4:14 PM
Anything that might help would be great when you're able to do it. I have a long ToDo list too so I understand 😉
u

_dhanos_

01/23/2024, 4:16 PM
Yeah, so we have the RK1 launch, RK1 32GB prototype testing, TPi2 v2.5 prototype, firmware updates, etc. I'll try to look at this in a few coming days
p

packman2008

01/23/2024, 4:17 PM
I've got two RK1s on the way...stuck in an airport warehouse somewhere between China and the UK.
u

_dhanos_

01/23/2024, 4:17 PM
They usually do not stay there for too long
p

packman2008

01/23/2024, 4:19 PM
Just checked again and they've arrived at the destination airport...hopefully that means in the UK!
u

_dhanos_

01/23/2024, 4:20 PM
If you check the tracking number in site like https://17track.net/ you might get more detailed information about the localization
p

packman2008

01/23/2024, 4:23 PM
17Track still show it waiting to be picked up in China! The link in my TP.com account gives a lot of movement info but no details of where it is at each step of the way.
j

jrudowski

01/23/2024, 6:05 PM
booting stuff is pretty much on a separate /boot partition so as long as you get to starting the kernel from it (in your case on emmc) - you should be (more or less) good. But, of course, that isn't really all - extending the short story a bit with more details - you may need to ensure that your kernel has SATA drivers compiled into it. This is likely the problem here. And I believe that may be what you were failing on (?) - in standard kernel (may depend on the system- here, based on ubuntu) they are likely compiled as modules, and those, indeed are on ext4 partition, which you hope to mount from SATA, without having SATA support provided, so you can't load those kernel modules - chicken and egg problem 🙂 This, however, can be changed - as above - kernel can be compiled with SATA support built in, instead of loadable modules. I totally understand the inconvenience of having the kernel recompiled, installed etc. - depends on how much determined you are to have your rootfs on SATA SSD. All in all - perhaps the easier option for now will be the other one - starting your system from emmc, just moving io-intensive filesystems to SATA/SSD. As to doubts about what to move - iostat/sar may help you with making decisions - pretty much whichever directory takes some regular writes. For binary/system updates once in a time - things like those in /usr or libraries here and there - they will be still good on emmc imho. Good candidates to move are perhaps things like: - /var/log - /tmp - /var/lib/docker - if you plan to use it - /home Maybe, potentially the whole /var, to make things a bit easier, with less dilemmas? (unless I forget about something, but the important system libs are anyway under /usr)
And either mounting under the right mountpoint or linking, like @_dhanos_ suggested - should perhaps work
u

_dhanos_

01/23/2024, 6:08 PM
> you may need to ensure that your kernel has SATA drivers compiled into it The Raspberry Pi OS does, but the Ubuntu image does not (you need to
apt install -y linux-modules-extra-raspi
), DietPi also does iirc
No kernel recompilation for these 3 will be needed
Oh wait, so this won's work if the SATA driver is a kernel module?
j

jrudowski

01/23/2024, 6:11 PM
Exactly - as modules are on the SATA drive which you can't use yet 🙂
Recompiling the kernel to make them built into the kernel image, instead of modules, should perhaps resolve that problem.
But shouldn't have anything to do with boot or overlays as such, as far as I'm aware
But I'm still kind of fresh to Linux on ARM, I may be still missing something yet. (not Linux in general though - old enough to remember recompiling kernels as "standard procedure" 🙂)
u

_dhanos_

01/23/2024, 6:14 PM
I am not sure if the modules do not land in the /boot partition. I've never checked that with Ubuntu
j

jrudowski

01/23/2024, 6:18 PM
This would be the location, I think: /lib/modules/5.15.0-1044-raspi/kernel/drivers/ata - and the module there is ahci for this purpose, if I'm not mistaken.
Example for my version of kernel, of course - depending on the system, that part of the path will change
So, say, in general, to see what's inside: ls -la /lib/modules/$(uname -r)/kernel/drivers/ata
u

_dhanos_

01/23/2024, 6:22 PM
Yes, yes, I get it, and this is how this may be made
But I believe these drivers are a part of the kernel binary in the Raspberry Pi OS, but I've never checked that actually.
j

jrudowski

01/23/2024, 6:30 PM
Loaded raspbian onto one of my modules meanwhile, out of curiosity - seems that it's actually also a module:
Copy code
#
# Controllers with non-SFF native interface
#
CONFIG_SATA_AHCI=m
CONFIG_SATA_MOBILE_LPM_POLICY=0
# CONFIG_SATA_AHCI_PLATFORM is not set
# CONFIG_AHCI_BRCM is not set
# CONFIG_AHCI_DWC is not set
# CONFIG_AHCI_CEVA is not set
# CONFIG_AHCI_QORIQ is not set
# CONFIG_SATA_INIC162X is not set
# CONFIG_SATA_ACARD_AHCI is not set
# CONFIG_SATA_SIL24 is not set
CONFIG_ATA_SFF=y
EDIT: as proved in further messages this doesn't seem to be a problem after all and rootfs mount from SATA disk seems possible with Raspbian bookworm 64bit without any need to recompile the kernel.
u

_dhanos_

01/23/2024, 6:32 PM
Yeah, damn. A docs page about moving the root onto a SATA disk might be useful then, containing all the steps
j

jrudowski

01/23/2024, 6:32 PM
Let's see actually what google say 🙂
There's a lot of documents describing rootfs on usb drive (can be still ssd, of course) - which may confuse a bit. But searching specifically for sata rootfs I found this - actually even on raspberry pi forum: https://forums.raspberrypi.com/viewtopic.php?t=317875 From 2021, but seem - had just a quick look - more or less accurate: https://forums.raspberrypi.com/viewtopic.php?t=317875
Raspberry website has also the section describing kernel recompilation/reinstallation - which may also help. Probably the most annoying part of this whole operation.
p

packman2008

01/23/2024, 8:17 PM
This https://www.jeffgeerling.com/blog/2021/raspberry-pi-os-now-has-sata-support-built#:~:text=After%20months%20of%20testing%20various,support%20out%20of%20the%20box. suggests that SATA support is now compiled into the kernel, which is what I've found...once the OS is up and running the SATA disk is accessible. This https://github.com/raspberrypi/firmware/issues/1653 says that booting from SATA isn't yet possible and probably won't be possible until the EEPROM size is increased. I assume that whilst loading the kernel reads the rootFS to pull drivers in and read config files from /etc, but at that point it's still 'running' under the bootloader and that doesn't have SATA capabilities.
I've got a copy of DietPi installed now with the SATA disk mounted into /mnt/sata. I've successfully created symlinks from /home, /root, /tmp and /var into sub-directories created below /mnt/sata. /home and /root were easy to do. /tmp initially objected but that's because it was mounted as a partition in it's own right. I unmounted it and then was able to delete it. /tmp was empty so nothing was lost. When doing /var I got an error saying that I couldn't move /var/log because the resource was busy. I moved the rest of /var and created a new /var/log directory. The new directory was used at the next boot so I was then able to delete the old /var/log (only a few tiny log files lost). This has survived a couple of reboots so it's working. I'll redo this on my second CM4 and document it for myself. If it's of use to others then I'll happily provide it to be put somewhere.
u

_dhanos_

01/23/2024, 8:35 PM
You could also just move the logs
p

packman2008

01/23/2024, 9:06 PM
It's probably better to move the whole of /var. It could have mail and a web site in it as well as the logs.
t

the_boom_boom

01/23/2024, 9:08 PM
What i've done is to update fstab to mount my drive on /, rsync the old root partition to the drive, change cmdline.txt and then reboot. I'm on my phone at the moment but I'll write it up in more detail when I get off work
j

jrudowski

01/23/2024, 9:16 PM
Yes, booting from SATA with CM4/RPI4 apparently won't be possible, but that's exactly why we're talking here about booting the system (kernel) from /boot still on emmc - details have been explained in earlier post and also how to check whether you're using modules (most likely) or kernel with drivers built-in to it directly - the procedure to boot with rootfs on SATA disk is actually described quite well in the raspberry pi forum page linked.
Just to double-check - are you using disk on SATA or USB connector (USB SATA adapter, to be clear - which simplifies things a bit)?
t

the_boom_boom

01/23/2024, 9:19 PM
SATA SSDs, not USB
d

dojo.major

01/23/2024, 9:25 PM
Have you considered using the TP2 BMC as PXE server? That way the system would not be tied to the NAS...
t

the_boom_boom

01/23/2024, 9:25 PM
Ah yes, I didn't see that you've already linked to an article that describes the procedure. That's exactly what I did minus the rebuild of the kernel. The standard raspbian distro already works
j

jrudowski

01/23/2024, 9:29 PM
Good to know - however that AHCI being a module also in raspbian made me expect otherwise. Need to test it myself in a free time then, with rootfs from SATA. That's still quite a decent option, even if actual boot from SATA isn't possible.
Also suggested earlier - https://discord.com/channels/754950670175436841/1199294692450369637/1199367169578897428 - RPI4/CM4 seem to be capable of booting from PXE, I'd still consider it worth trying, indeed. And no NAS/NFS should need to be involved - but of course, preparing things for PXE boot require a bit of additional knowledge and configuration in the (home?) network.
Missed that earlier - actually mounting into non-empty directory is possible, but the mounted partition with its data will mask the contents of mountpoint directory, until disk is unmounted again. (talking about Linux here - I can't tell for sure about all Unix or Unix-like systems, some of them perhaps the same as Linux - but no warranty. Irrelevant here, though, I guess)
p

packman2008

01/23/2024, 9:50 PM
I did set up a PXE boot system years ago at work, but it's long enough ago that numerous sleep cycles have wiped most of the details from my brain. As you've surmised this is a home network. I hadn't considered the BMC as a host for the PXE boot. I had assumed it wouldn't have had enough spare storage but I don't really know how much is needed, nor how much is free.
d

dojo.major

01/23/2024, 9:59 PM
The SDCard at the bottom of the board can be mounted into the BMC and there could be plenty of space for images, etc there. To me the bigger challenge is installing a TFTP server (and maybe a DHCP server) into the BMC in a way that does not impact the BMC negatively...
j

jrudowski

01/23/2024, 9:59 PM
Not sure personally if I would use BMC for that purpose - but may be possible. Space perhaps shouldn't be a problem anyway for rather small kernel and initrd themselves, but there's also a bit of binaries to bring as well to make it work. BMC's own storage is rather limited for its own software as far as I see. It seems possible to run BMC from SD card as well - where the limitation is not a case, but still... I'd leave BMC be a BMC, without acting as additional dhcpd or tftp. Those components might be perhaps built into a router, if offers tftp and configurable pxe. Otherwise - one of running nodes?
Yeah, exactly - not sure if BMC is such a great idea for it.
Depending on resources and chosen scenario - one of nodes might be acting as dhcp for other nodes, also tftp/pxe - other nodes booting from it and mounting their own disks of choice etc.
or external small SoC - another rpi, bpi or so maybe - replacing the dhcp from internet router or so.
p

packman2008

01/23/2024, 10:20 PM
OK. My brain hurts! I've just re-done the process of moving rootFS to SATA - a process I've already done many times and failed each time - and of course this time it's worked. I really don't understand what I did differently though. Perhaps a small un-noticed process change and maybe a particular OS image I've not tried before. This was with Raspbian 64-bit Bookworm Lite. It's getting late here so I'm going to sleep on this victory and then try rebuilding my other CM4/SATA pair in slot 1 tomorrow to see if I can replicate the process.
j

jrudowski

01/23/2024, 10:23 PM
Cool - that's anyway a good news, if it works this way. Glad to see that after all there's no need to recompile kernel to use rootfs on SATA, really good news. Cheers for update.
p

packman2008

01/24/2024, 2:23 PM
I've run through the migration process some more times, managing to get it wrong twice more along the way too! However, I've now documented the process and repeated it three times whilst writing the instructions so I'm now feeling confident that the instructions will work.
b

batz_10k

01/24/2024, 11:07 PM
I did get this working using a CM4-lite (no EMMC) and a USB to SATA adapter, connected to slot 4. The initial imaging was pretty simple as you can directly image the sata disk from the rpi imager, for a regular CM4 it might be possible to disable booting from EMMC, and just have it boot from USB. Performance seems reasonable.
Copy code
root@r004:~# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: KINGSTON Model:  SKC600256G      Rev: 0
  Type:   Direct-Access                    ANSI  SCSI revision: 06
root@r004:~# lsusb
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 002: ID 14b0:0207 StarTech.com Ltd.  SKC600256G
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 004: ID 1b1c:0c0b Corsair Lighting Node Pro
Bus 001 Device 005: ID 04a9:220e Canon, Inc. CanoScan N1240U/LiDE 30
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
- The adapter model is this one: https://www.amazon.co.uk/gp/product/B00XLAZODE/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1
u

_dhanos_

01/24/2024, 11:12 PM
CM4/RPi can boot off of USB devices for a very long time. The case here is what I'd call native SATA, where you have a PCIe-connected SATA controller.
p

packman2008

01/26/2024, 9:24 AM
I don't know if this means anything, but with the mPCIE to M2 adapter installed with the WD Black SN750 attached to it, if I try to flash an OS to the CM4 in that slot (using BMC flashing), it fails with a "USB driver init error". I can't flash the slot until I remove the adapter. This is trying to flash with RPi OS Lite (64-bit Bookworm). I'm wondering whether the adapter is trying to connect the NVME by USB rather than natively? Sort of mPCIE -> USB -> NVME all in one card? There's no electronics on the card though so it's a very simpe device.
u

_dhanos_

01/26/2024, 11:50 AM
This adapter is just a passive adapter. My best guess is it also carries USB signals and this causes signal integrity issues. This will be fixed in the new Turing Pi2 board revision
2 Views