Here are some rudimentary disk benchmarking results with a very limited set of disks. They cover * eMMC, * SATA SSD on node 3, * mdadm RAID-1 SATA SSD and HDD through a miniPCIe adapter on node 1/2, * mdadm RAID-1 SATA SSD and HDD through a mini PCIe adapter and 5-port multiplier on node 1/2, * mdadm RAID-1 SATA SSD through a USB to SATA adapter on node 4 If you care to expand on the number of runs, or number of drives, please do!

any plans to test a filesystem with CoW like ZFS?

Not unless I acquire a bunch of additional disks. But @DhanOS (Daniel Kukiela) has 10 1TB disks connected to two 5-port SATA multipliers so maybe he'll include some ZFS tests when he has time to do some benchmarking.

these are SATA HDDs right?

This value looks out of place.