Hardware: TPi2.4 w/ 4 cm4 (8GB, 32GB EEMC) & 2 One TB SATA drives on Node 3 Software: Talos 1.12.2, metallb, traefik, helm Objective: 1) Make SATA storage available as persistent storage to all nodes; and 2) optionally, make SATA storage available outside the cluster (NFS share?) I tried configuring the nfs-server-provisioner helm chart but became confused on how to associate the server with the storage. (I was able to get the nfs-subdir-external-provisioner [client] chart to work on another cluster.) I'm very much a noob so just saying "*Installing the Helm chart, and then inspecting the PersistentVolumeClaim's created will provide the necessary names for your PersistentVolume's to bind to*" just doesn't make any sense to me.

NFS hosted external to the cluster is the way I've always set it up. You can set it up on several different NAS storage offerings, or on a node in the cluster. It's still external that way however.

I want to go the other way and make cluster internal storage available to external nfs clients; therefore, I need a recommendation for which internal storage manager can make use of the SATA drives on node 3.

The way I'd look at setting that up, if I were so inclined, would be to setup an nfs server container as a pod. It'd need to have a local storage claim defined for it, I think. Then you could expose that via perhaps a node port or for sure a load balancer (I use metallb personally). You'd proabably also need to have a configmap setup to and mounted to the pod to setup the export file, but I don't know if that's completely necessary or not. Personally I don't think I'd go that way though. NFS has some performance and reliablity concerns.

I tried goiong the way you wish... let me tell you it is difficult to crate local storage (especially with that stupid talos). The problem with one of the nodes being the NFS server is cluster restarts (power fails and such), The nodes will start up 1,2,3,4 very quickly if you need nfs using it in a pod is almost worthless until the pod comes up. If you store anything on the drives needed for the cluster then your in a catch-22. Next I tried to do all the NFX at the OS level (part of the boot process). This cannot be done with talos (POS OS anyway) so I went with K3's running on unbutu. This mostly worked except the annoying cluster restarts didnt allways come up clean. Moved NFS to my NAS and that parially solved problem (and where i sit today). the NAS itakes some time to recover from power outages as well.

Yeah, that's my experience as well. And why I went to longhorn since if any node in the cluster is available the storage is available.

LongHorn is one Option if k8s/k3s based. Rook would be another, or plain Ceph.

Does anyone know if LongHorn, Rook, or Ceph tries to replicate all storage to every node? Since my TPiV2.4 cluster is running CM4 nodes with internal 32GB of EMMC storage, I am reluctant to use a storage solution that is constantly writing to the EMMC device.

Longhorn does 3 copies by default but you can increase or decrease it. Rook/Ceph is similar but I don't recall offhand the particulars.

@SometimesDreadfull Thanks for the info. My conclusion from all this is to use CM4s only on node 3. 🙂 My TPi2.5 is stuffed with RK1s and 1TB NVME drives, so I don't worry about it either.

longhorn/rook: you can label&select nodes which provide storage and you can set a number of maximal replicas ceph: depends how you set it up, if you use rook-ceph above applies, I would not run anythink write intensiv withou monitoring on the RK1 nodes from eMMC

@j0ju Did you mean "on the CM4 nodes from eMMC"? Anyway, thanks for the reply I actually think that intensive writes should be prohibited on the eMMC from any source.

eMMC is eMMC with or without an "e", these are WOFO devices (write often fail often) it does not matter if RK1, CM4 or anyother SBC