https://turingpi.com logo
#│forum
10Gb M.2 Ethernet
# │forum
t

terarex

05/25/2023, 6:23 PM
I've ordered one of these cards to test compatibility on the TPiv2: https://www.innodisk.com/en/products/embedded-peripheral/communication/egpl-t101. These aren't cheap, but if a Turing Machines customer has backend storage, high-speed Ethernet connectivity would be a better option than hosting storage locally. The TPiV2's onboard 1Gb Ethernet switch is sufficient for control plane functionality, but the data plane, particularly if it is cross-node, requires higher throughput. This isn't RDMAoF, but it gets closer. While this M.2 card will not work with the CM4 due to the way the PCIe lane is routed on the CM4 adapter board, it is very likely to work perfectly with RK1 and NVIDIA Jetson Orin SoMs. Haven't been able to determine which Marvel ACQ chip is being used as there are quite a few. The driver may or may not be integrated in upstream 3.10+ kernels, but source is provided and support is claimed for both x86 and ARM architectures. I'll report back here once I have the M.2 card and can test with Orin NX 16GB. Waiting patiently for Turing Machines to make the RK1 available for purchase ...
Sorry, the URL seems to have gotten mangled: www.innodisk.com/en/products/embedded-peripheral/communication/egpl-t101
u

_dhanos_

05/25/2023, 6:38 PM
Aren't you using M.2 SSD for the Orin NX?
t

terarex

05/26/2023, 5:32 AM
I am right now, but an alternative that uses a centralized storage subsystem might be interesting. Orin does support PXE and HTTP booting. iSCSI is also included in the boot manager.
u

_dhanos_

05/26/2023, 1:25 PM
I've also heard that someone managed to boot off of USB in Node , but I haven't tried that myself yet
t

terarex

05/26/2023, 2:44 PM
Using a USB SSD will probably be a part of the test process. The M.2 device has UEFI support, so there is a good chance the Orin bootloader will see it. The process for creating a network boot image with L4T using iSCSI will be explored.
Confirmed that this M.2 (B+M connector) card incorporates the Marvell AQC113. Technically, this chip supports PCIe Gen 4.0/3.X/2.0 x4. Innodisk's use of a B+M connector limits the chip to 2 lanes. The card can take full advantage of 10Gb Ethernet with PCIe Gen 3.X x2. PCIe Gen 2.0 does not have sufficient throughput to take full advantage of 10Gb. I do not see this as an issue since both NVIDIA Orin, including Orin Nano, and RK1 support PCIe Gen 3 x4 on the M.2 socket.
u

_dhanos_

05/27/2023, 11:21 PM
Orin NX actually supports PCIe Gen 4.0
t

terarex

05/27/2023, 11:26 PM
Yeah. Technically, the chip supports Gen 4, but I question whether there is much to be gained. I did several searches for a dual M.2 adapter in a U.2 form factor. There don't seem to be any that support bifurcation. That would support a 2-lane NVMe SSD and the 2-lane 10Gb NIC.
u

_dhanos_

05/27/2023, 11:28 PM
Bifurcation is a task of the controller, not adapter, so the module itself would need to support it
I believe the Orin modules do, but I may be wrong
So, if the module supports bifurcation, you'd need a way to split the PCIe lanes, which can be a challenge for such configuration 🙂
It was actually RK1, and more specifically RK358 where I have seen bifurcation
Copy code
Support aggregation and bifurcation with 1x 4lanes, 2x 2lanes, 4x 1lanes and 1x
2lanes + 2x 1lanes
Orin modules do not support bifurcation on this bus. The PCIe options are:
Which means only the x2 bus can be bifurcated
I would love to link the document, but the way Nvidia handles links, it'll start working shortly
https://developer.nvidia.com/embedded/downloads#?search=design&tx=$product,jetson_orin_nx,jetson_orin_nano ->
Jetson Orin NX Series and Orin Nano Series Design Guide
-> chapter2, page 4
s

silopolis

08/02/2023, 6:34 PM
hello guys 🙂 Looks like we're hunting for the same 5 leg sheep !
@terarex spent hours looking for dual M.2 adapter that could be connected one way or the other (Oculink, SFF-xxxx) to the 4 lanes of the modules' M.2 slots... To no luck 😢
Only solutions I've found are converting back to plain PCIe slots :/
But then as I'm almost PCI illiterate, I'm not even sure that would work
t

terarex

08/02/2023, 6:43 PM
I haven't gotten the Orin NX boot manager to see my 10Gb M.2 card. It's B+M so that might be the issue. I need to install L4T 35.3.1 on a USB stick to free up the M.2 slot. Then see whether lspci actually sees the card. If it does, I need to figure out how to inject the driver into the Orin NX's boot manager.
s

silopolis

08/02/2023, 6:45 PM
OK, hope that will doit... keep us posted
@_dhanos_ do you think the configurations with the hardware I just linked to could work with RK1 modules and their bifurcation support?
t

terarex

08/02/2023, 6:47 PM
Right now I'm working on something else, but I'll get back to that research project in the next week.
u

_dhanos_

08/03/2023, 10:31 AM
In theory it should, but I can't be sure, of course
s

silopolis

08/03/2023, 10:33 AM
OK, that's a start! 🙂 I'll see if I can get more clues from insiders...
Well digging a bit in channels and forum, it looks like I'll hardly find s/o better informed 😅 #shame
3 Views