10Gb M.2 Ethernet

Question

I ve ordered one of these cards to test compatibility on the TPiv2 https www innodisk com en products embedded peripheral communication egpl t101 These aren t cheap but if a Turing Machines customer has backend storage high speed Ethernet connectivity would be a better option than hosting storage locally The TPiV2 s onboard 1Gb Ethernet switch is sufficient for control plane functionality but the data plane particularly if it is cross node requires higher throughput This isn t RDMAoF but it gets closer While this M 2 card will not work with the CM4 due to the way the PCIe lane is routed on the CM4 adapter board it is very likely to work perfectly with RK1 and NVIDIA Jetson Orin SoMs Haven t been able to determine which Marvel ACQ chip is being used as there are quite a few The driver may or may not be integrated in upstream 3 10+ kernels but source is provided and support is claimed for both x86 and ARM architectures I ll report back here once I have the M 2 card and can test with Orin NX 16GB Waiting patiently for Turing Machines to make the RK1 available for purchase

terarex · Answer

Sorry the URL seems to have gotten mangled www innodisk com en products embedded peripheral communication egpl t101

_dhanos_ · Answer

Aren t you using M 2 SSD for the Orin NX

terarex · Answer

I am right now but an alternative that uses a centralized storage subsystem might be interesting Orin does support PXE and HTTP booting iSCSI is also included in the boot manager

_dhanos_ · Answer

I ve also heard that someone managed to boot off of USB in Node but I haven t tried that myself yet

terarex · Answer

Using a USB SSD will probably be a part of the test process The M 2 device has UEFI support so there is a good chance the Orin bootloader will see it The process for creating a network boot image with L4T using iSCSI will be explored

terarex · Answer

Confirmed that this M 2 B+M connector card incorporates the Marvell AQC113 Technically this chip supports PCIe Gen 4 0 3 X 2 0 x4 Innodisk s use of a B+M connector limits the chip to 2 lanes The card can take full advantage of 10Gb Ethernet with PCIe Gen 3 X x2 PCIe Gen 2 0 does not have sufficient throughput to take full advantage of 10Gb I do not see this as an issue since both NVIDIA Orin including Orin Nano and RK1 support PCIe Gen 3 x4 on the M 2 socket

_dhanos_ · Answer

Orin NX actually supports PCIe Gen 4 0

terarex · Answer

Yeah Technically the chip supports Gen 4 but I question whether there is much to be gained I did several searches for a dual M 2 adapter in a U 2 form factor There don t seem to be any that support bifurcation That would support a 2 lane NVMe SSD and the 2 lane 10Gb NIC

_dhanos_ · Answer

Bifurcation is a task of the controller not adapter so the module itself would need to support it

_dhanos_ · Answer

I believe the Orin modules do but I may be wrong

_dhanos_ · Answer

So if the module supports bifurcation you d need a way to split the PCIe lanes which can be a challenge for such configuration

_dhanos_ · Answer

It was actually RK1 and more specifically RK358 where I have seen bifurcation

_dhanos_ · Answer

``` Support aggregation and bifurcation with 1x 4lanes 2x 2lanes 4x 1lanes and 1x 2lanes + 2x 1lanes ```

_dhanos_ · Answer

https www cnx software com pdf Rockchip C2 A0RK3588 C2 A0Datasheet C2 A0V0 1 20210727 pdf > page 15

_dhanos_ · Answer

Orin modules do not support bifurcation on this bus The PCIe options are

_dhanos_ · Answer

Which means only the x2 bus can be bifurcated

_dhanos_ · Answer

I would love to link the document but the way Nvidia handles links it ll start working shortly

_dhanos_ · Answer

https developer nvidia com embedded downloads search=design tx=$product jetson orin nx jetson orin nano > `Jetson Orin NX Series and Orin Nano Series Design Guide` > chapter2 page 4

silopolis · Answer

hello guys Looks like we re hunting for the same 5 leg sheep

silopolis · Answer

< terarex> spent hours looking for dual M 2 adapter that could be connected one way or the other Oculink SFF xxxx to the 4 lanes of the modules M 2 slots To no luck

silopolis · Answer

Only solutions I ve found are converting back to plain PCIe slots

silopolis · Answer

Either with this https www ioi com tw products proddetail aspx AppID=1008 CatID=116 HostID=2113 ProdID=1160012

silopolis · Answer

Or https www microsatacables com pci e 1x 4x card to m 2 m key 4 lane adapter with clkreq m2 1522 clkreq + https www sfpcables com nvme ssd adapter card for m 2 x4 1 m 2 m key and 1 m 2 b key connectors

silopolis · Answer

But then as I m almost PCI illiterate I m not even sure that would work

terarex · Answer

I haven t gotten the Orin NX boot manager to see my 10Gb M 2 card It s B+M so that might be the issue I need to install L4T 35 3 1 on a USB stick to free up the M 2 slot Then see whether lspci actually sees the card If it does I need to figure out how to inject the driver into the Orin NX s boot manager

silopolis · Answer

OK hope that will doit keep us posted

silopolis · Answer

< dhanos > do you think the configurations with the hardware I just linked to could work with RK1 modules and their bifurcation support

terarex · Answer

Right now I m working on something else but I ll get back to that research project in the next week

_dhanos_ · Answer

In theory it should but I can t be sure of course

silopolis · Answer

OK that s a start I ll see if I can get more clues from insiders

silopolis · Answer

Well digging a bit in channels and forum it looks like I ll hardly find s o better informed shame