10Gb M.2 Ethernet
# │forumt
Terarex (Dan Donovan)
05/25/2023, 6:23 PMI've ordered one of these cards to test compatibility on the TPiv2: https://www.innodisk.com/en/products/embedded-peripheral/communication/egpl-t101. These aren't cheap, but if a Turing Machines customer has backend storage, high-speed Ethernet connectivity would be a better option than hosting storage locally. The TPiV2's onboard 1Gb Ethernet switch is sufficient for control plane functionality, but the data plane, particularly if it is cross-node, requires higher throughput. This isn't RDMAoF, but it gets closer.
While this M.2 card will not work with the CM4 due to the way the PCIe lane is routed on the CM4 adapter board, it is very likely to work perfectly with RK1 and NVIDIA Jetson Orin SoMs. Haven't been able to determine which Marvel ACQ chip is being used as there are quite a few.
The driver may or may not be integrated in upstream 3.10+ kernels, but source is provided and support is claimed for both x86 and ARM architectures.
I'll report back here once I have the M.2 card and can test with Orin NX 16GB. Waiting patiently for Turing Machines to make the RK1 available for purchase ...
Terarex (Dan Donovan)
05/25/2023, 6:25 PMSorry, the URL seems to have gotten mangled: www.innodisk.com/en/products/embedded-peripheral/communication/egpl-t101
d
DhanOS (Daniel Kukiela)
05/25/2023, 6:38 PMAren't you using M.2 SSD for the Orin NX?
t
Terarex (Dan Donovan)
05/26/2023, 5:32 AMI am right now, but an alternative that uses a centralized storage subsystem might be interesting.
Orin does support PXE and HTTP booting. iSCSI is also included in the boot manager.
d
DhanOS (Daniel Kukiela)
05/26/2023, 1:25 PMI've also heard that someone managed to boot off of USB in Node , but I haven't tried that myself yet
t
Terarex (Dan Donovan)
05/26/2023, 2:44 PMUsing a USB SSD will probably be a part of the test process.
The M.2 device has UEFI support, so there is a good chance the Orin bootloader will see it. The process for creating a network boot image with L4T using iSCSI will be explored.
Terarex (Dan Donovan)
05/27/2023, 11:17 PMConfirmed that this M.2 (B+M connector) card incorporates the Marvell AQC113.
Technically, this chip supports PCIe Gen 4.0/3.X/2.0 x4. Innodisk's use of a B+M connector limits the chip to 2 lanes. The card can take full advantage of 10Gb Ethernet with PCIe Gen 3.X x2. PCIe Gen 2.0 does not have sufficient throughput to take full advantage of 10Gb. I do not see this as an issue since both NVIDIA Orin, including Orin Nano, and RK1 support PCIe Gen 3 x4 on the M.2 socket.
d
DhanOS (Daniel Kukiela)
05/27/2023, 11:21 PMOrin NX actually supports PCIe Gen 4.0
t
Terarex (Dan Donovan)
05/27/2023, 11:26 PMYeah. Technically, the chip supports Gen 4, but I question whether there is much to be gained.
I did several searches for a dual M.2 adapter in a U.2 form factor. There don't seem to be any that support bifurcation. That would support a 2-lane NVMe SSD and the 2-lane 10Gb NIC.
d
DhanOS (Daniel Kukiela)
05/27/2023, 11:28 PMBifurcation is a task of the controller, not adapter, so the module itself would need to support it
DhanOS (Daniel Kukiela)
05/27/2023, 11:28 PMI believe the Orin modules do, but I may be wrong
DhanOS (Daniel Kukiela)
05/27/2023, 11:29 PMSo, if the module supports bifurcation, you'd need a way to split the PCIe lanes, which can be a challenge for such configuration 🙂
DhanOS (Daniel Kukiela)
05/27/2023, 11:31 PMIt was actually RK1, and more specifically RK358 where I have seen bifurcation
DhanOS (Daniel Kukiela)
05/27/2023, 11:31 PM Copy code
Support aggregation and bifurcation with 1x 4lanes, 2x 2lanes, 4x 1lanes and 1x
2lanes + 2x 1lanesDhanOS (Daniel Kukiela)
05/27/2023, 11:31 PMDhanOS (Daniel Kukiela)
05/27/2023, 11:34 PMOrin modules do not support bifurcation on this bus. The PCIe options are:
DhanOS (Daniel Kukiela)
05/27/2023, 11:34 PMDhanOS (Daniel Kukiela)
05/27/2023, 11:34 PMWhich means only the x2 bus can be bifurcated
DhanOS (Daniel Kukiela)
05/27/2023, 11:35 PMI would love to link the document, but the way Nvidia handles links, it'll start working shortly
DhanOS (Daniel Kukiela)
05/27/2023, 11:36 PMhttps://developer.nvidia.com/embedded/downloads#?search=design&tx=$product,jetson_orin_nx,jetson_orin_nano ->
Jetson Orin NX Series and Orin Nano Series Design Guides
silopolis
08/02/2023, 6:34 PMhello guys 🙂
Looks like we're hunting for the same 5 leg sheep !
silopolis
08/02/2023, 6:38 PM@Terarex (Dan Donovan) spent hours looking for dual M.2 adapter that could be connected one way or the other (Oculink, SFF-xxxx) to the 4 lanes of the modules' M.2 slots... To no luck 😢
silopolis
08/02/2023, 6:40 PMOnly solutions I've found are converting back to plain PCIe slots :/
silopolis
08/02/2023, 6:41 PMsilopolis
08/02/2023, 6:43 PMBut then as I'm almost PCI illiterate, I'm not even sure that would work
t
Terarex (Dan Donovan)
08/02/2023, 6:43 PMI haven't gotten the Orin NX boot manager to see my 10Gb M.2 card. It's B+M so that might be the issue.
I need to install L4T 35.3.1 on a USB stick to free up the M.2 slot. Then see whether lspci actually sees the card. If it does, I need to figure out how to inject the driver into the Orin NX's boot manager.
s
silopolis
08/02/2023, 6:45 PMOK, hope that will doit... keep us posted
silopolis
08/02/2023, 6:47 PM@DhanOS (Daniel Kukiela) do you think the configurations with the hardware I just linked to could work with RK1 modules and their bifurcation support?
t
Terarex (Dan Donovan)
08/02/2023, 6:47 PMRight now I'm working on something else, but I'll get back to that research project in the next week.
d
DhanOS (Daniel Kukiela)
08/03/2023, 10:31 AMIn theory it should, but I can't be sure, of course
s
silopolis
08/03/2023, 10:33 AMOK, that's a start! 🙂
I'll see if I can get more clues from insiders...
silopolis
08/03/2023, 10:58 AMWell digging a bit in channels and forum, it looks like I'll hardly find s/o better informed 😅 #shame
t
Terarex (Dan Donovan)
01/09/2024, 9:34 PMI'm resurrecting this topic after trying the 10GB Ethernet M.2 adapter with an NVIDIA Jetson Orin NX 16GB module. The onboard NVIDIA boot manager doesn't see the card and I have no idea how to inject a device driver into the boot manager.
The card has a B+M connector and requires 2 PCIe lanes. Wondering whether RK1 will support PCIe lane bifurcation (2x2).
While this M.2 board was pretty expensive in early to mid 2023, an AliExpress vendor is now selling an AQC107-based M.2 board for under $80USD.
t
Teslamax
01/15/2024, 10:20 PMPardon the tangent, but there may be some relevance…
Teslamax
01/15/2024, 10:24 PMI’m considering migrating my routing/network functions from pfSense (Proxmox VM) to my TP2
- add 1 or 2 Ethernet interfaces of 1 Gb or faster (node 1 or 2 for mPCIe)
- potentially switch to OpenWRT
Teslamax
01/15/2024, 10:25 PMany suggestions?
Teslamax
01/15/2024, 10:28 PMA 10Gb adapter is probably overkill for my uses… I was thinking 2.5Gb for future growth.
t
Terarex (Dan Donovan)
01/16/2024, 4:09 AMAdmittedly, I haven't searched for 2.5Gb M.2 Ethernet adapters. A 10Gb adapter will operate at 2.5Gb.
t
tuttocorsivo
03/21/2024, 10:47 PMdoes the rk1 module exposes the combo pipe interface of the rk3588(either pcie2.1 /sata/USB3.0)? i'm planning a carrier board for the module and i would need the an extra pcie lane and 2 sata lane instead of the ethernet connection
t
Terarex (Dan Donovan)
03/21/2024, 11:34 PM@tuttocorsivo I have no idea. The design of the RK1 seems to be closely held. Those outside Turing Machines had to sign an NDA. If you want it to work with your carrier, you'll probably need to do the same, particularly if you need to modify u-boot for some reason. The pinmux has to be close to the NVIDIA Jetson TX2 or Jetson Xavier NX. Turing Machines needs to sell the module outside of a purely Turing Pi v2 audience to recoup its investment.
I tried the Innodisk M.2 (Marvell/AQuantia AQC107 chip) 10GbE adapter yesterday with an RK1 in my TPiv2.4's node1 slot. After running the u-boot
pci enumlspci9 Views