https://turingpi.com logo
#│forum
10Gb M.2 Ethernet
# │forum
t

terarex

05/25/2023, 6:23 PM
I've ordered one of these cards to test compatibility on the TPiv2: https://www.innodisk.com/en/products/embedded-peripheral/communication/egpl-t101. These aren't cheap, but if a Turing Machines customer has backend storage, high-speed Ethernet connectivity would be a better option than hosting storage locally. The TPiV2's onboard 1Gb Ethernet switch is sufficient for control plane functionality, but the data plane, particularly if it is cross-node, requires higher throughput. This isn't RDMAoF, but it gets closer. While this M.2 card will not work with the CM4 due to the way the PCIe lane is routed on the CM4 adapter board, it is very likely to work perfectly with RK1 and NVIDIA Jetson Orin SoMs. Haven't been able to determine which Marvel ACQ chip is being used as there are quite a few. The driver may or may not be integrated in upstream 3.10+ kernels, but source is provided and support is claimed for both x86 and ARM architectures. I'll report back here once I have the M.2 card and can test with Orin NX 16GB. Waiting patiently for Turing Machines to make the RK1 available for purchase ...
Sorry, the URL seems to have gotten mangled: www.innodisk.com/en/products/embedded-peripheral/communication/egpl-t101
u

_dhanos_

05/25/2023, 6:38 PM
Aren't you using M.2 SSD for the Orin NX?
t

terarex

05/26/2023, 5:32 AM
I am right now, but an alternative that uses a centralized storage subsystem might be interesting. Orin does support PXE and HTTP booting. iSCSI is also included in the boot manager.
u

_dhanos_

05/26/2023, 1:25 PM
I've also heard that someone managed to boot off of USB in Node , but I haven't tried that myself yet
t

terarex

05/26/2023, 2:44 PM
Using a USB SSD will probably be a part of the test process. The M.2 device has UEFI support, so there is a good chance the Orin bootloader will see it. The process for creating a network boot image with L4T using iSCSI will be explored.
Confirmed that this M.2 (B+M connector) card incorporates the Marvell AQC113. Technically, this chip supports PCIe Gen 4.0/3.X/2.0 x4. Innodisk's use of a B+M connector limits the chip to 2 lanes. The card can take full advantage of 10Gb Ethernet with PCIe Gen 3.X x2. PCIe Gen 2.0 does not have sufficient throughput to take full advantage of 10Gb. I do not see this as an issue since both NVIDIA Orin, including Orin Nano, and RK1 support PCIe Gen 3 x4 on the M.2 socket.
u

_dhanos_

05/27/2023, 11:21 PM
Orin NX actually supports PCIe Gen 4.0
t

terarex

05/27/2023, 11:26 PM
Yeah. Technically, the chip supports Gen 4, but I question whether there is much to be gained. I did several searches for a dual M.2 adapter in a U.2 form factor. There don't seem to be any that support bifurcation. That would support a 2-lane NVMe SSD and the 2-lane 10Gb NIC.
u

_dhanos_

05/27/2023, 11:28 PM
Bifurcation is a task of the controller, not adapter, so the module itself would need to support it
I believe the Orin modules do, but I may be wrong
So, if the module supports bifurcation, you'd need a way to split the PCIe lanes, which can be a challenge for such configuration 🙂
It was actually RK1, and more specifically RK358 where I have seen bifurcation
Copy code
Support aggregation and bifurcation with 1x 4lanes, 2x 2lanes, 4x 1lanes and 1x
2lanes + 2x 1lanes
Orin modules do not support bifurcation on this bus. The PCIe options are:
Which means only the x2 bus can be bifurcated
I would love to link the document, but the way Nvidia handles links, it'll start working shortly
https://developer.nvidia.com/embedded/downloads#?search=design&tx=$product,jetson_orin_nx,jetson_orin_nano ->
Jetson Orin NX Series and Orin Nano Series Design Guide
-> chapter2, page 4
s

silopolis

08/02/2023, 6:34 PM
hello guys 🙂 Looks like we're hunting for the same 5 leg sheep !
@terarex spent hours looking for dual M.2 adapter that could be connected one way or the other (Oculink, SFF-xxxx) to the 4 lanes of the modules' M.2 slots... To no luck 😢
Only solutions I've found are converting back to plain PCIe slots :/
But then as I'm almost PCI illiterate, I'm not even sure that would work
t

terarex

08/02/2023, 6:43 PM
I haven't gotten the Orin NX boot manager to see my 10Gb M.2 card. It's B+M so that might be the issue. I need to install L4T 35.3.1 on a USB stick to free up the M.2 slot. Then see whether lspci actually sees the card. If it does, I need to figure out how to inject the driver into the Orin NX's boot manager.
s

silopolis

08/02/2023, 6:45 PM
OK, hope that will doit... keep us posted
@_dhanos_ do you think the configurations with the hardware I just linked to could work with RK1 modules and their bifurcation support?
t

terarex

08/02/2023, 6:47 PM
Right now I'm working on something else, but I'll get back to that research project in the next week.
u

_dhanos_

08/03/2023, 10:31 AM
In theory it should, but I can't be sure, of course
s

silopolis

08/03/2023, 10:33 AM
OK, that's a start! 🙂 I'll see if I can get more clues from insiders...
Well digging a bit in channels and forum, it looks like I'll hardly find s/o better informed 😅 #shame
t

terarex

01/09/2024, 9:34 PM
I'm resurrecting this topic after trying the 10GB Ethernet M.2 adapter with an NVIDIA Jetson Orin NX 16GB module. The onboard NVIDIA boot manager doesn't see the card and I have no idea how to inject a device driver into the boot manager. The card has a B+M connector and requires 2 PCIe lanes. Wondering whether RK1 will support PCIe lane bifurcation (2x2). While this M.2 board was pretty expensive in early to mid 2023, an AliExpress vendor is now selling an AQC107-based M.2 board for under $80USD.
t

teslamax

01/15/2024, 10:20 PM
Pardon the tangent, but there may be some relevance…
I’m considering migrating my routing/network functions from pfSense (Proxmox VM) to my TP2 - add 1 or 2 Ethernet interfaces of 1 Gb or faster (node 1 or 2 for mPCIe) - potentially switch to OpenWRT
any suggestions?
A 10Gb adapter is probably overkill for my uses… I was thinking 2.5Gb for future growth.
t

terarex

01/16/2024, 4:09 AM
Admittedly, I haven't searched for 2.5Gb M.2 Ethernet adapters. A 10Gb adapter will operate at 2.5Gb.
t

tuttocorsivo

03/21/2024, 10:47 PM
does the rk1 module exposes the combo pipe interface of the rk3588(either pcie2.1 /sata/USB3.0)? i'm planning a carrier board for the module and i would need the an extra pcie lane and 2 sata lane instead of the ethernet connection
t

terarex

03/21/2024, 11:34 PM
@tuttocorsivo I have no idea. The design of the RK1 seems to be closely held. Those outside Turing Machines had to sign an NDA. If you want it to work with your carrier, you'll probably need to do the same, particularly if you need to modify u-boot for some reason. The pinmux has to be close to the NVIDIA Jetson TX2 or Jetson Xavier NX. Turing Machines needs to sell the module outside of a purely Turing Pi v2 audience to recoup its investment. I tried the Innodisk M.2 (Marvell/AQuantia AQC107 chip) 10GbE adapter yesterday with an RK1 in my TPiv2.4's node1 slot. After running the u-boot
pci enum
command, u-boot could see the card. with `pci *`(see attached screen shot). Ubuntu's
lspci
command couldn't see the card. Might be a device tree issue. I'm not certain. Regardless, the atlantic.ko device driver is not present, so may need to be built from source. There is an issue with it that I'll debug at some point. Probably due to the driver being targeted for Red Hat Linux. I've ordered the IO Crest version of this card from AliExpress to see whether it functions the same. I'm going to try an RK1 on both a Waveshare and a Seeed Studio Jetson Nano carrier board just for "fun". 😜 https://cdn.discordapp.com/attachments/1111358860679192587/1220515981194166342/u-boot_pci-M.2_10Gb_Ethernet_adapter.jpg?ex=660f3916&is=65fcc416&hm=41c949d357159528aa6a03d66082ae95dab24790d7e093dc61ae201da2a6b3b0&
5 Views