Locked 10Gbps speed slow on windows 10

gspro2k · Apr 22, 2021

Hello, I am noticing slow speeds on my 10GbE computer with a Mellanox connectx3 card.
I see the link is 10Gbps FDX and when I test speeds locally to my truenas server i get about 3900mbps down and 5600 mbps upload.
Both machines are using the Mellanox connectx3 card. The have fans to prevent overheating/throttling.
FW on them is the latest Mellanox has also

Client PC Specs:
Intel Core i5 9500 @ 3.0Ghz
32GB DDR4 2400Mhz
256GB WD NVMe sn520

Server Specs:
Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz
223.9GiB total available (ECC)
512GB Inland NVMe TLC

Does anyone know of any tweaks to get the speeds closer to the line speed.

I have done a bunch of googling rss auto tuning and window size and others but I don't see a script that optimizes this.
In the past i have see windows 10 builds on there saying they have network optimizations.

Just looking to get this buttoned up before I start using this for file transfers at home.

Here is an example of the speed testing

https://ibb.co/BNN54tG

Cyler · Apr 22, 2021

Can we have the specs of your PC? Especially the hard disk(s)?

gspro2k · Apr 22, 2021

Thank you for responding

I have updated my first post with both machine specs

Please let me know if you want to know anything else about the machine

Light_Eater · Apr 22, 2021

Hello @gspro2k,

Are you using PowerLine adapters at any point in your network setup?

gspro2k · Apr 22, 2021

No,
Its setup like this:
Server 10Gbps SFP+ <> Unifi 10GBps switch <> Client 10Gbps SFP+

Server: Mellanox ConnectX-3 https://www.mellanox.com/products/ethernet-adapters/connectx-3-pro to 10Gbps SFP+ https://www.amazon.com/10G-SFP-DAC-Cable-SFP-H10GB-CU2M/dp/B00U8BL09Q
Switch: Unifi aggregation switch https://store.ui.com/collections/unifi-network-switching/products/unifi-switch-aggregation
Client: Mellanox ConnectX-3 https://www.mellanox.com/products/ethernet-adapters/connectx-3-pro to 10Gbps SFP+ https://www.amazon.com/10G-SFP-DAC-Cable-SFP-H10GB-CU2M/dp/B00U8BL09Q

Cyler · Apr 22, 2021

As a first approach to the problem, I'm afraid I have bad news for you. It looks to me like you already hit not your network limits but your disk/system limits. Let me explain.

Tho a lot of disks advertise crazy speeds, those speeds are also... kinda false. They show the speed from disk to cache and not to the actual system. When copying files speed may vary by A LOT and get affected by a lot of reasons. For example, it's different when you copy 1000 files of 1MB and different when you copy 1 file of 1000 MBs. Tho both are 1 GB technically, The OS needs to do 1000 additional read/writes to the system NTFS tables (or extf4, whatever your NAS is using), one for each file you send, and only 1 write in the case of the large file. When this happens over a network, Networks can send 1 file at a time (Thank you TCP/IP) and initiate a handshake of a sort for each file they send. So when you send 1 file, there is 1 socket that gets initiated. When you send 1000 files this needs to happen 1000 times. This is where an app will use multiplexing but that is an entirely different story and needs dedicated clients for transfer.

Now more specifically to your issue, your SSD disk, based on the image you provided, pushes (reads) close to 750 Mbyte per second (6000/8 = 750) and writes 500 Mb per second. which as good as you will get and that is part of why you don't see greater speeds. The disk cant send/receive more data to the NIC.

Please keep in mind that when you use ANY network card, the CPU is most often part of the problem and bottleneck as the cards rarely have their own hardware acceleration. That's why you can see in high-speed NAS to have multicore dedicated CPUs as the CPU is responsible for Read/Write to SSD as well as read/write to NICs concurrently and... even use multiple links. There are caching algorithms that can speed this process up, 1 core reads while the other sends, etc, but this kind of tuning needs to happen for your specific system, and even then I doubt you will see much improvement. The only way to push more data is to use RAID as then you will have multiple Disk reading to one NIC but also that will mean higher CPU usage fr the system and less for you.

Just to put things in perspective
10 GBit = 1250 MByte per second which is the theoretical max. You already can achieve more than half of that 6Gbit out of 10 Gbit or 750 MB out of 1250MB/s.

Light_Eater · Apr 22, 2021

WD claims reads up to 1700MB/s for 256GB WD NVMe sn520, the Inland NVMe TLC can write 1500MB/s but as @Cyler said they are quite often impossible to reach.
I have samsung 980 pro with advertised speeds of up to 7000MB/s and seen this speeds only in software provided by Samsung.

Maybe we should consider the overhead of the used protocol for the transfer as well here?

Cyler · Apr 22, 2021

Light_Eater said:
WD claims reads up to 1700MB/s for 256GB WD NVMe sn520, the Inland NVMe TLC can write 1500MB/s but as @Cyler said they are quite often impossible to reach.
I have samsung 980 pro with advertised speeds of up to 7000MB/s and seen this speeds only in software provided by Samsung.

Maybe we should consider the overhead of the used protocol for the transfer as well here?

It only happens under control conditions and... some if not all of the following:
* Only 1 device connected so they don't saturate the bus/controller.
* The "copy" is from disk to ram.
* Testing App priority is set to RealTime mode
* 1 huge file that is pre-created on the hard disk and nothing else, not even OS, to minimize OS overhead.

You get the idea. That's why they always say "UP TO", which means peek speed. When you add more things to the test, you get to see the real picture. That's why I always measure in Real life simulated conditions and not in ideal ones.

Edit: Forgot to add, some apps test with compressed data... don't need to say more.

gspro2k · Apr 23, 2021

Ill run crystaldiskmark on the two machines and see what the SSDs are doing.

gspro2k · Apr 27, 2021

Hello, here is the speedtests of the SSD on the client machine

Client PC Specs:
Intel Core i5 9500 @ 3.0Ghz
32GB DDR4 2400Mhz
256GB WD NVMe sn520

You must be registered for see links

Should I get a better SSD?

sandyss18 · May 18, 2021

Cyler said:
It only happens under control conditions and... some if not all of the following:
* Only 1 device connected so they don't saturate the bus/controller.
* The "copy" is from disk to ram.
* Testing App priority is set to RealTime mode
* 1 huge file that is pre-created on the hard disk and nothing else, not even OS, to minimize OS overhead.

You get the idea. That's why they always say "UP TO", which means peek speed. When you add more things to the test, you get to see the real picture. That's why I always measure in Real life simulated conditions and not in ideal ones.

Edit: Forgot to add, some apps test with compressed data... don't need to say more.

You have hit the nail on the head.

Cyler · May 18, 2021

Somehow I missed your answer @gspro2k . It's not a matter of a new SSD but rather a faster SSD or even better a Raid of SSD but the cost will be at the CPU side as whenever you copy/move files it will ask for more CPU time too. To understand better, a network transfer can go almost as fast as the disk can read and write data.

Now if you check your random read/write speeds (which is closer to real-life performance) you can see that your performance is:

565 MByte is 4.5 Gbit @ Reading (when your PC sends to Server) and 452 Mbyte is 3.6 Gbit when writing (From server to PC) so as you understand that more or less will be your max speeds. It will get far worse when you send small files as you can see from the last line. 44MByte = 0.3 Gbit. This is what I explained in my 1st answer, that the limits are your system and not the network.

Solutions: sadly not many. You will need A NVME array to start and if you plan to transfer a lot of files, a higher core CPU (and even a bit more RAM for caching?). Keep in mind that 10 Gbit and up networks are designed for multiple access, for example, multiple PCs to transfer to/from the server at high speed and not just 1.

Additionally, you may need a Network expert, as when you want a high-speed network, you need a bit different setup for PC/ router/Server than your average lan, and as you understand this can't be done over a forum site. There are several settings to modify/test such as jumbo frames, LSO (Large Send Offload), Windows Auto-Tuning, various QoS settings, and even things such as Antivirus, etc need to be changed in order to not interfere with networks transfer. Those need to be set for your specific hardware.