Tjyylli

I'm trying to understand a counter-intuitive phenomenon where I get only ~ 400 Mbit/s through a gigabit connection when the CPU is idle, but ~ 850 Mbit/s when the CPU is under full load (stress test utility). I suspect this is a symptom of some misconfiguration, in the end I would like to be able to saturate the link regardless of the CPU usage. The question is: what could cause the poor network performance, and why does it depend on the CPU usage in this strange way?

I'm glad for any hints, thanks for your time!

Systems and hardware

• Lynx SENYO Mini PC (GE5 PCF55) running up-to-date Arch Linux kernel version 5.0.8 (called the client for reference).


• 
  

  NIC is a Realtek RTL8168:

  
  
  

  # lspci -nn | grep Ethernet
  
  08:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 03)
  
  

  
  



• Connection: CAT 6 patch cable <-> CAT 7 in-wall cable <-> CAT 6 patch field <-> HPE OfficeConnect Switch <-> CAT 6 patch cable <-> supermicro home server (called the server for reference). Another PC gets ~ 950 Mbit/s on the exact same connection to the server with the exact same cables, so I guess this isolates the problem to the client.

Test setup

• 
  

  On the server:

  
  
  

  # iperf3 -D -s --bind 192.168.128.1
  
  

  
  


• 
  

  On the client while CPUs are idle:

  
  
  

  # iperf3 -c 192.168.128.1
  
  Connecting to host 192.168.128.1, port 5201
  
  [  5] local 192.168.128.100 port 60738 connected to 192.168.128.1 port 5201
  
  [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
  
  [  5]   0.00-1.00   sec  39.8 MBytes   333 Mbits/sec    0    310 KBytes       
  
  [  5]   1.00-2.00   sec  40.5 MBytes   339 Mbits/sec    0    322 KBytes       
  
  [  5]   2.00-3.00   sec  40.3 MBytes   338 Mbits/sec    0    322 KBytes       
  
  [  5]   3.00-4.00   sec  41.0 MBytes   344 Mbits/sec    0    322 KBytes       
  
  [  5]   4.00-5.00   sec  36.9 MBytes   310 Mbits/sec    0    322 KBytes       
  
  [  5]   5.00-6.00   sec  53.1 MBytes   445 Mbits/sec    0    322 KBytes       
  
  [  5]   6.00-7.00   sec  53.7 MBytes   450 Mbits/sec    0    322 KBytes       
  
  [  5]   7.00-8.00   sec  54.7 MBytes   459 Mbits/sec    0    338 KBytes       
  
  [  5]   8.00-9.00   sec  54.0 MBytes   453 Mbits/sec    0    338 KBytes       
  
  [  5]   9.00-10.00  sec  54.0 MBytes   453 Mbits/sec    0    338 KBytes       
  
  - - - - - - - - - - - - - - - - - - - - - - - - -
  
  [ ID] Interval           Transfer     Bitrate         Retr
  
  [  5]   0.00-10.00  sec   468 MBytes   393 Mbits/sec    0             sender
  
  [  5]   0.00-10.04  sec   467 MBytes   390 Mbits/sec                  receiver
  
  

  
  


• 
  

  On the client while running stress -c 4 (all four CPUs busy on 100%)

  
  
  

  # iperf3 -c 192.168.128.1
  
  Connecting to host 192.168.128.1, port 5201
  
  [  5] local 192.168.128.100 port 60742 connected to 192.168.128.1 port 5201
  
  [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
  
  [  5]   0.00-1.00   sec   102 MBytes   854 Mbits/sec    0    356 KBytes       
  
  [  5]   1.00-2.00   sec   101 MBytes   845 Mbits/sec    0    375 KBytes       
  
  [  5]   2.00-3.00   sec   101 MBytes   846 Mbits/sec    0    392 KBytes       
  
  [  5]   3.00-4.00   sec   101 MBytes   844 Mbits/sec    0    409 KBytes       
  
  [  5]   4.00-5.00   sec   100 MBytes   843 Mbits/sec    0    409 KBytes       
  
  [  5]   5.00-6.00   sec   101 MBytes   843 Mbits/sec    0    409 KBytes       
  
  [  5]   6.00-7.00   sec   101 MBytes   845 Mbits/sec    0    409 KBytes       
  
  [  5]   7.00-8.00   sec   100 MBytes   841 Mbits/sec    0    451 KBytes       
  
  [  5]   8.00-9.00   sec   101 MBytes   843 Mbits/sec    0    451 KBytes       
  
  [  5]   9.00-10.00  sec   101 MBytes   851 Mbits/sec    0    510 KBytes       
  
  - - - - - - - - - - - - - - - - - - - - - - - - -
  
  [ ID] Interval           Transfer     Bitrate         Retr
  
  [  5]   0.00-10.00  sec  1008 MBytes   846 Mbits/sec    0             sender
  
  [  5]   0.00-10.04  sec  1005 MBytes   840 Mbits/sec                  receiver
  
  

  
  

Unsuccessful attempts

• I thought maybe its a frequency scaling issue, but cpupower frequency-set -g performance makes no difference.


• Reading about issues with my specific NIC, I tried both drivers, the one present in the kernel (r8169) and the one from Realtek (r8168). The general picture is the same, the numbers are a bit different. With the Realtek driver I get 556 Mbits/s (idle CPU) vs 788 Mbits/s (busy CPU).


• Cross-checked a live Ubuntu 18.04 LTS (kernel 4.18.0) with the same results.


• Running irqbalance makes no difference.


• Disabling IPv6 makes no difference.

Additional information (glad to add more on request)

Output from ip

# ip addr show dev enp8s0

2: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000

    link/ether 00:14:0b:81:3f:8a brd ff:ff:ff:ff:ff:ff

    inet 192.168.128.100/24 brd 192.168.128.255 scope global noprefixroute enp8s0

       valid_lft forever preferred_lft forever

Output from ethtool:

# ethtool enp8s0

Settings for enp8s0:

        Supported ports: [ TP MII ]

        Supported link modes:   10baseT/Half 10baseT/Full 

                                100baseT/Half 100baseT/Full 

                                1000baseT/Half 1000baseT/Full 

        Supported pause frame use: Symmetric Receive-only

        Supports auto-negotiation: Yes

        Supported FEC modes: Not reported

        Advertised link modes:  10baseT/Half 10baseT/Full 

                                100baseT/Half 100baseT/Full 

                                1000baseT/Half 1000baseT/Full 

        Advertised pause frame use: Symmetric Receive-only

        Advertised auto-negotiation: Yes

        Advertised FEC modes: Not reported

        Link partner advertised link modes:  10baseT/Half 10baseT/Full 

                                             100baseT/Half 100baseT/Full 

                                             1000baseT/Full 

        Link partner advertised pause frame use: No

        Link partner advertised auto-negotiation: Yes

        Link partner advertised FEC modes: Not reported

        Speed: 1000Mb/s

        Duplex: Full

        Port: MII

        PHYAD: 0

        Transceiver: internal

        Auto-negotiation: on

        Supports Wake-on: pumbg

        Wake-on: g

        Current message level: 0x00000033 (51)

                               drv probe ifdown ifup

        Link detected: yes



# ethtool -i enp8s0

driver: r8169

version: 

firmware-version: rtl_nic/rtl8168d-1.fw

expansion-rom-version: 

bus-info: 0000:08:00.0

supports-statistics: yes

supports-test: no

supports-eeprom-access: no

supports-register-dump: yes

supports-priv-flags: no

Interrupts:

# cat /proc/interrupts 

           CPU0       CPU1       CPU2       CPU3       

...

 26:     145689          0          0     394154   PCI-MSI 4194304-edge      enp8s0

...



# cat /proc/softirqs 

                    CPU0       CPU1       CPU2       CPU3       

          HI:          1         42         46          1

       TIMER:      11550      32469      10688      11494

      NET_TX:      85221          4          7     189091

      NET_RX:     145692          0         14     394372

       BLOCK:         21        450         39      22765

    IRQ_POLL:          0          0          0          0

     TASKLET:     136092         93         97     384278

       SCHED:      10554      29735       8720       9726

     HRTIMER:          0          0          0          0

         RCU:       6154       8303       5838       6506