Contents
tests in LAN
Linux VM receiver
One data sender and one data receiver and both are Virtual Machines (VMs).
Each VM is hosted by Bhyve in a separate physical box (Beelink SER5 AMD Mini PC) that is using FreeBSD 14.1 release OS.
The physical boxes (Beelink SER5 AMD Mini PCs) are connected through a 1Gbps hub, which is connected to a 1Gbps router.
test config
sender kernel info |
FreeBSD 15.0-CURRENT (GENERIC) #135 main-n270731-62cb671705eb-dirty |
|
receiver kernel info |
Linux n2linuxvm 5.15.0-101-generic #111-Ubuntu SMP |
|
root@n1fbsd:~ # ifconfig vtnet0
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 58:9c:fc:0e:8c:fb
inet 192.168.50.37 netmask 0xffffff00 broadcast 192.168.50.255
inet6 fe80::5a9c:fcff:fe0e:8cfb%vtnet0 prefixlen 64 scopeid 0x1
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
root@n1fbsd:~ # ping -c 5 -S 192.168.50.37 192.168.50.89
PING 192.168.50.89 (192.168.50.89) from 192.168.50.37: 56 data bytes
64 bytes from 192.168.50.89: icmp_seq=0 ttl=64 time=0.485 ms
64 bytes from 192.168.50.89: icmp_seq=1 ttl=64 time=0.649 ms
64 bytes from 192.168.50.89: icmp_seq=2 ttl=64 time=0.778 ms
64 bytes from 192.168.50.89: icmp_seq=3 ttl=64 time=0.711 ms
64 bytes from 192.168.50.89: icmp_seq=4 ttl=64 time=0.799 ms
--- 192.168.50.89 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.485/0.685/0.799/0.113 msroot@n2linuxvm:~ # ip addr show enp0s5
2: enp0s5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 58:9c:fc:0d:d6:4e brd ff:ff:ff:ff:ff:ff
inet 192.168.50.89/24 metric 100 brd 192.168.50.255 scope global dynamic enp0s5
valid_lft 84995sec preferred_lft 84995sec
inet6 fe80::5a9c:fcff:fe0d:d64e/64 scope link
valid_lft forever preferred_lft forever- iperf command
iperf -B 192.168.50.37 -c 192.168.50.89 -l 1M -t 10 -i 1 -e
test result
- highlighted performance stats
version |
throughput |
congestions count |
data pkt retransmission |
RTO count |
||||
base |
940 Mbits/sec |
3 |
415 data packets (599508 bytes) retransmitted |
0 |
||||
patch |
940 Mbits/sec |
1 |
380 data packets (548828 bytes) retransmitted |
0 |
||||
newreno |
940 Mbits/sec |
4 |
264 data packets (382272 bytes) retransmitted |
0 |
||||
- base
------------------------------------------------------------ Client connecting to 192.168.50.89, TCP port 5001 with pid 1244 (1 flows) Write buffer size: 1048576 Byte TOS set to 0x0 (Nagle on) TCP window size: 32.0 KByte (default) ------------------------------------------------------------ [ 1] local 192.168.50.37%vtnet0 port 64036 connected with 192.168.50.89 port 5001 (sock=3) on 2024-06-21 16:00:09.453 (EDT) [ ID] Interval Transfer Bandwidth Write/Err [ 1] 0.00-1.00 sec 113 MBytes 948 Mbits/sec 113/0 [ 1] 1.00-2.00 sec 112 MBytes 940 Mbits/sec 112/0 [ 1] 2.00-3.00 sec 113 MBytes 948 Mbits/sec 113/0 [ 1] 3.00-4.00 sec 112 MBytes 940 Mbits/sec 112/0 [ 1] 4.00-5.00 sec 112 MBytes 940 Mbits/sec 112/0 [ 1] 5.00-6.00 sec 112 MBytes 940 Mbits/sec 112/0 [ 1] 6.00-7.00 sec 113 MBytes 948 Mbits/sec 113/0 [ 1] 7.00-8.00 sec 112 MBytes 940 Mbits/sec 112/0 [ 1] 8.00-9.00 sec 112 MBytes 940 Mbits/sec 112/0 [ 1] 9.00-10.00 sec 112 MBytes 940 Mbits/sec 112/0 [ 1] 0.00-10.03 sec 1.10 GBytes 940 Mbits/sec 1124/0
- after patch
------------------------------------------------------------ Client connecting to 192.168.50.89, TCP port 5001 with pid 1178 (1 flows) Write buffer size: 1048576 Byte TOS set to 0x0 (Nagle on) TCP window size: 32.0 KByte (default) ------------------------------------------------------------ [ 1] local 192.168.50.37%vtnet0 port 58453 connected with 192.168.50.89 port 5001 (sock=3) on 2024-06-21 12:47:14.333 (EDT) [ ID] Interval Transfer Bandwidth Write/Err [ 1] 0.00-1.00 sec 113 MBytes 948 Mbits/sec 113/0 [ 1] 1.00-2.00 sec 113 MBytes 948 Mbits/sec 113/0 [ 1] 2.00-3.00 sec 112 MBytes 940 Mbits/sec 112/0 [ 1] 3.00-4.00 sec 112 MBytes 940 Mbits/sec 112/0 [ 1] 4.00-5.00 sec 111 MBytes 931 Mbits/sec 111/0 [ 1] 5.00-6.00 sec 112 MBytes 940 Mbits/sec 112/0 [ 1] 6.00-7.00 sec 113 MBytes 948 Mbits/sec 113/0 [ 1] 7.00-8.00 sec 112 MBytes 940 Mbits/sec 112/0 [ 1] 8.00-9.00 sec 112 MBytes 940 Mbits/sec 112/0 [ 1] 9.00-10.00 sec 112 MBytes 940 Mbits/sec 112/0 [ 1] 0.00-10.02 sec 1.10 GBytes 940 Mbits/sec 1123/0
- also check newreno
------------------------------------------------------------ Client connecting to 192.168.50.89, TCP port 5001 with pid 1208 (1 flows) Write buffer size: 1048576 Byte TOS set to 0x0 (Nagle on) TCP window size: 32.0 KByte (default) ------------------------------------------------------------ [ 1] local 192.168.50.37%vtnet0 port 64082 connected with 192.168.50.89 port 5001 (sock=3) on 2024-06-21 15:57:54.557 (EDT) [ ID] Interval Transfer Bandwidth Write/Err [ 1] 0.00-1.00 sec 113 MBytes 948 Mbits/sec 113/0 [ 1] 1.00-2.00 sec 113 MBytes 948 Mbits/sec 113/0 [ 1] 2.00-3.00 sec 112 MBytes 940 Mbits/sec 112/0 [ 1] 3.00-4.00 sec 112 MBytes 940 Mbits/sec 112/0 [ 1] 4.00-5.00 sec 112 MBytes 940 Mbits/sec 112/0 [ 1] 5.00-6.00 sec 113 MBytes 948 Mbits/sec 113/0 [ 1] 6.00-7.00 sec 112 MBytes 940 Mbits/sec 112/0 [ 1] 7.00-8.00 sec 112 MBytes 940 Mbits/sec 112/0 [ 1] 8.00-9.00 sec 111 MBytes 931 Mbits/sec 111/0 [ 1] 9.00-10.00 sec 113 MBytes 948 Mbits/sec 113/0 [ 1] 0.00-10.03 sec 1.10 GBytes 940 Mbits/sec 1124/0
1% drop rate test in Linux VM receiver
test config
I added 1% packet drop rate in the Linux VM, so this test uses the same test method in testD40185.
sender kernel info |
FreeBSD 15.0-CURRENT #0 main-0667538b88-dirty: Thu Jun 20 08:45:01 MDT 2024 |
|
receiver kernel info |
Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-112-generic x86_64) Thu May 23 07:48:21 UTC 2024 |
|
root@n1fbsd:~ # ifconfig vtnet0
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 58:9c:fc:0e:8c:fb
inet 192.168.50.37 netmask 0xffffff00 broadcast 192.168.50.255
inet6 fe80::5a9c:fcff:fe0e:8cfb%vtnet0 prefixlen 64 scopeid 0x1
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
root@n1fbsd:~ # ping -c 5 -S 192.168.50.37 192.168.50.89
PING 192.168.50.89 (192.168.50.89) from 192.168.50.37: 56 data bytes
64 bytes from 192.168.50.89: icmp_seq=0 ttl=64 time=0.444 ms
64 bytes from 192.168.50.89: icmp_seq=1 ttl=64 time=0.804 ms
64 bytes from 192.168.50.89: icmp_seq=2 ttl=64 time=0.692 ms
64 bytes from 192.168.50.89: icmp_seq=3 ttl=64 time=0.815 ms
64 bytes from 192.168.50.89: icmp_seq=4 ttl=64 time=0.689 ms
--- 192.168.50.89 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.444/0.689/0.815/0.133 msroot@n2linuxvm:~ # ip addr show enp0s5
2: enp0s5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 58:9c:fc:0d:d6:4e brd ff:ff:ff:ff:ff:ff
inet 192.168.50.89/24 metric 100 brd 192.168.50.255 scope global dynamic enp0s5
valid_lft 86122sec preferred_lft 86122sec
inet6 fe80::5a9c:fcff:fe0d:d64e/64 scope link
valid_lft forever preferred_lft forever
root@n2linuxvm:~ # iptables -A INPUT -p tcp --dport 5001 -m statistic --mode nth --every 100 --packet 0 -j DROP
root@n2linuxvm:~ #- iperf command
iperf -B 192.168.50.37 -c 192.168.50.89 -l 1M -t 10 -i 1 -e
test result
- highlighted performance stats
version |
throughput |
congestions count |
data pkt retransmission |
RTO count |
||||
base |
193 Mbits/sec |
484 |
1895 data packets (2743960 bytes) retransmitted |
0 |
||||
patch |
23.8 Mbits/sec |
191 |
208 data packets (301184 bytes) retransmitted |
0 |
||||
newreno |
918 Mbits/sec |
1298 |
8222 data packets (11905456 bytes) retransmitted |
0 |
||||
- base
------------------------------------------------------------ Client connecting to 192.168.50.89, TCP port 5001 with pid 1006 (1 flows) Write buffer size: 1048576 Byte TOS set to 0x0 (Nagle on) TCP window size: 32.0 KByte (default) ------------------------------------------------------------ [ 1] local 192.168.50.37%vtnet0 port 57874 connected with 192.168.50.89 port 5001 (sock=3) on 2024-06-21 11:59:58.366 (EDT) [ ID] Interval Transfer Bandwidth Write/Err [ 1] 0.00-1.00 sec 27.0 MBytes 226 Mbits/sec 27/0 [ 1] 1.00-2.00 sec 21.0 MBytes 176 Mbits/sec 21/0 [ 1] 2.00-3.00 sec 24.0 MBytes 201 Mbits/sec 24/0 [ 1] 3.00-4.00 sec 23.0 MBytes 193 Mbits/sec 23/0 [ 1] 4.00-5.00 sec 22.0 MBytes 185 Mbits/sec 22/0 [ 1] 5.00-6.00 sec 24.0 MBytes 201 Mbits/sec 24/0 [ 1] 6.00-7.00 sec 23.0 MBytes 193 Mbits/sec 23/0 [ 1] 7.00-8.00 sec 22.0 MBytes 185 Mbits/sec 22/0 [ 1] 8.00-9.00 sec 23.0 MBytes 193 Mbits/sec 23/0 [ 1] 9.00-10.00 sec 22.0 MBytes 185 Mbits/sec 22/0 [ 1] 0.00-10.07 sec 232 MBytes 193 Mbits/sec 232/0
- after patch
------------------------------------------------------------ Client connecting to 192.168.50.89, TCP port 5001 with pid 1088 (1 flows) Write buffer size: 1048576 Byte TOS set to 0x0 (Nagle on) TCP window size: 32.0 KByte (default) ------------------------------------------------------------ [ 1] local 192.168.50.37%vtnet0 port 19749 connected with 192.168.50.89 port 5001 (sock=3) on 2024-06-21 12:23:03.058 (EDT) [ ID] Interval Transfer Bandwidth Write/Err [ 1] 0.00-1.00 sec 6.00 MBytes 50.3 Mbits/sec 6/0 [ 1] 1.00-2.00 sec 3.00 MBytes 25.2 Mbits/sec 3/0 [ 1] 2.00-3.00 sec 2.00 MBytes 16.8 Mbits/sec 2/0 [ 1] 3.00-4.00 sec 2.00 MBytes 16.8 Mbits/sec 2/0 [ 1] 4.00-5.00 sec 3.00 MBytes 25.2 Mbits/sec 3/0 [ 1] 5.00-6.00 sec 2.00 MBytes 16.8 Mbits/sec 2/0 [ 1] 6.00-7.00 sec 3.00 MBytes 25.2 Mbits/sec 3/0 [ 1] 7.00-8.00 sec 2.00 MBytes 16.8 Mbits/sec 2/0 [ 1] 8.00-9.00 sec 3.00 MBytes 25.2 Mbits/sec 3/0 [ 1] 9.00-10.00 sec 2.00 MBytes 16.8 Mbits/sec 2/0 [ 1] 0.00-10.24 sec 29.0 MBytes 23.8 Mbits/sec 29/0
- also check newreno
------------------------------------------------------------ Client connecting to 192.168.50.89, TCP port 5001 with pid 987 (1 flows) Write buffer size: 1048576 Byte TOS set to 0x0 (Nagle on) TCP window size: 32.0 KByte (default) ------------------------------------------------------------ [ 1] local 192.168.50.37%vtnet0 port 19490 connected with 192.168.50.89 port 5001 (sock=3) on 2024-06-21 11:58:49.535 (EDT) [ ID] Interval Transfer Bandwidth Write/Err [ 1] 0.00-1.00 sec 109 MBytes 914 Mbits/sec 109/0 [ 1] 1.00-2.00 sec 110 MBytes 923 Mbits/sec 110/0 [ 1] 2.00-3.00 sec 109 MBytes 914 Mbits/sec 109/0 [ 1] 3.00-4.00 sec 110 MBytes 923 Mbits/sec 110/0 [ 1] 4.00-5.00 sec 109 MBytes 914 Mbits/sec 109/0 [ 1] 5.00-6.00 sec 110 MBytes 923 Mbits/sec 110/0 [ 1] 6.00-7.00 sec 108 MBytes 906 Mbits/sec 108/0 [ 1] 7.00-8.00 sec 110 MBytes 923 Mbits/sec 110/0 [ 1] 8.00-9.00 sec 109 MBytes 914 Mbits/sec 109/0 [ 1] 9.00-10.00 sec 110 MBytes 923 Mbits/sec 110/0 [ 1] 0.00-10.01 sec 1.07 GBytes 918 Mbits/sec 1095/0
By using siftr data, some charts based on TCP metrics are shown here.
TCP CUBIC cwnd in base: 
TCP CUBIC cwnd in patch: 
TCP newreno cwnd: 
TCP srtt in base: 
TCP srtt in patch: 
TCP newreno srtt: 
1% drop rate test in Emulab Emulation over 1Gbps
test config
On a 1Gbps link between a FreeBSD-15-current node and a Ubuntu22.04 (Linux) node, this test uses the same test method in testD40185.
FreeBSD node is the traffic sender (s1) using iperf, and the Linux node (r1) is the traffic receiver.
sender kernel info |
FreeBSD 15.0-CURRENT #0 main-0667538b88-dirty: Thu Jun 20 08:45:01 MDT 2024 |
|
receiver kernel info |
Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-112-generic x86_64) Thu May 23 07:48:21 UTC 2024 |
|
cc@s1:~ % ifconfig bce3
bce3: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 00:24:e8:79:33:c8
inet 10.1.2.2 netmask 0xffffff00 broadcast 10.1.2.255
media: Ethernet 1000baseT <full-duplex>
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
cc@s1:~ % ping -c 5 r1
PING r1-link1 (10.1.2.3): 56 data bytes
64 bytes from 10.1.2.3: icmp_seq=0 ttl=64 time=0.172 ms
64 bytes from 10.1.2.3: icmp_seq=1 ttl=64 time=0.156 ms
64 bytes from 10.1.2.3: icmp_seq=2 ttl=64 time=0.163 ms
64 bytes from 10.1.2.3: icmp_seq=3 ttl=64 time=0.159 ms
64 bytes from 10.1.2.3: icmp_seq=4 ttl=64 time=0.153 ms
--- r1-link1 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.153/0.161/0.172/0.007 mscc@r1:~ % ip addr show eno4
5: eno4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:24:e8:79:19:e6 brd ff:ff:ff:ff:ff:ff
altname enp2s0f1
inet 10.1.2.3/24 brd 10.1.2.255 scope global eno4
valid_lft forever preferred_lft forever
inet6 fe80::224:e8ff:fe79:19e6/64 scope link
valid_lft forever preferred_lft forever
cc@r1:~ % sudo iptables -A INPUT -p tcp --dport 5001 -m statistic --mode nth --every 100 --packet 0 -j DROP
cc@r1:~ %- iperf command
iperf -B 10.1.2.2 -c 10.1.2.3 -l 1M -t 10 -i 1 -e
test result
- highlighted performance stats
version |
throughput |
|
base |
712 Mbits/sec |
|
patch |
180 Mbits/sec |
|
newreno |
783 Mbits/sec |
|
- base
------------------------------------------------------------ Client connecting to 10.1.2.3, TCP port 5001 with pid 1910 (1/0 flows/load) Write buffer size: 1048576 Byte TCP congestion control using cubic TOS defaults to 0x0 (dscp=0,ecn=0) (Nagle on) TCP window size: 32.8 KByte (default) ------------------------------------------------------------ [ 1] local 10.1.2.2%bce3 port 19899 connected with 10.1.2.3 port 5001 (sock=3) on 2024-06-20 09:18:48.877 (MDT) [ ID] Interval Transfer Bandwidth Write/Err [ 1] 0.00-1.00 sec 86.0 MBytes 721 Mbits/sec 86/0 [ 1] 1.00-2.00 sec 84.0 MBytes 705 Mbits/sec 84/0 [ 1] 2.00-3.00 sec 81.0 MBytes 679 Mbits/sec 81/0 [ 1] 3.00-4.00 sec 81.0 MBytes 679 Mbits/sec 81/0 [ 1] 4.00-5.00 sec 83.0 MBytes 696 Mbits/sec 83/0 [ 1] 5.00-6.00 sec 83.0 MBytes 696 Mbits/sec 83/0 [ 1] 6.00-7.00 sec 87.0 MBytes 730 Mbits/sec 87/0 [ 1] 7.00-8.00 sec 88.0 MBytes 738 Mbits/sec 88/0 [ 1] 8.00-9.00 sec 88.0 MBytes 738 Mbits/sec 88/0 [ 1] 9.00-10.00 sec 89.0 MBytes 747 Mbits/sec 89/0 [ 1] 0.00-10.02 sec 851 MBytes 712 Mbits/sec 851/0
- after patch
------------------------------------------------------------ Client connecting to 10.1.2.3, TCP port 5001 with pid 1721 (1/0 flows/load) Write buffer size: 1048576 Byte TCP congestion control using cubic TOS defaults to 0x0 (dscp=0,ecn=0) (Nagle on) TCP window size: 32.0 KByte (default) ------------------------------------------------------------ [ 1] local 10.1.2.2%bce3 port 53960 connected with 10.1.2.3 port 5001 (sock=3) on 2024-06-20 09:29:53.549 (MDT) [ ID] Interval Transfer Bandwidth Write/Err [ 1] 0.00-1.00 sec 15.0 MBytes 126 Mbits/sec 15/0 [ 1] 1.00-2.00 sec 22.0 MBytes 185 Mbits/sec 22/0 [ 1] 2.00-3.00 sec 22.0 MBytes 185 Mbits/sec 22/0 [ 1] 3.00-4.00 sec 22.0 MBytes 185 Mbits/sec 22/0 [ 1] 4.00-5.00 sec 22.0 MBytes 185 Mbits/sec 22/0 [ 1] 5.00-6.00 sec 22.0 MBytes 185 Mbits/sec 22/0 [ 1] 6.00-7.00 sec 22.0 MBytes 185 Mbits/sec 22/0 [ 1] 7.00-8.00 sec 22.0 MBytes 185 Mbits/sec 22/0 [ 1] 8.00-9.00 sec 22.0 MBytes 185 Mbits/sec 22/0 [ 1] 9.00-10.00 sec 23.0 MBytes 193 Mbits/sec 23/0 [ 1] 0.00-10.04 sec 215 MBytes 180 Mbits/sec 215/0
- also check newreno
------------------------------------------------------------ Client connecting to 10.1.2.3, TCP port 5001 with pid 1883 (1/0 flows/load) Write buffer size: 1048576 Byte TCP congestion control using newreno TOS defaults to 0x0 (dscp=0,ecn=0) (Nagle on) TCP window size: 32.8 KByte (default) ------------------------------------------------------------ [ 1] local 10.1.2.2%bce3 port 16278 connected with 10.1.2.3 port 5001 (sock=3) on 2024-06-20 09:13:48.960 (MDT) [ ID] Interval Transfer Bandwidth Write/Err [ 1] 0.00-1.00 sec 96.0 MBytes 805 Mbits/sec 96/0 [ 1] 1.00-2.00 sec 97.0 MBytes 814 Mbits/sec 97/0 [ 1] 2.00-3.00 sec 96.0 MBytes 805 Mbits/sec 96/0 [ 1] 3.00-4.00 sec 94.0 MBytes 789 Mbits/sec 94/0 [ 1] 4.00-5.00 sec 94.0 MBytes 789 Mbits/sec 94/0 [ 1] 5.00-6.00 sec 93.0 MBytes 780 Mbits/sec 93/0 [ 1] 6.00-7.00 sec 89.0 MBytes 747 Mbits/sec 89/0 [ 1] 7.00-8.00 sec 93.0 MBytes 780 Mbits/sec 93/0 [ 1] 8.00-9.00 sec 90.0 MBytes 755 Mbits/sec 90/0 [ 1] 9.00-10.00 sec 91.0 MBytes 763 Mbits/sec 91/0 [ 1] 0.00-10.01 sec 934 MBytes 783 Mbits/sec 934/0
1% drop rate test in Emulab Emulation over 10Gbps