• chengcui
• newtest

test patch

VM environment with a Ethernet switch

single path testbed

Virtual machines (VMs) as TCP traffic senders using iperf3 are hosted by Bhyve in a physical box (Beelink SER5 AMD Mini PC). This box uses FreeBSD 14.2 release. Another box that is the same type uses Ubuntu Linux desktop version. The two boxes are connected through a 5-Port Gigabit Ethernet Switch (TP-Link TL-SG105). The switch has a shared 1Mb (125KB) Packet Buffer Memory.

testbed:

functionality show case

This test will show a general TCP congestion control window (cwnd) growth pattern before/after the patch. And also compare with the cwnd growth pattern from the Linux kernel.

• additional 40ms latency is added/emulated in the receiver (Ubuntu Linux with traffic shaping: The minimum bandwidth delay product (BDP) is 1000Mbps x 40ms == 5 Mbytes. So I configured sender/receiver with 10MB buffer)

cc@Linux:~ % sudo tc qdisc add dev enp1s0 root netem delay 40ms
cc@Linux:~ % tc qdisc show dev enp1s0
qdisc netem 8001: root refcnt 2 limit 1000 delay 40ms
cc@Linux:~ %

root@n1fbsd:~ # ping -c 4 -S n1fbsd Linux
PING Linux (192.168.50.46) from n1fbsdvm: 56 data bytes
64 bytes from 192.168.50.46: icmp_seq=0 ttl=64 time=41.053 ms
64 bytes from 192.168.50.46: icmp_seq=1 ttl=64 time=41.005 ms
64 bytes from 192.168.50.46: icmp_seq=2 ttl=64 time=41.011 ms
64 bytes from 192.168.50.46: icmp_seq=3 ttl=64 time=41.168 ms

--- Linux ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 41.005/41.059/41.168/0.066 ms
root@n1fbsd:~ #

root@n1ubuntu24:~ # ping -c 4 -I 192.168.50.161 Linux
PING Linux (192.168.50.46) from 192.168.50.161 : 56(84) bytes of data.
64 bytes from Linux (192.168.50.46): icmp_seq=1 ttl=64 time=40.9 ms
64 bytes from Linux (192.168.50.46): icmp_seq=2 ttl=64 time=40.9 ms
64 bytes from Linux (192.168.50.46): icmp_seq=3 ttl=64 time=41.0 ms
64 bytes from Linux (192.168.50.46): icmp_seq=4 ttl=64 time=41.1 ms

--- Linux ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3061ms
rtt min/avg/max/mdev = 40.905/40.981/41.108/0.078 ms
root@n1ubuntu24:~ #

root@n1fbsd:~ # sysctl -f /etc/sysctl.conf
net.inet.tcp.hostcache.enable: 0 -> 0
kern.ipc.maxsockbuf: 10485760 -> 10485760
net.inet.tcp.sendbuf_max: 10485760 -> 10485760
net.inet.tcp.recvbuf_max: 10485760 -> 10485760
root@n1fbsd:~ #

root@n1ubuntu24:~ # sysctl -p
net.core.rmem_max = 10485760
net.core.wmem_max = 10485760
net.ipv4.tcp_rmem = 4096 131072 10485760
net.ipv4.tcp_wmem = 4096 16384 10485760
net.ipv4.tcp_no_metrics_save = 1
root@n1ubuntu24:~ #

cc@Linux:~ % sudo sysctl -p
net.core.rmem_max = 10485760
net.core.wmem_max = 10485760
net.ipv4.tcp_rmem = 4096 131072 10485760
net.ipv4.tcp_wmem = 4096 16384 10485760
net.ipv4.tcp_no_metrics_save = 1
cc@Linux:~ %

• info for switching to the FreeBSD RACK TCP stack

root@n1fbsd:~ # kldstat
Id Refs Address                Size Name
 1    5 0xffffffff80200000  1f75ca0 kernel
 2    1 0xffffffff82810000    368d8 tcp_rack.ko
 3    1 0xffffffff82847000     f0f0 tcphpts.ko
root@n1fbsd:~ # sysctl net.inet.tcp.functions_default=rack
net.inet.tcp.functions_default: freebsd -> rack
root@n1fbsd:~ #

• iperf3 command

receiver: iperf3 -s -p 5201 --affinity 1
sender:   iperf3 -B ${src} --cport ${tcp_port} -c ${dst} -p 5201 -l 1M -t 300 -i 1 -f m -VC ${name}

CUBIC growth pattern before/after patch in FreeBSD default stack:

CUBIC growth pattern before/after patch in FreeBSD RACK stack:

For reference, the CUBIC in Linux kernel growth pattern is also attached:

single flow 300 seconds with 1Gbps x 40ms BDP performance

CUBIC cwnd and throughput before patch in FreeBSD default stack:

CUBIC cwnd and throughput after patch in FreeBSD default stack:

CUBIC cwnd and throughput before patch in FreeBSD RACK stack:

CUBIC cwnd and throughput after patch in FreeBSD RACK stack:

CUBIC cwnd and throughput in Linux stack:

loss resilience in LAN test (no additional latency added)

Similar to testD46046, this configure tests TCP CUBIC in the VM traffic sender, with 1%, 2%, 3% or 4% incoming packet drop rate at the Linux receiver. No additional latency is added.

root@n1fbsd:~ # ping -c 4 -S n1fbsd Linux
PING Linux (192.168.50.46) from n1fbsdvm: 56 data bytes
64 bytes from 192.168.50.46: icmp_seq=0 ttl=64 time=0.499 ms
64 bytes from 192.168.50.46: icmp_seq=1 ttl=64 time=0.797 ms
64 bytes from 192.168.50.46: icmp_seq=2 ttl=64 time=0.860 ms
64 bytes from 192.168.50.46: icmp_seq=3 ttl=64 time=0.871 ms

--- Linux ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.499/0.757/0.871/0.151 ms
root@n1fbsd:~ #

root@n1ubuntu24:~ # ping -c 4 -I 192.168.50.161 Linux
PING Linux (192.168.50.46) from 192.168.50.161 : 56(84) bytes of data.
64 bytes from Linux (192.168.50.46): icmp_seq=1 ttl=64 time=0.528 ms
64 bytes from Linux (192.168.50.46): icmp_seq=2 ttl=64 time=0.583 ms
64 bytes from Linux (192.168.50.46): icmp_seq=3 ttl=64 time=0.750 ms
64 bytes from Linux (192.168.50.46): icmp_seq=4 ttl=64 time=0.590 ms

--- Linux ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3165ms
rtt min/avg/max/mdev = 0.528/0.612/0.750/0.082 ms
root@n1ubuntu24:~ #

## list of different packet loss rate
cc@Linux:~ % sudo iptables -A INPUT -p tcp --dport 5201 -m statistic --mode nth --every 100 --packet 0 -j DROP
cc@Linux:~ % sudo iptables -A INPUT -p tcp --dport 5201 -m statistic --mode nth --every 50 --packet 0 -j DROP
cc@Linux:~ % sudo iptables -A INPUT -p tcp --dport 5201 -m statistic --mode nth --every 33 --packet 0 -j DROP
cc@Linux:~ % sudo iptables -A INPUT -p tcp --dport 5201 -m statistic --mode nth --every 25 --packet 0 -j DROP
...

• iperf3 command

cc@Linux:~ % iperf3 -s -p 5201 --affinity 1
root@n1fbsd:~ # iperf3 -B n1fbsd --cport 54321 -c Linux -p 5201 -l 1M -t 30 -i 1 -f m -VC cubic

• iperf3 single flow 30s average throughput under different packet loss rate at the Linux receiver

tri-point topology testbed

Now there are two virtual machines (VMs) as traffic senders at the same time. They are hosted by Bhyve in two separate physical boxs (Beelink SER5 AMD Mini PCs). The traffic receiver is the same Linux box as before, for the simplicity of a tri-point topology.

The three physical boxes are connected through a 5 Port Gigabit Ethernet Switch (TP-Link TL-SG105). The switch has a shared 1Mb (0.125MB) Packet Buffer Memory, which is just 2.5% of the 5 Mbytes BDP (1000Mbps x 40ms).

testbed:

• additional 40ms latency is added/emulated in the receiver and senders/receiver have 2.5MB buffer (50% BDP)
• iperf3 commands

iperf3 -s -p 5201
iperf3 -B ${src} --cport ${tcp_port} -c ${dst} -p 5201 -l 1M -t 200 -i 1 -f m -VC ${name}
iperf3 -s -p 5202
iperf3 -B ${src} --cport ${tcp_port} -c ${dst} -p 5202 -l 1M -t 200 -i 1 -f m -VC ${name}

link utilization under two competing TCP flows

• the link utilization of two competing TCP flows starting at the same time from each VM