(I'm copy/past lot's of tips from mine BSDRP website)

How to bench a router

On a router we measure the throughput (maximum packet-per-second with smallest packet size) and not the bandwitdh (Gb/s).

A good full router benchmark should do all RFC2544 (Benchmarking Methodology for Network Interconnect Devices) and RFC3222 (Terminology for Forwarding Information Base (FIB) based Router Performance) tests. But this kind of full-tests are very time consuming: Starting by a more simple bench using a wire-speed (or line-rate) packet-generator using smallest packet size is a good start.

A wire-speed packet generator should be able to generate the maximum number of smallest packet per second on the tested media, this mean about 14.88Mpps on a TenGigabit media. Benchmarking smallest packet size permit to measure the "worse" case.

But using Packet-per-second (pps) as unique number isn't common for standard users that are waiting value in Gb/s.

We can estimated the bandwidth from the throughput using the standard Internet Mix packet size distribution.

The formula became: Bandwidth = PPS * ( 7*(40+14) + 4*(576+14) + (1500+14) )/12*8

Tools used

netmap pkt-gen

Netmap pkt-gen with an optional checksum & IPv6 patch is a great tool for benching router. It's a hight throughput packet generator using range of src/dst IP addresses or/and src/dst UDP ports mandatory for using the multi-queue feature of 10G NIC.

Switch configuration

When netmap pkt-gen runs as a packet receiver it will NEVER generate a frame:

Then during your bench test, the generator will generate line-rate 10Gb/s of traffic to the packet-receiver… but the MAC address of the receiver will age out on the switch table, and the switch will broadcast ALL traffic from the generator to ALL ports belonging to this VLAN ;-)

This is why it's always wise to statically configure MAC address of packet receiver on a switch before this kind of bench.

If you want to check the drivers statistics against the switchs stats, you should disable ALL advanced feature on the switch's ports used for the bench too, like:

Common configuration for all type of hardware

Disabling Ethernet Flow-control

Using Ethernet flow-control is a very bad idea on a router (and on a server): If your NIC is overloaded, it can ask an Ethernet "pause" to the direct peer (the switch). But this kind of mechanism already exist in TCP, and TCP have a better management than this very-basy Ethernet Flow-control. More information about this on When Flow Control is not a Good Thing

Disabling LRO and TSO

All modern NIC support LRO and TSO features that needs to be disabled on a router:

There is no real impact of disabling these feature on PPS.

x tso.lro.enabled
+ tso.lro.disabled
+--------------------------------------------------------------------------+
|   +  +     x+    *                          x+                    x     x|
|               |___________________________A_M_________________________|  |
||____________M___A________________|                                       |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5       1724046       1860817       1798145       1793343     61865.164
+   5       1702496       1798998       1725396     1734863.2     38178.905
No difference proven at 95.0% confidence

Reducing entropy harvest

Lot's of tuning guide indicate to disable these for getting best forwarding performance:

But here is the impact on a router (value in pps):

x harvest DISABLED
+ harvest ENABLED (default)
+--------------------------------------------------------------------------------+
|+                   x          x    x        x+        +   +   +               x|
|                    |_______________M_____A______________________|              |
|                   |_________________________A_________M______________|         |
+--------------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5       1918159       2036665       1950208       1963257     44988.621
+   5       1878893       2005333       1988952     1967850.8     51378.188
No difference proven at 95.0% confidence

=> No difference.

Enabling fastforwarding

If you're not using IPSec, fastforwarding NEEDs to be enabled, here is some performance impact regarding fastforwarding:

impact of fastforwarding, ipfw an pf on 4 cores Xeon on FreeBSD 10.0

10gFreeBSD/Router (last edited 2018-03-18 13:16:10 by MarkLinimon)