This is very much a work in progress. Please join in and improve it.

Please don't include references to other wiki pages - this will become a new *man* page

This page is the in the process of being converted to a man page with some of the changes here. Please don't modify it at this time until it is committed. At that point I will uncomment the text and sync the changes

<!-- This is initially just the current tuning(7) manpage with some annotations but hopefully we can pick out what's outdated and improve it to become the new tuning(7) manpage. Feel free to (in fact I want you to,) correct my mistakes :)

STORAGE SETUP

Partitioning isn't really performance tuning related, except for swap perhaps. Maybe the filesystem-specific discussion should move to its own page? --DougWhite FreeBSD/amd64 9.0 kitchen sink install = 2.7GB, of that ~1.2GB is ports and 750MB is src. --DougWhite

Swap

Size swap space to approximately twice the size of main memory on systems with less than 4GB RAM and the size of main memory for systems with more than 4GB. If in doubt, allocate more swap; allocating insufficient swap is far worse than allocating too much. If the system has multiple disks, reduce swap I/O contention by spreading swap across the disks, ideally in equally sized partitions.

/var

How you size your /var partition depends heavily on what you intend to use the machine for. This partition is primarily used to hold mailboxes, mail queues, print spools, and log files. If your machine will act as a mail or print server, or you are running a heavily visited web server, you should consider creating a much larger partition.

/tmp vs /var/tmp usage needs further consideration, especially now that bsdinstall creates a separate /tmp but leaves /var/tmp on /var. I deleted the existing text since it was unnecessarily dense, though. --DougWhite

/usr

The /usr partition holds the bulk of the files required to support the system and a subdirectory within it called /usr/local holds the bulk of the files installed from the ports(7) hierarchy. If you do not use ports all that much and do not intend to keep system source (/usr/src) on the machine, you can get away with a 1 gigabyte /usr partition. However, if you install a lot of ports (especially window managers and Linux-emulated binaries), we recommend at least a 30 GB /usr and if you also intend to keep system source on the machine, we recommend a 40 GB /usr. Do not underestimate the amount of space you will need in this partition, it can creep up and surprise you!

The /home partition is typically used to hold user-specific data. I usu- ally size it to the remainder of the disk.

Why partition at all? Why not create one big / partition and be done with it? Then I do not have to worry about undersizing things! Well, there are several reasons this is not a good idea. First, each partition has different operational characteristics and separating them allows the file system to tune itself to those characteristics. For example, the root and /usr partitions are read-mostly, with very little writing, while a lot of reading and writing could occur in /var and /var/tmp. By properly partitioning your system fragmentation introduced in the smaller more heavily write-loaded partitions will not bleed over into the mostly- read partitions. Additionally, keeping the write-loaded partitions closer to the edge of the disk (i.e., before the really big partitions instead of after in the partition table) will increase I/O performance in the partitions where you need it the most. Now it is true that you might also need I/O performance in the larger partitions, but they are so large that shifting them more towards the edge of the disk will not lead to a significant performance improvement whereas moving /var to the edge can have a huge impact. Finally, there are safety concerns. Having a small neat root partition that is essentially read-only gives it a greater chance of surviving a bad crash intact.

Properly partitioning your system also allows you to tune newfs(8), and tunefs(8) parameters. Tuning newfs(8) requires more experience but can lead to significant improvements in performance. There are two parameters that are relatively safe to tune: bytes/i-node, and cylinders/group.

If a large partition is intended to be used to hold fewer, larger files, such as database files, you can increase the bytes/i-node ratio which reduces the number of i-nodes (maximum number of files and directories that can be created) for that partition. Decreasing the number of i-nodes in a file system can greatly reduce fsck(8) recovery times after a crash. Do not use this option unless you are actually storing large files on the partition, because if you overcompensate you can wind up with a file system that has lots of free space remaining but cannot accommodate any more files. Using 64KiB, 128KiB, or 256KiB bytes/i-node is recommended. You can go higher but it will have only incremental effects on fsck(8) recovery times. For example, "newfs -i 65536 ...".

tunefs(8) may be used to further tune a file system. This command can be run in single-user mode without having to reformat the file system. However, this is possibly the most abused program in the system. Many people attempt to increase available file system space by setting the minfree percentage to 0. This can lead to severe file system fragmentation and we do not recommend that you do this. Really the only tunefs(8) option worthwhile here is turning on softupdates with 'tunefs -n enable /filesystem'. Softupdates can be turned on using the -U option to newfs(8), and bsdinstall(8) will typically enable softupdates automatically for non-root file systems). For 9.x softupdates journaling is also worth turning on, for SSDs it's worth enabling trim.

Softupdates drastically improves meta-data performance, mainly file creation and deletion. We recommend enabling softupdates on most file systems; however, there are two limitations to softupdates that you should be aware of when determining whether to use it on a file system. First, softupdates guarantees file system consistency in the case of a crash but could very easily be several seconds (even a minute!) behind on pending write to the physical disk. If you crash you may lose more work than otherwise. Secondly, softupdates delays the freeing of file system blocks. If you have a file system (such as the root file system) which is close to full, doing a major update of it, e.g. "make installworld", can run it out of space and cause the update to fail. For this reason, softupdates will not be enabled on the root file system during a typical install. There is no loss of performance since the root file system is rarely written to.

A number of run-time mount(8) options exist that can help you tune the system. The most obvious and most dangerous one is async. Only use this option in conjunction with gjournal(8), as it is far too dangerous on a normal file system. A less dangerous and more useful mount(8) option is called noatime. UNIX file systems normally update the last-accessed time of a file or directory whenever it is accessed. This operation is handled in FreeBSD with a delayed write and normally does not create a burden on the system. However, if your system is accessing a huge number of files on a continuing basis the buffer cache can wind up getting polluted with atime updates, creating a burden on the system. For example, if you are running a heavily loaded web site, or a news server with lots of readers, you might want to consider turning off atime updates on your larger partitions with this mount(8) option. However, you should not gratuitously turn off atime updates everywhere. For example, the /var file system customarily holds mailboxes, and atime (in combination with mtime) is used to determine whether a mailbox has new mail. You might as well leave atime turned on for mostly read-only partitions such as / and /usr as well. This is especially useful for / since some system utilities use the atime field for reporting.

STRIPING DISKS

In larger systems you can stripe partitions from several drives together to create a much larger overall partition. Striping can also improve the performance of a file system by splitting I/O operations across two or more disks. The gstripe(8), gvinum(8), and ccdconfig(8) utilities may be used to create simple striped file systems. Generally speaking, striping smaller partitions such as the root and /var/tmp, or essentially read- only partitions such as /usr is a complete waste of time. You should only stripe partitions that require serious I/O performance, typically /var, /home, or custom partitions used to hold databases and web pages. Choosing the proper stripe size is also important. File systems tend to store meta-data on power-of-2 boundaries and you usually want to reduce seeking rather than increase seeking. This means you want to use a large off-center stripe size such as 1152 sectors so sequential I/O does not seek both disks and so meta-data is distributed across both disks rather than concentrated on a single disk. If you really need to get sophisticated, we recommend using a real hardware RAID controller from the list of FreeBSD supported controllers.

SYSCTL TUNING

sysctl(8) variables permit system behavior to be monitored and controlled at run-time. Some sysctls simply report on the behavior of the system; others allow the system behavior to be modified; some may be set at boot time using rc.conf(5), but most will be set via sysctl.conf(5). There are several hundred sysctls in the system, including many that appear to be candidates for tuning but actually are not. In this document we will only cover the ones that have the greatest effect on the system.

IIRC mav@ wrote a mail as part of his MAXPHYS work where he explained that a value of 128 was a good value in his FS tests, needs to be looked up in the archives and investigated. http://kb.lsi.com/knowledgebasearticle14852.aspx lists increasing this as a recommended tuning although they use higher numbers, http://www.freebsdwiki.net/index.php/RAID,_performance_tests (found via google, last updated 2007) says "The read-ahead cache was changed from the default value of 8 to 128 for all tests performed, using sysctl -w vfs.read_max=128. Initial testing showed that dramatic performance increases occurred for all tested configurations, including baseline single-drive, with increases of vfs.read_max. The value of 128 was arrived at by continuing to double vfs.read_max until no further significant performance increase was to be seen (at vfs.read_max=256) and backing down to the last value tried." Finally, http://ivoras.sharanet.org/blog/tree/2010-11-19.ufs-read-ahead.html IvanVoras discusses tuning this here.

As an additional management tool you can use pipes in your firewall rules (see ipfw(8)) to limit the bandwidth going to or from particular IP blocks or ports. For example, if you have a T1 you might want to limit your web traffic to 70% of the T1's bandwidth in order to leave the remainder available for mail and interactive use. Normally a heavily loaded web server will not introduce significant latencies into other services even if the network link is maxed out, but enforcing a limit can smooth things out and lead to longer term stability. Many people also enforce artificial bandwidth limitations in order to ensure that they are not charged for using too much bandwidth.

Setting the send or receive TCP buffer to values larger than 65535 will result in a marginal performance improvement unless both hosts support the window scaling extension of the TCP protocol, which is controlled by the net.inet.tcp.rfc1323 sysctl. These extensions should be enabled and the TCP buffer size should be set to a value larger than 65536 in order to obtain good performance from certain types of network links; specifically, gigabit WAN links and high-latency satellite links. RFC1323 support is enabled by default.

net.inet.tcp.nolocaltimewait=1 -> stops creating any state (nor socket neither compressed tcpw) for the TCP connection where both endpoints were local. This saves resources on a server running HTTP accelerators or database servers+clients.

To add for desktop interactivity: kern.sched.preempt_thresh (=220)... see Message-ID 4C85D0C5.7060603@icyb.net.ua Tuning the scheduler? (freebsd-stable) for a description

net.link.ether.inet.maxhold (9+; for system <9 it was hardcoded to 1): system wide max number of packets in the ARP queue.

LOADER TUNABLES

Some aspects of the system behavior may not be tunable at runtime because memory allocations they perform must occur early in the boot process. To change loader tunables, you must set their values in loader.conf(5) and reboot the system.

Please note that some drivers require larger values, and in some cases the interfaces will not function; some drivers are known to hang on probe/configuration, e.g. some dual-port or quad-port igb(4), cards, or cause system panics, e.g. cxgb(4), if you set kern.ipc.nmbclusters to something less than 50000 and 75000, respectively.

More and more programs are using the sendfile(2) system call to transmit files over the network. The kern.ipc.nsfbufs sysctl controls the number of file system buffers sendfile(2) is allowed to use to perform its work. This parameter nominally scales with kern.maxusers so you should not need to modify this parameter except under extreme circumstances. See the TUNING section in the sendfile(2) manual page for details.

KERNEL CONFIG TUNING

There are a number of kernel options that you may have to fiddle with in a large-scale system. In order to change these options you need to be able to compile a new kernel from source. The config(8) manual page and the handbook are good starting points for learning how to do this. Generally the first thing you do when creating your own custom kernel is to strip out all the drivers and services you do not use. Removing things like INET6 and drivers you do not have will reduce the size of your kernel, sometimes by a megabyte or more, leaving more memory available for applications.

kern.cam.boot_delay may be used to reduce system boot times. The defaults are fairly high and can be responsible for 5+ seconds of delay in the boot process. Reducing kern.cam.boot_delay to something below 5 seconds could work (especially with modern drives).

There are a number of *_CPU options that can be commented out. If you only want the kernel to run on a Pentium class CPU, you can easily remove I486_CPU, but only remove I586_CPU if you are sure your CPU is being recognized as a Pentium II or better. Some clones may be recognized as a Pentium or even a 486 and not be able to boot without those options. If it works, great! The operating system will be able to better use higher-end CPU features for MMU, task switching, timebase, and even device operations. Additionally, higher-end CPUs support 4MB MMU pages, which the kernel uses to map the kernel itself into memory, increasing its efficiency under heavy syscall loads.

MAXPHYS=(1024*1024) for 9+, a mail from mav@ has to be searched for a good description.

IDE WRITE CACHING

FreeBSD 4.3 flirted with turning off IDE write caching. This reduced write bandwidth to IDE disks but was considered necessary due to serious data consistency issues introduced by hard drive vendors. Basically the problem is that IDE drives lie about when a write completes. With IDE write caching turned on, IDE hard drives will not only write data to disk out of order, they will sometimes delay some of the blocks indefinitely under heavy disk load. A crash or power failure can result in serious file system corruption. So our default was changed to be safe. Unfortunately, the result was such a huge loss in performance that we caved in and changed the default back to on after the release. You should check the default on your system by observing the hw.ata.wc sysctl variable. If IDE write caching is turned off, you can turn it back on by setting the hw.ata.wc loader tunable to 1. More information on tuning the ATA driver system may be found in the ata(4) manual page. If you need performance, go with SCSI.

CPU, MEMORY, DISK

The type of tuning you do depends heavily on where your system begins to bottleneck as load increases. If your system runs out of CPU (idle times are perpetually 0%) then you need to consider upgrading the CPU or moving to an SMP motherboard (multiple CPU's), or perhaps you need to revisit the programs that are causing the load and try to optimize them. If your system is paging to swap a lot you need to consider adding more memory. If your system is saturating the disk you typically see high CPU idle times and total disk saturation. systat(1) can be used to monitor this. As can gstat(8)

There are many solutions to saturated disks: increasing memory for caching, mirroring disks, distributing operations across several machines, and so forth. If disk performance is an issue and you are using IDE drives, switching to SCSI can help a great deal. While modern IDE drives compare with SCSI in raw sequential bandwidth, the moment you start seeking around the disk SCSI drives usually win.

NETWORK

See NetworkPerformanceTuning

POWER MANAGEMENT

A processor frequency (ticks per second) determines how much work (instructions processed per second) can be done. CPU-bound (intensive) tasks can run faster with higher CPU frequency. More work done (more tasks per second) causes more switchings in integrated circuits, which consumes more current, which causes more power consumption. Even when no CPU tasks are performed (idle state), higher CPU frequency causes some power consumption. From the point of view of source of power (AC or battery), higher voltage enables higher power supply, but in turn also possibly higher power consumption.

In order to reduce power consumption, it is advantageous to modulate CPU frequency and its voltage when:

The power management works by modulating CPU frequency and voltage by means of:

Power management with powerd daemon (P-states and C-states). POWERD(8). The default mode is adaptive for battery and hiadaptive for other (AC power).

An example of user configuration (be careful of using non-default values as they may shut down some system components, and the entire system as a result of it).

/etc/rc.conf:

powerd_flags="-n hadp"

performance_cx_lowest="C2"

economy_cx_lowest="C2"

performance_cpu_freq="HIGH"

PerformanceTweaks

NetworkPerformanceTuning

Performance

ZFSTuningGuide

http://people.freebsd.org/~kris/scaling/Help_my_system_is_slow.pdf

http://www.freebsdwiki.net/index.php/RAID,_performance_tests

http://ivoras.sharanet.org/blog/tree/2010-11-19.ufs-read-ahead.html -->

SystemTuning (last edited 2017-09-18 12:37:51 by KubilayKocak)