This page describes the requirements for a full cluster site. See also the requirements for a single-machine mirror site.
Full cluster sites host five machines:
- 2x firewalls
- 1x admin host
- 1x pkg mirror
- 1x ftp/git/svn mirror
The admin host serves replicas of cluster-internal services (DNS, LDAP, Kerberos, etc).
On layer 2, we need four VLANs - transit, internal, external and one to flip a CARP bit and sync pf state. We don't need our own switch if you can configure the VLANs on yours.
On layer 3, we need a /28 of routable IPv4 address space and a /64 of IPv6. We are completely flexible in terms of transit. Most of our existing sites set up CARP or VRRP, with routable addresses in the transit VLAN, and we set a default route on our firewalls. We are also happy to speak BGP with you or do whatever makes sense in your network.
We also require some sort of out-of-band console and power access to servers (either serial port or IPMI is fine).
This should be provisioned like a hosted customer network would be, where the customer provides and controls their own firewall.
Suggested minimum hardware specifications:
- igw0/1, admin - 2 CPU cores, 2x1Gbe network, 4G ram, 2xSATA HD 500GB space (mirror)
- pkg - 2 CPU cores, 1x1GbE network, 96G ram, 4xSATA HD (4TB) for 12TB space(raidz1)
- ftp/git/svn - 4 CPU cores, 1x1GbE network, 48G ram, 4xSATA HD (300 Gb minimum) for 1TB space (raidz1)
The git mirror depends on on being able to cache the back end git repository in ram to maintain a reasonable serving speed. At the time of writing, the two big back end repositories are 7GB for base, and 18GB for ports. Disk speed affects how quickly the cache will "warm up" after a reboot but having enough ram is far more important on an ongoing basis.
The package mirror doesn't require as fast disk I/O, but needs lots of storage. It should have a reasonable amount of memory. The pkg mirror is a downstream replica of a master with 14TB of usable space so we're looking for about that ballpark of usable space as a minimum. The configuration doesn't matter too much, it could just as easily be 8x2TB or 16x1TB if that was easier.
Ideally each of these servers would also have at least one hot spare disk available.
To begin configuration, we would like to have one machine installed with FreeBSD and then an account configured for SSH access. We will then take over the machine via the root password and proceed to install and configure all the machines via a netboot environment.
------------------------------------------------------------------------------- | | | Vendor Router | | xxx.xxx.xxx.xxx XXXX:XXXX:XXXX:XXXX::XXXX | | | | | [ xxx.xxx.xxx.xxx/29 XXXX:XXXX:XXXX:XXXX::/125 ] | | [ transit network vlan ] | | | | | igw0-ext.XXX igw-ext.XXX igw1-ext.XXX | | (xxx.xxx.xxx.2) xxx.xxx.xxx.1 (xxx.xxx.xxxx.3) | | (XXXX:XXXX:XXXX:XXXX::2) XXXX:XXXX:XXXX:XXXX::1 (XXXX:XXXX:XXXX:XXXX::3) | | FreeBSD ____________ Firewall ____________ FreeBSD | | Firewall 1 CARP addresses Firewall 2 | | (yyy.yyy.yyy.2) yyy.yyy.yyy.1 (yyy.yyy.yyy.3) | | (yyyy:yyyy:yyyy:yyyy::2) fe80::1 (yyyy:yyyy:yyyy:yyyy::3) | | igw0.SSS igw.SSS igw1.SSS | | 10.0.XXX.1/24 pf sync(vlan0003) 10.0.XXX.2/24 | | | | | [ Internal network vlan ] | | [ yyy.yyy.yyy.yyy/28 yyyy:yyyy:yyyy:yyyy::/64 ] | | | | | +----------------------+----------------------+ | | Admin/Netboot host Package Mirror git/www mirror | | yyy.yyy.yyy.4 yyy.yyy.yyy.5 yyy.yyy.yyy.6 | | (yyyy:yyyy:yyyy:yyyy::4) (yyyy:yyyy:yyyy:yyyy::5) (yyyy:yyyy:yyyy:yyyy::6) | |_____________________________________________________________________________|
After the basics are set up, we configure the additional jailed services on the backend machines, assign internal addresses to them and selectively open firewall access.