This page describes the requirements for a full cluster site. See also the requirements for a single-machine mirror site.

Full cluster sites host five machines:

The admin host serves replicas of cluster-internal services (DNS, LDAP, Kerberos, etc).

On layer 2, we need four VLANs - transit, internal, external and one to flip a CARP bit and sync pf state. We don't need our own switch if you can configure the VLANs on yours.

On layer 3, we need a /28 of routable IPv4 address space and a /64 of IPv6. We are completely flexible in terms of transit. Most of our existing sites set up CARP or VRRP, with routable addresses in the transit VLAN, and we set a default route on our firewalls. We are also happy to speak BGP with you or do whatever makes sense in your network.

We also require some sort of out-of-band console and power access to servers (either serial port or IPMI is fine).

This should be provisioned like a hosted customer network would be, where the customer provides and controls their own firewall.

Suggested minimum hardware specifications:

The git mirror depends on on being able to cache the back end git repository in ram to maintain a reasonable serving speed. At the time of writing, the two big back end repositories are 7GB for base, and 18GB for ports. Disk speed affects how quickly the cache will "warm up" after a reboot but having enough ram is far more important on an ongoing basis.

The package mirror doesn't require as fast disk I/O, but needs lots of storage. It should have a reasonable amount of memory. The pkg mirror is a downstream replica of a master with 14TB of usable space so we're looking for about that ballpark of usable space as a minimum. The configuration doesn't matter too much, it could just as easily be 8x2TB or 16x1TB if that was easier.

Ideally each of these servers would also have at least one hot spare disk available.

To begin configuration, we would like to have one machine installed with FreeBSD and then an account configured for SSH access. We will then take over the machine via the root password and proceed to install and configure all the machines via a netboot environment.

|                                                                             |
|                               Vendor Router                                 |
|             XXXX:XXXX:XXXX:XXXX::XXXX              |
|                                      |                                      |
|                 [  XXXX:XXXX:XXXX:XXXX::/125 ]           |
|                          [ transit network vlan ]                           |
|                                      |                                      |
|      igw0-ext.XXX               igw-ext.XXX             igw1-ext.XXX        |
|      (            (     |
|        FreeBSD   ____________    Firewall   ____________  FreeBSD           |
|       Firewall 1              CARP addresses             Firewall 2         |
|     (yyy.yyy.yyy.2)            yyy.yyy.yyy.1          (yyy.yyy.yyy.3)       |
| (yyyy:yyyy:yyyy:yyyy::2)         fe80::1          (yyyy:yyyy:yyyy:yyyy::3)  |
|        igw0.SSS                  igw.SSS                  igw1.SSS          |
|     10.0.XXX.1/24           pf sync(vlan0003)         10.0.XXX.2/24         |
|                                      |                                      |
|                          [ Internal network vlan ]                          |
|                [ yyy.yyy.yyy.yyy/28  yyyy:yyyy:yyyy:yyyy::/64 ]             |
|                                      |                                      |
|               +----------------------+----------------------+               |
|          Admin/Netboot host    Package Mirror        git/www mirror         |
|          yyy.yyy.yyy.4          yyy.yyy.yyy.5         yyy.yyy.yyy.6         |
|  (yyyy:yyyy:yyyy:yyyy::4) (yyyy:yyyy:yyyy:yyyy::5) (yyyy:yyyy:yyyy:yyyy::6) |

After the basics are set up, we configure the additional jailed services on the backend machines, assign internal addresses to them and selectively open firewall access.

Teams/clusteradm/generic-mirror-layout (last edited 2022-07-19T08:43:38+0000 by PhilipPaeps)