• Teams
• clusteradm
• new-site-standards

When bringing up future freebsd.org sites

Prerequisites:

• Clear site contact information
• Operational ownership and responsibility established and documented
• Clarity between foundation and project over the site
• Added to clusteradm site index page (Link TBD)

Basic standards:

• Remote power control
• Remote console
• BIOS console redirection enabled
• 115200 baud console
• Switch with ability to partition and remotely change vlan settings on a per-port basis

Internet for tier 1 sites:

• Naked internet, similar to how a site would provide a hosted customer.
• NO upstream firewall. We use a pf/pfsync/igw setup.
• Such sites can be part of global geo-routed services like pkg, www, svn etc.

Internet for tier 2 sites:

• If the upstream is providing firewalls, we can skip the igw/pf/pfsync setup.
• If the upstream is providing firewalls, our recourse for a site problem is to pull the site offline.
• Such sites MAY NOT be used for tier-1 services as we can't do immediate updates if required.
• Such sites MAY NOT carry cluster secrets - eg: copies of kerberos database, password hashes, etc.
• Such sites may be useful for country or site specific mirrors.

Initial bringup:

• Initial network access, basic services and netboot cabability

• all subsequent installs done via netboot to prove remote power, console and reinstall capability

• no memory stick / out of band installs unless it is absolutely impossible to netboot install, and documented as an unmanaged machine

• unmanaged machines are NOT used for anything critical.

Naming standards:

• 3 letter dns prefix
• understandable hostnames. not obscure acronyms.
• paired / redundant machines with a numerical suffix.
• if at all possible, use an existing naming scheme. Don't get cute. (especially peter@.. did you hear that?)

Jail naming and assignments:

• No **magic**, hacks, needless complexity, including pf hacks, loopback, baked in hostnames etc.
• Only use suffix of jail host for inherently non-portable jails. eg: an input mirror feed for per-machine use.
• machines like blue.freebsd.org can never happen again.
• If there is any possibility of a jail being portable or cloneable, then set it up that way. Don't bake in assumptions about its jail host.
• nullfs does NOT share locks, you cannot share databases across nullfs. Notably including svn.

Subnetting:

• Partition ipv4 space to allow mapping subnets to vlans
• Partition according to similar function. Don't put world+dog developers in the same vlan as security critical machines.

Cluster security:

• ssh keys for initial access
• ksu for elevated access
• NO password hashes, aside from root for console use.
• NO sudo
• NO g+w as a means of accessing a privileged subsystem's files
• NO logging in directly as role or privileged accounts, use ksu
• Privilege containment is to be done by having privileged processes safely pulling from work areas
• NO automated privilege escalation by scripts etc.

• ssh PermitRootLogin no

• ssh password authentication disabled.
• use integrated clusteradm scripts.
• Beat somebody up if clusteradm scripts are missing roles, don't use roles as an excuse to not use them.

IPv4 address assignment:

• Link TBD

IPv6 address assignment:

• Link TBD

Vlan assignment:

• Teams/clusteradm/vlanlayout

Cluster foothold requirements:

• A pair of routers
• An admin shell / netboot / general work box
• 3+ jail servers for cluster integration:
  • hosts/jails for replication of network services (dns, ntp, ldap, kerberos). these need to be redundant
  • a jail for sensitive things.. snmp monitoring, consoles, etc. (This may be able to be hosted on admin server.)

"Special" machines:

• Noted on the site index page.