installing a new machine at NYI
This information is correct as of November 2020.
Add the machine to https://rackmonkey.freebsd.org/. Document the physical connections and any peculiarities of the new machine in the notes field. Also note the MAC addresses so the machine can be found again if it inadvertently moves to a different switch port for whatever reason.
- Allocate IP addresses in the DNS.
- Remember to set up reverse DNS too.
- Also set up an IP address for IPMI.
- And reverse DNS for it.
- Poke holes in the firewall for IPMI
- Add IPMI to conserver
After committing to admin-svn, wait for the update emails to show up and resolve any alerts.
On admin.nyi, console to rpduXXXX/lpduXXXX and name the power outlets.
Modern machines will have IPMI rather than serial ports.
For a fresh installation, put both IPMI and network in the install vlan. Also remember to name the port.
On admin.nyi, ssh to rsw, and set things up.
set interfaces ge-N/0/XX description "hostname ipmi" set interfaces ge-N/0/YY description "hostname port 0" set interfaces ge-N/0/XX unit 0 family ethernet-switching vlan members install set interfaces ge-N/0/YY unit 0 family ethernet-switching vlan members install commit
Now grep /var/log/daemon for the IPMI MAC address on admin.nyi to find the DHCP lease in the install vlan. Then you can get to the console with ipmitool -U ADMIN -P ADMIN -H 10.0.7.XX sol activate.
Follow the instructions in /etc/motd of the pxeroot environment. The root password is in secrets/nyi.asc in the admin-svn repository.
Before rebooting the machine, jail into the install environment and set up production network information in /etc/rc.conf. You may also need to add something like comconsole_port="0x2f8" to /boot/loader.conf.
When you're happy the machine has a chance of showing up on the network following a reboot, exit the jail and reboot the server.
Move to production network
While the machine is booting, move the network interface (not IPMI!) to the production vlan on the switch:
delete interfaces ge-N/0/YY unit 0 family ethernet-switching vlan members install set interfaces ge-N/0/YY unit 0 family ethernet-switching vlan members XXXXX commit
When the machine comes up, verify that the network works by pinging freebsd.org on the console.
Follow instructions in /etc/motd for the new install.
You'll probably want to set up swap too.
gmirror label swap0 ada0p2 ada1p2 gmirror label swap1 ada2p2 ada3p2
And adjust /etc/fstab accordingly.
Do a sysrc local_unbound_enable="YES". The clusteradm scripts will configure it.
After /etc/motd is clear of warnings, reboot the machine to make sure everything still works.
Move IPMI to production
At this stage, the IPMI port is still in the install VLAN and you're talking to it with ipmitool on admin.nyi.
ssh to the new machine from somewhere else and become root and disconnect your ipmitool on admin.nyi (with ~.).
On the new machine, over ssh:
pkg install ipmitool ipmitool lan set 1 ipsrc static ipmitool lan set 1 ipaddr 10.0.0.XXX ipmitool lan set 1 netmask 255.255.255.0 ipmitool lan set 1 defgw ipaddr 10.0.0.1
Now move the IPMI port to the netmgmt vlan on rsw:
delete interfaces ge-N/0/XX unit 0 family ethernet-switching vlan members install set interfaces ge-N/0/XX unit 0 family ethernet-switching vlan members netmgmt commit
If you've done everything right, after about a minute, you should be able to console to the new machine from admin.nyi.
old information from before November 2020
(PhilipPaeps note: this information is very old but may still have value when debugging) (MarkLinimon note: WIP; needs some information from clusteradm/YahooNetinstall)
- set up the BIOS:
- figure out the MAC address
clone an entry in /usr/local/etc/dhcpd.conf and use that MAC address
machines which will pxeboot
make sure rsw02XX has the the interface's vlan set to portsbuild (e.g.)
make sure the /usr/local/etc/dhcpd.conf entry for your machine has entries for both fixed-address and option root-path
- boot the machine
- format the drive
edit drive information into files in (e.g.) /home/linimon/pxeroot/amd64-9/conf/<ipaddress>/etc/:
- reboot the machine
machines which will boot from hard drive
- if you have not done so already:
create a kickstart directory for pxebooting, something like /home/linimon/pxeroot/amd64-install-9.0
if you are installing via ZFS, this directory will need to be exported read-write.
copy over the /home/linimon/pxeroot/amd64-install-9.0/getbootableiso script to it and edit for that directory and release
sudo sh -x ./getbootableiso
(see BradDavis's writeup for more detail of what getbootableiso does)
make sure rsw02XX has the the proper interface's vlan set to install
make sure the /usr/local/etc/dhcpd.conf entry for your machine has entries for neither fixed-address nor option root-path so it will boot from the default install vlan
- reboot the machine.
- do the install.
- reboot the machine. Halt the boot in the BIOS and set the disk to have a higher boot priority than the Ethernet adapter.
if the machine comes up correctly, make sure the /usr/local/etc/dhcpd.conf entry for your machine has entries for both fixed-address and option root-path. The former should be an address that is in the range of the desired vlan. (See Teams/nyi-admin/Networks for the list of vlans).
- reboot the machine again to get the proper new address you picked.