OpenBSD Amsterdam Logo
880 VMs deployed 🐡

The setup

For all the people who want to know what our setup looks like. Below is a write-up of our setup and configuration. There are only a handful packages installed on the servers running the Virtual Machines.

The building blocks

As with most things OpenBSD you can build almost anything with a base install alone. Our setup is no different! The building blocks which OpenBSD Amsterdam is built on are:

Other building blocks used are pf(4), veb(4)/vport(4), tap(4) and in our case vlan(4).

All the VM configuration is stored in individual text files, on the host the VM is running on, with information provided via the contact form.

The configuation files which are generated:

vmm(4)/vmd(8)

It all starts, of course, with vmm(4)/vmd(8) and vm.conf(5). Our vm.conf(5) looks something like:

socket owner :vmdusers

switch "uplink_veb42" {
    interface veb42
}

vm "<vm-name>" {
    disable
    memory 1G
    owner <user>
    disk "/var/vmm/<vm-name>.qcow2" format qcow2
    interface tap {
        switch "uplink_veb42"
        lladdr fe:e1:bb:d1:c8:01
    }
}

socket owner was introduced in OpenBSD 6.4. It provides the option to change the owner of vmd.sock.

This is useful when multiple users need to control their own VM without adding them to group :wheel, the default owner.

A statically assigned MAC address (lladdr) is generated when a new VM is installed based on a MAC prefix assigned to the host. This to make the installation of a new VM easier with autoinstall(8) in combination with dhcpd(8).

In order to make sure we have enough tap(4) interfaces, by default there are four interfaces present (tap0 .. tap3), we run the following command on a new server:

# cd /dev
# for i in $(jot 50 4 50); do sh MAKEDEV tap$i; done
#

When running in a larger network you might need to increase the arpq, which is set to 50 by default. On most hosts we increased it to 256.

# sysctl net.inet.ip.arpq.maxlen=256
net.inet.ip.arpq.maxlen: 50 -> 256
#

Running in layer 3 you need to permit IPv4 and IPv6 forwarding.

# sysctl net.inet.ip.forwarding=1
net.inet.ip.forwarding: 0 -> 1
# sysctl net.inet6.ip6.forwarding=1
net.inet6.ip6.forwarding: 0 -> 1

dhcpd(8)

Our dhcpd.conf(5) is per vmd(8) server.

option domain-name "openbsd.amsterdam";
option domain-name-servers <dns>;

subnet <ip-space> netmask 255.255.255.0 {
    option routers <default-gateway-ipv4>;
    server-name "<serverX>.openbsd.amsterdam";
    range <start-ip end-ip>;

    host <vm-name> {
        hardware ethernet fe:e1:bb:d1:c8:01;
        fixed-address <assigned-ipv4>;
        filename "auto_install";
        option host-name "<vm-name>";
    }
}

doas(1)

For every user we add a line in doas.conf(5), this to provide the ability for the user to use doas(1) to kill the vm process when needed.

permit nopass <username> as root cmd pkill args -9 -xf "vmd: <vm name>"

The command to run would be:

$ doas pkill -9 -xf "vmd: vm42"
$

unbound(8)

The unbound.conf(5) is also per vmd(8) server.

server:
    interface: <serverX-ipv4>
    interface: <serverX-ipv6>

    access-control: 0.0.0.0/0 refuse
    access-control: 127.0.0.0/8 allow
    access-control: <serverX-ipv4-subnet> allow
    access-control: ::0/0 refuse
    access-control: ::1 allow
    access-control: <serverX-ipv6-subnet> allow

    hide-identity: yes
    hide-version: yes
    qname-minimisation: yes
    root-hints: "/var/unbound/db/root.hints"
    auto-trust-anchor-file: "/var/unbound/db/root.key"
    aggressive-nsec: yes
    cache-max-ttl: 14400
    cache-min-ttl: 300
    prefetch: yes
    rrset-roundrobin: yes
    verbosity: 0

remote-control:
    control-enable: yes
    control-interface: /var/run/unbound.sock

autoinstall(8)

When a new VM is being created autoinstall(8) is used for the installation of the VM. Per VM a config file is created based on the statically assigned MAC address. For example:

/var/www/htdocs/<install>/fe:e1:bb:d1:c8:01-install.conf

# <user> install.conf
System hostname = <hostname>
Password for root = <pwd>
Which speed should com0 = 115200
Network interfaces = vio0
IPv4 address for vio0 = autoconf
IPv6 address for vio0 = <assigned-ipv6>
IPv6 default router = <default-gateway-ipv6>
Setup a user = <user>
Password for user = <pwd>
Public ssh key for user = ssh-ed25519 AAAA...TxlrE5 <comment> <pwd>
Which disk is the root disk = sd0
What timezone are you in = Europe/Amsterdam
Location of sets = http
Server = server.openbsd.amsterdam
Set name(s) = +site*
Continue anyway = yes
Continue without verification = yes

siteXX.tgz

We are using siteXX.tgz to post-install configure a newly created VM. This also includes the install.site file to create rc.local(8) and to add syspatch(8) to rc.firsttime(8)

Current files and functions are:

installurl(5):

https://cdn.openbsd.org/pub/OpenBSD

rc.conf.local(8):

sndiod_flags=NO

sysmerge(8).ignore:

/etc/ttys

install.site:

echo "/usr/sbin/syspatch && touch /etc/rc.local.forcereboot" >> /etc/rc.firsttime

Due to the broken version of syspatch71-001_wifi.tgz we had modified this for OpenBSD 7.1 as it needed to run twice. However after the first time syspatch exists with 2 not 0.

echo "/usr/sbin/syspatch || /usr/sbin/syspatch && touch /etc/rc.local.forcereboot" >> /etc/rc.firsttime

rc.local(8):

if [ -r /etc/rc.local.forcereboot ]; then
    rm -f /install.site /etc/rc.local.forcereboot
    printf '\n*** Reboot after CPU microcode/OS updates\n\n'
    sleep 2
    reboot
fi

Hat tip to Stuart Henderson

httpd(8)

Since OpenBSD comes with httpd(8) in base we are using it to serve the <MAC address>-install.conf files needed for autoinstall(8) with a minimal config.

server "default" {
    listen on * port 80
    root "/htdocs/<install>"
}

We use our own mirror to serve all the installer sets during the installation of the VM.

server "mirror.openbsd.amsterdam" {
    location "/pub/OpenBSD/7.3/amd64/*" {
        root "/htdocs/openbsd/7.3/amd64"
        request strip 4
        directory { auto index }
    }
}

perl(1)

The text files used to capture all the VM data is processed by a perl script, deploy.pl. Both the MAC address as well as the temporary password are generated by this script when needed. Here is one example how we generate the temporary password:

my $_pass = qx(jot -rcs '' 20 46 125);

We assign a MAC prefix per server and the last bits are allocated based on the VM number unless one is specified.

my $_mac = $vms{$vm_name}{'mac'} || $conf{'conf'}{'MAC_PREFIX'}
    . ":" . $vms{$vm_name}{'vm_number'};

sensorsd(8)

To keep track of the hardware, especially disks, we are using sensorsd(8). The configuration for the disks we are using in sensorsd.conf(5) is as follows.

drive:command=/etc/sensorsd/drive %t %n %2 %s

The drive script looks like:

#!/bin/sh
echo "Current raid state: ${1}${2} ${3} ${4}" | mail -s "$(hostname) \
${1}${2} ${4}" -r noreply@example.com admin@example.com

SSHFP

All SSH fingerprints (SSHFP) records of all the hosts are added to DNS. You can verify the SSH fingerprint by adding “-o VerifyHostKeyDNS=yes” to the ssh command.

$ ssh -o VerifyHostKeyDNS=yes serverXX.openbsd.amsterdam
The authenticity of host 'serverXX.openbsd.amsterdam' can't be established.
ECDSA key fingerprint is SHA256:w3ZoL03eaY/2xdRd/7NvHHwfqIOjyv2O8xkvUnqEgps.
Matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes
...
serverXX$