Enterprise networking in the box

(this is in progress “livestream”. Header will be removed when completed).

I had OVH provision a /28 to the new dedi.

IP allocation:

  1. ovh-preproduction -01 wan ip
  2. ovh-preproduction-02 wan ip
  3. ovh-wanrtr-01 wan ip
  4. ovh-wanrtr-02 wan ip
  5. ovh-wanrtr wan float
  6. external web general (cloudflare origin) (route through opnsense dmz farm)
  7. external fnf (direct route to VM on brWan)
  8. external esign (direct route to VM on brWan)
  9. external discourse (direct route to VM on brWan)

This leaves me 4 spares (/32 using dedi IP .254 as gateway) makes the entire block usable. 🙂

Bare Metal – The base of it all

All stacks need a proper foundation or it all comes tumbling down.

Here’s the base:

Provider:

OVH (Canda data center)

TSYS has been with OVH (since the beginning) for several years and been exceptionally pleased with the service they provide. Rock solid, highly economical, fast provisioning.

Product choice:
RISE-2 – Intel Xeon D-1540 – RAM 64GB LEG – 2x 2TB SATA Datacenter Class Soft RAID (at 75.09 a month with two year commit, paid monthly) which is down from 80.99 with no commit.

OS Choice:

Ubuntu 18.04 . TSYS is a Ubuntu/debian shop (with a couple Centos appliance VM for some business applications like e-sign) .

OS configuration:

Packages

apt-get -y install qemu-kvm virt-manager fail2ban postfix glances htop tcpdump dstat mailutils molly-guard bridge-utils vlan openvswitch-switch build-essential autoconf automake libtool gawk alien fakeroot ksh zlib1g-dev uuid-dev libattr1-dev libblkid-dev libselinux-dev libudev-dev libacl1-dev libaio-dev libdevmapper-dev libssl-dev libelf-dev linux-headers-$(uname -r) python3 python3-dev python3-setuptools python3-cffi zsh openvswitch-switch-dpdk ladvd logwatch smartmontools lm-sensors snmpd haveged xfce4 xfce4-goodies xorg dbus-x11 x11-xserver-utils xrdp cpufrequtils chromium-browser

(configure postfix as internet site). It’s critical that the base system can send mail without any VM dependency.

xrdp/xfce will make your life MUCH easier when setting up virtual machines via virt-manager (x11 forwarding over ssh is horribly slow)

Setup netdata

( https://github.com/netdata/netdata ) This has been by far the most useful/detailed monitoring/alerting I’ve setup. It does require a fair amount of tweaking (I’m keeping the details of the tweaks under wraps , but am available for consulting at reasonable hourly rates to help optimize your netdata setup). It works out of the box but can definitely be a bit over verbose until you adjust it.

Setup hpn-ssh

( https://www.psc.edu/hpn-ssh ) I recommend replacing your system SSH with this. It’s so much faster for copying data etc.

Setup ZFS (I’m doing this build on 2/15/2019 and using zfs-0.8.3)

https://www.tonylykke.com/posts/2019/08/03/zfs-0.8.1-on-ubuntu-18.04/

https://zfsonlinux.org/

Drive layout

For the non data (this was done via OVH web installer, setup a mirrored 15G root partition)

[email protected]:~/iso# cat /etc/fstab

/dev/md3 / ext4 errors=remount-ro 0 1
/dev/md2 /boot ext4 errors=remount-ro 0 1
/dev/sda4 swap swap defaults 0 0
/dev/sdb4 swap swap defaults 0 0
proc /proc proc defaults 0 0
sysfs /sys sysfs defaults 0 0
UUID=89EC-EA13 /boot/efi vfat defaults 0 0

ZFS…

Create a partition on each hard drive (sda5 sdb5) in my case :

ovh-dedi-01# fdisk -l /dev/sda /dev/sdb
Disk /dev/sda: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EDAD6DE9-92E7-4CCE-8605-9E5724127D0D

Device Start End Sectors Size Type
/dev/sda1 2048 1048575 1046528 511M EFI System
/dev/sda2 1048576 2095103 1046528 511M Linux RAID
/dev/sda3 2095104 32813055 30717952 14.7G Linux RAID
/dev/sda4 32813056 33859583 1046528 511M Linux swap
/dev/sda5 33859584 3907029134 3873169551 1.8T Linux filesystem

Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: A04DBC0B-ADF9-4A01-8810-E30C18A0B2C9

Device Start End Sectors Size Type
/dev/sdb1 2048 1048575 1046528 511M EFI System
/dev/sdb2 1048576 2095103 1046528 511M Linux RAID
/dev/sdb3 2095104 32813055 30717952 14.7G Linux RAID
/dev/sdb4 32813056 33859583 1046528 511M Linux swap
/dev/sdb5 33859584 3907029134 3873169551 1.8T Linux filesystem
ovh-dedi-01#

It appears the packages for zfs that ship with 18.04.3 (as of 2/15/2020) don’t work (undefined symbols). So I built from source ( https://github.com/zfsonlinux/zfs/releases/download/zfs-0.8.3/zfs-0.8.3.tar.gz and doc at https://www.tonylykke.com/posts/2019/08/03/zfs-0.8.1-on-ubuntu-18.04/ ). I’ll skip over my ZFS setup (hire me if you want details and/or a beautiful production setup in your environment).

Network

Here be dragons…

WAN Setup:

So OVH 18.04 template sets up the network using systemd and injects your WAN IP (this allows the box to be on net and we can do basic setup). However it doesn’t let us do fancy pants network stuff.

[email protected]:/etc/systemd/network# ls
50-default.network 50-public-interface.link
[email protected]:/etc/systemd/network# cat 50-default.network

This file sets the IP configuration of the primary (public) network device.

You can also see this as “OSI Layer 3” config.

It was created by the OVH installer, please be careful with modifications.

Documentation: man systemd.network or https://www.freedesktop.org/software/systemd/man/systemd.network.html

[Match]
MACAddress=0c:c4:7a:77:1e:6e

[Network]
Description=network interface on public network, with default route
DHCP=no
Address=158.69.53.198/24
Gateway=158.69.53.254
IPv6AcceptRA=no
NTP=ntp.ovh.net
DNS=127.0.0.1
DNS=213.186.33.99
DNS=2001:41d0:3:163::1
Gateway=2607:5300:0060:82ff:ff:ff:ff:ff

[Address]
Address=2607:5300:0060:82c6::/64

[Route]
Destination=2607:5300:0060:82ff:ff:ff:ff:ff
Scope=link
[email protected]:/etc/systemd/network#

So follow the steps from
https://www.naut.ca/blog/2018/12/12/disabling-systemd-networking/

and setup a wan bridge:

Setup /etc/network/interfaces as such (check with ip a to see what interface you should bridge):

auto lo
iface lo inet loopback

auto eno3
iface eno3 inet manual

auto brWan
iface brWan inet static
address 158.69.53.198
netmask 255.255.255.0
network 158.69.53.0
gateway 158.69.53.254
bridge_ports eno3
bridge_stp off
bridge_fd 0
bridge_hello 2
bridge_maxage 12

Reboot and ensure you can still ssh to the dedi. We’ll setup the VLANs etc in the next step. Change one thing at a time. 🙂

Upon reboot you should have:

1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 0c:c4:7a:77:1c:16 brd ff:ff:ff:ff:ff:ff
3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 0c:c4:7a:77:1c:17 brd ff:ff:ff:ff:ff:ff
4: eno3: mtu 1500 qdisc mq master brWan state UP group default qlen 1000
link/ether 0c:c4:7a:77:1e:6e brd ff:ff:ff:ff:ff:ff
5: eno4: mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 0c:c4:7a:77:1e:6f brd ff:ff:ff:ff:ff:ff
6: brWan: mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 0c:c4:7a:77:1e:6e brd ff:ff:ff:ff:ff:ff
inet 158.69.53.198/24 brd 158.69.53.255 scope global brWan
valid_lft forever preferred_lft forever
inet6 fe80::ec4:7aff:fe77:1e6e/64 scope link
valid_lft forever preferred_lft forever
7: virbr0: mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:29:b8:fd brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
8: virbr0-nic: mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
link/ether 52:54:00:29:b8:fd brd ff:ff:ff:ff:ff:ff
ovh-dedi-01# brctl show
bridge name bridge id STP enabled interfaces
brWan 8000.0cc47a771e6e no eno3
virbr0 8000.52540029b8fd yes virbr0-nic

ovh-dedi-01# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 158.69.53.254 0.0.0.0 UG 0 0 0 brWan
158.69.53.0 0.0.0.0 255.255.255.0 U 0 0 0 brWan
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
ovh-dedi-01#

VLAN/LAN setup:

This is where things get pretty complicated, but also incredibly powerful. It allows for “enterprise LAN in a box” which eliminates need for anything except a single server (no switches, firewalls etc needed).

OpenVswitch is the core component.

(I’m keeping this mostly under wraps as it took a decent amount of time/effort to get correct. Hire me to implement it for you if you want all the details). Here’s a rough outline and you can fill in the blanks:

auto <bridge name>
allow-ovs <bridge-name>
iface <bridge-name> inet static
address <bare metal ip>
netmask <bare metal mask>
ovs_type OVSBridge
ovs_ports <vlan tag>

allow-mgmt <vlan tag>
iface <vlan tag> inet manual
ovs_bridge <bridge-name>
ovs_type OVSIntPort
ovs_options tag=<vlan tag>

You need todo this for all the vlans you want. I assign an IP to all the interfaces for monitoring and other purposes. This is optional and they can be strictly layer 2 “ports”.

Setup ladvd to announce on the ports (this is useful for nedi and librenms discovery / topology mapping purposes later.

Monitoring/security 

  • SSH key auth only
  • No account other than root
  • Set a strong root password (if you need to IPMI in , rescue mode etc)
  • Install fail2ban
  • Setup logwatch

ISOs/OVA you should have:

[email protected]:~/iso# ls -1
CentOS-7-x86_64-Minimal-1908.iso
librenms-ubuntu-18.04-amd64.ova
NeDian17C-Vmware.ova
openmanage_enterprise_kvm_format_3.3.1.zip
OPNsense-20.1-OpenSSL-vga-amd64.img.bz2
ubuntu-18.04.4-live-server-amd64.iso
UCS-Installation-amd64.iso
[email protected]:~/iso#

What I’m up to this long weekend – enterprise in a box build

I am doing a new server build. Preparing the full production stack to take TSYS live. A full HA VM stack.

I’ll do individual posts on each major piece and link them below.

The pieces involved:

  • Bare metal OS : Ubuntu 18.04 + KVM + ZFS (on encrypted block devices) + Netdata 
  • WAN Edge and core (VPN/BGP) routers: Opnsense 
  • Web servers/mysql servers/nfs servers : Ubuntu 18.04 
  • Monitoring: librenms OVA (Ubuntu 18.04 based) and Nedi.ch OVA (Debian based)
  • Utility server: Ubuntu 18.04 
  • ERP (ODOO) server : (legacy VM from old stack) Ubuntu 16.04

[email protected]:~# virsh list –all|grep running|awk ‘{print $2}’|sort
discourse
esign
fnfsrv
ovh-corertr-01
ovh-corertr-02
ovh-db-01
ovh-db-02
ovh-dc-01
ovh-dc-02
ovh-util-01
ovh-wanrtr-01
ovh-wanrtr-02
ovh-wwwcontent-01
ovh-wwwcontent-02
ovh-wwwsrv-01
ovh-wwwsrv-02

I couldn’t get the OVA to import into existing zvol (i’ll do a dedicated storage post shortly) so I just built ovh-util-01 (and i can lift/shift my existing libre/openvas/rundeck install) and add nedi on.