New desktop setup and taking ownership of data

I have switched to a new desktop setup . An iPad mini and a Dell Venue 11 pro (dual booting Windows 10/Ubuntu/Kali).

Utilizing Nextcloud for sync , Bitwarden (to my own self hosted bitwarden_rs server) for passwords, Firefox for browsing (using Chrome not signed in for sysadmin portals, Firefox for general net browsing). No account in Firefox, planning to setup the private firefox sync server. I keep iCloud turned off on iPad/iPhone for security reasons.

More details later (and I will finish up my previous posts on enterprise data center build out).

Enterprise networking in the box

(this is in progress “livestream”. Header will be removed when completed).

I had OVH provision a /28 to the new dedi.

IP allocation:

  1. ovh-preproduction -01 wan ip
  2. ovh-preproduction-02 wan ip
  3. ovh-wanrtr-01 wan ip
  4. ovh-wanrtr-02 wan ip
  5. ovh-wanrtr wan float
  6. external web general (cloudflare origin) (route through opnsense dmz farm)
  7. external fnf (direct route to VM on brWan)
  8. external esign (direct route to VM on brWan)
  9. external discourse (direct route to VM on brWan)

This leaves me 4 spares (/32 using dedi IP .254 as gateway) makes the entire block usable. 🙂

Bare Metal – The base of it all

All stacks need a proper foundation or it all comes tumbling down.

Here’s the base:


OVH (Canda data center)

TSYS has been with OVH (since the beginning) for several years and been exceptionally pleased with the service they provide. Rock solid, highly economical, fast provisioning.

Product choice:
RISE-2 – Intel Xeon D-1540 – RAM 64GB LEG – 2x 2TB SATA Datacenter Class Soft RAID (at 75.09 a month with two year commit, paid monthly) which is down from 80.99 with no commit.

OS Choice:

Ubuntu 18.04 . TSYS is a Ubuntu/debian shop (with a couple Centos appliance VM for some business applications like e-sign) .

OS configuration:


apt-get -y install qemu-kvm virt-manager fail2ban postfix glances htop tcpdump dstat mailutils molly-guard bridge-utils vlan openvswitch-switch build-essential autoconf automake libtool gawk alien fakeroot ksh zlib1g-dev uuid-dev libattr1-dev libblkid-dev libselinux-dev libudev-dev libacl1-dev libaio-dev libdevmapper-dev libssl-dev libelf-dev linux-headers-$(uname -r) python3 python3-dev python3-setuptools python3-cffi zsh openvswitch-switch-dpdk ladvd logwatch smartmontools lm-sensors snmpd haveged xfce4 xfce4-goodies xorg dbus-x11 x11-xserver-utils xrdp cpufrequtils chromium-browser

(configure postfix as internet site). It’s critical that the base system can send mail without any VM dependency.

xrdp/xfce will make your life MUCH easier when setting up virtual machines via virt-manager (x11 forwarding over ssh is horribly slow)

Setup netdata

( ) This has been by far the most useful/detailed monitoring/alerting I’ve setup. It does require a fair amount of tweaking (I’m keeping the details of the tweaks under wraps , but am available for consulting at reasonable hourly rates to help optimize your netdata setup). It works out of the box but can definitely be a bit over verbose until you adjust it.

Setup hpn-ssh

( ) I recommend replacing your system SSH with this. It’s so much faster for copying data etc.

Setup ZFS (I’m doing this build on 2/15/2019 and using zfs-0.8.3)

Drive layout

For the non data (this was done via OVH web installer, setup a mirrored 15G root partition)

[email protected]:~/iso# cat /etc/fstab

/dev/md3 / ext4 errors=remount-ro 0 1
/dev/md2 /boot ext4 errors=remount-ro 0 1
/dev/sda4 swap swap defaults 0 0
/dev/sdb4 swap swap defaults 0 0
proc /proc proc defaults 0 0
sysfs /sys sysfs defaults 0 0
UUID=89EC-EA13 /boot/efi vfat defaults 0 0


Create a partition on each hard drive (sda5 sdb5) in my case :

ovh-dedi-01# fdisk -l /dev/sda /dev/sdb
Disk /dev/sda: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EDAD6DE9-92E7-4CCE-8605-9E5724127D0D

Device Start End Sectors Size Type
/dev/sda1 2048 1048575 1046528 511M EFI System
/dev/sda2 1048576 2095103 1046528 511M Linux RAID
/dev/sda3 2095104 32813055 30717952 14.7G Linux RAID
/dev/sda4 32813056 33859583 1046528 511M Linux swap
/dev/sda5 33859584 3907029134 3873169551 1.8T Linux filesystem

Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: A04DBC0B-ADF9-4A01-8810-E30C18A0B2C9

Device Start End Sectors Size Type
/dev/sdb1 2048 1048575 1046528 511M EFI System
/dev/sdb2 1048576 2095103 1046528 511M Linux RAID
/dev/sdb3 2095104 32813055 30717952 14.7G Linux RAID
/dev/sdb4 32813056 33859583 1046528 511M Linux swap
/dev/sdb5 33859584 3907029134 3873169551 1.8T Linux filesystem

It appears the packages for zfs that ship with 18.04.3 (as of 2/15/2020) don’t work (undefined symbols). So I built from source ( and doc at ). I’ll skip over my ZFS setup (hire me if you want details and/or a beautiful production setup in your environment).


Here be dragons…

WAN Setup:

So OVH 18.04 template sets up the network using systemd and injects your WAN IP (this allows the box to be on net and we can do basic setup). However it doesn’t let us do fancy pants network stuff.

[email protected]:/etc/systemd/network# ls
[email protected]:/etc/systemd/network# cat

This file sets the IP configuration of the primary (public) network device.

You can also see this as “OSI Layer 3” config.

It was created by the OVH installer, please be careful with modifications.

Documentation: man or


Description=network interface on public network, with default route


[email protected]:/etc/systemd/network#

So follow the steps from

and setup a wan bridge:

Setup /etc/network/interfaces as such (check with ip a to see what interface you should bridge):

auto lo
iface lo inet loopback

auto eno3
iface eno3 inet manual

auto brWan
iface brWan inet static
bridge_ports eno3
bridge_stp off
bridge_fd 0
bridge_hello 2
bridge_maxage 12

Reboot and ensure you can still ssh to the dedi. We’ll setup the VLANs etc in the next step. Change one thing at a time. 🙂

Upon reboot you should have:

1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 0c:c4:7a:77:1c:16 brd ff:ff:ff:ff:ff:ff
3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 0c:c4:7a:77:1c:17 brd ff:ff:ff:ff:ff:ff
4: eno3: mtu 1500 qdisc mq master brWan state UP group default qlen 1000
link/ether 0c:c4:7a:77:1e:6e brd ff:ff:ff:ff:ff:ff
5: eno4: mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 0c:c4:7a:77:1e:6f brd ff:ff:ff:ff:ff:ff
6: brWan: mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 0c:c4:7a:77:1e:6e brd ff:ff:ff:ff:ff:ff
inet brd scope global brWan
valid_lft forever preferred_lft forever
inet6 fe80::ec4:7aff:fe77:1e6e/64 scope link
valid_lft forever preferred_lft forever
7: virbr0: mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:29:b8:fd brd ff:ff:ff:ff:ff:ff
inet brd scope global virbr0
valid_lft forever preferred_lft forever
8: virbr0-nic: mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
link/ether 52:54:00:29:b8:fd brd ff:ff:ff:ff:ff:ff
ovh-dedi-01# brctl show
bridge name bridge id STP enabled interfaces
brWan 8000.0cc47a771e6e no eno3
virbr0 8000.52540029b8fd yes virbr0-nic

ovh-dedi-01# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface UG 0 0 0 brWan U 0 0 0 brWan U 0 0 0 virbr0

VLAN/LAN setup:

This is where things get pretty complicated, but also incredibly powerful. It allows for “enterprise LAN in a box” which eliminates need for anything except a single server (no switches, firewalls etc needed).

OpenVswitch is the core component.

(I’m keeping this mostly under wraps as it took a decent amount of time/effort to get correct. Hire me to implement it for you if you want all the details). Here’s a rough outline and you can fill in the blanks:

auto <bridge name>
allow-ovs <bridge-name>
iface <bridge-name> inet static
address <bare metal ip>
netmask <bare metal mask>
ovs_type OVSBridge
ovs_ports <vlan tag>

allow-mgmt <vlan tag>
iface <vlan tag> inet manual
ovs_bridge <bridge-name>
ovs_type OVSIntPort
ovs_options tag=<vlan tag>

You need todo this for all the vlans you want. I assign an IP to all the interfaces for monitoring and other purposes. This is optional and they can be strictly layer 2 “ports”.

Setup ladvd to announce on the ports (this is useful for nedi and librenms discovery / topology mapping purposes later.


  • SSH key auth only
  • No account other than root
  • Set a strong root password (if you need to IPMI in , rescue mode etc)
  • Install fail2ban
  • Setup logwatch

ISOs/OVA you should have:

[email protected]:~/iso# ls -1
[email protected]:~/iso#

What I’m up to this long weekend – enterprise in a box build

I am doing a new server build. Preparing the full production stack to take TSYS live. A full HA VM stack.

I’ll do individual posts on each major piece and link them below.

The pieces involved:

  • Bare metal OS : Ubuntu 18.04 + KVM + ZFS (on encrypted block devices) + Netdata 
  • WAN Edge and core (VPN/BGP) routers: Opnsense 
  • Web servers/mysql servers/nfs servers : Ubuntu 18.04 
  • Monitoring: librenms OVA (Ubuntu 18.04 based) and OVA (Debian based)
  • Utility server: Ubuntu 18.04 
  • ERP (ODOO) server : (legacy VM from old stack) Ubuntu 16.04

[email protected]:~# virsh list –all|grep running|awk ‘{print $2}’|sort

I couldn’t get the OVA to import into existing zvol (i’ll do a dedicated storage post shortly) so I just built ovh-util-01 (and i can lift/shift my existing libre/openvas/rundeck install) and add nedi on.

Developing your direct reports – it takes a village

Over the last few months , the TSYS leadership team has come into their own. I’ve been very pleased with how everything has come together.

I’ve realized that it is one of the key aspects the BoD has been helping me with. They’ve been invaluable in providing perspective , context , stories of high and low performing teams / organizations , feedback on how I’m managing managers etc.

One key thing in delegation is you have to delegate both authority and responsibility. With the orbiter demo , I laid out a clear list of high level objectives (developed in concert with the engineering team, the BoD, outside advisors) , handed it over to the team and let go. Many organizations have the project management roll up to the top level. This is a recipe for disaster. It’s a big adjustment for the dev team to actually have static deliverables. Most of the time , devs don’t really get underway on something cause they’ll get yanked around by shifting priorities.

I have removed myself from visibility into the project boards and I muted the team category in discourse. I’m only an @ mention away if I’m really needed , and my CTO and COO have a great relationship , so my COO can handle any day to day (procurement , site work like rebooting something etc).

I have learned quite a bit about management , and building effective teams over the last 7 years. I’ve made numerous mistakes , I’ve learned from them. Key is to truly delegate , be prepared for loss/mistakes/failure and to incorporate results from that into the next sprint. Deadlines are fraught with conflict. Rough guidelines and blocks of time and clear objectives are key.

CEOs should manage capabilities not capacity

This post (and most of my posts actually) is an outgrowth of a conversation with someone I am advising (a fellow CEO).

Over the last few years , TSYS has experimented with a number of management structures. Specifically with reporting structures and day to day operational management.

One key take away , is that startup (in particular) CEOs shouldn’t manage capacity ( execution , amount of work and output etc) . That should be delegated one layer down. For too long I tried to manage capacity and it was ineffective, as I was also trying to manage capability and product etc. Basically I brought in a CTO much later than I needed to. Once I stopped managing capacity and I focused on closing capability gaps , we’ve been able to move forward much faster and with way less energy. Of course a company still needs fantastic talent , and that’s where my exclusive focus has been . Identifying , recruiting for , closing key material capability gaps. That’s also a critical pre requisite for outside funding (which also I’m finally raising , much later than I should have). Self funding a startup the first few years is a trade off. However it’s allowed us to establish a very strong culture , be incredibly frugal and have our internal systems setup right (no “we will do this thing later , just focus on product). We also have achieved some revenue. So team/revenue being on a good trajectory puts us in control with fundraising.

Some legal resources I consume

I’ve got various sources for more specialized material (ranging from capital and entity formation to regulatory expertise).

Remember most law is highly uniform , scales across situations etc. You very often do not actually need a lawyer , you always need todo your research though. Apply critical thinking, use the various ABA etc resources that are on the Net.

Oh has a great collection of material as well.

Self funded , sustainable business (pre profitability )CEO job description

I’ve been the CEO of an emerging LLC business for three years now. I want to share some of my journey and write a job description , this will be what I ask my board to evaluate me against.

Key deliverables for an early stage CEO:

1) formation issues. Deep thinking on entity structuring, balancing overhead , risk and execution.

2) building the initial board for submission to the LLC members for approval, growing it as needed over time and milestone achievement , changing the composition as warranted. Effectively utilize the board.

3) Putting the team together. Technology , ops, sales/marketing. In roughly that order .

4) Until you’ve achieved step 3, do those roles effectively. Found with no less than three people (one in each role), no more than five.

5) Fund the business. Be incredibly frugal , spend judiciously but invest where needed.

6) Set the culture, tone , character , tolerable behaviors etc.

The last three years saw us go from three untitled founders and a rough idea to :

1) Three solid lines of business

2) Four independent directors


4) Director of r&d, director of swdev, director of hwdev, director of marketing (who also serves as the gm of a revenue generating line of business), director of business development

We’ve had four separations over that time period. Team members have changed roles , changed commitment levels etc. Change is constant , embrace it quickly and ride the wave.

We are now raising capital. Self funding and early revenue is a fantastic approach , however it has limitations. None of us draw a salary , we maintain primary income sources (“dayjob”) at various levels.

I have come to properly value and manage my time. I deploy it where it will generate the most return. I’ve delegated very heavily (essentially to the point of abdication ala buffet) and focus my energy on filling in final recruitment gaps , partner relationship building / vendor sourcing and capital raising.