Hi all.

It’s been awhile since I’ve posted. Been busy with TSYS.

I returned to Austin in March and was on short list to re-deploy. The previous six months I’d been on the road with my Precision. It’s a great machine. Incredibly powerful. Terrible for lugging around.

I pressed the giant hunk of compute into service as pfv-vm2.turnsys.net (hosting lultix.turnsys.net my”laptop” (as most would call it) (where I hack on stuff) (see docs on it at https://github.com/charlesnw1/boilerplate/tree/master/notes/CNWWorkstation ).

I had purchased an iPad mini 5th gen in June 2019 or so. Used it to watch Netflix and such during deployment. Inspired by many “I ditched the lap space heater for iPad” blog posts, I decided todo the same.

Two use cases have emerged since March

1. Anchored/docked usage. I utilize the Apple Lightning to DVI adapter ( https://www.apple.com/shop/product/MD826AM/A/lightning-digital-av-adapter?afid=p238%7CpYD90RSt-dc_mtid_1870765e38482_pcrid_12352129613_&cid=aos-us-kwbi-pla-btb-slid–product-MD826AM/A) with a 24″ Dell monitor.
2. Road Warrior. I have a kickstand case, an Anker USB battery pack , life is good in the bars / coffee shops etc of Austin and the surrounding areas. I’m writing this post from one of my usual hang out.

In docked mode, I’m living 99% in RDP session on Lultix on the big screen. With my virtual desktops and all that goodness. Using the Jump app. It’s just like being on any other laptop I’ve used in the past I will (soon) finish up my blog post on the desktop setup.

In road warrior mode, I spend limited time in RDP. It’s quite workable on the small screen, and I’ve used it heavily, don’t get me wrong. It’s just far more useful on the big screen.

Road warrior is where I use: (among many other apps)

• Working copy (get my git fix)
• Buffer (a great text editor )
• Blink (CLI/SSH/telnet) (tried many, this one just completely wins)
• Firefox
• WordPress app
• MS Office
• (others I’ve forgotten)

on the iPad to work on a variety of deliverables.

In both docked/road warrior mode I heavily use:

• Apple Pencil First Gen
• Magic Mouse Second gen (configured for right click lol)
• iHome keyboard (yet to get the tilde / back tick working…. need to fix soon)

I have switched to a new desktop setup . An iPad mini and a Dell Venue 11 pro (dual booting Windows 10/Ubuntu/Kali).

Utilizing Nextcloud for sync , Bitwarden (to my own self hosted bitwarden_rs server) for passwords, Firefox for browsing (using Chrome not signed in for sysadmin portals, Firefox for general net browsing). No account in Firefox, planning to setup the private firefox sync server. I keep iCloud turned off on iPad/iPhone for security reasons.

More details later (and I will finish up my previous posts on enterprise data center build out).

(this is in progress “livestream”. Header will be removed when completed).

I had OVH provision a /28 to the new dedi.

IP allocation:

1. ovh-preproduction -01 wan ip
2. ovh-preproduction-02 wan ip
3. ovh-wanrtr-01 wan ip
4. ovh-wanrtr-02 wan ip
5. ovh-wanrtr wan float
6. external web general (cloudflare origin) (route through opnsense dmz farm)
7. external fnf (direct route to VM on brWan)
8. external esign (direct route to VM on brWan)
9. external discourse (direct route to VM on brWan)

This leaves me 4 spares (/32 using dedi IP .254 as gateway) makes the entire block usable. 🙂

All stacks need a proper foundation or it all comes tumbling down.

Here’s the base:

Provider:

OVH (Canda data center)

TSYS has been with OVH (since the beginning) for several years and been exceptionally pleased with the service they provide. Rock solid, highly economical, fast provisioning.

Product choice:
RISE-2 – Intel Xeon D-1540 – RAM 64GB LEG – 2x 2TB SATA Datacenter Class Soft RAID (at 75.09 a month with two year commit, paid monthly) which is down from 80.99 with no commit.

OS Choice:

Ubuntu 18.04 . TSYS is a Ubuntu/debian shop (with a couple Centos appliance VM for some business applications like e-sign) .

OS configuration:

Packages

apt-get -y install qemu-kvm virt-manager fail2ban postfix glances htop tcpdump dstat mailutils molly-guard bridge-utils vlan openvswitch-switch build-essential autoconf automake libtool gawk alien fakeroot ksh zlib1g-dev uuid-dev libattr1-dev libblkid-dev libselinux-dev libudev-dev libacl1-dev libaio-dev libdevmapper-dev libssl-dev libelf-dev linux-headers-$(uname -r) python3 python3-dev python3-setuptools python3-cffi zsh openvswitch-switch-dpdk ladvd logwatch smartmontools lm-sensors snmpd haveged xfce4 xfce4-goodies xorg dbus-x11 x11-xserver-utils xrdp cpufrequtils chromium-browser

(configure postfix as internet site). It’s critical that the base system can send mail without any VM dependency.

xrdp/xfce will make your life MUCH easier when setting up virtual machines via virt-manager (x11 forwarding over ssh is horribly slow)

Setup netdata

( https://github.com/netdata/netdata ) This has been by far the most useful/detailed monitoring/alerting I’ve setup. It does require a fair amount of tweaking (I’m keeping the details of the tweaks under wraps , but am available for consulting at reasonable hourly rates to help optimize your netdata setup). It works out of the box but can definitely be a bit over verbose until you adjust it.

Setup hpn-ssh

( https://www.psc.edu/hpn-ssh ) I recommend replacing your system SSH with this. It’s so much faster for copying data etc.

Setup ZFS (I’m doing this build on 2/15/2019 and using zfs-0.8.3)

https://www.tonylykke.com/posts/2019/08/03/zfs-0.8.1-on-ubuntu-18.04/

https://zfsonlinux.org/

Drive layout

For the non data (this was done via OVH web installer, setup a mirrored 15G root partition)

root@ovh-dedi-01:~/iso# cat /etc/fstab

/dev/md3 / ext4 errors=remount-ro 0 1
/dev/md2 /boot ext4 errors=remount-ro 0 1
/dev/sda4 swap swap defaults 0 0
/dev/sdb4 swap swap defaults 0 0
proc /proc proc defaults 0 0
sysfs /sys sysfs defaults 0 0
UUID=89EC-EA13 /boot/efi vfat defaults 0 0

ZFS…

Create a partition on each hard drive (sda5 sdb5) in my case :

ovh-dedi-01# fdisk -l /dev/sda /dev/sdb
Disk /dev/sda: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EDAD6DE9-92E7-4CCE-8605-9E5724127D0D

Device Start End Sectors Size Type
/dev/sda1 2048 1048575 1046528 511M EFI System
/dev/sda2 1048576 2095103 1046528 511M Linux RAID
/dev/sda3 2095104 32813055 30717952 14.7G Linux RAID
/dev/sda4 32813056 33859583 1046528 511M Linux swap
/dev/sda5 33859584 3907029134 3873169551 1.8T Linux filesystem

Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: A04DBC0B-ADF9-4A01-8810-E30C18A0B2C9

Device Start End Sectors Size Type
/dev/sdb1 2048 1048575 1046528 511M EFI System
/dev/sdb2 1048576 2095103 1046528 511M Linux RAID
/dev/sdb3 2095104 32813055 30717952 14.7G Linux RAID
/dev/sdb4 32813056 33859583 1046528 511M Linux swap
/dev/sdb5 33859584 3907029134 3873169551 1.8T Linux filesystem
ovh-dedi-01#

It appears the packages for zfs that ship with 18.04.3 (as of 2/15/2020) don’t work (undefined symbols). So I built from source ( https://github.com/zfsonlinux/zfs/releases/download/zfs-0.8.3/zfs-0.8.3.tar.gz and doc at https://www.tonylykke.com/posts/2019/08/03/zfs-0.8.1-on-ubuntu-18.04/ ). I’ll skip over my ZFS setup (hire me if you want details and/or a beautiful production setup in your environment).

Network

Here be dragons…

WAN Setup:

So OVH 18.04 template sets up the network using systemd and injects your WAN IP (this allows the box to be on net and we can do basic setup). However it doesn’t let us do fancy pants network stuff.

root@ovh-dedi-01:/etc/systemd/network# ls
50-default.network 50-public-interface.link
root@ovh-dedi-01:/etc/systemd/network# cat 50-default.network

This file sets the IP configuration of the primary (public) network device.

You can also see this as “OSI Layer 3” config.

It was created by the OVH installer, please be careful with modifications.

Documentation: man systemd.network or https://www.freedesktop.org/software/systemd/man/systemd.network.html

[Match]
MACAddress=0c:c4:7a:77:1e:6e

[Network]
Description=network interface on public network, with default route
DHCP=no
Address=158.69.53.198/24
Gateway=158.69.53.254
IPv6AcceptRA=no
NTP=ntp.ovh.net
DNS=127.0.0.1
DNS=213.186.33.99
DNS=2001:41d0:3:163::1
Gateway=2607:5300:0060:82ff:ff:ff:ff:ff

[Address]
Address=2607:5300:0060:82c6::/64

[Route]
Destination=2607:5300:0060:82ff:ff:ff:ff:ff
Scope=link
root@ovh-dedi-01:/etc/systemd/network#

So follow the steps from
https://www.naut.ca/blog/2018/12/12/disabling-systemd-networking/

and setup a wan bridge:

Setup /etc/network/interfaces as such (check with ip a to see what interface you should bridge):

auto lo
iface lo inet loopback

auto eno3
iface eno3 inet manual

auto brWan
iface brWan inet static
address 158.69.53.198
netmask 255.255.255.0
network 158.69.53.0
gateway 158.69.53.254
bridge_ports eno3
bridge_stp off
bridge_fd 0
bridge_hello 2
bridge_maxage 12

Reboot and ensure you can still ssh to the dedi. We’ll setup the VLANs etc in the next step. Change one thing at a time. 🙂

Upon reboot you should have:

1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 0c:c4:7a:77:1c:16 brd ff:ff:ff:ff:ff:ff
3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 0c:c4:7a:77:1c:17 brd ff:ff:ff:ff:ff:ff
4: eno3: mtu 1500 qdisc mq master brWan state UP group default qlen 1000
link/ether 0c:c4:7a:77:1e:6e brd ff:ff:ff:ff:ff:ff
5: eno4: mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 0c:c4:7a:77:1e:6f brd ff:ff:ff:ff:ff:ff
6: brWan: mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 0c:c4:7a:77:1e:6e brd ff:ff:ff:ff:ff:ff
inet 158.69.53.198/24 brd 158.69.53.255 scope global brWan
valid_lft forever preferred_lft forever
inet6 fe80::ec4:7aff:fe77:1e6e/64 scope link
valid_lft forever preferred_lft forever
7: virbr0: mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:29:b8:fd brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
8: virbr0-nic: mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
link/ether 52:54:00:29:b8:fd brd ff:ff:ff:ff:ff:ff
ovh-dedi-01# brctl show
bridge name bridge id STP enabled interfaces
brWan 8000.0cc47a771e6e no eno3
virbr0 8000.52540029b8fd yes virbr0-nic

ovh-dedi-01# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 158.69.53.254 0.0.0.0 UG 0 0 0 brWan
158.69.53.0 0.0.0.0 255.255.255.0 U 0 0 0 brWan
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
ovh-dedi-01#

VLAN/LAN setup:

This is where things get pretty complicated, but also incredibly powerful. It allows for “enterprise LAN in a box” which eliminates need for anything except a single server (no switches, firewalls etc needed).

OpenVswitch is the core component.

(I’m keeping this mostly under wraps as it took a decent amount of time/effort to get correct. Hire me to implement it for you if you want all the details). Here’s a rough outline and you can fill in the blanks:

auto <bridge name>
allow-ovs <bridge-name>
iface <bridge-name> inet static
address <bare metal ip>
netmask <bare metal mask>
ovs_type OVSBridge
ovs_ports <vlan tag>

allow-mgmt <vlan tag>
iface <vlan tag> inet manual
ovs_bridge <bridge-name>
ovs_type OVSIntPort
ovs_options tag=<vlan tag>

You need todo this for all the vlans you want. I assign an IP to all the interfaces for monitoring and other purposes. This is optional and they can be strictly layer 2 “ports”.

Setup ladvd to announce on the ports (this is useful for nedi and librenms discovery / topology mapping purposes later.

Monitoring/security 

• SSH key auth only
• No account other than root
• Set a strong root password (if you need to IPMI in , rescue mode etc)
• Install fail2ban
• Setup logwatch

ISOs/OVA you should have:

root@ovh-dedi-01:~/iso# ls -1
CentOS-7-x86_64-Minimal-1908.iso
librenms-ubuntu-18.04-amd64.ova
NeDian17C-Vmware.ova
openmanage_enterprise_kvm_format_3.3.1.zip
OPNsense-20.1-OpenSSL-vga-amd64.img.bz2
ubuntu-18.04.4-live-server-amd64.iso
UCS-Installation-amd64.iso
root@ovh-dedi-01:~/iso#

I am doing a new server build. Preparing the full production stack to take TSYS live. A full HA VM stack.

I’ll do individual posts on each major piece and link them below.

The pieces involved:

• Bare metal OS : Ubuntu 18.04 + KVM + ZFS (on encrypted block devices) + Netdata 
• WAN Edge and core (VPN/BGP) routers: Opnsense 
• Web servers/mysql servers/nfs servers : Ubuntu 18.04 
• Monitoring: librenms OVA (Ubuntu 18.04 based) and Nedi.ch OVA (Debian based)
• Utility server: Ubuntu 18.04 
• ERP (ODOO) server : (legacy VM from old stack) Ubuntu 16.04

oot@ovh-dedi-01:~# virsh list –all|grep running|awk ‘{print $2}’|sort
discourse
esign
fnfsrv
ovh-corertr-01
ovh-corertr-02
ovh-db-01
ovh-db-02
ovh-dc-01
ovh-dc-02
ovh-util-01
ovh-wanrtr-01
ovh-wanrtr-02
ovh-wwwcontent-01
ovh-wwwcontent-02
ovh-wwwsrv-01
ovh-wwwsrv-02

I couldn’t get the OVA to import into existing zvol (i’ll do a dedicated storage post shortly) so I just built ovh-util-01 (and i can lift/shift my existing libre/openvas/rundeck install) and add nedi on.

I’ve finally gotten my bookmarks freed from Chrome and put online. I like how it turned the folders into tags.

Here’s an example, my links on standby power:

https://bookmarks.knownelement.com/?searchtags=power

Overall root is here:

https://bookmarks.knownelement.com/

Over the last few months , the TSYS leadership team has come into their own. I’ve been very pleased with how everything has come together.

I’ve realized that it is one of the key aspects the BoD has been helping me with. They’ve been invaluable in providing perspective , context , stories of high and low performing teams / organizations , feedback on how I’m managing managers etc.

One key thing in delegation is you have to delegate both authority and responsibility. With the orbiter demo , I laid out a clear list of high level objectives (developed in concert with the engineering team, the BoD, outside advisors) , handed it over to the team and let go. Many organizations have the project management roll up to the top level. This is a recipe for disaster. It’s a big adjustment for the dev team to actually have static deliverables. Most of the time , devs don’t really get underway on something cause they’ll get yanked around by shifting priorities.

I have removed myself from visibility into the project boards and I muted the team category in discourse. I’m only an @ mention away if I’m really needed , and my CTO and COO have a great relationship , so my COO can handle any day to day (procurement , site work like rebooting something etc).

I have learned quite a bit about management , and building effective teams over the last 7 years. I’ve made numerous mistakes , I’ve learned from them. Key is to truly delegate , be prepared for loss/mistakes/failure and to incorporate results from that into the next sprint. Deadlines are fraught with conflict. Rough guidelines and blocks of time and clear objectives are key.

This post (and most of my posts actually) is an outgrowth of a conversation with someone I am advising (a fellow CEO).

Over the last few years , TSYS has experimented with a number of management structures. Specifically with reporting structures and day to day operational management.

One key take away , is that startup (in particular) CEOs shouldn’t manage capacity ( execution , amount of work and output etc) . That should be delegated one layer down. For too long I tried to manage capacity and it was ineffective, as I was also trying to manage capability and product etc. Basically I brought in a CTO much later than I needed to. Once I stopped managing capacity and I focused on closing capability gaps , we’ve been able to move forward much faster and with way less energy. Of course a company still needs fantastic talent , and that’s where my exclusive focus has been . Identifying , recruiting for , closing key material capability gaps. That’s also a critical pre requisite for outside funding (which also I’m finally raising , much later than I should have). Self funding a startup the first few years is a trade off. However it’s allowed us to establish a very strong culture , be incredibly frugal and have our internal systems setup right (no “we will do this thing later , just focus on product). We also have achieved some revenue. So team/revenue being on a good trajectory puts us in control with fundraising.

http://siliconhillslawyer.com/

https://www.chinalawblog.com/

https://writing.kemitchell.com/

I’ve got various sources for more specialized material (ranging from capital and entity formation to regulatory expertise).

Remember most law is highly uniform , scales across situations etc. You very often do not actually need a lawyer , you always need todo your research though. Apply critical thinking, use the various ABA etc resources that are on the Net.

Oh Docracy.com has a great collection of material as well.

I’ve been the CEO of an emerging LLC business for three years now. I want to share some of my journey and write a job description , this will be what I ask my board to evaluate me against.

Key deliverables for an early stage CEO:

1) formation issues. Deep thinking on entity structuring, balancing overhead , risk and execution.

2) building the initial board for submission to the LLC members for approval, growing it as needed over time and milestone achievement , changing the composition as warranted. Effectively utilize the board.

3) Putting the team together. Technology , ops, sales/marketing. In roughly that order .

4) Until you’ve achieved step 3, do those roles effectively. Found with no less than three people (one in each role), no more than five.

5) Fund the business. Be incredibly frugal , spend judiciously but invest where needed.

6) Set the culture, tone , character , tolerable behaviors etc.

The last three years saw us go from three untitled founders and a rough idea to :

1) Three solid lines of business

2) Four independent directors

3) CEO,CTO,CFOO

4) Director of r&d, director of swdev, director of hwdev, director of marketing (who also serves as the gm of a revenue generating line of business), director of business development

We’ve had four separations over that time period. Team members have changed roles , changed commitment levels etc. Change is constant , embrace it quickly and ride the wave.

We are now raising capital. Self funding and early revenue is a fantastic approach , however it has limitations. None of us draw a salary , we maintain primary income sources (“dayjob”) at various levels.

I have come to properly value and manage my time. I deploy it where it will generate the most return. I’ve delegated very heavily (essentially to the point of abdication ala buffet) and focus my energy on filling in final recruitment gaps , partner relationship building / vendor sourcing and capital raising.