(this is in progress “livestream”. Header will be removed when completed).

I had OVH provision a /28 to the new dedi.

IP allocation:

1. ovh-preproduction -01 wan ip
2. ovh-preproduction-02 wan ip
3. ovh-wanrtr-01 wan ip
4. ovh-wanrtr-02 wan ip
5. ovh-wanrtr wan float
6. external web general (cloudflare origin) (route through opnsense dmz farm)
7. external fnf (direct route to VM on brWan)
8. external esign (direct route to VM on brWan)
9. external discourse (direct route to VM on brWan)

This leaves me 4 spares (/32 using dedi IP .254 as gateway) makes the entire block usable. 🙂

All stacks need a proper foundation or it all comes tumbling down.

Here’s the base:

Provider:

OVH (Canda data center)

TSYS has been with OVH (since the beginning) for several years and been exceptionally pleased with the service they provide. Rock solid, highly economical, fast provisioning.

Product choice:
RISE-2 – Intel Xeon D-1540 – RAM 64GB LEG – 2x 2TB SATA Datacenter Class Soft RAID (at 75.09 a month with two year commit, paid monthly) which is down from 80.99 with no commit.

OS Choice:

Ubuntu 18.04 . TSYS is a Ubuntu/debian shop (with a couple Centos appliance VM for some business applications like e-sign) .

OS configuration:

Packages

apt-get -y install qemu-kvm virt-manager fail2ban postfix glances htop tcpdump dstat mailutils molly-guard bridge-utils vlan openvswitch-switch build-essential autoconf automake libtool gawk alien fakeroot ksh zlib1g-dev uuid-dev libattr1-dev libblkid-dev libselinux-dev libudev-dev libacl1-dev libaio-dev libdevmapper-dev libssl-dev libelf-dev linux-headers-$(uname -r) python3 python3-dev python3-setuptools python3-cffi zsh openvswitch-switch-dpdk ladvd logwatch smartmontools lm-sensors snmpd haveged xfce4 xfce4-goodies xorg dbus-x11 x11-xserver-utils xrdp cpufrequtils chromium-browser

(configure postfix as internet site). It’s critical that the base system can send mail without any VM dependency.

xrdp/xfce will make your life MUCH easier when setting up virtual machines via virt-manager (x11 forwarding over ssh is horribly slow)

Setup netdata

( https://github.com/netdata/netdata ) This has been by far the most useful/detailed monitoring/alerting I’ve setup. It does require a fair amount of tweaking (I’m keeping the details of the tweaks under wraps , but am available for consulting at reasonable hourly rates to help optimize your netdata setup). It works out of the box but can definitely be a bit over verbose until you adjust it.

Setup hpn-ssh

( https://www.psc.edu/hpn-ssh ) I recommend replacing your system SSH with this. It’s so much faster for copying data etc.

Setup ZFS (I’m doing this build on 2/15/2019 and using zfs-0.8.3)

https://www.tonylykke.com/posts/2019/08/03/zfs-0.8.1-on-ubuntu-18.04/

https://zfsonlinux.org/

Drive layout

For the non data (this was done via OVH web installer, setup a mirrored 15G root partition)

[email protected]:~/iso# cat /etc/fstab

/dev/md3 / ext4 errors=remount-ro 0 1
/dev/md2 /boot ext4 errors=remount-ro 0 1
/dev/sda4 swap swap defaults 0 0
/dev/sdb4 swap swap defaults 0 0
proc /proc proc defaults 0 0
sysfs /sys sysfs defaults 0 0
UUID=89EC-EA13 /boot/efi vfat defaults 0 0

ZFS…

Create a partition on each hard drive (sda5 sdb5) in my case :

ovh-dedi-01# fdisk -l /dev/sda /dev/sdb
Disk /dev/sda: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EDAD6DE9-92E7-4CCE-8605-9E5724127D0D

Device Start End Sectors Size Type
/dev/sda1 2048 1048575 1046528 511M EFI System
/dev/sda2 1048576 2095103 1046528 511M Linux RAID
/dev/sda3 2095104 32813055 30717952 14.7G Linux RAID
/dev/sda4 32813056 33859583 1046528 511M Linux swap
/dev/sda5 33859584 3907029134 3873169551 1.8T Linux filesystem

Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: A04DBC0B-ADF9-4A01-8810-E30C18A0B2C9

Device Start End Sectors Size Type
/dev/sdb1 2048 1048575 1046528 511M EFI System
/dev/sdb2 1048576 2095103 1046528 511M Linux RAID
/dev/sdb3 2095104 32813055 30717952 14.7G Linux RAID
/dev/sdb4 32813056 33859583 1046528 511M Linux swap
/dev/sdb5 33859584 3907029134 3873169551 1.8T Linux filesystem
ovh-dedi-01#

It appears the packages for zfs that ship with 18.04.3 (as of 2/15/2020) don’t work (undefined symbols). So I built from source ( https://github.com/zfsonlinux/zfs/releases/download/zfs-0.8.3/zfs-0.8.3.tar.gz and doc at https://www.tonylykke.com/posts/2019/08/03/zfs-0.8.1-on-ubuntu-18.04/ ). I’ll skip over my ZFS setup (hire me if you want details and/or a beautiful production setup in your environment).

Network

Here be dragons…

WAN Setup:

So OVH 18.04 template sets up the network using systemd and injects your WAN IP (this allows the box to be on net and we can do basic setup). However it doesn’t let us do fancy pants network stuff.

[email protected]:/etc/systemd/network# ls
50-default.network 50-public-interface.link
[email protected]:/etc/systemd/network# cat 50-default.network

This file sets the IP configuration of the primary (public) network device.

You can also see this as “OSI Layer 3” config.

It was created by the OVH installer, please be careful with modifications.

Documentation: man systemd.network or https://www.freedesktop.org/software/systemd/man/systemd.network.html

[Match]
MACAddress=0c:c4:7a:77:1e:6e

[Network]
Description=network interface on public network, with default route
DHCP=no
Address=158.69.53.198/24
Gateway=158.69.53.254
IPv6AcceptRA=no
NTP=ntp.ovh.net
DNS=127.0.0.1
DNS=213.186.33.99
DNS=2001:41d0:3:163::1
Gateway=2607:5300:0060:82ff:ff:ff:ff:ff

[Address]
Address=2607:5300:0060:82c6::/64

[Route]
Destination=2607:5300:0060:82ff:ff:ff:ff:ff
Scope=link
[email protected]:/etc/systemd/network#

So follow the steps from
https://www.naut.ca/blog/2018/12/12/disabling-systemd-networking/

and setup a wan bridge:

Setup /etc/network/interfaces as such (check with ip a to see what interface you should bridge):

auto lo
iface lo inet loopback

auto eno3
iface eno3 inet manual

auto brWan
iface brWan inet static
address 158.69.53.198
netmask 255.255.255.0
network 158.69.53.0
gateway 158.69.53.254
bridge_ports eno3
bridge_stp off
bridge_fd 0
bridge_hello 2
bridge_maxage 12

Reboot and ensure you can still ssh to the dedi. We’ll setup the VLANs etc in the next step. Change one thing at a time. 🙂

Upon reboot you should have:

1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 0c:c4:7a:77:1c:16 brd ff:ff:ff:ff:ff:ff
3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 0c:c4:7a:77:1c:17 brd ff:ff:ff:ff:ff:ff
4: eno3: mtu 1500 qdisc mq master brWan state UP group default qlen 1000
link/ether 0c:c4:7a:77:1e:6e brd ff:ff:ff:ff:ff:ff
5: eno4: mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 0c:c4:7a:77:1e:6f brd ff:ff:ff:ff:ff:ff
6: brWan: mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 0c:c4:7a:77:1e:6e brd ff:ff:ff:ff:ff:ff
inet 158.69.53.198/24 brd 158.69.53.255 scope global brWan
valid_lft forever preferred_lft forever
inet6 fe80::ec4:7aff:fe77:1e6e/64 scope link
valid_lft forever preferred_lft forever
7: virbr0: mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:29:b8:fd brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
8: virbr0-nic: mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
link/ether 52:54:00:29:b8:fd brd ff:ff:ff:ff:ff:ff
ovh-dedi-01# brctl show
bridge name bridge id STP enabled interfaces
brWan 8000.0cc47a771e6e no eno3
virbr0 8000.52540029b8fd yes virbr0-nic

ovh-dedi-01# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 158.69.53.254 0.0.0.0 UG 0 0 0 brWan
158.69.53.0 0.0.0.0 255.255.255.0 U 0 0 0 brWan
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
ovh-dedi-01#

VLAN/LAN setup:

This is where things get pretty complicated, but also incredibly powerful. It allows for “enterprise LAN in a box” which eliminates need for anything except a single server (no switches, firewalls etc needed).

OpenVswitch is the core component.

(I’m keeping this mostly under wraps as it took a decent amount of time/effort to get correct. Hire me to implement it for you if you want all the details). Here’s a rough outline and you can fill in the blanks:

auto <bridge name>
allow-ovs <bridge-name>
iface <bridge-name> inet static
address <bare metal ip>
netmask <bare metal mask>
ovs_type OVSBridge
ovs_ports <vlan tag>

allow-mgmt <vlan tag>
iface <vlan tag> inet manual
ovs_bridge <bridge-name>
ovs_type OVSIntPort
ovs_options tag=<vlan tag>

You need todo this for all the vlans you want. I assign an IP to all the interfaces for monitoring and other purposes. This is optional and they can be strictly layer 2 “ports”.

Setup ladvd to announce on the ports (this is useful for nedi and librenms discovery / topology mapping purposes later.

Monitoring/security 

• SSH key auth only
• No account other than root
• Set a strong root password (if you need to IPMI in , rescue mode etc)
• Install fail2ban
• Setup logwatch

ISOs/OVA you should have:

[email protected]:~/iso# ls -1
CentOS-7-x86_64-Minimal-1908.iso
librenms-ubuntu-18.04-amd64.ova
NeDian17C-Vmware.ova
openmanage_enterprise_kvm_format_3.3.1.zip
OPNsense-20.1-OpenSSL-vga-amd64.img.bz2
ubuntu-18.04.4-live-server-amd64.iso
UCS-Installation-amd64.iso
[email protected]:~/iso#

I am doing a new server build. Preparing the full production stack to take TSYS live. A full HA VM stack.

I’ll do individual posts on each major piece and link them below.

The pieces involved:

• Bare metal OS : Ubuntu 18.04 + KVM + ZFS (on encrypted block devices) + Netdata 
• WAN Edge and core (VPN/BGP) routers: Opnsense 
• Web servers/mysql servers/nfs servers : Ubuntu 18.04 
• Monitoring: librenms OVA (Ubuntu 18.04 based) and Nedi.ch OVA (Debian based)
• Utility server: Ubuntu 18.04 
• ERP (ODOO) server : (legacy VM from old stack) Ubuntu 16.04

[email protected]:~# virsh list –all|grep running|awk ‘{print $2}’|sort
discourse
esign
fnfsrv
ovh-corertr-01
ovh-corertr-02
ovh-db-01
ovh-db-02
ovh-dc-01
ovh-dc-02
ovh-util-01
ovh-wanrtr-01
ovh-wanrtr-02
ovh-wwwcontent-01
ovh-wwwcontent-02
ovh-wwwsrv-01
ovh-wwwsrv-02

I couldn’t get the OVA to import into existing zvol (i’ll do a dedicated storage post shortly) so I just built ovh-util-01 (and i can lift/shift my existing libre/openvas/rundeck install) and add nedi on.

I’ve finally gotten my bookmarks freed from Chrome and put online. I like how it turned the folders into tags.

Here’s an example, my links on standby power:

https://bookmarks.knownelement.com/?searchtags=power

Overall root is here:

https://bookmarks.knownelement.com/

Over the last few months , the TSYS leadership team has come into their own. I’ve been very pleased with how everything has come together.

I’ve realized that it is one of the key aspects the BoD has been helping me with. They’ve been invaluable in providing perspective , context , stories of high and low performing teams / organizations , feedback on how I’m managing managers etc.

One key thing in delegation is you have to delegate both authority and responsibility. With the orbiter demo , I laid out a clear list of high level objectives (developed in concert with the engineering team, the BoD, outside advisors) , handed it over to the team and let go. Many organizations have the project management roll up to the top level. This is a recipe for disaster. It’s a big adjustment for the dev team to actually have static deliverables. Most of the time , devs don’t really get underway on something cause they’ll get yanked around by shifting priorities.

I have removed myself from visibility into the project boards and I muted the team category in discourse. I’m only an @ mention away if I’m really needed , and my CTO and COO have a great relationship , so my COO can handle any day to day (procurement , site work like rebooting something etc).

I have learned quite a bit about management , and building effective teams over the last 7 years. I’ve made numerous mistakes , I’ve learned from them. Key is to truly delegate , be prepared for loss/mistakes/failure and to incorporate results from that into the next sprint. Deadlines are fraught with conflict. Rough guidelines and blocks of time and clear objectives are key.

This post (and most of my posts actually) is an outgrowth of a conversation with someone I am advising (a fellow CEO).

Over the last few years , TSYS has experimented with a number of management structures. Specifically with reporting structures and day to day operational management.

One key take away , is that startup (in particular) CEOs shouldn’t manage capacity ( execution , amount of work and output etc) . That should be delegated one layer down. For too long I tried to manage capacity and it was ineffective, as I was also trying to manage capability and product etc. Basically I brought in a CTO much later than I needed to. Once I stopped managing capacity and I focused on closing capability gaps , we’ve been able to move forward much faster and with way less energy. Of course a company still needs fantastic talent , and that’s where my exclusive focus has been . Identifying , recruiting for , closing key material capability gaps. That’s also a critical pre requisite for outside funding (which also I’m finally raising , much later than I should have). Self funding a startup the first few years is a trade off. However it’s allowed us to establish a very strong culture , be incredibly frugal and have our internal systems setup right (no “we will do this thing later , just focus on product). We also have achieved some revenue. So team/revenue being on a good trajectory puts us in control with fundraising.

http://siliconhillslawyer.com/

https://www.chinalawblog.com/

https://writing.kemitchell.com/

I’ve got various sources for more specialized material (ranging from capital and entity formation to regulatory expertise).

Remember most law is highly uniform , scales across situations etc. You very often do not actually need a lawyer , you always need todo your research though. Apply critical thinking, use the various ABA etc resources that are on the Net.

Oh Docracy.com has a great collection of material as well.

I’ve been the CEO of an emerging LLC business for three years now. I want to share some of my journey and write a job description , this will be what I ask my board to evaluate me against.

Key deliverables for an early stage CEO:

1) formation issues. Deep thinking on entity structuring, balancing overhead , risk and execution.

2) building the initial board for submission to the LLC members for approval, growing it as needed over time and milestone achievement , changing the composition as warranted. Effectively utilize the board.

3) Putting the team together. Technology , ops, sales/marketing. In roughly that order .

4) Until you’ve achieved step 3, do those roles effectively. Found with no less than three people (one in each role), no more than five.

5) Fund the business. Be incredibly frugal , spend judiciously but invest where needed.

6) Set the culture, tone , character , tolerable behaviors etc.

The last three years saw us go from three untitled founders and a rough idea to :

1) Three solid lines of business

2) Four independent directors

3) CEO,CTO,CFOO

4) Director of r&d, director of swdev, director of hwdev, director of marketing (who also serves as the gm of a revenue generating line of business), director of business development

We’ve had four separations over that time period. Team members have changed roles , changed commitment levels etc. Change is constant , embrace it quickly and ride the wave.

We are now raising capital. Self funding and early revenue is a fantastic approach , however it has limitations. None of us draw a salary , we maintain primary income sources (“dayjob”) at various levels.

I have come to properly value and manage my time. I deploy it where it will generate the most return. I’ve delegated very heavily (essentially to the point of abdication ala buffet) and focus my energy on filling in final recruitment gaps , partner relationship building / vendor sourcing and capital raising.

I recently invested into the TSYS Electronics Lab. It’s now quite nicely out fitted. Here’s a list of the items in the lab as of 12/9/2018 

• Soldering Iron
• DC Power Supply
• USB Microscope
• Bitscope
• SMT/SMD Assorted BIts
• ESD Safe Straight Tweezers
• ESD Safe Curved Tweezers
• SMT Component Testing Tweezers
• PCB Ruler
• ReFlow Oven Kit

Here’s a few pictures of the lab: 

So you’ve decided to start a for profit, cash flow positive, self funded, jump started business in the great state of TX.  You’ve done your research and found that you want to start a series LLC to provide maximum flexibility and asset protection. 

This is the exact route I took with Turn Net Systems. We were formed in the last days of 2015. Three years later (almost to the day) I have learned quite a bit about forming,  administering, governing, operating an LLC. Yes those are four distinct functions. 

Some presumptions:

1. You are not immediately taking outside capital. You want to reserve that option for later. 
2. You are granting equity to founders, with desire to retain option to pay salary later
3. You want to run it as a real business , with a Board Of Directors , officers etc 
4. You aren’t afraid to armchair legal/accounting all the standard off the shelf bits. 
5. You are starting with ~10 total people , across directors and founders (split perhaps into officers (managers) and individual contributors. For reference as of 12/9/2018, TSYS has four directors, three officers (CEO/CFOO/CTO) and four individual contributors. Officers are authorized in the operating agreement to enter into contracts , they report to the board quarterly etc. Individual contributors have broad authority (including budgetary) but do not have ability to bind the company into contracts (and some other pro forma items). 

Here’s what I used to get started , with all the above in mind (80% of that was in place day 1, 20% evolved over 3 years)

Legal: 

https://www.lonestarlandlaw.com/

Buy his book: https://www.lonestarlandlaw.com/book-info.html

Taxes:

Buy the Nolo books

Use Quickbooks Online, neat.com, get TurboTax . Your taxes won’t be that complicated the first few years as you are starting up.  Keep good records, scan them to neat. Integrate neat to QBO. 

Here’s a collection of links I used when forming the LLC/drafting operating agreement etc. 

Entity Formation

Formation & Founder Issues

Form Your Austin Startup Yourself Before Hiring a Cheap Lawyer

Watch out for this trap when drafting Texas series LLC docs

http://www.baylor.edu/content/services/document.php/181957.pdf << a great article on what exactly is liability from a legal perspective. capital calls are given extensive treatment.

http://www.sos.state.tx.us/corp/forms/205_boc.pdf

http://www.lexisnexis.com/legalnewsroom/tax-law/b/federaltaxation/archive/2012/06/11/substantial-economic-effect.aspx << here be dragons. You don’t want to be subject to a “substantial economic test” it’s where laywers and IRS act like cats playing with mice before killing them.

http://www.swlaw.com/assets/pdf/news/2010/02/18/Series%20LLCs%20(BXS)_BaharSchippel.pdf << the professionals are nervous about series llc, but that is their job no?

http://apps.americanbar.org/buslaw/committees/CL690000pub/newsletter/200807/kean.pdf << lots of good tax info in here

http://thestartuptoolkit.com/blog/2013/02/equity-basics-vesting-cliffs-acceleration-and-exits/ << vesting basics

http://www.mosaichub.com/answers/question/how-to-create-a-vesting-schedule-for-a-company << vesting llc practical

http://www.brightjourney.com/q/actually-setup-vesting-agreement-co-founders-w-breaking-budget << more llc vesting stuff

http://www.baylor.edu/content/services/document.php/181957.pdf << withdrawl/expulsion of members. ties into vesting.

http://foundrs.com/ << equity calculator

https://www.clerky.com/yc-stock-plan-forms << equity vesting for yc companies

http://www.nolo.com/legal-encyclopedia/llc-operating-agreement-30232.html

Bizops:

http://firstround.com/review/Positioning-Your-Startup-is-Vital-Heres-How-to-Do-It-Right/

http://www.stmaryslawjournal.org/pdfs/Outenreath_Step12.pdf

http://comptroller.texas.gov/taxinfo/taxpubs/tx96_114.html

Here is a good collection of curated links on accounting:

https://bench.co/syllabus/bookkeeping/catch-up-bookkeeping-guide/

https://bench.co/syllabus/tax-deductions/small-business-tax-deductions/

https://www.irs.gov/publications/p535/ch01.html#en_US_2013_publink1000208623

http://blog.bench.co/how-to-go-paperless

https://www.odoo.com/documentation/functional/accounting.html

https://www.irs.gov/publications/p583/ar02.html#en_US_201408_publink1000269308

http://www.dwmbeancounter.com/tutorial/Tutorial.html

Most of the links were sourced from:

https://news.ycombinator.com/item?id=11239437