Dear me, I have built a Kubernetes

Many moons ago, I came across a clever article Dear friend, you have built a Kubernetes, but eventually it drifted away in the tides of my mind. It was not until recently that I came across it again, but this time I realized that I became the subject. Here's how this played out...

Simple beginnings

It started with a really simple premise: I wanted to run Docker containers on something other than my local PC. Preferably a Linux server, always online. Sometimes it was an early version of this blog, other times a few development containers wired together when building a webapp.

This server became one of Intel's NUCs, a tiny laptop-grade computer without a dedicated GPU. It worked fantastic, and I would SSH in and start up various dev apps in the framework-of-the-day, usually some kind of NodeJS abomination.

Naturally, it couldn't stop there...

Monitoring

If I have a server, I might as well monitor its metrics right? What's the temperature like, is it under any load, what kind of containers are resource-intensive, etc...

I was going to need a fancy dashboard, and I decided that it was going to be Grafana. Of course, metrics were going to have to be collected somehow, so I installed node_exporter on the server and ran Prometheus + Grafana with Docker Compose.

This is when I caught my first whiff of what was to come. I didn't want to have to type http://arch.local:3000 just to pull up Grafana, and then be greeting by a certificate warning. It had to be fancy, something like https://grafana.local, and automatically routed to whatever host was running it...

More apps, new problems

In the meantime, I set up a couple more apps: Homebridge and unifi-docker.

The first was simple enough: I had a couple smarthome devices that weren't compatible with Apple's HomeKit that I still wanted to control from my phone.

unifi-docker was a little bit more convoluted. I was still hanging onto an old EdgeRouter 12, but bought a new Ubiquiti switch that I still wanted to manage with their new web interface. Ubiquiti's Unifi Controller normally runs on either their newer routers, a special Controller device, or, conveniently, in Docker.

Out of this, a problem arose. Despite all the configuration files living on my Mac, I was using Docker contexts to remotely apply them to my server. Of course, local files aren't visible to the remote Docker environment, so I needed them to be available remotely - but also locally for editing.

Syncthing

I thought I was really clever with this one. I had been using SCP and rsync to manually push files for a while. Every single time felt so primitive, I may as well be banging rocks together in a cave.

Syncthing, when installed to two computers, allows you to configure folders that you want to keep in sync. You can get really fancy with this, using glob patterns to include/exclude certain file/folder patterns, customize one-directional folder syncing, and even file versioning for deleted or changed files.

At this point, I now had three servers. Syncthing was such an elegant solution in my mind. All three servers had their own local Docker Compose folder, Syncthing installed, and I only had to wait ~2-3 seconds between file edits to have every configuration replicated.

It was a dream. But it didn't stop there.

Multicast DNS

I was getting pretty deep into this rabbithole. It was such a neat & elegant one, and I was tunneling so quickly that my field of vision couldn't pick up the sides closing in.

The next order of business was obvious: I had to stop addressing services by their hostname and port number. It was inelegant and ugly, and that red certificate warning got on my nerves.

Multicast DNS is a very convenient zero-configuration alternative to traditional DNS. Instead of having to assign a static IP to a host, the server itself broadcasts its hostname and address over IP multicast. A mental construct start slowly materializing - each service would have its own multicast name:

• grafana.local
• kibana.local
• homebridge.local

Individual services could even have their own per-host subdomains:

• node-exporter.server1.local
• node-exporter.server2.local
• node-exporter.server3.local

I even found an excellent article explaining how to do this with Avahi: Using mDNS aliases within your home network. This meant that regardless of what server a service was running on, the mDNS name would always point to the right one as long as the correct host was broadcasting it (I even envisioned a setup where a single host broadcast all mDNS names, even for non-local services).

However, at this point, I hit an intractable snag. While Avahi might allow me to broadcast multiple domains for a single host using avahi-publish, I was having serious problems trying to get it to resolve mDNS inside Docker containers. Since it was running on the host, there was no way to relay that information without extensive, per-container configuration at best.

Now systemd-resolved can interpret & resolve mDNS, even inside Docker due to how Docker picks up the host's /etc/resolv.conf. Unfortunately, broadcasting multiple domains is a feature that systemd does not currently support and doesn't seem likely to any time soon.

TLS

I treated the above as a temporary roadblock that would be quickly overcome and simultaneously starting figuring out the TLS component. This was primarily driven by two services I wanted to run: a local Docker Registry to cache container images, and an ELK stack.

Both required TLS configuration. I was determined to create and maintain an entire self-signed certificate authority chain so I wouldn't be reliant on external services that could go down.

I was so determined that I spent a couple months scouring the labyrinth of manpages, OpenSSL guides, and broken or severly outdated configurations to figure out how to follow modern TLS best practices. It got so far that I actually took the time to read multiple RFCs from top to bottom (the IETF RFC web browser has a fantastic design IMO).

Eventually, I even created a repository that contains a single OpenSSL configuration file (self-signed-ca-chain_openssl.cnf) that can be used to generate Root, Intermediate, and Leaf certificates. The README contains a terse explanation of how to use it (I planned to write a whole post on how to create and apply certs across multiple hosts). It should follow conventional modern standards that are broadly supported, but in the back of my mind, I'm not so confident I got it completely right.

Realization

You might think that after challenging the OpenSSL beast, I would realize the enormity and Sisyphean nature of this effort. But I still had one, last, "great" idea to implement.

If a host can broadcast multiple mDNS hostnames, then necessarily it will have to be correctly routed to its corresponding container. A perfect job for a reverse proxy. And not any reverse proxy, I wanted to cut my teeth with the new one on the block, Caddy.

You know what else a reverse proxy can do? Terminate TLS connections.

This was perfect. All the puzzle pieces were falling together in front of my eyes. I threw Caddy into another Docker Compose configuration, attached it to the host's network interface, and then built & attached network bridges to other Docker container services that were running on the same host. I started allocating local IPv4 and IPv6 subnets for various services. I started investigating step-ca to automatically rotate certificates.

The best part was that it worked. TLS worked. The mDNS component, while not completely solved, worked in a limited fashion. I started templating out the Caddy reverse proxy for each host, and then it hit me.

I had built an Ingress. This whole thing was Kubernetes.