From the sky to the bedrock.
Fifteen years across the whole stack — AI agents, applications, cloud platforms, networks and the hardware underneath. I write up what I build, including the parts that went wrong.
What I’ve been building.
Four of the last few months’ projects. Every one of them has a write-up behind it rather than a claim.
A Browser Farm for Agents
Ten Chrome browsers running in Kubernetes, each with a real profile and a real login, so an AI agent can use the web the way a person does.
Every Log in One Place
Container logs, hypervisor syslog and firewall events from three sites, in one lake — and the sizing decision that keeps the loud feed from erasing the quiet one.
A Network Where People Cannot Reach Infrastructure
Three sites, a software-defined mesh, and a zone policy that removed a rule with 6.3 million hits on it.
Fifteen Pipelines Into One
A full microservice estate rebuilt onto a single trunk-based pipeline, taking a release from most of a working day to thirteen minutes.
Enterprise SaaS, media and IoT, agencies, small business, nonprofits.
- IoT & enterprise SaaSTSI
- Media appliancesNorth Shore Automation
- Agency platform & hostingLuminFire
- Lake healthLittle Sand Lake Area Association
- Heritage nonprofitSteiger Heritage Club
Six things I keep ending up in.
Each topic is a set of case studies, not a service page.
What I’m building right now.
Short posts from the build log as the work happens, and long-form case studies once it’s done.
Build log Live from Mastodon
@jacob on feed.jacobnollette.com
- View on Mastodon ↗
The takeaway: on a 1 gig line, the UniFi Dream Machine Pro was the bottleneck, not the ISP. A consumer VPN on most networks plus intrusion prevention on the backup VLANs was more than it could encrypt and inspect. One QoS rule now puts people first and lets backups have the rest. How the lab's main AZ does VPN needs rethinking, probably somewhere other than the router.
- View on Mastodon ↗
Then the uploads hit the network. WAN latency spiked to 1.9 s, and I blamed inbound traffic, since QoS can't shape what has already crossed the ISP link, so I capped B2 downloads. Wrong. The router's CPU was at 93–97% from VPN encryption plus IPS inspecting every backup byte. With the VPN gone and IPS off the server VLANs, download hit 1 Gbit/s at 4–8 ms latency. Pulled the cap the same night.
- View on Mastodon ↗
Rebuilt the backups today. Everything now goes from the cluster straight to B2 with restic — no more mirroring to an offsite box first. Cloud drives are backed up from read-only rclone mounts, so nothing gets staged on Ceph on the way. An 11 TiB archive tree became B2-only once its pinned snapshot passed a restic read check and all 297,369 file paths matched the manifest before deleting the local copy.
Case studies
What the problem was, what broke, and what fixed it.
The Routing Flag That Took Five Sites Down
A VPN option swallowed the container network on a shared web server. Five sites went down for twenty hours, and the guard that should have prevented it had been silently dead for two months.
Infrastructure · ReliabilityWhen the Failure Domain Was a Lie
Five storage pools were told to keep three copies of every write. They were keeping all three on the same machine.
Infrastructure · ReliabilityEvery Disk in the Organisation, on One Dashboard
Fifty-eight drives across four sites, one collector — written because the standard one reports no age at all for the drives that matter most.
Software Donkey builds the agents and apps, and teaches small teams to do it too.
A standing team of six AI agents runs real work every day. The studio’s first apps are in private testing.
Let’s talk.
Whether that’s a role, a project, or a second opinion on something that keeps breaking.