The Maxwell Backstory


#1

Maybe John Lennon once said “Life is what happens when you’re making other plans.” Maybe. Maxwell is what happened when I was making other plans.

I was working at Zendesk, fresh off a successful project and ready to try something with a dubious success rate. I guess it’s how I learn: throw myself at the hardest problem I can find, fail, learn in the process. Zendesk had a boondoggle of a feature - the kind you regret building, the kind you know you couldn’t have really survived without - where customers had a build-your-own-SQL engine. I thought I could pull off an adaptive caching engine to fix some of the worst queries our customers had designed. The premise; you throw us any old terrible query, doesn’t matter if it’s indexed properly, we’ll cache it after we notice it’s slow, and then maintain a cache for it forever.

The first component I’d need was what I was calling the “firehose”; a stream of updates off the main database that we could process to keep caches in sync. A little bit of tinkering with the mysql binlog and a lot of hubris later and I had set out to build Maxwell. I finished an alpha version a few months in, launched, and then utterly solved all my scalability problems with a wonderful adaptive caching engine.

Or, no. That clearly didn’t happen; the adaptive caching project launched, but it just wasn’t a clear vision of a product in the way Maxwell was. Now I know what that project should have been, but, eh. Hindsight 20/20 and all that, maybe one day when I’m a much better engineer I’ll try to build that one again. What did happen was that the side benefits of Maxwell began to be more useful than I had anticipated. A team doing notifications picked it up for some edge-level triggering to send SMS messages. The search-index team poked their nose in. An edge team picked up the stream for some cache expiration.

I wrote a little streaming thingy that my company found useful, and the world found useful, and who knows how long the thingy will stay reasonably relevant for, but I’m grateful for the opportunity to contribute in whatever way I can to this mess of a world. I hope Maxwell solves a little problem for you too.