Under the Hood: Kill Bill

Billing & payments

This is the world Stéphane and I have been living in for well over 10 years now, ever since we started Kill Bill, the open-source subscription billing and payments platform.

During our journey, we've helped hundreds of companies leverage our technology to unlock new pricing strategies across a variety of industries: e-commerce, telecommunications, healthcare, insurance, etc. Whether you've checked the history of that used car you're thinking of buying, insured your latest mobile device, or even bought something on Groupon, it's quite likely you've paid for something at some point through Kill Bill.

Chapter 1: A Formal Introduction

At a high level, Kill Bill is an open-source programmable platform that enables you to build custom billing solutions. That's a mouthful, to say the least, but think of it as building blocks to implement your own billing workflows.

Similar to the Unix philosophy of composability over monolithic design, Kill Bill provides APIs to create various pricing models, manage subscriptions, generate invoices, integrate with various payment gateways, etc. Whether you are selling access to a digital service for $10 a month or selling a multi-tier cell-phone data plan, Kill Bill provides the primitives to do it so you don't have to start from scratch.

Chapter 2: Building the Next-Gen Cloud Billing Engine

After going through various implementations over the years, we thought we had seen it all. Still, when the Equinix Metal team asked us to join and help build the next generation of their cloud billing engine, we assumed it would be yet another day at the Kill Bill office. Boy, were we wrong!

Whether it's because of the scale, the advanced capabilities of the Metal pricing engine, or the nuanced requirements of big enterprises going digital, the project has turned out to be a real challenge. Happily, that’s exactly the way we like it.

If you think about it, billing for cloud computing resources seems, at first glance, is quite simple. Let's say I deploy a Quake III Arena server for 6 hours for my Saturday night LAN party. If the price is $1/hour, my bill at the end of the month will be $6. Simple, right?

In practice, however, customers deploy all sorts of special hardware, which all have different price points. They can also get discounts by reserving specific hardware, or by bidding on spare server capacity using our server features. Then, you add software licenses (e.g. Windows Server), storage arrays (e.g. Pure Storage), custom IP blocks, and custom region-based pricing for bandwidth that egresses our network in various regions. Now, imagine that matrix of use-cases in the context of a DevOps Automation pipeline that uses Terraform to launch hundreds of servers every hour.

Needless to say, apparently simple “usage” calculations can become complicated very quickly.

Chapter 3: Solving the Complex Problems

Besides coming up with the right numbers at the end of the month, we also need to deliver the billing experience that large enterprises expect. Our biggest customers consume Metal across a variety of sub-departments and business units, which don't always share the same budgets, purchase orders, or legal agreements. Acme, Inc. might have a dozen bills each month from Equinix, and each one of them could have different payment terms, discount rates, payment methods, or PO numbers.

Starting to appreciate the complexity? As my friend Sandeep Jain put it well, Quote-to-Cash (Q2C) has Become Exceptionally Hard.

Thankfully, Kill Bill already solves a lot of these problems, so our job over the last year has been to make it speak Metal. Here's a high-level architectural diagram of our Equinix deployment.

Chapter 4: The Nuts and Bolts

Our entire stack runs on Kubernetes, with a replicated bare metal PostgreSQL database. Our CI/CD pipeline is made of GitHub Actions and ArgoCD. All of our services are instrumented via OpenTelemetry.

BAS is a simple service that keeps track of our billing accounts metadata (e.g. tax registrations). It was separated mostly for GDPR and CCPA compliance purposes.

The core of the platform is the Equinix Metal Billing Service (EMBS), a Go microservice that implements the Kill Bill plugin APIs required to deeply integrate with the various parts of the Equinix Metal ecosystem. Instead of generic plans and subscriptions, it's the connective tissue that allows Kill Bill to speak to servers, networks, and metros.

Internally, EMBS listens to events for billable operations from our Control Plane, gathers additional metered data from the various sites (e.g. bandwidth usage captured via sFlow), and presents this in a way that is consumable by the billing engine. Kill Bill in turn manages the subscriptions (recurring and on-demand), computes the final bill that is sent to our customers, handles the payment lifecycle, and fulfills our accounting and compliance obligations.

While EMBS is Equinix’s proprietary secret sauce, the broader open-source project already benefited from this work. For example, the Kill Bill Analytics plugin, which enables financial reporting (think MRR, churn, ACV), now supports a connection to Trino, the distributed SQL query engine for big data. Historically, we've seen companies aggregate data (financial, CRM, traditional analytics) into a central data mart, where queries were typically run.

Due to the highly distributed nature of Equinix (200+ data center locations and growing!), we needed a way to distribute and run real-time queries across all of our sites around the globe. Trino allows us to do just this.

Chapter 5: Billing at Scale

As you can imagine, I'm just scratching the surface here, and you can expect more to come over the next few quarters as we scale our platform, replace old bits (from the Packet days) and grow our business requirements and team.

The vision that we’re building for is all about scale: millions of things in hundreds of places with billions of data points that need to be collected, understood, and billed for accurately.

If you’re interested in our work at Equinix Metal, or interested in Kill Bill, I’d love to chat. We have an exciting roadmap ahead of us, and I look forward to sharing more details in the coming months.