Oops, We Forgot to Build a Managed Kubernetes Service!

The Backstory

When we started Packet and began thinking about the shape and lines of our product we quickly came up with two touchpoints that have served us well over the last five years:

When everyone else looked up the stack, we would look down.
Don’t sell anything on our public cloud that a customer can’t take ‘home’ with them.

We have especially relied on these guiding principles when times get tough...or just plain confusing! Having these in our back pocket has provided us with a lot more agency and confidence to make tough product decisions.

So why these two rules? It all comes back to our ambition as a company to lead the next big wave in the cloud, instead of simply chasing the current one. If you are going to be the best in the world at something (in our case, automating fundamental infrastructure) you have to arm your team with some tools that are pragmatic and clear. Being explicitly infrastructure-focused and aggressive to software lock-in are pretty good litmus tests!

So, What Does it Mean to be a Cloud?

In our collaboration space we have a channel called “Great Debates”. The channel was created by one of our SRE’s to make room for those big, thorny discussions that seem to bubble up at least once per month. Sometimes the posts/threads are related to fleshing out key architectural directions or proposing investments in the way we work and build.

But more often than not, the discussions center on what kind of cloud we want to be when we grow up. The premise is easy to guess, but it often goes something like this:

Do we want to offer a load balancer as a service?
Do we want to offer object storage as a service?
Why not make an easy firewall as a service?
How about Logs? Backups? Databases? Serverless? NFV? Openstack? VMware?
And more recently, managed Kubernetes.

For those that don’t know Packet well, it’s not a question of "if" but "when" we will start tackling all these services. This is because the cloud experience is now well defined by a relatively small group of reigning champions: AWS, Azure, and GCP. These product models, from the customer viewpoint, are all essentially consistent. The increasing range and depth of managed services is simply the way it is done, and it has stretched the definition of cloud from the early days of VM’s and storage to a laundry list of robust “as a service” solutions.

The thing is, our company isn’t focused on winning today’s public cloud battle. We’re too small, we’re too late and (frankly) it’s not our jam.

We’re passionate about the infrastructure and operational building blocks that underlie the hyperscale public clouds. Arming the next wave of technology-empowered Enterprises with those capabilities is what gets us out of bed each day.

To K8s or Not to K8s: Is That the Question?

The problem is, you can’t swing a cat without running into Kubernetes. Every cloud platform I can think of has launched their own Kubernetes-as-a-Service offering, packaging it as an infrastructure primitive much like relational databases, load balancers and object storage. The message is clear and resonant: don’t waste your time with this stuff, we got it.

So what’s a pure play metal automation company to do? Our answer: wait it out. Don't worry. Software will evolve faster than we can or should. Software will solve this.

Think about it: just a few years ago KubeCon was still an ad hoc event run by Joseph Jacks and most people didn’t know how to pronounce Kubernetes. Now 10,000 movers and shakers as well as hundreds of sponsors pop up in Austin or Seattle or Copenhagen for the biggest open source conference in the world, and even my mom knows how to pronounce Kubernetes. Amazing eh?

While this ecosystem has developed at an extraordinary rate, it’s certainly not going to be the last software-based paradigm that changes the world. In fact, I’m quite sure the next one is right around the corner.

Between now and whatever the next wave looks like, we’ll see the market mature significantly and much of that will be made possible by the public clouds defining how easy it should be to run Kubernetes (we are especially big fans of the wonderful work done by the teams at Google and Digital Ocean in this respect).

My theory is that it will be hard to replicate this kind of polished experience in other places (multi-cloud, private datacenters, edge IoT devices, etc) and that Enterprise concerns will stretch and fracture that even further. But, like always, the ecosystem and the commercial offerings around it will, essentially, fix these problems.

Our belief is that running Kubernetes or container workloads anywhere will be solved by folks like Rancher (k3s), Google (GKE on Prem), Loodse (Kubermatic), Containership.io, Red Hat (Openshift), Platform9, VMware, Kontena, Cycle and Supergiant. These are the same kinds of companies that will jump in to help create the next innovation, whatever that ends up being!

Still, Why Not Ride the Wave?

Looking at our pipeline and the various ‘closed lost’ deals, it’s obvious that we’ve missed out on literally tens of millions of dollars of LTV due to our position on Kubernetes.

One client in particular comes to mind: a SaaS company with nearly $200k per month of spend mainly at GCP, and a big fan of Packet. In fact, running on bare metal would reduce their infrastructure spend by a significant amount while improving performance. And yet, the lack of a managed Kubernetes service is a deal blocker because it provides them an enormous amount of operational confidence at GCP, and allows them to be very portable to other clouds.

So, the money is there. I’m sure we could attract the talent or redirect current engineering resources and build our first real, important managed service. I can see the press release headline now: "Packet introduces Packenetes, a fully managed service that allows you to run Kubernetes the way Google does - on bare metal! Try our lovely certified distribution, MKS, today. The M stands for Metal!'

Such a service would undoubtedly unlock serious short term sales. But in any other context, it would conflict with our goals:

It would put us in a state of conflict with software partners.
It would break the glass ceiling we’ve imposed on managed services. Slippery slopes!
It would be a diversion from our focus of automating fundamental infrastructure.

So, we’re going to keep passing on riding this wave.

Focus on Primitives, Invest in Ecosystem

I guess it’s clear what we are not doing when it comes to Kubernetes. So what are we doing?

Through our outreach efforts we’ve put a significant amount of contractor and staff time into various integrations and primitives directly tied to Kubernetes on bare metal:

Released a CSI driver and a CCM;
Wrote a forthcoming Cluster Management API provider and also Kubernetes Autoscaler;
Created official providers for Kubermatic, Gardener and Kubespray;
Invested in our relationships with Containership, Cloud66, Mist.io, Kontena, Loodse and Platform9, among others.
Provided code examples on our Packet Labs GitHub repo for sample deployments with a mix of Terraform, BGP, Calico, MetalLB, k3s and more.

We’ve also contributed well over our promised $25,000 per month in infrastructure to the CNCF (through their Community Infrastructure Lab, the Cloud-Native Network Functions project, and the CNCF CI dashboard); supported commercial partners, meetup presenters, and open source projects with test infrastructure; and invited dozens of customers to take advantage of our "Unlimited POC" program for their own move to Kubernetes. (Not sure if this one counts but we’ve also commissioned a poster that illustrate the cloud native stack, with Kubernetes at the center.)

On the product side we’re working hard to make it easier to install and manage complex or licensed software on Packet, so that our end users and our partners can more easily offer a seamless experience on top of public cloud, on-premises or edge infrastructure.

Building a Reservoir of Goodwill

Much like a "B-Corp" the results of this kind of investment strategy can be hard to track, but the benefits have been clear from our vantage point. We’ve found productive ways to keep close to the Kubernetes ecosystem while adding value to the community and reinforcing our alignment with partners. This is a triple win in our book that more than offsets the potential lost revenue.

Importantly, the context and relationships we gain from engaging with the cloud native ecosystem (as well as other strategic ecosystems such as Openstack, VMware, Software Defined Storage, Traffic Management, and Security) is critical to our long term vision.

Our goal is to be the trusted brand of the software ecosystem and the developers who drive it forward. We believe that without this trust, long term we’re toast. Ed Vielmetti, one of our resident ecosystem experts, likes to remind me that ecosystem relationships aren’t built in a day. You can’t just show up one day and ask something of the ever-expanding ecosystem and expect to get it. Instead, you have to earn it by providing consistent value over time, and by building relationships.

That’s why we take a holistic view of the landscape, which includes not only a "blank check" commitment to open source, but also a religious-style alignment with partners.

Our message is clear: if you’re in the business of delivering software and services on top of infrastructure, we don't want to compete with you, we want to work with you.

In the short term, this stance has placed real limits on our ability to capture a certain kind of revenue. But by staying out of the software as a service game, we open up space for other companies to be the best in the world at doing that, and we showcase our belief in a vibrant, healthy and sustainable ecosystem.

We think this is the right trade off.

Know Which Game You are Playing

As a small company competing with giants, you only have so many cards in your hand to play.

We’ve decided to place bets we think we can win, but it’s important to note which game we are playing. In our minds, the game isn't called "Who Can Take 0.001% of the Public Cloud Market." That game is over!

Instead, we are playing to win a different game, which I like to call "Who Gets to Lead Their Industry Ten Years from Now". In this game, which does remind me a bit of Game of Thrones, we think one of the primary assets is the ability to efficiently deploy and manage your unique technology opinion at global scale. That's why we see such opportunity in mastering the low-level infrastructure fundamentals (the boring bits!) and of earning the unflinching trust of the software ecosystem.

We are taking a bet that we can play a central role in the next wave of the cloud, instead of an immeasurably small role in the current wave. That’s why when others look up, we put on our hard hats, look down and redouble our efforts on things like:

Hardware design and logistics
Hardware security
Asset management
Firmware management
Hardware health
Cloud interconnection
Network primitives like Native VLAN

It's also why we forgot to build a managed Kubernetes service. Oops!