Love Thy Neighbor? Maybe not in the Cloud

The past few months have been full of interesting security notices, keeping us all on our toes.  This week didn’t disappoint.  Welcome to 2018!

Meltdown in Summary

For those of you that were just trying to deal with #snow in Florida, add to your plate the announcement today (with technical details here) about a fairly major CPU and memory isolation bug that affects nearly everybody using an Intel CPU made in the last 10 years. That means most of you, yup.

The gist of the issue is that there is a bug in certain CPU designs that allows an attacker to exploit how both the Linux and Windows kernels isolate memory, enabling those memory tidbits to be read between processes, virtual machines or containers.  That’s the bad news, especially if you’re using a multi-tenant system that hasn’t been patched.  

Using #Meltdown to steal passwords in real time #intelbug #kaiser #kpti /cc @mlqxyz @lavados @StefanMangard @yuvalyarom https://t.co/gX4CxfL1Ax pic.twitter.com/JbEvQSQraP

— Michael Schwarz (@misc0110) January 4, 2018

The good news is that there is a fix, thanks to several months of secret work by the Linux and Windows kernel teams.  The catch?  The fix comes with some pretty massive performance hits to CPU and IO workload - basically anything that touches kernel page memory. Users of affected VM-based public clouds may notice some significant performance hits:

Impact of the patch for the Intel bug on my AWS EC2 instances running Linux, a t1.micro and m3.medium. #KPTI pic.twitter.com/n3uZsZ38iV

— Tim Gostony (@timgostony) January 3, 2018

The patch being introduced by cloud providers and OS vendors across the world effectively cuts the performance of syscalls significantly.  For workloads like Postgres, the initial reports show at best a 17% slowdown and at worst a 23% slowdown on common operations.

Instead of rehashing official statements from the major CPU vendors, I’ll just link them here for Intel, ARM and the-not-so-official, but juicy comments from AMD.  You can read them and form your own opinion (or you can take Linus’ side).

Why we Still Believe in Single Tenancy

For those of you that don't know much about Packet, it's handy to understand that since the beginning we’ve been laser focused on offering DevOps automation for single tenant, bare metal compute regardless of architecture. 

We don’t do multi-tenant servers, nor do we sell virtual machines or run containers for people.  We don’t force you to use Intel or Arm or AMD.  We certainly don't ask you to share a hypervisor with somebody you don’t know.

What we do offer is an elegant way for developers to have their own opinion about the software that they run and on what architecture they run it on -- from the OS, to the kernel, to the virtualization layer (or not). 

This isn't a marketing gimmick - it's about choice. We started our company in 2014 because we felt users should be able to define their entire stack, without losing the benefits of automation.  We're even more passionate about (and committed to) that mission today.

Implications at Packet

Packet users are not directly affected by either the Meltdown bug or the performance hits associated with the patches. Here's why:  

• We offer a variety of system architectures including an Armv8 Type2A machine and our upcoming AMD EPYC server.    
• We don’t force any multi-tenancy, so there is no risk of neighbor sidecar attacks from other Packet users (if you bring multi-tenancy to your Packet infrastructure, there will of course be considerations).
• We allow users to leverage official Packet images (e.g. Certified Ubuntu) or bring their own OS images via our Custom iPXE option.

We encourage users to make the best choice for their own businesses, workload and security situation - including looking at alternative architectures and running their OS without any forced patches.  Meanwhile: 

• We’ll stay engaged with Intel, our various Arm vendors and AMD about the implications of this attack method for our users so we can be as educated as possible. 
• We'll work to understand the upstream patch options being introduced and will communicate to our users any official image updates as they apply. 
• We’ll continue to show our support for the open source community by providing multi-architecture OS build capacity to Ubuntu, Debian, FreeBSD, CentOS, NixOS, Alpine, and Container Linux to ensure the broader community has access to the OS flavor of their choice on their hardware platform of choice.

P.S. Kudos to Google for it’s awesome work in identifying this security issue through its Project Zero efforts.