State of Cloud Native on Arm

With an emphasis on high core count and power efficiency under load, the Arm architecture provides an excellent substrate for cloud native workloads. Because of this, an increasing number of Cloud Native Computing Foundation (CNCF) projects — including Kubernetes — offer support for Arm-based systems. This software adoption is coming together with strong hardware support from cloud computing solutions like AWS Graviton and Ampere Altra, as well as the new Apple Silicon-based Mac computers that are the daily workhorse of many developers.

So, what’s the state of cloud native software on Arm? Let’s try to keep up!

(Looking for an interactive resource? Check out the results from Ampere Computing’s impressive CI/CD testing)

Operating Systems

The operating system sits between the hardware and the applications. It was one of the first pieces of software necessary to build a full Arm software ecosystem.

In the cloud native space, which in some ways was born alongside a vision for immutable operating systems (think CoreOS Container Linux), has helped to drive Arm support across the most popular operating systems, as well as specialized “Kubernetes” focused distributions.

Notable here is Ubuntu and CentOS, which have supported Arm systems for the past few years. Other standard Linux distributions like Debian, FreeBSD, and Alpine Linux now offer excellent support as well.

New to the scene are more purpose built operating systems such as Flatcar Linux (a successor to CoreOS Container Linux), NixOS, K3OS, and Talos. Even VMware, with their ESXi “fling” for Arm, has joined the parade. For those looking to learn more, a recent “Guide to Linux Operating Systems for Kubernetes” was published by our friends at The New Stack.

Kubernetes

Kubernetes has rapidly become the container orchestration software of choice for modern cloud native applications. Arm is a first-class architecture for running Kubernetes clusters, with all of Kubernetes components being built for Arm and functioning out-of-the-box. This has created possibilities for an ecosystem of multi-architecture focused tools for storage and network integrations that support Arm and x86 targets equally as well. Container Network Interface (CNI) plugins such as Flannel, Calico, and Weave are examples of major contributors to the ecosystem in the form of popular solutions prioritizing Arm systems.

The status of the components in Kubernetes, and prerequisite components, is straightforward:

• Container runtimes supported by Kubernetes like Docker and Containerd, both have Arm releases. CRI-O can be built and deployed on Arm systems, but does not yet have a pre-built release. Components like the key-value store Etcd, and SQLite (used in K3s in place of Etcd) are also dependent projects with first-class support for Arm, meaning that different node classes (cluster control plane nodes, replicas for HA control planes, and workers) can all be powered by Arm or x86 architecture.
• Many certified distributions of Kubernetes, such as K3s from Rancher and Lokomotive from Kinvolk, specifically have Arm in mind as a deployment target for use cases like edge computing. These target a high number of smaller clusters on resource constrained but highly performant hardware, a use-case where Arm systems are more and more ubiquitous.

The suitability for Kubernetes in emerging use cases is assisted by the growing ubiquity of Arm hardware: edge computing patterns, for example, are accelerated and can be generalized with patterns used in the datacenter by using containerized workloads. With Arm support, Kubernetes is more accessible than ever to users with edge deployments, lower power consumption machines, or a need for distributed, many core endpoints (i.e. for data pipelines, or points-of-presence for distributed web services).

Networking

The CNCF landscape includes many options for Kubernetes networks from CNI plugins to operate overlays, to those consuming external networks, to drivers for support of flat networking on your clusters’ pod networks. Many of the most popular choices already have actively maintained Arm support, making deploying a robust container network infrastructure across architectures simple and scalable.

Some of the most popular CNI plugins have out-of-box support for Arm architectures, including:

• Project Calico, which is a popular choice for a number of users with complex needs like BGP reflection, datastore, routing and security policies.
• Weave, which offers enhanced networking features like dynamic routing, fault tolerance, service discovery (as well as routing options for such discovered services), and network policies with a very thorough set of security and isolation options for applications.
• Flannel, which supports multiple network backend overlay types (i.e. vxlan), and a number of fault tolerance, failover, and routing features for use with quickly changing clusters.

Like many software packages that deal with networking, Service Meshes typically are written in languages requiring an OS and architecture target, so while a few do not support Arm architectures, most can be built without issue for Arm:

• Envoy is designed for cloud native applications. A graduated CNCF project, developers set up a CI build environment for Arm64 and enabled multi-architecture builds.
• Linkerd is another project with active compatibility for Arm builds atop RPC frameworks with supported Arm packages.
• NGINX offers better price performance with broad software support, so you can run them on any supported OS and container type to leverage AWS tools and services.
• Inlets - inlets is a Cloud Native Tunnel written in Go that supports Arm to connect services.

In addition to the networking packages above, Kubernetes has a number of features that can be extended with addons, for example, provider integrations or automation for external services for things like LoadBalancer-type Service resources. Packages like MetalLB and Kube-vip provide Layer 2 or Layer 3 network configuration options to the LoadBalancer resource and have full Arm support.

Storage

There are several Container native storage solutions and the most popular fully support Arm architecture.

• Longhorn is the official CNCF project to provide distributed storage to Kubernetes. It natively supports Arm architectures along with x86.
• Rook is an Arm ready cloud native storage service for Kubernetes that provides management for files, block, and object storage.
• OpenEBS has been a long time supporter of Arm architecture with support dating back to early 2018.
• Minio has also joined the Arm storage field with their S3 compatible object storage solution. They also released a great blog last year showing how Arm’s performance stands up to other systems.

Serverless

Serverless or Function-as-a-Service frameworks are based on an architectural pattern of abstracting away the domain-specific knowledge one might normally need to know about everything under the application. For example, one can run a task written in a language like Ruby or Python or JavaScript that has no dependency on a framework like Rails or Django or Node.js to produce web applications, just a script for the one function you’d like to complete, served by an HTTP interface. One chief benefit of this pattern is that it makes code incredibly portable between instances and between platforms.

Kubernetes is a platform that can be abstracted for use by developers hoping to decompose larger applications into a series of “serverless” functions. There are a few major projects bringing Serverless/FaaS capabilities to Kubernetes. Because a lot of these solutions are not platform-dependent, support on Arm systems will be less of an issue than with other solution types. However, because the features of the Kubernetes manifest style, and how the scheduler can be directed to target certain hardware on your fleet, Serverless frameworks can be used to help accelerate development for workloads that might take the most advantage of Arm many core architecture.

A few examples of Kubernetes-backed Serverless and FaaS frameworks that are in use today:

• Knative assists in serving application code, abstracting the underlying Kubernetes events and features into developer-friendly interfaces.
• OpenFaaS supports other container orchestration systems besides Kubernetes (i.e. Docker Swarm), which made a conscious effort to only use upstream components that support it, and contributes back to Arm projects that enhance its support.
• Fission is an open-source, Kubernetes native (integrating tightly with Kubernetes components, and extending features like those of Knative, for example) serverless framework that can be used to quickly spin-up and tear-down endpoints for functions, script workflows for a series of functions, and supports any language whose runtime can be packaged, but supports a number out of the box.
• Apache OpenWhisk is a distributed FaaS solution that runs on an event-driven functional model for task execution called Actions written in any supported programming language, that can be dynamically scheduled and run in response to associated events (via Triggers) from external sources (Feeds) or from HTTP requests.

Serverless computing puts the power of the underlying hardware to use in a manner that allows developers to write production code without the need to be a cloud native landscape domain expert. Cluster operators can automate provisioning the relevant resources while allowing developers to consume resources for the hardware/machine types their workloads may require while having Kubernetes doing the heavy lifting in directing those requests to the appropriate pool.

More to come

As with any ecosystem, change and evolution is a part of the process. Arm in the datacenter has had its share of evolution over the years. Participants have come and gone and come back while servers and processors have become more powerful and feature rich. In the past, access to Arm infrastructure limited the number of people that could participate in expanding Arm support for projects. However, this issue is resolved by an expanding presence for Arm in public cloud and now even on local developers’ machines.

Looking forward to 2021, areas of focus for the cloud native Arm community include storage, where MinIO is a container native storage solution that has long supported Arm-based systems but efforts are underway to add full support for OpenEBS and Rancher’s Longhorn. Other important projects, such as native Docker images for Istio and Arm support for Kubeflow (for machine learning), Knative (serverless), and Kubevirt (for virtualization) show interest and promise. This combined with ubiquitous Arm computing from the phone to the laptop to the datacenter will allow new projects and new applications to target and build for Arm first.