Kubernetes Management Tools for Bare Metal Deployments
Our handy roundup of the essential tools for bootstrapping and provisioning Kubernetes clusters on bare metal and managing container workloads.
Most people who deploy, scale and manage containerized applications have chosen Kubernetes as the platform for doing those things. The platform is complex and can be difficult to manage, so there is a vast ecosystem of Kubernetes management tools designed to help streamline cluster orchestration.
A subset of this ecosystem deals specifically with Kubernetes deployments on bare metal servers (as opposed to VMs). Sure, Kubernetes is a cloud-native technology, but it doesn’t have to run on cloud VMs, and this subset of management tools is for organizations that want to maximize performance by removing the hypervisor layer from their stack. The tools are designed to solve challenges like hardware heterogeneity, network complexity, bootstrapping Kubernetes clusters for reliable operations—in other words, things users of managed Kubernetes services on virtualized cloud platforms typically expect the providers to take care of.
More on Kubernetes deployment and management:
- Making the Right Choice: Kubernetes on Virtual Machines vs. Bare Metal
- So You Want to Run Kubernetes On Bare Metal
- Tutorial: Installing and Deploying Kubernetes on Ubuntu
Top Bare Metal Kubernetes Management Tools: an Overview
Let’s take a look at some of the most popular tools for working with Kubernetes on bare metal. We’ll cover their limitations and advantages and talk about how they stack up to your requirements. For good measure, we’ll highlight the reasons these tools are quickly overshadowing the more traditional configuration management tools.
Kubeadm
Kubeadm is one of the popular production-ready Kubernetes management tools, started by the Cluster Lifecycle Kubernetes Special Interest Group, or SIG. Designed for bootstrapping a Kubernetes cluster, it was first shipped with Kubernetes 1.5. Currently, Kubeadm focuses on automating, installing, upgrading, modifying or destroying a Kubernetes cluster and supports single and multi-master (high availability) modes.
Advantages of Kubeadm
Kubeadm automates a lot of the Kubernetes cluster setup work, such as generating certificates, setting up the control plane and joining worker nodes to the cluster. The automation makes using and setting up bare metal for your cluster straightforward with a simple CLI.
Kubeadm is so widely adopted that having hands-on experience upgrading clusters with Kubeadm is a requirement for becoming a Certified Kubernetes Administrator.
The robust tool can manage a massive number of nodes and supports best practices for setting up clusters. The latter is especially important when you’re dealing with bare metal Kubernetes deployments, where configuring nodes while maintaining best practices gets complicated.
Kubeadm can cater to individual requirements, such as different CRI implementations, network plugins and other configurations. It easily keeps a cluster up to date with the latest Kubernetes versions and security patches.
Limitations of Kubeadm
One limitation of Kubeadm is the absence of a GUI, which makes for a steep learning curve. Upgrading a cluster, for example, requires manually installing versions of kubelet and Kubeadm while draining the nodes—not a simple task for Kubernetes newcomers or users without any administrator skills. When things go wrong, it takes a seasoned Kubeadm expert to dive into logs and fix the problems. Not having such an expert in-house can be a major challenge for a company deploying K8s on bare metal.
Finally, while Kubeadm links everything together into a cluster, it doesn’t support infrastructure deployment or cluster creation. You also have to manually configure load balancing and storage once your cluster is bootstrapped.
Cluster API
Cluster API, or CAPI, is an infrastructure-agnostic way to provision and manage Kubernetes clusters using manifests declaratively. The project was initiated by the Kubernetes SIG Cluster Lifecycle with the intention to simplify Kubernetes cluster management by abstracting the underlying deployment complexity.
CAPI makes cluster management easier with custom resource definitions. The project uses a management cluster that stores all your configuration, cloud keys and state to deploy your workload cluster—it uses Kubernetes to deploy Kubernetes!
Advantages of Cluster API
CAPI automates infrastructure and cluster installation so you don't have to worry about provisioning infrastructure once you provide the API keys. One of its major benefits is a self-healing capability for your nodes; if one fails, CAPI ensures that a replacement is available by creating new nodes using API keys you’ve provided. To upgrade, simply increase the node count on your manifest and apply it. No more configuring infrastructure and then connecting nodes to your cluster, even if you’re on a bare metal provider.
CAPI supports multi-cluster management with declarative manifests. Each manifest reflects the state and upgrades easily. Rolling updates rather than inline upgrades improve uptime.
Limitations of Cluster API
CAPI is somewhat newer than other bootstrappers in the Kubernetes management tools category, making it a relatively weaker consideration for production environments. More importantly, the potential for a compromised management cluster is a security risk. Because it contains all your credentials and state data, it can be a single point of failure that can enable attackers to compromise all your workload clusters. Finally, if your management cluster fails, you lose the ability to manage all your workload clusters—even if your nodes are healthy and running.
MetalLB
MetalLB is a popular implementation for routing traffic to your cluster for bare metal. It exposes services via BGP (Border Gateway Protocol) or Layer 2 (with ARP Address Resolution Protocol) for fault-tolerant access to the cluster and can be installed with a few simple commands once the cluster is bootstrapped.
For IP allocation, MetalLB uses an address pool you provide, which you can lease from bare metal providers like Equinix Metal.
Advantages of MetalLB
MetalLB is a relatively easy solution to deploy for your bare metal clusters. You can install it using kubectl or Helm charts to implement the LoadBalancer service without having to deal with excessive infrastructure-level configuration.
Community support has helped this open source project expand, and it currently supports BGP and ARP for a variety of network setups. For example, you can use BGP to advertise routes between routers, and ARP to map IP addresses to MAC addresses at the data link layer.
Limitations of MetalLB
MetalLB's primary limitation is that it does not support dual stack networking in all modes. In such cases, each load balancer can only be assigned an IPv4 or an IPv6 address. This can be a significant disadvantage for organizations that require both IPv4 and IPv6 connectivity for their Kubernetes workloads.
Furthermore, MetalLB is limited to Layer 2 load balancing, which means it operates at the OSI model's data link layer. This can lead to limitations such as the network's bandwidth being limited to the bandwidth of your node. This means workloads that need high bandwidth will have performance issues when there are nodes with limited bandwidth.
MetalLB's Layer 2 load balancing mode can also limit its ability to manage IP failovers when a node fails and there’s a need for the IP to be reassigned to a new node without manual intervention.
Argo CD
The main function of Kubernetes is to manage your container workloads, so there should be tools for deploying workloads to Kubernetes. Enter Argo CD, a deployment tool that uses GitOps principles to store your manifests in a single Git repository and deploy changes to your cluster.
The open source project was released in 2018 and has been popular ever since for providing declarative continuous delivery. Argo CD supports Terraform, Helm and Kustomize, just to name a few.
Advantages of Argo CD
Argo CD is more than a means to implement your GitOps principles—it’s a unique and powerful CD tool. It can pull updates from Git and deploy them directly to your cluster, and its hard-working UI makes self-service for developers a breeze.
Limitations of Argo CD
One limitation of Argo CD that stands out is the complexity of dealing with so many YAMLs. A single broken YAML manifest complicates things for traditional DevOps teams, let alone several of them.
Argo CD also lacks built-in support for secret management. That can be challenging for organizations that require a more comprehensive solution for managing secrets in Kubernetes. Using Sealed Secrets or other providers can be a good alternative, but that also adds a layer of complexity.
Cilium
Cilium is a powerful modern container network interface (CNI) that uses eBPF to provide networking capabilities to Kubernetes clusters. eBPF, or extended Berkeley Packet Filter, is a revolutionary kernel technology that allows developers to write and load custom programs into the Linux kernel without modifying the kernel itself. When you can catch and analyze network packets in real time, you open up possibilities for advanced networking use cases.
Such use cases are crucial for cloud native systems, where container workloads are dispersed across clusters of nodes. A CNI is a specification for network plugins that let containers interact with external networks and each other using certain APIs. For example, once you have deployed your cluster with a bootstrapper like CAPI, you can install a CNI to inject networking capabilities (which doesn’t happen automatically with bare metal installations).
Cilium’s eBPF implementation provides features like load balancing, network security and observability. It maintains high performance and scalability with fine-grain capabilities, making it a perfect fit when you’re dealing with host machines directly, without a hypervisor for your nodes.
Advantages of Cilium
Using eBPF, Cilium replaces traditional networking sidecars, improving performance and scalability for Kubernetes clusters. Because it interacts directly with the Linux kernel, it can reduce the overhead of traditional networking approaches and accelerate network performance.
Cilium also enables administrators to create security policies based on the identification of the container orchestrator rather than IP subnets and container IPs. As a result, security policies are simpler to manage and apply to a wide variety of container workloads.
With Cilium’s observability features, developers can gain insights into their network traffic and performance. Its flow-level visibility, service-level metrics and network topology visualization make it easier to troubleshoot and optimize networking issues. You can also configure Cilium to initialize a load balancer.
Limitations of Cilium
Cilium offers a lot of modern capabilities, so the kernel versions it supports only go so far back. To install it, you need to be on a Linux kernel >= 4.8 (>= 4.9.17 LTS recommended), which might not be possible for all workloads.
Furthermore, installing and maintaining Cilium requires some specialized knowledge that isn’t readily available at every company.
Flux
Flux is a CNCF graduate project that’s similar to Argo CD. It provides a GitOps-based approach for streamlining and automating deployment of containerized apps on Kubernetes clusters. The tool's adaptability enables it to function in any Kubernetes environment like on bare metal.
Advantages of Flux
Flux is lightweight and simple to use, which makes it stand out among Kubernetes management tools as a fantastic option for smaller teams or businesses with less complex deployment needs. Its image automation feature allows the tool to automatically scan your image repository and update your container images without requiring Git commits.
You can use Flux for deployment without configuration by bringing along your manifests, such as Helm.
Also, thanks to a simple high-level setup, debugging is simpler than in other tools. Flux even offers to delete unnecessary resources from a cluster automatically.
Limitations of Flux
Deployments may be simple, but configuring and managing Flux is challenging. Access controls and installation procedures might require specific knowledge and skillsets for scenarios like multi-cluster deployments.
CD tools benefit from GUIs like Argo CD, which Flux (a CLI tool) lacks. Also, its developers have omitted certain features required in an industrial setting for the sake of simplicity. For example, the CD tool allows only one repository as the source control for each deployment instance, which might not be suitable for bigger organizations.
Another limitation of Flux is that it currently lacks an authorization layer beyond Kubernetes's RBAC. That can be a dealbreaker for organizations with more complex security and access control requirements.
Configuration Management Tools
Modern software delivery requirements have forced a shift from managing entire servers to managing software stacks. Classic configuration management tools like Chef, Puppet and CFEngine were popular for automating provisioning, configuration and management of servers and applications in traditional data centers with monolithic applications. However, as you've seen in this article, the requirements are different for distributed applications that use a microservices architecture.
You still need to manage configuration, but Kubernetes management tools like Kubeadm and Cluster API make cluster initialization and management simpler. Older tools become obsolete as configuration is managed automatically. When it comes to deployments and state management, declarative configuration is taking over.
The classic tools are, of course, very good at managing and configuring states, but as infrastructure grows, it drifts, requiring maintenance. Implementations like GitOps and Infrastructure as Code (IaC) prevent drift and provide self-service capabilities to developers to improve overall productivity.
Modern methods like GitOps and Kubernetes management tools tools like Argo CD and Flux ensure that your state is under version control, making audits and rollbacks easier. Tools like CAPI integrate well with private and public clouds, simplifying management of the entire infrastructure consistently, whereas previous solutions required a lot of work for different environments.
Conclusion
To choose the right Kubernetes management tools for your bare metal deployment, consider the level of automation you require and the custom configuration your use case demands. Don't just go for the popular tools; focus on the ones that can solve your specific problems effectively. Remember that a lot of configuration management tools that used to be popular have fallen behind in the modern software ecosystem.
Bare metal Kubernetes has been gaining popularity due to its improved cluster control, reduced hypervisor cost and enhanced security. Kubernetes management tools for bare metal deployments are likely to evolve and improve as the demand for more optimized compute nodes increases. Stay tuned for future tools to focus on solving problems like updating patches on the underlying OS of your bare metal and improving backup and restore capabilities.