Making the Right Choice: Kubernetes on Virtual Machines vs. Bare Metal

If you have only recently joined the ranks of organizations adopting Kubernetes for deploying, managing and scaling applications, an important question to answer is whether to run Kubernetes on bare metal or virtual machines.

Each option has its benefits and challenges, but the answer ultimately depends on your specific needs. So, what are the pros and cons of running Kubernetes on bare metal, and what are the pros and cons of running it on VMs? What implications will choosing one over the other have on cost, performance, deployment and management complexity, scalability and security? Let’s take a side-by-side look.

Kubernetes on VMs

Emulating the properties of a physical computer, a VM’s environment includes a virtual CPU, memory, storage and a network interface. This allows the VM to operate independently of its host operating system.

You can create multiple VMs on a single physical machine, each running its own OS and applications. Because the experience is similar to running applications on physical computers, VMs are useful for testing different software configurations, running legacy applications or creating sandbox environments for testing and development. VMs are common on cloud platforms, where a single physical server in a cloud provider's data center can run many virtual ones for its customers. This enables efficient use of shared computing resources by different users and for different purposes.

More on Kubernetes deployment and management:

• Kubernetes Management Tools for Bare Metal Deployments
• So You Want to Run Kubernetes On Bare Metal
• Tutorial: Installing and Deploying Kubernetes on Ubuntu

When Kubernetes runs on virtualized infrastructure, VM clusters can be relatively easily provisioned, managed and scaled up or down as needed. Cloud providers typically offer VMs as their basic units of compute, with services designed for running Kubernetes on top. Those would be services like Amazon Elastic Kubernetes Service, Google Kubernetes Engine and Azure Kubernetes Service.

Pros of Running Kubernetes on VMs

There’s a lot to like about running Kubernetes on VMs. At a high level, the benefits are relative ease of setup and deployment, great flexibility and scalability and cost effectiveness at small scale. Let’s unwrap these one at a time:

Straightforward Setup and Deployment

Scaling a Kubernetes cluster up or down as needed is simple, because you can easily create and destroy VMs without affecting the nodes that are running or the system overall.

Since Kubernetes is platform-agnostic, you can deploy a cluster on a variety of operating systems and VMs without tying it to the host machines; VMs provide a degree of isolation from the underlying hardware.

There are a variety of tools for automating deployment and management of Kubernetes clusters on VMs, such as kubeadm, kops and Rancher. Each can help simplify setup and lower the risk of errors and misconfigurations.

Flexibility and Scalability

The ability to provision and deprovision VMs based on demand adds a great degree of flexibility and scalability in running Kubernetes. Scale your cluster up when demand is high and down when it’s low.

Running Kubernetes on VMs across multiple hosts or data centers improves fault tolerance and uptime. If a host VM or physical server underneath it fails, Kubernetes automatically redistributes the workload to nodes that are healthy.

Cost Effective for Small-Scale Deployment

If your cluster doesn’t need a lot of nodes, you can create them all as VMs on one or very few physical servers. That shrinks the cost of hardware and the time it takes to configure each physical host. You save even more time because, as we’ve already mentioned, setting up Kubernetes on VMs is less complex than on bare metal.

Cons of Running Kubernetes on VMs

And here are some of the drawbacks of running Kubernetes on VMs, which may cause you to decide against this type of architecture.

Computing Overhead and Resource Contention

Virtualization adds a layer of processing overhead, which is small—and in many cases negligible—but can increase resource usage and reduce overall performance as workload grows. In the context of comparing virtual Kubernetes clusters to ones running directly on physical hardware, this is particularly true for memory and I/O.

Sharing the resources of a physical server among multiple VMs can lead to resource contention between them. As your cluster workload increases, the contention becomes more significant and may affect performance.

Complex Network Configurations

A VM needs to communicate, be it with its VM neighbors on the same physical server, VMs running elsewhere or other devices or services. Providing a VM with network access requires configuring virtual networks and switches. This can get especially complex when setting up Kubernetes, known for its involved configurations for network topologies and protocols.

Complex network configurations make troubleshooting network issues difficult. The more layers of abstraction and virtualization there are, the harder it is to identify the source of a network issue.

And let’s not forget that virtual networks can be less reliable than physical networks, particularly when it comes to latency and packet loss.

Kubernetes on Bare Metal

When we talk about Kubernetes on bare metal, we mean deploying a Kubernetes cluster directly on physical hardware—without a hypervisor abstracting the hardware from the cluster.

Running Kubernetes clusters on bare metal gives organizations more control over their infrastructure. Of course, that means you have to—or get to!—configure and manage the underlying hardware, network and storage resources. Running Kubernetes on bare metal requires more hardware management expertise. A cluster can also be more challenging to set up.

Pros of Running Kubernetes on Bare Metal

If your team has the skills, however, the benefits may outweigh the challenge!

Maximum Performance and Resource Utilization

Having direct access to the hardware underlying your Kubernetes cluster allows you to configure the hardware to make use of every bit of its resources, mainly CPU and memory.

Scalability and Reliability

Running Kubernetes on bare metal lets you scale your infrastructure both horizontally and vertically. You can add physical servers to a cluster to boost compute, memory and storage. You can upgrade each of the components individually as workload requirements change.

Since you have more control over hardware configuration and maintenance, you can optimize your infrastructure for reliability. You can closely monitor bare metal resources, ensuring that they’re properly maintained and updated to reduce hardware failure risk.

Better Security and Control

Greater control over your infrastructure gives you greater control over security. You can configure network and security settings on your servers to meet your organization’s specific requirements. Given that you have the right expertise, you can also ensure that there are no security vulnerabilities in the hardware.

Direct hardware access gives granular control over what network interfaces, storage, devices and memory resources are available to a cluster.

Cons of Running Kubernetes on Bare Metal

As you probably already see, many of the drawbacks of running Kubernetes on bare metal come down to the complexity involved. But there may also be cost implications. Further down, we’ll talk about a way to solve a lot of these drawbacks—by using dedicated cloud services—and still reap the benefits of bare metal. For now, however, let’s take a closer look at the factors that may lead one to avoid running Kubernetes on bare metal.

Higher Cost—When the Hardware Is Owned

Running Kubernetes on bare metal, of course, requires buying and maintaining the physical hardware. Buying hardware is a significant upfront cost, especially if you need a lot of servers to support your workloads. 

The cost of data center space, power and connectivity for your hardware is a significant ongoing expense. Maintaining and upgrading it is another ongoing cost, both in purchasing components and in team effort. Properly configuring, updating and maintaining servers may require you to hire more staff or outsource to a vendor. 

Running your own bare metal also requires infrastructure software licenses, for things like operating systems, storage solutions and network management tools. On the other hand, you save on buying virtualization software licenses—which can be costly!

More Complex Setup and Deployment

Bare metal servers require more configuration work than VMs. You have to set up hardware resources, networking, storage and security, all of which take time and expertise.

Managing updates and patches for a bare metal setup adds another layer of complexity. You have to ensure that operating systems, software dependencies and hardware drivers are all up to date and compatible with each other, which requires coordination and testing.

Difficulty Managing and Scaling Hardware Resources

Managing and scaling bare metal clusters involves adding and removing physical servers. This can result in downtime for applications hosted on your cluster. Configuring network and storage infrastructure also becomes increasingly complex as your cluster grows.

Scaling hardware resources on individual servers also requires careful planning and coordination to ensure the components are compatible with existing ones and meet current workload demands. Meanwhile, a virtualized Kubernetes cluster can be easily scaled with a tool like the Kubernetes Cluster Autoscaler, which automatically adjusts the number of VMs based on demand.

Dedicated Cloud: Best of Both Worlds

The complexity and cost aspects shrink significantly when the bare metal is “rented” from a dedicated cloud provider. In this scenario, the provider owns, manages and hosts the hardware, billing the user only for the capacity they consume and the length of time they consume it.

For example, if your Kubernetes cluster runs on three bare metal servers in a dedicated cloud for six months, you pay a monthly bill for those three servers. You avoid the upfront cost of buying your own servers and networking gear and setting up the nodes, as well as the ongoing cost of hardware management and data center space and power. 

A dedicated cloud provider, such as Equinix, also gives your cluster access to the internet or private network connections to whatever other public cloud services you may be using. You can choose hardware configurations and spin up your bare metal cluster remotely in any major metro of your choice and manage it through a web console or an API.

You would still need to set up your Kubernetes cluster on the rented bare metal, using your own resources if you have them or one of many managed Kubernetes services. And, of course, nothing would stop you from installing a hypervisor on your dedicated cloud infrastructure and running Kubernetes on VMs!

To sum it all up, the decision between running Kubernetes on VMs and running it on bare metal comes down to the degree of control over infrastructure your organization requires, the technical expertise it has access to and the resources it is willing to commit. Kubernetes on VMs is simpler to set up, manage and scale and can be very cost effective when you don’t need a large deployment. You should, however, be prepared to deal with some complexity on the networking side of things, as well as taking into account some processing overhead needed for the virtualization layer.

Meanwhile, running Kubernetes on bare metal ensures maximum performance and efficient hardware resource utilization, plus full control of hardware configuration and security. All those benefits, however, are only available to you if you have deep expertise in hardware configuration and management and the resources to buy and host your own hardware—unless you choose to go with a dedicated cloud provider.