Building Robust CI/CD Systems on Cloudy Bare Metal

The practice of Continuous Integration and Continuous Deployment, or CI/CD, has become vital to delivering high-quality software faster. By automating the build, test and deployment processes, development teams get shorter feedback loops, shorter time to market and simply better software.

Traditionally, CI/CD systems run in virtual machines or containers on general-purpose hardware operated by the major cloud providers. However, there are situations when for a variety of reasons it’s best to host CI/CD pipelines on hardware that’s tuned a certain way. Cloud providers do not provide the kind of low-level hardware access that’s needed in these situations, so development teams turn to bare metal servers.

Those who cannot or don’t want to own and operate their own hardware can get it as a service from a bare metal provider. Teams that need the ability to provision and manage their bare metal servers via an API, using Infrastructure-as-Code tools such as Terraform, can get that on demand from Equinix Metal. They can use declarative configuration files to spin up Metal servers at global scale, connect them privately to any major cloud platform or network provider and have full control over how data packets travel on their network.

On-Demand Bare Metal for CI/CD

So, when is an on-demand bare metal platform suitable for CI/CD? To better understand those scenarios, let's take a look at some of its capabilities.

Scale Up and Down as Needed

Scaling is a crucial aspect of modern infrastructure and one of the core benefits of the traditional cloud platforms. It enables you to manage workloads effectively as requirements change. One of the benefits of managing your own CI/CD infrastructure is being able to tailor your resources’ capacity to align with your project’s needs and the number of pipelines running.

A platform like Metal provides this kind of flexibility along with the low-level hardware access and control of the bare metal. You can have the right amount of resources that are tuned to your requirements running, whether you encounter a temporary increase in builds or need to scale down during low-activity periods.

High Performance

Bare metal servers offer superior performance to VMs because they lack a hypervisor layer, which adds a degree of performance overhead and often also causes noisy-neighbor issues, where VMs sharing a single physical machine contend for its resources. Your CI pipelines may also simply need more raw performance all at once than you can get from a major public cloud. (We’ll bring a real-world example of this below.)

Hardware Customization

Another compelling reason to use on-demand bare metal is the level of customization it offers. By simply making a few API calls, you can programmatically provision dedicated infrastructure tuned to your project’s requirements. This enables your workload to run automatically and seamlessly adapt as your demands evolve.

Bare metal gives you full control over the hardware configuration, operating system and software stack, enabling optimal compatibility and performance. This level of customization ensures maximum efficiency and productivity of your CI/CD processes by tailoring them to your needs. 

Moreover, not all popular CI solutions support all CPU architectures. GitHub Actions, for example, doesn’t support running code on Arm chips, so developers building software for Arm who want to use GitHub Runners in their CI pipelines must set up self-hosted runners. (Here’s a GitHub repo for doing that on Metal).

Equinix Metal offers choice of CPU architecture (x86 by AMD or Intel and Arm by Ampere), memory size, NVMe drives and NICs, with full dedicated platform provisioning time under 60 seconds. It natively supports a robust list of popular operating systems, and whatever OS isn’t on that list can be provisioned with custom iPXE. A server can be deployed with or without a public IP and quickly connected to any cloud or SaaS provider, or a network carrier of your choice.

Common Tools for CI Hosting on Bare Metal

There are many tools and approaches to running CI/CD pipelines on bare metal. Some of the popular ones rely on Docker, GitLab and Bash scripts:

• Docker: One of the benefits of Docker in the context of CI is the ability to package CI pipelines into containers, ensuring consistency across different hardware environments. With it, you can define your build environment, manage dependencies and isolate your CI/CD processes with ease. Docker provides portability, scalability and simplified deployment across a variety bare metal instances. As an example, here’s a good blog post describing the process of creating a simple CI server with Docker.
• GitLab: While Docker is a useful tool for CI/CD, you may need a more extensive solution, such as GitLab. If you have a lot of pipelines, you can define the requirements for each using a simple .gitlab-ci.yml that is stored in your source repository. By leveraging GitLab's extensive runner support and powerful deployment options, you can effectively implement parallel builds, ensuring smooth operations while leveraging the advantages of bare metal.
• Bash Scripts: While Bash scripts need more manual configuration and maintenance than Docker or GitLab, they're useful when you have special requirements or need to interface with current infrastructure. For example, with Bash, you can create custom scripts that send messages to Slack when certain state changes occur in your CI/CD pipelines. These scripts can be easily stored in your system and can help you patch or create an entire system from scratch. This page has some good examples of Bash CI you can reference.

Why Equinix Metal for CI/CD?

If you need full control of the hardware running your CI/CD pipelines, bare metal is the answer. However, if your organization isn’t prepared to deploy and manage dedicated CI/CD hardware in its own data centers, the answer is using an on-demand bare metal service provider.

Equinix Metal stands out among such providers because of its global scale, presence of virtually any third-party network or platform you may need to interconnect with in its data centers and its robust API, which enables maximum automation and programmability of your bare metal servers AND your network via familiar Infrastructure-as-Code tools.

On-demand bare metal is a way to get dedicated servers on a flexible pay-as-you-go basis. You can quickly adjust your infrastructure to match project requirements without the burden of hardware procurement and management. You can ramp up capacity (on a global basis if using Equinix) to run a series of builds, for example, and spin it back down when you’re finished, paying only for the time the capacity was used, like you do with traditional cloud services.

If you want to run your pipelines closer to your developers or need to distribute the latest versions of your software to users worldwide, you can do it quickly from facilities that are local to any particular concentration of users. It's sort of like being able to launch your own private global network of edge data centers on demand.

Mirantis uses Equinix in this way. The company runs tests and builds of its open source-based private cloud solutions for enterprises on Metal, and it’s not uncommon for its clusters to get up to 1,500 bare metal servers large at peak testing. Its high-end testing wouldn’t be practical on a virtualized cloud platform. The team requires full control over Layer 2 networking between nodes, so that it can validate complex networking configurations, as well as advanced SmartNICs and hardware accelerators to match its customers’ infrastructure.

Flexibility and Control, Without the Compromises

If your team is building software with highly specialized, high-performance infrastructure requirements, running CI/CD pipelines on a global, fully automated and flexible bare metal platform can greatly enhance the team’s productivity and quality of the software it produces. 

Traditional public cloud infrastructure and packaged CI/CD solutions that run on it work great for many use cases. However, if you require A LOT of compute, low-level access and full control of the hardware, as well as maximum network configurability without the burden of owning and managing your own data centers, you can tick all those boxes with an on-demand bare metal platform like Equinix’s.