Building Efficient Test Environments On Dedicated Cloud

Building efficient test environments is one of DevOps’ most challenging tasks. Tests must be reproducible, test environments must mimic production environments—including all the outside systems they interact with—and testing infrastructure costs must be kept under control.

In many testing scenarios public cloud services are ideal, enabling teams to spin up infrastructure at whatever scale their tests require and mimic the necessary interactions with external services’ APIs. They run the tests and then spin it all down, so they don’t have to pay for the services when they’re not testing. 

But there are many software products with testing needs beyond what shared public cloud infrastructure offers. Here are a few examples:

• Highly performance-sensitive applications: Resource contention between tenants on a host cloud server can be a problem for tests that require steady, predictable performance. The virtualization layer on multitenant cloud hosts also requires computational resources.
• Low-latency, or real-time, systems: Not having control over which host within a sprawling cloud data center campus your test runs on makes it hard to control for latency.
• Applications designed to run on hardware that’s configured a certain way: If you need your test to show how an operating system or hypervisor performs on a variety of server configurations, a cloud VM will hardly produce reliable test results. 
• Applications with stringent security and compliance requirements: If you’re in a highly regulated industry (healthcare, financial services, legal, government, etc.), tests that require access to sensitive data are subject to industry specific compliance requirements. That means you need full control over data security, access control, encryption, data segregation and so on.

One way to meet these needs is to test on your own on-premises infrastructure, which is costly and requires a lot of data center and IT management expertise. A more efficient way to go is to use dedicated cloud services that give you full control over infrastructure and the on-demand consumption model of cloud. More on this later; first, let’s go over some test environment basics.

Building Robust Software Test Environments

The key elements of a solid test environment are reproducibility, automation, scalability, accurate simulation of production scenarios and effective test data management.

Reproducibility

Reproducibility is the cornerstone of any test environment. Test results must be consistent across multiple runs if they are to be trusted.

Best practices for ensuring reproducibility include using version control systems like Git to track changes, containerization tools like Docker to isolate applications and their dependencies, Infrastructure-as-Code tools like Terraform to provision infrastructure consistently, configuration management tools like Ansible to automate configuration of the environment and Immutable Infrastructure principles.

Available and Scalable On Demand

A test environment should be available on demand and able to scale CPU, memory, storage and network resources dynamically and automatically. This ensures that applications can be tested under variable load conditions and avoids paying for idle resources.

Teams achieve this by using cloud-based infrastructure services in combination with infrastructure automation tools like Terraform and autoscaling features in container orchestration tools like Kubernetes.

Interaction with External Systems

Another important feature of a test environment is an ability to simulate interactions with external systems an application may interact with in production to identify and resolve issues around integration, compatibility and performance.

This is achieved by mocking interactions with external API endpoints, allowing developers to test their code without relying on external systems’ availability or stability. It also enables simulation of various scenarios and edge cases that may be challenging to reproduce with real systems.

There are tools (such as Postman) that facilitate creation and management of such test environments, mocking APIs and simulating responses from external systems, defining custom responses, simulating different states and controlling external system behavior during testing.

Test Data Management

Managing test data effectively is critical. Testing requires realistic and representative datasets, and organizations must have solid processes for generating, anonymizing and securing test data. Test data should be refreshed regularly to get rid of outdated information and maintain data integrity. Not to be neglected is a process for identifying sensitive information and implementing data masking techniques to protect privacy. A well-defined test data management strategy ensures that testing is accurate, reliable and compliant with data protection regulations.

Cost Management

In addition to controlling costs by provisioning and scaling test infrastructure when needed and then spinning it down, you can use specialized tools, such as Kubecost. The changing nature of testing environments calls for a flexible approach enabled by tools like Grafana and Prometheus. While not explicitly cost-management tools, these open source solutions can be customized for this purpose via integrations like OpenCost to monitor workload and egress costs.

With the right tools in place, your team can enforce best practices, including cost-overrun alerts, autoscaling and leveraging reserved instances for predictable workloads. These tools also facilitate decommissioning unused resources promptly.

Why Build Test Environments On Dedicated Cloud

As we already mentioned, traditional public cloud services provide the high degree of flexibility, scalability, extensibility and on-demand consumption model that testing environments need. They are designed to take much of the burden of infrastructure management off the shoulders’ of software developers. In return for this convenience, however, the user gives up a lot of control over infrastructure configuration (from compute hardware to network architecture), which can lead to both less than optimal testing infrastructure and higher overall costs. 

Teams that must have full control and maximum performance possible will find dedicated cloud a compelling option. It leaves them in control while providing the flexibility and the on-demand consumption model of cloud.

Developers working on things like operating systems, private cloud solutions and enterprise security services have used Equinix dedicated cloud to create just the kind of infrastructure their test, build and deployment environments need.

Dedicated cloud enables you to take reproducibility a step further by configuring the hardware underneath your test environment to mimic the production environment. You can use the same hardware resources every time, reducing the chances of inconsistencies between test and production environments. Choose your locations; choose your CPU (AMD, Arm or Intel), memory, network interface and storage; install an OS of your choice; select the public clouds, network carriers or ISPs you need to connect to; determine which connections should be private and which can use the public internet; and you’re off to the races. 

Run your tests, document the results and then spin it all down. Store your test data on private storage arrays available on demand to save on egress fees when you retrieve it and to avoid compliance issues that may arise from private data ending up in a public cloud or traveling over the internet.

This style of infrastructure is ideal for testing applications that are performance sensitive and expected to use a fluctuating amount of resources in production. You can use autoscaling and Infrastructure-as-Code tools together with the Equinix Metal API to scale bare metal compute capacity up and down on demand—in multiple global locations if necessary—to mimic production behavior.