Breaking Bond

If you’ve grown up in the cloud, traditional networking terms can feel weird and foreign. BGP? MLAG? Interface Counters? NNIs? VNIs? LOAs? But if you came of age building physical networks — maybe at a colocation facility down the street — the acronym soup of networking jargon rolls off the tongue quite easily.

With the advent of the public cloud, developers, application owners, and millions of others have built at-scale applications with programmatic access to IT infrastructure. This programmability has mainly been achieved through abstraction at various layers, and the network is one of the places where abstraction has been super helpful. This is because a lot can go wrong very quickly with physical or large-scale networking, and it can impact other tenants or even the internet at large.

On a more basic level, the “servers” or virtual machines within most public clouds have been normalized with virtual network cards that can be manipulated in software with relative ease. So when you start going physical, there is a learning curve to unlock new benefits and opportunities.

Why Hello, Physical World

With portable workloads and cloud native approaches, developers are increasingly discovering the alluring performance and efficiency benefits of physical infrastructure and networks. But along with the good comes a certain amount of unfamiliarity, which can trip up even the most experienced SREs and developers.

In the words of Walter White “If you don't know who I am, then maybe your best course would be to tread lightly.”

Sort of makes the prospect of deploying physical infrastructure like Equinix Metal a little nerve-racking. Fortunately, it’s not that complicated, but understanding the topology is critical.

Here’s what you need to know:

In a physical world, network interfaces map back to literal NICs (Network Interface Cards). This gives users a lot of control, but one of the tripping points I’ve seen is around network interfaces.

By default, servers at Equinix Metal are configured in a “bonded” mode using LACP (Link Aggregation Control Protocol). Each 2-port NIC is configured with a single bond (namely bond0) with both interfaces eth0 and eth1 as members of the bond in a default Layer 3 mode. Our automation builds the configuration at deployment across the switch and operating system to make getting up and running fast and easy.

This is a simple, high-performant, and redundant setup, but many environments benefit from Layer 2 networking to manage services like DHCP and routing within a private network. To suppor these use cases, we have a feature that allows users to add Layer 2 virtual networks to their Equinix Metal infrastructure. Our Layer 2 capabilities are exposed via encapsulated VXLANs sitting on top of Layer 3. VLANs can the nconnect your servers to each other or connect them to other infrastructures, like Equinix Fabric. 

In Layer-2 bonded mode, VLANs can be attached to the bonded network interface. Bonded modes offer redundancy and high-availability by supporting traffic across the 2 switches. An outage, maintenance, or reboot of one of them will not cause network interruptions. Need more networks? Just add more subinterfaces to the bond. 

In Layer-2 unbonded mode, the switch ports serving each of your server's NICs can be independently configured after dismantling the bond, enabling you to attach VLANs to the individual ports. This does introduce a single point of failure on the upstream switch or the network interface. An outage, maintenance, or reboot of one of the ports will cause network interruptions. 

You have options based on what sort of deployment you are working with within your private network. You might require a redundant system to not only support traffic but to ensure lossless operations in case one of the interfaces went down. While on the other hand, you might need the two interfaces to run independently.

In Layer-2 bonded and unbonded modes, access through the public internet is lost, and the host can only be reached by the SOS console or through the jump host attached to the same VLAN. If access through the internet is a requirement to your use case, we have a Hybrid Unbonded mode feature for you. The Hybrid Unbonded mode removes one network interface from the Layer-3 bond and places it in Layer 2 mode. This allows the user to attach VLANs to this interface for sole Layer 2 connectivity to your internal network and preserve Layer 3 connectivity to the server via the bond. Now the server is accessible through the public IP but with a single point of failure on the upstream switch or the network interface. An outage, maintenance, or reboot of one of the servers will cause network interruptions. 

Enter Hybrid Bonded

The good news is that there are more options available to you. In particular, we’ve added a new network interface configuration to our platform called “hybrid bonded”, which enables both L3 (Internet) and L2 (Private VLANs) on a single LACP bond.

This means you can now run a highly available mixed network setup on Metal! Firewall? Why not. Router? Of course! Interconnected Ingress Controller? Sure!

Hybrid Bonded mode enables a highly available “bonded” setup of 2 networking interfaces that supports both Layer 2 and Layer 3 modes at the same time. This keeps the functionality of supporting both Layer 2 and Layer 3, but does so while maintaining a highly available bonded networking interface that spans two diverse upstream switches.

Let’s say a customer has deployed their application hosted on Equinix Metal. The ingress traffic is coming from the internet through the network edge router into the customer's internal network. The northbound internet-facing interface of the network edge router needs to be in Layer-3 and the southbound interface connecting to the internal network in Layer-2. Customers can now set the network edge servers in the hybrid bonded mode bringing high availability and redundancy. The single bond connects to the internet over Layer 2 infrastructure. Any single fiber failure on the server or a single top of the rack (TOR) switch failure wouldn’t bring down the network edge device.

So go ahead! Go a little crazy. Be the one who knocks (another Walter White reference, of course) Break the bond.