Your Server's Networking¶

With Metal, every bare metal server connects to a redundant pair of switches using 802.3ad link aggregation. Under normal operating conditions, a server is provided a bonded 2x25 Gbps or 2x10 Gbps interface, with uplinks that have enough capacity to use them. If a switch experiences a mechanical failure, or undergoes maintenance, a server’s bandwidth is halved, however the server remains online.

Default server provisioning puts your server's networking in an all-Layer 3 mode, with a public IPv4 address set as the default route to the Internet and private IPv4 addresses for inter-server traffic within the same project and metro. You also get blocks of IPv6 addresses for applications that support it.

Our platform management software resides on isolated and secure physical devices separate from your server. There are no complex hypervisors or software agents running on the hardware that you provision from us.

More on our network design philosophy and implementation can be found in our blog, on How Equinix Metal is Bringing Cloud Networking to Bare Metal Servers

The Network at Rack-level¶

Each Equinix Metal server is attached to a top-of-rack (TOR) switch that functions as an edge device, isolating your server's traffic from surrounding neighbor traffic.

Our top-of-the-rack switches are configured in active mode and send Link Aggregation Control Protocol Data Units (LACPDUs) to your server, regardless of the server's network configuration.

Our Intelligent Platform Management Interface (IPMI), provides for “out of band” functions such as hardware telemetry, power cycling, and remote console. It's on a physically separate IPMI network accessible to our staff and internal management systems and we make its features available through the Equinix Metal console and API. It is not accessible to the open Internet.

Your Server's LACP Bonding¶

Bonding, also called link aggregation, combines several network interfaces (NICs) into a single link. This provides various potential benefits, including high availability, load balancing, maximum throughput, or a combination of these.

Equinix Metal's servers have dual NICs, and their network interfaces are bonded together using the Link Aggregation Control Protocol (LACP).

The LACP bond creates aggregation groups that share the same speed and duplex settings and utilizes all bond members in the active aggregator as specified by IEEE 802.3ad: Dynamic link aggregation.

If you provision a server using the default Layer 3 networking mode, the two interfaces are bonded into bond0. Our Hybrid and Layer 2 networking modes have flexible options for assigning Layer 2 VLAN traffic to the bonded interface, the individual interfaces, or both.

Technical Details¶

This is an example of the bond configuration on your server.

cat /proc/net/bonding/bond0
>
Ethernet Channel Bonding Driver: v5.15.0-30-generic

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 200
Peer Notification Delay (ms): 0

802.3ad info
LACP active: on
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 0c:c4:7a:86:35:b0
Active Aggregator Info:
    Aggregator ID: 1
    Number of ports: 2
    Actor Key: 9
    Partner Key: 12
    Partner Mac Address: 9c:cc:83:50:1d:a0

Transmit Hash Policy¶

Something important on that configuration is the Transmit Hash Policy: layer3+4 option, which is a balancing algorithm mode. layer3+4 is a policy that uses upper layer protocol information, when available, to generate the hash. This allows for traffic to a particular network peer to span multiple bond members, although a single connection will not span multiple bond members.

Note: Transmit Hash Policy is used in the bond member selection only for outgoing traffic. This means that you will be able to use the full network bandwidth when creating multiple connections.

Iperf Implications¶

When testing bandwidth throughput with iperf, it will use ONLY 1 interface on the client, hence, only half of the total bandwidth. But, if you test it with 1 master, and 2 clients, you will be able to see the full bandwidth used on the master.