Pure Layer 2 Modes¶
Equinix Metal™ allows users to change the networking mode of a server from the default Layer 3 Bonded mode to modes that only use Layer 2 networking over VLANs. This means all access to the public Internet is lost, and the host can only be reached by the Serial Over SSH (SOS) console or from other servers attached to the same VLAN.
You can either have both interface in a bonded configuration or you can have both interfaces separated so that you can assign different VLANs to each interface.
Layer 2 Bonded Mode
- This option will preserve the LACP bond and allow you to add Layer 2 networks to the bonded interface.
- This provides redundancy and HA, and an outage, maintenance, or reboot of one of the interfaces will not cause network interruptions.
- On the server, all traffic has to be tagged, and need to create sub-interfaces for each VLAN assigned to the bond. Untagged packets and native VLANs are not supported.
Layer 2 Unbonded Mode
- This option will destroy the LACP bond and allow you to place each interface into individual Layer 2 networks.
- This does introduce a single point of failure on the upstream switch or the network interface. An outage, maintenance, or reboot of one of the ports will cause network interruptions.
- On the server, you need to destroy the bond. If you have more than one VLAN assigned to a port, you can create sub-interfaces for each of them. Since the ports are not part of a bond in this mode, there is support for un-tagged packets and native VLANs.
Note: both of the Layer 2 networking configurations will permanently remove the server's public IP addresses. If the server is later converted back to Layer 3, new IP addresses will be assigned.
Converting to Layer 2 Bonded Mode¶
To use Bonded Layer 2 mode, you must first change your networking configuration to Layer 2. From the portal, navigate to the server's Network page, click Convert To Other Network Type and choose Layer 2.
In the Equinix Metal console, navigate to the server's Network page, click Convert To Other Network Type and choose Layer 2. Select the Bonded mode. Click Convert to Layer 2 to start the conversion.
When it is finished, you will be back on the server's Network page.
In the Layer 2 section, click Add New VLAN, which will allow you to assign a VLAN to the bond0
port. Click Add to start the changes.
Convert the bonded interface to Layer 2 with the metal port convert command, specifying the UUID of bond0
for the --port-id
.
metal port convert --port-id <bond0_id> --layer2 --bonded
Then, assign a VLAN to bond0
with the metal port vlan
command. Specify the UUID of bond0
for the --port-id
and the VLAN that you are assigning to --assign
.
metal ports vlan --port-id <bond0_id> --assign <VLAN_id>
In the API, you can convert the bonded interface to Layer 2 and assign the VLAN by submitting a POST
request to the /ports/{id}/convert/layer-2
endpoint, with the port ID specified in the path. For Layer 2 Bonded mode, it needs to be the UUID for bond0
as returned by the /devices/{id}
endpoint.
The VLAN_ID
is sent in the body of the request, and it can be either the VLAN's UUID as returned by the /projects/{id}/virtual-networks
endpoint or the ID that is in the console.
curl -X POST \
-H "Content-Type: application/json" \
-H "X-Auth-Token: <API_TOKEN>" \
"https://api.equinix.com/metal/v1/ports/{id}/convert/layer-2" \
-d '{
"vnid": "<VLAN_ID>"
}'
Remember: once you have clicked Convert to Layer 2 in the console, or submitted the request to the API, the existing IP Addresses will be removed and the server will no longer be accessible through the Internet. You will still be able to use the SOS console to connect.
Server Configuration - Bonded¶
Once you have assigned a VLAN to the interface, you will need to configure the networking on the server's operating system to use the VLAN on bond0
. Remember, you will have to use the SOS console to connect.
-
Enable VLAN support.
modprobe 8021q lsmod | grep 8021q echo "8021q" >> /etc/modules-load.d/networking.conf
-
Add the VLAN to
bond0
.VLAN_ID
should match the ID found on the console or the UUID from the API.ip link add link bond0 name bond0.<VLAN_ID> type vlan id <VLAN_ID>
For example,
ip link add link bond0 name bond0.1036 type vlan id 1036
-
Add
192.168.100.1
IP address to the VLAN. IP addresses in the 192.168 range are recommended, as the10.0.0.0/8
range is used internally by Equinix Metal. For example,ip addr add 192.168.100.1/24 brd 192.168.100.255 dev bond0.1036 ip link set dev bond0.1036 up
To make the changes permanent, configure
/etc/network/interfaces
with the IP address to the desired IP from your chosen block. For example,auto bond0.1036 iface bond0.1036 inet static pre-up sleep 5 address 192.168.100.1 netmask 255.255.255.0 vlan-raw-device bond0
Note: The line
pre-up sleep 5
helps to prevent conflicts onbond0
when the server boots. -
Ensure the IP address is configured.
ip -d link show bond0.1036
-
Enable VLAN support.
echo 8021q > /etc/modules-load.d/8021q.conf
-
Add the VLAN to
bond0
.VLAN_ID
should match the VLAN ID found on the console.nmcli connection add type vlan con-name bond0.<VLAN_ID> ifname bond0.<VLAN_ID> vlan.parent bond0 vlan.id <VLAN_ID>
For example,
nmcli connection add type vlan con-name bond0.1036 ifname bond0.1036 vlan.parent bond0 vlan.id 1036
-
Add
192.168.100.1
IP address to the VLAN. IP addresses in the 192.168 range are recommended, as the10.0.0.0/8
range is used internally by Equinix Metal. For example,nmcli connection modify bond0.1036 ipv4.addresses '192.168.100.1/24' nmcli connection modify bond0.1036 ipv4.gateway '192.168.100.1' nmcli connection modify bond0.1036 ipv4.method manual nmcli con up bond0.1036
You need to run through the same steps on all the servers that you are attaching to the VLAN, assigning a different IP address to each.
Attaching Multiple VLANs - Bonded¶
To assign multiple VLANs to an interface, navigate to the server's Network page. In the Layer 2 section, click Add New VLAN, which will allow you to assign additional VLANs to the bond0
port.
Click Add to start the changes. Note that if you assign multiple VLANs at once, they are added through an asynchronous batch process, which begins immediately, but may take some time to complete.
To assign another or multiple more VLANs, use the metal port vlan
command, and specify each VLAN you are assigning to bond0
.
metal port vlan --port-id <bond0_id> --assign <vlan> --assign <vlan>
The procedure for assigning a single additional VLAN to an interface in the API is the same as assigning the first VLAN. Send a POST
request to the /ports/{id}/assign
endpoint.
curl -X POST \
-H "Content-Type: application/json" \
-H "X-Auth-Token: <API_TOKEN> " \
"https://api.equinix.com/metal/v1/ports/{id}/assign" \
-d '{
"vnid": "c4032b18-5494-451b-a779-a7d3c536bfd7"
}'
It is also possible to assign VLANs to a port in bulk, as part of an asynchronous batch process. Send a POST
request to the /ports/{id}/vlan-assignments/batches
endpoint an array of VLAN assignments in the body of the request.
curl -X POST \
-H "Content-Type: application/json" \
-H "X-Auth-Token: <API_TOKEN> " \
"https://api.equinix.com/metal/v1/ports/{id}/vlan-assignments/batches" \
-d '{
"vlan_assignments": [
{
"vlan": "string",
"state": "assigned"
},
{
"vlan": "string",
"state": "assigned"
},
]
}'
Once the VLAN (or VLANs) is assigned to the port, you repeat the process in Server Configuration - Bonded to create the additional tagged interfaces.
Converting to Layer 2 Unbonded Mode¶
In the Equinix Metal console, navigate to the server's Network page, click Convert To Other Network Type and choose Layer 2. Select the Unbonded mode. Click Convert to Layer 2 to start the conversion.
When it is finished, you will be back on the server's Network page.
To assign a VLAN to one of the ports, click Add New VLAN, and select either the eth0
or eth1
interface. Click Add to start the changes.
Convert the bonded interface to Layer 2 while removing both port interfaces from the bond with the metal port convert command, specifying the UUID of bond0
for the --port-id
and using the --unbonded
flag.
metal port convert --port-id <bond0_id> --layer2 --unbonded
Then you can add VLANs to either eth0
or eth1
(or both) with the metal port vlan
command. Specify the UUID of eth0
or eth1
for the --port-id
and the VLAN that you are assigning to --assign
.
metal port vlan --port-id <eth1_id> --assign <VLAN_id>
The VLAN_id
can be either the VLAN's UUID or the VLAN ID number that is in the console.
In the API, converting to Layer 2 Unbonded mode is a two-step process. You have to first break the bond for all ports by sending a POST
request to the /ports/{id}/disbond
endpoint.
You have to specify the port ID of eth1
in the path, and it needs to be the UUID for that port as returned by the /devices/{id}
endpoint.
Set the bulk_disable
field to false
in the body of the request.
curl -X POST -H \
"Content-Type: application/json" \
-H "X-Auth-Token: <API_TOKEN>" \
"https://api.equinix.com/metal/v1/ports/{id}/disbond" \
-d '{
"bulk_disable": false
}'
Then, you need to assign the VLAN to eth1
. Send a POST
request to the /ports/{id}/assign
endpoint.
You have to specify the port ID of eth1
, and it needs to be the UUID for that port as returned by the /devices/{id}
endpoint.
The ID of the VLAN is sent in the body of the request, and it can be either the VLAN's UUID as returned by the /projects/{id}/virtual-networks
endpoint or the vxlan
ID that is in the console.
curl -X POST -H \
"Content-Type: application/json" \
-H "X-Auth-Token: <API_TOKEN>" \
"https://api.equinix.com/metal/v1/ports/{id}/assign" \
-d '{
"vnid": "1173"
}'
Server Configuration - Unbonded¶
The following configuration steps can be performed on both eth0
and eth1
, depending on which ports you are assigning a VLAN to. The examples use eth1
.
If only one VLAN is enabled on a port, packets are untagged. This means that the server's network configuration does not need to be VLAN-aware.
-
Make sure
eth1
has been removed frombond0
:cat /sys/class/net/bond0/bonding/slaves
If it hasn't been removed, remove it:
echo "-eth1" > /sys/class/net/bond0/bonding/slaves
-
Bring down the interface:
sudo ifdown eth1
-
Configure
/etc/sysconfig/network-scripts/ifcfg-eth1
on each of the servers, changing theIPADDR
field to the desired IP and network. Ensure the IP addresses are different on each server that you are attaching to the same VLAN. For example,DEVICE=eth1 ONBOOT=yes HWADDR=e4:1d:2d:11:22:33 IPADDR=192.168.1.2 NETMASK=255.255.255.0 NETWORK=192.168.1.0 BOOTPROTO=none
-
Bring up the interface:
sudo ifup eth1
-
Make sure eth1 has been removed from
bond0
:cat /sys/class/net/bond0/bonding/slaves
If it hasn't been removed, remove it:
echo "-eth1" > /sys/class/net/bond0/bonding/slaves
-
Bring down the eth1 interface:
sudo ifdown eth1
-
Configure
/etc/network/interfaces
on each server, changing the IP address to the desired IP from your chosen block. For example,auto eth1 iface eth1 inet static address 192.168.1.2 netmask 255.255.255.0
-
Bring up the interface:
sudo ifup eth1
Attaching Multiple VLANs - Unbonded¶
To assign multiple VLANs to a port, navigate to the server's Network page. In the Layer 2 section, click Add New VLAN, which will allow you to assign more VLANs to eth0
or eth1
.
Click Add to start the changes. Note that if you assign multiple VLANs at once, they are added through an asynchronous batch process, which begins immediately, but may take some time to complete.
To assign another or multiple more VLANs to either eth0
or eth1
, use the metal port vlan
command, specify each VLAN you are assigning, and with port you are assigning it to.
metal port vlan --port-id <eth0_id> --assign <vlan> --assign <vlan>
The procedure for assigning a single additional VLAN to a port is the same as assigning the first VLAN. Send a POST
request to the /ports/{id}/assign
endpoint.
You have to specify the port ID in the path, and it needs to be the UUID for that port as returned by the /devices/{id}
endpoint.
The ID of the VLAN is sent in the body of the request, and it can be either the VLAN's UUID as returned by the /projects/{id}/virtual-networks
endpoint or the VLAN ID that is in the console.
curl -X POST \
-H "Content-Type: application/json" \
-H "X-Auth-Token: <API_TOKEN> " \
"https://api.equinix.com/metal/v1/ports/{id}/assign" \
-d '{
"vnid": "c4032b18-5494-451b-a779-a7d3c536bfd7"
}'
It is also possible to assign VLANs to a port in bulk, as part of an asynchronous batch process. Send a POST
request to the /ports/{id}/vlan-assignments/batches
endpoint an array of VLAN assignments in the body of the request.
curl -X POST \
-H "Content-Type: application/json" \
-H "X-Auth-Token: <API_TOKEN> " \
"https://api.equinix.com/metal/v1/ports/{id}/vlan-assignments/batches" \
-d '{
"vlan_assignments": [
{
"vlan": "string",
"state": "assigned"
},
{
"vlan": "string",
"state": "assigned"
},
]
}'
In this scenario, IP packets that arrive at the server will have the VLAN ID populated. You will need to setup interfaces that will receive packets destined for each VLAN.
Note: If you need support for untagged packets, you can set a native VLAN for either port, since neither of them is part of a bond. More information is on the Setting a Native VLAN page.
-
Install the prerequisites for VLANs:
sudo modprobe 8021q sudo echo "8021q" >> /etc/modules
-
Bring down
eth1
:ifdown eth1
-
Configure
/etc/sysconfig/network-scripts/ifcfg-eth1.1000
and/etc/sysconfig/network-scripts/ifcfg-eth1.1001
on your server.1000
and1001
should match the VLANs you've configured on the host in the portal. For example,DEVICE=eth1.1000 BOOTPROTO=none ONBOOT=yes IPADDR=192.168.1.2 PREFIX=24 NETWORK=192.168.1.0 VLAN=yes
-
Restart networking:
sudo ifup eth1.1000 sudo ifup eth1.1001
-
Install the prerequisites for VLANs:
sudo apt-get install vlan sudo modprobe 8021q sudo echo "8021q" >> /etc/modules
-
Bring down
eth1
:ifdown eth1
Note: if you don't want
eth1
to come up after a reboot be sure to comment out theeth1
configuration in your/etc/network/interfaces
file. -
Add the new interface to
/etc/network/interfaces
.1000
and1001
should match the VLANs you've configured on the host in the portal. For example,auto eth1.1000 iface eth1.1000 inet static address 192.168.100.1 netmask 255.255.255.0 vlan-raw-device eth1 auto eth1.1001 iface eth1.1001 inet static address 172.16.100.1 netmask 255.255.255.0 vlan-raw-device eth1
-
Restart networking:
sudo ifup eth1.1000 sudo ifup eth1.1001
Example: Combining Hybrid and Layer 2 Modes¶
For this configuration you'll need two nodes, one in hybrid mode and one in Layer 2 networking mode, and one VLAN. The example assumes that the Hybrid mode server is in Hybrid Unbonded mode. The example also assumes that you have the server using exclusively Layer 2 in Layer 2 Unbonded mode, and assigns the VLAN to the eth1
interface.
-
Attach the VLAN to the hybrid node's interface,
eth1
in the case of a Hybrid Unbonded Mode. -
Attach that same VLAN to the server in Layer 2 mode. Remember, this node is in pure Layer 2 networking mode; there is no public connectivity to this server and you will have to use SOS to connect.
If you get locked out, you can always change the networking mode back to Layer 3, or hybrid mode and SSH back in via the public IPv4 address.
While connected to SOS to the server in Layer 2 mode, edit the network interfaces file and remove all but the eth1
interface, which should be configured with its own private IP from whichever block you choose to use (e.g. 192.168.2.0/24). You'll also need to specify the gateway address as the hybrid node's IP address. This process varies by operating system.
-
Tear down the bond0 interface:
sudo ifdown bond0
-
Configure
/etc/sysconfig/network-scripts/ifcfg-eth1
with any free IP from the IPv4 private block used by eth1 on the hybrid node. Ensure that the netmask, network, and gateway details are correct.DEVICE=eth1 ONBOOT=yes HWADDR=e4:1d:2d:11:22:32 IPADDR=192.168.2.2 NETMASK=255.255.255.0 GATEWAY=192.168.2.1 NETWORK=192.168.2.0 BOOTPROTO=none
-
Bring up eth1.
sudo ifup eth1
You can set the "ONBOOT" parameter for the rest of the network interfaces to "no" so they do not come up one reboots.
bond0
will not be used, andeth0
will only be used if you choose to connect it to another VLAN (perhaps connected to other isolated node). In which case, it should be configured with its own IP accordingly.
-
Tear down the bond0 interface:
sudo ifdown bond0
-
Configure
/etc/network/interfaces
with any free IP from the IPv4 private block used byeth1
on the hybrid node. Ensure that the netmask, network, and gateway details are correct.auto eth1 iface eth1 inet static address 192.168.2.2 netmask 255.255.255.0 gateway 192.168.2.1
-
Bring up eth1.
sudo ifup eth1
You can remove the other interfaces from this file.
bond0
will not be used, but if you connecteth0
to another VLAN (perhaps connected to other isolated nodes) then configure it with its own IP, accordingly.
At this point your Hybrid mode server and your Layer 2 mode server node can talk to each other, but the Layer 2 server cannot reach the Internet. To give it Internet access you must configure IP masquerading on the Hybrid mode server.
- Make sure IP forwarding is enabled on the Hybrid mode server.
sysctl net.ipv4.ip_forward=1
- Now add a new IP masquerade rule to the NAT table with
iptables
. We want this to route traffic from any of our private IPs through the Internet facing network interface on the Hybrid mode server, in this case,bond0
.
iptables -t nat -A POSTROUTING -s 192.168.2.0/24 -o bond0 -j MASQUERADE
Now your Layer 2 mode server should be able to ping outside the network.
ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp\_seq=1 ttl=120 time=1.85 ms
64 bytes from 8.8.8.8: icmp\_seq=2 ttl=120 time=1.93 ms
64 bytes from 8.8.8.8: icmp\_seq=3 ttl=120 time=1.87 ms
64 bytes from 8.8.8.8: icmp\_seq=4 ttl=120 time=1.86 ms
64 bytes from 8.8.8.8: icmp\_seq=5 ttl=120 time=1.81 ms
Converting Back to Layer 3¶
If you are in one of the Layer 2 modes and want to go back to Layer 3, you must first remove any attached VLANs.
To detach a VLAN in the console, navigate to the server's Network page. In the Layer 2 section, click Remove next to the VLAN you are detaching from the server. Confirm that you wish to remove it by clicking Yes.
Note that detaching the VLAN from this server does NOT delete it from your project. The VLAN will continue to exist after detaching it from the server.
Then, to convert back to Layer 3, click Convert To Other Network Type, select Layer 3. Click Convert to Layer 3 to start the process.