Connecting your Metal VLAN to AWS

How do you connect your private layer 2 VLAN on Equinix Metal with a VPC on Amazon Web Services (AWS)?

In this guide, we will walk you through connecting your Equinix Metal VLAN to a VPC that you own on AWS. This guide sets up the underlying networking elements directly, which gives you greater control. There is a simple, less-featured method using an Equinix Metal simplified connection in the console, which will be described in a later guide.

1. Architecture
2. Prerequisites
3. Deploy VRF
4. Deploy Equinix Metal Gateway
5. Deploy Fabric Virtual Connection (VC)
6. Test connection

1. Architecture

Before we get started, let's look at what the architecture of the connection looks like.

On the AWS side, we have a VPC, with multiple subnets. The IP address range for the VPC, and for each of the subnets, is as assigned by you. The VPC connects with the rest of the world via a standard AWS Transit Gateway.

On the Equinix Metal side, we have a VLAN, with IP addresses assigned by you as well. The VLAN may or may not connect to standard Equinix Metal Layer 3 networking via an Equinix Metal Gateway.

The VLAN is connected to an Equinix Metal Gateway, which is connected to an instance of Equinix Metal Virtual Routing and Forwarding (VRF). VRF is connected to an Equinix Fabric Virtual Connection (VC). The VC is connected on one end to the VRF, and on the other end to the VPC Gateway.

2. Prerequisites

The prerequisites for connecting your VLAN to your AWS VPC are:

• An AWS account
• A VPC created in your AWS account, along with a VPC Transit Gateway
• An Equinix Metal account, with a project in it
• A VLAN deployed to your Equinix Metal project
• Configuration information

The configuration information is the following:

Item	Purpose	Example Values
AWS VPC	CIDR range for the VPC	10.20.0.0/16
AWS Subnets	CIDR ranges for the subnets	10.20.10.0/24, 10.20.11.0/24, 10.20.12.0/24
Metal VRF	CIDR range for the VRF	10.50.0.0/16
Metal VLAN	CIDR range for the VLAN, which must be within the VRF range	10.50.10.0/24
AWS ASN	The Autonomous System Number for the AWS side	64550
Metal ASN	The Autonomous System Number for the Metal side	64600
Primary VC subnet	Subnet for the primary virtual connection, should be very small, as we will use just 2 addresses	192.168.15.0/30
Primary VC IP	IPs within the primary VC subnet to assign to each side	192.168.15.1 Metal and 192.168.15.2 AWS
Secondary VC subnet	Subnet for the secondary virtual connection, should be very small, as we will use just 2 addresses	192.168.15.4/30
Secondary VC IP	IPs within the secondary VC subnet to assign to each side	192.168.15.5 Metal and 192.168.15.6 AWS
MD5 password	A password to use for BGP peering, MD5-hashed	Clear text of "What A Great Guide!", hashed to dbc29529874661f92e089593c8adc10b

Since this is not an AWS users' guide, we will not walk through setting up the AWS components here. When complete, you should have your account, a VPC with a Transit Gateway, and several subnets in that VPC.

For Equinix Metal, we have guides to help with setting up your account, organization and project.

Once you have your project, select Networking:

then VLAN:

then "Add a VLAN":

Pick a Metro, and any VLAN ID that is convenient for you, or let Metal pick it automatically. For our example, we will use VLAN ID 200 in Washington. We are using Washington, because it is the same metro where Amazon's us-east-1 region is located.

3. Deploy VRF

With our VLAN in place, we now need a VRF that will be used to connect to Equinix Fabric. The VRF is under "Networking", like "VLANs". Click on "Virtual Routing and Forwarding":

then click "Create Virtual Router":

In order to create the Virtual Router, you need a few pieces of information:

• A name for the VRF. It doesn't have any inherent meaning; it just needs to be useful to you. We will call it "Cloud".
• The Metro. This must be the same metro as the VLAN you are connecting to. In our case, it is Washington.
• The ASN. This is the Metal-side ASN we selected earlier, 64600.
• The "allowed IP ranges". These are CIDR ranges that will be "behind" the VRF on the Metal side. Whatever ranges you pick here, these are the ranges that the VRF will tell Fabric, "send those addresses to me, I can handle them." This must include:
  • CIDRs used for Metal Gateways; we use the range we reserved earlier 10.50.0.0/16
  • CIDRs used for subnets for the Fabric VCs; we selected the primary and secondary ranges of 192.168.15.0/30 and 192.168.15.4/30, but VRF only allows up to /29, so use 192.168.15.0/29, which covers both.

Finally, once the VRF is created we need to reserve IP ranges to use from within the allowed IP ranges. For this particular VLAN we use the range we described above, 10.50.10.0/24, which is within the larger VRF allowed IP ranges.

Click on the VRF:

This brings up the VRF details. Click on "Add IP Reservation":

Then enter the range 10.50.10.0/24 and click "Submit Request":

4. Deploy Equinix Metal Gateway

We now have a VLAN and a VRF. Let's create a Metal Gateway to link them together. "Metal Gateway" is under "Networking", like "VLANs" and "Virtual Routing and Forwarding". Click on "Metal Gateways":

then click "Create a Metal Gateway":

In order to create the Metal Gateway, you need a few pieces of information:

• The Metro. This must be the same metro as the VLAN you are connecting to. In our case, it is Washington.
• The VLAN. This is the VLAN that the Metal Gateway will connect to. We will use VLAN 200 which we created earlier.
• The IP block. This is one of a reserved public IPv4 range, a private IPv4 range, or a VRF IP range. Since we are connecting to a VRF, we will pick "VRF IP", and then the IP address reservation from the VRF.

5. Deploy Fabric Virtual Connection

There are several steps to deploying the Fabric Virtual Connection (VC):

1. Create the connection in Equinix Metal to Fabric
2. Create the Fabric VC
3. Accept the connection in AWS
4. Update BGP peering in Metal

Create the connection in Equinix Metal to Fabric

From the left-hand-side, select "Interconnections":

then "Request a new interconnection":

There are two kinds of interconnection you can request, "AWS Direct Connect" and "Request Interconnection". Although we are connecting to AWS, do not select "AWS Direct Connect". That is a simplified version, easier to deploy but not as flexible. Instead, select "Request Interconnection":

The next screen gives us two options for interconnections, "Fabric VC" and "Dedicated Port". We won't cover dedicated ports in this guide, as we are focused on using Equinix Fabric. If you are interested in learning more about dedicated ports, please see our dedicated ports documentation, also available via the link in the "Dedicated Ports" box on the console.

Note that there are two kinds of Fabric VC, "Metal-billed" and "Fabric-billed". Either can be useful for this kind of interconnection, but we will use the Metal-billed one. Read the interconnections documentation for more information on the differences.

In order to create the connection, you need a few pieces of information:

• A name for the connection. It doesn't have any inherent meaning; it just needs to be useful to you. We will call it "AWS".
• The Metro. This must be the same metro as the VRF you are connecting to. In our case, it is Washington.
• The connection speed. Higher speeds have a higher cost. Since we are just creating a basic connection, we will use the lowest 50Mbps speed. You can select a higher one for your purposes.
• Single vs redundant. Each connection is exactly that, a single connection. If there are issues anywhere along the line, whether hardware such as a port on the AWS or Equinix end, or software, your connection will be out of service, at least until it is fixed. Like all good networking, you should plan for redundancy. We will use a redundant connection for this example.
• Connect to VLAN or VRF. The VC is a layer 2 connection between layer 3 routers with BGP support on each size. On the cloud provider side, the layer 3 is provided for you. On the Metal side, you can set up the layer 3 routing and BGP manually with software on Metal devices and additional VLANs connected directly to the VC, or you can let Metal do all of that for you with a VRF. We will use a VRF connection for this example.
• Primary and secondary connections. Since we selected redundant connections, we need to specify where each of the two connections attaches on the Equinix Metal side. In the case of a VLAN, that would require two distinct VLANs. Since we are using VRF, which is Virtual Routing and Forwarding, we can use the same VRF for both connections. This is a key advantage of VRF connections over VLAN connections. We will use our "Cloud" VRF for both.

Submit the request.

Once the request is submitted, you need two things:

1. An email to your address telling you that the service token is ready to use. You should get to such emails, one for each of the redundant connections. You know it is ready by the "Service Token Status" line.
2. The service token itself. This is a long string of characters that you will need to create the Fabric VC to Equinix Metal. It will appear in the emails, as well as in the console after you submit your request.

Create the Fabric VC

In this section, we create a Fabric VC between our VRF on Metal and our VPC at Amazon.

Hop over to the Fabric console at https://fabric.equinix.com and log in.

Once you are logged in, you will see the Equinix Fabric portal:

On the top menu bar, go to "Connections" and "Create Connection":

There are three main options for a connection: A Service Provider, An Equinix Fabric Customer, and My Own Assets. Click on "A Service Provider", then in the search box, type "AWS". This will filter the options below to just a few. Select "AWS".

This opens up the connection options for AWS. There are two types of connections, normal and high capacity, and single or redundant for each.

We don't need high capacity for our example, so we will select "Normal Capacity".

Once you do, you need to enter the following:

• The connection type, in our case, Redundant
• Your AWS account number
• The destination, which should be the same region as our Equinix Metro. Since we deployed to the Equinix Washington metro, we select "Ashburn (us-east-1)"

In the next step, configure the connection type. Since our interconnection request gave us a pair of service tokens, under "Origin Asset Type" select "Service Token".

Click "Enter Primary Service Token", which opens up the details window. Enter your first service token, and click "Enter Service Token".

Repeat for your secondary service token.

When done, you should have a page that looks like this:

Enter a name for each of the connections, select bandwidth to match what you selected earlier, and click "Next".

Review the details and, when ready, click "Create Connection".

You now should have a page showing the connections created!

Notice how it says that the connection configuration is incomplete, and you must accept it in AWS. That will be our next and final step.

Accept the connection in AWS

Log into your AWS console. In the search box, type "Direct Connect" and select it.

Once in the "Direct Connect" section, select "Connections":

You should see a list of connections there, including our new ones, named "AWS-VRF-primary" and "AWS-VRF-secondary". Both are in "ordering" state. Fabric has requested to connect to AWS, but your account must accept the connection.

Click on the primary connection, and then "Approve" it:

Repeat for the secondary connection.

When you are done, the "Connections" page will show both connections as "pending", until they are finished setting up.

Once they are done, they will show as "Available":

At this point, you have a connection from your VLAN, via an Metal Gateway through VRF over a Fabric VC to your AWS Direct Connect connection.

You now need to link those connections to your VPC Transit Gateway. This is a standard AWS operation.

If you do not have one, create an AWS Direct Connect gateway. Note that you need to provide the Amazon-side ASN. We will use the one we listed earlier, 64500.

When the Direct Connect gateway is setup, it will shows as "Ready". Click on it.

There are two sections to complete the connection:

• Virtual interface attachments are the connections between the Direct Connect gateway and Direct Connect links.
• Gateway attachments are the connections between the Direct Connect gateway and AWS side networks, in our case the Transit Gateway.

Start with the Virtual interface attachments. Click on "Create virtual interface".

• Type: Transit
• Name: pick a representative name; for our case, we will call it fabric-vrf-primary
• Connection: select the connection you want to use, in our case, the primary connection named AWS-VRF-primary
• Interface owner: my account
• Direct Connect gateway: the Gateway we created, i.e. equinix-fabric-vrf
• VLAN: can be any number, but leave it blank, and AWS will tell you what is available when you click "Create"
• BGP ASN: This is the ASN from the far end, i.e. what VRF announces, which we configured earlier, 64600

Expand the "Additional Settings" by clicking on the triangle to see the additional settings:

• "Address family": IPv4
• "Your router peer IP": The IP address on the Metal side 192.168.15.1/30
• "Amazon router peer IP": The IP address on the AWS side 192.168.15.2/30
• "BGP Authentication key": The hashed BGP password dbc29529874661f92e089593c8adc10b

Click "Create virtual interface".

Repeat it for the secondary connection, using secondary connection information:

• Type: Transit
• Name: pick a representative name; for our case, we will call it fabric-vrf-secondary
• Connection: select the connection you want to use, in our case, the primary connection named AWS-VRF-secondary
• Interface owner: my account
• Direct Connect gateway: the Gateway we created, i.e. equinix-fabric-vrf
• VLAN: can be any number, but leave it blank, and AWS will tell you what is available when you click "Create"
• BGP ASN: This is the ASN from the far end, i.e. what VRF announces, which we configured earlier, 64600

Expand the "Additional Settings" by clicking on the triangle to see the additional settings:

• "Address family": IPv4
• "Your router peer IP": The IP address on the Metal side 192.168.15.5/30
• "Amazon router peer IP": The IP address on the AWS side 192.168.15.6/30
• "BGP Authentication key": The hashed BGP password dbc29529874661f92e089593c8adc10b

Click "Create virtual interface".

Finally, connect the Direct Connect gateway to the Transit Gateway. Click on the "Gateway associations" tab and then "Associate gateway".

Select your Transit Gateway, and enter the IP addresses you want to make available. In our case, we will allow the entire VPC of 10.20.0.0/16, but you can have one or more subsets.

When you are done, the Transit Gateway will show as "associated". It may take a few minutes.

Update BGP peering in Metal

If you click on Interconnections, you now will see the connection in the list under "Fabric VCs", with a status of "Pending", which eventually will change to "Ready", when all of the other steps are done. Click on the connection.

Select the "Primary Port" tab:

At the bottom, you will see the "Virtual Circuits" listed. Select the 3 bars on the right to edit it:

This opens the "Manage Peering Details" window. Enter the following information from our configuration table above:

• "Peer ASN": AWS-side ASN 64550
• "Subnet": Primary VC subnet 192.168.15.0/30
• "Metal IP": IP address of the Metal side of the connection 192.168.15.1
• "Customer IP": IP address of the AWS side of the connection 192.168.15.2
• "MD5 Password": dbc29529874661f92e089593c8adc10b

Click "Update".

Repeat the process with "Second Port", using the secondary VC subnet and IP addresses.

• "Peer ASN": AWS-side ASN 64550
• "Subnet": Primary VC subnet 192.168.15.4/30
• "Metal IP": IP address of the Metal side of the connection 192.168.15.5
• "Customer IP": IP address of the AWS side of the connection 192.168.15.6
• "MD5 Password": dbc29529874661f92e089593c8adc10b

Click "Update".

6. Test Connection

We are done! Time to test the connection. We will deploy one server on each end, an EC2 instance in AWS, and a Metal server in our VLAN.

Since this is a guide on the connections, and not on deploying AWS instances or Metal servers, we won't give detailed descriptions here. There are a few things to keep in mind.

For the AWS server, ensure:

• It is in a subnet in the VPC that is connected to the Transit Gateway
• It has a public IP address
• It allows ICMP traffic from both of the private ranges, 10.20.0.0/16 for the VPC, and 10.50.0.0/16 for the Metal VLAN
• Save the private IP of the server. In our example, AWS assigned 10.20.11.170

For the Metal server:

1. Deploy the server normally in the Washington metro.
2. Switch the server networking type to hybrid bonded, so you can ssh to it from the Internet while also connecting to the VLAN, then attach it to the VLAN and assign an IP address. Use the Equinix Metal hybrid bonded networking documentation for detailed instructions.
3. Add the route to the AWS VPC via the Metal Gateway, via ip route add <AWS range> via <Metal Gateway>. In our example, that is ip route add 10.20.0.0/16 via 10.50.10.1.

root@c3-small-x86-01:~# apt-get install vlan
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
  grub-pc-bin
Use 'apt autoremove' to remove it.
The following NEW packages will be installed:
  vlan
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 10.4 kB of archives.
After this operation, 51.2 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 vlan all 2.0.5ubuntu5 [10.4 kB]
Fetched 10.4 kB in 0s (41.5 kB/s)
Selecting previously unselected package vlan.
(Reading database ... 74606 files and directories currently installed.)
Preparing to unpack .../vlan_2.0.5ubuntu5_all.deb ...
Unpacking vlan (2.0.5ubuntu5) ...
Setting up vlan (2.0.5ubuntu5) ...
Processing triggers for man-db (2.10.2-1) ...
Scanning processes...
Scanning processor microcode...
Scanning linux images...

Running kernel seems to be up-to-date.

The processor microcode seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

root@c3-small-x86-01:~# modprobe 8021q

root@c3-small-x86-01:~# echo "8021q" >> /etc/modules

root@c3-small-x86-01:~# ip link add link bond0 name bond0.200 type vlan id 200

root@c3-small-x86-01:~# ip address add 10.50.10.2/24 dev bond0.200

root@c3-small-x86-01:~# ip route add 10.20.0.0/16 via 10.50.10.1

With everything setup, first we will try to ping our Metal Gateway:

root@c3-small-x86-01:~# ping 10.50.10.1
PING 10.50.10.1 (10.50.10.1) 56(84) bytes of data.
64 bytes from 10.50.10.1: icmp_seq=1 ttl=64 time=0.169 ms
64 bytes from 10.50.10.1: icmp_seq=2 ttl=64 time=0.236 ms
64 bytes from 10.50.10.1: icmp_seq=3 ttl=64 time=0.215 ms
^C
--- 10.50.10.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2042ms
rtt min/avg/max/mdev = 0.169/0.206/0.236/0.027 ms

That worked. Now, let's try to ping the router in AWS:

root@c3-small-x86-01:~# ping 10.20.11.1
PING 10.20.11.1 (10.20.11.1) 56(84) bytes of data.
64 bytes from 10.20.11.1: icmp_seq=1 ttl=55 time=2.06 ms
64 bytes from 10.20.11.1: icmp_seq=2 ttl=55 time=1.78 ms
64 bytes from 10.20.11.1: icmp_seq=3 ttl=55 time=1.89 ms
^C
--- 10.20.11.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.780/1.909/2.064/0.117 ms

Great! Finally, let's ping our AWS instance, using the IP we saved from earlier:

root@c3-small-x86-01:~# ping 10.20.11.170
PING 10.20.11.170 (10.20.11.170) 56(84) bytes of data.
64 bytes from 10.20.11.170: icmp_seq=1 ttl=55 time=1.87 ms
64 bytes from 10.20.11.170: icmp_seq=2 ttl=55 time=1.64 ms
64 bytes from 10.20.11.170: icmp_seq=3 ttl=55 time=1.95 ms
^C
--- 10.20.11.170 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.640/1.821/1.952/0.109 ms