Elastic IPs with BGP

In our Elastic IPs Getting Started Guide, we got you started on what Elastic IPs are, how to use them, and how to assign them to servers using the Equinix Metal API and Console.

That works pretty well, but it has some downsides. First, it is a bit of a pain to assign the IPs to each server. Second, if the server goes down, you have to reassign the IP to another server. Until you do, your service is down. Third, if you want to have multiple servers handle the same IP, you cannot; sorry. Finally, the reassignment process can take a few minutes, even discounting the time it takes for you to notice that the original server is down and reassigning it.

That's where BGP comes in.

Equinix Metal supports using BGP for one or more servers to announce to the Equinix Metal routing infrastructure that they can handle IPs.

This is very similar to assigning the IPs using the Equinix Metal API, except that:

• the servers announce the routes they can handle autonomously
• the route updates are very quick
• multiple servers can handle the same IP in parallel.

Many people find BGP confusing, even intimidating. So, before we keep going on assigning the Elastic IP via BGP, we will spend a few moments describing what BGP is and how it works. If you already know all of that, feel free to jump ahead to the deploying section.

BGP

BGP is an internet protocol that lets routers and servers announce, "I can handle traffic for the following IP ranges."

These messages are called "announcements." The announcements are sent from a network device, such as a router or server, to a BGP peer. The BGP peer is usually a router upstream, which is expected to receive packets for the announced IP ranges and now know, "I can forward them to this device."

Most of the time, this is useful for general Internet routing. This is how, for example, your bank (or its Internet Service Provider) tells the rest of the Internet, "I can handle traffic for 100.125.150.175", allowing the rest of the Internet to send traffic to your bank's servers.

In order for it to work, you need three things:

• A router that can receive BGP announcements. This is usually a router that is connected to the Internet, and is usually provided by your ISP.
• A network device that can send BGP announcements. This is usually a router or server.
• An IP address range that you actually own and can announce.

In the diagram, 100.125.150.175 is the IP address that the servers can handle. Their router announces that address (the green "announcements") to the Internet routers (their "peers"). This in turn lets them send any traffic for 100.125.150.175 to the router, which will then forward it to the servers.

BGP Information

In order to send BGP announcements, you need to know several pieces of configuration information.

First, you need to know the IP address of the routers to which you are sending the announcements. These are called "BGP peers." In the diagram above, the BGP peers are the three routers that are connected to the Internet.

Second, you need to know your Autonomous System Number, or ASN. BGP is a Border Gateway Protocol, used between independent systems, hence, the name "Autonomous Systems." You need a unique number to identify your network to the peers. Those peers, in turn, may be configured to accept only valid ASNs, or only certain ASNs from certain senders.

This brings us to our third requirement: your local address. Some peers will accept announcements from any address, although this is not common. Most peers will accept announcements only from a specific set of addresses for specific ASNs that they recognize. In the case of Equinix Metal, the peers will accept only from:

1. your servers' private, management IPv4 addresses;
2. to specific peer IPs;
3. the ASN used when enabling BGP in the Equinix Metal project.

Why the complexity?

Routers are not exactly the easiest devices to configure, but they do already have BGP software installed. Indeed, their basic operating software is built around network protocols, like BGP.

In our case, we are not worrying too much about the upstream routers, or "peers." Fortunately, Equinix Metal takes care of those.

We do, however, have to configure the servers both to announce the IPs and to be prepared to receive traffic for those addresses. This is where it gets a bit tricky.

Configuring servers to announce the IPs means you need:

• the receiving peer router IPv4 addresses,
• the announcing local IPv4 addresses
• the ASN number,
• an MD5 password (optional),
• a BGP speaker (software that can handle the BGP protocol).

Deploying

Before you start using BGP for the Elastic IP, you'll need to log back into the console and detach the Elastic IP from whichever server currently has it.

To use BGP, you need to:

1. Enable BGP on the Equinix Metal project, if it was not already done.
2. Enable BGP on each server in the Equinix Metal project.
3. Configure BGP on each server to announce the routes.

To enable BGP on your project, select Networking, then BGP from the left navigation menu, then enable BGP. Once it is set, you will see a Local BGP Enabled message. To learn more about the difference between Local and Global BGP, check out the Local vs Global BGP documentation.

Next, for each server, select the server, and then select the BGP tab at the top.

You can see that there is no Status for the server, meaning there is no BGP enabled.

Click the arrow to the right of the IPv4 row, which brings up an Activate BGP sidebar.

Enable BGP for the server using the switch on the right:

The state for IPV4 will be Unknown until setup is complete:

Repeat the process for all three of your servers, and wait a few minutes for the status to change from Unknown to Down.

The status is Down, because BGP is ready, but has not received any communications from the servers. That is the next task.

Finally, you'll configure BGP on your servers to announce the Elastic IP address. We will use bird as the BGP speaker for our routes, meaning the bird software on each server will "announce" the IPs it can handle to the Equinix Metal routers.

Configuring bird can be a bit trickier. You need the following for each server:

• The elastic IP address you gathered earlier. This is shared across all of the servers. In our example, that is 145.40.77.88, but it will be different for you.
• The local private IP for each server. We are going to get that from Metal metadata.
• The IPs of the upstream BGP peers. We are going to get that from Metal metadata.
• The ASN of the upstream BGP peers. We are going to get that from Metal metadata.
• The IP of the upstream router.
• The interface to use. In all of our examples, we are going to use bond0.

On each server, do the following:

1. SSH to the server.
2. Get the information from the metadata with the following command: curl https://metadata.platformequinix.com/metadata | jq '.bgp_neighbors[0] | { customer_ip: .customer_ip, customer_as: .customer_as, multihop: .multihop, peer_ips: .peer_ips, peer_as: .peer_as }'
3. Create the bird configuration file.
4. Restart bird.

You added the Elastic IP to the lo loopback interface in the previous guide, so bird should just pick it up, if you configure it correctly.

root@c3-small-x86-01:~# curl -s https://metadata.platformequinix.com/metadata | jq '.bgp_neighbors[0] | { customer_ip: .customer_ip, customer_as: .customer_as, multihop: .multihop, peer_ips: .peer_ips, peer_as: .peer_as }'
{
  "customer_ip": "10.70.68.3",
  "customer_as": 65000,
  "multihop": true,
  "peer_ips": [
    "169.254.255.1",
    "169.254.255.2"
  ],
  "peer_as": 65530
}

The upstream peers are at 169.254.255.1 and 169.254.255.2, the customer IP, or the expected private IP, is 10.70.68.3, and the peer ASN is 65530.

Equinix Metal actually uses both the peer ASN and the upstream peer IPs as pretty much the same across its entire plant, but you should not rely upon it. Instead, use the metadata as we did above.

The only missing piece is the upstream router IP, which you can get via ip route list 10.0.0.0/8:

root@c3-small-x86-01:~# ip route list 10.0.0.0/8
10.0.0.0/8 via 10.70.68.2 dev bond0

So in this case, our upstream router IP is 10.70.68.2.

Next, you need to create the bird configuration file /etc/bird/bird.conf. Chances are pretty good that a default file was installed, so you'll need to replace it with this one. We put comments in to show where we placed our own information.

filter metal_bgp {
  if net = 145.40.77.88/32 then accept;  # Elastic IP
}

router id 10.70.68.2;   # Local private IP
protocol direct {
  interface "lo";
}

protocol kernel {
  persist;
  scan time 20;
  import all;
  export all;
}

protocol device {
  scan time 10;
}

protocol static {
  route 169.254.255.1/32 via 10.70.68.2; # Upstream router IP
  route 169.254.255.2/32 via 10.70.68.2; # Upstream router IP
}

protocol bgp neighbor_v4_1 {
  export filter metal_bgp;
  local as 65000;   # Our ASN
  multihop;
  neighbor 169.254.255.1 as 65530; # Upstream peer IP and ASN
}

protocol bgp neighbor_v4_2 {
  export filter metal_bgp;
  local as 65000;  # Our ASN
  multihop;
  neighbor 169.254.255.2 as 65530; # Upstream peer IP and ASN
}

To make your life easier, you can use this template, with fields to be filled in marked in ALL CAPS:

filter metal_bgp {
  if net = ELASTICIP/32 then accept;  # Elastic IP
}

router id PRIVATEIP;   # Local private IP
protocol direct {
  interface "lo";
}

protocol kernel {
  persist;
  scan time 20;
  import all;
  export all;
}

protocol device {
  scan time 10;
}

protocol static {
  route PEERIP1/32 via ROUTERIP; # Upstream router IP
  route PEERIP2/32 via ROUTERIP; # Upstream router IP
}

protocol bgp neighbor_v4_1 {
  export filter metal_bgp;
  local as 65000;   # Our ASN
  multihop;
  neighbor PEERIP1 as 65530; # Upstream peer IP and ASN
}

protocol bgp neighbor_v4_2 {
  export filter metal_bgp;
  local as 65000;  # Our ASN
  multihop;
  neighbor PEERIP2 as 65530; # Upstream peer IP and ASN
}

Now restart bird:

root@c3-small-x86-01:~# systemctl restart bird

bird should be working now and connected to the upstream BGP peers. You can check by asking it with birdc. If it is working, the connection should be listed as Established.

root@c3-small-x86-01:~# birdc show protocol neighbor_v4_1
BIRD 1.6.8 ready.
name     proto    table    state  since       info
neighbor_v4_1 BGP      master   up     18:52:38    Established
root@c3-small-x86-01:~# birdc show protocol neighbor_v4_2
BIRD 1.6.8 ready.
name     proto    table    state  since       info
neighbor_v4_2 BGP      master   up     18:52:39    Established

Next, repeat the process on the other two servers.

Test the configuration

At this point, you have:

• One Elastic IP
• Not assigned to any one server by Equinix Metal API
• With all three servers announcing it via BGP

The next step is to try to reach it from the outside world, using curl.

$ curl http://145.40.77.88
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>I am server 02</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

The test was successful. This time, server 02 answered. Now try it again:

$ curl http://145.40.77.88
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>I am server 01</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

This time server 01 answered, so you know that multiple servers are using the same Elastic IP address, as desired.

Conclusion

In this guide, you discovered the shortcomings of assigning an Elastic IP by hand. You also learned how BGP overcomes that limitation. You learned how to configure BGP in your Equinix Metal project, and how to configure it using bird. From here, you can add more servers to your project if you want, with confidence that BGP will use the same Elastic IP address for them.