Elastic IPs with BGP

In our Elastic IPs Getting Started Guide, we got you started on what Elastic IPs are, how to use them, and how to assign them to servers using the Equinix Metal API and Console.

That works pretty well, but it has some downsides. First, it is a bit of a pain to assign the IPs to each server. Second, if the server goes down, you have to reassign the IP to another server. Until you do, your service is down. Third, if you want to have multiple servers handle the same IP, you cannot; sorry. Finally, the reassignment process can take a few minutes, even discounting the time it takes for you to notice that the original server is down and reassigning it.

That's where BGP comes in.

Equinix Metal supports using BGP for one or more servers to announce to the Equinix Metal routing infrastructure that they can handle IPs.

This is very similar to assigning the IPs using the Equinix Metal API, as we did above, except that:

• the servers announce the routes they can handle autonomously
• the route updates are very quick
• multiple servers can handle the same IP in parallel.

Many people find BGP confusing, even intimidating. So, before we keep going on assigning the Elastic IP via BGP, we will spend a few moments describing what BGP is and how it works. If you already know all of that, feel free to jump ahead to the deploying section.

BGP

BGP is an Internet Protocol that lets routers and servers announce, "I can handle traffic for the following IP ranges".

These messages are called (surprise, surprise) "announcements". The announcements are sent from a network device, such as a router or server, to a BGP peer. The BGP peer is usually a router upstream, which is expected to receive packets for the announced IP ranges and now know, "I can forward them to this device".

Most of the time, this is useful for general Internet routing. This is how, for example, your bank (or its Internet Service Provider) tells the rest of the Internet, "I can handle traffic for 100.125.150.200", allowing the rest of the Internet to send traffic to your bank's servers.

In order for it to work, you need three things:

• A router that can receive BGP announcements. This is usually a router that is connected to the Internet, and is usually provided by your ISP.
• A network device that can send BGP announcements. This is usually a router or server.
• An IP address range that you actually own and can announce.

In our diagram, 100.125.150.175 is the IP address that the servers can handle. Their router announces that address (the green "announcements") to the Internet routers (their "peers"). This in turn lets them send any traffic for 100.125.150.175 to the router, which will then forward it to the servers.

BGP Information

In order to send BGP announcements, you need to know several pieces of configuration information.

First, you need to know the IP address of the routers to which you are sending the announcements. These are called "BGP peers". In our diagram above, the BGP peers are the three routers that are connected to the Internet.

Second, you need to know your Autonomous System Number, or ASN. BGP is a Border Gateway Protocol, used between independent systems, hence, the name "Autonomous Systems". You need a unique Number, or ASN, to identify your network to the peers. Those peers, in turn, may be configured to accept only valid ASNs, or only certain ASNs from certain senders.

This brings us to our third requirement: your local address. Some peers will accept announcements from any address, although this is not common. Most peers will accept announcements only from a specific set of addresses for specific ASNs that they recognize. In the case of Equinix Metal, the peers will accept only from:

1. Your servers
2. From the private IP addresses on them
3. To specific peer IPs
4. Using specific ASNs configured when setting up BGP on the Equinix Metal project

Why is it Difficult?

Well, it isn't too difficult, if you have all of the right software and configuration installed. Routers are not exactly the easiest devices to configure, but they do already have BGP software installed. Indeed, their basic operating software is built around network protocols, like BGP.

In our case, we are not worrying too much about the upstream routers, or "peers". Fortunately, Equinix Metal takes care of those for us.

We do, however, have to configure the servers both to announce the IPs and to be prepared to receive traffic for those addresses. This is where it gets a bit tricky.

Configuring servers to announce the IPs means we need:

• The correct configurations: peer IP addresses to which to announce; local IP from which to announce; ASN numbers; and, if enabled, MD5 passwords.
• BGP software to announce the IPs, since these are servers, not routers, so they don't come with it.
• Configuring the BGP software to announce the IPs according to the configuration.

Deploying

Before we start using BGP for the Elastic IP, be sure to log back into the console and detach the Elastic IP from whichever server currently has it.

In order to use BGP, we need to:

1. Enable BGP on the Equinix Metal project, if it was not already done.
2. Enable BGP on each server in the Equinix Metal project.
3. Configure BGP on each server to announce the routes.

To enable BGP on your project, select "Networking" -> "BGP" from the left-hand menu, then enable BGP. Once it is set, you will see "Local BGP Enabled".

Next, for each server, select the server, and then select the "BGP" tab at the top.

You can see above that there is no "Status" for the server, meaning there is no BGP enabled.

Click the arrow at the right of the IPv4 row, which brings up a BGP enable sidebar.

Enable BGP for the server using the switch on the right:

The next state will be unknown until setup is complete:

Repeat for all three of your servers, and wait a few minutes for the status to move from "Unknown" to "Down".

It is "Down", because it is ready, but has not received any communications from the servers. That is our next task.

Finally, we configure BGP on our servers, telling it to announce the Elastic IP address. We will use bird, which we installed earlier along with nginx, to announce our routes. This means that the bird software on each server will "announce", or report which IPs it can handle, to the Equinix Metal routing infrastructure.

Configuring bird can be a bit trickier. We need the following for each server:

• The elastic IP address we gathered earlier. This is shared across all of the servers. In our example, that is 145.40.77.88, but it will be different for you.
• The local private IP for each server. We are going to get that from Metal metadata.
• The IPs of our upstream BGP peers. We are going to get that from Metal metadata.
• The ASN of our upstream BGP peers. We are going to get that from Metal metadata.
• The IP of our upstream router.
• The interface we will use. In all of our examples, we are going to use bond0.

On each server, do the following:

1. ssh to the server.
2. Get the information from the metadata with the following command: curl https://metadata.platformequinix.com/metadata | jq '.bgp_neighbors[0] | { customer_ip: .customer_ip, customer_as: .customer_as, multihop: .multihop, peer_ips: .peer_ips, peer_as: .peer_as }'
3. Create the bird configuration file.
4. Restart bird.

We already added the Elastic IP to the lo loopback interface earlier, so bird should just pick it up, if we configure it correctly (which we will, of course)

root@c3-small-x86-01:~# curl -s https://metadata.platformequinix.com/metadata | jq '.bgp_neighbors[0] | { customer_ip: .customer_ip, customer_as: .customer_as, multihop: .multihop, peer_ips: .peer_ips, peer_as: .peer_as }'
{
  "customer_ip": "10.70.68.3",
  "customer_as": 65000,
  "multihop": true,
  "peer_ips": [
    "169.254.255.1",
    "169.254.255.2"
  ],
  "peer_as": 65530
}

Our upstream peers are at 169.254.255.1 and 169.254.255.2, our customer IP, or the expected private IP, is 10.70.68.3, and our peer ASN is 65530.

Equinix Metal actually uses both the peer ASN and the upstream peer IPs as pretty much the same across its entire plant, but you should not rely upon it. Instead, use the metadata as we did above.

The only missing pieces is our upstream router IP, which we can get via ip route list 10.0.0.0/8:

root@c3-small-x86-01:~# ip route list 10.0.0.0/8
10.0.0.0/8 via 10.70.68.2 dev bond0

So in this case, our upstream router IP is 10.70.68.2.

Next, we need to create the bird configuration file /etc/bird/bird.conf. Chances are pretty good that a default one was installed, so we will replace it with this one. We put comments in to show where we placed our own information.

filter metal_bgp {
if net = 145.40.77.88/32 then accept;  # Elastic IP
}

router id 10.70.68.2;   # Local private IP
protocol direct {
interface "lo";
}

protocol kernel {
persist;
scan time 20;
import all;
export all;
}

protocol device {
scan time 10;
}

protocol static {
  route 169.254.255.1/32 via 10.70.68.2; # Upstream router IP
  route 169.254.255.2/32 via 10.70.68.2; # Upstream router IP
}

protocol bgp neighbor_v4_1 {
  export filter metal_bgp;
  local as 65000;   # Our ASN
  multihop;
  neighbor 169.254.255.1 as 65530; # Upstream peer IP and ASN
}

protocol bgp neighbor_v4_2 {
  export filter metal_bgp;
  local as 65000;  # Our ASN
  multihop;
  neighbor 169.254.255.2 as 65530; # Upstream peer IP and ASN
}

To make your life easier, you can use this template, with fields to be filled in marked in ALL CAPS:

filter metal_bgp {
if net = ELASTICIP/32 then accept;  # Elastic IP
}

router id PRIVATEIP;   # Local private IP
protocol direct {
interface "lo";
}

protocol kernel {
persist;
scan time 20;
import all;
export all;
}

protocol device {
scan time 10;
}

protocol static {
  route PEERIP1/32 via ROUTERIP; # Upstream router IP
  route PEERIP2/32 via ROUTERIP; # Upstream router IP
}

protocol bgp neighbor_v4_1 {
  export filter metal_bgp;
  local as 65000;   # Our ASN
  multihop;
  neighbor PEERIP1 as 65530; # Upstream peer IP and ASN
}

protocol bgp neighbor_v4_2 {
  export filter metal_bgp;
  local as 65000;  # Our ASN
  multihop;
  neighbor PEERIP2 as 65530; # Upstream peer IP and ASN
}

Now restart bird:

root@c3-small-x86-01:~# systemctl restart bird

bird should be working now and connected to the upstream BGP peers. You can check by asking it with birdc. If it is working, the connection should be "Established".

root@c3-small-x86-01:~# birdc show protocol neighbor_v4_1
BIRD 1.6.8 ready.
name     proto    table    state  since       info
neighbor_v4_1 BGP      master   up     18:52:38    Established
root@c3-small-x86-01:~# birdc show protocol neighbor_v4_2
BIRD 1.6.8 ready.
name     proto    table    state  since       info
neighbor_v4_2 BGP      master   up     18:52:39    Established

Looks great!

Repeat on the other 2 servers.

Test it Out!

Now we have:

• One Elastic IP,
• Not assigned to any one server by Equinix Metal API,
• With all three of our servers announcing it via BGP.

Let's try to reach it from the outside world.

$ curl http://145.40.77.88
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>I am server 02</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Looks great! This time, server 02 answered. Let's try it again:

$ curl http://145.40.77.88
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>I am server 01</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

And this time server 01.

Success!