Network Path Troubleshooting with Linux

What’s in this guide?

Networking is Hard. Sure, plugging a bunch of boxes into other boxes is easy enough, but actually getting the right bits to flow from the right computers across the wilds of the internet and end up at the right destination, pointed to the correct port that’s open and listening for traffic gets really complicated really fast. And when you add on the fact that sometimes the Equinix Magic that can provide you with blazing interconnect speeds, can also cause some unfamiliar network behavior it can be really nice to have a friend or at least a guide to help you figure out where your packets are going.

This guide will provide you with a solid foundation of network troubleshooting on Linux. While this is generically useful for anyone sending bits between servers, the focus will be on troubleshooting connections between you (or your data center) and Equinix. Even if this doesn’t solve your problems, you’ll be able to provide all the information needed to our customer success team to get you the rest of the way there.

You will need

• Equinix Metal Account w/ access to machines in question
• A local Linux system (some tools may require root/sudo access)
• Optionally, the Metal CLI tool

IPv4 & IPv6

Since we’re talking about networks, we’re going to be dealing with IP addresses. You’re likely familiar with these numbers from daily use, but having a little deeper understanding will help with figuring out where things are going wrong. For the most part, everything we’ll be dealing with uses the long standing IPv4 format.

IPv4

In IPv4, any network connected device is given a unique identifier usually referred to as IP address. IP stands for internet protocol and points to a standard protocol used in networking connectivity. IPs consist of 32-bits usually portrayed as 4 octets in dot-decimal notation (xxx.xxx.xxx.xxx). Any given IP address can be divided into two halves, the first set of bits being the network portion of the address and the balk half representing the host portion. This is represented in either CIDR (Classless Inter-Domain Routing) notation or by use of a subnet mask, the former just listing how many bits are used by the latter. For example, let’s say you have an IP from Equinix of 55.82.200.45/29. You can work backwards to determine the proper subnet mask and the other IP addresses available in that block. With 29 bits (/29) dedicated to the network, that means we have 3 host bits giving us a subnet mask of 255.255.255.248 (11111111.11111111.11111111.11111000 in binary). And with a bit of binary conversion it’ll tell us that our 8 available IPs are going to be 55.82.200.40-48. Out of these, the very first (all 0s) host is usually reserved for the “network” address and the very last (all 1s) host being reserved for the network “broadcast” address. But that’s probably a bit deeper than this guide needs to go.

The main takeaway from breaking down our 55.82.200.45/29 address above is the device with that IP will treat any of 55.82.200.40 - .48 addresses as local to it, only forwarding requests outside of said network to its default gateway or router. What is a gateway, though? People tend to think of the internet as “one large network” – but that is slightly inadequate. Actually, the internet rather resembles a huge amount of small networks joined together. Every network segment with connectivity to other networks on the internet is assigned a so-called autonomous system number, also referred to as AS number or ASN. ASNs are used by routers to announce to all other routers on the internet what IP address ranges they are responsible for. When a computer on network A needs to talk to a computer on network B (i.e. any computer that is not part of its own local network), it will forward its request to the locally configured router, usually also referred to as gateway or default gateway for that matter. The gateway will forward the packets to the router having announced its public responsibility for the network to which the target host’s IP address belongs to. The path a packet needs to take to make it from its source to its target is usually referred to as route. Your system will usually only know the route to its default gateway, from where routing will happen mostly dynamically based on the worldwide routing tables. This allows any arbitrary host in any subnet of the internet to communicate with any other arbitrary host in general (specialities such as firewalls are left out here for the sake of simplicity.

So if you’re having trouble sending data from 55.82.200.45 to 55.82.200.46, you may need to double check they’re properly connected to the same network, or double check those subnet masks to make sure each computer has a proper picture of its network connections. If all of these conditions apply, checking whether a problem with the gateway connectivity exists is the next logical step.

IPv6

The binary inclined and curious may have done the math at this point to realize that with a 32bit number representing all devices on the internet, we’d only have room for about 4.3 billion internet connected devices, less when you consider the number of reserved IPs for local networks. And while even 3 billion feels like a big number, it’s certainly not when you consider the world population and the sheer number of internet connected devices we all have and interact with daily. Further, with the way IP address blocks can be subdivided and moved around, routing tables can get crowded quickly. In fact, most regional authorities have been down to their final block of /8 (~16 million) IP addresses since 2015.

IPv6 has been introduced to alleviate this with a 128-bit address providing 2128 or 3.4 x 1032 available IP addresses or “probably good enough for the foreseeable future”. And instead of a mutable subnet mask, IPv6 uses a set 64-bits for the host and network portions of the address. Envisioned with the original end-to-end nature of the internet in mind, IPv6 is routed a bit differently than IPv4 with different and slightly simpler demands from the router devices themselves and more work done on the end nodes. There are further differences between the two protocol versions, but the major difference for the purposes of this guide is notation.

While IPv4 is written as 4 decimal octets (groups of 8 bits or bytes) separated by dots or periods, IPv6 is written as 8 hexadecimal hextets (16-bit groups or chomps) separated by colons. For convenience and clarity, IPv6 also includes some rules about abbreviating: all leading 0s can be dropped in each hextet and consecutive hextets of all 0s can be replaced with two colons (::). The latter condition can only be applied once in an address as multiple uses would render the address vague and indeterminate.

For example, you might get the IPv6 address of 2001:0db8:0000:0000:0000:ff00:0042:8329. After removing the leading 0s we’d be left with 2001:db8:0:0:0:ff00:42:8329 and after compressing consecutive hextets of 0, we are left with 2001:db8::ff00:42:8329. Recognizing, expanding and abbreviating IPv6 addresses as well as remembering the 64-bits for network and 64-bits for host are going to be the biggest implications for troubleshooting IPv6, for this guide.

Finding your machine’s IP address

Using the Equinix Metal Web UI

Via Console: Open Organization > Project > Servers > Server Hostname (IPV4/6 address) in the Equinix Metal web interface.

Using the Equinix Metal CLI

Via Metal CLI:

To get all assigned IP addresses, use: metal devices get -o json | jq '.[0].ip_addresses'

To get IP addresses assigned to a specific device, use: metal devices get -i $device_id -o json | jq '.ip_addresses'

Verifying Web UI / CLI information with your systems

Linux-based Operating Systems

Most of the following tools will be available on standard installations, if not, all are available via your distribution’s package management (apt, snap, rpm)

Finding your IP from the machine command line:

ip a
ip add
ip addr
ip -br -c a brief + colors

For instance, to generate a concise overview of all interfaces and assigned addresses, use:

# ip -br -c a
lo               UNKNOWN        127.0.0.1/8 ::1/128
enp1s0f0         UP
enp1s0f1         UP
bond0            UP             139.178.69.191/31 10.88.19.129/31 2604:1380:1000:c00::1/127 fe80::ec4:7aff:feb2:98dc/64

Determine which ports are listening:

ss -l
ss -lp (show process)
lsof -i

For instance, to see a list of all listening ports on your current machine, use:

# ss -lp
Netid      State       Recv-Q      Send-Q     Local Address:Port         Peer Address:Port       Process
udp        UNCONN      0           0          127.0.0.53%lo:domain       0.0.0.0:*               users:(("systemd-resolve",pid=888,fd=12))
tcp        LISTEN      0           4096       127.0.0.53%lo:domain       0.0.0.0:*               users:(("systemd-resolve",pid=888,fd=13))
tcp        LISTEN      0           128        0.0.0.0:ssh                0.0.0.0:*               users:(("sshd",pid=1476,fd=3))
tcp        LISTEN      0           4096       *:5201                     *:*                     users:(("iperf3",pid=14162,fd=3))
tcp        LISTEN      0           128        [::]:ssh                   [::]:*                  users:(("sshd",pid=1476,fd=4))

For a list of all currently opened file descriptors, use:

# lsof -i
COMMAND    PID            USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
systemd-r  888 systemd-resolve   12u  IPv4  20923      0t0  UDP localhost:domain
systemd-r  888 systemd-resolve   13u  IPv4  20924      0t0  TCP localhost:domain (LISTEN)
sshd      1476            root    3u  IPv4  27081      0t0  TCP *:ssh (LISTEN)
sshd      1476            root    4u  IPv6  27083      0t0  TCP *:ssh (LISTEN)
sshd      1969            root    4u  IPv4  27199      0t0  TCP metal-guide-00:ssh->123-123-100-10.sbcglobal.net:51434 (ESTABLISHED)

Is a remote machine reachable?

ping [address]

If it is reachable, is time (latency) acceptable?

The following example shows reachability of the remote host packet.com with a latency of roughly one millisecond:

# ping packet.com
PING packet.com (104.18.26.66) 56(84) bytes of data.
64 bytes from 104.18.26.66 (104.18.26.66): icmp_seq=1 ttl=58 time=1.02 ms
64 bytes from 104.18.26.66 (104.18.26.66): icmp_seq=2 ttl=58 time=1.04 ms
64 bytes from 104.18.26.66 (104.18.26.66): icmp_seq=3 ttl=58 time=0.881 ms
64 bytes from 104.18.26.66 (104.18.26.66): icmp_seq=4 ttl=58 time=1.03 ms

Check if a specific remote port is reachable w/ netcat

nc -vz [address] [port]

To figure out if equinix.com port 443 is reachable from your machine, use:

# nc -vz equinix.com 443
Connection to equinix.com 443 port [tcp/https] succeeded!

Check if a specific remote port is reachable w/ nmap

nmap -p [port] [address]

The same task performed using the nmap tool:

# nmap -p 443 equinix.com
Starting Nmap 7.80 ( https://nmap.org ) at 2022-03-01 03:51 UTC
Nmap scan report for equinix.com (40.91.78.229)
Host is up (0.023s latency).

PORT    STATE SERVICE
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 0.53 seconds

What route is being used to a specified address w/ stats

mtr [address]
mtr -r -c 10 [address] (-cycles)
How many hops? Is one of them slower than acceptable?

Flags:
h for help
n to toggle DNS / ip address
q to exit

Here is the example output for one host, using an IPv6 IP, trying to reach another host.

metal-guide-00 (2604:1380:1000:c00::1)                            2022-03-01T03:56:07+0000
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                  Packets               Pings
 Host                                           Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. _gateway                                     0.0%    28    7.8   3.0   1.0   8.6   2.5
 2. 0.ae147.dsr1.r00c02.sjc1.packet.net          0.0%    28    9.4  23.0   1.5 401.7  75.8
 3. 0.ae11.bsr1.sjc1.packet.net                  0.0%    28   17.5   2.4   0.7  21.2   4.8
 4. 0.et-0-0-11.bsr1.sv5.packet.net              0.0%    28    1.5   4.4   1.3  40.5   9.2
 5. 2001:4860:1:1::1d7f                          0.0%    27    1.4   1.4   1.2   1.8   0.1
 6. 2607:f8b0:82c2::1                           48.1%    27    2.2   2.4   2.2   2.7   0.2
 7. any-in-2001-4860-4802-36--15.1e100.net       0.0%    27    1.5   1.4   1.2   1.5   0.1

Test connection speed w/speedtest:

Install speedtest.

apt install speedtest-cli
speedtest

To perform a speedtest on the CLI, use

# speedtest
Retrieving speedtest.net configuration...
Testing from Packet Host (139.178.69.191)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Ayera Technologies, Inc. (Modesto, CA) [97.79 km]: 7.473 ms
Testing download speed................................................................................
Download: 1142.73 Mbit/s
Testing upload speed......................................................................................................
Upload: 1656.41 Mbit/s

How fast is transfer between 2 machines?

iperf can be used to test transfer speeds between your machines.
apt install iperf3 (on both local & remote)

On the remote server, to start listening use:
iperf3 -s

On your local system use:
iperf3 -c [target address] iperf3 -c [target address] -R (in reverse, remote -> local)

The output could look like this:

# iperf3 -c 139.178.69.191
Connecting to host 139.178.69.191, port 5201
[  5] local 192.168.0.233 port 52106 connected to 139.178.69.191 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  35.2 MBytes   295 Mbits/sec
[  5]   1.00-2.00   sec  53.8 MBytes   451 Mbits/sec
[  5]   2.00-3.00   sec  27.0 MBytes   227 Mbits/sec
[  5]   3.00-4.00   sec  51.8 MBytes   435 Mbits/sec
[  5]   4.00-5.00   sec  43.6 MBytes   366 Mbits/sec
[  5]   5.00-6.00   sec  38.7 MBytes   324 Mbits/sec
[  5]   6.00-7.00   sec  41.5 MBytes   348 Mbits/sec
[  5]   7.00-8.00   sec  43.7 MBytes   367 Mbits/sec
[  5]   8.00-9.00   sec  44.5 MBytes   373 Mbits/sec
[  5]   9.00-10.00  sec  46.1 MBytes   386 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   426 MBytes   357 Mbits/sec                  sender
[  5]   0.00-10.06  sec   424 MBytes   354 Mbits/sec                  receiver

iperf Done.

Live view of network load by device

nload
arrows cycle through devices
q to quit

Again, here is some example output of said command:

Device bond0 [139.178.69.191] (1/5):
===========================================================================================
Incoming:
                                #####################
                                #####################
                               .#####################
                               ######################
                               ######################
                               ######################
                               ######################
                               ######################
                               ######################         Curr: 2.91 kBit/s
                               ######################         Avg: 125.98 MBit/s
                               ######################         Min: 2.90 kBit/s
                               ######################         Max: 317.84 MBit/s
                               ######################         Ttl: 1.20 GByte
Outgoing:









                                                              Curr: 30.23 kBit/s
                                 .               ..           Avg: 1.01 MBit/s
                                 ##################           Min: 7.57 kBit/s
                                ###################|          Max: 2.42 MBit/s
                               .####################          Ttl: 159.86 MByte

Who does a domain belong to?

whois [address]

The following example shows the whois information for the equinix.com domain:

# whois equinix.com
   Domain Name: EQUINIX.COM
   Registry Domain ID: 2603897_DOMAIN_COM-VRSN
   Registrar WHOIS Server: whois.corporatedomains.com
   Registrar URL: http://cscdbs.com
   Updated Date: 2021-08-22T05:11:35Z
   Creation Date: 1998-08-27T04:00:00Z
   Registry Expiry Date: 2022-08-26T04:00:00Z
   Registrar: CSC Corporate Domains, Inc.
   Registrar IANA ID: 299
   Registrar Abuse Contact Email: domainabuse@cscglobal.com
   Registrar Abuse Contact Phone: 8887802723
   Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
   Domain Status: serverDeleteProhibited https://icann.org/epp#serverDeleteProhibited
   Domain Status: serverTransferProhibited https://icann.org/epp#serverTransferProhibited
   Domain Status: serverUpdateProhibited https://icann.org/epp#serverUpdateProhibited
   Name Server: DNS1.P07.NSONE.NET
   Name Server: DNS2.P07.NSONE.NET
   Name Server: DNS3.P07.NSONE.NET
   Name Server: DNS4.P07.NSONE.NET
   DNSSEC: unsigned
   URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf/
>>> Last update of whois database: 2022-03-01T04:03:24Z <<<

What address does a domain name resolve to?

nslookup [address]
dig +trace [address]

An example using nslookup:

# nslookup equinix.com
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
Name:	equinix.com
Address: 40.91.78.229

The same example using dig:

# dig +trace equinix.com

; <<>> DiG 9.16.1-Ubuntu <<>> +trace equinix.com
;; global options: +cmd
.			79825	IN	NS	e.root-servers.net.
.			79825	IN	NS	d.root-servers.net.
.			79825	IN	NS	g.root-servers.net.
.			79825	IN	NS	a.root-servers.net.
.			79825	IN	NS	k.root-servers.net.
.			79825	IN	NS	b.root-servers.net.
.			79825	IN	NS	i.root-servers.net.
.			79825	IN	NS	m.root-servers.net.
.			79825	IN	NS	l.root-servers.net.
.			79825	IN	NS	h.root-servers.net.
.			79825	IN	NS	j.root-servers.net.
.			79825	IN	NS	c.root-servers.net.
.			79825	IN	NS	f.root-servers.net.
;; Received 262 bytes from 127.0.0.53#53(127.0.0.53) in 0 ms

com.			172800	IN	NS	c.gtld-servers.net.
com.			172800	IN	NS	m.gtld-servers.net.
com.			172800	IN	NS	k.gtld-servers.net.
com.			172800	IN	NS	b.gtld-servers.net.
com.			172800	IN	NS	g.gtld-servers.net.
com.			172800	IN	NS	d.gtld-servers.net.
com.			172800	IN	NS	e.gtld-servers.net.
com.			172800	IN	NS	f.gtld-servers.net.
com.			172800	IN	NS	h.gtld-servers.net.
com.			172800	IN	NS	i.gtld-servers.net.
com.			172800	IN	NS	j.gtld-servers.net.
com.			172800	IN	NS	a.gtld-servers.net.
com.			172800	IN	NS	l.gtld-servers.net.
com.			86400	IN	DS	30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF C41A5766
com.			86400	IN	RRSIG	DS 8 1 86400 20220313210000 20220228200000 9799 . l/Qi4jz9Mm/17YNIjJM8Z9vBxY2LNM4EwJykZsFLMXg8AnbJDxaAGB3h 47gGMj8mzTpTpgRrmD6ohAh/HOpbzR74YlQze6rVp+fn5oBZGzxKlVvW jVhm0L2vVCBwoilPJJ13sWdVYxBYJC4E1H23vT4khtzEUtjHdNnSbTbl GrFk3DDb6oIR5AhPQZII+zbYWPgLxJS7MJ20ch9ZXpby44IQ/0L+cgly Il/y84Ix/jq4qpfMexbxjTBWa9cK3S+vBIpWnxv4372lEFTiN6SPFYfF YCQVafKCnNZxWq3DIdwDNJcgcZS4ezCkm08Y4PZ/phfHBWHoFaqNMMpw BSY0cg==
;; Received 1199 bytes from 192.112.36.4#53(g.root-servers.net) in 44 ms

equinix.com.		172800	IN	NS	dns1.p07.nsone.net.
equinix.com.		172800	IN	NS	dns2.p07.nsone.net.
equinix.com.		172800	IN	NS	dns3.p07.nsone.net.
equinix.com.		172800	IN	NS	dns4.p07.nsone.net.
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0Q1GIN43N1ARRC9OSM6QPQR81H5M9A NS SOA RRSIG DNSKEY NSEC3PARAM
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400 20220304052328 20220225041328 38535 com. QBEVp5RQYWO4WbeNlRYtAg+ob5TeMdT9lidq3zTL5xoEWS8N9xGXMhjx JEnRCkf8+yirdIyLZvIqzKqKtvzr2WXVLpEDdoUsfpj3xdG1VDPXNbva 9bSOgIcOnfv0IUXJJXHhsp80NDWeFYbwR+wJMeEwgrHWQfyccyUejjpU JKK+uBrBP3D7wvHpnNIh1XnFk4f49bhdJCj5/ITzZyjR0Q==
2S6MCLRVGJJO7EJTAFM73G5O7INU7C84.com. 86400 IN NSEC3 1 1 0 - 2S6NAQI4JCBDB8UF5NFPLETQ9GTHE3P6 NS DS RRSIG
2S6MCLRVGJJO7EJTAFM73G5O7INU7C84.com. 86400 IN RRSIG NSEC3 8 2 86400 20220307055325 20220228044325 38535 com. RlR6BsuGKh937/apIR6FtvvQZ47Gv3CXUx0XotbbCGD3EtwsJfxY5bxa Yv3b2HnU+DqL1SChUD2MBAKr0SzNeTSa726Ymy6P75QDM7iGA9ObRRPP 6FEtkfcNxDqXx52pBQpiIQfVhRklW+VoRGD4V54vyC6jSq/PaOSoWCFE tD71VC4mvCbaox5KKUHPn2F8WH5rLFPqoRmHcFuAV01POw==
;; Received 678 bytes from 192.41.162.30#53(l.gtld-servers.net) in 24 ms

equinix.com.		300	IN	A	40.91.78.229
;; Received 56 bytes from 2a00:edc0:6259:7:7::4#53(dns4.p07.nsone.net) in 0 ms

What about arp, ifconfig, route, etc?

These were part of net-tools, unmaintained for years, deprecated and succeeded by ip, netstat -> ss.

To determine what hardware is in the machine

lshw
lshw -class network (only network )
lshw -class network -short

Here is example output from lshw, which will often look crowded and chaotic:

# lshw -class network
  *-network:0
       description: Ethernet interface
       product: I350 Gigabit Network Connection
       vendor: Intel Corporation
       physical id: 0
       bus info: pci@0000:01:00.0
       logical name: enp1s0f0
       version: 01
       serial: 0c:c4:7a:b2:98:dc
       size: 1Gbit/s
       capacity: 1Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pm msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=igb driverversion=5.6.0-k duplex=full firmware=1.52, 0x800007ae latency=0 link=yes multicast=yes port=twisted pair slave=yes speed=1Gbit/s
       resources: irq:16 memory:88400000-8847ffff ioport:e020(size=32) memory:88580000-88583fff memory:88500000-8853ffff memory:88584000-885a3fff memory:885a4000-885c3fff
  *-network:1
       description: Ethernet interface
       product: I350 Gigabit Network Connection
       vendor: Intel Corporation
       physical id: 0.1
       bus info: pci@0000:01:00.1
       logical name: enp1s0f1
       version: 01
       serial: 0c:c4:7a:b2:98:dc
       size: 1Gbit/s
       capacity: 1Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pm msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=igb driverversion=5.6.0-k duplex=full firmware=1.52, 0x800007ae latency=0 link=yes multicast=yes port=twisted pair slave=yes speed=1Gbit/s
       resources: irq:17 memory:88480000-884fffff ioport:e000(size=32) memory:885c4000-885c7fff memory:88540000-8857ffff memory:885c8000-885e7fff memory:885e8000-88607fff
  *-network
       description: Ethernet interface
       physical id: 3
       logical name: bond0
       serial: 0c:c4:7a:b2:98:dc
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=bonding driverversion=3.7.1 duplex=full firmware=2 ip=139.178.69.191 link=yes master=yes multicast=yes

Get further network card info / network card driver info

ethtool [DEVICE NAME]
ethtool -i [DEVICE NAME]

Figuring out firewall rules

Many Linux distributions pre-deploy firewall rules or services such as firewalld for basic system security. Those may get in your way when setting up networks. On Linux systems, packet filters are implemented using iptables or nftables, the latter being newer but having a compatibility interface to iptables. Here is how to get a list of all applied rules:

iptables -S

Remember that both iptables and nftables follow the “first rule match” principle. That means that an incoming packet is dealt with according to the first matching rule.