- Home /
- Developers /
- Technical guides /
- Network Path...
Network Path Troubleshooting with Linux
What’s in this guide?
Networking is Hard. Sure, plugging a bunch of boxes into other boxes is easy enough, but actually getting the right bits to flow from the right computers across the wilds of the internet and end up at the right destination, pointed to the correct port that’s open and listening for traffic gets really complicated really fast. And when you add on the fact that sometimes the Equinix Magic that can provide you with blazing interconnect speeds, can also cause some unfamiliar network behavior it can be really nice to have a friend or at least a guide to help you figure out where your packets are going.
This guide will provide you with a solid foundation of network troubleshooting on Linux. While this is generically useful for anyone sending bits between servers, the focus will be on troubleshooting connections between you (or your data center) and Equinix. Even if this doesn’t solve your problems, you’ll be able to provide all the information needed to our customer success team to get you the rest of the way there.
You will need
- Equinix Metal Account w/ access to machines in question
- A local Linux system (some tools may require root/sudo access)
- Optionally, the Metal CLI tool
IPv4 & IPv6
Since we’re talking about networks, we’re going to be dealing with IP addresses. You’re likely familiar with these numbers from daily use, but having a little deeper understanding will help with figuring out where things are going wrong. For the most part, everything we’ll be dealing with uses the long standing IPv4 format.
IPv4
In IPv4, any network connected device is given a unique identifier usually referred to as IP address. IP stands for internet protocol and points to a standard protocol used in networking connectivity. IPs consist of 32-bits usually portrayed as 4 octets in dot-decimal notation (xxx.xxx.xxx.xxx). Any given IP address can be divided into two halves, the first set of bits being the network portion of the address and the balk half representing the host portion. This is represented in either CIDR (Classless Inter-Domain Routing) notation or by use of a subnet mask, the former just listing how many bits are used by the latter. For example, let’s say you have an IP from Equinix of 55.82.200.45/29. You can work backwards to determine the proper subnet mask and the other IP addresses available in that block. With 29 bits (/29) dedicated to the network, that means we have 3 host bits giving us a subnet mask of 255.255.255.248 (11111111.11111111.11111111.11111000 in binary). And with a bit of binary conversion it’ll tell us that our 8 available IPs are going to be 55.82.200.40-48. Out of these, the very first (all 0s) host is usually reserved for the “network” address and the very last (all 1s) host being reserved for the network “broadcast” address. But that’s probably a bit deeper than this guide needs to go.
The main takeaway from breaking down our 55.82.200.45/29 address above is the device with that IP will treat any of 55.82.200.40 - .48 addresses as local to it, only forwarding requests outside of said network to its default gateway or router. What is a gateway, though? People tend to think of the internet as “one large network” – but that is slightly inadequate. Actually, the internet rather resembles a huge amount of small networks joined together. Every network segment with connectivity to other networks on the internet is assigned a so-called autonomous system number, also referred to as AS number or ASN. ASNs are used by routers to announce to all other routers on the internet what IP address ranges they are responsible for. When a computer on network A needs to talk to a computer on network B (i.e. any computer that is not part of its own local network), it will forward its request to the locally configured router, usually also referred to as gateway or default gateway for that matter. The gateway will forward the packets to the router having announced its public responsibility for the network to which the target host’s IP address belongs to. The path a packet needs to take to make it from its source to its target is usually referred to as route. Your system will usually only know the route to its default gateway, from where routing will happen mostly dynamically based on the worldwide routing tables. This allows any arbitrary host in any subnet of the internet to communicate with any other arbitrary host in general (specialities such as firewalls are left out here for the sake of simplicity.
So if you’re having trouble sending data from 55.82.200.45 to 55.82.200.46, you may need to double check they’re properly connected to the same network, or double check those subnet masks to make sure each computer has a proper picture of its network connections. If all of these conditions apply, checking whether a problem with the gateway connectivity exists is the next logical step.
IPv6
The binary inclined and curious may have done the math at this point to realize that with a 32bit number representing all devices on the internet, we’d only have room for about 4.3 billion internet connected devices, less when you consider the number of reserved IPs for local networks. And while even 3 billion feels like a big number, it’s certainly not when you consider the world population and the sheer number of internet connected devices we all have and interact with daily. Further, with the way IP address blocks can be subdivided and moved around, routing tables can get crowded quickly. In fact, most regional authorities have been down to their final block of /8 (~16 million) IP addresses since 2015.
IPv6 has been introduced to alleviate this with a 128-bit address providing 2128 or 3.4 x 1032 available IP addresses or “probably good enough for the foreseeable future”. And instead of a mutable subnet mask, IPv6 uses a set 64-bits for the host and network portions of the address. Envisioned with the original end-to-end nature of the internet in mind, IPv6 is routed a bit differently than IPv4 with different and slightly simpler demands from the router devices themselves and more work done on the end nodes. There are further differences between the two protocol versions, but the major difference for the purposes of this guide is notation.
While IPv4 is written as 4 decimal octets (groups of 8 bits or bytes) separated by dots or periods, IPv6 is written as 8 hexadecimal hextets (16-bit groups or chomps) separated by colons. For convenience and clarity, IPv6 also includes some rules about abbreviating: all leading 0s can be dropped in each hextet and consecutive hextets of all 0s can be replaced with two colons (::). The latter condition can only be applied once in an address as multiple uses would render the address vague and indeterminate.
For example, you might get the IPv6 address of 2001:0db8:0000:0000:0000:ff00:0042:8329. After removing the leading 0s we’d be left with 2001:db8:0:0:0:ff00:42:8329 and after compressing consecutive hextets of 0, we are left with 2001:db8::ff00:42:8329. Recognizing, expanding and abbreviating IPv6 addresses as well as remembering the 64-bits for network and 64-bits for host are going to be the biggest implications for troubleshooting IPv6, for this guide.
Finding your machine’s IP address
Using the Equinix Metal Web UI
Via Console: Open Organization > Project > Servers > Server Hostname (IPV4/6 address) in the Equinix Metal web interface.
Using the Equinix Metal CLI
Via Metal CLI:
To get all assigned IP addresses, use: metal devices get -o json | jq '.[0].ip_addresses'
To get IP addresses assigned to a specific device, use: metal devices get -i $device_id -o json | jq '.ip_addresses'
Verifying Web UI / CLI information with your systems
Linux-based Operating Systems
Most of the following tools will be available on standard installations, if not, all are available via your distribution’s package management (apt, snap, rpm)
Finding your IP from the machine command line:
ip aip addip addrip -br -c abrief + colors
For instance, to generate a concise overview of all interfaces and assigned addresses, use:
# ip -br -c a
lo UNKNOWN 127.0.0.1/8 ::1/128
enp1s0f0 UP
enp1s0f1 UP
bond0 UP 139.178.69.191/31 10.88.19.129/31 2604:1380:1000:c00::1/127 fe80::ec4:7aff:feb2:98dc/64
Determine which ports are listening:
ss -lss -lp (show process)lsof -i
For instance, to see a list of all listening ports on your current machine, use:
# ss -lp
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
udp UNCONN 0 0 127.0.0.53%lo:domain 0.0.0.0:* users:(("systemd-resolve",pid=888,fd=12))
tcp LISTEN 0 4096 127.0.0.53%lo:domain 0.0.0.0:* users:(("systemd-resolve",pid=888,fd=13))
tcp LISTEN 0 128 0.0.0.0:ssh 0.0.0.0:* users:(("sshd",pid=1476,fd=3))
tcp LISTEN 0 4096 *:5201 *:* users:(("iperf3",pid=14162,fd=3))
tcp LISTEN 0 128 [::]:ssh [::]:* users:(("sshd",pid=1476,fd=4))
For a list of all currently opened file descriptors, use:
# lsof -i
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-r 888 systemd-resolve 12u IPv4 20923 0t0 UDP localhost:domain
systemd-r 888 systemd-resolve 13u IPv4 20924 0t0 TCP localhost:domain (LISTEN)
sshd 1476 root 3u IPv4 27081 0t0 TCP *:ssh (LISTEN)
sshd 1476 root 4u IPv6 27083 0t0 TCP *:ssh (LISTEN)
sshd 1969 root 4u IPv4 27199 0t0 TCP metal-guide-00:ssh->123-123-100-10.sbcglobal.net:51434 (ESTABLISHED)
Is a remote machine reachable?
ping [address]
If it is reachable, is time (latency) acceptable?
The following example shows reachability of the remote host packet.com with a latency of roughly one millisecond:
# ping packet.com
PING packet.com (104.18.26.66) 56(84) bytes of data.
64 bytes from 104.18.26.66 (104.18.26.66): icmp_seq=1 ttl=58 time=1.02 ms
64 bytes from 104.18.26.66 (104.18.26.66): icmp_seq=2 ttl=58 time=1.04 ms
64 bytes from 104.18.26.66 (104.18.26.66): icmp_seq=3 ttl=58 time=0.881 ms
64 bytes from 104.18.26.66 (104.18.26.66): icmp_seq=4 ttl=58 time=1.03 ms
Check if a specific remote port is reachable w/ netcat
nc -vz [address] [port]
To figure out if equinix.com port 443 is reachable from your machine, use:
# nc -vz equinix.com 443
Connection to equinix.com 443 port [tcp/https] succeeded!
Check if a specific remote port is reachable w/ nmap
nmap -p [port] [address]
The same task performed using the nmap tool:
# nmap -p 443 equinix.com
Starting Nmap 7.80 ( https://nmap.org ) at 2022-03-01 03:51 UTC
Nmap scan report for equinix.com (40.91.78.229)
Host is up (0.023s latency).
PORT STATE SERVICE
443/tcp open https
Nmap done: 1 IP address (1 host up) scanned in 0.53 seconds
What route is being used to a specified address w/ stats
mtr [address]mtr -r -c 10 [address] (-cycles)
How many hops? Is one of them slower than acceptable?
Flags:
h for help
n to toggle DNS / ip address
q to exit
Here is the example output for one host, using an IPv6 IP, trying to reach another host.
metal-guide-00 (2604:1380:1000:c00::1) 2022-03-01T03:56:07+0000
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. _gateway 0.0% 28 7.8 3.0 1.0 8.6 2.5
2. 0.ae147.dsr1.r00c02.sjc1.packet.net 0.0% 28 9.4 23.0 1.5 401.7 75.8
3. 0.ae11.bsr1.sjc1.packet.net 0.0% 28 17.5 2.4 0.7 21.2 4.8
4. 0.et-0-0-11.bsr1.sv5.packet.net 0.0% 28 1.5 4.4 1.3 40.5 9.2
5. 2001:4860:1:1::1d7f 0.0% 27 1.4 1.4 1.2 1.8 0.1
6. 2607:f8b0:82c2::1 48.1% 27 2.2 2.4 2.2 2.7 0.2
7. any-in-2001-4860-4802-36--15.1e100.net 0.0% 27 1.5 1.4 1.2 1.5 0.1
Test connection speed w/speedtest:
Install speedtest.
apt install speedtest-clispeedtest
To perform a speedtest on the CLI, use
# speedtest
Retrieving speedtest.net configuration...
Testing from Packet Host (139.178.69.191)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Ayera Technologies, Inc. (Modesto, CA) [97.79 km]: 7.473 ms
Testing download speed................................................................................
Download: 1142.73 Mbit/s
Testing upload speed......................................................................................................
Upload: 1656.41 Mbit/s
How fast is transfer between 2 machines?
iperf can be used to test transfer speeds between your machines.apt install iperf3 (on both local & remote)
On the remote server, to start listening use:iperf3 -s
On your local system use:iperf3 -c [target address]iperf3 -c [target address] -R (in reverse, remote -> local)
The output could look like this:
# iperf3 -c 139.178.69.191
Connecting to host 139.178.69.191, port 5201
[ 5] local 192.168.0.233 port 52106 connected to 139.178.69.191 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 35.2 MBytes 295 Mbits/sec
[ 5] 1.00-2.00 sec 53.8 MBytes 451 Mbits/sec
[ 5] 2.00-3.00 sec 27.0 MBytes 227 Mbits/sec
[ 5] 3.00-4.00 sec 51.8 MBytes 435 Mbits/sec
[ 5] 4.00-5.00 sec 43.6 MBytes 366 Mbits/sec
[ 5] 5.00-6.00 sec 38.7 MBytes 324 Mbits/sec
[ 5] 6.00-7.00 sec 41.5 MBytes 348 Mbits/sec
[ 5] 7.00-8.00 sec 43.7 MBytes 367 Mbits/sec
[ 5] 8.00-9.00 sec 44.5 MBytes 373 Mbits/sec
[ 5] 9.00-10.00 sec 46.1 MBytes 386 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 426 MBytes 357 Mbits/sec sender
[ 5] 0.00-10.06 sec 424 MBytes 354 Mbits/sec receiver
iperf Done.
Live view of network load by device
nload
arrows cycle through devices
q to quit
Again, here is some example output of said command:
Device bond0 [139.178.69.191] (1/5):
===========================================================================================
Incoming:
#####################
#####################
.#####################
######################
######################
######################
######################
######################
###################### Curr: 2.91 kBit/s
###################### Avg: 125.98 MBit/s
###################### Min: 2.90 kBit/s
###################### Max: 317.84 MBit/s
###################### Ttl: 1.20 GByte
Outgoing:
Curr: 30.23 kBit/s
. .. Avg: 1.01 MBit/s
################## Min: 7.57 kBit/s
###################| Max: 2.42 MBit/s
.#################### Ttl: 159.86 MByte
Who does a domain belong to?
whois [address]
The following example shows the whois information for the equinix.com domain:
# whois equinix.com
Domain Name: EQUINIX.COM
Registry Domain ID: 2603897_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.corporatedomains.com
Registrar URL: http://cscdbs.com
Updated Date: 2021-08-22T05:11:35Z
Creation Date: 1998-08-27T04:00:00Z
Registry Expiry Date: 2022-08-26T04:00:00Z
Registrar: CSC Corporate Domains, Inc.
Registrar IANA ID: 299
Registrar Abuse Contact Email: domainabuse@cscglobal.com
Registrar Abuse Contact Phone: 8887802723
Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
Domain Status: serverDeleteProhibited https://icann.org/epp#serverDeleteProhibited
Domain Status: serverTransferProhibited https://icann.org/epp#serverTransferProhibited
Domain Status: serverUpdateProhibited https://icann.org/epp#serverUpdateProhibited
Name Server: DNS1.P07.NSONE.NET
Name Server: DNS2.P07.NSONE.NET
Name Server: DNS3.P07.NSONE.NET
Name Server: DNS4.P07.NSONE.NET
DNSSEC: unsigned
URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf/
>>> Last update of whois database: 2022-03-01T04:03:24Z <<<
What address does a domain name resolve to?
nslookup [address]dig +trace [address]
An example using nslookup:
# nslookup equinix.com
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
Name: equinix.com
Address: 40.91.78.229
The same example using dig:
# dig +trace equinix.com
; <<>> DiG 9.16.1-Ubuntu <<>> +trace equinix.com
;; global options: +cmd
. 79825 IN NS e.root-servers.net.
. 79825 IN NS d.root-servers.net.
. 79825 IN NS g.root-servers.net.
. 79825 IN NS a.root-servers.net.
. 79825 IN NS k.root-servers.net.
. 79825 IN NS b.root-servers.net.
. 79825 IN NS i.root-servers.net.
. 79825 IN NS m.root-servers.net.
. 79825 IN NS l.root-servers.net.
. 79825 IN NS h.root-servers.net.
. 79825 IN NS j.root-servers.net.
. 79825 IN NS c.root-servers.net.
. 79825 IN NS f.root-servers.net.
;; Received 262 bytes from 127.0.0.53#53(127.0.0.53) in 0 ms
com. 172800 IN NS c.gtld-servers.net.
com. 172800 IN NS m.gtld-servers.net.
com. 172800 IN NS k.gtld-servers.net.
com. 172800 IN NS b.gtld-servers.net.
com. 172800 IN NS g.gtld-servers.net.
com. 172800 IN NS d.gtld-servers.net.
com. 172800 IN NS e.gtld-servers.net.
com. 172800 IN NS f.gtld-servers.net.
com. 172800 IN NS h.gtld-servers.net.
com. 172800 IN NS i.gtld-servers.net.
com. 172800 IN NS j.gtld-servers.net.
com. 172800 IN NS a.gtld-servers.net.
com. 172800 IN NS l.gtld-servers.net.
com. 86400 IN DS 30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF C41A5766
com. 86400 IN RRSIG DS 8 1 86400 20220313210000 20220228200000 9799 . l/Qi4jz9Mm/17YNIjJM8Z9vBxY2LNM4EwJykZsFLMXg8AnbJDxaAGB3h 47gGMj8mzTpTpgRrmD6ohAh/HOpbzR74YlQze6rVp+fn5oBZGzxKlVvW jVhm0L2vVCBwoilPJJ13sWdVYxBYJC4E1H23vT4khtzEUtjHdNnSbTbl GrFk3DDb6oIR5AhPQZII+zbYWPgLxJS7MJ20ch9ZXpby44IQ/0L+cgly Il/y84Ix/jq4qpfMexbxjTBWa9cK3S+vBIpWnxv4372lEFTiN6SPFYfF YCQVafKCnNZxWq3DIdwDNJcgcZS4ezCkm08Y4PZ/phfHBWHoFaqNMMpw BSY0cg==
;; Received 1199 bytes from 192.112.36.4#53(g.root-servers.net) in 44 ms
equinix.com. 172800 IN NS dns1.p07.nsone.net.
equinix.com. 172800 IN NS dns2.p07.nsone.net.
equinix.com. 172800 IN NS dns3.p07.nsone.net.
equinix.com. 172800 IN NS dns4.p07.nsone.net.
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0Q1GIN43N1ARRC9OSM6QPQR81H5M9A NS SOA RRSIG DNSKEY NSEC3PARAM
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400 20220304052328 20220225041328 38535 com. QBEVp5RQYWO4WbeNlRYtAg+ob5TeMdT9lidq3zTL5xoEWS8N9xGXMhjx JEnRCkf8+yirdIyLZvIqzKqKtvzr2WXVLpEDdoUsfpj3xdG1VDPXNbva 9bSOgIcOnfv0IUXJJXHhsp80NDWeFYbwR+wJMeEwgrHWQfyccyUejjpU JKK+uBrBP3D7wvHpnNIh1XnFk4f49bhdJCj5/ITzZyjR0Q==
2S6MCLRVGJJO7EJTAFM73G5O7INU7C84.com. 86400 IN NSEC3 1 1 0 - 2S6NAQI4JCBDB8UF5NFPLETQ9GTHE3P6 NS DS RRSIG
2S6MCLRVGJJO7EJTAFM73G5O7INU7C84.com. 86400 IN RRSIG NSEC3 8 2 86400 20220307055325 20220228044325 38535 com. RlR6BsuGKh937/apIR6FtvvQZ47Gv3CXUx0XotbbCGD3EtwsJfxY5bxa Yv3b2HnU+DqL1SChUD2MBAKr0SzNeTSa726Ymy6P75QDM7iGA9ObRRPP 6FEtkfcNxDqXx52pBQpiIQfVhRklW+VoRGD4V54vyC6jSq/PaOSoWCFE tD71VC4mvCbaox5KKUHPn2F8WH5rLFPqoRmHcFuAV01POw==
;; Received 678 bytes from 192.41.162.30#53(l.gtld-servers.net) in 24 ms
equinix.com. 300 IN A 40.91.78.229
;; Received 56 bytes from 2a00:edc0:6259:7:7::4#53(dns4.p07.nsone.net) in 0 ms
What about arp, ifconfig, route, etc?
These were part of net-tools, unmaintained for years, deprecated and succeeded by ip, netstat -> ss.
To determine what hardware is in the machine
lshwlshw -class network (only network )lshw -class network -short
Here is example output from lshw, which will often look crowded and chaotic:
# lshw -class network
*-network:0
description: Ethernet interface
product: I350 Gigabit Network Connection
vendor: Intel Corporation
physical id: 0
bus info: pci@0000:01:00.0
logical name: enp1s0f0
version: 01
serial: 0c:c4:7a:b2:98:dc
size: 1Gbit/s
capacity: 1Gbit/s
width: 32 bits
clock: 33MHz
capabilities: pm msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=igb driverversion=5.6.0-k duplex=full firmware=1.52, 0x800007ae latency=0 link=yes multicast=yes port=twisted pair slave=yes speed=1Gbit/s
resources: irq:16 memory:88400000-8847ffff ioport:e020(size=32) memory:88580000-88583fff memory:88500000-8853ffff memory:88584000-885a3fff memory:885a4000-885c3fff
*-network:1
description: Ethernet interface
product: I350 Gigabit Network Connection
vendor: Intel Corporation
physical id: 0.1
bus info: pci@0000:01:00.1
logical name: enp1s0f1
version: 01
serial: 0c:c4:7a:b2:98:dc
size: 1Gbit/s
capacity: 1Gbit/s
width: 32 bits
clock: 33MHz
capabilities: pm msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=igb driverversion=5.6.0-k duplex=full firmware=1.52, 0x800007ae latency=0 link=yes multicast=yes port=twisted pair slave=yes speed=1Gbit/s
resources: irq:17 memory:88480000-884fffff ioport:e000(size=32) memory:885c4000-885c7fff memory:88540000-8857ffff memory:885c8000-885e7fff memory:885e8000-88607fff
*-network
description: Ethernet interface
physical id: 3
logical name: bond0
serial: 0c:c4:7a:b2:98:dc
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=bonding driverversion=3.7.1 duplex=full firmware=2 ip=139.178.69.191 link=yes master=yes multicast=yes
Get further network card info / network card driver info
ethtool [DEVICE NAME]ethtool -i [DEVICE NAME]
Figuring out firewall rules
Many Linux distributions pre-deploy firewall rules or services such as firewalld for basic system security. Those may get in your way when setting up networks. On Linux systems, packet filters are implemented using iptables or nftables, the latter being newer but having a compatibility interface to iptables. Here is how to get a list of all applied rules:
iptables -S
Remember that both iptables and nftables follow the “first rule match” principle. That means that an incoming packet is dealt with according to the first matching rule.
Last updated
05 April 2022Category
Ready to kick the tires?
Sign up and get going today, or request a demo to get a tour from an expert.