How Not to Lose (Most of) Your Packets

Distributed applications are the digital ecosystem's workhorses, carrying everything from e-commerce giants to bustling social networks. At its most basic, a distributed app consists of one or more local or remote clients that communicate with one or more servers on several machines linked through a network but appears to end users as one cohesive system.

As the distance between servers increases and the network becomes more complex, distributed applications can become vulnerable to network packet loss, leading to delays in data transfer, data corruption and diminished performance.

In this guide, you'll learn about packet loss, including its causes and strategies for minimizing its effect on distributed applications. You'll also learn about some best practices that platform engineers and DevOps teams can use to improve application resilience and performance in distributed applications.

More on networking:

• It Isn’t Magic, It’s TCP: Demystifying Networking for Developers
• How to Build Network Services in the User Space
• Locking Down Your Cloud Network: The Basics

Understanding Packet Loss

Packet loss occurs when one or more data packets transmitted across a network cannot reach their intended destination, resulting in information loss. In cloud networks, this slows performance and compromises data integrity, leading to errors, incomplete information and degraded user experiences.

How Packet Loss Occurs

Packet loss can stem from various factors, including:

• Network congestion: A leading cause of packet loss is network congestion, which happens when data transmission volume exceeds a network's bandwidth capacity—usually during peak traffic times. Load balancers and other components like cloud storage services and databases become overwhelmed, creating bottlenecks that cause dropped data packets or delayed delivery. This results in slower application response times.
• Hardware failures: When outdated physical components on your network fail, they can drop packets. Wear and tear, outdated firmware or incompatibilities with newer protocols can cause connectivity issues and increased lag that slows down network traffic enough to cause packet loss.
• Software bugs: Bugs in routing and switching software that control data packet movement can cause misrouted packets, loops, incorrect forwarding and increased congestion and delay, all of which lead to dropped packets. Software issues that cause memory leaks or buffer overflows can also degrade network performance and cause system crashes, resulting in packet loss. In addition, malicious actors can exploit security vulnerabilities through these bugs to create overwhelming traffic that causes congestion or manipulate protocol behavior to create inconsistencies in data flow, resulting in targeted packet loss.
• Configuration errors: Incorrectly configured network devices can also slow traffic enough to cause packet loss. Overly strict security settings can cause misrouted packets or block legitimate traffic. These errors, often made during manual setup, can significantly disrupt data flow and lead to packet loss.

The Impact of Packet Loss on Distributed Applications

The effects of packet loss can ripple through distributed applications in several ways:

• Increased latency: Packet loss magnifies latency (the delay between sending and receiving data across a network). When packets are lost, the missing data must be identified and resent. This is especially problematic for real-time applications, where lag and slow response times can mar the user experience.
• Performance degradation: The TCP protocol requires the retransmission of lost packets, and its congestion control mechanisms may also interpret packet loss as network congestion, triggering a reduction in data transmission rates that further impairs application performance.
• Potential data corruption: Lost packets can result in incomplete or incorrectly reassembled data at the destination, causing the end user to receive corrupted files or distorted messages. This disruption can compromise application logic, severely affecting applications like financial systems that depend on precise data.

Measuring Packet Loss

For platform engineers and DevOps teams, monitoring packet loss metrics (including rate, round-trip time, retransmission rate and error rate) is fundamental to maintaining the health and performance of distributed applications. Network monitoring tools like SolarWinds Network Performance Monitor, PRTG Network Monitor and Nagios Network Analyzer provide insight into network health by monitoring packet flow in real time and alerting users of issues as they arise.

Acceptable packet loss rates can vary depending on the use case and network topology. Real-time applications such as VoIP or live video streaming may have a lower tolerance for packet loss compared to non-real-time applications such as file transfers. Generally, a packet loss rate between 0.1 and 1 percent is acceptable for most applications. However, a rate exceeding 1 percent can lead to noticeable performance degradation.

Factors such as the time of day, the types of applications in use and the current network load can all influence what constitutes an acceptable packet loss rate. A thoughtful analysis of loss metrics therefore involves monitoring raw packet loss rates as well as understanding the context within which these rates occur. By monitoring the network's performance over time and carefully interpreting these metrics, your team can identify concerning packet loss rates and make informed decisions about when to intervene and apply corrective measures.

Strategies for Reducing Packet Loss

Improving network performance and the reliability of data transmission requires effective strategies for reducing packet loss across digital communications. The following sections outline a few strategies that can help reduce packet loss.

Optimize Your Network Infrastructure

Optimizing your network infrastructure is a key component of reducing packet loss and boosting the performance of your distributed applications. With complete control of your network, you can deploy several key strategies to ensure efficient delivery of data packets:

• Implement quality of service (QoS): QoS manages and prioritizes network traffic so that high-priority applications like VoIP and video conferencing receive the necessary bandwidth during congestion. QoS classifies traffic based on factors such as service type, source and destination IP addresses and content type, then applies policies to prioritize critical applications over less urgent ones such as email and file downloads.
• Use traffic shaping and prioritization: Traffic shaping regulates the amount and rate of data transmission on a network to ensure that bandwidth remains within its predefined limits. The aim is to prevent sudden bursts of data from causing bottlenecks, network congestion and eventual packet loss. Prioritization ensures high-priority data packets are processed first, preserving bandwidth for essential services, especially during peak times.
• Leverage direct cloud-to-cloud private networking: This approach connects cloud services via private links, bypassing the congested public internet. Direct connections reduce congestion-related packet loss and provide higher speeds with lower latency. This improves connectivity for applications that need fast, reliable communication between cloud platforms and minimizes data delay and data loss.

Improve Infrastructure Resilience

A solid network infrastructure with modern hardware and built-in redundancy can help keep packet loss to a minimum.

Modern routers, switches and network adapters come equipped with advanced traffic management capabilities and support for the latest protocols and features. These advanced capabilities and features improve system reliability, capacity and fault tolerance and directly contribute to reduced packet loss. 

In addition, relying on a single internet service provider (ISP) or cloud service provider (CSP) introduces a single point of failure and increases the risk of packet loss due to a lack of alternative transmission paths during disruptions. Integrating multiple ISPs and CSPs for a highly available, geographically diverse network creates redundancy and failover options that enhance resilience and ensure uninterrupted data flow and application performance.

Keep in mind that using multiple ISPs and CSPs can increase network complexity, which also contributes to packet loss. Managing multiple connections from different providers can complicate network configuration, monitoring and maintenance, leading to configuration errors that could cause packet loss. You should evaluate the advantages and drawbacks of this strategy based on your specific use cases and business requirements.

Best Practices for Handling Packet Loss in Distributed Application Design

To safeguard your distributed applications from the adverse effects of cloud network packet loss, consider the following best practices to bolster application resilience and reliability.

Architect for Redundancy and Failover

Anticipate system failures by integrating redundancy and fault tolerance into your design. Implement backup systems or alternate pathways for when primary paths fail.

Use load balancers to disperse traffic across multiple servers and diversify application infrastructure across multiple cloud providers and availability zones to help maintain application availability and mitigate provider-specific failures. 

Implement Application-Level Acknowledgments

While application-level acknowledgments introduce some latency due to acknowledgment wait times and data retransmissions, they also help safeguard data integrity and delivery during packet loss. If the system does not receive an acknowledgment for a sent packet from the application within a predefined window, the system assumes loss and initiates retransmission

This allows engineers to actively monitor and respond to packet delivery status to ensure that important data reaches its intended destination without being compromised by network issues. Despite the introduction of some latency, the trade-off for improved data reliability and integrity is often considered worthwhile, especially in applications where the accuracy of data delivery is a priority.

Through application acknowledgment techniques such as message queueing, custom acknowledgment messages, retry logic and sequence numbering, engineers can ensure data integrity and reliability even amid network disturbances.

Design Applications to Tolerate Minor Packet Loss

Not all applications demand perfect packet delivery. You can design distributed applications to tolerate minor packet loss, particularly in streaming or real-time data scenarios where slight packet loss is preferable to retransmission delays that can prevent operational disruptions.

Engineers can use adaptive algorithms to adjust data transmission rates to current network conditions or implement error correction codes for data reconstruction so applications can continue to function with reduced quality or speed.

Building Resilient Distributed Systems with Equinix

From the technical underpinnings of packet loss to practical steps for mitigation, we explored strategies and best practices for understanding, measuring and reducing cloud network packet loss in distributed applications.

Engineering teams can design and deploy highly available and performant distributed applications on Equinix Metal. They can use the Metal dedicated cloud platform to provision high-performance compute infrastructure in more than 30 global locations and have full control of how application traffic is managed on their network.