This document mainly introduces the main reasons that may cause CVM network access packet loss, as well as the corresponding troubleshooting and solutions.
Possible Reasons
The possible reasons for the CVM network access packet loss problem are as follows:
Triggering the speed limit causes TCP packet loss
Triggering the speed limit causes UDP packet loss
Triggering soft interrupt causes packet loss
UDP send buffer is full
UDP receive buffer is full
TCP full connection queue is full
TCP request overflow
The number of connections has reached the upper limit
iptables policy sets relevant rules
Prerequisites
You need to log in to the instance before locating and troubleshooting the problem. See Logging in to the Linux instance and Logging in to the Windows instance for details.
Troubleshooting
Triggering the speed limit causes TCP packet loss
CVM instances are available in various specifications, and different specifications have different network performance. When the bandwidth or packet volume of an instance exceeds the standard corresponding to the instance specification, the speed limit on the platform side will be triggered, resulting in packet loss. The troubleshooting and processing steps are as follows:
- Check the bandwidth and package size of the instance.
For Linux instances, you can perform the sar -n DEV 2 command to check the bandwidth and packet volume. Among them, the metrics rxpck/s and txpck/s are the number of the sending and receiving packets, and the indicators rxkB/s and txkB/s are the sending and receiving bandwidth. - Use the obtained bandwidth and package data to compare instance specifications to check whether the instance specification performance bottleneck has been reached.
- Yes, you need to upgrade the instance specifications or adjust the business volume.
- No, if the performance bottleneck of the instance specification has not been reached, further location processing can be performed.
Triggering the speed limit causes UDP packet loss
See the step Triggering the speed limit causes TCP packet loss to determine whether the packet loss is caused by the performance bottleneck of the instance specification.
- Yes, you need to upgrade the instance specifications or adjust the business volume.
- If the performance bottleneck of the instance specification is not reached, it may be caused by the platform's additional frequency limit on DNS requests. When the overall bandwidth or packet volume of the instance reaches the performance bottleneck of the instance specifications, the DNS request speed limit may be triggered and UDP packet loss may occur. It can be further located and processed.
Triggering soft interrupt causes packet loss
When the operating system detects that the count value in the second column of /proc/net/softnet_stat is increasing, it will be judged as soft interrupt packet loss. When your instance triggers soft interrupt packet loss, you can troubleshoot and handle it by following the steps below: Check whether RPS is enabled:
- If so, a small value of the kernel parameter net.core.netdev_max_backlog will cause packet loss and needs to be increased.
- If not, check whether the CPU single-core soft interrupt is high, resulting in failure to send and receive data in time. If so, you can:
- Enable RPS to make the soft interrupt distribution more balanced.
- Check whether the business program causes uneven distribution of soft interrupts.
UDP send buffer is full
If your instance has packet loss due to insufficient UDP buffer, you can troubleshoot by following these steps:
- Use the ss -nump command to check whether the UDP send buffer is full.
- If so, increase the kernel parameters net.core.wmem_max and net.core.wmem_default, and restart the UDP program for the changes to take effect.
- If the packet loss problem still exists, you can use the ss -nump command to check whether the send buffer has not increased as expected. At this time, you need to check whether the business code has set SO_SNDBUF through setsockopt. If so, modify the code to increase SO_SNDBUF.
UDP receive buffer is full
If your instance has packet loss due to insufficient UDP buffer, you can handle it by following these steps:
- Use the ss -nump command to check whether the UDP receive buffer is full.
- If so, increase the kernel parameters net.core.rmem_max and net.core.rmem_default, and restart the UDP program for the changes to take effect.
- If the packet loss problem still exists, you can use the ss -nump command to check whether the receive buffer has not increased as expected. At this time, you need to check whether the business code has set SO_RCVBUF through setsockopt. If so, modify the code to increase SO_RCVBUF.
TCP full connection queue is full
The length of the TCP full connection queue is the smaller value of net.core.somaxconn and the backlog parameter passed in when the business process calls listen. If your instance has packet loss due to the full TCP connection queue, you can handle it by following these steps:
Increase the kernel parameter net.core.somaxconn.
Check whether the backlog parameter is passed into the business process. If so, increase it accordingly.
TCP request overflow
When TCP receives data, if the socket is locked by the user, the data will be sent to the backlog queue. If this process fails, it will cause TCP request overflow and packet loss. Under normal conditions, assuming that the business program performance is normal, you can see the following methods to troubleshoot and solve the problem from the system level:
Check whether the business program has set the buffer size by itself through setsockopt:
- If it has been set and the value is not large enough, you can modify the business program to specify a larger value, or stop specifying the size through setsockopt.
Description
The value of setsockopt is limited by the kernel parameters net.core.rmem_max and net.core.wmem_max. When adjusting the business program, you can also adjust net.core.rmem_max and net.core.wmem_max simultaneously. After adjustment, restart the business program to make the configuration take effect. - If not set, you can increase the net.ipv4.tcp_mem, net.ipv4.tcp_rmem, and net.ipv4.tcp_wmem kernel parameters to adjust the TCP socket water level.
The number of connections has reached the upper limit
CVM instances are available in various specifications, and different specifications have different connection performance metrics. When the number of connections of an instance exceeds the standard corresponding to the instance specification, the platform's speed limit will be triggered, resulting in packet loss. The processing steps are as follows:
Description
The number of connections refers to the number of sessions of the CVM instance saved on the host, including TCP, UDP, and ICMP. This value is greater than the number of network connections obtained through the ss or netstat command on the CVM instance.
Check the number of connections on your instance and compare it to Strength Specifications, check whether the instance specification performance bottleneck has been reached.
- Yes, you need to upgrade the instance specifications or adjust the business volume.
- No, if the performance bottleneck of the instance specification has not been reached, further location processing can be performed.
iptables policy sets relevant rules
In the absence of relevant iptables rules set on the CVM, the iptables policy-related rule settings may be causing packets destined for the CVM to be discarded.
The processing steps are as follows:
- Perform the following command to check the iptables policy rules.
iptables -L | grep policy
The default iptables policy rule is ACCEPT. If the INPUT chain policy is not ACCEPT, all packets to the server will be lost. For example, if the following result is returned, it means that all packets entering the CVM will be dropped.
Chain INPUT (policy DROP)
Chain FORWARD (policy ACCEPT)
Chain OUTPUT (policy ACCEPT)
- Perform the following command and adjust the value after -P as needed.
iptables -P INPUT ACCEPT
After adjustment, you can perform the command in step 1 of iptables policy sets related ruless again to check the results as follows:
Chain INPUT (policy ACCEPT)
Chain FORWARD (policy ACCEPT)
Chain OUTPUT (policy ACCEPT)