Issue
Server does not answer some specific SYN Packages.
Webserver does not respond for some sessions (SYN requests). SYN Packets being ignored intermittently by RHEL Apache server
LDAP server occasionally ignoring SYNs
Any server, regardless of the application layer, may occasionally ignore TCP SYN packets (i.e. a TCP packet with the SYN flag set).
Environment
Red Hat Enterprise Linux 4 and 5 (Webserver/application server)
Red Hat Enterprise Linux 5.4 (openldap-2.3.43-3.el5)
This issue has been reported in a environment using NAT (Network Address Translation) like the following:
Client--->NAT Router--->Server
Resolution
Disable TCP time stamps using the following command:
echo 0 > /proc/sys/net/ipv4/tcp_timestamps
or by adding the following line to /etc/systcl.conf and rebooting the system:
net.ipv4.tcp_timestamps = 0
Disabling TCP time stamps will disable PAWS (Protect Against Wrapped Sequences), checking as defined in RFC 1323.
Root Cause
In a TCP dump of the affected packets, the TSVal of SYN packet is a lower value than the preceding packet.
Packet #30 from x.x.x.x to y.y.y.y has a TSval of 773695118
Packet #33 from x.x.x.x to y.y.y.y has a TSval of 470318331
In above TCP trace, packet #33 has a TSval of 470318331 which is lower than its preceding one. As PAWS checking is enabled in the TCP stack, it doesn't send an acknowledge (SYN-ACK) back to the peer.
PAWS assumes that every received TCP segment (including data and ACK segments) contains a timestamp (SEG.TSval) whose values are non-decreasing in time. The basic idea is that a segment can be discarded as an old duplicate if it is received with a timestamp (SEG.TSval) less than some timestamp recently received on this connection.
Timestamp issues often indicate a problem with firewalls or NAT routers.
Diagnostic Steps
Stage 1
Since the server is not heavily loaded, you may at first suspect this issue to be an issue with socket exhaustion. However, checking the output of "netstat -paunt" will reveal that there are only a couple of active connections.
Stage 2
Increasing the size of syn backlog queue (net.ipv4.tcp_max_syn_backlog) doesn't seem to help.
Stage 3
Collecting a TCP dump from the server using a command like the following:
tcpdump -s0 -w /tmp/tcpdump.pcap -i any host <client ip> and port 80
and generating traffic (HTTP or LDAP) to the server captures evidence of the server not responding to the TCP SYN packets. The output file can be analyzed with a command like
tcpdump -r /<path>/<to>/tcpdump.pcap
or Wireshark can be used to analyze the TCP dump file graphically. Wireshark is a standard package in most Red Hat Enterprise Linux and Fedora distributions. Analysis of the TCP dump should show cases where the client will send several TCP SYN packets to the host, but the host will not respond. Further analysis should show that the TSVal of the SYN packets is less than a preceding TCP packet.
