1

We have an apache instance sitting inside our DMZ which is configured to proxy requests to an internal NATed tomcat instance inside our network. It works fine, but then all of a sudden requests from apache to the tomcat instance stop getting through with the following in the apache logs:

[error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header

Investigating into the Cisco log viewer reveals the following:

Error Message %ASA-6-106015: Deny TCP (no connection) from IP_address/port to IP_address/port flags tcp_flags on interface interface_name. Explanation The adaptive security appliance discarded a TCP packet that has no associated connection in the adaptive security appliance connection table. The adaptive security appliance looks for a SYN flag in the packet, which indicates a request to establish a new connection. If the SYN flag is not set, and there is not an existing connection, the adaptive security appliance discards the packet.

Recommended Action None required unless the adaptive security appliance receives a large volume of these invalid TCP packets. If this is the case, trace the packets to the source and determine the reason these packets were sent.

All are machines are virtualised using VMware, and by default machines have been using the Intel E1000 emulated NIC. Our network administrator has changed this to a VMXNET3 driver in an attempt to correct the problem, we just have to wait and see if the problem persists as it's an intermittent problem.

Is there something else that could be causing this problem? This isn't the first service where we have had similar issues.

Our apache host is running Ubuntu 11.10 with a kernel version of 3.0.0-17-server. We have also had this issue on RHEL5 (5.8) running kernel 2.6.18-308.16.1.el5, this machine also has the E1000 NIC.

NOTE: I am not a network administrator and am a software architect and analyst programmer responsible for these systems.

Brett Ryan
  • 327
  • 1
  • 4
  • 15
  • How long is that connection open for before it has this problem? – Shane Madden Nov 14 '12 at 05:57
  • It can continue to work fine for around 30 minutes before it starts to fail randomly. The request being performed in our test scenario is a light weight request that returns static content, the request takes < 1s when working fine. When it fails, the requests will continue to fail for an undetermined period of time before they start to succeed again. Both machines in questions have no load. – Brett Ryan Nov 14 '12 at 06:03
  • Can you give some idea of the amount of time that it continues failing for? 10 seconds or a week? Also, what's the `timeout conn` configuration in the ASA look like? – Shane Madden Nov 14 '12 at 06:06
  • It's actually random, though it never seems to continue to fail for more than 30 minutes, sometimes just performing the same request again will work, but then another request after that may fail once again. My network administrator has just gone home so I will need to wait 'till tomorrow to get the `timeout conn` setting. – Brett Ryan Nov 14 '12 at 06:10

1 Answers1

1

Problem has been found to be the ASA closing persistent connections after a period of time, when it closes the connections it's also been configured to not send RST messages when a call is once again made.

To understand why this causes a problem I can illustrate it here.

  1. Apache creates first connection which succeeds.
  2. After a delay longer than ASA reset time the ASA closes the connection.
  3. Request is made and Apache tries sending on what it thinks is an open connection and times out after TimeOut - default 300 seconds
  4. Apache sends error to client

The problem here is amplified if there are a number of pooled connections still open. For example if Apache started with 5 pooled connections, and after the above closed one, it will still exhibit this behavior another 4 times before a client gets a successful request.

There are several ways to overcome this.

  1. Allow the ASA to send RST messages to clients it trusts.
  2. Set the configuration for mod_proxy:ProxyPass - keepalive to On
  3. Set the configuration for mod_proxy:ProxyPass - ttl to something lower than the firewalls reset time.

Don't try configuring mod_proxy:ProxyPass - timeout and mod_proxy:ProxyPass - connectiontimeout too low, as if you have any long running operations existing in your tomcat instance say for example any web services or ReST endpoints then they may start to fail if they take longer than this time.

Our solution is to do both of the first two options.

Brett Ryan
  • 327
  • 1
  • 4
  • 15