0

I have a CRM on a server on a network. We have a static IP and another server outward facing. We use port-forwarding to map to the CRM, so that when you go to the IP or the FQDN, you get to the CRM:

xxx.xxx.xxx.xxx

crm.example.com

Internally, we can access the CRM by going to

crm

or

crm.example.com

Lately, I've been noticing that accessing the server from outside the network times out or gives 503, bad gateway. During that time, I can also SSH (different port, so this works) into the outward facing computer and access the server just fine.

I have a robot monitoring the site and indeed via HTTP monitoring the site is going down periodically.

I looked through the Apache server access and error logs and nothing stuck out at me so I'm a bit confused as to what could be going on. I also searched the access logs for 503 and found nothing.

When I run tracert from outside the network, it appears the packets basically make it through the wider area servers (Comcast city and county servers) and end up dropping at the CRM server's front step.

I'm tempted to replace the server because it is older and underpowered but it would be nice to know what is going on.

Any ideas what to do next?

EDIT

Thanks for the feedback from everyone. This question is definitely vague, because I don't have much evidence to show. But I can say that to my knowledge there are no proxy servers on our network. If there are, I don't know about them or their configuration.

Also, the reason I mentioned port forwarding and SSH is because when these intermittent outages are happening, I can SSH into the server associated with the static IP. We just use port forwarding to access to CRM, because it's on another server.

I also have a DNS/DHCP server running and have gone through the configuration and nothing it out of the ordinary. The rest of the network is fine and ever since this intermittent trouble began, the CRM server configuration has been the same. This is one of those "nothing changed (to my knowledge) scenarios" so it's a bit tricky to troubleshoot. Plus, I'm not an expert in networking so I really don't know all what could be going on.

I will definitely go over what has been mentioned so far and try to solve the problem.

nicorellius
  • 565
  • 2
  • 5
  • 23
  • look for 503 bad-gateway errors in the logs of the front-facing server and of the crm server. Where are the 503 logged? – blau Jun 28 '13 at 08:23
  • Do you have multiple webapps hosted on the same CRM server? Are you using virtual hosts? Are you proxying through Apache? You may have a conflict in your virtual host and proxy pass configurations. – Jason Huntley Jun 30 '13 at 15:55
  • No virtual hosts, 503s are showing up in browser from outside when trying to access during outage. No proxies... No 503s in logs either... – nicorellius Jul 06 '13 at 15:21

2 Answers2

3

To answer your question in the same general way it's presented; your problem most likely lies between your apache daemon and the area defined as "external". I assume you are using private range IP Addresses internally, since you talk of port-forwarding (The next time, don't redact those. There is no point to it, which you should realize if you remember your IP Networking 101).

Keeping the above assumption in the mind, 503's generally happen when a intermediate proxy is misconfigured or runs out of resources, which is most likely what happens in your case, as you're seeing the error intermittently. This is why you don't see this error on your webserver: It doesn't happen there, it's on the proxy.

Note that this proxy can be located anywhere between your apache daemon and the "external" area: on the same host, on any machine, on a network element, etc.

Roman
  • 3,825
  • 3
  • 20
  • 33
  • Thanks for the feedback. To clarify, the IP mentioned above is not an internal IP; it's the static IP issued to us by our ISP. That's why I redacted it, because it might as well be unknown. Of course, you can also access the CRM at 192.168.10.20... But that's not what I was trying to convey. – nicorellius Jul 06 '13 at 15:13
1

Have you checked that the external DNS -> IP mapping is consistent? Perhaps the domain is configured for round robin.

Similarly network devices could be natting external -> internal in a round-robin or other method other than static.

Depending on your network config the same thing could be occurring between your proxy and the apache server.

Clint Priest
  • 31
  • 1
  • 5
  • I have checked DNS but I suppose I can double check. And, I suppose I could have a proxy that I don't know about, but that seems unlikely. Are proxies setup by default in any case? – nicorellius Jul 06 '13 at 15:22
  • Does this server run any other services? I had a few of my servers going down intermittently and discovered people were attempting to use them in dns amplification attacks (unsuccessfully) but they were bombarding them with DNS packets anyways. Do you have any monitoring software in place such as munin? A graph of network utilization during the 503's, or apache processes, etc may give you the insight you need. http://munin-monitoring.org/ is incredibly easy to setup. – Clint Priest Jul 07 '13 at 13:01
  • No, this server is relatively single-minded. I originally set it up to only handle the CRM, hoping this would cut down on hassle in the future. No networking monitoring, although I should probably install some tools to assist in troubleshooting, moving forward. I will read up on munin... Strangely, over the last couple weeks, uptime has been near 100%. Maybe it was an attack that has subsided... Thanks for the tips. – nicorellius Jul 08 '13 at 22:44