31

We have a web application (developed by a third party) that runs on Tomcat. We have been getting very bad performance from the application. The application developer is claiming that it is an Industry Best Practice to restart web servers every night, to free up all memory usage and start over.

From the customer perspective that alleviates their issue of the site crashing during the day, but from a SysAdmin perspective it is an awful solution.

We host 20 of these applications in different servers for different clients, and the coordination of making sure that all are being restarted every night just seems wrong.

Geo
  • 3,061
  • 11
  • 41
  • 52
  • 42
    Tell them that it's Industry Best Practice for application developers to find and fix their memory leaks. – Bart Silverstrim Feb 04 '10 at 00:31
  • 4
    @Bart Oh snap!! – mfinni Feb 04 '10 at 00:53
  • 1
    +1 just for making my day (PS: I am a developer myself) – RN. Apr 21 '10 at 05:57
  • 1
    Did he say servers or services? We hve a tomcat applicationm that needs the service restarting every night. If I don't do it, at some point in the future it will crash. I would rather not do it, but the service during the day is more important. – Tubs May 11 '10 at 12:58
  • 1
    Get some log file monitoring going and download some JVM monitoring tools. If stuff is crashing during the day you should see exceptions or something being logged--even if they're default exceptions. That'll give you some insight into the general nature of the error. Also, watch the JVM memory useage. Odds are really good they have a memory leak and you'll catch it if you watch the server JVM heap. Fight bad development with good sysadmin data. It destroys the "You just don't know what you're doing" defense and makes them actually have to answer for why things messed up. – FloppyDisk Oct 31 '12 at 14:28
  • 1
    Yes, it would be best practice because Java uses garbage collection to free memory. And a lot of times it gets itself in a twist and can't collect the memory. No serious server side web app (or client app) should be developed in Java anymore. Java is full of security flaws and is just plain slow. But if you must, restart it to get your memory and performance back. – hookenz Jul 23 '13 at 04:20
  • It is an industrial standard "feature" to leak memory and bring the server down to its knees by bad coding in Java... Lool... – Volkan Jul 23 '13 at 11:39

8 Answers8

32

This is certainly not a best practice. While it is good to restart your servers periodically just to make sure that everything comes up correctly, needing to restart nightly points to a very serious memory leak in the application.

EEAA
  • 108,414
  • 18
  • 172
  • 242
  • 1
    This is a very good point. If you never restart your servers as suggested below you might not know that you have certain services that dont start properly. Then, in the event of a power failure/hard restart your server may not come back right. – einstiien Feb 04 '10 at 06:54
  • 1
    +1. Monthly may make more sense - not only for a restart, but for a normal operation procedure to apply patches etc. I was once part of an admin team for about 1500 servers, 24/7, and every month there was a 3 night "rolling restart" scheduled, at which point all patches etc. would be put into the servers, too. This gicves some planning stability and a standard operating procedure. – TomTom Apr 21 '11 at 19:54
12

There's a difference between "Best Practice", things that many people do for good reasons, and "Common Practice", things that many people do because they're lazy and/or ignorant.

Applications and (worse) servers that need to be routinely restarted or rebooted to keep running well are fairly common. But it's also a clear indication that you have a critical bug.

By making it SOP to restart an application on a regular basis, your company is hiding a serious bug under the carpet. This is inexcusable, the bug needs to be faced down and squashed, or it will come back to bite you later.

Ideally, your company should find a better developer. Unfortunately, this may lead to rather a lot of work to rewrite large tracts of your code. The fact that the developer either thinks that poorly written code is acceptable, or doesn't know enough to recognize the symptoms of buggy code, suggests the quality of the code is low. A good developer will be constitutionally incapable of leaving it in that state.

Given that you may not be in a position to replace the developer, a few suggestions:

  • See if you can have a better developer review the code and report their assessment to someone who can do something about it,
  • Have a look into profiling tools. If you've got the skills and/or inclination, try profiling the code yourself to find the leak and report it.

Even without getting into developer-oriented profiling tools, there are plenty of sysadmin-oriented tools for profiling and monitoring memory usage on Java applications. You should really set up monitoring of memory (particularly heap) on your production servers in any case. I'd recommend this even if you were running quality code. It may give you advance warning when your buggy apps are about to topple over.

But better yet, these should help you to gather proof that there is a leak, and may even indicate where the issue is in the application. This will give you better ammunition to lobby for it to be fixed.

Kief
  • 301
  • 3
  • 3
  • 3
    Actually, a lot of times it is the infrastructure that has the bug, and not the developer's code. We've had no end of trouble with J2EE apps that go into garbage-collection hell periodically on JBoss but work fine on other commercail application servers. So it might not be the developer's fault, but rather the deployment environment. – rmalayter Mar 10 '11 at 22:40
6

The application developer is more likely claiming that it's in his own best interest for you to cover his ass by working around the unprofessional job he did. He may have stopped short of actually admitting that he wrote something with a whopping memory leak, but not very far short of it.

Maximus Minimus
  • 8,937
  • 1
  • 22
  • 36
3

Many answers here seem to fall way off the mark of practical solutions. They seem to eschew dogma - servers should never be restarted - why do we have 5 nines? fault tolerance? Well thats so when they are supposed to be up they stay up.

Also, to state its the cause of bad developers or bad development practice doesn't go to the root of the issue. It can be but more often than not its not bad application code. These issues are already built into much of the system code. Small memory leaks, Java heap and permgen issues if you are running lots of small apps as we do. Modern servers and the software they run are very complex. When you think of what a server like tomcat has to do - serve files, process web requests, network communications, database communications, etc., its doing a lot. IN that stack there a a heck of a lot of moving parts.

Proactively rebooting servers lets say once a week or month is smart and efficient in my view. If you are clustered and you rotate the servers you should not affect clients one bit. Clients will be much happier with the performance of your servers.

Claude
  • 39
  • 1
2

IMO Servers should be shut down as little as possible. It's more likely the App Developer built a shoddy application with a memory leak.

matpie
  • 453
  • 1
  • 5
  • 9
  • Absolutely - I think the OP needs to tell someone they need to find a better developer. – Helvick Feb 03 '10 at 23:38
  • 2
    There's a reason big companies pay big bucks for multiple nines uptime and why companies spend thousands on redundant power supplies, RAID, hot swap cages, etc., and it certainly isn't so that they only need to reboot once a day. – Bart Silverstrim Feb 04 '10 at 00:30
1

I have a script restart one of our webservers every night but that's more because of a poorly written java application rather than an industry standard. I would say that it isn't uncommon to restart the web services though. This might do the memory cleanup you're looking for and put less strain on the server as compared to a full restart.

einstiien
  • 2,538
  • 18
  • 18
0

The most frequent I've ever seen is weekly. Where I'm at now we're a windows shop, and we do it monthly over the weekend following Patch Tuesday.

Joel Coel
  • 12,910
  • 13
  • 61
  • 99
  • When I started working at one place, I found that they had nightly reboots in place... That was horrible, especially since the server had around a 1-2% chance of not coming back up right (timing bug in the hard drive driver). Took some time to fix the "causes" for the reboots. Time WELL spent. – Brian Knoblauch Apr 21 '11 at 19:53
0

While I agree it's not ideal to restart a server constantly, there are situations where it is neither the developer's fault nor the wrong thing to do. We have a well-behaved application that is leaking memory because of issues in the Python Popen library. It's an old application that will be retired soon, but it's business-critical. We have to keep it running with a minimum of fuss for our clients. So we've just decided to restart the server every night.

Nick Mellor
  • 1
  • 1