Server crashed - corporate web application is down - considering moving to an online host - any suggestions?

Question

I'm in a mess here. For a couple of years we've been running our own custom made application on a local server in our office. The servers had its well bad days and good days now and then - however the server literally died a couple of days ago - during routine maintenance we hired a freelancer to install a printer - the moron instead ended up destroying the server [I'm still speculating he tried to mess it up a bit so he could charge for fixing a self inflicted error only this time it went beyond his own ability].

Our system is down and we're seriously looking forward to hosting our web application on an online host. I'm quite open to suggestions here. Let me explain what is the nature of the system we are running here.

Our system is Php MySQL Based - its a critical data time centric collaboration system. Basically we have a corporate email address upon which we recieve client requests. Our application is used to collaborate those requests. We have a cron job running periodically in the background or actually Scheduled task running which actually physically downloads the emails from the email account and converts it into a format that can be input into a database for querying and then deletes the email from the actual account so its in our database and thus our system can collaborate using it instead of having to make an IMAP connection everytime we need an email. This cron job is set to run every 3 minutes considering the vast volume of requests we recieve - so its very time centric our application. Now and then the scheduled task does hang - so we sometimes have to manually restart the scheduled task.

The scheduled task also downloads attachments from the mail server and stores them as flat files on our web application. On average email requests have no attachments but in routine the average attachment varies between 200K to 10MB although attachments above 5MB are a rarity. Most of the downloading is between the mailserver to the web application.

I need a host that is reliable, has next to zero down time because we're not going to be running a website we'll be running a very critical application - I mean our business depends upon it and my boss is literally inches away from breaking down and loosing his sanity at this moment :.

Someone suggested rackspace's cloud servers - I'm looking at those but basically since I'm a programmer - my exposure to teh networking and hosting side of things doesn't extend beyond troubleshooting and debugging a web application. So I need some suggestion for the situation we are in - would an online host be a good idea or do we need to invest in better network hardware[for now we've scratched one freelancer off] - considering the option of an online host what options should we look out for which offer value for money. Cheap is good but not if theres going to be a huge compromise on quality...help please :( its urgent

EDIT ====

The advice is great though but right now I was considering this situation. Lets assume I go for an online service like cloud hosting or a dedicated server - would I still require a failover system to be set up in this case and if so - how would that failover system be set up or what would be the best way to handle it. We're a small company and the other IT guy has had some well not very good experience with Managed Services from another host - its like you need someone to constantly coordinate with the host on the manage service and if we have to do half the work then we could pretty much invest some time to manage our own server be it local or cloud.

In the situation defined above - if we consider an online dedicated host.. how would we set up a failover system?

Answer 1

I hate to say this, but if you really believe that your freelancer tried to screw you over, then your first call should be to the police, and probably a specialist in computer law.

Secondly, If you've already got network connectivity, and hosting space in a datacenter, I suggest that you keep it, and maintain your own network. I personally don't like the shared-hosting business model. I like to keep my own server hardware, network, and management in-house.

That said, you do need to hire someone you can trust to manage it. It's better to have a full-time in-house sysadmin, rather than hiring a freelancer every few times something comes up. Primarily, because having a go-to guy in the office is a massive time-saver, and secondly, because hopefully, he'll build up some loyalty to the company.

If the application you guys run is business-critical, and as important as you make it out to be, then why on earth was it only on a single server? You've just given yourself the biggest reason that it failed, wasn't because of the actions of one miscreant, but because your systems architect never said "hey, what about redundancy and resilience, let's have a pair of clustered servers".

While I'm on that note, was your server state and configuration backed up anywhere? If you had a bare-metal backup from before the day you got your contractor in, then restoration should give you a working machine. (unless he's physically broken the hardware, in which case, back to the first point, this time with a charge of criminal damage)

You don't need to worry too much about the networking side if you choose a good Datacentre. They're actually there to help, not to rip you off, and can provide you with diverse routes out to the internet, even if you don't have your own routing hardware.

My recommendation:

1. Hire a competant sysadmin.
2. Arrange 2U of colocation space in a good, carrier-neutral datacentre.
3. Buy two 1U dell (or HP or any other good brand) servers, set them up as a clustered failover pair.
4. Install your webapp on both, set up some monitoring, you might want to set up Puppet to aid deployments across multiple servers.
5. Set up a backup regime, and do a test restoration

Hopefully, you'll learn a vast amount about how to set up real server architectures, and make your application far more robust that it was before. Sadly, everyone has to make these mistakes at some point, the thing that makes you a good sysadmin is showing how you handle it.

Answer 2

I feel the need to chime in here about cloud based solutions and outsourcing your IT work as well. For full disclosure, I run an IT consulting business. I agree that at a certain point in time, a company will reach the size where having an internal IT person is more cost effective. However, I have also seen that usually when a company is at that point, they can only afford an entry level engineer. In those cases they can hire an IT firm to handle the higher level projects/tasks and also act as second level support to the internal tech when needed. When companies reach larger sizes, having internal staff usually makes a lot of sense, but it also depends on the culture of the business and how technologically savvy they are and want to be.

With that said, I can say I am quite paranoid about security and putting mission critical data out in the cloud, however, there are tremendous benefits to doing so in a lot of cases. The first thing is to choose a reliable company with a proven track record (i.e. Amazon Web Services or Rack Space). You want a company that stands behind their support as well and has an SLA. Either way, you should always keep good backups whether you host in-house or the cloud.

If you are going to consider spending thousands of dollars on servers and co-location space in a data center and want to learn how to install a server, you can do that in the cloud with a dramatic drop in overall cost and initial cash outlay. For instance, setting up a medium sized server at Amazon takes about 15 minutes to complete and you get full root access on Linux and full admin access on Windows. You can scale your memory, cpu and disk space up and down on the fly without having to wait for data center techs to do it for you. You have total control over your server instance. If you need to spin up another server like it, it's just as easy, you can even take a snapshot of your server and then store it and then use it to create another server just like it whenever you want in minutes. You only pay for what you use.

With Amazon and Rack Space (and others I presume) you can host servers on the east coast, west coast and even Europe and have them all talk to each other or use one for redundancy and only pay for it when you need it.

You pay less then pennies per hour to use a single instance server which adds up to the same if not a lot less then a data center solution and you can shut down your servers when you don't need them anymore. There are no term contracts, so you aren't stuck. You have flexibility there that you don't have with fixed asset solutions.

If you have trouble managing or don't want to learn, then you can either hire someone internally or outsource it. Either way, who ever manages it should be vetted properly and should know what they are doing and be able to prove it, before you pay them a salary or fees.

This is by no means a one size fits all approach, but a good IT consultant should strive to ask you a lot of questions about your needs, your future plans, your budget and where your business is headed so they can give you reasonable options you can live with.

I hope this helps.

Answer 3

Most online hosts tend to deal with websites, so what you'd probably be looking for is a managed service, especially if you're not good with the server administration yourself. You would typically sign a contract for the managed service which includes a service level agreement, and the company may also be willing to sign a non-disclosure agreement depending on the nature of your system.

If you do go down the route of maintaining it in-house, I would suggest having a failover system (it costs more, but if it's critical to your business it should be a no-brainer.) Then whatever maintenance you need to on the live server, you run your system on your failover system because, as you've seen, even the most trivial of tasks can go wrong.

I've worked with a couple of companies that provide managed servers - Every City are probably the best to date, as they install and manage the whole software stack (they run Solaris systems but can provide Windows or Linux systems if you request it.) However there prices are a little steep compared to others. NTT also provide custom hosting solutions.

Most other hosting providers provide the server and it's up to you to install and manage it. The best of these breed I feel are Redstation, Memset and Fasthosts (but there support is a bit dire.)

Answer 4

A key aspect of your failover solution has to be data replication. MySQL comes with replication built-in, and it is easy to learn. Once you have replication running between your two machines, the rest is easy.

You would get most value out of running your failover host in a different location. This will act as Disaster Recovery in the event of a fire in your office, or your servers getting stolen, or a power surge blowing up your servers.

A simple two-site solution in your case is probably to continue to run your service from your office, and to replicate data from there across a VPN to a server in a hosted or managed environment. You do not even necessarily need to ever run your service in the remote site, just keep a database replica running until something goes wrong with your office. At that point, you have an up-to-date recovery solution where you can either copy the data back to your office, or migrate your service into the hosting environment. Having a plan would help.

It is quite likely that the remote environment will be better provided than your office: good air con, clean power, good comms. In that case, migrate your service to the remote environment, and replicate the data back into your office and have your office act as the Disaster Recovery site.

Having your only two servers in the same rack / building puts you at risk of environment failure.

Answer 5

I'd recommend a managed VPS. These are generally more cost effective than co-location, provide support, and yet still give you full control. Most VPS hosts I've dealt with do automated backups in the event of an emergency. You could plug in as many websites as needed to fail over to.