-2

I hope I have what is a straightforward question. As I am sure the steps of setting this up are complicated I would like to know a few things. We have a team that has been working on a great platform that is near ready for completion. We have been developing on a standard lamp installation nothing special. The problem we have run into on the schema is how to plan for growth for the lamp configuration. It is easy to just buy a server and load the app on their but how do we plan when we need 3, 4, 5 servers and continuing to add to this scheme without breaking the site or taking it down every time? Essentially downtime needs to be avoided. We prefer to purchase our own servers and maintain them ourselves with co location.

Essentially how do we setup the site so that from the beginning it is made for parallelism ( I believe that is the word?).

DEVPROCB
  • 101
  • 3

1 Answers1

2

I hope I have what is a straightforward question.

Sorry, but this is pretty much the exact opposite. Scalability is a large set of design choices that pretty much need to get baked in from the start if you want it to be super easy. It's generally a lot of extra work, and if you've never built such a system before it's usually just easier to learn by "growing pains".

Our generic canned Answer to these questions is: deal with scalability later. Build your site/application/whatever, and throw a beefier server at it as long as you can. Once that option runs out, start worrying about how you'll split logical layers (which is generally the easiest), like front end web server from the database server. Then figure out how to run two web servers simultaneously, and so on.

A major part of the problem is that while I can waive my hand and say "database sharding" or "active-active clusters", the actual implementation of those is going to depend quite a bit on how your application works internally. LAMP isn't even very "standard" - I assume you mean PHP or Perl, neither of which play well with concurrency out of the box (and will probably be one of the first bottlenecks you'll deal with; at least these are common problems and there's plenty of articles around on how to deal with it).

Update: I'm having some trouble actually finding a concise outline of scalability topics. I ran across this:

  • Stateless application
  • Loose coupling
  • Asynchrony
  • Lazy loading
  • Caching
  • Parallelism
  • Partitioning
  • Routing

A similar outline from a different source:

  • Separation of Concerns (presentation, logic/app, data, transit/routing, management)
  • Minimize Distribution of State (ie, state doesn't cross the aforementioned separation)
  • Redundancy
  • Separate Environments (Dev, QA, Stage, Prod)
  • Build
    • Automation (scripting, build, management, etc)
    • Configuration Management
    • Central Identity Management
    • Deployment Management
  • Run
    • Monitor
    • Reassess Capacity, Plan
    • Logs, Logs, Logs
    • Continuous Improvement Cycle (Agile Dev at the extreme)

I could write a book breaking out each of the subjects in the first list into individual technologies. Each of those could have a book written about implementation on various platforms, different models, and more (many already do have books about them).

If anyone want to add more to this, or add details, feel free.

Chris S
  • 77,337
  • 11
  • 120
  • 212
  • 1
    And...having a problem like "too many people are hitting my website!" is a GOOD problem to have... – TheCleaner May 06 '13 at 13:19
  • i disagree with these question being closed as I am asking how do I setup the servers from the forefront to avoid running into downtime later. @ Chris I appreciate your insight, but what would be the solution to ensure server uptime as it will be vital at launch to ensure that we have the ability to essentially "just add servers" as huge demands come into play? – DEVPROCB May 07 '13 at 00:14
  • 1
    Uptime and Scalability are somewhat different. Having an application design that allows for greater redundancy will enable some horizontal scaling, but not necessarily what you're looking for. – Chris S May 07 '13 at 05:04
  • @chris, thank you so much for the massive insight on the information! I appreciate the information and that is what lead me to ubuntu JUJU. Because of you we were able to find the solution we feel will fit our needs. This may be the last time I come to this site though, because a question like this should allowed to continue to grow. – DEVPROCB May 07 '13 at 15:56
  • Part of the problem is that [SF] is for Systems Administration, not development. But I ask the guys on [SO] (which is for development) and was assured this sort of question was off-topic there too (I don't quite follow their logic, but have just as much say in their affairs as you do). – Chris S May 07 '13 at 17:06
  • @ChrisS who knows, while also researching I also came across another great system called freenas which will be needed for mass storage and can be integrated in a production environment as well. I think a question like this would have helped people that are not system admins divulge best practices and different technologies out there to help people get started instead of relying on hosting providers that charge an arm and a leg. I rather go to the source and place my own configured servers up, but make sure I have the proper equipment that will allow true growth. What happened to learning? – DEVPROCB May 08 '13 at 13:02
  • That's just it though; [SF] is explicitly not meant to help Non System Administrators (et al). Developers are implicitly "not welcome". [SF] is different from most [SE] sites in that regard. We're not trying to be nasty to the world (though at times it would appear so); but we can't be Free SysAdmin Support to the world either. I could go on for days about the history of how we got to the *very restrictive* topicality of the site, but it's a familiar story... Also, as much as I love FreeNAS (it's built on FreeBSD, my favorite OS), specific product recommendations are also explicit off-topic. – Chris S May 08 '13 at 13:22