5

My main area of work is programming, and I sometimes have to perform some actions on production servers (when deploying application or configuring something...).

So my question is, what should be done to be as sure as I can be that I don't mess up with something on production server that I intended to do on my machine/test environment? Wiping out some database or deleting logs, for instance.

What comes to my mind are few basic things, like:

  • having the production console open for only as much time as I need to do my job there, then close it
  • configuring the console on production server to be really diffrent then others - like red background or something like that
  • having every command there require an additional confirmation

Are there any other rules that help to stay sane and not mess up with production server?

Frederik
  • 3,293
  • 3
  • 30
  • 46
kender
  • 287
  • 1
  • 5
  • 1
    Colors even save time. With a quick glance of the font color (white background) I know where I am. No need to read. Actually this applies to general programming as well, colors add a lot. We configure the colors the same across all workstations, so even if I am helping a co-worker, I have my bearings. – 700 Software Apr 04 '11 at 20:30

5 Answers5

7

There is always a fine line between protecting from errors and hindering work. Although I am a dedicated admin, our organization is small enough so it is not practical to implement all the extra overhead and red tape that big enterprises like to have to keep their production safe.

Besides your rules, we also have the following:

  • never use the same credentials for test and production (i.e. login, database), if you can't afford to tighten the firewalls properly put drop rules for common mistakes, i.e. connecting to test backend from live frontend, etc.
  • implement as much of change management/version control as you can, not only on the application level, but on the OS level too. Here we use cfengine, but there is plenty of other options, i.e. puppet or even grow your own.
  • automate all routine tasks, i.e. you should not be deleting anything manually on production on a regular basis
  • keep an up-to-date documentation of everything, do not consider a task done until it is properly documented, combine wiki and bug tracking system. There has to be a "Why" for every change and for every weirdness in the config. I know this sounds like a Holy Grail, but any documentation is better than nothing at all.
Aleksandar Ivanisevic
  • 3,327
  • 19
  • 24
4

Use different passwords: apart from the fact that different users might have access, you can't log on to the production system database (and do something bad) thinking it is the test server database if the password is not the same.

Colour-coding: which is also a good idea, because it is hard to miss. Set the colour of the command-line prompt according to the host name, and set the desktop background colour for GUI access.

Community
  • 1
Peter Hilton
  • 475
  • 1
  • 11
  • 11
3

My main methods for doing this are as you stated, changing the colour of the background, changing the layout, making it significantly different to the development server.

Another thing you can do is change the password on the production server to something that you have to think about to input, that way you are conscious that you are logging on to the production server.

Another method I have seen used in other companies is to actually have a method for promoting work from development to production, using a web interface or batch script, so that you don't actually work on the production server, but just promote work when it is ready.

Sam Cogan
  • 38,158
  • 6
  • 77
  • 113
1

Rule 1, if dev's can get access to change things, it's not a production server.

Running proper version control (not just a VCS, but proper tagging) can really help here.

LapTop006
  • 6,466
  • 19
  • 26
1

One thing I've seen implemented is to use an "intermediary" terminal server between your operational network and your production network, so any changes have to be done (for production systems) when in a terminal services session. By not doing the same with test systems, the distinction is very, very clear.

Rob
  • 647
  • 1
  • 7
  • 27