8

We are running a production server based on Ubuntu 9.10 Karmic Koala, kernel is almost up-to-date (2.6.38.2-grsec-xxxx-grs-ipv6-64) but karmic package repository is now ridiculously outdated, eg. Nginx is 0.7.62 - really buggy - while latest stable is 1.0.x!

In addition, Karmic just reached its end of life.

This question: Best practices for keeping UNIX packages up to date? looks similar but actually only includes some suggestions about package managers; not at all what I need!

So the options that I see are:

  1. Get a new machine, install it from scratch, migrate
  2. Distribution upgrade
  3. Use a different repository (launchpad/ppa / backport / pinning)
  4. Build your own

The disadvantages of #1 are quite obvious.

I do not dare do a dist-upgrade path though, as downtime and possible catastrophic consequences are just impossible to predict for a production server, and currently am mostly re-building my own required packages. But I'm sure I might be missing some.

It is not really clear to me what the risks are (stability/compatibility) of using Ubuntu backports, in addition, nothing is officially provided for 9.10 anymore. Launchpad are individual-builds, similar question - how much better is this than compiling my own?

Building packages seems fine, but:

  1. Sometimes I have trouble reproducing the correct ./configure options in order to re-use my existing configuration files
  2. I am sure there are tons of packages and dependencies that are now pretty outdated and possible sources of bugs

Finally… what about "old" packages in a recent distribution? I guess there's no other way than re-building them myself? Is a combination of 2 and 4 finally the best path?

Is there any objective consensus on what is the best way to do this, or reasons why some of my options are fine/not fine?

If really there isn't, I will accept that the question gets closed before creating an endless thread!

Jonta
  • 103
  • 4
Stefano
  • 751
  • 1
  • 12
  • 23
  • 1
    It is for reasons that you are currently experiencing that only LTS versions should be used for servers. In answer to your question, until you can do either #1 or #2 you will be stuck with #4. I can see #3 starting to fail often as more time passes as dependencies are not available. – daemonofchaos May 25 '11 at 13:50
  • indeed @KayakJim, though we should have figured it out at the time - but when server load was low maintenance would have been acceptable, we did not think far ahead enough. lesson learned (hopefully). – Stefano May 25 '11 at 14:40

3 Answers3

4

Maintaining your own distribution is a lot of work. Even if you maintain the backports, you will soon be overwhelmed by security issues to fix, and have to pull low-level libraries to keep updating your software, which might break other things (I maintain servers running 6-year-old distros, it's not fun).

Upgrading is generally a good solution. do-release-upgrade is well made, and you should be able to upgrade without issues (especially if you only used official packages).

My favourite solution though might be the reinstall path. More specifically, your servers should be managed using a configuration management system such as Puppet, Cfengine or Chef. If all your configuration/package needs are specified using such a tool and your data are safe on a separate partition, it's much easier to reinstall quickly. You just install a new distribution without erasing the data partitions, and then run the configuration management tool to reset your packages/configurations. I believe this is the cleanest way to do, especially if you have several servers to manage.

If you are using non-official packages, you might want to identify them before you upgrade/reinstall. maintenance-check can help you identify the packages that are not officially maintained by Ubuntu:

$ bzr branch lp:ubuntu-maintenance-check
$ cd ubuntu-maintenance-check
$ ./maintenance-check -f n

If you want to reinstall, you can also export the list of installed packages:

$ dpkg --get-selections > myinstall.txt

and your debconf database:

$ debconf-get-selections > debconf.txt # from the debconf-utils package

As a note, since you're currently using Karmic, it might not be too violent to upgrade to Lucid, which is an LTS release, still supported until 2015 for the main server packages. This should leave you enough time to setup a viable automated installation for the future.

When you ask about Launchpad packages, I suppose you mean PPAs. There are tons of different PPAs. Some are experimental, some are stable. Some are maintained by official Ubuntu developers, some are maintained by people hardly know how to do a package properly. It's hard to say in general if packages you find on PPAs are good, there's no general rule. The best hint in this case might be too look at the owner of the PPAs to get an idea of the possible quality of their packages.

raphink
  • 11,337
  • 6
  • 36
  • 47
  • of course we did not think about puppet & co. at the time. But indeed you making a very good point that, should we go the re-install path, it's better to properly prepare an easy-to-replicate installation. PS. "especially if you only used official packages" of course not, unluckily! – Stefano May 25 '11 at 14:42
  • Then the first step you might want to take is to identify the non-official packages. `maintenance-check` can help you with that (see my edit). – raphink May 25 '11 at 14:53
  • Selected answer for the additional tips, including using conf management systems and maintenance check, and about PPAs. I just realized though, after looking into latest repositories, that packages are not always up-to-date, eg. even in 11.04 nginx version is OLD (0.8.54-4) and they'll never move to 1.0.x in that distrib. Any advise for those situations? – Stefano May 26 '11 at 09:08
  • 1
    @Stefano: Ubuntu uses the same kind of policy as Debian, which is that software do not get upgraded in the main repositories after release (after feature freeze to be precise). This can indeed lead to old versions of software in a still maintained release (especially if the software releases fast). You can use backports to get new versions, but you will probably lose in stability and security updates. There's no perfect solution for this, unless you're willing to maintain your own backports, which, as I stated before, is quite costly. – raphink May 26 '11 at 09:19
2

If the server is not exposed to the world, and you trust your users absolutely (generally that's not a good idea), then if it's working, you could just leave it be.

If it is in any way exposed to the outside world, and/or you entertain the idea of legitimate user playing with it in an illegitimate way, then you absolutely need fixes and patches to your installed software.

In this case, you have two options:

  1. Run a supported distribution, and get updates to your software, or

  2. Backport all fixes to your unsupported distribution, which, frankly speaking, doesn't seem feasible.

I'm not an Ubuntu user, so I cannot comment on completeness of patches you'd get through your option 3, but if you have any doubt, I'd assume you won't have complete coverage.

The best solution is to move to a LTS version of Ubuntu, which will give you support for the given package versions for some time to come. In time, some of the packages will be outdated, but your environment will have security patches and will be stable (no package version bumps). From my experience, stability of a known working environment is usually more valuable than new features.

It seems, that your current position is not maintainable, and you have to move. The only safe way is to get a second machine (or a virtual machine) and to test migrations until you have a repeatable successful procedure, then apply it to the production machine. If you use your backups to do test-migrations you'll have a good opportunity to test your backup procedures too.

Paweł Brodacki
  • 6,451
  • 19
  • 23
  • thanks @Pawel-Brodacki. The server is definitely exposed! I understand your point on stability over features.. problem is, often stability comes with major version steps. Eg. nginx 1.0.* series is more stable than the 0.8 series included even in latest natty. Any suggestion about that? – Stefano May 26 '11 at 09:09
  • 1
    If it currently is good enough, then "if it ain't broke, don't fix it" rule applies. After that it's business calculation: is added stability going to save you more than it is going to cost you to maintain a set of packages by yourself. If it's just nginx, or nginx and a handful of libraries, then compiling by yourself, testing and deploying is something that can be done. In that case, however, it would be prudent to follow development of the packages closely to quickly close any discovered bugs. – Paweł Brodacki May 26 '11 at 11:16
2

The only real way forward is a distribution upgrade. I can understand you being nervous about that, since by now you will be jumping several releases ahead (11.04 has just been released).

I would recommend to make a clone of the drives in this machine and then use a separate computer to run with the clones, and use that to do a series of test upgrades. Make notes of all the issues encountered and repeat until you have a clear procedure for all of them. Then apply this to your live server.

If you cannot afford any downtime, then migration is your only way out. Forget about the pinning and backports, that will only keep you alive for a limited period of time. And the "roll your own" option is not even worth considering. Just my 2 pennies' worth.

wolfgangsz
  • 8,767
  • 3
  • 29
  • 34