9

This is an open ended question, but I do wish to have a constructive and helpful discussion into this topic.

So to clarify on the question: On a server running CentOS 7 (or any other Linux distro/version for that matter) Is it best to stick with the package version in the Base/EPEL repo or is it fine to get the latest stable version form the package site? In this case I am more specifically referring to packages like nginx, MariaDB and PHP 7. For example, what would be the pros and cons of installing nginx 1.8.0 over the EPEL version 1.6.3? Are there any performance differences or security risks either way?

All discussion and experience is welcome, please try to cite resources and facts.

  • 4
    I'd avoid installing *over* an OS-supplied package. First, you don't really know what if any customizations the distro provider may have made - configuration file location, etc. For example, what happens if you do install nginx 1.8.0 over the distro-supplied 1.6.3 and then the distro jumps to 1.9.9? What's that going to do to your custom install? In general - don't screw with OS-supplied *anything* unless you want to commit to maintaining your customized OS install **for the life of the server**. For a later version of an OS-supplied application, install it in `/usr/local` or similar. – Andrew Henle Jan 17 '16 at 13:46
  • That's a good point, my retort to that would be: Again for example take nginx, the latest stable being 1.8.0 and the latest legacy version being 1.6.3, what if a security fault is discovered in the distro version of 1.6.3? – GiggleSquid Jan 17 '16 at 16:29
  • 5
    [Red Hat](https://www.centos.org/docs/5/html/Deployment_Guide-en-US/ch-security-updates.html): *As security vulnerabilities are discovered, the affected software must be updated in order to limit any potential security risks. If the software is part of a package within a Red Hat Enterprise Linux distribution that is currently supported, Red Hat, Inc. is committed to releasing updated packages that fix the vulnerability as soon as possible. ...* (cont) – Andrew Henle Jan 17 '16 at 17:24
  • 5
    *Often, announcements about a given security exploit are accompanied with a patch (or source code that fixes the problem). **This patch is then applied to the Red Hat Enterprise Linux package, tested by the Red Hat quality assurance team, and released as an errata update**. ...* Do you plan on signing up for all that entails? – Andrew Henle Jan 17 '16 at 17:25

3 Answers3

9

Generally, I try very hard to use system default packages.

However, this is sometime not possible. To do an educated choice you had to answer these questions:

  1. do the distribution's packages provide the features you require? If so, you don't even need to search for other packages; simply use the packages provided by system repositories.
  2. do you need official support and/or had you to comply to specific policies? If so, you can't use an unofficial repository. In this case, you are probably using the wrong distribution for your software project.
  3. if the answer to the previous questions was "no", you had to search for a more recent software version. Does exist a well-recognized repository with the required package? If so, use it.
  4. if no specific, reputable repositories exist, you had to use the upstream software. In this case, try very hard to use packaged software (eg: RPM, DEB, ecc) rather than plain tar.gz (or the likes).
shodanshok
  • 44,038
  • 6
  • 98
  • 162
  • 3
    Another thing you can add: Downside. Does your *employer* (if applicable) know you're doing this with company systems, especially if they're paying for support? Among other things, there may be *legal* reasons why a company is using a supported distribution, and rolling your own may very well open the company up to liability issues, if, for example, user data has to be protected. If your non-supported home-installed package leaks user data because you missed a security fix, you can no longer point to Red Hat and say, "We were running their OS and we paid them to keep it up-to-date." – Andrew Henle Jan 17 '16 at 17:38
6

Matthew Ife's and shodanshok's answers cover the issues in general, but I want to address your specific concern by putting the issues in context, as it is exactly these sorts of systems that I manage.

My current build for deploying PHP/MySQL web apps is:

First, let's consider why we choose a particular distribution or package set. Either we value stability over the latest features, or we value the latest features over stability. It's not generally possible to have both in the same distribution, as stabilizing software requires time to fix bugs, and adding new features introduces bugs, thus instability.

As a general rule I want the operating system on which the application runs to be as stable as possible, but with a reasonably modern feature set. Thus I'll choose CentOS 7 over CentOS 6, which is rather old at this point, and while it will work, it doesn't have as much time left in its support lifecycle, so I won't use it for a new project.

However, I then ran into the issue that the version of nginx included with CentOS was too old and did not have some required features and bug fixes. Thus I went searching for alternate packages, and found that nginx.org distributes their own. I switched to them almost immediately and have found them perfectly stable over the long haul.

Then there is PHP. I know from history that the version of PHP shipped with CentOS will be the only version it ever gets, and will only get security updates; no new features or bug fixes. Thus, once it is out of support upstream, I am eventually going to be unable to run modern PHP web applications if I use those packages. Thus it's necessary to replace these as well.

From long experience I've learned that it's best to track bugfix releases with PHP, not simply to freeze at one point release and take only security fixes, as the web applications I run will also be updated and will need those bugfixes. So after evaluating many different sets of PHP packages, I settled upon remi's pacakges. Remi happens to be a Red Hat employee and is also responsible for the PHP packages in RHEL/CentOS. So I know his packages will be high quality, and they have been. They are drop-in replacements for the system packages and work perfectly.

Finally we get to MariaDB. You can choose to keep the system packages here and suffer no ill effects. I chose to switch to MariaDB's 10.0 packages (and soon will go to 10.1) to take advantage of TokuDB and some other performance enhancements not available in the 5.5 version shipped with CentOS, and that it will never receive major upgrades for.


Overall you need stability in your base system, but web apps change much more rapidly than, say, line of business software, and your server will need to keep up. Thus I've chosen targeted points where upgrading packages will gain clear benefits with little additional administrative overhead (a.k.a. work).

Michael Hampton
  • 237,123
  • 42
  • 477
  • 940
5

The short answer is, always use whats provided by the system repositories. Be very careful what repositories you do install too. Some are just plain bad.

You shouldn't ovewrite the systems packages with newer versions, Redhat is designed and orchestrated very carefully and you might end up with strange bugs or problems if you do.

Some things to consider and look out for which can cause issues include.

  1. Some repositories are simply badly maintained. They dont update with security fixes for packages.
  2. People tend to write bad RPMs, they dont mark config files as config files, which overwrites your config each time you update, which could cause problems. I've seen this problem before.
  3. They do not sufficiently declare their dependencies properly. I've seen this before too, where a php package was put on the system but didn't update the pear package which introduced issues.
  4. Installing multiple repositories all offering the same package names can lead to unforeseen dependency problems on your system.
  5. Some packages overwrite or rewrite system configuration files that other packages depend or expect to exist. This leads to problems with other packages you might not expect.

Never build packages from source and install them over the top of the packages that are there. This breaks your systems package integrity which can lead to strange ABI problems like receiving unresolved symbol or undefined reference messages. It is pretty critical the system maintains a reliable and accurate index as to what software has been deployed on a given system to ensure that it all works with each other properly, this is the reason we use RPMs in the first place.

The viable (and Redhat blessed) way to go about resolving this is to use software collections.

www.softwarecollections.org

It installs is software and its 'new' dependencies in its own root. This can make it slightly harder to apply the package in your environment but does protect your system from weird errors or problems. It also installs the packages in their own namespace, letting you install multiple versions of a package in parallel.

The website gives instructions how to install and activate these packages, it contains most of what people miss on older versions of CentOS and Redhat (EL6 in particular). Some things I've used from this website successfully.

  • MySQL 5.6 and MySQL 5.7, MariaDB.
  • PHP 5.5 and PHP 5.6
  • Apache 2.4

Note that your default position on this matter should be not to adjust from what the Redhat repositories are pushing. Instead, make an assessment as to whether you really do need an updated version of a package, in particular what your specific requirements are, what problems it is supposed to fix and what risks it introduces.

As a general rule, if you find yourself constantly needing updated software and/or requires multiple parallel versions of the same packages to make things work it is usually an indicator you are doing something wrong.

Matthew Ife
  • 22,927
  • 2
  • 54
  • 71
  • 1
    _Some_ system packages, such as user applications, can be replaced with newer versions safely and with no ill effects, _if the packager knows what he is doing_. But it is _not_ safe to upgrade _libraries_ in this way; attempting to do so is what causes the ABI errors you mentioned. Unfortunately the knowledge of what can be upgraded safely and how to do it doesn't seem to be common among packagers. Even Red Hat has [occasionally gotten this wrong](http://serverfault.com/a/563055/126632). OTOH, Software Collections are a royal pain in the ass to use... – Michael Hampton Jan 17 '16 at 18:14
  • 1
    `nginx` is one of those 'all in one' packages like that. But, `httpd` (libapr dependencies) and `mysql` (libmysqlclient dependencies) in particular are not. Bad updates of both of these packages has caused errors in `python` and `php` for me in the past. The trouble here is its not simple to know how how one package interacts with another unless you know what to look for (translation: been burned before by it). – Matthew Ife Jan 17 '16 at 18:20
  • 2
    You can upgrade something like MariaDB with no real problem, as it carries a compatibility library to allow programs linked against the older system version to continue to work. You just have to remember to install the package (and yum should complain if you don't). PHP is more complex, as it has a lot of dependencies that must also be kept up to date along with it, and if the packager isn't doing this, the packages are worse than useless. Fortunately, since remi is also RHEL's PHP maintainer, he knows what all of those are, and his packages have been fine. – Michael Hampton Jan 17 '16 at 18:21