Manual updates are best as mentioned here in the sense that you can see what's happening. However, for very large numbers of servers that might become impractical. Dry run is a standard practice, in fact, most package managers will ask you before proceeding.
Updating regularly tends to be best though it can be a bit of a balancing act. Frequent updates means less in one go and less to go wrong at once. If things do go wrong there are fewer candidates to inspect. Packages are also slightly better at updating in smaller steps, as generally when the programmer updates they're looking at going from the last version to the next, whether they'll give any attention beyond the last version can vary, though this tends to matter mostly for software that's rapidly evolving.
Not all updates are non-disruptive. You'll want to watch out for this. Some will restart services leading to down time.
In an ideal setup you might have the following:
- A means of seemlesly switching servers (A/B or tick tock). This means you update one while it's on the bench, then simply swap the traffic from the current one to the new one. This may be more complicated for services such as databases.
- The ability to test updates. You should have test servers that are practically clones of production (but without connecting to any production services). These would allow you to test updates first.
- A good backup strategy, incremental is ideal. You never know. It's always better to be safe than sorry.
- Be aware of which times have the most activity and what level of downtime is tolerable.
- Know how to rollback an update or a specific package.
- Have your own package mirrors so updates are consistent and predictable across servers. This is the first step towards a decent unattended system that you can trust. It means you can update the mirror, run update on one or more test machine then if that's good let it go out automatically. I had an excellent time with aptly managing around 800 EPOS machines.
- A good level of consistency so that you can know that if something will work here, it'll work there.
Some of these can be overkill to varying degrees for small setups but should be kept in mind.
Generally speaking, updates are usually relatively painless for server distros. This is because they nearly always only stick to bug fixes and security updates. You may however have problems if people have done odd things to the system or you add additional package sources.
Although it's moderately rare, they do occasionally make mistakes and break compatibility between minor package versions.
