How bad is it really to install Linux on one big partition?

Question

We will be running CentOS 7 on our new server. We have 6 x 300GB drives in raid6 internal to the server. (Storage is largely external in the form of a 40TB raid box.) The internal volume comes to about 1.3TB if formatted as a single volume. Our sysadmin thinks it is a really bad idea to install the OS on one big 1.3TB partition.

I am a biologist. We constantly install new software to run and test, most of which lands in /usr/local. However, because we have about 12 non-computer savvy biologists using the system, we also collect a lot cruft in /home as well. Our last server had a 200GB partition for /, and after 2.5 years it was 90% full. I don't want that to happen again, but I also don't want to go against expert advice!

How can we best use the 1.3TB available to make sure that space is available when and where it's needed but not create a maintenance nightmare for the sysadmin??

Answer 1

The primary (historical) reasons for partitioning are:

• to separate the operating system from your user and application data. Until the release of RHEL 7 there was no supported upgrade path and a major version upgrade would require a re-install and then having for instance /home and other (application) data on separate partitions (or LVM volumes) allows you to easily preserve the user data and application data and wipe the OS partition(s).

• Users can't log in properly and your system starts to fail in interesting ways when you completely run out of disk space. Multiple partitions allow you to assign hard reserved disk space for the OS and keep that separate from the area's where users and/or specific applications are allowed to write (eg /home /tmp/ /var/tmp/ /var/spool/ /oradata/ etc.) , mitigating operational risk of badly behaved users and/or applications.

• Quota. Disk quota allow the administrator to prevent an individual user of using up all available space, disrupting service to all other users of the system. Individual disk quota is assigned per file system, so a single partition and thus a single file-system means only 1 disk quotum. Multiple (LVM) partitions means multiple file-systems allowing for more granular quota management. Depending on you usage scenario you may want for instance allow each user 10 GB in their home directory, 2TB in the /data directory on the external storage array and set up a large shared scratch area where anyone can dump datasets too large for their home directory and where the policy becomes "full is full" but when that happens nothing breaks either.

• Providing dedicated IO paths. You may have a combination of SSD's and spinning disks and would do well to address them differently. Not so much an issue in a general purpose server, but quite common in database setups is to also assign certain spindles (disks) to different purposes to prevent IO contention, e.g. seperate disk for the transaction logs, separate disks for actual database data and separate disks for temp space. .

• Boot You may have a need for a separate /boot partition. Historically to address BIOS problems with booting beyond the 1024 cylinder limit, nowadays more often a requirement to support encrypted volumes, to support certain RAID controllers, HBA's that don't support booting from SAN or file-systems not immediately supported by the installer etc.

• Tuning You may have a need for different tuning options or even completely different file-systems.

If you use hard partitions you more or less have to get it right at install time and then a single large partition isn't the worst, but it does come with some of the restrictions above.

Typically I recommend to partition your main volume as a single large Linux LVM physical volume and then create logical volumes that fit your current needs and for the remainder of your disk space, leave unassigned until needed.

You can than expand those volumes and their file-systems as needed (which is a trivial operation that can be done on a live system), or create additional ones as well.

Shrinking LVM volumes is trivial but often shrinking the file-systems on them is not supported very well and should probably be avoided.

Answer 2

The concept of using multiple partitions is that a full one in the the wrong place will not cause the whole system to work unexpected.

Consider a process on the machine filling up a logfile pretty fast up to the point that there is no free space available. On a single-partition machine, this could then, for example, prevent the system to write new data to /tmp. If there is another process that would want to write to /tmp it would probably exit with an error, causing unexpected behaviour.

This can be prevented if you use different partitions for places where users or processes normally write to (/home, /var, /tmp).

I would recommend that you check your old server which folders tend to get big. You can do that on the command line with

du -h -d 1 / 2> /dev/null

You'll see where the most data is accumulated and design your next system appropriately. The "-d 1" limits the output to only one level of folder depth making it more readable.

Answer 3

The main problem with having a single large partition is that filling the filesystem up it is possible that no login is possible anymore.

The user root has its home folder (/root) outside of /home because of this. If the filesystem is filled up under some circumstances even root cannot log in and cannot repair the system.

This is the reason why you normally create separate mounting points for /var, /tmp and /home to be able to login at least as root to repair the system when one of the other partitions is filled up.

Answer 4

IMHO, having one partition as / is quite reasonable.

But you can use lvm (logical volume manager). Use all disk as lvm group, but create small logical disks for /,/home,/usr and whatever your sysadmin prefers. Then put some monitoring on, that you know, when your system starts getting full and expand those disks that you need. lvresize and resize2fs is online tools and you can do expansion without restarting server. However you can not reduce disks, so you need to start reasonably small and increase where you see need.

Answer 5

There are minimal problems around the big-single-partition setup of linux, but it has big rewards.

Changing a partition layout is a little bit hard and risky thing, which you often can't do without long downtimes.

Its only advantage is that you have some protection against disk full problems. But you will find these problems much often. Imagine the situation, if one of your partitions are full, and you can't use the space on the other partitions, even if they are nearly empty!

Some professional system administrators have a total different opinion about this. They are saying, that having multiple partitions can make your system more reliable and you must know before your partitioning, how big your partitions will be. On my opinion, this simply can't be said, it is a terrible drawback on the flexibility on the system, and their real motivation is that they are simply liking to play with the partition maps.

There is a simple system named lvm, which enables the on-the-fly moving/resizing of "partitions" (in its terminology, volumes). But on a single local department server, IMHO it is normally not needed.

Answer 6

There are two main reasons for partitioning:

1. To keep static data away from non-static data
2. To keep public data away from private data

The first reason is the most obvious - you need to isolate areas that will fill up with files from those that don't, and you particularly want to protect /, to avoid an unbootable system. For example, the /var directory is typically where log files will be stored (var stands for "variable") and this is why /var tends to be mounted on a separate partition from /.

The second reason above is less cited (I last heard it on a Veritas Volume Manager course about 15 years ago) and it's really only relevant to systems where many people are logging in and performing work.

There is something of an art to effective paritioning, and this is perhaps why there are Sys Admins who take it a bit too far (IMO). Not only do you need to know the filesystem inside out, but you also have to know the intended use. I personally think it's a rather old-fashioned approach that's becoming less and less relevant to the way servers are used today.

As a software developer, I'm particularly fed up with the Ops department building Virtual Machines with thoughtless partitioning schemes that severely restrict the size of /tmp, /home, /var and /, regardless of the total disk space available, but then don't mount obvious choices like /usr or /opt separately. These machines will commonly put whatever is left out of the disk space you asked for into a "/stuff" volume that you inevitably end up installing and symlinking everything into, but that's hardly a consolation. The net result is that we often spend more time shuffling files around and sending out warning emails than doing any real work.

There is nothing inherently "bad" about having a single partition. On any system, you should be proactively monitoring your disk usage, and employing sensible housekeeping strategies (e.g. log rotation, quotas on home directories), so the only real question is: how many separate filesystems do you want to worry about?

I would therefore say: unless you're 100% confident in your ability to partition the system effectively for your particular use case, don't partition at all.

Answer 7

This allows to backup, restore or reinstall the operating system independently from the user data. This gives you freedom, independence and safety.

1. It is way easier to migrate to another Linux distribution still preserving the absolute majority of user data.

2. It is easy to revert buggy updates by applying the backup of the operating system partition (dual boot is required). This backup may even be rather old - you can apply it and later update to the knowingly stable version.

3. It is simple to revert to the previous version of the operating system if you do not like the recently applied "major upgrade" (dual boot is required).

For the biggest potential of this approach, you should also configure dual boot (can also be CentOS/CentOS), so that you could overwrite one operating system partition while running the operating system from another. And, surely, you must back up the system partition at least once a few months.

Answer 8

IMHO it is up to you entirely. First consider a few things, though entirely is somewhat relative.

• will this system be administered frequently ?
• will this system be used by one or more users ?
• will this system be serving as a desktop or a server, or both ?

Since one can consider (almost) any directory a mountpoint one should also consider what contains somewhat growing data and what contains growing data.

You'd be surprised how little a linux system ( somewhat growing data ) requires to run and how much is consumed by growing data ( typically /var /opt /home /srv )

It also depends on how you define the use for this system which outlines the partitioning requirements. The use for LVM included.

A typical desktop system would require about 20GB for installing loads of software, having all the rest assigned to a dedicated /home will do fine then. LVM causes minor overhead on your system and in this particular case is not of such great benefit. Though opinions might differ.

On a server it is less likely for installed software to be as dynamic as for a desktop system. It is also wiser to have actual mountpoints for typical filesystem components such as /tmp /var /usr /home /opt /srv The use of LVM here is recommended not to say mandatory.

This offers for great modularity of your system, it also permits to do silly things such as cloning that partition into a VM for example. Or creating a block-level backup using dd.

Based on some experience, here are a few notes. Also consider having multiple mount points permit for greater control, assigning a fast or slow disk device to a mountpoint can make a world of difference and can noteably increase cost efficiëncy.

Mounpoint /

• 1GB ( if using separate mountpoints for /var /usr /opt /home /tmp )
• +10 or even +20 GB if using as a desktop system with a separate /home

if using mountpoint /home

• assign all free space if used, /home ne

if using mountpoint /opt

if using mountpoint /usr

• this is a tricky one and hugely dependent on the installed software base

if using mountpoint /var

• this is a tricky one and hugely dependent on the installed software base
• for example, databases write their data here on Debian-based systems, if not all Linux
• having a separate /var/tmp is not unreasonable

if using mountpoint /tmp

• consider tmpfs exists and allocateds /tmp to RAM
• consider some applications may write a lot of data here

Answer 9

I have to question, first of all, why you are even posting this question here, being a biologist who is arguing with an apparently competent system administrator regarding the finer points of hard drive partitioning! (no offense, just really wondering why you are not trusting your sysadmin).

So, a few observations:

• 1.3 TB is not a large drive anymore. 2 TB is a more-or-less standard SATA drive size in the desktop world these days.

• an installation of any Linux Distro is unlikely to require more than 100GB. Certainly, the size for / (root) and (swap) should be easily determined as upper-bounded numbers by generously over-sizing them (for root) or by system configuration guidelines (swap).

• the mount point for /home should be pointing to something off on your 40TB RAID server. There is no need for that data, the users' home directories, to be anywhere on that root device. Putting them on the RAID server probably gives you better protection anyway. And, it's most likely a readily expandable NAS facility, whereas the little RAID built into the server box likely is not.

• you should probably put your special software into a separate partion (mount point for /usr/local/bin etc) so you can preserve this across OS updates and root partition wipes. Otherwise you're faced with the possibility that you will have to reinstall your "special" software apps after an OS upgrade/fix/whatever.

• if you want to worry about your system's administration, I'd ask a different question: what is the disaster recovery process after the building catches fire and the servers and RAID boxes are destroyed? Unless the data you care about remains entirely in your head, this is a question that every user should ask of their IT/sysadmin folks. The strategy should include questions such as "how will we replicate the hardware we need" and "how long will it take before we could be back up and running". Some discussion about virtualizing your servers might help resolve issues around hardware dependencies and getting things back up and running without the need to reconfigure your OS (since it could be configured to run in a "soft" device environment that does not change, even when the underlying hardware is completely different)

• similarly, you might want to ask what the strategy is for protecting user data against program and user error data losses. Saving an empty file over the really great draft of your research paper, or having a user typing the wrong command (rm -rf * for example) will cause data loss just as surely as an earthquake or fire or other physical damage. The solutions for individual file recovery are a bit different (or can be!) from those most useful for wholesale disaster recovery.

-

Answer 10

You don't have to install software to /usr/local, you can install all software in a different prefix, which might be in /home. Most software can do do this when you compile it from source, by running e.g. ./configure --prefix=/home/bin

Since you're a biologist you might be interested in a lot software that's not properly packaged in an rpm or deb and you will have to compile it from source anyway.

I'm a sysadmin for an HPC system with a lot of biologists among our users, we install all the software they request under a /apps/ filesystem, so I know it is possible to do this for most software, however, sometimes it might be very hard. To solve this problem my colleages and I have been writing on a tool called EasyBuild (Free and open source) It can compile and install software from source, and install it in a different folder, and automatically create an environment module file for you, so you can actually have 2 different versions of the same software installed, and have no conflicts.

Have a look at our list of packages we can install with just one command, as a biologist you might recognize a lot of them ;-)

Disclaimer: I'm a developer of EasyBuild

Answer 11

My short answer is that even a desktop should never use "one big partition". I recently tried that against my better judgement because it was "just a laptop" and the auto-installer used a single partition, and I just clicked the button.

When I went to install a different distro, I then had to repartition my drives because the installer isn't going to install over an existing distro. It can and WILL keep your /home intact if its on its own partition. So, I ended up booting a gparted live-cd and shrinking the partition, making new partitions for /home and others, moving my data to the new partitions, and then finally booting into the installer for the new OS.

Ideally, put / on an SSD, /home on a hard drive, /var on a hard drive, /usr could be on an SSD or HDD depending on how often you plan on upgrading. /tmp to a hard drive. I usually make another partition for shared media files like mp3s and movies with symlinks to it from my /home. Note that /sbin is part of root, and is /bin and /root. Thats the difference between /bin and /usr/bin, is /usr is stuff that might not be available until all your drives are mounted, so the mount command can't be in /usr! I usually keep a couple extra partitions for other linux distros, like that gparted one is on my hard drive just in case I screw up something really bad, I have another live system ready to do recovery work with.

For a server where you might need to move stuff around, add storage dynamically, and needs to stay up all the time, definately use LVM!!!

Answer 12

I think that in general for a beginner/introductory *ix user that having the fewest partitions can work until more is learned about the nature of the system. However, you can't simply have one paritition and in your case, sir, for multiple reasons.

The first and more publicly practical reason for this is that most all Linux Systems require a swap partition (generally it should between 1-2 * your RAM) also require a separate system, boot or home partitions and in the case of UEFI booting linux systems an EFI partition (just 500MB).

The second reason, specifically applicable to your situation, is that taking 6x 300GB drives and making them a raid 6 is not, to say the least, the optimal arraingment. Although, new tech, hails Raid 6 as a universally better system the striping algorithm is more required and the space required to store information (compared to RAID 5) is larger in size.

Not to mention that RAID 6 requires additional piece of hardware. Which should be used, in my respectful opinion, in your case to purchase larger disks to avoid disk failure, downtime, disaster recovery or added tech help costs. I understand that some, many maybe, will disagree with me and I want to re-affirm that when disks become larger in size over the next two years ( as they've dropped in price over the last few ) RAID6, for larger arrays and large income corporations will be an obvious choice. However in this case, I do not suggest the use of RAID 6.

To the second (non RAID based) issue, creating one giant partition when mirrored may work for you however, if you want to maximize efficiency use larger drives and multiple partitions. That way if you have a double-disk failure you will have no downtime on certain mountpoints or little depending on the /dev+/dir.

At least make your /sys (system, kernel, etc) run seperately so that if for any reason your kernel decides not to boot you can simply use a recovery kernel, remote boot or PXE, disk boot, etc your kernel and have company-wide access to your information while the d-recovery process takes places.

Your company may not care about these accents so long as the system works but I'm trying to explain reasons why people do things. I would also love to hear more from others, for and against this argument and to raise other points. If you don't agree, let me know why. Poster-- I will PM you some links as well.

One Love for our Linux Community Spencer Reiser

PS Especially on network servers or systems available to the general public, even if via credential, it would be truly best to seperate your partitions. Others can input more here, I need more coffee.