2

i have a Virtual Machine with Centos 7, on which i want to mount some ext4 partitions. Physical disks are actually hard disks provided by vSphere. All disks are located on the same NAS - so the three working ones (a,c,d) are virtually identical to the problematic ones (e,f,g)

content of fstab:

UUID=b6ebbca4-71d0-4d2e-bc1a-e465e5190698   /boot           ext4    defaults        1 2
UUID=2c3ab9f5-60a6-4a79-ada3-84737eef7748   /           ext4    defaults        1 1
UUID=e130758c-5108-44de-bbd8-7e003c9072bc   swap            swap    defaults        0 0
UUID=decbbdb6-8362-41ef-aa72-83066c172913   /home           ext4    defaults        1 2
UUID=5717b613-a9f4-43c9-95d2-cfbbb891bd19   /apps_home          ext4    defaults        1 2
UUID=e24df090-2dda-404c-8944-a28bd37d6c5e   /apps/var/progress  ext4    defaults        1 2
UUID=5f254c77-a91d-4255-8315-9325ddb7a9d8   /apps/var/standard  ext4    defaults        1 2
UUID=746c70c1-002a-4249-a06f-df393a99252c   /apps/var/custom    ext4    defaults        1 2

lsblk -o '+UUID,FSTYPE' output:

NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT         UUID                                 FSTYPE
sda       8:0    0   50G  0 disk
├─sda1    8:1    0    1G  0 part /boot              b6ebbca4-71d0-4d2e-bc1a-e465e5190698 ext4
└─sda2    8:2    0   49G  0 part /                  2c3ab9f5-60a6-4a79-ada3-84737eef7748 ext4
sdb       8:16   0   40G  0 disk
└─sdb1    8:17   0   40G  0 part [SWAP]             e130758c-5108-44de-bbd8-7e003c9072bc swap
sdc       8:32   0   50G  0 disk
└─sdc1    8:33   0   50G  0 part /home              decbbdb6-8362-41ef-aa72-83066c172913 ext4
sdd       8:48   0  600G  0 disk
└─sdd1    8:49   0  600G  0 part /apps_home              5717b613-a9f4-43c9-95d2-cfbbb891bd19 ext4
sde       8:64   0  250G  0 disk
└─sde1    8:65   0  250G  0 part /apps/var/progress e24df090-2dda-404c-8944-a28bd37d6c5e ext4
sdf       8:80   0    1T  0 disk
└─sdf1    8:81   0 1024G  0 part /apps/var/standard 5f254c77-a91d-4255-8315-9325ddb7a9d8 ext4
sdg       8:96   0    2T  0 disk
└─sdg1    8:97   0    2T  0 part /apps/var/custom   746c70c1-002a-4249-a06f-df393a99252c ext4
sr0      11:0    1 1024M  0 rom

The issue is that fstab is not mounting the three last partitions (that are supposed to be mounted under /apps/var/progress /apps/var/standard and /apps/var/custom respectively) consistently after reboots. It does mount one of them from time to time (it is totally random if any of these three is mounted, and it is always only one or none that is being mounted). The other partitions are being mounted consistently without any issues.

The mount -a option is not doing anything, however $mount /dev/sde1 (or dev/sdf1 or dev/sdg1) work like a charm. Similarly, mounting by using the command $mount /apps/var/progress works without any issues as too.

For now i just used cron to mount these three partitions after every reboot but in the long run i would like to know the root cause for this strange behaviour.

I tried replacing the UUID's with /dev/sd* names, i tried putting quotation marks.

mount always shows that the partitions are mounted on right catalogues, however umount reports that the partition is not mounted at the moment.

I noticed that the three problematic partitions had ext3 filesystems previously, and were somehow converted into ext4. Also, for these three partitions blkid shows the PARTUUID in addition to UUID (i think they were previously used with centos 6)

Mikołaj Stempniewicz
  • 23
  • 1
  • 6
  • I'm not sure how well systemd deals with nested mounts (fstab entries are converted to systemd mount units AFAIK) but are `/apps` `/apps/var/` and `/apps/var/[custom|progress|standard]` all actual directories on the root file-system? Because when they are for instance symlinked to subdirectories in /apps_home (which gets mounted concurrently to the apps/* mounts) the mount point may not yet be available when the mount when is attempted and therefore fail. – HBruijn Nov 06 '18 at 12:40
  • @MikołajStempniewicz Ok, no problem. Thank you for clarifying my observation about mount dependencies. Your question is still attracting my attention though, because of how your system behaves. I am very curious about the root cause. – Anderson Medeiros Gomes Nov 06 '18 at 13:58
  • @MikołajStempniewicz, please could you check if `/etc/mtab` is a symlink that targets to `/proc/self/mounts`? [That is required in CentOS 7](https://access.redhat.com/solutions/2640361). – Anderson Medeiros Gomes Nov 06 '18 at 14:02
  • @AndersonM.Gomes - the /etc/mtab is not a symlink, it is just a regular file, i will schedule the next downtime of the problematic centos for this week. I will delete the mtab file, and instead i will create a symlink to /proc/self/mounts and then i will try rebooting (beforehand i will clean the /etc/rc.local that i am using to mount these partitions at the moment instead of the cronjob) - does it look like a sensible approach? – Mikołaj Stempniewicz Nov 06 '18 at 16:07
  • @HBruijn - thanks! I checked, and now i think i found the issue... i will try to explain the situation as clearly as i can: 1) sdd1 is mounted under /apps_home 2) sde1 is mounted under /apps/progress 3) sdf1 is mounted under /apps/standard 4) sdg1 is mounted under /apps/custom 5) the apps is a symlink pointing to... /app_home/apps so in the end there is a stupid, confusing simlink, and also there are in fact the mount dependencies, as the sde,sdf and sdg are mounted under directory that is also mounted, but in the fstab the mounpoint is containing a symlink. – Mikołaj Stempniewicz Nov 06 '18 at 16:17
  • I figured that I won't delete my initial response about the syslinking the mtab, as it may be still usefull. Plan for the next reboot: edit the fstab, so the mount paths won't be containing symbolic links anymore, and if that won't help i will try syslinking the mtab. I don't want to implement both of these fixes at the same time, but i will totally try both of them separately, and then both of them at the same time just for the sake of completeness. I hope it will resolve the issue. I will keep this thread as updated as i can. Thanks for your help, it looks promising! – Mikołaj Stempniewicz Nov 06 '18 at 16:21
  • @MikołajStempniewicz Your proposed plan looks sensible. However, I recommend that you modify your `/etc/fstab`and change mount paths that contains symbolic links in the same maintenance window. I believe that both actions together are the solution to your problem. – Anderson Medeiros Gomes Nov 07 '18 at 01:04
  • @AndersonM.Gomes so far so good. I symlinked mtab and edited the fstab, and the fsck started, because sde, sdf and sdg were not checked since 40+ reboots. I don't really have time to wait for fsck to finish, so i will check if everything was mounted properly tommorow morning. Anyway, i am positive that the issue is resolved, i will post final answer tommorow with summary what to avoid and what to check in similar situations. Thanks for help everobody, you are the best ;^) – Mikołaj Stempniewicz Nov 07 '18 at 17:18
  • 1
    Symlinking mtab and getting rid of symlinks from the fstab resolved the issue, as predicted. Once again, thank you everybody for your help in resolving this (in the end) really trivial issue :) cheers! – Mikołaj Stempniewicz Nov 08 '18 at 08:56

2 Answers2

2

In summary from the discussion in the comments:

Your problem was essentially that you used symlinks in your mount point paths and at boot the system was not able to follow those properly to recognise the result as "nested mounts". Therefor systemd did not mount your file-systems in a proper sequential order to handle that dependancy.

  • You have a mount point /apps_home

  • You have a symlink /apps --> /apps_home/apps

  • And you have also try to mount volumes on /apps/var/progress /apps/var/custom and /apps/var/custom

The problem is that the mount points /apps/var/[custom|progress|standard] don't exist until /apps_home is mounted.

Solution:

Leave the symlink but mount your file-systems on the actual directory paths of the symlink target: i.e. convert your fstab entries to: 

UUID=5717b613-a9f4-43c9-95d2-cfbbb891bd19   /apps_home                    ext4 defaults 1 2
UUID=e24df090-2dda-404c-8944-a28bd37d6c5e   /apps_home/apps/var/progress  ext4 defaults 1 2
UUID=5f254c77-a91d-4255-8315-9325ddb7a9d8   /apps_home/apps/var/standard  ext4 defaults 1 2
UUID=746c70c1-002a-4249-a06f-df393a99252c   /apps_home/apps/var/custom    ext4 defaults 1 2

systemd-fstab-generato will generate the needed mount unit files and systemd.mount will implicitly add the correct dependancies:

If a mount unit is beneath another mount unit in the file system hierarchy, both a requirement dependency and an ordering dependency between both units are created automatically.

Alternative: remove the entries from /etc/fstab and create your own mount unit files and manually configure the requirement and ordering dependancies to ensure that /apps/var/progress /apps/var/custom and /apps/var/custom don't get mounted before /apps_home.

HBruijn
  • 72,524
  • 21
  • 127
  • 192
2

mount always shows that the partitions are mounted on right catalogues, however umount reports that the partition is not mounted at the moment.

From the discussion in the comments, I believe that your system presents that behavior due to /etc/mtab being a regular file instead of a symbolic link that targets to /proc/self/mount. I suggest you to restore the symlink, as documented in this Red Hat solution:

In Red Hat Enterprise Linux 7, /etc/mtab is no longer a flat file, it is instead a symlink to /proc/self/mounts. If a service for some reason uses the sed command to access or modify /etc/mtab, it is possible that it will remove the symlink, and create a flat file. This in turns causes the server not to fully boot normally it drops to emergency mode, df output shows everything mounted, but if /proc/mounts is checked there will be nothing but / is mounted.

Anderson Medeiros Gomes
  • 2,252
  • 8
  • 20