20

In mid-November, a VPS that I am renting from a hosting company stopped responding. When I contacted support, they explained that a power outage in the datacenter caused a forced reboot and fsck. Eventually, I asked why it was taking so long, and was told that the size of the volume is 30 TB. The last time I received an update was in February, and they have not responded to my most recent inquiry.

I understand that fsck can be very slow for some file systems, but is it possible for fsck to take 6 months on a 30 TB volume, or should I assume that this hosting company is lying to me so that I continue to pay my bill every month?

Brian Bi
  • 347
  • 2
  • 9
  • 39
    They were probably lying to you from the start. I would expect that to take _hours_. You should have stopped paying in December. – Michael Hampton May 07 '19 at 19:05
  • 15
    Even if they're not lying, choosing a HW + software setup that *could* require an FSCK that long shows they're incompetent. And whatever the reason, they're not providing the service you're paying for. – Peter Cordes May 08 '19 at 09:45
  • 35
    Sounds like a real cluster fsck! – JMK May 08 '19 at 11:40
  • 2
    @JMK Now I wish there was a way to flag comments for extra merit, maybe add to a hall-of-fame. – pipe May 08 '19 at 11:49
  • 2
    What @PeterCordes says is the key point. You're paying for a service. You're really sorry to hear that they're having problems but you're calling about the service that you're paying for and not getting. – Rob Moir May 08 '19 at 14:19
  • There is also the possibility that they are trying to cover up an issue caused by a human mistake and they just don't want their customers to know about. They perhaps think that customers are not smart enough to know about fsck. – lepe May 09 '19 at 05:29
  • @lepe which is why I suggest focusing on the lack of service. You're sorry to hear about their claimed "why" but (true or not) it isn't your problem. The lack of service is and you'd love to hear how they plan to fix that.... so to speak. – Rob Moir May 09 '19 at 07:48
  • When a company stops providing service and doesn't respond to you, you can usually get there attention by stopping payments to them. By that stage you've taken your business elsewhere. – hookenz May 09 '19 at 20:51

3 Answers3

33

fsck speed mainly depends on the number of files and how they are spread in the respective directory. That said, 6 month for a fsck is absolutely absurd: it should had completed in some hours at most, especially if using xfs which has the speedy xfs_repair utility. Here you can find some fsck run at a scale - all completed under one hour (3600s). So, it is not possible that your fsck is still running.

Anyway, an unexpected power loss will not cause a full-blow fsck, rather only a very fast (some seconds) journal replay. However, if some key files was damaged, the OS can be unbootable.

But they probably just lied to you. You should stop paying immediately, ask for an explanation and apply for a total refund.

shodanshok
  • 44,038
  • 6
  • 98
  • 162
  • 8
    If they're using `ext2`, then a power failure will require a full `fsck`, and I wouldn't be surprised if it takes days on a heavily-used 30TB volume. On the other hand, if they're using `ext2` on a 30TB volume, that in and of itself is a reason to look elsewhere for hosting services. – Mark May 07 '19 at 21:54
  • 14
    ext2 uses 32-bit block counter, with a maximum block size of 4096 bytes (ie: a page) on x86 and x86_64. This means ext2 (and ext3) are limited to 8TB volumes so no, the OP can not be using ext2/3. Anyway, using any non-journaled filesystem on a 30 TB volume would be absolutely *insane*. – shodanshok May 07 '19 at 22:05
  • I think ext4 fsck might be little better if one has a 30Tb FS containing vast numbers of tiny files. Lunacy to create that, so still a reason to look elsewhere. – nigel222 May 09 '19 at 10:51
8

Conjecture: Their system uses a BBU/FBWC-less RAID (or even software RAID) with all possible write caches (including these in the hard drives themselves) set at their most aggressive settings, in order to get maximum performance for minimal cost. A hard power outage on such a setup can leave a journaling filesystem in a condition where the journal cannot be trusted and cannot be used for recovery. The problem is that such a system aggressively reorders and postpones writes, which means that a journal entry can be written with the effect of the data action being lost ... or the journal entry being lost on a data action that was consequential.

Recovering such a system from a worst case outage can mean that you have to do a "slow" fsck/repair that actually examines all the filesystem structures as they are, which could indeed take a day or two for 30TB.... and it is not unlikely that you will have to run multiple repair cycles. Add to that that personnel might not be always available to monitor this, you could easily be down to one fsck being done per week. They probably gave up and forgot.

rackandboneman
  • 2,487
  • 10
  • 8
1

For most filesystems it will be much faster, even when there are errors, as normally only the metadata is checked.

In the worst case, it may read the whole disk, (e.g. something like fsck.ext4 -cc /dev/sda, which does a non-destructive write test on every block), that could take a few days for 30 TB. If you know the speed of the drives, you can calculate size/speed. For a consumer hard drive with about 100 MB/s copying a few TB can take more hours than most people would expect.

If it were your server, you could have the problem that it boots then hangs when fsck asks you if you want to fix an error. But the datacenter admin won't leave a fsck hanging for 6 month while all VPS are offline.

So they are either lying to you, or there is a huge misunderstanding. Or they were running fsck some time ago and did not update you about the new problem after it finished.

agc
  • 111
  • 4
allo
  • 1,524
  • 1
  • 19
  • 35
  • 4
    `fsck` traverses all the filesystem structures, which mostly means executing random i/o. So the above calculation, based on *sequential* transfer rate, is not very useful. – shodanshok May 08 '19 at 12:40
  • @shodanshok indeed file structure is irrelevant in a general drive check, as I've just explained in my answer. – Overmind May 08 '19 at 12:55
  • @shodanshok my worst-case assumption was based on a very extensive fsck. For example the typical xfs fsck doesn't do much. ext2 has a long running extensive check and old MS-DOS scandisk had a read-write test on each hard drive block when running it in full mode. So you have an upper bound at the size of the disk. – allo May 08 '19 at 13:20
  • 1
    @Overmind And you answer is irrelevant to the question which is about fsck and not a general drive check. – BlackJack May 08 '19 at 13:54
  • Please be aware that taking typical disk throughput as an indicator can be misleading. I've done the math when once re-syncing an array, which should (in my opinion) have taken less than a day, and it took over two weeks! Seeks are the one dominating factor for total time and even when you **think** that you're doing a strictly sequential operation, it sometimes isn't one. Now fsck is strictly non-sequential, so... no way can you judge from usual disk throughput to the length of the operation (still, _months_ is ridiculous... it's an obvious lie). – Damon May 09 '19 at 08:57