1

I was in the middle of an amanda backup, and after about 60GB, it exited with this error:

  localhost /home lev 0  FAILED [data write: Connection reset by peer]
  localhost /home lev 0  partial taper: No space left on device: No space left on device
  localhost /home lev 0  FAILED [data write: Connection reset by peer]
  localhost /home lev 0  partial taper: No space left on device: No space left on device

But it doesn't tell me which device. And I can't find any device that is full.

My backup ended after 60GB

-rw------- 1 amanda backup 36616372224 2016-11-02 23:42 00001.localhost._home.0
-rw------- 1 amanda backup 22800531456 2016-11-03 00:03 00002.localhost._home.0

My amanda.conf says

tapetype "HARD-DISK"
define tapetype HARD-DISK {
    comment "Dump onto hard disk"
    length 150 gbytes
}

My backup drive has space

Filesystem            Size  Used Avail Use% Mounted on
/dev/sdc1             2.7T  2.3T  277G  90% /storage

The disk being backed up has space

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/HermanGroup-Home
                       74G   57G   14G  81% /home

But now everything I try to do with amanda fails:

FAILURE DUMP SUMMARY:
  newlaptop.local.net /home lev 9  FAILED [too many taper retries]
  newlaptop.local.net /home lev 9  partial taper: No space left on device: No space left on device
  newlaptop.local.net /home lev 9  partial taper: No space left on device: No space left on device

How do I find out what happened?

UPDATE:

(in response to suggestion)

$ df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/root             262944   25713  237231   10% /
/dev/mapper/HermanGroup-Usr
                      655360  282687  372673   44% /usr
/dev/mapper/HermanGroup-Tmp
                      262144    2595  259549    1% /tmp
tmpfs                 219740       1  219739    1% /dev/shm
/dev/sdc1            183148544   28045 183120499    1% /storage
/dev/mapper/HermanGroup-Herman
                     3276800  298420 2978380   10% /herman
/dev/mapper/HermanGroup-Mysql
                      131072     191  130881    1% /var/lib/mysql
/dev/mapper/HermanGroup-UsrX11
                      393216   68534  324682   18% /usr/X11
/dev/mapper/HermanGroup-Home
                     4915200 1148476 3766724   24% /home
/dev/mapper/HermanGroup-WinGCC
                       65536    3536   62000    6% /usr/local/wingcc
/dev/mapper/HermanGroup-AmandaHolding
                      720896      11  720885    1% /storage/amanda/holding
192.168.42.10:/var/spool/mail
                      655360      17  655343    1% /var/spool/mail
hymie
  • 424
  • 2
  • 11

4 Answers4

0

I'm guessing your filesystem is running out of inodes. Try out df -i.

Janne Pikkarainen
  • 31,454
  • 4
  • 56
  • 78
  • I appreciate the suggestion, but no. 39992 inodes used, 183108552 inodes free. – hymie Nov 03 '16 at 10:44
  • Down vote for a bad answers or ones that are totally inaccurate not one that is potentially correct but wasn't in this case. – hookenz Nov 03 '16 at 22:33
0

It sounds like that your /home partition is being used while dumping? Then perhaps moved to your storage when finished. The message localhost /home lev 0 partial taper: No space left on device: No space left on device seem quite clear to me, that /home is being used somehow.

SteffenNielsen
  • 467
  • 4
  • 15
  • I don't think it's clear at all. The "partial taper: " part of the error message implies that "taper" is reporting the error -- on multiple machines (newlaptop and localhost). And no offense, but that doesn't make sense. What kind of backup system uses the area it's trying to back up to perform a backup of that same area? – hymie Nov 03 '16 at 11:12
  • @hymie i can't tell. But try keep an eye on your /home partition while performing the backup? If its growing it might be used for temporary files. – SteffenNielsen Nov 03 '16 at 12:57
0

You seem to be using LVM for your volumes on your machines to be backed up. Are these home directories highly transactional? I ask because it is typical to create an LVM snapshot when backing up. This is the case in most backup applications that can utilize LVM on the target machine for backup consistency. Amanda backup is one such product. That snapshot may be running out of space, if it's being created at all. Could you provide the output of vgdisplay and lvdisplay for all affected volume groups and logical volumes?

If it is indeed a snapshot causing this problem, your solution would be to add space to your volume group to compensate for the write transactions taking place during the backup process. This problem likely has nothing to do with your filesystem occupancy.

Spooler
  • 7,016
  • 16
  • 29
  • I am using LVM, but amanda is not (as far as I know) an LVM-aware backup program that makes snapshots and backs them up. The backup software, afaik, just sees a file system and uses tar to copy all of the files off of it. – hymie Nov 03 '16 at 22:26
  • You might want to make certain of that in your case. I've used Amanda for LVM based workloads in the past, and it's done a good job of backing up LAMP stacks *because* of its LVM snapshot features. If it doesn't find any room within a volume group to make a snapshot, it falls back on hashing diffs between modified files in a "chasing a running target" approach. Either is subject to running out of work space to store some kind of diff table to ensure workload consistency. Since these backups are failing about half way through with no space problems on filesystems, this problem is likely. – Spooler Nov 04 '16 at 00:13
  • Check LVM as it's making a backup, not after. Amanda should clean up snapshots after it's done or after it fails. – Spooler Nov 04 '16 at 00:14
0

I don't know if this is "the answer" or just a coincidence, but...

  • the size of a virtual tape is defined as 150GB
  • there are two virtual tape drives
  • the free space left on the drive when I got the error was just under 300GB, which is 2 x 150GB
  • erasing some old backups seems to have resolved the issue (at least for now).

So perhaps it's a bug (or feature) of amanda that it always expects to find enough free space to have one virtual tape loaded in each virtual tape drive.

hymie
  • 424
  • 2
  • 11