tar error to a Drobo FS: Cannot close: Input/output error, Exiting with failure status due to previous errors

Question

We've been running this tar backup from a Fedora 25 workstation to a Drobo FS for a couple of years and in the past few weeks it's starting to error out with the following but does seem to complete:

/bin/tar: /mypath/etc-new.tgz: Cannot close:
Input/output error
/bin/tar: Exiting with failure status due to previous errors

STANDARD OUTPUT:
Failed to rename /mypath/home-new.tgz to
/mypath/home.tgz: File exists

Backup of /etc FAILED

Backed up /etc to /mypath/etc-FAILED.tgz
Backed up /root to /mypath/root.tgz
Backed up /usr/local to /mypath/usr-local.tgz
Backed up /var/lib to /mypath/var-lib.tgz
Backed up /var/log to /mypath/var-log.tgz
Backed up /var/www to /mypath/var-www.tgz
Backed up /var/yp to /mypath/var-yp.tgz
Second try renamed /mypath/home-new.tgz to
/mypath/home.tgz

Here's the full tar command in action:

/usr/local/sbin/drobo-backup -n -v -c /etc/drobo-backup.conf
\Reading configfile /etc/drobo-backup.conf
/bin/mkdir /mypath
ionice -c2 -n7 nice -n19 /bin/tar -cf /mypath/home-new.tgz -C / --atime-preserve --one-file-system --warning=no-file-ignored --warning=no-file-changed --warning=no-file-removed home --exclude=.gvfs --exclude='Windows 7.vdi'
/bin/mkdir /mypath
ionice -c2 -n7 nice -n19 /bin/tar -cf /mypath/etc-new.tgz -C / --atime-preserve --one-file-system --warning=no-file-ignored --warning=no-file-changed --warning=no-file-removed etc
/bin/mkdir /mypath
ionice -c2 -n7 nice -n19 /bin/tar -cf /mypath/root-new.tgz -C / --atime-preserve --one-file-system --warning=no-file-ignored --warning=no-file-changed --warning=no-file-removed root
/bin/mkdir /mypath
ionice -c2 -n7 nice -n19 /bin/tar -cf /mypath/usr-local-new.tgz -C / --atime-preserve --one-file-system --warning=no-file-ignored --warning=no-file-changed --warning=no-file-removed usr/local
/bin/mkdir /mypath
ionice -c2 -n7 nice -n19 /bin/tar -cf /mypath/var-lib-new.tgz -C / --atime-preserve --one-file-system --warning=no-file-ignored --warning=no-file-changed --warning=no-file-removed var/lib --exclude=/var/lib/yum --exclude=/var/lib/rpm
/bin/mkdir /mypath
ionice -c2 -n7 nice -n19 /bin/tar -cf /mypath/var-log-new.tgz -C / --atime-preserve --one-file-system --warning=no-file-ignored --warning=no-file-changed --warning=no-file-removed var/log
/bin/mkdir /mypath
ionice -c2 -n7 nice -n19 /bin/tar -cf /mypath/var-www-new.tgz -C / --atime-preserve --one-file-system --warning=no-file-ignored --warning=no-file-changed --warning=no-file-removed var/www
/bin/mkdir /mypath
ionice -c2 -n7 nice -n19 /bin/tar -cf /mypath/var-yp-new.tgz -C / --atime-preserve --one-file-system --warning=no-file-ignored --warning=no-file-changed --warning=no-file-removed var/yp

The only logs that I can see that might be related are these which are about 30 minutes before the error report arrives by email:

Mar  3 01:08:31 myworkstation kernel: CIFS VFS: Server drobo has not responded in 120 seconds. Reconnecting...
Mar  3 01:09:29 myworkstation kernel: CIFS VFS: sends on sock ffff99407356fc00 stuck for 15 seconds
Mar  3 01:09:29 myworkstation kernel: CIFS VFS: Error -11 sending data on socket to server
Mar  3 01:09:44 myworkstation kernel: CIFS VFS: sends on sock ffff99407356fc00 stuck for 15 seconds
Mar  3 01:09:44 myworkstation kernel: CIFS VFS: Error -11 sending data on socket to server
Mar  3 01:10:25 myworkstation kernel: CIFS VFS: sends on sock ffff99407356fc00 stuck for 15 seconds
Mar  3 01:10:25 myworkstation kernel: CIFS VFS: Error -11 sending data on socket to server
Mar  3 01:12:09 myworkstation kernel: CIFS VFS: sends on sock ffff994089f78a00 stuck for 15 seconds
Mar  3 01:12:09 myworkstation kernel: CIFS VFS: Error -11 sending data on socket to server

A maintainer of GNU tar says this:

The 'close()' syscall returned non-zero value and errno was set to EIO. That sounds like some underlying issue with your hardware.

But the self tests on the Drobo FS do not indicate any errors or warnings. The tars do complete but perhaps there's something with the command or "position-sensitive" syntax? The error always happens on /etc/. We have a few other workstations doing these exact same backups to the same Drobo FS and no errors.

Answer 1

I believe I found the problem. It seems Tracker was the culprit. Here are some clues from logs:

ouruser journal: *** Unsupported operation detected on trash directory
ouruser journal:  dir: /home/users/username/.local/share/Trash/files, file: files, type: 4#012
ouruser journal: unable to create file '/run/user/1234/dconf/user': Permission denied.  dconf will not work properly.
ouruser journal: kde-recent-document-provider.vala:174: Couldn't find actor for 'kpdf'.
ouruser journal: Could not create FTS delete statement: table fts5 has no column named nco:hobby
ouruser journal: unable to create file '/run/user/1234/dconf/user': Permission denied.  dconf will not work properly.
ouruser journal: Could not create FTS delete statement: table fts5 has no column named nco:hobby
ouruser kernel: CIFS VFS: Server drobo-rh has not responded in 120 seconds. Reconnecting...
ouruser kernel: CIFS VFS: sends on sock ffff8d96e7af4000 stuck for 15 seconds
ouruser kernel: CIFS VFS: Error -11 sending data on socket to server

And from the cron job running the tar backup:

ERROR OUTPUT:
/bin/tar: /ourdomain/etc-new.tgz: Cannot close:
Input/output error
/bin/tar: Exiting with failure status due to previous errors

STANDARD OUTPUT:
Failed to rename /drobo/ourdomain/home-new.tgz to
/ourdomain/home.tgz: File exists

Backup of /etc FAILED

So /home is on its own partition:

/dev/mapper/vg_ouruser-LogVol02 /home                   ext4    defaults        1 2

I believe Tracker ends up running during the backup, and was trying to index the Trash. When the backup finishes /home and continues to /etc that's when the tar errors appear. I deleted the Trash of this user and then decided to remove the Tracker RPM. The error has disappeared (hopefully permanently!).