1

I have an ESXi 5.1 server (free version) with two 500G drives mounted and assigned to one datastore. I am not able to delete or move certain files in the datastore. I don't have vMotion.

This is not related to: VMWare ESX 3.5 Input/Output Error

Neither is it related to this VMWare KB

The output may be similar but the circumstances are different.

I've tried expanding the storage by adding another hard drive, so there are currently 3 500G drives assigned to the datastore, and there is a lot of space available to it.

The following is a listing of the datastore folder

-rw-r--r--    1 root     root          27 Feb  5  2015 DC-9f914f3e.hlog
-rw-------    1 root     root       16.0G Aug  5 15:20 DC-9f914f3e.vswp
-rw-------    1 root     root      995.6G Aug  5 16:01 DC-flat.vmdk
-rwxr-xr-x    1 root     root        3.5K Aug  3 05:36 DC.bakvmx
-rw-------    1 root     root        8.5K Aug  3 05:37 DC.nvram
-rw-------    1 root     root         505 Aug  3 05:36 DC.vmdk
-rw-r--r--    1 root     root           0 Jan 27  2014 DC.vmsd
-rwxr-xr-x    1 root     root        3.6K Aug  3 05:36 DC.vmx
-rw-------    1 root     root           0 Aug  3 05:36 DC.vmx.backup
-rw-r--r--    1 root     root         269 Aug  3 05:36 DC.vmxf
-rw-r--r--    1 root     root           0 Aug  6 08:19 DC2.vmsd
-rw-r--r--    1 root     root        3.5K Aug  6 12:47 DC2.vmx
-rw-r--r--    1 root     root         270 Aug  7 10:51 DC2.vmxf
-rw-------    1 root     root      500.0G Aug  6 12:51 DC_1-flat.vmdk
-rw-------    1 root     root         506 Aug  6 10:50 DC_1.vmdk
-rw-------    1 root     root        3.7M Jun 23 08:20 vmmcores-5.gz
-rw-------    1 root     root        2.9M Jun 23 08:41 vmmcores-6.gz
-rw-------    1 root     root        2.9M Jun 23 08:50 vmmcores-7.gz
-rw-------    1 root     root        2.3M Jun 23 08:56 vmmcores-8.gz
-rw-------    1 root     root        3.0M Jun 23 10:44 vmmcores-9.gz
-rw-r--r--    1 root     root      214.1K Aug  4 08:55 vmware-106.log
-rw-------    1 root     root       94.0M Aug  3 05:36 vmx-DC-2677100350-1.vswp
-rw-------    1 root     root       91.0M Oct 30  2014 vmx-DC-2677100350-2.vswp

I'm attempting to delete DC_1-flat.vmdk, or the DC.vmx.backup file which has a zero size. It cannot be deleted. I get the following output:

 rm: can't remove 'DC_1-flat.vmdk': Input/output error

The other files can be moved or deleted, but not these files.

I've tried everything, and I'm not sure what to do. The main vmdk has the same problem. The VM is not starting up because a similar Input/Output error is produced.

I've run the following check and the following output was produced:

# vmkfstools -v 0 -t 0 DC.vmdk
FileIOErrno2Result: Unexpected errno=5, Input/output error
OBJLIB-FILEBE : FileBEOpen: can't open './DC-flat.vmdk' : Input/output error (327683).
DISKLIB-VMFS  : "./DC-flat.vmdk" : failed to open (Input/output error): ObjLib_Open failed. Type 3
DISKLIB-LINK  : "DC.vmdk" : failed to open (Input/output error).
DISKLIB-CHAIN : "DC.vmdk" : failed to open (Input/output error).
DISKLIB-LIB   : Failed to open 'DC.vmdk' with flags 0xe Input/output error (327689).
Failed to open virtual disk: Input/output error 327689
AIOMGR-S : stat o=1 r=3 w=0 i=0 br=49152 bw=0
AIOMGR-U : stat o=1 r=0 w=0 i=0 br=0 bw=0
OBJLIB-LIB : ObjLib cleanup done.

I've checked the hard drive for errors using the following article

And the S.M.A.R.T. health check on the hard drives seem to check out okay.

Community
  • 1
Xenno
  • 21
  • 1
  • 1
  • 4
  • 2
    Sounds like bad disks. Can you verify that all disks are checking out okay? Do you monitor with anything? – TryTryAgain Aug 07 '15 at 17:38
  • What do you mean `with two 500G drives mounted in one datastore`? – GregL Aug 07 '15 at 18:41
  • I don't monitor with anything really. What could I use to check the disks? I tried some of the vmware disk utilities but they are apparently not necessary since vmware does an automatic health check. They checked out ok, but I do believe that it could be a disk fault. I just need to know how to identify it. – Xenno Aug 07 '15 at 19:49
  • Two 500g hard drives assigned to a single datatype, making 1TB available to the vm's – Xenno Aug 09 '15 at 07:54
  • I found the problem. One of the hard drives has a hardware fault. I scanned it using the manufacturer's disk utility and found a few bad sectors on the hard drive. I need to attempt to repair those sectors. I guess loss of data would be inevitable. – Xenno Aug 11 '15 at 10:46
  • `Two 500g hard drives assigned to a single datatype, making 1TB available to the vm's`.. This is a bad idea. The failure of the first extent will render the whole Datastore useless, not to mention that it's not recommended by [VMware](http://blogs.vmware.com/vsphere/2012/02/vmfs-extents-are-they-bad-or-simply-misunderstood.html) – GregL Aug 11 '15 at 11:55
  • Agreed, but in our case, we are only using one server with one VM on it, and this is a risk that we are prepared to take and mitigate. – Xenno Aug 19 '15 at 07:11

1 Answers1

1

TryTryAgain answered the question in a comment. The problem was the bad disks. One of the drives had a few bad sectors that could not be repaired.

We took the drive a data recovery centre and they cloned the drive to a new hard drive with the data as is. After installing it back into the server, and re-installing ESXi, the datastore was back online and the VM was restored. Thankfully, no data loss experienced.

Xenno
  • 21
  • 1
  • 1
  • 4
  • You should configure in RAID as this would eliminate that problem, rather than having them all striped. Remember: just because smart tool says PASSED, that merely means hat it was successful in checking the smart data and not that the drive is fine. You should pay particular attention to offline uncorrectable being anything other than 0, as well as bad sectors. – Timothy Frew Apr 28 '18 at 23:26