poor performance with vmdk mounted as loop device even after conversion to raw

Question

Edit 13/12/12 - At this stage it would appear to be related to Western Digital (and possibly others) Advanced Format 4K block sizes.

This link goes some way into describing the issue, however the disk I have reports 512 logical and 4096 physical blocks (as opposed to 512/512).

http://johannes-bauer.com/linux/wdc/?menuid=3

I will continue testing and if I ever find a solution I'll be sure to post it.

I have a requirement to write large amounts of data to a vmdk disk located a USB3 disk attached to a Debian Linux server. I am creating partition mappings using kpartx and then mounting the created loop devices e.g.

The USB disk is /dev/sdb4 mounted at /mnt/backup.

> kpartx -av /mnt/backup/3MSYDDP01/3MSYDDP01_3-flat.vmdk
add map loop2p1 (254:3): 0 273042 linear /dev/loop2 63
add map loop2p2 (254:4): 0 16787925 linear /dev/loop2 273105
add map loop2p3 (254:5): 0 8401995 linear /dev/loop2 17061030
add map loop2p4 (254:6): 0 184219648 linear /dev/loop2 25495552
add map loop2p5 : 0 227328 linear 254:6 2048
add map loop2p6 : 0 15998976 linear 254:6 231424
add map loop2p7 : 0 3059712 linear 254:6 16232448
add map loop2p8 : 0 2942976 linear 254:6 19294208
add map loop2p9 : 0 161978368 linear 254:6 22239232

Then I mount one of the partitions.

> mkdir /mnt/loop2p2
> mount /dev/mapper/loop2p2 /mnt/loop2p2

And run a test writing data to the mount point, aka the second partition in the vmdk.

> dd if=/dev/zero of=/mnt/loop2p2/zerofile.tst bs=1k count=1700000
1700000+0 records in
1700000+0 records out
1740800000 bytes (1.7 GB) copied, 135.871 s, 12.8 MB/s

A test to the USB3 disk itself runs well.

> dd if=/dev/zero of=/mnt/backup/zerofile.tst bs=1k count=1700000
1700000+0 records in
1700000+0 records out
1740800000 bytes (1.7 GB) copied, 19.5485 s, 89.1 MB/s

As a comparison I created a raw image file and ran the same test.

> qemu-img create -f raw /mnt/backup/raw.img 4G

After creating an ext3 partion on raw.img I mounted it and ran the test to it.

> kpartx -av /mnt/backup/raw.img
add map loop2p1 (254:3): 0 8386560 linear /dev/loop2 2048
> mkdir /mnt/loop2p1
> mount /dev/mapper/loop2p1 /mnt/loop2p1
> dd if=/dev/zero of=/mnt/loop2p1/zerofile.tst bs=1k count=1700000
1700000+0 records in
1700000+0 records out
1740800000 bytes (1.7 GB) copied, 23.3126 s, 74.7 MB/s

Good result! slightly slower than writing direct to the disk but that's to be expected.

So then I decided t convert the vmdk to a raw image using qemu-img as another comparison.

> qemu-img convert /mnt/backup/3MSYDDP01/3MSYDDP01_3-flat.vmdk -O raw /mnt/backup/3MSYDDP01/3MSYDDP01_3.raw

> kpartx -av /mnt/backup/3MSYDDP01/3MSYDDP01_3.raw
add map loop2p1 (254:3): 0 273042 linear /dev/loop2 63
add map loop2p2 (254:4): 0 16787925 linear /dev/loop2 273105
add map loop2p3 (254:5): 0 8401995 linear /dev/loop2 17061030
add map loop2p4 (254:6): 0 184219648 linear /dev/loop2 25495552
add map loop2p5 : 0 227328 linear 254:6 2048
add map loop2p6 : 0 15998976 linear 254:6 231424
add map loop2p7 : 0 3059712 linear 254:6 16232448
add map loop2p8 : 0 2942976 linear 254:6 19294208
add map loop2p9 : 0 161978368 linear 254:6 22239232
> mount /dev/mapper/loop2p2 /mnt/loop2p2
dd if=/dev/zero of=/mnt/loop2p2/zerofile.tst bs=1k count=1700000
1700000+0 records in
1700000+0 records out
1740800000 bytes (1.7 GB) copied, 129.653 s, 13.4 MB/s

Poor performance with the converted disk as well.

The fdisk of each disk as follows:

> fdisk -l /dev/sdb
Note: sector size is 4096 (not 512)

Disk /dev/sdb: 3000.6 GB, 3000558944256 bytes
255 heads, 63 sectors/track, 45599 cylinders, total 732558336 sectors
Units = sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x000246c6

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1             256      131327      524288   83  Linux
/dev/sdb2          131328     2228479     8388608   83  Linux
/dev/sdb3         2228480     2490623     1048576   82  Linux swap / Solaris
/dev/sdb4         2490624   732558335  2920270848   83  Linux

raw.img

> fdisk -l /mnt/backup/raw.img

Disk /mnt/backup/raw.img: 4294 MB, 4294967296 bytes
43 heads, 32 sectors/track, 6096 cylinders, total 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xd48075ee

              Device Boot      Start         End      Blocks   Id  System
/mnt/backup/raw.img1            2048     8388607     4193280   83  Linux

3MSYDDP01_3-flat.vmdk

> fdisk -l /mnt/backup/3MSYDDP01/3MSYDDP01_3-flat.vmdk

Disk /mnt/backup/3MSYDDP01/3MSYDDP01_3-flat.vmdk: 107.4 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders, total 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0005cbed

                                      Device Boot      Start         End      Blocks   Id  System
/mnt/backup/3MSYDDP01/3MSYDDP01_3-flat.vmdk1   *          63      273104      136521   83  Linux
/mnt/backup/3MSYDDP01/3MSYDDP01_3-flat.vmdk2          273105    17061029     8393962+  83  Linux
/mnt/backup/3MSYDDP01/3MSYDDP01_3-flat.vmdk3        17061030    25463024     4200997+  82  Linux swap / Solaris
/mnt/backup/3MSYDDP01/3MSYDDP01_3-flat.vmdk4        25495552   209715199    92109824    5  Extended
/mnt/backup/3MSYDDP01/3MSYDDP01_3-flat.vmdk5        25497600    25724927      113664   83  Linux
/mnt/backup/3MSYDDP01/3MSYDDP01_3-flat.vmdk6        25726976    41725951     7999488   83  Linux
/mnt/backup/3MSYDDP01/3MSYDDP01_3-flat.vmdk7        41728000    44787711     1529856   83  Linux
/mnt/backup/3MSYDDP01/3MSYDDP01_3-flat.vmdk8        44789760    47732735     1471488   83  Linux
/mnt/backup/3MSYDDP01/3MSYDDP01_3-flat.vmdk9        47734784   209713151    80989184   83  Linux

Edit: The Block size of the partitions in the vmdk are:

> dumpe2fs /dev/mapper/loop2p1 | grep -i 'block size'
dumpe2fs 1.42.5 (29-Jul-2012)
Block size:               1024
> dumpe2fs /dev/mapper/loop2p2 | grep -i 'block size'
dumpe2fs 1.42.5 (29-Jul-2012)
Block size:               4096
> dumpe2fs /dev/mapper/loop2p5 | grep -i 'block size'
dumpe2fs 1.42.5 (29-Jul-2012)
Block size:               1024
> dumpe2fs /dev/mapper/loop2p6 | grep -i 'block size'
dumpe2fs 1.42.5 (29-Jul-2012)
Block size:               4096
> dumpe2fs /dev/mapper/loop2p7 | grep -i 'block size'
dumpe2fs 1.42.5 (29-Jul-2012)
Block size:               4096
> dumpe2fs /dev/mapper/loop2p8 | grep -i 'block size'
dumpe2fs 1.42.5 (29-Jul-2012)
Block size:               4096
> dumpe2fs /dev/mapper/loop2p9 | grep -i 'block size'
dumpe2fs 1.42.5 (29-Jul-2012)
Block size:               4096

/dev/mapper/loop2p3 is swap.

/dev/mapper/loop2p4 is an extended partition.

Could the sector size of the USB disk have an impact? If so why would the raw image be fine? I will look at testing with the USB disk having a 512 sector size but that will take time as I have to copy the data off, format and copy back.

What block size for the VMDK? I wonder if these are aligned properly? — SpacemanSpiff, Dec 12 '12 at 01:08
@SpacemanSpiff I have added the block sizes to the question. — jelloir, Dec 12 '12 at 01:25
@Joel E Salas We will be using using rsync to syncronise a live VM to a copy stored on a the backup disk for offsite backups. — jelloir, Dec 12 '12 at 01:54
And the recommended way to efficiently backup a 1.6TB VM every night to offsite USB storage, keeping it in a vmdk format is? — jelloir, Dec 12 '12 at 22:02
Or [VMWare vSphere Replication](http://www.vmware.com/products/datacenter-virtualization/vsphere/replication) or [VMWare vSphere Data Protection](http://www.vmware.com/products/datacenter-virtualization/vsphere/data-protection.html). — ewwhite, Dec 14 '12 at 01:15

score 0 · Answer 1 · answered Dec 14 '12 at 00:06

0

Don't back up the entire VM. Depending on the application, you should be able to do some application-level replication. Also just filling a VMDK file via rsync is not the same as backing up the VM. There are intangibles like locked files in use, databases (if applicable) and configuration files that may need to be copied over separately.

Overall this is a poor idea and will probably not provide the recovery plan your employers are expecting. Also, external USB drives do not belong in production.

answered Dec 14 '12 at 00:06

Joel E Salas

5,562
15
25

This does not answer the problem I have with poor performance but for the sake of clarity... The VM on the Backup disk is a clone of the running VM taken when it was powered off. I am stopping any processes on the live VM that would cause inconsistency; mounting the cloned VM's vmdk's and then syncing the differences from the live VM, so in effect this IS an application level backup. Curious why you think USB drives do not belong in production? – jelloir Dec 14 '12 at 06:54

poor performance with vmdk mounted as loop device even after conversion to raw

1 Answers1