aligning truecrypt partition on 1.5TB 4kB sector drive

Question

Aligning partitions to start at real physical sector of ssds / stripped raids / 4kB drives is a 'good thing to do', but I've run into a problems when trying to do it for a truecrypt partition that will contain ext3 on it. Or so it seems.

When drive in question is partitioned properly and formatted with ext3 I get very reasonable write speeds around 70-80MB/s, but when I put truecrypt and ext3 on the top of it write performance becomes very unstable and goes between 1-25MB/s with very high io-wait. On the same server I don't have any performance issues with ext3 on the top of truecrypt on regular 512B-sector 500GB sata disks. So my best guess is that iowaits are caused by misalignment but I cannot really find reliable information on how to calculate optimal partition beginning. I've tried to start it at 128 logical sector, I've also tried 8132 sector as suggested here but both gave me very bad and unstable performance.

Do you have any experience with similar setup? Thanks!

ps - quote from truecrypt forum: When I encrypted the partition with Truecrypt, I only got 8Mbyte/sec because it does not place the start of the volume at sector 8192, but instead it places the volume at the end of the track which 8192 belongs to. I have 63 sectors per track, so sector 8192 is the second sector of the 130th track. Truecrypt started its volume at the end of this track (sector number 8252), which is 60 sectors too far. So the solution was to move the partition back by 60 sectors, so the partition started at 8132 instead of 8192. This caused the first sector of the Truecrypt volume to be located at the magic sector 8192.

Answer 1

after some googling and a few own tests i concluded it's faulty drive. others have similar problem too with WD15EADS disks. prolonged test on properly aligned ext3 partition also showed performance degradation.

i've run in loop:

dd if=/dev/zero of=/mnt/1_5tb0/out bs=1MB count=45000

and got deteriorating performance:

45000000000 bytes (45 GB) copied, 652.667 s, 68.9 MB/s
45000000000 bytes (45 GB) copied, 648.647 s, 69.4 MB/s
45000000000 bytes (45 GB) copied, 645.147 s, 69.8 MB/s
45000000000 bytes (45 GB) copied, 655.122 s, 68.7 MB/s
45000000000 bytes (45 GB) copied, 644.662 s, 69.8 MB/s
45000000000 bytes (45 GB) copied, 645.12 s, 69.8 MB/s
45000000000 bytes (45 GB) copied, 648.025 s, 69.4 MB/s
45000000000 bytes (45 GB) copied, 650.528 s, 69.2 MB/s
45000000000 bytes (45 GB) copied, 1247.87 s, 36.1 MB/s
45000000000 bytes (45 GB) copied, 1601.76 s, 28.1 MB/s
45000000000 bytes (45 GB) copied, 1776.75 s, 25.3 MB/s

at some moment system almost halts with very high iowait + very little disk activity in iostat [eg 2-10 iops/sec ].

ps.: previous 1.5TB disk was removed, new one was installed - all problems are gone. so it was hardware issue.

Answer 2

Does TrueCrypt add any header blocks before the rest of the partition? If it does then it could effectively be throwing your nicely aligned partitions back out of whack. You could try deliberately misalign the partition by each possible amount (this will mean up to 7 tests, for the different ways a 4k block can be off by a multiple of 0.5k) and repeating your tests. If the issue is header information pushing the main body of data away from being offset then on of these tests should show better results than the others. This assumes that any header info (or other reason for offset) being added is 512 bytes or a multiple there-of (which would be sensible, and seems to be the case as you see no similar performance degradation on drives with 512 byte sectors).