2

First a quick overview of the environment:

  • NetBackup running on Windows Servers (6.5.4 if you care) with LTO3 drives.

  • The backup target used to be a Solaris 9 server, on Sun hardware, with Veritas Volume Manger.

  • Rebuilt as RHEL5 box using LVM to manage the volumes, now on a Xiotech SAN. With a large number of volumes.

The nature of the data and the application that the box runs (Optix) is such that it used to write to a volume until it reached a certain size and then that volume was locked forever more. Hence we have \u01 \u02 \u03 ... \u50. A while back (still on the solaris build) we expanded and opened those volumes back up for writing so in any given day any or all of them might change. Backup throughput used to average 40MB/sec.

In the new Linux build we're averaging something closer to 8MB/sec. Given that here is 2.1TB of data here that's sort of wildly unacceptable, even running 4 streams it is taking 48+ hours to complete. I/O on the server is pegged. I am pretty sure it's not the SAN because other clients using the same class of storage and similar server hardware are backing up at a pokey but tolerable 20MB/sec.

I'm looking for ideas on improving throughput. The Solaris guys in the office next door are blaming LVM on Linux. Nobody thinks it's the backup environment, because that's still performing as expected everywhere else. The admin of the now very slow box says "I don't know it's not me, the users say it's fine." Which is probably true because it's a document management system and they're reading and writing small files.

Troubleshooting ideas? Has anybody seen LVM trash backup or other I/O performance? Especially given a largeish number of volumes holding a very large number (10 million maybe)of small files?

Edited to correct units.

Edited to add:

NIC is at 1000/Full (as checked from both the OS and Switch)

Filesystem is EXT3.

More new information....

The performance hit appears to be happening on several boxes running LVM and EXT3. Basically all the new RHEL5 boxes we built this summer.

Laura Thomas
  • 2,825
  • 1
  • 26
  • 24
  • What filesystem are you using on top of the LVM? – Cian Jul 23 '09 at 00:09
  • I don't know what file system is on top. I don't have access to it I'm just the backup administrator, so I'll have one of the Unix guys check it out in the morning. – Laura Thomas Jul 23 '09 at 03:30
  • There is nowhere near enough information in your post to answer your question. In any case LVM has at worst a few % perf. hit, certainly not 50%, ever. The FS has more impact. Hell the cabling to the SAN has more impact. – niXar Jul 23 '09 at 13:22

4 Answers4

2

The problem turns out to have been a NetBackup client version problem, more than it was a linux/LVM problem. When the box got rebuilt as a linux box the 6.5 client was installed. Today in response to another issue we upgraded client version to 6.5.4. I am back to pulling data off the box at 25-27mb/sec.

How it is I could have forgotten the number one rule of NetBackup, or probably any backup software, MAKE SURE YOUR CLIENT AND SERVER VERSIONS MATCH if you are having a problem I don't know. Maybe I need a tatoo.

Laura Thomas
  • 2,825
  • 1
  • 26
  • 24
2

Have you used sar or iostat to monitor the disk performance during the backup to see what Linux thinks about the disk performance?

What about using some sort of benchmark utility to test raw read performance of files on the system? I just came up with this, so this is probably a terrible way to do this, and this would really just be for sequential reading, but:

sudo dd if=/u1/some_large_file of=/dev/null

Basically, if you use a benchmarking utility to duplicate reading of all the small files, you can very if it is the disk, and go from there.

The following is no longer relevant:
If with 20 kb/s you mean kilobits, unless I messing this up because it is too early in the morning, your numbers don't add up. You said you have 2.1 terabytes at 20 kb/s:

Even if it was just 1 TeraByte:

1 TB = 8589934592 bits
8589934592 / 20 (bits a second) = 429496730 seconds
429496730 / 60 (seconds) = 7158278 minutes
7158278 minutes / 60 = 119,304 Hours
119,304 / 24 = 4971 (Days)

Or if you meant kilobytes:

1 terabyte = 1073741824 kilobytes
1073741824 / 20 kB/s = 53687091 seconds
53687091 seconds = 621 days

Am I messing up these calculations? ( will have to delete my post in shame if I am :-) )

Kyle Brandt
  • 82,107
  • 71
  • 302
  • 444
  • You are correct. I by default in my head convert the kilobytes/second number into megabytes/sec. But when I was verifying, I went with the column number. I'll go fix the original question now. How embarassing. – Laura Thomas Jul 23 '09 at 15:23
  • I have used iostat and basically the disks are thrashing with wait times up in the 60-85 range and they see the I/O rates to be about exactly the same as the backup system does. – Laura Thomas Jul 23 '09 at 15:26
  • Ah. So what does the storage look like? How is it attached? ISCSI, SCSI, Fiber? Is it in some sort of raid configuration? – Kyle Brandt Jul 23 '09 at 15:40
0

what file system are you using on the LVM volume(s)?

and how are the 10 million small files stored - all in one directory (or a small number of directories), or spread across many directories and subdirectories? ("many" being an arbitrarily large number)

the reason i ask is that some file systems have severe performance problems when you have thousands of files in them. this is one possible cause of your slow down.

for example, ext2 or ext3 without the dir_index feature turned on (IIRC, dir_index has been the default on ext3 for several years now. it helps a lot but it doesn't eliminate the problem entirely).

you can use tune2fs to query and/or set the dir_index feature for ext3. e.g. to query:

# tune2fs -l /dev/sda1 | grep feature
Filesystem features:      ext_attr resize_inode dir_index filetype sparse_super

if you don't see dir_index in that list, then you need to turn it on like so:

and to set:

# tune2fs -O dir_index /dev/sda1
tune2fs 1.41.8 (11-July-2009)

(yes, tune2fs only responds here by printing its version number...doesn't bother telling you whether the operation succeeded or failed. not good, but presumably it would print an error if it failed)



finally: if this does turn out to be the problem, and enabling dir_index doesn't help, then you probably need to consider using a different filesystem. XFS is a good general purpose filesystem, and AFAIK ext4 doesn't have this problem. either would be a reasonable choice for a replacement fs (although ext4 is quite new and even though many people are using it without problems, i'm not sure i'd trust it on production servers just yet)

cas
  • 6,653
  • 31
  • 34
  • The files are in a directory structure that is about 4 tiers down before you get to actual files. So while the number of files in a directory is large, but not insanely so. – Laura Thomas Jul 23 '09 at 03:52
  • 1
    if there's only a few hundred to maybe a thousand or two files in a directory, that would be OK. but there is a very steep and very noticable decline in ext3 performance when you have more than 1000 or so files in a dir (which is why it doesn't perform very well for usenet news spools or for very large Maildir folders) – cas Jul 23 '09 at 04:31
  • 1
    other things to check are the system's network card. have you tried speed tests with ftp or http transfers? checked that it's connecting at full-speed (100baseT or Gb) and full-duplex? (if it's running in half-duplex mode that would cause a massive performance loss). have you checked that there aren't lots of collisions or errors in ifconfig's output? any of these would indicate a fault with the NIC or the driver or with the configuration (e.g. some cards don't auto-detect duplex mode reliably and need to be explicitly set up for FD) – cas Jul 23 '09 at 04:36
  • The NIC is at 1000GB and Full duplex. The network appears to be pretty clean. The filesystem is EXT3. It's basically \vol\dir\hundreds of dirs\half a dozen dirs\100 or so files – Laura Thomas Jul 23 '09 at 16:06
  • ext3 should have no problem with a directory structure like that. next things to check are throughput with ftp or http (test both get and put, R & W. and don't try just one file, try dozens). also are jumbo frames enabled on the NIC? if ftp/http throughput is fine, then i'd be suspicious of the linux port of Netbackup. – cas Jul 23 '09 at 21:58
  • one more thing just occurred to me - how is Netbackup storing the files on the linux filesystems? via samba or some other file-server protocol? or with it's own in-built protocol? – cas Jul 23 '09 at 22:00
-1

LVM itself really shouldn't be impacting this. To my knowledge the LVM bits are not referenced on every metadata operation, which is where a delay would come in to play. It's at a different layer in the kernel. LVM would affect mount/unmount more than it would affect file open/close.

Far more likely is what Craig pointed out, large directories impairing performance. Linux is somewhat notorious for not handling the large-directory problem well. VxFS can handle up to 100K files/directory quickly, where ext2/ext3/reiserfs generally start slowing down well before then. This is one area where a poor choice in filesystem for migration target can seriously impair your backup performance.

That said, if this is your problem, just plain old access into and out of those directories should also be impaired. It may be the difference between 80ms to open a file and 210ms, which is barely perceptible to end-users, but it should be there.

sysadmin1138
  • 131,083
  • 18
  • 173
  • 296
  • yep, except for one detail. reiserfs doesn't have this problem at all. lots of files is one of the things reiser is optimised for. i still wouldn't use it on a production server though (or even on my home desktop). – cas Jul 23 '09 at 04:28
  • 1
    I disagree with the large directory thing, ext3 has had b-tree dir indexes for years now. It's not an issue. Reiser has one up on other FS in that it can store very small files in its tree instead of wasting a whole block to store a single byte, but this makes no difference at all if the files are of non-trivial sizes. – niXar Jul 23 '09 at 13:25
  • 8 Reiserfs partitions to 250GB error free for almost 4 years. (OpenSUSE 10). Watch, they're all gonna die when I get home tonight. Lol. – Chris K Jul 23 '09 at 16:54