I recently formatted a 20TB array as XFS. For file transfers < 10G performance is fantastic. However once I exceed 10G disk I/O decays logarithmically. I have the journal on the same disk. When I look at the disk IO usage, I see that jbd2flush is using nearly all disk IO. How can I resolve this? (Any way to do it w/o removing the data?) I am running Ubuntu 10.10. Thank you.
Asked
Active
Viewed 2,311 times
1
-
How big and how many log buggers did you create when you made the filesystem? – Decado Mar 29 '11 at 06:30
-
Is there a way I can check? I don't recall. – ensnare Mar 29 '11 at 06:31
-
xfs-info should do it. – Decado Mar 29 '11 at 06:49
-
Also, how many disks in the array and in what kind of raid array, and hardware or software controller? – Decado Mar 29 '11 at 06:50
-
8 disks hardware controlled. Doubtful it's the controller -- when I had 8x1TB drives in RAID5 formatted in ext4 I didn't have this problem. Only after my recent re-format with XFS. – ensnare Mar 29 '11 at 07:26
3 Answers
2
The journal is usually never the performance problem except for small random write IO, as the log is a circular buffer that is access sequentially as the log just gets appended.
How much memory does the system have? Whate are the watermarks on the pdflush side of things?
Can you produce a throughput diagram ie plot some of the columns of iostat -x -k 1 with gnuplot or something like that for the <10GB and for > 10GB cases?
pfo
- 5,630
- 23
- 36
-
It looks like the RAM on the system is getting eaten up. I have 4GB which may not be sufficient for a 20TB array. When I do free -m I see that the system starts eating away at swap for files < 10GB. Will put in more RAM to test. Do you happen to know the minimum requirement for a 20TB XFS array? Thanks. – ensnare Mar 29 '11 at 18:58
-
it may even bee "too" much RAM so that pdflush is keeping too much dirty memory that needs to be flushed frequently so that your IO suffers from very spiky throughput. – pfo Mar 29 '11 at 19:34
-
1
To reduce contention under high load, I would put the journal on a separate disk. Easiest would have been one disk from the array itself but probably you cannot do this anymore without backup restore. Setting a dedicated log device is explained in the tune2fs(8) manpage under the option -J.
Peter G.
- 333
- 5
- 12
-
1Is there a way to move the journal without re-formatting the array? Thanks. – ensnare Mar 29 '11 at 18:58
0
Is your XFS file system aligned to stripe size? This may not be directly related to your performance issue but it definitely improves performance. Please check this thread for some general ideas and samples.
