Centos 6 vs Centos 7 file access performance

Question

I'd like some assistance in trying to figure out what could explain Centos 7 being slower in accessing a file then Centos 6. It's about a 17% difference.

Here's a simple test:

[root@test-centos6 shm]# time for i in $(seq 0 50000); do stat file > /dev/null; done

real    0m42.283s
user    0m2.465s
sys     0m4.434s
[root@test-centos6 shm]# time for i in $(seq 0 50000); do stat file > /dev/null; done

real    0m41.890s
user    0m2.442s
sys     0m4.341s
[root@test-centos6 shm]# time for i in $(seq 0 50000); do stat file > /dev/null; done

real    0m41.795s
user    0m2.383s
sys     0m4.310s

----

[root@test-centos7 shm]# time for i in $(seq 0 50000); do stat file > /dev/null; done

real    0m49.081s
user    0m16.306s
sys     0m32.639s
[root@test-centos7 shm]# time for i in $(seq 0 50000); do stat file > /dev/null; done

real    0m48.379s
user    0m16.034s
sys     0m32.191s
[root@test-centos7 shm]# time for i in $(seq 0 50000); do stat file > /dev/null; done

real    0m48.054s
user    0m15.680s
sys     0m32.245s

strace seems to confirm C7 performs more operations per stat then C6.

[root@test-centos6 shm]# strace stat file 2>&1 | wc -l
145

[root@test-centos7 shm]# strace stat file 2>&1 | wc -l
168

Anyone knows how to reduce the number of operations for C7 so perfs are more inline with C6?

Thank you,

//EDIT:

I have put both in /dev/shm to eliminate a possible xfs/ext4 difference. These are 2 identical VMs on the same ESX host, with just a different OS. (but I have observed the same difference on hardware boxes as well, just used those VMs for my current testing)

Strace C6: https://paste.fedoraproject.org/paste/oCSUZqKgcoNLJubjeNbo9V5M1UNdIGYhyRLivL9gydE= Strace C7: https://paste.fedoraproject.org/paste/PvTXvxxvS~2UZ9LuuSVAA15M1UNdIGYhyRLivL9gydE=

I would have assumed the newer C7 would have better performances in it's stock form, not worse.

[root@test-centos6 shm]# uname -r
2.6.32-642.el6.x86_64
[root@test-centos6 shm]# getenforce
Disabled

[root@test-centos7 shm]# uname -r
3.10.0-514.6.1.el7.x86_64
[root@test-centos7 shm]# getenforce
Disabled

Looks like they are in /dev/shm (ramdisk) but it would be great to get confirmation. — Aaron, Mar 01 '17 at 17:18
I believe you may also wish to [upload your strace output](https://tinyvpn.org/) so people can take a peek. Not all folks may have both C6 and C7 images active somewhere. — Aaron, Mar 01 '17 at 17:22
Is this even a realistic test? How much does your actual workload rely on repeated `stat()` calls? — Michael Hampton, Mar 01 '17 at 21:36
Yes this is a realistic test as we heavily rely on file existence tests for this system (~ 90 million calls per day) — Bob, Mar 02 '17 at 14:51
Does your application use stat via a shell builtin or the command line? It may have a lot less overhead written in a different language, maybe even C. — John Mahowald, Mar 03 '17 at 13:38

ewwhite · Answer 1 · 2017-03-01T17:28:18.240

0

They are totally different operating systems with different kernels, compiled-defaults, filesystems, schedulers and tuning capabilities.

Why do you assume they should perform alike? There's a lot to account for, so this question will be difficult to answer in its current form.

edited Mar 01 '17 at 17:28

answered Mar 01 '17 at 17:21

ewwhite

194,921
91
434
799

I have assumed that the newer C7 would have better performances, or at least on-par with it's predecessor, not worse. I'm now looking for how to tune it to perform better. Thanks! – Bob Mar 01 '17 at 17:34

score 0 · Answer 2 · answered Mar 01 '17 at 20:21

The stat utility differs between CentOS6 and CentOS7. To confirm, try issuing stat --version on both machines.

The new strace version is more complex, perhaps. For example, from your strace output, it appears that CentOS7's strace maps libpcre.so.1 (a reguar expression library), while CentOS6's strace does not.

Anyway, this basically is a canned test: to really benchmark file access/manipulation speed, you should use tools as fs_mark or, even better, some real world test (eg: untar a large archive).

Centos 6 vs Centos 7 file access performance

2 Answers2