9

I wonder how much data in total does a freshly installed vanilla Linux system (like 32-bit CentOS 5.10) read in order to get to a virtual console shell prompt? You know, reading all the configuration files, loading binaries, kernel image, etc.

I am booting 100s of Linux machines, and 95% of their system contents are identical and will remain identical - the machines are clones. I would like to offload the identical/read-only shared part of the filesystem, to an NFS storage, mount it from there, and boot like that. Only writeable part of the filesystem, like /var, /tmp and /home, will remain local to each machine. Since potentially hundred machines may boot simultaneously as part of a "cluster", I need to estimate whether the accessed NFS storage link will be a bottleneck while booting from.

I am looking for order-of-magnitude estimates. I am aware that Linux boot varies greatly with regards to details of the process. Are we talking 10Mb? 100Mb? 1Gb?

amn
  • 325
  • 1
  • 16
  • 5
    Why are you asking? – Zoredache Mar 05 '14 at 17:20
  • 2
    The variability is (can be) orders of magnitude between systems - loading the kernel & drivers is the tiniest fraction of the boot process, and the init scripts on a system can do *literally anything* before you get a login prompt. Please explain the situation you're dealing with in terms of an actual, practical problem that we can help you solve. – voretaq7 Mar 05 '14 at 17:29
  • Let us assume a vanilla 32-bit CentOS 5.10. If no estimates down to 1/100th of a gigabyte can be given, can I at least sample the number somehow? I asked the question, because I need to evaluate mounting the read-only parts of the filesystem from an NFS storage, essentially booting from that. – amn Mar 05 '14 at 17:40
  • 1
    @amn Can you put the reason in your initial question? It'll help with context. Another reason people would ask a similar question is if they're using cycle-limited storage. More detail is always better. – ewwhite Mar 05 '14 at 17:47
  • @ewwhite We're talking about reading, not writing data. Flash storage shouldn't be limited on reading, but only on writing cycle. For the latter it would also be interessant to know the cost of rebooting a system on a SSD for example. – piernov Mar 05 '14 at 17:50
  • 1
    You're looking at the vicinity of 20 - 50 MB: kernel + modules (~10, but can be more depending on hardware), glibc, additional basic libs, around dozen of executables). – oakad Mar 06 '14 at 01:43
  • 5
    `I need to estimate...` then do one and measure it. – symcbean Mar 06 '14 at 12:44

2 Answers2

15

You say in your comments that you're evaluating a netboot / network root environment.

The first thing you must realize is there is no such thing as "vanilla" - you're not going to run CentOS 5.10 right out of the box with zero changes (if you think you are you're deluding yourself: NFS Root is already at least Strawberry, verging on Pistachio).

If you want an answer for your specific environment (which is what really counts) you're going to need to set up an NFS server and a client machine, boot it, and measure:

  1. The transfer (quantity)
  2. The throughput (rate)

Both values will be critically important for performance. You'll probably also want to set up several clients at some point and simulate normal use of the system to see what kind of steady-state demand they put on your NFS server / network when people are using the systems as they would in their everyday work.

See also: Our series on Capacity Planning - we don't talk specifically about NFS, but the general principles of "Build it, Test it, Stress it" apply.

Community
  • 1
voretaq7
  • 79,345
  • 17
  • 128
  • 213
  • 1
    If there is vanilla icecream, there is vanilla Linux! ;-) Seriously though, it is a fairly unaltered CentOS 5.10, and whatever was altered is part of the writeable filesystem, that which **won't** be mounted from NFS, so it's not a factor - yes, there is a gigantic Postgres database in /var/lib but /var is not mounted from NFS but is on the local physical startup disk. And if I wanted to profile it, I wouldn't be asking the question here :-) – amn Mar 05 '14 at 18:01
  • 10
    @amn I'm sorry you don't want to do profiling, but you gotta do what you gotta do - we can't pull applicable numbers out of our butts for you. Your solution (NFS root) is a sound, time-tested one, and honestly you can probably just deploy it blind with no problems (tens of thousands of Sun Microsystems environments were deployed blind like that back in the heyday of NFS-root & Netbooting Solaris & worked great). If you're concerned about performance though you'll need to do profiling to determine the demand and bottlenecks for your specific environment - that's just the way of the universe. – voretaq7 Mar 05 '14 at 18:58
  • 1
    @voretaq7 Can't argue with the profiling argument, and never did. I just wanted the next best thing before I roll up my sleeves and set up the NFS. Thanks for your valuable input. – amn Mar 06 '14 at 09:54
8

Install one system, boot it and check out the block layer statistics from /sys/block/${DEV}/stat e.g. /sys/block/sda/stat.

Quoting from the documentation:

The stat file consists of a single line of text containing 11 decimal values separated by whitespace. The fields are summarized in the following table, and described in more detail below:

Name            units         description
----            -----         -----------
read I/Os       requests      number of read I/Os processed
read merges     requests      number of read I/Os merged with in-queue I/O
read sectors    sectors       number of sectors read
read ticks      milliseconds  total wait time for read requests
write I/Os      requests      number of write I/Os processed
write merges    requests      number of write I/Os merged with in-queue I/O
write sectors   sectors       number of sectors written
write ticks     milliseconds  total wait time for write requests
in_flight       requests      number of I/Os currently in flight
io_ticks        milliseconds  total time this block device has been active
time_in_queue   milliseconds  total wait time for all requests

read sectors, write sectors

These values count the number of sectors read from or written to this block device. The "sectors" in question are the standard UNIX 512-byte sectors, not any device- or filesystem-specific block size. The counters are incremented when the I/O completes.

You can use this one-liner to get the number of bytes more easily:

awk '{printf("read %d bytes, wrote %d bytes\n", $3*512, $7*512)}' /sys/block/vda/stat

Results for Scientific Linux 6.1 i386

I tested this on a KVM/qemu virtual machine running Scientific Linux 6.1 i386 (which is similar to RHEL). The following services were enabled: acpid, auditd, crond, network, postfix, rsyslog, sshd and udev-post. The swap is on a separate disk, so it's not taken into account.

The stats for 85 boots, taken remotely with SSH a couple of seconds after the login prompt appeared, were:

    Name            Median   Average   Stdev
    -------------   ------   -------   -----
    read I/Os       1920     1920.2    2.6
    read merges     1158     1158.4    1.8
    read sectors    85322    85330.9   31.9
 >> read MiBytes    41.661   41.665    0.016
    read ticks      1165     1177.2    94.1
    write I/Os      33       32.6      1.7
    write merges    64       59.6      7.4
    write sectors   762      715.2     70.9
 >> write MiBytes   0.372    0.349     0.035
    write ticks     51       59.0      17.4
    in_flight       0        0.0       0.0
    io_ticks        895      909.9     57.8
    time_in_queue   1217     1235.2    98.5

The boot time was around 20 seconds.

Community
  • 1
Cristian Ciupitu
  • 6,226
  • 2
  • 41
  • 55
  • 2
    Note that this only seems to give you the transfer demand (quantity), not the throughput demand (rate). You could divide by the uptime to get an average number though. – voretaq7 Mar 05 '14 at 19:16