150

This is a software design question

I used to work on the following rule for speed

cache memory > memory > disk > network

With each step being 5-10 times the previous step (e.g. cache memory is 10 times faster than main memory).

Now, it seems that gigabit ethernet has latency less than local disk. So, maybe operations to read out of a large remote in-memory DB are faster than local disk reads. This feels like heresy to an old timer like me. (I just spent some time building a local cache on disk to avoid having to do network round trips - hence my question)

Does anybody have any experience / numbers / advice in this area?

And yes I know that the only real way to find out is to build and measure, but I was wondering about the general rule.

edit:

This is the interesting data from the top answer:

  • Round trip within same datacenter 500,000 ns

  • Disk seek 10,000,000 ns

This is a shock for me; my mental model is that a network round trip is inherently slow. And its not - its 10x faster than a disk 'round trip'.

Jeff attwood posted this v good blog on the topic http://blog.codinghorror.com/the-infinite-space-between-words/

Community
  • 1
pm100
  • 1,595
  • 3
  • 10
  • 9

13 Answers13

160

Here are some numbers that you are probably looking for, as quoted by Jeff Dean, a Google Fellow:

Numbers Everyone Should Know

L1 cache reference                             0.5 ns
Branch mispredict                              5 ns
L2 cache reference                             7 ns
Mutex lock/unlock                            100 ns (25)
Main memory reference                        100 ns
Compress 1K bytes with Zippy              10,000 ns (3,000)
Send 2K bytes over 1 Gbps network         20,000 ns
Read 1 MB sequentially from memory       250,000 ns
Round trip within same datacenter        500,000 ns
Disk seek                             10,000,000 ns
Read 1 MB sequentially from network   10,000,000 ns
Read 1 MB sequentially from disk      30,000,000 ns (20,000,000)
Send packet CA->Netherlands->CA      150,000,000 ns

It's from his presentation titled Designs, Lessons and Advice from Building Large Distributed Systems and you can get it here:

The talk was given at Large-Scale Distributed Systems and Middleware (LADIS) 2009.

Other Info

It's said that gcc -O4 emails your code to Jeff Dean for a rewrite.

David d C e Freitas
  • 2,712
  • 1
  • 16
  • 13
  • +1 Very interesting! – 9dan Feb 22 '11 at 09:35
  • 1
    Some presentations have different values indicated in the brackets. I assume the one's in the bracket were incorrect and he updated the values. – David d C e Freitas Feb 22 '11 at 12:56
  • 2
    Is this all pre-SSD era? see [here](https://gist.github.com/jboner/2841832) for further up-to-date numbers. – matt Jun 13 '16 at 19:26
  • I actually used these numbers to build a [presentation showing why SSD drives pay for themselves](https://prezi.com/j_ennbpapywo/how-fast-does-a-business-computer-need-to-be/), to convince our office manager that yes we needed faster machines to work on. Included the numbers for the technical info but geared it towards non-tech management as much as possible. – brichins Apr 14 '17 at 21:09
23

There are a lot of variables when it comes to network vs. disk, but in general, disk is faster.

The SATA 3.0 and SAS buses are 6 Gbps, vs. a networks 1Gbps minus protocol overhead. With RAID-10 15k SAS, the network is going to seem dog slow. In addition, you have disk cache and also the possibility of solid state harddrives, which depending on the scenario, could also increase speed. Random vs. Sequential data access plays a factor, as well as the block size in which data is being transferred. That all depends on the application that is being used to access the disk.

Now, I have not even touched on the fact that whatever you are transporting over the network is going to or coming from disk anyway...so.......again, disk is faster.

JakeRobinson
  • 2,886
  • 17
  • 26
  • 1
    Points for mentioning RAID which gives you parallel reads, something which you are unlikely to get on a network anytime soon. Of course, if we are talking about local laptop hard drives, then the combo of fast SAN and fast network might well be faster. Especially with the SSDs in that SAN. – Michael Dillon Feb 22 '11 at 01:47
  • 13
    Networks are inherently parallelizable -- what are you talking about? It's incredibly trivial to read from multiple systems on a network in aggregate; this is the entire point behind systems like Hadoop and MPI, not to mention the obvious BitTorrent. – jgoldschrafe Feb 22 '11 at 03:39
  • 2
    With SONET/SDH you can have 38Gbps still faster than SAS. And network aggregation can be done with something like http://en.wikipedia.org/wiki/Link_aggregation – Mircea Vutcovici Feb 22 '11 at 04:30
  • 1
    The question leaves too much context out -- but if the network is designed and implemented correctly, it won't be the bottleneck. But generally, the bottleneck is going to be the component you ran out of money to size optimally. – belacqua Feb 22 '11 at 08:39
  • 11
    @Jake When talking about 6 Gbps, you might want to make a clear distinction between interface bandwidth and the rate at which a disk can actually supply data. – NPE Feb 22 '11 at 09:28
  • I remember in the beginning of google. A search of google loaded faster than a page from disk :) – mplungjan Feb 22 '11 at 10:00
  • 1
    Network can easily be A LOT faster than disks, take a look at Infinband, its maximum speed is 300Gbps... – Kedare Feb 22 '11 at 15:09
  • 4
    i did say in my question that I was talking about a remote in memory database compared to a local on disk cache – pm100 Feb 22 '11 at 18:41
  • @pm100 The original question seemed to be asking about "local disk" instead of "local on-disk cache." If the data you want is already in the disk's cache, you should get speeds comparable to remote in-memory DB by avoiding any seek or spin (rotational latency) delays. – Paul Feb 22 '11 at 20:40
  • 1
    Latency != throughput. The OP was asking about the former. – Tobu Feb 23 '11 at 00:04
11

Well, that depends on whether the network resource has the data you are requesting readily available (in memory or similar) or if it would just, in turn, read it from a disk.

In any case, throughput may be higher in some cases but I believe latency will be higher.

3

The experience I have with this is that when you're on a 1Gbit connection and you try to download a file your harddisk is usually the bottleneck. A thing you have to keep in mind though is that you have to set up a connection first, which also takes time. So for sending big chunks of data network might actually be faster than disk.

teuneboon
  • 173
  • 1
  • 1
  • 7
  • 1
    Unless the disk is also the bottleneck on the other side of the network connection... –  Feb 21 '11 at 23:51
  • @Argote: True, but if the server software was written correctly, it'll buffer into memory before writing to the disk. – amphetamachine Feb 22 '11 at 01:32
2

IMX the disk is still faster. The theoretical transfer rate of the network is high but in practice you don't get close to that.

About two years ago I had hard drive trouble on my laptop and the DMA went out. This made the hard drive dramatically slower, and in particular slower than network. But when I switched to another computer I was back to my original state of HDD faster than Internet.

Charles
  • 363
  • 2
  • 3
  • 16
2

My experience with gigabit networks is, given the right server, that you can beat local performance in terms of throughput and latency. See Network Tests: Are We Getting Gigabit Performance?

For all practical purposes I would recommend treating network & local storage as equivalent and only use memory caches.

Standard caveat as you mentioned is true in that there are no general rules; and that actually most of the time one should be working with well configured servers and be using metrics to evaluate the best method of data transfer.

If you are using a low end machine with a slow hard drive then it will almost certainly be quicker to use a gigabit network connection to a server with a fast storage array.

Equally if you are working with two machines of near identical hardware then the latency and network overhead would make local storage quicker; it's common sense really.

Richard Harrison
  • 131
  • 4
2

It depends. If your I/O is primarily random access then its flat throughput is probably not that great compared to the network bandwidth that could be available. However, most network traffic is ultimately generated by processes that involve I/O. If the working set of whatever process is generating the network traffic fits into cache then it won't be constrained by disk bandwidth. If it thrashes the cache then disk will become a bottleneck.

I work on data warehouse systems, and the canonical DW query is a table scan. If your query hits more than a few percent of the rows in the fact table (or partition) then a table or partition scan using sequential I/O will be more efficient than a random access query plan using index lookups and seeks.

Networked storage (i.e. SANs) tends not to perform well on streaming workloads unless it is tuned appropriately. If the SAN is being used for a general purpose consolidation environment it will almost certainly be tuned quite sub-optimally for a streaming, spiky load like a data warehouse. I have seen a vendor white paper suggest that you need about 3x the number of disks to get the same throughput on a SAN that is not tuned for streaming I/O as for one that is.

My experience tallies with that. In fact, I have never deployed a data warehouse onto a consolidation environment where I could not run the same ETL process significantly quicker on my desktop PC. I've also had sales reps from a major vendor of SAN equipment say off the record that a lot of their customers use direct attach storage for the DW system because SANs aren't fast enough.

Networked storage is at least an order of magnitude more expensive per IOPS than direct attach storage for random access workloads and closer to two orders of magnitude more expensive for streaming.

ConcernedOfTunbridgeWells
  • 8,810
  • 2
  • 31
  • 52
1

Yes, in general, networks are now get faster than hard-drives, but this may chnage over time.

I think, therefore I am

When an application is running it means the host machine is working, while working over network needs a common protocol, checking for peer availability, channel security... and if the peers use different platforms, it's harder to achieve what you can do on a single machine.

I prefer to look at this in the terms of trade-offs rather than who is the strongest...

Xaqron
  • 208
  • 4
  • 13
1

You have to describe an exact use case for this comparison. Harddrives have seek time + transfer rate and cache. Networks have latency, transfer rate and protocol overhead...

I think your original cache memory>memory>disk>network still stands true in general though

Zepplock
  • 143
  • 1
  • 6
0

The disk is connected with the CPU via SCSI, SAS or IDE bus. Which is a internal network running a specific protocol - SCSI or ATAPI. Ethernet is designed to work on longer distances and can be much slower than SAS/SCSI/IDE. So which one is faster, dependes on which technologies are you comparing. If you compare a 20 years old laptop HDD with a 10Gbps in RAM storage, the winner will always be the networking. And when you buy a storage you have to compare it versus price and manageability.

Mircea Vutcovici
  • 16,706
  • 4
  • 52
  • 80
0

Well, there's Light Peak which is aiming for 100GBps networking speed, which is getting close to RAM speeds. Of course, the network can only deliver data as fast as the sender can generate the data, i.e. if the sender is reading the data from a hard disk then the receiver will only get the data at the same speed as the disk read, even with a superfast network.

Skizz
  • 111
  • 7
0

One thing to keep in mind is that it depends on the network. Say, for example, you're responsible for performance on a web site. This web site is of course connected to a database server over a local network, and is also connected to web surfers over the internet, which is also a kind of network.

In many cases, a dedicated link may be set up between the web server and the database server via static IPs and a crossover cable or automdx to keep latency down and provide a dedicated link for the traffic, since you want it to be very fast. The database server does all kinds of work to keep as much of the db as possible in memory, and in many cases often succeeds for the entire contents plus a few indexes. Queries to this database are going to be just as fast or even faster than queries to disk.

On the other hand, certain web technologies (asp.net webforms viewstate, I'm looking at you) like to push a lot of information to and from the client web browser as a cache (of sorts). If this is a local lan connection (and in asp.net webform's defense this is true much of the time) it's not all that bad, but on the public internet this can absolutely kill performance, such that you are often much better off pushing this to a database or local disk instead.

Joel Coel
  • 12,910
  • 13
  • 61
  • 99
0

Personally, I think there are several factors to consider. For instance, how fast is the memory or disk you are accessing locally vs. the one you would be accessing via network? If the remote data was on very fast SSD and faster than gigabit networking installed end to end, the remote might be faster for large streaming files.

However if you were randomly accessing small units of data and the network wasn't flawless or had many hops and more than just you accessing it, I'd bet that a local cache is faster even of its on a whirly mechanical disk drive almost 100% of the time. But you bring up an interesting point and how long will will need local storage of anything if network speeds continue to grow?

jim
  • 1