Why is writing to my NAS in small chunks so slow?

Question

I just discovered that the bottleneck of a Python script was writing a text file unbuffered line by line to our QNAP NAS. See attached Python snippet. Is this necessarily so slow or is there something wrong with our QNAP/network setup?

# local SSD: 2 seconds
with open(r'C:\Daten\numbers.txt', 'w') as f:

# local SSD with buffering: 2 seconds
with open(r'C:\Daten\numbers.txt', 'w', buffering=2**20) as f:

# Share on QNAP NAS: ### 36 ... 61 seconds! ###
with open(r'I:\numbers.txt', 'w') as f:

# Share on QNAP NAS with buffering: 2 ... 3 seconds
with open(r'I:\numbers.txt', 'w', buffering=2**20) as f:
    for i in range(1000000):
        print(i, file=f)

score 2 · Accepted Answer · answered Nov 10 '17 at 15:11

Any small I/O is going to be expensive compared to larger transfer. In this case, the situation is worsened by:

using unbuffered I/O
sending data down to the network rather than a fast local bus
using a remote filesystem protocol (ie: CIFS or NFS).

While you can somewhat improve performance by fine-tuning the NFS/CIFS/SMB server behavior, I strongly suggest you to avoid generic unbuffered I/O in favor of buffered I/O + explicit flush points (if required by your application).

On top the OP should read up on the difference performance characteristic of SSD and HDD and on the speed of his network ;) — TomTom, Nov 10 '17 at 15:20

Why is writing to my NAS in small chunks so slow?

1 Answers1