0
I guess I need some advanced advice on SSD, SATA, in/out limits etc.
I'm programming a video installation where I continously read and write quite a lot of data:
To the first SSD, drive A, I write 50 frames per second (= 50 files per second) plus simultaneously read some other 50 fps (from the same drive!). From a second SSD, drive B, I read an additional 50 frames per second.
The question is: Where is the bottle neck? I succeed running my program at 25 fps but not faster (and i would need to get it to 50 fps).
The total amount of data is actually not that excessive: Each frame is 1,8 MB so each stream is about 90 MB/s which comes down to a total of 180 MB/s read plus another 90 MB/s write = 270 MB/s total data transfer in 150 read/write operations per second.
What confuses me most is the following:
- I get the same performance when I run the patch from two separate SSDs as when I run all three streams from a single one without using the second SSD at all.
- The speed does not increase substantially if I make one or two of these streams smaller (and reduce the file size to 260 KB/file which equals only 13 MB/s per stream).
- It does increase however (to 40-60 fps) if I switch all three streams to 260 KB/file.
This makes me wonder what is going on... is it the drives? Is it SATA? Or some other stuff I maybe even never heard of? And what can I do about it?
I use a Macbook Pro (Retina, early 2013, i7 2,6 GHz, 16 GB RAM, OS 10.9.2), drive A is a Samsung 840 pro 256 GB SSD connected via thunderbolt, drive B is the internal 512 GB Macbook SSD (built-in from Apple).
My program is built in MAX/MSP and I use a an arbitrary, uncompressed binary data format (.jxf). I can rule out CPU-/GPU-load issues.
user3647558
Posted 2014-05-17T12:23:50.893
Reputation: 21
With each frame being its own file, you force a lot of write amplification as lots of file metadata gets updated. You may be able to tune the filesystem to buffer metadata writes more, but you probably should just use a more efficient storage design that doesn't require so many directory operations. (Also, shut off updating of file access/modification times. That makes lots of extra writes.) – David Schwartz – 2014-10-08T17:24:58.970
have you considered RAID as an option to minimize bottleneck? – None – 2014-10-08T17:18:52.463
One possibly issue is writing only 1.8mb at a time. Try, for testing purposes writing 2 frames into the same file. SSD like reading/writing in larger blocks >8mb some like as high as 32mb at once. Try ATTO benchmark to see what I mean. http://www.majorgeeks.com/mg/getmirror/atto_disk_benchmark,2.html
– cybernard – 2014-05-17T19:42:29.403thanks for your input! i tried that but since i read out these frames in a random fashion this basically just doubles my data rate (which makes things worse). regardless: in the meantime i did some more testing, as well with a pegasus j2 ssd-raid - which is blazingly fast but does not perform any better. by now i'm convinced i'm just hitting some io- or other limit of sata. ssd over pci might be a solution but since i work on a laptop, this is at least not an option right now... – user3647558 – 2014-05-23T01:54:26.313