The web application I'm working on, will be used to upload/download large number/amounts of smaller size files - I'm looking at close to 1B files with total size of > 10Pb. I'm currently struggling with deciding the scalable architecture that would support such amounts. And here's my question - is there a way of building some sort of storage that would be seen by a windows server as one huge (10Pb and up) network storage drive, so I can write all the files to subfolders of that virtual drive? And how would it perform?
Right now I'm trying to understand if that's even possible, or if I have to implement software level sharding - writing files to different drives based on some key.
I'm a developer, not a sys admin, so I apologize if it's a naive question, and thanks in advance for patience in explaining me possibly trivial things.
Andrey