8

A bunch of new files with unique filenames regularly "appears"1 on one server. (Like hundreds GBs of new data daily, solution should be scalable to terabytes. Each file is several megabytes large, up to several tens of megabytes.)

There are several machines that process those files. (Tens, should solution be scalable to hundreds.) It should be possible to easily add and remove new machines.

There are backup file storage servers on which each incoming file must be copied for archival storage. The data must not be lost, all incoming files must end up delivered on the backup storage server.

Each incoming file myst be delivered to a single machine for processing, and should be copied to the backup storage server.

The receiver server does not need to store files after it sent them on their way.

Please advise a robust solution to distribute the files in the manner, described above. Solution must not be based on Java. Unix-way solutions are preferable.

Servers are Ubuntu-based, are located in the same data-center. All other things can be adapted for the solution requirements.

1Note that I'm intentionally omitting information about the way files are transported to the filesystem. The reason is that the files are being sent by third parties by several different legacy means nowadays (strangely enough, via scp, and via ØMQ). It seems easier to cut the cross-cluster interface at the the filesystem level, but if one or another solution actually will require some specific transport — legacy transports can be upgraded to that one.

Shane Madden
  • 112,982
  • 12
  • 174
  • 248
Alexander Gladysh
  • 2,343
  • 7
  • 30
  • 47
  • 5
    I like this question. It's the kind of thing I talked about encouraging back to SF in my pre-election manifesto. – Tom O'Connor Jun 18 '13 at 10:11
  • I would very much appreciate if people who voted to close this question, elaborated on their motivation in the comments. Especially the off-topic vote. Thank you. – Alexander Gladysh Jun 18 '13 at 18:54
  • @AlexanderGladysh Historically, we haven't been too keen on "design me a system" style questions. It so happens that the problem here actually is solvable in a narrow enough scope, which is why I answered it. Not everyone agrees with me and Tom. – sysadmin1138 Jun 18 '13 at 19:00
  • Hmm. OK, well, is there a better place to ask this question? – Alexander Gladysh Jun 18 '13 at 19:01
  • @AlexanderGladysh ServerFault Chat seems to be the place open-ended questions like these end up. – sysadmin1138 Jun 18 '13 at 19:17
  • @sysadmin1138 Is that a good thing? Will Google find chat logs? Will they be readable enough to be a viable self-education material for other people?.. (That's off-topic, of course, so I'm shutting up. Thanks.) – Alexander Gladysh Jun 18 '13 at 19:20

2 Answers2

5

Here is one solution to what you're looking for. No java is involved in the making of this system, just readily available Open Source bits. The model presented here can work with other technologies than the ones I'm using as an example.

Scalable Upload

  1. Files are HTTP POSTed to a specific Round-Robin DNS address.
  2. The system POSTing the files then drops a job into an AMQP system (Rabbit MQ here), by way of another pair of load-balancers, to start the processing workflow.
  3. The Load Balancers receiving the HTTP POST are each in front of a group of OpenStack Swift object store servers.
    • The load-balancers each have two or more OpenStack Swift object-store servers behind them.
    • 'Round Robin is not HA' can be if the targets are HA themselves. YMMV.
    • For extra durability, the IPs in the RRDNS could be individual hot-standby LB clusters.
  4. The Object Store server that actually gets the POST delivers the file to a Gluster-based file-system.
    • The Gluster system should be both Distributed (a.k.a. sharded) and Replicated. This allows it to scale to silly densities.
  5. The AMQP system dispatches the first job, make the backup, to an available processing node.
  6. Processing node copies the file from main storage to backup storage and reports success/failure as needed.
    • Failure mode processing is not diagrammed here. Essentially, keep trying until it works. And if it never works, run through an exceptions process.
  7. Once the backup is complete AMQP then dispatches the Processing job to an available processing node.
  8. Processing node either pulls the file to its local file-system or processes it directly from Gluster.
  9. Processing node deposits processing product wherever that goes and reports success to AMQP.

This setup should be able to ingest files at extreme rates of speed given enough servers. Getting 10GbE aggregate ingestion speeds should be doable if you upsize it enough. Of course, processing that much data that fast will require even more servers in your Processing machine-class. This setup should scale up to a thousand nodes, and probably beyond (though how far depends on what, exactly, you're doing with all of this).

The deep engineering challenges will be in the workflow management process hidden inside the AMQP process. That's all software, and probably custom built to your system's demands. But it should be well fed with data!

sysadmin1138
  • 131,083
  • 18
  • 173
  • 296
3

Given that you've clarified that files will arrive via scp, I don't see any reason for the front-end server to exist at all, as the transport mechanism is something that can be redirected at layer 3.

I'd put an LVS director (pair) in front, with a processing server pool behind and a round-robin redirection policy. That makes it very easy to add and subtract servers to/from the pool, it increases reliability because there's no front-end server to fall over, and it means we don't have to address the pull/push question about getting the files from the front-end to the processing servers because there is no front-end.

Each pool server should then do two things when receiving a file - firstly, copy it to archival storage, then process the file and send it on its way.

MadHatter
  • 78,442
  • 20
  • 178
  • 229
  • 2
    What do you feel it lacks **given what's been asked**? If it fails only to address details which haven't been given in the question, then it's only not an answer if the question is not a question, surely? And you've made it very clear that you think that the question is a good one as it stands. – MadHatter Jun 18 '13 at 10:38
  • 1
    I just tend to ask questions about the question, as a comment on the question, but there we go. – Tom O'Connor Jun 18 '13 at 10:56
  • I rather agree with you; but since you canonised the question, I kind of feel you've at least beatified any answers fully based thereon ;-) – MadHatter Jun 18 '13 at 11:03
  • 2
    That would be an ecumenical matter. – Tom O'Connor Jun 18 '13 at 11:23
  • Thank you, @MadHatter, for your input. I added some info to the question. – Alexander Gladysh Jun 18 '13 at 19:07