Expert advice needed,technology for network to disk recording, 1+ Gbps

Is there any other internal storage technology than RAM that can achieve 1Gigabit or more write performance coupled with concurrent reads?

The use case is the following:
Many (100's) threads capturing IP input and writing to a filesystem , combined IP ingest being in the 1Gbps+ range.

This will be "continuous" write,so i doubt SSD could fit the requirement.

My analysis is that RAM is the only internal device that can cope with this requirement , however i'm not an expert in storage area.

Moreover,Is there any filesystem issues to be aware of ?

dweeves

Posted 2010-12-07T17:59:54.683

Reputation: 111

Answers

PCI-Express based ssds could do the trick.

Short of that, you could try adding ssds to a pool to get enough throughput.

dhiltonp

Posted 2010-12-07T17:59:54.683

Reputation: 121

The problem i see with ssd is write cycle limit.So , i know ssds are fast enough, the problem is may they live long enough for 24/7 write/delete operations on 100's of files done each few seconds. – dweeves – 2010-12-09T10:16:39.753

Depending on how much space you need, if your motherboard can support more RAM in it (or an upgrade) you may want to look into maxing it out and then creating a RAM drive to store the active file(s). then just make sure it periodically copies to a HDD.

Xantec

Posted 2010-12-07T17:59:54.683

Reputation: 2 303

Most SAN's will easily fill a 1Gb/s connection. Many Direct Attach SCSI devices can do this as well, with a good quality raid card and multiple drives. You can do it, and its not hard, but its not going to be "cheap"

Actually, come to think of it.. Thats really only 125MB/s 8 SATA drives in a raid array should be able to do that.. Probably less, it would depend on the sustained write speed.

Brian

Posted 2010-12-07T17:59:54.683

Reputation: 2 934

My requirement is for INTERNAL storage, i know external storage can do that, however i'm trying to design an autonomous box that can match my requirements so external storage is not the main use case for me. – dweeves – 2010-12-09T10:20:33.153

1 Gigabit/s isn't really all that fast. A single HD can almost handle that for sequential transfers, and a 4-disk RAID-10 or a mid-range SSD would definitely work. If you're talking about 1 Gigabyte/s, then that's a whole different story.

Unfortunately for you, you've got a lot more problems to address and questions to answer before you can arrive at a usable solution.

How much data are you capturing?
How are you storing the data?
What are you doing with it?
How long do you have to keep it?
How is it being written to disc?
What's your budget?

Just saying "throw a RamSan at it" doesn't do you much good if it's too small or too expensive.

Other things that floated through my head: If "IP data" is packet capture, and you're talking about 1 GB/s, how are you getting it all into a single system to capture? If you're multiplexing data from several systems, are you using some sort of shared storage? If not, what do you really need?

afrazier

Posted 2010-12-07T17:59:54.683

Reputation: 21 316