Cheap but Highly Available Shared Storage?

Question

my group is in the market for plentiful storage. We traditionally used big-iron FC-attached SANs, but they are way expensive and provide much more performance than we require.

We want highly available shared storage, and if it performs like local desktop-class drives, that's fine.

Are there shared disk solutions out there that can get us slower but bigger disks?

We like NexSan SataBeast but they don't have a fully global presence which makes support awkward (we're a global company).

Is there any storage of this type that you would recommend? We'd prefer to use FC-attached storage but we're open to suggestions.

Thanks,

Wout.

EDIT: To answer the requests for clarification, we'd like

• globally available hardware support
• tens of TB with the ability to grow
• Highly Available: Ability to provide service through common failures, so:
  • hotpluggable drives
  • multipath IO
  • active/active controller
  • Anything like this that has a low cost/benefit ratio
• NFS only might be an option depending on HA-ness and performance
• Price: whatever it costs, but as little as possible

Basically I'm wondering what the 3D graph of price-performance-availability looks like currently. If there are sharp exponential curves, I'd be interested in the systems in those elbows.

As noted below, this could be done by a farm of PC-level hardware serving iSCSI and host-level software RAID6. This proves that whatever an EMC top-end storage array costs, it is probably too much.

Answer 1

How much storage do you need? If you want something in the 10's to 100's of Terabytes you could look at the Sun X4540 (Thumper). Solaris comes with an iSCSI target if you need block level storage or it can be used as a NAS with NFS or Samba.

For those not familiar with Sun's Thumper, it's a 4U 2-socket Opteron box with 48 SATA disks. You can have up to 48TB of disk on a single machine, although with RAID-5 volumes this would be somewhat smaller.

This blog posting discusses some of the ins and outs of using a Thumper as an iSCSI target.

Another option is the secondhand market. You can get re-certified SANs quite cheaply and many (even older ones such as Clariion CXs) have the option of slower SATA or FC-SATA disks. If you don't mind the power bills you can get second hand Symmetrix gear quite cheaply and that has no issues around licensing for the management software - you can operate it through the console on the controller.

Answer 2

Depends on your definition of "Cheap". But you could look at HP LeftHand Starter SANs. It's an all iSCSI solution but provides better performance the more you scale it, uses standard servers for the hardware, has solutions in SATA and SAS varieties to fit your budget and performance needs, and is supported by a global, good customer service, company. The boxes can be setup with no redundancy for maximum disk space (and minimal cost), or up to several layers of redundancy so your data basically cannot be lost short of an act of God.

The performance is comparable to FC, and even can even surpass FC the more you scale it.

Answer 3

Ahh, haven't got enough reputation points to leave comments yet, so it's a seperate post I'm afraid.

With regard to the HP Lefthand pricing, a good site to grab the pricing on the Lefthand kit without calling HP is the StorageMojo, the StorageMojo also has list prices for most other manufacturers.

You're looking at about £40k for a 2 node setup with a total of 24 x 450GB SAS disks (10TB raw capacity), including the basic hardware support for 3 years. Slap on another £6k if you want the Premium support. This gives you pretty good redundancy and all the software features like Thin Provisioning, SnapShots, Remote Copy, Multi site DR which normally cost extra. The nodes hardware are based on the HP ProLiant DL185 G5 server (so that's 12 disks per node).

It came in at half the cost of an equivalent NetApp solution.

I think with HP having bought them out, the LeftHand solution now offers pretty good bang for the buck with some proper Enterprise grade support behind it, important when things go pear shaped and you need assistance in the wee hours.

Answer 4

Take a look at the Xiotech Emprise 5000. With their sata "diskpac"s you can get 16TB in 3U with redundant controllers and battery backed cache.

If you're willing to drop FC for iSCSI Dell's MD3000i or Equalogic products are the cheapest "name brand" SAN.

And if you really wanna get your freak on, check out Coraid's AoE. Though I'm not sure to what degree they meet your global presence requirements.

Answer 5

We set this up for less than $AU3000 (so, probably halve that for the US market):

• Dual Core 2.4Ghz
• 2x Adaptec SATA RAID cards
• 4x Gigabit NICs
• 4U Rackmount Case
• SATA Hotswap backplanes for case (total of 10x trays)
• Loaded her up with off-the-shelf Seagate SATAs (with 5 year warranties, but see this SF question if you plan on doing that)
• Installed FreeNAS and used as an iSCSI target (although you have your choice of connectivity)

Worked like a charm, cheap massive storage with connectivy out the wazoo.

Answer 6

You can get an iSCSI interface for Netgear's ReadyNAS range. I think the Pro supports it natively but you can get a plugin for the lower end systems.

If you don't want special hardware, you can use a bog standard box, use Starwind as an iSCSI target and connect that way.

Answer 7

If you really want cheap I'd look into DRDB along with your favorit Linux distro and iscsi-target in combination with linux-ha.

All you need is time, depending on how expensive that is four you the next best thing that provides ISCSI along with RAID, but not highly available (I think they don't have DRDB or linux-ha option) is probably OpenFiler (Linux based) or FreeNAS (FreeBSD based)

Next in the row of really cheap is probably stuff like the Drobo Pro or other ISCSI enable "SOHO storage devices".

All of those options are cheaper than the classic storage providers - at least for the initial investment. Would be nice if you could elaborate what your definition of highly available is thou

Answer 8

Well, I have a Nexsan SATABeast and I can highly recommend it. Extremely stable, fully redunant (with 2 controllers) and the lowest price/TB if you need FC or would like to keep it in your existing FC SAN.

It's true that they dont have engineers all over the world, but they provide excellent and quick response by mail/phone. The closest engineer for me is about 200 metric miles away (half a business day away w/airplane), and it has so far worked out excellent for me.

Edit: The controllers also have iSCSI

Answer 9

You might want to check out Scale Computing. They have a clustered storage solution that will do iSCSI, NFS and CIFS at the same time. The low-end is a 3TB cluster, and you can scale it 1 or 2TB at a time, up to something like 2PB.

Answer 10

iSCSI san do you? We're getting good results out of the Dell Equalogic. Still maybe not what you'd call "cheap" but from what I can see, definitely cheaper than the same sort of thing on FC.

Answer 11

Look for Direct Attached Storage using 10-15 SATA drives. Something like the Dell Powervault MD1000. It expands to 15 drives per enclosure and you can use up to 6 of them. With up to 1TB per drive it should be enough to keep you going for years to come.

Answer 12

A super cheap SAN is the Dell MD3000i Line you can expand them with the MD1000 storage head. I think it maxes around 28 TB with the 1000 and 7200 RPM 1TB drives in a RAID 5 config (15 drive bays in each enclosure) for sub 40K probably. It's an iSCSI solution and has served us well. Right now if you want faster drives they go up to 600GB 15K drives and 750 GB 10k Drives.

Answer 13

In terms of your requirements, pretty much any supplier will be able to get you a system which will hit each of your bullet points.

Obviously you're worried about price, which means you're not looking at EMC, NetApp, etc, but there's still a lot of choice.

If you want all the functionality of the high-end arrays, look at the Falconstor NSS SA storage appliances, they're a lot cheaper than the EMC's of the world but still have pretty much every feature you could want. Otherwise look at the Dell EqualLogic PS5000E iSCSI Array if you want comprehensive support, scales up to 200TB of storage, performs well, and will do all the tasks you've listed above.

http://www.falconstor.com/en/pages/?pn=NSS http://www1.euro.dell.com/uk/en/business/Storage/dell_ps5000e/pd.aspx?refid=dell_ps5000e&s=bsd&cs=ukbsdt1

Ewan

Answer 14

Re running (Open)Solaris on an x4540 I had some experience last year in running it as a cheaper iSCSI target and it did not go according to plan: http://scaleordie.com/2008/07/06/attempts-at-using-a-sunfire-x4500-thumper-as-an-iscsi-san/

Chances are the code has matured a bit since -- that's in substance what the sun support engineers told us back then: you have to wait until the code gets better.

Answer 15

There is a company that I’m sure you will found interesting, the StarWind software company.

It’s a global provider of storage virtualization and Open iSCSI SAN solutions.

StarWind will allow converting Windows PC into an iSCSI SAN that you can use to share storage as well as provide disaster recovery. The idea is that you can take any existing Windows PC, install StarWind on it and use it's storage as a SAN.

www.starwindsoftware.com

Answer 16

I personally adore HP P2000 class SAN's. They're intro level, and give you the flexibility to use FC, iSCSI, or both depending on the controllers you want to outfit it with. You can also trick them out for storage (larger slower disks, up to 36TB/2U shelf) or performance (lots of fast, high IO disks - 24x 15k disks/shelf). They're pretty robust, and they can be scaled up on the fly by adding additional chassis. They're also pretty fully featured, though lacking some of the super fancy features you'd find on really high end storage. They may be worth checking out - we use them for a number of projects here because they're so flexible. The only catch is you'll have 1 or 2 controllers handling all of your storage, which can be a bottleneck if you want to run tons of fast storage behind it.