ISCSI and SAN scaling

Question

The Problem: I have a photography studio, and I ran into a problem when I just ran out of space on a monthly basis. Every month I just purchased additional HDD for a file server. now I have approx 120 TB of data, and 120 TB for backup. I am looking for a solution that scales well since this storage will double almost every week from now on due to adding a 16-bit 8k video to the storage which takes up a lot of space.

My theoretical solution: I looked at NAS devices and one thing that i don't like is if it fails - the business fails. So then I started researching an alternative and found SAN. which is a bit more complicated and sound like it is a lot pricier then NAS solution. The reason I want to go with SAN is that if one block of the storage fails I have other blocks still available (with mirroring backups).

The Question: So after reading on how to set up SAN with ISCSI, I got very confused. SAN is a network of storage devices that runs ISCSI software. A target is a storage unit and initiator is a client. Target appears to be mounted like a local hard drive. that is a simple part of it, but What about scaling this whole thing. for example, if I have 5 servers that ran ISCSI target software can I merge them all together to be seen as 1 storage block.

2) Can I partition that storage?

3) if yes, can I manage accpartitions?partitions ?

or I completely didn't get an idea about SAN and ISCSI protocol.

And yes, I don't have money for fibre channel sadly. just yet :) Thanks for your attention

Answer 1

iSCSI is a protocol for accessing block devices over a network. No more, no less. Redundancy, scaling, device management, and so on, are not the protocol's concern. The answers to all your questions trivially derive from that.

On the larger point, SAN and NAS are not concrete technologies, but rather they are vaguely-defined concepts, and not even mutually-exclusive. NAS equipment can have significant redundancy, and SANs often have painful single-points-of-failure. Don't focus so much on "do I have a NAS, or a SAN?" but rather "does this specific product meet my requirements?", regardless of what buzzwords the vendor has applied to the box.

Given your storage requirements, the business-critical nature of the storage, and the cost and complexity of large-scale storage equipment, I strongly recommend you engage the services of someone with experience in deploying and managing these types of systems. Getting it wrong will cost you either a bucket load of money for unnecessary equipment, or your business (or both!).

Answer 2

There are various solutions for re-using your existing storage in a SAN pool (storage virtualization or hyper virtualization). Recommendations are off-topic here, however.

What you're asking is all possible, but may not be reasonable or within your budget. As it is, your requirements are still quite foggy and I'd suggest you consider the various dimensions of your project first:

Scalability

As TessellatingHeckler has pointed out, "storage will double almost every week from now" quickly puts a six-to-eight digit price tag on the project for the period of just a few years. You need to make up your mind how much storage you really need. In a large scale, you will require hierarchical storage. Some space for current production, some for recent production (larger but slower), and some for archive (slow but unlimited).

Options for current production include 10k HDDs or SSDs, for recent production large 7k HDDs, and for archive very large archive HDDs, tape or cloud.

Availability

There's a wide range for availability options and cost. The essential questions:

• which level of redundancy do you require?
• how fast do you need to recover your data in case of failure? (=what is your backup/restore performance?)
• how close to production does the recovery data need to be? (=what is your backup frequency?)

If your data has significant cost, you not only need to protect from hardware failure but also from disaster (fire, water, power failure, ...), from software failure, user failure and malware.

Security

Storing your (archive) data in a cloud may easily tick off scalability, but raises some questions about availability and definitely needs to be considered for security - who has or can gain access to your data?

You need to select an adequate location for your storage and for your backup, and implement access to everything in a secure fashion.

Going forward

All in all, this is a serious project. I think you need to put it in the perspective of the next few, maybe five years. The complexities are high, especially when it comes to security. Some you've already realized.

I'd seriously suggest talking to a professional who can look at your scenario in depth and (hopefully) provide significant insight into a possible solution. Assuming you don't have that trusted professional at hand you might need to talk to two or three people to get different options. However, the better you've made up your mind about your requirements the more productive those talks will be.