Is FreeNAS reliable?

Question

FreeNAS seems like a great product with a full checklist of features, even iSCSI. But how reliable is it? There are a few scary stories about lost data, for example here. Here is another example.

If you have used freeNAS for a longer period of time or even in a production setting, please share your experiences, good or bad. It would be great if you could also describe the setup, ie which hardware and features (software raid, zfs, iscsi etc) you are using.

Answer 1

I have been using freenas on a spare machine with 4x 1TB hard drives (2 raid 1's, so 2TB usable). It has been up 24/7 for 6 months.

I find it brilliant!

I tested many NAS's devices and only got a maximum of 10Mb/s on a gigabit port, and that was rare, typically it was around 3-4. My main reason for a device was to save energy, however 2x 2 drive nas's = more than a 80+% psu on a celeron system.

On freenas, I have a celeron based machine that cost me under £70, and on the internal 100Mb card, I can easily push 70Mb/s on samba.

The most expensive part was I bought a 4 drive enclosure to add/remove hard drives easily! Was a bit of a waste of money, but looks cool!

I can not complain at all about it and love the system. I did look at openfiler, but it seemed a bit OTT and freenas did what I needed...

To the others who recommended it, not saying Openfiler is bad, but freenas suited my needs perfectly, I boot the machine off of a USB stick and works well... The question was "is FreeNAS reliable" and my answer has to be yes.

The system is using software raid and even though the celeron is a single core 64 bit one, even during a raid rebuild + watching a HDTV episode across the network, it never goes above 60% cpu

To get it working, I downloaded the full iso, put a 1GB usb stick in my laptop, used usb pass through on Vmware Workstation and booted from the iso. I then used the install option and chose the USB stick. (You can do this on the actual machine and I have since however this was my first time using it and I couldn't find a blank cd!)

I put the usb stick in to the machine and booted. It worked fine first time!

Steps to actually get it usable as a nas were the following:

1. Go in to disk management and add each of the 4 drives.
2. Go to format and format all drives to software raid
3. Go to software raid and add disks 1 and 2, 3 and 4 to a new raid 1
4. Go to format and format both the new raid's to the standard os
5. Mount both raids
6. Set up Samba and choose both of the mount points as shares
7. Set up a couple of users

Then it was accessible over windows by \\ip and using the username and password I chose.

I will be looking at openfiler again soon as AD support is lacking a bit, however for a SOHO / domainless environment, you can not go wrong with freenas.

edit - Via request - Was to big to fit in comments

Answer 2

Reliability has different components to it. One is how reliable it is to go all the way with recommended hardware in mainstream configuration, set it up once, then NOT touch it, and watch how reliable things are. I'd rate FreeNAS pretty high on that.

Another is reliability taking into account various human error factors (non-standard configuration, experimenting, non-standard sequencse of changes in configuration, etc). I'd rate FreeNAS really low on that. It's so easy to shoot yourself in a foot and lose your data. You can read the forums to get the idea, but if you try it yourself, you'll have it too :)

As it was already mentioned above, FreeNAS seems to be really geared towards enterprise consumers and ignorant of home users and their problems. Speaking of expensive server grade hardware with have multiple vdevs tens of drives in each, FreeNAS server that manages it, and an operational engineer who manages the server.

If you are a home user or small business user willing to have simple file sharing over 3-9T array perhaps with all the nice ZFS redundancy and checksumming features, but you also want to save money and you're used to the idea that a fileserver can run on cheap hardware, don't even try FreeNAS, it's not cost-effective, unless you (a) want to make it a serious hobby project and it's fun for you to spend time on this, (b) you're ready to invest much more money on upgrading (or completely replacing) your setup as you figure out your hardware won't work.

I am one of such users, I've been trying FreeNAS for 2.5 months already. I've spent the entire Christmas and New Year's holidays on initial setup, plus a bunch of evenings and weekends. It's somewhat fun, but feels really dangerous. Due to bugs or too non-standard configuration, I've experienced loss of 1 drive in 4-drive Raid-Z some 5 times already, and during 2 of them I've hit unrecoverable read errors during resilvering. Most of that happened when I was detaching and re-importing volumes or reapplying encryption. I am somewhat worried that if 1 drive loss is possible due to noob-ish handling of server setup even without real hardware disk failure, you need Raid-Z2 or Raid-Z3 just to mitigate that. I've been very paranoid with continuous snapshotting to an external drive for backups, and it saved me multiple times. Though I did have a couple of unrecoverable read errors on external drive that I had to deal with, and I am guessing this is due to the fact I used USB 3.0 which is NOT recommended (read: it's not supported and there are possible bugs). I also had problems with an SSD drive I used for L2ARC reporting errors, which disappear after I remove and re-attach it, however, it was impossible to wipe it: I had to disable checks, wipe it with dd, and reboot so that OS would refresh its knowledge about disk metadata, only then it allowed me to re-partition the drive and re-add it. There are more things of this kind that I had to go through.

Common gotchas for new users:

1. FreeNAS forums have a list of recommended hardware (https://forums.freenas.org/index.php?threads/hardware-recommendations-read-this-first.23069/). You should be as close as possible to that, or you won't even get help on forums (negligence is punished).
2. ECC RAM is a must. It places restrictions on the type of motherboard and CPU you can use.
3. Server grade components is a strongly recommeded thing (incl. SAS card, NIC, drives). RAID cards should not be used in RAID mode, make sure to turn it off, in some cases it requires applying different firmware.
4. 8GB RAM is a true minimum (running on 4-6GB is barely possible, but chance of running into problems raise significantly - not only perf problems:)). As you enable more features or your data set grows, you'll need more. Look at how much your motherboard can support. I am using 24GB out of possible 32GB and I'd probably chose a different motherboard (to enable more growth) if I new all that. Dedupe needs 5GB of RAM per 1 TB of space.
5. Raid-Z(1) shouldn't be used at all. There are chances of a failure during resilvering reasons, but I don't think Raid-Z is truly supported. If you have 4 drives, go with Raid-Z2. Half of the space will be used on parity, that's okay.
6. Don't use USB disks. eSATA is a better option if you want external drives, but (a) make sure you're using server-grade SATA card, and make sure you detach your volume before unplugging the disk.
7. Don't plan on sharing FreeNAS box with any other functionality. Use a dedicated machine. Never run FreeNAS in a VM, unless you're an experienced professional in enterprise grade virtualization, you know what you're doing, or you just want a lot of fun (and possibly zero support on forums, people won't help you if they see you're using a VM).
8. Always backup your data, always backup your config (after every change), always backup your encryption keys incl. recovery keys (after every change). RAID redundancy is not a replacement for backups.
9. Permissions are hard, especially with Windows based systems, though Windows and CIFS is supported. Prepare for a lot of fun/mainteinance.
10. Don't expect home user specific bugs to be addressed, they're ignored with a smile. E.g. a complaint about drives not spinning down due to some unexpected disk activity is funny: enterprises run their servers 24/7 so you should do it, too. Then it stops being a problem.
11. ZFS is optimized for many (tens, hundreds) users accesing files simultaneously. Don't expect any optimizations for one or two user case (family, small office). E.g. there's no defragmentation other than recreating a dataset, because with hundreds of users it won't matter, what matters is IOPS and caching. Also default record size is 128K (compare to default cluster in Windows 4K) and it's better to not change it.
12. You'll find many people on forums generally advice against ZIL and L2ARC for home users, however I've found both ZIL and L2ARC very effective. ZIL maximizes write throughput to really saturate 1 GBit. L2ARC increases IOPS on repeated reads at least tenfold, and it enables very smooth experience accesing files from Windows, if you reboot your server rarely enough. Explorer does a lot of random/repeated reads to generate thumbnails/previews. Noticeable improvement in use cases like "watching photos straight from server directory".
13. Treat "updates" as "upgrades" with necessary babysitting, verification, etc. After last update I somehow lost my settings for automated snapshots, therefore replication to backup disk stopped working. Do configure all sorts of alerting, don't just assume things will work. FreeNAS can send e-mails.

Answer 3

I'm running FreeNAS 7 in a pair of office file servers. I use the built-in rsync to periodically back up the primary to an identical machine in the next rack over. And I have scripts that run remote backups to the rsync target on the FreeNAS box. It's pretty solid for iSCSI, NFS, and CIFS as well.

I went with FreeNAS primarily for hardware support... the OpenFiler release that was available didn't support the NIC or the storage adapter on my servers, nor did it support ZFS. I'm probably going to be upgrading to FreeNAS 8 soon (upgrading the backup, letting it sync, then making it the master, and upgrading the former-master).

I now see that this is an ancient thread, but if you're still in the SF Bay Area, one of the FreeNAS 8 team will be demonstrating the brand new release at BayLISA this Thursday (6/16/11) ... check out www.baylisa.org or www.meetup.com/baylisa for details (disclaimer: I'm the secretary of BayLISA so I like seeing lots of people show up).

Answer 4

I dont have direct direct experience with FreeNas, but I would suggest that you try OpenFiler instead. Like FreeNas, it is is Free and Unix based. OpenFiler is a more commercialized product with many commercial production implementations.

If you are about to commit data that you care about to this product please consider the following points:

1. As with all free software - use the most commercialized, supported and still free product. Think OpenSuse, MySQL, OpenFiler and so on.
2. As with all free software - trust but verify!
3. As with all NAS products - RAID is a must.

I also suggest that you host your OpenFiler on ESXi, and you will be able to share the machine with other VMs if the system is powerful enough and will not be heavily loaded by other VM's.

You can also load OpenFiler successfully onto Hyper-V - just setup the ethernet adapter as legacy adapter.

Good luck - if this helps please give me credits.

Answer 5

I've been running Freenas for a couple of years now. First i ran 3 disks in a RAID 5 with a few extras for temporary storage.

I really like the reliability of it, it's rock solid, and once it's set up, it's fantastic.

The basic setup is very easy, especially if you install to a hard drive or CF card (i chose the latter). However, i tried for a while to get it to boot from a USB thumbdrive, and gave up. There are now new instructions on how to get this to work, but my CF setup works fine, so i haven't tried again.

Adding drives, creating RAID arrays, etc is very simple. Setting up Samba/CIFS is also very easy. This means you can quickly set up a server that plays very well with a mixed Mac and PC network.

When you start to customise a bit more, you can run into trouble however. I recently tried to upgrade my Samba transfer speed by tweaking settings, and ended up having to reset to factory defaults and restoring the config from a backup (this is actually easy to do, but i don't think that it should have been a necessary step).

Freenas is a bit fussy about hardware, it doesn't seem to like the onboard Nvidia NIC that comes with my ASUS motherboard, as i said before, it doesn't make using a USB thumbdrive easy.

Recently i've upgraded the disks and changed to 1TB drives backing up from disk to disk using the included Rsync, as the whole RAID 5 thing scared me (i worried that if it went wrong, the risk of making a mistake and losing all my data was too great). BTW, Rsync is ridiculously easy, in contrast to my experiences with trying to use it with Windows.

Support is through the forum on Sourceforge, and if you don't ask questions in the right way, you will be ignored. People are very helpful if you make the effort.

Finally, is it worth the hassle? Absolutely, i have a reliable server that once set up runs and runs. I use it for backing up various machines, as well as holding my photo, music, and video libraries, which stream to various machines. I touch the config about once a year, to upgrade to a newer version, or add disks, etc, and between those times i never need to worry about it.

Answer 6

FREENAS is sublime!

I've been using FreeNAS 8 for about six months with pure joy. Still I haven't finished testing all functions yet, but till now all is working perfectly as described in the manuals.

Every single function can be configured by my grandmother without my help. The GUI is great and a VERY BIG + is the independence of hardware platforms.

I use CIFS, NFS, ISCSI and FTP. To get started I used the instruction videos on YouTube, succeeded on the first try and have managed to recreate the setup many times after for testing and demo purposes.

I recommend FREENAS to everyone and I do! Technology should be FREE for the world to make it a better place (and not make the few richer thus making many others poorer.)

REMARK: I very strongly feel that the product FREENAS should remain a shared storage solution. So please do not combine other functions like for instance QNAP does. It is elegantly the best storage solution I have encountered when searching for it for over 6 months.

My congrats with your achievements!!

Answer 7

For a while I ran OpenFiler against ESXi. I measured the IOPS I could produce locally (on the machine itself, against a raid volume) and remotely from an ESXi server over a software iScsi initiator and the results were within 95% of local performance -- and it scaled to 4 drives in a raid5 array.

YMMV -- but for me -- it provided near local performance over iSCSI, even with multiple servers attached (* only tested with 3 servers).

Answer 8

I chose FreeNAS instead of OpenFiler because of hardware requirements. FreeNAS basically needs less of everything. I was able to run a 1.28TB RAID1 configuration with just 256Mb of RAM on a 400MHz VIA board with a 128Mb CF card. It's just some really old hardware I had lying around. So far, it has been working well enough. No real complaints yet.

Answer 9

I have run FreeNAS boxes for nearly a year with good success. I generally use them as backup devices to local storage so performance isn't of the utmost important. The current systems I have are HPaq 2 GHz P4 with a single 1.5 TB drive. As I remember when we benchmarked about 6 months ago we could get something like 40 MB sustained over a GB connection using NFS. I have used iSCSI as well but not at this point.

I have recently configured a unit connected via NFS to 2 VMware hosts and so far my daily management guy can't see any performance issues.

We have experimented with more cutting-edge hardware using RAID and more drives. Getting newer hardware running has been difficult, so we may end up using Ubuntu for its excellent hardware support and OpenFiler on top.

Answer 10

I tried to use freenas under VMWare Fusion on my Mac Mini because I want to continue using ZFS and Apple dropped the ball on that one. However, the disk I'm using is a 1TB USB external disk and the FreeNas kernel can't talk to it, the kernel says the USB port times out. A small flash USB disk worked fine.

This was on the unstable v7 release (for zfs), but I can't shake the feeling that the FreeBSD kernel isn't as comfortable with various sorts of hardware as Linux.

I tried an OpenSolaris VM and it could see the disk just fine. I'm thinking about using that since knowing that ZFS keeps my data checksummed gives me peace of mind.

UPDATE: This post got me wondering, so I got around the USB problem by attaching the USB disk as a raw disk. I couldn't get that to work on VMWare Fusion, but I could make it work on Virtualbox 3.0.4.

To make a long story short: With FreeNAS 7RC1 I was able to mount the zpool and I was transferring files to my host systems over NFS at 7MB/s. I then used the same raw disk setup with OpenSolaris 11/08 and I was only able to get 4MB/s.

It's not a very professional test, I know :) The Solaris VM seems to be halting every so often, I'm not sure why.

So in conclusion: FreeNAS on a VirtualBox VM outperforms OpenSolaris by almost 100%. Works as an endorsement for me even though it was only the one test...

Answer 11

On 2 occasions now I have setup a freenas box with 2 - 1 tb drives as a software raid, copied all my data to the share, and then the power in my house went out from a storm, longer than a UPS would handle it, and each time, I have had to rebuild it from scratch. It kills the raid and there is nothing you can do to get it back... luckily I have rStudio recovery software that I was able to use to recover my data. It is all a very time consuming process and really annoying that you basically have to start over from scratch. If you use FreeNAS, it would be worth to have not only the raid, but also an external drive to copy all your data to because you could lose it all using FreeNAS. I am looking for an alternative solution for my file server.

Answer 12

Use FreeNAS from USB stick so saved another disk bay, and it work perfectly for years. Recently moved to 8 branch (clear install, then import volumes and re-setup - took me 2 hrs), still a bit 'too nice' interface, but still the same robustness.

USB flash is fairly good choice. Here are the steps:

1. Download .iso file.
2. Find .xz file in the iso, extract it (I used 7-zip).
3. Extract .xz file (7-zip helped again). The file you have now is an embedded version of FreeNAS.
4. Put it onto USB flash (at least 2Gb). Use Physdiskwrite to do that, run physdiskwrite -u image-file, and answer which disk you need to write info to. You can check the disk by PhysGUI.exe (in the same archive).
5. That's all, boot from that USB flash.

P.S. If something go messy (under Windows it may be so), you may want to clear USB stick. Use diskpart for that:

1. Open a command window as admin (cmd), type diskpart and press Enter.
2. Issue list disk command. Find out which disk you need.
3. Now do select disk # (replace # with the number of your drive).
4. Type clean and press Enter.

Answer 13

The only gotchas that I've run into in the past is that sometimes complex ACLs don't behave as expected, although I ran into this personally about two years ago, so it is very possible that those bugs are fixed.

If you have complex ACL requirements, I'd recommend testing.

Answer 14

I'd rather go for openfiler. It's even more complete than FreeNAS, and has an excellent reliability track.

Answer 15

I've been running a setup similar to sybreon's for over a year at the office with no problems. The only annoyance's I've had are with Samba user management (though I haven't tried new versions).

Answer 16

Maybe ok for lab, far from usable for production

1) so many bugs... and developers really don't care... As far as I know, most developers on this project are iXsystems employees and they focus on selling their hardware... This is the worst organized open source project I have seen. Developers keep closing tickets leaving bugs not fixed.. Reboot is a solution for some bugs but hey this is a storage server... You can't just randomly reboot it.... Spend 10 minutes on http://support.freenas.org/ then you probably don't want to waste your time on FreeNAS

2) FreeNAS is so insecure.. If you can't put it into a totally secured network (note internal != secure), don't use it... I doubt it was even designed with security in mind.

3) For CIFS, the permission management is totally a mess.... I can't find an easy way to allow each user to have their private storage and a public storage. And if you use it with AD, usually the configuration file cannot be correctly generated... The AD permission related features are so broken in the web interface reboot is the fix....And the performance can be awful (memory related issue I guess, i could hit 950mbps+ with iperf but CIFS write can get be as slow as a few mbps)....

4) For iSCSI, the performance is really just ok... I use it with ESXi and get a lot of warnings on degraded I/O (latency over 1s...). With Intel PRO/1000 PT Dual Port I expected a lot better....

Due to some bugs inherited from previous release, I have to manually edit the config file over ssh to enable the web interface every time I reboot the box. I guess that's some good security measure...

In short, if you have $, get EMC/NetApp If all you need is RAID, get a hardware RAID card and save the time for something that matters. If you don't, just use FreeBSD/Solaris to create a storage server. Then you can have a better picture of what might work.

IMO using FreeNAS for production is a good way to lose customers/get fired....