Latest interface: 0.3.1
Latest system: 010

1 posts

Posted on 9 November 2014 @ 17:03edited 17:14 15s
Hello everyone, ive been looking into expanding my storage for a while now. I currently have 8TB...but its not enough for what i need, here are the parts i have thus far and i will go into my questions after that :)

CPU's: Intel Xeon X5560 Quad core @ 2.80Ghz x2
CPU coolers: Noctua's (1366 Xeon)
RAM 24GB 1333Mhz DDR3 ECC
OS harddrives: RAID1 150GB raptors
HBA Cards: M1015 Crossflashed P16 IT Mode x3

Also possible use of SAS expander eventually.

My plan is to purchase x12 4TB hdds and create x2 6 hdd RAIDZ2's and then another x12 2TB hdd and create x2 6 hdd RAIDZ2's.
I.E: Group 1 x6 4TB hdds , Group 2 x6 4TB hdds , Group 3 x6 2TB hdds , Group 4 2TB hdds giving me 24 hdds total.

I plan to use x2 Rosewell LV-4500 cases as they have 15 3.5" internal bays, one case will house the motherboard, HBA's and 12 hdd's, this case will also have a PCI bracket that will take my internal SAS to external SAS. I will be using 2 PCI brackets for Internal SAS to external SAS going from case 1 to case 2 which will then connect to another 12 hdds. This was the easiest method so that in the event i need to replace a harddrive in one of the cases i can simple power it down, remove the external SAS cables, and then unrack the server to do the replacement. This would also allow me to in theory use a SAS expander or 2 ( 3? :D ) and purchase one or 2 more internal to external SAS brackets and connect to another case ( Case 3 ) and have another 12 harddrives.

Some background:

I have all the parts listed above already and have been doing testing on a small scale, only x6 500GB hdds for right now configured in RAIDZ2. I am currently in the process of finding SSD's that i like/will function the best with ZFS caching. The reason i have selected Ubuntu for the OS is i have 2 dual port 10GB NICS i will be using and they are not supported in FreeNAS which was my original choice...with that being said the cards are supported with Ubuntu, i will be create the ZFS arrays in Ubuntu and then serving them to my main server ( Windows 2012 R2) via a direct 10GB connection between the 2 servers, the ubuntu server will be serving the storage via iSCSI and the windows server 2012 R2 will be running iSCSI initiator.

The plan is to offload all the storage on my windows server that currently contains my movies, tv shows, ISO's , and VM's + VM data.

At the moment i have my iSCSI target serving my server a zvol that is located on the RAIDZ2 of x6 500GB hdds
My objective is to have ZFS as the underlying file system and NTFS ontop of it....Now here is where the fun begins!

1) Is using a zvol an appropriate way/solution to serve the volume i need with iscsi?
2) I plan to use RAID 1 or RAID10 for 2-4 SSD's for caching, i would like recommendations on what SSD's (Brands / make / model) and best practices
3) Is there any issues with how i plan to setup my ZFS server? I.E. Hardware choices, possible bottle necks , etc
4) I would like some all around general input on this build, positive , negative, neutral and constructive criticism is welcome :)

I know its alot to read so thanks for taking the time to read it all and i look forward to reading some replies :)

1199 posts

Posted on 12 November 2014 @ 19:32
ZFS platforms
I do not really like the choice for Ubuntu or other Linux-based ZFS implementation. But that is for your consideration.

You have opted for FreeNAS. But FreeNAS is lagging behind in terms of BSD system support. ZFSguru instead is quite ahead and offers 10.0, 10.1 and 11-CURRENT builds. You can use ZFSguru as you would use a vanilla BSD installation, even stripping away the ZFSguru specific configuration files and web-interface if you like.

The main advantages are true Root-on-ZFS implementation and mature ZFS implementation. FreeBSD has the best ZFS implementation available with features not found elsewhere.

I also do not like using ZFS as virtual harddrive container and using NTFS on top of an iSCSI-powered container, such as is the case with iSCSI. You lose part of the protection that ZFS offers and you do not have centralized storage anymore but rather you have SAN storage instead of NAS storage.

NAS is better because the filesystem is powered by the server, and can be shared to an infinite number of clients. The clients do not control the filesystem with NAS protocols like NFS and CIFS (Samba).

Have you considered NAS4Free or ZFSguru to use instead of using Linux?

I like Crucial MX100 because of their price and the protection mechanisms they employ. The power-safe capacitors are quite unique for a consumer SSD product and the RAID5 bitcorrection is a nice extra. The MX100 has 1/127 bitcorrection while the M500 has 1/16 bitcorrection. The latter is superior, but the MX100 is a tad cheaper and better available. The 512GB MX100 is also very fast with 400MB/s of sequential write, almost saturating the SATA/600.

But for L2ARC caching you do not need sequential performance. Any modern SSD will do. The MLC flash is good enough for some L2ARC pleasure, and the cheap price makes it not too bad to replace them after tough usage.

When you buy two SSDs, you can partition them to suit your needs. ZFSguru supports this very well; FreeNAS does not:

1) partition 10GiB-20GiB for the system (adjust space if you want service addons)
2) partition 4GiB for the sLOG (separate log device also called separate ZIL)
3) partition 40GiB for L2ARC
4) do not partition at least 25% of the SSD; or the best: 40%

So assuming you get two 128GB MX100 SSDs, you have:
1) a mirror pool of 10-20GiB for the operating system (ZFSguru supported Root-on-ZFS the best you can have)
2) a mirror of the 4GiB sLOG partitions meaning a mirrored sLOG device
3) a stripe of the two 40GiB L2ARC partitions meaning 80GiB total L2ARC
4) about 40% of overprovisioning; which you need if you want L2ARC and maintain high performance as well as make sure your SSDs do not burn through their write cycles that fast; write amplification will be very high if you use no overprovisioning!

Any ideas what kind of HDDs you want to use?
You do know about 4K optimized RAID-Z1/2/3 combinations? Generally 6-disk RAID-Z2 vdevs or 10-disk RAID-Z2 vdevs are the best configuration. Do not be afraid to span your disks across different controllers. You can utilise the onboard chipset-powered SATA just as well. Make sure your SSDs are on AHCI chipset SATA because of TRIM support and DIPM support.

I advise not to use any expanders but instead buy proper HBA controllers and have enough PCI-express to suit your expansion needs. A consumer board with 16 PCI-express lanes can have two 16-port HBA for total 32 drives along with the 6 onboard SATA offered for a total of 36 ports.

41 posts

Posted on 18 January 2015 @ 23:20
I just joined the forum as I'm trying to build my server running ZFSGuru. I have two M500/240GB SSDs which I thought I'm going to use in a root mirror, but I've also been debating with myself how I could use them also as l2arc/zil, at the same time. Searching around came across your dialogue, but i'm not clear what is the process to accomplish what CIPHER described (the web interface doesn't seem to offer that functionality). Would you mind giving additional details?
Thank you

1199 posts

Posted on 19 January 2015 @ 20:25
Partition your disks as discussed in my previous post. Then you have multiple partitions each with a name attached to them (the label name). It becomes then available as 'gpt/mylabelname'. Then you use that as an available disk when adding sLOG/L2ARC or creating a pool on the relevant pages in the Pools section.

If you need more help, let me know.

17 posts

Posted on 22 January 2015 @ 06:32
What type of 10 GE nics are you using ?
I have intel x520da2 and they work very well under freebsd. Only need some buffer tuning. This is well documented on freebsd support fprums.
Last Page

Valid XHTML 1.1