Latest interface: 0.3.1
Latest system: 010

75 posts

Posted on 28 June 2013 @ 20:02
I'm currently running 8 3tb spinning drives with an 80gb intel 320 for zil and 250gb samsung 830 as l2arc. I run one filesystem as nfs shared to a few esxi hosts as well as a few other filesystems for smb. Lately I've noticed my number of VM's have grown and my smb performance is lacking with all the VM's constantly chattering away. SMB can sustain about 25MB/s when I know it used to be double that. Its not the nic as the nfs is shared over a seperate nic.

Anyway, my though was to take 4 x samsung 840 pro 512gb ssd's and make a 1tb mirrored pair pool. That way I can run all my VMs on that (in total they are only 330gb now, I keep data mostly separate from the vms). I've read there is no need for a zil or l2arc and those can actually slow things down. Has anyone done an all ssd pool and have any advice for me? Pretty sure I need to put in another HBA as I don't think I have 4 ports available anyway.

75 posts

Posted on 19 July 2013 @ 14:13
as I still haven't bought anything, I'm gonna hold off until the new samsung 840 evo 1tb drives come out next month.

Anyone else done an all ssd pool?

75 posts

Posted on 7 February 2014 @ 14:53
Alright, I've bought my new toys and will be installing zfsguru soon.

Hardware (about $4300):
supermicro SC846B 4U 920W RPS Chassis
x9DRH-7TF-O (dual onboard 10g-BaseT)
2x xeon E5-2609
6x 16gb DDR3-1333 (should have gotten 8 to balance out the slots, whatever)
2x LSI 9211-8I
Drives (about $5300):
10x seagate 4tb sas constellation ST4000NM0023 (using 8 as 4 mirrored pairs, 2 for spare. I like maxing at 8 drives in the pool so I can move to a single lsi in a whitebox if things go bad)
2x Intel S3700 200gb zil/l2arc. I know the size is overkill for zil but the performance is sick
4x Samsung 840 EVO 1TB. This will be a raid10 pool just for nfs export for running VMs

With my current setup I have AD to windows server 2012 working great for smb shares, no folder redirection or anything, just permissions on shares.
By default I add tmux. I'm hoping this or something like this is included in the new 9.2 image. I'm currently running 9.0-004 web .2 beta 8

I have rsbackup to pull using rsync from other servers
I have zfs-periodic to make snapshots but now that I have it working with windows previous versions, I want to do something different to match the shadow copy name and I can't figure out how to match the hourly AND daily AND monthly right now. So my users can only see the daily versions without getting me to look back through the snapshot folders.
I then add a modified zfsrep set of scripts. The main changes I made were to facilitate the syncing of the periodic snapshots.
Recently I added rsyslog pushing to an elasticsearch vm and use kibana to view stuff. I have some of my sync logs pushing there, but working on adding more monitoring stuff. Hopefully I can get drive failures and other zfs alerts pushing this route as I'm starting to have too much stuff to login to each thing everyday and see if my pool is degraded or not.

If anyone has something better than zfs-periodic and zfsrep with mbuffer to snapshot and push to a second box, let me know.

I'll keep this thread updated as I go, and I'll be taking better notes and posting my scripts here as well. I have seen the trouble people have had doing some stuff I've pulled off like AD integration and my backup stuff so I'm happy to share what I've learned.

if anyone has any questions or comments please post and I'll try to reply in a timely fashion. This forum doesn't seem to have email alerts that I can find.

9 posts

Posted on 15 May 2015 @ 00:10

I know this is an old thread, but I guess it couldn't hurt to try and get in contact ;-)

I am currently in the process of upgrading a system of mine to host VMs (in favor of a crappy Open-E install I have now) and since you're doing a bit of the same, I was wondering if you could shed some light on some the questions I had:

- Is a second processor really beneficial to the I/O if money allows? I currently have 1x 2609, wondering if I should fill up the second socket.
- Are you using the memory as L2ARC? Any specific tuning settings you did to make use of the bigger memory? I only have 32G right now, but since the price of memory is so low, I guess it couldn't hurt to put in a multiple of this.
- Did you do any tuning on the 10Gbit nics? Jumbo frames? Also, curious as to which switch you are using.
- Lastly: what kind of random write performance are you getting on the ssd volumes from inside the VM's?

Nice setup, hope to read more in the future!

Kind regards,
Last Page

Valid XHTML 1.1