Latest interface: 0.3.1
Latest system: 010

37 posts

Posted on 12 November 2014 @ 17:46edited 18:00 45s
Hello everyone.
My initial build had a 5x2TB on Raidz. Two yeras later, disks were replaced with 3TB ones. However, in less than two years the pool is almost full. So I've laid my possible solutions:

Option 1:
Replace current disks with 4TB, being the most reasonably priced.
Cons: will give only 4TB of free space that will pretty be full in less than two years again

Option 2:
Get 3 of 5TB or 6TB and add it to the current pool.
Pros: quick upgrade path
Cons: Not space efficient. I read that mixing different vdev sizes & disk count isn't recommended?

Option 3:
For 30% less the price, get another 5x3TB Raidz and attach it to the current pool
Pros: cheap! Lowest price to space ration. A lot of space that will last 4 years. Possiblity to upgrade 5 disks only at a time instead of 10 disks. Also, will be able to find a much larger and reasonably priced disks by then.
Cons: a lot of drives (heat, power?) don't know. Again, mixing different sizes isn't recommended? And having multiple vdev doesn't necessary mean better protection.

which lead me to the most preferred
Option 4:
Get another 5x3TB drives and make a 10x3TB Raidz2!
Pros: 1 pool, extra protection up to two drives failure.
Cons: a lot of drives to upgrade at once. And most importantly, I don't have space to backup my existing data!

So my thought is to create a Raidz2 using partitions of 5 drives. Migrate data and replace partitions with disks. Risky business.

What is your opinion overall, experience or suggestions.

1199 posts

Posted on 12 November 2014 @ 19:39
Hi redhams,

You can mix vdev capacity/diskcount/redundancy level just fine. But remember that adding a new (empty) vdev to a pool with a vdev that is almost full, is not very optimal. It is best to have vdevs that have similar level of free space as well as matching in performance level.

I myself have just bought WD60EZRX WD Green 6TB and i'm very happy with it. They are relatively cheap, they are very silent, they are quite fast (175MB/s) and very low power consumption; 3W for 6TB is great!

You talk about a RAID-Z2 of 5 drives, but that is not optimal. You should limit yourself to these configurations which are 4K optimal:

RAID-Z1: 3,5,9,17 disks
RAID-Z2: 4,6,10,18 disks
RAID-Z3: 5,7,11,19 disks

The most common are:
RAID-Z: 3 disks
RAID-Z2: 6 or 10 disks
RAID-Z3: 19 disks

I think the 10-disk RAID-Z2 is the best allround ZFS configuration, because you have low overhead (20%) and still double parity protection.

Optimal configuration mentioned above mean the best space efficiency when having 4K optimized performance level. So the best of both capacity and performance. Other configurations mean you have to choose between low performance or lower capacity. Capacity loss can be seen in this graph:

broken image

BEWARE: above graph lists the number of data disks. A RAID-Z with 5 disks means 4 data disks! So do not count the parity disks. Note the optimal number of power-of-two disks: 2, 4, 8, 16, 32, etc.

37 posts

Posted on 13 November 2014 @ 08:05edited 08:06 47s
Hello CiPHER,
thanks for your explanation. Actually it is pretty good that it should be put in documentation or linked in main page.

Here I'm interested to know more about the non-optimal part:
- Why adding a new vdev to an almost full vdev isn't optimal?
guessing here: each vdev has independent IOPS. So when comparing a pool with a full vdev and an empty one to another pool with enough space in both vdevs, then the write speed is halved. But read speed shouldn't be affected.
If that is the case then it isn't really bad for a home server. Is that correct? What other solutions or workarounds other than destroying and recreating the pool?

Currently what I'm looking into is either
- Option 3: add another 5x3TB Raidz to the current pool (same disk number and configuration)
- Option 4: get another 5x3TB, destroy the current pool and recreate a 10 disks Raidz2

The 3rd option is easy and straight forward. In the future I can upgrade a single vdev to 6TB. But coming back to the non-optimal part above: is having a pool with 5x3TB and 5x6TB a bad thing?

For clarification on Option 4, I wasn't planning to make Raidz2 of 5 disks. But I need to migrate the data from old pool to new pool. So I was thinking of splitting each drive into two partitions (1.5TB). That's 10 partitions. Then create a Raidz2 pool out of these partitions. Copy the data. Destroy old Raidz pool. Then replace 2nd partitions with drives from destroyed pool. That's a lot of replace and scrub command.
Now is there an advantage of having Raidz2 over a Raidz with two vdev to make it worthwhile the trouble? In the end they're the same number of disks and utilization (minus protection level, which I'm confident Raidz2 has the upper hand)

1199 posts

Posted on 13 November 2014 @ 15:23
The issue about 4K optimal pool configurations is also listed on the Pools->Create page on the ZFSguru web-interface.

Adding a new vdev to an almost full pool means that new data written will be mostly stored on the new vdev. This means you will have less performance than if your vdevs are filled about the same. Both IOps and throughput scale with more vdevs; but only if there is enough free space to put your data on all vdevs. In most cases a single vdev does provide enough performance, though. So it's not terrible, just less ideal.

But there is a workaround: move your data to a different filesystem. This means you write the data again. If there is enough space on all vdevs, it will be spread across all vdevs. So after expanding by adding a new vdev, free up enough space by deleting what you do not need. Then, when you have some free space on your old vdev, you can move data from tank/video filesystem to tank/video2 filesystem for example. Then, after moving everything, you can remove tank/video and rename video2 to video. The result is that all data will be written again, and by writing it again you will have placed it on all available vdevs and have optimal performance.

Option 4 is best because two times RAID-Z is pretty vulnerable, while RAID-Z2 even with 10 disks is pretty safe.

What you can do though: create RAID-Z2 with 10 member disks, but with 2 disks missing. This means you can copy data to the new RAID-Z2 with two disks missing. Those two disks can contain your old data. So what you can do:

1) buy 5 more disks
2) copy data from your current pool to 2 of the new disks, it has to fit on those two disks
3) destroy the old pool
4) create 10-disk pool with only 8 disks, by using 2 memory disks of similar size
5) after pool creation, DO NOT WRITE to the pool but instead reboot; this will remove the memory disks from the pool. Otherwise the memory will starve because you do not have 2*3TB of RAM memory.
6) copy data from the 2 disks to the new pool
7) add the 2 disks to the pool, making it complete instead of degraded

You can create memory disks on the Disks->Memory disks page.

37 posts

Posted on 14 November 2014 @ 12:04
I was already planning to move shared files to a new file system to keep things organized. So will go with a 2 vdev raidz setup.

Thanks for your time. It has been very helpful and informative.
Last Page

Valid XHTML 1.1