Latest interface: 0.3.1
Latest system: 010
Pages: 1 2
Jason
Developer

806 posts

Posted on 2 January 2011 @ 02:50edited 02:50 49s
This thread is about the Benchmark feature of ZFSguru web-interface.

It can do benchmarking of your disks in various ZFS configurations, including RAID-Z and RAID-Z, and some nested configurations. These benchmarks are destructive, and thus destroy any existing data on your disks. They may also run for several hours in the default configuration, especially when you have many disks to test.

Use this thread to ask questions about the functionality of the benchmark feature. Please use other threads to discuss ZFS performance itself!
nick
User

2 posts

Posted on 9 January 2011 @ 04:16
Here is my current setup and some outputs of benchmarks I'm seeing:

Product name ZFSguru
Product version 0.1.7
Product build date 2011-Jan-01 @ 21:57 UTC
Distribution type rootonzfs
System version 8.2-001 (checksum: b16729227805deb5ceb6b2eb0d9b55ea)
FreeBSD version 8.2-RC1
Processor AMD Athlon(tm) II X2 250 Processor (3000.14-MHz K8-class CPU)
Processor architecture amd64 (64-bit)
Memory size 7.7 GiB, of which 7.5 GiB kernel memory (307.2 GiB max.)
ZFS version This system is currently running ZFS pool version 15.

OS: MicroCenter Store Brand USB 2.0 4gig (rootonzfs)
Pool: 5x HITACHI Deskstar 7K1000.C HDS721010CLA332 1TB 7200 RPM (RAID5)

Read throughput: 482.7 MB/s
Write throughput: 283.1 MB/s

-----------------------------------------------------------------------
CrystalDiskMark 3.0 x64 (C) 2007-2010 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
  • MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]


  • Sequential Read : 66.424 MB/s
    Sequential Write : 100.227 MB/s
    Random Read 512KB : 65.558 MB/s
    Random Write 512KB : 101.803 MB/s
    Random Read 4KB (QD=1) : 9.432 MB/s [ 2302.7 IOPS]
    Random Write 4KB (QD=1) : 9.521 MB/s [ 2324.6 IOPS]
    Random Read 4KB (QD=32) : 38.132 MB/s [ 9309.6 IOPS]
    Random Write 4KB (QD=32) : 31.128 MB/s [ 7599.5 IOPS]

    Test : 1000 MB [Z: 0.0% (0.0/3648.6 GB)] (x5)
    Date : 2011/01/08 21:14:28
    OS : Windows 7 [6.1 Build 7600] (x64)

    Much faster than my OpenIndiana install from last week; internal and over smb.
    Kristan
    User

    14 posts

    Posted on 3 February 2011 @ 10:58edited 11:00 55s
    I've gotten a very odd benchmark result :

    broken image
    broken image

    ZFSGURU-benchmark, version 1
    Test size: 32.000 gigabytes (GiB)
    Test rounds: 3
    Cooldown period: 2 seconds
    Sector size override: 4096 bytes
    Number of disks: 4 disks
    disk 1: gpt/Pool1.nop
    disk 2: gpt/Pool2.nop
    disk 3: gpt/Pool3.nop
    disk 4: gpt/Pool4.nop


  • Test Settings: TS32; SECT4096;

  • Tuning: KMEM=1.7g; AMIN=0.6g; AMAX=0.8g;

  • Stopping background processes: sendmail, moused, syslogd and cron

  • Stopping Samba service


  • Now testing RAID0 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 21 MiB/sec 20 MiB/sec 22 MiB/sec = 21 MiB/sec avg
    WRITE: 271 MiB/sec 258 MiB/sec 277 MiB/sec = 269 MiB/sec avg

    Now testing RAID0 configuration with 3 disks: cWmRd@cWmRd@cWmRd@
    READ: 21 MiB/sec 19 MiB/sec 21 MiB/sec = 20 MiB/sec avg
    WRITE: 217 MiB/sec 211 MiB/sec 206 MiB/sec = 211 MiB/sec avg

    Now testing RAID0 configuration with 2 disks: cWmRd@cWmRd@cWmRd@
    READ: 166 MiB/sec 178 MiB/sec 190 MiB/sec = 178 MiB/sec avg
    WRITE: 192 MiB/sec 190 MiB/sec 184 MiB/sec = 189 MiB/sec avg

    What on earth is going on with the read speed? And why only on the 3 and 4 disk tests? Sequential reads from a stipe should fly.

    Each drive benchmarked individually all give around the same speed (110-130Mib/sec) and there's no SMART problems reported so I don't think it's a bad drive. I'm a bit concerned as my intended configuration of RAIDZ seems to suffer from slower reads (though not as extreme as RAID0):

    Now testing RAIDZ configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 101 MiB/sec 107 MiB/sec 109 MiB/sec = 106 MiB/sec avg
    WRITE: 156 MiB/sec 154 MiB/sec 154 MiB/sec = 155 MiB/sec avg

    Now testing RAIDZ configuration with 3 disks: cWmRd@cWmRd@cWmRd@
    READ: 89 MiB/sec 90 MiB/sec 79 MiB/sec = 86 MiB/sec avg
    WRITE: 150 MiB/sec 147 MiB/sec 117 MiB/sec = 138 MiB/sec avg

    Now testing RAIDZ configuration with 2 disks: cWmRd@cWmRd@cWmRd@
    READ: 101 MiB/sec 91 MiB/sec 88 MiB/sec = 93 MiB/sec avg
    WRITE: 101 MiB/sec 102 MiB/sec 101 MiB/sec = 101 MiB/sec avg

    Any advice or suggestions on what to try?
    Jason
    Developer

    806 posts

    Posted on 3 February 2011 @ 22:59
    Those are odd results, but you have assigned very little memory to ZFS.

    This probably happened because you installed in a virtual environment with little RAM available. The automatic memory tuning selects values based on the memory available at that time. You can rerun this automatic tuning on the System->Tuning page. Simply click the "Reset to recommended values" and reboot the server.

    When trying to do the benchmarks again, you would see a warning about you didn't reboot yet and loadersettings have been contaminated. Ignore that error, it doesn't detect you rebooting properly yet. When you get your first new benchmark images, you will see them displaying higher KMEM and ARC values, and it should considerably improve your score.
    bed
    User

    20 posts

    Posted on 4 February 2011 @ 00:03edited 26 February 2011 @ 00:43
    Ok this might be a long post, hope it formats well enough.
    \

    All tests did not finish and I included the entire output text of the benchmark.

    My first series of tests was with the LSI 9260-8i presenting each disk as its own raid 0 array with various settings.

    broken image

    broken image


    ZFSGURU-benchmark, version 1
    Test size: 32.000 gigabytes (GiB)
    Test rounds: 3
    Cooldown period: 2 seconds
    Sector size override: 4096 bytes
    Number of disks: 8 disks
    disk 1: gpt/disk1.nop
    disk 2: gpt/disk2.nop
    disk 3: gpt/disk3.nop
    disk 4: gpt/disk4.nop
    disk 5: gpt/disk5.nop
    disk 6: gpt/disk6.nop
    disk 7: gpt/disk7.nop
    disk 8: gpt/disk8.nop

  • Test Settings: TS32; SECT4096;

  • Tuning: KMEM=12g; AMIN=4g; AMAX=6g;

  • Stopping background processes: sendmail, moused, syslogd and cron

  • Stopping Samba service


  • Now testing RAID0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 614 MiB/sec 587 MiB/sec 591 MiB/sec = 597 MiB/sec avg
    WRITE: 278 MiB/sec 309 MiB/sec 605 MiB/sec = 397 MiB/sec avg

    Now testing RAIDZ configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 468 MiB/sec 492 MiB/sec 469 MiB/sec = 477 MiB/sec avg
    WRITE: 437 MiB/sec 444 MiB/sec 448 MiB/sec = 443 MiB/sec avg

    Now testing RAIDZ2 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 417 MiB/sec 421 MiB/sec 424 MiB/sec = 421 MiB/sec avg
    WRITE: 381 MiB/sec 381 MiB/sec 389 MiB/sec = 384 MiB/sec avg

    Now testing RAID1 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 471 MiB/sec 477 MiB/sec 482 MiB/sec = 476 MiB/sec avg
    WRITE: 115 MiB/sec 126 MiB/sec 116 MiB/sec = 119 MiB/sec avg

    Now testing RAID1+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 468 MiB/sec 494 MiB/sec 479 MiB/sec = 481 MiB/sec avg
    WRITE: 472 MiB/sec 458 MiB/sec 482 MiB/sec = 471 MiB/sec avg

    Now testing RAIDZ+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 520 MiB/sec 443 MiB/sec 516 MiB/sec = 493 MiB/sec avg
    WRITE: 408 MiB/sec 413 MiB/sec 411 MiB/sec = 411 MiB/sec avg

    Now testing RAID0 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 409 MiB/sec 395 MiB/sec 381 MiB/sec = 395 MiB/sec avg
    WRITE: 497 MiB/sec 482 MiB/sec 482 MiB/sec = 487 MiB/sec avg

    Now testing RAID0 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 512 MiB/sec 523 MiB/sec 500 MiB/sec = 511 MiB/sec avg
    WRITE: 569 MiB/sec 580 MiB/sec 566 MiB/sec = 572 MiB/sec avg

    Now testing RAID0 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 581 MiB/sec 569 MiB/sec 557 MiB/sec = 569 MiB/sec avg
    WRITE: 618 MiB/sec 652 MiB/sec 612 MiB/sec = 628 MiB/sec avg

    Now testing RAID0 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 581 MiB/sec 588 MiB/sec 623 MiB/sec = 597 MiB/sec avg
    WRITE: 652 MiB/sec 667 MiB/sec 648 MiB/sec = 656 MiB/sec avg

    Now testing RAIDZ configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 333 MiB/sec 317 MiB/sec 313 MiB/sec = 321 MiB/sec avg
    WRITE: 337 MiB/sec 335 MiB/sec 351 MiB/sec = 341 MiB/sec avg

    Now testing RAIDZ configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 422 MiB/sec 425 MiB/sec 433 MiB/sec = 427 MiB/sec avg
    WRITE: 401 MiB/sec 400 MiB/sec 393 MiB/sec = 398 MiB/sec avg

    Now testing RAIDZ configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 441 MiB/sec 448 MiB/sec 432 MiB/sec = 440 MiB/sec avg
    WRITE: 423 MiB/sec 422 MiB/sec 424 MiB/sec = 423 MiB/sec avg

    Now testing RAIDZ configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 448 MiB/sec 432 MiB/sec 465 MiB/sec = 448 MiB/sec avg
    WRITE: 433 MiB/sec 446 MiB/sec 437 MiB/sec = 439 MiB/sec avg

    Now testing RAIDZ2 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 228 MiB/sec 227 MiB/sec 230 MiB/sec = 229 MiB/sec avg
    WRITE: 221 MiB/sec 216 MiB/sec 230 MiB/sec = 222 MiB/sec avg

    Now testing RAIDZ2 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 317 MiB/sec 332 MiB/sec 320 MiB/sec = 323 MiB/sec avg
    WRITE: 314 MiB/sec 325 MiB/sec 316 MiB/sec = 318 MiB/sec avg

    Now testing RAIDZ2 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 468 MiB/sec 438 MiB/sec 440 MiB/sec = 449 MiB/sec avg
    WRITE: 328 MiB/sec 344 MiB/sec 334 MiB/sec = 335 MiB/sec avg

    Now testing RAIDZ2 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 423 MiB/sec 480 MiB/sec 475 MiB/sec = 459 MiB/sec avg
    WRITE: 369 MiB/sec 370 MiB/sec 364 MiB/sec = 368 MiB/sec avg

    Now testing RAID1 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 293 MiB/sec 293 MiB/sec 296 MiB/sec = 294 MiB/sec avg
    WRITE: 113 MiB/sec 116 MiB/sec 125 MiB/sec = 118 MiB/sec avg

    Now testing RAID1 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 368 MiB/sec 354 MiB/sec 369 MiB/sec = 364 MiB/sec avg
    WRITE: 112 MiB/sec 123 MiB/sec 117 MiB/sec = 118 MiB/sec avg

    Now testing RAID1 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 416 MiB/sec 406 MiB/sec 417 MiB/sec = 413 MiB/sec avg
    WRITE: 114 MiB/sec 115 MiB/sec 112 MiB/sec = 113 MiB/sec avg

    Now testing RAID1 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 472 MiB/sec 466 MiB/sec 466 MiB/sec = 468 MiB/sec avg
    WRITE: 117 MiB/sec 115 MiB/sec 120 MiB/sec = 117 MiB/sec avg

    Now testing RAID1+0 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 357 MiB/sec 356 MiB/sec 367 MiB/sec = 360 MiB/sec avg
    WRITE: 256 MiB/sec 249 MiB/sec 240 MiB/sec = 248 MiB/sec avg

    Now testing RAID1+0 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 473 MiB/sec 482 MiB/sec 481 MiB/sec = 479 MiB/sec avg
    WRITE: 388 MiB/sec 363 MiB/sec 379 MiB/sec = 377 MiB/sec avg

    Now testing RAIDZ+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 499 MiB/sec 483 MiB/sec 489 MiB/sec = 321 MiB/sec avg
    WRITE: 416 MiB/sec 410 MiB/sec 419 MiB/sec = 341 MiB/sec avg

    Now testing RAID0 configuration with 1 disks: cWmRd@cWmRd@cWmRd@
    READ: 116 MiB/sec 116 MiB/sec 118 MiB/sec = 117 MiB/sec avg
    WRITE: 117 MiB/sec 125 MiB/sec 124 MiB/sec = 122 MiB/sec avg

    Now testing RAID0 configuration with 2 disks: cWmRd@cWmRd@cWmRd@
    READ: 237 MiB/sec 242 MiB/sec 240 MiB/sec = 240 MiB/sec avg
    WRITE: 257 MiB/sec 252 MiB/sec 250 MiB/sec = 253 MiB/sec avg

    Now testing RAID0 configuration with 3 disks: cWmRd@cWmRd@cWmRd@
    READ: 339 MiB/sec 334 MiB/sec 356 MiB/sec = 343 MiB/sec avg
    WRITE: 375 MiB/sec 387 MiB/sec 372 MiB/sec = 378 MiB/sec avg

    Now testing RAIDZ configuration with 2 disks: cWmRd@cWmRd@cWmRd@
    READ: 97 MiB/sec 97 MiB/sec 97 MiB/sec = 97 MiB/sec avg
    WRITE: 114 MiB/sec 117 MiB/sec 111 MiB/sec = 114 MiB/sec avg

    Now testing RAIDZ configuration with 3 disks: cWmRd@cWmRd@cWmRd@
    READ: 189 MiB/sec 196 MiB/sec 204 MiB/sec = 196 MiB/sec avg
    WRITE: 248 MiB/sec 246 MiB/sec 249 MiB/sec = 248 MiB/sec avg

    Now testing RAIDZ2 configuration with 3 disks: cWmRd@cWmRd@cW


    Next up is the same card with what I like to call "128k_ReadAhead_DirectIO_AlwaysWriteBack_CacheOn"

    broken image

    broken image


    ZFSGURU-benchmark, version 1
    Test size: 32.000 gigabytes (GiB)
    Test rounds: 3
    Cooldown period: 2 seconds
    Sector size override: 4096 bytes
    Number of disks: 8 disks
    disk 1: gpt/disk1.nop
    disk 2: gpt/disk2.nop
    disk 3: gpt/disk3.nop
    disk 4: gpt/disk4.nop
    disk 5: gpt/disk5.nop
    disk 6: gpt/disk6.nop
    disk 7: gpt/disk7.nop
    disk 8: gpt/disk8.nop

  • Test Settings: TS32; SECT4096;

  • Tuning: KMEM=12g; AMIN=4g; AMAX=6g;

  • Stopping background processes: sendmail, moused, syslogd and cron

  • Stopping Samba service


  • Now testing RAID0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 546 MiB/sec 534 MiB/sec 539 MiB/sec = 540 MiB/sec avg
    WRITE: 640 MiB/sec 646 MiB/sec 669 MiB/sec = 652 MiB/sec avg

    Now testing RAIDZ configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 481 MiB/sec 527 MiB/sec 475 MiB/sec = 494 MiB/sec avg
    WRITE: 456 MiB/sec 441 MiB/sec 456 MiB/sec = 451 MiB/sec avg

    Now testing RAIDZ2 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 430 MiB/sec 429 MiB/sec 435 MiB/sec = 432 MiB/sec avg
    WRITE: 379 MiB/sec 388 MiB/sec 381 MiB/sec = 382 MiB/sec avg

    Now testing RAID1 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 502 MiB/sec 507 MiB/sec 520 MiB/sec = 510 MiB/sec avg
    WRITE: 118 MiB/sec 116 MiB/sec 123 MiB/sec = 119 MiB/sec avg

    Now testing RAID1+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 557 MiB/sec 543 MiB/sec 535 MiB/sec = 545 MiB/sec avg
    WRITE: 484 MiB/sec 492 MiB/sec 478 MiB/sec = 484 MiB/sec avg

    Now testing RAIDZ+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 455 MiB/sec 488 MiB/sec 456 MiB/sec = 466 MiB/sec avg
    WRITE: 422 MiB/sec 417 MiB/sec 421 MiB/sec = 420 MiB/sec avg

    Now testing RAID0 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 388 MiB/sec 415 MiB/sec 389 MiB/sec = 397 MiB/sec avg
    WRITE: 501 MiB/sec 504 MiB/sec 494 MiB/sec = 499 MiB/sec avg

    Now testing RAID0 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 500 MiB/sec 477 MiB/sec 520 MiB/sec = 499 MiB/sec avg
    WRITE: 591 MiB/sec 598 MiB/sec 583 MiB/sec = 591 MiB/sec avg

    Now testing RAID0 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 570 MiB/sec 550 MiB/sec 571 MiB/sec = 564 MiB/sec avg
    WRITE: 614 MiB/sec 605 MiB/sec 625 MiB/sec = 614 MiB/sec avg

    Now testing RAID0 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 632 MiB/sec 601 MiB/sec 623 MiB/sec = 619 MiB/sec avg
    WRITE: 644 MiB/sec 637 MiB/sec 638 MiB/sec = 640 MiB/sec avg

    Now testing RAIDZ configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 327 MiB/sec 329 MiB/sec 317 MiB/sec = 324 MiB/sec avg
    WRITE: 341 MiB/sec 352 MiB/sec 344 MiB/sec = 346 MiB/sec avg

    Now testing RAIDZ configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 433 MiB/sec 441 MiB/sec 430 MiB/sec = 435 MiB/sec avg
    WRITE: 397 MiB/sec 407 MiB/sec 393 MiB/sec = 399 MiB/sec avg

    Now testing RAIDZ configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 458 MiB/sec 438 MiB/sec 447 MiB/sec = 447 MiB/sec avg
    WRITE: 417 MiB/sec 431 MiB/sec 427 MiB/sec = 425 MiB/sec avg

    Now testing RAIDZ configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 445 MiB/sec 443 MiB/sec 447 MiB/sec = 445 MiB/sec avg
    WRITE: 446 MiB/sec 446 MiB/sec 446 MiB/sec = 446 MiB/sec avg

    Now testing RAIDZ2 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 227 MiB/sec 231 MiB/sec 227 MiB/sec = 229 MiB/sec avg
    WRITE: 228 MiB/sec 225 MiB/sec 232 MiB/sec = 228 MiB/sec avg

    Now testing RAIDZ2 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 331 MiB/sec 315 MiB/sec 324 MiB/sec = 324 MiB/sec avg
    WRITE: 315 MiB/sec 317 MiB/sec 321 MiB/sec = 318 MiB/sec avg

    Now testing RAIDZ2 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 451 MiB/sec 447 MiB/sec 451 MiB/sec = 450 MiB/sec avg
    WRITE: 340 MiB/sec 337 MiB/sec 341 MiB/sec = 340 MiB/sec avg

    Now testing RAIDZ2 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 457 MiB/sec 476 MiB/sec 437 MiB/sec = 457 MiB/sec avg
    WRITE: 368 MiB/sec 368 MiB/sec 369 MiB/sec = 368 MiB/sec avg

    Now testing RAID1 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 300 MiB/sec 295 MiB/sec 289 MiB/sec = 295 MiB/sec avg
    WRITE: 120 MiB/sec 117 MiB/sec 109 MiB/sec = 115 MiB/sec avg

    Now testing RAID1 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 382 MiB/sec 377 MiB/sec 379 MiB/sec = 380 MiB/sec avg
    WRITE: 122 MiB/sec 113 MiB/sec 121 MiB/sec = 119 MiB/sec avg

    Now testing RAID1 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 421 MiB/sec 419 MiB/sec 418 MiB/sec = 419 MiB/sec avg
    WRITE: 123 MiB/sec 121 MiB/sec 122 MiB/sec = 122 MiB/sec avg

    Now testing RAID1 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 494 MiB/sec 478 MiB/sec 479 MiB/sec = 484 MiB/sec avg
    WRITE: 122 MiB/sec 114 MiB/sec 123 MiB/sec = 120 MiB/sec avg

    Now testing RAID1+0 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 372 MiB/sec 377 MiB/sec 359 MiB/sec = 369 MiB/sec avg
    WRITE: 251 MiB/sec 257 MiB/sec 255 MiB/sec = 254 MiB/sec avg

    Now testing RAID1+0 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 458 MiB/sec 479 MiB/sec 475 MiB/sec = 471 MiB/sec avg
    WRITE: 380 MiB/sec 378 MiB/sec 382 MiB/sec = 380 MiB/sec avg

    Now testing RAIDZ+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 493 MiB/sec 464 MiB/sec 528 MiB/sec = 324 MiB/sec avg
    WRITE: 414 MiB/sec 417 MiB/sec 411 MiB/sec = 346 MiB/sec avg

    Now testing RAID0 configuration with 1 disks: cWmRd@cWmRd@cWmRd@
    READ: 118 MiB/sec 118 MiB/sec 117 MiB/sec = 118 MiB/sec avg
    WRITE: 117 MiB/sec 121 MiB/sec 126 MiB/sec = 121 MiB/sec avg

    Now testing RAID0 configuration with 2 disks: cWmRd@cWmRd@cWmRd@
    READ: 233 MiB/sec 238 MiB/sec 234 MiB/sec = 235 MiB/sec avg
    WRITE: 51 MiB/sec 63 MiB/sec 58 MiB/sec = 57 MiB/sec avg

    Now testing RAID0 configuration with 3 disks: cWmRd@cWmRd@cWmRd@
    READ: 343 MiB/sec 345 MiB/sec 347 MiB/sec = 345 MiB/sec avg
    WRITE: 89 MiB/sec 88 MiB/sec 97 MiB/sec = 91 MiB/sec avg

    Now testing RAIDZ configuration with 2 disks: cWmRd@cWmRd@cWmRd@
    READ: 95 MiB/sec 97 MiB/sec 95 MiB/sec = 96 MiB/sec avg
    WRITE: 25 MiB/sec 25 MiB/sec 23 MiB/sec = 24 MiB/sec avg

    Now testing RAIDZ configuration with 3 disks: cWmRd@cWmRd@cWmRd@
    READ: 195 MiB/sec 195 MiB/sec 197 MiB/sec = 196 MiB/sec avg
    WRITE: 51 MiB/sec 53 MiB/sec 54 MiB/sec = 53 MiB/sec avg


    Next up is the same card with what I call "128k_ReadAhead_CachedIO_AlwaysWriteBack_CacheOn"

    broken image

    broken image

    ZFSGURU-benchmark, version 1
    Test size: 32.000 gigabytes (GiB)
    Test rounds: 3
    Cooldown period: 2 seconds
    Sector size override: 4096 bytes
    Number of disks: 8 disks
    disk 1: gpt/disk1.nop
    disk 2: gpt/disk2.nop
    disk 3: gpt/disk3.nop
    disk 4: gpt/disk4.nop
    disk 5: gpt/disk5.nop
    disk 6: gpt/disk6.nop
    disk 7: gpt/disk7.nop
    disk 8: gpt/disk8.nop

  • Test Settings: TS32; SECT4096;

  • Tuning: KMEM=12g; AMIN=4g; AMAX=6g;

  • Stopping background processes: sendmail, moused, syslogd and cron

  • Stopping Samba service


  • Now testing RAID0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 630 MiB/sec 636 MiB/sec 628 MiB/sec = 631 MiB/sec avg
    WRITE: 247 MiB/sec 259 MiB/sec 259 MiB/sec = 255 MiB/sec avg

    Now testing RAIDZ configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 445 MiB/sec 479 MiB/sec 478 MiB/sec = 467 MiB/sec avg
    WRITE: 227 MiB/sec 235 MiB/sec 271 MiB/sec = 244 MiB/sec avg

    Now testing RAIDZ2 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 424 MiB/sec 472 MiB/sec 432 MiB/sec = 443 MiB/sec avg
    WRITE: 379 MiB/sec 384 MiB/sec 382 MiB/sec = 381 MiB/sec avg

    Now testing RAID1 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 519 MiB/sec 500 MiB/sec 509 MiB/sec = 510 MiB/sec avg
    WRITE: 116 MiB/sec 126 MiB/sec 121 MiB/sec = 121 MiB/sec avg

    Now testing RAID1+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 475 MiB/sec 551 MiB/sec 562 MiB/sec = 529 MiB/sec avg
    WRITE: 479 MiB/sec 492 MiB/sec 484 MiB/sec = 485 MiB/sec avg

    Now testing RAIDZ+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 501 MiB/sec 459 MiB/sec 464 MiB/sec = 474 MiB/sec avg
    WRITE: 415 MiB/sec 419 MiB/sec 421 MiB/sec = 418 MiB/sec avg

    Now testing RAID0 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 405 MiB/sec 389 MiB/sec 385 MiB/sec = 393 MiB/sec avg
    WRITE: 492 MiB/sec 507 MiB/sec 490 MiB/sec = 496 MiB/sec avg

    Now testing RAID0 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 529 MiB/sec 506 MiB/sec 524 MiB/sec = 520 MiB/sec avg
    WRITE: 565 MiB/sec 577 MiB/sec 578 MiB/sec = 573 MiB/sec avg

    Now testing RAID0 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 587 MiB/sec 576 MiB/sec 578 MiB/sec = 580 MiB/sec avg
    WRITE: 624 MiB/sec 642 MiB/sec 667 MiB/sec = 644 MiB/sec avg

    Now testing RAID0 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 589 MiB/sec 604 MiB/sec 603 MiB/sec = 599 MiB/sec avg
    WRITE: 665 MiB/sec 662 MiB/sec 668 MiB/sec = 665 MiB/sec avg

    Now testing RAIDZ configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 333 MiB/sec 311 MiB/sec 339 MiB/sec = 328 MiB/sec avg
    WRITE: 350 MiB/sec 349 MiB/sec 350 MiB/sec = 350 MiB/sec avg

    Now testing RAIDZ configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 460 MiB/sec 423 MiB/sec 431 MiB/sec = 438 MiB/sec avg
    WRITE: 406 MiB/sec 397 MiB/sec 400 MiB/sec = 401 MiB/sec avg

    Now testing RAIDZ configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 486 MiB/sec 431 MiB/sec 439 MiB/sec = 452 MiB/sec avg
    WRITE: 427 MiB/sec 422 MiB/sec 427 MiB/sec = 426 MiB/sec avg

    Now testing RAIDZ configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 442 MiB/sec 445 MiB/sec 445 MiB/sec = 444 MiB/sec avg
    WRITE: 439 MiB/sec 442 MiB/sec 440 MiB/sec = 440 MiB/sec avg

    Now testing RAIDZ2 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 226 MiB/sec 230 MiB/sec 227 MiB/sec = 227 MiB/sec avg
    WRITE: 237 MiB/sec 224 MiB/sec 234 MiB/sec = 232 MiB/sec avg

    Now testing RAIDZ2 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 313 MiB/sec 322 MiB/sec 315 MiB/sec = 317 MiB/sec avg
    WRITE: 318 MiB/sec 321 MiB/sec 311 MiB/sec = 317 MiB/sec avg

    Now testing RAIDZ2 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 445 MiB/sec 477 MiB/sec 440 MiB/sec = 454 MiB/sec avg
    WRITE: 338 MiB/sec 340 MiB/sec 353 MiB/sec = 344 MiB/sec avg

    Now testing RAIDZ2 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 433 MiB/sec 438 MiB/sec 435 MiB/sec = 435 MiB/sec avg
    WRITE: 368 MiB/sec 366 MiB/sec 371 MiB/sec = 368 MiB/sec avg

    Now testing RAID1 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 297 MiB/sec 300 MiB/sec 301 MiB/sec = 299 MiB/sec avg
    WRITE: 113 MiB/sec 122 MiB/sec 115 MiB/sec = 117 MiB/sec avg

    Now testing RAID1 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 376 MiB/sec 370 MiB/sec 350 MiB/sec = 365 MiB/sec avg
    WRITE: 123 MiB/sec 116 MiB/sec 115 MiB/sec = 118 MiB/sec avg

    Now testing RAID1 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 421 MiB/sec 420 MiB/sec 416 MiB/sec = 419 MiB/sec avg
    WRITE: 118 MiB/sec 116 MiB/sec 124 MiB/sec = 119 MiB/sec avg

    Now testing RAID1 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 496 MiB/sec 486 MiB/sec 472 MiB/sec = 485 MiB/sec avg
    WRITE: 116 MiB/sec 119 MiB/sec 118 MiB/sec = 117 MiB/sec avg

    Now testing RAID1+0 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 358 MiB/sec 370 MiB/sec 338 MiB/sec = 356 MiB/sec avg
    WRITE: 254 MiB/sec 250 MiB/sec 266 MiB/sec = 257 MiB/sec avg

    Now testing RAID1+0 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 491 MiB/sec 494 MiB/sec 485 MiB/sec = 490 MiB/sec avg
    WRITE: 370 MiB/sec 379 MiB/sec 380 MiB/sec = 376 MiB/sec avg

    Now testing RAIDZ+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 457 MiB/sec 470 MiB/sec 458 MiB/sec = 328 MiB/sec avg
    WRITE: 414 MiB/sec 412 MiB/sec 419 MiB/sec = 350 MiB/sec avg

    Now testing RAID0 configuration with 1 disks: cWmRd@cWmRd@cWmRd@
    READ: 120 MiB/sec 119 MiB/sec 116 MiB/sec = 118 MiB/sec avg
    WRITE: 128 MiB/sec 120 MiB/sec 111 MiB/sec = 120 MiB/sec avg

    Now testing RAID0 configuration with 2 disks: cWmRd@cWmRd@cWmRd@
    READ: 230 MiB/sec 238 MiB/sec 235 MiB/sec = 235 MiB/sec avg
    WRITE: 56 MiB/sec 59 MiB/sec 59 MiB/sec = 58 MiB/sec avg

    Now testing RAID0 configuration with 3 disks: cWmRd@cWmRd@cWmRd@
    READ: 351 MiB/sec 342 MiB/sec 343 MiB/sec = 345 MiB/sec avg
    WRITE: 90 MiB/sec 88 MiB/sec 91 MiB/sec = 90 MiB/sec avg

    Now testing RAIDZ configuration with 2 disks: cWmRd@cWmRd@cWmRd@
    READ: 97 MiB/sec 96 MiB/sec 96 MiB/sec = 96 MiB/sec avg
    WRITE: 24 MiB/sec 25 MiB/sec 25 MiB/sec = 25 MiB/sec avg

    Now testing RAIDZ configuration with 3 disks: cWmRd@cWmRd@cWmRd@
    READ: 188 MiB/sec 193 MiB/sec 197 MiB/sec = 192 MiB/sec avg
    WRITE: 51 MiB/sec 51 MiB/sec 53 MiB/sec = 52 MiB/sec avg

    Now testing RAIDZ2 configuration with 3 disks: cW


    Next up I put in an Intel SASUC8i (All default controller settings)

    First test is without any tuning directly off the live CD.
    broken image

    broken image

    ZFSGURU-benchmark, version 1
    Test size: 32.000 gigabytes (GiB)
    Test rounds: 3
    Cooldown period: 2 seconds
    Sector size override: 4096 bytes
    Number of disks: 8 disks
    disk 1: gpt/disk1.nop
    disk 2: gpt/disk2.nop
    disk 3: gpt/disk3.nop
    disk 4: gpt/disk4.nop
    disk 5: gpt/disk5.nop
    disk 6: gpt/disk6.nop
    disk 7: gpt/disk7.nop
    disk 8: gpt/disk8.nop

  • Test Settings: TS32; SECT4096;

  • Tuning: none

  • Stopping background processes: sendmail, moused, syslogd and cron

  • Stopping Samba service


  • Now testing RAID0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 647 MiB/sec 639 MiB/sec 675 MiB/sec = 654 MiB/sec avg
    WRITE: 184 MiB/sec 194 MiB/sec 191 MiB/sec = 190 MiB/sec avg

    Now testing RAIDZ configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 491 MiB/sec 458 MiB/sec 459 MiB/sec = 469 MiB/sec avg
    WRITE: 146 MiB/sec 149 MiB/sec 153 MiB/sec = 149 MiB/sec avg

    Now testing RAIDZ2 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 417 MiB/sec 448 MiB/sec 412 MiB/sec = 426 MiB/sec avg
    WRITE: 133 MiB/sec 128 MiB/sec 130 MiB/sec = 130 MiB/sec avg

    Now testing RAID1 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 514 MiB/sec 490 MiB/sec 526 MiB/sec = 510 MiB/sec avg
    WRITE: 23 MiB/sec 22 MiB/sec 22 MiB/sec = 22 MiB/sec avg

    Now testing RAID1+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 635 MiB/sec 641 MiB/sec 623 MiB/sec = 633 MiB/sec avg
    WRITE: 103 MiB/sec 101 MiB/sec 103 MiB/sec = 102 MiB/sec avg

    Now testing RAIDZ+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 547 MiB/sec 544 MiB/sec 548 MiB/sec = 546 MiB/sec avg
    WRITE: 140 MiB/sec 142 MiB/sec 141 MiB/sec = 141 MiB/sec avg

    Now testing RAID0 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 457 MiB/sec 460 MiB/sec 447 MiB/sec = 455 MiB/sec avg
    WRITE: 101 MiB/sec 98 MiB/sec 103 MiB/sec = 101 MiB/sec avg

    Now testing RAID0 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 548 MiB/sec 565 MiB/sec 551 MiB/sec = 555 MiB/sec avg
    WRITE: 127 MiB/sec 128 MiB/sec 126 MiB/sec = 127 MiB/sec avg

    Now testing RAID0 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 651 MiB/sec 621 MiB/sec 630 MiB/sec = 634 MiB/sec avg
    WRITE: 156 MiB/sec 150 MiB/sec 148 MiB/sec = 151 MiB/sec avg

    Now testing RAID0 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 682 MiB/sec 686 MiB/sec 720 MiB/sec = 696 MiB/sec avg
    WRITE: 176 MiB/sec 170 MiB/sec 168 MiB/sec = 172 MiB/sec avg

    Now testing RAIDZ configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 344 MiB/sec 334 MiB/sec 331 MiB/sec = 336 MiB/sec avg
    WRITE: 68 MiB/sec 68 MiB/sec 67 MiB/sec = 68 MiB/sec avg

    Now testing RAIDZ configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 392 MiB/sec 451 MiB/sec 402 MiB/sec = 415 MiB/sec avg
    WRITE: 91 MiB/sec 95 MiB/sec 89 MiB/sec = 92 MiB/sec avg

    Now testing RAIDZ configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 476 MiB/sec 509 MiB/sec 489 MiB/sec = 491 MiB/sec avg
    WRITE: 110 MiB/sec 105 MiB/sec 108 MiB/sec = 108 MiB/sec avg

    Now testing RAIDZ configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 452 MiB/sec 430 MiB/sec 502 MiB/sec = 461 MiB/sec avg
    WRITE: 129 MiB/sec 129 MiB/sec 123 MiB/sec = 127 MiB/sec avg

    Now testing RAIDZ2 configuration with 4 disks: cW


    Next up is the Intel SASUC8i again but installed a usb stick using default tunings

    broken image

    broken image

    ZFSGURU-benchmark, version 1
    Test size: 32.000 gigabytes (GiB)
    Test rounds: 3
    Cooldown period: 2 seconds
    Sector size override: 4096 bytes
    Number of disks: 8 disks
    disk 1: gpt/disk1.nop
    disk 2: gpt/disk2.nop
    disk 3: gpt/disk3.nop
    disk 4: gpt/disk4.nop
    disk 5: gpt/disk5.nop
    disk 6: gpt/disk6.nop
    disk 7: gpt/disk7.nop
    disk 8: gpt/disk8.nop

  • Test Settings: TS32; SECT4096;

  • Tuning: KMEM=12g; AMIN=4g; AMAX=6g;

  • Stopping background processes: sendmail, moused, syslogd and cron

  • Stopping Samba service


  • Now testing RAID0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 712 MiB/sec 754 MiB/sec 722 MiB/sec = 729 MiB/sec avg
    WRITE: 201 MiB/sec 185 MiB/sec 203 MiB/sec = 196 MiB/sec avg

    Now testing RAIDZ configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 476 MiB/sec 476 MiB/sec 471 MiB/sec = 474 MiB/sec avg
    WRITE: 150 MiB/sec 147 MiB/sec 152 MiB/sec = 150 MiB/sec avg

    Now testing RAIDZ2 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 433 MiB/sec 408 MiB/sec 402 MiB/sec = 414 MiB/sec avg
    WRITE: 127 MiB/sec 122 MiB/sec 126 MiB/sec = 125 MiB/sec avg

    Now testing RAID1 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 484 MiB/sec 511 MiB/sec 498 MiB/sec = 498 MiB/sec avg
    WRITE: 22 MiB/sec 24 MiB/sec 21 MiB/sec = 22 MiB/sec avg

    Now testing RAID1+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 618 MiB/sec 591 MiB/sec 603 MiB/sec = 604 MiB/sec avg
    WRITE: 96 MiB/sec 93 MiB/sec 92 MiB/sec = 94 MiB/sec avg

    Now testing RAIDZ+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 543 MiB/sec 518 MiB/sec 530 MiB/sec = 530 MiB/sec avg
    WRITE: 137 MiB/sec 146 MiB/sec 140 MiB/sec = 141 MiB/sec avg

    Now testing RAID0 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 455 MiB/sec 460 MiB/sec 444 MiB/sec = 453 MiB/sec avg
    WRITE: 100 MiB/sec 113 MiB/sec 98 MiB/sec = 104 MiB/sec avg

    Now testing RAID0 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 543 MiB/sec 545 MiB/sec 560 MiB/sec = 550 MiB/sec avg
    WRITE: 125 MiB/sec 127 MiB/sec 124 MiB/sec = 126 MiB/sec avg

    Now testing RAID0 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 636 MiB/sec 630 MiB/sec 625 MiB/sec = 630 MiB/sec avg
    WRITE: 153 MiB/sec 160 MiB/sec 150 MiB/sec = 154 MiB/sec avg

    Now testing RAID0 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 649 MiB/sec 692 MiB/sec 713 MiB/sec = 685 MiB/sec avg
    WRITE: 166 MiB/sec 174 MiB/sec 188 MiB/sec = 176 MiB/sec avg

    Now testing RAIDZ configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 354 MiB/sec 348 MiB/sec 334 MiB/sec = 345 MiB/sec avg
    WRITE: 75 MiB/sec 70 MiB/sec 73 MiB/sec = 73 MiB/sec avg

    Now testing RAIDZ configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 458 MiB/sec 421 MiB/sec 443 MiB/sec = 441 MiB/sec avg
    WRITE: 105 MiB/sec 100 MiB/sec 90 MiB/sec = 99 MiB/sec avg

    Now testing RAIDZ configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 421 MiB/sec 427 MiB/sec 456 MiB/sec = 435 MiB/sec avg
    WRITE: 118 MiB/sec 113 MiB/sec 116 MiB/sec = 115 MiB/sec avg

    Now testing RAIDZ configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 482 MiB/sec 444 MiB/sec 450 MiB/sec = 459 MiB/sec avg
    WRITE: 138 MiB/sec 146 MiB/sec 140 MiB/sec = 141 MiB/sec avg

    Now testing RAIDZ2 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 227 MiB/sec 225 MiB/sec 224 MiB/sec = 225 MiB/sec avg
    WRITE: 48 MiB/sec 46 MiB/sec 42 MiB/sec = 45 MiB/sec avg

    Now testing RAIDZ2 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 324 MiB/sec 330 MiB/sec 326 MiB/sec = 326 MiB/sec avg
    WRITE: 75 MiB/sec 68 MiB/sec 75 MiB/sec = 73 MiB/sec avg

    Now testing RAIDZ2 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 481 MiB/sec 476 MiB/sec 504 MiB/sec = 487 MiB/sec avg
    WRITE: 92 MiB/sec 96 MiB/sec 93 MiB/sec = 94 MiB/sec avg

    Now testing RAIDZ2 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 440 MiB/sec 462 MiB/sec 437 MiB/sec = 446 MiB/sec avg
    WRITE: 110 MiB/sec 119 MiB/sec 108 MiB/sec = 112 MiB/sec avg

    Now testing RAID1 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 305 MiB/sec 310 MiB/sec 307 MiB/sec = 308 MiB/sec avg
    WRITE: 24 MiB/sec 23 MiB/sec 22 MiB/sec = 23 MiB/sec avg

    Now testing RAID1 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 372 MiB/sec 376 MiB/sec 378 MiB/sec = 375 MiB/sec avg
    WRITE: 23 MiB/sec 22 MiB/sec 23 MiB/sec = 23 MiB/sec avg

    Now testing RAID1 configuration with 6 disks: cWmRd@cW

    Jason
    Developer

    806 posts

    Posted on 4 February 2011 @ 02:55
    Quite interesting, if not confusing!

    Your hardware should be capable of very decent speeds and at times it manages to show that, but i also see unexplained wild dips to 20-ish MB/s write throughput, which is terrible and means some issue is capping your writes.

    Can you think of any reason why that might be? Was the system unused during benchmarking? Are there any disks in the array that perform worse? Are you running the Intel controller in IT-firmware mode? Did you try without any controller readahead? (Especially not 128K - ZFS can readahead much more intelligently; or should be able to).
    bed
    User

    20 posts

    Posted on 4 February 2011 @ 03:14
    I agree its confusing.

    All drives are equal in every way. I have run a full surface scan with estools from samsung to verify drive integrity. System was completely unused during testing (closed browser during tests while I slept). This controller is new and running firmware it came with, dated 2008 If I recall correctly. Did not see any real options to change the config in the bios of the card. If there is a better firmware to use I can find it and flash it.
    The tests that have 128k in them mean the stripe on the 9260 was using a 128k stripe. Performance was much better using the 9260 but as you know I have to create each disk as an array since that controller does not support JBOD.
    For the intel card its using factory defaults.

    Let me know if you want me to test anything (or anyway) in particular.
    Jason
    Developer

    806 posts

    Posted on 4 February 2011 @ 14:29
    You could check whether the Intel controller was using IT mode. If not, you can use the SuperMicro/LSI firmware to flash that controller and make it a non-RAID controller working in IT mode. The site servethehome.com(ext) has a document describing that procedure, might be a good start.

    If you can rerun the benchmark on Intel SASUC8i in IT mode, i would be very interested if that changes anything to your scores.

    If you rerun the benchmarks, do you see the dips happening at the same benchmarks? Or do you see low performance fluctuating? In that case it could be that for some reason performance slows down, not directly related to how many disks in what kind of pool configuration. If your RAID0 scores with just a few disks are bad, then this should receive further scrutiny.

    If your system has "CPU C1E power savings" option in BIOS, try to disable it just for the benchmarking, to see if that solves your wild dips in performance.

    Since I see good scores as well, there must be some issue on your system since the hardware is capable of quite decent speeds.
    Kristan
    User

    14 posts

    Posted on 4 February 2011 @ 23:27
    Installed ZFSGuru on a spare disk so it's running from HDD and memory settings seem right, but I'm still getting crappy read performance :

    ZFSGURU-benchmark, version 1
    Test size: 32.000 gigabytes (GiB)
    Test rounds: 1
    Cooldown period: 2 seconds
    Sector size override: 4096 bytes
    Number of disks: 4 disks
    disk 1: gpt/Pool1.nop
    disk 2: gpt/Pool2.nop
    disk 3: gpt/Pool3.nop
    disk 4: gpt/Pool4.nop


  • Test Settings: TS32; TR1; SECT4096;

  • Tuning: KMEM=5.9g; AMIN=2g; AMAX=2.9g;

  • Stopping background processes: sendmail, moused, syslogd and cron

  • Stopping Samba service


  • Now testing RAID0 configuration with 4 disks: cWmRd@
    READ: 21 MiB/sec = 21 MiB/sec avg
    WRITE: 291 MiB/sec = 291 MiB/sec avg

    Any ideas what could be causing this? It's *very* strange!
    bed
    User

    20 posts

    Posted on 5 February 2011 @ 00:15
    CPU C1E was disabled for all benchmarks. I don't believe any disks are bad as they all pass Samsung diagnostics.
    Flashed the IT firmware as instructed at the site servethehome.com and about to restart the benchmark, will post results when completed.
    Kristan
    User

    14 posts

    Posted on 5 February 2011 @ 03:05
    Just for a bit of fun I benchmarked each drive in RAID0 individually and got about 130MiB/sec READ 110MiB/sec WRITE for each, so I'm happy there's no bad disks. Did the same with 512 byte & 4K sectors and no real difference.

    So, doing three disks gave me :

    ZFSGURU-benchmark, version 1
    Test size: 32.000 gigabytes (GiB)
    Test rounds: 1
    Cooldown period: 2 seconds
    Sector size override: default (no override)
    Number of disks: 3 disks
    disk 1: gpt/Pool1
    disk 2: gpt/Pool2
    disk 3: gpt/Pool3


  • Test Settings: TS32; TR1;

  • Tuning: KMEM=5.9g; AMIN=2g; AMAX=2.9g;

  • Stopping background processes: sendmail, moused, syslogd and cron

  • Stopping Samba service


  • Now testing RAID0 configuration with 1 disks: cWmRd@
    READ: 130 MiB/sec = 130 MiB/sec avg
    WRITE: 103 MiB/sec = 103 MiB/sec avg

    Now testing RAID0 configuration with 2 disks: cWmRd@
    READ: 187 MiB/sec = 187 MiB/sec avg
    WRITE: 180 MiB/sec = 180 MiB/sec avg

    Now testing RAID0 configuration with 3 disks: cWmRd@
    READ: 28 MiB/sec = 28 MiB/sec avg
    WRITE: 220 MiB/sec = 220 MiB/sec avg

    Now testing RAIDZ configuration with 2 disks: cWmRd@
    READ: 90 MiB/sec = 90 MiB/sec avg
    WRITE: 93 MiB/sec = 93 MiB/sec avg

    Now testing RAIDZ configuration with 3 disks: cWmRd@
    READ: 72 MiB/sec = 72 MiB/sec avg
    WRITE: 123 MiB/sec = 123 MiB/sec avg

    (at which point the webUI stopped responding, and I had to console on to start lighthttpd).

    Again we have shocking pool READ performance when >2 disks are in RAID0, and not great performance when in RAIDZ. Some interesting (and consistent) figures from the READ portion of 3 disk RAID0 test:

    ----------------------------------------
    last pid: 15836; load averages: 0.00, 0.06, 0.20 up 0+03:44:05 00:35:17
    19 processes: 1 running, 18 sleeping

    Mem: 21M Active, 5032K Inact, 3085M Wired, 152K Cache, 801M Free

    ----------------------------------------
    0% CPU?? Very strange!! The most I saw it at during the test was around 10%.
    And then in the I/O screen..:
    ----------------------------------------
    dT: 1.001s w: 1.000s filter: ^(gpt|label)\/
    L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
    0 0 0 0 0.0 0 0 0.0 0.0 gpt/Pool1.nop
    0 0 0 0 0.0 0 0 0.0 0.0 gpt/systemdisk
    0 0 0 0 0.0 0 0 0.0 0.0 gpt/Pool2.nop
    0 0 0 0 0.0 0 0 0.0 0.0 gpt/Pool3.nop
    0 8 8 1023 499.4 0 0 0.0 100.0 gpt/Pool1
    4 3 3 384 1.2 0 0 0.0 0.2 gpt/Pool2
    0 0 0 0 0.0 0 0 0.0 0.0 gpt/Pool4.nop
    0 5 5 639 1.2 0 0 0.0 0.3 gpt/Pool3
    0 0 0 0 0.0 0 0 0.0 0.0 gpt/Pool4
    ----------------------------------------

    This tended to be the case while I was watching the test - one disk would be at 100% while the rest were idle, but the data transfer was very low. On average I'd say they were at around 3000kpbs, with the (very) rare spike to 20,000kpbs, or even 40,000kbps at one point.

    Also, a little side issue. I ran the test forced to 4K sector size the first time round. 2nd time I left it as default... But it seemed to still be 4K when I checked the disks screen? Is this just it not refreshing or did it not reset the sector size after the first time? I reformatted them and they seemed to go back to 512bytes..

    Hope this is useful feedback!
    bed
    User

    20 posts

    Posted on 5 February 2011 @ 15:27edited 26 February 2011 @ 00:44
    Interesting results after flashing with IT Firmware.

    broken image

    broken image

    ZFSGURU-benchmark, version 1
    Test size: 32.000 gigabytes (GiB)
    Test rounds: 3
    Cooldown period: 2 seconds
    Sector size override: 4096 bytes
    Number of disks: 8 disks
    disk 1: gpt/disk1.nop
    disk 2: gpt/disk2.nop
    disk 3: gpt/disk3.nop
    disk 4: gpt/disk4.nop
    disk 5: gpt/disk5.nop
    disk 6: gpt/disk6.nop
    disk 7: gpt/disk7.nop
    disk 8: gpt/disk8.nop

  • Test Settings: TS32; SECT4096;

  • Tuning: KMEM=12g; AMIN=4g; AMAX=6g;

  • Stopping background processes: sendmail, moused, syslogd and cron

  • Stopping Samba service


  • Now testing RAID0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 682 MiB/sec 635 MiB/sec 703 MiB/sec = 673 MiB/sec avg
    WRITE: 202 MiB/sec 205 MiB/sec 209 MiB/sec = 205 MiB/sec avg

    Now testing RAIDZ configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 473 MiB/sec 466 MiB/sec 477 MiB/sec = 472 MiB/sec avg
    WRITE: 166 MiB/sec 168 MiB/sec 166 MiB/sec = 167 MiB/sec avg

    Now testing RAIDZ2 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 452 MiB/sec 442 MiB/sec 400 MiB/sec = 432 MiB/sec avg
    WRITE: 139 MiB/sec 145 MiB/sec 150 MiB/sec = 145 MiB/sec avg

    Now testing RAID1 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 489 MiB/sec 478 MiB/sec 485 MiB/sec = 484 MiB/sec avg
    WRITE: 71 MiB/sec 107 MiB/sec 98 MiB/sec = 92 MiB/sec avg

    Now testing RAID1+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 494 MiB/sec 532 MiB/sec 510 MiB/sec = 512 MiB/sec avg
    WRITE: 415 MiB/sec 383 MiB/sec 393 MiB/sec = 397 MiB/sec avg

    Now testing RAIDZ+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 462 MiB/sec 477 MiB/sec 519 MiB/sec = 486 MiB/sec avg
    WRITE: 418 MiB/sec 415 MiB/sec 419 MiB/sec = 417 MiB/sec avg

    Now testing RAID0 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 413 MiB/sec 390 MiB/sec 388 MiB/sec = 397 MiB/sec avg
    WRITE: 395 MiB/sec 415 MiB/sec 402 MiB/sec = 404 MiB/sec avg

    Now testing RAID0 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 528 MiB/sec 518 MiB/sec 510 MiB/sec = 519 MiB/sec avg
    WRITE: 466 MiB/sec 502 MiB/sec 480 MiB/sec = 483 MiB/sec avg

    Now testing RAID0 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 575 MiB/sec 558 MiB/sec 568 MiB/sec = 567 MiB/sec avg
    WRITE: 567 MiB/sec 561 MiB/sec 573 MiB/sec = 567 MiB/sec avg

    Now testing RAID0 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 575 MiB/sec 551 MiB/sec 606 MiB/sec = 577 MiB/sec avg
    WRITE: 603 MiB/sec 579 MiB/sec 596 MiB/sec = 593 MiB/sec avg

    Now testing RAIDZ configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 319 MiB/sec 313 MiB/sec 306 MiB/sec = 313 MiB/sec avg
    WRITE: 281 MiB/sec 267 MiB/sec 289 MiB/sec = 279 MiB/sec avg

    Now testing RAIDZ configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 400 MiB/sec 393 MiB/sec 422 MiB/sec = 405 MiB/sec avg
    WRITE: 357 MiB/sec 362 MiB/sec 362 MiB/sec = 360 MiB/sec avg

    Now testing RAIDZ configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 408 MiB/sec 419 MiB/sec 458 MiB/sec = 428 MiB/sec avg
    WRITE: 405 MiB/sec 406 MiB/sec 398 MiB/sec = 403 MiB/sec avg

    Now testing RAIDZ configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 418 MiB/sec 453 MiB/sec 422 MiB/sec = 431 MiB/sec avg
    WRITE: 434 MiB/sec 427 MiB/sec 436 MiB/sec = 432 MiB/sec avg

    Now testing RAIDZ2 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 217 MiB/sec 216 MiB/sec 213 MiB/sec = 215 MiB/sec avg
    WRITE: 178 MiB/sec 179 MiB/sec 178 MiB/sec = 178 MiB/sec avg

    Now testing RAIDZ2 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 295 MiB/sec 322 MiB/sec 313 MiB/sec = 310 MiB/sec avg
    WRITE: 270 MiB/sec 259 MiB/sec 271 MiB/sec = 267 MiB/sec avg

    Now testing RAIDZ2 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 450 MiB/sec 474 MiB/sec 435 MiB/sec = 453 MiB/sec avg
    WRITE: 325 MiB/sec 318 MiB/sec 308 MiB/sec = 317 MiB/sec avg

    Now testing RAIDZ2 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 412 MiB/sec 415 MiB/sec 424 MiB/sec = 417 MiB/sec avg
    WRITE: 362 MiB/sec 350 MiB/sec 367 MiB/sec = 360 MiB/sec avg

    Now testing RAID1 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 302 MiB/sec 296 MiB/sec 297 MiB/sec = 298 MiB/sec avg
    WRITE: 111 MiB/sec 103 MiB/sec 102 MiB/sec = 105 MiB/sec avg

    Now testing RAID1 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 374 MiB/sec 373 MiB/sec 368 MiB/sec = 372 MiB/sec avg
    WRITE: 104 MiB/sec 110 MiB/sec 107 MiB/sec = 107 MiB/sec avg

    Now testing RAID1 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 421 MiB/sec 397 MiB/sec 413 MiB/sec = 410 MiB/sec avg
    WRITE: 101 MiB/sec 102 MiB/sec 100 MiB/sec = 101 MiB/sec avg

    Now testing RAID1 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 475 MiB/sec 486 MiB/sec 474 MiB/sec = 478 MiB/sec avg
    WRITE: 105 MiB/sec 105 MiB/sec 98 MiB/sec = 103 MiB/sec avg

    Now testing RAID1+0 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 344 MiB/sec 339 MiB/sec 364 MiB/sec = 349 MiB/sec avg
    WRITE: 222 MiB/sec 216 MiB/sec 216 MiB/sec = 218 MiB/sec avg

    Now testing RAID1+0 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 499 MiB/sec 454 MiB/sec 465 MiB/sec = 473 MiB/sec avg
    WRITE: 310 MiB/sec 324 MiB/sec 313 MiB/sec = 316 MiB/sec avg

    Now testing RAIDZ+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 437 MiB/sec 472 MiB/sec 430 MiB/sec = 313 MiB/sec avg
    WRITE: 423 MiB/sec 417 MiB/sec 424 MiB/sec = 279 MiB/sec avg

    Now testing RAID0 configuration with 1 disks: cWmRd@cWmRd@cWmRd@
    READ: 118 MiB/sec 119 MiB/sec 119 MiB/sec = 119 MiB/sec avg
    WRITE: 106 MiB/sec 105 MiB/sec 107 MiB/sec = 106 MiB/sec avg

    Now testing RAID0 configuration with 2 disks: cWmRd@cWmRd@cWmRd@
    READ: 244 MiB/sec 245 MiB/sec 242 MiB/sec = 244 MiB/sec avg
    WRITE: 47 MiB/sec 48 MiB/sec 46 MiB/sec = 47 MiB/sec avg

    Now testing RAID0 configuration with 3 disks: cWmRd@cWmRd@cWmRd@
    READ: 355 MiB/sec 365 MiB/sec 359 MiB/sec = 360 MiB/sec avg
    WRITE: 71 MiB/sec 77 MiB/sec 72 MiB/sec = 73 MiB/sec avg

    Now testing RAIDZ configuration with 2 disks: cW



    I will next run the benchmark using disks 2-8 instead of 1-8 to see if performance is affected by drive 8. Any other tuning I could do to improve writes?
    Jason
    Developer

    806 posts

    Posted on 6 February 2011 @ 01:18edited 01:19 20s
    Kristan wrote: Installed ZFSGuru on a spare disk so it's running from HDD and memory settings seem right, but I'm still getting crappy read performance : You have 4GiB RAM right? You need to make sure the vfs.zfs.prefetch_disable set to "1". Then re-run the tests and see if your read performance improves.


    bed wrote: Interesting results after flashing with IT Firmware.

    http://intelbytes.com/img/intel_SASUC8i/it_firm/running_seqread.png

    http://intelbytes.com/img/intel_SASUC8i/it_firm/running_seqwrite.png
    Could you try again using no sectorsize override this time? Do the weird performance fluctuations disappear?
    bed
    User

    20 posts

    Posted on 6 February 2011 @ 19:45edited 26 February 2011 @ 00:45
    Jason wrote: Could you try again using no sectorsize override this time? Do the weird performance fluctuations disappear?
    Fist up is the test utilizing disks 2-8...

    broken image

    broken image

    ZFSGURU-benchmark, version 1
    Test size: 32.000 gigabytes (GiB)
    Test rounds: 3
    Cooldown period: 2 seconds
    Sector size override: 4096 bytes
    Number of disks: 7 disks
    disk 1: gpt/disk2.nop
    disk 2: gpt/disk3.nop
    disk 3: gpt/disk4.nop
    disk 4: gpt/disk5.nop
    disk 5: gpt/disk6.nop
    disk 6: gpt/disk7.nop
    disk 7: gpt/disk8.nop

  • Test Settings: TS32; SECT4096;

  • Tuning: KMEM=12g; AMIN=4g; AMAX=6g;

  • Stopping background processes: sendmail, moused, syslogd and cron

  • Stopping Samba service


  • Now testing RAID0 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 440 MiB/sec 438 MiB/sec 432 MiB/sec = 437 MiB/sec avg
    WRITE: 97 MiB/sec 98 MiB/sec 100 MiB/sec = 98 MiB/sec avg

    Now testing RAID0 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 519 MiB/sec 522 MiB/sec 523 MiB/sec = 521 MiB/sec avg
    WRITE: 120 MiB/sec 124 MiB/sec 123 MiB/sec = 122 MiB/sec avg

    Now testing RAID0 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 631 MiB/sec 625 MiB/sec 610 MiB/sec = 622 MiB/sec avg
    WRITE: 155 MiB/sec 161 MiB/sec 155 MiB/sec = 157 MiB/sec avg

    Now testing RAID0 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 627 MiB/sec 663 MiB/sec 650 MiB/sec = 647 MiB/sec avg
    WRITE: 197 MiB/sec 219 MiB/sec 211 MiB/sec = 209 MiB/sec avg

    Now testing RAIDZ configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 343 MiB/sec 316 MiB/sec 331 MiB/sec = 330 MiB/sec avg
    WRITE: 91 MiB/sec 271 MiB/sec 260 MiB/sec = 208 MiB/sec avg

    Now testing RAIDZ configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 425 MiB/sec 411 MiB/sec 434 MiB/sec = 423 MiB/sec avg
    WRITE: 351 MiB/sec 343 MiB/sec 345 MiB/sec = 346 MiB/sec avg

    Now testing RAIDZ configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 423 MiB/sec 438 MiB/sec 423 MiB/sec = 428 MiB/sec avg
    WRITE: 413 MiB/sec 404 MiB/sec 404 MiB/sec = 407 MiB/sec avg

    Now testing RAIDZ configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 435 MiB/sec 422 MiB/sec 419 MiB/sec = 426 MiB/sec avg
    WRITE: 440 MiB/sec 436 MiB/sec 440 MiB/sec = 439 MiB/sec avg

    Now testing RAIDZ2 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 218 MiB/sec 216 MiB/sec 220 MiB/sec = 218 MiB/sec avg
    WRITE: 174 MiB/sec 168 MiB/sec 158 MiB/sec = 167 MiB/sec avg

    Now testing RAIDZ2 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 320 MiB/sec 300 MiB/sec 310 MiB/sec = 310 MiB/sec avg
    WRITE: 84 MiB/sec 81 MiB/sec 81 MiB/sec = 82 MiB/sec avg

    Now testing RAIDZ2 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 465 MiB/sec 444 MiB/sec 463 MiB/sec = 457 MiB/sec avg
    WRITE: 97 MiB/sec 96 MiB/sec 90 MiB/sec = 94 MiB/sec avg

    Now testing RAIDZ2 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 427 MiB/sec 403 MiB/sec 448 MiB/sec = 426 MiB/sec avg
    WRITE: 106 MiB/sec 113 MiB/sec 109 MiB/sec = 109 MiB/sec avg

    Now testing RAID1 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 291 MiB/sec 288 MiB/sec 290 MiB/sec = 289 MiB/sec avg
    WRITE: 23 MiB/sec 23 MiB/sec 23 MiB/sec = 23 MiB/sec avg

    Now testing RAID1 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 349 MiB/sec 358 MiB/sec 347 MiB/sec = 351 MiB/sec avg
    WRITE: 23 MiB/sec 21 MiB/sec 25 MiB/sec = 23 MiB/sec avg

    Now testing RAID1 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 382 MiB/sec 384 MiB/sec 390 MiB/sec = 385 MiB/sec avg
    WRITE: 23 MiB/sec 22 MiB/sec 24 MiB/sec = 23 MiB/sec avg

    Now testing RAID1 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 468 MiB/sec 441 MiB/sec 444 MiB/sec = 451 MiB/sec avg
    WRITE: 23 MiB/sec 23 MiB/sec 22 MiB/sec = 23 MiB/sec avg

    Now testing RAID1+0 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 342 MiB/sec 358 MiB/sec 351 MiB/sec = 350 MiB/sec avg
    WRITE: 48 MiB/sec 48 MiB/sec 45 MiB/sec = 47 MiB/sec avg

    Now testing RAID1+0 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 467 MiB/sec 471 MiB/sec 495 MiB/sec = 478 MiB/sec avg
    WRITE: 73 MiB/sec 75 MiB/sec 76 MiB/sec = 75 MiB/sec avg

    Now testing RAID0 configuration with 1 disks: cWmRd@cWmRd@cWmRd@
    READ: 126 MiB/sec 127 MiB/sec 125 MiB/sec = 126 MiB/sec avg
    WRITE: 25 MiB/sec 22 MiB/sec 22 MiB/sec = 23 MiB/sec avg

    Now testing RAID0 configuration with 2 disks: cWmRd@cWmRd@cWmRd@
    READ: 237 MiB/sec 238 MiB/sec 243 MiB/sec = 239 MiB/sec avg
    WRITE: 47 MiB/sec 49 MiB/sec 48 MiB/sec = 48 MiB/sec avg

    Now testing RAID0 configuration with 3 disks: cW



    When I tested without any sectorsize override. These were the results...

    broken image

    broken image


    ZFSGURU-benchmark, version 1
    Test size: 32.000 gigabytes (GiB)
    Test rounds: 3
    Cooldown period: 2 seconds
    Sector size override: default (no override)
    Number of disks: 8 disks
    disk 1: gpt/disk1
    disk 2: gpt/disk2
    disk 3: gpt/disk3
    disk 4: gpt/disk4
    disk 5: gpt/disk5
    disk 6: gpt/disk6
    disk 7: gpt/disk7
    disk 8: gpt/disk8

  • Test Settings: TS32;

  • Tuning: KMEM=12g; AMIN=4g; AMAX=6g;

  • Stopping background processes: sendmail, moused, syslogd and cron

  • Stopping Samba service


  • Now testing RAID0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 803 MiB/sec 812 MiB/sec 759 MiB/sec = 791 MiB/sec avg
    WRITE: 182 MiB/sec 200 MiB/sec 189 MiB/sec = 191 MiB/sec avg

    Now testing RAIDZ configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 545 MiB/sec 560 MiB/sec 542 MiB/sec = 549 MiB/sec avg
    WRITE: 153 MiB/sec 148 MiB/sec 154 MiB/sec = 152 MiB/sec avg

    Now testing RAIDZ2 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 500 MiB/sec 512 MiB/sec 524 MiB/sec = 512 MiB/sec avg
    WRITE: 123 MiB/sec 129 MiB/sec 126 MiB/sec = 126 MiB/sec avg

    Now testing RAID1 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 535 MiB/sec 539 MiB/sec 515 MiB/sec = 529 MiB/sec avg
    WRITE: 21 MiB/sec 23 MiB/sec 22 MiB/sec = 22 MiB/sec avg

    Now testing RAID1+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 626 MiB/sec 645 MiB/sec 655 MiB/sec = 642 MiB/sec avg
    WRITE: 87 MiB/sec 96 MiB/sec 92 MiB/sec = 92 MiB/sec avg

    Now testing RAIDZ+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 510 MiB/sec 521 MiB/sec 538 MiB/sec = 523 MiB/sec avg
    WRITE: 124 MiB/sec 128 MiB/sec 138 MiB/sec = 130 MiB/sec avg

    Now testing RAID0 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 484 MiB/sec 488 MiB/sec 480 MiB/sec = 484 MiB/sec avg
    WRITE: 104 MiB/sec 97 MiB/sec 94 MiB/sec = 99 MiB/sec avg

    Now testing RAID0 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 549 MiB/sec 567 MiB/sec 572 MiB/sec = 562 MiB/sec avg
    WRITE: 124 MiB/sec 119 MiB/sec 117 MiB/sec = 120 MiB/sec avg

    Now testing RAID0 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 647 MiB/sec 649 MiB/sec 673 MiB/sec = 656 MiB/sec avg
    WRITE: 148 MiB/sec 146 MiB/sec 141 MiB/sec = 145 MiB/sec avg

    Now testing RAID0 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 744 MiB/sec 761 MiB/sec 743 MiB/sec = 749 MiB/sec avg
    WRITE: 166 MiB/sec 163 MiB/sec 165 MiB/sec = 165 MiB/sec avg

    Now testing RAIDZ configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 349 MiB/sec 362 MiB/sec 362 MiB/sec = 358 MiB/sec avg
    WRITE: 73 MiB/sec 66 MiB/sec 70 MiB/sec = 70 MiB/sec avg

    Now testing RAIDZ configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 443 MiB/sec 442 MiB/sec 441 MiB/sec = 442 MiB/sec avg
    WRITE: 85 MiB/sec 94 MiB/sec 90 MiB/sec = 90 MiB/sec avg

    Now testing RAIDZ configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 499 MiB/sec 490 MiB/sec 498 MiB/sec = 496 MiB/sec avg
    WRITE: 114 MiB/sec 114 MiB/sec 106 MiB/sec = 111 MiB/sec avg

    Now testing RAIDZ configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 523 MiB/sec 516 MiB/sec 513 MiB/sec = 517 MiB/sec avg
    WRITE: 134 MiB/sec 125 MiB/sec 131 MiB/sec = 130 MiB/sec avg

    Now testing RAIDZ2 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 233 MiB/sec 221 MiB/sec 232 MiB/sec = 229 MiB/sec avg
    WRITE: 45 MiB/sec 43 MiB/sec 44 MiB/sec = 44 MiB/sec avg

    Now testing RAIDZ2 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 336 MiB/sec 327 MiB/sec 338 MiB/sec = 334 MiB/sec avg
    WRITE: 66 MiB/sec 73 MiB/sec 70 MiB/sec = 69 MiB/sec avg

    Now testing RAIDZ2 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 485 MiB/sec 454 MiB/sec 464 MiB/sec = 468 MiB/sec avg
    WRITE: 83 MiB/sec 93 MiB/sec 89 MiB/sec = 88 MiB/sec avg

    Now testing RAIDZ2 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 454 MiB/sec 476 MiB/sec 468 MiB/sec = 466 MiB/sec avg
    WRITE: 114 MiB/sec 110 MiB/sec 109 MiB/sec = 111 MiB/sec avg

    Now testing RAID1 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 312 MiB/sec 305 MiB/sec 311 MiB/sec = 309 MiB/sec avg
    WRITE: 21 MiB/sec 23 MiB/sec 23 MiB/sec = 23 MiB/sec avg

    Now testing RAID1 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
    READ: 383 MiB/sec 377 MiB/sec 388 MiB/sec = 383 MiB/sec avg
    WRITE: 23 MiB/sec 23 MiB/sec 21 MiB/sec = 22 MiB/sec avg

    Now testing RAID1 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 359 MiB/sec 349 MiB/sec 346 MiB/sec = 351 MiB/sec avg
    WRITE: 22 MiB/sec 20 MiB/sec 23 MiB/sec = 22 MiB/sec avg

    Now testing RAID1 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
    READ: 412 MiB/sec 444 MiB/sec 426 MiB/sec = 427 MiB/sec avg
    WRITE: 21 MiB/sec 23 MiB/sec 21 MiB/sec = 22 MiB/sec avg

    Now testing RAID1+0 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
    READ: 372 MiB/sec 389 MiB/sec 381 MiB/sec = 381 MiB/sec avg
    WRITE: 48 MiB/sec 46 MiB/sec 53 MiB/sec = 49 MiB/sec avg

    Now testing RAID1+0 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
    READ: 496 MiB/sec 487 MiB/sec 494 MiB/sec = 492 MiB/sec avg
    WRITE: 89 MiB/sec 242 MiB/sec 294 MiB/sec = 208 MiB/sec avg

    Now testing RAIDZ+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
    READ: 468 MiB/sec 404 MiB/sec 457 MiB/sec = 358 MiB/sec avg
    WRITE: 316 MiB/sec 402 MiB/sec 392 MiB/sec = 70 MiB/sec avg

    Now testing RAID0 configuration with 1 disks: cWmRd@cWmRd@cWmRd@
    READ: 123 MiB/sec 120 MiB/sec 122 MiB/sec = 121 MiB/sec avg
    WRITE: 102 MiB/sec 99 MiB/sec 101 MiB/sec = 101 MiB/sec avg

    Now testing RAID0 configuration with 2 disks: cWmRd@cWmRd@cWmRd@
    READ: 248 MiB/sec 242 MiB/sec 243 MiB/sec = 244 MiB/sec avg
    WRITE: 83 MiB/sec 200 MiB/sec 205 MiB/sec = 162 MiB/sec avg

    Now testing RAID0 configuration with 3 disks: cWmRd@cWmRd@cWmRd@
    READ: 356 MiB/sec 358 MiB/sec 358 MiB/sec = 357 MiB/sec avg
    WRITE: 294 MiB/sec 308 MiB/sec 307 MiB/sec = 303 MiB/sec avg

    Now testing RAIDZ configuration with 2 disks: cWmRd@cWmRd@cWmRd@
    READ: 101 MiB/sec 102 MiB/sec 102 MiB/sec = 101 MiB/sec avg
    WRITE: 91 MiB/sec 96 MiB/sec 91 MiB/sec = 93 MiB/sec avg

    Now testing RAIDZ configuration with 3 disks: cWmRd@cWmRd@cWmRd@
    READ: 246 MiB/sec 239 MiB/sec 234 MiB/sec = 240 MiB/sec avg
    WRITE: 186 MiB/sec 195 MiB/sec 192 MiB/sec = 191 MiB/sec avg

    Now testing RAIDZ2 configuration with 3 disks: cW


    I don't know why the benchmarks do not finish.
    Hope this info helps.
    Jason
    Developer

    806 posts

    Posted on 6 February 2011 @ 22:46
    To me this looks like a bad disk or something, dragging scores down. Could you try testing on only 4 disks, then after that run a test on the other 4 disks, and compare the results. If one would be really good, and the other shows weird performance dips, that might indeed be a bad disk that performs poorly. Perhaps not always, but sometimes?

    I've not seen such performance behavior yet. Could you tell me exactly what hardware you use?

    There would be another test you can do, which is starting a dd command on each disk manually, and then monitor performance using gstat (the I/O monitor in the web-interface is the same thing, basically).

    You should run these commands as root:

    dd if=/dev/gpt/<GPTLABEL> of=/dev/null bs=1m &

    Note:
    1) triple check this command before executing; if you mix up the if= and of=, then you might actually write to the device, destroying your data. The of= is the device to write to, this should be /dev/null. So the above command is a sequential read-only test.
    2) The ampersand (&) at the end is important, this will execute the command on the background. Now you should issue the same command but on the next GPT-labeled disk. Continue until you have a dd running on each of your disks. Then observe gstat behavior (either in ssh shell or using I/O monitor on the web-interface).

    On gstat you should see very steady performance of at least 80MB/s on each drive. Check if you see any downward spikes. Also check your SMART data again; every drive is clean?
    bed
    User

    20 posts

    Posted on 7 February 2011 @ 00:40edited 8 February 2011 @ 01:01
    System Hardware

    HBA: http://www.newegg.com/Product/Product.aspx?Item=N82E16816117157

    HDD: 8 x http://www.newegg.com/Product/Product.aspx?Item=N82E16822152245

    CPU: http://www.newegg.com/Product/Product.aspx?Item=N82E16819103681

    MB: http://www.newegg.com/Product/Product.aspx?Item=N82E16813130223

    MEM: 2 x pairs http://www.newegg.com/Product/Product.aspx?Item=N82E16820211409

    Case: http://www.newegg.com/Product/Product.aspx?Item=N82E16811192169
    edit:
    NIC: http://www.newegg.com/Product/Product.aspx?Item=N82E16833106034

    I included the case incase the HDD housings might be an issue. I will run the 4 x tests in a bit but I did do the dd command on all disks. While disk 6 & 7 varied iops all drives maintained a speed around or above 136182 kBps. I ran those for about 20 minutes. All disks show Healthy on the SMART tab.

    You have my email, so if you want to send me your IP I will add it to the allow list and let you take a direct look at it.
    Jason
    Developer

    806 posts

    Posted on 7 February 2011 @ 01:38
    Your motherboard has 6 chipset-powered SATA ports, and 2 add-on ports supplied by JMicron JMB322 controller (ports 7/8 + eSATA port). If you have any disks connected to this controller, try to connect them elsewhere instead, or test without disks being part on this controller.

    The idea is, something is causing your terrible scores since they are really poor while at other times you get quite good scores; once you find out what your speeds should return to normal.
    bed
    User

    20 posts

    Posted on 7 February 2011 @ 02:13
    Not using any of the Motherboard supplied SATA ports for the the hard drives. They all connect to the Intel SASUC8I (with IT firmware). I will let you know the results of the 4 x tests when they are completed. If there is any component I can replace to bump speeds up I will. Just need to isolate it I guess.
    bed
    User

    20 posts

    Posted on 8 February 2011 @ 00:59
    I am still running tests but so far I have done disks 1-4 and 5-8 and a 4-7 test is running right now. The first four disks benchmarks very nice. the 5-8 test has a huge dip when all four disks are used with a huge bump in speed when the 4th disk is taken out of the test. When the 4-7 test is completed I am going to move some disks around and see if the problem moves also.

    To help reduce page bloat, is there any single test I should focus on? I plan on implementing raidz. Are the other tests really helpful in determining problems in general?

    Thinking of replacement drives, how do you think the Hitachi 5k3000 would work with these samsungs F4's?
    danswartz
    User

    252 posts

    Posted on 5 March 2011 @ 19:56
    I just have to say this benchmark rocks. Between all the windows boxes I have to back up, the 600GB mirror is running low, so I was thinking about doubling the size. The question was: go with a raid10 or raidz with 3 drives? I had 4 160GB PATA drives sitting around, so I threw them in, replacing the 2 640GB SATA drives, and am running the benchmark now. I am assuming it will test everything, including raid10 and raidz. We shall see...
    danswartz
    User

    252 posts

    Posted on 7 March 2011 @ 03:45
    Question: what release did the sync property get added? The stable version 3/14 doesn't have it. I was curious to see what that would do for my writes on crystal disk mark...
    Jason
    Developer

    806 posts

    Posted on 7 March 2011 @ 20:19
    I'm a little bit confused; what exactly do you mean with sync property?
    danswartz
    User

    252 posts

    Posted on 7 March 2011 @ 21:25edited 21:39 07s
    The previous zfs NAS I tried using was nexentastor, based on some flavor of opensolaris. They had a per-folder attribute called 'sync'. Values were 'always', 'never' or 'standard' (IIRC). Basically had to do with whether the filer treated the request as synchronous or asynchronous. The first two are obvious. The 'standard' was basically 'if the client says it is synchronous, we make them wait, otherwise we do not'. Here is an article where someone was describing their benchmarking using nexentastor, and the sync property.

    http://www.livingonthecloud.net/2011/01/nexentastor-community-edition.html

    an example from the post is:

    zfs set sync=disabled testpool/testing


    danswartz
    User

    252 posts

    Posted on 7 March 2011 @ 21:53
    Found this too:

    http://static.opensolaris.org/on/flagdays/pages/20100503093847.html

    I haven't yet found which version of zfs supports this...
    The_Dave
    User

    221 posts

    Posted on 7 March 2011 @ 23:11
    Interesting articles
    Next Page »

    Valid XHTML 1.1