Latest interface: 0.3.1
Latest system: 010
Zoldan
User

49 posts

Posted on 19 February 2013 @ 16:21
Hi,

I have 2 identical Dell PE R410 servers, both with 14Gb memory, just the disk setup is different:

Server1: FreeNAS 8.3 installed with 4x 1Tb WD black series harddisks in a RAID-Z1 config

Server2: ZFSGuru latest with 3x 250Gb WD (Dell ones) and a 60Gb Intel 520 SSD as boot and ARC2 cache

Both without tuning.

When I run iperf on both (iperf -s) I get rather different performance from both.

Server1 with the 4 disks and FreeNAS gives me a ~600-640Mbit speed

Server2 with 3 disks and the SSD as cache gives me a ~295-320Mbit speed

Tuning server 2 in automatic tuning and fastest performance did not make any effect.

Both servers use the same Dell gigabit switch, same just 1 NIC connected (no bridging yet), same server series (bought same time).

Another installation where I also used ZFSGuru, but on a Supermicro server with 16Gb, 3x WD black RAID-Z1 and 1x 32Gb SSD as boot and cache, gave me about the same values in performance; about 330 Mbit.

Needless to say I´d like know what is the difference here and how I can get ZFSGuru to be "speedier"
Also 640Mbit on FreeNAS is not that wonderful I think?

Let me here your thoughts!
CiPHER
Developer

1199 posts

Posted on 19 February 2013 @ 17:16
Did you test with the same client? Can i see the iperf output of both servers?
Zoldan
User

49 posts

Posted on 19 February 2013 @ 18:52
yes, same client, I use my laptop to test.
Next time I´m on site I will make a printout from both.
Zoldan
User

49 posts

Posted on 21 February 2013 @ 06:24
Hi CiPHER,

I will paste the output from both here now:
Server 1 with FreeNAS output server side:
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 4] local 172.18.1.100 port 5001 connected with 172.18.1.254 port 57796
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-60.0 sec 4.35 GBytes 623 Mbits/sec

Client side:
bin/iperf.exe -c 172.18.1.100 -P 1 -i 1 -p 5001 -f m -t 60
------------------------------------------------------------
Client connecting to 172.18.1.100, TCP port 5001
TCP window size: 0.01 MByte (default)
------------------------------------------------------------
[156] local 172.18.1.254 port 57796 connected with 172.18.1.100 port 5001
[ ID] Interval Transfer Bandwidth
[156] 0.0- 1.0 sec 74.4 MBytes 624 Mbits/sec
[156] 1.0- 2.0 sec 74.3 MBytes 623 Mbits/sec
[156] 2.0- 3.0 sec 72.9 MBytes 612 Mbits/sec
[156] 3.0- 4.0 sec 74.2 MBytes 623 Mbits/sec
[156] 4.0- 5.0 sec 74.6 MBytes 626 Mbits/sec
[156] 5.0- 6.0 sec 72.4 MBytes 607 Mbits/sec
[156] 6.0- 7.0 sec 74.7 MBytes 627 Mbits/sec
[156] 7.0- 8.0 sec 74.0 MBytes 621 Mbits/sec
[156] 8.0- 9.0 sec 74.1 MBytes 621 Mbits/sec
[156] 9.0-10.0 sec 74.5 MBytes 625 Mbits/sec
[156] 10.0-11.0 sec 72.7 MBytes 609 Mbits/sec
[156] 11.0-12.0 sec 73.9 MBytes 620 Mbits/sec
[156] 12.0-13.0 sec 73.4 MBytes 616 Mbits/sec
[156] 13.0-14.0 sec 74.6 MBytes 626 Mbits/sec
[156] 14.0-15.0 sec 73.4 MBytes 616 Mbits/sec
[156] 15.0-16.0 sec 74.3 MBytes 623 Mbits/sec
[156] 16.0-17.0 sec 73.3 MBytes 614 Mbits/sec
[156] 17.0-18.0 sec 73.6 MBytes 618 Mbits/sec
[156] 18.0-19.0 sec 73.5 MBytes 616 Mbits/sec
[156] 19.0-20.0 sec 73.3 MBytes 615 Mbits/sec
[ ID] Interval Transfer Bandwidth
[156] 20.0-21.0 sec 74.3 MBytes 623 Mbits/sec
[156] 21.0-22.0 sec 73.3 MBytes 615 Mbits/sec
[156] 22.0-23.0 sec 73.4 MBytes 616 Mbits/sec
[156] 23.0-24.0 sec 74.4 MBytes 624 Mbits/sec
[156] 24.0-25.0 sec 74.4 MBytes 624 Mbits/sec
[156] 25.0-26.0 sec 73.2 MBytes 614 Mbits/sec
[156] 26.0-27.0 sec 73.5 MBytes 617 Mbits/sec
[156] 27.0-28.0 sec 74.1 MBytes 621 Mbits/sec
[156] 28.0-29.0 sec 74.4 MBytes 624 Mbits/sec
[156] 29.0-30.0 sec 74.7 MBytes 627 Mbits/sec
[156] 30.0-31.0 sec 75.1 MBytes 630 Mbits/sec
[156] 31.0-32.0 sec 75.2 MBytes 631 Mbits/sec
[156] 32.0-33.0 sec 75.3 MBytes 632 Mbits/sec
[156] 33.0-34.0 sec 74.0 MBytes 621 Mbits/sec
[156] 34.0-35.0 sec 75.0 MBytes 629 Mbits/sec
[156] 35.0-36.0 sec 74.6 MBytes 626 Mbits/sec
[156] 36.0-37.0 sec 74.7 MBytes 627 Mbits/sec
[156] 37.0-38.0 sec 74.8 MBytes 627 Mbits/sec
[156] 38.0-39.0 sec 75.3 MBytes 632 Mbits/sec
[156] 39.0-40.0 sec 74.8 MBytes 628 Mbits/sec

Server 2 ZFSGuru latest:

server side:
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 4] local 172.18.1.2 port 5001 connected with 172.18.1.254 port 57758
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-60.0 sec 2.30 GBytes 329 Mbits/sec

client side:

bin/iperf.exe -c 172.18.1.2 -P 1 -i 1 -p 5001 -f m -t 60
------------------------------------------------------------
Client connecting to 172.18.1.2, TCP port 5001
TCP window size: 0.01 MByte (default)
------------------------------------------------------------
[156] local 172.18.1.254 port 57758 connected with 172.18.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[156] 0.0- 1.0 sec 38.4 MBytes 322 Mbits/sec
[156] 1.0- 2.0 sec 40.9 MBytes 343 Mbits/sec
[156] 2.0- 3.0 sec 40.5 MBytes 340 Mbits/sec
[156] 3.0- 4.0 sec 40.0 MBytes 335 Mbits/sec
[156] 4.0- 5.0 sec 38.0 MBytes 319 Mbits/sec
[156] 5.0- 6.0 sec 37.4 MBytes 314 Mbits/sec
[156] 6.0- 7.0 sec 36.1 MBytes 303 Mbits/sec
[156] 7.0- 8.0 sec 38.9 MBytes 326 Mbits/sec
[156] 8.0- 9.0 sec 36.6 MBytes 307 Mbits/sec
[156] 9.0-10.0 sec 36.0 MBytes 302 Mbits/sec
[156] 10.0-11.0 sec 41.5 MBytes 348 Mbits/sec
[156] 11.0-12.0 sec 37.8 MBytes 317 Mbits/sec
[156] 12.0-13.0 sec 38.8 MBytes 326 Mbits/sec
[156] 13.0-14.0 sec 37.8 MBytes 317 Mbits/sec
[156] 14.0-15.0 sec 40.7 MBytes 341 Mbits/sec
[156] 15.0-16.0 sec 36.4 MBytes 305 Mbits/sec
[156] 16.0-17.0 sec 36.9 MBytes 310 Mbits/sec
[156] 17.0-18.0 sec 39.7 MBytes 333 Mbits/sec
[156] 18.0-19.0 sec 37.8 MBytes 317 Mbits/sec
[156] 19.0-20.0 sec 36.0 MBytes 302 Mbits/sec
[ ID] Interval Transfer Bandwidth
[156] 20.0-21.0 sec 38.6 MBytes 324 Mbits/sec
[156] 21.0-22.0 sec 38.1 MBytes 319 Mbits/sec
[156] 22.0-23.0 sec 40.8 MBytes 342 Mbits/sec
[156] 23.0-24.0 sec 41.3 MBytes 346 Mbits/sec
[156] 24.0-25.0 sec 38.1 MBytes 319 Mbits/sec
[156] 25.0-26.0 sec 37.2 MBytes 312 Mbits/sec
[156] 26.0-27.0 sec 38.6 MBytes 324 Mbits/sec
[156] 27.0-28.0 sec 40.4 MBytes 339 Mbits/sec
[156] 28.0-29.0 sec 37.6 MBytes 316 Mbits/sec
[156] 29.0-30.0 sec 41.2 MBytes 345 Mbits/sec
[156] 30.0-31.0 sec 39.3 MBytes 330 Mbits/sec
[156] 31.0-32.0 sec 41.1 MBytes 345 Mbits/sec
[156] 32.0-33.0 sec 38.6 MBytes 324 Mbits/sec
[156] 33.0-34.0 sec 41.0 MBytes 344 Mbits/sec
[156] 34.0-35.0 sec 39.9 MBytes 334 Mbits/sec
[156] 35.0-36.0 sec 39.9 MBytes 334 Mbits/sec
[156] 36.0-37.0 sec 38.7 MBytes 325 Mbits/sec
[156] 37.0-38.0 sec 41.2 MBytes 346 Mbits/sec
[156] 38.0-39.0 sec 39.8 MBytes 334 Mbits/sec
[156] 39.0-40.0 sec 38.3 MBytes 321 Mbits/sec

performance is about half..... (exact same hardware)
Hakker
User

77 posts

Posted on 22 February 2013 @ 05:58edited 06:03 11s
hmm this bring a whole new light upon that article from ZFSbuild then. There might actually be some things that could be improved in ZFSguru in the end.
I for one am at least glad it's something that has now been reproduced. Sure is a bit disappointing there is such a difference but glad that there is more data now. This way no one is looking for something that might not have existed at all.
I'm sure Jason and or Cipher will hunt it down in a future release, hopefully next one. Still as long as I don't have 10gbit I don't mind it but I'm sure they want to be on par with other ZFS solutions.
Zoldan
User

49 posts

Posted on 22 February 2013 @ 08:40
I can produce more data on different machines, but not all in Gigabit network environment. In the gigabit environments where I tested we observe the same "slow" speeds around the 300-350Mbit/s even with the SSD as cashe, while the FreeNAS one doesn´t have.
Other thing to observe is the stability in the speed, I tested for 60secs (posted only 20) and see a flat horizontal line for FreeNAS while ZFSGuru fluctuates more.
There is a big difference I realize now, FreeNAS uses FBSD8.3 while ZFSGuru uses FBSD9.1.
In all cases I use jperf 2.0.2 as test client on the same Dell Notebook (Vostro 3460 Core i7 8Gb RAM)

If Jason and CiPHER need have more things tested, just say so!
DVD_Chef
User

128 posts

Posted on 22 February 2013 @ 13:50
The zfsbuild group does mention that something must have been optimized in FreeNAS to increase its speed, as the older version was remarkably slower in their previous testing.
moussar
User

2 posts

Posted on 22 February 2013 @ 17:26
Microserver N40L w/ 3 WD green hdd's

server process running on a "hackintosh"

[ssh@zfsguru /usr/ports]$ iperf -c 192.168.1.34 P 1 -i 1 -p 5001 -f m -t 60
iperf: ignoring extra argument -- P
iperf: ignoring extra argument -- 1
------------------------------------------------------------
Client connecting to 192.168.1.34, TCP port 5001
TCP window size: 0.03 MByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.101 port 14235 connected with 192.168.1.34 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 112 MBytes 937 Mbits/sec
[ 3] 1.0- 2.0 sec 112 MBytes 937 Mbits/sec
[ 3] 2.0- 3.0 sec 110 MBytes 923 Mbits/sec
[ 3] 3.0- 4.0 sec 112 MBytes 938 Mbits/sec
[ 3] 4.0- 5.0 sec 112 MBytes 937 Mbits/sec
[ 3] 5.0- 6.0 sec 111 MBytes 931 Mbits/sec
[ 3] 6.0- 7.0 sec 112 MBytes 938 Mbits/sec
[ 3] 7.0- 8.0 sec 112 MBytes 936 Mbits/sec
[ 3] 8.0- 9.0 sec 112 MBytes 937 Mbits/sec
[ 3] 9.0-10.0 sec 112 MBytes 936 Mbits/sec
[ 3] 10.0-11.0 sec 112 MBytes 938 Mbits/sec
[ 3] 11.0-12.0 sec 112 MBytes 937 Mbits/sec
[ 3] 12.0-13.0 sec 112 MBytes 938 Mbits/sec
[ 3] 13.0-14.0 sec 112 MBytes 937 Mbits/sec
[ 3] 14.0-15.0 sec 112 MBytes 938 Mbits/sec
[ 3] 15.0-16.0 sec 112 MBytes 938 Mbits/sec
[ 3] 16.0-17.0 sec 111 MBytes 933 Mbits/sec
[ 3] 17.0-18.0 sec 112 MBytes 936 Mbits/sec
[ 3] 18.0-19.0 sec 112 MBytes 937 Mbits/sec
[ 3] 19.0-20.0 sec 112 MBytes 938 Mbits/sec
[ 3] 20.0-21.0 sec 112 MBytes 935 Mbits/sec
[ 3] 21.0-22.0 sec 110 MBytes 920 Mbits/sec
[ 3] 22.0-23.0 sec 112 MBytes 938 Mbits/sec
[ 3] 23.0-24.0 sec 112 MBytes 938 Mbits/sec
[ 3] 24.0-25.0 sec 111 MBytes 932 Mbits/sec
[ 3] 25.0-26.0 sec 112 MBytes 935 Mbits/sec
[ 3] 26.0-27.0 sec 111 MBytes 932 Mbits/sec
[ 3] 27.0-28.0 sec 112 MBytes 938 Mbits/sec
[ 3] 28.0-29.0 sec 111 MBytes 933 Mbits/sec
[ 3] 29.0-30.0 sec 112 MBytes 938 Mbits/sec
[ 3] 30.0-31.0 sec 112 MBytes 937 Mbits/sec
[ 3] 31.0-32.0 sec 112 MBytes 937 Mbits/sec
[ 3] 32.0-33.0 sec 107 MBytes 899 Mbits/sec
[ 3] 33.0-34.0 sec 110 MBytes 927 Mbits/sec
[ 3] 34.0-35.0 sec 111 MBytes 932 Mbits/sec
[ 3] 35.0-36.0 sec 111 MBytes 934 Mbits/sec
[ 3] 36.0-37.0 sec 110 MBytes 925 Mbits/sec
[ 3] 37.0-38.0 sec 112 MBytes 938 Mbits/sec
[ 3] 38.0-39.0 sec 112 MBytes 936 Mbits/sec
[ 3] 39.0-40.0 sec 112 MBytes 936 Mbits/sec
[ 3] 40.0-41.0 sec 112 MBytes 938 Mbits/sec
[ 3] 41.0-42.0 sec 109 MBytes 912 Mbits/sec
[ 3] 42.0-43.0 sec 112 MBytes 938 Mbits/sec
[ 3] 43.0-44.0 sec 112 MBytes 936 Mbits/sec
[ 3] 44.0-45.0 sec 112 MBytes 937 Mbits/sec
[ 3] 45.0-46.0 sec 112 MBytes 937 Mbits/sec
[ 3] 46.0-47.0 sec 111 MBytes 928 Mbits/sec
[ 3] 47.0-48.0 sec 111 MBytes 932 Mbits/sec
[ 3] 48.0-49.0 sec 110 MBytes 927 Mbits/sec
[ 3] 49.0-50.0 sec 112 MBytes 938 Mbits/sec
[ 3] 50.0-51.0 sec 112 MBytes 935 Mbits/sec
[ 3] 51.0-52.0 sec 111 MBytes 929 Mbits/sec
[ 3] 52.0-53.0 sec 112 MBytes 935 Mbits/sec
[ 3] 53.0-54.0 sec 110 MBytes 922 Mbits/sec
[ 3] 54.0-55.0 sec 112 MBytes 936 Mbits/sec
[ 3] 55.0-56.0 sec 109 MBytes 913 Mbits/sec
[ 3] 56.0-57.0 sec 112 MBytes 936 Mbits/sec
[ 3] 57.0-58.0 sec 111 MBytes 934 Mbits/sec
[ 3] 58.0-59.0 sec 112 MBytes 937 Mbits/sec
[ 3] 59.0-60.0 sec 112 MBytes 937 Mbits/sec
[ 3] 0.0-60.0 sec 6678 MBytes 934 Mbits/sec
CiPHER
Developer

1199 posts

Posted on 22 February 2013 @ 17:56
Like Moussar is what you should get, above 900 megabits. Then the gigabit works as it should on a low level. If you get less, something is wrong!

The case of Zoldan is very interesting because he claims two similar hardware setups running only different software get different results, then you can draw some conclusions. But to me, both scores are too low! Why are you not getting 900+ megabits? Something mus be wrong and we can find out what.

Zoldan, you can try rerunning the benchmark with -w 64k parameter to set 64KiB client TCP window instead of the 0.01MB (i guess 8K). Next you can run another benchmark with -P 8 to see if parallel I/O changes anything.

One more thing you can do is let the benchmark run for quite some time and run the 'top' command. Then look at the percentage of interrupts and idle and system. Try to get good accurate results at where those lie.

Can you also tell me about your netwerk setup? Can you list ifconfig and also this command: dmesg | grep "em0"
(replace em0 with the name of your network driver you see in ifconfig)

I am no expert in networks, but like you i find out in the past about the same problems and struggled with the same issues. At the moment every hardware i own works near perfect out of the box in terms of performance, but i know that varies per hardware and setup. In some cases you have to dig deep to find the real cause. I have not been able to in many cases. But i am still very curious, so we can try. :)
Zoldan
User

49 posts

Posted on 24 February 2013 @ 04:48
I can only test again later this week, but I will explain a little more about the setups:

FreeNAS server 1:
Dell PowerEdge R410 Xeon L5520 14Gb memory 4x WD black in Raid-Z1 (no SSD and cache)
2x NIC BCE0 & BCE1 both static IP, not in Link aggregation yet (will be later this week in LACP)
~# dmesg | grep "bce0"
bce0: Gigabit link up!

iperf speed ~620Mbit/s

ZFSGuru server 2 v0.2.0-beta8 system: 9.1-005:
Dell PowerEdge R410 Xeon L5520 14Gb memory 3x WD enterprise 250gb in Raid-Z1 & 1x Intel 520 60Gb SSD (boot & cache)
2x NIC BCE0 & BCE1 both static IP
dmesg | grep "bce0"
bce0: <Broadcom NetXtreme II BCM5716 1000Base-T (C0)> mem 0xda000000-0xdbffffff irq 36 at device 0.0 on pci1
miibus0: <MII bus> on bce0
bce0: Ethernet address: 84:2b:2b:19:02:a6
bce0: ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.2); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.8)
bce0: Gigabit link up!
bce0: Gigabit link up!
bce0: Gigabit link up!
bce0: promiscuous mode enabled
bce0: promiscuous mode disabled
bce0: promiscuous mode enabled

iperf speed: ~320Mbit/s

Other site with ZFSGuru server with about the same speed as the server 2 above:

ZFSGuru Server 3 v0.2.0-beta7 system: 9.1-003:
Supermicro server Xeon E5606 16Gb memory 3x WD Enterprise 500Gb & 1x Kingston 32Gb SSD (boot & cache)
2x NIC igb0 & igb1 in LACP lagg bound together (lagg0)
dmesg | grep "igb0"
igb0: <Intel(R) PRO/1000 Network Connection version - 2.3.4> port 0xec00-0xec1f mem 0xfbde0000-0xfbdfffff,0xfbdc0000-0xfbddffff,0xfbd9c000-0xfbd9ffff irq 28 at device 0.0 on pci1
igb0: Using MSIX interrupts with 5 vectors
igb0: Ethernet address: 00:25:90:6c:75:98
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
igb0: link state changed to UP
igb0: promiscuous mode enabled

dmesg | grep "lagg0"
lagg0: link state changed to UP
lagg0: promiscuous mode enabled

iperf speed ~340Mbit/s

First 2 servers are connected to the same Dell 2824 gigabit switch, the other connected to a HP 24p gigabit switch, all switches are manageable.

Both ZFSGuru servers are tested with automatic tuning and automatic tuning "performance tuning" which did not make any difference.

I testted iperf in a time interval from 60secs, when I have the chance later this week I will run the tests CiPHER requests.
DVD_Chef
User

128 posts

Posted on 25 February 2013 @ 15:57edited 16:08 05s
Microserver N36L with 4 3TB WD green drives.
The first client is another ZFSguru box connected to the same gigabit switch.
The second is a Win7prox64 workstation running iperf.exe

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.1.66 port 5001 connected with 192.168.1.74 port 26422
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-60.0 sec 6.55 GBytes 937 Mbits/sec

$ iperf -c 192.168.1.66 -P 1 -i 1 -p 5001 -f m -t 60
------------------------------------------------------------
Client connecting to 192.168.1.66, TCP port 5001
TCP window size: 0.03 MByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.74 port 26422 connected with 192.168.1.66 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 83.8 MBytes 703 Mbits/sec
[ 3] 1.0- 2.0 sec 112 MBytes 942 Mbits/sec
[ 3] 2.0- 3.0 sec 112 MBytes 942 Mbits/sec
[ 3] 3.0- 4.0 sec 112 MBytes 942 Mbits/sec
[ 3] 4.0- 5.0 sec 112 MBytes 942 Mbits/sec
[ 3] 5.0- 6.0 sec 112 MBytes 941 Mbits/sec
[ 3] 6.0- 7.0 sec 112 MBytes 942 Mbits/sec
[ 3] 7.0- 8.0 sec 112 MBytes 942 Mbits/sec
[ 3] 8.0- 9.0 sec 112 MBytes 942 Mbits/sec
[ 3] 9.0-10.0 sec 112 MBytes 942 Mbits/sec
[ 3] 10.0-11.0 sec 112 MBytes 942 Mbits/sec
.....
[ 3] 49.0-50.0 sec 112 MBytes 942 Mbits/sec
[ 3] 50.0-51.0 sec 112 MBytes 942 Mbits/sec
[ 3] 51.0-52.0 sec 112 MBytes 941 Mbits/sec
[ 3] 52.0-53.0 sec 112 MBytes 942 Mbits/sec
[ 3] 53.0-54.0 sec 112 MBytes 942 Mbits/sec
[ 3] 54.0-55.0 sec 112 MBytes 942 Mbits/sec
[ 3] 55.0-56.0 sec 112 MBytes 942 Mbits/sec
[ 3] 56.0-57.0 sec 112 MBytes 942 Mbits/sec
[ 3] 57.0-58.0 sec 112 MBytes 942 Mbits/sec
[ 3] 58.0-59.0 sec 112 MBytes 942 Mbits/sec
[ 3] 59.0-60.0 sec 112 MBytes 942 Mbits/sec
[ 3] 0.0-60.0 sec 6706 MBytes 938 Mbits/sec


Windows 7 pro x64 running iperf.exe as client

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.1.66 port 5001 connected with 192.168.1.74 port 26422
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-60.0 sec 2.67 GBytes 383 Mbits/sec

iperf.exe -c 192.168.1.66 -P 1 -i 1 -p 5001 -f m -t 60
------------------------------------------------------------
Client connecting to 192.168.1.66, TCP port 5001
TCP window size: 0.01 MByte (default)
------------------------------------------------------------
[156] local 192.168.1.111 port 49175 connected with 192.168.1.66 port 5001
[ ID] Interval Transfer Bandwidth
[156] 0.0- 1.0 sec 38.8 MBytes 326 Mbits/sec
[156] 1.0- 2.0 sec 47.4 MBytes 397 Mbits/sec
[156] 2.0- 3.0 sec 46.6 MBytes 391 Mbits/sec
[156] 3.0- 4.0 sec 24.7 MBytes 207 Mbits/sec
[156] 4.0- 5.0 sec 24.1 MBytes 203 Mbits/sec
[156] 5.0- 6.0 sec 24.5 MBytes 206 Mbits/sec
[156] 6.0- 7.0 sec 47.8 MBytes 401 Mbits/sec
[156] 7.0- 8.0 sec 48.8 MBytes 410 Mbits/sec
[156] 8.0- 9.0 sec 47.4 MBytes 398 Mbits/sec
[156] 9.0-10.0 sec 49.8 MBytes 418 Mbits/sec
[156] 10.0-11.0 sec 47.6 MBytes 400 Mbits/sec
....
[156] 50.0-51.0 sec 46.4 MBytes 389 Mbits/sec
[156] 51.0-52.0 sec 46.7 MBytes 392 Mbits/sec
[156] 52.0-53.0 sec 47.3 MBytes 397 Mbits/sec
[156] 53.0-54.0 sec 47.8 MBytes 401 Mbits/sec
[156] 54.0-55.0 sec 47.3 MBytes 396 Mbits/sec
[156] 55.0-56.0 sec 46.9 MBytes 393 Mbits/sec
[156] 56.0-57.0 sec 46.5 MBytes 390 Mbits/sec
[156] 57.0-58.0 sec 46.1 MBytes 387 Mbits/sec
[156] 58.0-59.0 sec 46.8 MBytes 393 Mbits/sec
[156] 59.0-60.0 sec 46.4 MBytes 389 Mbits/sec
[156] 0.0-60.0 sec 2738 MBytes 383 Mbits/sec


Win7 again with tcp window manually set to 0.03MB to match FreeBSD

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.1.66 port 5001 connected with 192.168.1.74 port 26422
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-60.0 sec 4.36 GBytes 625 Mbits/sec

iperf.exe -c 192.168.1.66 -P 1 -i 1 -p 5001 -f m -t 60 -w 0.03MB
------------------------------------------------------------
Client connecting to 192.168.1.66, TCP port 5001
TCP window size: 0.03 MByte
------------------------------------------------------------
[156] local 192.168.1.111 port 49190 connected with 192.168.1.66 port 5001
[ ID] Interval Transfer Bandwidth
[156] 0.0- 1.0 sec 87.8 MBytes 737 Mbits/sec
[156] 1.0- 2.0 sec 68.7 MBytes 576 Mbits/sec
[156] 2.0- 3.0 sec 62.0 MBytes 520 Mbits/sec
[156] 3.0- 4.0 sec 61.7 MBytes 517 Mbits/sec
[156] 4.0- 5.0 sec 105 MBytes 877 Mbits/sec
[156] 5.0- 6.0 sec 69.9 MBytes 587 Mbits/sec
[156] 6.0- 7.0 sec 60.3 MBytes 505 Mbits/sec
[156] 7.0- 8.0 sec 61.6 MBytes 516 Mbits/sec
[156] 8.0- 9.0 sec 102 MBytes 853 Mbits/sec
[156] 9.0-10.0 sec 73.9 MBytes 620 Mbits/sec
[156] 10.0-11.0 sec 60.7 MBytes 509 Mbits/sec
[156] 50.0-51.0 sec 61.3 MBytes 514 Mbits/sec
[156] 51.0-52.0 sec 61.0 MBytes 512 Mbits/sec
[156] 52.0-53.0 sec 69.9 MBytes 586 Mbits/sec
[156] 53.0-54.0 sec 105 MBytes 879 Mbits/sec
[156] 54.0-55.0 sec 61.2 MBytes 513 Mbits/sec
[156] 55.0-56.0 sec 62.8 MBytes 527 Mbits/sec
[156] 56.0-57.0 sec 66.7 MBytes 559 Mbits/sec
[156] 57.0-58.0 sec 109 MBytes 914 Mbits/sec
[156] 58.0-59.0 sec 62.4 MBytes 524 Mbits/sec
[156] 59.0-60.0 sec 61.4 MBytes 515 Mbits/sec
[156] 0.0-60.0 sec 4469 MBytes 625 Mbits/sec


Increasing the window does increase the number, but it still does not reach the speed seen from the the other ZFSguru box. Forcing the window to 0.01MB on the ZFSguru box does drop its speed down to 330 Mbits/sec, which is slower than win7.

Not really sure what all this means, but wanted to add some data to the discussion.
roef33
User

2 posts

Posted on 26 February 2013 @ 03:20
Did some tests on mij HP DC7700 with 3 WD 160GB disks in RaidZ

E:\>iperf -c zfsguru -P 1 -i 1 -p 5001 -f m -t 5 -w 16KB
------------------------------------------------------------
Client connecting to zfsguru, TCP port 5001
TCP window size: 0.02 MByte
------------------------------------------------------------
[292] local 192.168.137.1 port 57565 connected with 192.168.137.114 port 5001
[ ID] Interval Transfer Bandwidth
[292] 0.0- 1.0 sec 29.7 MBytes 249 Mbits/sec
[292] 1.0- 2.0 sec 60.0 MBytes 503 Mbits/sec
[292] 2.0- 3.0 sec 65.4 MBytes 549 Mbits/sec
[292] 3.0- 4.0 sec 60.8 MBytes 510 Mbits/sec
[292] 4.0- 5.0 sec 62.2 MBytes 522 Mbits/sec
[292] 0.0- 5.0 sec 278 MBytes 466 Mbits/sec

E:\>iperf -c zfsguru -P 1 -i 1 -p 5001 -f m -t 5 -w 16KB
------------------------------------------------------------
Client connecting to zfsguru, TCP port 5001
TCP window size: 0.02 MByte
------------------------------------------------------------
[292] local 192.168.137.1 port 57572 connected with 192.168.137.114 port 5001
[ ID] Interval Transfer Bandwidth
[292] 0.0- 1.0 sec 54.0 MBytes 453 Mbits/sec
[292] 1.0- 2.0 sec 58.1 MBytes 487 Mbits/sec
[292] 2.0- 3.0 sec 55.8 MBytes 468 Mbits/sec
[292] 3.0- 4.0 sec 53.9 MBytes 452 Mbits/sec
[292] 4.0- 5.0 sec 57.0 MBytes 478 Mbits/sec
[292] 0.0- 5.0 sec 279 MBytes 467 Mbits/sec

E:\>iperf -c zfsguru -P 1 -i 1 -p 5001 -f m -t 5 -w 32KB
------------------------------------------------------------
Client connecting to zfsguru, TCP port 5001
TCP window size: 0.03 MByte
------------------------------------------------------------
[292] local 192.168.137.1 port 57582 connected with 192.168.137.114 port 5001
[ ID] Interval Transfer Bandwidth
[292] 0.0- 1.0 sec 99.9 MBytes 838 Mbits/sec
[292] 1.0- 2.0 sec 100 MBytes 843 Mbits/sec
[292] 2.0- 3.0 sec 98.0 MBytes 822 Mbits/sec
[292] 3.0- 4.0 sec 100 MBytes 841 Mbits/sec
[292] 4.0- 5.0 sec 100 MBytes 842 Mbits/sec
[292] 0.0- 5.0 sec 499 MBytes 836 Mbits/sec

E:\>iperf -c zfsguru -P 1 -i 1 -p 5001 -f m -t 5 -w 32KB
------------------------------------------------------------
Client connecting to zfsguru, TCP port 5001
TCP window size: 0.03 MByte
------------------------------------------------------------
[292] local 192.168.137.1 port 57590 connected with 192.168.137.114 port 5001
[ ID] Interval Transfer Bandwidth
[292] 0.0- 1.0 sec 91.9 MBytes 771 Mbits/sec
[292] 1.0- 2.0 sec 95.4 MBytes 801 Mbits/sec
[292] 2.0- 3.0 sec 95.2 MBytes 798 Mbits/sec
[292] 3.0- 4.0 sec 94.1 MBytes 790 Mbits/sec
[292] 4.0- 5.0 sec 91.8 MBytes 770 Mbits/sec
[292] 0.0- 5.0 sec 468 MBytes 785 Mbits/sec

E:\>iperf -c zfsguru -P 1 -i 1 -p 5001 -f m -t 5 -w 64KB
------------------------------------------------------------
Client connecting to zfsguru, TCP port 5001
TCP window size: 0.06 MByte
------------------------------------------------------------
[292] local 192.168.137.1 port 57598 connected with 192.168.137.114 port 5001
[ ID] Interval Transfer Bandwidth
[292] 0.0- 1.0 sec 104 MBytes 870 Mbits/sec
[292] 1.0- 2.0 sec 103 MBytes 864 Mbits/sec
[292] 2.0- 3.0 sec 103 MBytes 867 Mbits/sec
[292] 3.0- 4.0 sec 70.5 MBytes 591 Mbits/sec
[292] 4.0- 5.0 sec 103 MBytes 867 Mbits/sec
[292] 0.0- 5.0 sec 484 MBytes 810 Mbits/sec

E:\>iperf -c zfsguru -P 1 -i 1 -p 5001 -f m -t 5 -w 64KB
------------------------------------------------------------
Client connecting to zfsguru, TCP port 5001
TCP window size: 0.06 MByte
------------------------------------------------------------
[292] local 192.168.137.1 port 57606 connected with 192.168.137.114 port 5001
[ ID] Interval Transfer Bandwidth
[292] 0.0- 1.0 sec 104 MBytes 875 Mbits/sec
[292] 1.0- 2.0 sec 104 MBytes 870 Mbits/sec
[292] 2.0- 3.0 sec 104 MBytes 874 Mbits/sec
[292] 3.0- 4.0 sec 103 MBytes 864 Mbits/sec
[292] 4.0- 5.0 sec 102 MBytes 856 Mbits/sec
[292] 0.0- 5.0 sec 517 MBytes 867 Mbits/sec

E:\>iperf -c zfsguru -P 1 -i 1 -p 5001 -f m -t 5 -w 96KB
------------------------------------------------------------
Client connecting to zfsguru, TCP port 5001
TCP window size: 0.09 MByte
------------------------------------------------------------
[292] local 192.168.137.1 port 57614 connected with 192.168.137.114 port 5001
[ ID] Interval Transfer Bandwidth
[292] 0.0- 1.0 sec 94.7 MBytes 794 Mbits/sec
[292] 1.0- 2.0 sec 91.9 MBytes 771 Mbits/sec
[292] 2.0- 3.0 sec 91.4 MBytes 767 Mbits/sec
[292] 3.0- 4.0 sec 94.7 MBytes 794 Mbits/sec
[292] 4.0- 5.0 sec 94.6 MBytes 794 Mbits/sec
[292] 0.0- 5.0 sec 467 MBytes 783 Mbits/sec

E:\>iperf -c zfsguru -P 1 -i 1 -p 5001 -f m -t 5 -w 96KB
------------------------------------------------------------
Client connecting to zfsguru, TCP port 5001
TCP window size: 0.09 MByte
------------------------------------------------------------
[292] local 192.168.137.1 port 57622 connected with 192.168.137.114 port 5001
[ ID] Interval Transfer Bandwidth
[292] 0.0- 1.0 sec 94.1 MBytes 789 Mbits/sec
[292] 1.0- 2.0 sec 93.3 MBytes 782 Mbits/sec
[292] 2.0- 3.0 sec 94.0 MBytes 788 Mbits/sec
[292] 3.0- 4.0 sec 92.3 MBytes 774 Mbits/sec
[292] 4.0- 5.0 sec 92.0 MBytes 772 Mbits/sec
[292] 0.0- 5.0 sec 466 MBytes 780 Mbits/sec

my 2cent
stompa
User

2 posts

Posted on 6 May 2013 @ 12:54
Hi Moussar
any chance you can share any tuneables & sysctls you used on your n40l with freenas (apologies for hijacking this zfsguru forum). I'm getting burtsts ~20MB/s on my 1 gigabit network but want to get >70MB+. my n40l has 3 x2TB WD green and 1 x 1TB Samsung configured as a 4 disk RAIDZ pool.

i have the following tuneables with no success:
vfs.zfs.arc_max="6442450944" # Generated by autotune
vm.kmem_size_max="12884901888" # Generated by autotune
vm.kmem_size="8589934592" # Generated by autotune
vfs.zfs.txg.timeout="10" # manual
vfs.zfs.txg.write_limit_override="629145600"
vfs.zfs.zil_disable="1"
vfs.zfs.txg.synctime="5"
vfs.zfs.vdev.min_pending="4"

& the pool was created with force 4k.

thanks in advance
[quote=3826]Microserver N40L w/ 3 WD green hdd's
Last Page

Valid XHTML 1.1