Lab notes on 10 Gbit/s network tests with OS X
(C) 2009 Jan Wagner, Guifre Molera
24 June 2009 - OS X Setup
With some efforts we tested our OS X Leopard (10.5) install/recovery DVD together with several nVidia Driver patches from the OSx86 project on the Abidal computer (nForce680a, 2xAMD2212). The setup was a total failure. OS X simply loves platforms based on an Intel chipset but shies away from anything with nForce. So on the second attempt we used Lionel (Core 2 Quad Q9550, X48). Worked perfect. Only had to install Myri 10G drivers and Addonics ADSA3GPX8-4E eSATA controller drivers for OS X. Both are available as a DMG or as a package from each vendor website.
There is a fatal OS X bug regarding port multipliers and RAID: power saving for disks must be disabled, otherwise OS X will shut down idle RAID members without first ejecting the RAID. This leads to instant file system corruption on the RAID. So we had the pleasure of experiencing the bufsize 512 not a multiple of block size 4096 kernel panic every time the disks were connected. Automounting could not be disabled, so reformatting was impossible because of the instant crashes.
A Ubuntu USB stick rescued the situation. We ran a 'dd if=/dev/zero' and zeroed the first 100MB of each disk -- this area seems to contain the GPT partition table, os x bootloader partition, helper partition and the start of the actual data partition. After all disk were blank at the start, OS X could be used safely without kernel panics and it was possible to re-create the Striped RAID.
After the finally successful installation and disk configuration on Lionel we ported the realtime-extended 'wr-nexgen.c' version onto OS X (source code in wr-nexgen_mac.c). Certain things are missing in this port: setting near maximum task priority, locking the source memory buffer to prevent swapping, and no posix fadvise to advise the OS that we plan to do sequential write I/O with no seeking and the OS can optimize itself accordingly. In Linux using these preparing calls gives the best performance. In OS X these API functions do not exist or exist under a different name I couldn't figure out yet. Thus for now these are missing from wr-nexgen_mac.c
But even without those, the performance of OS X is already better than that of Linux:
| Disks | Result | Log file |
|---|---|---|
| 20 |
Took 8552.99 seconds for 5000000.00 MBytes: 4903.902 Mbits(dec)/s (464.38% of PCI33). Volumes/4GEXPReS/wrtest.bin max. 3.599229 seconds. |
log |
| 12 evenly | 4.665 Gbps (cancelled at 20%) |
log |
| 10 | todo |
|
| 2x10 | 3.000 Gbps x 2 |
log I | log II |
It is notable that in 'top', wr-nexgen took about 48% of the single core it was tied to. The mflush operating system tasks took only 3% each.
With two dual-PMP 2x10 raid configurations being written in parallel the CPU load looks like:
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
593 wr-nexgen_ 35.4% 17:52.12 1 13 0 0 0 201M 788M
592 wr-nexgen_ 35.4% 17:40.89 1 13 0 0 0 201M 788M
0 kernel_tas 21.9% 11:12.78 57 2 0 0 0 107M 878M
125 WindowServ 1.4% 1:31.63 9 183 0 0 0 26M 740M
519 Terminal 1.0% 0:20.00 3 104- 0 0 0 30M 797M
745 top 0.7% 0:02.80 1 19 0 0 0 3796K 596M
218 SiCoreServ 0.4% 0:15.08 1 20 0 0 0 6264K 615M
746 Safari 0.2% 0:05.06 5 125+ 0 0 0 80M 852M
...
2331 mdworker 0.0% 0:00.12 3 51 0 0 0 10M 654M
718 mdworker 0.0% 0:00.98 4 56 0 0 0 21M 697M
...
This is in stark contrast to the poor Linux 2.6.27 wr-nexgen performance on the same computer, where with any file system (ext3, ext4, xfs, jfs) wr-nexgen will hit 100% CPU and the pdflush'es alternate between 30% and 80%. Worse, earlier we had noticed that the performance in Linux 2.6.30 and 2.6.31 of Ubuntu Karmic is below the performance of 2.6.27. See the Bug #13408 LKML discussion with no resolution yet as of 07/2009.
 |
 |
This work has received financial support under the EU FP6 Integrated Infrastructure Initiative contract number #026642, EXPReS. |