10 and 40 Gbit/s burst mode observations
02-Oct-2009 Jan Wagner
Burst mode observations are used to record small amounts of DAQ data at rates faster than the final storage or network path can cope with. VLBI quality improvement over continuous mode observations are outlined in Development of the Burst Mode VLBI (Matsumoto, K.; Kawaguchi, N.; 1993).
Short 10s recordings at 4 Gbit/s require 5GB of storage.
The emerging Solid State Disks (2009) reside between magnetic disk and volatile memory in terms of capacity, I/O bandwidth and random access latency. Disks provide high capacity (>1 TB) but only medium bandwidth (<150MB/s) and they have high latency (5ms average seek time). RAM on the other hand is low capacity (≤12 GB on desktops) but has highest bandwidth (>30 GB/s) and nanosecond latency.
Burst mode recording with simultaneous transfer to the final storage results in random I/O with competing read and write operations. Mechanic disks even in RAID0 configuration are impractical due to the high random access latency. Desktops can handle recording bursts into RAM at 4 Gbit/s for some ten seconds. Servers can fit more RAM. In network-based data acquisition it is natural to use broadcast or multicast to extend the supported burst duration by employing suitable time division multiplexing or staggering over a group of computers.
Upcoming SSD disks can support 4 Gbit/s writes with ease. High rates are accomplished using either 2-3 SSDs with SATA-II, SAS-3G or FC interfaces, combined into a single RAID-0. Single PCI Express -based SSD cards are even more efficient. Ultralow latency ensures full performace even with random I/O.
Software
Normal network recording tools can be used (vdifcapture, udp2raid, ...). Since the burst storage (SSD) is per definition faster than the final storage, one can simply start the recording tool to begin recording a file onto SSD and start a fraction of time later a normal disk 'copy' or network transfer of that same growing file to another location.
Example:
$ vdifcapture 193.166.42.11 46220 10s /mnt/ssd/scanname.vdif & $ do while [ ! -f /mnt/ssd/scanname.vdif ]; done $ cp /mnt/ssd/scanname.vdif /raid/scanname.vdif # happens while vdifcapture is still writing $ del /mnt/ssd/scanname.vdif
The ATA7 'TRIM' command and/or manufacturer provided automatic trimming utilities will be useful to maintain the speed of the SSD in VLBI data capture.
Hardware
Hardware as of 10/2009 as a very brief overview:
- OCZ Z-Drive P48: PCIe x8, 250GB - 1TB, MLC, 900k hours MTBF, 600MB/s sustained write: OCZSSDPCIE-ZDP841T
- OCZ Z-Drive E48: PCIe x8, 250GB - 512GB, SLC, 900k hours MTBF, 750MB/s sustained write: OCZSSDPCIE-ZDE84512G
- SuperTalent RAIDdrive GS: PCIe x8, 2TB, MLC, 1.5M hours MTBF, 1200MB/s sustained write
- ioDrive Duo: PCIe 2.0 x4, 160GB - 1.28TB, SLC/MLC, unknown MTBF, 1400 MB/s sustained write: (link with nice photo)
- Photofast G-Monster Promise: PCIe x8, 128GB - 1TB, MLC, 1.5M hours MTBF, 1000(?) MB/s sustained write
- XceedIOPS: PCIe 2.0 x8, 400GB, SLC, unknown MTBF, 500 MB/s(?) sustained write
- StorExpress: dual PCIe x8 cards with 4U external flash, 3TB - 4TB, SLC, 2500 MB/s sustained write: DSE4XM: $20k
- HyperOs Systems HyperDrive5: SATA-II, 8*DDR2 max 64GB, 145MB/s sustained write
Some of the PCIe hardware such as early Z-Drives are based on RocketRaid 3520 and four SSDs simply crammed into a nice plastic package. Others like ioDrive are designed from ground up. Enterprise Linux support unknown.
Current NAND flash types:
- Multi-level cell (MLC): area efficient, lower cost per GB, survives fewer write cycles, slower to write
- Enterprise MLC (SMLC): attempt to re-sell plain MLC at higher price by reserving more memory for error resilience
- Single-level cell (SLC): traditional flash memory, high cost per GB, more write cycles, faster to write
The SSD MTBF reliability specs compare reasonably but not really favourably to cheapest SATA crap 900k hour MTBF and medium to high end 1.2M hour SATA, 1.4M hour SAS MTBF. It is unclear whether manufacturers use write-only access patterns for their MTBF estimate. Still, even inexpensive SSDs should outlast their usability timespan in VLBI without too many of them failing. Unless the trend of manufacturers reducing SSD MTBFs continues.
As can be seen from the hardware list, components cope easily with 4 Gbit/s and some even with 8 Gbit/s burst modes.
The fast 512GB SSDs can capture up to 8 minute "bursts" at 8 Gbit/s. The price is less than that of a Mark5C.
Five such SSDs in RAID0 allow 40 Gbit/s burst mode together with a 100GbE or 40GbE NIC.
Main speed improvements in SSDs in the future will come from using more, not faster, parallel flash memory segments.
End of 2010 16 Gbit/s burst modes with single SSDs may be possible.
This concludes our burst mode intro.