Several sections: - no UDP checksums and diskless and 9000 MTU - same with better server IPD spacing code - same but with 8192 blocksize (MTU still 9000) - client writes to disk, no UDP checksums and 32kB blocksize and 9000 MTU ------------------------------------------------------------------------------------------- With UDP Checksumming disabled, Diskless ------------------------------------------------------------------------------------------- Applied the UDP checksum setting setsockopt ( inSettings->mSock, SOL_SOCKET, SO_NO_CHECK, &yes, sizeof(yes)); to the server network.c, then tested a lossy (no retransmissions) and diskless Tsunami transfer first from the slower Juliano to faster Abidal and then the other way round. --------- From Juliano to Abidal client ----------- oper@abidal:/usr/src/tsunami-udp-v12/client$ ./tsunami connect juliano set rate 6g set lossless no set losswindow 0 Tsunami Client for protocol rev 20071109 Revision: v1.2 totally-alpha ;) cvsbuild 1 Compiled: Dec 12 2007 14:40:22 Connected. rate = 6000000000 lossless = no losswindow = 0 msec tsunami> get * Requesting all available files Server is sharing 1 files Multi-GET of 1 files: 6gigfile GET *: now requesting file '6gigfile' Warning: overwriting existing file '6gigfile' Receiving data on UDP port 46224 last_interval transfer_total buffers transfer_remaining OS UDP time blk data rate rexmit blk data rate rexmit queue ring blk rt_len err 00:00:00.354 2850 0.00M 2106.1Mbps 0.0% 2850 0.1G 2106.1Mbps 0.0% 0 0 192463 0 0 00:00:00.708 3450 0.00M 2557.4Mbps 0.0% 6300 0.2G 2330.5Mbps 0.0% 0 0 189013 0 0 00:00:01.059 4150 0.00M 3103.5Mbps 0.0% 10450 0.3G 2585.8Mbps 0.0% 0 0 184863 0 0 00:00:01.410 5050 0.00M 3774.0Mbps 0.0% 15500 0.5G 2880.9Mbps 0.0% 0 0 179813 0 0 00:00:01.761 6200 0.00M 4639.4Mbps 0.0% 21700 0.7G 3230.3Mbps 0.0% 0 0 173613 0 0 00:00:02.112 6649 0.00M 4959.2Mbps 0.0% 28349 0.9G 3517.4Mbps 0.0% 0 0 167813 0 1105 00:00:02.463 8552 0.00M 6404.5Mbps 0.0% 36901 1.1G 3927.3Mbps 0.0% 0 0 162413 0 4507 00:00:02.815 8893 0.00M 6613.8Mbps 0.0% 45794 1.4G 4263.2Mbps 0.0% 0 0 157013 0 7989 00:00:03.169 8837 0.00M 6559.9Mbps 0.0% 54631 1.7G 4518.7Mbps 0.0% 0 0 151613 0 11440 00:00:03.522 8888 0.00M 6612.3Mbps 0.0% 63519 1.9G 4727.7Mbps 0.0% 0 0 146213 0 14923 00:00:03.873 8838 0.00M 6601.1Mbps 0.0% 72357 2.2G 4897.1Mbps 0.0% 0 0 140813 0 18348 00:00:04.226 8829 0.00M 6564.7Mbps 0.0% 81186 2.5G 5035.9Mbps 0.0% 0 0 135363 0 21729 00:00:04.578 8835 0.00M 6581.2Mbps 0.0% 90021 2.7G 5154.3Mbps 0.0% 0 0 129963 0 25192 00:00:04.928 9188 0.00M 6879.5Mbps 0.0% 99209 3.0G 5276.5Mbps 0.0% 0 0 122563 0 26591 00:00:05.280 8792 0.00M 6564.4Mbps 0.0% 108001 3.3G 5361.9Mbps 0.0% 0 0 115063 0 27895 00:00:05.631 8869 0.00M 6612.9Mbps 0.0% 116870 3.6G 5439.8Mbps 0.0% 0 0 107563 0 29267 00:00:05.983 8829 0.00M 6588.8Mbps 0.0% 125699 3.8G 5507.1Mbps 0.0% 0 0 100063 0 30587 00:00:06.334 8800 0.00M 6568.4Mbps 0.0% 134499 4.1G 5565.8Mbps 0.0% 0 0 92563 0 31894 00:00:06.687 8529 0.00M 6342.7Mbps 0.0% 143028 4.4G 5606.6Mbps 0.0% 0 0 85963 0 34190 00:00:07.040 8855 0.00M 6587.5Mbps 0.0% 151883 4.6G 5655.4Mbps 0.0% 0 0 80563 0 37631 00:00:07.392 8820 0.00M 6570.0Mbps 0.0% 160703 4.9G 5698.7Mbps 0.0% 0 0 75163 0 41069 00:00:07.743 8837 0.00M 6608.9Mbps 0.0% 169540 5.2G 5739.7Mbps 0.0% 0 0 69813 0 44566 00:00:08.096 8863 0.00M 6583.6Mbps 0.0% 178403 5.4G 5776.2Mbps 0.0% 0 0 64413 0 48013 last_interval transfer_total buffers transfer_remaining OS UDP time blk data rate rexmit blk data rate rexmit queue ring blk rt_len err 00:00:08.447 9191 0.00M 6863.4Mbps 0.0% 187594 5.7G 5821.2Mbps 0.0% 0 0 58063 0 50490 !!!! Mbits of data transmitted : 48828.12 Duration in seconds : 8.77 THROUGHPUT (Mbps) : 5838.85 OS UDP packet rx errors : delta 50490 Transfer type : lossy Missing data blocks count : 52268 (26.76% of data) per user-specified time window constraint > set rate 5g ... Mbits of data transmitted : 48828.12 Duration in seconds : 10.34 THROUGHPUT (Mbps) : 4954.04 OS UDP packet rx errors : delta 22572 Transfer type : lossy Missing data blocks count : 22573 (11.56% of data) per user-specified time window constraint > set rate 4g ... Mbits of data transmitted : 48828.12 Duration in seconds : 12.79 THROUGHPUT (Mbps) : 4002.96 OS UDP packet rx errors : delta 0 Transfer type : lossy Missing data blocks count : 1 (0.00% of data) per user-specified time window constraint > set rate 4500m ... Mbits of data transmitted : 48828.12 Duration in seconds : 11.49 THROUGHPUT (Mbps) : 4456.83 OS UDP packet rx errors : delta 8013 Transfer type : lossy Missing data blocks count : 8206 (4.20% of data) per user-specified time window constraint Note the high "OS UDP packet rx errors"... --------- From Abidal to Juliano client ----------- (the served file as little bit larger) oper@juliano:/usr/src/tsunami-udp-v12/client$ ./tsunami connect 193.166.42.22 set rate 6g set lossless no set losswindow 0 ... Mbits of data transmitted : 51200.00 Duration in seconds : 9.24 THROUGHPUT (Mbps) : 5810.22 OS UDP packet rx errors : delta 45726 Transfer type : lossy Missing data blocks count : 46127 (22.52% of data) per user-specified time window constraint > set rate 5g ... Mbits of data transmitted : 51200.00 Duration in seconds : 11.05 THROUGHPUT (Mbps) : 4860.15 OS UDP packet rx errors : delta 13332 Transfer type : lossy Missing data blocks count : 13436 (6.56% of data) per user-specified time window constraint > set rate 4g ... Mbits of data transmitted : 51200.00 Duration in seconds : 13.67 THROUGHPUT (Mbps) : 3925.94 OS UDP packet rx errors : delta 0 Transfer type : lossy Missing data blocks count : 1 (0.00% of data) per user-specified time window constraint > set rate 4500m ... Mbits of data transmitted : 51200.00 Duration in seconds : 12.28 THROUGHPUT (Mbps) : 4371.11 OS UDP packet rx errors : delta 0 Transfer type : lossy Missing data blocks count : 1 (0.00% of data) per user-specified time window constraint > set rate 4900m ... Mbits of data transmitted : 51200.00 Duration in seconds : 11.25 THROUGHPUT (Mbps) : 4771.41 OS UDP packet rx errors : delta 11467 Transfer type : lossy Missing data blocks count : 12486 (6.10% of data) per user-specified time window constraint ------------------------------------------------------------------------------------------- Modified server IPD spacing code: ------------------------------------------------------------------------------------------- In ./server/main.c /* delay for the next packet */ ipd_time = ((ipd_time + 50) < xfer->ipd_current) ? ((u_int64_t) (xfer->ipd_current - ipd_time - 50)) : 0; usleep_that_works(ipd_time); Changed it into: /* delay for the next packet */ ipd_time = get_usec_since(&delay); if (ipd_time < xfer->ipd_current) { usleep_that_works(xfer->ipd_current - ipd_time); } oper@juliano:/usr/src/tsunami-udp-v12/client$ ./tsunami connect 193.166.42.22 set rate 4g set lossless no set losswindow 0 ... Mbits of data transmitted : 51200.00 Duration in seconds : 14.66 THROUGHPUT (Mbps) : 3662.21 OS UDP packet rx errors : delta 0 Transfer type : lossy Missing data blocks count : 1 (0.00% of data) per user-specified time window constraint > set rate 5g ... Mbits of data transmitted : 51200.00 Duration in seconds : 12.02 THROUGHPUT (Mbps) : 4465.81 OS UDP packet rx errors : delta 0 Transfer type : lossy Missing data blocks count : 1 (0.00% of data) per user-specified time window constraint > set rate 5500m ... Mbits of data transmitted : 51200.00 Duration in seconds : 11.00 THROUGHPUT (Mbps) : 4882.38 OS UDP packet rx errors : delta 15248 Transfer type : lossy Missing data blocks count : 15420 (7.53% of data) per user-specified time window constraint > set blocksize 40960 > set rate 5500m ... Mbits of data transmitted : 51200.00 Duration in seconds : 10.98 THROUGHPUT (Mbps) : 4887.38 OS UDP packet rx errors : delta 7490 Transfer type : lossy Missing data blocks count : 7563 (4.62% of data) per user-specified time window constraint > set blocksize 32768 > set speedup 1/6 > set rate 5500m ... Mbits of data transmitted : 51200.00 Duration in seconds : 10.32 THROUGHPUT (Mbps) : 5199.80 OS UDP packet rx errors : delta 20279 Transfer type : lossy Missing data blocks count : 20499 (10.01% of data) per user-specified time window constraint > set blocksize 32768 > set speedup 1/6 > set rate 5g ... Mbits of data transmitted : 51200.00 Duration in seconds : 11.31 THROUGHPUT (Mbps) : 4747.94 OS UDP packet rx errors : delta 1986 Transfer type : lossy Missing data blocks count : 1987 (0.97% of data) per user-specified time window constraint ------------------------------------------------------------------------------------------- With Tsunami blocksize to match MTU: ------------------------------------------------------------------------------------------- 9000 MTU so using 8192 blocksize oper@juliano:/usr/src/tsunami-udp-v12/client$ ./tsunami connect 193.166.42.22 set rate 4g set lossless no set losswindow 0 set speedup 1/6 set blocksize 8192 ... Mbits of data transmitted : 51200.00 Duration in seconds : 15.07 THROUGHPUT (Mbps) : 3562.68 OS UDP packet rx errors : delta 90946 Transfer type : lossy Missing data blocks count : 92443 (11.28% of data) per user-specified time window constraint compared to 32kB blocksize Mbits of data transmitted : 51200.00 Duration in seconds : 13.97 THROUGHPUT (Mbps) : 3843.35 OS UDP packet rx errors : delta 0 Transfer type : lossy Missing data blocks count : 1 (0.00% of data) per user-specified time window constraint ------------------------------------------------------------------------------------------- With client RAID ------------------------------------------------------------------------------------------- oper@abidal:/raid$ tsunami connect juliano set rate 4g set lossless no set losswindow 0 set speedup 1/6 FULL! -- ring_reserve() blocking. space_ready = 0, data_ready = 1 a lot... Mbits of data transmitted : 48828.12 Duration in seconds : 13.00 THROUGHPUT (Mbps) : 3939.12 OS UDP packet rx errors : delta 94563 Transfer type : lossy Missing data blocks count : 92079 (47.14% of data) per user-specified time window constraint Increased ringbuffer size from 4096 to 8192 in ./include/tsunami-client.h FULL! -- ring_reserve() blocking. space_ready = 0, data_ready = 1 a lot... Mbits of data transmitted : 48828.12 Duration in seconds : 13.77 THROUGHPUT (Mbps) : 3717.52 OS UDP packet rx errors : delta 88739 Transfer type : lossy Missing data blocks count : 90485 (46.33% of data) per user-specified time window constraint tsunami> set buffer 64000000 FULL! -- ring_reserve() blocking. ... Mbits of data transmitted : 48828.12 Duration in seconds : 13.77 THROUGHPUT (Mbps) : 3717.23 OS UDP packet rx errors : delta 65036 Transfer type : lossy Missing data blocks count : 63462 (32.49% of data) per user-specified time window constraint oper@abidal:/raid$ tsunami connect juliano set rate 2g set lossless no set losswindow 0 set speedup 1/6 Mbits of data transmitted : 48828.12 Duration in seconds : 24.97 THROUGHPUT (Mbps) : 2050.60 OS UDP packet rx errors : delta 0 Transfer type : lossy Missing data blocks count : 1 (0.00% of data) per user-specified time window constraint oper@abidal:/raid$ tsunami connect juliano set rate 3g set lossless no set losswindow 0 set speedup 1/6 Mbits of data transmitted : 48828.12 Duration in seconds : 16.35 THROUGHPUT (Mbps) : 3131.12 OS UDP packet rx errors : delta 0 Transfer type : lossy Missing data blocks count : 1 (0.00% of data) per user-specified time window constraint oper@abidal:/raid$ tsunami connect juliano set rate 3500m set lossless no set losswindow 0 set speedup 1/6 Mbits of data transmitted : 48828.12 Duration in seconds : 13.82 THROUGHPUT (Mbps) : 3705.35 OS UDP packet rx errors : delta 102 Transfer type : lossy Missing data blocks count : 103 (0.05% of data) per user-specified time window constraint oper@abidal:/raid$ tsunami connect juliano set rate 4g set speedup 1/6 FULL! -- ring_reserve() blocking. ... Mbits of data transmitted : 48828.12 Duration in seconds : 25.07 THROUGHPUT (Mbps) : 2042.62 OS UDP packet rx errors : delta 4331 Transfer type : lossless => we get at most 3.7 Gbit/s for memory-to-disk