Fermilab Computing Division

CS Document 4966-v3

Optimizing Large Data Transfers over 100Gbps Wide Area Networks

Document #:
CS-doc-4966-v3
Document type:
Conference
Submitted by:
Parag A Mhashilkar
Updated by:
Parag A Mhashilkar
Document Created:
29 Nov 2012, 13:54
Contents Revised:
14 May 2013, 08:52
Metadata Revised:
14 May 2013, 08:56
Viewable by:
  • Public document
Modifiable by:

Quick Links:
Latest Version

Other Versions:
Abstract:
Abstract— The Advanced Networking Initiative (ANI) project from the Energy Services Network provides a 100 Gbps testbed, which offers the opportunity for evaluating applications and middleware used by scientific experiments. This testbed is a prototype of a 100 Gbps wide-area network backbone, which links several Department of Energy (DOE) national laboratories, universities and other research institutions. These scientific experiments involve movement of large datasets for collaborations among researchers at different sites and thus require advanced infrastructure for supporting large and fast data transfers. A 100 Gbps network testbed is a key component of the ANI project and is used for DOE’s science research programs. This work presents results towards obtaining maximum throughput in large data transfers by optimizing and fine-tuning scientific applications and middleware to use this advanced infrastructure efficiently. A detailed performance evaluation is discussed measuring both applications, from High Energy Physics (HEP) and from data transfer middleware (GridFTP, Globus Online, Storage Resource Management, XrootD and Squid) at 100 Gbps speeds and 53 ms of latency. Results show that up to 97% efficiency of such high bandwidth high latency network is possible, achieving 80-90 Gbps in most test cases with a peak transfer rate of 100 Gbps.
Keywords:
HTDP Network CCGrid 100G
DocDB Home ]  [ Search ] [ Authors ] [ Events ] [ Topics ]

DocDB Version 8.8.9, contact Document Database Administrators