Fermilab Computing Division

CS Document 4492-v1

End-To-End Solution for Integrated Workload and Data Management using glideinWMS and Globus Online

Document #:
Document type:
Submitted by:
Parag A Mhashilkar
Updated by:
Parag A Mhashilkar
Document Created:
24 Oct 2011, 10:41
Contents Revised:
24 Oct 2011, 10:41
Metadata Revised:
12 Jun 2012, 14:43
Viewable by:
  • Public document
Modifiable by:

Quick Links:
Latest Version

Parag Mhashilkar, Gabriele Garzoglio, Burt Holzman, Xi Duan, Zach Miller, Cathrin Weiss, Raj Kettimuthu, Lukasz Lacinski

Grid computing has enabled scientific communities to effectively share computing resources distributed over many independent sites. Several such communities, or Virtual Organizations (VO), in the Open Science Grid and the European Grid Infrastructure use the glideinWMS system to run complex application work-flows. GlideinWMS is a pilot-based workload management system (WMS) that creates on demand, dynamically-sized overlay Condor batch system on Grid resources. While the WMS addresses the management of compute resources, however, data management in the Grid is still the responsibility of the VO. In general, large VOs have resources to develop complex custom solutions, while small VOs would rather push this responsibility to the infrastructure. The latter requires a tight integration of the WMS and the data management layers, an approach still not common in modern Grids. In this paper we describe a solution developed to address this shortcoming in the context of Center for Enabling Distributed Petascale Science (CEDPS) by integrating glideinWMS with Globus Online (GO). GO is a fast, reliable file transfer service that makes it easy for any user to move data. The solution eliminates the need for the users to provide custom data transfer solutions in the application by making this functionality part of the glideinWMS infrastructure. To achieve this, glideinWMS uses the file transfer plug-in architecture of Condor. The paper describes the system architecture and how this solution can be extended to support data transfer services other than GO when used with Condor or glideinWMS.

Files in Document:
Other Files:
Notes and Changes:
Duplicate of #4744. Refer to #4744 for talk and paper
Associated with Events:
CHEP 2012 held on 21 May 2012 in New York, New York
DocDB Home ]  [ Search ] [ Authors ] [ Events ] [ Topics ]

DocDB Version 8.8.9, contact Document Database Administrators