Fermilab Computing Division

CS Document 6733-v5

Distributing User Code with the CernVM FileSystem

Document #:
CS-doc-6733-v5
Document type:
Conference
Submitted by:
Dave Dykstra
Updated by:
Dave Dykstra
Document Created:
03 May 2019, 17:29
Contents Revised:
05 Nov 2019, 14:29
Metadata Revised:
05 Nov 2019, 14:29
Viewable by:
  • Public document
Modifiable by:

Quick Links:
Latest Version

Other Versions:
CS-doc-6733-v4
04 Nov 2019, 16:08
CS-doc-6733-v3
30 Oct 2019, 17:18
CS-doc-6733-v2
30 Oct 2019, 17:13
CS-doc-6733-v1
29 Oct 2019, 17:44
CS-doc-6733-v0
03 May 2019, 17:31
Abstract:
The CernVM FileSystem (CVMFS) is widely used in High Throughput Computing to efficiently distributed experiment code. However, the standard CVMFS publishing tools are designed for a small group of people from each experiment to maintain common software, and the tools don't work well for the majority of users that submit jobs related to each experiment. As a result, most user code, such as code to do specific physics analyses, is still sent with jobs. That process is relatively inefficient, especially when the user code is large. To overcome these limitations, we have built a CVMFS user code publication system. Users submit code as a tarball with their jobs, and the job submission client sends the tarball to one of a pair of user code publishing servers. The code is published in one of a small number of shared CVMFS repositories, and updates are expedited through the standard CVMFS infrastructure to be available on worker nodes worldwide within several minutes. A job wrapper automatically passes the location of the code to the jobs. The user code is automatically deleted from CVMFS after a period of no use. Most of the software for the system is available as a single self-contained open source rpm called cvmfs-user-pub, and is available for other deployments.
Files in Document:
  • Talk pdf (CHEP19_Talk_Userpub.pdf, 178.0 kB)
Other Files:
Associated with Events:
CHEP 2019 held on 04 Nov 2019 in Adelaide, Australia
DocDB Home ]  [ Search ] [ Authors ] [ Events ] [ Topics ]

DocDB Version 8.8.9, contact Document Database Administrators