Fermilab Computing Division

CS Document 6733-v6

Distributing User Code with the CernVM FileSystem

Document #:
CS-doc-6733-v6
Document type:
Conference
Submitted by:
Dave Dykstra
Updated by:
Dave Dykstra
Document Created:
03 May 2019, 17:29
Contents Revised:
12 Mar 2020, 12:59
Metadata Revised:
12 Mar 2020, 12:59
Viewable by:
  • Public document
Modifiable by:

Quick Links:
Latest Version

Other Versions:
CS-doc-6733-v5
04 Feb 2020, 17:04
CS-doc-6733-v4
04 Nov 2019, 16:08
CS-doc-6733-v3
30 Oct 2019, 17:18
CS-doc-6733-v2
30 Oct 2019, 17:13
CS-doc-6733-v1
29 Oct 2019, 17:44
CS-doc-6733-v0
03 May 2019, 17:31
Abstract:
The CernVM FileSystem (CVMFS) is widely used in High Throughput Computing to efficiently distributed experiment code. However, the standard CVMFS publishing tools are designed for a small group of people from each experiment to maintain common software, and the tools are not a good fit for publishing software from numerous users in each experiment. As a result, most user code, such as code to do specific physics analyses, is still sent with every job to the place the job is run. That process is relatively inefficient, especially when the user code is large. To overcome these limitations, we have built a CVMFS user code publication system. This publication system enables users to still submit their code with their jobs but the code is distributed and accessed through the standard CVMFS infrastructure. The user code is automatically deleted from CVMFS after a period of no use. Most of the software for the system is available as a single self-contained open source rpm called cvmfs user pub and is available for other deployments.
Files in Document:
  • Paper pdf (CHEP19_Paper_Userpub.pdf, 415.4 kB)
  • Talk pdf (CHEP19_Talk_Userpub.pdf, 178.0 kB)
Other Files:
Associated with Events:
CHEP 2019 held on 04 Nov 2019 in Adelaide, Australia
DocDB Home ]  [ Search ] [ Authors ] [ Events ] [ Topics ]

DocDB Version 8.8.9, contact Document Database Administrators