Fermilab Computing Division

CS Document 5771-v1

CHEP 2016 - Accessing Data Federations with CVMFS

Document #:
Document type:
Submitted by:
Dave Dykstra
Updated by:
Dave Dykstra
Document Created:
05 Jul 2016, 15:05
Contents Revised:
03 Oct 2017, 12:45
Metadata Revised:
03 Oct 2017, 12:45
Viewable by:
  • Public document
Modifiable by:

Quick Links:
Latest Version

Other Versions:
05 Jul 2016, 15:09
Data federations have become an increasingly common tool for large collaborations such as CMS and Atlas to efficiently distribute large data files. Unfortunately, these typically are implemented with weak namespace semantics and a non-POSIX API. On the other hand, CVMFS has provided a POSIX-compliant read-only interface for use cases with a small working set size (such as software distribution). The metadata required for the CVMFS POSIX interface is distributed through a caching hierarchy, allowing it to scale to the level of about a hundred thousand hosts. In this paper, we will describe our contributions to CVMFS that merges the data scalability of XRootD-based data federations (such as AAA) with metadata scalability and POSIX interface of CVMFS. We modified CVMFS so it can serve unmodified files without copying them to the repository server. CVMFS 2.2.0 is also able to redirect requests for data files to servers outside of the CVMFS content distribution network. Finally, we added the ability to manage authorization and authentication using security credentials such as X509 proxy certificates. We combined these modifications with the OSGs StashCache regional XRootD caching infrastructure to create a cached data distribution network. We will show performance metrics accessing the data federation through CVMFS compared to direct data federation access. Additionally, we will discuss the improved user experience of providing access to a data federation through a POSIX filesystem.
Files in Document:
Associated with Events:
CHEP 2016 held from 10 Oct 2016 to 14 Oct 2016 in San Francisco
DocDB Home ]  [ Search ] [ Authors ] [ Events ] [ Topics ]

DocDB Version 8.8.9, contact Document Database Administrators