Fermilab Computing Division

CS Document 5239-v1

The last mile of data handling: Fermilab's IFDH tools

Document #:
CS-doc-5239-v1
Document type:
Technical Note
Submitted by:
Marc Mengel
Updated by:
Marc Mengel
Document Created:
29 Oct 2013, 13:40
Contents Revised:
29 Oct 2013, 13:40
Metadata Revised:
29 Oct 2013, 13:40
Viewable by:
  • Public document
Modifiable by:

Quick Links:
Latest Version

Abstract:
IFDH (Intensity Frontier Data Handling), is a suite of tools for data
movement tasks for Fermilab experiments and is an important part of
the FIFE (Fabric for Frontier Experiments) initiative described at
this conference. IFDH encompasses moving input data from caches or
storage elements to compute nodes (the "last mile"
of data movement) and moving output data potentially to those caches
as part of the journey back to the user. IFDH also involves throttling
and locking to ensure that large numbers of jobs do not cause data
movement bottlenecks. IFDH is realized as an easy to use layer that
users call in their job scripts (e.g. "ifdh cp"),
hiding the low level data movement tools. One advantage of this layer
is that the underlying low level tools can be selected or changed
without the need for the user to alter their scripts. Logging and
performance monitoring can also be added easily. This system will
be presented in detail as well as its impact on the ease of data handling
at Fermilab experiments.
Files in Document:
DocDB Home ]  [ Search ] [ Authors ] [ Events ] [ Topics ]

DocDB Version 8.8.9, contact Document Database Administrators