Fermilab Computing Division

CS Document 4596-v1

File Aggregation in Enstore

Document #:
Document type:
Submitted by:
Alexander Moibenko
Updated by:
Alexander Moibenko
Document Created:
05 Jan 2012, 17:27
Contents Revised:
05 Jan 2012, 17:27
Metadata Revised:
05 Jan 2012, 17:27
Viewable by:
  • Public document
Modifiable by:

Quick Links:
Latest Version

Tape drive transfer rates depend on the size of the files. If the size of a file is relatively small then overall average data throughput rate is less than for the relatively big file size. There are several reasons for this: seeking to the files position, starts/ stops etc. The optimal size for a file on tape depends on the tape technology is 2 GB. Today's large file is tomorrow's small file.
Users can not always easily control the size of files they write to tapes, and many storage systems provide transparent aggregation of files for the user. Enstore does not provide this functionality and this project is intended to provide this. It is of particular interest to existing and upcoming Neutrino experiments, whose data files are typically small.
Another fact to take into considerations is that there are already many files with relatively small sizes stored in the Fermilab enstore tape systems. We are migrating from tapes with small files to tapes with much larger capacity, resulting in tapes with tens of thousands of small files on them. The access to such files can be quite slow and inefficient, tying up valuable resources. To optimize access and transfer rates for relatively small files we need to create a mechanism of packaging such files as a single entity (package), stored on tape, while at the same time permitting transparent access to each file in a package.
This document describes the structure details of design details and operation of Enstore file aggregation feature.
Files in Document:
DocDB Home ]  [ Search ] [ Authors ] [ Events ] [ Topics ]

DocDB Version 8.8.9, contact Document Database Administrators