Fermilab Computing Division

CS Document 5750-v1

HEPCloud, a new paradigm for HEP facilities: CMS Amazon Web Services Investigation

Document #:
Document type:
Technical Note
Submitted by:
Burt Holzman
Updated by:
Burt Holzman
Document Created:
12 May 2016, 11:08
Contents Revised:
12 May 2016, 11:08
Metadata Revised:
14 Mar 2019, 14:55
Viewable by:
  • Public document
Modifiable by:

Quick Links:
Latest Version

The HEPCloud Facility concept is envisioned to be a portal to an ecosystem of computing resources---commercial or academic---that will help our facilities move away from standalone, siloed solutions. It will provide "complete solutions" to all users, with agreed-upon levels of service, routing user workflows to local ("owned") or remote ("rental") resources based on efficiency, cost, workflow requirements and the policies of the facilities hosting the resources. This will be done transparently to the users, utilizing a sophisticated decision engine and cost model, and incorporating the necessary policies and infrastructure to manage allocations to potential computing resources (local or remote). The portal concept could provide the means for all laboratories to provide shared resources in the ecosystem, resulting in a large pool of offerings for compute, data archival capabilities, database services, data management, etc., potentially linking all US HEP computing. In order to investigate the merit of this approach, we established the Fermilab HEPCloud project. The goal is to integrate rented resources into the current Fermilab computing facility in a manner transparent to the user. The first type of external resources considered was commercial clouds, partnering with Amazon Web Services as the provider. For our studies, we identified use cases that both demonstrate the necessary aspects of the concept, and that are also useful to the experimenters. One of the use cases focused on CMS Monte Carlo generation and reconstruction, targeting physics results for the Moriond conference in March 2016. This use case studied the scalability and sustainability of elastic provisioning of AWS resources through the portal, and exercised the prototype decision engine and cost model.
Files in Document:
Publication Information:
DocDB Home ]  [ Search ] [ Authors ] [ Events ] [ Topics ]

DocDB Version 8.8.10, contact Document Database Administrators