Fermilab Computing Division

CS Document 6887-v1

Concurrent Conditions Access across Validity Intervals in CMSSW

Document #:
CS-doc-6887-v1
Document type:
Conference
Submitted by:
W. David Dagenhart
Updated by:
W. David Dagenhart
Document Created:
30 Oct 2019, 11:43
Contents Revised:
30 Oct 2019, 11:43
Metadata Revised:
30 Oct 2019, 11:43
Viewable by:
  • Public document
Modifiable by:

Quick Links:
Latest Version

Abstract:
The CMS software system, known as CMSSW, has a generalized conditions, calibration, and geometry data products system called the EventSetup. The EventSetup caches results of reading or calculating data products based on the 'interval of validity', IOV, which is based on the time period for which that data product is appropriate. With the original single threaded CMSSW framework, updating only on an IOV boundary meant we only required memory for a single data product of a given type at any time during the program execution. In 2016 CMS transitioned to using a multi-threaded framework as a way to save on memory during processing. This was accomplished by amortizing the memory cost of EventSetup data products across multiple concurrent events. To initially accomplish that goal required synchronizing event processing across IOV boundaries, thereby decreasing the scalability of the system. In this presentation we will explain how we used 'limited concurrent task queues' to allow concurrent IOVs while still being able to limit the memory utilized. In addition, we will present performance measurements for both threading scalability of event throughtput and memory utilization.
Files in Document:
  • Poster (CHEP2019Poster_3.pdf, 2.3 MB)
Associated with Events:
CHEP 2019 held on 04 Nov 2019 in Adelaide, Australia
DocDB Home ]  [ Search ] [ Authors ] [ Events ] [ Topics ]

DocDB Version 8.8.9, contact Document Database Administrators