Fermilab Computing Division

CS Document 6598-v1

The Open High Throughput Computing Content Delivery Network

Document #:
Document type:
Submitted by:
Dave Dykstra
Updated by:
Dave Dykstra
Document Created:
28 Nov 2018, 15:37
Contents Revised:
28 Nov 2018, 15:38
Metadata Revised:
28 Nov 2018, 15:44
Viewable by:
  • Public document
Modifiable by:

Quick Links:
Latest Version

LHC experiments make extensive use of web proxy caches, especially for software distribution via the CernVM File System and for conditions data via the Frontier Distributed Database Caching system. Since many jobs read the same data, cache hit rates are high and hence most of the traffic flows efficiently over Local Area Networks. However, it is not always possible to have local web caches, particularly for opportunistic cases where experiments have little control over site services. The Open High Throughput Computing (HTC) Content Delivery Network (CDN), openhtc.io, aims to address this by using web proxy caches from a commercial CDN provider. Cloudflare provides a simple interface for registering DNS aliases of any web server and does reverse proxy web caching on those aliases. The openhtc.io domain is hosted on Cloudflare's free tier CDN which has no bandwidth limit and makes use of data centers throughout the world, so the average performance for clients is much improved compared to reading from CERN or a Tier 1. The load on WLCG servers is also significantly reduced. WLCG Web Proxy Auto Discovery is used to select local web caches when they are available and otherwise select openhtc.io caching. This paper describes the Open HTC CDN in detail and provides initial results from its use for LHC@Home and USCMS opportunistic computing.
Files in Document:
Notes and Changes:
Associated with Events:
CHEP 2018 held from 09 Jul 2018 to 13 Jul 2018 in Sofia, Bulgaria
DocDB Home ]  [ Search ] [ Authors ] [ Events ] [ Topics ]

DocDB Version 8.8.9, contact Document Database Administrators