Fermilab Computing Division

CS Document 2989-v2

Greatly improved cache update times for conditions data with Frontier/Squid

Document #:
Document type:
Submitted by:
Lee Lueking
Updated by:
Lee Lueking
Document Created:
12 Nov 2008, 11:42
Contents Revised:
15 May 2009, 15:07
Metadata Revised:
15 May 2009, 15:11
Viewable by:
  • Public document
Modifiable by:

Quick Links:
Latest Version

Other Versions:
13 Nov 2008, 10:32
The CMS experiment requires worldwide access to conditions data by nearly a hundred thousand processing jobs daily. This is accomplished using a software subsystem called Frontier. This system translates database queries into http, looks up the results in a central database at CERN, and caches the results in an industry-standard http proxy/caching server called Squid. One of the most challenging aspects of any cache system is coherency, that is, ensuring that changes made to the underlying data get propagated out to all clients in a timely manner. Recently, the Frontier system was enhanced to drastically reduce the time for changes to be propagated everywhere, typically as low as 10 minutes for some kinds of data and no more than 60 minutes for the rest of the data, without overloading servers. This was done by taking advantage of an http and Squid feature called "If-Modified-Since" in which the "Last-Modified" timestamp of cached data is sent back to the central server. The server responds to this with a very short message if data has not been modified, which is the case most of the time, and re-validates the cache. In order to use this feature, the Frontier server has to send the "Last-Modified" timestamp, but that information is not normally stored by the Oracle databases so a PL/SQL program was developed to keep track of the modification times of database tables. We discuss the details of this caching scheme and the obstacles overcome including Oracle database and Squid bugs.
Files in Document:
Associated with Events:
CHEP 2009 held from 21 Mar 2009 to 27 Mar 2009 in Prague, Czech Republic
DocDB Home ]  [ Search ] [ Authors ] [ Events ] [ Topics ]

DocDB Version 8.8.9, contact Document Database Administrators