IRIS Home  
site map contact search  
data software stations and instrumentation  
You are here: IRIS > Data > Data Notice
 

Data

 

 

 

 

22 second Timing Issue with IRIS DMC EarthWorm Real-Time Data:

The 11-step recovery program

 

On Sept 11, 2006, the staff at the IRIS DMC was notified by Chuck Ammon that data being collected in the IRIS DMC’s BUD filesystem were plagued with “strange time shifts” in some data. His first clue, and good observation, was revealed when plotting record sections of well-recorded earthquakes, illustrating that the data was routinely time-stamped 22 seconds earlier than was predicted using tau-p software. Most users that submitted requests to the DMC archive will receive an email guiding them to get these data resent. (Read the section below entitled “The Solution” for more detail.) Below is an example to illustrate the nature of the problem, where you can observe that the true arrival is clearly multiple seconds before the predicted travel times:

 

The Problem:
Upon observing this behavior in the data, research was begun to find the source of the problem. It was determined that one piece of software, called ew2mseed, had a compiled version difference in the way the software was compiled, and there was a difference between the two real-time systems that are behind collecting real-time data at the IRIS DMC. We have real-time data flowing into two machines, called usra and bud. This allowed us to compare outputs, and it was diagnosed that only this one piece of code (fortunately) was not introducing the sum of the leap seconds since 1972 (when leap seconds were introduced), on the machine bud. This value is exactly 22 seconds. Just for reference, below is a plot of the accumulation of seconds related to keeping uniform time between atomic clocks and the Earth’s rotational time. The important thing to know is that the ew2mseed software utilizes this value when writing time values into the miniSEED data as it comes in from the EarthWorm waveservers.

So, between July 19 and Sept 11, 2006, the IRIS real-time system, located on the machine bud, was mis-timing data with an offset of 22 seconds early. The summary of affected stations can be found online at  http://www.iris.edu/data/problem/badstas.txt

The Solution:
The staff at IRIS developed a remedy to fix these data, with both software and a process that became affectionately named the “11-step process”, related to the interwoven data management environment that protects data in our data center. (there are multiple copies on multiple mass storage systems). The process meant that all 35,000 station/day files that contained “R” quality data, (“R” quality means that the data was collected in the BUD filesystem) that were initially ingested using the earthworm software, were corrected by adding 22 seconds, and re-archived. The script that documents this and which was used to do this is called
dbserv1:/usr/local/bin/ops/ew2mseed22.csh for the /Isilon copies, and
dbserv1:/usr/local/bin/ops/ew2mseed22.purg.csh
if the data were not ever archived, leaving them in “purgatory”.
The completion date for this was Nov 3, 2006.

IMPORTANT NOTE:

Users who requested data from the archive during this time interval are going to be contacted via email that was supplied when they submitted requests. However, there may have been requests for data via the DHI request mechanism (examples are Vase and JWeed) that are not tracked back to individual users, so if there is ANY question, you should contact rick@iris.washington.edu and/or re-request data from the networks listed in the document mentioned earlier, http://www.iris.edu/data/problem/badstas.txt.

Rick Benson
Director of Operations
IRIS DMC