Data Services Newsletter

Volume 2 : No 4 : December 2000

Data Transfer to the DMC

Note: This method is from 2000 and is outdated. Please visit the page describing the current method.

Network operators play an important role in the flow of data from both temporary and permanent network installations. In order to make the transfer of data to the DMC more efficient (on both ends), the DMC is continually working to improve the information given to network operators about data flow to and from the DMC.

Data can be submitted to the DMC by tape or over the Internet. Network operators are encouraged to submit data in SEED format in near real-time. This “concurrent” method of data submission is beneficial to the network operator because much of the data handling and archiving issues are removed from their routine and they are able to request their data from the DMC almost immediately (depending on how the data is sent to the DMC).

The following is a breakdown of the steps that network operators need to take before and during data transfer to the DMC:

Before data is sent:

  1. Request FDSN Network code
  2. If temporary array, complete mobilization form
  3. Submit test volume(s)
  4. Get a tape label from the DMC and correctly state number of volumes/tapes OR contact engine_room@iris.washington.edu to set up an FTP directory
  5. Make clear any special considerations

Accepted Data Formats

  • Full SEED
    • must pass verseed check
    • must be at least SEED version 2.3
  • Dataless Seed and miniSEED
    • miniSEED must contain Blockette 1000’s and be at least SEED v2.4
    • dataless must represent entire network for all time
    • ending effective times should be for the last station day of the last year of deployment
    • data record length = 512 byte, 1k, or 4k for miniSEED, but dataless SEED volumes should always be written with 4k logical record length

Tape formats

  • 4mm DAT
  • 8mm Exabyte, either 2 or 5 gigabyte
  • DLT, either type III or IV

The data on tape should preferably be:

  1. Station/day, including all streams per tape file
  2. Include 5 EOF marks at End of Data (EOT)
  3. Written with the UNIX utility dd:
DD if=seed.file of=/dev/nrst# bs=32k

On-line transfers

1. Before shipping data, contact Rick Benson so that a directory can be created on the machine:

ftp.iris.washington.edu:/pub/dropoff/buffer/NC

(where NC is the ‘network code’ of your experiment)

2. Dataless SEED volumes are required prior to ANY archiving of timeseries data. The IRIS DMC is reliant upon the metadata found in dataless SEED volumes to archive, manage, and distribute the waveform data. MiniSEED data by itself is incomplete. For further discussion of this and/or there are special circumstances that would bypass this requirement, please email rick@iris.washington.edu

The data should preferably include:

1. Entire network days of data, with the exception of large networks where data volumes would be too large and unwieldy for internet transfers or where networks receive data from sites within their network at various times.
2. Checksum file for each file transferred, using the ‘cksum’ utility. Example:

00097.CI.2000.279.seed 00097.CI.2000.279.cksum

The contents of the cksum file is the output of this:

cksum 00097.CI.2000.279.seed >& 00097.CI.2000.279.cksum

When the two files arrive to the DMC, we generate a checksum file and compare it with the contents of what is sent:

cksum 00097.CI.2000.279.seed |diff - 00097.CI.2000.279.cksum

What happens to the data once it arrives at the DMC

  1. Data information is entered into a spreadsheet and assigned a unique internal DMC tracking number
  2. Data is sent to operations for processing
  3. Tapes are tagged with color coded “Network” label
  4. The meta data from the dataless is loaded into the database
  5. The data is archived, sent to mass store and time series information is loaded into the database
  6. The data is quality checked, then exported to a “public” machine for publication and release (if not under restricted access rules)
  7. Requested by users and distributed

by Deborah Barnes and Rick Benson (IRIS Data Management Center)

07:47:59 v.22510d55