Data Services Newsletter

Volume 24 : No 3 : Winter 2022

The Last IRIS Data Services Newsletter

This is the last newsletter that will be published by IRIS Data Services. IRIS is merging with our geodetic counterpart (UNAVCO) and will become EarthScope Consortium on January 1, 2023. The new organization will be much larger than IRIS and will provide a more diverse range of skills to draw upon for providing the data services that support the community’s research, monitoring, and educational needs. At the recommendation of the NSF after a 2019 joint review of our data services, IRIS and UNAVCO data services staff have been working together for the last few years to integrate our respective seismological and geodetic systems and move the resulting system to the cloud. This has been a huge undertaking that has given us an opportunity to rethink how we can provide data services that not only service the traditional seismic and geodetic requests for data and products, but that also remove data handling barriers for research that requires access to massive amounts of data. The new system, which we have named the Common Cloud Platform (CCP), is an evolutionary step that is helping us retire our technical debt and improve our services.

As this is the last newsletter from IRIS data services, it is appropriate to look back upon the 34-year history of the DMC and celebrate the remarkable success in providing data, products, and services to the community.

1988

  • The first, interim IRIS Data Management Center is formed in Austin, TX.
  • The most common request format, BREQ_FAST (Batch Request, Fast) was developed and is still an active request mechanism today.

1991

  • DMC moves to Seattle and is affiliated with the UW. The staff numbers 3 people.
  • IRIS DMC acquired its first mass storage system, a Metrum RSS-600, capable of 6 TB of storage.

1994

  • DMC establishes a WWW home page for public access. The FARM event data filesystem assembles easy access to earthquake datasets.

1997

  • The 2nd generation of tape storage, the StorageTek Wolfcreek robotic archive (partially donated by StorageTek) is purchased using NSF MRI and Keck Foundation funds and put into production.
  • WILBER is released to support discovery and access to FARM data products.

2000

  • Operations moves into Electronic Data Reception to make it easier to ingest and manage data. This eliminates tape-based shipments to the DMC.

2001

  • The 3rd generation of tape storage, the StorageTek Powderhorn tape library, is put into production.
  • IRIS DMC software engineers release a CORBA-based data delivery service called DHI. WILBER II is put into production.

2003

  • The first-generation quality metrics system, QUACK (QUality Analysis Control Kit), is introduced.

2004

  • The USArray data pulse begins, and staff is hired to manage and perform QC on this special collection, including a systems administrator.

2005

  • The 4th generation of storage is purchased, this time RAID disk from Isilon Systems. The price point of disk now makes it possible to store all data on RAID disk to assist with increases in user requests for data.

2006

  • IRIS participates in the EarthScope Portal project, combining USArray data with the SAFOD, PBO, and USArray program data in a cooperative effort.

2009

  • The 5th generation of storage is purchased, the Isilon Systems RAID cluster with higher capacity disk trays, enabling disk storage for the doubling of distributed data seen annually.

2010

  • DMC unveils IRIS Web Services in public beta.
  • The GMV (Ground Motion Visualization) and EARS (EarthScope Automated Receiver Survey, originally developed at the University of South Carolina) products are launched.

2011

  • Six web services are made production-ready: event (earthquake), processed time series, station metadata, SAC poles and zeros, instrument response evaluation, Flinn-Engdahl region codes, and distance-azimuth calculation. Command line client scripts are developed for web service data access.
  • The next-generation quality metrics system, MUSTANG, begins development.
  • Several new products are released: Event Plots, Moment Tensor (from the GCMT Project at Lamont-Doherty Earth Observatory), Back-projection, EMTF (Magnetotelluric Transfer Functions, calculated by Oregon State University), and EMC (Earth Model Collaboration).

2012

  • The 6th generation of storage is purchased from Hitachi.
  • Auxiliary Data Center Enabled: The Active Backup was moved to the Lawrence Livermore National Laboratories Facilities and upgraded to a functional failover, load-balanced, webservice-capable endpoint. The benefit also includes the proximity to High Performance Computing at LLNL that could be used to test data extraction.
  • The FDSN approves a standard specification for three web service components: event, station, and dataselect. Tools that utilize the standard are released: the IRIS Java Web Services Library and the MATLAB® interface for web services.
  • Two new services are released: virtual network definitions and travel time calculator.
  • Products released include: Event Bulletins, Film Chips, Calibration, Station Digest, SWS-DBs (Shear-wave splitting databases from the Geosciences Montpellier SplitLab and the Missouri University of Science and Technology), SeisSound, ANCC-CIEI, (Western US Ambient Noise Cross-Correlations), Infrasound, and ShakeMovieSynthetics.

2013

  • IRIS makes its primary web services FDSN-compliant. Web services are moved to their own domain address and old web services are retired.
  • A Web Service Shell module is developed that allows international data centers to easily deploy their own FDSN-compliant web services. Three data centers soon have web services deployed to their own datasets.
  • The CORBA-based DHI system at IRIS DMC is retired. WILBER 3 is released.
  • EMERALD, and EQEnergy are released as products.

2014

  • BREQ_FAST batch request processing is migrated over to web services. A metadata-change service is released.
  • MUSTANG is released and replaces QUACK as the primary source of quality metrics and PDFs for seismic data in the IRIS archive. The initial MUSTANG web services are measurements, noise-psd, noise-pdf, metrics, and targets.
  • Product releases include: Envelope Functions (computed by the Earth and Space Sciences Department of the University of Washington), ASWMS (Automated Surface Wave Phase Velocity Measuring System, from Columbia University), globalstacks, SourceTimeFunction, Aftershocks, and Noise Toolkit.

2015

  • IRIS releases the Fedcatalog service, which catalogs data offered by FDSN-compliant data centers around the world.
  • Event and Station services are updated to support the GeoCSV format.
  • The Syngine synthetic data service is released along with the GlobalEmpiricalGreensTensors (from the University of Rhode Island).

2016

  • Enhanced data plotting capabilities are provided from the Timeseriesplot service.
  • The MUSTANG web services noise-pdf-browser and noise-mode-timeseries are introduced. QUACK is officially retired.
  • AELUMA (Automated Event Location Using a Mesh of Arrays, developed by de Groot-Hedlin and Hedlin, 2015) is released.

2017

  • An immense effort to catalog every archived sample of data is expressed in the new Availability service. Dataselect efficiency is greatly improved. PH5 experiment data is released with FDSN-compliant web services. Research Ready Datasets (RRDS) web service for quality-controlled data access is released.
  • Inexpensive RAID is acquired to store the original versions of data received at the DMC.
  • ESEC (Exotic Seismic Events Catalog) is released as a product.

2017- 2018

  • The primary archive at the DMC is upgraded to contain what is termed the “best” version of data merging the original versions of the data, which may contain duplicate, overlapping, or quality-controlled segments.

2018

  • The 7th Generation of Hitachi storage controller heads are installed to increase performance and expand the feature set.
  • Large JBOD disk subsystems from SuperMicro are acquired and put into operations to manage PH5 data. These systems offer a lower cost per byte for data management and perform well for temporary deployment data in PH5 format.
  • A global load balancer makes web services fully available between primary and secondary data centers. A validator/converter tool is released for the benefit of network operators to migrate their metadata from Dataless SEED to StationXML.
  • Numerous request tools support federated data access to multiple data centers.
  • The MUSTANG web service noise-spectrogram is introduced. The first QA Section workshop “North American Seismic Network Training Workshop, Quality Assurance” is held in Seattle, WA.
  • The Surface-Wave Radiation Pattern product is released.

2019

  • The Hitachi RAID system is replaced by the 8th generation of the storage system.
  • An NSF review of IRIS and UNAVCO Data Services results in a recommendation to integrate the seismic and geodetic data management systems and move them to the cloud.
  • Seismometer data from the Mars Insight mission are first made available from IRIS.
  • Full SEED data volumes are retired.
  • ROVER, an application for retrieving large data sets is released along with an FDSN availability service for accessing time series data availability information.

2020

  • COVID strikes and DMC staff transition to remote work.
  • Work begins on integrating the seismic and geodetic data services systems for deployment in a cloud environment.
  • QuARG, the Quality Assurance Report Generator is released; it is intended for network operators to help keep track of problems in their network as they arise and are resolved.
  • The IRIS availability service is retired in favor of the FDSN availability service.

2021

  • PIQQA, the PI’s Quick Quality Assessment for PASSCAL experiments is released to leverage selected IRIS MUSTANG metrics to provide a broad overview of data availability and noise levels across the experiment.
  • MERMAID data are made available through the DMC.

2022

  • A web service is launched to deliver responses from the Nominal Response Library.
    IRIS and UNAVCO enter the final year of preparation for their merger into EarthScope Consortium.
  • The final DMC newsletter is published.

Through engaged community governance and diligent work by the dedicated staff the DMC has succeeded in delivering high-quality data, products, and services. While the IRIS name is being retired, the DMC will continue under its new EarthScope Consortium banner with even better services for meeting the future needs of the geophysical community.

by Jerry Carter (IRIS)

23:23:25 v.22510d55