Hello IRIS Web Service Support,
Our group with Univ of Utah MesoWest also appears to be seeing the same
issue as described below when utilizing the timeseries service. If a query
is made with a starting timestamp prior to 2020-02-11 00:00:00 it appears
to return properly, for example:
http://service.iris.edu/irisws/timeseries/1/query?net=N4&sta=060A&loc=30&cha=LDO&start=2020-02-10T23:00:00&end=2020-02-12T00:00:00&output=ascii
However if the starting timestamp is later than 2020-02-11 00:00:00, a 404
error is produced, for example:
http://service.iris.edu/irisws/timeseries/1/query?net=N4&sta=060A&loc=30&cha=LDO&start=2020-02-11T01:00:00&end=2020-02-12T00:00:00
&output=ascii
Regards
Alexander Jacques
Postdoctoral Research Associate
Department of Atmospheric Sciences
University of Utah
alexander.jacques<at>utah.edu
http://home.chpc.utah.edu/~u0790486/
---------- Forwarded message ---------
From: Daniel Hook <daniel.hook<at>esgsolutions.com>
Date: Thu, Feb 13, 2020 at 10:16 AM
Subject: Re: [IRIS][webservices] irisws timeseries query problem
To: Web Services <webservices<at>lists.ds.iris.edu>
We have observed the same problem for all stations that we monitor in our
software systems.
The issue started just before midnight on the 11th. Since then we find
that
• Querying data from times after 2020-02-11 00:00:00 result in a 404
error
• Querying data from times before 2020-02-11 00:00:00 will return
results
• Querying data that crosses this boundary will sometimes work (as
long as it includes time before 2020-02-11 00:00:00), but it is inconsistent
Note that queries built using the URL builder (
http://service.iris.edu/irisws/timeseries/docs/1/builder/) exhibit this
problem, so it's not a user issue.
WORKING ->
http://service.iris.edu/irisws/timeseries/1/query?net=IU&sta=ANMO&cha=BHZ&start=2020-02-10T23:00:50&end=2020-02-11T23:59:59&format=plot&loc=00
BROKEN ->
http://service.iris.edu/irisws/timeseries/1/query?net=IU&sta=ANMO&cha=BHZ&start=2020-02-11T00:00:00&end=2020-02-11T00:01:00&format=plot&loc=00
BROKEN ->
http://service.iris.edu/irisws/timeseries/1/query?loc=00&output=miniseed&sta=PB08&net=TX&cha=HHZ&start=2020.044T14:39:50&end=2020.044T14:45:10
----------------------
Web Services
Topic home: http://ds.iris.edu/message-center/topic/webservices/ |
Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
Update subscription preferences at http://ds.iris.edu/account/profile/
Our group with Univ of Utah MesoWest also appears to be seeing the same
issue as described below when utilizing the timeseries service. If a query
is made with a starting timestamp prior to 2020-02-11 00:00:00 it appears
to return properly, for example:
http://service.iris.edu/irisws/timeseries/1/query?net=N4&sta=060A&loc=30&cha=LDO&start=2020-02-10T23:00:00&end=2020-02-12T00:00:00&output=ascii
However if the starting timestamp is later than 2020-02-11 00:00:00, a 404
error is produced, for example:
http://service.iris.edu/irisws/timeseries/1/query?net=N4&sta=060A&loc=30&cha=LDO&start=2020-02-11T01:00:00&end=2020-02-12T00:00:00
&output=ascii
Regards
Alexander Jacques
Postdoctoral Research Associate
Department of Atmospheric Sciences
University of Utah
alexander.jacques<at>utah.edu
http://home.chpc.utah.edu/~u0790486/
---------- Forwarded message ---------
From: Daniel Hook <daniel.hook<at>esgsolutions.com>
Date: Thu, Feb 13, 2020 at 10:16 AM
Subject: Re: [IRIS][webservices] irisws timeseries query problem
To: Web Services <webservices<at>lists.ds.iris.edu>
We have observed the same problem for all stations that we monitor in our
software systems.
The issue started just before midnight on the 11th. Since then we find
that
• Querying data from times after 2020-02-11 00:00:00 result in a 404
error
• Querying data from times before 2020-02-11 00:00:00 will return
results
• Querying data that crosses this boundary will sometimes work (as
long as it includes time before 2020-02-11 00:00:00), but it is inconsistent
Note that queries built using the URL builder (
http://service.iris.edu/irisws/timeseries/docs/1/builder/) exhibit this
problem, so it's not a user issue.
WORKING ->
http://service.iris.edu/irisws/timeseries/1/query?net=IU&sta=ANMO&cha=BHZ&start=2020-02-10T23:00:50&end=2020-02-11T23:59:59&format=plot&loc=00
BROKEN ->
http://service.iris.edu/irisws/timeseries/1/query?net=IU&sta=ANMO&cha=BHZ&start=2020-02-11T00:00:00&end=2020-02-11T00:01:00&format=plot&loc=00
BROKEN ->
http://service.iris.edu/irisws/timeseries/1/query?loc=00&output=miniseed&sta=PB08&net=TX&cha=HHZ&start=2020.044T14:39:50&end=2020.044T14:45:10
----------------------
Web Services
Topic home: http://ds.iris.edu/message-center/topic/webservices/ |
Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
Update subscription preferences at http://ds.iris.edu/account/profile/
-
Hi everyone,
I'm encountering similar problems querying IRISWS timeseries and IRIS FDSN
dataselect. Attempts fail with both the web interface and my scripts in
SeisIO for Julia language.
- Web requests yield 404 Not Found.
- HTTP POST requests in Julia yield no data but no error is thrown.
- Haven't tried ObsPy.
As of this afternoon (2020-02-13), I get no data from requests with start
times more recent than 2020-02-13T0-:00:00. Earlier this morning, the
cutoff time was midnight UTC on the 11th, as others are reporting. Two
sample queries are below.
This returns no data. Note that I don't get a 404 using Julia with HTTP
POST, but the return is empty.
request url: http://service.iris.edu/fdsnws/dataselect/1/query
request body:
format=miniseed
UW TDH * EHZ 2020-02-13T01:00:00.000000 2020-02-13T02:00:00.000000
UW VLL * EHZ 2020-02-13T01:00:00.000000 2020-02-13T02:00:00.000000
CC VALT * * 2020-02-13T01:00:00.000000 2020-02-13T02:00:00.000000
This works perfectly and returns good data, including the full time window
of the previous query, on all channels.
request url: http://service.iris.edu/fdsnws/dataselect/1/query
request body:
format=miniseed
UW TDH * EHZ 2020-02-13T00:00:00.000000 2020-02-13T12:00:00.000000
UW VLL * EHZ 2020-02-13T00:00:00.000000 2020-02-13T12:00:00.000000
CC VALT * * 2020-02-13T00:00:00.000000 2020-02-13T12:00:00.000000
Checking the PNSN web tools, these stations are online and recording good
data.
Station and event queries appear to work at any start time, so I don't
think those services are affected.
-josh
---------------------------
Mail: Dr. Joshua P. Jones, 4509 NE Sumner St, Portland, OR 97218-1547,
United States of America
OrcID: https://orcid.org/0000-0002-3518-263
ResearchGate: https://www.researchgate.net/profile/Joshua_Jones4
GitHub: https://github.com/jpjones76
Note: there are several Earth science researchers named Joshua Jones.
On Thu, Feb 13, 2020 at 9:32 AM Alex Jacques <alexander.jacques<at>utah.edu>
wrote:
Hello IRIS Web Service Support,
Our group with Univ of Utah MesoWest also appears to be seeing the same
issue as described below when utilizing the timeseries service. If a query
is made with a starting timestamp prior to 2020-02-11 00:00:00 it appears
to return properly, for example:
http://service.iris.edu/irisws/timeseries/1/query?net=N4&sta=060A&loc=30&cha=LDO&start=2020-02-10T23:00:00&end=2020-02-12T00:00:00&output=ascii
However if the starting timestamp is later than 2020-02-11 00:00:00, a 404
error is produced, for example:
http://service.iris.edu/irisws/timeseries/1/query?net=N4&sta=060A&loc=30&cha=LDO&start=2020-02-11T01:00:00&end=2020-02-12T00:00:00
&output=ascii
Regards
Alexander Jacques
Postdoctoral Research Associate
Department of Atmospheric Sciences
University of Utah
alexander.jacques<at>utah.edu
http://home.chpc.utah.edu/~u0790486/
---------- Forwarded message ---------
From: Daniel Hook <daniel.hook<at>esgsolutions.com>
Date: Thu, Feb 13, 2020 at 10:16 AM
Subject: Re: [IRIS][webservices] irisws timeseries query problem
To: Web Services <webservices<at>lists.ds.iris.edu>
We have observed the same problem for all stations that we monitor in our
software systems.
The issue started just before midnight on the 11th. Since then we find
that
• Querying data from times after 2020-02-11 00:00:00 result in a 404
error
• Querying data from times before 2020-02-11 00:00:00 will return
results
• Querying data that crosses this boundary will sometimes work (as
long as it includes time before 2020-02-11 00:00:00), but it is inconsistent
Note that queries built using the URL builder (
http://service.iris.edu/irisws/timeseries/docs/1/builder/) exhibit this
problem, so it's not a user issue.
WORKING ->
http://service.iris.edu/irisws/timeseries/1/query?net=IU&sta=ANMO&cha=BHZ&start=2020-02-10T23:00:50&end=2020-02-11T23:59:59&format=plot&loc=00
BROKEN ->
http://service.iris.edu/irisws/timeseries/1/query?net=IU&sta=ANMO&cha=BHZ&start=2020-02-11T00:00:00&end=2020-02-11T00:01:00&format=plot&loc=00
BROKEN ->
http://service.iris.edu/irisws/timeseries/1/query?loc=00&output=miniseed&sta=PB08&net=TX&cha=HHZ&start=2020.044T14:39:50&end=2020.044T14:45:10
----------------------
Web Services
Topic home: http://ds.iris.edu/message-center/topic/webservices/ |
Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
Update subscription preferences at http://ds.iris.edu/account/profile/
----------------------
Web Services
Topic home: http://ds.iris.edu/message-center/topic/webservices/ |
Unsubscribe: webservices-unsubscribe<at>lists.ds.iris.edu
Sent from the IRIS Message Center (http://ds.iris.edu/message-center/)
Update subscription preferences at http://ds.iris.edu/account/profile/
-
Hi Josh,
It appears that the time range for your first request (2020-02-13T01:00:00 to 2020-02-13T02:00:00) falls entirely outside the window of data availability that we currently have archived, which would explain the No Data errors you’re seeing. However the second time window (2020-02-13T00:00:00 to 2020-02-13T12:00:00) overlaps with the few seconds of data at the start of the day that we do have available. So, it makes sense that you would receive the first seconds of data on day 2020-02-13 with a bigger time window.
If you (or anyone else) would like to know what is the most recent data that is available through IRIS web services, please consult with the fdsnws-availability service (http://service.iris.edu/fdsnws/availability/1/) and use the “extent” endpoint. In the example below, I’ve outlined the window of data availability in RED.
http://service.iris.edu/fdsnws/availability/1/extent?net=UW&sta=TDH,VLL&cha=EHZ&nodata=404&starttime=2020-02-01 http://service.iris.edu/fdsnws/availability/1/extent?net=UW&sta=TDH,VLL&cha=EHZ&nodata=404&starttime=2020-02-01>
#Network Station Location Channel Quality SampleRate Earliest Latest Updated TimeSpans Restriction
UW TDH -- EHZ M 100.0 2020-02-01T00:00:00.000000Z 2020-02-13T00:00:03.381312Z 2020-02-13T10:19:16Z 5 OPEN
UW VLL -- EHZ M 100.0 2020-02-01T00:00:00.000000Z 2020-02-13T00:00:05.971341Z 2020-02-13T10:21:09Z 6 OPEN
These time periods of data availability are updated once a day as we work to injest and archive data. However if you are in need of accessing (near-) realtime data, I’d highly recommend going through our SeedLink server (https://ds.iris.edu/ds/nodes/dmc/services/seedlink/) instead of using the web services.
Thanks for bringing this to our attention.
Regards,
Robert
_______________________________
Robert Weekly, PhD
Quality and Deployment System Engineer
rtweekly<at>iris.washington.edu http://iris.washington.edu/
1-206-547-0393 ext. 129
On Feb 13, 2020, at 2:15 PM, Joshua Jones <jpjones.gphys<at>gmail.com> wrote:
UW VLL * EHZ 2020-02-13T00:00:00.000000 2020-02-13T12:00:00.000000
-
Hi Robert,
Other users were reporting that data availability stopped at
2020-02-11T00:00:00 until early this afternoon. I don't want to speak for
others, but I think the concern was that data availability lagged by three
days; seems mostly resolved now, e.g., for David Love's message, a query
like
http://service.iris.edu/fdsnws/availability/1/extent?net=AU&sta=BBOO&cha=?HZ&nodata=404&starttime=2020-02-01
shows that this channel is caught up.
However, checking fdsnws-availability, I find two issues that would benefit
from official clarification:
1. extent returned by fdsnws-availability isn't the extent of available
data. It's the extent of *request start times* for which *queries
succeed*. These aren't equivalent and that isn't explained in the
fdsnws-availability documentation; hence my confusion. For example, in the
attached figure, I grab 24 hours starting at 2020-02-13T00:00:00 UTC (note,
*x*-axis is UTC -8); at most two samples fall within the stated extent.
Yet my query returns timeseries data as recent as six minutes before the
POST operation.
2. Is there an expected lag for FDSNWS timeseries extent? In other
words, should users *always* expect extent to end at 00:00:00 UTC on the
current day, or are occasional 3-day lags the expected behavior?
Thanks,
-josh
---------------------------
Mail: Dr. Joshua P. Jones, 4509 NE Sumner St, Portland, OR 97218-1547,
United States of America
OrcID: https://orcid.org/0000-0002-3518-263
ResearchGate: https://www.researchgate.net/profile/Joshua_Jones4
GitHub: https://github.com/jpjones76
Note: there are several Earth science researchers named Joshua Jones.
On Thu, Feb 13, 2020 at 2:43 PM Robert Weekly <rtweekly<at>iris.washington.edu>
wrote:
Hi Josh,
It appears that the time range for your first request (2020-02-13T01:00:00
to 2020-02-13T02:00:00) falls entirely outside the window of data
availability that we currently have archived, which would explain the No
Data errors you’re seeing. However the second time window
(2020-02-13T00:00:00 to 2020-02-13T12:00:00) overlaps with the few seconds
of data at the start of the day that we do have available. So, it makes
sense that you would receive the first seconds of data on day 2020-02-13
with a bigger time window.
If you (or anyone else) would like to know what is the most recent data
that is available through IRIS web services, please consult with the
fdsnws-availability service (
http://service.iris.edu/fdsnws/availability/1/) and use the “extent”
endpoint. In the example below, I’ve outlined the window of data
availability in RED.
http://service.iris.edu/fdsnws/availability/1/extent?net=UW&sta=TDH,VLL&cha=EHZ&nodata=404&starttime=2020-02-01
#Network Station Location Channel Quality SampleRate Earliest Latest Updated TimeSpans Restriction
UW TDH -- EHZ M 100.0 2020-02-01T00:00:00.000000Z 2020-02-13T00:00:03.381312Z 2020-02-13T10:19:16Z 5 OPEN
UW VLL -- EHZ M 100.0 2020-02-01T00:00:00.000000Z 2020-02-13T00:00:05.971341Z 2020-02-13T10:21:09Z 6 OPEN
These time periods of data availability are updated once a day as we work
to injest and archive data. However if you are in need of accessing (near-)
realtime data, I’d highly recommend going through our SeedLink server (
https://ds.iris.edu/ds/nodes/dmc/services/seedlink/) instead of using the
web services.
Thanks for bringing this to our attention.
Regards,
Robert
_______________________________
Robert Weekly, PhD
Quality and Deployment System Engineer
rtweekly<at>iris.washington.edu
1-206-547-0393 ext. 129
On Feb 13, 2020, at 2:15 PM, Joshua Jones <jpjones.gphys<at>gmail.com> wrote:
UW VLL * EHZ 2020-02-13T00:00:00.000000 2020-02-13T12:00:00.000000
Attachments
-
Hi Josh,
The word “extent” is used in this context to describe the first and last sample for which waveform data is available. However, this time period should not be assumed to always have continuous data, so even if you request data for a time window that falls within the extent window, there is still no guarantee of a successful request. If you want the full segment-by-segment detail, try the same request but use the “query?” endpoint of the availability service.
Also, the availability service does not continually update as it would be a strain on our infrastructure. I believe it is currently updated once per day. Over the course of today, it may have been the case that more data was curated and made available in our database, but not yet communicated to the service. Which could potentially explain why you received data timestamped at a later date than the availability service claimed.
And yes, there is a lag between when the various network operators send data to IRIS for ingestion and when it is made available through our web services. The times vary, depending on network, but data are typically made available within ~24 hours. This recent multi-day stretch was definitely an outlier, unfortunately, and we certainly don’t expect this to occur again. For uses where real-time data are needed, again I would highly suggest setting up a SEEDLink server instead of polling the web services for such data.
Hope this helps clear up some of the confusion. I’ll take a look at the service documentation and make some edits for clarity.
Thanks for the feedback.
Regards,
Robert
On Feb 13, 2020, at 4:08 PM, Joshua Jones <jpjones.gphys<at>gmail.com> wrote:
Hi Robert,
Other users were reporting that data availability stopped at 2020-02-11T00:00:00 until early this afternoon. I don't want to speak for others, but I think the concern was that data availability lagged by three days; seems mostly resolved now, e.g., for David Love's message, a query like
http://service.iris.edu/fdsnws/availability/1/extent?net=AU&sta=BBOO&cha=?HZ&nodata=404&starttime=2020-02-01
shows that this channel is caught up.
However, checking fdsnws-availability, I find two issues that would benefit from official clarification:
extent returned by fdsnws-availability isn't the extent of available data. It's the extent of request start times for which queries succeed. These aren't equivalent and that isn't explained in the fdsnws-availability documentation; hence my confusion. For example, in the attached figure, I grab 24 hours starting at 2020-02-13T00:00:00 UTC (note, x-axis is UTC -8); at most two samples fall within the stated extent. Yet my query returns timeseries data as recent as six minutes before the POST operation.
Is there an expected lag for FDSNWS timeseries extent? In other words, should users always expect extent to end at 00:00:00 UTC on the current day, or are occasional 3-day lags the expected behavior?
Thanks,
-josh
---------------------------
Mail: Dr. Joshua P. Jones, 4509 NE Sumner St, Portland, OR 97218-1547, United States of America
OrcID: https://orcid.org/0000-0002-3518-263
ResearchGate: https://www.researchgate.net/profile/Joshua_Jones4
GitHub: https://github.com/jpjones76
Note: there are several Earth science researchers named Joshua Jones.
On Thu, Feb 13, 2020 at 2:43 PM Robert Weekly <rtweekly<at>iris.washington.edu <rtweekly<at>iris.washington.edu>> wrote:
Hi Josh,
It appears that the time range for your first request (2020-02-13T01:00:00 to 2020-02-13T02:00:00) falls entirely outside the window of data availability that we currently have archived, which would explain the No Data errors you’re seeing. However the second time window (2020-02-13T00:00:00 to 2020-02-13T12:00:00) overlaps with the few seconds of data at the start of the day that we do have available. So, it makes sense that you would receive the first seconds of data on day 2020-02-13 with a bigger time window.
If you (or anyone else) would like to know what is the most recent data that is available through IRIS web services, please consult with the fdsnws-availability service (http://service.iris.edu/fdsnws/availability/1/) and use the “extent” endpoint. In the example below, I’ve outlined the window of data availability in RED.
http://service.iris.edu/fdsnws/availability/1/extent?net=UW&sta=TDH,VLL&cha=EHZ&nodata=404&starttime=2020-02-01 http://service.iris.edu/fdsnws/availability/1/extent?net=UW&sta=TDH,VLL&cha=EHZ&nodata=404&starttime=2020-02-01>
#Network Station Location Channel Quality SampleRate Earliest Latest Updated TimeSpans Restriction
UW TDH -- EHZ M 100.0 2020-02-01T00:00:00.000000Z 2020-02-13T00:00:03.381312Z 2020-02-13T10:19:16Z 5 OPEN
UW VLL -- EHZ M 100.0 2020-02-01T00:00:00.000000Z 2020-02-13T00:00:05.971341Z 2020-02-13T10:21:09Z 6 OPEN
These time periods of data availability are updated once a day as we work to injest and archive data. However if you are in need of accessing (near-) realtime data, I’d highly recommend going through our SeedLink server (https://ds.iris.edu/ds/nodes/dmc/services/seedlink/) instead of using the web services.
Thanks for bringing this to our attention.
Regards,
Robert
_______________________________
Robert Weekly, PhD
Quality and Deployment System Engineer
rtweekly<at>iris.washington.edu http://iris.washington.edu/
1-206-547-0393 ext. 129
On Feb 13, 2020, at 2:15 PM, Joshua Jones <jpjones.gphys<at>gmail.com <jpjones.gphys<at>gmail.com>> wrote:
<query_plot.png>
UW VLL * EHZ 2020-02-13T00:00:00.000000 2020-02-13T12:00:00.000000
-
-
-
-
Hi all,
I tried some more tests this morning and got varied results so it does not appear to be working yet. For example, I can obtain data for the PB network. Interestingly, obtaining data for the TX network appears to depend on the time window (see below).
PB test (WORKING) -> http://service.iris.edu/irisws/timeseries/1/query?net=PB&sta=B082&cha=HHZ&start=2020-02-09T00:00:00&end=2020-02-10T00:00:00&format=plot&loc=--
PB test (WORKING) -> http://service.iris.edu/irisws/timeseries/1/query?net=PB&sta=B082&cha=HHZ&start=2020-02-10T00:00:00&end=2020-02-12T00:00:00&format=plot&loc=--
Also the time window seems to affect whether it works:
TX test (NOT WORKING) for a short time window -> http://service.iris.edu/irisws/timeseries/1/query?net=TX&sta=PB01&cha=HHZ&start=2020-02-13T13:39:00&end=2020-02-13T13:45:00&format=plot&loc=00
TX test (NOT WORKING) for slightly longer window -> http://service.iris.edu/irisws/timeseries/1/query?net=TX&sta=PB01&cha=HHZ&start=2020-02-13T23:00:00&end=2020-02-13T23:49:00&format=plot&loc=00
TX test (NOT WORKING) even longer -> http://service.iris.edu/irisws/timeseries/1/query?net=TX&sta=PB01&cha=HHZ&start=2020-02-13T20:00:00&end=2020-02-13T23:49:00&format=plot&loc=00
TX test (WORKING) for same day but longer time window -> http://service.iris.edu/irisws/timeseries/1/query?net=TX&sta=PB01&cha=HHZ&start=2020-02-13T00:00:00&end=2020-02-13T23:49:00&format=plot&loc=00
If I push that last window start time 3 seconds past midnight, then it no longer works:
http://service.iris.edu/irisws/timeseries/1/query?net=TX&sta=PB01&cha=HHZ&start=2020-02-13T00:01:00&end=2020-02-13T23:49:00&format=plot&loc=00
But still works for 2 seconds past midnight:
http://service.iris.edu/irisws/timeseries/1/query?net=TX&sta=PB01&cha=HHZ&start=2020-02-13T00:00:02&end=2020-02-13T23:49:00&format=plot&loc=00
Sorry for the wall of links, I wanted to provide some testable examples. The fact that the data is clearly available e.g. for the final TX tests on PB01.00.HHZ, but breaks depending on the start time is particularly odd.
Thanks,
Joshua Williams, MGeophys, Ph.D.
ESG Solutions
20 Hyperion Ct., Kingston, ON, Canada, K7K 7K2
Office: 613.548.8287 ext: 307, fax: 613.548.8917
joshua.williams<at>esgsolutions.com, www.esgsolutions.com