Thread: continuous data

Started: 2011-12-20 19:11:53
Last activity: 2011-12-20 19:14:30
Topics: Web Services
John West
2011-12-20 19:11:53
Hi.

If I want to request continuous data via the web services, is there an
optimal block size (i.e., ~1 day) to request at a time?

I overheard a bit of conversation on this at AGU, but didn't get the
complete story.

Thanks!

-- John

  • Rich Karstens
    2011-12-20 16:41:43
    Hi John,

    The data in our archive is generally stored in files comprised of all
    the channels of a station for a specific day. Thus, there are
    advantages to requesting 'blocks' of exactly one day. Another
    advantageous technique would be to request all channels for a particular
    station in the same request. All of this pretty much comes down to
    disk I/O, file seeks and parsing.

    I'm sure we'd love to see any throughput numbers you come up with.

    Hope that helps,

    Rich


    John D. West wrote:
    Hi.

    If I want to request continuous data via the web services, is there an
    optimal block size (i.e., ~1 day) to request at a time?

    I overheard a bit of conversation on this at AGU, but didn't get the
    complete story.

    Thanks!

    -- John
    _______________________________________________
    webservices mailing list
    webservices<at>iris.washington.edu
    http://www.iris.washington.edu/mailman/listinfo/webservices

    • Bruce Weertman
      2011-12-20 19:14:30
      John:

      A day is probably a good choice for a "block" size.

      Also, take a look at the bottom of this web page:

      http://www.iris.edu/ws/doc/bulkdataselect_help.htm

      As Rich pointed out, our file organization is all data from one station for one day into one file.
      The code that pulls the data out of one of the files works more efficiently if you ask for all of the information
      that you want for that one station-day at once, rather than in a bunch of requests.

      So if you were say getting all BHZ, BH1, BH2 for a network, you would want to
      request station by station rather than channel by channel

      ie
      one request
      IU ANMO 00 BHZ time-range
      IU ANMO 00 BH1 time-range
      IU ANMO 00 BH2 time-range

      then next station request
      etc
      ...

      but not
      IU ANMO 00 BHZ time-range
      IU QSPA 00 BHZ time-range
      IU ADK 00 BHZ time-range
      etc
      ...
      then the next channel request

      -Bruce

      On Dec 20, 2011, at 10:41 AM, Rich Karstens wrote:

      Hi John,

      The data in our archive is generally stored in files comprised of all the channels of a station for a specific day. Thus, there are advantages to requesting 'blocks' of exactly one day. Another advantageous technique would be to request all channels for a particular station in the same request. All of this pretty much comes down to disk I/O, file seeks and parsing.

      I'm sure we'd love to see any throughput numbers you come up with.

      Hope that helps,

      Rich


      John D. West wrote:
      Hi.

      If I want to request continuous data via the web services, is there an optimal block size (i.e., ~1 day) to request at a time?

      I overheard a bit of conversation on this at AGU, but didn't get the complete story.

      Thanks!

      -- John
      _______________________________________________
      webservices mailing list
      webservices<at>iris.washington.edu
      http://www.iris.washington.edu/mailman/listinfo/webservices
      _______________________________________________
      webservices mailing list
      webservices<at>iris.washington.edu
      http://www.iris.washington.edu/mailman/listinfo/webservices



22:30:41 v.01697673