International Federation of Digital Seismograph Networks

Thread: Next generation miniSEED - 2016-3-30 straw man change proposal 7 - Change start time to custom structure

None
Started: Aug. 11, 2016, 12:47 p.m.
Last activity: Aug. 23, 2016, 1:42 p.m.

Hi all,

Change proposal #7 to the 2016-3-30 straw man (iteration 1) is attached: Change start time to custom structure.

Please use this thread to provide your feedback on this proposal by Wednesday August 24th.

thanks,
Chad




  • Two issues that should be pointed out this this change. The first is
    that a "leap second occurs IN this record" is required in either case,
    and positive/negative direction is needed. A record that overlaps a
    leap second is different from a record where the start time is IN a
    leap second. Consider a 1sps record containing 11 samples with a start
    time of 23:59:50 on 31 Dec. The starttime is not a leap second in
    either case, but the last sample could occur either at 00:00:00 on 1
    Jan or at 23:59:60 on 31 Dec. Allowing the seconds field to be 60 is
    insufficient in this case and so some extra information beyond the
    start time is needed even in the case of the BTime structure.

    The second comment is that users do not interact directly with
    miniseed currently, and will not do so in the future. There will be
    software libraries in each of the popular languages that do this for
    them, and those libraries will convert times into the languages
    default representation. A BTime might be a little less confusing for
    the developer, but the user is going to use a value in software that
    looks a lot like the IEEE format when they initiate any processing.
    Having the file format doesn't make this problem go away or really
    even make it less likely. What would help is if libraries were created
    with functions such as getTimeOfSample(27) or
    timeDifferenceBetweenSamples(sample1, sample2) so that the library
    does the math correctly taking into account the leap seconds.

    I was originally somewhat against the IEEE format, but the more I
    think about it, the less I think it matters. Math with time is hard
    and the file format is not the right place to protect users against
    doing wrong subtraction across leap seconds. The software that uses
    the file format is the right place for that.

    thanks
    Philip

    On Thu, Aug 11, 2016 at 3:49 PM, Chad Trabant <chad<at>iris.washington.edu> wrote:

    Hi all,

    Change proposal #7 to the 2016-3-30 straw man (iteration 1) is attached:
    Change start time to custom structure.

    Please use this thread to provide your feedback on this proposal by
    Wednesday August 24th.

    thanks,
    Chad





    ----------------------
    Posted to multiple topics:
    FDSN Working Group II
    (http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)
    FDSN Working Group III
    (http://www.fdsn.org/message-center/topic/fdsn-wg3-products/)

    Sent via IRIS Message Center (http://www.fdsn.org/message-center/)
    Update subscription preferences at http://www.fdsn.org/account/profile/



    • Hi,

      I strongly agree with Philip's two points. In my understanding the two existing leap second flags are no more or less optional than they were before as stated in the Rationale. There is no way to leave these flags out, aka required. Granted they are often not set when they should be, but that is not relevant to start time representation.

      As to the argument that making the time representation appear to be continuous would encourage improper usage, bear in mind that checking the 3 leap second bits in the straw man is a much lower burden than the current SEED 2.x time interpretation, which is more likely to go wrong. Currently, to understand time in a record, the developer must check the 2 existing leap second flags (or check with an external reference), check for optional microseconds, check for non-zero time correction and check the flag to see if the time correction has been applied. Yet seismology is not riddled with improper time interpretations, mostly, I suspect, because users use libraries that take care of these details as Philip writes.

      Also, the epoch time scale between any two consecutive records in a stream is continuous in the vast, vast, vast majority of existing cases. Unless we start inserting leap seconds every few minutes this will continue to be true for future records. Designing the time stamp for ease of use in what are some of the most common operations with miniSEED, e.g. reconstructing a time series from multiple records, is worth strong consideration. In the case of an epoch time scale the "cost" is the addition of a flag that must be checked to know if the time stamp is a leap second; which is added to the 2 existing leap second flags that should be consulted anyway. With that one additional flag we have a complete time representation that is easy to use for time calculations. On the contrary, the proposed MSEED3 Time Structure requires the developer to either build operators to perform time calculations or convert the representation to something more amenable to such calculations.

      Of minor importance is that the microsecond epoch time representation is smaller (8 bytes) than the proposed MSEED3 Time Structure (12 bytes). The later continues to waste a byte for the necessary alignment as the SEED 2.4 BTime structure does.

      The assertion that the proposed time representation (I assume this is in reference to the epoch time representation) has no way to represent 2 consecutive leap seconds is true. But I'm confused as to why that is relevant, the time stamp represents an instant, not a range. Also, the proposed MSEED3 Time Structure does not appear to be able to represent consecutive leap seconds. Maybe I've misunderstood the point. I predict there would be larger fish to fry upstream of miniSEED should consecutive leap seconds occur!

      Chad

      On Aug 11, 2016, at 2:36 PM, Philip Crotwell <crotwell<at>seis.sc.edu> wrote:

      Two issues that should be pointed out this this change. The first is
      that a "leap second occurs IN this record" is required in either case,
      and positive/negative direction is needed. A record that overlaps a
      leap second is different from a record where the start time is IN a
      leap second. Consider a 1sps record containing 11 samples with a start
      time of 23:59:50 on 31 Dec. The starttime is not a leap second in
      either case, but the last sample could occur either at 00:00:00 on 1
      Jan or at 23:59:60 on 31 Dec. Allowing the seconds field to be 60 is
      insufficient in this case and so some extra information beyond the
      start time is needed even in the case of the BTime structure.

      The second comment is that users do not interact directly with
      miniseed currently, and will not do so in the future. There will be
      software libraries in each of the popular languages that do this for
      them, and those libraries will convert times into the languages
      default representation. A BTime might be a little less confusing for
      the developer, but the user is going to use a value in software that
      looks a lot like the IEEE format when they initiate any processing.
      Having the file format doesn't make this problem go away or really
      even make it less likely. What would help is if libraries were created
      with functions such as getTimeOfSample(27) or
      timeDifferenceBetweenSamples(sample1, sample2) so that the library
      does the math correctly taking into account the leap seconds.

      I was originally somewhat against the IEEE format, but the more I
      think about it, the less I think it matters. Math with time is hard
      and the file format is not the right place to protect users against
      doing wrong subtraction across leap seconds. The software that uses
      the file format is the right place for that.

      thanks
      Philip

      On Thu, Aug 11, 2016 at 3:49 PM, Chad Trabant <chad<at>iris.washington.edu> wrote:

      Hi all,

      Change proposal #7 to the 2016-3-30 straw man (iteration 1) is attached:
      Change start time to custom structure.

      Please use this thread to provide your feedback on this proposal by
      Wednesday August 24th.

      thanks,
      Chad





      ----------------------
      Posted to multiple topics:
      FDSN Working Group II
      (http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)
      FDSN Working Group III
      (http://www.fdsn.org/message-center/topic/fdsn-wg3-products/)

      Sent via IRIS Message Center (http://www.fdsn.org/message-center/)
      Update subscription preferences at http://www.fdsn.org/account/profile/



  • I agree very strongly with Doug Neuhauser’s comments with regard to use of
    64-bit “unix” time.



    In addition to the specific issues Doug has raised with regard to handling
    of leap seconds, there are general concerns.


    1. Unix time is referential; without a qualified conversion, the time value
    per se is useless. The POSIX standard is deliberately non-committal in
    defining how it should be interpreted, stating:



    “It is sufficient to require that applications be allowed to treat this time
    as if it represented the number of seconds between the referenced time and
    the Epoch. It is the responsibility of the vendor of the system, and the
    administrator of the system, to ensure that this value represents the number
    of seconds between the referenced time and the Epoch as closely as necessary
    for the application being run on that system.” (underline mine)



    The proposed straw-man approach would presumably attempt to represent UTC
    with a (modified) Unix time that, when conversion to a broken-down Gregorian
    time/date, the correct UTC date/time is returned. This approach is subject
    to errors of interpretation, requiring, in effect, that each time tag must
    be reconstructed for valid interpretation.

    The approach is convenient for data center management, but forces onto the
    user the valid development and interpretation of the time tag and leap
    second management.
    In contrast, there is no ambiguity in the archive, and the content is
    essentially self-contained and self-documenting as to the presence of a leap
    second using the broken-down time representation.



    2. A monolithic 64-bit tag is opaque to inspection. It is prone to subtle
    errors in construction and manipulation, for example, loss of 8 bits or 16
    bits, with no apparent symptom.





    The use of a 64-bit tag inappropriately confuses processing convenience with
    archival integrity.

    An archival representation should be as free as possible from ambiguous, or
    outright inaccurate (as in the case of leap-second handling using Unix
    time), representations.
    A broken-down UTC time tag, as in the present format (with additional
    microsecond resolution), serves as an identifiable, unambiguous stand-alone
    time mark in each archived data record.





    Joseph Steim






    From: Chad Trabant [chad<at>iris.washington.edu]
    Sent: Thursday, August 11, 2016 3:48 PM
    To: Tim Ahern <tim<at>iris.washington.edu>; Bob Hutt <Hutt<at>asl.cr.usgs.gov>;
    Bruce Beaudoin <bruce<at>passcal.nmt.edu>; Pete Davis
    <pdavis<at>epicenter.ucsd.edu>; Kent Anderson <kent<at>iris.edu>; Bob Woodward
    <woodward<at>iris.edu>; Katrin Hafner <hafner<at>iris.edu>; Justin Sweet
    <jrsweet<at>uw.edu>; Brent.Evers<at>iris.edu; David Wilson <dwilson<at>usgs.gov>;
    Mathias Franke <mathias.franke<at>kmi.com>; Ogie Kuraica <ogie<at>kmi.com>; Neil
    Spriggs <NeilSpriggs<at>nanometrics.ca>; Angel Rodriguez
    <angel<at>volcanbaru.com>; Leonid Zimakov <L.Zimakov<at>reftek.com>; Edelvays
    Spassov <ens<at>kmi.com>; Branden Christensen
    <branden.christensen<at>osop.com.pa>; Dieter Stoll
    <dstoll<at>lennartz-electronic.de>; Seiji Tsuboi <tsuboi<at>jamstec.go.jp>; Jara
    Salvador, Jose Antonio <JoseAntonio.Jara<at>icgc.cat>; David Easton
    <davideaston<at>nanometrics.ca>; timparker<at>nanometrics.ca; John Clinton
    <jclinton<at>sed.ethz.ch>; Vallee Martin <vallee<at>ipgp.fr>; Constantin Ionescu
    <viorel<at>info.ro>; Marmureanu.Alexandru<at>dst.units.it; Bogdan Grecu
    <bgrecu<at>infp.ro>; Cristian Neagoe <cristian.neagoe<at>infp.ro>; Helle Pedersen
    <Helle.Pedersen<at>ujf-grenoble.fr>; Catherine Pequegnat
    <catherine.pequegnat<at>obs.ujf-grenoble.fr>; Pierre Volcke
    <pierre.volcke<at>obs.ujf-grenoble.fr>; Angelo Strollo
    <strollo<at>gfz-potsdam.de>; Sébastien Judenherc
    <Sebastien.Judenherc<at>fedd-scientific.com>; Tony Russell
    <tony.russell<at>kenda.co.uk>; Robert Leugoud <rleugoud<at>eentec.com>; Jiang Li
    <lijiang01<at>geodevice.cn>; Lani Oncescu <lani<at>kmi.com>; Shawn Goessen
    <support<at>guralp.com>; Dennis Pumphrey <dp<at>kmi.com>; Suzan Kowalski
    <suzankowalski<at>nanometrics.ca>; Bruce Townsend
    <brucetownsend<at>nanometrics.ca>; Joseph Steim <steim<at>quanterra.com>; Ian
    Billings <reftek_support<at>trimble.com>; Claudio Parma <c.parma<at>solgeo.it>
    Cc: fdsn Group II <fdsn-wg2-data<at>lists.fdsn.org>; FDSN Working Group III
    <fdsn-wg3-products<at>lists.fdsn.org>
    Subject: Next generation miniSEED - 2016-3-30 straw man change proposal 7 -
    Change start time to custom structure





    Hi all,



    Change proposal #7 to the 2016-3-30 straw man (iteration 1) is attached:
    Change start time to custom structure.



    Please use this thread to provide your feedback on this proposal by
    Wednesday August 24th.



    thanks,

    Chad