International Federation of Digital Seismograph Networks

Thread: miniSEED: progress on a new FDSN standard

None
Started: July 26, 2017, 9:22 p.m.
Last activity: Aug. 1, 2017, 3:12 p.m.
Reinoud Sleeman
July 26, 2017, 9:22 p.m.
Dear Goran and members of WG2,

the outcome of the successful FDSN WG2 meeting in the Netherlands, earlier this year, on the future of mini-SEED is
a White Paper which describes the ongoing discussion within WG2 on the motivation and vision for a new format, the
identification of current needs for a new format, a description of several proposals and a proposal for the way forward.

This White Paper, which is attached to this e-mail, will be the basis for further discussion during the FDSN WG2 meeting
in Kobe. At this moment, however, there is no consensus yet about the adoption and implementation of a new FDSN
standard. Feedback from several stakeholders show very strong, different opinions which reflect that the FDSN community
is not ready for a new format at this stage.

In order to get broader support to work towards a new format, the FDSN should first reflect on and take decisions on the
following question:

- What is the scope of FDSN? Does the FDSN envision/support the need to prepare our infrastructure for large N experiments,
and other more complex experiments, often with lower quality sensors, that may go beyond seismic data ?

- Can we solve the identification limitation without major overhaul ?
Introducing an disruptive change of the format has an impact on all stakeholders. Do we accept version incompatibility ?

Because of the different opinions, that we need to respect, and the potential big impact on the current seismological infrastructure
a careful process must be followed. We only can move forward when the motivation for a new format is well understood, accepted and
supported by the FDSN. The white paper proposes a path forward, starting with a requirements list (to be reviewed and approved by WG2)
and the setup of a Technical Working Group to implement draft realisations of the different proposals with the aim to identify
problems in an early stage and evaluate performance. If the FDSN supports the design of a new format it will have to be a long
term sustainable format and this takes time to design it carefully, and even can be more ambitious than the current proposals.
Most likely, SEED was not build in a short time either.

Finally, as soon as the FDSN supports the motivation for a new format it is required to define a WG2 agreed requirements
list for the new format. This must be the starting point for the evaluation of the different proposals. That requirements list
will take away current differences in views and opinions of what the new format should be designed for (e.g. supporting
real-time or not).

Please find attached the White Paper: The Future of mini-SEED.

Best regards,
Reinoud



_______________________________________
From: fdsn-wg2-data-bounce<at>lists.fdsn.org<fdsn-wg2-data-bounce<at>lists.fdsn.org> [fdsn-wg2-data-bounce<at>lists.fdsn.org] on behalf of Goran Ekstrom [ekstrom<at>ldeo.columbia.edu]
Sent: 21 July 2017 19:54
To: FDSN Working Group II
Subject: [fdsn-wg2-data] miniSEED: progress on a new FDSN standard

Dear Colleagues and Members of Working Group II:

About a year ago (August 6, 2016) I wrote to you concerning the recognized current need for an updated miniSEED format, and in support of the ongoing efforts in WGII to develop a new FDSN standard. While I did not attend the Utrecht workshop on this topic earlier this year, I understand that it was productive and that much progress has been made on the detailed definition of a new standard.

I now write again to request that you use the Kobe meeting to take the next steps toward adoption and implementation of a new FDSN standard.

My understanding is that a nearly complete, though not final, specification for a new format - miniSEED3 - exists and has been discussed by the team of technical developers identified at the meeting in Utrecht. Input came from a number of groups that commented on the initial strawman proposed by IRIS, and modifications were made based on input from the Utrecht meeting and through further discussions in the technical evaluation team.

I will not delve into, or argue for, specific features of the proposed format. Most of you are more familiar with its details than I. I will, however, highlight some important aspects of the new format, as presented by its developers and proponents:

1. It solves the imminent data-center need to identify and accommodate larger numbers of networks, stations, and sensors.
[Some modern data sets relevant for the FDSN community cannot be accommodated with the current miniSEED format. ]

2. Conversion of miniSEED2 to miniSEED3, for archiving or distribution, is simple and lossless.

3. Networks and data centers that do not need the new features offered by miniSEED3 may not see any benefit in adopting miniSEED3 and can continue relying on miniSEED2 and their current infrastructure in their operation.

4. By design, miniSEED3 is not intended to be a low-latency real-time data format. It primarily addresses issues in data archiving, data management and data exchange.

Point (1) addresses the critical issue that some data centers have to deal with now. Having an FDSN standard will strongly encourage data centers to implement the same standard, facilitating continued sharing of software and development.

Points (2-3) make it clear that the new format does not impose a burden on those data centers and networks that do not need miniSEED3.

Point (4) clarifies the limitations of the scope of miniSEED3 and identifies the separate need for an effort to clarify/develop standards in the field recording and transmission of data and to address related issues, such as data latency.

I urge you to use the WGII meeting in Kobe to review and consider the
miniSEED3 format proposal. While doing so, I hope you will not forget the strategic benefits of having an FDSN standard and also to consider the risks associated with not adopting a standard. I also hope that your deliberations will lead to a recommendation that the FDSN Steering Committee can consider during the second plenary meeting.

My assessment is that the FDSN is ready to, and needs to, move forward with a solution.

Best regards,
Göran


----------------------
FDSN Working Group II (http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)

Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
Update subscription preferences at http://www.fdsn.org/account/profile/


  • Philipp Kästli
    July 31, 2017, 9:22 a.m.
    Dear colleagues,

    i'd like to add a few comments to the discussion of the next generation waveform format:


    a) Much of the variability of the 5 proposals are, imho, the result of different assumptions on the scope of the development: Should we develop for

    - FDSN only (mostly measured data from +/- long living stations, organized in defined networks, with a community organized well enough to be able to rely, to some extent, also on conventions)

    - Seismology in general (including any type of waveform modelling, and any type of motion sensors, also poorly identified or very mobile ones) This requires additional flexibility for identifying streams from instruments not organized in networks or

    - Regular environmental time series in general (leveraging the possibility of software and data exchange with other domains). This requires a better separation of time series and domain metadata and probably an extension of temporal scopes)

    ð In order to focus further discussion, I'd like to invite WG2 to take a decision on this.


    b) The question of (backward) compatibility
    Miniseed2 started from a subset of seed, with seed covering all waveforms, data quality metadata, instrumentation metadata, and event information. In the recent years, separate formats and services for these topics have developed and were adopted as standards, as it proved to be more lightweight and flexible to handle different types of information separately: mseed for waveforms, QuakeML BED for event information, StationXML/InventoryXML for instrumentation metadata, and Mustang/WFcatalog with their respective delivery formats for data quality metadata. So when calling for compatibility of mseed3, we should more aim at the semantically correct distribution of information into these four domains than to maintain decisions inherited from full seed. Otherwise we risk overlaps/collision or gaps with regards to the other domain formats and respective services

    ð I'd like to invite WG2 to aim for a future proof affiliation of information to these different domains and the respective formats & services, even at the price of breaking formal compatibility with seed/mseed2.



    c) Application scope and requirements list
    There seems to be an implicit agreement that the new format should cover different application cases (near-real time transfer, economic storage, efficient seek and retrieval), and that the format should accommodate for this with different representation variations (including variable block size, compression, and variable alignment [storage or time]). However, we define only a file format but none of I transfer protocol, II subformat conversion rules, or III data access strategy.

    ð In order to avoid ambiguities or shortcoming in these different applications, I'd like to propose 3 add-ons to the requirements list:
    -- forward writeability: You should be able to start writing or transferring a record before having acquired or read all data intended to be added to it. This detaches the transfer protocol from the block structure (I) and reduces the needs for subformat conversion. It disables for data integrity (e.g. crc) in block headers, as those are available only with all data available.
    -- content invariance with subformat conversion. This allows to losslessly recode the same information in different subformats. It disallows for features IMplicitly referring to the edges, or extent, of a data block (as this would change such implicit information)
    -- explicit alignment strategy. This allows the user to see whether block size is aligned to storage units or time intervals, and whether or not it is subject of change between blocks of the same data source & stream. It requires subformat indicators, and potentially stream ID aliases

    Kind regards,

    Philipp




    From: fdsn-wg2-data-bounce<at>lists.fdsn.org [fdsn-wg2-data-bounce<at>lists.fdsn.org] On Behalf Of Reinoud Sleeman
    Sent: Mittwoch, 26. Juli 2017 23:23
    To: FDSN Working Group II
    Subject: [fdsn-wg2-data] miniSEED: progress on a new FDSN standard

    Dear Goran and members of WG2,

    the outcome of the successful FDSN WG2 meeting in the Netherlands, earlier this year, on the future of mini-SEED is
    a White Paper which describes the ongoing discussion within WG2 on the motivation and vision for a new format, the
    identification of current needs for a new format, a description of several proposals and a proposal for the way forward.

    This White Paper, which is attached to this e-mail, will be the basis for further discussion during the FDSN WG2 meeting
    in Kobe. At this moment, however, there is no consensus yet about the adoption and implementation of a new FDSN
    standard. Feedback from several stakeholders show very strong, different opinions which reflect that the FDSN community
    is not ready for a new format at this stage.

    In order to get broader support to work towards a new format, the FDSN should first reflect on and take decisions on the
    following question:

    - What is the scope of FDSN? Does the FDSN envision/support the need to prepare our infrastructure for large N experiments,
    and other more complex experiments, often with lower quality sensors, that may go beyond seismic data ?

    - Can we solve the identification limitation without major overhaul ?
    Introducing an disruptive change of the format has an impact on all stakeholders. Do we accept version incompatibility ?

    Because of the different opinions, that we need to respect, and the potential big impact on the current seismological infrastructure
    a careful process must be followed. We only can move forward when the motivation for a new format is well understood, accepted and
    supported by the FDSN. The white paper proposes a path forward, starting with a requirements list (to be reviewed and approved by WG2)
    and the setup of a Technical Working Group to implement draft realisations of the different proposals with the aim to identify
    problems in an early stage and evaluate performance. If the FDSN supports the design of a new format it will have to be a long
    term sustainable format and this takes time to design it carefully, and even can be more ambitious than the current proposals.
    Most likely, SEED was not build in a short time either.

    Finally, as soon as the FDSN supports the motivation for a new format it is required to define a WG2 agreed requirements
    list for the new format. This must be the starting point for the evaluation of the different proposals. That requirements list
    will take away current differences in views and opinions of what the new format should be designed for (e.g. supporting
    real-time or not).

    Please find attached the White Paper: The Future of mini-SEED.

    Best regards,
    Reinoud



    _______________________________________
    From: fdsn-wg2-data-bounce<at>lists.fdsn.org<fdsn-wg2-data-bounce<at>lists.fdsn.org> [fdsn-wg2-data-bounce<at>lists.fdsn.org] on behalf of Goran Ekstrom [ekstrom<at>ldeo.columbia.edu]
    Sent: 21 July 2017 19:54
    To: FDSN Working Group II
    Subject: [fdsn-wg2-data] miniSEED: progress on a new FDSN standard

    Dear Colleagues and Members of Working Group II:

    About a year ago (August 6, 2016) I wrote to you concerning the recognized current need for an updated miniSEED format, and in support of the ongoing efforts in WGII to develop a new FDSN standard. While I did not attend the Utrecht workshop on this topic earlier this year, I understand that it was productive and that much progress has been made on the detailed definition of a new standard.

    I now write again to request that you use the Kobe meeting to take the next steps toward adoption and implementation of a new FDSN standard.

    My understanding is that a nearly complete, though not final, specification for a new format - miniSEED3 - exists and has been discussed by the team of technical developers identified at the meeting in Utrecht. Input came from a number of groups that commented on the initial strawman proposed by IRIS, and modifications were made based on input from the Utrecht meeting and through further discussions in the technical evaluation team.

    I will not delve into, or argue for, specific features of the proposed format. Most of you are more familiar with its details than I. I will, however, highlight some important aspects of the new format, as presented by its developers and proponents:

    1. It solves the imminent data-center need to identify and accommodate larger numbers of networks, stations, and sensors.
    [Some modern data sets relevant for the FDSN community cannot be accommodated with the current miniSEED format. ]

    2. Conversion of miniSEED2 to miniSEED3, for archiving or distribution, is simple and lossless.

    3. Networks and data centers that do not need the new features offered by miniSEED3 may not see any benefit in adopting miniSEED3 and can continue relying on miniSEED2 and their current infrastructure in their operation.

    4. By design, miniSEED3 is not intended to be a low-latency real-time data format. It primarily addresses issues in data archiving, data management and data exchange.

    Point (1) addresses the critical issue that some data centers have to deal with now. Having an FDSN standard will strongly encourage data centers to implement the same standard, facilitating continued sharing of software and development.

    Points (2-3) make it clear that the new format does not impose a burden on those data centers and networks that do not need miniSEED3.

    Point (4) clarifies the limitations of the scope of miniSEED3 and identifies the separate need for an effort to clarify/develop standards in the field recording and transmission of data and to address related issues, such as data latency.

    I urge you to use the WGII meeting in Kobe to review and consider the
    miniSEED3 format proposal. While doing so, I hope you will not forget the strategic benefits of having an FDSN standard and also to consider the risks associated with not adopting a standard. I also hope that your deliberations will lead to a recommendation that the FDSN Steering Committee can consider during the second plenary meeting.

    My assessment is that the FDSN is ready to, and needs to, move forward with a solution.

    Best regards,
    Göran


    ----------------------
    FDSN Working Group II (http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)

    Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
    Update subscription preferences at http://www.fdsn.org/account/profile/


    • Jean-Marie Saurel
      Aug. 1, 2017, 3:12 p.m.
      Hello Philipp, hello everyone,

      I just wanted to add a few comments to some of your points, mainly the
      first one.

      a) Much of the variability of the 5 proposals are, imho, the result
      of different assumptions on the scope of the development: Should we
      develop for

      - FDSN only (mostly measured data from +/- long living
      stations, organized in defined networks, with a community organized well
      enough to be able to rely, to some extent, also on conventions)

      - Seismology in general (including any type of waveform
      modelling, and any type of motion sensors, also poorly identified or
      very mobile ones) This requires additional flexibility for identifying
      streams from instruments not organized in networks or

      - Regular environmental time series in general (leveraging the
      possibility of software and data exchange with other domains). This
      requires a better separation of time series and domain metadata and
      probably an extension of temporal scopes)
      SEED standard (and therefore miniseed 2.4) already applies to such
      general use, by defining channel codes for almost every geophysical or
      environmental sensor, which are often in the vicinity of seismic
      stations and may be useful to the scientist to interpret the seismic data.

      The appendix A at least already list :
      - high quality and low quality seismometers (high and low gain, short
      period and broadband, geophones);
      - accelerometer and gravimeter;
      - tiltmeter, creepmeter and strainmeter;
      - pressure, humidity, temperature, rainfall, cloud and wind sensors;
      - tide and water current sensors;
      - electronic test points and magnetometer.

      Some of the miniseed 2.4 users might already use those possibilities (we
      do at IPGP) and, as you mention, the possibility to use the same data
      exchange softwares.

      The new standard, if there is one, should at least cover the same range
      of instruments and use cases as the actual miniseed 2.4.

      ð In order to focus further discussion, I’d like to invite WG2 to take
      a decision on this.



      b) The question of (backward) compatibility
      Miniseed2 started from a subset of seed, with seed covering all
      waveforms, data quality metadata, instrumentation metadata, and event
      information. In the recent years, separate formats and services for
      these topics have developed and were adopted as standards, as it proved
      to be more lightweight and flexible to handle different types of
      information separately: mseed for waveforms, QuakeML BED for event
      information, StationXML/InventoryXML for instrumentation metadata, and
      Mustang/WFcatalog with their respective delivery formats for data
      quality metadata. So when calling for compatibility of mseed3, we should
      more aim at the semantically correct distribution of information into
      these four domains than to maintain decisions inherited from full seed.
      Otherwise we risk overlaps/collision or gaps with regards to the other
      domain formats and respective services

      ð I’d like to invite WG2 to aim for a future proof affiliation of
      information to these different domains and the respective formats &
      services, even at the price of breaking formal compatibility with
      seed/mseed2.

      Could this affiliation of information could serve as a replacement for
      the full SEED volumes ?

      Regards.

      Jean-Marie SAUREL.



      c) Application scope and requirements list
      There seems to be an implicit agreement that the new format should cover
      different application cases (near-real time transfer, economic storage,
      efficient seek and retrieval), and that the format should accommodate
      for this with different representation variations (including variable
      block size, compression, and variable alignment [storage or time]).
      However, we define only a file format but none of I transfer protocol,
      II subformat conversion rules, or III data access strategy.

      ð In order to avoid ambiguities or shortcoming in these different
      applications, I’d like to propose 3 add-ons to the requirements list:
      -- forward writeability: You should be able to start writing or
      transferring a record before having acquired or read all data intended
      to be added to it. This detaches the transfer protocol from the block
      structure (I) and reduces the needs for subformat conversion. It
      disables for data integrity (e.g. crc) in block headers, as those are
      available only with all data available.
      -- content invariance with subformat conversion. This allows to
      losslessly recode the same information in different subformats. It
      disallows for features IMplicitly referring to the edges, or extent, of
      a data block (as this would change such implicit information)
      -- explicit alignment strategy. This allows the user to see whether
      block size is aligned to storage units or time intervals, and whether or
      not it is subject of change between blocks of the same data source &
      stream. It requires subformat indicators, and potentially stream ID aliases



      Kind regards,



      Philipp









      *From:*fdsn-wg2-data-bounce<at>lists.fdsn.org
      [fdsn-wg2-data-bounce<at>lists.fdsn.org] *On Behalf Of *Reinoud Sleeman
      *Sent:* Mittwoch, 26. Juli 2017 23:23
      *To:* FDSN Working Group II
      *Subject:* [fdsn-wg2-data] miniSEED: progress on a new FDSN standard



      Dear Goran and members of WG2,



      the outcome of the successful FDSN WG2 meeting in the Netherlands,
      earlier this year, on the future of mini-SEED is

      a White Paper which describes the ongoing discussion within WG2 on the
      motivation and vision for a new format, the

      identification of current needs for a new format, a description of
      several proposals and a proposal for the way forward.



      This White Paper, which is attached to this e-mail, will be the basis
      for further discussion during the FDSN WG2 meeting

      in Kobe. At this moment, however, there is no consensus yet about the
      adoption and implementation of a new FDSN

      standard. Feedback from several stakeholders show very strong, different
      opinions which reflect that the FDSN community

      is not ready for a new format at this stage.



      In order to get broader support to work towards a new format, the FDSN
      should first reflect on and take decisions on the

      following question:



      - What is the scope of FDSN? Does the FDSN envision/support the need to
      prepare our infrastructure for large N experiments,

      and other more complex experiments, often with lower quality sensors,
      that may go beyond seismic data ?



      - Can we solve the identification limitation without major overhaul ?

      Introducing an disruptive change of the format has an impact on all
      stakeholders. Do we accept version incompatibility ?



      Because of the different opinions, that we need to respect, and the
      potential big impact on the current seismological infrastructure

      a careful process must be followed. We only can move forward when the
      motivation for a new format is well understood, accepted and

      supported by the FDSN. The white paper proposes a path forward, starting
      with a requirements list (to be reviewed and approved by WG2)

      and the setup of a Technical Working Group to implement draft
      realisations of the different proposals with the aim to identify

      problems in an early stage and evaluate performance. If the FDSN
      supports the design of a new format it will have to be a long

      term sustainable format and this takes time to design it carefully, and
      even can be more ambitious than the current proposals.

      Most likely, SEED was not build in a short time either.



      Finally, as soon as the FDSN supports the motivation for a new format it
      is required to define a WG2 agreed requirements

      list for the new format. This must be the starting point for the
      evaluation of the different proposals. That requirements list

      will take away current differences in views and opinions of what the new
      format should be designed for (e.g. supporting

      real-timeor not).



      Please find attached the White Paper: The Future of mini-SEED.



      Best regards,

      Reinoud







      _______________________________________

      From: fdsn-wg2-data-bounce<at>lists.fdsn.org
      <fdsn-wg2-data-bounce<at>lists.fdsn.org>
      [fdsn-wg2-data-bounce<at>lists.fdsn.org] on behalf of Goran Ekstrom
      [ekstrom<at>ldeo.columbia.edu]

      Sent: 21 July 2017 19:54

      To: FDSN Working Group II

      Subject: [fdsn-wg2-data] miniSEED: progress on a new FDSN standard



      Dear Colleagues and Members of Working Group II:



      About a year ago (August 6, 2016) I wrote to you concerning the
      recognized current need for an updated miniSEED format, and in support
      of the ongoing efforts in WGII to develop a new FDSN standard. While I
      did not attend the Utrecht workshop on this topic earlier this year, I
      understand that it was productive and that much progress has been made
      on the detailed definition of a new standard.



      I now write again to request that you use the Kobe meeting to take the
      next steps toward adoption and implementation of a new FDSN standard.



      My understanding is that a nearly complete, though not final,
      specification for a new format - miniSEED3 - exists and has been
      discussed by the team of technical developers identified at the meeting
      in Utrecht. Input came from a number of groups that commented on the
      initial strawman proposed by IRIS, and modifications were made based on
      input from the Utrecht meeting and through further discussions in the
      technical evaluation team.



      I will not delve into, or argue for, specific features of the proposed
      format. Most of you are more familiar with its details than I. I will,
      however, highlight some important aspects of the new format, as
      presented by its developers and proponents:



      1. It solves the imminent data-center need to identify and accommodate
      larger numbers of networks, stations, and sensors.

      [Some modern data sets relevant for the FDSN community cannot be
      accommodated with the current miniSEED format. ]



      2. Conversion of miniSEED2 to miniSEED3, for archiving or distribution,
      is simple and lossless.



      3. Networks and data centers that do not need the new features offered
      by miniSEED3 may not see any benefit in adopting miniSEED3 and can
      continue relying on miniSEED2 and their current infrastructure in their
      operation.



      4. By design, miniSEED3 is not intended to be a low-latency real-time
      data format. It primarily addresses issues in data archiving, data
      management and data exchange.



      Point (1) addresses the critical issue that some data centers have to
      deal with now. Having an FDSN standard will strongly encourage data
      centers to implement the same standard, facilitating continued sharing
      of software and development.



      Points (2-3) make it clear that the new format does not impose a burden
      on those data centers and networks that do not need miniSEED3.



      Point (4) clarifies the limitations of the scope of miniSEED3 and
      identifies the separate need for an effort to clarify/develop standards
      in the field recording and transmission of data and to address related
      issues, such as data latency.



      I urge you to use the WGII meeting in Kobe to review and consider the

      miniSEED3 format proposal. While doing so, I hope you will not forget
      the strategic benefits of having an FDSN standard and also to consider
      the risks associated with not adopting a standard. I also hope that your
      deliberations will lead to a recommendation that the FDSN Steering
      Committee can consider during the second plenary meeting.



      My assessment is that the FDSN is ready to, and needs to, move forward
      with a solution.



      Best regards,

      Göran





      ----------------------

      FDSN Working Group II
      (http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)



      Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)

      Update subscription preferences at http://www.fdsn.org/account/profile/






      ----------------------
      FDSN Working Group II (http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)

      Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
      Update subscription preferences at http://www.fdsn.org/account/profile/



      --
      --------------------------------------
      Institut de Physique du Globe de Paris
      Observatoires Volcanologiques
      1 rue Jussieu
      75238 Paris Cédex 05
      +33(0)183957437
      France