International Federation of Digital Seismograph Networks

Thread: response in Sensor and DataLogger

None
Started: Aug. 3, 2015, 9:34 a.m.
Last activity: Aug. 11, 2015, 12:30 p.m.
Philip Crotwell
Aug. 3, 2015, 9:34 a.m.
One issue I have with the existing StationXML is that it can be very large
and with much repeated information, particularly for responses. For
example, it is very common to have 3 components of motion recorded at a
station on a single "sensor" for which the response of the 3 channels are
identical, but the information is repeated for each channel. Moreover, the
response is very often the nominal response of that model of sensor, and so
is repeated for all stations with that model sensor. Similar issues of
course for the DataLogger.

One thing that might help with this is to make use of externally defined
responses, such as the Nominal Response Library, and link to it from within
the Sensor and DataLogger elements, and make this an alternative way of
expressing the response. This could be done by adding a ResponseLink
element to the EquipmentType with a URL to an external response, probably
along with a starting stage and ending stage. Not only does this convey the
same information as currently but in much less space, but it also conveys
more information in that is says that the network operator is just using
the nominal response. And so if their are errors found in the nominal
response and it is updated, end users have the option of getting an update
and being confident that they new nominal response is more correct than the
previous.

Also, because it is useful to have fully self contained statilxml, it might
be useful to create a separate element parallel to Network, that could
contain these same responses as a "named response" where the name would be
the same as the URL used in the ResponseLink element. That way if I make a
request for staitonxml for 99 stations that all have STS-2 sensors, I only
need to get the STS-2 response once, and all the channels link to it.

Lastly, because a URL is very small, this information could be returned
always as part of a channel level request, and so a client that already has
the response associated with that URL does not need to ask for the response
level for that channel.

I will create a pull request for this, but thought some discussion might be
good first.

Philip

  • Jeremy Fee
    Aug. 3, 2015, 8:12 a.m.
    One option, instead of a parallel element within the same document, is to
    make that information optional in the response and make a separate request
    for only that information?


    Thanks,

    Jeremy


    On Monday, August 3, 2015, Philip Crotwell <crotwell<at>seis.sc.edu> wrote:


    One issue I have with the existing StationXML is that it can be very large
    and with much repeated information, particularly for responses. For
    example, it is very common to have 3 components of motion recorded at a
    station on a single "sensor" for which the response of the 3 channels are
    identical, but the information is repeated for each channel. Moreover, the
    response is very often the nominal response of that model of sensor, and so
    is repeated for all stations with that model sensor. Similar issues of
    course for the DataLogger.

    One thing that might help with this is to make use of externally defined
    responses, such as the Nominal Response Library, and link to it from within
    the Sensor and DataLogger elements, and make this an alternative way of
    expressing the response. This could be done by adding a ResponseLink
    element to the EquipmentType with a URL to an external response, probably
    along with a starting stage and ending stage. Not only does this convey the
    same information as currently but in much less space, but it also conveys
    more information in that is says that the network operator is just using
    the nominal response. And so if their are errors found in the nominal
    response and it is updated, end users have the option of getting an update
    and being confident that they new nominal response is more correct than the
    previous.

    Also, because it is useful to have fully self contained statilxml, it
    might be useful to create a separate element parallel to Network, that
    could contain these same responses as a "named response" where the name
    would be the same as the URL used in the ResponseLink element. That way if
    I make a request for staitonxml for 99 stations that all have STS-2
    sensors, I only need to get the STS-2 response once, and all the channels
    link to it.

    Lastly, because a URL is very small, this information could be returned
    always as part of a channel level request, and so a client that already has
    the response associated with that URL does not need to ask for the response
    level for that channel.

    I will create a pull request for this, but thought some discussion might
    be good first.

    Philip


    • Philip Crotwell
      Aug. 3, 2015, 10:20 a.m.
      Yes, my intention was that the parallel element was not required, but
      allowed to make it possible to have a single self contained XML with all
      needed information for long term storage perhaps or if it was important to
      get everything in a single http request. I am not sure that is really
      required, I would tend to use it with the separate request pattern myself.

      thanks
      PHilip


      On Mon, Aug 3, 2015 at 10:13 AM, Jeremy Fee <jmfee<at>usgs.gov> wrote:

      One option, instead of a parallel element within the same document, is to
      make that information optional in the response and make a separate request
      for only that information?


      Thanks,

      Jeremy


      On Monday, August 3, 2015, Philip Crotwell <crotwell<at>seis.sc.edu> wrote:


      One issue I have with the existing StationXML is that it can be very
      large and with much repeated information, particularly for responses. For
      example, it is very common to have 3 components of motion recorded at a
      station on a single "sensor" for which the response of the 3 channels are
      identical, but the information is repeated for each channel. Moreover, the
      response is very often the nominal response of that model of sensor, and so
      is repeated for all stations with that model sensor. Similar issues of
      course for the DataLogger.

      One thing that might help with this is to make use of externally defined
      responses, such as the Nominal Response Library, and link to it from within
      the Sensor and DataLogger elements, and make this an alternative way of
      expressing the response. This could be done by adding a ResponseLink
      element to the EquipmentType with a URL to an external response, probably
      along with a starting stage and ending stage. Not only does this convey the
      same information as currently but in much less space, but it also conveys
      more information in that is says that the network operator is just using
      the nominal response. And so if their are errors found in the nominal
      response and it is updated, end users have the option of getting an update
      and being confident that they new nominal response is more correct than the
      previous.

      Also, because it is useful to have fully self contained statilxml, it
      might be useful to create a separate element parallel to Network, that
      could contain these same responses as a "named response" where the name
      would be the same as the URL used in the ResponseLink element. That way if
      I make a request for staitonxml for 99 stations that all have STS-2
      sensors, I only need to get the STS-2 response once, and all the channels
      link to it.

      Lastly, because a URL is very small, this information could be returned
      always as part of a channel level request, and so a client that already has
      the response associated with that URL does not need to ask for the response
      level for that channel.

      I will create a pull request for this, but thought some discussion might
      be good first.

      Philip



      ----------------------
      FDSN Working Group II (
      http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)

      Sent via IRIS Message Center (http://www.fdsn.org/message-center/)
      Update subscription preferences at http://www.fdsn.org/account/profile/



  • Joachim Saul
    Aug. 4, 2015, 1:38 p.m.
    Philip Crotwell wrote on 08/03/2015 03:35 PM:
    Also, because it is useful to have fully self contained statilxml, it
    might be useful to create a separate element parallel to Network

    In other words, turn (at least) Response into a "top-level" element.
    That would allow to have fully self contained XML, while referencing
    Response from Sensor, DataLogger et al. Note that EquipmentType already
    has a resourceID attribute to facilitate this.

    It might actually make sense to turn all EquipmentType's into top-level
    elements.

    On a side note, similar considerations led to making for instance Pick a
    top-level element in QuakeML. Within an event there are often several to
    many Origins (e.g., evolving in time), each computed from strongly
    overlapping sets of picks. It would have been very inefficient to
    duplicate the very redundant sets of picks again and again as child
    elements in each new origin for a given event. Just think of the
    bandwidth required for that in a real-time messaging system! Instead, we
    decided to make Pick, Amplitude etc. top-level elements that are only
    /referenced/ by an Origin through it's publicID. This makes the Origins
    smaller, reduces redundancy and -- of course -- allows Pick elements to
    be sent independent of a Origin element, which is important, too.

    Regards,
    Joachim


    • Philipp Kästli
      Aug. 6, 2015, 4:14 p.m.

      Philip Crotwell wrote on 08/03/2015 03:35 PM:

      One thing that might help with this is to make use of externally defined responses, such as the Nominal
      Response Library, and link to it from within the Sensor and DataLogger elements, and make this an
      alternative way of expressing the response.
      [...]
      Also, because it is useful to have fully self contained statilxml, it
      might be useful to create a separate element parallel to Network

      Joachim Saul wrote:

      In other words, turn (at least) Response into a "top-level" element.
      That would allow to have fully self contained XML, while referencing
      Response from Sensor, DataLogger et al

      You would need two top level elements:
      - sensor (with model, manufacturer, vendor, serial, etc. reference to response) -> this would allow not to repeat identical information with 3 streams, but just reference it by publicid
      - response (with the response to be considered). -> if a network operator provides just standard response per sensor model, all sensors of model x may reference to a single instance of response. If individual sensors were individually measured/calibrated, then every sensor may point to its own response (however only once, without repetition for each stream and each deployment of the same sensor [with the same serial number/publicID] at a different station).

      Such a refactoring would also allow to describe a sensor if it is not actually deployed. This is exactly one of the changes I had in mind when proposing to do a larger refactoring of StationXML towards 2.0 and represent information as properties of natural entities, rather than for a single application case (in this case: "I need the response of stream MYSTATION.HHZ at date x"). Such structural changes are not backward compatible (and should not be...); thus, they should not be done incrementally, but in a (hopefully rare), large review.

      In this context, InventoryXML has a nice feature to be considered also for StationXML:
      A sensor refers to its default (either individual or type-specific response). However, it can also have multiple calibration periods as children, with individual timespan and reference to individual instances of response. These override the standard response.
      Thus, if you have a stream pointing to sensor, and the sensor has calibration periods, you do time matching in order to get the calibrated response, if it does not have any, you are happy to use the sensor's default response.

      Best regards,

      Philipp (Kästli)


      • Philip Crotwell
        Aug. 6, 2015, 5:45 p.m.
        Hi

        Here is a first cut at allowing stationXML to refer to responses instead of
        embedding them directly in a channel.
        https://github.com/FDSN/StationXML/pull/6

        thanks
        Philip

        On Thu, Aug 6, 2015 at 12:16 PM, <"Philipp Kästli
        <kaestli<at>sed.ethz.ch>"@fdsn.fdsn.org> wrote:


        Philip Crotwell wrote on 08/03/2015 03:35 PM:

        One thing that might help with this is to make use of externally
        defined responses, such as the Nominal
        Response Library, and link to it from within the Sensor and DataLogger
        elements, and make this an
        alternative way of expressing the response.
        [...]
        Also, because it is useful to have fully self contained statilxml, it
        might be useful to create a separate element parallel to Network

        Joachim Saul wrote:

        In other words, turn (at least) Response into a "top-level" element.
        That would allow to have fully self contained XML, while referencing
        Response from Sensor, DataLogger et al

        You would need two top level elements:
        - sensor (with model, manufacturer, vendor, serial, etc. reference to
        response) -> this would allow not to repeat identical information with 3
        streams, but just reference it by publicid
        - response (with the response to be considered). -> if a network operator
        provides just standard response per sensor model, all sensors of model x
        may reference to a single instance of response. If individual sensors were
        individually measured/calibrated, then every sensor may point to its own
        response (however only once, without repetition for each stream and each
        deployment of the same sensor [with the same serial number/publicID] at a
        different station).

        Such a refactoring would also allow to describe a sensor if it is not
        actually deployed. This is exactly one of the changes I had in mind when
        proposing to do a larger refactoring of StationXML towards 2.0 and
        represent information as properties of natural entities, rather than for a
        single application case (in this case: "I need the response of stream
        MYSTATION.HHZ at date x"). Such structural changes are not backward
        compatible (and should not be...); thus, they should not be done
        incrementally, but in a (hopefully rare), large review.

        In this context, InventoryXML has a nice feature to be considered also for
        StationXML:
        A sensor refers to its default (either individual or type-specific
        response). However, it can also have multiple calibration periods as
        children, with individual timespan and reference to individual instances of
        response. These override the standard response.
        Thus, if you have a stream pointing to sensor, and the sensor has
        calibration periods, you do time matching in order to get the calibrated
        response, if it does not have any, you are happy to use the sensor's
        default response.

        Best regards,

        Philipp (Kästli)


        ----------------------
        FDSN Working Group II (
        http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)

        Sent via IRIS Message Center (http://www.fdsn.org/message-center/)
        Update subscription preferences at http://www.fdsn.org/account/profile/


        • Chad Trabant
          Aug. 7, 2015, 1:57 p.m.

          Hi Philip, Joachim & Philipp,

          Rearranging the response details into re-usable entities within a document is certainly worth discussing. We should keep in mind that it would be pretty disruptive to existing reading software, more disruptive than any other changes proposed so far. I'm not saying it wouldn't be worth it.

          On the other hand, referring to and relying on external definitions for a complete document sounds like a bad idea to me. There is a significant added burden to any reader of such information to get a complete response, especially if the external information is in another format like RESP. Beyond this reader burden there are issues of longevity and versioning. If the external document goes away (there is no guarantee that NRL entries are permanent) then the StationXML now refers to a dead link and is incomplete. If the external document changes then the complete, re-constituted document changes and it may not be obvious to any given reader or user.

          Chad

          On Aug 6, 2015, at 2:47 PM, Philip Crotwell <crotwell<at>seis.sc.edu> wrote:

          Hi

          Here is a first cut at allowing stationXML to refer to responses instead of embedding them directly in a channel.
          https://github.com/FDSN/StationXML/pull/6

          thanks
          Philip

          On Thu, Aug 6, 2015 at 12:16 PM, <"Philipp Kästli <kaestli<at>sed.ethz.ch>"@fdsn.fdsn.org <mailto:"Philipp K%C3%A4stli <kaestli<at>sed.ethz.ch>"@fdsn.fdsn.org>> wrote:

          Philip Crotwell wrote on 08/03/2015 03:35 PM:

          One thing that might help with this is to make use of externally defined responses, such as the Nominal
          Response Library, and link to it from within the Sensor and DataLogger elements, and make this an
          alternative way of expressing the response.
          [...]
          Also, because it is useful to have fully self contained statilxml, it
          might be useful to create a separate element parallel to Network

          Joachim Saul wrote:

          In other words, turn (at least) Response into a "top-level" element.
          That would allow to have fully self contained XML, while referencing
          Response from Sensor, DataLogger et al

          You would need two top level elements:
          - sensor (with model, manufacturer, vendor, serial, etc. reference to response) -> this would allow not to repeat identical information with 3 streams, but just reference it by publicid
          - response (with the response to be considered). -> if a network operator provides just standard response per sensor model, all sensors of model x may reference to a single instance of response. If individual sensors were individually measured/calibrated, then every sensor may point to its own response (however only once, without repetition for each stream and each deployment of the same sensor [with the same serial number/publicID] at a different station).

          Such a refactoring would also allow to describe a sensor if it is not actually deployed. This is exactly one of the changes I had in mind when proposing to do a larger refactoring of StationXML towards 2.0 and represent information as properties of natural entities, rather than for a single application case (in this case: "I need the response of stream MYSTATION.HHZ at date x"). Such structural changes are not backward compatible (and should not be...); thus, they should not be done incrementally, but in a (hopefully rare), large review.

          In this context, InventoryXML has a nice feature to be considered also for StationXML:
          A sensor refers to its default (either individual or type-specific response). However, it can also have multiple calibration periods as children, with individual timespan and reference to individual instances of response. These override the standard response.
          Thus, if you have a stream pointing to sensor, and the sensor has calibration periods, you do time matching in order to get the calibrated response, if it does not have any, you are happy to use the sensor's default response.

          Best regards,

          Philipp (Kästli)


          ----------------------
          FDSN Working Group II (http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)

          Sent via IRIS Message Center (http://www.fdsn.org/message-center/)
          Update subscription preferences at http://www.fdsn.org/account/profile/


          ----------------------
          FDSN Working Group II (http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)

          Sent via IRIS Message Center (http://www.fdsn.org/message-center/)
          Update subscription preferences at http://www.fdsn.org/account/profile/


          • Philip Crotwell
            Aug. 7, 2015, 5:45 p.m.
            Hi

            I guess their are two ways of using StationXML, as a "message" and as a
            "document", and they have different needs and goals. In a messaging use,
            the xml exists only for a short time as it is transmitted from server to
            client and it is perfectly fine to convey some information directly and
            refer to related information via links, think web page with linked images.
            For a document use, you want long term storage and links external links are
            "bad" in the sense that you can't rely on them existing in the future.
            Think PDF with embedded images.

            It sounds like you are viewing stationxml primarily as a long lived
            document, but I question whether that is really the only, or even most
            common use. And the argument that the response must live in the same file
            as the other net/station/channel metadata is not that far from the argument
            that the metadata and the waveform data must live in the same file (a la
            full seed).

            I am not saying that a large, all encompassing document is a wrong use of
            stationxml, and my proposal does not remove the existing "stages in
            response in channel" style. But stationxml as lots of small transient
            messages between client and server is also a valid use, and in some ways
            may be more common. In my particular use case, SOD would greatly benefit
            from not getting a response for channel A when it is exactly the same as
            the response for channel B. If the ID is the same, I get it from the local
            cache.

            That said, it may be that this type of short term messaging may be
            better/easier to accommodate via something like StationJSON. In that case
            then the JSON needs to be able to carry all the same information, and of
            course actually be implemented as part of the servers.

            Maybe that is a fair way to divide the use, xml as deep, long term, self
            contained files, and JSON as small, short term, linked messages?

            Philip



            On Fri, Aug 7, 2015 at 4:57 PM, Chad Trabant <chad<at>iris.washington.edu>
            wrote:


            Hi Philip, Joachim & Philipp,

            Rearranging the response details into re-usable entities within a document
            is certainly worth discussing. We should keep in mind that it would be
            pretty disruptive to existing reading software, more disruptive than any
            other changes proposed so far. I'm not saying it wouldn't be worth it.

            On the other hand, referring to and relying on external definitions for a
            complete document sounds like a bad idea to me. There is a significant
            added burden to any reader of such information to get a complete response,
            especially if the external information is in another format like RESP.
            Beyond this reader burden there are issues of longevity and versioning. If
            the external document goes away (there is no guarantee that NRL entries are
            permanent) then the StationXML now refers to a dead link and is
            incomplete. If the external document changes then the complete,
            re-constituted document changes and it may not be obvious to any given
            reader or user.

            Chad

            On Aug 6, 2015, at 2:47 PM, Philip Crotwell <crotwell<at>seis.sc.edu> wrote:

            Hi

            Here is a first cut at allowing stationXML to refer to responses instead
            of embedding them directly in a channel.
            https://github.com/FDSN/StationXML/pull/6

            thanks
            Philip

            On Thu, Aug 6, 2015 at 12:16 PM, <"Philipp Kästli
            <kaestli<at>sed.ethz.ch>"@fdsn.fdsn.org> wrote:


            Philip Crotwell wrote on 08/03/2015 03:35 PM:

            One thing that might help with this is to make use of externally
            defined responses, such as the Nominal
            Response Library, and link to it from within the Sensor and
            DataLogger elements, and make this an
            alternative way of expressing the response.
            [...]
            Also, because it is useful to have fully self contained statilxml, it
            might be useful to create a separate element parallel to Network

            Joachim Saul wrote:

            In other words, turn (at least) Response into a "top-level" element.
            That would allow to have fully self contained XML, while referencing
            Response from Sensor, DataLogger et al

            You would need two top level elements:
            - sensor (with model, manufacturer, vendor, serial, etc. reference to
            response) -> this would allow not to repeat identical information with 3
            streams, but just reference it by publicid
            - response (with the response to be considered). -> if a network operator
            provides just standard response per sensor model, all sensors of model x
            may reference to a single instance of response. If individual sensors were
            individually measured/calibrated, then every sensor may point to its own
            response (however only once, without repetition for each stream and each
            deployment of the same sensor [with the same serial number/publicID] at a
            different station).

            Such a refactoring would also allow to describe a sensor if it is not
            actually deployed. This is exactly one of the changes I had in mind when
            proposing to do a larger refactoring of StationXML towards 2.0 and
            represent information as properties of natural entities, rather than for a
            single application case (in this case: "I need the response of stream
            MYSTATION.HHZ at date x"). Such structural changes are not backward
            compatible (and should not be...); thus, they should not be done
            incrementally, but in a (hopefully rare), large review.

            In this context, InventoryXML has a nice feature to be considered also
            for StationXML:
            A sensor refers to its default (either individual or type-specific
            response). However, it can also have multiple calibration periods as
            children, with individual timespan and reference to individual instances of
            response. These override the standard response.
            Thus, if you have a stream pointing to sensor, and the sensor has
            calibration periods, you do time matching in order to get the calibrated
            response, if it does not have any, you are happy to use the sensor's
            default response.

            Best regards,

            Philipp (Kästli)


            ----------------------
            FDSN Working Group II (
            http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)

            Sent via IRIS Message Center (http://www.fdsn.org/message-center/)
            Update subscription preferences at http://www.fdsn.org/account/profile/



            ----------------------
            FDSN Working Group II (
            http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)

            Sent via IRIS Message Center (http://www.fdsn.org/message-center/)
            Update subscription preferences at http://www.fdsn.org/account/profile/




            • Philip Crotwell
              Aug. 7, 2015, 8:23 p.m.
              Hi

              There is one other use case I forgot to mention, but I think is important
              and was part of my original motivation. Consider the case of a network
              operator uploading metadata to a datacenter. I really hope that some day
              soon we can dispense with dataless seed and use stationxml for this. In the
              case of uploading metadata, I really think there is an advantage for the
              network operator to be able to say "the response of this channel is the
              nominal response for this configuration of this sensor and that logger". It
              makes the submission easier, gives the datacenter more information, and
              potentially allows for future updates to a nominal response to be
              incorporated.

              The way I have done uploaded metadata in the past, for example, is to
              download the NRL response from IRIS, have PDCC create dataless that
              incorporates that response but without any information that connects the
              response back to the NRL, and then send the very same (hopefully) response
              back to IRIS? Seems kind of weird to download nominal responses from IRIS
              just to send them right back, especially with the very real potential of me
              and my fat fingers screwing something up in the transition?

              Thoughts?

              Philip


              On Fri, Aug 7, 2015 at 5:47 PM, Philip Crotwell <crotwell<at>seis.sc.edu>
              wrote:


              Hi

              I guess their are two ways of using StationXML, as a "message" and as a
              "document", and they have different needs and goals. In a messaging use,
              the xml exists only for a short time as it is transmitted from server to
              client and it is perfectly fine to convey some information directly and
              refer to related information via links, think web page with linked images.
              For a document use, you want long term storage and links external links are
              "bad" in the sense that you can't rely on them existing in the future.
              Think PDF with embedded images.

              It sounds like you are viewing stationxml primarily as a long lived
              document, but I question whether that is really the only, or even most
              common use. And the argument that the response must live in the same file
              as the other net/station/channel metadata is not that far from the argument
              that the metadata and the waveform data must live in the same file (a la
              full seed).

              I am not saying that a large, all encompassing document is a wrong use of
              stationxml, and my proposal does not remove the existing "stages in
              response in channel" style. But stationxml as lots of small transient
              messages between client and server is also a valid use, and in some ways
              may be more common. In my particular use case, SOD would greatly benefit
              from not getting a response for channel A when it is exactly the same as
              the response for channel B. If the ID is the same, I get it from the local
              cache.

              That said, it may be that this type of short term messaging may be
              better/easier to accommodate via something like StationJSON. In that case
              then the JSON needs to be able to carry all the same information, and of
              course actually be implemented as part of the servers.

              Maybe that is a fair way to divide the use, xml as deep, long term, self
              contained files, and JSON as small, short term, linked messages?

              Philip



              On Fri, Aug 7, 2015 at 4:57 PM, Chad Trabant <chad<at>iris.washington.edu>
              wrote:


              Hi Philip, Joachim & Philipp,

              Rearranging the response details into re-usable entities within a
              document is certainly worth discussing. We should keep in mind that it
              would be pretty disruptive to existing reading software, more disruptive
              than any other changes proposed so far. I'm not saying it wouldn't be
              worth it.

              On the other hand, referring to and relying on external definitions for a
              complete document sounds like a bad idea to me. There is a significant
              added burden to any reader of such information to get a complete response,
              especially if the external information is in another format like RESP.
              Beyond this reader burden there are issues of longevity and versioning. If
              the external document goes away (there is no guarantee that NRL entries are
              permanent) then the StationXML now refers to a dead link and is
              incomplete. If the external document changes then the complete,
              re-constituted document changes and it may not be obvious to any given
              reader or user.

              Chad

              On Aug 6, 2015, at 2:47 PM, Philip Crotwell <crotwell<at>seis.sc.edu> wrote:

              Hi

              Here is a first cut at allowing stationXML to refer to responses instead
              of embedding them directly in a channel.
              https://github.com/FDSN/StationXML/pull/6

              thanks
              Philip

              On Thu, Aug 6, 2015 at 12:16 PM, <"Philipp Kästli
              <kaestli<at>sed.ethz.ch>"@fdsn.fdsn.org> wrote:


              Philip Crotwell wrote on 08/03/2015 03:35 PM:

              One thing that might help with this is to make use of externally
              defined responses, such as the Nominal
              Response Library, and link to it from within the Sensor and
              DataLogger elements, and make this an
              alternative way of expressing the response.
              [...]
              Also, because it is useful to have fully self contained statilxml, it
              might be useful to create a separate element parallel to Network

              Joachim Saul wrote:

              In other words, turn (at least) Response into a "top-level" element.
              That would allow to have fully self contained XML, while referencing
              Response from Sensor, DataLogger et al

              You would need two top level elements:
              - sensor (with model, manufacturer, vendor, serial, etc. reference to
              response) -> this would allow not to repeat identical information with 3
              streams, but just reference it by publicid
              - response (with the response to be considered). -> if a network
              operator provides just standard response per sensor model, all sensors of
              model x may reference to a single instance of response. If individual
              sensors were individually measured/calibrated, then every sensor may point
              to its own response (however only once, without repetition for each stream
              and each deployment of the same sensor [with the same serial
              number/publicID] at a different station).

              Such a refactoring would also allow to describe a sensor if it is not
              actually deployed. This is exactly one of the changes I had in mind when
              proposing to do a larger refactoring of StationXML towards 2.0 and
              represent information as properties of natural entities, rather than for a
              single application case (in this case: "I need the response of stream
              MYSTATION.HHZ at date x"). Such structural changes are not backward
              compatible (and should not be...); thus, they should not be done
              incrementally, but in a (hopefully rare), large review.

              In this context, InventoryXML has a nice feature to be considered also
              for StationXML:
              A sensor refers to its default (either individual or type-specific
              response). However, it can also have multiple calibration periods as
              children, with individual timespan and reference to individual instances of
              response. These override the standard response.
              Thus, if you have a stream pointing to sensor, and the sensor has
              calibration periods, you do time matching in order to get the calibrated
              response, if it does not have any, you are happy to use the sensor's
              default response.

              Best regards,

              Philipp (Kästli)


              ----------------------
              FDSN Working Group II (
              http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)

              Sent via IRIS Message Center (http://www.fdsn.org/message-center/)
              Update subscription preferences at http://www.fdsn.org/account/profile/



              ----------------------
              FDSN Working Group II (
              http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)

              Sent via IRIS Message Center (http://www.fdsn.org/message-center/)
              Update subscription preferences at http://www.fdsn.org/account/profile/





              ----------------------
              FDSN Working Group II (
              http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)

              Sent via IRIS Message Center (http://www.fdsn.org/message-center/)
              Update subscription preferences at http://www.fdsn.org/account/profile/



            • Joachim Saul
              Aug. 10, 2015, 7:13 p.m.
              Hello Philip,

              when developing QuakeML on one hand, and the SeisComP software on the other hand, around nine years ago we had very similar discussions. Data needed to be represented as (XML) documents containing huge earthquake catalogs in some application cases (e.g., reference catalog for a hazard assessment), but very small snippets of information in others (e.g., one software informing another one on the appearance of a new Pick). Furthermore, you need some way to store the information in a database, for which you might not want to use XML strings.

              Therefore, what IMHO we are looking for may not be just an XML format. In fact, like in the case of QuakeML, we are actually more generically looking for a data model. XML is just one of many ways to serialize the data, which in many use cases may not be the most efficient. For instance in SeisComP, we initially used XML for messages containing e.g. picks and amplitudes, resulting in 90% of the CPU time being spent on (de)serializing XML. Therefore now binary messages are used. I am certainly not in favor of using binary formats for station metadata exchange in an FDSN context. But as you say, depending on the use case, it may well be JSON (or YAML or some future format). The representation / serialization format should not really matter; different representations are converted easily as long as they use a standard *data* *model*. I propose to standardize the model itself (by the way of a UML diagram), and an XML representation (as a well defined format for high-level data ex
              change).


              Regards
              Joachim


              Philip Crotwell wrote on 08/07/2015 11:47 PM:

              Hi

              I guess their are two ways of using StationXML, as a "message" and as a
              "document", and they have different needs and goals. In a messaging use,
              the xml exists only for a short time as it is transmitted from server to
              client and it is perfectly fine to convey some information directly and
              refer to related information via links, think web page with linked
              images. For a document use, you want long term storage and links
              external links are "bad" in the sense that you can't rely on them
              existing in the future. Think PDF with embedded images.

              It sounds like you are viewing stationxml primarily as a long lived
              document, but I question whether that is really the only, or even most
              common use. And the argument that the response must live in the same
              file as the other net/station/channel metadata is not that far from the
              argument that the metadata and the waveform data must live in the same
              file (a la full seed).

              I am not saying that a large, all encompassing document is a wrong use
              of stationxml, and my proposal does not remove the existing "stages in
              response in channel" style. But stationxml as lots of small transient
              messages between client and server is also a valid use, and in some ways
              may be more common. In my particular use case, SOD would greatly benefit
              from not getting a response for channel A when it is exactly the same as
              the response for channel B. If the ID is the same, I get it from the
              local cache.

              That said, it may be that this type of short term messaging may be
              better/easier to accommodate via something like StationJSON. In that
              case then the JSON needs to be able to carry all the same information,
              and of course actually be implemented as part of the servers.

              Maybe that is a fair way to divide the use, xml as deep, long term, self
              contained files, and JSON as small, short term, linked messages?

              Philip



              On Fri, Aug 7, 2015 at 4:57 PM, Chad Trabant <chad<at>iris.washington.edu
              <chad<at>iris.washington.edu>> wrote:


              Hi Philip, Joachim & Philipp,

              Rearranging the response details into re-usable entities within a
              document is certainly worth discussing. We should keep in mind that
              it would be pretty disruptive to existing reading software, more
              disruptive than any other changes proposed so far. I'm not saying
              it wouldn't be worth it.

              On the other hand, referring to and relying on external definitions
              for a complete document sounds like a bad idea to me. There is a
              significant added burden to any reader of such information to get a
              complete response, especially if the external information is in
              another format like RESP. Beyond this reader burden there are
              issues of longevity and versioning. If the external document goes
              away (there is no guarantee that NRL entries are permanent) then the
              StationXML now refers to a dead link and is incomplete. If the
              external document changes then the complete, re-constituted document
              changes and it may not be obvious to any given reader or user.

              Chad

              On Aug 6, 2015, at 2:47 PM, Philip Crotwell <crotwell<at>seis.sc.edu
              <crotwell<at>seis.sc.edu>> wrote:

              Hi

              Here is a first cut at allowing stationXML to refer to responses
              instead of embedding them directly in a channel.
              https://github.com/FDSN/StationXML/pull/6

              thanks
              Philip

              On Thu, Aug 6, 2015 at 12:16 PM, <"Philipp Kästli
              <kaestli<at>sed.ethz.ch>"@fdsn.fdsn.org
              <mailto:%22Philipp+K%C3%A4stli+%3Ckaestli<at>sed.ethz.ch%3E%22<at>fdsn.fdsn.org>>
              wrote:


              Philip Crotwell wrote on 08/03/2015 03:35 PM:

              One thing that might help with this is to make use of externally defined responses, such as the Nominal
              Response Library, and link to it from within the Sensor and DataLogger elements, and make this an
              alternative way of expressing the response.
              [...]
              Also, because it is useful to have fully self contained statilxml, it
              might be useful to create a separate element parallel to Network

              Joachim Saul wrote:

              In other words, turn (at least) Response into a "top-level" element.
              That would allow to have fully self contained XML, while referencing
              Response from Sensor, DataLogger et al

              You would need two top level elements:
              - sensor (with model, manufacturer, vendor, serial, etc.
              reference to response) -> this would allow not to repeat
              identical information with 3 streams, but just reference it by
              publicid
              - response (with the response to be considered). -> if a
              network operator provides just standard response per sensor
              model, all sensors of model x may reference to a single
              instance of response. If individual sensors were individually
              measured/calibrated, then every sensor may point to its own
              response (however only once, without repetition for each
              stream and each deployment of the same sensor [with the same
              serial number/publicID] at a different station).

              Such a refactoring would also allow to describe a sensor if it
              is not actually deployed. This is exactly one of the changes I
              had in mind when proposing to do a larger refactoring of
              StationXML towards 2.0 and represent information as properties
              of natural entities, rather than for a single application case
              (in this case: "I need the response of stream MYSTATION.HHZ at
              date x"). Such structural changes are not backward compatible
              (and should not be...); thus, they should not be done
              incrementally, but in a (hopefully rare), large review.

              In this context, InventoryXML has a nice feature to be
              considered also for StationXML:
              A sensor refers to its default (either individual or
              type-specific response). However, it can also have multiple
              calibration periods as children, with individual timespan and
              reference to individual instances of response. These override
              the standard response.
              Thus, if you have a stream pointing to sensor, and the sensor
              has calibration periods, you do time matching in order to get
              the calibrated response, if it does not have any, you are
              happy to use the sensor's default response.

              Best regards,

              Philipp (Kästli)


              ----------------------
              FDSN Working Group II
              (http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)

              Sent via IRIS Message Center (http://www.fdsn.org/message-center/)
              Update subscription preferences at
              http://www.fdsn.org/account/profile/



              ----------------------
              FDSN Working Group II
              (http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)

              Sent via IRIS Message Center (http://www.fdsn.org/message-center/)
              Update subscription preferences at
              http://www.fdsn.org/account/profile/





              ----------------------
              FDSN Working Group II (http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)

              Sent via IRIS Message Center (http://www.fdsn.org/message-center/)
              Update subscription preferences at http://www.fdsn.org/account/profile/



          • Joachim Saul
            Aug. 11, 2015, 12:30 p.m.
            Hi Chad,

            Chad Trabant wrote on 08/07/2015 10:58 PM:
            On the other hand, referring to and relying on external definitions for
            a complete document sounds like a bad idea to me.

            It is a bad idea to refer to external data *if* they are not guaranteed
            to be persistent or if they are not in a standard format.

            There is a
            significant added burden to any reader of such information to get a
            complete response, especially if the external information is in another
            format like RESP. Beyond this reader burden there are issues of
            longevity and versioning. If the external document goes away (there is
            no guarantee that NRL entries are permanent) then the StationXML now
            refers to a dead link and is incomplete. If the external document
            changes then the complete, re-constituted document changes and it may
            not be obvious to any given reader or user.

            I agree. On the other hand, "external" may refer to a Response that is
            not part of a Channel, hence from the Channel point of view it is
            "external" even though it resides within the same, self-contained XML
            document. Where self-containedness is a requirement it can always be
            achieved.

            One could think of persistent identifiers (e.g., DOIs) for responses,
            too. In fact, given the decreasing costs for assigning persistent
            identifiers, it might become feasible to use them for addressing
            specific responses as part of the NRL. Probably not within the very near
            future, but I have the impression that developments will move into that
            direction.

            Cheers
            Joachim

            • Philip Crotwell
              Aug. 11, 2015, 10:37 a.m.
              Hi

              One more thought, a reference is cheap, and so can be returned in a
              "channel" level request. If the client asks for "response" level, they get
              the stages, possibly referred to within the same file. This for existing
              software and longevity I think.

              On the other hand, all this is meaningless unless datacenters are actually
              willing to coalesce identical (often nominal) responses and there is a URL
              to retrieve an individual response.

              The upload use case may still be worth pursuing.

              Philip

              On Tue, Aug 11, 2015 at 6:32 AM, Joachim Saul <saul<at>gfz-potsdam.de> wrote:

              Hi Chad,

              Chad Trabant wrote on 08/07/2015 10:58 PM:
              On the other hand, referring to and relying on external definitions for
              a complete document sounds like a bad idea to me.

              It is a bad idea to refer to external data *if* they are not guaranteed
              to be persistent or if they are not in a standard format.

              There is a
              significant added burden to any reader of such information to get a
              complete response, especially if the external information is in another
              format like RESP. Beyond this reader burden there are issues of
              longevity and versioning. If the external document goes away (there is
              no guarantee that NRL entries are permanent) then the StationXML now
              refers to a dead link and is incomplete. If the external document
              changes then the complete, re-constituted document changes and it may
              not be obvious to any given reader or user.

              I agree. On the other hand, "external" may refer to a Response that is
              not part of a Channel, hence from the Channel point of view it is
              "external" even though it resides within the same, self-contained XML
              document. Where self-containedness is a requirement it can always be
              achieved.

              One could think of persistent identifiers (e.g., DOIs) for responses,
              too. In fact, given the decreasing costs for assigning persistent
              identifiers, it might become feasible to use them for addressing
              specific responses as part of the NRL. Probably not within the very near
              future, but I have the impression that developments will move into that
              direction.

              Cheers
              Joachim

              ----------------------
              FDSN Working Group II (
              http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)

              Sent via IRIS Message Center (http://www.fdsn.org/message-center/)
              Update subscription preferences at http://www.fdsn.org/account/profile/