diff --git a/guide/images/architecture/c4.container.png b/guide/images/architecture/c4.container.png index 825831cf..9118f280 100644 Binary files a/guide/images/architecture/c4.container.png and b/guide/images/architecture/c4.container.png differ diff --git a/guide/index.adoc b/guide/index.adoc index 10bbf261..9ee9bb72 100644 --- a/guide/index.adoc +++ b/guide/index.adoc @@ -1,7 +1,7 @@ -:title: Guide to the WMO Information System (WMO-No. 1061), Volume II - WMO Information System 2.0 (APPROVED) +:title: Guide to the WMO Information System (WMO-No. 1061), Volume II - WMO Information System 2.0 (DRAFT-INFCOM-4) :titletext: {title} -:version: 1.3.0 -:date: 2026-06-15 +:version: 1.4.0-draft +:date: {docdate} :copyright: Copyright © 2026 World Meteorological Organization (WMO) :doctype: book :encoding: utf-8 @@ -17,11 +17,11 @@ |=== |{set:cellbgcolor:#FFFFFF} |[big]*World Meteorological Organization* -|Date: 2026-06-15 +|Date: {docdate} |Version: {version} -|Document status: APPROVED +|Document status: DRAFT-INFCOM-4 |Revision history: see <<_revision_history,Annex A>> -|Document location: https://wmo-im.github.io/wis2-guide/guide/wis2-guide-APPROVED.html +|Document location: https://wmo-im.github.io/wis2-guide/guide/infcom/guide/wis2-guide-DRAFT-INFCOM-4.html |WMO publication location: https://library.wmo.int/idurl/4/69130 |Standing Committee on Information Management and Technology (SC-IMT)footnote:[https://community.wmo.int/governance/commission-membership/commission-observation-infrastructures-and-information-systems-infcom/commission-infrastructure-officers/infcom-management-group/standing-committee-information-management-and-technology-sc-imt] |Commission for Observation, Infrastructure and Information Systems (INFCOM)footnote:[https://community.wmo.int/governance/commission-membership/infcom] @@ -43,8 +43,13 @@ include::sections/part4/index.adoc[] include::sections/part5/index.adoc[] -include::sections/annex_a.adoc[] +include::sections/part6/index.adoc[] + +include::sections/appendix_a.adoc[] +include::sections/appendix_b.adoc[] + +include::sections/annex_a.adoc[] // include::sections/other-considerations.adoc[] diff --git a/guide/sections/annex_a.adoc b/guide/sections/annex_a.adoc index b93305f3..b0243e2b 100644 --- a/guide/sections/annex_a.adoc +++ b/guide/sections/annex_a.adoc @@ -9,4 +9,5 @@ |November 2024 | 1.1.0 | SC-IMT |https://github.com/wmo-im/wis2-guide/milestone/3?closed=1[See GitHub]| Fast-track 2024-2 |February 2026 | 1.2.0 | SC-IMT |https://github.com/wmo-im/wis2-guide/milestone/7?closed=1[See GitHub] |Fast-track 2025-2 |June 2026 | 1.3.0 | SC-IMT |https://github.com/wmo-im/wis2-guide/milestone/4?closed=1[See GitHub] |Fast-track 2026-1 +|August 2026 | 1.4.0-DRAFT | SC-IMT | |INFCOM-4 |=== diff --git a/guide/sections/appendix_a.adoc b/guide/sections/appendix_a.adoc new file mode 100644 index 00000000..df1e40f3 --- /dev/null +++ b/guide/sections/appendix_a.adoc @@ -0,0 +1,94 @@ +[appendix] +:appendix-caption: Annex + +== APPENDIX A: PER-COMPONENT OPERATIONAL CRITERIA + +This appendix contains additional information to section 2.7.2.9. The operational criteria specific for each category of designated Global Services are set out below. Criteria for other types of Global Service will be added in subsequent revisions of this Guide. + +=== A.1 Global Broker + +The operational status of a Global Broker is assessed against two KPIs, both measured through Sensor Centre Global Broker (SCGB) instances operating in different parts of the globe, so as to reduce the influence of localized network conditions or of any individual Sensor Centre. Hourly KPI evaluation is based on the best result across all SCGB instances; KPIs are calculated at the Global Monitor, not at the Sensor Centre itself. + +==== A.1.1 Message completeness KPI + +This KPI measures the percentage of WIS2 Notification Messages (WNM) published by the Global Broker, within a defined cut-off time, compared against a baseline derived from all Global Brokers. + +*Measure (calculated each hour):* + +The percentage of WNM messages published by the Global Broker within the cut-off time of 60 seconds, compared to the baseline, is assessed using a randomized sample of approximately 10 per cent of messages. The number of messages published by each Global Broker within the cutoff time is recorded using the metric `wmo_wis2_scgb_messages_published_total`. The baseline is the highest number of messages published by any Global Broker, as recorded by any SCGB instance, within the previous hour. A message is considered "missed" if a Global Broker has not published that message within 60-seconds of it first being published by any other Global Broker. + +*KPI evaluation (hourly):* + +The message completeness KPI is set to "pass" for the hour if the percentage of messages published is not less than 95 per cent of the baseline; otherwise it is set to "fail" for the hour. + +==== A.1.2 Connectivity KPI + +This KPI measures the proportion of the hour during which the public MQTT endpoint of the Global Broker are reachable, as determined from the connection status recorded by SCGB instances. + +*Measure (calculated each hour):* + +The proportion of time that the Global Broker’s MQTT end-point is accessible, published by the Sensor Centre Global Broker using metric `wmo_wis2_scgb_connected_flag`. + +*KPI evaluation (hourly):* + +The connectivity KPI is set to "pass" for the hour if the `wmo_wis2_scgb_connected_flag` is set to `1` ("True", connected) for not less than 95 per cent of the time; otherwise it is set to "fail" for the hour. + +==== A.1.3 Combined hourly and monthly KPI + +* For each hour, the Global Broker is considered operational where *both* the message completeness KPI *and* the connectivity KPI are set to "pass". If either KPI fails for the hour, the Global Broker is considered non-operational for that hour. +* Each month, the overall KPI is set to "pass" if the combined hourly KPI passes for not less than 98 per cent of hours in the month (i.e. the Global Broker may experience up to 14 "failure" hours in a calendar month and still be considered performing within the service level). +* Hours coinciding with planned outages, or other _force majeure_ agreed by SC-IMT, are excluded from the monthly KPI calculation. +* The monthly KPI is used in the assessment of Global Broker performance informing the determination of downtime in accordance with section 2.7.2.6. + +''' + +=== A.2 Global Cache + +The operational status of a Global Cache is assessed against two KPIs, both measured through Sensor Centre Global Cache (SCGC) instances operating in different parts of the globe, so as to reduce the influence of localized network conditions or of any individual Sensor Centre. Hourly KPI evaluation is based on the best result across all SCGC instances; KPIs are calculated at the Global Monitor, not at the Sensor Centre itself. + +==== A.2.1 Cache completeness KPI + +This KPI measures the percentage of data items published by the Global Cache compared with the total number of data items published by all WIS2 Nodes (on `origin` topics). + +*Measure (calculated each hour):* + +The number of data items published by all origin WIS2 Nodes is recorded using the metric `wmo_wis2_scgc_messages_published_total`. Only those data items expected to be cached are included in this measure. + +The number of data items cached by a Global Cache is recorded using the metric `wmo_wis2_scgc_messages_cached_total`. Data items that are cached more than 600-seconds after publication by the origin WIS2 Node are excluded from this measure. + +Because Global Caches are designed to discard out-of-sequence updates, and counting such discards as "misses" would not be a fair reflection of performance, KPI performance is evaluated only for "canonical" data items (i.e., where the Notification Message specifies the link-relation `canonical`). Both `wmo_wis2_scgc_messages_published_total` and `wmo_wis2_scgc_messages_cached_total` metrics include the label `rel` to distinguish the link-relation. + +The percentage of data items published by a Global Cache is calculated as the increase in metric `wmo_wis2_scgc_messages_cached_total (rel=canonical)` over the previous hour ÷ the increase in metric `wmo_wis2_scgc_messages_published_total (rel=canonical)` over the previous hour. + +*KPI evaluation (hourly):* + +The cache completeness KPI is set to "pass" for the hour if the percentage of data items cached is not less than 95 per cent of the number published by origin; otherwise it is set to "fail" for the hour. + +==== A.2.2 Successful downloads KPI + +This KPI measures the proportion of cached data items that can be successfully downloaded from the Global Cache, including, where a checksum is available in the message, validation of data integrity. + +*Measure (calculated each hour):* + +This measure is evaluated using a sample set of 300 randomly selected data items each hour. Data items must be cached by all Global Caches to be included in the sample set. The same sample set is used for all Global Caches. A small sample size is used to avoid overloading both Global Caches and Sensor Centres. + +The total number of data items sampled for download validation is recorded using the metric `wmo_wis2_scgc_download_selected_total`. + +The total number of data items in the sample set successfully downloaded by each Global Cache is recorded using the metric `wmo_wis2_scgc_download_successful_total`. + +The percentage of data items successfully downloaded is calculated as the increase in metric `wmo_wis2_scgc_download_successful_total` over the previous hour ÷ the increase in metric `wmo_wis2_scgc_download_selected_total` over the previous hour. + +*KPI evaluation (hourly):* + +The successful download KPI is set to "pass" for the hour if the percentage of successful downloads is not less than 95 per cent; otherwise it is set to "fail" for the hour. + +Note that if `wmo_wis2_scgc_download_selected_total` is zero for the previous hour (i.e., no data items were cached by any Global Cache - perhaps because one Global Cache is offline), the successful downloads KPI cannot be calculated. In this case, the KPI is automatically set to "pass" for all Global Caches. The Global Cache that is offline will fail the cache completeness KPI, thus the combined KPI (see below) for the offline Global Cache will always be a fail. + +==== A.2.3 Combined hourly and monthly KPI + +* For each hour, the Global Cache is considered operational where *both* the cache completeness KPI *and* the successful downloads KPI are set to "pass". If either KPI fails for the hour, the Global Cache is considered non-operational for that hour. +* Each month, the overall KPI is set to "pass" if the combined hourly KPI passes for not less than 98 per cent of hours in the month (i.e. the Global Cache may experience up to 14 "failure" hours in a calendar month and still be considered performing within the service level). +* Hours coinciding with planned outages, or other _force majeure_ agreed by SC-IMT, are excluded from the monthly KPI calculation. +* The monthly KPI is used in the assessment of Global Cache performance informing the determination of downtime in accordance with section 2.7.2.6. + + diff --git a/guide/sections/appendix_b.adoc b/guide/sections/appendix_b.adoc new file mode 100644 index 00000000..8d8dd0c3 --- /dev/null +++ b/guide/sections/appendix_b.adoc @@ -0,0 +1,3 @@ +== APPENDIX B: WMO CORE METADATA PROFILE (VERSION 2), KEY PERFORMANCE INDICATORS + +https://wmo-im.github.io/wcmp2/kpi/wcmp2-kpi-DRAFT.html \ No newline at end of file diff --git a/guide/sections/part1/data-consumer.adoc b/guide/sections/part1/data-consumer.adoc index 9ca32db6..709cb3d8 100644 --- a/guide/sections/part1/data-consumer.adoc +++ b/guide/sections/part1/data-consumer.adoc @@ -54,9 +54,9 @@ A discovery metadata record or notification message may provide more than one do A data consumer should expect to receive multiple notification messages about the same data item. Furthermore, data consumers should not assume that messages arrive in time-sequence order (i.e., newest published message, as indicated by ``properties.pubtime``, arriving last). Consequently, for notification messages relating to the same data item (i.e., notification messages with identical ``properties.data_id``), they should: -1. Prioritise the message with the latest publication time. +1. Prioritise the message with the latest publication time. 2. Discard any notification messages earlier publication time that have not yet been processed or may be received in the future. -3. Where a previous notification message with earlier publication time has already been processed, any data or metadata associated with the earlier notification message should be overwritten. +3. Where a previous notification message with earlier publication time has already been processed, any data or metadata associated with the earlier notification message should be overwritten. This means that a data consumer should keep track of notification messages that have been processed so that newly arrived notification messages can be compared with earlier messages identify which notification message relates to the latest version of the data or metadata. Given that messages propagate rapidly through WIS2, a record of previous messages need not exceed a 60-minute duration. diff --git a/guide/sections/part1/data-publisher.adoc b/guide/sections/part1/data-publisher.adoc index 9dd80304..3197c897 100644 --- a/guide/sections/part1/data-publisher.adoc +++ b/guide/sections/part1/data-publisher.adoc @@ -183,11 +183,11 @@ Data publishers planning to operate WIS2 Nodes, at a minimum, should read the fo The following sections are recommended for further reading: -* <<_part_iii_information_management>> -* <<_part_iv_security>> -* <<_part_v_competencies>> +* <<_part_iv_information_management>> +* <<_part_v_security>> +* <<_part_vi_competencies>> -Note that _PART IV. Security_ and _PART V. Competencies_ reference content originally published for WIS1. These sections remain largely applicable and will be updated in subsequent releases of this Guide. +Note that _PART V. Security_ and _PART VI. Competencies_ reference content originally published for WIS1. These sections remain largely applicable and will be updated in subsequent releases of this Guide. Data publishers publishing aviation weather data via WIS2 for onward transmission through the International Civil Aviation Organization (ICAO) System Wide Information Management (SWIM), should also read <<_2_8_1_1_publishing_meteorological_data_through_wis2_into_icao_swim>>. diff --git a/guide/sections/part2/global-services.adoc b/guide/sections/part2/global-services.adoc index d0e7c466..acc0b22c 100644 --- a/guide/sections/part2/global-services.adoc +++ b/guide/sections/part2/global-services.adoc @@ -26,7 +26,11 @@ The WMO Secretariat and other Global Services will make the required changes to ==== 2.7.2 Performance management and monitoring of a Global Service -===== 2.7.2.1 Monitoring and metrics for WIS2 operations +===== 2.7.2.1 General + +The performance management of designated Global Services is governed by 2.7–2.9 of the Manual on WIS, Volume II. This section of the Guide provides operational guidance on the service level, the methodology used to assess it, the reporting cadence, and the remediation and restoration practice. Section 2.7.2.9 sets out the per-component operational criteria specific to each category of Global Service. The framework set out in this section and in sections 2.7.2.8 and 2.7.2.9 applies for the purposes of the suspension procedure set out in 2.8 of the Manual and the revocation procedure set out in 2.9 of the Manual; routine incident management for Global Services continues to be coordinated by GISCs under 4.7.4 of the Manual. + +===== 2.7.2.2 Monitoring and metrics for WIS2 operations The availability of data and the performance of system components within WIS2 are actively monitored by GISCs and the Global Monitor service to ensure proactive responses to incidents and effective capacity planning for future operations. @@ -43,10 +47,11 @@ Metrics for WIS2 monitoring should follow the naming convention ``wmo__ The full set of the WIS2 monitoring metrics is given in WMO: WIS2 Metric Hierarchy footnote:[See https://github.com/wmo-im/wis2-metric-hierarchy.] -===== 2.7.2.2 Service levels, performance indicators, and fair usage policies +===== 2.7.2.3 Service levels, performance indicators, and fair usage policies * Each WIS centre operating a WIS2 Node is responsible for achieving the highest possible level of service based on its resources and capabilities. * All Global Services, in particular Global Brokers and Global Caches, are collectively responsible for making WIS a reliable and efficient means of exchanging the data required for the operation of all WIS centres. The architecture provides a redundant solution where the failure of one component will not impact the overall level of service of WIS. * Each Global Service should aim to achieve at least 99.5% availability of the service it provides. This is not a contractual target. It should be considered by the entity providing the Global Service as a guideline when designing and operating that service. +* Any Global Service operator anticipating an outage that may exceed 4 hours should notify other WIS2 centres (including other Global Services and the WMO Secretariat) in advance to allow them to prepare for the disruption. A monitor message will be published by the Global Monitor informing about the outage. * A Global Broker: ** Should support a minimum of 200 WIS2 Nodes or Global Services; ** Should support a minimum of 1 000 subscribers; @@ -54,45 +59,88 @@ The full set of the WIS2 monitoring metrics is given in WMO: WIS2 Metric Hierarc * A Global Cache: ** Should support a minimum of 100 GB of data in the cache; ** Should support a minimum of 1 000 simultaneous downloads; -** Could limit the number of simultaneous connections from a user (known by its originating source IP) to five; +** Could limit the number of simultaneous connections from a user (known by its originating source IP) to not less than sixteen; ** Could limit the bandwidth usage of the service to 1 Gb/s. +* A Global Discovery Catalogue: +** Should support a minimum of 20 000 metadata records; +** Should support a minimum of 25 requests per second to the API endpoint. * A Global Monitor: ** Should support a minimum of 50 metrics providers; ** Should support 200 simultaneous "access" to the dashboard; ** Could limit the bandwidth usage of the service to 100 Mb/s. -* A Global Discovery Catalogue: -** Should support a minimum of 20 000 metadata records; -** Should support a minimum of 50 requests per second to the API endpoint. -===== 2.7.2.3 Metrics for Global Services +===== 2.7.2.4 Metrics for Global Services In the following sections, and for each Global Service, a set of metrics is defined. Each Global Service will provide those metrics. They will then be ingested by the Global Monitor. +===== 2.7.2.5 Service level + +The service level is met in a calendar month where the designated Global Service is operational for not less than the proportion of that month set out, for the relevant category of Global Service, in the per-component operational criteria in section 2.7.2.9. The conditions under which a Global Service is considered operational are set out in those criteria, and the proportion of operational time over the month is determined in accordance with section 2.7.2.6. + +The service level is the reference for the suspension and revocation procedures set out in 2.8 and 2.9 of the Manual; the conditions under which those procedures are initiated are set out in the Manual. + +===== 2.7.2.6 Assessment methodology + +For each calendar month, the proportion of time during which the Global Service was operational is determined. The conditions under which a Global Service is considered operational vary by category and are set out in the per-component operational criteria in section 2.7.2.9. The time during which the Global Service is not operational is determined from the operational metrics published by the Global Service and by the Sensor Centres against those criteria, by aggregating the time intervals during which one or more of the per-component operational criteria are not met. Scheduled maintenance windows notified in advance to the WMO Secretariat are excluded from this time, provided that the total duration of such windows does not exceed four hours per calendar month, unless an exception has been approved by SC-IMT. SC-IMT will also exclude hours where Global Service operation is adversely affected by conditions beyond the control of the operator. The Global Service meets the service level for the calendar month where the proportion of operational time is not less than the value set out in section 2.7.2.5 and 2.7.2.9. + +In the event of an incident affecting the operation of Global Service, the operating Member should maintain a record of the start time, the end time, and affected criteria. These incident records, while not themselves binding, support the determination of operational time and provide operational context for the dashboards and statistical summaries provided by the Global Monitors. + +===== 2.7.2.7 Reporting cadence + +Operational metrics for each designated Global Service are published in near-real-time by the Global Service and by the Sensor Centres. The Global Monitors provide dashboards and statistical summaries based on these metrics. The monthly service-level results of each designated Global Service, including performance against the service level over the relevant assessment period, are published openly by the WMO Secretariat in accordance with 2.7.3.1 of the Manual, so that the performance of all designated Global Services is publicly visible, and are also communicated directly to each operating Member in accordance with 2.7.3.3 of the Manual. SC-IMT reviews aggregated performance results at each of its meetings and considers any indications of sustained deterioration in accordance with 2.7.3.4 of the Manual. + +===== 2.7.2.8 Remediation and restoration + +====== 2.7.2.8.1 Deadline for restoring performance + +Following notification under 2.8.2.1 of the _Manual on WIS_, Volume II, SC-IMT establishes a deadline under 2.8.2.2 by which the operating Member is expected to re-establish the performance of the Global Service so that it meets the service level. When establishing the deadline, SC-IMT considers the nature of the failure to meet the service level, the operational impact on WIS 2.0 and its users, and the typical time required to address comparable issues. The deadline reflects the operational nature of the Global Service: it may be set in hours where the issue can be addressed by routine operational measures, and in days where more substantive corrective action is required, but it should not normally extend beyond what is needed to address the immediate issue. + +The operating Member should keep SC-IMT and the WMO Secretariat informed of the progress towards re-establishing performance. Failure to meet the deadline triggers suspension in accordance with 2.8.3.1 of the Manual. + +====== 2.7.2.8.2 Remediation plan following suspension + +Following suspension under 2.8.3.1 of the Manual, the operating Member should submit a remediation plan to SC-IMT and the WMO Secretariat. The remediation plan should describe the root cause analysis of the breach, the corrective actions to be taken, the proposed timeline for the restoration of the Global Service to WIS 2.0 operations, and the arrangements for monitoring and reporting progress. SC-IMT and the operating Member agree the timeline for restoration under 2.8.4.1 of the Manual. + +The operating Member should report on the implementation of the remediation plan at intervals not exceeding thirty calendar days. While a suspended Global Service does not meet the service level, each such calendar month counts towards the rolling window in 2.7.2.3 of the Manual, so that persistent failure to restore the Global Service may result in the initiation of the procedure for the revocation of the designation, in accordance with 2.8.4.3 of the Manual. + +====== 2.7.2.8.3 Restoration + +Restoration of a suspended Global Service to WIS 2.0 operations is confirmed by SC-IMT, with the assistance of the WMO Secretariat and using the dashboards and statistical summaries provided by the Global Monitors, once the operational performance has met the service level for not less than one calendar month following completion of the remediation actions. + +==== 2.7.2.9 Per-component operational criteria + +The operational criteria specific to each category of designated Global Service are set out below. These criteria define what constitutes operational performance for the purposes of the service level and inform the determination of downtime for each component. The service level stated for each Global Service is used for the performance-management procedures of the Manual. The design-capacity figures for each category remain as set out in 2.7.2.3. The detailed assessment methodology and Key Performance Indicators (KPIs) corresponding to each criterion are set out in Appendix A, and the detailed technical checks are maintained by the WMO Secretariat on behalf of SC-IMT and published as a technical reference. + +For a Global Broker, the service level is not less than 98 per cent of operational time per calendar month. + +For a Global Cache, the service level is not less than 98 per cent of operational time per calendar month. + ==== 2.7.3 Global Broker ===== 2.7.3.1 Technical considerations * As detailed above, there will be at least three Global Brokers to ensure that messages within WIS2 are highly available and delivered globally with low latency. -* A Global Broker subscribes to messages from WIS2 Nodes and other Global Services. The Global Broker should aim to subscribe to all WIS centres. If this is not possible, the Global Broker should inform the WMO Secretariat so that the situation can be documented. +* A Global Broker subscribes to messages and monitoring events from WIS2 Nodes and other Global Services. The Global Broker should aim to subscribe to all WIS centres. If this is not possible, the Global Broker should inform the WMO Secretariat so that the situation can be documented. * Every WIS2 Node or Global Service must have subscriptions from at least two Global Brokers. -* For full global coverage, a Global Broker will subscribe to messages from at least two other Global Brokers. -* When subscribing to messages from WIS2 Nodes and other Global Services, a Global Broker must authenticate using the valid credentials managed by the WIS centre and available at WMO Secretariat. +* For full global coverage, a Global Broker will subscribe to messages and monitoring events from at least two other Global Brokers. +* When subscribing to messages and monitoring events from WIS2 Nodes and other Global Services, a Global Broker must authenticate using the valid credentials managed by the WIS centre and available at WMO Secretariat. * A Global Broker is built around two software components: ** An off the shelf broker implementing both MQTT 3.1.1 and MQTT 5.0 in a highly available setup, typically in a cluster mode. Tools such as EMQX, HiveMQ, VerneMQ, RabbitMQ (in its latest versions) are compliant with these requirements. The open source version of Mosquitto cannot be clustered and therefore should not be used as part of a Global Broker. ** Additional features, including anti-loop detection, notification message format compliance, validation of the published topic, and metrics provision. -* When connected to a local WIS centre broker or a Global Service broker, the metric ``wmo_wis2_gb_connected_flag`` will be equal to 1. When the connection cannot be established or is interrupted the metric ``wmo_wis2_gb_connected_flag`` will be equal to 0. +* When connected to the local WIS centre broker, the ``wmo_wis2_gb_connected_flag`` will be set to 1. When the connection to the local WIS centre broker is not possible, the ``wmo_wis2_gb_connected_flag`` will be set to 0. * When receiving a message from a local WIS centre broker or a Global Service broker, the metric ``wmo_wis2_gb_messages_received_total`` will be increased by 1. * A Global Broker will check if a discovery metadata record exists corresponding to the topic on which a message has been published. If there is no corresponding discovery metadata record, the Global Broker will discard non-compliant messages and will raise an alert. The metric ``wmo_wis2_gb_messages_no_metadata_total`` will be increased by 1. The Global Broker should not request information from a Global Discovery Catalogue for each notification message but should keep a cache of all valid topics for every ``centre-id``. -* A Global Broker will check that the topic on which the message is received is valid. If the topic is invalid, the Global Broker will discard non-compliant messages and will raise an alert. The metric ``wmo_wis2_gb_invalid_topic_total`` will be increased by 1. -* During the pre-operational phase (2024), a Global Broker will not discard the message but instead will send a message on the `monitor` topic hierarchy to inform the originating centre and its GISC. -* A Global Broker will validate notification messages against the standard format (see _Manual on WIS_, Volume II – Appendix E. WIS2 Notification Message), discarding non-compliant messages and raising an alert. The metric ``wmo_wis2_gb_invalid_format_total`` will be increased by 1. -* A Global Broker will republish a message only once. It will record the message identifier (``id``) (as defined in the WIS2 Notification Message) of messages already published and will discard subsequent identical messages (those with the same message ``id``). This is the anti-loop feature of the Global Broker. +* A Global Broker will check that the topic on which the message is received is valid. If the topic is invalid, the Global Broker will discard non-compliant messages and will raise an alert. The metric ``wmo_wis2_gb_messages_invalid_topic_total`` will be increased by 1. +* A Global Broker will validate notification messages against the standard format (see _Manual on WIS_, Volume II – Appendix E. WIS2 Notification Message), discarding non-compliant messages and raising an alert. The metric ``wmo_wis2_gb_messages_invalid_format_total`` will be increased by 1. +* A Global Broker will validate monitoring events against the standard format (see _Manual on WIS_, Volume II – Appendix G. WIS2 Monitoring Events), discarding non-compliant messages and raising an alert. The metric ``wmo_wis2_gb_messages_invalid_format_total`` will be increased by 1. +* A Global Broker will republish a message or monitoring event only once. It will record the message identifier (``id``) (as defined in the WIS2 Notification Message) of messages already published and will discard subsequent identical messages (those with the same message ``id``). This is the anti-loop feature of the Global Broker. * When publishing a message to the local broker, the metric ``wmo_wis2_gb_messages_published_total`` will be increased by 1. +* To improve the service level, a WIS centre may decide to operate two nodes sharing the same ``centre-id``. One will be considered as _primary_ and the other one as _backup_. In this case, when the primary WIS centre broker is reachable (the MQTT connection is up and running), the Global Broker will only process the notification messages from this primary node. When the connection is dropped (the MQTT connection is down) for at least one minute, the Global Broker will start processing the notification messages received from the backup WIS centre broker. When the connection to the primary WIS centre broker is active for at least one minute, the Global Broker will stop processing the messages coming from the backup WIS centre broker and will only process the messages received from the primary WIS centre broker. When the Global Broker is processing the messages from the backup WIS centre broker the metric ``wmo_wis2_gb_connected_backup_flag`` will be set to 1. When the backup WIS centre broker is defined but is not used as the source of notification message, the metric ``wmo_wis2_gb_connected_backup_flag`` will be set to 0. * A Global Broker will update its metrics at least every 60 seconds. * A Global Broker should not reset its metrics counts. -* All above-defined metrics will be made available on HTTPS endpoints that the Global Monitor will ingest from regularly. +* All above-defined metrics will be made available on HTTPS endpoints that the Global Monitor will ingest regularly. * As a convention, the Global Broker centre-id will be ``{tld}-{centre-name}-global-broker``. * A Global Broker should operate with a fixed IP address so that WIS2 Nodes can permit access to download resources based on IP address filtering. A Global Broker should also operate with a publicly resolvable Domain Name System (DNS) name pointing to that IP address. The WMO Secretariat must be informed of the IP address and/or hostname and any subsequent changes. @@ -131,13 +179,13 @@ In WIS2, Global Caches provide access to WMO core data for data consumers. This * There is no requirement for a Global Cache to provide a browsable interface to the files in its repository in order to allow data consumers to discover what content is available. However, a Global Cache may choose to provide such a capability (for example, implemented as a WAF), along with documentation to inform data consumers of how the capability works. * The default behaviour for a Global Cache is to cache all data published under the ``origin/a/wis2/{centre-id}/data/core`` topic. A data publisher may indicate that data should not be cached by adding the ``"cache": false`` assertion in the WIS2 Notification Message. * A Global Cache should make sure that data are downloaded in parallel and that downloads are not blocking each other. -* A Global Cache may decide not to cache data, for example, if the data are considered too large, or if a WIS2 Node publishes an excessive number of small files. If a Global Cache decides not to cache data, it should behave as though the cache property is set to false and send a message on the ``monitor`` topic hierarchy to inform the originating centre and its GISC. The Global Cache operator should work with the originating WIS2 Node and its GISC to remedy this issue. +* A Global Cache may decide not to cache data, for example, if the data are considered too large, or if a WIS2 Node publishes an excessive number of small files. If a Global Cache decides not to cache data, it should behave as though the cache property is set to false and shall send a WME (WIS2 Monitoring Event) message on the ``monitor`` topic hierarchy to inform other WIS centres about it. The Global Cache operator should work with the originating WIS2 Node and its GISC to remedy this issue. The deduplication mechanism mentioned above applies; only one WME per non-cached data instance shall be generated, regardless of the number of received notification messages. * A Global Cache should operate with a fixed IP address so that WIS2 Nodes can permit access to download resources based on IP address filtering. A Global Cache should also operate with a publicly resolvable DNS name pointing to that IP address. The WMO Secretariat must be informed of the IP address and/or hostname, and any subsequent changes. -* A Global Cache will emit the following metrics for each WIS2 component (e.g., WIS2 Node or other Global Cache, identified with its ``centre-id``) from which it attempts to download and cache resources (e.g., data objects or metadata records): -** ``wmo_wis2_gc_downloaded_total``: This metric is incremented by 1 each time the Global Cache successfully downloads and validates a new resource or an update to a resource published more recently than any previously received messages for that resource. This metric is not incremented for duplicate resources or out-of-sequence updates. -** ``wmo_wis2_gc_downloaded_errors_total``: This metric is incremented by 1 each time the Global Cache fails to download a resource that it is attempting to cache. +* A Global Cache will emit the following metrics for each WIS2 component (e.g., WIS2 Node or other Global Cache, identified with its ``centre-id``) from which it attempts to download and cache resources (e.g., data objects or metadata records): +** ``wmo_wis2_gc_downloaded_total``: This metric is incremented by 1 each time the Global Cache successfully downloads and validates a new resource or an update to a resource published more recently than any previously received messages for that resource. This metric is not incremented for duplicate resources or out-of-sequence updates. +** ``wmo_wis2_gc_downloaded_errors_total``: This metric is incremented by 1 each time the Global Cache fails to download a resource that it is attempting to cache. ** ``wmo_wis2_gc_last_download_timestamp_seconds``: This metric is set to the current time (timestamp - the number of seconds since Unix epoch) each time the Global Cache successfully downloads a new or updated resource. -** ``wmo_wis2_gc_integrity_failed_total``: This metric is incremented by 1 each time the Global Caches fails to validate the integrity of a new or updated resource that it is attempting to cache. +** ``wmo_wis2_gc_integrity_failed_total``: This metric is incremented by 1 each time the Global Caches fails to validate the integrity of a new or updated resource that it is attempting to cache. * A Global Cache will update its metrics at least every 60 seconds. * A Global Cache should not reset its metrics counts. * As a convention, the Global Cache centre-id will be ``{tld}-{centre-name}-global-cache``. @@ -164,6 +212,8 @@ In WIS2, Global Caches provide access to WMO core data for data consumers. This * A Global Discovery Catalogue should connect to and subscribe to more than one Global Broker to ensure that no messages are lost in the event of a Global Broker failure. A Global Discovery Catalogue will discard duplicate messages as needed. * A Global Discovery Catalogue will verify that a discovery metadata record identifier’s centre-id token (see Manual on WIS, Volume II – Appendix F. WMO Core Metadata Profile (Version 2)) matches the centre-id level of the topic from which it was published (see Manual on WIS, Volume II – Appendix D. WIS2 Topic Hierarchy) to ensure that discovery metadata are published by the authoritative organization. * A Global Discovery Catalogue will validate discovery metadata records against the WCMP2. Valid WCMP2 records will be ingested into the catalogue. Invalid or malformed records will be discarded and reported to the Global Monitor against the centre-id associated with the discovery metadata record. +* A Global Discovery Catalogue may perform quality assessment of discovery metadata against the WCMP2 Key Performance Indicators (KPIs) footnote:[https://wmo-im.github.io/wcmp2/kpi/wcmp2-kpi-STABLE.html], providing the following metrics as defined in the WIS2 Metric Hierarchy footnote:[https://github.com/wmo-im/wis2-metric-hierarchy/blob/main/metrics/gdc.csv]: ``wmo_wis2_gdc_kpi_percentage_total``, ``wmo_wis2_gdc_kpi_percentage_average``, ``wmo_wis2_gdc_kpi_percentage_over80_total``. +* A single Global Discovery Catalogue perfoming quality assessment is sufficient for WIS2. * A Global Discovery Catalogue will only update discovery metadata records to replace links for dataset subscription and notification (origin), with their equivalent links for subscription at Global Brokers (cache). * A Global Discovery Catalogue will periodically assess discovery metadata provided by NCs and DCPCs against a set of key performance indicators (KPIs) in support of continuous improvement. Suggestions for improvement will be reported to the Global Monitor against the centre identifier associated with the discovery metadata record. * A Global Discovery Catalogue will remove discovery metadata that are marked for deletion as specified in the data notification message. diff --git a/guide/sections/part2/index.adoc b/guide/sections/part2/index.adoc index 7ef71906..0abb5999 100644 --- a/guide/sections/part2/index.adoc +++ b/guide/sections/part2/index.adoc @@ -7,3 +7,5 @@ include::wis2node.adoc[] include::global-services.adoc[] include::operations.adoc[] + +include::monitoring.adoc[] diff --git a/guide/sections/part2/monitoring.adoc b/guide/sections/part2/monitoring.adoc new file mode 100644 index 00000000..df681c1f --- /dev/null +++ b/guide/sections/part2/monitoring.adoc @@ -0,0 +1,10 @@ +=== 2.9 Monitoring for Global Service providers + +WIS2 is comprised of Global Services providing highly available services for discovery, subscription, notification and download, based on the publication of data by WIS2 Nodes. Global Services need to be able to provide notifications of events WIS2 (examples include, but are not limited to: validation reports on notification messages, discovery metadata, notices of outages, maintenance). + +Using the WIS2 Monitoring Events (WME) standard, Global Services can publish events using MQTT as the publish-subscribe protocol as follows: + +* dedicated topic structure (``monitor/a/wis2/``), where ```` is the centre identifier of the subject that the event notification is pertaining to +* WIS2 Monitoring Event Message (WMEM) leveraging the CloudEvents specification + +For more information, see _Manual on WIS_, Volume II – APPENDIX G. WIS2 MONITORING EVENTS. diff --git a/guide/sections/part2/operations.adoc b/guide/sections/part2/operations.adoc index 00890bd1..8fbc49bd 100644 --- a/guide/sections/part2/operations.adoc +++ b/guide/sections/part2/operations.adoc @@ -277,7 +277,7 @@ harvest and merge these catalogues, creating a global map of the ocean data. IODE harvests all metadata shared by ODIS partners, combines these metadata and creates a knowledge graph, and processes these metadata to export derivative products (for example, diagnostic reports and cloud-optimized data products). -The Ocean InfoHub (OIH) system is IODE's Reference Implementation of a +The Ocean InfoHub (OIH) system is IODE's Reference Implementation of a discovery system leveraging ODIS. ODIS architecture and tools are free and open-source software (FOSS), with regular releases published for the community. diff --git a/guide/sections/part2/wis2-architecture.adoc b/guide/sections/part2/wis2-architecture.adoc index 15621ada..977c58aa 100644 --- a/guide/sections/part2/wis2-architecture.adoc +++ b/guide/sections/part2/wis2-architecture.adoc @@ -38,10 +38,10 @@ These roles are outlined below. ==== 2.2.3 Global Service operator * To ensure highly available global data exchange, a WIS centre may operate one or more Global Services: - i) The Global Discovery Catalogue enables users to search all datasets provided by data publishers and discover where and how to interact with those datasets (for example, to subscribe to updates, to access/download/visualize data, to access more detailed information about the dataset); - ii) A Global Broker provides highly available messaging services through which users may subscribe to notifications about all datasets provided by data publishers; - iii) A Global Cache provides a highly available download service for cached copies of core data downloaded from data publishers’ web services; - iv) A Global Monitor gathers and displays system performance, data availability, and other metrics from all WIS2 Nodes and Global Services. + i) The Global Discovery Catalogue enables users to search all datasets provided by data publishers and discover where and how to interact with those datasets (for example, to subscribe to updates, to access/download/visualize data, to access more detailed information about the dataset); + ii) A Global Broker provides highly available messaging services through which users may subscribe to notifications about all datasets provided by data publishers; + iii) A Global Cache provides a highly available download service for cached copies of core data downloaded from data publishers’ web services; + iv) A Global Monitor gathers and displays system performance, data availability, and other metrics from all WIS2 Nodes and Global Services. ==== 2.2.4 Data consumer * This role represents anyone wanting to find, access, and use data from WIS2. Examples include NMHSs, government agencies, research institutions, private sector organizations, and so forth. @@ -70,6 +70,10 @@ Leveraging existing open standards, WIS2 defines the following specifications in |Dataset metadata, dataset granules |Global Broker, WIS2 Nodes +|WIS2 Monitoring Events (WME) +|Monitoring event metadata, datasets, dataset granules +|All Global Services, WIS2 Nodes + |=== Refer to the _Manual on WIS_, Volume II for details. @@ -88,12 +92,12 @@ Refer to the _Manual on WIS_, Volume II for details. ==== 2.4.2 Global Broker * WIS2 incorporates several Global Brokers, ensuring the highly resilient distribution of notification messages across the globe. -* A Global Broker subscribes to the Message Broker operated by each WIS2 Node and republishes notification messages. -* A Global Broker subscribes to notifications from other Global Brokers to ensure that it receives a copy of all notification messages. -* A Global Broker republishes notification messages from every WIS2 Node and Global Service. +* A Global Broker subscribes to the Message Broker operated by each WIS2 Node and republishes notification and monitoring messages. +* A Global Broker subscribes to notifications and monitoring events from other Global Brokers to ensure that it receives a copy of all notification messages. +* A Global Broker republishes notification messages and monitoring events from every WIS2 Node and Global Service. * A Global Broker operates a highly available, high-performance Message Broker. * A Global Broker uses the WIS2 Topic Hierarchy, enabling data consumers to easily find topics relevant to their needs. -* Data consumers should subscribe to notifications from a Global Broker, not directly from the Message Brokers operated by WIS2 Nodes. +* Data consumers should subscribe to notifications and monitoring events from a Global Broker, not directly from the Message Brokers operated by WIS2 Nodes. ==== 2.4.3 Global Cache * WIS2 incorporates several Global Caches, ensuring the highly resilient distribution of data across the globe. diff --git a/guide/sections/part2/wis2node.adoc b/guide/sections/part2/wis2node.adoc index 8dc5c448..d8de2d07 100644 --- a/guide/sections/part2/wis2node.adoc +++ b/guide/sections/part2/wis2node.adoc @@ -76,6 +76,14 @@ During registration, the WMO Secretariat will provide host names and IP addresse Access controls may be implemented for recommended data. Only the security schemes for authentication and authorization specified in OpenAPI should be used.footnote:[See OpenAPI Specification - Security Scheme Object: https://spec.openapis.org/oas/v3.1.0#security-scheme-object.] +===== 2.6.1.4 Improving Level of Service of a WIS Centre + +It is possible to improve the level of service of a WIS centre by running a primary and a backup node. When the MQTT endpoint provided by the primary node is reachable, the Global Broker will subscribe to the primary node broker. When the MQTT connection is lost, the Global Broker will wait for one minute. If after one minute, the connection to the primary node broker is still down, the connection to the backup node will be activated. When the connection to the primary node is reestablished and stable for one minute, the connection to the backup node will be stopped. + +It must be noted that the switch between the primary and the backup node is *only* controlled by the connectivity to the MQTT endpoint. The permanent absence of messages on the primary node or the failure to download core data from the Global Cache will not trigger the switch to the backup node. + +It is suggested to run the two instances, primary and backup, as independently as possible (different hardware, different location – but same ``centre-id``) and permanently (the backup node should always publish the notification messages). This ensures the maximum possible level of service and limit the need for manual intervention to switch between primary and backup, or vice-versa. + ==== 2.6.2 Performance management ===== 2.6.2.1 Service levels and performance indicators @@ -93,7 +101,11 @@ A WIS2 Node must be able to publish datasets and compliant discovery metadata. T If contacted by a Global Monitor for a performance issue via a GISC, the WIS2 Node should provide metrics to the GISC and the Global Monitor when service is restored to inform them of the resolution of the issue. -==== 2.6.3 WIS2 Node Reference Implementation: WIS2 in a box +==== 2.6.3 Global Service Monitoring considerations + +A WIS2 Node can obtain feedback pertaining to its data and metadata publishing from Global Services using the WIS2 Monitoring Events (WME) specification. By subscribing to the WIS2 Monitoring topic (``monitor/a/wis2/``, where ```` is the centre identifier of the WIS2 Node), a WIS2 Node can receive feedback to trigger corrective action (for example, a metadata publication error reported by a Global Discovery Catalogue). For more information, see <<_2_9_monitoring_for_global_service_providers>>. + +==== 2.6.4 WIS2 Node Reference Implementation: WIS2 in a box When providing a WIS2 Node, Members may use whichever software components they consider most appropriate to comply with the WIS2 technical regulations. diff --git a/guide/sections/part3/index.adoc b/guide/sections/part3/index.adoc index 626dfd3e..0c8f33f8 100644 --- a/guide/sections/part3/index.adoc +++ b/guide/sections/part3/index.adoc @@ -1,3 +1,7 @@ -== PART III. Information management +== PART III. Additional components -include::information-management.adoc[] +include::introduction.adoc[] + +include::other-global-services.adoc[] + +include::sensor-centres.adoc[] diff --git a/guide/sections/part3/introduction.adoc b/guide/sections/part3/introduction.adoc new file mode 100644 index 00000000..92aa6039 --- /dev/null +++ b/guide/sections/part3/introduction.adoc @@ -0,0 +1,44 @@ +=== 3.1 Overview + +==== 3.1.1 Value-added capabilities + +Alongside the core set of Global Services defined, Additional Global Services and Sensor Centres provide value-added capabilities within WIS2. + +Additional Global Services: + +* provide value added capabilities and insights into WIS2 +* are not required components of WIS2 +* may be provided with various access control policies depending on their scope + +Sensor Centres: + +* provide continuous measurement and assessment on WIS2 functions + +==== 3.1.2 Designation procedure for Additional Components + +Any WMO Member or partner organization may request to operate an Additional Component that is defined in this Guide. + +Additional Components provide value to the entire WIS2 ecosystem. As such, all instances of Additional Components must be assessed to assure their correct operation before being designated as a WIS2 component. + +The designation of an Additional Component involves the following steps: + +* Request to host an Additional Component: the "service offer". +** A service offer shall be made by the Permanent Representative of the Member country or territory from where the Additional Component will operate, or by the organisation responsible for WMO Programme or Activity that will operate the Additional Component. +** A service offer shall include: (i) the type of Additional Component - this shall be a type that is defined in this Guide, (ii) an outline the Centre’s capacity to deliver the proposed component, and (iii) contact details for the focal point responsible for the Additional Component. +* Complete the WIS2 Register. +** Once the Additional Component is ready for assessment, the focal point shall update the WIS2 Register providing the information requested by the Secretariat, including but not limited to: the proposed centre identifier ("centre-id"), and technical details for the Additional Component required to connect with other WIS2 components. +* Provide details of WIS2 components. +** The Secretariat provides connection details (such as IP addresses or hostnames) for the WIS2 services or components to which Additional Component needs to connect. +* Additional Component assessment. +** In coordination with the WMO Secretariat and SC-IMT, the principal GISC of the Member country or territory from where the Additional Component will operate will assess technical compliance with specifications of WIS2 (e.g., provision of metrics, provision of WIS2 Monitor Event messages) and the specification of the Additional Component defined in this Guide. +** Following successful technical compliance validation, the principal GISC undertakes a pre-operational assessment of the Additional Component for a period not less than one month. The GISC, in coordination with WMO Secretariat and SC-IMT, will facilitate connection with the Global Services if required. This assessment includes verifying interoperability with Global Services and validating performance against any KPIs defined for the Additional Component. During the pre-operational testing the Additional Component and any outputs it provides will be marked as experimental. +* Approve the designation of the Additional Component. +** Based on the findings of the technical validation and operational assessment, SC-IMT will consider the service offer and provide a recommendation to President of INFCOM for approval. In cases where the proposal is rejected, SC-IMT will provide constructive feedback to the requestor. +* Add new Additional Component to WIS2 (if approved). +** The Additional Component is added to the list of designated Additional Components maintained and published by the WMO Secretariat. +** Experimental marking is removed. + + +==== 3.1.3 Decommissioning procedure for Additional Components + +When a Member seeks to stop operating an Additional Component they must contact the Secretariat to inform WMO of their wish to decommission the service. Members should provide sufficient lead-time between submitting the request to decommission and cessation of the service. Where the service contributes to the effective functioning of WIS2, the Member seeking to stop the service should attempt to find another Member to whom they can transfer their service obligations. On the date agreed for cessation of the service, the Secretariat will: (i) remove the Additional Component from the list designated services, and (ii) coordinate with the Global Service providers to disconnect the Additional Component from WIS2. diff --git a/guide/sections/part3/other-global-services.adoc b/guide/sections/part3/other-global-services.adoc new file mode 100644 index 00000000..9ad560d4 --- /dev/null +++ b/guide/sections/part3/other-global-services.adoc @@ -0,0 +1,69 @@ +=== 3.2 Other Global Services + +A Member wishing to operate a Service defined in this Guide must submit a service offer as per the designation process described in 3.1.2 Designation procedure for Additional Components. + +==== 3.2.1 Global Replay + +A Global Replay provides access and subscription to past WIS Notification Messages. + +===== 3.2.1.1 Component definition + +* WIS2 may include a Global Replay. +* The Global Replay provides the Global Cache, Global Discovery Catalogue and data consumers with a mechanism to search, query and subscribe to past notification messages and monitoring events of interest as published by the Global Broker. +* A Global Replay subscribes to all notification messages and monitoring events as published by Global Broker. It stores the notification message and monitoring events. +* A Global Replay shall store copies of notification messages and monitoring events for a duration compatible with the real-time or near real-time schedule of the data and not less than: +** 1 day for notification messages. +** 7 days for monitoring events. +* A Global Replay will discard notification messages and monitoring events once the retention period has expired. + +===== 3.2.1.2 Technical considerations + +* The Global Replay provides Global Services and data consumers with a mechanism to search, query and subscribe to notification messages and monitoring events of interest. +* The Global Replay implements the OGC API – Features – Part 1: Core standardfootnote:[https://docs.ogc.org/is/17-069r4/17-069r4.html], adhering to the following conformance classes and their dependencies: +** Core +** GeoJSON +* The Global Replay implements the OGC API – Processes - Part 1: Core standardfootnote:[https://docs.ogc.org/is/18-062r2/18-062r2.html], adhering to the following conformance classes and their dependencies: +** Core +** OGC Process Description +** JSON +* A Global Replay shall subscribe to all Global Broker instances, to the topics `+origin/a/wis2/#+`, `+cache/a/wis2/#+` and `+monitor/a/wis2/#+`. +* A Global Replay shall use the notification message and monitoring event identifier to deduplicate notifications and events. +* The Global Replay will make notification messages available via the API collection identifier `wis2-notification-messages`. +* The Global Replay will make monitoring events available via the API collection identifier `wis2-monitoring-events`. +* The Global Replay will provide an API process to trigger the republication of previously sent messages, allowing users to subscribe to messages with various criteria (spatial and / or temporal extent, topic selection) via the Global Broker. +** the API process request accepts the following inputs: +*** `datetime`: a temporal range of messages to republish (example ``2025-04-06T17:21:26Z/2025-04-06T19:21:26Z``). +*** `subscriber-id`: a unique identifier (UUID) defined by the user/client. This identifier may be reused. +*** `topic`: the topic on which to trigger message replication (centre-id required, allowing `+` and `#` for subsequent topic levels) +** the resulting topic subscription in the API process response will be `+replay/a/wis2///+`, where: +*** ``topic`` is the full topic of the message as previously published to WIS2. +** the API process will set temporal boundary limits to 15 minutes. When exceeded, it will result in an error message prohibiting the republishing of messages. +** an example topic would be `+replay/a/wis2/ca-eccc-msc-global-replay/fdb6dae4-95fe-465a-80fb-498ebd61dce8/cache/a/wis2/br-inmet/data/core/weather/surface-based-observations/synop+` +* A single Global Replay instance is sufficient for WIS2. +* Multiple Global Replay instances may be deployed for resilience. +* Global Replay instances operate independently of each other; each Global Replay instance will hold notification messages and monitoring events according to the required retention period. Global Replays do not need to synchronize with each other. +* A Global Replay is populated with notification messages and monitoring events received via a Global Broker instance. +* A Global Replay is connected to at least two (2) Global Brokers. +* A Global Replay instance will discard duplicate messages as needed. +* A Global Replay can validate notification messages against the standard format (see _Manual on WIS_, Volume II – Appendix E. WIS2 Notification Message). Valid messages will be stored. Invalid or malformed messages will be discarded. +* A Global Replay can validate monitoring events against the WIS Monitoring Events Message Encoding (see _Manual on WIS_, Volume II – Appendix G. WIS2 Monitoring Events Message Encoding). Valid WIS Monitoring Events Message Encoding Messages will be stored. Invalid or malformed monitoring events will be discarded. +* A Global Replay will remove notification messages after the required retention period. +* A Global Replay will remove monitoring events after the required retention period. +* A Global Replay may choose to implement access control for controlling usage. +* As a convention Global Replay centre-id will be ``tld-{centre-name}-global-replay``. + +===== 3.2.1.3 Global Replay Reference Implementation: wis2-grep + +To provide a Global Replay, members may use whichever software components they consider most appropriate to comply with WIS2 Technical Regulations. + +To assist Members participation in WIS2, a free and open-source Global Replay Reference Implementation is made available for download and use. wis2-grep builds on mature and robust free and open-source software components that are widely adopted for operational use. + +wis2-grep provides functionality required for the Global Replay, providing the following technical functions: + +* notification messages and monitoring event subscription from a Global Broker instance +* notification message and monitoring event ingest and API publication +* user defined subscription for MQTT message replay from a Global Broker instance +* OGC API - Features - Part 1: Core compliance +* OGC API - Processes - Part 1: Core compliance + +wis2-grep is managed as a free and open source project. Source code, issue tracking and discussions are hosted in the open on GitHub: https://github.com/wmo-im/wis2-grep. diff --git a/guide/sections/part3/sensor-centres.adoc b/guide/sections/part3/sensor-centres.adoc new file mode 100644 index 00000000..d73339c1 --- /dev/null +++ b/guide/sections/part3/sensor-centres.adoc @@ -0,0 +1,162 @@ +=== 3.3 Sensor Centres + +Sensor Centres are components external to WIS 2.0, which provide continuous assessments on functionality. Sensor +Centres can provide external feedback on WIS 2.0 functions to trigger corrective action in support of continuous improvement. + +==== 3.3.1 Overview + +As described in the Manual on WIS, Volume II, WIS 2.0 is a collective, distributed data exchange system that requires monitoring. + +All WIS 2.0 components have defined metrics and conditions on those metrics to trigger alerts. + +The Global Monitor collects the metrics and presents them a suite of dashboards enabling observability, visuaization and analysis. + +Beyond WIS 2.0 itself, a Sensor Centre relies on WIS 2.0 Global Services and Nodes. It implements further processing on any component of the WIS 2.0 architecture. This processing results in additional metrics which can also be provided to dashboards, within or external to WIS 2.0. + +"Sensor Centre" is a generic term to describe additional components to measure various aspects including (but not limited to): + +* the quality and performance of Global Services +* the conformance of the products exchanged with the agreed WMO Regulations (e.g., RBON) + +Each type of Sensor Centre will result in a set of metrics that will be used to measure and evaluate WIS 2.0 Global Services and Nodes. + +A Sensor Centre may also provide logs, in addition to the defined metrics, through download links announced using the WIS2 Monitoring Events standard in support of further analysis and corrective action. + +Sensor Centres monitoring the Global Services that are critical to the function of WIS 2.0 are described below: Sensor Centre Global Broker, Sensor Centre Global Cache, and Sensor Centre Global Discovery Catalogue. Multiple instances of each of these Sensor Centre types are deployed globally to provide a Regional perspective on Global Service performance. + +Given that Sensors Centres may be used to monitor any function in WIS 2.0, additional Sensor Centre types may be used to meet other requirements, such as validation of GBON or RBON compliance. + +Any Member may choose to design and operate a Sensor Centre to meet the needs they identify. They may also choose to propose a new recipe for the WIS2 Cookbook that describes what their Sensor Centre monitors and how it functions. Where a Sensor Centre has an official role unrelated to WIS2 (e.g., in support of RBON compliance) the responsible body for that function may assert their own governance requirements regarding their designation and operation. Sensor Centres supporting the formal monitoring of WIS2 - including contributing to the objective evaluation of Global Service performance - are governed by INFCOM. SC-IMT welcomes proposals for new Sensor Centre types that contribute to the formal monitoring of WIS2. Members are invited to contact the Secretariat with details of their Sensor Centre and implementation evidence. + +A Member wishing to operate a Sensor Centre defined in this Guide must submit a service offer as per the designation process described in 3.1.2 Designation procedure for Additional Components, and undergo assessment to assure their correct operation. + +==== 3.3.2 Sensor Centres for Global Brokers (SCGB) + +The aim of the SCGB is to compare the behaviour of Global Broker(s):: + +Through direct subscription to WIS2 Nodes or via other Global Brokers, each Global Broker aims at providing all messages available on WIS 2.0 to users subscribing to their MQTT(S) endpoint. + +Therefore, the primary function of a SCGB is to test and evaluate that all GBs in WIS 2.0 are making available the same number of messages. In addition, it will provide a connectivity flag to detect the status of the connection toward each GB, as well as a timestamp of the last message being received from each GB. + +As a convention, the Sensor Centre for Global Broker centre-id will be ``{tld}-{centre-name}-sensor-centre-global-broker``. + +For such a centre operated by Direction de la Météorologie of Congo, the centre-id would be cg-dirmet-sensor-centre-global-cache + +===== 3.3.2.1 Technical considerations + +* The SCGB will connect to all Global Brokers. +* The SCGB subscribes to `+origin/a/wis2/#+`, `+cache/a/wis2/#+`, `+monitor/a/wis2/#+` on all GBs +* When the SCGB is successfully connected to the GB, the value of ``wmo_wis2_scgb_connected_flag`` will be set to ``1``. When the SCGB cannot connect to the GB, the value of ``wmo_wis2_scgb_connected_flag`` will be set to ``0``. +* The SCGB validates Global Broker performance using a randomised sample of approximately 10 per cent of all messages. The randomised sample is selected using bytes (or hex digits) from the the message identifier (`message-id`) UUID. Sampling at 10 per cent keeps the processing workload of an SCGB instance at a manageable level. +* Messages with the same `message-id` should be received from all Global Brokers. A message from a Global Broker is considered "received" if it is the first time that message has been seen, or it arrives within 60-seconds of the when that message was first received from another Global Broker. +* When a message from the randomized sample is received from a Global Broker, the SCGB will compare the arrival time of the message with those from other Global Brokers. If the message arrived within the 60-second cutoff time, the SCGB will increment the metric ``wmo_wis2_scgb_messages_received_total`` by ``1`` and will set the metric ``wmo_wis2_scgb_last_message_timestamp_seconds`` to the current timestamp in seconds. Otherwise, the message is considered to be "missed" by the Global Broker. +* The SCGB should retain the identifiers of received messages for a minimum of two hours, so that previously seen messages are not reprocessed. + +All the metrics must be exposed for scraping by the Global Monitor. + +==== 3.3.3 Sensor Centres for Global Caches (SCGC) + +The aim of the SCGC is to compare the behaviour of Global Cache(s):: + +In WIS 2.0 each Global Cache is independent of the other Global Caches. According to the specification of WIS 2.0, Global Cache (see https://wmo-im.github.io/wis2-guide/guide/wis2-guide-APPROVED.html#_2_7_4_1_technical_considerations ) : + +`Global Caches will operate independently of one another. Each Global Cache will hold a full copy of the cache – although there may be small differences between the various Global Caches as data availability notification messages propagate through WIS to each one. There is no formal synchronization between Global Caches.` + +it is therefore interesting to verify, for example: + +. what is the average delay to cache the data made available by WIS2 Node +. are all files published as _core data_ (and with `cache: true` in the Notification Message) by WIS2 Nodes are effectively available in all Global Caches +. are files missed by some Global Cache or more critically by all Global Caches + +Such metrics would provide useful information to identify problems, to help Global Caches to fix them, to define KPI that could be used to objectively measure the effective performance of the Cache. + +It could also be used to detect anomalies from the WIS2 node, such as the reuse too frequently of the same `data_id`. + +As a convention, the Sensor Centre for Global Cache centre-id will be ``{tld}-{centre-name}-sensor-centre-global-cache``. + +For such a centre operated by Météo-France the centre-id would be fr-meteofrance-sensor-centre-global-cache + +===== 3.3.3.1 Technical considerations + +A list of metrics has been defined for this Sensor Centre at https://github.com/wmo-im/wis2-metric-hierarchy/blob/main/metrics/scgc.csv. + +The processing of this Sensor Centre is as follows: + +* Subscribe to a given `origin/a/wis2/...` and to the equivalent on the cache topic (`cache/a/wis2/...`). The last subscription level could be `#` or a particular centre-id or any subtopic. + +For the messages received on the `origin` topic subscription: + +* Discard the Notification Message if data policy is `recommended` or `cache` is set to false: these data are not cached +* Detect any duplicates `data_id` not including a `rel: update` within a period of at least 3 hours +* Increase by 1 `wmo_wis2_scgc_messages_published_total` metric +* Store the time where the message as been received +* (optional) Store the full Notification Message - this can be useful to analyse systematic issues + +For the messages received on the `cache` topic subscription and using the `properties.global-cache` value to select the Global Cache: + +* Check if the link with `rel="canonical"` or `rel="update"` in `links` is reachable by doing a HTTP HEAD request (The HTTP HEAD must provide a response status code equal to `200`). If this is not the case, discard the Notification Message. Otherwise: + ** For each Notification Message and using `data_id`, calculate the difference between the time was received on `origin` and the same `data_id` on `cache`: Time~Cache~ - Time~Origin~ + ** Update the `wmo_wis2_scgc_cache_delay_seconds` metric with this value + ** Compare this value with the three thresholds defined in the metric table above. Increase by 1 `wmo_wis2_scgc_messages_cached_delay_total` using the threshold as a label (so less than 120s, less than 300s, less than 600s) + ** If the message was received within the 600-second time limit, increase by 1 `wmo_wis2_scgc_messages_cached_total` +* If no Notification Message for the `data_id` is received after the highest threshold (here 600s) or if the link provided is not valid, increase by 1 `wmo_wis2_scgc_messages_missed_total` + +The `rel` metric label is used to distinguish between messages relating to "canonical" and "updated" data items. This allows Global Cache performance to be evaluated only using "canonical" data items: Given that Global Caches are designed to discard out-of-sequence updates, counting such discards as "misses" would not be a fair reflection of performance. + +As a Global Cache can decide not to cache a data granule, and in order to assess if a Global Cache has decided to do so, the Sensor Centre will compare the ``links`` array published on `++origin/a/wis/#++` and in the Notification Message received on `++cache/a/wis2/#++` if the two ``links`` are identical, then the Global Cache has decided not to cache the data. In this case, increase by 1 ``wmo_wis2_scgc_messages_cache_override_total``. + +If no Global Cache has cached the data, increase by 1 `wmo_wis2_scgc_messages_missed_all_total` + +In order to assess the downloading feature of a Global Cache, and to provide the required information to determine the KPI on downloads, the Sensor Centre will: + +* Select randomly every minute a sample of at least 5 data granules successfully cached by all Global Caches +* Increment ``wmo_wis2_scgc_download_selected_total`` by the number of selected data granules +* Attempt a download for that same granule against all Global Caches +* Evaluate the checksum for the data granule if provided in the Notification Message +* Increment ``wmo_wis2_scgc_download_successful_total`` by 1 if the download is successful and the (optional) checksum validation passes + +All the metrics SHALL be exposed for scraping by the Global Monitor. + +In addition to the metrics, the SCGC will also make available an hourly logfile containing at least: + +* the centre-id +* the full topic where the WIS2 Notification Message has been originally published +* the full WIS2 Notification Message as published by the origin centre-id +* for each Global Cache a clear indication when the message pertaining to this data-id has been published on the cache topic (the timestamp), and if the data-id was not republished by a particular Global Cache `9999999999999` will be used instead of the timestamp +* each entry in the log file will be a valid JSON + +Then, at the end of the hour, the SCGC SHALL publish a notification message on `monitor/a/wis2/{tld}-{centre-name}-sensor-centre-global-cache` providing a URL to download the logfile. The notification message published will be conformant with the WMO WIS2 Monitoring Events specifications. The monitor message will be of `type` : `int.wmo.wis.wme.event.item.download`. + +==== 3.3.4 Sensor Centres for Global Discovery Catalogues (SCGDC) + +The primary function of an SCGDC is to test and evaluate content parity across all GDCs in WIS 2.0. + +A Global Discovery Catalogue (GDC) must publish, every 24 hours, a zip file containing all metadata records available via the GDC API. + +A GDC Sensor Centre checks all GDCs are publishing the zip file and then analyzes the content of the zip file (all metadata records) to detect anomalies (i.e. missing metadata records, inconsistent data records between GDCs, validating metadata records for WCMP2 compliance). + +Metadata content must be identical for all GDCs. The SCGDC verifies this consistency provides the necessary insights to trigger corrective action of the discrepancies. + +===== 3.3.4.1 Technical considerations + +* The SCGDC will connect to a minimum of two Global Brokers. +* The SCGDC subscribes to the metadata topic for all GDCs, by one of the following methods: +** subscribing to ``cache/a/wis2/+/metadata``, discarding all Notification Messages whose centre identifier (in the topic) does not end with ``-global-discovery-catalogue``. +** subscribing to each GDC via ``cache/a/wis2/centre-id/metadata``. +* The SCGDC will use the link provided in the WIS2 Notification Message received to download the zip file published by the GDC. +* For each GDC centre identifier, upon successful download of the zip file, the SCGDC will update the metric ``wmo_wis2_scgdc_last_download_timestamp_seconds`` with the timestamp (in seconds) of the last successful download. +* Once a day, the SCGDC will analyze the content of the latest zip files received across all GDCs +** Files older than 24 hours will not be analyzed. +* The SCGDC will set the metric ``wmo_wis2_scgdc_records_incorrect_total`` to ``0`` at the beginning of the analysis for all GDCs where the zip file is less than 24 hours old. *Note*: the metric ``wmo_wis2_scgdc_records_incorrect_total`` is a gauge (not a counter) whose value can be reset to ``0`` at the beginning of the analysis. +* After unpacking the zip file, the SCGDC will update the metric ``wmo_wis2_scgdc_records_received_total`` with the number of records found in the zip file. +* For each record, the SCGDC will: +** prune the `links` array. The `links` array is modified by each GDC and modifications are dependent on the GDC itself. It is therefore required to remove all links that are not `rel=license` before further processing for parity checking. +** validate the record against the latest approved version of WCMP2 +*** If the record is not compliant, the SCGDC will update the metric ``wmo_wis2_scgdc_record_invalid_total``, with the WCMP2 id as well as centre id of the GDC +** compare each metadata record with the same WCMP2 `id` received from all other GDCs. +** If the content is the same for all GDCs, the SCGDC will increase the metric ``wmo_wis2_scgdc_records_identical_total`` by ``1``. +** If the content is different, the SCGDC will compare the record's ``properties.updated`` property, if present, across GDCs. If more than one value of ``properties.updated`` is present, the SCGDC will consider the GDC with the record of the most recent date is the canonical record. +*** In this case, the SCGDC will increase the metric ``wmo_wis2_scgdc_records_incorrect_total`` by ``1`` for all GDCs except the GDC having the most recent record. +** If the content is different, and if no GDC holds a record with ``properties.updated``, the SCGDC will consider that it is not possible to ensure which GDC has the correct record. +*** In this case, the SCGDC will increase the metric ``wmo_wis2_scgdc_records_incorrect_total`` by ``1`` for all GDCs. +* As a convention, the Sensor Centre for Global Discovery Catalogue centre-id will be ``{tld}-{centre-name}-sensor-centre-global-discovery-catalogue``. diff --git a/guide/sections/part4/index.adoc b/guide/sections/part4/index.adoc index 1386fa0b..0cd80489 100644 --- a/guide/sections/part4/index.adoc +++ b/guide/sections/part4/index.adoc @@ -1,3 +1,3 @@ -== PART IV. Security +== PART IV. Information management -include::security.adoc[] +include::information-management.adoc[] diff --git a/guide/sections/part3/information-management.adoc b/guide/sections/part4/information-management.adoc similarity index 94% rename from guide/sections/part3/information-management.adoc rename to guide/sections/part4/information-management.adoc index 1f2f6e25..94b1f3e7 100644 --- a/guide/sections/part3/information-management.adoc +++ b/guide/sections/part4/information-management.adoc @@ -1,16 +1,16 @@ -=== 3.1 Introduction +=== 4.1 Introduction -==== 3.1.1 Background +==== 4.1.1 Background The efficient and effective provision of services relying on meteorological, climatological, hydrological and oceanographic information depends on a reliable information infrastructure. This infrastructure should be guided by community best practices and standards, including recommendations and requirements on sourcing, securing, managing, archiving, exchanging and providing easy access to information. This part of the Guide aims to provide high-level guidance on these activities, which can be grouped under the term “information management”. This is done by identifying and describing the fundamental principles of good information management and by highlighting the different stages of the information management life cycle. _Note: The term "information" is used in a general sense and includes data and products._ -==== 3.1.2 Scope +==== 4.1.2 Scope High-level guidance on information management practices that apply in the context of information related to the Earth system is provided in this part of the Guide. Detailed technical information, such as the specification of data formats or quality control and assurance methods, is provided in other parts of the Guide and in other WMO publications. These are referenced where applicable. -The principles of information management are described below. Section <<_3_3_the_information_management_life_cycle>> describes five focus areas. +The principles of information management are described below. Section <<_4_3_the_information_management_life_cycle>> describes five focus areas. 1. Planning, information creation and acquisition. The creation of information using internal and external data sources and the acquisition of information from various sources. 2. Representation and metadata. The use of standards to represent metadata, data and information; this is of primary importance to enable the interoperability and long-term usability of the information. @@ -18,7 +18,7 @@ The principles of information management are described below. Section <<_3_3_the 4. Usage and communication. The publication of guidance material on the use of published information, including on the limitations and suitability of the information and any licensing terms. 5. Storage, archival and disposal. The policies and procedures for business continuity and disaster recovery, as well as retention and disposal. -===== 3.1.3 Intended audience +===== 4.1.3 Intended audience This guidance is primarily aimed at personnel within WMO centres, who are responsible for planning and undertaking the creation or acquisition, stewardship, exchange and provision of information related to the Earth system. @@ -30,27 +30,27 @@ Specifically, the guidance has five main target audiences across the information 4. Service providers (those who disseminate the information) - they are responsible for ensuring information availability and maintaining capability for easy and secure access to the information; 5. Information consumers (those who utilize the information) - they need to understand the restrictions, rights, responsibilities and limitations associated with the information together with the suitability for the intended usage or purpose. -==== 3.2 Principles of information management +==== 4.2 Principles of information management The effective management of information is essential for WMO centres to deliver operational services and information that is authoritative, seamless, secure and timely. The principles below underpin and provide a framework for information management across the full information life cycle. These principles are independent of the information type and are largely independent of technology; they are therefore expected to remain stable over time. -===== 3.2.1 Principle 1: Information is a valued asset +===== 4.2.1 Principle 1: Information is a valued asset * An information asset is information that has value. This value may be related to the cost of generating and collecting the information, it may be associated with the immediate use of the information, or it may be associated with the longer -term preservation and subsequent reuse of the information. * This value should be recognizable and quantifiable and the asset should have an identifiable life cycle. Risks associated with, and to, an information asset should also be identified. As such, information management must be considered an integral part of a WMO centre’s responsibilities and needs to be adequately resourced over the full life cycle of the information. -===== 3.2.2 Principle 2: Information must be managed +===== 4.2.2 Principle 2: Information must be managed * An information asset must be managed throughout its life cycle, from creation to use to eventual disposal, in a way that makes it valuable, maximizes its benefits and reflects its value in time and its different uses. * Information managers must consider the entire information life cycle, from identifying needs and business cases to creation, quality assurance, maintenance, reuse, archival, and disposal. Careful consideration must be given to disposal, ensuring that information is destroyed only when it has ceased to be useful for all categories of users. * Professionally qualified and adequately skilled staff with clear roles and responsibilities should apply a sound custodianship framework concerning security, confidentiality and other statutory requirements of different types of information. -===== 3.2.3 Principle 3: Information must be fit for purpose +===== 4.2.3 Principle 3: Information must be fit for purpose * Information should be developed and managed in accordance with its function and use for internal and external users. * WMO centres should regularly assess information to ensure that it is fit for its purpose and that the related processes, procedures and documentation are adequate. * Processes should be consistent with the general provisions and principles of quality management as described in the WMO https://library.wmo.int/idurl/4/35722[_Technical Regulations_] (WMO-No. 49). -===== 3.2.4 Principle 4: Information must be standardized and interoperable +===== 4.2.4 Principle 4: Information must be standardized and interoperable * Information must be stored and exchanged in standardized formats to ensure wide usability in the short and long-term. It is essential for long-term archiving that information be stored in a form that can be understood and used after several decades. * Standardization is essential for structured information such as dataset definitions and metadata to support interoperability. @@ -58,18 +58,18 @@ The effective management of information is essential for WMO centres to deliver * Which standards to use depends on the user community and organizational policies. Interoperability requirements should be considered when selecting the standard for internal use and broader dissemination. * The use of closed and proprietary standards is strongly discouraged. -===== 3.2.5 Principle 5: Information must be well documented +===== 4.2.5 Principle 5: Information must be well documented * WMO centres should comprehensively document information processes, policies, and procedures to facilitate broad and long-term use. * WMO centres should keep documentation up to date to ensure full traceability of processes along the information life cycle, particularly for its creation. * Previous versions of the documentation should be retained, versioned, archived and made readily available for future use. In addition, versions should be assigned a unique and persistent identifier for future unambiguous identification. -===== 3.2.6 Principle 6: Information must be discoverable, accessible and retrievable +===== 4.2.6 Principle 6: Information must be discoverable, accessible and retrievable * Information should be easy to find through the Web, and for this purpose, the publisher should share discovery metadata with a catalogue service. The catalogue service should include a web API to be used by other applications in order to offer user-tailored search portals. * For information to be easily retrievable once discovered, it should be accessible using standard data exchange protocols. -===== 3.2.7 Principle 7: Information should be reusable +===== 4.2.7 Principle 7: Information should be reusable * In order to maximize the economic benefits of an information asset, it should be made as widely available and as accessible as possible. * Resolution 1 (Cg-Ext(2021)) encourages the reuse of data and information through the open and unrestricted exchange of core WMO data. WMO encourages the free and unrestricted exchange of information in all circumstances. @@ -78,7 +78,7 @@ The effective management of information is essential for WMO centres to deliver _Note: Information on the FAIR data principles can be found at: FAIR Principles - GO FAIR_footnote:[https://go-fair.org] -===== 3.2.8 Principle 8: Information management is subject to accountability and governance +===== 4.2.8 Principle 8: Information management is subject to accountability and governance * Information management processes must be governed as the information moves through its life cycle. All information must have a designated owner, steward, curator and custodian. These roles may be invested in the same person but should be clearly defined at the time of creation. A WMO centre with responsibility of managing information must ensure: ** The implementation of general information management practices, procedures and protocols, including well-defined roles, responsibilities and restrictions on managing the information; @@ -86,9 +86,9 @@ _Note: Information on the FAIR data principles can be found at: FAIR Principles ** The establishment of licensing and the definition and enforcement of any access restrictions. ** The designated owner should have budget and decision-making authority with respect to preservation and data usage, including the authority to pass ownership to another entity. -==== 3.3 The information management life cycle +==== 4.3 The information management life cycle -===== 3.3.1 Overview +===== 4.3.1 Overview All information should be subject to a well -defined and documented life cycle. The governance of this process is often referred to as the information management life cycle; it helps organizations manage information from planning, creation and acquisition through usage and exchange to archival and disposal. @@ -102,22 +102,22 @@ The following sections describe two overarching themes, governance and documenta Governance covers the rules that apply to managing information in a secure and transparent manner; documentation covers the act of recording the reasons for, and details of, all operations in the information management process. -===== 3.3.2 Overarching requirements +===== 4.3.2 Overarching requirements -====== 3.3.2.1 Governance +====== 4.3.2.1 Governance * Information management governance defines a set of organizational procedures, policies and processes for the management of information. This includes defining accountabilities and compliance mechanisms. * Effective governance helps ensure that all aspects of the information management process are conducted in a rigorous, standardized and transparent manner and that the information is secure, accessible and usable. * WMO centres should establish a board or leadership group to develop and regularly review such a governance structure and ensure compliance with its requirements. -====== 3.3.2.2 Documentation +====== 4.3.2.2 Documentation * Documentation describing the who, what, why, when, where and how with respect to the various actions that are undertaken in the management of information is required to ensure the traceability and integrity of the information and to ensure operations can continue if key staff leave. * This documentation is required for all aspects of the information life cycle and should be clear, well -communicated, regularly updated and easy to find. Guidance relating to the documentation should be provided to new staff taking on responsibilities for information management and be a key component of training. -===== 3.3.3 Aspects of the information management life cycle +===== 4.3.3 Aspects of the information management life cycle -====== 3.3.3.1 Planning, information creation and acquisition +====== 4.3.3.1 Planning, information creation and acquisition Before the creation or acquisition of new information a business case plan and an information management plan should be developed, covering both the input information sources and any derived information. The plans should include: @@ -136,7 +136,7 @@ Information created rather than acquired should undergo the same processes as ac To ensure traceability and reproducibility, the information and documents at this and subsequent stages, should be version controlled and clearly labelled with version information. Similarly, software or computer code used to generate or process information should be version controlled with the version information recorded in the documentation and metadata. Where possible, software should be maintained within a code repository. -====== 3.3.3.2 Representation and metadata +====== 4.3.3.2 Representation and metadata The formats used to store and exchange information should be standardized to ensure its usability in both the short and the long-term. It is essential that the information be accessible many years after archival if required. To ensure this usability, the format and version of the information should be recorded in the information metadata record and included within the information itself where the format allows. @@ -146,7 +146,7 @@ These formats have been developed within the WMO community to enable the efficie The WMO formats specified in the manuals are subject to strong governance processes, and changes to the formats can be traced through the versions of the manuals. The code tables and controlled vocabularies are also maintained in a code repository. To enable future reuse, the technical information, including detailed format specifications, should be archived alongside information for future access. This includes any controlled vocabulary, such as BUFR tables or WIGOS metadata code lists, associated with the format. -====== 3.3.3.3 Publication and exchange of information +====== 4.3.3.3 Publication and exchange of information To maximize the benefits and return on investment in the acquisition and generation of information, there needs to be a clear method as to how the information will be published, exchanged and accessed by users. @@ -160,7 +160,7 @@ Updates to the information exchanged on WIS, including the publication of new in _Note: The newsletter is available from: https://community.wmo.int/news/operational-newsletter_. -====== 3.3.3.4 Usage and communication +====== 4.3.3.4 Usage and communication For information to have value, it must inform users, aid knowledge discovery and have an impact through informed decision -making. Ensuring that the user can make effective use of the information is an important step in the information management life cycle. This is accomplished in two ways: @@ -169,13 +169,13 @@ For information to have value, it must inform users, aid knowledge discovery and For common types of information, the guides may be generic or link to standard documentation. Information on the observations available from WIGOS is provided in the https://library.wmo.int/idurl/4/55063[_Manual on the WMO Integrated Global Observing System_] (WMO-No. 1160) and the https://library.wmo.int/idurl/4/55696[_Guide to the WMO Integrated Global Observing System_] (WMO-No. 1165). This includes information on the expected uses and quality of the data. Similarly, information on the data and products available through the WMO Integrated Processing and Prediction System is provided in the https://library.wmo.int/idurl/4/35703[_Manual on the WMO Integrated Processing and Prediction System_] (formerly the Manual on the Global Data Processing and Forecasting System) (WMO-No. 485). -For non-standard and specialist products, targeted user guides may be more appropriate. These should be accessible and retrievable via a link within the discovery metadata and should include a plain text summary for the non-technical user. Any user guide should be in addition to the technical documentation described in <<_3_3_3_1_planning_information_creation_and_acquisition>>. +For non-standard and specialist products, targeted user guides may be more appropriate. These should be accessible and retrievable via a link within the discovery metadata and should include a plain text summary for the non-technical user. Any user guide should be in addition to the technical documentation described in <<_4_3_3_1_planning_information_creation_and_acquisition>>. -Updates and the availability of new information should be announced and published via the WMO Operational Newsletter (see <<_3_3_3_3_publication_and_exchange_of_information>>). Other communication methods may also be used, but these should not be in place of the operational newsletter. It is also recommended that users be allowed to subscribe to the newsletter to receive updates directly. +Updates and the availability of new information should be announced and published via the WMO Operational Newsletter (see <<_4_3_3_3_publication_and_exchange_of_information>>). Other communication methods may also be used, but these should not be in place of the operational newsletter. It is also recommended that users be allowed to subscribe to the newsletter to receive updates directly. The discovery metadata should include a valid point of contact, enabling users to provide feedback and ask questions about the information provided. -====== 3.3.3.5 Storage, archival and disposal +====== 4.3.3.5 Storage, archival and disposal The type of storage used should be appropriate to the type of information stored. Core information exchanged operationally should be stored and made available via high-availability and low-latency media and services. For some operation-critical information, such as hazard warnings, there is a requirement for the end-to-end global distribution of the information to be completed in two minutes. For other operational data, there is a requirement for the global exchange to be completed in 15 minutes. @@ -191,12 +191,12 @@ Earth system information, especially observational data, is often irreplaceable. When an information resource is marked for disposal, the reasons for disposal, including the outcome of the consultation with stakeholders and users, must clearly be documented. The disposal must be authorized by the identified owner and custodian of the information. Information relating to the disposal must be included in the metadata associated with the information resource. The metadata must be retained for future reference. -==== 3.4 Other considerations +==== 4.4 Other considerations -===== 3.4.1 Technology and technology migration +===== 4.4.1 Technology and technology migration Information managers must be aware of the need to ensure that the technologies, hardware and software used do not become obsolete, and they must be aware of emerging data issues. This topic is discussed further in the https://library.wmo.int/idurl/4/56904[_WMO Guidelines on Emerging Data Issues_] (WMO-No. 1239). -===== 3.4.2 Information security +===== 4.4.2 Information security Further information on information security and best practices can be found in the https://library.wmo.int/idurl/4/51145[_Guide to Information Technology Security_] (WMO-No. 1115). diff --git a/guide/sections/part5/index.adoc b/guide/sections/part5/index.adoc index ab69cf6c..f1dc23af 100644 --- a/guide/sections/part5/index.adoc +++ b/guide/sections/part5/index.adoc @@ -1,3 +1,3 @@ -== PART V. Competencies +== PART V. Security -include::competencies.adoc[] +include::security.adoc[] diff --git a/guide/sections/part4/security.adoc b/guide/sections/part5/security.adoc similarity index 100% rename from guide/sections/part4/security.adoc rename to guide/sections/part5/security.adoc diff --git a/guide/sections/part5/competencies.adoc b/guide/sections/part6/competencies.adoc similarity index 100% rename from guide/sections/part5/competencies.adoc rename to guide/sections/part6/competencies.adoc diff --git a/guide/sections/part6/index.adoc b/guide/sections/part6/index.adoc new file mode 100644 index 00000000..20b078ab --- /dev/null +++ b/guide/sections/part6/index.adoc @@ -0,0 +1,3 @@ +== PART VI. Competencies + +include::competencies.adoc[]