Skip to content
BranchesTags

Repository files navigation

Aviso for Destination Earth — Extremes Digital Twin

This repository provides documentation and example scripts for using Aviso to receive data-availability notifications for Destination Earth (DestinE) Digital Twin data, with a focus on the Extremes Digital Twin (Extremes-DT).

Table of Contents

1. What is Aviso?

Aviso is an open-source notification system developed by ECMWF that broadcasts time-critical events across HPC and cloud systems, enabling event-driven workflows that span multiple domains.

In short, Aviso lets you:

  • Subscribe to events you care about (e.g. "a new forecast step is ready").
  • Define triggers that execute when a matching notification arrives — print, log, run a shell command, POST a CloudEvent, or call a Python function.
  • React in near real time rather than polling for new data.

2. Aviso in the Destination Earth Context

Aviso is the mechanism that informs users the moment new forecast data becomes available, so that downstream workflows — post-processing, visualisation, model coupling, alerting — can react immediately.

Aviso is developed and used extensively at ECMWF across internal operational workflows, so much of the upstream documentation is aimed at ECMWF-internal users. The examples and documentation in this repository are tailored specifically to Destination Earth. Generic pyaviso examples from the upstream docs may reference events, endpoints, or authentication methods that do not apply in the DestinE context.

Warning

Although Digital Twin data is produced on multiple EuroHPC systems (Mare Nostrum, LUMI, Leonardo), the Aviso server is currently deployed only on the LUMI Databridge and exposes notifications for the Extremes Digital Twin only. Climate DT notifications are not currently available through Aviso.

See Section 9 for details.

3. Requirements and Access

Note

Listening to Aviso notifications does not require authentication — the current endpoint uses auth_type: none. However, downloading data via Polytope (used in the Earthkit example) does require valid DESP credentials.

To obtain DESP credentials:

  1. Register at the DestinE Platform.
  2. Apply for upgraded access as described in the access policy.

Once upgraded access is granted, run the desp-authentication.py script. It will prompt for your DestinE username and password, then retrieve a long-lived offline token from the Destination Earth Service Platform (DESP) and store it at ~/.polytopeapirc. Re-authentication is only needed when that token expires.

4. Installation

Python ≥ 3.6 is required.

Install all dependencies used in this repository:

pip install -r requirements.txt

For a minimal installation (listener only, no data download or plotting):

pip install pyaviso

5. Service Endpoints and Connectivity

Setting Value
Aviso host aviso.lumi.apps.dte.destination-earth.eu
Port 443 (HTTPS)
Polytope host (data) polytope.lumi.apps.dte.destination-earth.eu

Verify connectivity before running the examples:

curl -v https://aviso.lumi.apps.dte.destination-earth.eu

If you are on a corporate or institutional network, ensure outbound TCP 443 to the Aviso host is permitted by your firewall or proxy.

6. Running the Examples

All scripts can be run directly from the repository root.

Real-time listener (echo trigger):

python3 aviso-extremes-dt.py

Replay from a past publish time, then continue live:

python3 aviso-extremes-dt-from-time.py

End-to-end Earthkit/Polytope workflow:

python3 aviso-extremes-dt-earthkit-example.py

On startup you should see output similar to:

loaded config:
{'auth_type': 'none',
 'configuration_engine': {'host': 'aviso.lumi.apps.dte.destination-earth.eu',
                          'https': True,
                          'port': 443},
 'notification_engine': {'host': 'aviso.lumi.apps.dte.destination-earth.eu',
                         'https': True,
                         'port': 443},
 'remote_schema': True,
 'schema_parser': 'generic'}
Listening to /de/data/ at aviso.lumi.apps.dte.destination-earth.eu:443...

Notifications matching your filter will be printed to stdout as they arrive. Stop with Ctrl + C.

Important

On every startup, Aviso checks the last notification it received and replays any that were missed before switching to real-time listening. On the very first run there is no prior state, so no historical notifications are returned. This catch-up behaviour ensures no notifications are lost across restarts or outages. See Section 8 for full details.

7. Core Concepts

7.1 Listener

A listener is a Python dictionary that tells Aviso what to watch for and what to do when a match is found. It has three required parts:

  1. event — the kind of event to listen for.
  2. request — a filter dictionary; only notifications whose metadata matches all keys are delivered.
  3. triggers — one or more actions to execute per matching notification.
listener = {
    "event": "data", # always use "event": "data" for destine
    "request": { ... },        # filter keys
    "triggers": [{...}, ...],  # one or more triggers
}
listeners_config = {"listeners": [listener]}

7.2 Event

For DestinE Digital Twin data, the event is always:

listener = {
    "event": "data",
    ...
    }

The pyaviso schema also defines mars and dissemination events used in ECMWF operational contexts — these are not relevant for DT data.

7.3 Request Keys (MARS-like)

The aviso request dictionary uses keys from the MARS language. A notification is delivered only when every specified key matches. The fewer keys you include, the broader the subscription.

Do not confuse the aviso request dictionary with a polytope request dictionary. Although they are similar, the aviso request dictionary only accepts the following keys:

Key Typical value Meaning
class "d1" Destination Earth class
expver "0001" Experiment version (operational)
stream "oper", "wave" Data stream
type "fc" Forecast
levtype "sfc", "pl", "sol" Level type (surface, pressure levels, soil)
date "YYYYMMDD" Forecast base date
time "00" Forecast base time (only 00 available)
step 0, [0, 3, 6, ...] Forecast lead time in hours

Each value can be a scalar or a list — for example, "step": [0, 3, 6, 9, 12, 15, 18, 21, 24] matches any of those steps.

Note

Polytope request dictionaries accept additional keys (e.g., "param", "feature") and follow different validation rules since they are processed by separate software. Refer to the Polytope documentation and polytope-examples repository for details on building valid polytope requests.

7.4 Triggers

Triggers define what happens when a notification matches. Multiple triggers per listener are supported; each runs as an independent process.

Type Use case
echo Print to stdout — simplest, good for testing.
log Append notifications to a log file.
command Run a shell command; supports substitutions like ${request.step}.
post Forward as a CloudEvent over HTTP or AWS SNS.
function Call a Python function directly — used in the Earthkit example.

Example trigger definitions:

# Echo (no extra config)
trigger = {"type": "echo"}

# Log to file
trigger = {"type": "log", "path": "aviso.log"}

# Shell command with request field substitution
trigger = {
    "type": "command",
    "command": "./process.sh --date ${request.date} --step ${request.step}",
}

# Python callback
def on_notification(notification): ...
trigger = {"type": "function", "function": on_notification}

7.5 Notification Payload

The dictionary received by a function trigger (and sent by the post trigger as JSON) looks like this:

{
    "event": "data",
    "request": {
        "class":   "d1",
        "expver":  "0001",
        "stream":  "oper",
        "type":    "fc",
        "levtype": "sfc",
        "date":    "20251104",
        "time":    "0000",
        "step":    "6",
    },
}

In a function trigger, the values from notification["request"] are typically used to construct a Polytope or earthkit.data request to retrieve the data.

8. Catch-up and Historical Replay

When nm.listen(...) is called, Aviso by default:

  1. Checks the last notification received (persisted locally per user).
  2. Replays any notifications missed since then.
  3. Switches to real-time listening.

After a reboot or transient outage, no notifications are lost. On the very first run there is no prior state, so no catch-up occurs.

To explicitly set a starting point, use from_date:

nm.listen(
    listeners=listeners_config,
    from_date=datetime(2025, 11, 4), #optional
    to_date=datetime(2025, 11, 30), #optional
    config=config,
)

An optional to_date bounds the replay window; when set, Aviso exits after processing the historical range instead of continuing into real-time.

Important

If using to_date, it should be set to a date in the past (not the current moment). Aviso will exit once all notifications up to that time have been processed.

8.1 from_date and to_date are Publication Times, Not Forecast Base Times

Warning

from_date and to_date refer to the wall-clock time when the notification was published on the Aviso server—i.e., when the data producer announced that a product was available. They are not the forecast base time (date + time in the request).

Example scenario: Data for the forecast initialized at 2025-11-04 00 UTC typically becomes available around 8:00–11:00 UTC that same day. However, due to queuing delays or system issues, the corresponding notifications may be published several hours or even days later.

Correct approach:

  • To replay all notifications for the 2025-11-04 00 UTC cycle, set from_date to a time before the cycle started (e.g., datetime(2025, 11, 3, 12)) and in the Polytope request dictionary filter to target the specific cycle: "date": "20251104", "time": "0000".
  • If you set from_date=datetime(2025, 11, 4, 12), you will only capture notifications published from noon UTC onward, which may miss early steps and include products from prior forecast cycles.
  • When in doubt, set from_date conservatively early and rely on the request filter to select the correct forecasts.

The same principle applies to to_date: it bounds the publication-time window, not the forecast base time window.

9. Scope: Extremes Digital Twin Only

Notifications available through the endpoint used in these examples are limited to the Extremes Digital Twin:

  • class: d1, expver: 0001
  • stream: oper (and wave for the wave component)
  • 4 km global resolution; operational production began on 2023-12-11

Climate DT data is not available through Aviso at this time.

Consult the Extremes DT Data Catalogue to identify available variables, levels, and forecast steps before constructing your request filter.

10. Examples in this Repository

Script Description
aviso-extremes-dt.py Minimal real-time listener with the echo trigger.
aviso-extremes-dt-from-time.py Replay from a given publish time, then continue listening live.
aviso-extremes-dt-earthkit-example.py End-to-end workflow: notification → Polytope download → regrid → Europe map plot.
aviso-extremes-dt-log.py Persist every notification to a structured JSON-lines log file.
aviso-extremes-dt-multi-listener.py Two listeners in one process with different filters (surface vs. wave).
aviso-extremes-dt-replay-window.py Replay a bounded historical window (from_date + to_date) and exit.
aviso-extremes-dt-polytope-download.py Download GRIB data to disk for every matching step (no plotting).
desp-authentication.py Obtain a DESP offline token and store it for Polytope access.

11. Quotas, Throttling, and Best Practices

  • Rate limit: up to 50 requests/second to the Aviso server (may be adjusted under load).
  • Concurrent operations: currently unlimited, but please be considerate.
  • Use the narrowest request filter possible (stream, levtype, step) to avoid triggering on irrelevant notifications.
  • In a function trigger, do not block the main thread with lengthy operations. Hand off work to a queue or worker pool (e.g. concurrent.futures, Celery, or use the post trigger to forward to an external system).
  • Each notification typically corresponds to one forecast step — download volume scales directly with how broad your filter is.
  • For production deployments that should ignore catch-up replay, run aviso listen --now (CLI) to reset the local state and listen only to new notifications.

12. Troubleshooting

Symptom Likely cause / fix
Script prints Listening to /de/data/ … and no notifications arrive No matching data published since you started, or filter is too narrow. Try a from_date in the past.
ConnectionError / TLS errors Outbound port 443 to aviso.lumi.apps.dte.destination-earth.eu is blocked by your network.
Polytope download fails with 401 / 403 Refresh the DESP token by re-running desp-authentication.py.
from_date replay returns fewer events than expected from_date is a publish time, not a forecast base time — see §8.1.
Catch-up replays the same notifications on every restart The local state file is missing or being reset — check the Aviso config directory (default: ~/aviso/).

13. References

About

No description, website, or topics provided.

Resources

Readme
Apache-2.0 license
Activity
Custom properties

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Used by

Contributors

Languages