Skip to content

Commit d5294c7

Browse files
authored
Add docs about localprocessing feature (PR #413)
1 parent 5dd398c commit d5294c7

File tree

2 files changed

+98
-0
lines changed

2 files changed

+98
-0
lines changed
93.9 KB
Loading

docs/cookbook/localprocessing.rst

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,3 +4,101 @@ Client-side (local) processing
44

55
.. warning::
66
This is a new experimental feature and API, subject to change.
7+
8+
Background
9+
----------
10+
11+
The client-side processing functionality allows to test and use openEO with its processes locally, i.e. without any connection to an openEO back-end.
12+
It relies on the projects `openeo-pg-parser-networkx <https://github.com/Open-EO/openeo-pg-parser-networkx>`_, which provides an openEO process graph parsing tool, and `openeo-processes-dask <https://github.com/Open-EO/openeo-processes-dask>`_, which provides an Xarray and Dask implementation of most openEO processes.
13+
14+
Installation
15+
------------
16+
17+
.. note::
18+
This feature requires ``Python>=3.9`` and has been tested
19+
with ``openeo-pg-parser-networkx==2023.3.1`` and
20+
``openeo-processes-dask==2023.3.2``
21+
22+
.. code:: bash
23+
24+
pip install openeo[localprocessing]
25+
26+
Usage
27+
-----
28+
29+
Local Collections
30+
~~~~~~~~~~~~~~~~~
31+
32+
Every openEO process graph relies on data, which was always provided by a cloud infrastructure (the openEO back-end) until now.
33+
The client-side processing adds the possibility to read and use local netCDFs, geoTIFFs and ZARR files for your experiments.
34+
35+
If you want to use our sample data, please clone this repository:
36+
37+
.. code:: bash
38+
39+
git clone https://github.com/Open-EO/openeo-localprocessing-data.git
40+
41+
With some sample data we can now check the STAC metadata for the local files by doing:
42+
43+
.. code:: python
44+
45+
from openeo.local import LocalConnection
46+
local_data_folders = [
47+
"./openeo-localprocessing-data/sample_netcdf",
48+
"./openeo-localprocessing-data/sample_geotiff",
49+
]
50+
local_conn = LocalConnection(local_data_folders)
51+
local_conn.list_collections()
52+
53+
This code will parse the metadata content of each netCDF, geoTIFF or ZARR file in the provided folders and return a JSON object containing the STAC representation of the metadata.
54+
If this code is run in a Jupyter Notebook, the metadata will be rendered nicely.
55+
56+
57+
.. tip::
58+
The code expects local files to have a similar structure to the sample files provided `here <https://github.com/Open-EO/openeo-localprocessing-data.git>`_.
59+
If the code can not handle you special netCDF, you can still modify the function that reads the metadata from it `here <https://github.com/Open-EO/openeo-python-client/blob/90c0505fae47c56746e49c91476be5147be6e1d0/openeo/local/collections.py#L19>`_ and the function that reads the data `here <https://github.com/Open-EO/openeo-python-client/blob/90c0505fae47c56746e49c91476be5147be6e1d0/openeo/local/processing.py#L26>`_.
60+
61+
Local Processing
62+
~~~~~~~~~~~~~~~~
63+
64+
Let's start with the provided sample netCDF of Sentinel-2 data:
65+
66+
.. code-block:: pycon
67+
68+
>>> local_collection = "openeo-localprocessing-data/sample_netcdf/S2_L2A_sample.nc"
69+
>>> s2_datacube = local_conn.load_collection(local_collection)
70+
>>> # Check if the data is loaded correctly
71+
>>> s2_datacube.execute()
72+
<xarray.DataArray (bands: 5, t: 12, y: 705, x: 935)>
73+
dask.array<stack, shape=(5, 12, 705, 935), dtype=float32, chunksize=(1, 12, 705, 935), chunktype=numpy.ndarray>
74+
Coordinates:
75+
* t (t) datetime64[ns] 2022-06-02 2022-06-05 ... 2022-06-27 2022-06-30
76+
* x (x) float64 6.75e+05 6.75e+05 6.75e+05 ... 6.843e+05 6.843e+05
77+
* y (y) float64 5.155e+06 5.155e+06 5.155e+06 ... 5.148e+06 5.148e+06
78+
crs |S1 ...
79+
* bands (bands) object 'B04' 'B03' 'B02' 'B08' 'SCL'
80+
Attributes:
81+
Conventions: CF-1.9
82+
institution: openEO platform - Geotrellis backend: 0.9.5a1
83+
description:
84+
title:
85+
...
86+
87+
As you can see in the previous example, we are using a call to execute() which will execute locally the generated openEO process graph.
88+
In this case, the process graph consist only in a single load_collection, which performs lazy loading of the data. With this first step you can check if the data is being read correctly by openEO.
89+
90+
Looking at the metadata of this netCDF sample, we can see that it contains the bands B04, B03, B02, B08 and SCL.
91+
Additionally, we also see that it is composed by more than one element in time and that it covers the month of June 2022.
92+
93+
We can now do a simple processing for demo purposes, let's compute the median NDVI in time and visualize the result:
94+
95+
.. code:: python
96+
97+
b04 = s2_datacube.band("B04")
98+
b08 = s2_datacube.band("B08")
99+
ndvi = (b08-b04)/(b08+b04)
100+
ndvi_median = ndvi.reduce_dimension(dimension="t",reducer="median")
101+
result_ndvi = ndvi_median.execute()
102+
result_ndvi.plot.imshow(cmap="Greens")
103+
104+
.. image:: ../_static/images/local/local_ndvi.jpg

0 commit comments

Comments
 (0)