diff --git a/.buildinfo b/.buildinfo
index ae03e151..6f2ab479 100644
--- a/.buildinfo
+++ b/.buildinfo
@@ -1,4 +1,4 @@
# Sphinx build info version 1
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
-config: 813e97c72189bc3d5936c5d851f0940f
+config: fe4de2f98f2fb236d4feccd422113bbf
tags: 645f666f9bcd5a90fca523b33c5a78b7
diff --git a/_sources/auth/index.rst.txt b/_sources/auth/index.rst.txt
new file mode 100644
index 00000000..bb7d9c6c
--- /dev/null
+++ b/_sources/auth/index.rst.txt
@@ -0,0 +1,319 @@
+.. _auth:
+Authentication
+==============
+
+Some functionality of the freva-rest API and the client library is only
+accessible after successful authentication. This authentication is realised
+with OAuth2 token creation. You can create new access and refresh tokens.
+Refresh tokens can be used to create new access tokens without needing to log
+in via username and password, thereby minimising the risk of exposing login
+credentials. Bear in mind that both login and refresh tokens have a limited
+lifetime.
+
+Generally speaking, you have three options to interact with the authorization
+system:
+
+
+- via the REST API ``/api/auth/v2`` endpoints
+- via the :py:func:`freva_client.authenticate` function
+- via the ``freva-client auth`` command-line interface
+
+Using the restAPI endpoints
+---------------------------
+The API supports token-based authentication using OAuth2. To obtain an access
+token, clients can use the ``/api/auth/v2/token`` endpoint by providing
+valid username and password credentials. The access token should then be
+included in the authorization header for secured endpoints.
+
+.. http:post:: /api/auth/v2/token
+
+ Create an new login token from a username and password.
+ You should either set a username and password or an existing refresh token.
+ You can also set the client_id. Client id's are configured to gain access,
+ specific access for certain users. If you don't set the client_id, the
+ default id will be chosen.
+
+ :form username: The username for the login
+ :type username: str
+ :form password: The password for the login
+ :type password: str
+ :form refresh_token: The refresh token that is used to create a new token
+ the refresh token can be used instead of authorizing
+ via user creentials.
+ :type refresh_token: str
+ :form client_id: The unique identifier for your application used to
+ request an OAuth2 access token from the authentication
+ server, this form parameter is optional.
+ :type client_id: str
+ :form client_secret: An optional client secret used for authentication
+ this parameters is optional an in most cases not neeede
+ :type client_secret: str
+ :statuscode 200: no error
+ :statuscode 401: unauthorized
+ :resheader Content-Type: ``application/json``: access and refresh token.
+
+
+ Example Request
+ ~~~~~~~~~~~~~~~
+
+ .. sourcecode:: http
+
+ POST /api/auth/v2/token HTTP/1.1
+ host: www.freva.dkrz.de
+
+ {
+ "username": "your_username",
+ "password": "your_password"
+ }
+
+ Example Request
+ ~~~~~~~~~~~~~~~
+
+ .. sourcecode:: http
+
+ HTTP/1.1 200 OK
+ Content-Type: application/json
+
+ {
+ "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6.."
+ "token_type": "Bearer",
+ "expires_in": 300,
+ "refresh_token": "eyJhbGciOiJIUzUxMiIsInR5cCIgOiAiSldUIiwia2lkIi.."
+ "refresh_expires_in": 1800,
+ "scope": "email",
+ "not-before-policy": 0
+ }
+
+ Code examples
+ ~~~~~~~~~~~~~
+ Below you can find example usages of this request in different scripting and
+ programming languages
+
+ .. tabs::
+
+ .. code-tab:: bash
+ :caption: Shell
+
+ curl -X POST https://www.freva.dkrz.de/api/auth/v2/token \
+ -d "username=janedoe" \
+ -d "password=janedoe123"
+
+ .. code-tab:: python
+ :caption: Python
+
+ import requests
+ response = requests.post(
+ "https://www.freva.dkrz.de/api/auth/v2/token",
+ data={"username": "janedoe", "password": "janedoe123"}
+ )
+ token_data = response.json()
+
+ .. code-tab:: r
+ :caption: gnuR
+
+ library(httr)
+
+ url <- "https://freva.dkrz.de/api/auth/v2/token"
+ response <- POST(
+ "https://www.freva.dkrz.de/api/auth/v2/token",
+ body = list(username = "janedoe", password = "janedoe123"),
+ encode = "form"
+ )
+ token_data <- content(response, "parsed")
+
+ .. code-tab:: julia
+ :caption: Julia
+
+ using HTTP
+ using JSON
+
+ response = HTTP.POST(
+ "https://www.freva.dkrz.de/api/auth/v2/token",
+ body = Dict("username" => "janedoe", "password" => "janedoe123")
+ )
+ token_data = JSON.parse(String(response.body))
+
+ .. code-tab:: c
+ :caption: C/C++
+
+ #include
+ #include
+
+ int main() {
+ CURL *curl;
+ CURLcode res;
+
+ curl_global_init(CURL_GLOBAL_DEFAULT);
+ curl = curl_easy_init();
+ if(curl) {
+ struct curl_slist *headers = NULL;
+ headers = curl_slist_append(headers, "Content-Type: application/x-www-form-urlencoded");
+
+ curl_easy_setopt(curl, CURLOPT_URL, "https://www.freva.dkrz.de/api/auth/v2/token");
+ curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
+ curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "username=janedoe&password=janedoe123");
+
+ res = curl_easy_perform(curl);
+ curl_easy_cleanup(curl);
+ }
+ curl_global_cleanup();
+ return 0;
+ }
+
+---
+
+
+.. http:get:: /api/auth/v2/status
+
+ Check the status of an access token.
+
+
+
+ :reqheader Authorization: The OAuth2 access token
+ :statuscode 200: no error
+ :statuscode 401: unauthorized
+ :resheader Content-Type: ``application/json``: access and refresh token.
+
+
+ Example Request
+ ~~~~~~~~~~~~~~~
+
+ .. sourcecode:: http
+
+ POST /api/auth/v2/status HTTP/1.1
+ host: www.freva.dkrz.de
+ Authorization: Bearer your_access_token
+
+ Example Request
+ ~~~~~~~~~~~~~~~
+
+ .. sourcecode:: http
+
+ HTTP/1.1 200 OK
+ Content-Type: application/json
+
+ {
+ "sub": "648692af-aaed-4f82-9f74-2d6baf96f5ea",
+ "exp": 1719261824,
+ "email": "jane@example.com"
+ }
+
+ Code examples
+ ~~~~~~~~~~~~~
+ Below you can find example usages of this request in different scripting and
+ programming languages
+
+ .. tabs::
+
+ .. code-tab:: bash
+ :caption: Shell
+
+ curl -X GET https://www.freva.dkrz.de/api/auth/v2/status \
+ -H "Authorization: Bearer YOUR_ACCESS_TOKEN"
+
+ .. code-tab:: python
+ :caption: Python
+
+ import requests
+ response = requests.get(
+ "https://www.freva.dkrz.de/api/auth/v2/status",
+ headers={"Authorization": "Bearer YOUR_ACCESS_TOKEN"}
+ )
+ token_data = response.json()
+
+ .. code-tab:: r
+ :caption: gnuR
+
+ library(httr)
+
+ response <- GET(
+ "https://www.freva.dkrz.de/api/auth/v2/status",
+ add_headers(Authorization = paste("Bearer", "YOUR_ACCESS_TOKEN"))
+ )
+ token_data <- content(response, "parsed")
+
+ .. code-tab:: julia
+ :caption: Julia
+
+ using HTTP
+ using JSON
+
+ response = HTTP.get(
+ "https://www.freva.dkrz.de/api/auth/v2/status",
+ headers = Dict("Authorization" => "Bearer YOUR_ACCESS_TOKEN")
+ )
+ token_data = JSON.parse(String(response.body))
+
+ .. code-tab:: c
+ :caption: C/C++
+
+ #include
+ #include
+
+ int main() {
+ CURL *curl;
+ CURLcode res;
+
+ curl_global_init(CURL_GLOBAL_DEFAULT);
+ curl = curl_easy_init();
+ if(curl) {
+ struct curl_slist *headers = NULL;
+ headers = curl_slist_append(headers, "Authorization: Bearer YOUR_ACCESS_TOKEN");
+
+ curl_easy_setopt(curl, CURLOPT_URL, "https://www.freva.dkrz.de/api/auth/v2/status");
+ curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
+
+ res = curl_easy_perform(curl);
+ curl_easy_cleanup(curl);
+ }
+ curl_global_cleanup();
+ return 0;
+ }
+
+---
+
+Using the freva-client python library
+--------------------------------------
+The freva-client python library offers a very simple interface to interact
+with the authentication system.
+
+.. automodule:: freva_client
+ :members: authenticate
+
+Using the command line interface
+--------------------------------
+
+Token creation and refreshing can also be achieved with help of the ``auth``
+sub command of the command line interface
+
+.. code:: console
+
+ freva-client auth --help
+
+.. execute_code::
+ :hide_code:
+
+ from subprocess import run, PIPE
+ res = run(["freva-client", "auth", "--help"], check=True, stdout=PIPE, stderr=PIPE)
+ print(res.stdout.decode())
+
+You can create a token using your user name and password. For security reasons
+you can not pass your password as an argument to the command line interface.
+This means that you can only create a new token with help of a valid refresh
+token in a non-interactive session. Such as a batch job.
+
+Therefore you want to store your token data securely in a file, and use the
+refresh token to create new tokens:
+
+.. code:: console
+
+ freva-client auth -u janedoe > ~/.mytoken.json
+ chmod 600 ~/.mytoken.json
+
+Later you can use the `jq json command line parser `_
+to read the refresh token from and use it to create new access tokens.
+
+.. code:: console
+
+ export re_token=$(cat ~/.mytoken.json | jq -r .refresh_token)
+ freva-client auth -r $re_token > ~/.mytoken.json
diff --git a/_sources/databrowser/APIRef.rst.txt b/_sources/databrowser/APIRef.rst.txt
index a4256947..25615f3f 100644
--- a/_sources/databrowser/APIRef.rst.txt
+++ b/_sources/databrowser/APIRef.rst.txt
@@ -31,7 +31,7 @@ Getting an overview
:resheader Content-Type: ``application/json``: the available DRS search standards
and their search facets.
- - ``flavours``: array of available DRS stanards.
+ - ``flavours``: array of available DRS standards.
- ``attributes``: array of search facets for each available DRS standard.
Example Request
@@ -40,7 +40,7 @@ Getting an overview
.. sourcecode:: http
GET /api/databrowser/overview HTTP/1.1
- Host: api.freva.example
+ Host: www.freva.dkrz.de
Example Response
~~~~~~~~~~~~~~~~
@@ -96,20 +96,20 @@ Getting an overview
# Parse the json-content with jq
curl -X GET \
- http://api.freva.example/api/databrowser/overview | jq .attributes.cordex
+ https://www.freva.dkrz.de/api/databrowser/overview | jq .attributes.cordex
.. code-tab:: python
:caption: Python
import requests
- response = requests.get("http://api.freva.example/api/databrowser/overview")
+ response = requests.get("https://www.freva.dkrz.de/api/databrowser/overview")
data = response.json()
.. code-tab:: r
:caption: gnuR
library(httr)
- response <- GET("http://api.freva.example/api/databrowser/overview")
+ response <- GET("https://www.freva.dkrz.de/api/databrowser/overview")
data <- jsonlite::fromJSON(content(response, as = "text", encoding = "utf-8"))
.. code-tab:: julia
@@ -117,7 +117,7 @@ Getting an overview
using HTTP
using JSON
- response = HTTP.get("http://api.freva.example/api/databrowser/overview")
+ response = HTTP.get("https://www.freva.dkrz.de/api/databrowser/overview")
data = JSON.parse(String(HTTP.body(response)))
.. code-tab:: c
@@ -132,7 +132,7 @@ Getting an overview
curl = curl_easy_init();
if (curl) {
- char url[] = "https://api.freva.example/api/databrowser/overview";
+ char url[] = "https://www.freva.dkrz.de/api/databrowser/overview";
curl_easy_setopt(curl, CURLOPT_URL, url);
res = curl_easy_perform(curl);
@@ -198,7 +198,7 @@ Searching for datasets locations
.. sourcecode:: http
GET /api/databrowser/data_search/freva/file?product=EUR-11&fs_type=swift HTTP/1.1
- Host: api.freva.example
+ Host: www.freva.dkrz.de
Example Response
~~~~~~~~~~~~~~~~
@@ -228,14 +228,14 @@ Searching for datasets locations
:caption: Shell
curl -X GET \
- 'http://api.freva.example/api/databrowser/data_search/freva/file?product=EUR-11&fs_type=swift'
+ 'https://www.freva.dkrz.de/api/databrowser/data_search/freva/file?product=EUR-11&fs_type=swift'
.. code-tab:: python
:caption: Python
import requests
response = requests.get(
- "http://api.freva.example/api/databrowser/data_search/freva/file",
+ "https://www.freva.dkrz.de/api/databrowser/data_search/freva/file",
pramas={"product": "EUR-11", "fs_type": "swift"}
)
data = list(response.iter_lines(decode_unicode=True))
@@ -245,7 +245,7 @@ Searching for datasets locations
library(httr)
response <- GET(
- "http://api.freva.example/api/databrowser/data_search/freva/file",
+ "https://www.freva.dkrz.de/api/databrowser/data_search/freva/file",
query = list(product = "EUR-11", fs_type = "swift")
)
data <- strsplit(content(response, as = "text", encoding = "UTF-8"), "\n")[[1]]
@@ -257,7 +257,7 @@ Searching for datasets locations
using HTTP
response = HTTP.get(
- "http://api.freva.example/api/databrowser/data_search/freva/file",
+ "https://www.freva.dkrz.de/api/databrowser/data_search/freva/file",
query = Dict("product" => "EUR-11", "fs_type" => "swift")
)
data = split(String(HTTP.body(response)),"\n")
@@ -271,7 +271,7 @@ Searching for datasets locations
int main() {
CURL *curl;
CURLcode res;
- const char *url = "https://api.freva.example/api/databrowser/data_search/freva/file";
+ const char *url = "https://www.freva.dkrz.de/api/databrowser/data_search/freva/file";
// Query parameters
const char *product = "EUR-11";
@@ -372,7 +372,7 @@ Searching for metadata
- ``facet_mapping``: Translation rules describing
how to map the freva DRS standard to the desired
standard. This can be useful if ``GET /search_facets``
- was instructed to *not* tranlate the facet entries
+ was instructed to *not* translate the facet entries
and the translation should be done from client side.
- ``primary_facets``: Array of facets that are most
important. This can be useful for building clients
@@ -389,7 +389,7 @@ Searching for metadata
.. sourcecode:: http
GET /api/databrowser/metadata_search/freva/file?product=EUR-11 HTTP/1.1
- Host: api.freva.example
+ Host: www.freva.dkrz.de
Example Response
~~~~~~~~~~~~~~~~
@@ -459,7 +459,7 @@ Searching for metadata
.. code-tab:: bash
:caption: Shell
- curl -X GET 'http://api.freva.example/api/databrowser/metadata_search/freva/file?product=EUR-11'
+ curl -X GET 'https://www.freva.dkrz.de/api/databrowser/metadata_search/freva/file?product=EUR-11'
.. code-tab:: python
@@ -467,7 +467,7 @@ Searching for metadata
import requests
response = requests.get(
- "http://api.freva.example/api/databrowser/metadata_search/freva/file",
+ "https://www.freva.dkrz.de/api/databrowser/metadata_search/freva/file",
pramas={"product": "EUR-11"}
)
data = response.json()
@@ -477,7 +477,7 @@ Searching for metadata
library(httr)
response <- GET(
- "http://api.freva.example/api/databrowser/metadata_search/freva/file",
+ "https://www.freva.dkrz.de/api/databrowser/metadata_search/freva/file",
query = list(product = "EUR-11")
)
data <- jsonlite::fromJSON(content(response, as = "text", encoding = "utf-8"))
@@ -488,7 +488,7 @@ Searching for metadata
using HTTP
using JSON
response = HTTP.get(
- "http://api.freva.example/api/databrowser/metadata_search/freva/file",
+ "https://www.freva.dkrz.de/api/databrowser/metadata_search/freva/file",
query = Dict("product" => "EUR-11")
)
data = JSON.parse(String(HTTP.body(response)))
@@ -502,7 +502,7 @@ Searching for metadata
int main() {
CURL *curl;
CURLcode res;
- const char *url = "https://api.example.com/api/databrowser/metadata_search/freva/file";
+ const char *url = "https://www.freva.dkrz.de/api/databrowser/metadata_search/freva/file";
// Query parameters
const char *product = "EUR-11";
@@ -599,7 +599,7 @@ Generating an intake-esm catalogue
.. sourcecode:: http
GET /api/databrowser/intake_catalogue/freva/file?product=EUR-11 HTTP/1.1
- Host: api.freva.example
+ Host: www.freva.dkrz.de
Example Response
~~~~~~~~~~~~~~~~
@@ -669,7 +669,7 @@ Generating an intake-esm catalogue
:caption: Shell
curl -X GET \
- 'http://api.freva.example/api/databrowser/intake_catalogue/freva/file?product=EUR-11' > catalogue.json
+ 'https://www.freva.dkrz.de/api/databrowser/intake_catalogue/freva/file?product=EUR-11' > catalogue.json
.. code-tab:: python
:caption: Python
@@ -677,7 +677,7 @@ Generating an intake-esm catalogue
import requests
import intake
response = requests.get(
- "http://api.freva.example/api/databrowser/intake_catalogue/freva/file",
+ "https://www.freva.dkrz.de/api/databrowser/intake_catalogue/freva/file",
pramas={"product": "EUR-11"}
)
cat = intake.open_esm_datastore(cat)
@@ -687,7 +687,7 @@ Generating an intake-esm catalogue
library(httr)
response <- GET(
- "http://api.freva.example/api/databrowser/intake_catalogue/freva/file",
+ "https://www.freva.dkrz.de/api/databrowser/intake_catalogue/freva/file",
query = list(product = "EUR-11")
)
json_content <- content(response, "text", encoding="utf-8")
@@ -699,7 +699,7 @@ Generating an intake-esm catalogue
using HTTP
using JSON
response = HTTP.get(
- "http://api.freva.example/api/databrowser/intake_catalogue/freva/file",
+ "https://www.freva.dkrz.de/api/databrowser/intake_catalogue/freva/file",
query = Dict("product" => "EUR-11")
)
data = JSON.parse(String(HTTP.body(response)))
@@ -720,7 +720,7 @@ Generating an intake-esm catalogue
curl = curl_easy_init();
if (curl) {
- char url[] = "http://api.freva.example/api/databrowser/intake_catalogue/freva/file?product=EUR-11";
+ char url[] = "https://www.freva.dkrz.de/api/databrowser/intake_catalogue/freva/file?product=EUR-11";
curl_easy_setopt(curl, CURLOPT_URL, url);
fp = fopen("intake_catalogue.json", "w");
@@ -739,9 +739,181 @@ Generating an intake-esm catalogue
---
+.. _databrowser-api-zarr:
+
+Creating zarr endpoints for streaming data
+-------------------------------------------
+
+.. http:get:: /api/databrowser/load/(str:flavour)
+
+ This endpoint searches for datasets and streams the results as Zarr data.
+ The Zarr format allows for efficient storage and retrieval of large,
+ multidimensional arrays. This endpoint can be used to query datasets and
+ receive the results in a format that is suitable for further analysis and
+ processing with Zarr. If the ``catalogue-type`` parameter is set to "intake",
+ it can generate Intake-ESM catalogues that point to the generated Zarr
+ endpoints.
+
+ :param flavour: The Data Reference Syntax (DRS) standard specifying the
+ type of climate datasets to query. The available
+ DRS standards can be retrieved using the
+ ``GET /api/datasets/overview`` method.
+ :type flavour: str
+ :query start: Specify the starting point for receiving search results.
+ Default is 0.
+ :type start: int
+ :type max-results: int
+ :query multi-version: Use versioned datasets for querying instead of the
+ latest datasets. Default is false.
+ :type multi-version: bool
+ :query translate: Translate the metadata output to the required DRS flavour.
+ Default is true
+ :type translate: bool
+ :query catalogue-type: Set the type of catalogue you want to create from
+ this query.
+ :type catalogue-type: str
+ :query \**search_facets: With any other query parameters you refine your
+ data search. Query parameters could be, depending
+ on the DRS standard flavour ``product``, ``project``
+ ``model`` etc.
+ :type \**search_facets: str, list[str]
+ :reqheader Authorization: Bearer token for authentication.
+ :reqheader Content-Type: application/json
+
+ :statuscode 200: no error
+ :statuscode 400: no entries found for this query
+ :statuscode 422: invalid query parameters
+ :resheader Content-Type: ``text/plain``: zarr endpoints for the data
+
+
+ Example Request
+ ~~~~~~~~~~~~~~~
+
+ The logic works just like for the ``data_search`` and ``intake_catalogue``
+ endpoints. We constrain the data search by ``key=value`` search pairs.
+ The only difference is that we have to authenticate by using an access token.
+ You will also have to use a valid access token if you want to access the
+ zarr data via http. Please refer to the :ref:`auth` chapter for more details.
+
+ .. sourcecode:: http
+
+ GET /api/databrowser/load/freva/file?dataset=cmip6-fs HTTP/1.1
+ Host: www.freva.dkrz.de
+ Authorization: Bearer your_access_token
+
+ Example Response
+ ~~~~~~~~~~~~~~~~
+
+ .. sourcecode:: http
+
+ HTTP/1.1 200 OK
+ Content-Type: plain/text
+
+ https://www.freva.dkrz.de/api/freva-data-portal/zarr/dcb608a0-9d77-5045-b656-f21dfb5e9acf.zarr
+ https://www.freva.dkrz.de/api/freva-data-portal/zarr/f56264e3-d713-5c27-bc4e-c97f15b5fe86.zarr
+
+
+ Example
+ ~~~~~~~
+ Below you can find example usages of this request in different scripting and
+ programming languages.
+
+ .. tabs::
+
+ .. code-tab:: bash
+ :caption: Shell
+
+ curl -X GET \
+ 'https://www.freva.dkrz.de/api/databrowser/load/freva?dataset=cmip6-fs'
+ -H "Authorization: Bearer YOUR_ACCESS_TOKEN"
+
+ .. code-tab:: python
+ :caption: Python
+
+ import requests
+ import intake
+ response = requests.get(
+ "https://www.freva.dkrz.de/api/databrowser/load/freva",
+ pramas={"dataset": "cmip6-fs"},
+ headers={"Authorization": "Bearer YOUR_ACCESS_TOKEN"},
+ stream=True,
+ )
+ files = list(res.iterlines(decode_unicode=True)
+
+ .. code-tab:: r
+ :caption: gnuR
+
+ library(httr)
+ response <- GET(
+ "https://www.freva.dkrz.de/api/databrowser/load/freva",
+ query = list(dataset = "cmip6-fs")
+ )
+ data <- strsplit(content(response, as = "text", encoding = "UTF-8"), "\n")[[1]]
+
+
+ .. code-tab:: julia
+ :caption: Julia
+
+ using HTTP
+ response = HTTP.get(
+ "https://www.freva.dkrz.de/api/databrowser/load/freva",
+ query = Dict("dataset" => "cmip6-fs")
+ )
+ data = split(String(HTTP.body(response)),"\n")
+
+ .. code-tab:: c
+ :caption: C/C++
+
+ #include
+ #include
+
+ int main() {
+ CURL *curl;
+ CURLcode res;
+ const char *url = "https://www.freva.dkrz.de/api/databrowser/load/freva";
+
+ // Query parameters
+ const char *dataset = "cmip6-fs";
+ const int start = 0;
+ const int multi_version = 0; // 0 for false, 1 for true
+
+ // Build the query string
+ char query[256];
+ snprintf(query, sizeof(query),
+ "?dataset=%s&start=%d&multi-version=%d",product , start, multi_version);
+
+ // Initialize curl
+ curl = curl_easy_init();
+ if (!curl) {
+ fprintf(stderr, "Failed to initialize curl\n");
+ return 1;
+ }
+
+ // Construct the full URL with query parameters
+ char full_url[512];
+ snprintf(full_url, sizeof(full_url), "%s%s", url, query);
+
+ // Set the URL to fetch
+ curl_easy_setopt(curl, CURLOPT_URL, full_url);
+
+ // Perform the request
+ res = curl_easy_perform(curl);
+ if (res != CURLE_OK) {
+ fprintf(stderr, "curl_easy_perform() failed: %s\n", curl_easy_strerror(res));
+ }
+
+ // Clean up
+ curl_easy_cleanup(curl);
+
+ return 0;
+ }
+
+
+---
+
.. note::
Please note that in these examples,
- I used "https://api.freva.example" as a placeholder URL.
+ I used "https://www.freva.dkrz.de" as a placeholder URL.
You should replace it with the actual URL of your
Freva Databrowser REST API. The response above is truncated for brevity.
The actual response will include more datasets in the `catalog_dict` list.
diff --git a/_sources/databrowser/cli.rst.txt b/_sources/databrowser/cli.rst.txt
index 317ffd5f..5ef93ba5 100644
--- a/_sources/databrowser/cli.rst.txt
+++ b/_sources/databrowser/cli.rst.txt
@@ -5,22 +5,23 @@ The databrowser command line interface
.. toctree::
:maxdepth: 3
-This section introduces the usage of the ``freva-client`` command.
+This section introduces the usage of the ``freva-client databrowser`` sub command.
Please see the :ref:`install+configure` section on how to install and
configure the command line interface.
-After successful installation you will have the ``freva-client`` command
+After successful installation you can use the ``freva-client databrowser`` sub
+command
.. code:: console
- freva-client --help
+ freva-client databrowser --help
.. execute_code::
:hide_code:
from subprocess import run, PIPE
- res = run(["freva-client", "--help"], check=True, stdout=PIPE, stderr=PIPE)
+ res = run(["freva-client", "databrowser", "--help"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())
@@ -32,13 +33,13 @@ databrowser application. You can search for data locations by applying the
.. code:: console
- freva-client data-search --help
+ freva-client databrowser data-search --help
.. execute_code::
:hide_code:
from subprocess import run, PIPE
- res = run(["freva-client", "data-search", "--help"], check=True, stdout=PIPE, stderr=PIPE)
+ res = run(["freva-client", "databrowser", "data-search", "--help"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())
@@ -54,13 +55,13 @@ variables available that satisfies a certain constraint (e.g. sampled
.. code:: console
- freva-client project=observations variable=pr model=cp*
+ freva-client databrowser data-search project=observations variable=pr model=cp*
.. execute_code::
:hide_code:
from subprocess import run, PIPE
- res = run(["freva-client", "data-search", "experiment=cmorph"], check=True, stdout=PIPE, stderr=PIPE)
+ res = run(["freva-client", "databrowser", "data-search", "experiment=cmorph"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())
There are many more options for defining a value for a given key:
@@ -96,26 +97,24 @@ There are many more options for defining a value for a given key:
| | expression to find |
| | what you want.) |
+-------------------------------------------------+------------------------+
-| ``attribute=value1 [...] attribute=valueN`` | Search for files |
-| | containing *any* of N |
-| OR: | given values of same |
-| | attribute. |
-| ``attribute={value1,..,valueN}`` | |
-| | |
+| ``attribute=value1 attribute=value2`` | Search for files |
+| | containing either |
+| OR: | value1 OR value2 for |
+| | the given attribute |
+| ``attribute={value1,value2}`` | (note that's the same |
+| | attribute twice!) |
+-------------------------------------------------+------------------------+
| ``attribute1=value1 attribute2=value2`` | Search for files |
| | containing value1 for |
| | attribute1 AND value2 |
| | for attribute2 |
+-------------------------------------------------+------------------------+
-| ``attribute=-value`` ``attribute=not value`` | Search for files NOT |
+| ``attribute_not_=value`` | Search for files NOT |
| | containing value |
+-------------------------------------------------+------------------------+
-| ``attribute=-value1 attribute=not value2`` | Search for files |
-| | *not* caintaning given |
-| OR | values. You can also |
-| | combine search |
-| ``attribute1_not_=value _not_attribute2=value`` | attributes |
+| ``attribute_not_=value1 attribute_not_=value2`` | Search for files |
+| | containing neither |
+| | value1 nor value2 |
+-------------------------------------------------+------------------------+
.. note::
@@ -124,6 +123,38 @@ There are many more options for defining a value for a given key:
different meaning (normally it will try to match files with that name)
to turn that off you can use backslash \ (key=\*) or use quotes (key='*').
+Streaming files via zarr
+~~~~~~~~~~~~~~~~~~~~~~~~
+Instead of getting the file locations on disk or tape, you can instruct the
+system to register zarr streams. Which means that instead of opening the
+data directly you can open it via zarr from anywhere. To do so simply add
+the ``--zarr`` flag.
+
+.. note::
+
+ Before you can use the ``--zarr`` flag you will have
+ to create an access token and use that token to log on to the system
+ see also the :ref:`auth` chapter for more details on token creation.
+
+.. code:: console
+
+ token=$(freva-client auth -u janedoe|jq -r .access_token)
+ freva-client databrowser data-search dataset=cmip6-fs --zarr --access-token $token
+
+.. execute_code::
+ :hide_code:
+
+ from subprocess import run, PIPE
+ from freva_client import authenticate
+ token = authenticate(username="janedoe")
+ res = run(["freva-client", "databrowser", "data-search",
+ "--zarr", "dataset=cmip6-fs",
+ "--access-token", token["access_token"],
+ ], check=True, stdout=PIPE, stderr=PIPE)
+ print(res.stdout.decode())
+
+
+
Special cases: Searching for times
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -133,13 +164,13 @@ ranges:
.. code:: console
- freva-client data-search project=observations -t '2016-09-02T22:15 to 2016-10'
+ freva-client databrowser data-search project=observations -t '2016-09-02T22:15 to 2016-10'
.. execute_code::
:hide_code:
from subprocess import run, PIPE
- res = run(["freva-client", "data-search",
+ res = run(["freva-client", "databrowser", "data-search",
"-t", "2016-09-02T22:15 to 2016-10",
], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())
@@ -153,26 +184,26 @@ start of the time period:
.. code:: console
- freva-client data-search project=observations -t '2016-09-02T22:15 to 2016-10' -ts strict
+ freva-client databrowser data-search project=observations -t '2016-09-02T22:15 to 2016-10' -ts strict
.. execute_code::
:hide_code:
from subprocess import run, PIPE
- res = run(["freva-client", "data-search", "-t", "2016-09-02T22:15 to 2016-10", "-ts", "strict"], check=True, stdout=PIPE, stderr=PIPE)
+ res = run(["freva-client", "databrowser", "data-search", "-t", "2016-09-02T22:15 to 2016-10", "-ts", "strict"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())
Giving single time steps is also possible:
.. code:: console
- freva-client data-search project=observations -t 2016-09-02T22:10
+ freva-client databrowser data-search project=observations -t 2016-09-02T22:10
.. execute_code::
:hide_code:
from subprocess import run, PIPE
- res = run(["freva-client", "data-search", "-t", "2016-09-02T22:00"], check=True, stdout=PIPE, stderr=PIPE)
+ res = run(["freva-client", "databrowser", "data-search", "-t", "2016-09-02T22:00"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())
.. note::
@@ -183,6 +214,29 @@ Giving single time steps is also possible:
``2000-01 to 2100-12`` and alike. Single time steps are given without the
``to`` keyword.
+Creating intake-esm catalouges
+-------------------------------
+The ``intake-catalogue`` sub command allows you to create an
+`intake-esm catalogue _` from
+the current search. This can be useful to share the catalogue with others
+or merge datasets.
+
+.. code:: console
+
+ freva-client databrowser intake-catalogue --help
+
+.. execute_code::
+ :hide_code:
+
+ from subprocess import run, PIPE
+ res = run(["freva-client", "databrowser", "intake-catalogue", "--help"], check=True, stdout=PIPE, stderr=PIPE)
+ print(res.stdout.decode())
+
+
+You can either set the ``--filename`` flag to save the catalogue to a ``.json``
+file or pipe the catalogue to stdout (default). Just like for the ``data-search``
+sub command you can instruct the system to create zarr file streams to access
+the data via zarr.
Query the number of occurrences
-------------------------------
@@ -193,13 +247,13 @@ the files themselves.
.. code:: console
- freva-client data-count --help
+ freva-client databrowser data-count --help
.. execute_code::
:hide_code:
from subprocess import run, PIPE
- res = run(["freva-client", "data-count", "--help"], check=True, stdout=PIPE, stderr=PIPE)
+ res = run(["freva-client", "databrowser", "data-count", "--help"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())
By default the ``data-count`` sub command will display the total number of items
@@ -207,13 +261,13 @@ matching your search query. For example:
.. code:: console
- freva-client data-count project=observations
+ freva-client databrowser data-count project=observations
.. execute_code::
:hide_code:
from subprocess import run, PIPE
- res = run(["freva-client", "data-count", "project=observations"], check=True, stdout=PIPE, stderr=PIPE)
+ res = run(["freva-client", "databrowser", "data-count", "project=observations"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())
If you want to group the number of occurrences by search categories (facets)
@@ -221,13 +275,13 @@ use the ``-d`` or ``--detail`` flag:
.. code:: console
- freva-client data-count -d project=observations
+ freva-client databrowser data-count -d project=observations
.. execute_code::
:hide_code:
from subprocess import run, PIPE
- res = run(["freva-client", "data-count", "-d", "project=observations"], check=True, stdout=PIPE, stderr=PIPE)
+ res = run(["freva-client", "databrowser", "data-count", "-d", "project=observations"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())
@@ -239,13 +293,13 @@ For this you use the ``metadata-search`` sub command:
.. code:: console
- freva-client metadata-search --help
+ freva-client databrowser metadata-search --help
.. execute_code::
:hide_code:
from subprocess import run, PIPE
- res = run(["freva-client", "metadata-search", "--help"], check=True, stdout=PIPE, stderr=PIPE)
+ res = run(["freva-client", "databrowser", "metadata-search", "--help"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())
Just like with any other databrowser command you can apply different search
@@ -253,13 +307,13 @@ constraints when acquiring metadata
.. code:: console
- freva-client metadata-search project=observations
+ freva-client databrowser metadata-search project=observations
.. execute_code::
:hide_code:
from subprocess import run, PIPE
- res = run(["freva-client", "metadata-search", "project=observations"], check=True, stdout=PIPE, stderr=PIPE)
+ res = run(["freva-client", "databrowser", "metadata-search", "project=observations"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())
@@ -269,13 +323,13 @@ flag.
.. code:: console
- freva-client metadata-search -e project=observations
+ freva-client databrowser metadata-search -e project=observations
.. execute_code::
:hide_code:
from subprocess import run, PIPE
- res = run(["freva-client", "metadata-search", "-e", "project=observations"], check=True, stdout=PIPE, stderr=PIPE)
+ res = run(["freva-client", "databrowser", "metadata-search", "-e", "project=observations"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())
Sometimes you don't exactly know the exact names of the search keys and
@@ -284,13 +338,13 @@ for getting all ocean reanalysis datasets you can apply the ``--facet`` flag:
.. code:: console
- freva-client metadata-search -e realm=ocean --facet 'rean*'
+ freva-client databrowser metadata-search -e realm=ocean --facet 'rean*'
.. execute_code::
:hide_code:
from subprocess import run, PIPE
- res = run(["freva-client", "metadata-search","--facet", "rean*", "realm=ocean"], check=True, stdout=PIPE, stderr=PIPE)
+ res = run(["freva-client", "databrowser", "metadata-search","--facet", "rean*", "realm=ocean"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())
@@ -304,13 +358,13 @@ metadata of those files on tape:
.. code:: console
- freva-client metadata-search -e file="/arch/*"
+ freva-client databrowser metadata-search -e file="/arch/*"
.. execute_code::
:hide_code:
from subprocess import run, PIPE
- res = run(["freva-client", "metadata-search", "-e", "file=/arch*"], check=True, stdout=PIPE, stderr=PIPE)
+ res = run(["freva-client","databrowser", "metadata-search", "-e", "file=/arch*"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())
Parsing the command output
@@ -324,13 +378,13 @@ search to the `command line json processor jq `_:
.. code:: console
- freva-client metadata-search -e file="/arch/*" --json
+ freva-client databrowser metadata-search -e file="/arch/*" --json
.. execute_code::
:hide_code:
from subprocess import run, PIPE
- res = run(["freva-client", "metadata-search", "-e", "file=/arch*", "--json"], check=True, stdout=PIPE, stderr=PIPE)
+ res = run(["freva-client", "databrowser", "metadata-search", "-e", "file=/arch*", "--json"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())
By using the pipe operator ``|`` the JSON output of the `freva-client`
@@ -338,7 +392,7 @@ commands can be piped and processed by ``jq``:
.. code:: console
- freva-client metadata-search -e file="/arch/*" --json | jq -r .ensemble[0]
+ freva-client databrowser metadata-search -e file="/arch/*" --json | jq -r .ensemble[0]
.. execute_code::
:hide_code:
diff --git a/_sources/index.rst.txt b/_sources/index.rst.txt
index 9a682b5f..6d4fde53 100644
--- a/_sources/index.rst.txt
+++ b/_sources/index.rst.txt
@@ -123,9 +123,9 @@ Table of Content
.. toctree::
:maxdepth: 1
+ auth/index
databrowser/index
-
Feedback and Support
====================
We value your feedback and are committed to improving the API continuously.
diff --git a/_static/__pycache__/__init__.cpython-311.pyc b/_static/__pycache__/__init__.cpython-311.pyc
deleted file mode 100644
index ca766edd..00000000
Binary files a/_static/__pycache__/__init__.cpython-311.pyc and /dev/null differ
diff --git a/_static/__pycache__/__init__.cpython-312.pyc b/_static/__pycache__/__init__.cpython-312.pyc
new file mode 100644
index 00000000..4198c54e
Binary files /dev/null and b/_static/__pycache__/__init__.cpython-312.pyc differ
diff --git a/_static/documentation_options.js b/_static/documentation_options.js
index 25e09a46..2629b9e1 100644
--- a/_static/documentation_options.js
+++ b/_static/documentation_options.js
@@ -1,5 +1,5 @@
const DOCUMENTATION_OPTIONS = {
- VERSION: '2404.0.1',
+ VERSION: '2408.0.0.dev1',
LANGUAGE: 'en',
COLLAPSE_INDEX: false,
BUILDER: 'html',
diff --git a/auth/index.html b/auth/index.html
new file mode 100644
index 00000000..a5916bd8
--- /dev/null
+++ b/auth/index.html
@@ -0,0 +1,905 @@
+
+
+
+
+
+
+
+
+
+
+ Authentication — Freva Databrowser 2408.0.0.dev1 documentation
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Some functionality of the freva-rest API and the client library is only
+accessible after successful authentication. This authentication is realised
+with OAuth2 token creation. You can create new access and refresh tokens.
+Refresh tokens can be used to create new access tokens without needing to log
+in via username and password, thereby minimising the risk of exposing login
+credentials. Bear in mind that both login and refresh tokens have a limited
+lifetime.
+
Generally speaking, you have three options to interact with the authorization
+system:
The API supports token-based authentication using OAuth2. To obtain an access
+token, clients can use the /api/auth/v2/token endpoint by providing
+valid username and password credentials. The access token should then be
+included in the authorization header for secured endpoints.
Create an new login token from a username and password.
+You should either set a username and password or an existing refresh token.
+You can also set the client_id. Client id’s are configured to gain access,
+specific access for certain users. If you don’t set the client_id, the
+default id will be chosen.
+
+
Form Parameters:
+
+
username – The username for the login
+
password – The password for the login
+
refresh_token – The refresh token that is used to create a new token
+the refresh token can be used instead of authorizing
+via user creentials.
+
client_id – The unique identifier for your application used to
+request an OAuth2 access token from the authentication
+server, this form parameter is optional.
+
client_secret – An optional client secret used for authentication
+this parameters is optional an in most cases not neeede
The freva-client python library offers a very simple interface to interact
+with the authentication system.
+
Client software freva evaluation system framework (freva):
+
Freva, the free evaluation system framework, is a data search and analysis
+platform developed by the atmospheric science community for the atmospheric
+science community. With help of Freva researchers can:
+
+
quickly and intuitively search for data stored at typical data centers that
+host many datasets.
+
create a common interface for user defined data analysis tools.
+
apply data analysis tools in a reproducible manner.
+
+
The code described here is currently in testing phase. The client and server
+library described in the documentation only support searching for data. If you
+need to apply data analysis plugins, please visit the
Token creation and refreshing can also be achieved with help of the auth
+sub command of the command line interface
+
freva-client auth --help
+
+
+
Results
+
[1m [0m
+[1m [0m[1;33mUsage: [0m[1mfreva-client auth [OPTIONS][0m[1m [0m[1m [0m
+[1m [0m
+ Create OAuth2 access and refresh token.
+
+[2m╭─[0m[2m Options [0m[2m───────────────────────────────────────────────────────────────────[0m[2m─╮[0m
+[2m│[0m [1;36m-[0m[1;36m-host[0m [1;33mTEXT [0m Set the hostname of the databrowser, if [2m│[0m
+[2m│[0m not set (default) the hostname is read [2m│[0m
+[2m│[0m from a config file [2m│[0m
+[2m│[0m [2m[default: None] [0m [2m│[0m
+[2m│[0m [1;36m-[0m[1;36m-username[0m [1;32m-u[0m [1;33mTEXT [0m The username used for authentication. [2m│[0m
+[2m│[0m [2m[default: runner] [0m [2m│[0m
+[2m│[0m [1;36m-[0m[1;36m-refresh[0m[1;36m-token[0m [1;32m-r[0m [1;33mTEXT [0m Instead of using a password, you can use a [2m│[0m
+[2m│[0m refresh token. refresh the access token. [2m│[0m
+[2m│[0m This is recommended for non-interactive [2m│[0m
+[2m│[0m environments. [2m│[0m
+[2m│[0m [2m[default: None] [0m [2m│[0m
+[2m│[0m [1;36m-[0m[1;36m-force[0m [1;32m-f[0m [1;33m [0m Force token recreation, even if current [2m│[0m
+[2m│[0m token is still valid. [2m│[0m
+[2m│[0m [1;32m-v[0m [1;33mINTEGER[0m Increase verbosity [2m[default: 0][0m [2m│[0m
+[2m│[0m [1;36m-[0m[1;36m-version[0m [1;32m-V[0m [1;33m [0m Show version an exit [2m│[0m
+[2m│[0m [1;36m-[0m[1;36m-help[0m [1;33m [0m Show this message and exit. [2m│[0m
+[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m
+
+
+
+
+
You can create a token using your user name and password. For security reasons
+you can not pass your password as an argument to the command line interface.
+This means that you can only create a new token with help of a valid refresh
+token in a non-interactive session. Such as a batch job.
+
Therefore you want to store your token data securely in a file, and use the
+refresh token to create new tokens:
This endpoint searches for datasets and streams the results as Zarr data.
+The Zarr format allows for efficient storage and retrieval of large,
+multidimensional arrays. This endpoint can be used to query datasets and
+receive the results in a format that is suitable for further analysis and
+processing with Zarr. If the catalogue-type parameter is set to “intake”,
+it can generate Intake-ESM catalogues that point to the generated Zarr
+endpoints.
+
+
Parameters:
+
+
flavour (str) – The Data Reference Syntax (DRS) standard specifying the
+type of climate datasets to query. The available
+DRS standards can be retrieved using the
+GET/api/datasets/overview method.
+
+
+
Query Parameters:
+
+
start – Specify the starting point for receiving search results.
+Default is 0.
+
multi-version – Use versioned datasets for querying instead of the
+latest datasets. Default is false.
+
translate – Translate the metadata output to the required DRS flavour.
+Default is true
+
catalogue-type – Set the type of catalogue you want to create from
+this query.
+
**search_facets – With any other query parameters you refine your
+data search. Query parameters could be, depending
+on the DRS standard flavour product, project
+model etc.
The logic works just like for the data_search and intake_catalogue
+endpoints. We constrain the data search by key=value search pairs.
+The only difference is that we have to authenticate by using an access token.
+You will also have to use a valid access token if you want to access the
+zarr data via http. Please refer to the Authentication chapter for more details.
#include<stdio.h>
+#include<curl/curl.h>
+
+intmain(){
+CURL*curl;
+CURLcoderes;
+constchar*url="https://www.freva.dkrz.de/api/databrowser/load/freva";
+
+// Query parameters
+constchar*dataset="cmip6-fs";
+constintstart=0;
+constintmulti_version=0;// 0 for false, 1 for true
+
+// Build the query string
+charquery[256];
+snprintf(query,sizeof(query),
+"?dataset=%s&start=%d&multi-version=%d",product,start,multi_version);
+
+// Initialize curl
+curl=curl_easy_init();
+if(!curl){
+fprintf(stderr,"Failed to initialize curl\n");
+return1;
+}
+
+// Construct the full URL with query parameters
+charfull_url[512];
+snprintf(full_url,sizeof(full_url),"%s%s",url,query);
+
+// Set the URL to fetch
+curl_easy_setopt(curl,CURLOPT_URL,full_url);
+
+// Perform the request
+res=curl_easy_perform(curl);
+if(res!=CURLE_OK){
+fprintf(stderr,"curl_easy_perform() failed: %s\n",curl_easy_strerror(res));
+}
+
+// Clean up
+curl_easy_cleanup(curl);
+
+return0;
+}
+
+
+
+
+
—
Note
Please note that in these examples,
-I used “https://api.freva.example” as a placeholder URL.
+I used “https://www.freva.dkrz.de” as a placeholder URL.
You should replace it with the actual URL of your
Freva Databrowser REST API. The response above is truncated for brevity.
The actual response will include more datasets in the catalog_dict list.
This section introduces the usage of the freva-client command.
+
This section introduces the usage of the freva-clientdatabrowser sub command.
Please see the Installation and configuration section on how to install and
configure the command line interface.
-
After successful installation you will have the freva-client command
-
freva-client --help
+
After successful installation you can use the freva-clientdatabrowser sub
+command
+
freva-client databrowser --help
Results
[1m [0m
-[1m [0m[1;33mUsage: [0m[1mfreva-client [OPTIONS] COMMAND [ARGS]...[0m[1m [0m[1m [0m
+[1m [0m[1;33mUsage: [0m[1mfreva-client databrowser [OPTIONS] COMMAND [ARGS]...[0m[1m [0m[1m [0m
[1m [0m
- Freva the Free Evaluation System command line interface.
+ Data search related commands
[2m╭─[0m[2m Options [0m[2m───────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m [1;36m-[0m[1;36m-help[0m Show this message and exit. [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m
[2m╭─[0m[2m Commands [0m[2m──────────────────────────────────────────────────────────────────[0m[2m─╮[0m
-[2m│[0m [1;36mdata-count [0m[1;36m [0m Count the databrowser search results [2m│[0m
-[2m│[0m [1;36mdata-overview [0m[1;36m [0m Get an overview over what is available in the databrowser. [2m│[0m
-[2m│[0m [1;36mdata-search [0m[1;36m [0m Search the databrowser for datasets. [2m│[0m
-[2m│[0m [1;36mmetadata-search [0m[1;36m [0m Search databrowser for metadata (facets). [2m│[0m
+[2m│[0m [1;36mdata-count [0m[1;36m [0m Count the databrowser search results [2m│[0m
+[2m│[0m [1;36mdata-overview [0m[1;36m [0m Get an overview over what is available in the [2m│[0m
+[2m│[0m [1;36m [0m databrowser. [2m│[0m
+[2m│[0m [1;36mdata-search [0m[1;36m [0m Search the databrowser for datasets. [2m│[0m
+[2m│[0m [1;36mintake-catalogue [0m[1;36m [0m Create an intake catalogue from the search. [2m│[0m
+[2m│[0m [1;36mmetadata-search [0m[1;36m [0m Search databrowser for metadata (facets). [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m
@@ -427,12 +444,12 @@
Instead of getting the file locations on disk or tape, you can instruct the
+system to register zarr streams. Which means that instead of opening the
+data directly you can open it via zarr from anywhere. To do so simply add
+the --zarr flag.
+
+
Note
+
Before you can use the --zarr flag you will have
+to create an access token and use that token to log on to the system
+see also the Authentication chapter for more details on token creation.
Give password for server authentication: *****
+http://localhost:7777/api/freva-data-portal/zarr/c48f9024-f61e-5682-9651-7b9a21d81048.zarr
+http://localhost:7777/api/freva-data-portal/zarr/ff8b9acf-b652-5ce3-a26c-58f9bc4884a3.zarr
+
+
For example you want to know how many files we have between a certain time range
To do so you can use the time search key to subset time steps and whole time
ranges:
-
freva-client data-search project=observations -t '2016-09-02T22:15 to 2016-10'
+
freva-client databrowser data-search project=observations -t '2016-09-02T22:15 to 2016-10'
The intake-catalogue sub command allows you to create an
+intake-esm catalogue <https://intake-esm.readthedocs.io/en/stable/>_ from
+the current search. This can be useful to share the catalogue with others
+or merge datasets.
[1m [0m
+[1m [0m[1;33mUsage: [0m[1mfreva-client databrowser intake-catalogue [OPTIONS] [SEARCH_KEYS]...[0m[1m [0m[1m [0m
+[1m [0m
+ Create an intake catalogue from the search.
+
+[2m╭─[0m[2m Arguments [0m[2m─────────────────────────────────────────────────────────────────[0m[2m─╮[0m
+[2m│[0m search_keys [1;2;33m[[0m[1;33mSEARCH_KEYS]...[0m Refine your data search with this [2m│[0m
+[2m│[0m `key=value` pair search parameters. The [2m│[0m
+[2m│[0m parameters could be, depending on the [2m│[0m
+[2m│[0m DRS standard, flavour product, project [2m│[0m
+[2m│[0m model etc. [2m│[0m
+[2m│[0m [2m[default: None] [0m [2m│[0m
+[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m
+[2m╭─[0m[2m Options [0m[2m───────────────────────────────────────────────────────────────────[0m[2m─╮[0m
+[2m│[0m [1;36m-[0m[1;36m-facet[0m [1;33mTEXT [0m If you are not sure about [2m│[0m
+[2m│[0m the correct search key's [2m│[0m
+[2m│[0m you can use the [2m│[0m
+[2m│[0m ``[1;36m-[0m[1;36m-facet[0m`` flag to [2m│[0m
+[2m│[0m search of any matching [2m│[0m
+[2m│[0m entries. For example [2m│[0m
+[2m│[0m [1;36m-[0m[1;36m-facet[0m 'era5' would [2m│[0m
+[2m│[0m allow you to search for [2m│[0m
+[2m│[0m any entries containing [2m│[0m
+[2m│[0m era5, regardless of [2m│[0m
+[2m│[0m project, product etc. [2m│[0m
+[2m│[0m [2m[default: None] [0m [2m│[0m
+[2m│[0m [1;36m-[0m[1;36m-uniq[0m[1;36m-key[0m [1;32m-u[0m [1;2;33m[[0m[1;33mfile[0m[1;2;33m|[0m[1;33muri[0m[1;2;33m][0m[1;33m [0m The type of search [2m│[0m
+[2m│[0m result, which can be [2m│[0m
+[2m│[0m either “file” or “uri”. [2m│[0m
+[2m│[0m This parameter determines [2m│[0m
+[2m│[0m whether the search will [2m│[0m
+[2m│[0m be based on file paths or [2m│[0m
+[2m│[0m Uniform Resource [2m│[0m
+[2m│[0m Identifiers [2m│[0m
+[2m│[0m [2m[default: file] [0m [2m│[0m
+[2m│[0m [1;36m-[0m[1;36m-flavour[0m [1;32m-f[0m [1;2;33m[[0m[1;33mfreva[0m[1;2;33m|[0m[1;33mcmip6[0m[1;2;33m|[0m[1;33mcmip5[0m[1;2;33m|[0m[1;33mcord[0m The Data Reference Syntax [2m│[0m
+[2m│[0m [1;33mex[0m[1;2;33m|[0m[1;33mnextgems[0m[1;2;33m][0m[1;33m [0m (DRS) standard specifying [2m│[0m
+[2m│[0m the type of climate [2m│[0m
+[2m│[0m datasets to query. [2m│[0m
+[2m│[0m [2m[default: freva] [0m [2m│[0m
+[2m│[0m [1;36m-[0m[1;36m-time[0m[1;36m-select[0m [1;32m-ts[0m [1;2;33m[[0m[1;33mstrict[0m[1;2;33m|[0m[1;33mflexible[0m[1;2;33m|[0m[1;33mfile[0m[1;2;33m][0m[1;33m [0m Operator that specifies [2m│[0m
+[2m│[0m how the time period is [2m│[0m
+[2m│[0m selected. Choose from [2m│[0m
+[2m│[0m flexible (default), [2m│[0m
+[2m│[0m strict or file. [2m│[0m
+[2m│[0m ``strict`` returns only [2m│[0m
+[2m│[0m those files that have the [2m│[0m
+[2m│[0m *entire* time period [2m│[0m
+[2m│[0m covered. The time search [2m│[0m
+[2m│[0m ``2000 to 2012`` will not [2m│[0m
+[2m│[0m select files containing [2m│[0m
+[2m│[0m data from 2010 to 2020 [2m│[0m
+[2m│[0m with the ``strict`` [2m│[0m
+[2m│[0m method. ``flexible`` will [2m│[0m
+[2m│[0m select those files as [2m│[0m
+[2m│[0m ``flexible`` returns [2m│[0m
+[2m│[0m those files that have [2m│[0m
+[2m│[0m either start or end [2m│[0m
+[2m│[0m period covered. ``file`` [2m│[0m
+[2m│[0m will only return files [2m│[0m
+[2m│[0m where the entire time [2m│[0m
+[2m│[0m period is contained [2m│[0m
+[2m│[0m within *one single* file. [2m│[0m
+[2m│[0m [2m[default: flexible] [0m [2m│[0m
+[2m│[0m [1;36m-[0m[1;36m-time[0m [1;32m-t[0m [1;33mTEXT [0m Special search facet to [2m│[0m
+[2m│[0m refine/subset search [2m│[0m
+[2m│[0m results by time. This can [2m│[0m
+[2m│[0m be a string [2m│[0m
+[2m│[0m representation of a time [2m│[0m
+[2m│[0m range or a single time [2m│[0m
+[2m│[0m step. The time steps have [2m│[0m
+[2m│[0m to follow ISO-8601. Valid [2m│[0m
+[2m│[0m strings are [2m│[0m
+[2m│[0m ``%Y-%m-%dT%H:%M`` to [2m│[0m
+[2m│[0m ``%Y-%m-%dT%H:%M`` for [2m│[0m
+[2m│[0m time ranges and [2m│[0m
+[2m│[0m ``%Y-%m-%dT%H:%M``. [2m│[0m
+[2m│[0m **Note**: You don't have [2m│[0m
+[2m│[0m to give the full string [2m│[0m
+[2m│[0m format to subset time [2m│[0m
+[2m│[0m steps ``%Y``, ``%Y-%m`` [2m│[0m
+[2m│[0m etc are also valid. [2m│[0m
+[2m│[0m [2m[default: None] [0m [2m│[0m
+[2m│[0m [1;36m-[0m[1;36m-zarr[0m [1;33m [0m Create zarr stream files, [2m│[0m
+[2m│[0m as catalogue targets. [2m│[0m
+[2m│[0m [1;36m-[0m[1;36m-access[0m[1;36m-token[0m [1;33mTEXT [0m Use this access token for [2m│[0m
+[2m│[0m authentication when [2m│[0m
+[2m│[0m creating a zarr based [2m│[0m
+[2m│[0m intake catalogue. [2m│[0m
+[2m│[0m [2m[default: None] [0m [2m│[0m
+[2m│[0m [1;36m-[0m[1;36m-filename[0m [1;32m-f[0m [1;33mPATH [0m Path to the file where [2m│[0m
+[2m│[0m the catalogue, should be [2m│[0m
+[2m│[0m written to. if None given [2m│[0m
+[2m│[0m (default) the catalogue [2m│[0m
+[2m│[0m is parsed to stdout. [2m│[0m
+[2m│[0m [2m[default: None] [0m [2m│[0m
+[2m│[0m [1;36m-[0m[1;36m-host[0m [1;33mTEXT [0m Set the hostname of the [2m│[0m
+[2m│[0m databrowser, if not set [2m│[0m
+[2m│[0m (default) the hostname is [2m│[0m
+[2m│[0m read from a config file [2m│[0m
+[2m│[0m [2m[default: None] [0m [2m│[0m
+[2m│[0m [1;32m-v[0m [1;33mINTEGER [0m Increase verbosity [2m│[0m
+[2m│[0m [2m[default: 0] [0m [2m│[0m
+[2m│[0m [1;36m-[0m[1;36m-multi[0m[1;36m-version[0m [1;33m [0m Select all versions and [2m│[0m
+[2m│[0m not just the latest [2m│[0m
+[2m│[0m version (default). [2m│[0m
+[2m│[0m [1;36m-[0m[1;36m-version[0m [1;32m-V[0m [1;33m [0m Show version an exit [2m│[0m
+[2m│[0m [1;36m-[0m[1;36m-help[0m [1;33m [0m Show this message and [2m│[0m
+[2m│[0m exit. [2m│[0m
+[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m
+
+
+
+
+
You can either set the --filename flag to save the catalogue to a .json
+file or pipe the catalogue to stdout (default). Just like for the data-search
+sub command you can instruct the system to create zarr file streams to access
+the data via zarr.
In some cases it might be useful to know how many files are found in the
databrowser for certain search constraints. In such cases you can use the
data-count sub command to count the number of found files instead of getting
the files themselves.
You can either search for files or uri’s. Uri’s give you an information
on the storage system where the files or objects you are looking for are
@@ -487,6 +501,9 @@
If you don’t have direct access to the data, for example because you are
+not directly logged in to the computer where the data is stored you can
+set stream_zarr=True. The data will then be
+provisioned in zarr format and can be opened from anywhere. But bear in
+mind that zarr streams if not accessed in time will expire. Since the
+data can be accessed from anywhere you will also have to authenticate
+before you are able to access the data. Refer also to the
+freva_client.authenticate() method.
Create an intake esm catalogue object from the search.
+
This method creates a intake-esm catalogue from the current object
+search. Instead of having the original files as target objects you can
+also choose to stream the files via zarr.
Get the metadata (facets) for the current databrowser query.
You can retrieve all information that is associated with your current
databrowser search. This can be useful for reverse searches for example
-for retrieving metadata of object sotres or file/directory names.
If you are not sure about the correct search key’s you can use
positional arguments to search of any matching entries. For example
‘era5’ would allow you to search for any entries
containing era5, regardless of project, product etc.
If you don’t know what search flavours or search keys you can use
for searching the data you can use this method to get an overview
over what is available.