Skip to content

Commit

Permalink
Merge pull request #35 from FREVA-CLINT/release-prep
Browse files Browse the repository at this point in the history
Release prep
  • Loading branch information
antarcticrainforest authored May 14, 2024
2 parents 30d5bbf + a7923b7 commit 463163e
Show file tree
Hide file tree
Showing 30 changed files with 287 additions and 258 deletions.
3 changes: 3 additions & 0 deletions .coveragerc
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ exclude_lines =
except ImportError:
raise SystemExit

# Don't complain about the Completer class
class Completer

# Don't complain if non-runnable code isn't run:
if 0:
if perf_file *:
Expand Down
30 changes: 30 additions & 0 deletions .github/workflows/build_job.yml
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,36 @@ jobs:
tags: |
ghcr.io/freva-clint/freva-rest-api:${{ steps.repository.outputs.tag }}
ghcr.io/freva-clint/freva-rest-api:latest
pypi:
name: Create Pip package
permissions:
id-token: write
needs: [tests, build-and-push-vault-image]
runs-on: ubuntu-latest
steps:
-
name: Checkout
uses: actions/checkout@v4
-
name: Setup Python
uses: actions/setup-python@v4
with:
python-version: "3.x"
-
name: Install flit
run: python -m pip install flit
-
name: Building freva-deployment 📦 with flit
run: flit build
working-directory: freva-client
-
name: publish distribution 📦 to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
password: ${{ secrets.PYPI_API_TOKEN }}
skip-existing: true
verbose: true
packages-dir: freva-client/dist

bump-databrowser-version:
runs-on: ubuntu-latest
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@ __pycache__/
# C extensions
*.so


# Distribution / packaging
freva-client/LICENSE
.Python
build/
develop-eggs/
Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

[![License](https://img.shields.io/badge/License-BSD-purple.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.12-red.svg)](https://www.python.org/downloads/release/python-312/)
[![Docs](https://img.shields.io/badge/API-Doc-green.svg)](https://freva-clint.github.io/databrowserAPI)
[![Tests](https://github.com/FREVA-CLINT/databrowserAPI/actions/workflows/ci_job.yml/badge.svg)](https://github.com/FREVA-CLINT/databrowserAPI/actions)
[![Test-Coverage](https://codecov.io/github/FREVA-CLINT/databrowserAPI/branch/init/graph/badge.svg?token=dGhXxh7uP3)](https://codecov.io/github/FREVA-CLINT/databrowserAPI)
[![Docs](https://img.shields.io/badge/API-Doc-green.svg)](https://freva-clint.github.io/freva-nextgen)
[![Tests](https://github.com/FREVA-CLINT/freva-nextgen/actions/workflows/ci_job.yml/badge.svg)](https://github.com/FREVA-CLINT/freva-nextgen/actions)
[![Test-Coverage](https://codecov.io/github/FREVA-CLINT/freva-nextgen/branch/init/graph/badge.svg?token=dGhXxh7uP3)](https://codecov.io/github/FREVA-CLINT/freva-nextgen)

This repository contains the *freva-rest services* defining rest endpoints
that make up the freva server services as well as the client
Expand All @@ -25,8 +25,8 @@ rest service counterparts.
2. Clone this repository:

```console
git clone [email protected]:FREVA-CLINT/databrowserAPI.git
cd databrowserAPI
git clone [email protected]:FREVA-CLINT/freva-nextgen.git
cd freva-nextgen
```

3. Install flit:
Expand Down
72 changes: 36 additions & 36 deletions docs/source/databrowser/cli.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,22 +5,22 @@ The databrowser command line interface
.. toctree::
:maxdepth: 3

This section introduces the usage of the ``freva-databrowser`` command.
This section introduces the usage of the ``freva-client`` command.
Please see the :ref:`install+configure` section on how to install and
configure the command line interface.


After successful installation you will have the ``freva-databrowser`` command
After successful installation you will have the ``freva-client`` command

.. code:: console
freva-databrowser --help
freva-client --help
.. execute_code::
:hide_code:

from subprocess import run, PIPE
res = run(["freva-databrowser", "--help"], check=True, stdout=PIPE, stderr=PIPE)
res = run(["freva-client", "--help"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())


Expand All @@ -32,13 +32,13 @@ databrowser application. You can search for data locations by applying the

.. code:: console
freva-databrowser --help
freva-client data-search --help
.. execute_code::
:hide_code:

from subprocess import run, PIPE
res = run(["freva-databrowser", "data-search", "--help"], check=True, stdout=PIPE, stderr=PIPE)
res = run(["freva-client", "data-search", "--help"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())


Expand All @@ -54,13 +54,13 @@ variables available that satisfies a certain constraint (e.g. sampled

.. code:: console
freva-databrowser project=observations variable=pr model=cp*
freva-client project=observations variable=pr model=cp*
.. execute_code::
:hide_code:

from subprocess import run, PIPE
res = run(["freva-databrowser", "data-search", "experiment=cmorph"], check=True, stdout=PIPE, stderr=PIPE)
res = run(["freva-client", "data-search", "experiment=cmorph"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())

There are many more options for defining a value for a given key:
Expand Down Expand Up @@ -131,13 +131,13 @@ ranges:

.. code:: console
freva-databrowser data-search project=observations -t '2016-09-02T22:15 to 2016-10'
freva-client data-search project=observations -t '2016-09-02T22:15 to 2016-10'
.. execute_code::
:hide_code:

from subprocess import run, PIPE
res = run(["freva-databrowser", "data-search",
res = run(["freva-client", "data-search",
"-t", "2016-09-02T22:15 to 2016-10",
], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())
Expand All @@ -151,26 +151,26 @@ start of the time period:

.. code:: console
freva-databrowser data-search project=observations -t '2016-09-02T22:15 to 2016-10' -ts strict
freva-client data-search project=observations -t '2016-09-02T22:15 to 2016-10' -ts strict
.. execute_code::
:hide_code:

from subprocess import run, PIPE
res = run(["freva-databrowser", "data-search", "-t", "2016-09-02T22:15 to 2016-10", "-ts", "strict"], check=True, stdout=PIPE, stderr=PIPE)
res = run(["freva-client", "data-search", "-t", "2016-09-02T22:15 to 2016-10", "-ts", "strict"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())

Giving single time steps is also possible:

.. code:: console
freva-databrowser data-search project=observations -t 2016-09-02T22:10
freva-client data-search project=observations -t 2016-09-02T22:10
.. execute_code::
:hide_code:

from subprocess import run, PIPE
res = run(["freva-databrowser", "count", "-t", "2016-09-02T22:00"], check=True, stdout=PIPE, stderr=PIPE)
res = run(["freva-client", "data-search", "-t", "2016-09-02T22:00"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())

.. note::
Expand All @@ -186,46 +186,46 @@ Query the number of occurrences
-------------------------------
In some cases it might be useful to know how many files are found in the
databrowser for certain search constraints. In such cases you can use the
``count`` sub command to count the number of *found* files instead of getting
``data-count`` sub command to count the number of *found* files instead of getting
the files themselves.

.. code:: console
freva-databrowser count --help
freva-client data-count --help
.. execute_code::
:hide_code:

from subprocess import run, PIPE
res = run(["freva-databrowser", "count", "--help"], check=True, stdout=PIPE, stderr=PIPE)
res = run(["freva-client", "data-count", "--help"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())

By default the ``count`` sub command will display the total number of items
By default the ``data-count`` sub command will display the total number of items
matching your search query. For example:

.. code:: console
freva-databrowser count project=observations
freva-client data-count project=observations
.. execute_code::
:hide_code:

from subprocess import run, PIPE
res = run(["freva-databrowser", "count", "project=observations"], check=True, stdout=PIPE, stderr=PIPE)
res = run(["freva-client", "data-count", "project=observations"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())

If you want to group the number of occurrences by search categories (facets)
use the ``-d`` or ``--detail`` flag:

.. code:: console
freva-databrowser count -d project=observations
freva-client data-count -d project=observations
.. execute_code::
:hide_code:

from subprocess import run, PIPE
res = run(["freva-databrowser", "count", "-d", "project=observations"], check=True, stdout=PIPE, stderr=PIPE)
res = run(["freva-client", "data-count", "-d", "project=observations"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())


Expand All @@ -237,27 +237,27 @@ For this you use the ``metadata-search`` sub command:

.. code:: console
freva-databrowser metadata-search --help
freva-client metadata-search --help
.. execute_code::
:hide_code:

from subprocess import run, PIPE
res = run(["freva-databrowser", "metadata-search", "--help"], check=True, stdout=PIPE, stderr=PIPE)
res = run(["freva-client", "metadata-search", "--help"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())

Just like with any other databrowser command you can apply different search
constraints when acquiring metadata

.. code:: console
freva-databrowser metadata-search project=observations
freva-client metadata-search project=observations
.. execute_code::
:hide_code:

from subprocess import run, PIPE
res = run(["freva-databrowser", "metadata-search", "project=observations"], check=True, stdout=PIPE, stderr=PIPE)
res = run(["freva-client", "metadata-search", "project=observations"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())


Expand All @@ -267,13 +267,13 @@ flag.

.. code:: console
freva-databrowser metadata-search -e project=observations
freva-client metadata-search -e project=observations
.. execute_code::
:hide_code:

from subprocess import run, PIPE
res = run(["freva-databrowser", "metadata-search", "-e", "project=observations"], check=True, stdout=PIPE, stderr=PIPE)
res = run(["freva-client", "metadata-search", "-e", "project=observations"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())

Sometimes you don't exactly know the exact names of the search keys and
Expand All @@ -282,13 +282,13 @@ for getting all ocean reanalysis datasets you can apply the ``--facet`` flag:

.. code:: console
freva-databrowser metadata-search -e realm=ocean --facet 'rean*'
freva-client metadata-search -e realm=ocean --facet 'rean*'
.. execute_code::
:hide_code:

from subprocess import run, PIPE
res = run(["freva-databrowser", "metadata-search","--facet", "rean*", "realm=ocean"], check=True, stdout=PIPE, stderr=PIPE)
res = run(["freva-client", "metadata-search","--facet", "rean*", "realm=ocean"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())


Expand All @@ -302,13 +302,13 @@ metadata of those files on tape:

.. code:: console
freva-databrowser metadata-search -e file="/arch/*"
freva-client metadata-search -e file="/arch/*"
.. execute_code::
:hide_code:

from subprocess import run, PIPE
res = run(["freva-databrowser", "metadata-search", "-e", "file=/arch*"], check=True, stdout=PIPE, stderr=PIPE)
res = run(["freva-client", "metadata-search", "-e", "file=/arch*"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())

Parsing the command output
Expand All @@ -322,21 +322,21 @@ search to the `command line json processor jq <https://jqlang.github.io/jq/>`_:

.. code:: console
freva-databrowser metadata-search -e file="/arch/*" --json
freva-client metadata-search -e file="/arch/*" --json
.. execute_code::
:hide_code:

from subprocess import run, PIPE
res = run(["freva-databrowser", "metadata-search", "-e", "file=/arch*", "--json"], check=True, stdout=PIPE, stderr=PIPE)
res = run(["freva-client", "metadata-search", "-e", "file=/arch*", "--json"], check=True, stdout=PIPE, stderr=PIPE)
print(res.stdout.decode())

By using the pipe operator ``|`` the JSON output of the `freva-databrowser`
By using the pipe operator ``|`` the JSON output of the `freva-client`
commands can be piped and processed by ``jq``:

.. code:: console
freva-databrowser metadata-search -e file="/arch/*" --json | jq -r .ensemble[0]
freva-client metadata-search -e file="/arch/*" --json | jq -r .ensemble[0]
.. execute_code::
:hide_code:
Expand Down
12 changes: 6 additions & 6 deletions docs/source/databrowser/python-lib.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ TLDR: Too long didn't read
To query data databrowser and search for data you have three different options.
You can the to following methods

- :py:class:`databrowser`: The main class for searching data is the
:py:class:`freva.databrowser` class. After creating in instance of the
- :py:class:`freva_client.databrowser`: The main class for searching data is the
:py:class:`freva_client.databrowser` class. After creating in instance of the
databrowser class with your specific search constraints you can get retrieve
all *files* or *uris* that matching your search constraints. You can also
retrieve a count of the number objects matching the search, as well as
Expand All @@ -24,11 +24,11 @@ You can the to following methods
can be useful to get information on the storage system where the *files*
or object stores are located.

- :py:meth:`databrowser.metadata_search`: This class method lists all search
categories (facets) and their values.
- :py:meth:`freva_client.databrowser.metadata_search`: This class method lists
all search categories (facets) and their values.

- :py:meth:`freva.count_values`: You can count the occurrences of
search results with this method.
- :py:meth:`freva_client.databrowser.count_values`: You can count the
occurrences of search results with this method.


Library Reference
Expand Down
Loading

0 comments on commit 463163e

Please sign in to comment.