Migrate Docs pages to Markdown

doc/developer_guide/code_reviews.md

@@ -0,0 +1,25 @@
+# Code reviews
+
+Before anything is merged into the release branch (`RC_*`), we require that one reviewer accepts the code changes of a pull request.
+
+## How to do a code review
+
+* Checkout out pull request locally ([how to checkout a pull request locally](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally))
+
+* Run tests locally
+
+* Go through code and see if it is readable and easy to understand
+
+* Not required, but often useful: test new features with your own data
+
+## Tips and expectations
+
+Doing a code review can be very challenging if you are unfamiliar with the process. Here is a set of documents which might provide a good resource on how to get started:
+
+https://github.com/google/eng-practices
+
+## Conventional comments
+
+The comments in a code review should be clear and constructive.
+
+A useful way of highlighting the intention of specific comments is to label them according to [conventional comments](https://conventionalcomments.org/).

doc/developer_guide/code_structure.rst → doc/developer_guide/code_structure.md

@@ -1,74 +1,51 @@
-.. _code_structure:
+(code_structure)=
+# Code structure and key design concepts
 
-Code structure and key design concepts
---------------------------------------
-
-==================================
-Modules
-==================================
+## Modules
 
 **tobac** aims to provide a flexible and modular framework which can be seen as a toolbox to create tracking algorithms according to the user's specific research needs. 
 
 The **tobac** package currently consists of three **main modules**:
 
-1. The :py:mod:`tobac.feature_detection` contains methods to identify objects (*features*) in 2D or 3D (3D or 4D when including the time dimensions) gridded data. This is done by identifying contiguous regions above or below one or multiple user-defined thresholds. The module makes use of :py:mod:`scipy.ndimage.label`, a generic image processing method that labels features in an array. The methods in :py:mod:`tobac.feature_detection` are high-level functions that enable a fast and effective feature detection and create easy-to-use output in form of a :py:mod:`pandas.DataFrame` that contains the coordinates and some basic information on each detected feature. The most high-level methods that is commonly used by users is :py:func:`tobac.feature_detection_multithreshold`. 
+1. The {py:mod}`tobac.feature_detection` contains methods to identify objects (*features*) in 2D or 3D (3D or 4D when including the time dimensions) gridded data. This is done by identifying contiguous regions above or below one or multiple user-defined thresholds. The module makes use of {py:mod}`scipy.ndimage.label`, a generic image processing method that labels features in an array. The methods in {py:mod}`tobac.feature_detection` are high-level functions that enable a fast and effective feature detection and create easy-to-use output in form of a {py:mod}`pandas.DataFrame` that contains the coordinates and some basic information on each detected feature. The most high-level methods that is commonly used by users is {py:func}`tobac.feature_detection_multithreshold`. 
 
-2. The :py:mod:`tobac.segmentation` module contains methods to define the extent of the identified feature areas or volumes. This step is needed to create a mask of the identified features because the feature detection currently only saves the center points of the features. The segmentation procedure is performed by using the watershedding method, but more methods are to be implemented in the future. Just as the feature detection, this module can handle both 2D and 3D data. 
+2. The {py:mod}`tobac.segmentation` module contains methods to define the extent of the identified feature areas or volumes. This step is needed to create a mask of the identified features because the feature detection currently only saves the center points of the features. The segmentation procedure is performed by using the watershedding method, but more methods are to be implemented in the future. Just as the feature detection, this module can handle both 2D and 3D data. 
 
-3. The :py:mod:`tobac.tracking` module is responsible for linking identified features over time. This module makes primarily use of the python package :py:mod:`trackpy`. Note that the linking using :py:mod:`trackpy` is based on particle tracking principles which means that only the feature center positions (not the entire area or volume associated with each feature) are needed to link features over time. Other methods such as tracking based on overlapping areas from the segmented features are to be implemented.
+3. The {py:mod}`tobac.tracking` module is responsible for linking identified features over time. This module makes primarily use of the python package {py:mod}`trackpy`. Note that the linking using {py:mod}`trackpy` is based on particle tracking principles which means that only the feature center positions (not the entire area or volume associated with each feature) are needed to link features over time. Other methods such as tracking based on overlapping areas from the segmented features are to be implemented.
 
 In addition to the main modules, there are three **postprocessing modules**: 
 
-4. The :py:mod:`tobac.merge_split` module provides functionality to identify mergers and splitters in the tracking output and to add labels such that one can reconstruct the parent and child tracks of each cell. 
+4. The {py:mod}`tobac.merge_split` module provides functionality to identify mergers and splitters in the tracking output and to add labels such that one can reconstruct the parent and child tracks of each cell. 
 
-5. The :py:mod:`tobac.analysis` module contains methods to analyze the tracking output and derive statistics about individual tracks as well as summary statistics of the entire populations of tracks or subsets of the latter. 
+5. The {py:mod}`tobac.analysis` module contains methods to analyze the tracking output and derive statistics about individual tracks as well as summary statistics of the entire populations of tracks or subsets of the latter. 
 
-6. The :py:mod:`tobac.plotting` module provides methods to visualize the tracking output, for example for creating maps and animations of identified features, segmented areas and tracks.
+6. The {py:mod}`tobac.plotting` module provides methods to visualize the tracking output, for example for creating maps and animations of identified features, segmented areas and tracks.
 
 
 Finally, there are two modules that are primarily **important for developers**:
 
-7. The :py:mod:`tobac.utils` module is a collection of smaller, not necessarily tracking-specific methods that facilitate and support the methods of the main modules. This module has multiple submodules. We separate methods that are rather generic and could also be practical for tobac users who build their own tracking algorithms (:py:mod:`tobac.utils.general`) and methods that mainly facilitate the development of **tobac** (:py:mod:`tobac.utils.internal`). Sometimes, new features come with the need of a whole set of new methods, so it could make sense to save these in their own submodule (see e.g. :py:mod:`tobac.periodic_boundaries`)
+7. The {py:mod}`tobac.utils` module is a collection of smaller, not necessarily tracking-specific methods that facilitate and support the methods of the main modules. This module has multiple submodules. We separate methods that are rather generic and could also be practical for tobac users who build their own tracking algorithms ({py:mod}`tobac.utils.general`) and methods that mainly facilitate the development of **tobac** ({py:mod}`tobac.utils.internal`). Sometimes, new features come with the need of a whole set of new methods, so it could make sense to save these in their own submodule (see e.g. {py:mod}`tobac.periodic_boundaries`)
 
-8. The :py:mod:`tobac.testing` module provides support for writing of unit tests. This module contains several methods to create simplified test data sets on which the various methods and parameters for feature detection, segmentation, and tracking can be tested. 
+8. The {py:mod}`tobac.testing` module provides support for writing of unit tests. This module contains several methods to create simplified test data sets on which the various methods and parameters for feature detection, segmentation, and tracking can be tested. 
 
 For more information on each submodule, refer to the respective source code documentation.
 
 One thing to note is that **tobac** as of now is purely functional. The plan is, however, to move towards a more object-oriented design with base classes for the main operations such as feature detection and tracking. 
 
+## Examples
 
-========
-Examples
-========
-
-To help users get started with **tobac** and to demonstrate the various functionalities, **tobac** hosts several detailed and **illustrated examples** in the form of Jupyter notebooks. They are hosted under the directory `examples/` and be executed by the user. Our readthedocs page also hosts a rendered version of our examples as `gallery <https://tobac.readthedocs.io/en/latest/examples.html>`_
+To help users get started with **tobac** and to demonstrate the various functionalities, **tobac** hosts several detailed and **illustrated examples** in the form of Jupyter notebooks. They are hosted under the directory `examples/` and be executed by the user. Our readthedocs page also hosts a rendered version of our examples as [gallery](https://tobac.readthedocs.io/en/latest/examples.html)
 
+## Migrating to xarray and dask
 
-============================
-Migrating to xarray and dask
-============================
-
-Currently, **tobac** uses `iris cubes <https://scitools-iris.readthedocs.io/en/latest/userguide/iris_cubes.html>`_ as the 
+Currently, **tobac** uses [iris cubes](https://scitools-iris.readthedocs.io/en/latest/userguide/iris_cubes.html) as the 
 primary data container. However, we are currently working on migrating the source code to 
-`xarray <https://docs.xarray.dev/en/stable/>`_ such that all internal functions are based on `xr.DataArray 
-objects <https://docs.xarray.dev/en/stable/generated/xarray.DataArray.html>`_. 
+[xarray](https://docs.xarray.dev/en/stable/) such that all internal functions are based on [xr.DataArray 
+objects](https://docs.xarray.dev/en/stable/generated/xarray.DataArray.html). 
 
 To ensure a robust transition from **iris** to **xarray**, we make use of various decorators that convert input and 
-output data for the main functions without changing their actual code. These decorators are located in the `decorator 
-submodule <https://github.com/tobac-project/tobac/blob/main/tobac/utils/decorators.py>`_. 
-
-In addition, one of our main goals for the future is to fully support `dask <https://www.dask.org/>`_, in order to scale 
-to large datasets and enable parallelization.  
-
-
-
-
-
-
-
-
-
-
-
-
+output data for the main functions without changing their actual code. These decorators are located in the [decorator 
+submodule](https://github.com/tobac-project/tobac/blob/main/tobac/utils/decorators.py). 
 
+In addition, one of our main goals for the future is to fully support [dask](https://www.dask.org/), in order to scale 
+to large datasets and enable parallelization.

doc/developer_guide/testing_sphinx-based_rendering.md

@@ -0,0 +1,139 @@
+(testing-sphinx-rendering)=
+# How to check the Sphinx-based rendering
+
+The workflow has been tested in a linux system. We aim to build a static
+website out of the documentation material present in `tobac`.
+
+## 1. Preparing the Local Environment
+
+- **choose a separate place for your testing**
+
+  I will use the temporary directory `/tmp/website-testing` which I
+  need to create. You can use a dedicated place of your choice …
+
+  ```bash
+  > mkdir /tmp/website-testing
+  > cd /tmp/website-testing
+  ```
+
+  I will indicate my position now with the `/tmp/website-testing>`
+  prompt.
+
+- **get the official repository**
+
+  ```bash
+  /tmp/website-testing> git clone https://github.com/tobac-project/tobac
+  ```
+
+  You might like to test a certain remote branch `<BRANCH>` then do:
+
+  ```bash
+  /tmp/website-testing> cd tobac
+  /tmp/website-testing/tobac> git fetch --all
+  /tmp/website-testing/tobac> git checkout -t origin/<BRANCH>
+  /tmp/website-testing/tobac> cd .. 
+  ```
+
+- **Python environment**
+
+  - create a python virtual env
+
+    ```bash
+    /tmp/website-testing> python -m venv .python3-venv
+    ```
+
+  - and install requirements
+
+    ```bash
+    # deactivation conda is only necessary if your loaded conda before … 
+    /tmp/website-testing> conda deactivate
+
+    # activate the new env and upgrade ``pip`` 
+    /tmp/website-testing> source .python3-venv/bin/activate 
+    /tmp/website-testing> pip install --upgrade pip
+
+    # now everything is installed into the local python env!
+    /tmp/website-testing> pip install -r tobac/doc/requirements.txt
+    ```
+
+    `pip`-based installation takes a bit of time, but is much faster than `conda`.
+
+
+If the installation runs without problems, you are ready to build the website.
+
+## 2. Building the Website
+
+Actually, only few steps are needed to build the website, i.e.
+
+- **running sphinx for rendering**
+
+  ```bash
+  /tmp/website-testing> cd tobac
+
+  /tmp/website-testing/tobac> sphinx-build -b html doc doc/_build/html
+  ```
+
+  If no severe error appeared
+
+- **view the HTML content**
+
+  ```bash
+  /tmp/website-testing/tobac> firefox doc/_build/html/index.html
+  ```
+
+## 3. Parsing Your Local Changes
+
+Now, we connect to your locally hosted `tobac` repository and your
+development branch.
+
+- **connect to your local repo**: Assume your repo is located at
+  `/tmp/tobac-testing/tobac`, then add a new remote alias and fetch
+  all content with
+
+  ```bash
+  /tmp/website-testing/tobac> git remote add local-repo /tmp/tobac-testing/tobac
+  /tmp/website-testing/tobac> git fetch --all
+  ```
+
+- **check your development branch out**: Now, assume the your
+  development branch is called `my-devel`, then do
+
+  ```bash
+  # to get a first overview on available branches
+  /tmp/website-testing/tobac> git branch --all
+
+  # and then actually get your development branch
+  /tmp/website-testing/tobac> git checkout -b my-devel local-repo/my-devel
+  ```
+
+  You should see your developments, now …
+
+- **build and view website again**
+
+  ```bash
+  /tmp/website-testing/tobac> sphinx-build -M clean doc doc/_build
+  /tmp/website-testing/tobac> sphinx-build -b html doc doc/_build/html
+  /tmp/website-testing/tobac> firefox _build/html/index.html
+  ```
+
+## Option: Check Rendering of a Pull requests
+
+- **check the pull request out**: Now, assume the PR has the ID `<ID>` and you define the branch name `BRANCH_NAME` as you like
+
+  ```bash
+  # to get PR shown as dedicated branch
+  /tmp/website-testing/tobac> git fetch upstream pull/ID/head:BRANCH_NAME
+
+  # and then actually get this PR as branch
+  /tmp/website-testing/tobac> git checkout BRANCH_NAME
+  ```
+
+  You should see the PR now ...
+
+- **build and view website again**
+
+  ```bash
+  /tmp/website-testing/tobac> sphinx-build -M clean doc doc/_build
+  /tmp/website-testing/tobac> sphinx-build -b html doc doc/_build/html
+  /tmp/website-testing/tobac> firefox _build/html/index.html
+  ```