Some changes to the docs. (#538)

Author: tobiasraabe

.pre-commit-config.yaml

@@ -76,8 +76,8 @@ repos:
         additional_dependencies: [
             mdformat-gfm,
             mdformat-black,
+            mdformat-pyproject,
         ]
-        args: [--wrap, "88"]
         files: (README\.md)
 -   repo: https://github.com/executablebooks/mdformat
     rev: 0.7.17
@@ -86,31 +86,9 @@ repos:
         additional_dependencies: [
             mdformat-myst,
             mdformat-black,
+            mdformat-pyproject,
         ]
-        args: [--wrap, "88"]
         files: (docs/.)
-        exclude: |
-            (?x)^(
-                docs/source/index.md|
-                docs/source/how_to_guides/bp_structure_of_task_files.md|
-                docs/source/how_to_guides/how_to_influence_build_order.md|
-                docs/source/how_to_guides/migrating_from_scripts_to_pytask.md|
-                docs/source/how_to_guides/repeating_tasks_with_different_inputs_the_pytest_way.md|
-                docs/source/how_to_guides/using_task_returns.md|
-                docs/source/how_to_guides/writing_custom_nodes.md|
-                docs/source/how_to_guides/hashing_inputs_of_tasks.md|
-                docs/source/how_to_guides/remote_files.md|
-                docs/source/reference_guides/hookspecs.md|
-                docs/source/tutorials/configuration.md|
-                docs/source/tutorials/debugging.md|
-                docs/source/tutorials/defining_dependencies_products.md|
-                docs/source/tutorials/making_tasks_persist.md|
-                docs/source/tutorials/repeating_tasks_with_different_inputs.md|
-                docs/source/tutorials/selecting_tasks.md|
-                docs/source/tutorials/set_up_a_project.md|
-                docs/source/tutorials/using_a_data_catalog.md|
-                docs/source/tutorials/write_a_task.md
-            )$
 -   repo: https://github.com/nbQA-dev/nbQA
     rev: 1.7.1
     hooks:

docs/source/changes.md

@@ -23,6 +23,8 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
 - {pull}`528` improves the codecov setup and coverage.
 - {pull}`535` reenables and fixes tests with Jupyter.
 - {pull}`536` allows partialed functions to be task functions.
+- {pull}`538` updates the documentation. For example, colon fences are replaced by
+  backticks to allow formatting all pages by mdformat.
 - {pull}`539` implements the {confval}`hook_module` configuration value and
   `--hook-module` commandline option to register hooks.
 - {pull}`540` changes the CLI entry-point and allow `pytask.build(tasks=task_func)` as
@@ -124,7 +126,7 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
 - {pull}`413` removes scripts to generate `.svg`s.
 - {pull}`414` allow more ruff rules.
 - {pull}`416` removes `.from_annot` again.
-- {pull}`417` deprecates {func}`pytask.mark.task` in favor of {func}`pytask.task`.
+- {pull}`417` deprecates `pytask.mark.task` in favor of {func}`pytask.task`.
 - {pull}`418` fixes and error and simplifies code in `dag.py`.
 - {pull}`420` converts `DeprecationWarning`s to `FutureWarning`s for the deprecated
   decorators.
@@ -170,7 +172,7 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
 - {pull}`376` enhances the documentation for `pytask dag`.
 - {pull}`378` conditionally skips test on MacOS.
 - {pull}`381` deprecates `@pytask.mark.parametrize`. (Closes {issue}`233`.)
-- {pull}`501` removes {class}`pytask.MetaNode`.
+- {pull}`501` removes `pytask.MetaNode`.
 
 ## 0.3.1 - 2023-12-25
 
@@ -254,7 +256,7 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
   arguments. It also implements {class}`_pytask.models.CollectionMetadata` to carry
   parametrized arguments to the task class.
 - {pull}`228` removes `task.pytaskmark` and moves the information to
-  {attr}`_pytask.models.CollectionMetadata.markers`.
+  {attr}`pytask.CollectionMetadata.markers`.
 - {pull}`229` implements a new loop-based approach to parametrizations using the
   {func}`@pytask.mark.task <pytask.mark.task>` decorator.
 - {pull}`230` implements {class}`_pytask.logging._TimeUnit` as a
@@ -337,8 +339,8 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
   the direction of the DAG.
 - {pull}`186` enhance live displays by deactivating auto-refresh, among other things.
 - {pull}`187` allows to enable and disable showing tracebacks and potentially different
-  styles in the future with {confval}`show_traceback=True|False`.
-- {pull}`188` refactors some code related to {class}`_pytask.enums.ExitCode`.
+  styles in the future with `show_traceback=True|False`.
+- {pull}`188` refactors some code related to {class}`pytask.ExitCode`.
 - {pull}`189` do not display a table in the execution if no task was run.
 - {pull}`190` updates the release notes.
 
@@ -389,8 +391,8 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
 
 ## 0.1.1 - 2021-08-25
 
-- {pull}`138` changes the default {confval}`verbosity` to `1` which displays the live
-  table during execution and `0` display the symbols for outcomes (e.g. `.`, `F`, `s`).
+- {pull}`138` changes the default `verbosity` to `1` which displays the live table
+  during execution and `0` display the symbols for outcomes (e.g. `.`, `F`, `s`).
 - {pull}`139` enables rich's auto-refresh mechanism for live objects which causes almost
   no performance penalty for the live table compared to the symbolic output.
 

docs/source/conf.py

@@ -85,14 +85,16 @@
     "click": ("https://click.palletsprojects.com/en/8.0.x/", None),
     "deepdiff": ("https://zepworks.com/deepdiff/current/", None),
     "networkx": ("https://networkx.org/documentation/stable", None),
+    "nx": ("https://networkx.org/documentation/stable", None),
     "pandas": ("https://pandas.pydata.org/docs", None),
+    "pd": ("https://pandas.pydata.org/docs", None),
     "pluggy": ("https://pluggy.readthedocs.io/en/latest", None),
     "pygraphviz": ("https://pygraphviz.github.io/documentation/stable/", None),
     "python": ("https://docs.python.org/3.10", None),
 }
 
 # MyST
-myst_enable_extensions = ["colon_fence", "deflist", "dollarmath"]
+myst_enable_extensions = ["deflist", "dollarmath"]
 myst_footnote_transition = False
 
 # Open Graph

docs/source/how_to_guides/bp_structure_of_task_files.md

@@ -13,9 +13,9 @@ are looking for orientation or inspiration, here are some tips.
 - Task functions should be at the top of a task module to easily identify what the
   module is for.
 
-  :::{seealso}
+  ```{seealso}
   The only exception might be for {doc}`repetitions <bp_scaling_tasks>`.
-  :::
+  ```
 
 - The purpose of the task function is to handle IO operations like loading and saving
   files and calling Python functions on the task's inputs. IO should not be handled in
@@ -44,11 +44,11 @@ module are re-run. If the runtime of all tasks in the module is high, you wait l
 for your tasks to finish or until an error occurs which prolongs your feedback loops and
 hurts your productivity.
 
-:::{seealso}
+```{seealso}
 Use {func}`@pytask.mark.persist <pytask.mark.persist>` if you want to avoid accidentally
 triggering an expensive task. It is also explained in [this
 tutorial](../tutorials/making_tasks_persist).
-:::
+```
 
 ### Structure of the module
 
@@ -86,11 +86,11 @@ Here is an example of a task module which conforms to all advices.
 ```{literalinclude} ../../../docs_src/how_to_guides/bp_structure_of_task_files.py
 ```
 
-:::{seealso}
+```{seealso}
 The structure of the task module is greatly inspired by John Ousterhout's "A Philosopy
 of Software Design" in which he coins the name "deep modules". In short, deep modules
 have simple interfaces which are defined by one or a few {term}`public functions <public
 function>` (or classes) which provide the functionality. The complexity is hidden inside
 the module in {term}`private functions <private function>` which are called by the
 {term}`public functions <public function>`.
-:::
+```

docs/source/how_to_guides/hashing_inputs_of_tasks.md

@@ -11,15 +11,15 @@ If an input is not parsed by any more specific node type, the general
 In the following example, the argument `text` will be parsed as a
 {class}`~pytask.PythonNode`.
 
-:::::{tab-set}
+`````{tab-set}
 
-::::{tab-item} Python 3.10+
+````{tab-item} Python 3.10+
 
 ```{literalinclude} ../../../docs_src/how_to_guides/hashing_inputs_of_tasks_example_1_py310.py
 ```
 
-::::
-:::::
+````
+`````
 
 By default, pytask does not detect changes in {class}`~pytask.PythonNode` and if the
 value would change (without changing the task module), pytask would not rerun the task.
@@ -28,15 +28,15 @@ We can also hash the value of {class}`~pytask.PythonNode` s so that pytask knows
 the input changed. For that, we need to use the {class}`~pytask.PythonNode` explicitly
 and set `hash = True`.
 
-:::::{tab-set}
+`````{tab-set}
 
-::::{tab-item} Python 3.10+
+````{tab-item} Python 3.10+
 
 ```{literalinclude} ../../../docs_src/how_to_guides/hashing_inputs_of_tasks_example_2_py310.py
 ```
 
-::::
-:::::
+````
+`````
 
 When `hash=True`, pytask will call the builtin {func}`hash` on the input that will call
 the `__hash__()` method of the object.
@@ -62,10 +62,10 @@ from interpreter session to interpreter session for security reasons (see
 ```
 
 {class}`list` and {class}`dict` are not hashable by default. Luckily, there are
-libraries who provide this functionality like {mod}`deepdiff`. We can use them to pass a
+libraries who provide this functionality like `deepdiff`. We can use them to pass a
 function to the {class}`~pytask.PythonNode` that generates a stable hash.
 
-First, install {mod}`deepdiff`.
+First, install `deepdiff`.
 
 ```console
 $ pip install deepdiff
@@ -74,12 +74,12 @@ $ conda install deepdiff
 
 Then, create the hash function and pass it to the node.
 
-:::::{tab-set}
+`````{tab-set}
 
-::::{tab-item} Python 3.10+
+````{tab-item} Python 3.10+
 
 ```{literalinclude} ../../../docs_src/how_to_guides/hashing_inputs_of_tasks_example_3_py310.py
 ```
 
-::::
-:::::
+````
+`````

docs/source/how_to_guides/how_to_influence_build_order.md

@@ -1,23 +1,23 @@
 # How to influence the build order
 
-:::{important}
+```{important}
 This guide shows how to influence the order in which tasks are executed. The feature
 should be treated with caution since it might make projects work whose dependencies and
 products are not fully specified.
-:::
+```
 
 You can influence the order in which tasks are executed by assigning preferences to
 tasks. Use {func}`@pytask.mark.try_first <pytask.mark.try_first>` to execute a task as
 early as possible and {func}`@pytask.mark.try_last <pytask.mark.try_last>` to defer
 execution.
 
-:::{note}
+```{note}
 A little bit more background: Tasks, dependencies and products form a directed acyclic
 graph (DAG). A [topological ordering](https://en.wikipedia.org/wiki/Topological_sorting)
 determines the order in which tasks are executed such that tasks are not run until their
 predecessors have been executed. You should not assume a fixed ordering in which tasks
 are executed.
-:::
+```
 
 As an example, here are two tasks where the decorator ensures that the output of the
 second task is always shown before the output of the first task.

docs/source/how_to_guides/migrating_from_scripts_to_pytask.md

@@ -55,7 +55,7 @@ An `if __name__ == "__main__"` block must be deleted.
 To let pytask know the order in which to execute tasks and when to re-run them, you'll
 need to specify task dependencies and products. Add dependencies as arguments to the
 function with default values. Do the same for products, but also add the special
-{obj}`~pytask.Product` annotation with {obj}`Annotated[Path, Product]`. For example:
+{obj}`~pytask.Product` annotation with `Annotated[Path, Product]`. For example:
 
 ```{literalinclude} ../../../docs_src/how_to_guides/migrating_from_scripts_to_pytask_4.py
 ```
@@ -65,10 +65,10 @@ You can also use a dictionary to group multiple dependencies or products.
 ```{literalinclude} ../../../docs_src/how_to_guides/migrating_from_scripts_to_pytask_5.py
 ```
 
-:::{seealso}
+```{seealso}
 If you want to learn more about dependencies and products, read the
 [tutorial](../tutorials/defining_dependencies_products.md).
-:::
+```
 
 ## Execution
 
@@ -88,10 +88,10 @@ then automatically spawn multiple processes to run the workflow in parallel.
 $ pytask -n 4
 ```
 
-:::{seealso}
+```{seealso}
 You can find more information on pytask-parallel in the
 [readme](https://github.com/pytask-dev/pytask-parallel) on Github.
-:::
+```
 
 ## Bonus: From R script to task
 
@@ -109,10 +109,10 @@ $ pip install pytask-r
 $ conda install -c conda-forge pytask-r
 ```
 
-:::{seealso}
+```{seealso}
 Checkout [pytask-julia](https://github.com/pytask-dev/pytask-julia) and
 [pytask-stata](https://github.com/pytask-dev/pytask-stata), too!
-:::
+```
 
 And here is the R script `prepare_data.r` that we want to integrate.
 
@@ -131,8 +131,8 @@ products.
 ```{literalinclude} ../../../docs_src/how_to_guides/migrating_from_scripts_to_pytask_6.py
 ```
 
-pytask automatically makes the paths to the dependencies and products available to the
-R file via a JSON file. Let us amend the R script to load the information from the JSON
+pytask automatically makes the paths to the dependencies and products available to the R
+file via a JSON file. Let us amend the R script to load the information from the JSON
 file.
 
 ```r

docs/source/how_to_guides/remote_files.md

@@ -12,11 +12,11 @@ with remote files. The package provides a {mod}`pathlib`-like interface, making
 easy to interact with files from an HTTP(S)-, Dropbox-, S3-, GCP-, Azure-based
 filesystem, and many more.
 
-:::{warning}
+```{warning}
 universal_pathlib does currently not support Python 3.12. To track progress, see [this
 PR](https://github.com/fsspec/universal_pathlib/pull/152) and check the [releases
 `>0.1.4`](https://github.com/fsspec/universal_pathlib/releases)
-:::
+```
 
 ## HTTP(S)-based filesystem
 
@@ -63,11 +63,10 @@ After installing s3fs, rerun the command.
 ```
 
 Usually, you will need credentials to access files. Search in
-[fsspec's documentation](https://filesystem-spec.readthedocs.io/en/latest)
-or the plugin's documentation, here
-[s3fs](https://s3fs.readthedocs.io/en/latest/#credentials), for information on
-authentication. One way would be to set the environment variables `AWS_ACCESS_KEY_ID`
-and `AWS_SECRET_ACCESS_KEY`.
+[fsspec's documentation](https://filesystem-spec.readthedocs.io/en/latest) or the
+plugin's documentation, here [s3fs](https://s3fs.readthedocs.io/en/latest/#credentials),
+for information on authentication. One way would be to set the environment variables
+`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
 
 ## Detecting changes in remote files