Skip to content

Commit ac44212

Browse files
authoredDec 31, 2023
Some changes to the docs. (#538)
1 parent 846d0ab commit ac44212

25 files changed

+378
-410
lines changed
 

‎.pre-commit-config.yaml

+2-24
Original file line numberDiff line numberDiff line change
@@ -76,8 +76,8 @@ repos:
7676
additional_dependencies: [
7777
mdformat-gfm,
7878
mdformat-black,
79+
mdformat-pyproject,
7980
]
80-
args: [--wrap, "88"]
8181
files: (README\.md)
8282
- repo: https://github.com/executablebooks/mdformat
8383
rev: 0.7.17
@@ -86,31 +86,9 @@ repos:
8686
additional_dependencies: [
8787
mdformat-myst,
8888
mdformat-black,
89+
mdformat-pyproject,
8990
]
90-
args: [--wrap, "88"]
9191
files: (docs/.)
92-
exclude: |
93-
(?x)^(
94-
docs/source/index.md|
95-
docs/source/how_to_guides/bp_structure_of_task_files.md|
96-
docs/source/how_to_guides/how_to_influence_build_order.md|
97-
docs/source/how_to_guides/migrating_from_scripts_to_pytask.md|
98-
docs/source/how_to_guides/repeating_tasks_with_different_inputs_the_pytest_way.md|
99-
docs/source/how_to_guides/using_task_returns.md|
100-
docs/source/how_to_guides/writing_custom_nodes.md|
101-
docs/source/how_to_guides/hashing_inputs_of_tasks.md|
102-
docs/source/how_to_guides/remote_files.md|
103-
docs/source/reference_guides/hookspecs.md|
104-
docs/source/tutorials/configuration.md|
105-
docs/source/tutorials/debugging.md|
106-
docs/source/tutorials/defining_dependencies_products.md|
107-
docs/source/tutorials/making_tasks_persist.md|
108-
docs/source/tutorials/repeating_tasks_with_different_inputs.md|
109-
docs/source/tutorials/selecting_tasks.md|
110-
docs/source/tutorials/set_up_a_project.md|
111-
docs/source/tutorials/using_a_data_catalog.md|
112-
docs/source/tutorials/write_a_task.md
113-
)$
11492
- repo: https://github.com/nbQA-dev/nbQA
11593
rev: 1.7.1
11694
hooks:

‎docs/source/changes.md

+9-7
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,8 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
2323
- {pull}`528` improves the codecov setup and coverage.
2424
- {pull}`535` reenables and fixes tests with Jupyter.
2525
- {pull}`536` allows partialed functions to be task functions.
26+
- {pull}`538` updates the documentation. For example, colon fences are replaced by
27+
backticks to allow formatting all pages by mdformat.
2628
- {pull}`539` implements the {confval}`hook_module` configuration value and
2729
`--hook-module` commandline option to register hooks.
2830
- {pull}`540` changes the CLI entry-point and allow `pytask.build(tasks=task_func)` as
@@ -124,7 +126,7 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
124126
- {pull}`413` removes scripts to generate `.svg`s.
125127
- {pull}`414` allow more ruff rules.
126128
- {pull}`416` removes `.from_annot` again.
127-
- {pull}`417` deprecates {func}`pytask.mark.task` in favor of {func}`pytask.task`.
129+
- {pull}`417` deprecates `pytask.mark.task` in favor of {func}`pytask.task`.
128130
- {pull}`418` fixes and error and simplifies code in `dag.py`.
129131
- {pull}`420` converts `DeprecationWarning`s to `FutureWarning`s for the deprecated
130132
decorators.
@@ -170,7 +172,7 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
170172
- {pull}`376` enhances the documentation for `pytask dag`.
171173
- {pull}`378` conditionally skips test on MacOS.
172174
- {pull}`381` deprecates `@pytask.mark.parametrize`. (Closes {issue}`233`.)
173-
- {pull}`501` removes {class}`pytask.MetaNode`.
175+
- {pull}`501` removes `pytask.MetaNode`.
174176

175177
## 0.3.1 - 2023-12-25
176178

@@ -254,7 +256,7 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
254256
arguments. It also implements {class}`_pytask.models.CollectionMetadata` to carry
255257
parametrized arguments to the task class.
256258
- {pull}`228` removes `task.pytaskmark` and moves the information to
257-
{attr}`_pytask.models.CollectionMetadata.markers`.
259+
{attr}`pytask.CollectionMetadata.markers`.
258260
- {pull}`229` implements a new loop-based approach to parametrizations using the
259261
{func}`@pytask.mark.task <pytask.mark.task>` decorator.
260262
- {pull}`230` implements {class}`_pytask.logging._TimeUnit` as a
@@ -337,8 +339,8 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
337339
the direction of the DAG.
338340
- {pull}`186` enhance live displays by deactivating auto-refresh, among other things.
339341
- {pull}`187` allows to enable and disable showing tracebacks and potentially different
340-
styles in the future with {confval}`show_traceback=True|False`.
341-
- {pull}`188` refactors some code related to {class}`_pytask.enums.ExitCode`.
342+
styles in the future with `show_traceback=True|False`.
343+
- {pull}`188` refactors some code related to {class}`pytask.ExitCode`.
342344
- {pull}`189` do not display a table in the execution if no task was run.
343345
- {pull}`190` updates the release notes.
344346

@@ -389,8 +391,8 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
389391

390392
## 0.1.1 - 2021-08-25
391393

392-
- {pull}`138` changes the default {confval}`verbosity` to `1` which displays the live
393-
table during execution and `0` display the symbols for outcomes (e.g. `.`, `F`, `s`).
394+
- {pull}`138` changes the default `verbosity` to `1` which displays the live table
395+
during execution and `0` display the symbols for outcomes (e.g. `.`, `F`, `s`).
394396
- {pull}`139` enables rich's auto-refresh mechanism for live objects which causes almost
395397
no performance penalty for the live table compared to the symbolic output.
396398

‎docs/source/conf.py

+3-1
Original file line numberDiff line numberDiff line change
@@ -85,14 +85,16 @@
8585
"click": ("https://click.palletsprojects.com/en/8.0.x/", None),
8686
"deepdiff": ("https://zepworks.com/deepdiff/current/", None),
8787
"networkx": ("https://networkx.org/documentation/stable", None),
88+
"nx": ("https://networkx.org/documentation/stable", None),
8889
"pandas": ("https://pandas.pydata.org/docs", None),
90+
"pd": ("https://pandas.pydata.org/docs", None),
8991
"pluggy": ("https://pluggy.readthedocs.io/en/latest", None),
9092
"pygraphviz": ("https://pygraphviz.github.io/documentation/stable/", None),
9193
"python": ("https://docs.python.org/3.10", None),
9294
}
9395

9496
# MyST
95-
myst_enable_extensions = ["colon_fence", "deflist", "dollarmath"]
97+
myst_enable_extensions = ["deflist", "dollarmath"]
9698
myst_footnote_transition = False
9799

98100
# Open Graph

‎docs/source/how_to_guides/bp_structure_of_task_files.md

+6-6
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,9 @@ are looking for orientation or inspiration, here are some tips.
1313
- Task functions should be at the top of a task module to easily identify what the
1414
module is for.
1515

16-
:::{seealso}
16+
```{seealso}
1717
The only exception might be for {doc}`repetitions <bp_scaling_tasks>`.
18-
:::
18+
```
1919

2020
- The purpose of the task function is to handle IO operations like loading and saving
2121
files and calling Python functions on the task's inputs. IO should not be handled in
@@ -44,11 +44,11 @@ module are re-run. If the runtime of all tasks in the module is high, you wait l
4444
for your tasks to finish or until an error occurs which prolongs your feedback loops and
4545
hurts your productivity.
4646

47-
:::{seealso}
47+
```{seealso}
4848
Use {func}`@pytask.mark.persist <pytask.mark.persist>` if you want to avoid accidentally
4949
triggering an expensive task. It is also explained in [this
5050
tutorial](../tutorials/making_tasks_persist).
51-
:::
51+
```
5252

5353
### Structure of the module
5454

@@ -86,11 +86,11 @@ Here is an example of a task module which conforms to all advices.
8686
```{literalinclude} ../../../docs_src/how_to_guides/bp_structure_of_task_files.py
8787
```
8888

89-
:::{seealso}
89+
```{seealso}
9090
The structure of the task module is greatly inspired by John Ousterhout's "A Philosopy
9191
of Software Design" in which he coins the name "deep modules". In short, deep modules
9292
have simple interfaces which are defined by one or a few {term}`public functions <public
9393
function>` (or classes) which provide the functionality. The complexity is hidden inside
9494
the module in {term}`private functions <private function>` which are called by the
9595
{term}`public functions <public function>`.
96-
:::
96+
```

‎docs/source/how_to_guides/hashing_inputs_of_tasks.md

+14-14
Original file line numberDiff line numberDiff line change
@@ -11,15 +11,15 @@ If an input is not parsed by any more specific node type, the general
1111
In the following example, the argument `text` will be parsed as a
1212
{class}`~pytask.PythonNode`.
1313

14-
:::::{tab-set}
14+
`````{tab-set}
1515
16-
::::{tab-item} Python 3.10+
16+
````{tab-item} Python 3.10+
1717
1818
```{literalinclude} ../../../docs_src/how_to_guides/hashing_inputs_of_tasks_example_1_py310.py
1919
```
2020
21-
::::
22-
:::::
21+
````
22+
`````
2323

2424
By default, pytask does not detect changes in {class}`~pytask.PythonNode` and if the
2525
value would change (without changing the task module), pytask would not rerun the task.
@@ -28,15 +28,15 @@ We can also hash the value of {class}`~pytask.PythonNode` s so that pytask knows
2828
the input changed. For that, we need to use the {class}`~pytask.PythonNode` explicitly
2929
and set `hash = True`.
3030

31-
:::::{tab-set}
31+
`````{tab-set}
3232
33-
::::{tab-item} Python 3.10+
33+
````{tab-item} Python 3.10+
3434
3535
```{literalinclude} ../../../docs_src/how_to_guides/hashing_inputs_of_tasks_example_2_py310.py
3636
```
3737
38-
::::
39-
:::::
38+
````
39+
`````
4040

4141
When `hash=True`, pytask will call the builtin {func}`hash` on the input that will call
4242
the `__hash__()` method of the object.
@@ -62,10 +62,10 @@ from interpreter session to interpreter session for security reasons (see
6262
```
6363

6464
{class}`list` and {class}`dict` are not hashable by default. Luckily, there are
65-
libraries who provide this functionality like {mod}`deepdiff`. We can use them to pass a
65+
libraries who provide this functionality like `deepdiff`. We can use them to pass a
6666
function to the {class}`~pytask.PythonNode` that generates a stable hash.
6767

68-
First, install {mod}`deepdiff`.
68+
First, install `deepdiff`.
6969

7070
```console
7171
$ pip install deepdiff
@@ -74,12 +74,12 @@ $ conda install deepdiff
7474

7575
Then, create the hash function and pass it to the node.
7676

77-
:::::{tab-set}
77+
`````{tab-set}
7878
79-
::::{tab-item} Python 3.10+
79+
````{tab-item} Python 3.10+
8080
8181
```{literalinclude} ../../../docs_src/how_to_guides/hashing_inputs_of_tasks_example_3_py310.py
8282
```
8383
84-
::::
85-
:::::
84+
````
85+
`````

‎docs/source/how_to_guides/how_to_influence_build_order.md

+4-4
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,23 @@
11
# How to influence the build order
22

3-
:::{important}
3+
```{important}
44
This guide shows how to influence the order in which tasks are executed. The feature
55
should be treated with caution since it might make projects work whose dependencies and
66
products are not fully specified.
7-
:::
7+
```
88

99
You can influence the order in which tasks are executed by assigning preferences to
1010
tasks. Use {func}`@pytask.mark.try_first <pytask.mark.try_first>` to execute a task as
1111
early as possible and {func}`@pytask.mark.try_last <pytask.mark.try_last>` to defer
1212
execution.
1313

14-
:::{note}
14+
```{note}
1515
A little bit more background: Tasks, dependencies and products form a directed acyclic
1616
graph (DAG). A [topological ordering](https://en.wikipedia.org/wiki/Topological_sorting)
1717
determines the order in which tasks are executed such that tasks are not run until their
1818
predecessors have been executed. You should not assume a fixed ordering in which tasks
1919
are executed.
20-
:::
20+
```
2121

2222
As an example, here are two tasks where the decorator ensures that the output of the
2323
second task is always shown before the output of the first task.

‎docs/source/how_to_guides/migrating_from_scripts_to_pytask.md

+9-9
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ An `if __name__ == "__main__"` block must be deleted.
5555
To let pytask know the order in which to execute tasks and when to re-run them, you'll
5656
need to specify task dependencies and products. Add dependencies as arguments to the
5757
function with default values. Do the same for products, but also add the special
58-
{obj}`~pytask.Product` annotation with {obj}`Annotated[Path, Product]`. For example:
58+
{obj}`~pytask.Product` annotation with `Annotated[Path, Product]`. For example:
5959

6060
```{literalinclude} ../../../docs_src/how_to_guides/migrating_from_scripts_to_pytask_4.py
6161
```
@@ -65,10 +65,10 @@ You can also use a dictionary to group multiple dependencies or products.
6565
```{literalinclude} ../../../docs_src/how_to_guides/migrating_from_scripts_to_pytask_5.py
6666
```
6767

68-
:::{seealso}
68+
```{seealso}
6969
If you want to learn more about dependencies and products, read the
7070
[tutorial](../tutorials/defining_dependencies_products.md).
71-
:::
71+
```
7272

7373
## Execution
7474

@@ -88,10 +88,10 @@ then automatically spawn multiple processes to run the workflow in parallel.
8888
$ pytask -n 4
8989
```
9090

91-
:::{seealso}
91+
```{seealso}
9292
You can find more information on pytask-parallel in the
9393
[readme](https://github.com/pytask-dev/pytask-parallel) on Github.
94-
:::
94+
```
9595

9696
## Bonus: From R script to task
9797

@@ -109,10 +109,10 @@ $ pip install pytask-r
109109
$ conda install -c conda-forge pytask-r
110110
```
111111

112-
:::{seealso}
112+
```{seealso}
113113
Checkout [pytask-julia](https://github.com/pytask-dev/pytask-julia) and
114114
[pytask-stata](https://github.com/pytask-dev/pytask-stata), too!
115-
:::
115+
```
116116

117117
And here is the R script `prepare_data.r` that we want to integrate.
118118

@@ -131,8 +131,8 @@ products.
131131
```{literalinclude} ../../../docs_src/how_to_guides/migrating_from_scripts_to_pytask_6.py
132132
```
133133

134-
pytask automatically makes the paths to the dependencies and products available to the
135-
R file via a JSON file. Let us amend the R script to load the information from the JSON
134+
pytask automatically makes the paths to the dependencies and products available to the R
135+
file via a JSON file. Let us amend the R script to load the information from the JSON
136136
file.
137137

138138
```r

‎docs/source/how_to_guides/remote_files.md

+6-7
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,11 @@ with remote files. The package provides a {mod}`pathlib`-like interface, making
1212
easy to interact with files from an HTTP(S)-, Dropbox-, S3-, GCP-, Azure-based
1313
filesystem, and many more.
1414

15-
:::{warning}
15+
```{warning}
1616
universal_pathlib does currently not support Python 3.12. To track progress, see [this
1717
PR](https://github.com/fsspec/universal_pathlib/pull/152) and check the [releases
1818
`>0.1.4`](https://github.com/fsspec/universal_pathlib/releases)
19-
:::
19+
```
2020

2121
## HTTP(S)-based filesystem
2222

@@ -63,11 +63,10 @@ After installing s3fs, rerun the command.
6363
```
6464

6565
Usually, you will need credentials to access files. Search in
66-
[fsspec's documentation](https://filesystem-spec.readthedocs.io/en/latest)
67-
or the plugin's documentation, here
68-
[s3fs](https://s3fs.readthedocs.io/en/latest/#credentials), for information on
69-
authentication. One way would be to set the environment variables `AWS_ACCESS_KEY_ID`
70-
and `AWS_SECRET_ACCESS_KEY`.
66+
[fsspec's documentation](https://filesystem-spec.readthedocs.io/en/latest) or the
67+
plugin's documentation, here [s3fs](https://s3fs.readthedocs.io/en/latest/#credentials),
68+
for information on authentication. One way would be to set the environment variables
69+
`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
7170

7271
## Detecting changes in remote files
7372

‎docs/source/how_to_guides/using_task_returns.md

+25-25
Original file line numberDiff line numberDiff line change
@@ -16,32 +16,32 @@ One way to declare the returns of functions as products is annotating the return
1616
In the following example, the second value of {class}`typing.Annotated` is a path that
1717
defines where the return of the function, a string, should be stored.
1818

19-
::::{tab-set}
19+
`````{tab-set}
2020
21-
:::{tab-item} Python 3.10+
21+
````{tab-item} Python 3.10+
2222
:sync: python310plus
2323
2424
```{literalinclude} ../../../docs_src/how_to_guides/using_task_returns_example_1_py310.py
2525
```
2626
27-
:::
27+
````
2828
29-
:::{tab-item} Python 3.8+
29+
````{tab-item} Python 3.8+
3030
:sync: python38plus
3131
3232
```{literalinclude} ../../../docs_src/how_to_guides/using_task_returns_example_1_py38.py
3333
```
3434
35-
:::
36-
::::
35+
````
36+
`````
3737

3838
It works because internally the path is converted to a {class}`pytask.PathNode` that is
3939
able to store objects of type {class}`str` and {class}`bytes`.
4040

41-
:::{seealso}
41+
```{seealso}
4242
Read the explanation on {doc}`nodes <../how_to_guides/writing_custom_nodes>` to learn
4343
more about how nodes work.
44-
:::
44+
```
4545

4646
## Task decorator
4747

@@ -58,31 +58,31 @@ If a task function has multiple returns, you can use multiple nodes to store eac
5858
returns in a different place. The following example shows how to accomplish it with both
5959
of the previous interfaces.
6060

61-
::::{tab-set}
61+
`````{tab-set}
6262
63-
:::{tab-item} Python 3.10+
63+
````{tab-item} Python 3.10+
6464
:sync: python310plus
6565
6666
```{literalinclude} ../../../docs_src/how_to_guides/using_task_returns_example_3_py310.py
6767
```
6868
69-
:::
69+
````
7070
71-
:::{tab-item} Python 3.8+
71+
````{tab-item} Python 3.8+
7272
:sync: python38plus
7373
7474
```{literalinclude} ../../../docs_src/how_to_guides/using_task_returns_example_3_py38.py
7575
```
7676
77-
:::
77+
````
7878
79-
:::{tab-item} &#8203;`@pytask.task`
79+
````{tab-item} @pytask.task
8080
8181
```{literalinclude} ../../../docs_src/how_to_guides/using_task_returns_example_3_task.py
8282
```
8383
84-
:::
85-
::::
84+
````
85+
`````
8686

8787
Each return is mapped to its node by respecting its position in the tuple.
8888

@@ -93,35 +93,35 @@ is a shallower tree.
9393
The following example shows how a task function with a complex structure of returns is
9494
mapped to the defined nodes.
9595

96-
::::{tab-set}
96+
`````{tab-set}
9797
98-
:::{tab-item} Python 3.10+
98+
````{tab-item} Python 3.10+
9999
:sync: python310plus
100100
101101
```{literalinclude} ../../../docs_src/how_to_guides/using_task_returns_example_4_py310.py
102102
```
103103
104-
:::
104+
````
105105
106-
:::{tab-item} Python 3.8+
106+
````{tab-item} Python 3.8+
107107
:sync: python38plus
108108
109109
```{literalinclude} ../../../docs_src/how_to_guides/using_task_returns_example_4_py38.py
110110
```
111111
112-
:::
112+
````
113113
114-
:::{tab-item} &#8203;`@pytask.task`
114+
````{tab-item} @pytask.task
115115
116116
```{literalinclude} ../../../docs_src/how_to_guides/using_task_returns_example_4_task.py
117117
```
118118
119-
:::
120-
::::
119+
````
120+
`````
121121

122122
The returns are mapped to the nodes as follows.
123123

124-
```python
124+
```text
125125
PythonNode(name="dict1") <- "a"
126126
PythonNode(name="dict2") <- {"b": 1, "c": 2}
127127
PythonNode(name="tuple1") <- 3

‎docs/source/how_to_guides/writing_custom_nodes.md

+42-37
Original file line numberDiff line numberDiff line change
@@ -17,47 +17,47 @@ to inputs and outputs and call {func}`~pandas.read_pickle` and
1717
```{literalinclude} ../../../docs_src/how_to_guides/writing_custom_nodes_example_1.py
1818
```
1919

20-
To remove IO operations from the task and delegate them to pytask, we will write a
21-
`PickleNode` that automatically loads and stores Python objects.
20+
To remove IO operations from the task and delegate them to pytask, we will replicate the
21+
{class}`~pytask.PickleNode` that automatically loads and stores Python objects.
2222

2323
And we pass the value to `df` via {obj}`~typing.Annotated` to preserve the type hint.
2424

2525
The result will be the following task.
2626

27-
::::{tab-set}
27+
`````{tab-set}
2828
29-
:::{tab-item} Python 3.10+
29+
````{tab-item} Python 3.10+
3030
:sync: python310plus
3131
3232
```{literalinclude} ../../../docs_src/how_to_guides/writing_custom_nodes_example_2_py310.py
3333
```
3434
35-
:::
35+
````
3636
37-
:::{tab-item} Python 3.10+ & Return
37+
````{tab-item} Python 3.10+ & Return
3838
:sync: python310plus
3939
4040
```{literalinclude} ../../../docs_src/how_to_guides/writing_custom_nodes_example_2_py310_return.py
4141
```
4242
43-
:::
43+
````
4444
45-
:::{tab-item} Python 3.8+
45+
````{tab-item} Python 3.8+
4646
:sync: python38plus
4747
4848
```{literalinclude} ../../../docs_src/how_to_guides/writing_custom_nodes_example_2_py38.py
4949
```
5050
51-
:::
51+
````
5252
53-
:::{tab-item} Python 3.8+ & Return
53+
````{tab-item} Python 3.8+ & Return
5454
:sync: python38plus
5555
5656
```{literalinclude} ../../../docs_src/how_to_guides/writing_custom_nodes_example_2_py38_return.py
5757
```
5858
59-
:::
60-
::::
59+
````
60+
`````
6161

6262
## Nodes
6363

@@ -73,54 +73,61 @@ follow at least the protocol {class}`~pytask.PNode` or, even better,
7373

7474
## `PickleNode`
7575

76-
Since our {class}`PickleNode` will only vary slightly from {class}`~pytask.PathNode`, we
77-
use it as a template, and with some minor modifications, we arrive at the following
78-
class.
76+
Since our {class}`~pytask.PickleNode` will only vary slightly from
77+
{class}`~pytask.PathNode`, we use it as a template, and with some minor modifications,
78+
we arrive at the following class.
7979

80-
::::{tab-set}
80+
`````{tab-set}
8181
82-
:::{tab-item} Python 3.10+
82+
````{tab-item} Python 3.10+
8383
:sync: python310plus
8484
8585
```{literalinclude} ../../../docs_src/how_to_guides/writing_custom_nodes_example_3_py310.py
8686
```
8787
88-
:::
88+
````
8989
90-
:::{tab-item} Python 3.8+
90+
````{tab-item} Python 3.8+
9191
:sync: python38plus
9292
9393
```{literalinclude} ../../../docs_src/how_to_guides/writing_custom_nodes_example_3_py38.py
9494
```
9595
96-
:::
97-
::::
96+
````
97+
`````
9898

9999
Here are some explanations.
100100

101101
- The node does not need to inherit from the protocol {class}`~pytask.PPathNode`, but
102102
you can do it to be more explicit.
103+
103104
- The node has two attributes
105+
104106
- `name` identifies the node in the DAG, so the name must be unique.
105107
- `path` holds the path to the file and identifies the node as a path node that is
106108
handled slightly differently than normal nodes within pytask.
109+
107110
- The node has an additional property that computes the signature of the node. The
108111
signature is a hash and a unique identifier for the node. For most nodes it will be a
109112
hash of the path or the name.
110-
- The {func}`classmethod` {meth}`PickleNode.from_path` is a convenient method to
113+
114+
- The {func}`classmethod` {meth}`~pytask.PickleNode.from_path` is a convenient method to
111115
instantiate the class.
112-
- The method {meth}`PickleNode.state` yields a value that signals the node's state. If
113-
the value changes, pytask knows it needs to regenerate the workflow. We can use
114-
the timestamp of when the node was last modified.
115-
- pytask calls {meth}`PickleNode.load` when it collects the values of function arguments
116-
to run the function. The argument `is_product` signals that the node is loaded as a
117-
product with a {class}`~pytask.Product` annotation or via `produces`.
116+
117+
- The method {meth}`~pytask.PickleNode.state` yields a value that signals the node's
118+
state. If the value changes, pytask knows it needs to regenerate the workflow. We can
119+
use the timestamp of when the node was last modified.
120+
121+
- pytask calls {meth}`~pytask.PickleNode.load` when it collects the values of function
122+
arguments to run the function. The argument `is_product` signals that the node is
123+
loaded as a product with a {class}`~pytask.Product` annotation or via `produces`.
118124

119125
When the node is loaded as a dependency, we want to inject the value of the pickle
120126
file. In the other case, the node returns itself so users can call
121-
{meth}`PickleNode.save` themselves.
122-
- {meth}`PickleNode.save` is called when a task function returns and allows to save the
123-
return values.
127+
{meth}`~pytask.PickleNode.save` themselves.
128+
129+
- {meth}`~pytask.PickleNode.save` is called when a task function returns and allows to
130+
save the return values.
124131

125132
## Conclusion
126133

@@ -135,14 +142,12 @@ databases. [^kedro]
135142

136143
## References
137144

138-
[^structural-subtyping]:
139-
Structural subtyping is similar to ABCs an approach in Python to
140-
enforce interfaces, but it can be considered more pythonic since it is closer to
141-
duck typing. Hynek Schlawack wrote a comprehensive
145+
[^structural-subtyping]: Structural subtyping is similar to ABCs an approach in Python to enforce interfaces, but
146+
it can be considered more pythonic since it is closer to duck typing. Hynek Schlawack
147+
wrote a comprehensive
142148
[guide on subclassing](https://hynek.me/articles/python-subclassing-redux/) that
143149
features protocols under "Type 2". Glyph wrote an introduction to protocols called
144150
[I want a new duck](https://glyph.twistedmatrix.com/2020/07/new-duck.html).
145151

146-
[^kedro]:
147-
Kedro, another workflow system, provides many adapters to data sources:
152+
[^kedro]: Kedro, another workflow system, provides many adapters to data sources:
148153
https://docs.kedro.org/en/stable/kedro_datasets.html.

‎docs/source/index.md

+13-12
Original file line numberDiff line numberDiff line change
@@ -49,10 +49,11 @@ there.
4949

5050
If you want to know more about pytask, dive into one of the following topics
5151

52-
::::{grid} 1 2 2 2
53-
:gutter: 3
54-
55-
:::{grid-item-card}
52+
`````{grid} 1 2 2 2
53+
---
54+
gutter: 3
55+
---
56+
````{grid-item-card}
5657
:text-align: center
5758
:img-top: _static/images/light-bulb.svg
5859
:class-img-top: index-card-image
@@ -69,9 +70,9 @@ Tutorials
6970
7071
Tutorials help you to get started with pytask and how you manage your first project.
7172
72-
:::
73+
````
7374
74-
:::{grid-item-card}
75+
````{grid-item-card}
7576
:text-align: center
7677
:img-top: _static/images/book.svg
7778
:class-img-top: index-card-image
@@ -89,9 +90,9 @@ How-to Guides
8990
How-to guides provide instructions for very specific and advanced tasks and document
9091
best-practices.
9192
92-
:::
93+
````
9394
94-
:::{grid-item-card}
95+
````{grid-item-card}
9596
:text-align: center
9697
:img-top: _static/images/books.svg
9798
:class-img-top: index-card-image
@@ -108,9 +109,9 @@ Explanations
108109
109110
Explanations deal with key topics and concepts which underlie the package.
110111
111-
:::
112+
````
112113
113-
:::{grid-item-card}
114+
````{grid-item-card}
114115
:text-align: center
115116
:img-top: _static/images/coding.svg
116117
:class-img-top: index-card-image
@@ -127,9 +128,9 @@ Reference Guides
127128
128129
Reference guides explain the implementation and provide an entry-point for developers.
129130
130-
:::
131+
````
131132
132-
::::
133+
`````
133134

134135
```{toctree}
135136
---

‎docs/source/reference_guides/api.md

+44-58
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,39 @@ To write to the terminal, use pytask's console.
2828
.. class:: pytask.console
2929
```
3030

31+
## Exceptions
32+
33+
Exceptions all inherit from
34+
35+
```{eval-rst}
36+
.. autoclass:: pytask.PytaskError
37+
```
38+
39+
The following exceptions can be used to interrupt pytask's flow, emit reduced tracebacks
40+
and return the correct exit codes.
41+
42+
```{eval-rst}
43+
.. autoclass:: pytask.CollectionError
44+
.. autoclass:: pytask.ConfigurationError
45+
.. autoclass:: pytask.ExecutionError
46+
.. autoclass:: pytask.ResolvingDependenciesError
47+
```
48+
49+
The remaining exceptions convey specific errors.
50+
51+
```{eval-rst}
52+
.. autoclass:: pytask.NodeNotCollectedError
53+
.. autoclass:: pytask.NodeNotFoundError
54+
```
55+
56+
## General classes
57+
58+
```{eval-rst}
59+
.. autoclass:: pytask.Session
60+
.. autoclass:: pytask.DataCatalog
61+
:members:
62+
```
63+
3164
## Marks
3265

3366
pytask uses marks to attach additional information to task functions which is processed
@@ -45,15 +78,11 @@ by the host or by plugins. The following marks are available by default.
4578
Can be any valid Python object or an iterable of any Python objects. To be
4679
valid, it must be parsed by some hook implementation for the
4780
:func:`_pytask.hookspecs.pytask_collect_node` entry-point.
48-
```
4981
50-
```{eval-rst}
5182
.. function:: pytask.mark.persist()
5283
5384
A marker for a task which should be persisted.
54-
```
5585
56-
```{eval-rst}
5786
.. function:: pytask.mark.produces(objects: Any | Iterable[Any] | dict[Any, Any])
5887
5988
Specify products of a task.
@@ -63,40 +92,30 @@ by the host or by plugins. The following marks are available by default.
6392
Can be any valid Python object or an iterable of any Python objects. To be
6493
valid, it must be parsed by some hook implementation for the
6594
:func:`_pytask.hookspecs.pytask_collect_node` entry-point.
66-
```
6795
68-
```{eval-rst}
6996
.. function:: pytask.mark.skipif(condition: bool, *, reason: str)
7097
7198
Skip a task based on a condition and provide a necessary reason.
7299
73100
:param bool condition: A condition for when the task is skipped.
74101
:param str reason: A reason why the task is skipped.
75-
```
76102
77-
```{eval-rst}
78103
.. function:: pytask.mark.skip_ancestor_failed(reason: str = "No reason provided")
79104
80105
An internal marker for a task which is skipped because an ancestor failed.
81106
82107
:param str reason: A reason why the task is skipped.
83-
```
84108
85-
```{eval-rst}
86109
.. function:: pytask.mark.skip_unchanged()
87110
88111
An internal marker for a task which is skipped because nothing has changed.
89112
90113
:param str reason: A reason why the task is skipped.
91-
```
92114
93-
```{eval-rst}
94115
.. function:: pytask.mark.skip()
95116
96117
Skip a task.
97-
```
98118
99-
```{eval-rst}
100119
.. function:: pytask.mark.task(name, *, id, kwargs)
101120
102121
The task decorator allows to mark any task function regardless of its name as a task
@@ -118,9 +137,6 @@ by the host or by plugins. The following marks are available by default.
118137
A dictionary containing keyword arguments which are passed to the task when it
119138
is executed.
120139
121-
```
122-
123-
```{eval-rst}
124140
.. function:: pytask.mark.try_first
125141
126142
Indicate that the task should be executed as soon as possible.
@@ -136,9 +152,6 @@ by the host or by plugins. The following marks are available by default.
136152
dependencies and products and automatic inference may be incomplete like with
137153
pytask-latex and latex-dependency-scanner.
138154
139-
```
140-
141-
```{eval-rst}
142155
.. function:: pytask.mark.try_last
143156
144157
Indicate that the task should be executed as late as possible.
@@ -207,39 +220,6 @@ These functions help you to handle marks.
207220
.. autofunction:: pytask.set_marks
208221
```
209222

210-
## Exceptions
211-
212-
Exceptions all inherit from
213-
214-
```{eval-rst}
215-
.. autoclass:: pytask.PytaskError
216-
```
217-
218-
The following exceptions can be used to interrupt pytask's flow, emit reduced tracebacks
219-
and return the correct exit codes.
220-
221-
```{eval-rst}
222-
.. autoclass:: pytask.CollectionError
223-
.. autoclass:: pytask.ConfigurationError
224-
.. autoclass:: pytask.ExecutionError
225-
.. autoclass:: pytask.ResolvingDependenciesError
226-
```
227-
228-
The remaining exceptions convey specific errors.
229-
230-
```{eval-rst}
231-
.. autoclass:: pytask.NodeNotCollectedError
232-
.. autoclass:: pytask.NodeNotFoundError
233-
```
234-
235-
## General classes
236-
237-
```{eval-rst}
238-
.. autoclass:: pytask.Session
239-
.. autoclass:: pytask.DataCatalog
240-
:members:
241-
```
242-
243223
## Protocols
244224

245225
Protocols define how tasks and nodes for dependencies and products have to be set up.
@@ -261,11 +241,11 @@ Nodes are the interface for different kinds of dependencies or products.
261241

262242
```{eval-rst}
263243
.. autoclass:: pytask.PathNode
264-
:members: load, save
244+
:members:
265245
.. autoclass:: pytask.PickleNode
266-
:members: load, save
246+
:members:
267247
.. autoclass:: pytask.PythonNode
268-
:members: load, save
248+
:members:
269249
```
270250

271251
To parse dependencies and products from nodes, use the following functions.
@@ -301,6 +281,7 @@ attribute of the task function.
301281

302282
```{eval-rst}
303283
.. autoclass:: pytask.CollectionMetadata
284+
:members:
304285
```
305286

306287
## Outcomes
@@ -370,8 +351,12 @@ resolution and execution.
370351
## Tree utilities
371352

372353
```{eval-rst}
373-
.. automodule:: pytask.tree_util
374-
:members:
354+
.. autoclass:: pytask.tree_util.PyTree
355+
.. autofunction:: pytask.tree_util.tree_flatten_with_path
356+
.. autofunction:: pytask.tree_util.tree_leaves
357+
.. autofunction:: pytask.tree_util.tree_map
358+
.. autofunction:: pytask.tree_util.tree_map_with_path
359+
.. autofunction:: pytask.tree_util.tree_structure
375360
```
376361

377362
## Typing
@@ -384,6 +369,7 @@ resolution and execution.
384369
>>> def task_example(path: Annotated[Path, Product]) -> None:
385370
... path.write_text("Hello, World!")
386371
372+
.. autofunction:: pytask.is_task_function
387373
```
388374

389375
## Tracebacks

‎docs/source/reference_guides/configuration.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -35,10 +35,10 @@ strongly discouraged, you can deactivate the warning in the configuration file w
3535
check_casing_of_paths = false
3636
```
3737
38-
:::{note}
38+
```{note}
3939
An error is only raised on Windows when a case-insensitive path is used. Contributions
4040
are welcome to also support macOS.
41-
:::
41+
```
4242
4343
````
4444

‎docs/source/reference_guides/hookspecs.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -67,12 +67,12 @@ The following hooks traverse directories and collect tasks from files.
6767
The following hooks are designed to build a DAG from tasks and dependencies and check
6868
which files have changed and need to be re-run.
6969

70-
:::{warning}
70+
```{warning}
7171
This step is still experimental and likely to change in the future. If you are planning
7272
to write a plugin which extends pytask in this dimension, please, start a discussion
7373
before writing a plugin. It may make your life easier if changes in pytask anticipate
7474
your plugin.
75-
:::
75+
```
7676

7777
```{eval-rst}
7878
.. autofunction:: pytask_dag

‎docs/source/tutorials/configuration.md

+3-3
Original file line numberDiff line numberDiff line change
@@ -39,10 +39,10 @@ The second option is to let pytask try to find the configuration itself.
3939

4040
1. Find the common base directory of all paths passed to pytask (default to the current
4141
working directory).
42-
2. Starting from this directory, look at all parent directories, and return the file if
42+
1. Starting from this directory, look at all parent directories, and return the file if
4343
it exists.
44-
3. Stop searching if a directory contains a `.git` directory/file or a valid configuration file with
45-
the right section.
44+
1. Stop searching if a directory contains a `.git` directory/file or a valid
45+
configuration file with the right section.
4646

4747
## The options
4848

‎docs/source/tutorials/debugging.md

+4-4
Original file line numberDiff line numberDiff line change
@@ -12,15 +12,15 @@ find out the cause of the exception.
1212
```{include} ../_static/md/pdb.md
1313
```
1414

15-
:::{tip}
15+
```{tip}
1616
If you do not know about the Python debugger or pdb yet, check out this [explanation
1717
from RealPython](https://realpython.com/python-debugging-pdb/).
18-
:::
18+
```
1919

20-
:::{note}
20+
```{note}
2121
A following tutorial explains {doc}`how to select a subset of tasks <selecting_tasks>`.
2222
Combine it with the {option}`pytask build --pdb` flag to debug specific tasks.
23-
:::
23+
```
2424

2525
## Tracing
2626

‎docs/source/tutorials/defining_dependencies_products.md

+52-57
Original file line numberDiff line numberDiff line change
@@ -11,15 +11,15 @@ You find a tutorial on type hints {doc}`here <../type_hints>`.
1111

1212
If you want to avoid type annotations for now, look at the tab named `produces`.
1313

14-
The `Decorators` tab documents the deprecated approach that should not be used
15-
anymore and will be removed in version v0.5.
14+
The `Decorators` tab documents the deprecated approach that should not be used anymore
15+
and will be removed in version v0.5.
1616

1717
In this tutorial, we only deal with local files. If you want to use pytask with files
1818
online, S3, GCP, Azure, etc., read the
1919
{doc}`guide on remote files <../how_to_guides/remote_files>`.
2020

21-
First, we focus on defining products that should already be familiar to you. Then,
22-
we focus on how you can declare task dependencies.
21+
First, we focus on defining products that should already be familiar to you. Then, we
22+
focus on how you can declare task dependencies.
2323

2424
We use the same project as before and add a `task_plot_data.py` module.
2525

@@ -47,9 +47,9 @@ my_project
4747
Let's revisit the task from the {doc}`previous tutorial <write_a_task>` that we defined
4848
in `task_data_preparation.py`.
4949

50-
::::{tab-set}
50+
`````{tab-set}
5151
52-
:::{tab-item} Python 3.10+
52+
````{tab-item} Python 3.10+
5353
:sync: python310plus
5454
5555
```{literalinclude} ../../../docs_src/tutorials/defining_dependencies_products_products_py310.py
@@ -59,9 +59,9 @@ in `task_data_preparation.py`.
5959
{class}`~pytask.Product` allows marking an argument as a product. After the
6060
task has finished, pytask will check whether the file exists.
6161
62-
:::
62+
````
6363
64-
:::{tab-item} Python 3.8+
64+
````{tab-item} Python 3.8+
6565
:sync: python38plus
6666
6767
```{literalinclude} ../../../docs_src/tutorials/defining_dependencies_products_products_py38.py
@@ -71,9 +71,9 @@ task has finished, pytask will check whether the file exists.
7171
{class}`~pytask.Product` allows marking an argument as a product. After the
7272
task has finished, pytask will check whether the file exists.
7373
74-
:::
74+
````
7575
76-
:::{tab-item} &#8203;`produces`
76+
````{tab-item} prodouces
7777
:sync: produces
7878
7979
```{literalinclude} ../../../docs_src/tutorials/defining_dependencies_products_products_produces.py
@@ -84,9 +84,9 @@ Tasks can use `produces` as a "magic" argument name. Every value, or in this cas
8484
passed to this argument is automatically treated as a task product. Here, we pass the
8585
path as the default argument.
8686
87-
:::
87+
````
8888
89-
:::{tab-item} Decorators
89+
````{tab-item} Decorators
9090
:sync: decorators
9191
9292
```{warning}
@@ -103,13 +103,15 @@ task. After the task has finished, pytask will check whether the file exists.
103103
Add `produces` as an argument of the task function to get access to the same path inside
104104
the task function.
105105
106-
:::
107-
::::
106+
````
108107
109-
:::{tip}
110-
If you do not know about {mod}`pathlib` check out [^id3] and [^id4]. The module is
111-
beneficial for handling paths conveniently and across platforms.
112-
:::
108+
`````
109+
110+
```{tip}
111+
If you do not know about {mod}`pathlib` check out this guide by
112+
[RealPython](https://realpython.com/python-pathlib/). The module is beneficial for
113+
handling paths conveniently and across platforms.
114+
```
113115

114116
## Dependencies
115117

@@ -119,24 +121,24 @@ To show how dependencies work, we extend our project with another task that plot
119121
data generated with `task_create_random_data`. The task is called `task_plot_data`, and
120122
we will define it in `task_plot_data.py`.
121123

122-
::::{tab-set}
124+
`````{tab-set}
123125
124-
:::{tab-item} Python 3.10+
126+
````{tab-item} Python 3.10+
125127
:sync: python310plus
126128
127129
To specify that the task relies on the data set `data.pkl`, you can add the path
128130
to the function signature while choosing any argument name, here `path_to_data`.
129131
130-
pytask assumes that all function arguments that do not have a {class}`~pytask.`Product`
132+
pytask assumes that all function arguments that do not have a {class}`~pytask.Product`
131133
annotation are dependencies of the task.
132134
133135
```{literalinclude} ../../../docs_src/tutorials/defining_dependencies_products_dependencies_py310.py
134136
:emphasize-lines: 11
135137
```
136138
137-
:::
139+
````
138140
139-
:::{tab-item} Python 3.8+
141+
````{tab-item} Python 3.8+
140142
:sync: python38plus
141143
142144
To specify that the task relies on the data set `data.pkl`, you can add the path
@@ -149,9 +151,9 @@ annotation are dependencies of the task.
149151
:emphasize-lines: 11
150152
```
151153
152-
:::
154+
````
153155
154-
:::{tab-item} &#8203;`produces`
156+
````{tab-item} prodouces
155157
:sync: produces
156158
157159
To specify that the task relies on the data set `data.pkl`, you can add the path to the
@@ -164,9 +166,9 @@ pytask assumes that all function arguments that are not passed to the argument
164166
:emphasize-lines: 9
165167
```
166168
167-
:::
169+
````
168170
169-
:::{tab-item} Decorators
171+
````{tab-item} Decorators
170172
:sync: decorators
171173
172174
```{warning}
@@ -182,8 +184,8 @@ access the dependency path inside the function and load the data.
182184
:emphasize-lines: 9, 11
183185
```
184186
185-
:::
186-
::::
187+
````
188+
`````
187189

188190
Now, let us execute the two paths.
189191

@@ -195,36 +197,36 @@ Now, let us execute the two paths.
195197
Dependencies and products do not have to be absolute paths. If paths are relative, they
196198
are assumed to point to a location relative to the task module.
197199

198-
::::{tab-set}
200+
`````{tab-set}
199201
200-
:::{tab-item} Python 3.10+
202+
````{tab-item} Python 3.10+
201203
:sync: python310plus
202204
203205
```{literalinclude} ../../../docs_src/tutorials/defining_dependencies_products_relative_py310.py
204206
:emphasize-lines: 8
205207
```
206208
207-
:::
209+
````
208210
209-
:::{tab-item} Python 3.8+
211+
````{tab-item} Python 3.8+
210212
:sync: python38plus
211213
212214
```{literalinclude} ../../../docs_src/tutorials/defining_dependencies_products_relative_py38.py
213215
:emphasize-lines: 8
214216
```
215217
216-
:::
218+
````
217219
218-
:::{tab-item} &#8203;`produces`
220+
````{tab-item} prodouces
219221
:sync: produces
220222
221223
```{literalinclude} ../../../docs_src/tutorials/defining_dependencies_products_relative_produces.py
222224
:emphasize-lines: 4
223225
```
224226
225-
:::
227+
````
226228
227-
:::{tab-item} Decorators
229+
````{tab-item} Decorators
228230
:sync: decorators
229231
230232
```{warning}
@@ -241,16 +243,16 @@ You can also use absolute and relative paths as strings that obey the same rules
241243
If you use `depends_on` or `produces` as arguments for the task function, you will have
242244
access to the paths of the targets as {class}`pathlib.Path`.
243245
244-
:::
245-
::::
246+
````
247+
`````
246248

247249
## Multiple dependencies and products
248250

249251
Of course, tasks can have multiple dependencies and products.
250252

251-
::::{tab-set}
253+
`````{tab-set}
252254
253-
:::{tab-item} Python 3.10+
255+
````{tab-item} Python 3.10+
254256
:sync: python310plus
255257
256258
```{literalinclude} ../../../docs_src/tutorials/defining_dependencies_products_multiple1_py310.py
@@ -263,9 +265,9 @@ structures if needed.
263265
```{literalinclude} ../../../docs_src/tutorials/defining_dependencies_products_multiple2_py310.py
264266
```
265267
266-
:::
268+
````
267269
268-
:::{tab-item} Python 3.8+
270+
````{tab-item} Python 3.8+
269271
:sync: python38plus
270272
271273
```{literalinclude} ../../../docs_src/tutorials/defining_dependencies_products_multiple1_py38.py
@@ -278,9 +280,9 @@ structures if needed.
278280
```{literalinclude} ../../../docs_src/tutorials/defining_dependencies_products_multiple2_py38.py
279281
```
280282
281-
:::
283+
````
282284
283-
:::{tab-item} &#8203;`produces`
285+
````{tab-item} prodouces
284286
:sync: produces
285287
286288
If your task has multiple products, group them in one container like a dictionary
@@ -294,9 +296,9 @@ You can do the same with dependencies.
294296
```{literalinclude} ../../../docs_src/tutorials/defining_dependencies_products_multiple2_produces.py
295297
```
296298
297-
:::
299+
````
298300
299-
:::{tab-item} Decorators
301+
````{tab-item} Decorators
300302
:sync: decorators
301303
302304
```{warning}
@@ -405,8 +407,8 @@ def task_fit_model(depends_on, produces):
405407
}
406408
```
407409
408-
:::
409-
::::
410+
````
411+
`````
410412

411413
(after)=
412414

@@ -431,8 +433,7 @@ def task_plot_data(...):
431433
You can also pass a list of task functions.
432434

433435
The second mode is to pass an expression, a substring of the name of the dependent
434-
tasks. Here, we can pass the function name or a significant part of the function
435-
name.
436+
tasks. Here, we can pass the function name or a significant part of the function name.
436437

437438
```python
438439
@task(after="random_data")
@@ -450,9 +451,3 @@ You will learn more about expressions in {doc}`selecting_tasks`.
450451
- An overview of all ways to specify dependencies and products and their strengths and
451452
weaknesses can be found in
452453
{doc}`../how_to_guides/interfaces_for_dependencies_products`.
453-
454-
## References
455-
456-
[^id3]: The official documentation for {mod}`pathlib`.
457-
458-
[^id4]: A guide for pathlib by [RealPython](https://realpython.com/python-pathlib/).

‎docs/source/tutorials/making_tasks_persist.md

+7-7
Original file line numberDiff line numberDiff line change
@@ -19,10 +19,10 @@ in the database such that the subsequent execution will skip the task successful
1919
- You want to integrate a task that you have already run elsewhere. Copy over the
2020
dependencies and products and the task definition and make the task persist.
2121

22-
:::{caution}
22+
```{caution}
2323
This feature can corrupt the integrity of your project. Document why you have applied
2424
the decorator out of consideration for yourself and other contributors.
25-
:::
25+
```
2626

2727
## How to do it?
2828

@@ -45,15 +45,15 @@ Running pytask will execute the task since the product is missing.
4545
```
4646

4747
After that, we accidentally changed the task's source file by formatting the file with
48-
Black. Without the {func}`@pytask.mark.persist <pytask.mark.persist>` decorator, the task
49-
would run again since the source has changed. With the decorator, a green p signals that
50-
the execution is skipped.
48+
Black. Without the {func}`@pytask.mark.persist <pytask.mark.persist>` decorator, the
49+
task would run again since the source has changed. With the decorator, a green p signals
50+
that the execution is skipped.
5151

5252
```{include} ../_static/md/persist-persisted.md
5353
```
5454

55-
If we rerun the task, it is skipped because nothing has changed and not because
56-
it is marked with {func}`@pytask.mark.persist <pytask.mark.persist>`.
55+
If we rerun the task, it is skipped because nothing has changed and not because it is
56+
marked with {func}`@pytask.mark.persist <pytask.mark.persist>`.
5757

5858
```{include} ../_static/md/persist-skipped.md
5959
```

‎docs/source/tutorials/repeating_tasks_with_different_inputs.md

+63-65
Original file line numberDiff line numberDiff line change
@@ -11,33 +11,33 @@ reproducible samples.
1111
Apply the {func}`@task <pytask.task>` decorator, loop over the function and supply
1212
different seeds and output paths as default arguments of the function.
1313

14-
::::{tab-set}
14+
`````{tab-set}
1515
16-
:::{tab-item} Python 3.10+
16+
````{tab-item} Python 3.10+
1717
:sync: python310plus
1818
1919
```{literalinclude} ../../../docs_src/tutorials/repeating_tasks_with_different_inputs1_py310.py
2020
```
2121
22-
:::
22+
````
2323
24-
:::{tab-item} Python 3.8+
24+
````{tab-item} Python 3.8+
2525
:sync: python38plus
2626
2727
```{literalinclude} ../../../docs_src/tutorials/repeating_tasks_with_different_inputs1_py38.py
2828
```
2929
30-
:::
30+
````
3131
32-
:::{tab-item} &#8203;`produces`
32+
````{tab-item} produces
3333
:sync: produces
3434
3535
```{literalinclude} ../../../docs_src/tutorials/repeating_tasks_with_different_inputs1_produces.py
3636
```
3737
38-
:::
38+
````
3939
40-
:::{tab-item} Decorators
40+
````{tab-item} Decorators
4141
:sync: decorators
4242
4343
```{warning}
@@ -47,8 +47,8 @@ This approach is deprecated and will be removed in v0.5
4747
```{literalinclude} ../../../docs_src/tutorials/repeating_tasks_with_different_inputs1_decorators.py
4848
```
4949
50-
:::
51-
::::
50+
````
51+
`````
5252

5353
Executing pytask gives you this:
5454

@@ -59,33 +59,33 @@ Executing pytask gives you this:
5959

6060
You can also add dependencies to repeated tasks just like with any other task.
6161

62-
::::{tab-set}
62+
`````{tab-set}
6363
64-
:::{tab-item} Python 3.10+
64+
````{tab-item} Python 3.10+
6565
:sync: python310plus
6666
6767
```{literalinclude} ../../../docs_src/tutorials/repeating_tasks_with_different_inputs2_py310.py
6868
```
6969
70-
:::
70+
````
7171
72-
:::{tab-item} Python 3.8+
72+
````{tab-item} Python 3.8+
7373
:sync: python38plus
7474
7575
```{literalinclude} ../../../docs_src/tutorials/repeating_tasks_with_different_inputs2_py38.py
7676
```
7777
78-
:::
78+
````
7979
80-
:::{tab-item} &#8203;`produces`
80+
````{tab-item} produces
8181
:sync: produces
8282
8383
```{literalinclude} ../../../docs_src/tutorials/repeating_tasks_with_different_inputs2_produces.py
8484
```
8585
86-
:::
86+
````
8787
88-
:::{tab-item} Decorators
88+
````{tab-item} Decorators
8989
:sync: decorators
9090
9191
```{warning}
@@ -95,8 +95,8 @@ This approach is deprecated and will be removed in v0.5
9595
```{literalinclude} ../../../docs_src/tutorials/repeating_tasks_with_different_inputs2_decorators.py
9696
```
9797
98-
:::
99-
::::
98+
````
99+
`````
100100

101101
(how-to-repeat-a-task-with-different-inputs-the-id)=
102102

@@ -131,33 +131,33 @@ a combination of the argument name and the iteration counter.
131131

132132
For example, the following function is parametrized with tuples.
133133

134-
::::{tab-set}
134+
`````{tab-set}
135135
136-
:::{tab-item} Python 3.10+
136+
````{tab-item} Python 3.10+
137137
:sync: python310plus
138138
139139
```{literalinclude} ../../../docs_src/tutorials/repeating_tasks_with_different_inputs3_py310.py
140140
```
141141
142-
:::
142+
````
143143
144-
:::{tab-item} Python 3.8+
144+
````{tab-item} Python 3.8+
145145
:sync: python38plus
146146
147147
```{literalinclude} ../../../docs_src/tutorials/repeating_tasks_with_different_inputs3_py38.py
148148
```
149149
150-
:::
150+
````
151151
152-
:::{tab-item} &#8203;`produces`
152+
````{tab-item} produces
153153
:sync: produces
154154
155155
```{literalinclude} ../../../docs_src/tutorials/repeating_tasks_with_different_inputs3_produces.py
156156
```
157157
158-
:::
158+
````
159159
160-
:::{tab-item} Decorators
160+
````{tab-item} Decorators
161161
:sync: decorators
162162
163163
```{warning}
@@ -167,8 +167,8 @@ This approach is deprecated and will be removed in v0.5
167167
```{literalinclude} ../../../docs_src/tutorials/repeating_tasks_with_different_inputs3_decorators.py
168168
```
169169
170-
:::
171-
::::
170+
````
171+
`````
172172

173173
Since the tuples are not converted to strings, the ids of the two tasks are
174174

@@ -181,36 +181,36 @@ task_data_preparation.py::task_create_random_data[seed1]
181181

182182
### User-defined ids
183183

184-
The {func}`@task <pytask.task>` decorator has an `id` keyword, allowing
185-
the user to set a unique name for the iteration.
184+
The {func}`@task <pytask.task>` decorator has an `id` keyword, allowing the user to set
185+
a unique name for the iteration.
186186

187-
::::{tab-set}
187+
`````{tab-set}
188188
189-
:::{tab-item} Python 3.10+
189+
````{tab-item} Python 3.10+
190190
:sync: python310plus
191191
192192
```{literalinclude} ../../../docs_src/tutorials/repeating_tasks_with_different_inputs4_py310.py
193193
```
194194
195-
:::
195+
````
196196
197-
:::{tab-item} Python 3.8+
197+
````{tab-item} Python 3.8+
198198
:sync: python38plus
199199
200200
```{literalinclude} ../../../docs_src/tutorials/repeating_tasks_with_different_inputs4_py38.py
201201
```
202202
203-
:::
203+
````
204204
205-
:::{tab-item} &#8203;`produces`
205+
````{tab-item} produces
206206
:sync: produces
207207
208208
```{literalinclude} ../../../docs_src/tutorials/repeating_tasks_with_different_inputs4_produces.py
209209
```
210210
211-
:::
211+
````
212212
213-
:::{tab-item} Decorators
213+
````{tab-item} Decorators
214214
:sync: decorators
215215
216216
```{warning}
@@ -220,8 +220,8 @@ This approach is deprecated and will be removed in v0.5
220220
```{literalinclude} ../../../docs_src/tutorials/repeating_tasks_with_different_inputs4_decorators.py
221221
```
222222
223-
:::
224-
::::
223+
````
224+
`````
225225

226226
produces these ids
227227

@@ -237,8 +237,8 @@ and arguments. Here are three tips to organize the repetitions.
237237

238238
1. Use suitable containers to organize your ids and the function arguments.
239239

240-
::::{tab-set}
241-
:::{tab-item} NamedTuple
240+
`````{tab-set}
241+
````{tab-item} NamedTuple
242242
243243
{obj}`typing.NamedTuple` or {obj}`collections.namedtuple` are useful containers to
244244
organize the arguments of the parametrizations. They also provide better support for
@@ -260,9 +260,9 @@ and arguments. Here are three tips to organize the repetitions.
260260
}
261261
```
262262
263-
:::
263+
````
264264
265-
:::{tab-item} Dictionary
265+
````{tab-item} Dictionary
266266
267267
```python
268268
ID_TO_KWARGS = {
@@ -271,44 +271,44 @@ and arguments. Here are three tips to organize the repetitions.
271271
}
272272
```
273273
274-
:::
275-
::::
274+
````
275+
`````
276276

277-
2. {func}`@task <pytask.task>` has a `kwargs` argument that allows you
278-
inject arguments to the function instead of adding them as default arguments.
277+
1. {func}`@task <pytask.task>` has a `kwargs` argument that allows you inject arguments
278+
to the function instead of adding them as default arguments.
279279

280-
3. If the generation of arguments for the task function is complex, we should use a
280+
1. If the generation of arguments for the task function is complex, we should use a
281281
function.
282282

283283
Following these three tips, the parametrization becomes
284284

285-
::::{tab-set}
285+
`````{tab-set}
286286
287-
:::{tab-item} Python 3.10+
287+
````{tab-item} Python 3.10+
288288
:sync: python310plus
289289
290290
```{literalinclude} ../../../docs_src/tutorials/repeating_tasks_with_different_inputs5_py310.py
291291
```
292292
293-
:::
293+
````
294294
295-
:::{tab-item} Python 3.8+
295+
````{tab-item} Python 3.8+
296296
:sync: python38plus
297297
298298
```{literalinclude} ../../../docs_src/tutorials/repeating_tasks_with_different_inputs5_py38.py
299299
```
300300
301-
:::
301+
````
302302
303-
:::{tab-item} &#8203;`produces`
303+
````{tab-item} produces
304304
:sync: produces
305305
306306
```{literalinclude} ../../../docs_src/tutorials/repeating_tasks_with_different_inputs5_produces.py
307307
```
308308
309-
:::
309+
````
310310
311-
:::{tab-item} Decorators
311+
````{tab-item} Decorators
312312
:sync: decorators
313313
314314
```{warning}
@@ -318,12 +318,11 @@ This approach is deprecated and will be removed in v0.5
318318
```{literalinclude} ../../../docs_src/tutorials/repeating_tasks_with_different_inputs5_decorators.py
319319
```
320320
321-
:::
322-
::::
321+
````
322+
`````
323323

324324
Unpacking all the arguments can become tedious. Instead, use the `kwargs` argument of
325-
the {func}`@task <pytask.task>` decorator to pass keyword arguments to
326-
the task.
325+
the {func}`@task <pytask.task>` decorator to pass keyword arguments to the task.
327326

328327
```python
329328
for id_, kwargs in ID_TO_KWARGS.items():
@@ -354,8 +353,7 @@ for id_, kwargs in ID_TO_KWARGS.items():
354353
...
355354
```
356355

357-
The
358-
{doc}`best-practices guide on parametrizations <../how_to_guides/bp_scaling_tasks>`
356+
The {doc}`best-practices guide on parametrizations <../how_to_guides/bp_scaling_tasks>`
359357
goes into even more detail on how to scale parametrizations.
360358

361359
## A warning on globals

‎docs/source/tutorials/selecting_tasks.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -105,7 +105,7 @@ pytask uses booleans, floats, integers, and strings in the task id. It replaces
105105
Python objects like tuples with a combination of the argument name and an iteration
106106
counter and separates multiple arguments via dashes.
107107

108-
:::{seealso}
108+
```{seealso}
109109
Read this {ref}`section <how-to-repeat-a-task-with-different-inputs-the-id>` for more
110110
information on how ids for repeated tasks are created and can be customized.
111-
:::
111+
```

‎docs/source/tutorials/set_up_a_project.md

+11-11
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
# Set up a project
22

3-
Assuming you want to use pytask for a more extensive project, you want
4-
to organize the project as a Python package. This tutorial explains the minimal setup.
3+
Assuming you want to use pytask for a more extensive project, you want to organize the
4+
project as a Python package. This tutorial explains the minimal setup.
55

6-
If you want to use pytask with a collection of scripts, you can skip this lesson
7-
and move to the next section of the tutorials.
6+
If you want to use pytask with a collection of scripts, you can skip this lesson and
7+
move to the next section of the tutorials.
88

99
The following directory tree gives an overview of the project's different parts.
1010

@@ -49,11 +49,11 @@ SRC = Path(__file__).parent.resolve()
4949
BLD = SRC.joinpath("..", "..", "bld").resolve()
5050
```
5151

52-
:::{seealso}
52+
```{seealso}
5353
If you want to know more about the "`src` layout" and why it is NASA-approved, read
5454
[this article by Hynek Schlawack](https://hynek.me/articles/testing-packaging/) or this
5555
[setuptools article](https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#src-layout).
56-
:::
56+
```
5757

5858
## The `bld` directory
5959

@@ -70,7 +70,7 @@ The `pyproject.toml` file is the modern configuration file for most Python packa
7070
apps. It contains
7171

7272
1. the configuration for our Python package.
73-
2. pytask's configuration.
73+
1. pytask's configuration.
7474

7575
Let us start with the configuration of the Python package, which contains general
7676
information about the package, like its name and version, the definition of the package
@@ -93,10 +93,10 @@ where = ["src"]
9393
namespaces = false
9494
```
9595

96-
:::{seealso}
96+
```{seealso}
9797
You can find more extensive information about this metadata in the documentation of
9898
[setuptools](https://setuptools.pypa.io/en/latest/userguide/quickstart.html).
99-
:::
99+
```
100100

101101
Alongside the package information, we include pytask's configuration under the
102102
`[tool.pytask.ini_options]` section. We only tell pytask to look for tasks in
@@ -153,12 +153,12 @@ mode and it means any changes in the package's source files are immediately avai
153153
the installed version. Again, setuptools makes
154154
[a good job explaining it](https://setuptools.pypa.io/en/latest/userguide/development_mode.html).
155155

156-
:::{important}
156+
```{important}
157157
Do not forget to rerun the editable install every time after you recreate your Python
158158
environment.
159159
160160
Also, do not mix it up with the regular installation command `pip install .` because it
161161
will likely not work. Then, paths defined with the variables `SRC` and `BLD` in
162162
`config.py` will look for files relative to the location where your package is installed
163163
like `/home/user/miniconda3/envs/...` and not in the folder you are working in.
164-
:::
164+
```

‎docs/source/tutorials/using_a_data_catalog.md

+24-24
Original file line numberDiff line numberDiff line change
@@ -56,9 +56,9 @@ create a {class}`~pytask.PickleNode` that allows you to save any Python object t
5656

5757
The following tabs show you how to use the data catalog given the interface you prefer.
5858

59-
::::{tab-set}
59+
`````{tab-set}
6060
61-
:::{tab-item} Python 3.10+
61+
````{tab-item} Python 3.10+
6262
:sync: python310plus
6363
6464
Use `data_catalog["key"]` as an default argument to access the
@@ -69,9 +69,9 @@ Use `data_catalog["key"]` as an default argument to access the
6969
:emphasize-lines: 11, 22
7070
```
7171
72-
:::
72+
````
7373
74-
:::{tab-item} Python 3.8+
74+
````{tab-item} Python 3.8+
7575
:sync: python38plus
7676
7777
Use `data_catalog["key"]` as an default argument to access the
@@ -82,9 +82,9 @@ Use `data_catalog["key"]` as an default argument to access the
8282
:emphasize-lines: 10, 21
8383
```
8484
85-
:::
85+
````
8686
87-
:::{tab-item} ​`produces`
87+
````{tab-item} ​produces
8888
:sync: produces
8989
9090
Use `data_catalog["key"]` as an default argument to access the
@@ -95,9 +95,9 @@ Use `data_catalog["key"]` as an default argument to access the
9595
:emphasize-lines: 7, 17
9696
```
9797
98-
:::
98+
````
9999
100-
:::{tab-item} ​Python 3.10+ & Return
100+
````{tab-item} ​Python 3.10+ & Return
101101
:sync: return
102102
103103
An elegant way to use the data catalog is via return type annotations. Add
@@ -111,18 +111,18 @@ You can read more about return type annotations in
111111
:emphasize-lines: 8, 17
112112
```
113113
114-
:::
115-
::::
114+
````
115+
`````
116116

117117
## `task_plot_data`
118118

119119
Next, we will define the second task that consumes the data set from the previous task.
120120
Following one of the interfaces gives you immediate access to the
121121
{class}`~pandas.DataFrame` in the task without any additional line to load it.
122122

123-
::::{tab-set}
123+
`````{tab-set}
124124
125-
:::{tab-item} Python 3.10+
125+
````{tab-item} Python 3.10+
126126
:sync: python310plus
127127
128128
Use `data_catalog["key"]` as an default argument to access the
@@ -133,9 +133,9 @@ Use `data_catalog["key"]` as an default argument to access the
133133
:emphasize-lines: 12
134134
```
135135
136-
:::
136+
````
137137
138-
:::{tab-item} Python 3.8+
138+
````{tab-item} Python 3.8+
139139
:sync: python38plus
140140
141141
Use `data_catalog["key"]` as an default argument to access the
@@ -146,8 +146,8 @@ Use `data_catalog["key"]` as an default argument to access the
146146
:emphasize-lines: 12
147147
```
148148
149-
:::
150-
::::
149+
````
150+
`````
151151

152152
Finally, let's execute the two tasks.
153153

@@ -192,35 +192,35 @@ The path can be absolute or relative to the module of the data catalog.
192192
You can now use the data catalog as in previous example and use the
193193
{class}`~~pathlib.Path` in the task.
194194

195-
::::{tab-set}
195+
`````{tab-set}
196196
197-
:::{tab-item} Python 3.10+
197+
````{tab-item} Python 3.10+
198198
:sync: python310plus
199199
200200
```{literalinclude} ../../../docs_src/tutorials/using_a_data_catalog_5_py310.py
201201
:emphasize-lines: 11, 12
202202
```
203203
204-
:::
204+
````
205205
206-
:::{tab-item} Python 3.8+
206+
````{tab-item} Python 3.8+
207207
:sync: python38plus
208208
209209
```{literalinclude} ../../../docs_src/tutorials/using_a_data_catalog_5_py38.py
210210
:emphasize-lines: 11, 12
211211
```
212212
213-
:::
213+
````
214214
215-
:::{tab-item} ​Python 3.10+ & Return
215+
````{tab-item} ​Python 3.10+ & Return
216216
:sync: return
217217
218218
```{literalinclude} ../../../docs_src/tutorials/using_a_data_catalog_5_py310_return.py
219219
:emphasize-lines: 9, 10
220220
```
221221
222-
:::
223-
::::
222+
````
223+
`````
224224

225225
## Developing with the `DataCatalog`
226226

‎docs/source/tutorials/write_a_task.md

+25-25
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,8 @@ Starting from the project structure in the {doc}`previous tutorial <set_up_a_pro
44
you will learn how to write your first task.
55

66
The task, `task_create_random_data`, will be defined in
7-
`src/my_project/task_data_preparation.py`, and it will generate a data set stored
8-
in `bld/data.pkl`.
7+
`src/my_project/task_data_preparation.py`, and it will generate a data set stored in
8+
`bld/data.pkl`.
99

1010
The `task_` prefix for modules and task functions is important so that pytask
1111
automatically discovers them.
@@ -34,57 +34,57 @@ The following interfaces are different ways to specify the products of a task wh
3434
necessary for pytask to correctly run a workflow. The interfaces are ordered from most
3535
(left) to least recommended (right).
3636

37-
:::{important}
37+
```{important}
3838
You cannot mix different interfaces for the same task. Choose only one.
39-
:::
39+
```
4040

41-
:::::{tab-set}
41+
`````{tab-set}
4242
43-
::::{tab-item} Python 3.10+
43+
````{tab-item} Python 3.10+
4444
4545
The task accepts the argument `path` that points to the file where the data set will be
4646
stored. The path is passed to the task via the default value, `BLD / "data.pkl"`. To
4747
indicate that this file is a product we add some metadata to the argument.
4848
4949
Look at the type hint `Annotated[Path, Product]`. It uses the
5050
{obj}`~typing.Annotated` syntax. The first entry is the type of the argument,
51-
{class}`~pathlib.Path`. The second entry is {class}`pytask.Product` that marks this
51+
{class}`~pathlib.Path`. The second entry is {class}`~pytask.Product` that marks this
5252
argument as a product.
5353
5454
```{literalinclude} ../../../docs_src/tutorials/write_a_task_py310.py
5555
:emphasize-lines: 3, 11
5656
```
5757
58-
:::{tip}
58+
```{tip}
5959
If you want to refresh your knowledge about type hints, read
6060
[this guide](../type_hints.md).
61-
:::
61+
```
6262
63-
::::
63+
````
6464
65-
::::{tab-item} Python 3.8+
65+
````{tab-item} Python 3.8+
6666
6767
The task accepts the argument `path` that points to the file where the data set will be
6868
stored. The path is passed to the task via the default value, `BLD / "data.pkl"`. To
6969
indicate that this file is a product we add some metadata to the argument.
7070
7171
Look at the type hint `Annotated[Path, Product]`. It uses the
7272
{obj}`~typing.Annotated` syntax. The first entry is the type of the argument,
73-
{class}`~pathlib.Path`. The second entry is {class}`pytask.Product` that marks this
73+
{class}`~pathlib.Path`. The second entry is {class}`~pytask.Product` that marks this
7474
argument as a product.
7575
7676
```{literalinclude} ../../../docs_src/tutorials/write_a_task_py38.py
7777
:emphasize-lines: 8, 11
7878
```
7979
80-
:::{tip}
80+
```{tip}
8181
If you want to refresh your knowledge about type hints, read
8282
[this guide](../type_hints.md).
83-
:::
83+
```
8484
85-
::::
85+
````
8686
87-
::::{tab-item} &#8203;`produces`
87+
````{tab-item} produces
8888
8989
Tasks can use `produces` as an argument name. Every value, or in this case path, passed
9090
to this argument is automatically treated as a task product. Here, the path is given by
@@ -94,9 +94,9 @@ the default value of the argument.
9494
:emphasize-lines: 9
9595
```
9696
97-
::::
97+
````
9898
99-
::::{tab-item} Decorators
99+
````{tab-item} Decorators
100100
101101
```{warning}
102102
This approach is deprecated and will be removed in v0.5
@@ -113,8 +113,8 @@ an argument name to use the path inside the task function.
113113
To let pytask track the product of the task, you need to use the
114114
{func}`@pytask.mark.produces <pytask.mark.produces>` decorator.
115115
116-
::::
117-
:::::
116+
````
117+
`````
118118

119119
Now, execute pytask to collect tasks in the current and subsequent directories.
120120

@@ -125,9 +125,9 @@ Now, execute pytask to collect tasks in the current and subsequent directories.
125125

126126
## Customize task names
127127

128-
Use the {func}`@task <pytask.task>` decorator to mark a function as a
129-
task regardless of its function name. You can optionally pass a new name for the task.
130-
Otherwise, pytask uses the function name.
128+
Use the {func}`@task <pytask.task>` decorator to mark a function as a task regardless of
129+
its function name. You can optionally pass a new name for the task. Otherwise, pytask
130+
uses the function name.
131131

132132
```python
133133
from pytask import task
@@ -148,10 +148,10 @@ def create_random_data():
148148
...
149149
```
150150

151-
:::{warning}
151+
```{warning}
152152
Since v0.4 users should use {func}`@task <pytask.task>` over
153153
{func}`@pytask.mark.task <pytask.mark.task>` which will be removed in v0.5.
154-
:::
154+
```
155155

156156
## Customize task module names
157157

‎pyproject.toml

+4
Original file line numberDiff line numberDiff line change
@@ -198,3 +198,7 @@ exclude_also = [
198198
"\\.\\.\\.",
199199
"def __repr__",
200200
]
201+
202+
[tool.mdformat]
203+
wrap = 88
204+
end_of_line = "keep"

‎src/_pytask/nodes.py

+2-4
Original file line numberDiff line numberDiff line change
@@ -221,12 +221,10 @@ class PythonNode(PNode):
221221
objects.
222222
node_info
223223
The infos acquired while collecting the node.
224-
signature
225-
The signature of the node.
226224
227225
Examples
228226
--------
229-
To allow a :class:`~pytask.PythonNode` to hash a dictionary, you need to pass your
227+
To allow a :class:`PythonNode` to hash a dictionary, you need to pass your
230228
own hashing function. For example, from the :mod:`deepdiff` library.
231229
232230
>>> from deepdiff import DeepHash
@@ -293,7 +291,7 @@ def state(self) -> str | None:
293291

294292

295293
@define
296-
class PickleNode:
294+
class PickleNode(PPathNode):
297295
"""A node for pickle files.
298296
299297
Attributes

0 commit comments

Comments
 (0)
Please sign in to comment.