You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- dask's [`ClientExecutor`](https://distributed.dask.org/en/stable/api.html#distributed.Client.get_executor) allows in combination with [coiled](https://docs.coiled.io/user_guide/index.html) to spawn clusters and workers on AWS, GCP, and other providers with minimal configuration.
39
-
40
-
## Usage
41
-
42
-
To parallelize your tasks across many workers, pass an integer greater than 1 or
43
-
`'auto'` to the command-line interface.
44
-
45
-
```console
46
-
$ pytask -n 2
47
-
$ pytask --n-workers 2
48
-
49
-
# Starts os.cpu_count() - 1 workers.
50
-
$ pytask -n auto
51
-
```
52
-
53
-
Using processes to parallelize the execution of tasks is useful for CPU-bound tasks such
54
-
as numerical computations. ([Here](https://stackoverflow.com/a/868577/7523785) is an
55
-
explanation of what CPU- or IO-bound means.)
56
-
57
-
For IO-bound tasks, tasks where the limiting factor is network latency and access to
58
-
files, you can parallelize via threads.
59
-
60
-
```console
61
-
pytask --parallel-backend threads
62
-
```
63
-
64
-
You can also set the options in a `pyproject.toml`.
65
-
66
-
```toml
67
-
# This is the default configuration. Note that, parallelization is turned off.
68
-
69
-
[tool.pytask.ini_options]
70
-
n_workers = 1
71
-
parallel_backend = "loky"# or processes or threads
72
-
```
73
-
74
-
## Parallelization and Debugging
75
-
76
-
It is not possible to combine parallelization with debugging. That is why `--pdb` or
77
-
`--trace` deactivate parallelization.
78
-
79
-
If you parallelize the execution of your tasks using two or more workers, do not use
80
-
`breakpoint()` or `import pdb; pdb.set_trace()` since both will cause exceptions.
81
-
82
18
## Documentation
83
19
84
-
You find the documentation at <https://pytask-parallel.readthedocs.io/en/stable>.
20
+
You find the documentation at <https://pytask-parallel.readthedocs.io> with a
0 commit comments