Merge branch 'main' into uv

Author: tobiasraabe

docs/source/changes.md

@@ -24,6 +24,21 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
 - {pull}`571` removes redundant calls to `PNode.state()` which causes a high penalty for
   remote files.
 - {pull}`573` removes the `pytask_execute_create_scheduler` hook.
+- {pull}`579` fixes an interaction with `--pdb` and `--trace` and task that return. The
+  debugging modes swallowed the return and `None` was returned. Closes {issue}`574`.
+- {pull}`581` simplifies the code for tracebacks and unpublishes some utility functions.
+- {pull}`586` improves linting.
+- {pull}`587` improves typing of `capture.py`.
+- {pull}`588` resets class variables of `ExecutionReport` and `Traceback`.
+- {pull}`590` fixes an error introduced in {pull}`588`.
+- {pull}`591` invalidates the cache of fsspec when checking whether a remote file
+  exists. Otherwise, a remote file might be reported as missing although it was just
+  created. See https://github.com/fsspec/s3fs/issues/851 for more info.
+
+## 0.4.6 - 2024-03-13
+
+- {pull}`576` fixes accidentally collecting `pytask.MarkGenerator` when using
+  `from pytask import mark`.
 
 ## 0.4.5 - 2024-01-09
 
@@ -66,6 +81,7 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
 - {pull}`485` adds missing steps to unconfigure pytask after the job is done, which
   caused flaky tests.
 - {pull}`486` adds default names to {class}`~pytask.PPathNode`.
+- {pull}`487` implements task generators and provisional nodes.
 - {pull}`488` raises an error when an invalid value is used in a return annotation.
 - {pull}`489` and {pull}`491` simplifies parsing products and does not raise an error
   when a product annotation is used with the argument name `produces`. And allow

docs/source/how_to_guides/index.md

@@ -19,6 +19,7 @@ capture_warnings
 how_to_influence_build_order
 hashing_inputs_of_tasks
 using_task_returns
+provisional_nodes_and_task_generators
 writing_custom_nodes
 extending_pytask
 the_data_catalog

docs/source/how_to_guides/provisional_nodes_and_task_generators.md

@@ -0,0 +1,94 @@
+# Provisional nodes and task generators
+
+pytask's execution model can usually be separated into three phases.
+
+1. Collection of tasks, dependencies, and products.
+1. Building the DAG.
+1. Executing the tasks.
+
+But, in some situations, pytask needs to be more flexible.
+
+Imagine you want to download a folder with files from an online storage. Before the task
+is completed you do not know the total number of files or their filenames. How can you
+still describe the files as products of the task?
+
+And how would you define another task that depends on these files?
+
+The following sections will explain how you use pytask in these situations.
+
+## Producing provisional nodes
+
+As an example for the aforementioned scenario, let us write a task that downloads all
+files without a file extension from the root folder of the pytask GitHub repository. The
+files are downloaded to a folder called `downloads`. `downloads` is in the same folder
+as the task module because it is a relative path.
+
+```{literalinclude} ../../../docs_src/how_to_guides/provisional_products.py
+---
+emphasize-lines: 4, 22
+---
+```
+
+Since the names of the files are not known when pytask is started, we need to use a
+{class}`~pytask.DirectoryNode` to define the task's product. With a
+{class}`~pytask.DirectoryNode` we can specify where pytask can find the files. The files
+are described with a root path (default is the directory of the task module) and a glob
+pattern (default is `*`).
+
+When we use the {class}`~pytask.DirectoryNode` as a product annotation, we get access to
+the `root_dir` as a {class}`~pathlib.Path` object inside the function, which allows us
+to store the files.
+
+```{note}
+The {class}`~pytask.DirectoryNode` is a provisional node that implements
+{class}`~pytask.PProvisionalNode`. A provisional node is not a {class}`~pytask.PNode`,
+but when its {meth}`~pytask.PProvisionalNode.collect` method is called, it returns
+actual nodes. A {class}`~pytask.DirectoryNode`, for example, returns
+{class}`~pytask.PathNode`.
+```
+
+## Depending on provisional nodes
+
+In the next step, we want to define a task that consumes and merges all previously
+downloaded files into one file.
+
+The difficulty here is how can we reference the downloaded files before they have been
+downloaded.
+
+```{literalinclude} ../../../docs_src/how_to_guides/provisional_task.py
+---
+emphasize-lines: 9
+---
+```
+
+To reference the files that will be downloaded, we use the
+{class}`~pytask.DirectoryNode` is a dependency. Before the task is executed, the list of
+files in the folder defined by the root path and the pattern are automatically collected
+and passed to the task.
+
+If we use a {class}`~pytask.DirectoryNode` with the same `root_dir` and `pattern` in
+both tasks, pytask will automatically recognize that the second task depends on the
+first. If that is not true, you might need to make this dependency more explicit by
+using {func}`@task(after=...) <pytask.task>`, which is explained {ref}`here <after>`.
+
+## Task generators
+
+What if we wanted to process each downloaded file separately instead of dealing with
+them in one task?
+
+For that, we have to write a task generator to define an unknown number of tasks for an
+unknown number of downloaded files.
+
+A task generator is a task function in which we define more tasks, just as if we were
+writing functions in a task module.
+
+The code snippet shows each task takes one of the downloaded files and copies its
+content to a `.txt` file.
+
+```{literalinclude} ../../../docs_src/how_to_guides/provisional_task_generator.py
+```
+
+```{important}
+The generated tasks need to be decoratored with {func}`@task <pytask.task>` to be
+collected.
+```

docs/source/reference_guides/api.md

@@ -190,6 +190,8 @@ Protocols define how tasks and nodes for dependencies and products have to be se
    :show-inheritance:
 .. autoprotocol:: pytask.PTaskWithPath
    :show-inheritance:
+.. autoprotocol:: pytask.PProvisionalNode
+   :show-inheritance:
 ```
 
 ## Nodes
@@ -203,6 +205,8 @@ Nodes are the interface for different kinds of dependencies or products.
    :members:
 .. autoclass:: pytask.PythonNode
    :members:
+.. autoclass:: pytask.DirectoryNode
+   :members:
 ```
 
 To parse dependencies and products from nodes, use the following functions.
@@ -330,7 +334,6 @@ resolution and execution.
 ## Tracebacks
 
 ```{eval-rst}
-.. autofunction:: pytask.remove_internal_traceback_frames_from_exc_info
 .. autoclass:: pytask.Traceback
 ```
 

docs/source/tutorials/skipping_tasks.md

@@ -13,18 +13,18 @@ skip tasks during development that take too much time to compute right now.
 ```{literalinclude} ../../../docs_src/tutorials/skipping_tasks_example_1.py
 ```
 
-Not only will this task be skipped, but all tasks that depend on
+Not only will this task be skipped, but all tasks depending on
 `time_intensive_product.pkl`.
 
 ## Conditional skipping
 
 In large projects, you may have many long-running tasks that you only want to execute on
-a remote server but not when you are not working locally.
+a remote server, but not when you are not working locally.
 
 In this case, use the {func}`@pytask.mark.skipif <pytask.mark.skipif>` decorator, which
 requires a condition and a reason as arguments.
 
-Place the condition variable in a different module than the task, so you can change it
+Place the condition variable in a module different from the task so you can change it
 without causing a rerun of the task.
 
 ```python

docs_src/how_to_guides/provisional_products.py

@@ -0,0 +1,33 @@
+from pathlib import Path
+
+import httpx
+from pytask import DirectoryNode
+from pytask import Product
+from typing_extensions import Annotated
+
+
+def get_files_without_file_extensions_from_repo() -> list[str]:
+    url = "https://api.github.com/repos/pytask-dev/pytask/git/trees/main"
+    response = httpx.get(url)
+    elements = response.json()["tree"]
+    return [
+        e["path"]
+        for e in elements
+        if e["type"] == "blob" and Path(e["path"]).suffix == ""
+    ]
+
+
+def task_download_files(
+    download_folder: Annotated[
+        Path, DirectoryNode(root_dir=Path("downloads"), pattern="*"), Product
+    ],
+) -> None:
+    """Download files."""
+    # Contains names like CITATION or LICENSE.
+    files_to_download = get_files_without_file_extensions_from_repo()
+
+    for file_ in files_to_download:
+        url = "raw.githubusercontent.com/pytask-dev/pytask/main"
+        response = httpx.get(url=f"{url}/{file_}", timeout=5)
+        content = response.text
+        download_folder.joinpath(file_).write_text(content)

docs_src/how_to_guides/provisional_task.py

@@ -0,0 +1,14 @@
+from pathlib import Path
+
+from pytask import DirectoryNode
+from typing_extensions import Annotated
+
+
+def task_merge_files(
+    paths: Annotated[
+        list[Path], DirectoryNode(root_dir=Path("downloads"), pattern="*")
+    ],
+) -> Annotated[str, Path("all_text.txt")]:
+    """Merge files."""
+    contents = [path.read_text() for path in paths]
+    return "\n".join(contents)

docs_src/how_to_guides/provisional_task_generator.py

@@ -0,0 +1,21 @@
+from pathlib import Path
+
+from pytask import DirectoryNode
+from pytask import task
+from typing_extensions import Annotated
+
+
+@task(is_generator=True)
+def task_copy_files(
+    paths: Annotated[
+        list[Path], DirectoryNode(root_dir=Path("downloads"), pattern="*")
+    ],
+) -> None:
+    """Create tasks to copy each file to a ``.txt`` file."""
+    for path in paths:
+        # The path of the copy will be CITATION.txt, for example.
+        path_to_copy = path.with_suffix(".txt")
+
+        @task
+        def copy_file(path: Annotated[Path, path]) -> Annotated[str, path_to_copy]:
+            return path.read_text()

pyproject.toml

@@ -112,14 +112,12 @@ extend-include = ["*.ipynb"]
 [tool.ruff.lint]
 select = ["ALL"]
 ignore = [
-    "FBT",  # flake8-boolean-trap
-    "TRY",  # ignore tryceratops.
-    # Others.
-    "ANN101",  # type annotating self
-    "ANN102",  # type annotating cls
+    "ANN101",
+    "ANN102",
     "ANN401",  # flake8-annotate typing.Any
     "COM812",  # Comply with ruff-format.
     "ISC001",  # Comply with ruff-format.
+    "FBT",
     "PD901",  # Avoid generic df for dataframes.
     "S101",  # raise errors for asserts.
     "S603",  # Call check with subprocess.run.

scripts/update_plugin_list.py

@@ -70,7 +70,7 @@
 )
 
 
-_EXCLUDED_PACKAGES = ["pytask-io"]
+_EXCLUDED_PACKAGES = ["pytask-io", "pytask-list"]
 
 
 def _escape_rst(text: str) -> str:

src/_pytask/_inspect.py

@@ -106,7 +106,7 @@ def get_annotations(  # noqa: C901, PLR0912, PLR0915
 
         if not isinstance(ann, dict):
             msg = f"{obj!r}.__annotations__ is neither a dict nor None"
-            raise ValueError(msg)
+            raise ValueError(msg)  # noqa: TRY004
 
         if not ann:
             return {}

src/_pytask/build.py

@@ -13,7 +13,6 @@
 from typing import Literal
 
 import click
-from rich.traceback import Traceback
 
 from _pytask.capture_utils import CaptureMethod
 from _pytask.capture_utils import ShowCapture
@@ -34,7 +33,7 @@
 from _pytask.session import Session
 from _pytask.shared import parse_paths
 from _pytask.shared import to_list
-from _pytask.traceback import remove_internal_traceback_frames_from_exc_info
+from _pytask.traceback import Traceback
 
 if TYPE_CHECKING:
     from typing import NoReturn
@@ -66,7 +65,7 @@ def pytask_unconfigure(session: Session) -> None:
     path.write_text(json.dumps(HashPathCache._cache))
 
 
-def build(  # noqa: C901, PLR0912, PLR0913, PLR0915
+def build(  # noqa: C901, PLR0912, PLR0913
     *,
     capture: Literal["fd", "no", "sys", "tee-sys"] | CaptureMethod = CaptureMethod.FD,
     check_casing_of_paths: bool = True,
@@ -257,9 +256,7 @@ def build(  # noqa: C901, PLR0912, PLR0913, PLR0915
         session = Session.from_config(config_)
 
     except (ConfigurationError, Exception):
-        exc_info = remove_internal_traceback_frames_from_exc_info(sys.exc_info())
-        traceback = Traceback.from_exception(*exc_info)
-        console.print(traceback)
+        console.print(Traceback(sys.exc_info()))
         session = Session(exit_code=ExitCode.CONFIGURATION_FAILED)
 
     else:
@@ -279,9 +276,7 @@ def build(  # noqa: C901, PLR0912, PLR0913, PLR0915
             session.exit_code = ExitCode.FAILED
 
         except Exception:  # noqa: BLE001
-            exc_info = remove_internal_traceback_frames_from_exc_info(sys.exc_info())
-            traceback = Traceback.from_exception(*exc_info)
-            console.print(traceback)
+            console.print(Traceback(sys.exc_info()))
             session.exit_code = ExitCode.FAILED
 
         session.hook.pytask_unconfigure(session=session)