A new interface for pytask

[tobiasraabe]

Introduction

pytask will become three years old soon 🥳. What better way to celebrate this birthday than with an open discussion about pytask's interface and maybe a revision of significant components?

We should agree on some principles to guide the design decisions in this discussion.

• pytask has a lot of users that are beginners in programming. For them, the interface should be intuitive and use familiar language constructs. Also, changes need to be justified since we do not want to alienate existing users.
• pytask should also appeal to Python experts and power users that want to use features like type checking.
• Compromises may be possible by supporting a couple of interfaces.

Each section will discuss one detail, and sections might build up on each other.

The problem with depends_on and produces

• depends_on and produces are magic keywords the users need to know about. They must learn to use decorators that inject the correct values at runtime.
• depends_on and produces obfuscate what's behind the values. It can be a path, a dictionary of paths, or a list of paths. Just look at these two signatures; what example is clearer?

from pathlib import Path


@pytask.mark.depends_on({"a": Path(...), "b": Path(...)})
@pytask.mark.produces(Path(...))
def task_example(depends_on:dict[str, Path], produces: Path) -> None:
    depends_on["a"], depends["b"]


@pytask.mark.produces(Path(...))
def task_example(a: Path = Path(...), b: Path = Path(...), produces: Path) -> None:
    ...

Getting rid of the decorators

We can eliminate the decorators by parsing depends_on and produces from the default arguments of a task function. This is already possible if a task is marked with @pytask.mark.task.

Getting rid of depends_on

As you saw in the example before and here, depends_on is not necessary nor clearer. The values can be better attached as default arguments.

@pytask.mark.produces(Path(...))
def task_example(a: Path = Path(...), b: Path = Path(...), produces: Path) -> None:
    ...

This interface is not new. The task decorator already allows exploiting default arguments and some extra features (no task_ prefix necessary, passing kwargs).

@pytask.mark.task
def task_example(a: Path = Path(...), b: Path = Path(...), produces: Path = Path(...)) -> None:
    ...

But, pytask currently only looks for path dependencies in depends_on. The implementation could be changed without probably causing many problems to treat every task input except produces as a pytree and when a pathlib.Path is encountered, it is parsed as a dependency.

str should probably not be supported outside of depends_on. Using str in depends_on should be deprecated as well for keeping everything simpler.

Since pathlib.Paths are always assumed to be PathNodes, how can I pass a pathlib.Path as a normal function argument? Just wrap the path in a PythonNode, the neutral element for pytask. A PythonNode is explained in the next section.

Possible changes

The following are non-breaking changes.

• If depends_on is present, do not change behavior. Also, strings are still allowed instead of paths.
• If depends_on is not present, try to parse from all arguments of the task function that are not produces and only if the value in the pytree is a pathlib.Path. Strings as paths are not allowed anymore.

Allowing for more dependency and product types.

Currently, pytask builds the DAG only from paths, and it cannot handle different objects, significantly limiting users.

To add a new type like a PythonNode, the user creates a new class that inherits from MetaNode.

The metaclass requires a new type to implement

• a state property. The state is a hash or some almost unique value signaling changes in the node's value. It is None if the node does not exist.
• a value property to retrieve the value from the node and pass it inside the task.

from abc import ABCMeta


class Node(metaclass=ABCMeta):

    @property
    @abstractmethod
    def state(self):
        pass

    @property
    @abstractmethod
    def value(self):
        pass

The Python node could then be

from attrs import define


@define
class PythonNode(Node):
    hash: bool = False
    value: Any = None

    @property
    def state(self) -> Any:
        if self.hash:
            return hash(self.value)
        return None

    @property
    def value(self) -> Any:
        return self.value

Hashing Python inputs

What is a usecase for this new PythonNode?

Currently, python inputs are not processed as dependencies. If they alter the signature of the task, changing them might trigger a rerun of the task, but it is not a given.

Here, we declare that the task receives additional_kwargs. Internally, pytask will treat them in a PythonNode container. We say the dictionary should be hashed to detect changes.

(We do not want hashing by default since some inputs can be substantial and hashing too costly.)

from pytask import PythonNode
from pathlib import Path


def task_example(
    path_to_csv: Path = Path(...),
    additional_kwargs: PythonNode = PythonNode({"a": 1, "b": 2}, hash=True),
    produces: Path = Path(...),
) -> None:
    ...

Clearer types

The previous example introduced PythonNodes and how they can be used to declare and hash Python dependencies.

A problem with the previous example is that types of function inputs are obfuscated by pytask's node types and do not reveal the original types anymore. Especially, for testing and other purposes it would be advantageous if you can call the function without any knowledge about pytask.

Luckily, a new Python feature recently dropped, typing.Annotated, was introduced in PEP 593. It allows adding arbitrary metadata to type hints.

The previous example can be rewritten to

from pytask import PythonNode
from typing import Annotated
from pathlib import Path


def task_example(
    path_to_csv: Path = Path(...),
    additional_kwargs: Annotated[dict[str, int], PythonNode(hash=True)] = {"a": 1, "b": 2},
    produces: Path = Path(...),
) -> None:
    ...

The example is heavily inspired by FastAPI, which adopted PEP 593 recently. Here are some links from the docs:

• https://fastapi.tiangolo.com/python-types/#type-hints-with-metadata-annotations
• https://fastapi.tiangolo.com/tutorial/query-params-str-validations/#advantages-of-annotated

A general concern is that users are unfamiliar with type hints, so any interface built on top of that asks a lot from users.

How to get rid of (path) products?

The solution to replace products is less straightforward than for dependencies. Currently, there are two ways to declare products. First, continue using @pytask.mark.produces or using only produces as a magical argument of the task function which already removes the decorator.

If we want to declare products independent from the produces argument name, we could use our nodes with a new argument.

from pytask import PathNode
from pathlib import Path
from pytask import Product


def task_example(
    path_to_csv: Path = Path(...),
    additional_kwargs: dict[str, int] = {"a": 1, "b": 2},
    path_to_picke: Path = PathNode(Path(...), product=True),
) -> None:
    ...

# or

def task_example(
    ...,
    path_to_picke: Annotated[Path, PathNode(product=True)] = Path(...),
) -> None:
    ...

[axtimhaus]

Sounds basically well to me.

Python Inputs

I often encountered the problem of tasks not running after changing some config values, ended up in adding a config.py with all that constants as a dep. Unfortunately, this often triggers more tasks to run than needed. So this change would be pleasant.

Get Rid of depends_on and produces

Would also be fine, especially when dealing with multiple deps and products it would be fine to have distinct arguments, which allows for easier typing due to syntax completion (avoids dict keys).

Also the old way often requires redundant typing of parameter names, which is avoided by the new one.

A suggestion: to make products easier to type, use a wrapper function instead of the class constructor (which calls the constructor internally with the respective arguments). Perhaps, this may be omitted for deps. But requiring explicit dependencies would allow to pass in parameters without "depending" on them.

def task_example(
    dep1: Path = dependency(Path(...)), # explicit dep
    dep2: Path = Path(...), # implicit dep
    prod1: Path = product(Path(...)), # explicit product
) -> None:
    ...

Additional Suggestions

Recently, I encountered the problem of accessing the tasks metadata from within the function (f.e. the id). I would appreciate, if it were possible to obtain the Task instance or some of its properties (like id) as a magic function argument.

for i in ["a", "b", "c"]:
    @pytask.mark.task(id=i)
    def task_example(
        id: str,
        ...
    ):
        print(id) # silly example

[tobiasraabe]

Hi @axtimhaus, thanks for your input, and sorry for my delayed answer.

Get Rid of depends_on and produces

I think I see where your proposal is more elegant than mine, but correct me if I am misinterpreting what you meant.

So, instead of knowing about PathNode and other possible node types, the user only needs to know about the wrapper functions. And, internally, I imagine there is something like a type dispatch that finds the right node type, right?

I find it more elegant. There could also be a type registry where users can register their custom nodes following some interface for nodes.

Additional Suggestions

What kind of metadata are you missing? There are multiple solutions to your id problem, but maybe you are looking for something else. Here are the possibilities and the "interface creep" that I also want to get rid of:

As a default argument

for i in ["a", "b", "c"]:
    @pytask.mark.task(id=i)
    def task_example(
        id: str = i,
        ...
    ):
        ...

As kwargs

for i in ["a", "b", "c"]:
    @pytask.mark.task(id=i, kwargs={"i": i})
    def task_example(
        id: str,
        ...
    ):
        ...

Of course, there could be something like a context object that you can get within the task function by using the argument context. But, since it was not needed, I do not want to add another magic keyword.

[axtimhaus]

Get Rid of depends_on and produces

Yes something like that. The type registry could be done with the already present plugin system.

Additional Suggestions

Yes, that is the way I have done it in the end, but this is a bit redundant. However, I understand if you do not want to pollute the magic namespace. It is a very rare need so far.

But in general, also the old way provided me a very useful tool. My criticism is just for simplification and optimization of workflow. I use pytask excessively in my scientific work. Almost everything is running within pytask: data collection and processing, simulation routines, graphics and LaTeX document generation. So just type pytask and a journal paper drops, awesome.