Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement task generators and provisional nodes. #487

Merged
merged 109 commits into from
Mar 15, 2024
Merged
Changes from 1 commit
Commits
Show all changes
109 commits
Select commit Hold shift + click to select a range
2cd96d7
Simplify TopologicalSorter.
tobiasraabe Oct 21, 2023
6f2aaef
Remove static_order.
tobiasraabe Oct 21, 2023
e37abd6
Remove backup.
tobiasraabe Oct 21, 2023
789e0d8
Add instantiation from another sorter and a DAG.
tobiasraabe Oct 22, 2023
b196584
Add a delayed node that can serve as a product.
tobiasraabe Oct 23, 2023
7bbb5b8
Raise errors when not delayed tasks receive delayed dependencies.
tobiasraabe Oct 23, 2023
af7eb2c
Everything works except tasks depending on delayed nodes.
tobiasraabe Oct 23, 2023
18318a3
Fix.
tobiasraabe Oct 23, 2023
5dd1be1
Merge branch 'main' into dag
tobiasraabe Oct 23, 2023
0436b3d
Merge branch 'dag' into delayed-nodes
tobiasraabe Oct 23, 2023
b8a16c4
fix.
tobiasraabe Oct 23, 2023
c96d58d
Add draft.
tobiasraabe Oct 25, 2023
7433efb
Merge branch 'main' into delayed-nodes
tobiasraabe Nov 8, 2023
ecb2e60
fix.
tobiasraabe Nov 8, 2023
cb32db1
Fix.
tobiasraabe Nov 8, 2023
61bda5c
Merge branch 'dag' into delayed-nodes
tobiasraabe Nov 8, 2023
41c76cf
Temp.
tobiasraabe Nov 8, 2023
918557a
Merge branch 'main' into delayed-nodes
tobiasraabe Nov 9, 2023
e71bc51
Allow executing tasks with delayednodes.
tobiasraabe Nov 9, 2023
3ca95df
Allow tasks to depend on delayed nodes.
tobiasraabe Nov 9, 2023
ca77a19
to changes.
tobiasraabe Nov 10, 2023
4337700
fix.
tobiasraabe Nov 10, 2023
c2db3c5
Fix.
tobiasraabe Nov 10, 2023
104aa5f
remove allow_delayed.
tobiasraabe Nov 10, 2023
43afe7b
Merge remote-tracking branch 'origin/main' into delayed-nodes
tobiasraabe Nov 10, 2023
9d70e3f
Fix test.
tobiasraabe Nov 10, 2023
0928a7e
extend test.
tobiasraabe Nov 10, 2023
a7ccb23
Merge branch 'main' into delayed-nodes
tobiasraabe Nov 10, 2023
f2d236b
Add new hook.
tobiasraabe Nov 11, 2023
fc60e00
Merge branch 'main' into delayed-nodes
tobiasraabe Nov 11, 2023
3b8ef38
fix.
tobiasraabe Nov 11, 2023
318a0b7
Merge branch 'main' into delayed-nodes
tobiasraabe Nov 11, 2023
948b743
Fixes.
tobiasraabe Nov 12, 2023
9d14eff
fix.
tobiasraabe Nov 12, 2023
701b46d
Allow tasks to depend on other tasks.
tobiasraabe Nov 13, 2023
ab9fd8d
Fix.
tobiasraabe Nov 13, 2023
8cf360d
Fix.
tobiasraabe Nov 13, 2023
9f61f92
Fix paths.
tobiasraabe Nov 13, 2023
bd0bb6f
Fix.
tobiasraabe Nov 14, 2023
413ffba
fix.
tobiasraabe Nov 14, 2023
fc9d40f
Merge remote-tracking branch 'origin/main' into allow-to-depend-on-taks
tobiasraabe Nov 14, 2023
649aa24
Fix.
tobiasraabe Nov 14, 2023
a013135
Merge branch 'allow-to-depend-on-taks' into delayed-nodes
tobiasraabe Nov 14, 2023
8353c7e
fix.
tobiasraabe Nov 14, 2023
e1cd75a
Fix.
tobiasraabe Nov 15, 2023
137144c
Fix.
tobiasraabe Nov 15, 2023
4903307
Fix.
tobiasraabe Nov 15, 2023
92e9990
Raise errors gracefully when recreating the DAG.
tobiasraabe Nov 15, 2023
71dd738
Fix.
tobiasraabe Nov 15, 2023
e078927
temp.
tobiasraabe Nov 15, 2023
640b465
Merge branch 'main' into allow-to-depend-on-taks
tobiasraabe Nov 16, 2023
880e95a
Merge branch 'allow-to-depend-on-taks' into delayed-nodes
tobiasraabe Nov 16, 2023
e7ab6ce
fix test.
tobiasraabe Nov 16, 2023
4ab7e0f
Get preliminary version of task generators to work.
tobiasraabe Nov 16, 2023
cabb531
Temp.
tobiasraabe Nov 16, 2023
3ab8bac
Temp.
tobiasraabe Nov 16, 2023
11b18ec
Merge branch 'main' into allow-to-depend-on-taks
tobiasraabe Nov 16, 2023
7fe2f27
Merge branch 'allow-to-depend-on-taks' into delayed-nodes
tobiasraabe Nov 16, 2023
7644b35
Fix.
tobiasraabe Nov 16, 2023
f4c78c6
Merge branch 'main' into delayed-nodes
tobiasraabe Nov 16, 2023
2bff286
Fix.
tobiasraabe Nov 17, 2023
c1c1e7a
Refine task generators even more.
tobiasraabe Nov 17, 2023
421db8e
fix some.
tobiasraabe Nov 18, 2023
1d9e85b
Add docs.
tobiasraabe Nov 22, 2023
ac51dcf
Merge remote-tracking branch 'origin/main' into delayed-nodes
tobiasraabe Nov 23, 2023
4da17c8
Fix.
tobiasraabe Nov 25, 2023
6b256d3
Merge branch 'main' into delayed-nodes
tobiasraabe Nov 25, 2023
f172992
Rename delayed nodes to provisional nodes.
tobiasraabe Nov 25, 2023
6cc4a28
Make PDelayedNode independent from PNode.
tobiasraabe Nov 25, 2023
bfd7d38
Rename DelayedPathNode to DirectoryNode.
tobiasraabe Nov 25, 2023
80bd1ed
Add test for data catalog.
tobiasraabe Nov 25, 2023
23c7362
Adjust more types.
tobiasraabe Nov 25, 2023
89e1014
FIx types.
tobiasraabe Nov 25, 2023
c5eb99e
Update docs.
tobiasraabe Nov 25, 2023
dd4ab17
Remove talking about delayed.
tobiasraabe Nov 25, 2023
89e0588
Merge branch 'main' into delayed-nodes
tobiasraabe Jan 28, 2024
432f782
update
tobiasraabe Jan 28, 2024
fe64020
Fix.
tobiasraabe Jan 28, 2024
e7ca69d
Fix.
tobiasraabe Jan 29, 2024
5b4cba2
Rename generator to is_generator.
tobiasraabe Jan 30, 2024
deb45d1
Fix side-effect in tests.
tobiasraabe Feb 4, 2024
86f227d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Feb 4, 2024
11e3484
Fix.
tobiasraabe Feb 4, 2024
ab47e5d
Fix.
tobiasraabe Feb 4, 2024
b683da5
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Feb 4, 2024
f56fa68
Fix docs and coverage.
tobiasraabe Feb 4, 2024
017ec9c
Merge remote-tracking branch 'refs/remotes/origin/delayed-nodes' into…
tobiasraabe Feb 4, 2024
240c47e
Add test for task generator in notebook.
tobiasraabe Feb 4, 2024
7d8829f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Feb 4, 2024
6f18094
Merge branch 'main' into delayed-nodes
tobiasraabe Feb 4, 2024
11f5647
Merge remote-tracking branch 'refs/remotes/origin/delayed-nodes' into…
tobiasraabe Feb 4, 2024
01a00c2
FIx.
tobiasraabe Feb 4, 2024
0bff197
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Feb 4, 2024
c3b6122
Fix.
tobiasraabe Feb 4, 2024
d76c716
no pragma.
tobiasraabe Feb 4, 2024
c001a9d
Remove ways to generate tasks based on returns.
tobiasraabe Feb 4, 2024
f80d3ef
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Feb 4, 2024
315f354
Fix.
tobiasraabe Feb 4, 2024
0d41ac6
Merge remote-tracking branch 'refs/remotes/origin/delayed-nodes' into…
tobiasraabe Feb 4, 2024
ebabd84
Merge branch 'main' into delayed-nodes
tobiasraabe Mar 9, 2024
ef42e6e
Fix.
tobiasraabe Mar 9, 2024
eacecf4
Change.
tobiasraabe Mar 9, 2024
43486d7
Fix.
tobiasraabe Mar 10, 2024
c3b5596
Improve docs.
tobiasraabe Mar 13, 2024
d218e5a
Merge branch 'main' into delayed-nodes
tobiasraabe Mar 13, 2024
696e931
Merge branch 'main' into delayed-nodes
tobiasraabe Mar 15, 2024
bf95f75
Fix.
tobiasraabe Mar 15, 2024
bf4fcbd
Merge branch 'delayed-nodes' of https://github.com/pytask-dev/pytask …
tobiasraabe Mar 15, 2024
6e5679c
Last fixes.
tobiasraabe Mar 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'main' into delayed-nodes
tobiasraabe committed Mar 15, 2024
commit 696e9311a8c4dfbf1e9355d986d96ff6ff1215fd
6 changes: 3 additions & 3 deletions src/_pytask/execute.py
Original file line number Diff line number Diff line change
@@ -140,8 +140,8 @@ def pytask_execute_task_setup(session: Session, task: PTask) -> None: # noqa: C
node = dag.nodes[node_signature].get("task") or dag.nodes[
node_signature
].get("node")

if node_signature in predecessors and not node.state():
node_state = node.state()
if node_signature in predecessors and not node_state:
msg = f"{task.name!r} requires missing node {node.name!r}."
if IS_FILE_SYSTEM_CASE_SENSITIVE:
msg += (
@@ -156,7 +156,7 @@ def pytask_execute_task_setup(session: Session, task: PTask) -> None: # noqa: C
):
continue

has_changed = has_node_changed(task=task, node=node)
has_changed = has_node_changed(task=task, node=node, state=node_state)
if has_changed:
needs_to_be_executed = True
break
2 changes: 1 addition & 1 deletion src/_pytask/nodes.py
Original file line number Diff line number Diff line change
@@ -369,7 +369,7 @@ def collect(self) -> list[Path]:
return list(self.root_dir.glob(self.pattern)) # type: ignore[union-attr]


def _get_state(path: Path) -> str:
def _get_state(path: Path) -> str | None:
"""Get state of a path.

A simple function to handle local and remote files.
You are viewing a condensed version of this merge commit. You can view the full changes here.