feat[cartesian]: DaCe bridge refactor: OIR -> TreeIR -> ScheduleTree -> SDFG

docs/development/ADRs/cartesian/backend-cuda-feature-freeze.md

@@ -0,0 +1,17 @@
+# Cuda backend: Feature freeze
+
+In the context of (backend) feature development, facing maintainability/duplication concerns, we decided to put a feature freeze on the `cuda` backend and focus on the `dace:gpu` backends instead to keep the number of backends manageable.
+
+## Context
+
+The introduction of the [`dace:*`](./backend-dace.md) backends brought up the question of backend redundancy. In particular, it seems that `cuda` and `dace:gpu` backends serve similar purposes.
+
+`dace:gpu` backends not only generate code for different graphics cards, they also share substantial code paths with the `dace:cpu` backend. This simplifies (backend) feature development.
+
+## Decision
+
+We decided to put a feature freeze on the `cuda` backend, focusing on the `dace:*` backends instead. While we don't drop the backend, new DSL features won't be able in the `cuda` backend. New features will error out cleanly and suggest to use the `dace:gpu` backend instead.
+
+## Consequences
+
+While the `cuda` backend only targets NVIDIA cards, the `dace:*` backends allow to generate code for NVIDIA and AMD graphics cards. Furthermore, `dace:cpu` and `dace:gpu` backends share large parts of the transpilation layers because code generation is deferred to DaCe and only depending on the SDFG. This allows us to develop many (backend) features for the `dace:*` backends in one place.

docs/development/ADRs/cartesian/backend-dace-schedule-tree.md

@@ -0,0 +1,73 @@
+# DaCe backends: Schedule tree
+
+In the context of [DaCe backends](./backend-dace.md), facing tech-debt, a lack of understanding of the current stack, and under performing map- & state fusion, we decided to rewrite substantial parts of the DaCe backends with so called "Schedule Trees" to achieve hardware dependent macro-level optimizations (e.g. loop merging and loop re-ordering) at a new IR level before going down to SDFGs. We considered writing custom SDFG fusion passes and accept that we have to contribute a conversion from Schedule Tree to SDFG in DaCe.
+
+## Context
+
+Basically three forces were driving this drastic change:
+
+1. We were unhappy with the performance of the DaCe backends, especially on CPU.
+2. We had little understanding of the previous GT4Py-DaCe bridge.
+3. The previous GT4Py-DaCe bridge accumulated a lot of tech debt, making it clumsy to work with and hard to inject major changes.
+
+## Decision
+
+We chose to directly translate GT4Py's optimization IR (OIR) to DaCe's schedule tree (and from there to SDFG and code generation) because this allows to separate macro-level and data-specific optimizations. DaCe's schedule tree is ideally suited for schedule-level optimizations like loop re-ordering or loop merges with over-computation. The (simplified) pipeline looks like this:
+
+```mermaid
+flowchart LR
+oir["
+    OIR
+    (GT4Py)
+"]
+treeir["
+    Tree IR
+    (GT4Py)
+"]
+stree["
+    Schedule tree
+    (DaCe)
+"]
+sdfg["
+    SDFG
+    (DaCe)
+"]
+codegen["
+    Code generation
+    (per target)
+"]
+
+oir --> treeir --> stree --> sdfg --> codegen
+```
+
+OIR to Tree IR conversion has two visitors in separate files:
+
+1. `dace/oir_to_treeir.py` transpiles control flow elements
+2. `dace/oir_to_tasklet.py` transpiles computations (i.e. bodies of control flow elements)
+
+While this incurs a bit of code duplications (e.g. resolving index accesses), it allows for separation of concerns: Everything that is related to the schedule is handled in `oir_to_treeir.py`. Note, for example, that we keep the distinction between horizontal mask and general `if` statements. This distinction is kept because horizontal regions might influence scheduling decisions, while general `if` statements do not.
+
+The subsequent conversion from Tree IR to schedule tree is a straight forward visitor located in `dace/treeir_to_stree.py`. Notice the simplicity of that visitor.
+
+## Consequences
+
+The schedule tree introduces a transpilation layer ideally suited for macro-level optimizations, which are targeting the program's execution schedule. This is particularly interesting for the DaCe backends because we use the same backend pipeline to generate code for CPU and GPU targets.
+
+In particular, the schedule tree allows to easily re-order/modify/change the loop structure. This not only allows us to generate hardware-specific loop order and tile-sizes, but also gives us fine grained control over loop merges and/or which loops to generate in the first place. For example, going directly from OIR to Tree IR allows us to translate horizontal regions to either `if` statements inside a bigger horizontal loop (for small regions) or break them out into separate loops (for bigger regions) if that makes sense for the target architecture.
+
+## Alternatives considered
+
+### OIR -> SDFG -> schedule tree -> SDFG
+
+- Seems smart because it allows to keep the current OIR -> SDFG bridge, i.e. no need to write and OIR -> schedule tree bridge,
+- but the first SDFG is unnecessary and translation times are a real problem
+- and we were unhappy with the OIR -> SDFG bridge anyway
+- and ,in addition, we loose some context between OIR and schedule tree (e.g. horizontal regions).
+
+### Improve the existing SDFG map fusion
+
+GT4Py next has gone this route and an improved version is merged in the mainline version of DaCe. We think we'll need a custom map fusion pass which lets us decide low-level things like under which circumstances over-computation is desirable. A general map fusion pass will never be able to allow this.
+
+### Write custom map fusion based on SDFG syntax
+
+Possible, but a lot more cumbersome than writing the same transformation based on the schedule tree syntax.

docs/development/ADRs/cartesian/backend-dace-version.md

@@ -0,0 +1,26 @@
+# DaCe backends: DaCe version
+
+In the context of the [DaCe backend](./backend-dace.md) and the [schedule tree](./backend-dace-schedule-tree.md), facing time pressure, we decided to stay at the `v1.x` branch of DaCe to minimize up-front cost and deliver CPU performance as fast as possible. We considered updating to the mainline version of DaCe and accept follow-up cost of partial rewrites once DaCe `v2` releases.
+
+## Context
+
+The currently released version of DaCe is on the `v1.x` branch. However, the mainline branch moved on (with breaking changes) to what is supposed to be DaCe `v2`. All feature development is supposed to be merged against mainline. Only bug fixes are allowed on the `v1.x` branch.
+
+The [schedule tree](./backend-dace-schedule-tree.md) feature will need changes in DaCe, in particular to translate schedule trees into SDFG. We are unfamiliar with the breaking changes in DaCe.
+
+## Decision
+
+We decided to build a first version of the schedule tree feature against the `v1.x` version of DaCe.
+
+## Consequences
+
+- We'll be able to code against familiar API (e.g. same as the previous GT4Py-DaCe bridge).
+- In DaCe, we won't be able to merge changes into `v1.x`. We'll work on a branch and later refactor the schedule tree -> SDFG transformation to code flow regions in DaCe `v2`.
+
+## Alternatives considered
+
+### Update to DaCe mainline first
+
+- Good because mainline DaCe is accepting new features while `v1.x` is closed for new feature development.
+- Bad because it incurs an up-front cost, which we are trying to minimize to get results fast.
+- Bad because we aren't trained to use the new control flow regions.

docs/development/ADRs/cartesian/backend-dace.md

@@ -0,0 +1,25 @@
+# DaCe backends
+
+In the context of performance optimization, facing the fragmentedness of NWP code, we decided to implemented a backend based on DaCe to unlock full-program optimization. We accept the downside of having to maintain that (additional) performance backend.
+
+## Context
+
+NWP codes aren't like your typical optimization problem homework where 80% of runtime is spent within a single stencil which you can then optimize to oblivion. Instead, computations in NWP codes are fragmented and scattered all over the place with parts in-between that move memory around. Stencil-only optimizations don't cut through this. DaCe allows us to do (data-flow) optimization on the full program, not only inside stencils. As a nice side-effect, DaCe offers code generation to CPU and GPU targets.
+
+## Decision
+
+We chose to add DaCe backends,`dace:cpu` and `dace:gpu`, for CPU and GPU targets because we need full-program optimization to get the best possible performance.
+
+## Consequences
+
+We will need to maintain the `dace:*` backends. If we keep adding more and more backends, maintainability will be a question down the road. We thus decided to put a [feature freeze](./backend-cuda-feature-freeze.md) on the `cuda` backend, focussing on `dace:*` backends instead.
+
+Compared to the [`cuda` backend](./backend-cuda-feature-freeze.md), which only targets NVIDIA cards, we get support for both, NVIDIA and AMD cards, with the `dace:gpu` backends.
+
+## Alternatives considered
+
+@Florian: Did we consider alternatives (back then)?
+
+## References
+
+[DaCe Promo Website](http://dace.is/fast) | [DaCe GitHub](https://github.com/spcl/dace) | [DaCe Documentation](https://spcldace.readthedocs.io/en/latest/)

noxfile.py

@@ -85,7 +85,7 @@
 )
 CodeGenTestSettings: Final[dict[str, dict[str, list[str]]]] = {
     "internal": {"extras": [], "markers": ["not requires_dace"]},
-    "dace": {"extras": ["dace"], "markers": ["requires_dace"]},
+    "dace": {"extras": ["dace-cartesian"], "markers": ["requires_dace"]},
 }
 # Use dace-next for GT4Py-next, to install a different dace version than in cartesian
 CodeGenNextTestSettings = CodeGenTestSettings | {

pyproject.toml

@@ -135,7 +135,7 @@ all = ['gt4py[dace,formatting,jax,performance,testing]']
 cuda11 = ['cupy-cuda11x>=12.0']
 cuda12 = ['cupy-cuda12x>=12.0']
 # features
-dace = ['dace>=1.0.2,<1.1.0']  # v1.x will contain breaking changes, see https://github.com/spcl/dace/milestone/4
+dace-cartesian = ['dace']  # pull dace version with support for schedule trees
 dace-next = ['dace']  # pull dace latest version from the git repository
 formatting = ['clang-format>=9.0']
 jax = ['jax>=0.4.26']
@@ -421,11 +421,12 @@ conflicts = [
     {extra = 'rocm5_0'}
   ],
   [
-    {extra = 'dace'},
+    {extra = 'dace-cartesian'},
     {extra = 'dace-next'}
   ],
   [
     {extra = 'all'},
+    {extra = 'dace-cartesian'},
     {extra = 'dace-next'}
   ]
 ]
@@ -438,7 +439,10 @@ url = 'https://test.pypi.org/simple/'
 
 [tool.uv.sources]
 atlas4py = {index = "test.pypi"}
-dace = {git = "https://github.com/GridTools/dace", tag = "__gt4py-next-integration_2025_07_03", extra = "dace-next"}
+dace = [
+  {git = "https://github.com/romanc/dace", branch = "romanc/stree-to-sdfg", extra = "dace-cartesian"},
+  {git = "https://github.com/GridTools/dace", tag = "__gt4py-next-integration_2025_07_03", extra = "dace-next"}
+]
 
 # -- versioningit --
 [tool.versioningit]