Open
Conversation
|
Documentation for this branch can be viewed at https://sites.ecmwf.int/docs/loki/649/index.html |
…g module and file item
… 'process_transformation' optionally (item) 'mode'-aware
… tree-like for relevant kernels called from drivers with different modes
… be handled by 'process'
MichaelSt98
force-pushed
the
nams_multimode_processing
branch
from
3f83ad3 to
2a1af8f
Compare
…on that actually uses it
MichaelSt98
force-pushed
the
nams_multimode_processing
branch
from
968886c to
85c7247
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #649 +/- ##
==========================================
- Coverage 96.39% 96.38% -0.01%
==========================================
Files 266 266
Lines 46417 46724 +307
==========================================
+ Hits 44744 45037 +293
- Misses 1673 1687 +14
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
This PR introduces multi-mode (driver-specific) pipeline processing in Loki’s batch scheduler, aiming to produce a more mode-separated (tree-like) dependency graph and to generate correct build-system “plan” outputs for replicated/duplicated items.
Changes:
- Add scheduler support to run a dict of pipelines keyed by mode, including mode propagation/separation and mode-filtered traversals.
- Introduce
SeparateModesKerneltransformation to duplicate/retarget dependencies by inherited modes and rename calls/imports accordingly. - Extend CLI and CMake wrappers with a
--multimode/MULTIMODEoption and add new test fixtures + scheduler tests for multi-mode behavior and plan generation.
Reviewed changes
Copilot reviewed 35 out of 35 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| loki/transformations/dependency.py | Adds SeparateModesKernel transformation for mode-based duplication and call/import rewriting. |
| loki/transformations/build_system/plan.py | Ensures plan generation can include original sources for replicated items via orig_path. |
| loki/transformations/build_system/dependency.py | Fixes suffix stripping to only remove the last suffix occurrence when renaming imports. |
| loki/sourcefile.py | Introduces Sourcefile.orig_path to preserve original file path across cloning/renaming. |
| loki/cli/loki_transform.py | Adds --multimode option to run all configured pipelines (dict) instead of a single mode pipeline. |
| loki/batch/scheduler.py | Adds dict-of-pipelines processing and mode propagation/separation hooks; adds mode-filtering to transformation traversal. |
| loki/batch/sgraph.py | Adds helpers to find descendants and corresponding module/file items for mode propagation. |
| loki/batch/sfilter.py | Adds optional mode filter to traversal to process items for a specific mode. |
| loki/batch/item_factory.py | Tracks orig_path on cloned sources; adds helper to find file items by Sourcefile identity. |
| loki/batch/item.py | Exposes Item.orig_path forwarding to Sourcefile.orig_path. |
| loki/batch/configure.py | Adds inherited_mode property for multi-mode processing metadata. |
| cmake/loki_transform.cmake | Adds MULTIMODE option wiring through to the CLI. |
| loki/batch/tests/test_scheduler.py | Adds multi-mode scheduler test (including plan validation) and imports CMakePlanTransformation. |
| loki/tests/sources/projMultiModeModules/driver_0_mod.F90 | New test source fixture for module-based multi-mode callgraphs. |
| loki/tests/sources/projMultiModeModules/driver_1_mod.F90 | New test source fixture for module-based multi-mode callgraphs. |
| loki/tests/sources/projMultiModeModules/driver_2_mod.F90 | New test source fixture for module-based multi-mode callgraphs. |
| loki/tests/sources/projMultiModeModules/driver_3_mod.F90 | New test source fixture for module-based multi-mode callgraphs. |
| loki/tests/sources/projMultiModeModules/driver_4_mod.F90 | New test source fixture for module-based multi-mode callgraphs. |
| loki/tests/sources/projMultiModeModules/subroutine_1_mod.F90 | New test source fixture for module-based multi-mode callgraphs. |
| loki/tests/sources/projMultiModeModules/subroutine_2_mod.F90 | New test source fixture for module-based multi-mode callgraphs. |
| loki/tests/sources/projMultiModeModules/subroutine_3_mod.F90 | New test source fixture for module-based multi-mode callgraphs. |
| loki/tests/sources/projMultiModeModules/nested_subroutine_1_mod.F90 | New test source fixture for module-based multi-mode callgraphs. |
| loki/tests/sources/projMultiModeModules/nested_subroutine_2_mod.F90 | New test source fixture for module-based multi-mode callgraphs. |
| loki/tests/sources/projMultiModeModules/nested_subroutine_3_mod.F90 | New test source fixture for module-based multi-mode callgraphs. |
| loki/tests/sources/projMultiMode/driver_0_mod.F90 | New test source fixture for include-based multi-mode callgraphs. |
| loki/tests/sources/projMultiMode/driver_1_mod.F90 | New test source fixture for include-based multi-mode callgraphs. |
| loki/tests/sources/projMultiMode/driver_2_mod.F90 | New test source fixture for include-based multi-mode callgraphs. |
| loki/tests/sources/projMultiMode/driver_3_mod.F90 | New test source fixture for include-based multi-mode callgraphs. |
| loki/tests/sources/projMultiMode/driver_4_mod.F90 | New test source fixture for include-based multi-mode callgraphs. |
| loki/tests/sources/projMultiMode/subroutine_1.F90 | New test source fixture for include-based multi-mode callgraphs. |
| loki/tests/sources/projMultiMode/subroutine_2.F90 | New test source fixture for include-based multi-mode callgraphs. |
| loki/tests/sources/projMultiMode/subroutine_3.F90 | New test source fixture for include-based multi-mode callgraphs. |
| loki/tests/sources/projMultiMode/nested_subroutine_1.F90 | New test source fixture for include-based multi-mode callgraphs. |
| loki/tests/sources/projMultiMode/nested_subroutine_2.F90 | New test source fixture for include-based multi-mode callgraphs. |
| loki/tests/sources/projMultiMode/nested_subroutine_3.F90 | New test source fixture for include-based multi-mode callgraphs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Comment on lines
+514
to
+519
| if rename_calls: | ||
| new_dependencies = CaseInsensitiveDict((new_item.local_name, new_item) | ||
| for new_item in new_item_new_items) | ||
| new_dependencies = CaseInsensitiveDict((k.local_name, v) for k, v in | ||
| new_item.trafo_data.get('SeparateModes', {}).items() if not isinstance(k, str)) | ||
| self._adapt_calls_imports(new_item.ir, new_dependencies) |
Comment on lines
+546
to
+552
| # Modify module import if it imports any call targets | ||
| if new_dependencies and im.symbols and any(s in new_dependencies for s in im.symbols): | ||
| relevant_symbol = [s for s in im.symbols if s in new_dependencies][0] | ||
| _new_symbol = new_dependencies[relevant_symbol] | ||
| new_symbol = _new_symbol.scope_ir.procedure_symbol # local_name | ||
| new_module_name = _new_symbol.scope_name | ||
| im._update(module=new_module_name, symbols=(new_symbol,)) |
|
|
||
| # Create a new FileItem for the new source | ||
| new_source.path = item.path.with_name(f'{scope_name or local_name}{item.path.suffix}') | ||
| new_source.orig_path = item.path |
| sys.exit(msg) | ||
| # apply those pipelines | ||
| for mode in modes: | ||
| self.process_pipeline(pipeline=self.config.pipelines[mode], proc_strategy=proc_strategy, mode=mode) |
Comment on lines
+335
to
+355
| class SeparateModesKernel(Transformation): | ||
| """ | ||
| Duplicate subroutines which includes the creation of new :any:`Item`s | ||
| as well as the addition of the corresponding new dependencies. | ||
|
|
||
| Therefore, this transformation creates a new item and also implements | ||
| the relevant routines for dry-run pipeline planning runs. | ||
|
|
||
| Parameters | ||
| ---------- | ||
| duplicate_kernels : str|tuple|list, optional | ||
| Kernel name(s) to be duplicated. | ||
| duplicate_suffix : str, optional | ||
| Suffix to be used to append the original kernel name(s). | ||
| duplicate_module_suffix : str, optional | ||
| Suffix to be used to append the original module name(s), | ||
| if defined, otherwise `duplicate_suffix` | ||
| duplicate_subgraph : bool, optional | ||
| Whether or not duplicate the subgraph beneath the kernel(s) | ||
| that are duplicated. | ||
| """ |
Comment on lines
+488
to
+523
| modes_to_duplicate = sorted(list(set(self._get_item_modes(child)))) | ||
| if item.mode in child.trafo_data['SeparateModes']: | ||
| new_item = child.trafo_data['SeparateModes'][item.mode] | ||
| removed_items += (child,) | ||
| new_items += as_tuple(new_item) | ||
| item.trafo_data['SeparateModes'][child] = new_item | ||
| elif len(modes_to_duplicate) > 1: | ||
| mode_to_duplicate = item.mode | ||
| new_item = self._get_or_create_or_rename_item(child, mode_to_duplicate, item_factory, config) | ||
| new_item.config = child.config.copy() | ||
| new_item.config['mode'] = mode_to_duplicate | ||
| parent_items = self.get_parent_items(new_item, item_factory) | ||
| for parent_item in parent_items: | ||
| parent_item.config['mode'] = mode_to_duplicate | ||
| child.inherited_mode.discard(mode_to_duplicate) | ||
| removed_items += (child,) | ||
| new_items += as_tuple(new_item) | ||
| item.trafo_data['SeparateModes'][child] = new_item | ||
| child.trafo_data['SeparateModes'][mode_to_duplicate] = new_item | ||
| # recurse | ||
| new_item.plan_data.setdefault('removed_dependencies', ()) | ||
| new_item.plan_data.setdefault('additional_dependencies', ()) | ||
| new_item_new_items, new_item_removed_items = self._create_new_items(sub_sgraph.successors(child), | ||
| item_factory, config, new_item, sub_sgraph) | ||
| new_item.plan_data['additional_dependencies'] += new_item_new_items | ||
| new_item.plan_data['removed_dependencies'] += new_item_removed_items | ||
| if rename_calls: | ||
| new_dependencies = CaseInsensitiveDict((new_item.local_name, new_item) | ||
| for new_item in new_item_new_items) | ||
| new_dependencies = CaseInsensitiveDict((k.local_name, v) for k, v in | ||
| new_item.trafo_data.get('SeparateModes', {}).items() if not isinstance(k, str)) | ||
| self._adapt_calls_imports(new_item.ir, new_dependencies) | ||
| else: | ||
| mode_to_duplicate = modes_to_duplicate[0] | ||
| child.config['mode'] = item.mode | ||
| parent_items = self.get_parent_items(child, item_factory) |
Comment on lines
+520
to
+525
| else: | ||
| mode_to_duplicate = modes_to_duplicate[0] | ||
| child.config['mode'] = item.mode | ||
| parent_items = self.get_parent_items(child, item_factory) | ||
| for parent_item in parent_items: | ||
| parent_item.config['mode'] = mode_to_duplicate |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
allow for multi-mode/driver-specific Loki pipelines, which includes possibly making the graph more tree-like for relevant routines/items.
E.g, starting from:
with
getting to: