Skip to content

add a first draft of D6 - filedicts design #1448

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mandresm wants to merge 2 commits into
base: sprint/filedicts/main
Choose a base branch
from
Open

add a first draft of D6 - filedicts design #1448

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8c6477a
add a first draft of D6 - filedicts design
mandresm Feb 25, 2026
c01708d
add some suggestions from @pgierz into the document
mandresm Mar 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
352 changes: 352 additions & 0 deletions filedicts_design.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,352 @@
# ESM-Tools File Operations Redesign — Filedicts

This deliverable has been using as its base the old GitHub project for the refactoring of file handling https://github.com/orgs/esm-tools/projects/12, that already included many of the current requirements and design considerations reflected in this document. It is also based in existing unfinished developments in `sprint/filedicts/main` branch that correspond to that old GitHub project. The deliverable has been written using @claude -code in planning mode, using the existing resources mentioned above and new prompted design considerations.

## Overview

ESM-Tools currently handles file operations during `esm_runscripts` through a fragmented system of parallel flat dictionaries (`input_files`, `input_sources`, `input_in_work`, `forcing_files`, etc.) distributed across multiple sections of
component configs. This approach is hard to read, difficult to extend, and couples file metadata to intermediate directory structures that add unnecessary complexity and file duplication.

This document describes the new design — **filedicts** — which unifies file definitions into structured objects, removes intermediate staging directories inside `run_DATE/`, and simplifies the operation model, where each phase declares in the backend the file operation directions.

----

## Design

### Syntax

#### `files:` block

Partly taken from https://github.com/orgs/esm-tools/projects/12/views/1?pane=issue&itemId=8130162

Discussion in: [esm-tools/esm-design#2](https://github.com/esm-tools/esm-design/issues/2)

Files are defined in a `files:` block, grouped by type. The type is inferred from the group key: no `type:` attribute needed per file. Each group may contain a `defaults:` sub-block for shared attributes. Individual file entries can be scalars (shorthand) or full objects.

```yaml
files:
input:
defaults:
path_in_pool: ${echam.input_dir}
prepare: copy
cldoptprops: # null → name = label in all locations
janspec: janspec.nc # scalar → name_in_pool
jansurf: # object → explicit overrides only
name_in_pool: jansurf.nc
name_in_run: unit.24
rrtmglw: /other/pool/path/rrtmg.nc # has "/" → full path, ignores defaults
forcing:
defaults:
path_in_pool: ${echam.forcing_dir}
prepare: link
sst: pisst.nc
sic:
name_in_pool: pisic${current_date.year}.nc
name_in_run: unit.96
include_years_before: 1
outdata:
defaults:
tidy: copy
histogram: # null → name_in_run = label
atm_data: atmosphere_output.nc # scalar → name_in_run
restart:
defaults:
prepare: copy
tidy: copy
jan_restart:
name_in_run: restart.nc
```

#### File object attributes

| Attribute | Description |
|---|---|
| `name_in_pool` | Filename in the pool/source location |
| `name_in_run` | Filename in `run_DATE/` (the working directory) |
| `name_in_exp` | Filename in the experiment tree (defaults to `name_in_run`) |
| `path_in_pool` | Path to the file in the pool (excluding filename) |
| `prepare` | Operation for `pool/exp → run`: `copy`, `link`, `move` |
| `tidy` | Operation for `run → exp/<type>`: `copy`, `link`, `move` |
Copy link
Member

@pgierz pgierz Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah...ok...I see. Then perhaps: prepare_op and tidy_op? We can discuss...

Copy link
Contributor Author

@mandresm mandresm Mar 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. What about prepare_operation, tidy_operation, etc.? I'm also fine with op.

| `include_years_before` | Years before current to include (requires `StringWithDate`) |
| `include_years_after` | Years after current to include (requires `StringWithDate`) |
| `description` | Human-readable description |
| `allowed_to_be_missing` | If `true`, missing file does not raise an error |
| `is_reusable` | if `true` copy from exp instead of from pool, like bins and inputs (default: false) |
| description | a file description [optional] |
| filetype | filetype, like NetCDF [optional], not sure if we should implement this attribute |

#### Files with varying paths depending on dates

For files whose paths change depending on dates the syntax will be:
```yaml
files:
<file1_label>:
description: "some string"
path_in_pool:
"/path/to/first/file${year}":
from: <year>
to: <year>
"/path/to/second/file${year}":
from: <year>
```

#### File selection via `choose_`

All files defined in `files:` are active by default. Scenarios override specific attributes via `choose_`. Only the differing attributes need to be specified:

```yaml
files:
forcing:
defaults:
path_in_pool: ${echam.forcing_dir}
sst:
name_in_pool: pisst.nc # default: PI-CTRL
name_in_run: unit.20

choose_scenario:
historical:
files:
forcing:
sst:
name_in_pool: histsst.nc # only override what changes

ssp585:
add_files: # add a file not in base config
forcing:
ozone:
name_in_pool: ozone_ssp585.nc
name_in_run: ozone.nc
```

A reserved `include:` list mechanism per type could be implemented in the future if explicit whitelisting is needed, or perhaps it can be implemente also as an attribute of file objects.
Copy link
Member

@pgierz pgierz Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am afraid I don't understand what you mean here. You could have an attribute on your file object that defines a conditional, and the parser could then spit out a true or false for that. I'll see if I can invent an example...

Copy link
Member

@pgierz pgierz Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not happy that I am about to suggest this, but we already have a mechanism to do eval... we really really should not use it further, because eval is dangerous for all kinds of reasons. Buuuuuttttttt this would be one way:

  files:
      input:
          ozone_ssp:
              name_in_pool: ozone_ssp585.nc
              include: "$(( '${scenario}' == 'ssp585' ))"

          debug_dump:
              name_in_run: debug.nc
              # imaginary switch that might turn on some namelist setting
              # and write a new output file
              include: "$(( ${debug_mode} and ${verbosity} > 2 ))"


#### Scalar shorthand rules direction-awareness:

In order to simplify the amount of writing needed for specifying file operations we allow for shorthand file definitions, consisting of a label (key) followed by a path (value). The path can be an absolute path (starting with `/`) or a relative path. The filedicts attribute that takes the value of the shorthand-path is determined by the source of file operations for that specific phase. For example:

- For `prepare` input-like file types `input`, `forcing`, `config` and `restart` the scalar is `name_in_pool`, or `name_in_exp`.
- For `tidy` output-like file types `outdata`, `restart`, `log`, and `mon` the scalar is `name_in_run`.

Other shorthand rules are:
- For `null` values the label is used as name in all relevant locations. The label cannot have placeholders (no `${}`).
- Plain string (no `/`) → filename only; path comes from `defaults.path_in_pool`
- String with `/` → full path, overrides `defaults.path_in_pool`
Comment on lines +133 to +136
Copy link
Member

@pgierz pgierz Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above, I feel like null is dangerous, we need to think about this

Missing here is nested relative paths. If I remember correctly, some models have a nested subfolder in work (or, I guess now run):

files:
    restart:
        my_fancy_nested_restart: run_${DATE}/component_${year}/my_restart.nc

If I understand it as designed, that will then end in the restart folder during the tidy phase as my_restart.nc, correct?

Copy link
Member

@pgierz pgierz Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something my LLM came up with which I think is worth considering:

Line 133: Nested relative paths syntax

The question: how to handle run_DATE/subdir/file.nc?

Proposed syntax:

  files:
      outdata:
          my_nested_output:
              name_in_run: component_output/year_${year}/output.nc
              name_in_exp: output_${year}.nc  # flattened to basename

Rules:

  • name_in_run can contain path separators (relative to run_DATE/)
  • name_in_exp can contain path separators (relative to exp/<type>/<component>/)
  • If name_in_exp omitted, default to basename of name_in_run

This means nested structure is explicitly controlled:

  • Flatten (default behavior)
    name_in_run: subdir/output.nc
    --> exp/outdata/component/output.nc

  • Preserve structure (explicit)
    name_in_run: subdir/output.nc
    name_in_exp: subdir/output.nc
    --> exp/outdata/component/subdir/output.nc

Recommendation: Add this to the document with explicit rules about path handling.


#### Date-varying files

Files whose names contain date variables resolve to `StringWithDate` objects,
produced by `esm_parser` (see esm_parser dependency below). The
`include_years_before/after` attributes instruct filedicts to expand the file
for multiple years around the given date on the string (if there is only one date, if there are several dates throws and error).

```yaml
files:
forcing:
defaults:
path_in_pool: ${echam.forcing_dir}
prepare: link
ozone:
name_in_pool: ozon${current_year}.nc
name_in_run: ozon${current_year}.nc
include_years_before: 1
include_years_after: 0
```

If `include_years_before/after` is set but the name resolves to a plain `str`
(i.e. the string contained no date variable), filedicts raises a `FileDictError`.
Comment on lines +138 to +159
Copy link
Member

@pgierz pgierz Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this a lot. As above, my only thought is to do this in a general way for other temporal frequencies as well. Specifying just one year becomes trivial, you just need some wordier syntax. I am not opposed to it, and it makes it more flexible.

What do the others think?

Copy link
Member

@pgierz pgierz Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again from the LLM:

Line 156: Generic timedelta

Three options:

  1. Keep simple: include_years_before: 1 (current design)
  2. Go generic: include_timedelta_before: {years: 1, months: 6}
  3. Support both: shorthand for years, verbose for complex cases

Option 2 syntax:

  files:
      forcing:
          ozone:
              include_timedelta_before:
                  years: 1
              include_timedelta_after:
                  months: 6

Recommendation: Go with option 2 (generic only). It's slightly more verbose for the common case but avoids adding a second syntax later. Years become {years: 1}.

❯ Can we write files.forcing.ozone.include_timedelta_before: {years: 1} in YAML? Is that allowed, like square brackets for lists?

⏺ Yes, that's valid YAML. Curly braces are "flow style" for mappings, just like square brackets are flow style for sequences.

  # Flow style (inline)
  include_timedelta_before: {years: 1}
  include_timedelta_before: {years: 1, months: 6}

  # Block style (expanded)
  include_timedelta_before:
      years: 1
      months: 6

  # Both parse identically to Python:
  # {'include_timedelta_before': {'years': 1, 'months': 6}}

So the compact version works fine:

  files:
      forcing:
          ozone:
              name_in_pool: ozon${current_year}.nc
              include_timedelta_before: {years: 1}
              include_timedelta_after: {months: 6}

Copy link
Member

@pgierz pgierz Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My own note here would be to keep it simple: if you put an integer on RHS instead of a dict, you automatically mean years. Then, include_years_before: 1 internally becomes include_timedelta_before: {years: 1} and something (I guess the parser?) beeps at you to please be nice and write it correctly. It would not break though.


#### `simulation_files` (removed)

The `simulation_files` / `<type>_files` selection mechanism is replaced by the
grouped `files:` block with Option C selection (all defined files active,
scenario-specific changes via `choose_`).
Comment on lines +163 to +165
Copy link
Member

@pgierz pgierz Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes! 🥳


---

### File Locations

Three locations replace the previous multi-level structure:

| Location | Description |
|---|---|
| `pool` | Source on the HPC system — input data, forcing data |
| `run` | `run_DATE/` — the flat working directory during simulation |
| `exp` | Experiment tree — persistent storage, structure preserved |

**`run_DATE/` is now flat.** Intermediate staging directories (`run_DATE/input/`, `run_DATE/work/`, `run_DATE/outdata/`, etc.) will be removed. All files land directly in `run_DATE/`. The `exp` tree structure won't be changed (`exp/outdata/<component>/`, `exp/restart/<component>/`, etc.).
Comment on lines +171 to +179
Copy link
Member

@pgierz pgierz Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes! 🥳 🎊 🚀


---

### File Operations

`file_movements` special key is entirely removed, no functionality depends on this key anymore. Instead, key/values of phase/file operation can be defined as attributes in the file dictionary:

```yaml
files:
forcing:
ozone:
name_in_pool: ozon${current_year}.nc
prepare: copy
```

The direction is defined in the phase itself as a system invariant, not user-configurable. This is subject to change in the future if needed. The `FileTypes` enum declares which phases apply to each type and the `exp` subdirectory name. Default operations (`copy`/`link`/`move`) per type are defined in `per_model_defaults` in the system defaults YAML.
Copy link
Member

@pgierz pgierz Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't foresee needing to change the direction. What did you have in mind here?


| Phase | Direction | Applicable types |
|---|---|---|
| `prepare` | `pool/exp → run` | `input`, `forcing`, `config`, `restart` |
| `tidy` | `run → exp/<type>` | `outdata`, `restart`, `log`, `mon` |

Restart files use both phases: `prepare` stages the previous restart into `run/`, `tidy` archives the new restart to `exp/restart/`.

---

### `StringWithDate` (esm_parser dependency)

When `esm_parser` resolves a string containing a date variable (e.g. `${current_year}`), it produces a `StringWithDate` — a `str` subclass that also carries the `Date` object and the original template. This allows filedicts to re-resolve the string for year offsets without re-running the parser:

```python
class StringWithDate(str):
# Behaves as a normal string — resolves to current date value
# Also exposes:
# .for_year_offset(n: int) → StringWithDate
# .date → esm_calendar.Date
```

**This is an `esm_parser` task.** Filedicts consumes `StringWithDate`; it does not create it. This dependency must be resolved before date-varying file features can be implemented in filedicts.

----

## Major Changes

- [ ] Syntax rework: grouped `files:` block with `defaults:` and scalar
shorthand
- [ ] Remove intermediate directories inside `run_DATE/`
- [ ] Replace `file_movements` with `prepare`/`tidy` per-file attributes
- [ ] Remove `simulation_files` / `<type>_files` file selection. If it is present in the config a file operation is needed.
- [ ] `StringWithDate` in `esm_parser`

---

## Features

- [ ] Report missing values. This feature should not change, except that if a globbing has 0 files then it should report a missing value https://github.com/orgs/esm-tools/projects/12/views/1?sortedBy%5Bdirection%5D=desc&sortedBy%5BcolumnId%5D=Status&pane=issue&itemId=8312692
- [ ] File type movements per file https://github.com/orgs/esm-tools/projects/12/views/1?sortedBy%5Bdirection%5D=desc&sortedBy%5BcolumnId%5D=Status&pane=issue&itemId=8307345
- [ ] Ignore files https://github.com/orgs/esm-tools/projects/12/views/1?sortedBy%5Bdirection%5D=desc&sortedBy%5BcolumnId%5D=Status&pane=issue&itemId=8308020
- [ ] _check_fesom_missing_files https://github.com/orgs/esm-tools/projects/12/views/1?sortedBy%5Bdirection%5D=desc&sortedBy%5BcolumnId%5D=Status&pane=issue&itemId=8308173
- [ ] Remove `@YEAR@` placeholder → `StringWithDate` + `include_years_before/after` https://github.com/orgs/esm-tools/projects/12/views/1?sortedBy%5Bdirection%5D=desc&sortedBy%5BcolumnId%5D=Status&pane=issue&itemId=8308849
- [ ] Calculate absolute paths for relative paths https://github.com/orgs/esm-tools/projects/12/views/1?sortedBy%5Bdirection%5D=desc&sortedBy%5BcolumnId%5D=Status&pane=issue&itemId=8309908
- [ ] Reuse sources https://github.com/orgs/esm-tools/projects/12/views/1?sortedBy%5Bdirection%5D=desc&sortedBy%5BcolumnId%5D=Status&pane=issue&itemId=8310094
- [ ] ~Include property (switch on/off files)~ Probably to not be implemented [#209](https://github.com/esm-tools/esm_runscripts/issues/209)
- [ ] `include_years_before/after` properties https://github.com/orgs/esm-tools/projects/12/views/1?sortedBy%5Bdirection%5D=desc&sortedBy%5BcolumnId%5D=Status&pane=issue&itemId=8312260
- [ ] `FileDictError` https://github.com/orgs/esm-tools/projects/12/views/1?sortedBy%5Bdirection%5D=desc&sortedBy%5BcolumnId%5D=Status&pane=issue&itemId=8312649
- [ ] Checksums computation https://github.com/orgs/esm-tools/projects/12/views/1?sortedBy%5Bdirection%5D=desc&sortedBy%5BcolumnId%5D=Status&pane=issue&itemId=8313953
- [ ] Unit test cleanup https://github.com/orgs/esm-tools/projects/12/views/1?sortedBy%5Bdirection%5D=desc&sortedBy%5BcolumnId%5D=Status&pane=issue&itemId=8956628
- [ ] Paths of linked files should be shown in the finished config file https://github.com/orgs/esm-tools/projects/12/views/1?sortedBy%5Bdirection%5D=desc&sortedBy%5BcolumnId%5D=Status&pane=issue&itemId=11438417
- [ ] Make sure the config is not populated with a huge amount of repeated key/values
- [ ] Parallelization of file operations https://github.com/orgs/esm-tools/projects/12/views/1?sortedBy%5Bdirection%5D=desc&sortedBy%5BcolumnId%5D=Status&pane=issue&itemId=11632805&issue=esm-tools%7Cesm_tools%7C811
- [ ] Better merging strategy for from/to in files https://github.com/orgs/esm-tools/projects/12/views/1?sortedBy%5Bdirection%5D=desc&sortedBy%5BcolumnId%5D=Status&pane=issue&itemId=21671561
- [ ] Support files with `.` (config_files.namelist.config) https://github.com/orgs/esm-tools/projects/12/views/1?sortedBy%5Bdirection%5D=desc&sortedBy%5BcolumnId%5D=Status&pane=issue&itemId=23111901
- [ ] Allow missing wildcards https://github.com/orgs/esm-tools/projects/12/views/1?sortedBy%5Bdirection%5D=desc&sortedBy%5BcolumnId%5D=Status&pane=issue&itemId=23198986
- [ ] define category file movements (or file operations) https://github.com/orgs/esm-tools/projects/12/views/1?sortedBy%5Bdirection%5D=desc&sortedBy%5BcolumnId%5D=Status&pane=issue&itemId=30153684
- [ ] Add intermediate step into _gather_file_movements https://github.com/orgs/esm-tools/projects/12/views/1?sortedBy%5Bdirection%5D=desc&sortedBy%5BcolumnId%5D=Status&pane=issue&itemId=30207127
- [ ] more binary info in finished_config #990

---

## Testing

### Unit tests

One unit test per feature, developed alongside each feature following the
red/green/refactor cycle (see Development Workflow below).
Comment on lines +263 to +264
Copy link
Member

@pgierz pgierz Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say "one unit test collection", not "one unit test". It may be we want to nail down edge case logic in some features


### Integration tests (CI)

AWI-ESM 2 and AWICM3 are the reference cases. For each, a dry-run produces the
full resolved filedicts state (all file attributes and absolute paths). This
state is asserted against expected snapshot values. Features are built
incrementally until the full dry-run state matches the real case scenario.
Runs in CI on every PR to `sprint/filedicts/main`.

### Checksum tests (esm_tests)

To verify the refactor is behaviour-preserving — identical files end up in
identical locations:

1. Compute checksums for all files moved/copied/linked on the `release` branch
2. Compute checksums for the same run on `sprint/filedicts/main`
3. Compare checksums for both `prepare` and `tidy` phases

Any mismatch is a regression. Checksum tests run via `esm_tests`.
Copy link
Member

@pgierz pgierz Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏻


---

## Retrocompatibility

The new syntax will substitute to the old syntax. The old syntax won't be supported anymore from the new release version. That means that old runscripts referencing the old file lists will have to be changed to work with the new version containing this refactoring. Same goes for the configs, however, the configs' syntax will be changed to the new one as part of this release, and include test runs with esm_tests. Resuming or branching off old simulations will still be possible because the file structure in `exp` won't change.

The users will be informed of such changes.
Comment on lines +287 to +291
Copy link
Member

@pgierz pgierz Mar 9, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏻 (actually +100)

If we spent any effort on making this backward compatible, I would vote that it will be in automatically translating and gently telling the user "Hey, I have updated all of your configs" via <translation tool>.

maintaining two syntaxes when the old one is supposed to be replaced sounds like a bad time.


---

## Refactoring guidelines

### General

Taken from https://github.com/orgs/esm-tools/projects/12/views/1?pane=issue&itemId=8002017

- One task, one function
Copy link
Member

@pgierz pgierz Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

- Do not change the YAML syntax excessively
Copy link
Member

@pgierz pgierz Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes again

- Runscripts and config files need minimal changes
Copy link
Member

@pgierz pgierz Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure I agree with this one. By nature, some of these changes will "look bloated". they aren't, they are making it more precise.

- Changes should make configs easier to write and understand
Copy link
Member

@pgierz pgierz Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bigger yes

- All tests pass (unit tests, esm_tests, red/green/refactor cycle)
- Do not refactor a feature before fully understanding its behaviour and
dependencies

### Development workflow per feature

1. New branch: `sprint/filedicts/<descriptive_name>`
2. Unit test development
3. PR draft
4. Feature development (push early and often)
5. Pull request to `sprint/filedicts/main`
6. Review
7. Merge

### Task groups

1. **Preparation** — codebase understanding, test infrastructure
2. **New syntax / functionality** — filedicts features
3. **Back-compatibility** — migration layer for old YAML syntax
4. **Deployment** — YAML file adaptation, release preparation

### Filedicts design guidelines

Taken from https://github.com/orgs/esm-tools/projects/12/views/1?pane=issue&itemId=8306419

- Internally, use `source`/`target` terminology once directions are resolved by
phase
- Use `pathlib` instead of string path concatenation
- Use `esm_parser.user_error/user_note` for user-facing errors and warnings
(higher-level functions only; lower-level functions should raise exceptions
for testability)
- Write numpy-style docstrings
- Use `.get()` for extracting file properties
- `FileDictionary` class: file-specific logic, checks, attribute completion
- file specific
- checks
- autocomplete info
- `FileDictionaries` class: general functionality, initialising file objects
- general functionalities
- initializing the file objects

### Naming conventions

Taken from https://github.com/orgs/esm-tools/projects/12/views/1?pane=issue&itemId=8308503

Taken from "Clean Code with Python" (https://www.amazon.de/Clean-Code-Python-maintainable-efficient/dp/1800560214/ref=asc_df_1800560214/?tag=googshopde-21&linkCode=df0&hvadid=473997534442&hvpos=&hvnetw=g&hvrand=17247415010170938050&hvpone=&hvptwo=&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=9068390&hvtargid=pla-1124354993243&psc=1&th=1&psc=1). Ask Paul if you want to borrow it

* Functions that should be used "outside" have `regular_names`
* Functions that should only be used "inside" have `_private_names`

Python does not make explicit between public and private functions, but these guidelines are used elsewhere as well. I would denote "outside" as a step to be included in the run recipe, and "inside" as some small thing you just need, but whoever is designing a run recipe does not need to know about.
Loading