Skip to content

Fix staging of Path fields of included record types#7226

Open
robsyme wants to merge 1 commit into
masterfrom
fix-record-include-staging
Open

Fix staging of Path fields of included record types#7226
robsyme wants to merge 1 commit into
masterfrom
fix-record-include-staging

Conversation

@robsyme

@robsyme robsyme commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

Fixes #7225

Problem

When a typed process (nextflow.enable.types = true) declares an input whose type is a record included from another module file, the implicit stageAs directives for the record's Path fields are not generated. The fields are interpolated into .command.sh as raw source paths instead of staged paths. On a shared local filesystem the raw path still resolves, which masks the bug, but on an object-store work dir (e.g. Fusion/S3) the task fails with No such file or directory.

Isolating the variables with the reproducer shows the trigger is the record type being included from a separate file (the nextflow.enable.moduleBinaries flag suspected in the original report is unrelated):

Record definition Field staged?
included from types.nf no — raw source path
inline in main.nf yes — TaskPath

Cause

ScriptCompiler performs resolution and Groovy conversion per source file, and the entry script is analyzed before the modules it includes. When ProcessToGroovyVisitorV2 inspects a record input to infer the implicit file inputs, the included record node is found (isRecordType is true) but its field types are still unresolved ClassNodes, so isPathType returns false and no stager is generated for the Path fields. The unstaged field then bypasses the staged-path substitution in TaskInputResolver.normalizeValue and reaches the task script as a raw source path.

Solution

  • ModuleResolver now records the include dependencies of each source file, returning existing source units on repeat encounters so the dependency graph is complete.
  • ScriptCompiler analyzes each source file only after the modules it includes from, so included types are fully resolved before they are used. Guards handle include cycles and unparseable modules.

Verification

  • New unit test ScriptLoaderV2Test."should stage path fields of an included record type" fails before the change (0 file inputs generated) and passes after (2 file inputs).
  • The issue reproducer (https://github.com/robsyme/nf-record-fusion-repro) now renders the record fields as staged TaskPath names instead of raw paths.
  • Transitive includes (process in module A consuming a record included from module B) also stage correctly.
  • Circular includes still fail with a StackOverflowError, but this is pre-existing behavior at runtime module loading (IncludeDef.loadModuleV2), unchanged by this PR.

🤖 Generated with Claude Code

When a typed process declares an input whose type is a record imported
from another module, the implicit stageAs directives for the record's
Path fields were not generated, so the fields were interpolated into
the task script as raw source paths instead of staged paths. On a
shared local filesystem the raw path still resolves, masking the bug,
but on an object-store work dir (e.g. Fusion/S3) the task fails with
'No such file or directory'.

The cause is the per-source analysis order in the script compiler:
the entry script was resolved and converted to Groovy before the
modules it includes, so the field types of an included record were
still unresolved ClassNodes when ProcessToGroovyVisitorV2 inspected
them to infer the implicit file inputs, and the Path fields were not
recognized as such.

Fix by tracking the include dependencies of each source file during
module resolution and analyzing each source only after the modules it
includes from, so that included types are fully resolved before they
are used.

Note: issue #7225 attributes the failure to the moduleBinaries feature
flag, but the actual trigger is the record type being included from a
separate module file; the flag is unrelated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Rob Syme <rob.syme@gmail.com>
@robsyme robsyme requested a review from a team as a code owner June 11, 2026 23:33
@netlify

netlify Bot commented Jun 11, 2026

Copy link
Copy Markdown

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit 9274276
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/6a2b45d6fbf660000887b75f

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Record Path fields are not staged when the record type is included from another module

1 participant