Order of 'includeConfig' and 'profiles' in nextflow.config seems to affect nested property overrides

[danoreper]

Bug report

Expected behavior and actual behavior

Hi Nextflow team,

I expected that the order in which 'includeConfig' and 'profiles' are declared in nextflow.config would have no effect on property values.
However, I think I'm seeing that 'profiles' must be declared after any 'includeConfig' statements, in order for profile overrides to work correctly. In particular, this seems to affect what I'd describe as nested properties.

Steps to reproduce the problem

Given the following two files:

nextflow.config:

profiles {
    hpc {
        process {

         process_nested.value = "hpc.config"
         process_nonest       = "hpc.config"
       
         withLabel: "labelExample" {
             memory = 400
        }
       }
    }
}

includeConfig "memory.config"

and memory.config:

process {
    process_nested.value = "memory.config"
    process_nonest       = "memory.config"

    withLabel: "labelExample" {
        memory = 4
    }
}

and using the following command line invocation:

nextflow config -profile hpc

The nested properties 'process_nested.value' and 'labelExample' fail to be overridden by the hpc profile, while 'process_nonest' is correctly overridden.

(Of note, the problematic overrides work as expected once the 'includeConfig' declaration is moved above the 'profiles' declaration)

Program output

the unexpected output is as follows:

process {
   process_nested {
      value = 'memory.config'
   }
   process_nonest = 'hpc.config'
   withLabel:labelExample {
      memory = 4
   }
}

(No .nextflow.log is generated by nextflow config)

Environment

• Nextflow version: Both 23.10.1 and 23.04.4
• Java version: Java/17.0.4
• Operating system: Linux
• Bash version: GNU bash, version 4.2.46(2)-release

Additional context

The example provided is for the nextflow config command, but I'm also seeing similar behavior for nextflow run

[bentsherman]

This will be fixed by #4744

[danoreper]

Thanks @bentsherman !

[pditommaso]

I'm not 100% sure this is going to be by #4744. We advice to avoid the pattern you are using for some limitation of the underlying parsing library.

See the red box at this link in the docs.

A better way to handle your need is to use the standard config file profile e.g.

profiles {
    standard {
      includeConfig "memory.config"
    }
    hpc {
        process {

         process_nested.value = "hpc.config"
         process_nonest       = "hpc.config"
       
         withLabel: "labelExample" {
             memory = 400
        }
       }
    }
}

I get the following config

» nextflow config . 
process {
   process_nested {
      value = 'memory.config'
   }
   process_nonest = 'memory.config'
   withLabel:labelExample {
      memory = 4
   }
}


» nextflow config . -profile hpc
process {
   process_nested {
      value = 'hpc.config'
   }
   process_nonest = 'hpc.config'
   withLabel:labelExample {
      memory = 400
   }
}

» nextflow config . -profile standard,hpc
process {
   process_nested {
      value = 'hpc.config'
   }
   process_nonest = 'hpc.config'
   withLabel:labelExample {
      memory = 400
   }
}

[bentsherman]

It is a suboptimal pattern, but it will be fixed by the new config parser

[pditommaso]

The may new config parser solve this problem, however is still under development and surely it's not going delivered soon.

The current best practice consists of using separate profiles as reported in the documentation

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[awgymer]

I have just stumbled on the same problems as this issue.

With a nextflow.config:

includeConfig './nextflow2.config'

process {
    withName: '(.*):?A' {
        ext.args = 'a'
        ext.args2 = 'a'
    }
}

Where the included config contains profiles, the profiles when selected are applied before - or possibly overwritten entirely by - the general top-level process block.

Contrast to:

process {
    withName: '(.*):?A' {
        ext.args = 'a'
        ext.args2 = 'a'
    }
}

includeConfig './nextflow2.config'

Here when using a profile from the included config it overrides whatever is in the bare top-level process definition.

I tested this both with and without NXF_SYNTAX_PARSER=v2 and also with the latest edge version 26.03.2-edge.

This indicates to me that Paolo is correct in saying that the new parser did not solve this issue? Interestingly the "red box" in the docs for config profiles is not present in the latest docs as far as I can tell, and the docs are not explicit one way or another whether profiles are applied after all top-level config only that they are applied in order with regards to other profiles.

The behaviour certainly seems a little squiffy when you consider this case:

includeConfig './nextflow2.config'   // defines profile `override`

process {
    withName: '(.*):?A' {
        ext.args = 'a'
        ext.args2 = 'a'
    }

    withName: '(.*):?B' {
        ext.args = 'b'
        ext.args2 = 'b'
    }
}

includeConfig './nextflow3.config'   // defines profile `override2`

And if you run NXF_SYNTAX_PARSER=v2 nextflow run main.nf -profile override2,override then you see that override2 does indeed get applied and overrides the top-level config, but override despite being after override2 on the command line and subject to Nextflow applies profiles in the order you specify them on the command line does not actually override the top-level config

[bentsherman]

@awgymer your issue is slightly different but very related

the original issue was that nested config options don't get resolved correctly between profiles and top-level config

the v2 syntax parser does fix this issue, but it also introduces another issue, which is that profiles are being applied too early, allowing them to be overridden by subsequent config. which also prevents it from actually solving the original issue 😅

we just need to update the v2 syntax parser to apply profiles at the very end. I will prepare a fix

[bentsherman]

The issue is also independent of config includes. Here is a minimal example using a single config:

profiles {
    special {
        process {
            ext.args = "special"
            cpus = 8

            withLabel: example {
                memory = 16.GB
            }
        }
    }
}

process {
    ext.args = ""
    cpus = 2

    withLabel: example {
        memory = 4.GB
    }
}