Support include in (sub) workflow body (DSL2)

[mjhipp]

I am wondering if it would be possible to support include statements within the main body of a sub-workflow. My use case is that I would like to implicitly pass parameters to downstream processes/workflows that can determine the publishing directory (a workaround for not having publish at workflow level: #1933 (comment)).

For example, a single sample workflow takes a sample name, and uses that value to assign params.publishDir for all sub-workflows and processes within the single sample scope. Here is an example where a single-sample workflow calls a single process, foo:

Process:

process foo {
  publishDir params.samplePublishDir, enabled: params.samplePublishDir
  
  output:
    path 'foo.txt', emit: foo_ch

  "touch foo.txt"
}

Single sample workflow:

nextflow.enable.dsl=2

workflow SINGLESAMPLE {
  take:
    sample_name

  main:
    // Assign here so I have access to `sample_name` input variable.
    params.samplePublishDir = "$params.publishRoot/$sample_name"
    // By including here, it inherits the params defined right above implicitly
    include { foo } from './foo.nf'
    foo()
}

// for cli access
workflow {
  SINGLESAMPLE(params.sample_name)
}

This actually works great when calling that single sample workflow:

❯ ./nextflow run SINGLESAMPLE.nf --sample_name sample1 --publishRoot ./test-output
N E X T F L O W  ~  version 21.03.0-edge
Launching `SINGLESAMPLE.nf` [dreamy_lovelace] - revision: b9c0bee9f8
executor >  local (1)
[5b/5394ec] process > SINGLESAMPLE:foo [100%] 1 of 1 ✔

❯ tree test-output
test-output
└── sample1
    └── foo.txt

However, it breaks when importing that workflow into an outer (multi-sample for example) workflow:

nextflow.enable.dsl=2

include { SINGLESAMPLE } from './singlesample'

workflow MULTISAMPLE {
  take:
    sample_names // list of sample names

  main:
    SINGLESAMPLE(channel.fromList(sample_names))
}


// for cli access
params.sample_names = ['sample1', 'sample2', 'sample3']

workflow {
  MULTISAMPLE(params.sample_names)
}

❯ nextflow run MULTISAMPLE.nf  --publishRoot ./testmulti
N E X T F L O W  ~  version 21.03.0-edge
Launching `MULTISAMPLE.nf` [thirsty_jepsen] - revision: b2065896e9
Missing process or function with name 'foo'

 -- Check script 'MULTISAMPLE.nf' at line: 11 or see '.nextflow.log' file for more details

.nextflow.log

Mar-25 16:31:46.316 [main] DEBUG nextflow.Session - Session start invoked
Mar-25 16:31:46.530 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Mar-25 16:31:46.628 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: 
Mar-25 16:31:46.789 [main] DEBUG nextflow.Session - Session aborted -- Cause: Missing process or function with name 'foo'
Mar-25 16:31:46.803 [main] ERROR nextflow.cli.Launcher - @unknown
nextflow.exception.MissingProcessException: Missing process or function with name 'foo'
	at nextflow.script.WorkflowDef.run(WorkflowDef.groovy:173)
	at nextflow.script.BindableDef.invoke_a(BindableDef.groovy:52)
	at nextflow.script.ComponentDef.invoke_o(ComponentDef.groovy:41)
	at nextflow.script.WorkflowBinding.invokeMethod(WorkflowBinding.groovy:95)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:397)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:339)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:61)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:185)
	at Script_204df2aa$_runScript_closure1$_closure3.doCall(Script_204df2aa:11)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
	at groovy.lang.Closure.call(Closure.java:412)
	at groovy.lang.Closure.call(Closure.java:406)
	at nextflow.script.WorkflowDef.run0(WorkflowDef.groovy:186)
	at nextflow.script.WorkflowDef.run(WorkflowDef.groovy:170)
	at nextflow.script.BindableDef.invoke_a(BindableDef.groovy:52)
	at nextflow.script.ComponentDef.invoke_o(ComponentDef.groovy:41)
	at nextflow.script.WorkflowBinding.invokeMethod(WorkflowBinding.groovy:95)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:397)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:339)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:61)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:185)
	at Script_204df2aa$_runScript_closure2$_closure4.doCall(Script_204df2aa:18)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
	at groovy.lang.Closure.call(Closure.java:412)
	at groovy.lang.Closure.call(Closure.java:406)
	at nextflow.script.WorkflowDef.run0(WorkflowDef.groovy:186)
	at nextflow.script.WorkflowDef.run(WorkflowDef.groovy:170)
	at nextflow.script.BindableDef.invoke_a(BindableDef.groovy:52)
	at nextflow.script.ChainableDef$invoke_a.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:139)
	at nextflow.script.BaseScript.runDsl2(BaseScript.groovy:191)
	at nextflow.script.BaseScript.run(BaseScript.groovy:200)
	at nextflow.script.ScriptParser.runScript(ScriptParser.groovy:221)
	at nextflow.script.ScriptRunner.run(ScriptRunner.groovy:212)
	at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:120)
	at nextflow.cli.CmdRun.run(CmdRun.groovy:297)
	at nextflow.cli.Launcher.run(Launcher.groovy:475)
	at nextflow.cli.Launcher.main(Launcher.groovy:657)
Caused by: groovy.lang.MissingMethodException: No signature of method: nextflow.script.WorkflowBinding.foo() is applicable for argument types: () values: []
Possible solutions: any(), find(), use([Ljava.lang.Object;), is(java.lang.Object), tap(groovy.lang.Closure), find(groovy.lang.Closure)
	at nextflow.script.WorkflowBinding.invokeMethod(WorkflowBinding.groovy:103)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:397)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:339)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:61)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:176)
	at Script_7cd8d3ac$_runScript_closure1$_closure3.doCall(Script_7cd8d3ac:12)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
	at groovy.lang.Closure.call(Closure.java:412)
	at groovy.lang.Closure.call(Closure.java:406)
	at nextflow.script.WorkflowDef.run0(WorkflowDef.groovy:186)
	at nextflow.script.WorkflowDef.run(WorkflowDef.groovy:170)
	... 57 common frames omitted

I am wondering if it would be possible to add the ability to recognize include within a sub-workflow body, rather than just the top of the file.

Alternatively, I would like to know if there is a better way of achieving my goal, which is to let a single sample workflow define the publishing behavior of all sub-workflows/processes, while also being part of an upstream multi-sample workflow.