Cypher Questions

[rstyd]

This discussion is for asking questions about BEE's cypher code.

[kabir-vats]

@Boogie3D I have some clarification questions from the cypher code:

What does a 'BEGINS' edge mean? It appears to connect all tasks to the central workflow node, but there seems to be a complicated query to create the edge.

How does the workflow know a task is ready to be run with set_runnable_tasks_to_ready? The query checks if the input.values are not NULL, but wouldn’t the values always not be NULL? In this case, why have a separate method for set_init_tasks_to_ready?

Thank you for your help, I can tell you have put in a lot of work to implement neo4j on this project.

[Boogie3D]

Hi,

So the BEGINS edge connects the Workflow node specifically to the tasks which take Workflow Inputs as specified in the Workflow CWL file as their task inputs. If a task depends entirely on task outputs as their inputs then it will not have a BEGINS edge.

In general, all BEGINS edges are determined by matching task input sources to workflow inputs and DEPENDS_ON edges are determined by matching task input sources to task outputs.

set_init_tasks_to_ready specifically sets to ready the tasks whose inputs depend entirely on workflow inputs and not any other tasks, i.e. those tasks which have no DEPENDS_ON edges. set_runnable_tasks_to_ready checks if a task's inputs are entirely satisfied, either by workflow inputs or task outputs. I believe tasks which depend on other tasks for their input will have a NULL input.value field until set_task_input is invoked to set the task input.value. It may be possible for set_runnable_tasks_to_ready to subsume the role of set_init_tasks_to_ready, I just didn't really think of it at the time.

Let me know if you have any other questions!

Thanks,
Steven

[rstyd]

Hey @Boogie3D,

I was wondering if it's possible for the parser to parse inline inputs instead of them being included with a separate YAML file. The same way that the run section for a workflow doesn't necessarily need to be a link to another file.

As an example,

The lulesh workflow is

inputs:
  size: int
  iterations: int

outputs:
  lulesh_stdout:
    type: File
    outputSource: lulesh/lulesh_stdout

steps:
  lulesh:
    run:
      class: CommandLineTool
      baseCommand: [/lulesh2.0]
      stdout: lulesh_stdout.txt
      inputs:
        size:
          type: int
          inputBinding:
            prefix: -s
        iterations:
          type: int
          inputBinding:
            prefix: -i
      outputs:
        lulesh_stdout:
          type: stdout
    in:
      size: size
      iterations: iterations
    out: [lulesh_stdout]
    hints:
      DockerRequirement:
        beeflow:useContainer: /usr/projects/beedev/mpi/lulesh-x86_64.tgz
      beeflow:MPIRequirement:
        ntasks: 27

And the accompanying YAML file looks like

size: 20
iterations: 10

I'd like to change the cwl file so it looks like:

inputs:
  size: int
  iterations: int

outputs:
  lulesh_stdout:
    type: File
    outputSource: lulesh/lulesh_stdout

steps:
  lulesh:
    run:
      class: CommandLineTool
      baseCommand: [/lulesh2.0]
      stdout: lulesh_stdout.txt
      inputs:
        size:
          type: int
          inputBinding:
            prefix: -s
        iterations:
          type: int
          inputBinding:
            prefix: -i
      outputs:
        lulesh_stdout:
          type: stdout
    in:
      size: 20
      iterations: 10
    out: [lulesh_stdout]
    hints:
      DockerRequirement:
        beeflow:useContainer: /usr/projects/beedev/mpi/lulesh-x86_64.tgz
      beeflow:MPIRequirement:
        ntasks: 27

Is that actually possible/reasonable? We have a customer that would like to avoid using input YAML files if at all possible.

[Boogie3D]

Apologies for the late response. It looks like the CWL way to do it is to add a default field to the workflow input(s) (https://cwl.discourse.group/t/constructing-a-default-value-for-a-tool-input/161/4).

I don't think it would be hard to modify the CWL parser to handle this.

[Boogie3D]

However, I think the more conventional way to do this type of "quick-and-dirty" workflow parameter is to use the arguments field, which would need to be implemented in the parser. https://www.commonwl.org/user_guide/topics/additional-arguments-and-parameters.html

[rstyd]

@Boogie3D Hey it's been quite a while! Just had something come up, and I'm not entirely sure. What's the difference between the cwl outputs in the main cwl file and the step outputs in the other files?

[rstyd]

My impression is that we define step outputs so that they can then be used to define dependencies in the main clamr file. Is that basically correct?

[Boogie3D]

So the main cwl outputs are the outputs of the entire workflow. What you specify there is what should be output into the workdir, such as the clamr movie, time log, stdout, and ffmpeg stderr for example. Those are specified from the step outputs that you want to be output from the workflow by using the outputSource field. The remaining step outputs that are not designated as workflow outputs are only used by intermediate steps and should be stored only temporarily elsewhere.

[rstyd]

That's super helpful thanks!

[Boogie3D]

It should be noted these are also part of the CWL spec.