The defined --defaultMemory does not apply to CWL jobs without memory values and 256 is always used instead

[ndonyapour]

It seems that the specified value for defaultMemory is being overridden by the default value of 1GiB. Below is the command I used to run my workflow

toil-cwl-runner --outdir outdir_toil --jobStore file:./jobStore_bbbc --batchSystem slurm --defaultMemory 20GiB --defaultDisk 5Gi --statePollingWait 0 --retryCount 0 --maxCores 128 --maxLocalJobs 128 --disableAutoDeployment True --stats --clusterStats clusterStats.json --singularity --workDir /data1 --coordinationDir /data1 --disableCaching --disableProgress --cleanWorkDir never --clean never --slurmArgs --export=ALL --partition=normal_gpu --error=serror.txt --relax-path-checks --logLevel DEBUG autogenerated/bbbc.cwl autogenerated/bbbc_inputs.yml```

and this what I see when the jobs are submitted where --mem=256

[2024-11-19T14:06:11-0500] [Thread-3] [D] [toil.lib.misc] run command: sbatch -J toil_job_2_bbbc.cwl.bbbc__step__1__BbbcDownload.BbbcDownload.cwl --signal=B:INT@30 --export=ALL --partition=normal_gpu --error=serror.txt --export=ALL,TOIL_RT_LOGGING_ADDRESS=172.16.1.110:55351,TOIL_RT_LOGGING_LEVEL=INFO,OMP_NUM_THREADS=1 --mem=256 --cpus-per-task=1 -o /data1/toil_5aaa976f-2aa1-4300-8987-84102afb8284.2.%j.out.log -e 

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1676

[adamnovak]

I think this is a CWL spec feature (and maybe a bug in Toil that we expose a flag called defaultMemory from the CWL runner at all?). See #4971 (comment); the CWL specification is where that 256 number is coming from. The Toil --defaultMemory applies to Toil's internal jobs that don't have a memory requirement assigned, but CWL jobs always have a memory requirement assigned: either 256 from the spec or whatever the user specified.

This also might be related to #2696 where we talked about making the Toil default parameters do something for CWL.

@ndonyapour Presumably you are looking for a feature in Toil that would let you override the CWL specification's required behavior, and use a different memory limit on jobs that don't have a CWL-level memory value set?

The other solution is probably hiding the --defaultX flags from toil-cwl-runner, to not make it look like we have this feature when we don't.

[ndonyapour]

This makes sense; thank you for your response! Adding ramMin to the ResourceRequirement of the CWL tool fixed the issue. However, having a feature in Toil that allows overriding the CWL specification would be a great addition!