Passing parameters into the workflow?

[BioComSoftware]

Morning!

So, we want to run a pipeline that uses multiple files. Is there a way to pass the file list into the workflow?

Using your tutorial example,

class MyWorkflow(sciluigi.WorkflowTask):
    # Maybe...
    # repl = sciluigi.Parameter()
    # ...or....
    # repl = luigi.parameter.ListParameter([])        

    # ...or ...
    def workflow(self, repl):
    # ...instead of ...    
    # def workflow(self):
    # or something like that ? :D
        foowriter = self.new_task('foowriter', MyFooWriter)
        # Use the parameter here
        fooreplacer = self.new_task('fooreplacer', MyFooReplacer, replacement=repl)
        fooreplacer.in_foo = foowriter.out_foo
        return fooreplacer        
        
if __name__ == '__main__':
    sciluigi.run_local(main_task_cls=MyWorkflow, repl = 'thisTextInstead')    

[BioComSoftware]

Ive tried a couple of variations of the above and, while some of them are allowed by Luigi or sciluigi...none of them allow me to actually pass the parameter at the sciluigi.run_local line....

[BioComSoftware]

OK. Nevermind :) It works if I use the command line with 'luigi etc etc'. I was trying to get it to run from "if name == main" :)

If I run it from the command line using ...

bash-3.2# luigi --module Bioproximity.sciluigi_tasks.PipelineTest1 MyWorkflow  --workers 2 --local-scheduler --f 10

...it accepts the parameters fine.

[BioComSoftware]

So, I reopened this instead of creating a new issue.

While I am able to pass in parameters from command line, I'm still looking at addressing the issue of the workflow needing to run multiple files (not just one file only).

So, the workflow concept (where the workflow method builds all the tasks first, and then runs then in sequence at the 'return' call) means looping the workflow method won't work.

Since there will always be an unknown number of files passed in as a list (from 1 to n) ... is there an existing way to have the workflow run each file independently (without having to call the luigi command line 1 to 'n' times)?

Thanks!