Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Expand to work with other parameter types. #28

Open
cpaulik opened this issue Oct 28, 2016 · 3 comments
Open

Expand to work with other parameter types. #28

cpaulik opened this issue Oct 28, 2016 · 3 comments

Comments

@cpaulik
Copy link

cpaulik commented Oct 28, 2016

It would be great if SciLuigi could work with parameters other than strings.

I tried to use DictParameters and IntParameters in the MetaWF example and both failed.

Am I doing something wrong or is this just not supported?

e.g. for IntParameter

'''
An example showing how you can run multiple workflow tasks, from a "Meta workflow" (MetaWF below)
'''

import sciluigi as sl
import luigi

class MetaWF(sl.WorkflowTask):
    '''
    Meta workflow
    '''
    def workflow(self):
        tasks = []
        for r in [3, 2, 1]:
            wf = self.new_task('wf', WF, replacement=r)
            tasks.append(wf)
        return tasks

class WF(sl.WorkflowTask):
    '''
    Main workflow, which is run in multiple instances above
    '''
    replacement = luigi.IntParameter()
    def workflow(self):
        t1 = self.new_task('foowriter', FooWriter)
        t2 = self.new_task('foo2bar', Foo2Bar, replacement=self.replacement)
        t2.in_foo = t1.out_foo
        return t2

class FooWriter(sl.Task):
    '''
    A dummy task
    '''
    def out_foo(self):
        return sl.TargetInfo(self, 'foo.txt')
    def run(self):
        self.ex('echo foo > {foo}'.format(
            foo=self.out_foo().path))

class Foo2Bar(sl.Task):
    '''
    Another dummy task
    '''
    replacement = luigi.IntParameter()
    in_foo = sl.TargetInfo(None, 'None')
    def out_bar(self):
        return sl.TargetInfo(self, self.in_foo().path + '.{r}.txt'.format(r=str(self.replacement)))
    def run(self):
        self.ex('sed "s/foo/{r}/g" {inf} > {outf}'.format(
            r=type(self.replacement),
            inf=self.in_foo().path,
            outf=self.out_bar().path)
        )

# Run as script
if __name__ == '__main__':
    sl.run_local(main_task_cls=MetaWF)
@samuell
Copy link
Member

samuell commented Nov 2, 2016 •
edited
Loading

Thanks for reporting this @cpaulik !

IIRC, there were some issues related to the dynamic task instantiation that sciluigi does, that was a bit problematic with non-string types. But I need to check back into this, whether it was really a problem, or if we just didn't try hard enough.

So far, we have been going with just string representation of any parameters, because it was just simpler, but I agree that it would be nice with proper typed parameters.

Hope to find time to look at this soon.

@samuell
Copy link
Member

samuell commented Nov 8, 2016

Hi @cpaulik , in what sense did the workflow fail?

When I run the script, I do get the files 'foo.txt', 'foo.txt.1.txt', 'foo.txt.2.txt', 'foo.txt.3.txt' ... in spite of the warning messages (for which I've created issue #29). Did you also get this result?

Output from running this, and listing the files:

[samuel sciluigi_metawf]$ python metawf.py 
No handlers could be found for logger "luigi-interface"
2016-11-08 18:11:35 |     INFO | --------------------------------------------------------------------------------
2016-11-08 18:11:35 |     INFO | SciLuigi: MetaWF Workflow Started (logging to log/workflow_metawf_started_20161108_171135_535883.log)
2016-11-08 18:11:35 |     INFO | --------------------------------------------------------------------------------
2016-11-08 18:11:35 |     INFO | --------------------------------------------------------------------------------
2016-11-08 18:11:35 |     INFO | SciLuigi: WF Workflow Started (logging to log/workflow_wf_started_20161108_171135_539076.log)
2016-11-08 18:11:35 |     INFO | --------------------------------------------------------------------------------
/home/samuel/opt/miniconda2/lib/python2.7/site-packages/luigi/parameter.py:259: UserWarning: Parameter WF(instance_name=wf, replacement=1) is not of type string.
  warnings.warn("Parameter {0} is not of type string.".format(str(x)))
2016-11-08 18:11:35 |     INFO | --------------------------------------------------------------------------------
2016-11-08 18:11:35 |     INFO | SciLuigi: WF Workflow Started (logging to log/workflow_wf_started_20161108_171135_538974.log)
2016-11-08 18:11:35 |     INFO | --------------------------------------------------------------------------------
/home/samuel/opt/miniconda2/lib/python2.7/site-packages/luigi/parameter.py:259: UserWarning: Parameter WF(instance_name=wf, replacement=2) is not of type string.
  warnings.warn("Parameter {0} is not of type string.".format(str(x)))
2016-11-08 18:11:35 |     INFO | --------------------------------------------------------------------------------
2016-11-08 18:11:35 |     INFO | SciLuigi: WF Workflow Started (logging to log/workflow_wf_started_20161108_171135_538853.log)
2016-11-08 18:11:35 |     INFO | --------------------------------------------------------------------------------
/home/samuel/opt/miniconda2/lib/python2.7/site-packages/luigi/parameter.py:259: UserWarning: Parameter WF(instance_name=wf, replacement=3) is not of type string.
  warnings.warn("Parameter {0} is not of type string.".format(str(x)))
2016-11-08 18:11:35 |     INFO | Task foowriter started
2016-11-08 18:11:35 |     INFO | Executing command: echo foo > foo.txt
2016-11-08 18:11:35 |     INFO | Task foowriter finished after 0.004s
2016-11-08 18:11:36 |     INFO | Task foo2bar started
2016-11-08 18:11:36 |     INFO | Executing command: sed "s/foo/<type 'int'>/g" foo.txt > foo.txt.1.txt
2016-11-08 18:11:36 |     INFO | Task foo2bar finished after 0.005s
2016-11-08 18:11:36 |     INFO | --------------------------------------------------------------------------------
2016-11-08 18:11:36 |     INFO | SciLuigi: WF Workflow Finished (workflow log at log/workflow_wf_started_20161108_171135_539076.log)
2016-11-08 18:11:36 |     INFO | --------------------------------------------------------------------------------
2016-11-08 18:11:36 |     INFO | Task foowriter started
2016-11-08 18:11:36 |     INFO | Executing command: echo foo > foo.txt
2016-11-08 18:11:36 |     INFO | Task foowriter finished after 0.004s
2016-11-08 18:11:37 |     INFO | Task foo2bar started
2016-11-08 18:11:37 |     INFO | Executing command: sed "s/foo/<type 'int'>/g" foo.txt > foo.txt.2.txt
2016-11-08 18:11:37 |     INFO | Task foo2bar finished after 0.005s
2016-11-08 18:11:37 |     INFO | --------------------------------------------------------------------------------
2016-11-08 18:11:37 |     INFO | SciLuigi: WF Workflow Finished (workflow log at log/workflow_wf_started_20161108_171135_538974.log)
2016-11-08 18:11:37 |     INFO | --------------------------------------------------------------------------------
2016-11-08 18:11:37 |     INFO | Task foowriter started
2016-11-08 18:11:37 |     INFO | Executing command: echo foo > foo.txt
2016-11-08 18:11:37 |     INFO | Task foowriter finished after 0.004s
2016-11-08 18:11:37 |     INFO | Task foo2bar started
2016-11-08 18:11:37 |     INFO | Executing command: sed "s/foo/<type 'int'>/g" foo.txt > foo.txt.3.txt
2016-11-08 18:11:37 |     INFO | Task foo2bar finished after 0.005s
2016-11-08 18:11:37 |     INFO | --------------------------------------------------------------------------------
2016-11-08 18:11:37 |     INFO | SciLuigi: WF Workflow Finished (workflow log at log/workflow_wf_started_20161108_171135_538853.log)
2016-11-08 18:11:37 |     INFO | --------------------------------------------------------------------------------
2016-11-08 18:11:37 |     INFO | --------------------------------------------------------------------------------
2016-11-08 18:11:37 |     INFO | SciLuigi: MetaWF Workflow Finished (workflow log at log/workflow_metawf_started_20161108_171135_535883.log)
2016-11-08 18:11:37 |     INFO | --------------------------------------------------------------------------------
[samuel sciluigi_metawf]$ lltr
total 28K
-rw-rw-r--  1 samuel samuel 1,5K nov  8 18:08 metawf.py
drwxrwxr-x  2 samuel samuel 4,0K nov  8 18:11 log
-rw-rw-r--  1 samuel samuel   13 nov  8 18:11 foo.txt.1.txt
-rw-rw-r--  1 samuel samuel   13 nov  8 18:11 foo.txt.2.txt
-rw-rw-r--  1 samuel samuel    4 nov  8 18:11 foo.txt
-rw-rw-r--  1 samuel samuel   13 nov  8 18:11 foo.txt.3.txt
drwxrwxr-x 14 samuel samuel 4,0K nov  8 18:11 audit
[samuel sciluigi_metawf]$

@cpaulik
Copy link
Author

cpaulik commented Nov 8, 2016

The example might have been too simple. It fails in the sense that it does not produce the correct result. In its current state it runs through but all the files say string instead of int since the type is not respected. Originally I've tried with a DictParameter in which case it really did fail with a backtrace.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@samuell @cpaulik