Enabling original Luigi web view?

[dioh]

When I run example2 I cannot see the results in the static visualiser.
http://localhost:8082/static/visualiser/index.html

Is there any way to get tasks written in sciluigi to be visualized in the webapp?

Kindly, D.

[samuell]

Hi @dioh ,

Tasks have always shown up for us. For example, figure 7 in our paper is from running with SciLuigi.

I think what might happen could be that the tasks run so fast, that they are not retained in the web view.

I have some memory that we neede to add some specific setting to have finished tasks remain in the view. Otherwise, you could try adding a 60 second sleep inside the tasks?

[samuell]

@dioh Ah, yes, now I found the setting. It is the remove-delay setting, under the [scheduler] section, in your client.cfg file.

You can see how we set it to a large number here.

Cheers

[dioh]

Dear @samuell, thanks for your help.
I enabled the configuration as you stated:

export LUIGI_CONFIG_PATH=luigi.cfg
luigid --background --port=8082 --logdir=logs

Then I run example2_ngi from your repo.
python2 example2_ngi.py

I see the workflow stdout messages:

2018-02-05 13:36:16 | INFO | --------------------------------------------------------------------------------
2018-02-05 13:36:16 | INFO | SciLuigi: NGITestWF Workflow Started (logging to log/workflow_ngitestwf_started_20180205_163616_935151.log)
2018-02-05 13:36:16 | INFO | --------------------------------------------------------------------------------
/home/danito/.local/lib/python2.7/site-packages/luigi/parameter.py:261: UserWarning: Parameter "workflow_task" with value "NGITestWF(instance_name=sciluigi_workflow, task=merge)" is not of type string.
warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
2018-02-05 13:36:16 | INFO | Task run10min started
2018-02-05 13:36:16 | INFO | Executing command: wc -l data/acgt.txt
2018-02-05 13:36:16 | INFO | COMMAND: head -n 2 data/acgt.txt > data/acgt.txt.part1
2018-02-05 13:36:16 | INFO | Executing command: head -n 2 data/acgt.txt > data/acgt.txt.part1
2018-02-05 13:36:16 | INFO | Executing command: tail -n 2 data/acgt.txt > data/acgt.txt.part2
2018-02-05 13:36:16 | INFO | Task run10min finished after 0.027s
2018-02-05 13:36:17 | INFO | Task dosth2 started
2018-02-05 13:36:17 | INFO | Task dosth2 finished after 0.001s
2018-02-05 13:36:17 | INFO | Task dosth1 started
2018-02-05 13:36:17 | INFO | Task dosth1 finished after 0.001s
2018-02-05 13:36:17 | INFO | Task merge started
2018-02-05 13:36:17 | INFO | Executing command: cat data/acgt.txt.part1.something_done data/acgt.txt.part2.something_done > data/acgt.txt.part1.something_done.merged
2018-02-05 13:36:17 | INFO | Task merge finished after 0.009s
2018-02-05 13:36:17 | INFO | --------------------------------------------------------------------------------
2018-02-05 13:36:17 | INFO | SciLuigi: NGITestWF Workflow Finished (workflow log at log/workflow_ngitestwf_started_20180205_163616_935151.log)
2018-02-05 13:36:17 | INFO | --------------------------------------------------------------------------------

and I verify the output in data/*merged

So the example executes correctly but no data is shown in the visualiser. Am I missing something?

[samuell]

Hi @dioh ,

no data is shown in the visualiser

What do you mean with "data"? Do you mean that the visualizer is not showing anything, or that it is not showing the data outputs?

I don't think Luigi can show data outputs. I think it only shows tasks.

Does that clarify the situation?

[dioh]

To clarify I'm looking for the executed tasks. Not the data itself. I want to see the executed workflow in the Luigi Task Visualizer.

[samuell]

@dioh Ah, ok, good, then we're on the same page.

Well that's strange. Could you paste here the exact content of your client.cfg file?

[dioh]

Sure! It's the same as the one you shared but with my email.

cat $LUIGI_CONFIG_PATH

[core]
rpc-connect-timeout: 864000
worker-ping-interval: 30
; worker-keep-alive: True
error-email: [email protected]

[scheduler]
; The time before re-run objects in the web UI
; disappear. It is here set to one year (in seconds)
remove-delay=31536000

And my library versions:

pip2 freeze | grep luigi

luigi==2.7.2
sciluigi==0.9.6b7

[samuell]

Ok, thanks!

Oh, well ... please note that the examples are running without connecting to the luigi daemon, by default.

See hos it is running the run_local(...) command here.

Try changing that to run(...)

It might also work by specifying to use the daemon on the commandline, but I'm not sure which setting will then take precedence, so I'd recommend trying first to change it to run(...) in the code, to see whether that helps.

[dioh]

🤦‍♂️
It was that.

Thanks a lot. One quick question now that I have your attention,
If I work in a pull request regarding #7, or a simple version to do composition of workflows (ie, given A, B workflows, create C workflow that runs A and then B doing the wiring "automatically") would you consider adding it? Do you continue with this project or you changed entirely to scipipe?

[samuell]

If I work in a pull request regarding #7, or a simple version to do composition of workflows (ie, given A, B workflows, create C workflow that runs A and then B doing the wiring "automatically") would you consider adding it? Do you continue with this project or you changed entirely to scipipe?

Sure, I would be happy to consider adding that.

I think the best is if you can first provide a "proof of principle" implementation, to demonstrate the approach you plan to take, before spending too much time on it, so that I can review and possibly give any comments on that as early as possible. But any concrete solution that works, will be a great starting point for any further modifications of course, and thus very welcome.

Also, I think I need to set up some continuous integration, to be able to merge things with ease, first. I'll have a look if I can get that up shortly ... like this week (CircleCI has been pretty quick to set up for some of my other repos).

Regarding the situation for the project, we are not currently actively using it (we run scipipe now in production), but we'd like to keep it in good shape at least during this year or so, when I'm finishing my PhD studies. After that we'll have to look at what is the best future for the project ... whether I, or someone within @pharmbio wants to continue maintaining it, or if there's someone else interested in maintaining it.