Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-enable subdags, but in a dynamic way #66

Open
johanneskoester opened this issue Feb 16, 2017 · 1 comment
Open

Re-enable subdags, but in a dynamic way #66

johanneskoester opened this issue Feb 16, 2017 · 1 comment
Assignees

Comments

@johanneskoester
Copy link
Contributor

johanneskoester commented Feb 16, 2017

It would be nice to use multiple build jobs (remember our SUBDAG feature) in case of bigger pull requests. However, for the regular pull request, this is not necessary and creates a lot of overhead. We should try to do the following:

  • the regular, automatic travis job investigates the DAG of packages that need to be build.
  • if the DAG is small, go on with building it
  • if the DAG is big, and can be split up into several disconnected subdags, trigger a new travis build (see API) with an appropriate build matrix set (e.g. defining n SUBDAG jobs). Then stop the current job.

This way, we get parallelism for connected subdags in case of large builds, and don't have additional overhead in case of small builds. If this works, it is a much superior solution compared to the feedstock approach, because we are able to automatically deal with the dependencies between packages without the downside that all builds run in a single job.

@bioconda/core, what do you think?

@epruesse
Copy link
Member

Work on this begun in #433

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants