[ROS] trigger rebuild request

[ruffsl]

Most of the current tags for ROS are about a month old. We've since then releases several syncs into current releases that have not yet made their way into images in the docker up registry. We would like to request a rebuild for the current set of tags. This disjoint in pinned version and releases is I think in part to the limitations of our debian repositories system: recent context.

I've notice that pipeline for ROS has remained active, however are those purly test jobs I'm seeing rather than build jobs?

By the way, would there be webhook yet that trusted maintainers could use to ping for this. I think the last time I ask I was suggested to simply message the librarians directly.

Thanks!

[yosifkit]

The pipeline you linked is the build we would poke to rebuild the image. We do not have a way to manually trigger a build beyond that. Since the base image, and bits from the library have not changed (like the GitCommit and Directory), it uses our bashbrew cache tags and skips building entirely. Even if it did get to a docker build the lines of the Dockerfile have not changed and thus would be entirely covered by docker build cache. I believe there should be Debian and Ubuntu image refreshes in the next week or so that will cause rebuilds of the images based upon them (since it has been about a month since the last refresh).

If you change your Dockerfiles to use the full version of the ROS packages, rather than 1.3.1-0* it would provide a natural cache bust when you want to update the images to contain the new changes.

As for your query in your linked post:

I haven’t found a good way to retrieve the archived list of available digests for a given repo from the docker registry as of yet, but would like to know

Since the Docker Hub does not display this sort of information, see the ros directory in docker-library/repo-info. Which is also linked from the Docker Hub description under "Published image artifact details".

[tianon]

(for example, the current full version number for ros-kinetic-ros-core is 1.3.1-0xenial-20171117-082647-0800, which would be a very natural cache bust)

[mikaelarguedas]

@ruffsl Once we automate the dockerfile generation on each sync, it should be easy to provide a full version number in each dockerfile (I mean we could also update the scripts right now independently of how we trigger the Dockerfiles generation).
The upside is that this would ensure that we get a rebuild every time we sync a distribution.
The downside is that this means creating PRs here every 2 weeks (sync frequency) for each ROS distribution (~8 PRs a month) so more overhead for ROS maintainers as well as Docker maintainers.

One blocker is that we have different version numbers for each architecture as there is also almost zero probability that the packages on different architectures are built at the exact same time.
Not sure what would be best there? should we provide different dockerfiles for each architecture?

[ruffsl]

Since the Docker Hub does not display this sort of information, see the ros directory in docker-library/repo-info. Which is also linked from the Docker Hub description under "Published image artifact details".

Oh ya! This is so cool, learn something new every day! I think that repo should be referenced in the docker doc page I linked about downloading by digest for decoverabilty. I've ticketed such here: docker/docs#5721

The downside is that this means creating PRs here every 2 weeks (sync frequency) for each ROS distribution (~8 PRs a month) so more overhead for ROS maintainers as well as Docker maintainers.

True, that would be rather frequent for maintainers. We could potentially automate the upstream PR that point here, as such to be triggered by our syncs, via our osrf-docker-pr bot. That being if the librarians here would be ok with that.

One blocker is that we have different version numbers for each architecture as there is also almost zero probability that the packages on different architectures are built at the exact same time.
Not sure what would be best there? should we provide different dockerfiles for each architecture?

Oh no... I would not look forward to polethra of more dockerfiles and bookkeeping that would result. Could we use more creative wildcards with apt-get or regex to circumvent the issue?

[tianon]

True, that would be rather frequent for maintainers. We could potentially automate the upstream PR that point here, as such to be triggered by our syncs, via our osrf-docker-pr bot. That being if the librarians here would be ok with that.

~8 PRs a month would be perfectly fine via bot (as long as you monitor comments for the bot's PRs, or auto-subscribe yourself) -- we've got a few image maintainers that already do that today.

Oh no... I would not look forward to polethra of more dockerfiles and bookkeeping that would result. Could we use more creative wildcards with apt-get or regex to circumvent the issue?

Yeah, please don't do a separate Dockerfile per arch. Worst case, you could do something like the following:

RUN set -eux; \
	dpkgArch="$(dpkg --print-architecture)"; \
	case "$dpkgArch" in \
		armhf) version='1.3.1-0xenial-...' ;; \
		amd64) version='...' ;; \
		... \
		*) echo >&2 "unsupported architecture: $dpkgArch"; exit 1 ;; \
	esac; \
	apt-get update; \
	apt-get install -y "...=$version"; \
	...

Or even something like this, if you wanted to use explicit ENV vars instead:

ENV ROS_VERSION_armhf="1.3.1-0xenial-..." ROS_VERSION_amd64="..." ROS_...
RUN set -eux; \
	dpkgArch="$(dpkg --print-architecture)"; \
	eval "version=\"\$ROS_VERSION_$dpkgArch\""; \
	apt-get update; \
	apt-get install -y "...=$version"; \
	...

[mikaelarguedas]

@tianon Thanks I'll look into doing that 👍
As I'm not sure of when I'll get around implementing this, in the meantime is it ok for you if I open issues like this one when we need to trigger a rebuild ?

[yosifkit]

@mikaelarguedas, an issue requesting a rebuild would not cause any change to the image since the library/ros unique bits for the image did not change (as noted above). The only way to cause an image to rebuild is for the base image to change (an update by Ubuntu) or to change the Dockerfile to break Docker build cache and make a PR to update the library file (like embedding a more specific version and making a bump).

[mikaelarguedas]

Oh right, I'll work on implementing the logic to provide complete version numbers as soon as I have spare cycles. Issue moved to osrf/docker_images#112, feel free to close this one.

[tianon]

👍