Plugins added from local storage or pulled from self hosted registry are not scanned for commands

[ghost]

When adding an image from Docker Hub with a LABEL containing the command, the CS plugin automatically populates the command list as it should. When adding an image that exists locally or pulling from a locally hosted registry this step does not occur. After adding the container I can manually copy/paste the command JSON in and the container will be available and runs fine. Also, doing an inspect on the container shows the LABEL is present and properly formatted.

To confirm it wasn't an error on my part I created a simple test container, pushed to my local registry, and added it to XNAT. The command did not appear. I then pushed it to Docker Hub, added to XNAT from there, and the command populated fine.

I also reversed the process and pulled one of the example containers from the XNAT Docker public repository and the command populated automatically. I then used the corresponding Dockerfile from github, built the container locally, pushed to my local repo, and added to XNAT but no command populated. And just for completeness sake, pushed the same image to a public Docker Hub repo, added it again, and the command populated just fine.

This behavior is consistent in versions 1.5.1, 2.0.0-RC1, and 2.0.0-RC2 running on XNAT 1.7.4.1

[johnflavin]

To reproduce this, I would need to have a local registry running. Do you know an easy way to set one up?

As a workaround, there is an API to tell container service to check for commands in the labels of an image stored locally: /xapi/docker/images/save?image=<image>. Can you see if that works correctly for images pulled from a local registry that weren't automatically saved?

[ghost]

Running a local registry is as simple as docker run -d -p 5000:5000 --restart=always --name registry registry:2 then tag and push an image specifying the new registry. More documentation can be found here explaining everything in a little more detail.

Manually calling that API endpoint produces mixed results.

Image stored in local registry: API returns command JSON and XNAT updates the command list.

Image stored on local machine (never pushed to a registry, just built and tagged on the machine running XNAT): API returns command JSON but XNAT does NOT update the command list.

I realize running XNAT, a container registry, and doing my dev work all on the same machine probably isn't something you planned for when developing the plugin. Thankfully it isn't a permanent setup, just a quick environment I cobbled together to test getting our pipelines into containers and executed using your plugin. It is looking like we can accomplish everything we need so I'll be expanding my efforts and changing to a slightly more sane configuration this week. That may make this no longer an issue for me, but if the behavior is unintended I'll do what I can to continue testing and providing feedback for this particular scenario.

[johnflavin]

In my efforts to reproduce I have found problems trying to use a local registry, but they are different from the problems you've pointed out. I'm trying to catalog all the different ways this is going wrong before figuring out how to fix it.

• How did you add the image to the registry? What tag did you give it?
• How did you pull that image? Did you use the container service UI? The REST API? If the latter, what API and what params did you give it?

[ghost]

Well this one is on me. Using the XNAT UI to get an image from a local registry is broken. I'm assuming because the : used for the host:port is being misinterpreted as the tag specifier, but haven't checked through the source to verify that. Since the CS plugin enumerates every container present on the host I've been abusing that to get images into XNAT by pulling from the registry myself as part of the deploy process I put in place. Sorry for the incomplete reproduction steps, I've been going at this for a few days now testing workflows strictly local and some that hit an outside registry (poorly simulated with a localhost registry, probably not the best choice...) Some of the details have blurred together.

For timeliness sake I created a free account with quay.io and pushed an image there, cleared all local copies of it, confirmed the CS plugin had no mention of the image, then used the CS plugin to successfully pull the image. It completed without issue and the command is populated.

This makes me think there is no issue with populating from an outside, 'unofficial' registry, only with the CS plugin not supporting image names that contain a host:port in their specifier. I will still be creating a local registry on a dedicated host that can be accessed without needing a port specified. Will update once I can confirm it works as intended, and if so I'll create a second issue for the host:port issue to keep this from getting any messier than it already has.

This issue can probably be reduced to just tracking down why locally created images are being automatically enumerated, but their commands not parsed and added.

[ghost]

TL;DR:

• Specifying a port when pointing to a registry is breaking the CS plugin. It will pull the image but not parse the command.
• Locally built images are automatically enumerated and listed, but their command is not parsed out.
• Discovered that entering a local registry in the UI is only partially broken. You can specify a port in the Image Name field without issue. You can also specify a port and a tag in the Image Name box. However, if you enter a port in the Image Name but put the tag in the Version Tag box XNAT returns Error 400: Cannot pull an image by ID. This would explain my failures during earlier testing as I was specifying a tag to keep track of changes as I iterated on my container and pipeline.

Long version:
Just tested a few registry setups I felt would be most common. Here are the results, with steps taken along the way. Assumes you have an account on Docker Hub and a 3rd party registry (again I chose quay.io, they were the first one I found on Google this morning...). Really one or the other will suffice.

1. Cleared all containers and images from XNAT server and registry server to give a clean slate:
   docker rm $(docker ps -a -q)
docker rmi $(docker images -q)

2. Started self-hosted registries:
   On dedicated registry server:
   docker run -d --restart=always -p 80:5000 registry:2
docker run -d --restart=always -p 5000:5000 registry:2
   On XNAT server:
   docker run -d --restart=always -p 5000:5000 registry:2
   Note: Did not test on port 80 on XNAT server as it is setup to auto-forward to 443 when serving XNAT to enforce HTTPS. Seemed unlikely anyone would want to disable that.
   Note 2: Be sure to add the both dedicated registries to your /etc/docker/daemon.json as an insecure repository
   or if you have certs already you can adjust the above to specify them Docker will check HTTPS before falling back to HTTP, so no need to alter any commands below if you do add certs.

3. Pull sample image from Docker Hub to XNAT server, retag, and push to each registry:
   docker pull xnat/debug-command
docker tag xnat/debug-command quay.io/<username>/my-debug
docker tag xnat/debug-command localhost:5000/my-debug
docker tag xnat/debug-command registry.foo.com/my-debug
docker tag xnat/debug-command registry.foo.com:5000/my-debug
docker push quay.io/<username>/my-debug
docker push localhost:5000/my-debug
docker push registry.foo.com/my-debug
docker push registry.foo.com:5000/my-debug

4. Clean up local images so CS plugin doesn't see them
   docker rmi quay.io/mjbledsoe/my-debug
docker rmi localhost:5000/my-debug
docker rmi registry.foo.com/my-debug
docker rmi registry.foo.com:5000/my-debug

5. (Optional) Verify container is in each repo (JSON returned should list your image):
   login to quay.io like some kind of caveman and check manually
   (Also while you're checking this, change the repo to public if it isn't already)
   curl -X GET http://localhost:5000/v2/_catalog
curl -X GET http://registry.foo.com:5000/v2/_catalog
curl -X GET http://registry.foo.com/v2/_catalog

6. Login to XNAT and check Images and Commands to see the xnat/debug-command listed from the docker pull done earlier. It should have no command populated confirming the 'local build is broken' report above.

7. Delete the xnat/debug-command Image.

8. Using Add New Image pop-up, test all the registries using these Image Name entries:
   (Note: you need to delete the image pulled at each step to make it clear which registry is having which outcome)
   xnat/debug-command to test Docker Hub
   image pulls, command populates
   quay.io/<username>/my-debug to test 3rd party registry
   image pulls, command populates
   localhost:5000/my-debug to test local registry
   image pulls, command DOES NOT populate
   registry.foo.com/my-debug to test dedicated registry without port
   image pulls, command populates
   registry.foo.com:5000/my-debug to test dedicated registry with specified port
   image pulls, command DOES NOT populate

[johnflavin]

Thanks so much for testing all of this! It's a lot to go through, so it might take me some time to fix things. I can give my general impressions now, though:

• I will verify the problem with port numbers in registries and track that down. 👍
• Locally-built images won't be automatically saved by container service. This is the expected behavior. To save them, you need to explicitly tell container service to go read the image labels and save the commands. There currently is no button for this in the user interface, but there is a REST API. It would not be too hard to get a button in the UI to do this, though.
• The "Add a new image" UI is due for a refresh. When we do that I'll keep the problems with specifying registries in mind.

[ghost]

Glad to be able to give something back. Thanks for all the work you (and the rest of the XNAT devs) have put into this. Looking forward to the next release.