docker compose pull has no options for pull policy

[FFdhorkin]

Description

I was having issues with my docker compose project when I tried to run it from within a Github Action. Seems like for some reason, on Github, I get rate limited for pulls, but don't locally.

This led to me coming up with the workaround of running docker compose --parallel 1 pull before the docker compose up

I noticed that even though the docker cache github action was showing corresponding cached images, the docker compose --parallel 1 pull was resulting in images being pulled even though they were already in the cache (I was still getting occasional rate limit pull failures on cached images)

In other words, there's no way to say docker compose pull --missing-only the way you can do docker compose up --pull missing

This led to me attempting the following even hackier workaround:

docker compose --parallel 1 up --pull missing --no-start
# the above line breaks docker compose up; I guess it's expecting docker compose start, but I need to use my makefile task
docker compose rm --force --volumes
# the following line, among other things, runs docker compose up
make setup

I'm not happy with this solution, but COMPOSE_PARALLEL_LIMIT=1 make setup would cause performance issues. Doing docker compose --parallel 1 pull --ignore-pull-failures also is not an ideal solution, since it will attempt unnecessary pulls.

TL;DR

there should be some sort of flag for docker compose pull that corresponds to docker compose up --pull ____ so that you can pull only missing images

[glours]

Hey @FFdhorkin
Did you try add a step in your GHA workflow to be logged and avoid to be rate limited? You can try to use our official action for this https://github.com/docker/login-action

[thaJeztah]

@FFdhorkin do you have more details about your issue?

• Perhaps more information about your compose file as well (for example, does it have pull-policies set in the compose file?)

Seems like for some reason, on Github, I get rate limited for pulls, but don't locally.

Do you have more information about how the "rate limit" exposed itself? Do you have the error message (if any) that you got? (want to be sure these are Docker Hub rate limits, or other rate-limits: I've seen cases where GitHub itself was producing 429 statuses).

w.r.t.

the docker compose --parallel 1 pull was resulting in images being pulled even though they were already in the cache (I was still getting occasional rate limit pull failures on cached images)

and

docker compose --parallel 1 up --pull missing --no-start
# the above line breaks docker compose up; I guess it's expecting docker compose start, but I need to use my makefile task

Are these all "distinct" images, or are there multiple services that use the same image? (So without your workarounds it would be (say) pull ubuntu:latest for the first service, and again for the second service)

• Also see the discussion on introduce pull --policy flag to only pull images not present in cache #10981

[FFdhorkin]

Hey @FFdhorkin Did you try add a step in your GHA workflow to be logged and avoid to be rate limited? You can try to use our official action for this https://github.com/docker/login-action

I have not; I'll be switching over to using a private ECR once things are set up, so I didn't explore that route. Incidentally, I am getting rate limits on Amazon Public ECR images, too, which is weird, since I thought the whole point of that was to avoid that.

Perhaps more information about your compose file as well (for example, does it have pull-policies set in the compose file?)

I did not explicitly set any pull_policy; the docs say the default is missing, though, so if it's intended that docker compose pull should respect pull_policy, then that suggests there's a bug either in docker compose pull or in pull_policy

Do you have more information about how the "rate limit" exposed itself? Do you have the error message (if any) that you got?

The specific error message I get is Error response from daemon: toomanyrequests: Rate exceeded which Google hits suggest could be an Amazon Public ECR thing, but I get this error sometimes on images pulled from DockerHub, too.

Are these all "distinct" images, or are there multiple services that use the same image? (So without your workarounds it would be (say) pull ubuntu:latest for the first service, and again for the second service)

They are all distinct: