-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docker image push retry mechanism #5275
Comments
Can you post the output of (Also note that the retry happens on the daemon side, so probably better to report in https://github.com/moby/moby/issues) That said, I'd have to check on the retry mechanism; I am aware of a retry for pulling, but not sure about pushing (but it's possible that I'd not configurable); from the
|
@thaJeztah No, the retry mechanism was available a few months back. Below is the example.
|
There were no changes to the retry mechanism. However, the retry is only performed when it makes sense to. Can you provide the error reported after a push that wasn't retried? |
@vvoland The docker image push command would retry 5 times before erroring out, saying the repository doesn't exist. Now, it simply errors out without retrying. We have custom integration in place in ECR registry to create the repository if it doesn't exist. In the past, it performed retries before exiting with any type of error; now it exits immediately. Current behavior:
past behavior:
In both of the above scenarios, the repository in the ECR did not exist. The custom integration created a repository, and when Docker retried for the second or third time, the repository would be created, allowing Docker to push the image without any issues. |
I checked the code, and there were no changes to the retry mechanism. I see that, as mentioned before by @.vvoland, the retry mechanism only works if the user is not authorized. Could you please specify which version works this way? Also, please make sure user successfully authorized. 🙏 |
Yes, I'm authenticated to the ECR registry before pushing the images. |
do you think upstream ECR api might have an effect on the retry mechanism, by any chance? |
@vvoland could this be related to the bugfix in the registry client?
That one was related to moby/moby#45415, which revealed that a fatal (non-retriable) error fell through. |
Hmmm.. probably not; that patch already went in registry v2.8.2 (https://github.com/distribution/distribution/blob/v2.8.2/registry/client/errors.go), which was included in docker 20.10.26 already, so ... unlikely moby/moby#45980 |
Hmmm, could be. @madhu2852 what was the Docker version in case where it worked for you? |
@vvoland I think it was 24.x, but the strange thing is that even if I download old versions dating back to 2022/2023, it’s not retrying anymore. Could it be that the responses it was getting from the ECR upstream API caused the retries now something changed and its not getting the same response back causing the client not to retry? |
Definitely possible something changed in their registry, perhaps it didn't return correct errors before |
I was bothered by the same problem. I found now that when pushing to ECR but the repository does not exist, it will return a 404 error code to the POST request sent from the client. |
I think a 404 is allowed for these endpoints, looking at https://github.com/opencontainers/distribution-spec/blob/v1.1.0/spec.md#endpoints Depending on the registry, a non-existing repository may be created automatically, or require it to be created first. Not sure if the error-code to return is defined as part of the spec though (if either 404 or 403 (e.g.) are acceptable). That said, I don't think a retry would help here; if the registry does not allow a missing repository to be created as part of a push, then retrying won't help. |
Description
A couple of months ago, I observed that Docker had a built-in retry mechanism for docker push operations, which was beneficial in handling intermittent network issues or transient errors during image pushes. However, recently I noticed that this functionality seems to be either altered or unavailable.
Could you please provide clarification on the following points:
Has there been any recent update or change to the retry mechanism in Docker image pushing operations?
If the retry mechanism has been modified or removed, what is the reason behind this change?
Are there any alternative recommendations or best practices for managing retry behavior during Docker image pushes?
I appreciate any insights or information you can provide on this matter. If possible, I would also like to understand if there are plans to reintroduce or enhance retry capabilities in future Docker releases.
Reproduce
docker push : a
Expected behavior
docker should retry 5 times before exiting with an error.
docker version
ocker version Client: Version: 27.0.3 API version: 1.46 Go version: go1.21.11 Git commit: 7d4bcd8 Built: Fri Jun 28 23:59:41 2024 OS/Arch: darwin/amd64 Context: desktop-linux Server: Docker Desktop 4.32.0 (157355) Engine: Version: 27.0.3 API version: 1.46 (minimum version 1.24) Go version: go1.21.11 Git commit: 662f78c Built: Sat Jun 29 00:02:50 2024 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.7.18 GitCommit: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e runc: Version: 1.7.18 GitCommit: v1.1.13-0-g58aa920 docker-init: Version: 0.19.0 GitCommit: de40ad0
docker info
Additional Info
The text was updated successfully, but these errors were encountered: