fix(reaper): refactor to allow retries and fix races #2728
+858
−705
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for testcontainers-go ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
Depends on #2738 |
stevenh
force-pushed
the
fix/reaper-retries-race
branch
from
0dc52af
to
0b62224
Compare
mdelapenya
force-pushed
the
fix/reaper-retries-race
branch
from
d3c9c9a
to
0589878
Compare
mdelapenya
reviewed
Sep 6, 2024
|
@mdelapenya reminder this is dependent on #2728 which is why it has missing components still. I'll rebase once #2738 is merged. |
stevenh
force-pushed
the
fix/reaper-retries-race
branch
from
e7b948d
to
5de803c
Compare
|
Rebased and picking up again, hopefully wont require many fixes. Will convert from draft when ready. |
stevenh
force-pushed
the
fix/reaper-retries-race
branch
3 times, most recently
from
bf961f0
to
7649e0c
Compare
mdelapenya
reviewed
Sep 13, 2024
There was a problem hiding this comment.
I only added a few super minor comments, other than that, my only concern is about the implications of these changes with the reuse mode we are proposing in #2768
I'm fine merging this, although I'd like to discuss with you about reuse more in depth, as it could apply to the reaper code as well.
stevenh
force-pushed
the
fix/reaper-retries-race
branch
from
7649e0c
to
4182991
Compare
|
@stevenh is there anything I can do to move forward with this PR? |
stevenh
force-pushed
the
fix/reaper-retries-race
branch
from
4182991
to
7bc2a87
Compare
|
Blocked on #2786 |
stevenh
force-pushed
the
fix/reaper-retries-race
branch
from
7bc2a87
to
12c4483
Compare
mdelapenya
reviewed
Sep 20, 2024
mdelapenya
reviewed
Sep 20, 2024
mdelapenya
reviewed
Sep 20, 2024
mdelapenya
reviewed
Sep 20, 2024
stevenh
force-pushed
the
fix/reaper-retries-race
branch
2 times, most recently
from
a304b09
to
1f4141c
Compare
stevenh
force-pushed
the
fix/reaper-retries-race
branch
from
1f4141c
to
0e34b71
Compare
|
There are failing tests for the reaper. Could you take a look? |
Thanks for the poke I was expecting some issues here, but hadn't got round to checking yet. |
Refactor how the reaper is created to allow for proper retries of temporary errors such as container not found issues during startup or shutdown races. This eliminates the use of sync.Once which wasn't solving the problem at hand and replaces it with a singleton spawner with locked access. Wrap reaper errors so we can determine the cause of failures more easily. Fix race condition in port wait when loading from container by always waiting for the port first. Remove unnecessary use of buffering and invalid retry logic in reaper connection handling which could never recover correctly from a partial read. Move the reaper creation just before connections are established in compose to ensure its still running when the Connect calls are made. Previously the reaper was requested in NewDockerComposeWith which means it could have already shutdown before connections are made during the later sections of Up if the startup took over 1 minute. This was causing consistent failures for: TestDockerComposeAPIWithWaitLogStrategy Ensure that resource labels are correct so that resources aren't reaped when the reaper is disabled by excluding session id when reaper is disabled. Error when creating a reaper when the config says it's disabled so that we avoid hard to debug issues because a reaper is running when it shouldn't be. Set org.testcontainers.reap label for containers which should be reaped by the reaper, to prevent containers which disable the reaper from being incorrectly reaped.
stevenh
force-pushed
the
fix/reaper-retries-race
branch
from
0e34b71
to
4218dcc
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refactor how the reaper is created to allow for proper retries of temporary errors such as container not found issues during startup or shutdown races.
This eliminates the use of sync.Once which wasn't solving the problem at hand and replaces it with a singleton spawner with locked access.
Wrap reaper errors so we can determine the cause of failures more easily.
Fix race condition in port wait when loading from container by always waiting for the port first.
Remove unnecessary use of buffering and invalid retry logic in reaper connection handling which could never recover correctly from a partial read.
Move the reaper creation just before connections are established in compose to ensure its still running when the Connect calls are made.
Previously the reaper was requested in NewDockerComposeWith which means it could have already shutdown before connections are made during the later sections of Up if the startup took over 1 minute.
This was causing consistent failures for:
TestDockerComposeAPIWithWaitLogStrategy
Ensure that resource labels are correct so that resources aren't reaped when the reaper is disabled by excluding session id when reaper is disabled.
Error when creating a reaper when the config says it's disabled so that we avoid hard to debug issues because a reaper is running when it shouldn't be.