pageserver: reorder upload queue when possible #10218
Draft
+224
−44
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
erikgrinaker
force-pushed
the
erik/upload-reorder
branch
from
11ee986 to
b457d21
Compare
jcsp
reviewed
Dec 20, 2024
| upload_queue.num_inprogress_deletions == upload_queue.inprogress_tasks.len() | ||
| } | ||
| /// TODO: consider moving this and other associated logic into UploadOp and UploadQueue. | ||
| fn can_bypass(a: &UploadOp, b: &UploadOp) -> bool { |
There was a problem hiding this comment.
Let's unit test this against the most important invariants:
- A layer upload must happen before an index that references it
- A layer deletion must happen after an index that de-references it
- If a layer name that is re-used, the second upload must come after an index that de-references the earlier layer of the same name
- ...whichever others we can think of
7095 tests run: 6793 passed, 5 failed, 297 skipped (full report)Failures on Postgres 17
Flaky tests (5)Postgres 17
Postgres 14
Test coverage report is not availableThe comment gets automatically updated with the latest test results
7c3ae0e at 2024-12-23T13:08:56.921Z :recycle: |
erikgrinaker
force-pushed
the
erik/upload-reorder
branch
from
b457d21 to
c2215f1
Compare
erikgrinaker
force-pushed
the
erik/upload-reorder
branch
from
ebb4f72 to
7c3ae0e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
The upload queue currently sees significant head-of-line blocking. For example, index uploads act as upload barriers, and for every layer flush we schedule a layer and index upload, which effectively serializes layer uploads.
Requires #10227.
Requires #10228.
Resolves #10096.
Summary of changes
Allow upload queue operations to bypass the queue if they don't conflict with preceding operations.