Improve throughput for immediate tasks (1/3) #5841
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pedro-psb
force-pushed
the
feat/run-immediate-tasks-on-parent
branch
from
7fb9d32
to
d18b071
Compare
pedro-psb
changed the title
gerrod3
reviewed
Sep 27, 2024
pulpcore/tasking/worker.py
Outdated
Comment on lines
349
to
352
| .first() | ||
| ) | ||
| with contextlib.suppress(AdvisoryLockError), task: | ||
| task.refresh_from_db() |
There was a problem hiding this comment.
.first() can return None. Will this not error trying to call a method on None?
There was a problem hiding this comment.
Yes, nice catch.
But anyway, this was a sketch of the approach I had in mind, which won't be used.
I force pushed with the ideas I discussed with @mdellweg yesterday.
pedro-psb
force-pushed
the
feat/run-immediate-tasks-on-parent
branch
2 times, most recently
from
9ab7302
to
589053b
Compare
|
(there is a refator and a impl commit) |
pedro-psb
force-pushed
the
feat/run-immediate-tasks-on-parent
branch
from
589053b
to
b417979
Compare
|
Curious how the tests passed on the lowerbound but got stuck in the others scenarios 🤔 |
pedro-psb
force-pushed
the
feat/run-immediate-tasks-on-parent
branch
3 times, most recently
from
ce087b4
to
ff396e0
Compare
pedro-psb
force-pushed
the
feat/run-immediate-tasks-on-parent
branch
from
ff396e0
to
40fd2a0
Compare
pedro-psb
force-pushed
the
feat/run-immediate-tasks-on-parent
branch
2 times, most recently
from
d710146
to
697f97a
Compare
* immediate prioritization This defines tasks of the type "immediate" by adding a Task.immediate(bool) field to the Task model. These type of task are ordered first in the worker's query to pick up a new available task. * misc This also adds a "deferred" field to the Task model, so we can later harden the constraint that 'deferred == False' and immediate == True' shouldn't get past the API level. This is possible today in some edge case failure scenarios on dispatch. Closes pulp#5767
pedro-psb
force-pushed
the
feat/run-immediate-tasks-on-parent
branch
from
697f97a
to
ef6ddb0
Compare
|
Ready to review. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In busy tasking environments, "immediate" running tasks (which are only dispatched to workers due to resource locking requirements) can wait for very long behind long-running tasks, such as syncs and publishes.
This tries to solve this implementing the solution proposed in #5767, as discussed in the Pulp team.