Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow receivers to run in the background. #2

Open
ddanier opened this issue Sep 11, 2022 · 1 comment
Open

Allow receivers to run in the background. #2

ddanier opened this issue Sep 11, 2022 · 1 comment

Comments

@ddanier
Copy link
Member

ddanier commented Sep 11, 2022

Allow receivers to run in the background by just adding the task to the event loop queue, but not awaiting it. Could do something similar to https://fastapi.tiangolo.com/tutorial/background-tasks/.

Note: Background tasks must be configurable by each receiver, as we still need to have receivers that are required to be finished before continuing. This might for example be true for a receiver that sets the modification date of something before it is stored to the database. Also we should add an option to send() so disable background tasks.

@ddanier
Copy link
Member Author

ddanier commented Sep 11, 2022

starlette actually seems to await the background tasks just after sending the response, see:
https://github.com/encode/starlette/blob/master/starlette/background.py
https://github.com/encode/starlette/blob/master/starlette/responses.py#L159

If we want to really just add the task to the task queue we should probably do something like:
https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task

Meaning:

  • We can just use asyncio.create_task(...) to schedule the task
  • The signal instance needs to keep track of all running tasks in a set, so the tasks won't get garbage collected
  • We can use a add_done_callback to clean the task from the set when finished

We however need to add docs so people won't add CPU heavy tasks as background jobs, this really is just for receivers that are ok to be handles some microseconds later. If something really needs the CPU (or worse calls some sync IO functions) this has to be done in celery/arq.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant