Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't load entire distribution into memory #7

Open
dstufft opened this issue Jun 6, 2022 · 1 comment
Open

Don't load entire distribution into memory #7

dstufft opened this issue Jun 6, 2022 · 1 comment
Labels
enhancement New feature or request

Comments

@dstufft
Copy link
Member

dstufft commented Jun 6, 2022

Currently this fetches the distribution from PyPI into a BytesIO object, after doing a requests.get() call (not streaming).

That means that while we're inside of _get_dist, we'll currently be using 2x the file size of the distribution worth of extra RAM, and outside of it we'll be using 1x the file size of extra RAM.

This should probably buffer to a temporary file and use streaming requests so that a large distribution doesn't kill us on memory.

This might just be #5 but I wanted to call it out explicitly since this applies even if we're storing the files somewhere.

@di
Copy link
Member

di commented Jun 6, 2022

I think we could probably improve this before we get to fixing #5.

@di di added the enhancement New feature or request label Jun 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants