Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

potential slowdown when using run_iter over thousands of paths #56

Open
amyreese opened this issue Jun 24, 2022 · 0 comments
Open

potential slowdown when using run_iter over thousands of paths #56

amyreese opened this issue Jun 24, 2022 · 0 comments
Assignees

Comments

@amyreese
Copy link
Member

amyreese commented Jun 24, 2022

Walking paths with thousands of files, and then calling run_iter on those paths, seems to get slower and slower when pushing past ~10k paths, and eventually grinds to a near halt around 14/15k. Batching into groups of 1k paths eliminates the slowdown, so doesn't seem related to filesystem issues. Perhaps something in concurrent.futures.as_completed when dealing with that many futures.

@amyreese amyreese self-assigned this Jun 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant