You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Walking paths with thousands of files, and then calling run_iter on those paths, seems to get slower and slower when pushing past ~10k paths, and eventually grinds to a near halt around 14/15k. Batching into groups of 1k paths eliminates the slowdown, so doesn't seem related to filesystem issues. Perhaps something in concurrent.futures.as_completed when dealing with that many futures.
The text was updated successfully, but these errors were encountered:
Walking paths with thousands of files, and then calling
run_iter
on those paths, seems to get slower and slower when pushing past ~10k paths, and eventually grinds to a near halt around 14/15k. Batching into groups of 1k paths eliminates the slowdown, so doesn't seem related to filesystem issues. Perhaps something in concurrent.futures.as_completed when dealing with that many futures.The text was updated successfully, but these errors were encountered: