Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance improvement : Cache pre-pickled documents #12883

Open
ArthurAttout opened this issue Sep 11, 2024 · 1 comment · May be fixed by #12882
Open

Performance improvement : Cache pre-pickled documents #12883

ArthurAttout opened this issue Sep 11, 2024 · 1 comment · May be fixed by #12882
Labels

Comments

@ArthurAttout
Copy link

We have enormous documents in which some individual files .. include:: hundreds of external .rst files.

This sometimes leads to individual .doctrees files exceeding 5MB. Under this scenario, the build procedure is particularly slow (+5 hours).

After profiling the code, repeated calls to pickle.loads() targeting those 5MB files where found. It appears that Sphinx will pickle.loads() the 5MB file at each cross-reference .

While sphinx/environment/__init__.py already caches the raw bytes for each pickled doctree it would be more efficient to cache the result of pickle.loads() instead.

Caching the pre-pickled nodes.document instead of the raw bytes sped up the build process from +5 hours to around 10 minutes (including transformation to PDF with MikTex).

I have not compared the overhead of both caching methods. But I suspect it would be worth the speedup.

I have opened a pull request with my workaround. Feel free to let me know your thoughts !

@ArthurAttout ArthurAttout added the type:proposal a feature suggestion label Sep 11, 2024
ArthurAttout pushed a commit to ArthurAttout/sphinx that referenced this issue Sep 11, 2024
@ArthurAttout
Copy link
Author

See #12882

ArthurAttout pushed a commit to ArthurAttout/sphinx that referenced this issue Sep 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants