Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Faster test session reruns & microoptimize CI #13126

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Faster test session reruns & microoptimize CI #13126

wants to merge 2 commits into from
+26 −26

Conversation

ichard26
Copy link
Member

@ichard26 ichard26 commented Dec 24, 2024 •
edited
Loading

See commit messages for more details.

@ichard26
test: Skip build/install steps with nox's --no-install flag
c340d7e 
This saves time when you want to rerun the test suite with different
pytest arguments but you haven't made any code changes in-between.
@ichard26 ichard26 added C: tests Testing and related things skip news Does not need a NEWS file entry (eg: trivial changes) labels Dec 24, 2024
@ichard26 ichard26 force-pushed the microoptimize-ci branch 3 times, most recently from 34c73ee to dd054a9 Compare December 24, 2024 23:09
@ichard26
CI: micro-optimize test collection & pass nox's --no-install
5ce1145 
Pytest can be pretty slow to collect pip's entire test suite and
prepare for test execution. I've observed a ~15s delay from invoking
pytest to the first test running in CI in the worst case. This can be
improved by reducing how many files pytest has to process while
collecting tests. In short, passing tests/unit is faster than -m unit.

In addition, use nox's --no-install flag to skip redundant build and
install steps on the 2nd nox session invocation (for the integration
tests), which was made possible by the previous commit.
@ichard26 ichard26 changed the title CI optimizations Faster test session reruns & microoptimize CI Dec 25, 2024
@ichard26
Copy link
Member Author

Replacing pytest -m unit -n auto with pytest tests/unit -n auto does reduce the delay before the first test is executed (try it locally if you don't believe me!). It's pretty small to begin with, at worse 15-20s in CI, but shrinking the search space can (at most) cut that in half. Before logs and after logs.

For nox's --no-install, it saves another 10-20 seconds by skipping the slow build and install set-up steps. They're redundant as the pip code under test has already been installed in the nox test environment earlier during the unit tests.

Using the example runs linked above, 10 seconds are saved between nox being invoked to the test suite starting.

image
image

For the Windows 3.8 (1) job, which is one of our slowest jobs, 20 seconds are saved. Before logs, after logs.

image
image

In total, these optimizations probably improve CI times by 1-3%... which is rather small, but IMO it's still worth it given the low complexity (and the nox --no-install change is actually more beneficial for local development anyway).

@ichard26 ichard26 marked this pull request as ready for review December 25, 2024 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
C: tests Testing and related things skip news Does not need a NEWS file entry (eg: trivial changes)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@ichard26