Stability-focused test infra strategy

[webknjaz]

What's the problem this feature will solve?

The CI has been prone to external influence for a long time. This is due to unpinned deps. Pip specifically. Historically, we've been running the tests against a pip version that was pinned to a minor release, and the latest release, plus the main branch of the pip repo.

This turned out to be unsustainable since it causes PR runs to become unpredictable over time. Frankly, this is at odds with the reproducibility promise that pip-tools itself seeks to implement for its users.

It also contributes to other CI problems accumulating over time and blocks progress in many other aspects.

Describe the solution you'd like

I want to make sure that the CI runs in PRs and pushes use pinned pip, short term. With the future possibility to have more lock files, long-term.

With that in mind, I want to let the nightly runs keep hitting newer pip versions.

Alternative Solutions

…suffering the status quo?

Additional context

N/A

[webknjaz]

One of the problems found while looking into fixing the CI: #2141.

N.B. I'm going to prepare a series of separate PRs and then a PR merging-in all those branches into one. The combined one is what will need to be approved and merged.

[webknjaz]

Found out that we're hitting a problem with the conditionals relying on ${{ github.job_workflow_sha }} which would have to be rewritten as a hack: actions/runner#2417 (comment).

[webknjaz]

Another PR: #2142.

[webknjaz]

I've found a weird behavior that I thought might be a bug in contextlib.contextmanager: https://github.com/jazzband/pip-tools/pull/2106/files#r1870250007. But I can't replicate it outside the pip-compile invocation (by importing and making the underlying calls separately). I now suspect that Click might be influencing that.
Here's how the problem manifests itself:

$ pip-compile --build-deps-for wheel --find-links tests/test_data/minimal_wheels /tmp/pytest-of-${USER}/pytest-current/test_compile_build_targets_setcurrent/pyproject.toml --output-file - -vvvvvv
Using pip-tools configuration defaults found in 'pyproject.toml'.
Traceback (most recent call last):
  File "/home/${USER}/src/github/jazzband/pip-tools/some-venv-py313/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
             ~~~^^
  File "/home/${USER}/src/github/jazzband/pip-tools/some-venv-py313/lib/python3.13/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/${USER}/src/github/jazzband/pip-tools/some-venv-py313/lib/python3.13/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/${USER}/src/github/jazzband/pip-tools/some-venv-py313/lib/python3.13/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/${USER}/src/github/jazzband/pip-tools/some-venv-py313/lib/python3.13/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/${USER}/src/github/jazzband/pip-tools/some-venv-py313/lib/python3.13/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/${USER}/src/github/jazzband/pip-tools/piptools/scripts/compile.py", line 370, in cli
    metadata = build_project_metadata(
        src_file=Path(src_file),
    ...<4 lines>...
        quiet=log.verbosity <= 0,
    )
  File "/home/${USER}/src/github/jazzband/pip-tools/piptools/build.py", line 164, in build_project_metadata
    with _create_project_builder(
    ...<22 lines>...
        )
  File "/home/${USER}/.pyenv/versions/3.13.0rc1/lib/python3.13/contextlib.py", line 143, in __enter__
    raise RuntimeError("generator didn't yield") from None
RuntimeError: generator didn't yield

I've made a branch to preserve the breaking state: https://github.com/webknjaz/pip-tools/tree/maintenance/contextlib.contextmanager-repro. Perhaps, I'll get to inspect it deeper one day.

[webknjaz]

UPD: the CI has been made green. The issue remains open to track long-term improvements.