Recursive extra dependencies compiles wrong requirements.txt

[maxime1907]

When pyproject.toml references its own local package to recursively include extra dependencies, it outputs a requirements.txt file which references the local package with the absolute path which is a problem as its not portable and should instead just list the dependencies

It was previously possible with setup.py with this configuration:

from setuptools import find_packages, setup

extra_require_tools = ["build"]
extra_require_test = ["pyyaml"]

setup(
    name='package',
    version='1.0.0',
    packages=find_packages(),
    extras_require={
        "dev": extra_require_tools + extra_require_test,
        "tools": extra_require_tools,
        "test": extra_require_test,
    },
)

Environment Versions

1. OS Type: Ubuntu 22.04.4 LTS
2. Python version: Python 3.11.6
3. pip version: pip 23.2.1
4. pip-tools version: pip-compile, version 7.4.1

Steps to replicate

1. Create a pyproject.toml with this content:

[project]
name = "package"
version = "1.0.0"

[build-system]
requires = ["setuptools>=69.1", ]
build-backend = "setuptools.build_meta"

[project.optional-dependencies]
dev = ["package[test,tools]"]
tools = ["build"]
test = ["pyyaml"]

2. Generate the requirements-dev.txt

pip-compile --extra=dev --no-emit-index-url --output-file=requirements-dev.txt pyproject.toml

Expected result

#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile --extra=dev --no-emit-index-url --output-file=requirements-dev.txt pyproject.toml
#
build==1.2.1
    # via package
packaging==24.1
    # via build
pyproject-hooks==1.1.0
    # via build
pyyaml==6.0.1
    # via package

Actual result

#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile --extra=dev --no-emit-index-url --output-file=requirements-dev.txt pyproject.toml
#
build==1.2.1
    # via package
package[test,tools] @ file:///tmp/example
    # via package (pyproject.toml)
packaging==24.1
    # via build
pyproject-hooks==1.1.0
    # via build
pyyaml==6.0.1
    # via package

Linked issues

• pip-compile absolute links when given relative path #204 (comment)
• Recursive extra dependencies don't work for a private package #1685
• Eventually implement support for the PEP 735 "dependency groups" #2062

[chrysle]

This is an edge case; we probably need to add some logic to drop the origin requirements if they are recursive extras, as they are redundant if the dependencies they imply are already part of the requirements file. That should happen directly after the resolution process. It's rather low priority, I presume, we have a lot of other issues to tackle right now - you're welcome to try and draft a fix, of course.

[ckristo]

I encountered this issue too - as workaround for now, I stopped using nested optional-dependencies groups. Using the example above:

[project.optional-dependencies]
dev = ["build", "pyyaml"] # instead of `["package[test,tools]"]`
tools = ["build"]
test = ["pyyaml"]

[webknjaz]

optional-dependencies groups

To eliminate the ambiguity: these are not dependency groups, but extras. Dependency groups are a separate concept that basically standardize what pip's requirements files solve across the ecosystem through PEP 735. They are exposed to project contributors (those who work with Git checkouts, usually). And extras are pieces of metadata that are exposed to all end-users, especially those that get dists from PyPI.

People are so used to abusing extras as dependency groups because there was no standard in the past, and also they didn't realize that extras are public APIs for the end-users essentially.

[WhyNotHugo]

The line:

dev = ["package[test,tools]"]

Declares "a dependency of this package is package with extras test,tools".

That makes the package package a dependency of package. I don't think that this should be considered valid input.

I understand the intent of wanting to specify that an extra depends on other extras, but that's not what this syntax means, and not how pip will interpret it.

[webknjaz]

The line:

dev = ["package[test,tools]"]

Declares "a dependency of this package is package with extras test,tools".

That makes the package package a dependency of package. I don't think that this should be considered valid input.

I understand the intent of wanting to specify that an extra depends on other extras, but that's not what this syntax means, and not how pip will interpret it.

@WhyNotHugo I think, pip started supporting this at some point, but it's tribal knowledge. Hynek said it's since pip 21.2: https://hynek.me/articles/python-recursive-optional-dependencies/. The change log does not call it out explicitly, though: https://pip.pypa.io/en/stable/news/#v21-2. It's not really documented prominently anywhere: pypa/pip#11296. And the maintainers don't even know what enabled the feature: pypa/pip#10393 (comment).
It's supported in pip-tools since v6.13.0: #1685 (comment). And uv has it too: astral-sh/uv#1987 (comment).

[WhyNotHugo]

Thanks for the clarification. I had no idea. I hope you'll forgive my ignorance, given that the devs themselves didn't even know about it 😂