Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DUB aggregates and fetch all dependencies regardless of selected configuration #1706

Open
schveiguy opened this issue May 11, 2019 · 9 comments
Labels

Comments

@schveiguy
Copy link
Member

schveiguy commented May 11, 2019

Currently when dub is run on a project that has multiple configurations, some of which select optional dependencies, dub will determine packages that should be downloaded in case those dependencies are selected, and download them proactively. However, this makes little sense if the configuration is not intended to be used in common cases. For example, vibe.d's tls:botan configuration will select botan to download, even if you never intend to compile the botan version.

In addition, different configurations may require different versions of dependencies, creating a conflict of which version of a package should be downloaded for those versions. This can even happen within subconfigurations of packages that you will NEVER build. For example, the test configurations for 2 dependency projects may depend on DIFFERENT versions of a subdependency, even if I'm not going to build the specific subconfiguration.

Example:

...
         "dependencies": {
		"mysql-native": "~>2.3.0"
	},
        "configurations" : [
            {
                "name" : "unittest",
                "dependencies" : {
                    "unit-threaded": "==0.7.37"
                }
            }
        ]
...

To solve this problem, I propose that we:

  1. Do not download packages until needed for build (this will save from unnecessarily downloading never-used packages)
  2. Change dub.selections.json into a key/value array, where the specific dub.selections.json configuration depends on which configurations were selected for building. In practice, you mainly only build one or two configurations, so this will still make the dub.selections.json file relatively static.

For an example of the proposed format, it might be something like:

{
    "fileVersion": 2,
    "selections" : [
        {
            "configuration" : {
                "myproj" : "unittest",
                "mysql-native" : "default",
                "unit-threaded" : "default"
            },
            "versions" : {
                "mysql-native" : "2.3.0",
                "unit-threaded" : "0.7.40"
            }
        },
        ...
    ]
}
@schveiguy schveiguy changed the title Proposal for avoiding downloading optional dependencies Proposal for avoiding downloading optional dependencies and avoiding subproject configuration conflicts May 11, 2019
@schveiguy
Copy link
Member Author

@wilzbach @ghost91-

@andre2007
Copy link
Contributor

In the FAQ the first 2 topics are related to your proposal and explains the difficulties
https://github.com/dlang/dub/wiki/FAQ

Are you aware of this FAQ?

@schveiguy
Copy link
Member Author

Thank you for the FAQ, I was not aware of it.

The topics do not explain why the current model is better than one which would select the packages needed when needed. As an end user, I don't care AT ALL how dub actually figures out what packages to select, and how the internal workings happen. I just want it to build things that make sense, and only download packages which it needs.

The problems identified in the first part are not made up -- if I want to build for example, using unit threaded of a particular version for my project, and another dependency has a NON-SELECTED configuration where it is using a different version of unit-threaded, the build will succeed. BUT dub does not allow this. It's a limitation that is not easily fixed by me, as I can't affect what the dependency puts in their dub.json file.

Note also that downloading packages that are not going to be used can and should be delayed until actually needed. Either one of these problems could potentially be tackled as separate issues, but trying to fix them both at once can result in a much more cohesive solution.

Other language build systems have solved these problems. If dub cannot do this, it simply will stop being used. See threads like this for instance: https://forum.dlang.org/post/[email protected]

@andre2007
Copy link
Contributor

I also want to see this issue fixed.
There was a pull request in the past to solve this issue
#1148

@schveiguy
Copy link
Member Author

Thanks for the link! I will take a look.

@Kozzi11
Copy link

Kozzi11 commented Sep 18, 2019

Please this ASAP or I will really need to stop using dub, it is impossible to use right now. Everytime I need tu upgrade to new version of dub packages I will end up with:
Could not resolve configuration for package tachograf_bridge

@Geod24
Copy link
Member

Geod24 commented Sep 19, 2019

Please this ASAP or I will really need to stop using dub, it is impossible to use right now.

A workaround we are using is to provide a path-based dependency with a dummy dub.json.
E.g. for botan:

{
    "name": "botan",
    "targetType": "none"
}

dub.selections.json:

{
	"fileVersion": 1,
	"versions": {
                // ...
		"botan": { "path": "submodules/botan/" },
                // ...
	}
}

@Geod24 Geod24 added the bug label Jan 2, 2020
CyberShadow added a commit to CyberShadow/ae that referenced this issue Jul 25, 2020
This allows using ae:openssl on systems where /usr/lib/libssl.so is
1.1 or newer.

To do so, run e.g.: dub build --override-config=ae:openssl/lib-1.0

At some point, this configuration will become the default.

This commit would also have added configurations to select the openssl
D library version, if not for a failure in Dub's design which
prohibits doing so: dlang/dub#1706
@Geod24 Geod24 changed the title Proposal for avoiding downloading optional dependencies and avoiding subproject configuration conflicts DUB aggregates and fetch all dependencies regardless of selected configuration Jul 27, 2022
@Geod24 Geod24 pinned this issue Jul 27, 2022
@Geod24
Copy link
Member

Geod24 commented Jul 27, 2022

Renamed and pinned the issue as it is the most common one.

@ichordev
Copy link

ichordev commented Dec 6, 2023

Does anyone have any plans to resolve this issue or put in a DEP?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants