Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Track conda-forge migrations with automated tooling #100

Open
bdice opened this issue Sep 10, 2024 · 1 comment
Open

Track conda-forge migrations with automated tooling #100

bdice opened this issue Sep 10, 2024 · 1 comment

Comments

@bdice
Copy link
Contributor

bdice commented Sep 10, 2024

Problem Statement

Today, RAPIDS is not built with conda-forge infrastructure, but aims to be compatible with the conda-forge ecosystem.

There is an extensive set of conda-forge automated bots that perform migration work (for example, most of the rebuilds of libarrow are from migrators). Today, RAPIDS does not benefit from conda-forge's automation for version migrations, yet RAPIDS must track conda-forge migrations in order to be compatible with the latest conda-forge packages.

Tracking conda-forge migrations in RAPIDS has historically been a pain point. We've had conflicts with spdlog and fmt numerous times, and updating those pinnings has been difficult to do smoothly (typically because of patches needed in the upstream repos). We used to have similar issues with Arrow, though libcudf has recently dropped the build requirement on libarrow (at least in part to ease this pain).

See also:

We need to improve our integration with the conda-forge ecosystem by following conda-forge migrations in RAPIDS builds.

Possible solutions

We could implement similar automation to what conda-forge uses, with various degrees of complexity. We might be happy with plain notifications when a migration is needed on our end. The following are not mutually exclusive. I am open to any possible improvements that fix problems (ideally with some automation) and/or give us more information about conflicts or work to be done.

Solution 1: conda_build_config.yaml alignment

Use https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml as a "base version set" for RAPIDS conda builds. This would make RAPIDS match the state of conda-forge's completed migrations. This would keep RAPIDS up to date, but could be painful if the rollover breaks CI. We could issue our own mini-migrations that keep RAPIDS aligned with rapids-reviser, perhaps.

Solution 2: notifications of migrations

Set up a system that tracks a few core dependencies that we flag (e.g. spdlog, fmt, aws-sdk-cpp, libarrow), watches for migrations that touch those, and files task issues on a central RAPIDS repository.

Solution 3: notifications of conflicts with "latest"

Set up a system that notifies us when RAPIDS nightlies cannot be used with the latest versions of conda-forge dependencies. This could look something like rapidsai/integration#690, with a few tweaks.

@bdice bdice mentioned this issue Sep 10, 2024
3 tasks
@vyasr
Copy link
Contributor

vyasr commented Sep 20, 2024

Perhaps @jakirkham or @msarahan could comment on the feasibility of copying over some of the cf bots. I'm guessing it would be more work than it's wroth, but I don't know enough offhand to be 100% sure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants