update fmt and spdlog across RAPIDS
#56
Comments
|
@bdice drew my attention to these global pins in fmt:
- '10'
spdlog:
- '1.12'I think the I think we should NOT update any RAPIDS packages to depending on the References:
|
|
Yes, we should try to keep RAPIDS in sync with conda-forge. For these, I would move them forward once the migrator passes some critical threshold (for spdlog, I might base it on when packages like For the purpose of fixing our conda clobbering issues, we may want to work on some kind of vendoring solution (centralized via Also, if we vendor these and remove the dependencies from the RAPIDS conda packages, it would also make it possible to decouple from the global conda-forge pinnings and manage version updates solely through rapids-cmake (the status quo requires changes in both rapids-cmake and conda recipes). |
Modifies pinning tests from #530: * to only test projects that were downloaded by CPM (e.g. ignoring the `fmt` that might already exist in the build environment) * to echo out `pinned_versions.json` and `versions.json` in logs from failed tests, to make debugging faster ## Notes for Reviewers #592 proposes some testing changes that aren't specific to the goals of that PR. Since that PR might be stuck for a bit (rapidsai/build-planning#56 (comment)), this proposes pulling those out into a separate PR: * so that other changes in this project benefit from them * to shrink the diff of #592 and therefore the risk of merge conflicts ### How I tested this Pushed a commit with the new test error message content changes but keeping `fmt` in the failing tests, to confirm that the expected tests failed. <details><summary>got the expected outputs (click me)</summary> ```text The following tests FAILED: 698 - cpm_generate_pins-nested-makefile (Failed) 700 - cpm_generate_pins-nested-ninja (Failed) 702 - cpm_generate_pins-nested-ninja_multi-config (Failed) 722 - cpm_generate_pins-simple-makefile (Failed) 724 - cpm_generate_pins-simple-ninja (Failed) 726 - cpm_generate_pins-simple-ninja_multi-config (Failed) ``` And they failed in the expected way more informative logs! ```text CMake Error at CMakeLists.txt:51 (message): pinned fmt tag (10.2.1) should differ compared to baseline 10.2.1 pinned_versions.json: { "always_download" : true, "git_shallow" : false, "git_tag" : "${version}", "git_url" : "https://github.com/fmtlib/fmt.git", "version" : "10.2.1" } versions.json: { "git_tag" : "${version}", "git_url" : "https://github.com/fmtlib/fmt.git", "version" : "10.2.1" } ``` ([build link](https://github.com/rapidsai/rapids-cmake/actions/runs/8837613213/job/24267079292?pr=592)) </details> # Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Robert Maynard (https://github.com/robertmaynard) - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #599
|
FYI -- There's active migrations for spdlog 1.14 and fmt 11 going on in conda-forge right now:
I believe RAPIDS is now multiple versions behind on spdlog which will make solving environments painful. |
|
Thanks for the heads-up @kkraus14! We'll discuss and try to land an update for fmt/spdlog in the next RAPIDS release. |
|
Adding links to the migration statuses: |
|
Alright, I think this set of changes is ready for review! See "How I tested this" in rapidsai/rapids-cmake#689 ... I'm able to build all of RAPIDS with these new versions of
Putting a plan for how to roll this out in writing, will start an internal chat thread to coordinate. Phase 1: reviews
Phase 2: run tests
Phase 3: merge changes
Note These will all require admin merges, because of the use of Phase 4: clean up
|
## Description Replaces #15603 Contributes to: * rapidsai/build-planning#54 * rapidsai/build-planning#56 * rapidsai/rapids-cmake#387 Now that most of `conda-forge` has been updated to `fmt >=11.0.1,<12` and `spdlog>=1.14.1,<1.15` (rapidsai/build-planning#56 (comment)), we're attempting to upgrade RAPIDS to similar versions of those libraries. This improves the likelihood that RAPIDS will be installable alongside newer versions of its dependencies and complementary packages on conda-forge. ## Notes for Reviewers This PR is testing changes made in rapidsai/rapids-cmake#689. It shouldn't be merged until those `rapids-cmake` changes are merged and any testing-specific details have been removed.
This PR removes the NumPy<2 pin which is expected to work for RAPIDS projects once CuPy 13.3.0 is released (CuPy 13.2.0 had some issues preventing the use with NumPy 2). Also contributes to rapidsai/build-planning#56 by removing the pin on `proj` (which allows this project to work with newer `fmt` / `spdlog`, in sync with conda-forge). Authors: - Sebastian Berg (https://github.com/seberg) - Bradley Dice (https://github.com/bdice) - James Lamb (https://github.com/jameslamb) - https://github.com/jakirkham Approvers: - Bradley Dice (https://github.com/bdice) URL: #1441
… libcuml conda host dependencies (#6071) Contributes to rapidsai/build-planning#56 * updates `fmt` and `spdlog` to newer versions, to match the rest of RAPIDS * adds `fmt` and `spdlog` to `host:` dependencies for `libcuml` conda packages (see #6071 (comment)) Now that most of `conda-forge` has been updated to `fmt >=11.0.1,<12` and `spdlog>=1.14.1,<1.15` (rapidsai/build-planning#56 (comment)), we're attempting to upgrade RAPIDS to similar versions of those libraries. This improves the likelihood that RAPIDS will be installable alongside newer versions of its dependencies and complementary packages on conda-forge. ## Notes for Reviewers This PR is testing changes made in rapidsai/rapids-cmake#689. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #6071
Contributes to rapidsai/build-planning#56 With rapidsai/cuspatial#1441, it should be possible to revert some of the workarounds introduced in #719. ## Notes for Reviewers ### How to test this if this is working, we should see the following in the conda solves: * `fmt >=11.0.2` * `spdlog >=1.14.1` We won't see `numpy >=2` yet, because `cugraph` doesn't support it yet (rapidsai/cugraph#4615). Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #722
|
This is complete! RAPIDS is now building successfully against Put up one follow-up item here: rapidsai/rapids-cmake#695 |
Description
RAPIDS projects that need
fmtandspdloguserapids-cmaketo find it, via functions likerapids_cpm_fmt()(docs link).rapids-cmakeis currently carrying patches to both of those libraries, and as a result always downloads sources of them. (see rapidsai/rapids-cmake#525 for details).Those patches have since been upstreamed and made it into official releases of those projects (in their main source control and on
conda-forge). This proposes upgrading to those new versions across RAPIDS and dropping the patches inrapids-cmake.Benefits of this work
condabuilds across many RAPIDS projectsrmmto ensure future such conflicts raise a loud errorrapids-cmakeand therefore the need to reason about them in future upgradesAcceptance Criteria
rapids-cmakeis pulling infmt >= 11.0.1rapids-cmakeis pulling inspdlog >= 1.14fmtandspdloginrmm's conda build CI logsApproach
On a branch, modify
rapids-cmake/cpm/versions.json(code link) such that:fmtandspdlogare bumped to these new onesFollow "Overriding RAPIDS CMake" (rapids-cmake docs) to point builds of at least
rmm,cudf, andraftat your branch ofrapids-cmakewith these changes. Also modify those projects'dependencies.yamland/or conda recipemeta.yamlto change their version constraints onfmtandspdlog.NOTE: this is a great task for
rapids-reviser(https://github.com/rapidsai/rapids-reviser/tree/main/examples/fmt-10-spdlog-1.12).Confirm that CI succeeds and that the correct versions of
fmtandspdlogare being pulled in.For
rmm, re-use this PR for that purpose: rapidsai/rmm#1508. If this change also results in there being 0 remaining conda clobbering warnings inrmm, leave in theconda config --set path_conflict preventin that PR's build scripts, so we'll be alerted by a CI failure if something around this changes in the future.Notes
What makes you confident that these patches can be removed?
From #54 (comment)
fmt's most recent release was11.0.2, so that patch we're carrying around for the10.1.1version could be dropped.rapids-cmake: (fmt/fix_10_1_1_version.diff)v10.2.1release on conda-forge removing a similar patch: fmt v10.2.1 conda-forge/fmt-feedstock#48The
spdlogpatch we've been carrying around has been upstreamedrapids-cmake: (spdlog/nvcc_constexpr_fix.diff)v1.13.0of the conda-forge recipe is up: spdlog v1.13.0 conda-forge/spdlog-feedstock#61What are "clobber warnings" and how does this help with them?
As described in #54 and rapidsai/rmm#1508, conda builds of
rmmand other RAPIDS packages downstream of it are emitting dozens of warnings like these:These are saying that the
librmmconda package contains files whose paths exactly conflict with those from thefmtandspdlogpackages fromconda-forge. When such packages are installed together, one will overwrite the other, which could lead to runtime issues.Removing the patches in
rapids-cmakemakes it more likely that builds will find the files they need from thefmtandspdlogconda-forge packages already present in the build environment, and therefore not vendor them, and therefore not ship a copy that causes conflicts.Code changes (these PRs should be merged)
Testing dependencies (do not merge)
Post-migration
The text was updated successfully, but these errors were encountered: