Repairing wheel (delocate-wheel) failed to find arm64 binary (macos-13), but the file exists

[patrikhuber]

Description

Hi,

I'm building wheels of a C++/CMake project (https://github.com/patrikhuber/eos) with cibuildwheel. The Windows and Linux cibuildwheel builds work fine, and macos cp37-macosx_x86_64 and cp38-macosx_x86_64 build fine. But the first macos arm64 wheel, cp38-macosx_arm64, fails at the "Reparing wheel" step:

Building wheel...
  
  + python -m pip wheel /Users/runner/work/eos/eos --wheel-dir=/private/var/folders/x4/dc0wnrqs60v8cfdj3vx06s1h0000gn/T/cibw-run-67h7fric/cp38-macosx_arm64/built_wheel --no-deps
  Processing /Users/runner/work/eos/eos
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Building wheels for collected packages: eos-py
    Building wheel for eos-py (pyproject.toml): started
    Building wheel for eos-py (pyproject.toml): still running...
    Building wheel for eos-py (pyproject.toml): finished with status 'done'
    Created wheel for eos-py: filename=eos_py-1.5.0-cp38-cp38-macosx_11_0_arm64.whl size=607694 sha256=577bb44df95eac705098e3911712f6e6a5f3c9863d9ff12b044a28d61bd8590a
    Stored in directory: /private/var/folders/x4/dc0wnrqs60v8cfdj3vx06s1h0000gn/T/pip-ephem-wheel-cache-pv7_rlly/wheels/88/16/da/1b56921bcca532fb8d60b9de70e13ab41100089e255fc6708a
  Successfully built eos-py
                                                             ✓ 79.13s
Repairing wheel...
  
  + delocate-wheel --require-archs arm64 -w /private/var/folders/x4/dc0wnrqs60v8cfdj3vx06s1h0000gn/T/cibw-run-67h7fric/cp38-macosx_arm64/repaired_wheel -v /private/var/folders/x4/dc0wnrqs60v8cfdj3vx06s1h0000gn/T/cibw-run-67h7fric/cp38-macosx_arm64/built_wheel/eos_py-1.5.0-cp38-cp38-macosx_11_0_arm64.whl
  Fixing: /private/var/folders/x4/dc0wnrqs60v8cfdj3vx06s1h0000gn/T/cibw-run-67h7fric/cp38-macosx_arm64/built_wheel/eos_py-1.5.0-cp38-cp38-macosx_11_0_arm64.whl
  Traceback (most recent call last):
    File "/private/var/folders/x4/dc0wnrqs60v8cfdj3vx06s1h0000gn/T/cibw-run-67h7fric/cp38-macosx_arm64/build/venv/bin/delocate-wheel", line 8, in <module>
      sys.exit(main())
    File "/private/var/folders/x4/dc0wnrqs60v8cfdj3vx06s1h0000gn/T/cibw-run-67h7fric/cp38-macosx_arm64/build/venv/lib/python3.8/site-packages/delocate/cmd/delocate_wheel.py", line 116, in main
      copied = delocate_wheel(
    File "/private/var/folders/x4/dc0wnrqs60v8cfdj3vx06s1h0000gn/T/cibw-run-67h7fric/cp38-macosx_arm64/build/venv/lib/python3.8/site-packages/delocate/delocating.py", line 1090, in delocate_wheel
      out_wheel_fixed = _check_and_update_wheel_name(
    File "/private/var/folders/x4/dc0wnrqs60v8cfdj3vx06s1h0000gn/T/cibw-run-67h7fric/cp38-macosx_arm64/build/venv/lib/python3.8/site-packages/delocate/delocating.py", line 914, in _check_and_update_wheel_name
      new_name, problematic_files = _calculate_minimum_wheel_name(
    File "/private/var/folders/x4/dc0wnrqs60v8cfdj3vx06s1h0000gn/T/cibw-run-67h7fric/cp38-macosx_arm64/build/venv/lib/python3.8/site-packages/delocate/delocating.py", line 848, in _calculate_minimum_wheel_name
      raise DelocationError(
  delocate.libsana.DelocationError: Failed to find any binary with the required architecture: 'arm64'
  Error: Command delocate-wheel --require-archs arm64 -w /private/var/folders/x4/dc0wnrqs60v8cfdj3vx06s1h0000gn/T/cibw-run-67h7fric/cp38-macosx_arm64/repaired_wheel -v /private/var/folders/x4/dc0wnrqs60v8cfdj3vx06s1h0000gn/T/cibw-run-67h7fric/cp38-macosx_arm64/built_wheel/eos_py-1.5.0-cp38-cp38-macosx_11_0_arm64.whl failed with code 1. 

I fail to see why delocate-wheel would fail to find the binary: It looks for eos_py-1.5.0-cp38-cp38-macosx_11_0_arm64.whl, and that is exactly the file that the "Building wheel" step built.

This might be related to #2080, but in that other issue, the user reports that the build step actually builds a wheel named x86_64 instead of an arm64 wheel. In my case, the file name looks correct and contains arm64. But perhaps it still built an x64 wheel and just named it "...arm64"?

I should note that all my builds ran through fine with cibuildwheel 2.15.0 (though I don't have a Mac to actually test whether the built wheels are arm64, like the filename says they are - https://pypi.org/project/eos-py/#files), this issue only started today when I updated my CI script to use cibuildwheel~=2.22.0.

Build log

https://productionresultssa11.blob.core.windows.net/actions-results/9fade6ef-cf43-4756-ab68-9cad1b9fa2d3/workflow-job-run-0e352a2a-a03f-559b-855c-21904474bf95/logs/job/job-logs.txt?rsct=text%2Fplain&se=2024-12-11T17%3A58%3A14Z&sig=3Yaqd%2FICtElPCeRoAnwbP%2BWkdjwkKFbyFlXVaMwoxOA%3D&ske=2024-12-12T03%3A02%3A09Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2024-12-11T15%3A02%3A09Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2024-11-04&sp=r&spr=https&sr=b&st=2024-12-11T17%3A48%3A09Z&sv=2024-11-04

CI config

https://github.com/patrikhuber/eos/blob/master/.github/workflows/cibuildwheel.yml

[Czaki]

Ok. So on 2.15.0 it is not working. The delocate version that cibuildwheel use then, do not properly validate wheels.

You are using macos-13 GitHub action runner that is x86_64 one. So to build ARM wheel there is a need to cross-compile from x86-64. However, you are using a custom cmake wrapper and I guess that your wrapper is not properly pass cross compile options to cmake and in reality, you are building x86_64 wheels labeled as ARM one.

The simplest solution for you will be to build ARM wheels on macos-14 (macos-latest) runner. Then it will be building wheels on native architecture.

I suggest stopping using custom cmake wrapper and use something like scikit-build-core. https://pypi.org/project/scikit-build-core/. Then its maintainers will take care of features like cross compilation.

[patrikhuber]

@Czaki Thanks a lot for the thorough reply and checking my build, and all the information. That is very useful. I'll have to set aside a bit of time to go through it in more details, this turns out to be more effort than I thought it would be.

Thanks for the suggestion to use scikit-build-core. I think I've looked at it before but haven't used it, as I didn't want to add more complexity by introducing yet another tool/framework to my build (and usually these tools die or become unmaintained after a few years, or there's a new kid around the block, then you have to spend time on migrating and rewriting things etc). However it does look pretty good and well-maintained and you made me tempted to try it. One thing I did plan to add to my Python build though is a few custom lines of code that clone an installation-local copy of vcpkg and then run that in manifest mode to fetch the dependencies of my library (instead of using git submodules, as I am now). The advantage of that is that a user doesn't have to install any C++ library build dependencies locally themselves (like with submodules), but I can use a package manager (vcpkg) instead of submodules. When a user does pip install eos-py, my setup.py clones vcpkg (via shell invokes) and then calls cmake with CMAKE_TOOLCHAIN_FILE=./build_local_vcpkg/....vcpkg.cmake. I've tested this locally and it works great. However I think scikit-build-core won't allow this as its build file is declarative (toml). Do you have any views on this?

Thanks a lot!