Build-dependent environment variables

[Time0o]

Description

I would like to set some build environment variables that depend CIBW_BEFORE_BUILD, that seems like it would be a very common requirement, e.g. in my case I want to:

• Install a Python wheel "foo" from some remote location in CIBW_BEFORE_BUILD.
• Set an environment variable to a path inside the installed package, e.g. CIBW_ENVIRONMENT'FOO_DIR=$(python "print(__import__(\"foo\").__path__[0])")'

Obviously this does not work since CIBW_ENVIRONMENT is evaluated before CIBW_BEFORE_BUILD, the next logical thing to try would then be CIBW_BEFORE_BUILD='<fetch wheel> && export FOO_DIR="..."'. But this also does not work since unlike with CIBW_ENVIRONMENT, FOO_DIR is then not available during the build step. CIBW_BEFORE_ALL is also not very helpful since I would then have to "manually" iterate over all Python versions and their locations in /opt for which to install the foo wheel.

I'm not sure if this is necessarily a bug but it feels like an oversight unless there is a way to do this I have not considered.

Build log

No response

CI config

No response

[Czaki]

Why cannot you import foo package during the build phase?

[Time0o]

Why cannot you import foo package during the build phase?

You are probably right, do you mean moving this logic into setup.py? Up until now I have had setup.py read FOO_DIR from the environment and pass that to CMake via DFOO_DIR=$FOO_DIR. But I could also find foo.__path__[0] in there if the foo module exists.

[Czaki]

Hm. You are using custom wrapper around cmake?
Based on my expertise, it may be worth checking if you could migrate to scikit-build-core. https://pypi.org/project/scikit-build-core/

And resolve your problem by cmake extension like https://github.com/scikit-build/cython-cmake

If you provide more detailed description, maybe @henryiii could suggest something.

But for minimal required change, I'm suggest migrating logic to setup.py.

[Time0o]

Hm. You are using custom wrapper around cmake? Based on my expertise, it may be worth checking if you could migrate to scikit-build-core. https://pypi.org/project/scikit-build-core/

And resolve your problem by cmake extension like https://github.com/scikit-build/cython-cmake

I will look into that, thanks.

If you provide more detailed description, maybe @henryiii could suggest something.

What I want to do is build Python wrappings for an out of tree MLIR-based project. The issue here is that these are built by CMake and dumped into the CMake binary directory as a directory structure consisting of a bunch of Python files and Python extension modules. So my approach to making this work with setup.py is to simply provide a custom bdist_wheel step that runs CMake and then points distribution.packages/package_dir/package_data to the generated files. And now, in order for CMake to find MLIR, I want users to be able to specify MLIR_DIR if they build locally and to use existing MLIR Python wheels when building under cibuildwheel.

[Czaki]

So, based on my knowledge, using scikit-build-core and implementing your own find_MLIR cmake function will be the best solution.
It will require more work, but should be the most stable solution in future.

Optionally, you may read env variable in cmake. See: https://github.com/4DNucleome/PartSegCore-compiled-backend/blob/42c1b46104e02c5c5793245df35b868666da51ad/CMakeLists.txt#L19

[Time0o]

After making some progress in on this it seems like cibuildwheel cannot handle this case anyways. It fails with "Build failed because a pure Python wheel was generated" (which is not true but I can sort of see why cibuildwheel thinks it is). I assume scikit-build-core won't help me here?

This is now deviating from my original question quite a bit but this is where I am at:

  adding 'mlir/execution_engine.py'                                                                                                                                             
  adding 'mlir/ir.py'                                                                                                                                                           
  adding 'mlir/passmanager.py'                                                                                                                                                  
  adding 'mlir/rewrite.py'                                                                                                                                                      
  adding 'mlir/_mlir_libs/__init__.py'                                                                                                                                          
  adding 'mlir/_mlir_libs/_mlir.cpython-311-x86_64-linux-gnu.so'                                                                                                                
  adding 'mlir/_mlir_libs/_mlirDialectsFoo.cpython-311-x86_64-linux-gnu.so'                                                                                              
  adding 'mlir/_mlir_libs/_mlirExecutionEngine.cpython-311-x86_64-linux-gnu.so'                                                                                                 
  adding 'mlir/_mlir_libs/_mlirExecutionEngine.pyi'                                                                                                                             
  adding 'mlir/_mlir_libs/_mlirRegisterEverything.cpython-311-x86_64-linux-gnu.so'                                                                                              
  adding 'mlir/_mlir_libs/libMLIRFooPythonCAPI.so'                                                                                                                       
  adding 'mlir/_mlir_libs/libMLIRFooPythonCAPI.so.20.0'                                                                                                                  
  adding 'mlir/_mlir_libs/_mlir/__init__.pyi'                                                                                                                                   
  adding 'mlir/_mlir_libs/_mlir/ir.pyi'                                                                                                                                         
  adding 'mlir/_mlir_libs/_mlir/passmanager.pyi'                                                                                                                                
  adding 'mlir/dialects/_builtin_ops_gen.py'                                                                                                                                    
  adding 'mlir/dialects/_foo_ops_gen.py'                                                                                                                                 
  adding 'mlir/dialects/_func_ops_gen.py'                                                                                                                                       
  adding 'mlir/dialects/_ods_common.py'                                                                                                                                         
  adding 'mlir/dialects/builtin.py'                                                                                                                                             
  adding 'mlir/dialects/foo.py'                                                                                                                                          
  adding 'mlir/dialects/func.py'                                                                                                                                                
  adding 'mlir/extras/meta.py'                                                                                                                                                  
  adding 'mlir/extras/types.py'                                                                                                                                                 
  adding 'mlir-0.1.0.dist-info/METADATA'
  adding 'mlir-0.1.0.dist-info/NOTICE'
  adding 'mlir-0.1.0.dist-info/WHEEL'
  adding 'mlir-0.1.0.dist-info/top_level.txt'
  adding 'mlir-0.1.0.dist-info/RECORD'
  removing build/bdist.linux-x86_64/wheel
  Building wheel for mlir (pyproject.toml): finished with status 'done'
  Created wheel for mlir: filename=mlir-0.1.0-py3-none-any.whl size=106597762 sha256=dc9164aedfed158500a31d527cac16f4200dc05a2f9bd57dfaeae4373b42c3b4
  Stored in directory: /tmp/pip-ephem-wheel-cache-fatjr1fs/wheels/aa/0c/49/7c4d39428e78274ff9f1db5b9c539aa0d6bd6becda70207fa7
Successfully built mlir
    + /opt/python/cp39-cp39/bin/python -c 'import sys, json, glob; json.dump(glob.glob('"'"'/tmp/cibuildwheel/built_wheel/*.whl'"'"'), sys.stdout)'
    + rm -rf /tmp/cibuildwheel/repaired_wheel
    + mkdir -p /tmp/cibuildwheel/repaired_wheel

                                                                     ✕ 350.39s
Error:
Build failed because a pure Python wheel was generated.