From 144f98b8ed3f164aa8eb6596575844238ead3354 Mon Sep 17 00:00:00 2001 From: Pieter Pas Date: Mon, 20 May 2024 19:44:04 +0200 Subject: [PATCH] [Docs] updated cross-compilation instructions --- docs/Cross-compilation.md | 298 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 286 insertions(+), 12 deletions(-) diff --git a/docs/Cross-compilation.md b/docs/Cross-compilation.md index f5361c9..285cfeb 100644 --- a/docs/Cross-compilation.md +++ b/docs/Cross-compilation.md @@ -5,6 +5,28 @@ Cross-compiling Python extension modules is supported out of the box through [CMake toolchain files](https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html). +When building packages using a tool like [`cibuildwheel`](https://github.com/pypa/cibuildwheel), +cross-compilation will also be enabled automatically whenever appropriate +(e.g. when building ARM64 packages on an Intel Mac). + +> Table of contents +>  [Terminology](#terminology) +>  [Simple example](#simple-example) +>   [Caveats](#caveats) +>  [Complete cross-compilation workflow](#complete-cross-compilation-workflow) +>   [Set up the environment](#set-up-the-environment) +>   [Download a cross-compilation toolchain for your _host_ system](#download-a-cross-compilation-toolchain-for-your-host-system) +>   [Download Python for your _host_ system](#download-python-for-your-host-system) +>   [Inspect and customize the toolchain files and py-build-cmake configuration](#inspect-and-customize-the-toolchain-files-and-py-build-cmake-configuration) +>   [Cross-compile the pybind11-project example package using py-build-cmake](#cross-compile-the-pybind11-project-example-package-using-py-build-cmake) +>   [Automated Bash scripts](#automated-bash-scripts) +>   [A closer look at the CMake toolchain files](#a-closer-look-at-the-cmake-toolchain-files) +>   [Cross-compilation of dependencies](#cross-compilation-of-dependencies) +>  [Automatic cross-compilation](#automatic-cross-compilation) +>   [Windows](#windows) +>   [macOS](#macos) +>   [Linux](#linux) + ## Terminology - Build system: the system used for building, on which py-build-cmake is @@ -30,23 +52,22 @@ relative path of the CMake toolchain file to use. For example: ```toml implementation = 'cp' # CPython -version = '39' # 3.9 -abi = 'cp39' # (default ABI for CPython 3.9) +version = '312' # 3.12 +abi = 'cp312' # (default ABI for CPython 3.12) arch = 'linux_aarch64' toolchain_file = 'aarch64-linux-gnu.cmake' +library = "/path-to-sysroot/usr/lib/aarch64-linux-gnu/libpython3.12.so" +include_dir = "/path-to-sysroot/usr/include/python3.12" ``` -This will generate a package for CPython 3.9 on Linux for 64-bit ARM, using the +This will generate a package for CPython 3.12 on Linux for 64-bit ARM, using the compilers defined by the toolchain file `aarch64-linux-gnu.cmake` (which you should provide as well). The format for the values in this file is the same as the format used for the -tags in wheel filenames, for example `package-1.2.3-cp39-cp39-linux_aarch64.whl`. +tags in wheel filenames, for example `pkg-1.2.3-cp312-cp312-linux_aarch64.whl`. For details about platform compatibility tags, see the PyPA specification: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags. -For more information about cross-compilation, ready-to-use toolchains, and -example toolchain files, see . - ### Caveats Note that `py-build-cmake` does not check the Python version when @@ -58,16 +79,269 @@ e.g. by setting the appropriate hints and artifacts variables: - - -You can either specify these in your toolchain file, or in the -`py-build-cmake.cross.toml` configuration, for example: +You can either specify these in your `py-build-cmake.cross.toml` configuration +as shown above, or in your CMake toolchain file. + +--- + +## Complete cross-compilation workflow + +This section will guide you through the full process of cross-compiling your +Python package. + +You'll need the following: + +- A cross-compilation toolchain that provides compilers for your _host_ system + (while running on your _build_ system). +- A (pre-compiled) Python installation for the _host_ system. +- A py-build-cmake configuration file and a CMake toolchain file with the + appropriate settings for your use case. + +Let's go over these requirements step by step: + +### Set up the environment + +We'll first clone `py-build-cmake` and its example projects: + +```sh +git clone https://github.com/tttapa/py-build-cmake --branch=0.2.0a13 +cd py-build-cmake +``` + +### Download a cross-compilation toolchain for your _host_ system + +It is important that the toolchain has an older version of glibc and the linux +headers, so that the resulting package is compatible with a wide range of +Linux distributions. The toolchains in your system's package manager are usually +not compatible with older systems. + +You can find ready-to-use toolchains with good compatibility at https://github.com/tttapa/toolchains. + +```sh +# Create a directory to save the cross-compilation toolchains into +mkdir -p toolchains +# Download and extract the toolchain for AArch64 +url="https://github.com/tttapa/toolchains/releases/download/0.1.0" +wget "$url/x-tools-aarch64-rpi3-linux-gnu-gcc14.tar.xz" -O- | tar xJ -C toolchains +# Verify that the toolchain works +./toolchains/x-tools/aarch64-rpi3-linux-gnu/bin/aarch64-rpi3-linux-gnu-gcc --version +``` + +### Download Python for your _host_ system + +CMake needs to be able to locate the Python header files, and in some cases the +Python shared library before you can build your package. + +You can download these from https://github.com/tttapa/python-dev. + +```sh +# The toolchain is read-only by default, make it writable to add Python to it +chmod u+w "toolchains/x-tools/aarch64-rpi3-linux-gnu" +# Download and extract the Python binaries for AArch64 +url="https://github.com/tttapa/python-dev/releases/download" +wget "$url/0.2.0/python-dev-aarch64-rpi3-linux-gnu.tar.xz" -O- | tar xJ -C toolchains +``` + +### Inspect and customize the toolchain files and py-build-cmake configuration + +The Python installations from https://github.com/tttapa/python-dev already +include the necessary CMake toolchain files and `py-build-cmake` configuration +files. Inspect them and customize to your specific setup if necessary. +(No changes necessary when just following this guide using the example projects). + +```sh +# List the available CMake toolchain files. +ls toolchains/x-tools/*.cmake +# List the available py-build-cmake cross-compilation configuration files. +ls toolchains/x-tools/*.py-build-cmake.cross.toml +``` + +We'll have a quick look at the `toolchains/x-tools/aarch64-rpi3-linux-gnu.python3.11.py-build-cmake.cross.toml` as an example: ```toml +implementation = 'cp' +version = '311' +abi = 'cp311' +arch = 'manylinux_2_27_aarch64' +toolchain_file = 'aarch64-rpi3-linux-gnu.python.toolchain.cmake' + [cmake.options] -Python3_LIBRARY = "/path-to-sysroot/usr/lib/aarch64-linux-gnu/libpython3.9.so" -Python3_INCLUDE_DIR = "/path-to-sysroot/usr/include/python3.9" +TOOLCHAIN_PYTHON_VERSION = '3.11' ``` -You can find a full example in [examples/pybind11-project/py-build-cmake.cross.example.toml](https://github.com/tttapa/py-build-cmake/blob/main/examples/pybind11-project/py-build-cmake.cross.example.toml). +The Python implementation, version, ABI and architecture were already discussed +in a previous section. The CMake toolchain file simply points to a file in the +same directory as the configuration file. We'll have a look at what it does +shortly. Finally, the `TOOLCHAIN_PYTHON_VERSION` variable tells the toolchain +file which version of Python to add to the CMake search paths. + +### Cross-compile the pybind11-project example package using py-build-cmake + +We now have our toolchain, the Python installation, and a configuration file +that selects the correct toolchain and Python version, so we are ready to +cross-compile our first package. We'll use the `pybind11-project` example that's +included with `py-build-cmake`. + +```sh +# Install PyPA build as a build front-end +python3 -m pip install -U build +# Cross-compile the example package using our cross-compilation configuration +python3 -m build -w examples/pybind11-project \ + -C cross="$PWD/toolchains/x-tools/aarch64-rpi3-linux-gnu.python3.11.py-build-cmake.cross.toml" +``` +If everything worked as expected, you should see output similar to the following. +```sh +-- The C compiler identification is GNU 14.1.0 +-- The CXX compiler identification is GNU 14.1.0 +-- Check for working C compiler: py-build-cmake/toolchains/x-tools/aarch64-rpi3-linux-gnu/bin/aarch64-rpi3-linux-gnu-gcc - skipped +[...] +-- Found Python3: py-build-cmake/toolchains/x-tools/aarch64-rpi3-linux-gnu/python3.11/usr/local/include/python3.11 (found version "3.11.3") found components: Development.Module +-- Detecting pybind11 CMake location +-- pybind11 CMake location: /tmp/build-env-xxxxx/lib/python3.9/site-packages/pybind11/share/cmake/pybind11 +-- Performing Test HAS_FLTO +-- Performing Test HAS_FLTO - Success +-- Found pybind11: /tmp/build-env-xxxxx/lib/python3.9/site-packages/pybind11/include (found version "2.12.0") +-- Configuring done (1.4s) +-- Generating done (0.0s) +-- Build files have been written to: py-build-cmake/examples/pybind11-project/.py-build-cmake_cache/cp311-cp311-manylinux_2_27_aarch64 +[ 50%] Building CXX object CMakeFiles/_add_module.dir/src/add_module.cpp.o +[100%] Linking CXX shared module _add_module.cpython-311-aarch64-linux-gnu.so +[100%] Built target _add_module +-- Installing: /tmp/xxxxx/staging/pybind11_project/_add_module.cpython-311-aarch64-linux-gnu.so +[...] +Successfully built pybind11_project-0.2.0a13-cp311-cp311-manylinux_2_27_aarch64.whl +``` +You can see that CMake is using the cross-compiler we downloaded, and that it +managed to locate the version of Python we requested (CPython 3.11 for AArch64). +It is important to verify the module extension suffix +(`.cpython-311-aarch64-linux-gnu.so` in this case) and the Wheel tags +(`cp311-cp311-manylinux_2_27_aarch64`). + +You can now copy the Wheel package in `examples/pybind11-project/dist/pybind11_project-0.2.0a13-cp311-cp311-manylinux_2_27_aarch64.whl` +to e.g. a Raspberry Pi and install it using `pip install`. + +### Automated Bash scripts + +The included `scripts/download-cross-toolchains-linux.sh` script downloads the +toolchains and Python installations for common architectures and current +versions of Python. You can then run the `examples/pybind11-project/cross-compile-linux.sh` +and `examples/nanobind-project/cross-compile-linux.sh` scripts to cross-compile +these example packages for this wide range of configurations. + +```sh +./scripts/download-cross-toolchains-linux.sh +./examples/pybind11-project/cross-compile-linux.sh +./examples/nanobind-project/cross-compile-linux.sh +``` + +### A closer look at the CMake toolchain files + +Let us now look at `toolchains/x-tools/aarch64-rpi3-linux-gnu.python.toolchain.cmake` +more closely: +```cmake +include("${CMAKE_CURRENT_LIST_DIR}/aarch64-rpi3-linux-gnu.toolchain.cmake") + +# [...] + +# User options +set(TOOLCHAIN_PYTHON_VERSION "3" CACHE STRING "Python version to locate") +option(TOOLCHAIN_NO_FIND_PYTHON "Do not set the FindPython hints" Off) +option(TOOLCHAIN_NO_FIND_PYTHON3 "Do not set the FindPython3 hints" Off) + +# [...] + +# Internal variables +set(TOOLCHAIN_PYTHON_ROOT "${CMAKE_CURRENT_LIST_DIR}/${CROSS_GNU_TRIPLE}/python${TOOLCHAIN_PYTHON_VERSION}") +list(APPEND CMAKE_FIND_ROOT_PATH "${TOOLCHAIN_PYTHON_ROOT}") + +# Determine the paths and other properties of the Python installation +function(toolchain_locate_python) + # [...] +endfunction() + +if (NOT TOOLCHAIN_NO_FIND_PYTHON OR NOT TOOLCHAIN_NO_FIND_PYTHON3) + # [...] + # Set FindPython hints and artifacts + if (NOT TOOLCHAIN_NO_FIND_PYTHON) + set(Python_ROOT_DIR ${TOOLCHAIN_PYTHON_ROOT_DIR} CACHE PATH "" FORCE) + set(Python_LIBRARY ${TOOLCHAIN_PYTHON_LIBRARY} CACHE FILEPATH "" FORCE) + set(Python_INCLUDE_DIR ${TOOLCHAIN_PYTHON_INCLUDE_DIR} CACHE PATH "" FORCE) + set(Python_INTERPRETER_ID "Python" CACHE STRING "" FORCE) + endif() + # Set FindPytho3 hints and artifacts + if (NOT TOOLCHAIN_NO_FIND_PYTHON3) + set(Python3_ROOT_DIR ${TOOLCHAIN_PYTHON_ROOT_DIR} CACHE PATH "" FORCE) + set(Python3_LIBRARY ${TOOLCHAIN_PYTHON_LIBRARY} CACHE FILEPATH "" FORCE) + set(Python3_INCLUDE_DIR ${TOOLCHAIN_PYTHON_INCLUDE_DIR} CACHE PATH "" FORCE) + set(Python3_INTERPRETER_ID "Python" CACHE STRING "" FORCE) + endif() + # Set pybind11 hints + # [...] + set(PYTHON_MODULE_DEBUG_POSTFIX ${PYTHON_MODULE_DEBUG_POSTFIX} CACHE INTERNAL "") + set(PYTHON_MODULE_EXTENSION ${PYTHON_MODULE_EXTENSION} CACHE INTERNAL "") + set(PYTHON_IS_DEBUG ${PYTHON_IS_DEBUG} CACHE INTERNAL "") + # Set nanobind hints + # [...] + set(NB_SUFFIX ${NB_SUFFIX} CACHE INTERNAL "") + set(NB_SUFFIX_S ${NB_SUFFIX_S} CACHE INTERNAL "") +endif() +``` + +First, it includes the standard toolchain file for the platform (the one that +sets the platform properties and compiler paths, see below). +Next, it declares the options that you might want to configure as a user, such +as the version of Python to make available, and whether to set CMake's +[FindPython](https://cmake.org/cmake/help/latest/module/FindPython.html) and +[FindPython3](https://cmake.org/cmake/help/latest/module/FindPython3.html) hints. +Then it adds the selected Python installation to CMake's search path, it +locates the Python library and include paths, its properties such as the +ABI and the extension suffix, and sets the FindPython hints. +Finally, it sets some specific cache variables that are needed by the +[pybind11](https://github.com/pybind/pybind11) and [nanobind](https://github.com/wjakob/nanobind) +frameworks. + +The file `toolchains/x-tools/aarch64-rpi3-linux-gnu.toolchain.cmake` contains: +```cmake +# System information +set(CMAKE_SYSTEM_NAME "Linux") +set(CMAKE_SYSTEM_PROCESSOR "aarch64") +set(CROSS_GNU_TRIPLE "aarch64-rpi3-linux-gnu" + CACHE STRING "The GNU triple of the toolchain to use") +set(CMAKE_LIBRARY_ARCHITECTURE "aarch64-linux-gnu") + +# Compiler flags +set(CMAKE_C_FLAGS_INIT "-mcpu=cortex-a53+crc+simd") +set(CMAKE_CXX_FLAGS_INIT "-mcpu=cortex-a53+crc+simd") +set(CMAKE_Fortran_FLAGS_INIT "-mcpu=cortex-a53+crc+simd") + +# Search path configuration +set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) +set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) +set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) +set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY) + +# Packaging +set(CPACK_DEBIAN_PACKAGE_ARCHITECTURE "arm64") + +# Compiler binaries +set(TOOLCHAIN_DIR "${CMAKE_CURRENT_LIST_DIR}/${CROSS_GNU_TRIPLE}") +set(CMAKE_C_COMPILER "${TOOLCHAIN_DIR}/bin/${CROSS_GNU_TRIPLE}-gcc" + CACHE FILEPATH "C compiler") +set(CMAKE_CXX_COMPILER "${TOOLCHAIN_DIR}/bin/${CROSS_GNU_TRIPLE}-g++" + CACHE FILEPATH "C++ compiler") +set(CMAKE_Fortran_COMPILER "${TOOLCHAIN_DIR}/bin/${CROSS_GNU_TRIPLE}-gfortran" + CACHE FILEPATH "Fortran compiler") +``` + +### Cross-compilation of dependencies + +If your Python package depends on native libraries, you'll have to cross-compile +these dependencies as well. You can either compile them from source yourself, +using the CMake toolchain files included with the toolchain, or you can use +the Conan package manager to build these dependencies for you. A Conan profile +is included as well, see e.g. `toolchains/x-tools/aarch64-rpi3-linux-gnu.profile.conan`. + +--- ## Automatic cross-compilation