Missing source distribution for python dependencies - yt-fts

[NotJoeMartinez]

Output of brew config

HOMEBREW_VERSION: 4.2.17-25-ga21c71f
ORIGIN: https://github.com/Homebrew/brew
HEAD: a21c71fa97aba7fafd7e41a7b1dbb2a6f9d54a57
Last commit: 5 hours ago
Core tap HEAD: 4951517b150814fd57ad3df98a88fb83d5a31fb5
Core tap last commit: 87 minutes ago
Core tap JSON: 09 Apr 07:11 UTC
Core cask tap HEAD: 2033f09727226794c15daac722b372cb01d8f62b
Core cask tap last commit: 32 minutes ago
Core cask tap JSON: 09 Apr 07:11 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_EDITOR: nvim
HOMEBREW_MAKE_JOBS: 8
HOMEBREW_SORBET_RUNTIME: set
Homebrew Ruby: 3.1.4 => /opt/homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/bin/ruby
CPU: octa-core 64-bit arm_ibiza
Clang: 15.0.0 build 1500
Git: 2.39.3 => /Applications/Xcode.app/Contents/Developer/usr/bin/git
Curl: 8.4.0 => /usr/bin/curl
macOS: 14.2-arm64
CLT: 15.3.0.0.1.1708646388
Xcode: 15.3

Output of brew doctor

Your system is ready to brew.

Description of issue

Howdy, I've been trying to get my python command line tool yt-fts published on homebrew/core and I've been having some trouble building my Formula from source. I believe this is due to some package dependencies for chroma not having source distributions available on pypi, specifically onnxruntime and pulsar-client.

I first tried using the brew create command and it failed when it reached the onnxruntime package.

brew create --verbose --python https://github.com/NotJoeMartinez/yt-fts/releases/download/v0.1.43/yt-fts-0.1.43.tar.gz
...
==> Getting PyPI info for "oauthlib==3.2.2"
==> Getting PyPI info for "onnxruntime==1.17.1"
Error: Unable to resolve some dependencies. Please update the resources for "yt-fts" manually.

I then tried using the homebrew-pypi-poet tool recommended by the docs and it managed to fetch all of my dependencies but onnxruntime and pulsar-client, leaving them defined with empty url and sha256 strings.

• https://pypi.org/project/onnxruntime/
• https://pypi.org/project/pulsar-client/

I've also tried just using the .whl files and hashes for these packages instead but after attempting to build with

HOMEBREW_NO_INSTALL_FROM_API=1 brew install --build-from-source --verbose --debug yt-fts.rb

I get this bcrypt build error.

  running build_ext
  running build_rust

      =============================DEBUG ASSISTANCE=============================
      If you are seeing a compilation error please try the following steps to
      successfully install bcrypt:
      1) Upgrade to the latest pip and try again. This will fix errors for most
         users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
      2) Ensure you have a recent Rust toolchain installed. bcrypt requires
         rustc >= 1.64.0. (1.63 may be used by setting the BCRYPT_ALLOW_RUST_163
         environment variable)

      Python: 3.12.2
      platform: macOS-14.2-arm64-arm-64bit
      pip: n/a
      setuptools: 69.2.0
      setuptools_rust: 1.9.0
      rustc: n/a
      =============================DEBUG ASSISTANCE=============================

  error: can't find Rust compiler

  If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.

  To update pip, run:

      pip install --upgrade pip

  and then retry package installation.

  If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.

  This package requires Rust >=1.63.0.
  error: subprocess-exited-with-error
  
  × Building wheel for bcrypt (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /opt/homebrew/Cellar/yt-fts/0.1.43/libexec/bin/python /opt/homebrew/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /private/tmp/tmpkroofmcm
  cwd: /private/tmp/yt-fts--bcrypt-20240409-35062-yxp785/bcrypt-4.1.2
  Building wheel for bcrypt (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for bcrypt
Failed to build bcrypt
ERROR: Could not build wheels for bcrypt, which is required to install pyproject.toml-based projects
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/git --version
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --version
/opt/homebrew/Library/Homebrew/ignorable.rb:29:in `block in raise'
BuildError: Failed executing: python3.12 -m pip --python=/opt/homebrew/Cellar/yt-fts/0.1.43/libexec/bin/python install --verbose --no-deps --no-binary=:all: --ignore-installed --no-compile /private/tmp/yt-fts--bcrypt-20240409-35062-yxp785/bcrypt-4.1.2

I'm not sure how this got to rust compilation errors for a package that already has it as an sdist in my formula. I'm a bit lost on where to go from here and was hoping someone could take a look at my formula.

• https://github.com/NotJoeMartinez/homebrew-yt-fts/blob/828452656e4b58b31308606be851a894e758ec23/Formula/yt-fts.rb

Thanks in advance :)

[ZhongRuoyu]

Some Python packages require additional dependencies to build. In this case bcrypt is missing rust dependency; to add that you can add depends_on "rust" => :build to the formula. Check out dvc.rb from the core tap for an example.

Also, you may want to use numpy and certifi dependencies from Homebrew (i.e., via depends_on); see https://docs.brew.sh/Python-for-Formula-Authors#libraries.

[NotJoeMartinez]

This does solve the rust error but now I'm getting one related to chroma-hnswlib and numpy with and without the build flags

depends_on "numpy"
depends_on "certify"

Processing /private/tmp/yt-fts--chroma-hnswlib-20240505-16474-6dvmqi/chroma-hnswlib-0.7.3
  Installing build dependencies: started
  Running command pip subprocess to install build dependencies
  Collecting setuptools>=42
    Using cached setuptools-69.5.1-py3-none-any.whl
  Collecting wheel
    Using cached wheel-0.43.0-py3-none-any.whl
  Collecting numpy>=1.10.0
    Using cached numpy-1.26.4.tar.gz (15.8 MB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'error'
    error: subprocess-exited-with-error

I also tried removing the dependencies from the download list but got the same error

  # resource "certifi" do
  #   url "https://files.pythonhosted.org/packages/71/da/e94e26401b62acd6d91df2b52954aceb7f561743aa5ccc32152886c76c96/certifi-2024.2.2.tar.gz"
  #   sha256 "0569859f95fc761b18b45ef421b1290a0f65f147e92a1e5eb3e635f9a5e4e66f"
  # end

  # resource "numpy" do
  #   url "https://files.pythonhosted.org/packages/65/6e/09db70a523a96d25e115e71cc56a6f9031e7b8cd166c1ac8438307c14058/numpy-1.26.4.tar.gz"
  #   sha256 "2a02aba9ed12e4ac4eb3ea9421c420301a0c6460d9830d74a9df87efa4912010"
  # end

Edit:
The above build error is not related to onnxruntime, I and I can't say it's related until I fix the numpy error

I think this is related to the onnxruntime and pulsar-client dependences. They don't have source distributions, so I tried adding their wheels which does nothing. I'm not sure if this is because I'm not building them correctly in the install function or because the wheels are not supported.

  resource "onnxruntime" do
    url "https://files.pythonhosted.org/packages/a1/a2/00ea929ccf9b4702af11e13fa52e9ac607aecc867b041ffcbe646e29f880/onnxruntime-1.17.1-cp312-cp312-macosx_11_0_universal2.whl"
    sha256 "40f08e378e0f85929712a2b2c9b9a9cc400a90c8a8ca741d1d92c00abec60843"
  end

  resource "pulsar-client" do
    url "https://files.pythonhosted.org/packages/0f/a3/02b50d7abc5537438d44ba94342dbb78261ee4a0ae5620b4553ae233525c/pulsar_client-3.4.0-cp312-cp312-macosx_10_15_universal2.whl"
    sha256 "1e077a4839be3ead3de3f05b4c244269dca2df07f47cea0b90544c7e9dc1642f"
  end

  def install
    virtualenv_install_with_resources
  end

There don't seem to be any formulas that use chroma as a dependency and the last PR that tried ran into an error related to
onnxruntime not being supported by python3.12.

Homebrew/homebrew-core#160501

[NotJoeMartinez]

I got it to build by adding these.

  depends_on "rust" => :build #for bcrypt
  depends_on "cmake" => :build # for hnswlib
  depends_on "onnxruntime"
  depends_on "numpy" 
  depends_on "certifi" 

Apparently chroma-hnswlib needs cmake to use numpy properly? It takes about 30 minuets to build but at least I solved this part.