For a formula that wraps a Python package, why did `brew bump-formula-pr` stop passing for a Test PyPI URL? #5193

PawelLipski · 2024-03-11T12:40:33Z

PawelLipski
Mar 11, 2024

Output of `brew config`

HOMEBREW_VERSION: 4.2.11-105-gae0ceee
ORIGIN: https://github.com/Homebrew/brew
HEAD: ae0ceee74f1570083a24e21be2509443d3a4337f
Last commit: 12 hours ago
Core tap HEAD: 9177ba51b781f5874dd6af7f651a3b50f607c487
Core tap last commit: 2 days ago
Core tap JSON: 11 Mar 12:21 UTC
Core cask tap HEAD: bb5569edf0e89f5a685397472aa5d2fdbc025825
Core cask tap last commit: 2 days ago
Core cask tap JSON: 11 Mar 12:00 UTC
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_EDITOR: vim
HOMEBREW_MAKE_JOBS: 16
HOMEBREW_SORBET_RUNTIME: set
Homebrew Ruby: 3.1.4 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/bin/ruby
CPU: 16-core 64-bit kabylake
Clang: 15.0.0 build 1500
Git: 2.44.0 => /usr/local/bin/git
Curl: 8.4.0 => /usr/bin/curl
macOS: 14.3.1-x86_64
CLT: 15.3.0.0.1.1708646388
Xcode: N/A

Output of `brew doctor`

Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: Some installed kegs have no formulae!
This means they were either deleted or installed manually.
You should find replacements for the following formulae:
  python-click
  python-attrs
  nss
  python-toml
  unison

Warning: Some installed formulae are deprecated or disabled.
You should find replacements for the following formulae:
  [email protected]
  cowsay
  [email protected]

Warning: Unbrewed dylibs were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected dylibs:
  /usr/local/lib/libcrypto.3.dylib
  /usr/local/lib/libykcs11.2.5.1.dylib
  /usr/local/lib/libykpiv.2.5.1.dylib
  /usr/local/lib/libz.1.dylib

Warning: Unbrewed header files were found in /usr/local/include.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected header files:
  /usr/local/include/openssl/aes.h
  /usr/local/include/openssl/asn1.h
  /usr/local/include/openssl/asn1_mac.h
  /usr/local/include/openssl/asn1err.h
  /usr/local/include/openssl/asn1t.h
  /usr/local/include/openssl/async.h
  /usr/local/include/openssl/asyncerr.h
  /usr/local/include/openssl/bio.h
  /usr/local/include/openssl/bioerr.h
  /usr/local/include/openssl/blowfish.h
  /usr/local/include/openssl/bn.h
  /usr/local/include/openssl/bnerr.h
  /usr/local/include/openssl/buffer.h
  /usr/local/include/openssl/buffererr.h
  /usr/local/include/openssl/camellia.h
  /usr/local/include/openssl/cast.h
  /usr/local/include/openssl/cmac.h
  /usr/local/include/openssl/cmp.h
  /usr/local/include/openssl/cmp_util.h
  /usr/local/include/openssl/cmperr.h
  /usr/local/include/openssl/cms.h
  /usr/local/include/openssl/cmserr.h
  /usr/local/include/openssl/comp.h
  /usr/local/include/openssl/comperr.h
  /usr/local/include/openssl/conf.h
  /usr/local/include/openssl/conf_api.h
  /usr/local/include/openssl/conferr.h
  /usr/local/include/openssl/configuration.h
  /usr/local/include/openssl/conftypes.h
  /usr/local/include/openssl/core.h
  /usr/local/include/openssl/core_dispatch.h
  /usr/local/include/openssl/core_names.h
  /usr/local/include/openssl/core_object.h
  /usr/local/include/openssl/crmf.h
  /usr/local/include/openssl/crmferr.h
  /usr/local/include/openssl/crypto.h
  /usr/local/include/openssl/cryptoerr.h
  /usr/local/include/openssl/cryptoerr_legacy.h
  /usr/local/include/openssl/ct.h
  /usr/local/include/openssl/cterr.h
  /usr/local/include/openssl/decoder.h
  /usr/local/include/openssl/decodererr.h
  /usr/local/include/openssl/des.h
  /usr/local/include/openssl/dh.h
  /usr/local/include/openssl/dherr.h
  /usr/local/include/openssl/dsa.h
  /usr/local/include/openssl/dsaerr.h
  /usr/local/include/openssl/dtls1.h
  /usr/local/include/openssl/e_os2.h
  /usr/local/include/openssl/e_ostime.h
  /usr/local/include/openssl/ebcdic.h
  /usr/local/include/openssl/ec.h
  /usr/local/include/openssl/ecdh.h
  /usr/local/include/openssl/ecdsa.h
  /usr/local/include/openssl/ecerr.h
  /usr/local/include/openssl/encoder.h
  /usr/local/include/openssl/encodererr.h
  /usr/local/include/openssl/engine.h
  /usr/local/include/openssl/engineerr.h
  /usr/local/include/openssl/err.h
  /usr/local/include/openssl/ess.h
  /usr/local/include/openssl/esserr.h
  /usr/local/include/openssl/evp.h
  /usr/local/include/openssl/evperr.h
  /usr/local/include/openssl/fips_names.h
  /usr/local/include/openssl/fipskey.h
  /usr/local/include/openssl/hmac.h
  /usr/local/include/openssl/hpke.h
  /usr/local/include/openssl/http.h
  /usr/local/include/openssl/httperr.h
  /usr/local/include/openssl/idea.h
  /usr/local/include/openssl/kdf.h
  /usr/local/include/openssl/kdferr.h
  /usr/local/include/openssl/lhash.h
  /usr/local/include/openssl/macros.h
  /usr/local/include/openssl/md2.h
  /usr/local/include/openssl/md4.h
  /usr/local/include/openssl/md5.h
  /usr/local/include/openssl/mdc2.h
  /usr/local/include/openssl/modes.h
  /usr/local/include/openssl/obj_mac.h
  /usr/local/include/openssl/objects.h
  /usr/local/include/openssl/objectserr.h
  /usr/local/include/openssl/ocsp.h
  /usr/local/include/openssl/ocsperr.h
  /usr/local/include/openssl/opensslconf.h
  /usr/local/include/openssl/opensslv.h
  /usr/local/include/openssl/ossl_typ.h
  /usr/local/include/openssl/param_build.h
  /usr/local/include/openssl/params.h
  /usr/local/include/openssl/pem.h
  /usr/local/include/openssl/pem2.h
  /usr/local/include/openssl/pemerr.h
  /usr/local/include/openssl/pkcs12.h
  /usr/local/include/openssl/pkcs12err.h
  /usr/local/include/openssl/pkcs7.h
  /usr/local/include/openssl/pkcs7err.h
  /usr/local/include/openssl/prov_ssl.h
  /usr/local/include/openssl/proverr.h
  /usr/local/include/openssl/provider.h
  /usr/local/include/openssl/quic.h
  /usr/local/include/openssl/rand.h
  /usr/local/include/openssl/randerr.h
  /usr/local/include/openssl/rc2.h
  /usr/local/include/openssl/rc4.h
  /usr/local/include/openssl/rc5.h
  /usr/local/include/openssl/ripemd.h
  /usr/local/include/openssl/rsa.h
  /usr/local/include/openssl/rsaerr.h
  /usr/local/include/openssl/safestack.h
  /usr/local/include/openssl/seed.h
  /usr/local/include/openssl/self_test.h
  /usr/local/include/openssl/sha.h
  /usr/local/include/openssl/srp.h
  /usr/local/include/openssl/srtp.h
  /usr/local/include/openssl/ssl.h
  /usr/local/include/openssl/ssl2.h
  /usr/local/include/openssl/ssl3.h
  /usr/local/include/openssl/sslerr.h
  /usr/local/include/openssl/sslerr_legacy.h
  /usr/local/include/openssl/stack.h
  /usr/local/include/openssl/store.h
  /usr/local/include/openssl/storeerr.h
  /usr/local/include/openssl/symhacks.h
  /usr/local/include/openssl/thread.h
  /usr/local/include/openssl/tls1.h
  /usr/local/include/openssl/trace.h
  /usr/local/include/openssl/ts.h
  /usr/local/include/openssl/tserr.h
  /usr/local/include/openssl/txt_db.h
  /usr/local/include/openssl/types.h
  /usr/local/include/openssl/ui.h
  /usr/local/include/openssl/uierr.h
  /usr/local/include/openssl/whrlpool.h
  /usr/local/include/openssl/x509.h
  /usr/local/include/openssl/x509_vfy.h
  /usr/local/include/openssl/x509err.h
  /usr/local/include/openssl/x509v3.h
  /usr/local/include/openssl/x509v3err.h
  /usr/local/include/ykpiv/ykpiv-config.h
  /usr/local/include/ykpiv/ykpiv.h
  /usr/local/include/zlib.h

Warning: Unbrewed '.pc' files were found in /usr/local/lib/pkgconfig.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected '.pc' files:
  /usr/local/lib/pkgconfig/ykcs11.pc
  /usr/local/lib/pkgconfig/ykpiv.pc

Warning: Unbrewed static libraries were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected static libraries:
  /usr/local/lib/libykcs11.a
  /usr/local/lib/libykpiv.a

Warning: You have unlinked kegs in your Cellar.
Leaving kegs unlinked can lead to build-trouble and cause formulae that depend on
those kegs to fail to run properly once built. Run `brew link` on these:
  python-packaging
  openssl@3
  kubernetes-cli

Warning: You have uncommitted modifications to Homebrew/homebrew-core.
If this is a surprise to you, then you should stash these modifications.
Stashing returns Homebrew to a pristine state but can be undone
should you later need to do so for some reason.
  cd /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core && git stash -u && git clean -d -f

Uncommitted files:
   M Formula/g/git-machete.rb

Warning: Homebrew's "sbin" was not found in your PATH but you have installed
formulae that put executables in /usr/local/sbin.
Consider setting your PATH for example like so:
  echo 'export PATH="/usr/local/sbin:$PATH"' >> /Users/pawel_lipski/.bash_profile

Description of issue

I'm the maintainer of https://github.com/VirtusLab/git-machete, corresponding to git-machete formula. Each released version is first released to production PyPI (https://pypi.org/project/git-machete), and then our CI runs brew bump-formula-pr. This is working fine.

However, for develop builds (snapshot versions), CI releases a snapshot a to test PyPI (https://test.pypi.org/project/git-machete) and then it used to run brew bump-formula-pr --write-only to check whether the subsequent brew bump-formula-pr will work without an issue. This used to work... until March 9th, when I've first seen the error:

==> Retrieving PyPI dependencies for "git-machete==3.24.0"...
Error: Unable to determine dependencies for "git-machete==3.24.0" because of a failure when running
`/home/linuxbrew/.linuxbrew/opt/[email protected]/bin/python3 -m pip install -q --dry-run --ignore-installed --report=/dev/stdout git-machete==3.24.0`.
Please update the resources manually.

This wasn't the case with Homebrew itself being updated, as the CI used the pinned 4.1.21 version both before and after this error.

Has there been a recent change in Homebrew ecosystem that causes production PyPI to be checked, even if the url in formula refers to test PyPI?

To repro, check out VirtusLab/git-machete at develop, then run

brew tap homebrew/core
brew bump-formula-pr --write-only --verbose --url https://test.pypi.org/packages/source/g/git-machete/git-machete-3.24.3.tar.gz --sha256 61bedb38b135148681b60ed5cada2e54d7842bf64ffd1d9eebeee874e7d2ae1c git-machete

As of March 11th, git-machete v3.24.3 is released to test PyPI but not to production PyPI. Note the URL corresponds to test.pypi.org, not pypi.org.

This ends up in the same error as I've seen on CI previously:

/usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/3.1.0/bin/bundle clean
==> replace /https:\/\/files\.pythonhosted\.org\/packages\/39\/b5\/4bf952712817b4a6fb6fdfc04f294e143b2a0da56170d9486c1ff3d1dbeb\/git\-machete\-3\.24\.0\.tar\.gz/ with "https://test.pypi.org/packages/source/g/git-machete/git-machete-3.24.3.tar.gz"
==> replace "9f09fd3eea46cf4bbabdf4ff237d54e90cbfcb32439d2becd8a6d353c831dd8d" with "61bedb38b135148681b60ed5cada2e54d7842bf64ffd1d9eebeee874e7d2ae1c"
==> Downloading https://formulae.brew.sh/api/formula.jws.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
==> Retrieving PyPI dependencies for "git-machete==3.24.3"...
Error: Unable to determine dependencies for "git-machete==3.24.3" because of a failure when running
`/usr/local/opt/[email protected]/libexec/bin/python -m pip install -q --disable-pip-version-check --dry-run --ignore-installed --report=/dev/stdout git-machete==3.24.3`.
Please update the resources manually.

Answered by cho-m

Mar 11, 2024

Was this happening when we were using pythonhosted URL? i.e. Homebrew/homebrew-core@6237b76

In that case, brew bump-formula-pr will try to automatically figure out dependency tree. It looks like we look at url of the original formula to determine this constraint, not the input url.

Also, it looks like URL was changed back in Homebrew/homebrew-core@c8c31cd so I would assume original behavior would be restored.

View full answer

PawelLipski · 2024-03-11T12:43:25Z

PawelLipski
Mar 11, 2024
Author

For the record, an execution of brew bump-formula-pr --write-only from before March 9th that succeeded (link):

+ brew bump-formula-pr --write-only --force --no-browse --verbose --url https://test.pypi.org/packages/source/g/git-machete/git-machete-3.24.0.tar.gz --sha256 1421f96ba1296d6a6929ae99ae81924c0c0f131d4bf9811825e82457252000a2 git-machete
Warning: bump-formula-pr is a developer command, so Homebrew's
developer mode has been automatically turned on.
To turn developer mode off, run:
  brew developer off

==> Installing 'bundler' gem
Fetching bundler-2.4.18.gem
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/bin/bundle install
Fetching gem metadata from https://rubygems.org/.......
Fetching minitest 5.20.0
Fetching public_suffix 5.0.4
Fetching concurrent-ruby 1.2.2
Fetching zeitwerk 2.6.12
Installing zeitwerk 2.6.12
Installing public_suffix 5.0.4
Installing minitest 5.20.0
Installing concurrent-ruby 1.2.2
Fetching ast 2.4.2
Installing ast 2.4.2
Fetching bindata 2.4.15
Fetching did_you_mean 1.6.3
Fetching json 2.6.3
Installing did_you_mean 1.6.3
Installing bindata 2.4.15
Installing json 2.6.3 with native extensions
Fetching parallel 1.23.0
Installing parallel 1.23.0
Fetching racc 1.7.3
Installing racc 1.7.3 with native extensions
Fetching plist 3.7.0
Installing plist 3.7.0
Fetching rack 3.0.8
Installing rack 3.0.8
Fetching rainbow 3.1.1
Installing rainbow 3.1.1
Fetching regexp_parser 2.8.2
Fetching rexml 3.2.6
Installing regexp_parser 2.8.2
Installing rexml 3.2.6
Fetching ruby-progressbar 1.13.0
Fetching unicode-display_width 2.5.0
Installing unicode-display_width 2.5.0
Installing ruby-progressbar 1.13.0
Fetching ruby-macho 4.0.0
Fetching sorbet-runtime 0.5.10461
Installing ruby-macho 4.0.0
Installing sorbet-runtime 0.5.10461
Fetching warning 1.3.0
Installing warning 1.3.0
Fetching addressable 2.8.5
Installing addressable 2.8.5
Fetching elftools 1.2.0
Fetching i18n 1.14.1
Installing elftools 1.2.0
Installing i18n 1.14.1
Fetching tzinfo 2.0.6
Installing tzinfo 2.0.6
Fetching patchelf 1.4.0
Installing patchelf 1.4.0
Fetching activesupport 6.1.7.6
Installing activesupport 6.1.7.6
Fetching parser 3.2.2.4
Installing parser 3.2.2.4
Fetching rubocop-ast 1.30.0
Installing rubocop-ast 1.30.0
Fetching rubocop 1.50.2
Installing rubocop 1.50.2
Fetching rubocop-performance 1.17.1
Fetching rubocop-rails 2.19.1
Fetching rubocop-capybara 2.18.0
Fetching rubocop-sorbet 0.7.5
Installing rubocop-capybara 2.18.0
Installing rubocop-performance 1.17.1
Installing rubocop-sorbet 0.7.5
Installing rubocop-rails 2.19.1
Fetching rubocop-rspec 2.20.0
Installing rubocop-rspec 2.20.0
Bundle complete! 37 Gemfile dependencies, 35 gems now installed.
Bundled gems are installed into `../../linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle`
==> replace /https:\/\/pypi\.org\/packages\/source\/g\/git\-machete\/git\-machete\-3\.23\.2\.tar\.gz/ with "https://test.pypi.org/packages/source/g/git-machete/git-machete-3.24.0.tar.gz"
==> replace "2766b677bae7f2f7dc596ff6dcc7b6bcc06bc8e3c75a4ca8d826de5619cbc406" with "1421f96ba1296d6a6929ae99ae81924c0c0f131d4bf9811825e82457252000a2"
/home/linuxbrew/.linuxbrew/bin/brew audit --formula git-machete
==> try to fork repository with GitHub API
==> git add /home/linuxbrew/.linuxbrew/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/g/git-machete.rb
==> git checkout --no-track -b bump-git-machete-3.24.0 origin/master
==> git commit --no-edit --verbose --message='git-machete 3.24.0' -- /home/linuxbrew/.linuxbrew/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/g/git-machete.rb
==> git push --set-upstream FORK_URL bump-git-machete-3.24.0:bump-git-machete-3.24.0
==> git checkout --quiet -
==> create pull request with GitHub API (base branch: master)
+ export HOMEBREW_NO_INSTALL_FROM_API=1
+ HOMEBREW_NO_INSTALL_FROM_API=1
+ brew config
HOMEBREW_VERSION: 4.1.21
ORIGIN: https://github.com/Homebrew/brew
HEAD: ad1bb178d4a478840c4524063ac5cd52d8667d25
Last commit: 4 months ago
Core tap origin: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 8f3d5c308d9ca2e2db8b73b38b2ef7fd54fc3e0a
Core tap last commit: 58 minutes ago
Core tap branch: master
Core tap JSON: 04 Mar 06:43 UTC
HOMEBREW_PREFIX: /home/linuxbrew/.linuxbrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: :99
HOMEBREW_GITHUB_API_TOKEN: set
HOMEBREW_MAKE_JOBS: 4
HOMEBREW_NO_AUTO_UPDATE: set
HOMEBREW_NO_INSTALL_FROM_API: set
HOMEBREW_SORBET_RUNTIME: set
Homebrew Ruby: 2.6.10 => /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/bin/ruby
CPU: quad-core 64-bit icelake
Clang: N/A
Git: 2.38.1 => /bin/git
Curl: 7.81.0 => /bin/curl
Kernel: Linux 5.15.0-1022-aws x86_64 GNU/Linux
OS: Ubuntu 22.04.1 LTS (jammy)
Host glibc: 2.35
/usr/bin/gcc: 11.3.0
/usr/bin/ruby: N/A
glibc: N/A
gcc@11: N/A
gcc: N/A
xorg: N/A

0 replies

cho-m · 2024-03-11T21:29:59Z

cho-m
Mar 11, 2024

Was this happening when we were using pythonhosted URL? i.e. Homebrew/homebrew-core@6237b76

In that case, brew bump-formula-pr will try to automatically figure out dependency tree. It looks like we look at url of the original formula to determine this constraint, not the input url.

Also, it looks like URL was changed back in Homebrew/homebrew-core@c8c31cd so I would assume original behavior would be restored.

1 reply

PawelLipski Mar 12, 2024
Author

Indeed, it looks that when the URL in formula is of https://pypi.org/... form, then brew bump-formula-pr --write-only with test PyPI version works. Thanks!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Homebrew

For a formula that wraps a Python package, why did `brew bump-formula-pr` stop passing for a Test PyPI URL? #5193

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 1 reply

{{title}}

{{title}}

{{title}}

Select a reply

Homebrew

For a formula that wraps a Python package, why did brew bump-formula-pr stop passing for a Test PyPI URL? #5193

PawelLipski Mar 11, 2024

Output of brew config

Output of brew doctor

Description of issue

Replies: 2 comments · 1 reply

PawelLipski Mar 11, 2024 Author

cho-m Mar 11, 2024

PawelLipski Mar 12, 2024 Author

For a formula that wraps a Python package, why did `brew bump-formula-pr` stop passing for a Test PyPI URL? #5193

PawelLipski
Mar 11, 2024

Output of `brew config`

Output of `brew doctor`

Replies: 2 comments 1 reply

PawelLipski
Mar 11, 2024
Author

cho-m
Mar 11, 2024

PawelLipski Mar 12, 2024
Author