[TRIAGE] The bottle for $foo has an invalid build provenance attestation #177384

noelleleigh · 2024-07-15T14:08:52Z

`brew gist-logs <formula>` link OR `brew config` AND `brew doctor` output

brew config

HOMEBREW_VERSION: 4.3.9-227-gd4f808f
ORIGIN: https://github.com/Homebrew/brew
HEAD: d4f808ffb56cf47d8ee96b973bb15afd48cf8ba2
Last commit: 14 minutes ago
Core tap JSON: 15 Jul 13:58 UTC
Core cask tap JSON: 15 Jul 13:58 UTC
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_EDITOR: code
HOMEBREW_MAKE_JOBS: 16
HOMEBREW_SORBET_RUNTIME: set
Homebrew Ruby: 3.3.4 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.3.4/bin/ruby
CPU: 16-core 64-bit kabylake
Clang: 15.0.0 build 1500
Git: 2.45.2 => /usr/local/bin/git
Curl: 8.6.0 => /usr/bin/curl
macOS: 14.5-x86_64
CLT: 15.3.0.0.1.1708646388
Xcode: N/A

brew doctor

Your system is ready to brew.

Verification

My brew doctor output says Your system is ready to brew. and am still able to reproduce my issue.
I ran brew update and am still able to reproduce my issue.
I have resolved all warnings from brew doctor and that did not fix my problem.
I searched for recent similar issues at https://github.com/Homebrew/homebrew-core/issues?q=is%3Aissue and found no duplicates.

What were you trying to do (and why)?

Upgrade nano from 8.0 to 8.1

What happened (include all command output)?

> % brew upgrade nano  
==> Downloading https://formulae.brew.sh/api/formula.jws.json

==> Downloading https://formulae.brew.sh/api/cask.jws.json

Warning: Treating nano as a formula. For the cask, use homebrew/cask/nano or specify the `--cask` flag.
==> Upgrading 1 outdated package:
nano 8.0 -> 8.1
==> Downloading https://ghcr.io/v2/homebrew/core/nano/manifests/8.1
Already downloaded: /Users/noelle/Library/Caches/Homebrew/downloads/c235906dbe78ab367ac80fa098363c4312bc50c3945eb8154535b340d8394f94--nano-8.1.bottle_manifest.json
==> Fetching nano
==> Downloading https://ghcr.io/v2/homebrew/core/nano/blobs/sha256:78947cd54c0938695fd01dd784f3f0033c0af0532627
Already downloaded: /Users/noelle/Library/Caches/Homebrew/downloads/ac5e4929fdb43b3e506e55e05d011b7f77c9148f1abc5f20f9927bd46f1ce721--nano--8.1.sonoma.bottle.tar.gz
==> Upgrading nano
  8.0 -> 8.1 
==> Verifying attestation for nano
Error: The bottle for nano has an invalid build provenance attestation.

This may indicate that the bottle was not produced by the expected
tap, or was maliciously inserted into the expected tap's bottle
storage.

Additional context:

attestation verification failed: Failure while executing; `/usr/bin/env GH_TOKEN=****** /usr/local/bin/gh attestation verify /Users/noelle/Library/Caches/Homebrew/downloads/ac5e4929fdb43b3e506e55e05d011b7f77c9148f1abc5f20f9927bd46f1ce721--nano--8.1.sonoma.bottle.tar.gz --repo trailofbits/homebrew-brew-verify --format json` exited with 1. Here's the output:

Error: failed to fetch attestations from trailofbits/homebrew-brew-verify: HTTP 401: Bad credentials (https://api.github.com/repos/trailofbits/homebrew-brew-verify/attestations/sha256:78947cd54c0938695fd01dd784f3f0033c0af053262712e2d34bef6cd7653513?per_page=30)

What did you expect to happen?

Install without error

Step-by-step reproduction instructions (by running `brew` commands)

brew upgrade nano

The text was updated successfully, but these errors were encountered:

woodruffw · 2024-07-15T15:05:08Z

Thanks for the report @noelleleigh! Could you attempt to run the following for me locally?

gh auth status

...that should help me determine why the GitHub API call failed to authenticate here.

woodruffw · 2024-07-15T15:06:13Z

In the mean time as well: you can disable this feature (it's in beta, but you're receiving it because you have developer mode enabled) by setting HOMEBREW_NO_VERIFY_ATTESTATIONS=1.

mitchblank · 2024-07-15T15:27:47Z

I had the same issue, and a gh auth login fixed it.

It seems that the attestation-checking code is sensitive to a stale gh token in a way that other parts of brew upgrade are not.

Also the way the error surfaces isn't very helpful

woodruffw · 2024-07-15T15:32:28Z

Also the way the error surfaces isn't very helpful

Agreed; we rolled this out to people with developer mode enabled to discover exactly these kinds of rough edges. I'll look into improving this error message.

lblackstone · 2024-07-15T16:27:06Z

I encountered a similar error for the goreleaser package:

Error: failed to fetch attestations from trailofbits/homebrew-brew-verify: HTTP 401: Bad credentials (https://api.github.com/repos/trailofbits/homebrew-brew-verify/attestations/sha256:2a5e4f6a0352af1989c5516ad2b742e132bf5112e2574190b6b706f7b1697f35?per_page=30)

Running gh auth login did not fix it, but HOMEBREW_NO_VERIFY_ATTESTATIONS=1 worked as suggested.

I'm using the SSH git protocol if that makes a difference.

woodruffw · 2024-07-15T16:29:14Z

Using SSH for git shouldn't make a difference 🙂

@lblackstone could you run gh auth status and post the output here?

lblackstone · 2024-07-15T16:30:04Z

$ gh auth status

github.com
  ✓ Logged in to github.com account lblackstone (/Users/levi/.config/gh/hosts.yml)
  - Active account: true
  - Git operations protocol: ssh
  - Token: gho_************************************
  - Token scopes: 'admin:public_key', 'gist', 'read:org', 'repo'

woodruffw · 2024-07-15T16:31:55Z

Thanks. That looks pretty close to what I have, so I don't think that's the source of problems here. It's possible that there's another (stale) credential elsewhere that Homebrew is giving priority to; I'll look into that.

Edit: @lblackstone do you happen to have a different API credential configured via HOMEBREW_GITHUB_API_TOKEN? That's the one credential source that has priority over the gh-held token.

lblackstone · 2024-07-15T16:38:58Z

do you happen to have a different API credential configured via HOMEBREW_GITHUB_API_TOKEN? That's the one credential source that has priority over the gh-held token.

Ah, sure enough. It looks like my ~/.zshrc file was setting that, but I don't see the token on my GH settings. I assume it expired since I used it awhile back.

woodruffw · 2024-07-15T16:41:50Z

Yep, that'll do it. I'll look into improving the error message on that case as well. In the mean time, you should be able to re-enable attestations and delete that old env var (Homebrew will use your gh token).

noelleleigh · 2024-07-15T18:49:49Z

Thanks for the report @noelleleigh! Could you attempt to run the following for me locally?
gh auth status
...that should help me determine why the GitHub API call failed to authenticate here.

Sorry for the delay:

$ gh auth status
github.com
  X Failed to log in to github.com account noelleleigh (keyring)
  - Active account: true
  - The token in keyring is invalid.
  - To re-authenticate, run: gh auth login -h github.com
  - To forget about this account, run: gh auth logout -h github.com -u noelleleigh

woodruffw · 2024-07-15T19:09:22Z

No problem, thanks for checking. Could you try running gh auth login and seeing if the command works after that?

(This feature won't require this kind of auth flow once it's out of beta; you can leave the beta either by disabling Homebrew's developer mode or by explicitly setting HOMEBREW_NO_VERIFY_ATTESTATIONS=1.)

woodruffw · 2024-07-15T20:29:29Z

Hi, I'm trying to install otf2bdf and I'm getting a similar error, but a 404 instead of a 401. Is this related?

That's unrelated, but thank you for raising it. You can use the same HOMEBREW_NO_VERIFY_ATTESTATIONS=1 env var mentioned above to disable that failing check.

Edit: I've kicked off a rebottle for otf2bdf that should resolve the above.

Edit 2: The rebottle has completed and otf2bdf's bottles should now be attested. Thank you @jackson-57!

noelleleigh · 2024-07-15T20:35:41Z

No problem, thanks for checking. Could you try running gh auth login and seeing if the command works after that?

After completing the gh auth login flow, I can now upgrade nano successfully. Thanks!

angelus2014 · 2024-07-16T18:31:58Z

I use set HOMEBREW_NO_VERIFY_ATTESTATIONS 1 in fish and a check with env shows me that it's set.

woodruffw · 2024-09-25T17:30:48Z

Thanks @rlucas7. Could you please provide the logging information that you elided here? I need that information to triage this.

Additional context:
... output elided...

Edit: for context, I'm unable to reproduce this with the ventura tag for wget:

$ brew verify wget --bottle-tag=ventura
==> Downloading https://ghcr.io/v2/homebrew/core/wget/blobs/sha256:1b7e2f76c90553543a5e25dadf031c6fcfe280f52bf27d89e04006f9d33fd20b
###################################################################################################################################################### 100.0%
==> wget--1.24.5.ventura.bottle.tar.gz has a valid attestation

rlucas7 · 2024-09-25T17:55:47Z

Thanks @rlucas7. Could you please provide the logging information that you elided here? I need that information to triage this.
Additional context:
... output elided...
Edit: for context, I'm unable to reproduce this with the ventura tag for wget:
$ brew verify wget --bottle-tag=ventura
==> Downloading https://ghcr.io/v2/homebrew/core/wget/blobs/sha256:1b7e2f76c90553543a5e25dadf031c6fcfe280f52bf27d89e04006f9d33fd20b
###################################################################################################################################################### 100.0%
==> wget--1.24.5.ventura.bottle.tar.gz has a valid attestation

yep, apologies for omitting here it is:

Additional context:

attestation verification failed: Failure while executing; `/usr/bin/env GH_TOKEN=****** GH_HOST=github.com /usr/local/bin/gh attestation verify /Users/rlucas/Library/Caches/Homebrew/downloads/a69f6815965ac498390ce6a33fa2b0f3f67a970097aa33e329f401a79698e073--wget--1.24.5.ventura.bottle.tar.gz --repo Homebrew/homebrew-core --format json` exited with 1. Here's the output:
unknown command "attestation" for "gh"

Usage:  gh <command> <subcommand> [flags]

Available commands:
  alias
  api
  auth
  browse
  cache
  co
  codespace
  completion
  config
  extension
  gist
  gpg-key
  issue
  label
  org
  pr
  project
  release
  repo
  ruleset
  run
  search
  secret
  ssh-key
  status
  variable
  workflow

In case you want/need this I'm using

Lucass-MacBook:data rlucas$ gh --version
gh version 2.42.1 (2024-01-15)
https://github.com/cli/cli/releases/tag/v2.42.1
Lucass-MacBook:data rlucas$

Also FWIW I'm on ventura 13.6.7

woodruffw · 2024-09-25T17:57:52Z

Thanks, that's helpful! That looks like a case of your gh being too old; we used to detect that, but doing so reliably is somewhat challenging. Could you try brew update && brew upgrade gh and see if that resolves the problem?

rlucas7 · 2024-09-25T21:58:52Z

brew upgrade gh

Yes. I already had installed via the HOMEBREW_NO_VERIFY_ATTESTATIONS=1 -mentioned in comment so I uninstalled and confirmed I could reproduce the error on install of wget. Then updating the gh cli tool and reinstall worked fine. The brew upgrade gh bumped the version from
2.42.1 -> 2.57.0 to fix this issue.

Thanks @woodruffw 👍

bash stuff for posterity

Lucass-MacBook:1 rlucas$ brew uninstall wget Uninstalling /usr/local/Cellar/wget/1.24.5... (92 files, 4.5MB)

Warning: The following may be wget configuration files and have not been removed!
If desired, remove them manually with rm -rf:
/usr/local/etc/wgetrc
Lucass-MacBook:1 rlucas$ which wget
Lucass-MacBook:1 rlucas$ brew install wget
==> Auto-updating Homebrew...
Adjust how often this is run with HOMEBREW_AUTO_UPDATE_SECS or disable with
HOMEBREW_NO_AUTO_UPDATE. Hide these hints with HOMEBREW_NO_ENV_HINTS (see man brew).
==> Auto-updated Homebrew!
==> Updated Homebrew from 2f2ca82035 to 465a7352f3.
No changes to formulae or casks.

==> Downloading https://ghcr.io/v2/homebrew/core/wget/manifests/1.24.5
Already downloaded: /Users/rlucas/Library/Caches/Homebrew/downloads/885bce6d1c81de373d8479681e3ccbb748be9ec a29a8a3b3c95e6479a98458bd--wget-1.24.5.bottle_manifest.json
==> Fetching wget
==> Downloading https://ghcr.io/v2/homebrew/core/wget/blobs/ sha256:1b7e2f76c90553543a5e25dadf031c6fcfe280f52bf27d89e04006f9d33fd20b
Already downloaded: /Users/rlucas/Library/Caches/Homebrew/downloads/ a69f6815965ac498390ce6a33fa2b0f3f67a970097aa33e329f401a79698e073--wget--1.24.5.ventura.bottle.tar.gz
==> Verifying attestation for wget
Error: The bottle for wget has an invalid build provenance attestation.

This may indicate that the bottle was not produced by the expected
tap, or was maliciously inserted into the expected tap's bottle
storage.

Additional context:

attestation verification failed: Failure while executing; /usr/bin/env GH_TOKEN=****** GH_HOST=github.com /usr/local/bin/gh attestation verify /Users/rlucas/Library/Caches/Homebrew/downloads/ a69f6815965ac498390ce6a33fa2b0f3f67a970097aa33e329f401a79698e073--wget--1.24.5.ventura.bottle.tar.gz --repo Homebrew/homebrew-core --format json exited with 1. Here's the output:
unknown command "attestation" for "gh"

Usage: gh [flags]

Available commands:
alias
api
auth
browse
cache
co
codespace
completion
config
extension
gist
gpg-key
issue
label
org
pr
project
release
repo
ruleset
run
search
secret
ssh-key
status
variable
workflow

Lucass-MacBook:1 rlucas$ brew upgrade gh
==> Upgrading 1 outdated package:
gh 2.42.1 -> 2.57.0
==> Downloading https://ghcr.io/v2/homebrew/core/gh/manifests/2.57.0
########################################################################################################### ############################ 100.0%
==> Fetching gh
==> Downloading https://ghcr.io/v2/homebrew/core/gh/blobs/ sha256:ee67b4ecaf27b815f29ae80c82388a4fb43254ba2c9ca840795fe05404e04814
########################################################################################################### ############################ 100.0%
==> Upgrading gh
2.42.1 -> 2.57.0
==> Pouring gh--2.57.0.ventura.bottle.tar.gz
==> Caveats
Bash completion has been installed to:
/usr/local/etc/bash_completion.d
==> Summary
🍺 /usr/local/Cellar/gh/2.57.0: 201 files, 40.5MB
==> Running brew cleanup gh...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see man brew).
Removing: /usr/local/Cellar/gh/2.42.1... (192 files, 43.6MB)
Lucass-MacBook:1 rlucas$ brew install wget
==> Downloading https://ghcr.io/v2/homebrew/core/wget/manifests/1.24.5
Already downloaded: /Users/rlucas/Library/Caches/Homebrew/downloads/885bce6d1c81de373d8479681e3ccbb748be9ec a29a8a3b3c95e6479a98458bd--wget-1.24.5.bottle_manifest.json
==> Fetching wget
==> Downloading https://ghcr.io/v2/homebrew/core/wget/blobs/ sha256:1b7e2f76c90553543a5e25dadf031c6fcfe280f52bf27d89e04006f9d33fd20b
Already downloaded: /Users/rlucas/Library/Caches/Homebrew/downloads/ a69f6815965ac498390ce6a33fa2b0f3f67a970097aa33e329f401a79698e073--wget--1.24.5.ventura.bottle.tar.gz
==> Verifying attestation for wget
==> Pouring wget--1.24.5.ventura.bottle.tar.gz
🍺 /usr/local/Cellar/wget/1.24.5: 92 files, 4.5MB
==> Running brew cleanup wget...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see man brew).
Lucass-MacBook:1 rlucas$

101Dude · 2024-10-10T18:36:50Z

Have the same issue with wakonlan bottle.

I added notes to the closed issue above

I was able to install by using export HOMEBREW_NO_VERIFY_ATTESTATIONS=1 in my session and then brew install wakeonlan

hashhar · 2024-12-05T18:37:54Z

[ashhar.hasan:~] 14s 130 % gh auth status
github.com
  ✓ Logged in to github.com account hashhar (keyring)
  - Active account: true
  - Git operations protocol: https
  - Token: gho_************************************
  - Token scopes: 'gist', 'read:org', 'repo', 'workflow'

[ashhar.hasan:~] 6s % brew upgrade
==> Upgrading 2 outdated packages:
awscli 2.22.10 -> 2.22.12
[email protected] 3.11.10 -> 3.11.11
==> Downloading https://ghcr.io/v2/homebrew/core/awscli/manifests/2.22.12
####################################################################################################################################################################################### 100.0%
==> Fetching awscli
==> Downloading https://ghcr.io/v2/homebrew/core/awscli/blobs/sha256:76814f5d8b8959a9744c2dd02314794a2cfc0cd95091ac1c8c720dfb0df6b3c1
####################################################################################################################################################################################### 100.0%
==> Verifying attestation for awscli
Warning: Failed to verify attestation. Retrying in 1s...
Warning: Failed to verify attestation. Retrying in 3s...
Warning: Failed to verify attestation. Retrying in 9s...
Warning: Failed to verify attestation. Retrying in 27s...
Warning: Failed to verify attestation. Retrying in 81s...
Error: The bottle for awscli has an invalid build provenance attestation.

This may indicate that the bottle was not produced by the expected
tap, or was maliciously inserted into the expected tap's bottle
storage.

Additional context:

no attestation matches subject: awscli--2.22.12.arm64_sonoma.bottle.tar.gz

Nothing other than disabling attestation fixes this for me. Is there something wrong with these specific packages? GH APIs also seem fully operational.

woodruffw · 2024-12-05T18:42:19Z

Thanks for the report @hashhar -- it looks like awscli--2.22.12.arm64_sequoia.bottle.tar.gz has an attestation but arm64_sonoma doesn't for some reason. I'm looking into it now.

Edit: Bump here: f38c833

hashhar · 2024-12-05T18:43:15Z

Deleting the manifest locally fixed it for me somehow - rm /Users/ashhar.hasan/Library/Caches/Homebrew/downloads/8a8898a42ed57965cbfcfbbb6071d245678db8887c6a6eea89e241eed49e08cf--python@3.11-3.11.11.bottle_manifest.json and the same for awscli.

woodruffw · 2024-12-05T18:46:19Z

Deleting the manifest locally fixed it for me somehow - rm /Users/ashhar.hasan/Library/Caches/Homebrew/downloads/8a8898a42ed57965cbfcfbbb6071d245678db8887c6a6eea89e241eed49e08cf--python@3.11-3.11.11.bottle_manifest.json and the same for awscli.

Hmm, that's interesting -- I can't reproduce that fix. Can you confirm that it's still emitting a "has a valid attestation" message on installation?

mitchblank · 2024-12-05T18:47:15Z

I've had nothing but those "Failed to verify attestation. Retrying in ..." errors all day today on several packages. Something definitely broke in a new way.

Right now I'm seeing failures (on x68_64) vttest, molecule, [email protected], etc...

hashhar · 2024-12-05T18:47:33Z

let me recheck by uninstalling + installing.

woodruffw · 2024-12-05T18:48:16Z

Right now I'm seeing failures (on x68_64) vttest, molecule, [email protected], etc...

Can you please paste the exact output you're seeing? That will help me pin down the exact tags/bottles that aren't working for you.

woodruffw · 2024-12-05T18:51:09Z

I think I've identified a likely break point here: Homebrew began generating multi-subject attestations yesterday. I'm looking into this now.

hashhar · 2024-12-05T18:52:05Z

yep, no longer works.

[ashhar.hasan:~] 2s % brew uninstall awscli
Uninstalling /opt/homebrew/Cellar/awscli/2.22.12... (14,103 files, 128.7MB)
==> Autoremoving 3 unneeded formulae:
cffi
cryptography
pycparser
Uninstalling /opt/homebrew/Cellar/cryptography/44.0.0... (262 files, 9MB)
Uninstalling /opt/homebrew/Cellar/cffi/1.17.1_1... (96 files, 1.7MB)
Uninstalling /opt/homebrew/Cellar/pycparser/2.22_1... (98 files, 1.8MB)
[ashhar.hasan:~] 3s % brew cleanup --prune=all -s
.........................................
Pruned 0 symbolic links and 6 directories from /opt/homebrew
==> This operation has freed approximately 35.0MB of disk space.
[ashhar.hasan:~] 5s % rm -rf "$(brew --cache)"
[ashhar.hasan:~] % brew install awscli
==> Downloading https://formulae.brew.sh/api/formula.jws.json
####################################################################################################################################################################################### 100.0%
==> Downloading https://formulae.brew.sh/api/formula_tap_migrations.jws.json
####################################################################################################################################################################################### 100.0%
==> Downloading https://formulae.brew.sh/api/cask.jws.json
####################################################################################################################################################################################### 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/awscli/manifests/2.22.12
####################################################################################################################################################################################### 100.0%
==> Fetching dependencies for awscli: pycparser, cffi and cryptography
==> Downloading https://ghcr.io/v2/homebrew/core/pycparser/manifests/2.22_1
####################################################################################################################################################################################### 100.0%
==> Fetching pycparser
==> Downloading https://ghcr.io/v2/homebrew/core/pycparser/blobs/sha256:96eddd22a812be4f919562d6525ab8fa19b03b700426d2fbf892da0f4fd02e73
####################################################################################################################################################################################### 100.0%
==> Verifying attestation for pycparser
==> Downloading https://ghcr.io/v2/homebrew/core/cffi/manifests/1.17.1_1
####################################################################################################################################################################################### 100.0%
==> Fetching cffi
==> Downloading https://ghcr.io/v2/homebrew/core/cffi/blobs/sha256:7818f620936fb017c68eb02c8985dfecd297349b97e67550d4915cef440dd2fe
####################################################################################################################################################################################### 100.0%
==> Verifying attestation for cffi
==> Downloading https://ghcr.io/v2/homebrew/core/cryptography/manifests/44.0.0
####################################################################################################################################################################################### 100.0%
==> Fetching cryptography
==> Downloading https://ghcr.io/v2/homebrew/core/cryptography/blobs/sha256:e79a38ec32ea9db4d0c3174a987446a2d48e50dece37717fb7f8347cc5bc6691
####################################################################################################################################################################################### 100.0%
==> Verifying attestation for cryptography
==> Fetching awscli
==> Downloading https://ghcr.io/v2/homebrew/core/awscli/blobs/sha256:76814f5d8b8959a9744c2dd02314794a2cfc0cd95091ac1c8c720dfb0df6b3c1
####################################################################################################################################################################################### 100.0%
==> Verifying attestation for awscli
Warning: Failed to verify attestation. Retrying in 1s...
Warning: Failed to verify attestation. Retrying in 3s...
Warning: Failed to verify attestation. Retrying in 9s...
Warning: Failed to verify attestation. Retrying in 27s...
Warning: Failed to verify attestation. Retrying in 81s...
Error: The bottle for awscli has an invalid build provenance attestation.

This may indicate that the bottle was not produced by the expected
tap, or was maliciously inserted into the expected tap's bottle
storage.

Additional context:

no attestation matches subject: awscli--2.22.12.arm64_sonoma.bottle.tar.gz

mitchblank · 2024-12-05T18:52:27Z

Can you please paste the exact output you're seeing?

==> Fetching vttest
==> Downloading https://ghcr.io/v2/homebrew/core/vttest/blobs/sha256:da1330ee7b6c0d892a5639f26cd1f9045c975596e50f9156458c6fbef937
########################################################################################################################## 100.0%
==> Verifying attestation for vttest
Warning: Failed to verify attestation. Retrying in 1s...
Warning: Failed to verify attestation. Retrying in 3s...
Warning: Failed to verify attestation. Retrying in 9s...
Warning: Failed to verify attestation. Retrying in 27s...
Warning: Failed to verify attestation. Retrying in 81s...
Error: The bottle for vttest has an invalid build provenance attestation.

This may indicate that the bottle was not produced by the expected
tap, or was maliciously inserted into the expected tap's bottle
storage.

Additional context:

no attestation matches subject: vttest--20241204.sonoma.bottle.tar.gz

[...]

==> Fetching molecule
==> Downloading https://ghcr.io/v2/homebrew/core/molecule/blobs/sha256:4f26feff60895f759951874fb7b8bc7423348aa80f538b0bb8a6370d34
########################################################################################################################## 100.0%
==> Verifying attestation for molecule
Warning: Failed to verify attestation. Retrying in 1s...
Warning: Failed to verify attestation. Retrying in 3s...
Warning: Failed to verify attestation. Retrying in 9s...
Warning: Failed to verify attestation. Retrying in 27s...
Warning: Failed to verify attestation. Retrying in 81s...
Error: The bottle for molecule has an invalid build provenance attestation.

This may indicate that the bottle was not produced by the expected
tap, or was maliciously inserted into the expected tap's bottle
storage.

Additional context:

no attestation matches subject: molecule--24.12.0.sonoma.bottle.tar.gz

[...]

==> Downloading https://ghcr.io/v2/homebrew/core/python/3.11/blobs/sha256:f6d08d2fd8405af14cf6b1c1f9f86cbcb24837920482124dcf25bc7
########################################################################################################################## 100.0%
==> Verifying attestation for [email protected]
Warning: Failed to verify attestation. Retrying in 1s...
Warning: Failed to verify attestation. Retrying in 3s...
Warning: Failed to verify attestation. Retrying in 9s...
Warning: Failed to verify attestation. Retrying in 27s...
Warning: Failed to verify attestation. Retrying in 81s...
Error: The bottle for [email protected] has an invalid build provenance attestation.

This may indicate that the bottle was not produced by the expected
tap, or was maliciously inserted into the expected tap's bottle
storage.

Additional context:

no attestation matches subject: [email protected]

[...]

==> Downloading https://ghcr.io/v2/homebrew/core/python/3.9/blobs/sha256:ed8adf7e7e85490d7cf057aafaf293dddf830c421b656d4fd0e7fab6
########################################################################################################################## 100.0%
==> Verifying attestation for [email protected]
Warning: Failed to verify attestation. Retrying in 1s...
Warning: Failed to verify attestation. Retrying in 3s...
Warning: Failed to verify attestation. Retrying in 9s...
Warning: Failed to verify attestation. Retrying in 27s...
Warning: Failed to verify attestation. Retrying in 81s...
Error: The bottle for [email protected] has an invalid build provenance attestation.

This may indicate that the bottle was not produced by the expected
tap, or was maliciously inserted into the expected tap's bottle
storage.

Additional context:

no attestation matches subject: [email protected]

A few packages that were failing a few hours ago worked this time (eza, ansible, [email protected] worked on this run) Not sure if that is because people are manually fixing something or if it's an issue where attestations just take awhile to sync up for new updates?

woodruffw · 2024-12-05T18:59:28Z

Yeah, I think I can see what happened here: when we switched to multi-subject attestations, we didn't sufficiently generalize the subject handling to allow any subject to match. As a result, verification will only accept whichever subject happens to be first, which is typically arm64_sequoia but probably variant.

I'm working on a MVR and fix now. In the meantime, you can disable developer mode or attestations to work around this, if you'd like.

This should fix the behavior observed in Homebrew/homebrew-core#177384 (comment) and below. Signed-off-by: William Woodruff <[email protected]>

woodruffw · 2024-12-05T19:14:22Z

Homebrew/brew#18883 should address the above, once merged.

hashhar · 2024-12-05T19:15:37Z

Thanks for the super quick triage and fix @woodruffw.

woodruffw · 2024-12-05T19:17:46Z

No problem, thank you and @mitchblank for your reports -- it's only with that kind of detail that I can triage effectively 🙂

woodruffw · 2024-12-05T19:39:33Z

The fix has landed on master, and I now see verifications succeeding for awscli on that bottle tag:

==> Downloading https://ghcr.io/v2/homebrew/core/awscli/blobs/sha256:76814f5d8b8959a9744c2dd02314794a2cfc0cd95091ac1c8c720dfb0df6b3c1
Already downloaded: /Users/william/Library/Caches/Homebrew/downloads/78517e2e72a07be96b5b8cd6a9655bcbdc7f2ad9eecf7b31026861d8bc7bcc8b--awscli--2.22.12.arm64_sonoma.bottle.tar.gz
==> awscli--2.22.12.arm64_sonoma.bottle.tar.gz has a valid attestation

Edit: Similar for the x86-64 bottles that @mitchblank reported:

==> Downloading https://ghcr.io/v2/homebrew/core/python/3.11/blobs/sha256:f6d08d2fd8405af14cf6b1c1f9f86cbcb24837920482124dcf25bc720f440649
Already downloaded: /Users/william/Library/Caches/Homebrew/downloads/187717ee812cad7e33df527ba810711ffccef119a6fccefa7699eb3b29ed3724--python@3.11--3.11.11.sonoma.bottle.tar.gz
==> [email protected] has a valid attestation

woodruffw · 2024-12-05T20:31:42Z

@hashhar @mitchblank could one or both of you confirm that the fix on master works for you? We can prep a point fix/tag once we know it's working for other people as well.

mitchblank · 2024-12-05T20:37:48Z

No change... or is an auto-update not enough to get this fix?

$ brew upgrade
==> Auto-updating Homebrew...
Adjust how often this is run with HOMEBREW_AUTO_UPDATE_SECS or disable with
HOMEBREW_NO_AUTO_UPDATE. Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
==> Auto-updated Homebrew!

[...]

==> Fetching vttest
==> Downloading https://ghcr.io/v2/homebrew/core/vttest/blobs/sha256:da1330ee7b6c0d892a5639f26cd1f9045c975596e50f9156458c6fbef937
########################################################################################################################## 100.0%
==> Verifying attestation for vttest
Warning: Failed to verify attestation. Retrying in 1s...
Warning: Failed to verify attestation. Retrying in 3s...
Warning: Failed to verify attestation. Retrying in 9s...
Warning: Failed to verify attestation. Retrying in 27s...
Warning: Failed to verify attestation. Retrying in 81s...
Error: The bottle for vttest has an invalid build provenance attestation.

This may indicate that the bottle was not produced by the expected
tap, or was maliciously inserted into the expected tap's bottle
storage.

Additional context:

no attestation matches subject: vttest--20241204.sonoma.bottle.tar.gz

woodruffw · 2024-12-05T20:40:35Z

No change... or is an auto-update not enough to get this fix?

Do you have developer mode enabled? brew developer on should make you switch to master (and you can brew developer off afterwards).

mitchblank · 2024-12-05T21:54:10Z

Sorry, was away for a bit there.

Do you have developer mode enabled?

Yes, I did have it enabled already.

I just did another brew upgrade though and all of the attestation problems went away. Not sure why it was still giving me errors an hour ago but not now. Maybe the previous auto-update didn't actually bring me up to HEAD?

woodruffw · 2024-12-05T22:33:52Z

I just did another brew upgrade though and all of the attestation problems went away. Not sure why it was still giving me errors an hour ago but not now. Maybe the previous auto-update didn't actually bring me up to HEAD?

Ah yeah, that's possible. We just did a point release of brew as well since we got independent confirmation of the fix as well, so you may have received it via that.

noelleleigh added the bug Reproducible Homebrew/homebrew-core bug label Jul 15, 2024

Bo98 assigned Bo98 and woodruffw and unassigned Bo98 Jul 15, 2024

This comment was marked as resolved.

Sign in to view

This comment was marked as duplicate.

Sign in to view

This comment was marked as resolved.

Sign in to view

This comment was marked as duplicate.

Sign in to view

This comment was marked as resolved.

Sign in to view

carlocab mentioned this issue Oct 10, 2024

Error: The bottle for wakeonlan has an invalid build provenance attestation. #191180

Closed

SMillerDev mentioned this issue Nov 22, 2024

brew install swift-format #198632

Closed

SMillerDev mentioned this issue Dec 5, 2024

Cannot verify ansible 11.1.0 attestation #200152

Closed

4 tasks

woodruffw added a commit to Homebrew/brew that referenced this issue Dec 5, 2024

attestation: handle multiple subjects

e5d47f8

This should fix the behavior observed in Homebrew/homebrew-core#177384 (comment) and below. Signed-off-by: William Woodruff <[email protected]>

woodruffw mentioned this issue Dec 5, 2024

attestation: handle multiple subjects Homebrew/brew#18883

Merged

7 tasks

[TRIAGE] The bottle for $foo has an invalid build provenance attestation #177384

[TRIAGE] The bottle for $foo has an invalid build provenance attestation #177384

Comments

noelleleigh commented Jul 15, 2024 • edited Loading

brew gist-logs <formula> link OR brew config AND brew doctor output

Verification

What were you trying to do (and why)?

What happened (include all command output)?

What did you expect to happen?

Step-by-step reproduction instructions (by running brew commands)

woodruffw commented Jul 15, 2024

woodruffw commented Jul 15, 2024

mitchblank commented Jul 15, 2024

woodruffw commented Jul 15, 2024

lblackstone commented Jul 15, 2024 • edited Loading

woodruffw commented Jul 15, 2024

lblackstone commented Jul 15, 2024

woodruffw commented Jul 15, 2024 • edited Loading

lblackstone commented Jul 15, 2024 • edited Loading

woodruffw commented Jul 15, 2024 • edited Loading

noelleleigh commented Jul 15, 2024

woodruffw commented Jul 15, 2024

This comment was marked as resolved.

woodruffw commented Jul 15, 2024 • edited Loading

noelleleigh commented Jul 15, 2024

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as duplicate.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as duplicate.

This comment was marked as duplicate.

This comment was marked as duplicate.

This comment was marked as duplicate.

angelus2014 commented Jul 16, 2024 • edited Loading

This comment was marked as resolved.

woodruffw commented Sep 25, 2024 • edited Loading

rlucas7 commented Sep 25, 2024

woodruffw commented Sep 25, 2024

rlucas7 commented Sep 25, 2024

101Dude commented Oct 10, 2024 • edited Loading

hashhar commented Dec 5, 2024 • edited Loading

woodruffw commented Dec 5, 2024 • edited Loading

hashhar commented Dec 5, 2024

woodruffw commented Dec 5, 2024

mitchblank commented Dec 5, 2024

hashhar commented Dec 5, 2024

woodruffw commented Dec 5, 2024

woodruffw commented Dec 5, 2024

hashhar commented Dec 5, 2024

mitchblank commented Dec 5, 2024

woodruffw commented Dec 5, 2024 • edited Loading

woodruffw commented Dec 5, 2024

hashhar commented Dec 5, 2024

woodruffw commented Dec 5, 2024

woodruffw commented Dec 5, 2024 • edited Loading

woodruffw commented Dec 5, 2024

mitchblank commented Dec 5, 2024

woodruffw commented Dec 5, 2024

mitchblank commented Dec 5, 2024

woodruffw commented Dec 5, 2024

noelleleigh commented Jul 15, 2024 •

edited

Loading

`brew gist-logs <formula>` link OR `brew config` AND `brew doctor` output

Step-by-step reproduction instructions (by running `brew` commands)

lblackstone commented Jul 15, 2024 •

edited

Loading

woodruffw commented Jul 15, 2024 •

edited

Loading

lblackstone commented Jul 15, 2024 •

edited

Loading

woodruffw commented Jul 15, 2024 •

edited

Loading

woodruffw commented Jul 15, 2024 •

edited

Loading

angelus2014 commented Jul 16, 2024 •

edited

Loading

woodruffw commented Sep 25, 2024 •

edited

Loading

101Dude commented Oct 10, 2024 •

edited

Loading

hashhar commented Dec 5, 2024 •

edited

Loading

woodruffw commented Dec 5, 2024 •

edited

Loading

woodruffw commented Dec 5, 2024 •

edited

Loading

woodruffw commented Dec 5, 2024 •

edited

Loading