APT/RPM PGP key for some repositories at pkgs.k8s.io expired at 2024-11-02

[kungfooflea]

Starting around 3:30am PST we noticed our automated pipelines are failing to build the docker image.

Errors below:

#6 1.692 Err:6 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease
#6 1.692 The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project isv:[email protected]
#6 1.704 Get:13 https://dl.k6.io/deb stable/main amd64 Packages [3556 B]
#6 2.267 Reading package lists...
#6 2.686 W: GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease: The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project isv:[email protected]
#6 2.686 E: The repository 'https://pkgs.k8s.io/core:/stable:/v1.26/deb InRelease' is not signed.
#6 ERROR: process "/bin/sh -c sudo gpg -k && sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69 && echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list && sudo apt update && sudo apt install -y k6 && sudo npm install --global yarn tslint" did not complete successfully: exit code: 100

[2/2] RUN sudo gpg -k && sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69 && echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list && sudo apt update && sudo apt install -y k6 && sudo npm install --global yarn tslint:
1.376 Get:10 https://deb.nodesource.com/node_16.x buster/main amd64 Packages [776 B]
1.410 Get:6 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease [1192 B]
1.533 Get:11 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [307 kB]
1.643 Get:12 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [18.8 kB]
1.692 Err:6 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease
1.692 The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project isv:[email protected]
1.704 Get:13 https://dl.k6.io/deb stable/main amd64 Packages [3556 B]

2.686 W: GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease: The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project isv:[email protected]
2.686 E: The repository 'https://pkgs.k8s.io/core:/stable:/v1.26/deb InRelease' is not signed.

debian:bullseye is the base

[petedyerarm]

I see the same issue with v1.27

[Creteil]

« apt dist-upgrade » under ubuntu 22.04 output same error related...

oem@oem-ThinkPad-P15-Gen-1:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.5 LTS
Release:	22.04
Codename:	jammy
oem@oem-ThinkPad-P15-Gen-1:~$

Err:16 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.31/deb  InRelease                                                                                                          
  The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project <isv:[email protected]>
Fetched 3917 B in 8s (491 B/s)                                                                                                                                                                                    
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
1 package can be upgraded. Run 'apt list --upgradable' to see it.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.31/deb  InRelease: The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project <isv:[email protected]>
W: Failed to fetch https://pkgs.k8s.io/core:/stable:/v1.31/deb/InRelease  The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project <isv:[email protected]>
W: Some index files failed to download. They have been ignored, or old ones used instead.

[xmudrii]

The original key has expired on 2024-11-02, however all repositories have been resigned with a new key. At the moment, the way forward is to download and install the new key from the latest Kubernetes repository (v1.31):

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

We'll try to figure out if we can handle this in a better way in the future.

Update 2024-11-12:

On the RPM-based systems, you need to modify the Kubernetes repository definition to point to the latest GPG key. Open /etc/yum.repos.d/kubernetes.repo in your preferred text editor, then make sure that the gpgkey URL is set to https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key (which is the latest Kubernetes repository at the moment).

As described in #3818 (comment), we at the moment recommend using the GPG key from the latest Kubernetes repository, as older Kubernetes repositories might not have the latest key. This GPG key from the latest Kubernetes repository should work with all other repositories under pkgs.k8s.io. We're debugging that and we'll follow up as we understand what's the best way forward.

[petedyerarm]

In my use case I have a loop which gets the Release.key for 1.27, 1.28, 1.29 and 1.30, followed by an apt update which fails for 1.27. I chose later the version I actually want to install.
So have all the keys expired and been replaced?

[drzewiec]

The original key has expired on 2024-11-02, however all repositories have been resigned with a new key. At the moment, the way forward is to download and install the new key:

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

We'll try to figure out if we can handle this in a better way in the future.

There seems to be some problem beyond that, because one of the steps in my docker build is to download that key fresh. So basically I'm doing this:

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.24/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.24/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list
apt-get update && apt-get install kubectl

That fails (with the errors mentioned in this issue) even though I am not caching the key.

[petedyerarm]

It appears that you can use the v1.31 Release.key to pull the v1.24 version, but I am doubtful this is correct.

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.24/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list
apt-get update && apt-get install kubectl

[xmudrii]

@drzewiec I was able to reproduce the issue, I'll check what's going on and report back ASAP.

[Buuuntyyy]

The original key has expired on 2024-11-02, however all repositories have been resigned with a new key. At the moment, the way forward is to download and install the new key:

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

We'll try to figure out if we can handle this in a better way in the future.

Hey, I got the same issue. Unfortunately, when i try to reset my key with the previous command, i got an error too:

File '/etc/apt/keyrings/kubernetes-apt-keyring.gpg' exists. Overwrite? (y/N) curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to pkgs.k8s.io:443
gpg: no valid OpenPGP data found.

Update: any curl -fsSL to the Release.key results to a connection refused on port 443

Waiting for your report, thanks for your reactivity.
Regards

[xmudrii]

/retitle APT key for some repositories at pkgs.k8s.io expired at 2024-11-02
I'm also going to pin this issue for time being.

[xmudrii]

/priority critical-urgent

[xmudrii]

/assign

[cameronbroe]

It looks like the RPM repositories are experiencing the same issue.

[seifrajhi]

+1
Same here

ubuntu $ echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /
ubuntu $ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
gpg: can't create '/etc/apt/keyrings/kubernetes-apt-keyring.gpg': No such file or directory
gpg: no valid OpenPGP data found.
gpg: dearmoring failed: No such file or directory
(23) Failed writing body

[BenTheElder]

Please refrain from +1 comments as opposed to upvoting existing comments. It's hard enough to keep up with the GitHub notifications and this is a volunteer effort.

[Pandey23000Om]

For installing K8s 1.27, I am having to get the key for 1.28 and then install 1.27

curl -fsSL https://pkgs.k8s.io/core:/stable:/**v1.28**/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

apt update
apt install -y kubelet=1.27.11-1.1 kubeadm=1.27.11-1.1 kubectl=1.27.11-1.1

[xmudrii]

At the moment, the GPG key is the same for all Kubernetes repositories under pkgs.k8s.io. This means that you can take the key from the v1.31 repository, and use it with let's say v1.27 or any other repository.

We're confirming if it's supposed to work that away or if there's something wrong with our service provider (OpenBuildService). At this time, you can mitigate the issue by downloading the key from the v1.31 repository as described in #3818 (comment)

We'll keep this issue updated as we figure out if there's a better way to handle this in the future, or if not, to properly document this.

Update: #3818 (comment) has been updated with the latest details and changes.

[primeos-work]

On RPM based systems it might not be enough to update the repo (e.g., /etc/yum.repos.d/kubernetes.repo) since the key was only updated to extend the expiration date instead of rotating to a new key. I opened #3840 to discuss that in more details.

For RPM based systems (in my case Fedora 40) I used the following workaround to remove the old version of the key which forces dnf update to (re)import it:

• rpm -qa | grep gpg-pubkey-9a296436 | xargs rpm --erase
• dnf update (or similar)

Then DNF will prompt to import the key based on (gpgkey in the repo definition (e.g. /etc/yum.repos.d/kubernetes.repo)):

Kubernetes                                                                                                                                                            4.7 kB/s | 1.7 kB     00:00
Importing GPG key 0x9A296436:
 Userid     : "isv:kubernetes OBS Project <isv:[email protected]>"
 Fingerprint: DE15 B144 86CD 377B 9E87 6E1A 2346 54DA 9A29 6436
 From       : https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key
Is this ok [y/N]: y
Key imported successfully

[BenTheElder]

/retitle APT/RPM PGP key for some repositories at pkgs.k8s.io expired at 2024-11-02

[ulidtko]

Note: the current key ID 234654DA9A296436 (full DE15B14486CD377B9E876E1A234654DA9A296436) has not changed, but the .gpg file has to be redownloaded, because it gained a reissued subkey.

Before:

~> gpg /etc/apt/trusted.gpg.d/kubernetes-OBS.234654DA9A296436.gpg
pub   rsa2048 2022-08-25 [SC] [expired: 2024-11-02]
      DE15B14486CD377B9E876E1A234654DA9A296436
uid           isv:kubernetes OBS Project <isv:[email protected]>

After:

~> gpg /etc/apt/trusted.gpg.d/kubernetes-OBS.234654DA9A296436.gpg
pub   rsa2048 2022-08-25 [SC] [expires: 2026-12-29]
      DE15B14486CD377B9E876E1A234654DA9A296436
uid           isv:kubernetes OBS Project <isv:[email protected]>

— notice [expires: 2026-12-29] VS [expired: 2024-11-02].

[saiteneffekt]

Running Ubuntu 22.04.5 LTS on WSL, I had to change the target directory from /etc/apt/keyrings/ to /etc/apt/trusted.gpg.d/ to make it work:
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/kubernetes-apt-keyring.gpg

[meyerkev]

1.27 key expired today. Mitigated via using the 1.31 key in place of the 1.27 key and we are now on k8s 1.31

[BenTheElder]

@xmudrii raised this problem the December 3rd SIG Release meeting, when we discussed it there I think one of the main action items was to figure out how other projects on OBS are handling this / "best practice". This is still TODO AFAIK and if anyone knows more, help would be welcome.

Barring that, one workaround @xmudrii raised was having a root/dummy package built frequently to emit a current key and then in Kubernetes's CDN infra copying it out over the other versions.

meeting notes: https://docs.google.com/document/d/1Fu6HxXQu8wl6TwloGUEOXVzZ1rwZ72IAhglnaAMCPqA/edit?tab=t.0

[cunningr]

We're hitting this for v1.24. Using the key for v1.31 is a workaround but we are using upstream https://github.com/kubernetes-sigs/image-builder which requires some ugly patching. Are there any plans to renew the keys for v1.24 where they are expiring?

[ulidtko]

Hey @BenTheElder, if I may suggest some ideas...

First and foremost, don't version the package-signing key. It makes zero sense to distribute one key for v1.24 and [potentially] another for v1.31. That's because PSKs have nothing to do with packaged software neither changes therein — they have all to do with release-engineering, authenticating packages' publisher, establishing a secure link in the supply chain. Coupling the key to kubectl version is gratuitous and completely unnecessary. Perhaps I'm missing some additional concerns of the SIG — but IMO doing this just creates complications for no good reason.

You'd asked for example: here's Mozilla team, publishing releases in Launchpad PPA as a step before they land in main repos: https://launchpad.net/~mozillateam — observe 2 things:

• there's 15-ish software products versioned & packaged separately (Firefox, Thunderbird, etc) — this is analogous to, how pkgs.k8s.io publishes cri-tools, kubeadm, kubelet, kubectl, kubernetes-cni;
• all of them are signed by the single package-signing-key. The single key identifies the single team.

I tried OBS once, long time ago, unfortunately not a frequenter there currently. But I'm absolutely sure there's nothing Launchpad-specific in that practice; it's just the first relatable example that came to mind.

For another example, here's how NodeJS deb bindist gets published by NodeSource Co. — https://github.com/nodesource/distributions/blob/master/README.md — same idea: single package signing key. They keep just 2 packages in the repo, but maintain multiple LTS release lines just like Kubernetes does — yet, there's only 1 key, as there's only 1 team making the signatures.

In fact, as a package consumer, I'd become alerted if I spotted suddenly two teams signing off releases with two different keys. Unless the keys are cross-signed, that's a clear sign of takeover attempt.

Heck, even Microsoft can do it! Can you believe that 😆 — https://packages.microsoft.com/

• 30-ish repos just for Debian, I didn't count how many packages;
• again, all signed by the single key BC528686B50D79E339D3721CEB3E94ADBE1229CF, as far as I'm checking.

---

[...] one workaround @xmudrii raised was having a root/dummy package built frequently to emit a current key and then in Kubernetes's CDN infra copying it out over the other versions.

Pardon my frank critique, hoping I won't offend anyone, but inventive bicycles that this completely miss the point.

Key distribution must happen out of band, through an outside channel relative to the artifacts that the key signs. Because you don't want an event of access compromise granting file-upload-equivalent capability to automatically escalate into capability of signing-key substitution (and consequently, attacker being able to stamp valid signatures onto their uploads). Which means: no, distributing the key itself via a "keychain" dummy package — is a bad idea.

Less theoretically and more constructively: please look into PGP SKS & HKP; that's the canonical approach to pgp key distribution. Here's a view into SKS keyserver network https://spider.pgpkeys.eu/graphs/ hope that helps. But also: just a plain old HTTPS URL hosting the package signing key file (again: outside the repos!) is common too, simple and well-accepted.

[BenTheElder]

First and foremost, don't version the package-signing key. It makes zero sense to distribute one key for v1.24 and [potentially] another for v1.31. That's because PSKs have nothing to do with packaged software neither changes therein — they have all to do with release-engineering, authenticating packages' publisher, establishing a secure link in the supply chain. Coupling the key to kubectl version is gratuitous and completely unnecessary. Perhaps I'm missing some additional concerns of the SIG — but IMO doing this just creates complications for no good reason.

yes, I don't think we want to, my understanding is that the current layout comes from OBS (also to be clear I don't work on this, but I do lead SIG K8s Infra which provides the CDN infra and manages sustainable cloud credit usage)

I think we understand what other deb/rpm repos look like, before OBS we used an instance of Google's package host which looked like what you're describing -- single key.

The problem is either to manage this within the constraints of OBS (hence the ask for prior art from other OBS users) or to outline replacing OBS (a potentially much bigger but doable task, we use our own build & release system for the other bits).

personally I didn't think we should be hosting/maintauning distro style packages at all, we have binaries and container images and distros could package kubectl / kubelet on their own and we already spend substantial resources on the other more popular binary/container image hosts.

[BenTheElder]

to figure out how other projects on OBS are handling this

unless I missed something, we still don't know this (I don't see evidence that node or Microsoft are using openBuildService)

[BenTheElder]

... ideally the project would fully control the key and we could just publish it directly ...

[ulidtko]

I think @BenTheElder OBS as the CI / build plaform is not material to the overall practice of package-signing-key management... It's more of an operational (Ops) matter

[ulidtko]

Again: I'm not an active user of OBS; but it "seems easy"™ and perfectly doable to apply the generic principles to OBS specifics. E.g. finding this guide — https://en.opensuse.org/openSUSE:Build_Service_Signer — just set the $keyfile of all repos to 1 shared key, and bob's your uncle, right?..

[BenTheElder]

OBS as the CI / build plaform is not material to the overall practice of package-signing-key management... It's more of an operational (Ops) matter

I don't think that's true, the package signing happens as part of the build service.

E.g. finding this guide — https://en.opensuse.org/openSUSE:Build_Service_Signer — just set the $keyfile of all repos to 1 shared key, and bob's your uncle, right?..

Those docs appear to me to be related to operating your own instance of the OBS software (specifically the signing service), releng / SIG Release is using the hosted public instance at https://build.opensuse.org/

@xmudrii would know more.