Project cloud-provider-vsphere has been deleted.

[saintdle]

What happened?

Booting up cluster, unable to get CSI pods running which has taken cluster offline, when troubleshooting, unable to pull the image, testing with docker pull, get the following

❯ docker pull gcr.io/cloud-provider-vsphere/csi/release/driver:v3.3.0
Error response from daemon: Head "https://gcr.io/v2/cloud-provider-vsphere/csi/release/driver/manifests/v3.3.0": denied: Project cloud-provider-vsphere has been deleted.

What did you expect to happen?

Image to be available

How can we reproduce it (as minimally and precisely as possible)?

docker pull gcr.io/cloud-provider-vsphere/csi/release/driver:v3.3.0

Anything else we need to know (please consider providing level 4 or above logs of CPI)?

No response

Kubernetes version

❯ kubectl version
Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.7
Kubecolor Version: v0.4.0

Cloud provider or hardware configuration

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Kernel (e.g. uname -a)

Install tools

Container runtime (CRI) and and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

Others

[hyorch]

Images have been migrated to another registry.
https://github.com/kubernetes/cloud-provider-vsphere#warning-kubernetes-image-registry-migration-for-cloud-provider-vsphere

[addyess]

@hyorch thanks for pointing at the note about the new registry. I don't see the older releases (1.27.0 for example) nor do I quite understand where the various syncer, driver, or manager for the vsphere-storage drivers

effectively -- where did these images go?

gcr.io/cloud-provider-vsphere/cpi/release/manager:v1.18.0
gcr.io/cloud-provider-vsphere/cpi/release/manager:v1.19.0
gcr.io/cloud-provider-vsphere/cpi/release/manager:v1.2.1
gcr.io/cloud-provider-vsphere/cpi/release/manager:v1.20.0
gcr.io/cloud-provider-vsphere/cpi/release/manager:v1.21.1
gcr.io/cloud-provider-vsphere/cpi/release/manager:v1.22.3
gcr.io/cloud-provider-vsphere/cpi/release/manager:v1.23.1
gcr.io/cloud-provider-vsphere/cpi/release/manager:v1.24.2
gcr.io/cloud-provider-vsphere/cpi/release/manager:v1.25.0
gcr.io/cloud-provider-vsphere/cpi/release/manager:v1.26.0
gcr.io/cloud-provider-vsphere/cpi/release/manager:v1.27.0
gcr.io/cloud-provider-vsphere/csi/release/driver:v2.5.3
gcr.io/cloud-provider-vsphere/csi/release/driver:v2.5.4
gcr.io/cloud-provider-vsphere/csi/release/driver:v2.6.0
gcr.io/cloud-provider-vsphere/csi/release/driver:v2.6.1
gcr.io/cloud-provider-vsphere/csi/release/driver:v2.6.2
gcr.io/cloud-provider-vsphere/csi/release/driver:v2.6.3
gcr.io/cloud-provider-vsphere/csi/release/driver:v2.7.0
gcr.io/cloud-provider-vsphere/csi/release/driver:v3.0.0
gcr.io/cloud-provider-vsphere/csi/release/driver:v3.0.2
gcr.io/cloud-provider-vsphere/csi/release/driver:v3.0.3
gcr.io/cloud-provider-vsphere/csi/release/driver:v3.1.0
gcr.io/cloud-provider-vsphere/csi/release/driver:v3.1.1
gcr.io/cloud-provider-vsphere/csi/release/driver:v3.1.2
gcr.io/cloud-provider-vsphere/csi/release/driver:v3.2.0
gcr.io/cloud-provider-vsphere/csi/release/driver:v3.3.0
gcr.io/cloud-provider-vsphere/csi/release/driver:v3.3.1
gcr.io/cloud-provider-vsphere/csi/release/syncer:v2.5.3
gcr.io/cloud-provider-vsphere/csi/release/syncer:v2.5.4
gcr.io/cloud-provider-vsphere/csi/release/syncer:v2.6.0
gcr.io/cloud-provider-vsphere/csi/release/syncer:v2.6.1
gcr.io/cloud-provider-vsphere/csi/release/syncer:v2.6.2
gcr.io/cloud-provider-vsphere/csi/release/syncer:v2.6.3
gcr.io/cloud-provider-vsphere/csi/release/syncer:v2.7.0
gcr.io/cloud-provider-vsphere/csi/release/syncer:v3.0.0
gcr.io/cloud-provider-vsphere/csi/release/syncer:v3.0.2
gcr.io/cloud-provider-vsphere/csi/release/syncer:v3.0.3
gcr.io/cloud-provider-vsphere/csi/release/syncer:v3.1.0
gcr.io/cloud-provider-vsphere/csi/release/syncer:v3.1.1
gcr.io/cloud-provider-vsphere/csi/release/syncer:v3.1.2
gcr.io/cloud-provider-vsphere/csi/release/syncer:v3.2.0
gcr.io/cloud-provider-vsphere/csi/release/syncer:v3.3.0
gcr.io/cloud-provider-vsphere/csi/release/syncer:v3.3.1

[erSitzt]

@hyorch nice warning, but why are the manifests for those versions still referencing the old gcr.io images ?
And is there a list/mapping of old to new images so i can replace them manually ?

These issues regarding the registry switch are now all two month old and still there is no well documented way to install the csi-driver using the new registry ?

This registry switch does not seem to be planned and executed well...

P.S. @hyorch after a second read this sounded like im blaming you :) sorry, this was meant to be a general rant on how bad this was handeled.

[hyorch]

I not working for Vsphere. I think they have deleted images from gcr.io before gcr.io shutdown planed for May 2025. https://cloud.google.com/artifact-registry/docs/transition/gcr-repositories

The problem is if you are using and old version of the containers and try just to change container version on your deployments. New releases needs new deployments parameters. So, if you just upgrde container images version on old deployments, your deployments will not work. You have to upgrade both: container versions and deployment paramentes (Statefullset, Deployments, etc).
You can review new parameters/configuration on GitHub Kubernetes-Sig page: https://github.com/kubernetes-sigs/vsphere-csi-driver/tree/master/manifests/vanilla

[erSitzt]

@hyorch yeah i know :)

i just saw that the other repository has better info on the change and the new images... and correct manifests

[DanielXiao]

Sorry that CPI images below v1.28.0 were lost during the registry switch. We have a plan to restore them but before that you can use mirrors from dockerhub like https://hub.docker.com/r/rancher/mirrored-cloud-provider-vsphere-cpi-release-manager/tags.