RKE2 on VMware vSphere and configuring the CPI and CSI Cloud Providers during installation

[vu3oim]

Environmental Info:
RKE2 Version: rke2r1
rke2 version v1.25.10+rke2r1 (e0c376c)
go version go1.19.9 X:boringcrypto

Node(s) CPU architecture, OS, and Version:
[root@rke2vm1 ~]# ssh rke2vm1 uname -a
Linux rke2vm1 4.18.0-305.19.1.el8_4.x86_64 #1 SMP Tue Sep 7 07:07:31 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

[root@rke2vm1 ~]# ssh rke2vm2 uname -a
Linux rke2vm2 4.18.0-305.19.1.el8_4.x86_64 #1 SMP Tue Sep 7 07:07:31 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

[root@rke2vm1 ~]# ssh rke2vm3 uname -a
Linux rke2vm3 4.18.0-305.19.1.el8_4.x86_64 #1 SMP Tue Sep 7 07:07:31 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
I have a 3 node RKE2 cluster:
[root@rke2vm1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
rke2vm1 Ready control-plane,etcd,master 19h v1.25.10+rke2r1
rke2vm2 Ready control-plane,etcd,master 19h v1.25.10+rke2r1
rke2vm3 Ready control-plane,etcd,master 19h v1.25.10+rke2r1

Describe the bug:
I have setup my RKE2 cluster following: https://docs.rke2.io/install/quickstart
Without configuring a Cloud Provider (rancher-vsphere) during installation, there is no problem in installation, all good.

But when I want to have an integrated Cloud Provider (in my case vSphere) setup during installation, I am unable to get the installation done.

The problem I see is, there is no enough documentation about this.
I tried setting these 2 via rke2 command line and also tried putting the details of these 2 variables into /etc/rancher/rke2/config.yaml in yam format, but no luck.

--cloud-provider-name value (cloud provider) Cloud provider name [$RKE2_CLOUD_PROVIDER_NAME]
--cloud-provider-config value (cloud provider) Cloud provider configuration file path [$RKE2_CLOUD_PROVIDER_CONFIG]

--cloud-provider-name rancher-vsphere
The above is clear.

But I want to get information how to setup --cloud-provider-config and its configuration file contents.

Steps To Reproduce:
Install RKE2 by enabling the Cloud Provider as VMware vSphere. Due to missing / lack of documentation which I feel is the case, I am not able to understand how to supply all the needed vSphere related information (vCenter IP, credentials, datastore, etc) to the RKE2 installer.

Please point me to the relevant documentation.

Expected behavior:

Actual behavior:

Additional context / logs:

Due to the above, the needed CPI and CSI pods related to vSphere is not starting up. Logs are here.
Its clear that those pods are not able to gather the vSphere details, but my main point is how to supply them during installer?

kubectl logs -n kube-system helm-install-rancher-vsphere-csi-sw6md

• helm_v3 install --set-string 'global.clusterCIDR=10.128.0.0/14,fd02::/48' --set-string global.clusterCIDRv4=10.128.0.0/14 --set-string global.clusterCIDRv6=fd02::/48 --set-string global.clusterDNS=172.30.0.10 --set-string global.clusterDomain=cluster.local --set-string global.rke2DataDir=/var/lib/rancher/rke2 --set-string 'global.serviceCIDR=172.30.0.0/16,fd03::/112' rancher-vsphere-csi /tmp/rancher-vsphere-csi.tgz
  Error: INSTALLATION FAILED: template: rancher-vsphere-csi/templates/secret.yaml:8:23: executing "rancher-vsphere-csi/templates/secret.yaml" at <tpl .Values.vCenter.configSecret.configTemplate .>: error calling tpl: error during tpl function execution for "[Global]\ncluster-id = {{ required ".Values.vCenter.clusterId must be provided" (default .Values.vCenter.clusterId .Values.global.cattle.clusterId) | quote }}\nuser = {{ .Values.vCenter.username | quote }}\npassword = {{ .Values.vCenter.password | quote }}\nport = {{ .Values.vCenter.port | quote }}\ninsecure-flag = {{ .Values.vCenter.insecureFlag | quote }}\n\n[VirtualCenter {{ .Values.vCenter.host | quote }}]\ndatacenters = {{ .Values.vCenter.datacenters | quote }}\n": execution error at (rancher-vsphere-csi/templates/secret.yaml:2:16): .Values.vCenter.clusterId must be provided
• exit

kubectl logs -n kube-system rancher-vsphere-cpi-cloud-controller-manager-kmpsx

I0730 11:54:15.789480 1 serving.go:348] Generated self-signed cert in-memory
W0730 11:54:15.789517 1 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0730 11:54:16.408104 1 requestheader_controller.go:244] Loaded a new request header values for RequestHeaderAuthRequestController
I0730 11:54:16.408618 1 main.go:160] vsphere-cloud-controller-manager version: v1.25.2
E0730 11:54:16.408791 1 config_yaml.go:136] vsphere.conf does not have the VirtualCenter IP address specified
E0730 11:54:16.408803 1 config_yaml.go:214] validateConfig failed: vsphere.conf does not have the VirtualCenter IP address specified
W0730 11:54:16.408808 1 config.go:69] ReadCPIConfigYAML failed: vsphere.conf does not have the VirtualCenter IP address specified
E0730 11:54:16.408827 1 config.go:73] ReadConfigINI failed: 3:1: expected section header
F0730 11:54:16.408836 1 main.go:265] Cloud provider could not be initialized: could not init cloud provider "vsphere": 3:1: expected section header

[brandond]

I want to get information how to setup --cloud-provider-config and its configuration file contents.

Our documentation does not cover all of the many bits provided by different projects. Each cloud provider wants different information in different formats. You can find the cloud provider configuration file format required by the vsphere cloud provider in the vsphere cloud provider's documentation: https://github.com/kubernetes/cloud-provider-vsphere/blob/master/docs/book/cloud_config.md

[qdrddr]

What's the recommended way to use rke2 installed ontop of vSphere?

[marvinlnnx]

Today after a year still I am struggling with this !

[ghost]

I bumped into this thought also (others too #2089) (and I think is not well-documented, but we are free to PR docs). RKE2's cloud-provider-name: vsphere forces an in-tree cloud provider (as it is deprecated, kube-controller-xxx SIGSEVs miserably on 1.25.x, at least in my environment).
It is because of #1115, you may reproduce:

1. RKE2 install. Do not set up cloud-provider or related config
2. Install CPI through Helm charts or manual. In theory this could be done through Rancher Charts if cluster is lcoal, but I am not sure if it is recommended.
3. Check you have your vsphere controllers up and running, and no taints.
4. Set up

kubelet-arg:
  - "cloud-provider=external"
cloud-provider-name: vsphere

And restart kubelet (systemctl restart rke2-server)
6. Here we expect to see no taints and out-of-tree CPI configured. But I can see that's not the case. kube-controller-manager-* are crashing:

2023-08-23T12:38:21.309171682+02:00 W0823 10:38:21.309113       1 plugins.go:132] WARNING: vsphere built-in cloud provider is now deprecated. The vSphere provider is deprecated and will be removed in a future release. Please use https://github.com/kubernetes/cloud-provider-vsphere
I0823 10:38:21.311868       1 shared_informer.go:252] Waiting for caches to sync for tokens
panic: runtime error: invalid memory address or nil pointer dereference
2023-08-23T12:38:21.317119390+02:00 [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x3595902]

Whoah! We are sticking to deprecated in-tree provider. This could be expected as per #1115 breaking behaviour. Modify again your RKE2 config:

kubelet-arg:
  - "cloud-provider=external"

And restart kubelet
7. Now our kube-controller-manager-* seem fine, and Jobs are schedulable again.

But we have another method! This is for me the non-documented and opaque one.

1. make sure you have no helm chart vSphere CPI/CSI installed from vanilla/manual install
2. Config your rke2:

kubelet-arg:
  - "cloud-provider=external"
cloud-provider-name: rancher-vsphere

3. When you restart your kubelet some static charts will be installed through integrated helm-controller. This has some advantages, bundled cpi/csi will be well-testet with your rke2 version.
4. You can see that your configuration out-of the box does not work! You have to modify a pair of ConfigMap - Secret that will contain your vsphere information and credentials. Check for related/mounted resources in kube-system pods: rancher-vsphere-cpi-cloud-controller-*. I cannot remember exact names: vsphere-creds/vsphere-cloud-conf/... How to configure it is not SUSE-Rancher related, see https://cloud-provider-vsphere.sigs.k8s.io/cloud_config
5. If you modify your configMap and Secret and force rancher-vsphere-cpi-cloud-controller-* reinstatiation (delete existing pods) you will see that new rancher-vsphere-controller's are working again.
6. If I restart a rke2-server with cloud-provider-name: rancher-vsphere. CPI is reinstalled and I lost my configuration. As this is not documented I do not certainly know how this should be addressed. An option may be:
   • Static HelmChartConfig to override values and manually supply a ConfigMap/Secret.
   • Use of --cloud-provider-config if we belive is vsphere.conf format (ini/yaml based). Have not tested this.

Conclusion

'rancher-vsphere' provides a somewhat (and not documented) opaque method to auto-install out-of-tree vSphere CPI. Which I believe people use but I did not find reliable resources about it. Another drawback is that it also forces CSI install, which may not be desirable in some cases.

'rancher-vsphere' with static helm charts works, but when rke2-server is restarted, it resets is values. Some persistent static config must be configured in a server node's filesystem.

If you need more flexibility, you can succeed with vSphere CPI/CSI installation with a manual/chart store install. But you do not have to set cloud-provider-name in your RKE2 config.

[gabrielgbs97]

I changed my nickname, and the author of the comment disappeared. Just to say that nowadays our approach is to use static YAML HelmChartConfig (s) to bootstrap managemnt cluster for Rancher, or manually installed CPI standalone. @qwiatu-lp has a good example below for HelmChartConfig path.

[qdrddr]

Are there any workarounds or solutions to fix this issue so it continues to work after the ke2-server is restarted? @gbarceloPIB @brandond

[ghost]

Yes, if you are using static rancher-vsphere, you need to modify secrets and config maps related to vsphere CPI/CSI (through Rancher UI is ok), when they are working, export them as single yaml. When you restart you will lose the values set. You need to reference the yaml file in cloud-provider-config.

You may try to set up first vsphere CPI/CSI without rancher-vsphere help to practice. Just follow vsphere docs. https://cloud-provider-vsphere.sigs.k8s.io/tutorials/kubernetes-on-vsphere-with-helm

[rfx77]

i have the same problem and the documentation is lacking very much.

this is my config.yaml in /etc/rancher/rke2/

token: my-shared-secret
tls-san:
  - office.local

kubelet-arg:
  - "cloud-provider=external"

cloud-provider-name: rancher-vsphere
cloud-provider-config: "/etc/rancher/rke2/vsphere.yaml"

No matter what the contents of the cloud-provider-conf are, they are never used. i tried various formats.

the contents of the configmap vsphere-cloud-config are always the sample content.

i cannot find any error. can someone please clarify how the cloud-provider-config should look like in the case of rancher-vsphere and why it is not used by the klipper-helm install job helm-install-rancher-vsphere-cpi and also not by the rancher-vsphere-cpi-cloud-controller-manager.

same situation with the csi driver.

thanks.

[brandond]

When using an external cloud-provider, you cannot pass config in via CLI args. You need to configure the cloud provider HelmChartConfig.
Ref:

• https://docs.rke2.io/helm#customizing-packaged-components-with-helmchartconfig
• https://github.com/rancher/rke2-charts/blob/main/charts/rancher-vsphere-cpi/rancher-vsphere-cpi/1.5.100/values.yaml

[qwiatu-lp]

As @brandond mentioned you can use helmchartconfig to parametrize vsphere cpi/csi. Below example config uses vsphere cpi/csi provided by rke2 but you can use always more configurable edition from partner's site.

mkdir -p /etc/rancher/rke2/
cat <<EOF> /etc/rancher/rke2/config.yaml
cloud-provider-name: rancher-vsphere
cni:
- calico
protect-kernel-defaults: false
selinux: true
token: random-it

EOF


# just for developer use
curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="server" sh -


mkdir -p /var/lib/rancher/rke2/server/manifests

cat << EOF > /var/lib/rancher/rke2/server/manifests/config-rancher-vsphere-cpi-csi.yaml
---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rancher-vsphere-cpi
  namespace: kube-system
spec:
  valuesContent: |-
    vCenter:
      host: "vcenter"
      port: 443
      insecureFlag: true
      datacenters: "vmware-dc"
      username: "[email protected]"
      password: "passwd"
      credentialsSecret:
        name: "vsphere-cpi-creds"
        generate: true
---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rancher-vsphere-csi
  namespace: kube-system
spec:
  valuesContent: |-
    vCenter:
      host: "vcenter"
      port: 443
      insecureFlag: "1"
      clusterId: "randomize-it"
      datacenters: "vmware-dc"
      username: "[email protected]"
      password: "passwd"
EOF