Cant uninstall cilium.... it keeps getting reinstalled... how to uninstall? #5769

Smithx10 · 2024-04-12T19:52:30Z

Smithx10
Apr 12, 2024

I've recently installed Cilium to replace kube-proxy, and wanted to revert that change.

For some reason after I've deleted all things cilium eventually rke2 or something is reinstalling it.

I uninstalled it all..... and it keeps coming back lol..... my cluster feels haunted.

#!/bin/bash
rm /var/lib/rancher/rke2/server/manifests/cilium.yaml
rm -rf /var/run/cilium/cilium.pid
rm /opt/cni/bin/cilium-cni
./cilium-dbg post-uninstall-cleanup -f
rm -rf /root/.kube/cache/discovery/127.0.0.1_6443/cilium.io
rm -rf /var/lib/cni/results/cilium-*
rm -rf /etc/sysctl.d/99-zzz-override_cilium.conf
rm -rf /run/live/overlay/rw/etc/sysctl.d/99-zzz-override_cilium.conf
rm -rf /var/lib/rancher/rke2/data/v1.29.0-rke2r1-491b1a599021/charts/rke2-cilium.yaml

for r in $(k get crd | grep cilium | awk '{print $1}'); do kubectl delete crd $r; done
kubectl delete configmap chart-content-cilium -n kube-system
kubectl delete addon cilium-proxy -n kube-system

brandond · 2024-04-12T20:16:46Z

brandond
Apr 12, 2024
Maintainer

Did you restart rke2 with a different CNI, or with cni: none? The selected CNI will be redeployed every time RKE2 restarts...

0 replies

Smithx10 · 2024-04-12T20:37:03Z

Smithx10
Apr 12, 2024
Author

We currently run kube-ovn. We installed the cluster with cni: none in order to install kube-ovn. After installing Kube-OVN we went to experiment with removing kube-proxy for ciliums proxy by adding "cilium-proxy.yaml" to
/var/lib/rancher/rke2/server/manifests/cilium-proxy.yaml.

After failing and creating the following issue: kubeovn/kube-ovn#3908 we attempted to uninstall.... and now we are stuck with it coming back to life, and not really knowing from where.

cilium-proxy.yaml

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: cilium
  namespace: kube-system
spec:
  bootstrap: true
  chart: cilium
  targetNamespace: kube-system
  version: 1.15.3
  repo: https://helm.cilium.io
  valuesContent: |-
    routingMode: native
    devices: nodeip
    kubeProxyReplacement: true
    k8sServiceHost: 172.16.0.100
    k8sServicePort: 6443
    loadBalancer:
      algorithm: maglev
      mode: dsr
      dsrDispatch: opt

config.yaml

token: $mytoken
data-dir: /var/lib/rancher/rke2
cni: none
tls-san:
        - cluster.local
        - 172.16.0.1
        - 172.16.0.2
        - 172.16.0.3
        - 172.16.0.100
        - 10.91.230.10
        - 10.91.230.11
        - 10.91.230.12
        - 10.91.230.100
node-ip: 172.16.0.1
node-external-ip: 10.91.230.10
cluster-cidr: 172.16.128.0/17
cluster-dns: 172.16.64.10
service-cidr: 172.16.64.0/18
cluster-domain: cluster.local
snapshotter: zfs
node-taint:
        - CriticalAddonsOnly=true:NoExecute
node-label:
        - kube-ovn/role=master
disable:
  - cilium-proxy
  - cilium

0 replies

Smithx10 · 2024-04-12T22:05:57Z

Smithx10
Apr 12, 2024
Author

After restarting the rke2-server on headnode-02 it resulted in the job firing off:

[ use1 ] root@headnode-01:/$ rg -l -g '!proc/' -g '!sys/' -g '!run/' cilium
root/cleanup.sh
root/post-boot/manifests/cilium-proxy.yaml
etc/rancher/rke2/config.yaml
var/log/sysstat/sar11
var/log/pods/kube-system_kube-apiserver-headnode-01_3cc1d912047f1dec88e5dae71d2a7852/kube-apiserver/1.log
var/log/pods/kube-system_kube-apiserver-headnode-01_3cc1d912047f1dec88e5dae71d2a7852/kube-apiserver/0.log
var/log/pods/kube-system_etcd-headnode-01_17a1768bd2a7812d1dff1c8c540f0e60/etcd/0.log
var/log/pods/kube-system_etcd-headnode-01_17a1768bd2a7812d1dff1c8c540f0e60/etcd/1.log
var/lib/rancher/rke2/agent/logs/kubelet.log
var/lib/rancher/rke2/agent/containerd/containerd.log
var/lib/rancher/rke2/agent/containerd/io.containerd.content.v1.content/blobs/sha256/1272788a112d1333fe684ab53b2784dc833e7bf6733238d651c7c9d0b7af6a49

[ use1 ] root@headnode-02:/$ rg -l -g '!proc/' -g '!sys/' -g '!run/' cilium
etc/rancher/rke2/config.yaml
root/cleanup.sh
var/lib/rancher/rke2/agent/containerd/containerd.log
var/lib/rancher/rke2/agent/logs/kubelet.log
var/lib/rancher/rke2/agent/containerd/io.containerd.content.v1.content/blobs/sha256/1272788a112d1333fe684ab53b2784dc833e7bf6733238d651c7c9d0b7af6a49
var/log/sysstat/sar11
var/log/pods/kube-system_helm-delete-cilium-drws9_c5e0f755-eb2f-4220-ad00-b3c7408012c3/helm/0.log
var/log/pods/kube-system_kube-apiserver-headnode-02_bed643f3c80b5bd7ec85a4816d8160f4/kube-apiserver/1.log
var/log/pods/kube-system_etcd-headnode-02_13710d86e3a9a4b1bec724e886acce45/etcd/1.log
var/log/pods/kube-system_etcd-headnode-02_13710d86e3a9a4b1bec724e886acce45/etcd/0.log
var/log/pods/kube-system_kube-apiserver-headnode-02_bed643f3c80b5bd7ec85a4816d8160f4/kube-apiserver/0.log


[ use1 ] root@headnode-03:/$ rg -l -g '!proc/' -g '!sys/' -g '!run/' cilium
root/cleanup.sh
etc/rancher/rke2/config.yaml
var/lib/rancher/rke2/agent/logs/kubelet.log
var/lib/rancher/rke2/agent/containerd/containerd.log
var/lib/rancher/rke2/agent/containerd/io.containerd.content.v1.content/blobs/sha256/1272788a112d1333fe684ab53b2784dc833e7bf6733238d651c7c9d0b7af6a49
var/log/pods/kube-system_kube-apiserver-headnode-03_7418626ca441c2de0dd853960c871205/kube-apiserver/1.log
var/log/pods/kube-system_kube-apiserver-headnode-03_7418626ca441c2de0dd853960c871205/kube-apiserver/0.log
var/log/pods/kube-system_etcd-headnode-03_8fdeacb098631c555b69c7d66dbc73f7/etcd/0.log
var/log/pods/kube-system_kube-controller-manager-headnode-03_6785504cf290f47af5e0262cd05c371e/kube-controller-manager/2.log
var/log/sysstat/sar11

[ use1 ] root@headnode-01:~$ tail -f /var/lib/rancher/rke2/agent/logs/kubelet.log | grep cilium

I0412 21:41:57.143021 2631195 topology_manager.go:215] "Topology Admit Handler" podUID="5c819549-cade-4999-9028-7b9b194474cf" podNamespace="kube-system" podName="helm-install-cilium-97ldt"
I0412 21:41:57.191635 2631195 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zhmx\" (UniqueName: \"kubernetes.io/projected/5c819549-cade-4999-9028-7b9b194474cf-kube-api-access-4zhmx\") pod \"helm-install-cilium-97ldt\" (UID: \"5c819549-cade-4999-9028-7b9b194474cf\") " pod="kube-system/helm-install-cilium-97ldt"
I0412 21:41:57.191834 2631195 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started for volume \"content\" (UniqueName: \"kubernetes.io/configmap/5c819549-cade-4999-9028-7b9b194474cf-content\") pod \"helm-install-cilium-97ldt\" (UID: \"5c819549-cade-4999-9028-7b9b194474cf\") " pod="kube-system/helm-install-cilium-97ldt"
I0412 21:41:57.191940 2631195 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started for volume \"values\" (UniqueName: \"kubernetes.io/secret/5c819549-cade-4999-9028-7b9b194474cf-values\") pod \"helm-install-cilium-97ldt\" (UID: \"5c819549-cade-4999-9028-7b9b194474cf\") " pod="kube-system/helm-install-cilium-97ldt"
I0412 21:41:58.838113 2631195 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/helm-install-cilium-97ldt" podStartSLOduration=1.838024083 podStartE2EDuration="1.838024083s" podCreationTimestamp="2024-04-12 21:41:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2024-04-12 21:41:58.836382803 +0000 UTC m=+229.880711535" watchObservedRunningTime="2024-04-12 21:41:58.838024083 +0000 UTC m=+229.882352799"
I0412 21:42:01.709964 2631195 topology_manager.go:215] "Topology Admit Handler" podUID="3e8e3c20-7979-4196-9292-730b561b9a68" podNamespace="kube-system" podName="cilium-6z4xh"
I0412 21:42:01.731856 2631195 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cilium-run\" (UniqueName: \"kubernetes.io/host-path/3e8e3c20-7979-4196-9292-730b561b9a68-cilium-run\") pod \"cilium-6z4xh\" (UID: \"3e8e3c20-7979-4196-9292-730b561b9a68\") " pod="kube-system/cilium-6z4xh"
I0412 21:42:01.731987 2631195 reconciler_common.go:258] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e8e3c20-7979-4196-9292-730b561b9a68-lib-modules\") pod \"cilium-6z4xh\" (UID: \"3e8e3c20-7979-4196-9292-730b561b9a68\") " pod="kube-system/cilium-6z4xh"

0 replies

Smithx10 · 2024-04-12T23:22:36Z

Smithx10
Apr 12, 2024
Author

Just to clarify,

cni: none has been consistent in the config the entire time. Only adding cilium-proxy.yaml to and removed from the server/manifests directory.

0 replies

Smithx10 · 2024-04-12T23:49:23Z

Smithx10
Apr 12, 2024
Author

After a while, I'm not sure why or what triggered this, but it appears there is a helm-delete-cilium job.

Is this a function of removing the helmchart crd?

[ use1 ] root@headnode-02:/$ k get helmchart -A
NAMESPACE     NAME                               JOB                                             CHART          TARGETNAMESPACE   VERSION   REPO                                    HELMVERSION   BOOTSTRAP
kube-system   cert-manager                       helm-install-cert-manager                       cert-manager   cert-manager      v1.14.4   https://charts.jetstack.io                            true
kube-system   openebs                            helm-install-openebs                            zfs-localpv    openebs           2.4.2     https://openebs.github.io/zfs-localpv                 true
kube-system   rke2-coredns                       helm-install-rke2-coredns                                                                                                                        true
kube-system   rke2-ingress-nginx                 helm-install-rke2-ingress-nginx                                                                                                                  false
kube-system   rke2-metrics-server                helm-install-rke2-metrics-server                                                                                                                 false
kube-system   rke2-snapshot-controller           helm-install-rke2-snapshot-controller                                                                                                            false
kube-system   rke2-snapshot-controller-crd       helm-install-rke2-snapshot-controller-crd                                                                                                        false
kube-system   rke2-snapshot-validation-webhook   helm-install-rke2-snapshot-validation-webhook                                                                                                    false

[ use1 ] root@headnode-02:/$ k get job helm-delete-cilium -n kube-system
NAME                 COMPLETIONS   DURATION   AGE
helm-delete-cilium   1/1           4s         108m

3 replies

brandond Apr 13, 2024
Maintainer

Is this a function of removing the helmchart crd?

Yes.

Smithx10 Apr 13, 2024
Author

Does this run on a timer, or have a controller? What actor actually resolves this? I don't see a controller. Is this built into the rke2-server daemon itself?

brandond Apr 15, 2024
Maintainer

Yes, the helm controller is built into rke2.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cant uninstall cilium.... it keeps getting reinstalled... how to uninstall? #5769

{{title}}

Replies: 5 comments 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Cant uninstall cilium.... it keeps getting reinstalled... how to uninstall? #5769

Smithx10 Apr 12, 2024

Replies: 5 comments · 3 replies

brandond Apr 12, 2024 Maintainer

Smithx10 Apr 12, 2024 Author

Smithx10 Apr 12, 2024 Author

Smithx10 Apr 12, 2024 Author

Smithx10 Apr 12, 2024 Author

brandond Apr 13, 2024 Maintainer

Smithx10 Apr 13, 2024 Author

brandond Apr 15, 2024 Maintainer

Smithx10
Apr 12, 2024

Replies: 5 comments 3 replies

brandond
Apr 12, 2024
Maintainer

Smithx10
Apr 12, 2024
Author

Smithx10
Apr 12, 2024
Author

Smithx10
Apr 12, 2024
Author

Smithx10
Apr 12, 2024
Author

brandond Apr 13, 2024
Maintainer

Smithx10 Apr 13, 2024
Author

brandond Apr 15, 2024
Maintainer