Add documentation about cilium and aws-ebs-csi-driver

[sylr]

Description

How can the docs be improved?

We have had issues in our clusters caused by dandling EBS volumes attachements on karpenter disrupted nodes.

Because of the taint karpenter.sh/disrupted:NoSchedule applied by karpenter on disrputed nodes, the cilium and aws-ebs-csi-driver agents where evicted before the aws-ebs-csi-driver could properly unmount the PVC and remove the volumeattachments.storage.k8s.io.

To resolve our issue we had to add the following tolerations to cilium and aws-ebs-csi-driver multiple daemonsets:

- effect: NoSchedule
  key: karpenter.sh/disrupted
  operator: Exists

[z0rc]

aws-ebs-csi-driver installed as eks addon already tolerates all taints. aws-ebs-csi-driver installed as a helm chart already does so.

In addition karpenter v1 watches VolumeAttachments to ensure they are detached prior to shutdown, read up at https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/faq.md#6-minute-delays-in-attaching-volumes.

WRT Cilium and tolerations. Taint already explained at https://karpenter.sh/docs/concepts/disruption/#termination-controller, if you believe it's worth mentioning specific services there, then PRs are open.

[github-actions]

This issue has been inactive for 14 days. StaleBot will close this stale issue after 14 more days of inactivity.