NATS Operator and K8S v1.20 compatibility

[wallyqs]

Users have reported that some of the features from the operator like service roles have stopped working on K8S 1.20

E0527 15:18:55.677376       1 generic.go:108] error syncing "nats-io/nats-cluster": failed to reconcile pods: Operation cannot be fulfilled on secrets "nats-box-pod-nats-cluster-bound-token": StorageError: invalid object, Code: 4, Key: /registry/secrets/app/nats-box-pod-nats-cluster-bound-token, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: a60354a1-6a3d-40d2-a370-61b3c93c362b, UID in object meta:
E0527 15:18:56.210752       1 generic.go:108] error syncing "nats-io/nats-cluster": failed to update auth data in config secret: secrets "nats-box-pod-nats-cluster-bound-token" not found

[rayjanoka]

I can reproduce this when using the ServiceAccounts feature.

Install k8s version 1.20.0

minikube start --kubernetes-version=v1.20.0 \
          --extra-config=apiserver.service-account-signing-key-file=/var/lib/minikube/certs/sa.key \
          --extra-config=apiserver.service-account-key-file=/var/lib/minikube/certs/sa.pub \
          --extra-config=apiserver.service-account-issuer=api \
          --extra-config=apiserver.service-account-api-audiences=api,spire-server \
          --extra-config=apiserver.authorization-mode=Node,RBAC \
          --extra-config=kubelet.authentication-token-webhook=true

Install operator and example deployment

curl -sSL https://raw.githubusercontent.com/nats-io/nats-operator/master/example/nats-operator-cluster-scoped.yaml | kubectl apply -f -
curl -sSL https://raw.githubusercontent.com/nats-io/nats-operator/master/example/nats-operator-cluster-scoped-rbac.yaml | kubectl apply -f -
echo "sleeping for 15s..."; sleep 15
curl -sSL https://raw.githubusercontent.com/nats-io/nats-operator/master/example/example-svc-accounts-diff-namespaces.yaml | kubectl apply -f -

No bound-tokens are created on k8s 1.20.0

➜ k get secret -A | grep -E "nats|app-ns"
my-admin-app-ns   default-token-qlh8q                              kubernetes.io/service-account-token   3      2m2s
my-admin-app-ns   nats-admin-user-token-6splc                      kubernetes.io/service-account-token   3      2m2s
my-app-ns         default-token-ms9bg                              kubernetes.io/service-account-token   3      2m2s
my-app-ns         nats-user-token-8qngh                            kubernetes.io/service-account-token   3      2m2s
nats-io           default-token-5fhj5                              kubernetes.io/service-account-token   3      2m53s
nats-io           nats-operator-token-8n64n                        kubernetes.io/service-account-token   3      2m52s
nats-io           nats-server-token-tmb7m                          kubernetes.io/service-account-token   3      2m52s
nats-system       default-token-8wdtz                              kubernetes.io/service-account-token   3      2m2s

Using minikube k8s version 1.19.10 this same procedure works, the bound-tokens are created

➜ k get secret -A | grep -E "nats|app-ns"
my-admin-app-ns   default-token-tg7cd                              kubernetes.io/service-account-token   3      16s
my-admin-app-ns   nats-admin-user-nats-cluster-bound-token         Opaque                                1      11s
my-admin-app-ns   nats-admin-user-token-4rs9d                      kubernetes.io/service-account-token   3      16s
my-app-ns         default-token-dwhjd                              kubernetes.io/service-account-token   3      16s
my-app-ns         nats-user-nats-cluster-bound-token               Opaque                                1      16s
my-app-ns         nats-user-token-jnqnt                            kubernetes.io/service-account-token   3      16s
nats-io           default-token-s5lwr                              kubernetes.io/service-account-token   3      111s
nats-io           nats-operator-token-pzcbx                        kubernetes.io/service-account-token   3      111s
nats-io           nats-server-token-z59vw                          kubernetes.io/service-account-token   3      111s
nats-system       default-token-vm8wm                              kubernetes.io/service-account-token   3      16s
nats-system       nats-cluster                                     Opaque                                1      16s

It would be nice if there was a guide to enable debug on the operator because there are hardly any logs to view.

Thanks!

[jeesmon]

We are looking for a fix to this. Any idea if anyone looking into fixing it? Thanks.