kubernetes pod after upgrade EKS to 1.26

[Awot83]

What happened?

After upgrading EKS cluster to 1.26, the pod of dashboard is in error : Back-off restarting failed container / crashLoopBackoff
the EC2 (compute) is ok,
how can i trouble shoot ?

logs of the pod :
goroutine 1 [running]:
github.com/kubernetes/dashboard/src/app/backend/client/csrf.(*csrfTokenManager).init(0xc00051fae8)
/home/runner/work/dashboard/dashboard/src/app/backend/client/csrf/manager.go:41 +0x30e
github.com/kubernetes/dashboard/src/app/backend/client/csrf.NewCsrfTokenManager(...)
/home/runner/work/dashboard/dashboard/src/app/backend/client/csrf/manager.go:66
github.com/kubernetes/dashboard/src/app/backend/client.(*clientManager).initCSRFKey(0xc000166180)
/home/runner/work/dashboard/dashboard/src/app/backend/client/manager.go:527 +0x94
github.com/kubernetes/dashboard/src/app/backend/client.(*clientManager).init(0x19a62f9?)
/home/runner/work/dashboard/dashboard/src/app/backend/client/manager.go:495 +0x32
github.com/kubernetes/dashboard/src/app/backend/client.NewClientManager(...)
/home/runner/work/dashboard/dashboard/src/app/backend/client/manager.go:594
main.main()
/home/runner/work/dashboard/dashboard/src/app/backend/dashboard.go:96 +0x1cf
2024/11/12 14:53:33 Starting overwatch
2024/11/12 14:53:33 Using namespace: kubernetes-dashboard
2024/11/12 14:53:33 Using in-cluster config to connect to apiserver
2024/11/12 14:53:33 Using secret token for csrf signing
2024/11/12 14:53:33 Initializing csrf token from kubernetes-dashboard-csrf secret
panic: Get "https://XXXXXX/api/v1/namespaces/kubernetes-dashboard/secrets/kubernetes-dashboard-csrf": dial tcp XXXXX: connect: no route to host

What did you expect to happen?

Kubernetes Dashboard is accessible (pod is ok)

How can we reproduce it (as minimally and precisely as possible)?

upgrade cluster control plan from 1.24 to 1.26

Anything else we need to know?

helm.sh/chart=kubernetes-dashboard-6.0.8

What browsers are you seeing the problem on?

No response

Kubernetes Dashboard version

helm.sh/chart=kubernetes-dashboard-6.0.8

Kubernetes version

1.26

Dev environment

No response

[floreks]

Please use the latest released chart.

[Awot83]

hello, i did the upgrade : now i am in v2.7.10
but i have this error

Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service account's configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get "[https://172.20.0.1:443/version](https://172.20.0.1/version)": dial tcp [172.20.0.1:443](https://172.20.0.1/): connect: no route to host

Refer to our FAQ and wiki pages for more information: https://github.com/kubernetes/dashboard/wiki/FAQ

the cluster is a EKS upgraded to 1.26, what misconfiguration could it be ?

[floreks]

hello, i did the upgrade : now i am in v2.7.10

There is no such version available.

[Awot83]

sorry my mistake : please find the "helm show chart kubernetes-dashboard/kubernetes-dashboard" result :
(it's version: 7.10.0)

apiVersion: v2
dependencies:

• alias: nginx
  condition: nginx.enabled
  name: ingress-nginx
  repository: https://kubernetes.github.io/ingress-nginx
  version: 4.10.1
• condition: cert-manager.enabled
  name: cert-manager
  repository: https://charts.jetstack.io
  version: v1.14.5
• condition: metrics-server.enabled
  name: metrics-server
  repository: https://kubernetes-sigs.github.io/metrics-server/
  version: 3.12.1
• condition: kong.enabled
  name: kong
  repository: https://charts.konghq.com
  version: 2.38.0
  description: General-purpose web UI for Kubernetes clusters
  home: https://github.com/kubernetes/dashboard
  icon: https://raw.githubusercontent.com/kubernetes/kubernetes/master/logo/logo.svg
  keywords:
• kubernetes
• dashboard
  kubeVersion: '>=1.21.0-0'
  maintainers:
• email: [email protected]
  name: desaintmartin
• email: [email protected]
  name: floreks
  name: kubernetes-dashboard
  sources:
• https://github.com/kubernetes/dashboard
  version: 7.10.0

[Awot83]

since the update the pods are running

• kubernetes-dashboard-api-588dfb6895-9g46g
• kubernetes-dashboard-auth-68fd5ff854-t4jmz
• kubernetes-dashboard-kong-65476f87d4-gmv2x
• kubernetes-dashboard-web-584f878c66-hjsjw

but i have no ingress for kubernetes-dashboard
despite the fact i used a value.yaml with an ingress specified :

ingress:
  ## If true, Kubernetes Dashboard Ingress will be created.
  ##
  enabled: true

  ## Kubernetes Dashboard Ingress labels
  # labels:
  #   key: value

  ## Kubernetes Dashboard Ingress annotations
  annotations:
   kubernetes.io/ingress.class: kong
  #   kubernetes.io/ingress.class: nginx
  #   kubernetes.io/tls-acme: 'true'

  ## If you plan to use TLS backend with enableInsecureLogin set to false
  ## (default), you need to uncomment the below.
  ## If you use ingress-nginx < 0.21.0
  #   nginx.ingress.kubernetes.io/secure-backends: "true"
  ## if you use ingress-nginx >= 0.21.0
  #   nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

  ## Kubernetes Dashboard Ingress Class
  # className: "example-lb"
  ingressClassName: kong
  ## Kubernetes Dashboard Ingress paths
  ## Both `/` and `/*` are required to work on gce ingress.
  # paths:
  #  - /
  #  - /*

  ## Custom Kubernetes Dashboard Ingress paths. Will override default paths.
  ##
  customPaths:
  #  - pathType: ImplementationSpecific
    - backend:
       service:
        name: kubernetes-dashboard
        port:
         number: 443
 #          name: use-annotation
      pathType: ImplementationSpecific
  #    backend:
  #      service:
  #        name: >-
  #          {{ include "kubernetes-dashboard.fullname" . }}
  #        port:
  #          # Don't use string here, use only integer value!
  #          number: 443
  ## Kubernetes Dashboard Ingress hostnames
  ## Must be provided if Ingress is enabled
  ##
  hosts:
    - k8s-dashboard.production.xxxxxxx

could you please tell me why i have no ingress here ?

[floreks]

Do you mean that the actual ingress resource hasn't been created in the kubernetes cluster and is not visible via kubectl -n <namespace> get ing?

[Awot83]

hello, yes exactly :

kubectl -n kubernetes-dashboard get ing
No resources found in kubernetes-dashboard namespace.

[Awot83]

before the upgrade of the dashboard in v7.10.0 i used a value file with the url i want to access the dashboard to in "host"
since the upgrade the ingress is not created,
how can i access to the dasboard with this url now ?
do i have to modify the value file and add this section (that was not in my value file before) ?:

kong:  
    proxy:
      type: NodePort
    http:
      enabled: true 

My old value file in the ingress section is :

ingress:
  ## If true, Kubernetes Dashboard Ingress will be created.
  ##
  enabled: true

  ## Kubernetes Dashboard Ingress labels
  # labels:
  #   key: value

  ## Kubernetes Dashboard Ingress annotations
  annotations:
   kubernetes.io/ingress.class: kong
  #   kubernetes.io/ingress.class: nginx
  #   kubernetes.io/tls-acme: 'true'

  ## If you plan to use TLS backend with enableInsecureLogin set to false
  ## (default), you need to uncomment the below.
  ## If you use ingress-nginx < 0.21.0
  #   nginx.ingress.kubernetes.io/secure-backends: "true"
  ## if you use ingress-nginx >= 0.21.0
  #   nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

  ## Kubernetes Dashboard Ingress Class
  # className: "example-lb"
  ingressClassName: kong
  ## Kubernetes Dashboard Ingress paths
  ## Both `/` and `/*` are required to work on gce ingress.
  # paths:
  #  - /
  #  - /*

  ## Custom Kubernetes Dashboard Ingress paths. Will override default paths.
  ##
  customPaths:
  #  - pathType: ImplementationSpecific
    - backend:
       service:
        name: kubernetes-dashboard
        port:
         number: 443
 #          name: use-annotation
      pathType: ImplementationSpecific
  #    backend:
  #      service:
  #        name: >-
  #          {{ include "kubernetes-dashboard.fullname" . }}
  #        port:
  #          # Don't use string here, use only integer value!
  #          number: 443
  ## Kubernetes Dashboard Ingress hostnames
  ## Must be provided if Ingress is enabled
  ##
  hosts:
    **- k8s-dashboard.xxxxx.xxxxxxxx.xx =>  the URL i want to access the dashboard** 
  ## Kubernetes Dashboard Ingress TLS configuration
  ## Secrets must be manually created in the namespace
  ##
  # tls:
  #   - secretName: kubernetes-dashboard-tls
  #     hosts:
  #       - kubernetes-dashboard.domain.com

# Global dashboard settings

[Awot83]

Hi
I will describe how i fix this .
I used the last value file .
And modified : Kong http to true

kong:
  proxy:
    http:
      enabled: **true**

Then i adjusted the Ingress created with port number to 80 :

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dashboard-ingress
spec:
  rules:
  - host: <your-hostname>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard-kong-proxy
            port:
              **number: 80**
  tls:
  - hosts:
    - my-hostname 
    secretName: <your-tls-secret>

Then i was able to login to dashboard with the token .

So why do we need to be in http between ingress and kong-proxy to make it work i dont know..