Dashboard says its all healthy, yet Kube API is unreachable

[felipesere]

I've been trying to setup talos on my TuringPi 2 with 2x RK1 nodes using https://github.com/nberlee/talos
which has images that work for TPI2/RK1

From the screenshot you can see that it looks like the controlplane node is happy and ready to go, at least according to the health messages. But if you look at the log output, the EndpointController can't talk to the Kube API server:

user: warning: [2024-08-13T18:43:36.005502323Z]: [talos] kubernetes endpoint watch error {"component": "controller-runtime", "controller":
 "k8s.EndpointController", "error": "failed to list *v1.Endpoints: Get \"https://192.168.1.201:6433/api/v1/namespaces/default/endpoints?
 fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0\": dial tcp 192.168.1.201:6433: connect: connection refused"}

I've also tried to run nmap and can confirm that nothing is listening on 192.168.1.201:

▧  ❯ nmap 192.168.1.201
Starting Nmap 7.95 ( https://nmap.org ) at 2024-08-13 19:49 BST
Nmap scan report for node1 (192.168.1.201)
Host is up (0.10s latency).
Not shown: 998 closed tcp ports (conn-refused)
PORT      STATE SERVICE
50000/tcp open  ibm-db2
50001/tcp open  unknown

Nmap done: 1 IP address (1 host up) scanned in 23.91 seconds

These are the running services:

homelab/talos cluster:admin@callisto @ ns:default on  main ◍◍
▧  ❯ talosctl -n 192.168.1.201 service
NODE            SERVICE      STATE     HEALTH   LAST CHANGE         LAST EVENT
192.168.1.201   apid         Running   OK       21h32m30s ago       Health check successful
192.168.1.201   containerd   Running   OK       478770h52m9s ago    Health check successful
192.168.1.201   cri          Running   OK       21h32m33s ago       Health check successful
192.168.1.201   etcd         Running   OK       21h29m40s ago       Health check successful
192.168.1.201   kubelet      Running   OK       21h32m12s ago       Health check successful
192.168.1.201   machined     Running   OK       478770h52m16s ago   Health check successful
192.168.1.201   syslogd      Running   OK       478770h52m15s ago   Health check successful
192.168.1.201   trustd       Running   OK       21h32m33s ago       Health check successful
192.168.1.201   udevd        Running   OK       478770h52m12s ago   Health check successful

Not sure in which direction to explore further to get my cluster working?

[felipesere]

I find this line surprising from
talosctl -n 192.168.1.201 logs --kubernetes kube-system/kube-apiserver-node1:kube-apiserver:810fa8e0f1a0

192.168.1.201: 2024-08-12T21:23:05.081139301Z stderr F I0812 21:23:05.081033       1 secure_serving.go:213] Serving securely on [::]:6443

Isn't [::] IPv6?

[smira]

It is "all addresses".

[smira]

talosctl ss provides information about listening ports.

Run talosctl health, and in general follow Troubleshooting guide.

[felipesere]

I've tried going through the troubleshoot section but I couldn't spot anything out of the ordinary.

here is the health output.
The dashboard still thinks its all healthy.

[smira]

At this point this is a mystery to me... Either you reconfigured kube-apiserver to listen on a different port, or you have something else going on here.

This doesn't make any sense that kube-apiserver is up, but it doesn't actually listen.

[felipesere]

I got something working by following https://www.youtube.com/watch?v=VKfE5BuqlSc
and at the point of the disks I did switch talos to wipe the disks.
That makes me think I may have had some stale state somewhere?

[miran248]

Isn't the port 6443? Your logs show 6433.

[thedaringdev]

Ha! I some how managed to make the same typo.

[felipesere]

Yeee! I am going to give myself a pass and blame old state 🤣

on the last install I exported IPs as environment variables to avoid typos.

[thedaringdev]

I'm experiencing the same problem actually, multiple re-installs.

[smira]

Every problem can't be same, this issue is about specifying wrong port for the Kubernetes endpoint.