RKE2 HA Not working

[J1a-wei]

Hi team

I installed RKE2 v1.26.13+rke2r1, three master nodes and six workers.
Due to a failure in our data center, two out of the three masters are inaccessible and need to be moved to another data center, which is expected to take one day. Currently, only one master is accessible, but when I try to access the kube-api, it shows a port refusal error. After checking the logs, it appears that the master is not functioning properly.

kubectl get po
E0520 17:28:04.921207 1442154 memcache.go:265] couldn't get current server API group list: Get "https://10.70.24.149:6443/api?timeout=32s": dial tcp 127.0.0.1:6443: connect: connection refused

journalctl rke2-server logs

May 20 14:22:29 rke-master-02 rke2[3550652]: {"level":"warn","ts":"2024-05-20T14:22:29.514516Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00064fdc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
May 20 14:22:29 rke-master-02 rke2[3550652]: {"level":"info","ts":"2024-05-20T14:22:29.51466Z","logger":"etcd-client","caller":"[email protected]/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
May 20 14:22:44 rke-master-02 rke2[3550652]: {"level":"warn","ts":"2024-05-20T14:22:44.516432Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00064fdc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
May 20 14:22:44 rke-master-02 rke2[3550652]: {"level":"info","ts":"2024-05-20T14:22:44.516558Z","logger":"etcd-client","caller":"[email protected]/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
May 20 14:22:59 rke-master-02 rke2[3550652]: {"level":"warn","ts":"2024-05-20T14:22:59.517412Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00064fdc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
May 20 14:22:59 rke-master-02 rke2[3550652]: {"level":"info","ts":"2024-05-20T14:22:59.517459Z","logger":"etcd-client","caller":"[email protected]/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
May 20 14:23:14 rke-master-02 rke2[3550652]: {"level":"warn","ts":"2024-05-20T14:23:14.518694Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00064fdc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
May 20 14:23:14 rke-master-02 rke2[3550652]: {"level":"info","ts":"2024-05-20T14:23:14.51882Z","logger":"etcd-client","caller":"[email protected]/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
May 20 14:23:29 rke-master-02 rke2[3550652]: {"level":"warn","ts":"2024-05-20T14:23:29.520737Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00064fdc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
May 20 14:23:29 rke-master-02 rke2[3550652]: {"level":"info","ts":"2024-05-20T14:23:29.520877Z","logger":"etcd-client","caller":"[email protected]/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
May 20 14:23:44 rke-master-02 rke2[3550652]: {"level":"warn","ts":"2024-05-20T14:23:44.521634Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00064fdc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
May 20 14:23:44 rke-master-02 rke2[3550652]: {"level":"info","ts":"2024-05-20T14:23:44.521771Z","logger":"etcd-client","caller":"[email protected]/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
May 20 14:23:59 rke-master-02 rke2[3550652]: {"level":"warn","ts":"2024-05-20T14:23:59.523563Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00064fdc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
May 20 14:23:59 rke-master-02 rke2[3550652]: {"level":"info","ts":"2024-05-20T14:23:59.52372Z","logger":"etcd-client","caller":"[email protected]/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
May 20 14:24:14 rke-master-02 rke2[3550652]: {"level":"warn","ts":"2024-05-20T14:24:14.525861Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00064fdc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
May 20 14:24:14 rke-master-02 rke2[3550652]: {"level":"info","ts":"2024-05-20T14:24:14.525996Z","logger":"etcd-client","caller":"[email protected]/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
May 20 14:24:29 rke-master-02 rke2[3550652]: {"level":"warn","ts":"2024-05-20T14:24:29.527398Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00064fdc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
May 20 14:24:29 rke-master-02 rke2[3550652]: {"level":"info","ts":"2024-05-20T14:24:29.527526Z","logger":"etcd-client","caller":"[email protected]/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
May 20 14:24:35 rke-master-02 rke2[3550652]: time="2024-05-20T14:24:35Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp $PUBLIC_IP:9345: connect: connection timed out"
May 20 14:24:35 rke-master-02 rke2[3550652]: time="2024-05-20T14:24:35Z" level=error msg="Remotedialer proxy error" error="dial tcp $PUBLIC_IP:9345: connect: connection timed out"
May 20 14:24:40 rke-master-02 rke2[3550652]: time="2024-05-20T14:24:40Z" level=info msg="Connecting to proxy" url="wss://$PUBLIC_IP:9345/v1-rke2/connect"
May 20 14:24:44 rke-master-02 rke2[3550652]: {"level":"warn","ts":"2024-05-20T14:24:44.528185Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00064fdc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
May 20 14:24:44 rke-master-02 rke2[3550652]: {"level":"info","ts":"2024-05-20T14:24:44.528325Z","logger":"etcd-client","caller":"[email protected]/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
May 20 14:24:59 rke-master-02 rke2[3550652]: {"level":"warn","ts":"2024-05-20T14:24:59.530202Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00064fdc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
May 20 14:24:59 rke-master-02 rke2[3550652]: {"level":"info","ts":"2024-05-20T14:24:59.530312Z","logger":"etcd-client","caller":"[email protected]/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
May 20 14:25:14 rke-master-02 rke2[3550652]: {"level":"warn","ts":"2024-05-20T14:25:14.532337Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00064fdc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
May 20 14:25:14 rke-master-02 rke2[3550652]: {"level":"info","ts":"2024-05-20T14:25:14.532455Z","logger":"etcd-client","caller":"[email protected]/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
May 20 14:25:29 rke-master-02 rke2[3550652]: {"level":"warn","ts":"2024-05-20T14:25:29.533388Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00064fdc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
May 20 14:25:29 rke-master-02 rke2[3550652]: {"level":"info","ts":"2024-05-20T14:25:29.533528Z","logger":"etcd-client","caller":"[email protected]/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
May 20 14:25:44 rke-master-02 rke2[3550652]: {"level":"warn","ts":"2024-05-20T14:25:44.535264Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00064fdc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
May 20 14:25:44 rke-master-02 rke2[3550652]: {"level":"info","ts":"2024-05-20T14:25:44.535413Z","logger":"etcd-client","caller":"[email protected]/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
May 20 14:25:59 rke-master-02 rke2[3550652]: {"level":"warn","ts":"2024-05-20T14:25:59.536581Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00064fdc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
May 20 14:25:59 rke-master-02 rke2[3550652]: {"level":"info","ts":"2024-05-20T14:25:59.536776Z","logger":"etcd-client","caller":"[email protected]/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
May 20 14:26:14 rke-master-02 rke2[3550652]: {"level":"warn","ts":"2024-05-20T14:26:14.538702Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00064fdc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
May 20 14:26:14 rke-master-02 rke2[3550652]: {"level":"info","ts":"2024-05-20T14:26:14.538822Z","logger":"etcd-client","caller":"[email protected]/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
May 20 14:26:29 rke-master-02 rke2[3550652]: {"level":"warn","ts":"2024-05-20T14:26:29.539735Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00064fdc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
May 20 14:26:29 rke-master-02 rke2[3550652]: {"level":"info","ts":"2024-05-20T14:26:29.539912Z","logger":"etcd-client","caller":"[email protected]/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
May 20 14:26:44 rke-master-02 rke2[3550652]: {"level":"warn","ts":"2024-05-20T14:26:44.54215Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00064fdc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}

How should I handle this situation?

[brandond]

I installed RKE2 v1.26.13+rke2r1, three master nodes and six workers.
Due to a failure in our data center, two out of the three masters are inaccessible

etcd requires a majority of nodes be available for quorum. 1/3 nodes does not meet quorum; you'll need to get at least one more server node online.
Ref: https://etcd.io/docs/v3.5/faq/#what-is-failure-tolerance