Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The periodic change in MACAddressPolicy is causing network disruptions between pods. #3155

Open
peaaceChoi opened this issue Dec 17, 2024 · 1 comment

Comments

@peaaceChoi
Copy link

peaaceChoi commented Dec 17, 2024

What happened:
It seems that the MACAddressPolicy is periodically changing or being updated on my Ubuntu 22.04 cluster.

When I create the 22.04 image, I set the file /usr/lib/systemd/network/99-default.link to MACAddressPolicy=none and generate the image.

However, after a certain period of time (24 hours?), the value changes back to MACAddressPolicy=persistent. Strangely, even though the value has changed, the date still points to 2022.

-rw-r--r-- 1 root root 499 Mar 11 2022 /usr/lib/systemd/network/99-default.link
Is there a specific systemd daemon that might be changing this value?

As a solution, I can set the �file with a higher priority, but I am curious about why it is being changed.

[Match]
OriginalName=*

[Link]
NamePolicy=keep kernel database onboard slot path
AlternativeNamesPolicy=database onboard slot path
MACAddressPolicy=persistent

thank you.

Environment:

  • kubernetes:
    Client Version: v1.30.4
    Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
    Server Version: v1.30.4
  • cni는 amazon-k8s-cni-init:v1.11.4 : amazon-k8s-cni-init:v1.11.4
  • OS (e.g: cat /etc/os-release): "Ubuntu 22.04.1 LTS"
  • Kernel (e.g. uname -a): 5.15.0-47-generic
@orsenthil
Copy link
Member

It seems that the MACAddressPolicy is periodically changing or being updated on my Ubuntu 22.04 cluster.

This is strange.

Is there a specific systemd daemon that might be changing this value?

CNI does not introduce any systemd daaemon that can change this. Perhaps it is some other dependency?

We have this information in our troubleshooting guide tough.

- **systemd-udev** - Linux distributions that install the `systemd-udev` package create `/usr/lib/systemd/network/99-default.link` with `Link.MACAddressPolicy` set to `persistent`.
This policy may cause the MAC address assigned to the host veth interface for a pod to change after the interface is moved to the host network namespace. The CNI plugin installs a static ARP binding for the default gateway in the pod network namespace pointing to the host veth MAC, so the MAC changing leads to pod connectivity issues.
The workaround for this issue is to set `MACAddressPolicy=none`, as shown [here](https://github.com/aws/amazon-vpc-cni-k8s/issues/2103#issuecomment-1321698870). This issue is known to affect Ubuntu 22.04+, and long-term solutions are being evaluated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants