eks with secondary network for pods

[davirezendegb]

Following the documentation, we end up in the following scenario.
https://aws.github.io/aws-eks-best-practices/networking/custom-networking/

The pod receives the IPs from the secondary subnet and to try to access resources outside the VPC, the main IP of the node in the main network is used, but for resources in the same VPC it tries to use the secondary IP that the pod receives. Is there a way to mask the pod IP to the main IP for internal resources in the same VPC?

pod ip: 100.127.35.58
node ip: 192.168.1.44

when trying to access an RDS resource in the same vpc the pod uses the ip 100.127.35.58

to access a resource outside the vpc a masking is done so that the pod uses the node ip 192.168.1.44

We would like this masking to also be done for internal resources in the same VPC

[orsenthil]

Is there a way to mask the pod IP to the main IP for internal resources in the same VPC?

No, there is not a way to mask the pod ip to the main ip (or node ip) of internal resources in the same VPC.

If you use the hostnetworking pod, then the ip of the pod is the same as the node's primary ip.

[davirezendegb]

do we not have the possibility to create a function for this?
"AWS_VPC_K8S_CNI_INTERNALSNAT=true"

[orsenthil]

Do you mean, create a new flag to have an Internal SNAT? I am not sure how that will work with the property of hostnetwork pods.

[davirezendegb]

the intention is to make the pods receive the ips of the secondary subnet as expected, but to communicate internally use the ip of the main subnet that is also tied to the node

for example, the pods receive the range 100.127.xxx.xxx and intermanete they use this ip 100.127.xxx.xxx to communicate with an RDS.

the idea is to use the primary ip of the node: 10.72.xxx.xxx (this occurs for when the resource is outside the vpc)

[davirezendegb]

in GCP and azure there is this way of performing this configuration

https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent

https://github.com/Azure/ip-masq-agent-v2

[orsenthil]

@davirezendegb - can you raise this feature request containers-roadmap. It needs to be evaluated and prioritized as a product requirement for CNI.

[arehmandev]

Hi this is a blocker for us too, we want to use a seperate subnet for pods whilst maintaining routing for nodes via internalsnat

This was very difficult to debug as the documentation was lacking. This is also a feature GCP has documented well for GKE (via ipmasqagent) - its surprising to not see this in EKS.