-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Linkerd Multi-Cluster service-mirroring to give option to mirror EndpointSlices as well #12660
Comments
Hi @ryanhristovski, thanks for the write-up. I suppose this is in relation to the |
Hi @mateiidavid yes exactly - we're currently not able to use multi-cluster Linkerd due to this limitation but I can definitely give context on how we would like to use it. We're hoping to use the multi-cluster extension mostly for mTLS & cross-cluster service discovery (using endpointslices, the way envoy gateway currently does SD by default). Since we're using Envoy, we intend to use it's load balancing capabilities for upstream and don't really need it for the proxy aspect - just mTLS & cross-cluster on the ingress services. We hope for linkerd to behave in the same way it would for an external proxy using headless services, where the load balancing is managed by the external proxy and not linkerd proxy. Below is a quick diagram to help describe how we would hope to use it. The main blocker now is that we won't see ingress metrics for each backend. Since we don't rely on Linkerd for the proxying aspect, not having observability into which backends Envoy has up is debilitating. |
Hey @ryanhristovski! After kicking this around with the team, this may not be a thing that's really possible. EndpointSlice resources are structurally different from Endpoints resources, obviously. Where this matters is that Endpoints only contains IP addresses and port numbers, but each endpoint in an EndpointSlice is required to have a hostname and a targetRef. Unfortunately, the targetRef cannot point to a target that's not in the same cluster -- and Linkerd actually uses the targetRef for identity, so we can't just point them all to a Link object or something. The end result is that mirroring the EndpointSlices turns out be fiendishly difficult; extending Envoy Gateway to be able to use the Linkerd control plane for discovery is probably actually simpler. Also, Envoy Gateway v1.1 should include the ability to have Envoy Gateway route to Service IPs rather than directly to endpoints (see envoyproxy/gateway#3543), which may be another option here. I'm going to go ahead and close this one as a thing without a great answer; let me know if you think there's more to it than that. |
What problem are you trying to solve?
When meshing an ingress pod (in our case, Envoy Gateway) which handles large amounts of requests, we want our proxy to handle load-balancing and to see each individual backend, aka endpoint, rather than just ClusterIP it is connected to. As of now, Linkerd is handling all of the upstream load-balancing and active backends are only seen through Linkerd metrics. This takes away critical observability and load balancing from our proxy layer.
We want the option to run Linkerd on idle for our ingress proxies and use it strictly for service-discovery and mTLS.
How should the problem be solved?
Linkerd multi-cluster service-mirroring is currently only mirroring the ClusterIP of each service, and then the Linkerd proxy side car is handling the load balancing.
Linkerd should also mirror the
EndpointSlice
(a more scalable and extensible alternative toEndpoint
) so proxies can handle their own load-balancing and also have more visibility into their backends.Any alternatives you've considered?
Using an alternative service mirroring tool or building out our own service discovery.
How would users interact with this feature?
Potentially by adding a label to services that are to be mirrored to include endpointslice like so:
kubectl label svc foobar mirror.linkerd.io/endpointslices=true
Would you like to work on this feature?
maybe
The text was updated successfully, but these errors were encountered: