Add IPv4/IPv6 Dual Stack Support

[danehans]

Currently, EG only supports IPv4. For example, an Infra IR proxy listen address represents an IPv4 address. The code should be audited for IPv4-ism's and updated to support a dual-stack IPv4/IPv6 environment properly.

xref: #183 (comment)

https://kubernetes.io/docs/concepts/services-networking/dual-stack/

[Baenimyr]

On my IPv6 cluster, the readiness probe fails.

Readiness probe failed: Get "http://[IPv6]:19001/ready": dial tcp [IPv6]:19001: connect: connection refused

Test

I ran the test from https://gateway.envoyproxy.io/latest/user/http-routing/

Fix

I can patch the deployment to use "::" as the listening address (solution discussed here), but the envoygateway controller tries to restore it.
The fixed replicaset is not deleted because the restored value is never "Ready".

I can replace the bootstrap values thanks to an envoy proxy but I must rewrite the whole value.

Complete solution

Can you give me a EnvoyProxy patch to replace only the wrong values.
It would be nice if the listening address could use a "any IPv4 or IPv6" syntax.

[owenhaynes]

Yeah the KubernetesServiceSpec is a bit limited as you can not even configure the ipFamilyPolicy let alone additional labels so you can not make the exposed enovy service a dual stack.

[Uburro]

Yeah the KubernetesServiceSpec is a bit limited as you can not even configure the ipFamilyPolicy let alone additional labels so you can not make the exposed enovy service a dual stack.

i have same problem

[Uburro]

just for guys who wanna make ipv6 with envoy-gateway and found this issue
pathing Envoy

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  annotations:
  name: only-patch
  namespace: envoy-gateway-system
spec:
  jsonPatches:
  - name: envoy-gateway-system/${GATEWAY_NAME}/${GATEWAY_NAME_PORT}
    operation:
      op: replace
      path: /address
      value:
        socket_address:
          address: "::"
          port_value: 10080
          protocol: TCP
          ipv4_compat: true
    type: type.googleapis.com/envoy.config.listener.v3.Listener
  targetRef:
    group: gateway.networking.k8s.io
    kind: GatewayClass
    name: ${GATEWAY_NAME_CLASS}
    namespace: envoy-gateway-system
  type: JSONPatch

pathing EnvoyProxy

kind: EnvoyProxy
metadata:
  name: test-only-proxy-config
  namespace: envoy-gateway-system
spec:
  provider:
    kubernetes:
      envoyDeployment:
        patch:
          type: StrategicMerge
          value:
            spec:
              template:
                spec:
                  containers:
                  - args:
                    - --service-cluster test-only
                    - --log-level debug
                    - --local-address-ip-version
                    - v6
                    - --cpuset-threads
                    - --service-node $(ENVOY_POD_NAME)
                    - |
                      --config-yaml admin:
                        access_log:
                        - name: envoy.access_loggers.file
                          typed_config:
                            "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                            path: /dev/null
                        address:
                          socket_address:
                            address: 127.0.0.1
                            port_value: 19000
                      layered_runtime:
                        layers:
                        - name: global_config
                          static_layer:
                            envoy.restart_features.use_eds_cache_for_ads: true
                            re2.max_program_size.error_level: 4294967295
                            re2.max_program_size.warn_level: 1000
                      dynamic_resources:
                        ads_config:
                          api_type: DELTA_GRPC
                          transport_api_version: V3
                          grpc_services:
                          - envoy_grpc:
                              cluster_name: xds_cluster
                          set_node_on_first_message_only: true
                        lds_config:
                          ads: {}
                          resource_api_version: V3
                        cds_config:
                          ads: {}
                          resource_api_version: V3
                      static_resources:
                        listeners:
                        - name: envoy-gateway-proxy-ready-0.0.0.0-19001
                          address:
                            socket_address:
                              address: "::"
                              port_value: 19001
                              protocol: TCP
                              ipv4_compat: true
                          filter_chains:
                          - filters:
                            - name: envoy.filters.network.http_connection_manager
                              typed_config:
                                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                                stat_prefix: eg-ready-http
                                route_config:
                                  name: local_route
                                  virtual_hosts:
                                  - name: prometheus_stats
                                    domains:
                                    - "*"
                                    routes:
                                    - match:
                                        prefix: /stats/prometheus
                                      route:
                                        cluster: prometheus_stats
                                http_filters:
                                - name: envoy.filters.http.health_check
                                  typed_config:
                                    "@type": type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck
                                    pass_through_mode: false
                                    headers:
                                    - name: ":path"
                                      string_match:
                                        exact: /ready
                                - name: envoy.filters.http.router
                                  typed_config:
                                    "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
                        clusters:
                        - name: prometheus_stats
                          connect_timeout: 0.250s
                          type: STATIC
                          lb_policy: ROUND_ROBIN
                          load_assignment:
                            cluster_name: prometheus_stats
                            endpoints:
                            - lb_endpoints:
                              - endpoint:
                                  address:
                                    socket_address:
                                      address: 127.0.0.1
                                      port_value: 19000
                        - connect_timeout: 10s
                          load_assignment:
                            cluster_name: xds_cluster
                            endpoints:
                            - load_balancing_weight: 1
                              lb_endpoints:
                              - load_balancing_weight: 1
                                endpoint:
                                  address:
                                    socket_address:
                                      address: envoy-gateway
                                      port_value: 18000
                          typed_extension_protocol_options:
                            envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
                              "@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
                              explicit_http_config:
                                http2_protocol_options:
                                  connection_keepalive:
                                    interval: 30s
                                    timeout: 5s
                          name: xds_cluster
                          type: STRICT_DNS
                          transport_socket:
                            name: envoy.transport_sockets.tls
                            typed_config:
                              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
                              common_tls_context:
                                tls_params:
                                  tls_maximum_protocol_version: TLSv1_3
                                tls_certificate_sds_secret_configs:
                                - name: xds_certificate
                                  sds_config:
                                    path_config_source:
                                      path: "/sds/xds-certificate.json"
                                    resource_api_version: V3
                                validation_context_sds_secret_config:
                                  name: xds_trusted_ca
                                  sds_config:
                                    path_config_source:
                                      path: "/sds/xds-trusted-ca.json"
                                    resource_api_version: V3
                    name: envoy
      envoyService:
        externalTrafficPolicy: Local
        patch:
          value:
            spec:
              ipFamilies:
              - IPv6
              - IPv4
             ipFamilyPolicy: PreferDualStack
        type: LoadBalancer
    type: Kubernetes
    ```

[zirain]

cc @zhlsunshine

[zhlsunshine]

I think you may want some suggestion from me, so @zirain mentioned me. I think envoy already can support dual stack, below is my suggestion:

1. Please refer to Listener for listener configuration in Envoy. IPv4 address can be filled in address field and IPv6 address can be filled in additional_addresses field, then the envoy can automatically pick the correct IP family for you.

2. Please refer to Cluster for cluster configuration in Envoy. It's similar with the Listener configuration.

[arkodg]

thanks for the input @zhlsunshine

this issue requires improvements in the infra and xds layers to support this feature, hoping someone from the community can step in and take this forward

[EraKin575]

I intend to participate in LFX Term 3 for this project proposal. Where can I get started to learn more about this project

[octonawish-akcodes]

Can i get some guidance and supporting docs regarding this issue, I am interested in this issue for LFX

[Ayush9026]

@arkodg sir i am also interested in this issue for LFX Mentorship Term 3 .

[juwon8891]

@arkodg
Hello, I would like to resolve this issue.

[arkodg]

thanks @juwon8891 for applying and being selected for the LFX Mentorship Program for Envoy Gateway
https://github.com/cncf/mentoring/tree/main/programs/lfx-mentorship/2024/03-Sep-Nov#envoy-gateway

looking forward to working with you !

[alessiodionisi]

Waiting this feature to switch an IPv6 first cluster to Envoy Gateway 🔥 I'm available if you need help with code or tests.

[juwon8891]

@alessiodionisi Thank you for your interest in IPv6 support for Envoy Gateway. We're actively improving the code to fully support IPv6 and dual-stack environments. This is an ongoing process, and we're making steady progress.

[juwon8891]

IPv6 우선 클러스터를 Envoy Gateway로 전환하는 이 기능을 기다리고 있습니다 🔥 코드나 테스트에 도움이 필요하면 저에게 연락하세요.

@alessiodionisi Have you tried to connect to ipv6 before? If you have, can you tell me how to reproduce it?

[alessiodionisi]

@juwon8891 I started with these patches #184 (comment) as a base and incorporated @zhlsunshine's suggestions, but I wasn't able to get it working correctly.

[zhlsunshine]

Hi @alessiodionisi, there is a prerequisite by following my suggestion: you need to install Istio with Dual Stack feature enabled. Can you confirm it? BTW, please provide your Istio version, thanks!