AWS Network Load Balancer does not allow multiple certs albeit the functionality existing

[AzySir]

What happened:

• I tried to add two certificates to the NLB and there was only the default certificate that was added. I did this by creating a stringList on the helmChart

What you expected to happen:

• I expected there to be 2 certificates added

NGINX Ingress controller version (exec into the pod and run /nginx-ingress-controller --version):

Helm Chart Version 4.11.3

Kubernetes version (use kubectl version):

• client - v1.31.2
• server v1.30.7-eks-56e63d8

Environment:

• Cloud provider or hardware configuration:

• AWS EKS

• How was the ingress-nginx-controller installed:
  Kustomize:

helmCharts:
  - name: ingress-nginx
    version: 4.11.3
    repo: https://kubernetes.github.io/ingress-nginx
    includeCRDs: true
    releaseName: ingress-nginx
    valuesFile: nginx_ingress_values.yaml
    namespace: networking

How to reproduce this issue:

Use these annotations and add in your own certs

commonLabels: {}

controller:
  name:dev-kubernetes-ingress
  fullnameOverride:dev-kubernetes-ingress
  # defaultBackend:
  service:
    type: LoadBalancer
    ports:
      http: 80
    targetPorts:
      http: http
      https: http
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-name:dev-kubernetes-ingress
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: true
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
      service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-west-2:XXXX:certificate/XXX,arn:aws:acm:eu-west-2:YYYY:certificate/YYYY"
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
      externalTrafficPolicy: Local

Error on svc description

  Normal   EnsuringLoadBalancer    3m20s (x8 over 10m)  service-controller  Ensuring load balancer
  Warning  SyncLoadBalancerFailed  3m19s                service-controller  Error syncing load balancer: failed to ensure load balancer: error updating load balancer listener: "ValidationError: Certificate ARN 'arn:aws:acm:eu-west-2:XXX:certificate/XXXX,arn:aws:acm:eu-west-2:YYYY:certificate/YYYY' is not valid\n\tstatus code: 400, request id: 177e39f2-3195-437d-9f93-066cf99dc449"

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[longwuyuan]

/remove-kind bug

Request that you answer all the questions asked in a new bug report template so readers have a lot more info to comment on.

I am guessing that you are terminating TLS on the LB and not on the controller. If true, then the assumed related data is that the annotations used in the service object created by the controller, that work on the AWS resources, are not part of the controller code. So there is nothing we can do about it. You have to talk to the support team at AWS for multiple certs, configured on the NLB, when TLS is terminated on the NLB.

For users who terminate TLS on the controller, the upstream K8S KEP Ingress-API as defined the spec for multiple certs. kubectl explain ingress.spec.tls

[k8s-ci-robot]

@longwuyuan: Those labels are not set on the issue: kind/bug

In response to this:

/remove-kind bug

Request that you answer all the questions asked in a new bug report template so readers have a lot more info to comment on.

I am guessing that you are terminating TLS on the LB and not on the controller. If true, then the assumed related data is that the annotations used in the service object created by the controller, that work on the AWS resources, are not part of the controller code. So there is nothing we can do about it. You have to talk to the support team at AWS for multiple certs, configured on the NLB, when TLS is terminated on the NLB.

For users who terminate TLS on the controller, the upstream K8S KEP Ingress-API as defined the spec for multiple certs. kubectl explain ingress.spec.tls

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[strongjz]

Which cloud controller is installed in the cluster and what version? Ingress-nginx uses whatever cloud controller is available, I would check if that annotation is supported in the version being used. The 400 is either coming from AWS API or from the cloud controller.