Skip to content

Latest commit

 

History

History
393 lines (363 loc) · 9.72 KB

aws-public-private-route53.md

File metadata and controls

393 lines (363 loc) · 9.72 KB
Raw

AWS Route53 with same domain for public and private zones

This tutorial describes how to setup ExternalDNS using the same domain for public and private Route53 zones and nginx-ingress-controller. It also outlines how to use cert-manager to automatically issue SSL certificates from Let's Encrypt for both public and private records.

Deploy public nginx-ingress-controller

You may be interested with GKE with nginx ingress for installation guidelines.

Specify ingress-class in nginx-ingress-controller container args:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: external-ingress
  name: external-ingress-controller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: external-ingress
  template:
    metadata:
      labels:
        app: external-ingress
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
        - --configmap=$(POD_NAMESPACE)/external-ingress-configuration
        - --tcp-services-configmap=$(POD_NAMESPACE)/external-tcp-services
        - --udp-services-configmap=$(POD_NAMESPACE)/external-udp-services
        - --annotations-prefix=nginx.ingress.kubernetes.io
        - --ingress-class=external-ingress
        - --publish-service=$(POD_NAMESPACE)/external-ingress
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.11.0
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: external-ingress-controller
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1

Set type: LoadBalancer in your public nginx-ingress-controller Service definition.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
  labels:
    app: external-ingress
  name: external-ingress
spec:
  externalTrafficPolicy: Cluster
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app: external-ingress
  sessionAffinity: None
  type: LoadBalancer

Deploy private nginx-ingress-controller

Make sure to specify ingress-class in nginx-ingress-controller container args:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: internal-ingress
  name: internal-ingress-controller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: internal-ingress
  template:
    metadata:
      labels:
        app: internal-ingress
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
        - --configmap=$(POD_NAMESPACE)/internal-ingress-configuration
        - --tcp-services-configmap=$(POD_NAMESPACE)/internal-tcp-services
        - --udp-services-configmap=$(POD_NAMESPACE)/internal-udp-services
        - --annotations-prefix=nginx.ingress.kubernetes.io
        - --ingress-class=internal-ingress
        - --publish-service=$(POD_NAMESPACE)/internal-ingress
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.11.0
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: internal-ingress-controller
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1

Set additional annotations in your private nginx-ingress-controller Service definition to create an internal load balancer.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
    service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
  labels:
    app: internal-ingress
  name: internal-ingress
spec:
  externalTrafficPolicy: Cluster
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app: internal-ingress
  sessionAffinity: None
  type: LoadBalancer

Deploy the public zone ExternalDNS

Consult AWS ExternalDNS setup docs for installation guidelines.

In ExternalDNS containers args, make sure to specify aws-zone-type and ingress-class:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: external-dns-public
  name: external-dns-public
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: external-dns-public
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: external-dns-public
    spec:
      containers:
      - args:
        - --source=ingress
        - --provider=aws
        - --registry=txt
        - --txt-owner-id=external-dns
        - --ingress-class=external-ingress
        - --aws-zone-type=public
        image: registry.k8s.io/external-dns/external-dns:v0.15.1
        name: external-dns-public

Deploy the private zone ExternalDNS

Consult AWS ExternalDNS setup docs for installation guidelines.

In ExternalDNS containers args, make sure to specify aws-zone-type and ingress-class:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: external-dns-private
  name: external-dns-private
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: external-dns-private
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: external-dns-private
    spec:
      containers:
      - args:
        - --source=ingress
        - --provider=aws
        - --registry=txt
        - --txt-owner-id=dev.k8s.nexus
        - --ingress-class=internal-ingress
        - --aws-zone-type=private
        image: registry.k8s.io/external-dns/external-dns:v0.15.1
        name: external-dns-private

Create application Service definitions

For this setup to work, you need to create two Ingress definitions for your application.

At first, create a public Ingress definition:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app: app
  name: app-public
spec:
  ingressClassName: external-ingress
  rules:
  - host: app.domain.com
    http:
      paths:
      - backend:
          service:
            name: app
            port:
              number: 80
        pathType: Prefix

Then create a private Ingress definition:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app: app
  name: app-private
spec:
  ingressClassName: internal-ingress
  rules:
  - host: app.domain.com
    http:
      paths:
      - backend:
          service:
            name: app
            port:
              number: 80
        pathType: Prefix

Additionally, you may leverage cert-manager to automatically issue SSL certificates from Let's Encrypt. To do that, request a certificate in public service definition:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/acme-challenge-type: "dns01"
    certmanager.k8s.io/acme-dns01-provider: "route53"
    certmanager.k8s.io/cluster-issuer: "letsencrypt-production"
    kubernetes.io/tls-acme: "true"
  labels:
    app: app
  name: app-public
spec:
  ingressClassName: "external-ingress"
  rules:
  - host: app.domain.com
    http:
      paths:
      - backend:
          service:
            name: app
            port:
              number: 80
        pathType: Prefix
  tls:
  - hosts:
    - app.domain.com
    secretName: app-tls

And reuse the requested certificate in private Service definition:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app: app
  name: app-private
spec:
  ingressClassName: "internal-ingress"
  rules:
  - host: app.domain.com
    http:
      paths:
      - backend:
          service:
            name: app
            port:
              number: 80
        pathType: Prefix
  tls:
  - hosts:
    - app.domain.com
    secretName: app-tls