Custom cluster domain

[xeon22]

• EKS 1.23
• BottleRocket AMIs
• v1.1.0

We have a cluster with a customized cluster domain name. The agent insists on connecting to https://brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local/bottlerocket-node-resource which does not resolve in that cluster. Here is the error:

2023-03-23T13:47:39.786059Z ERROR agent::agentclient: error: Unable to operate on BottlerocketShadow: 'Unable to create the BottlerocketShadow associated with this node: 'Unable to create BottlerocketShadow (ip-10-250-82-160.us-east-2.compute.internal, 4be7304a-81c6-491d-b354-9eaac8ea3410): 'error sending request for url (https://brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local/bottlerocket-node-resource): error trying to connect: dns error: failed to lookup address information: Name does not resolve'''
    at agent/src/agentclient.rs:211
    in agent::agentclient::create_metadata_shadow
    in agent::agentclient::create_shadow_if_not_exist
    in agent::agentclient::run

  2023-03-23T13:47:39.786127Z  WARN agent::agentclient: An error occurred when try to create BottlerocketShadow. Restarting event loop.
    at agent/src/agentclient.rs:498
    in agent::agentclient::run

Is there a setting I am missing that can make the domain configurable? Or is this something that will need to change to accommodate?

[gthao313]

Hi @xeon22 Thanks for opening this issue. I don't think we support customized cluster domain name and there is not a setting which can make the domain configurable currently. The error looks like it failed to reach out to brupop api.

Can you provide more information so we can try to reproduce this issue? what cluster domain name you were using and what bottlerocket-update-operator.yaml you were using?

Thanks :)

[xeon22]

I deployed: https://github.com/bottlerocket-os/bottlerocket-update-operator/releases/download/v1.1.0/bottlerocket-update-operator-v1.1.0.yaml

And the custom cluster domain is: ue2.green.aws

If the url is contructed without the FQDN eg. https://brupop-apiserver.brupop-bottlerocket-aws.svc this woudn't be an an issue. Or provide the facility to define the hostname.

Nothing to reproduce. The condition occurs with a vanilla install of the operator in the pre-set namespace in a cluster with a custom domain defined. I went through the code and realized the uri is hardcoded to that default value above.

[cbgbt]

I'm not confident that this is all that's needed, but in order to support this we'd at the very least need to:

• Our self-signed certs need to be created for the correct domain, so modify all of our CA cert templates to allow for configuration via custom values (e.g. these sections)
• Modify the apiserver client implementation to use the same values to derive the services domain here

[xeon22]

The chart can address both items by way of setting an env var for the code to respect the fqdn or otherwise and the cert templates with the correct list of dnsNames to match.