Unable to discover AMI by alias

[maxsxu]

Description

Observed Behavior:

{"level":"ERROR","time":"2024-12-19T15:32:00.620Z","logger":"controller","message":"Reconciler error","commit":"3298d91","controller":"nodeclass.status","controllerGroup":"karpenter.k8s.aws","controllerKind":"EC2NodeClass","EC2NodeClass":{"name":"default"},"namespace":"","name":"default","reconcileID":"a7f7f4f5-0c9b-4c29-a11f-263dafa81493","error":"getting amis, getting AMI queries, failed to discover AMIs for alias \"al2023@v20241213\""}

Expected Behavior:

Able to discover AMI by alias.

Reproduction Steps (Please include YAML):

Using following EC2NodeÇlass will cause this issue:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  role: test-cluster-ng-role
  amiSelectorTerms:
    - alias: al2023@v20241213   # Or al2023@latest
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "test-cluster"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "test-cluster"

Will result following status

status:
  conditions:
  - lastTransitionTime: "2024-12-19T17:58:19Z"
    message: object is awaiting reconciliation
    observedGeneration: 1
    reason: AwaitingReconciliation
    status: Unknown
    type: AMIsReady
  - lastTransitionTime: "2024-12-19T17:58:19Z"
    message: ""
    observedGeneration: 1
    reason: SubnetsReady
    status: "True"
    type: SubnetsReady
  - lastTransitionTime: "2024-12-19T17:58:19Z"
    message: ""
    observedGeneration: 1
    reason: SecurityGroupsReady
    status: "True"
    type: SecurityGroupsReady
  - lastTransitionTime: "2024-12-19T17:58:19Z"
    message: ""
    observedGeneration: 1
    reason: InstanceProfileReady
    status: "True"
    type: InstanceProfileReady
  - lastTransitionTime: "2024-12-19T17:58:19Z"
    message: AMIsReady=Unknown
    observedGeneration: 1
    reason: ReconcilingDependents
    status: Unknown
    type: Ready
  instanceProfile: test-cluster_4961034478281494142
  securityGroups:
  - id: sg-0acaf62b6e189b666
    name: test-cluster-node-20241218154546808000000001
  subnets:
  - id: subnet-0e4f85f692b99ffff
    zone: us-west-1c
    zoneID: usw1-az3
  - id: subnet-0494a6c43245d6666
    zone: us-west-1a
    zoneID: usw1-az1

The workaround for me is specifying the AMI ID and AMI Family with following spec:

spec:
  amiFamily: AL2023
  amiSelectorTerms:
  - id: ami-0784a08b412b69c00

Versions:

• Chart Version: 1.1.1
• Kubernetes Version (kubectl version): 1.30.7

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jmdeal]

Are you able to check your CloudTrail logs for failed GetParameter calls made by Karpenter? Can you also verify that you have sufficient permissions for ssm:GetParameter on your KarpenterControllerPolicy (ref)?

[maxsxu]

Thanks @jmdeal 👍 I've fixed it and the root cause is my Karpenter role is restricted to perform ssm actions by a permissions boundary.

And I suggest append the the permission errors to the controller log, so that we can better know what happened.

[jmdeal]

Agreed, they shouldn't omitted. Do you want to update the title to reflect the issue, e.g. something along the lines of "authorization failures aren't provided in logs for amiSelectorTerm aliases"?