Error retrieving credentials for long Kaniko builds

[maxf9]

Kaniko leverages the ECR Credentials Helper under the hood to retrieve AWS credentials for authenticating on ECR and pushing Docker images.

I run Kaniko builds by using a private Gitlab CI runner that deploys CI jobs on EKS Fargate. The Kaniko CI jobs leverage the IAM role specified in the default AWS profile, whereas the build image has to be pushed to the private ECR repo in the other account (see AWS profile ecr).

The contents of the ~/.aws/config file:

[default]
region=eu-central-1
role_arn=arn:aws:iam::build_account_id:role/gitlab-runner-testing
duration_seconds=1800
web_identity_token_file=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

[profile ecr]
region=eu-central-1
role_arn=arn:aws:iam::repo_account_id:role/CiPipelineAccessRole
duration_seconds=1800
source_profile=default

So ECR Credentials Helper should use the credentials obtained from the default AWS profile to assume the IAM role arn:aws:iam::repo_account_id:role/CiPipelineAccessRole in the repo_account_id account where the ECR repo resides.

Kaniko is configured to use AWS credentials provided by the ECR credentionals helper. Check out the file /kaniko/.docker/config.json:

{"credHelpers":{"repo_account_id.dkr.ecr.eu-central-1.amazonaws.com":"ecr-login"}}

Kaniko builds work fine when the image build time is relatively short (for example, 3 minutes). ECR credentials helper assumes the required IAM role arn:aws:iam::repo_account_id:role/CiPipelineAccessRole, successfully logs into ECR in the repo_account_id AWS account.

time="2023-10-06T05:32:54Z" level=debug msg="Retrieving credentials" region=eu-central-1 registry=repo_account_id serverURL=repo_account_id.dkr.ecr.eu-central-1.amazonaws.com service=ecr
time="2023-10-06T05:32:54Z" level=debug msg="Checking file cache" registry=repo_account_id
time="2023-10-06T05:32:54Z" level=debug msg="Calling ECR.GetAuthorizationToken" registry=repo_account_id
time="2023-10-06T05:32:54Z" level=debug msg="Saving credentials to file cache" registry=repo_account_id service=ecr
time="2023-10-06T05:33:26Z" level=debug msg="Retrieving credentials" region=eu-central-1 registry=repo_account_id serverURL=repo_account_id.dkr.ecr.eu-central-1.amazonaws.com service=ecr
time="2023-10-06T05:33:26Z" level=debug msg="Checking file cache" registry=repo_account_id
time="2023-10-06T05:33:26Z" level=debug msg="Calling ECR.GetAuthorizationToken" registry=repo_account_id
time="2023-10-06T05:33:26Z" level=debug msg="Saving credentials to file cache" registry=repo_account_id service=ecr

However when the image build time takes longer (for example, 15 minutes), Kaniko becomes unable to push images to ECR. The reason is: ECR Credentials Helper cannot retrieve AWS credentials to log into ECR in the ECR repo account.

time="2023-10-06T06:41:29Z" level=debug msg="Could not fetch credentials for cache prefix, disabling cache" error="failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, access disabled to EC2 IMDS via client option, or \"AWS_EC2_METADATA_DISABLED\" environment variable"
time="2023-10-06T06:41:29Z" level=debug msg="Retrieving credentials" region=eu-central-1 registry=repo_account_id serverURL=repo_account_id.dkr.ecr.eu-central-1.amazonaws.com service=ecr
time="2023-10-06T06:41:29Z" level=debug msg="Calling ECR.GetAuthorizationToken" registry=repo_account_id
time="2023-10-06T06:41:29Z" level=error msg="Error retrieving credentials" error="ecr: Failed to get authorization token: operation error ECR: GetAuthorizationToken, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, access disabled to EC2 IMDS via client option, or \"AWS_EC2_METADATA_DISABLED\" environment variable"
time="2023-10-06T06:41:29Z" level=debug msg="Could not fetch credentials for cache prefix, disabling cache" error="failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, access disabled to EC2 IMDS via client option, or \"AWS_EC2_METADATA_DISABLED\" environment variable"
time="2023-10-06T06:41:29Z" level=debug msg="Retrieving credentials" region=eu-central-1 registry=repo_account_id serverURL=repo_account_id.dkr.ecr.eu-central-1.amazonaws.com service=ecr
time="2023-10-06T06:41:29Z" level=debug msg="Calling ECR.GetAuthorizationToken" registry=repo_account_id
time="2023-10-06T06:41:29Z" level=error msg="Error retrieving credentials" error="ecr: Failed to get authorization token: operation error ECR: GetAuthorizationToken, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, access disabled to EC2 IMDS via client option, or \"AWS_EC2_METADATA_DISABLED\" environment variable"

Moreover, the error message is misleading, clearly there is no EC2 IMDS available on EKS Fargate. How to solve this issue to make ECR Credentials Helper work in this case?

The configuration of the Gitlab CI job that runs Kaniko backed by ECR Credentials Helper:

.push:
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [ "" ]
  variables:
    ROLE_ARN: "arn:aws:iam::repo_account_id:role/CiPipelineAccessRole"
    AWS_CONFIG_FILE: "/root/.aws/config"
    AWS_PROFILE: "ecr"
    AWS_SDK_LOAD_CONFIG: "true"
  before_script:
    - mkdir -p /root/.aws
  script:
    - echo -e "[default]\nregion=eu-central-1\nrole_arn=$AWS_ROLE_ARN\nduration_seconds=1800\nweb_identity_token_file=$AWS_WEB_IDENTITY_TOKEN_FILE\n[profile ecr]\nregion=eu-central-1\nrole_arn=arn:aws:iam::repo_account_id:role/CiPipelineAccessRole\nduration_seconds=1800\nsource_profile=default\n" > /root/.aws/config
    - mkdir -p /kaniko/.docker
    - echo "{\"credHelpers\":{\"repo_account_id.dkr.ecr.eu-central-1.amazonaws.com\":\"ecr-login\"}}" > /kaniko/.docker/config.json
    - unset AWS_WEB_IDENTITY_TOKEN_FILE
    - unset AWS_ROLE_ARN
    - AWS_PROFILE=ecr AWS_SDK_LOAD_CONFIG=true AWS_EC2_METADATA_DISABLED=true /kaniko/executor --context . --dockerfile Dockerfile --destination repo_account_id.dkr.ecr.eu-central-1.amazonaws.com/ecr_repo_name:$CI_PIPELINE_ID --verbosity=info