Bug: docker with aws' sam build on M1 for amd64 cannot build

[stationname]

Disclaimer

This issue is merely for reference purposes. It's not necessarily intended to be used as triaging for fixing.

Reason: This might be a difficult edge case and I don't have a from-scratch-like clean environment to make sure the steps will lead to a failure.

Description:

Docker build fails when triggered by AWS' sam build on Mac M1 with target amd64 (cross-plattform).

Error: Build Failed Error: NotFound: content digest sha256:[...]: not found

Notes:

• The build succeeds if it's directly invoked by docker build.
• The build succeeds if the target is arm64
• It's not clear whether this is a sam problem or a docker problem.

Workaround stated at the bottom.

Steps to reproduce:

• Configuration
  • AWS SAM version: version 1.125.0
  • Host: Mac M1
  • Docker 4.34.3 or any other version until 4.29
  • Target platform: amd64
  • buildkit: activated

Files

*template.yaml`

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: asdf

Resources:
  HjksDataCrawler:
    Type: AWS::Serverless::Function
    Properties:
      Description: function
      FunctionName: AFunction
      PackageType: Image
      Architectures:
        - x86_64
    Metadata:
      Dockerfile: Dockerfile
      DockerContext: ./

Dockerfile

FROM --platform=linux/amd64 public.ecr.aws/lambda/python:3.11
COPY app.py /var/task
CMD [ "app.lambda_handler" ]

(↑ I hope I didn't strip too much)

app.py

def lambda_handler():
  print("hello world")

samconfig.toml

version = 0.1
[dev]
[dev.build]
[dev.build.parameters]
region="ap-east-1"
skip_pull_image=true
use_container=true
template = "template.yml"
base_dir = "."

# s3_bucketやimage_repositoryは仮置き
[dev.deploy]
[dev.deploy.parameters]
stack_name = "teststack"
image_repository = # TODO: NEEDS TO BE FILLED IN
region = "ap-east-1"
confirm_changeset = true
capabilities = "CAPABILITY_IAM"
s3_bucket = # TODO: NEEDS TO BE FILLED IN
s3_prefix = "docker_bug_test_m1_x86"

Command to run

export DOCKER_HOST=unix://"$HOME/.docker/run/docker.sock"
sam build --config-env dev 

Note:

• For the DOCKER_HOST setting you can also use this more cryptic command, which leads to the same result

Observed result:

Error output: Build Failed Error: NotFound: content digest sha256:[...]: not found

I can't reproduce the debug output, because I already workaround-ed the problem. 🙏

Expected result:

Build should succeed.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: macOS Sonoma 14.5 (14.5 (23F79))
2. sam --version: version 1.125.0
3. AWS region: ap-east-1
4. Docker 4.34.3 or any other version until 4.29

sam --info

{
  "version": "1.125.0",
  "system": {
    "python": "3.12.6",
    "os": "macOS-14.5-arm64-arm-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "20.10.14",
    "aws_cdk": "2.61.1 (build d319d9c)",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

Workaround

Downgrade to Docker 4.7, downloadable here: https://desktop.docker.com/mac/main/arm64/77141/Docker.dmg
( @docker maintainers: Please don't remove the download link in order to keep the workaround working 🙏 )

[hnnasit]

Hi @stationname, thanks for reporting the issue and providing how you managed to get it to work. Marking it as needs more investigation to see if this is a sam or docker issue.

[baptiste-mnh]

+1

[arvindh-manian]

+1

[arvindh-manian]

Example debug output

2024-12-24 01:22:38,708 | Config file location: /Users/arvindh/cs/projects/kp-restructure/lmsys/samconfig.toml
2024-12-24 01:22:38,709 | Loading configuration values from [default.['build'].parameters] (env.command_name.section) in config file at
'/Users/arvindh/cs/projects/kp-restructure/lmsys/samconfig.toml'...
2024-12-24 01:22:38,710 | Configuration values successfully loaded.
2024-12-24 01:22:38,710 | Configuration values are: {'stack_name': 'lmsys', 'parallel': True}
2024-12-24 01:22:38,713 | Using SAM Template at /Users/arvindh/cs/projects/kp-restructure/lmsys/template.yaml
2024-12-24 01:22:38,732 | Using config file: samconfig.toml, config environment: default
2024-12-24 01:22:38,732 | Expand command line arguments to:
2024-12-24 01:22:38,732 | --template_file=/Users/arvindh/cs/projects/kp-restructure/lmsys/template.yaml --parallel --mount_with=READ
--build_dir=.aws-sam/build --cache_dir=.aws-sam/cache
2024-12-24 01:22:38,757 | 'build' command is called
2024-12-24 01:22:38,759 | No Parameters detected in the template
2024-12-24 01:22:38,771 | There is no customer defined id or cdk path defined for resource ${Redacted}, so we will use the resource logical id
as the resource id
2024-12-24 01:22:38,771 | There is no customer defined id or cdk path defined for resource ${Redacted}, so we will use the resource logical
id as the resource id
2024-12-24 01:22:38,772 | 0 stacks found in the template
2024-12-24 01:22:38,772 | No Parameters detected in the template
2024-12-24 01:22:38,781 | There is no customer defined id or cdk path defined for resource ${Redacted}, so we will use the resource logical id
as the resource id
2024-12-24 01:22:38,781 | There is no customer defined id or cdk path defined for resource ${Redacted}, so we will use the resource logical
id as the resource id
2024-12-24 01:22:38,782 | 2 resources found in the stack
2024-12-24 01:22:38,783 | Found Serverless function with name='${Redacted}' and ImageUri='None'
2024-12-24 01:22:38,783 | --base-dir is not presented, adjusting uri ./${Redacted}-lambda relative to
/Users/arvindh/cs/projects/kp-restructure/lmsys/template.yaml
2024-12-24 01:22:38,784 | --base-dir is not presented, adjusting uri . relative to /Users/arvindh/cs/projects/kp-restructure/lmsys/template.yaml
2024-12-24 01:22:38,787 | 2 resources found in the stack
2024-12-24 01:22:38,788 | Found Serverless function with name='${Redacted}' and ImageUri='None'
2024-12-24 01:22:38,788 | Error occurred while trying to track an event: Event 'BuildFunctionRuntime' does not accept value 'None'.
2024-12-24 01:22:38,789 | Instantiating build definitions
2024-12-24 01:22:38,791 | Unique function build definition found, adding as new (Function Build Definition: BuildDefinition(None,
/Users/arvindh/cs/projects/kp-restructure/lmsys, Image, , b8e87180-e974-4d8d-b2e9-5d09841c1aa4, {'DockerTag': 'provided.al2023-v1', 'DockerContext':
'/Users/arvindh/cs/projects/kp-restructure/lmsys/lmsys-${Redacted}-lambda', 'Dockerfile': 'Dockerfile'}, {}, x86_64, []), Function: Function({'function_id':
'${Redacted}', 'name': '${Redacted}', 'functionname': '${Redacted}', 'runtime': None, 'memory': 128, 'timeout': 5, 'handler':
None, 'imageuri': None, 'packagetype': 'Image', 'imageconfig': None, 'codeuri': '/Users/arvindh/cs/projects/kp-restructure/lmsys', 'environment': None,
'rolearn': 'arn:aws:lambda:us-east-1:123456789012:function:${Redacted}', 'layers': [], 'events': {'QueueScheduler': {'Type': 'Schedule',
'Properties': {'Schedule': 'rate(5 minutes)', 'Enabled': True, 'Name': '${Redacted}', 'Description': 'Schedule to trigger
${Redacted} every 5 minutes'}}}, 'metadata': {'DockerTag': 'provided.al2023-v1', 'DockerContext':
'/Users/arvindh/cs/projects/kp-restructure/lmsys/${Redacted}-lambda', 'Dockerfile': 'Dockerfile', 'SamResourceId': '${Redacted}'},
'inlinecode': None, 'codesign_config_arn': None, 'architectures': ['x86_64'], 'function_url_config': None, 'function_build_info':
<FunctionBuildInfo.BuildableImage: ('BuildableImage', 'Regular IMAGE function which can be build with SAM CLI')>, 'stack_path': '',
'runtime_management_config': None, 'logging_config': None}))
2024-12-24 01:22:38,810 | Async execution started
2024-12-24 01:22:38,810 | Invoking function functools.partial(<bound method ParallelBuildStrategy.build_single_function_definition of
<samcli.lib.build.build_strategy.ParallelBuildStrategy object at 0x103c22090>>, <samcli.lib.build.build_graph.FunctionBuildDefinition object at
0x10684b410>)
2024-12-24 01:22:38,811 | Building codeuri: /Users/arvindh/cs/projects/kp-restructure/lmsys runtime: None architecture: x86_64 functions:
${Redacted}
2024-12-24 01:22:38,811 | Building to following folder /Users/arvindh/cs/projects/kp-restructure/lmsys/.aws-sam/build/${Redacted}
2024-12-24 01:22:38,811 | Building image for ${Redacted} function
2024-12-24 01:22:38,812 | Waiting for async results
2024-12-24 01:22:38,816 | Setting DockerBuildArgs for ${Redacted} function
2024-12-24 01:22:40,307 | Failed building function ${Redacted}
2024-12-24 01:22:40,310 | Exception raised during the execution

Build Failed
2024-12-24 01:22:40,311 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-12-24 01:22:40,371 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-12-24 01:22:40,372 | Unable to find Click Context for getting session_id.
2024-12-24 01:22:40,372 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '7b1de4c5-37ea-4c50-a6f4-c97797106dfc', 'installationId':
'55c6fc5a-513a-4288-a9a2-475796b2a85c', 'sessionId': 'f87faa4b-c244-4fef-a67f-f454731481d4', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion':
'3.11.10', 'samcliVersion': '1.132.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build',
'metricSpecificAttributes': {'projectType': 'CFN', 'gitOrigin': None, 'projectName': '636fa9664f878a4de79227421c3c5199b1d7ced700dc2171f7e22d4253accd25',
'initialCommit': None}, 'duration': 1579, 'exitReason': 'DockerBuildFailed', 'exitCode': 1}}]}
2024-12-24 01:22:40,375 | Sending Telemetry: {'metrics': [{'events': {'requestId': '64d09680-a58a-49e4-a99a-13f112d2ffa4', 'installationId':
'55c6fc5a-513a-4288-a9a2-475796b2a85c', 'sessionId': 'f87faa4b-c244-4fef-a67f-f454731481d4', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion':
'3.11.10', 'samcliVersion': '1.132.0', 'commandName': 'sam build', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension',
'event_value': '.toml', 'thread_id': 'a8fa6c4f6c584fe084a942f01640b64f', 'time_stamp': '2024-12-24 06:22:38.708', 'exception_name': None},
{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '27b2a0b86cae42e6b020ea0980ac17ac', 'time_stamp': '2024-12-24
06:22:38.731', 'exception_name': None}]}}}]}
2024-12-24 01:22:40,690 | Telemetry response: 200
2024-12-24 01:22:40,693 | Telemetry response: 200
Error: failed to get destination image "sha256:8de8595beec4ae7521d32e3527054f4336b10f0e4246ec7b0cf6584702b7ae7b": No such image: sha256:8de8595beec4ae7521d32e3527054f4336b10f0e4246ec7b0cf6584702b7ae7b