PythonPipBuilder:ResolveDependencies errors on arm64, Incorrect Layer build behaviors

[brysontyrrell]

Note: this is another open issue on this build error #1840 but there have been no updates since December and there are other behaviors I'm noticing that are not a part of that issue.

Description:

I have a SAM template where I have defined the architecture at arm64 for my functions and for my shared Lambda Layer.

I am working on an Apple Silicon MacBook Pro (arm64 host).

I am installing a private package from CodeArtifact that has a dependency of jsonschema that has a dependency of pyrsistent.

The Lambda Layer is configured to use the builder:

  SharedLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: !Ref AWS::StackName
      ContentUri: ./src/layers/shared
      CompatibleArchitectures:
        - arm64
      CompatibleRuntimes:
        - python3.9
    Metadata:
      BuildMethod: python3.9

If I run sam build without a container and view the build files in .aws-sam/build I can see that it compiled for x86 by some of the binaries that are present: pvectorc.cpython-39-x86_64-linux-gnu.so. The layer's compatibility is clearly set for arm64.

I then tried to run the build in a container. The Layer pulled the x86_64 image!

I added the package to a requirements file for the Lambda function to compare. It did pull the arm64, but then it hit the following error:

Error: PythonPipBuilder:ResolveDependencies - {pyrsistent==0.18.1(wheel)}

Running sam build without using a container yielded the same error.

Switching the Lambda Function to use x86_64 results in a successful build.

I launched a container to test installing the package ( docker run --rm -it build-python3.9:latest-arm64 ) and it did so without an issue.

Here's a summary of the sam build issues:

• Lambda Layer - On an arm64 host builds are locked to x86_64.
• Lambda Layer - Builds are locked to the x86_64 build image even if compatibility is only arm64.
• Lambda Function - The build fails at the pyrsistent dependency (error above) using the arm64 image or building on an arm64 host. This is NOT reproducible inside the arm64 image or on an arm64 host using pip directly.

These issues have halted all attempts to migrate to Graviton for Lambda Functions with my group.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: macOS 12.3
2. sam --version: 1.40.1
3. AWS region: N/A

Add --debug flag to command you are running

[brysontyrrell]

Found some related issues/PRs in the lambda builder repo:

aws/aws-lambda-builders#302

aws/aws-lambda-builders#321

@jfuss The latest version of the.builders was in use and I still hit this issue.

[mndeveci]

Thanks for reporting this issue @brysontyrrell ,

For your layer build, you need to provide BuildArchitecture metadata (just like BuildMethod), in order to use correct one which is documented here. Your use case actually seems like a bug, so I will check with the team whether if we can use architecture, if only one is defined under CompatibleArchitectures.

For the rest of the cases used following template;

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sam-app-cli-3747

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.8
      Architectures:
        - arm64

  HelloWorldLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      ContentUri: layer/
      CompatibleArchitectures:
        - arm64
      CompatibleRuntimes:
        - python3.8
    Metadata:
      BuildMethod: python3.8
      BuildArchitecture: arm64

And I used jsonschema as the only dependency for both function and layer.

If i tried to build it on my Mac (which is an X86), it fails with following error;

Error: PythonPipBuilder:ResolveDependencies - {pyrsistent==0.18.1(wheel)}

Since jsonschema contains a native dependency, we need to run build inside container. So running sam build --use-container fixes issue.

To test this on an arm64 machine, I spun up a new EC2 instance with arm64 architecture, and tried to build it there. Since that machine is a linux/arm64, I am able to build it either running sam build or sam build --use-container.

So regarding to your findings;

• Lambda Layer - On an arm64 host builds are locked to x86_64.
  • For this case, it is downloading wrong type of the library since BuildArchitecture: arm64 is missing in the Metadata. But even if you add this, you can't build something which requires native dependency compilation, since your target lambda runs on linux/arm64 but you are building it on macosx/arm64. And looking at pyrsistent==0.18.1 files, they don't provide wheels that is compatible with arm64.
• Lambda Layer - Builds are locked to the x86_64 build image even if compatibility is only arm64.
  • For now, you can fix this by adding BuildArchitecture: arm64 into your metadata. I will check with the team whether we can set this by checking CompatibleArchitectures.
• Lambda Function - The build fails at the pyrsistent dependency (error above) using the arm64 image or building on an arm64 host. This is NOT reproducible inside the arm64 image or on an arm64 host using pip directly.
  • I think this issue is fixed by the latest release of the lambda builders with this PR Fix python ARM build issue aws-lambda-builders#321

Please let us know if you have other questions or concerns.

[github-actions]

Patch is released in v1.104.0. Closing