Build failing - Error in ResolveDependencies for fastparquet

[Monal5031]

Description:

I have a piece of code which works with fastparquet==0.6.3 and python 3.9 locally and when I try to deploy it to AWS lambda via aws-sam-cli, the build fails.

Steps to reproduce:

• Create a new python environment
• Add fastparquet==0.6.3 as part of requirement
• Deploy with aws-sam-cli using: sam build --use-container --template ./template.yaml

Observed result:

Running PythonPipBuilder:ResolveDependencies
Error: PythonPipBuilder:ResolveDependencies - {fastparquet==0.6.3(sdist)}
Build failed!

Expected result:

Successful build.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: macOS Monterey 12.1
2. sam --version: 1.28.0
3. AWS region: eu-central-1
4. Python: 3.9

Add --debug flag to command you are running

[hoffa]

What Lambda architecture are you using? Also see potentially related #1840.

I ran the following:

sam init --name sam-app --runtime python3.9 --dependency-manager pip --app-template hello-world
cd sam-app
echo 'fastparquet==0.6.3' > hello_world/requirements.txt
sam build --use-container --debug

Which succeeds:

2021-12-29 10:49:55,205 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-12-29 10:49:55,205 | Using config file: samconfig.toml, config environment: default
2021-12-29 10:49:55,205 | Expand command line arguments to:
2021-12-29 10:49:55,205 | --template_file=/Users/rehnc/Desktop/sam-app/template.yaml --use_container --build_dir=.aws-sam/build --cache_dir=.aws-sam/cache 
2021-12-29 10:49:55,406 | 'build' command is called
2021-12-29 10:49:55,406 | Starting Build inside a container
2021-12-29 10:49:55,411 | No Parameters detected in the template
2021-12-29 10:49:55,429 | 2 stacks found in the template
2021-12-29 10:49:55,430 | No Parameters detected in the template
2021-12-29 10:49:55,440 | 2 resources found in the stack 
2021-12-29 10:49:55,440 | No Parameters detected in the template
2021-12-29 10:49:55,452 | Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/'
2021-12-29 10:49:55,452 | --base-dir is not presented, adjusting uri hello_world/ relative to /Users/rehnc/Desktop/sam-app/template.yaml
2021-12-29 10:49:55,452 | No Parameters detected in the template
2021-12-29 10:49:56,301 | Instantiating build definitions
2021-12-29 10:49:56,304 | Same function build definition found, adding function (Previous: BuildDefinition(python3.9, /Users/rehnc/Desktop/sam-app/hello_world, Zip, , 8a71a0d3-1d39-4e5e-abc5-77460e6a6ea5, {}, {}, x86_64, []), Current: BuildDefinition(python3.9, /Users/rehnc/Desktop/sam-app/hello_world, Zip, , ebab27bf-ee66-4802-afab-2c025c463d7f, {}, {}, x86_64, []), Function: Function(name='HelloWorldFunction', functionname='HelloWorldFunction', runtime='python3.9', memory=None, timeout=3, handler='app.lambda_handler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/Users/rehnc/Desktop/sam-app/hello_world', environment=None, rolearn=None, layers=[], events={'HelloWorld': {'Type': 'Api', 'Properties': {'Path': '/hello', 'Method': 'get', 'RestApiId': 'ServerlessRestApi'}}}, metadata=None, inlinecode=None, codesign_config_arn=None, architectures=['x86_64'], stack_path=''))
2021-12-29 10:49:56,305 | Building codeuri: /Users/rehnc/Desktop/sam-app/hello_world runtime: python3.9 metadata: {} architecture: x86_64 functions: ['HelloWorldFunction']
2021-12-29 10:49:56,305 | Building to following folder /Users/rehnc/Desktop/sam-app/.aws-sam/build/HelloWorldFunction

Fetching public.ecr.aws/sam/build-python3.9:latest-x86_64 Docker container image......
2021-12-29 10:49:57,468 | Mounting /Users/rehnc/Desktop/sam-app/hello_world as /tmp/samcli/source:ro,delegated inside runtime container
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i
Full dependency closure: {fsspec==2021.11.1(wheel), fastparquet==0.6.3(sdist), numpy==1.21.5(wheel), python-dateutil==2.8.2(wheel), pytz==2021.3(wheel), cramjam==2.5.0(wheel), six==1.16.0(wheel), thrift==0.15.0(sdist), pandas==1.3.5(wheel)}
initial compatible: {fsspec==2021.11.1(wheel), numpy==1.21.5(wheel), python-dateutil==2.8.2(wheel), pytz==2021.3(wheel), cramjam==2.5.0(wheel), six==1.16.0(wheel), pandas==1.3.5(wheel)}
initial incompatible: {fastparquet==0.6.3(sdist), thrift==0.15.0(sdist)}
Downloading missing wheels: {fastparquet==0.6.3(sdist), thrift==0.15.0(sdist)}
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_x86_64 --implementation cp --abi cp39 --dest /tmp/samcli/scratch fastparquet==0.6.3
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_x86_64 --implementation cp --abi cp39 --dest /tmp/samcli/scratch thrift==0.15.0
compatible wheels after second download pass: {fsspec==2021.11.1(wheel), numpy==1.21.5(wheel), python-dateutil==2.8.2(wheel), pytz==2021.3(wheel), cramjam==2.5.0(wheel), six==1.16.0(wheel), pandas==1.3.5(wheel)}
Build missing wheels from sdists (C compiling True): {fastparquet==0.6.3(sdist), thrift==0.15.0(sdist)}
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/fastparquet-0.6.3.tar.gz
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/thrift-0.15.0.tar.gz
compatible after building wheels (no C compiling): {fsspec==2021.11.1(wheel), fastparquet==0.6.3(wheel), numpy==1.21.5(wheel), python-dateutil==2.8.2(wheel), pytz==2021.3(wheel), cramjam==2.5.0(wheel), six==1.16.0(wheel), thrift==0.15.0(wheel), pandas==1.3.5(wheel)}
Build missing wheels from sdists (C compiling False): set()
compatible after building wheels (C compiling): {fsspec==2021.11.1(wheel), fastparquet==0.6.3(wheel), numpy==1.21.5(wheel), python-dateutil==2.8.2(wheel), pytz==2021.3(wheel), cramjam==2.5.0(wheel), six==1.16.0(wheel), thrift==0.15.0(wheel), pandas==1.3.5(wheel)}
Final compatible: {fsspec==2021.11.1(wheel), fastparquet==0.6.3(wheel), numpy==1.21.5(wheel), python-dateutil==2.8.2(wheel), pytz==2021.3(wheel), cramjam==2.5.0(wheel), six==1.16.0(wheel), thrift==0.15.0(wheel), pandas==1.3.5(wheel)}
Final incompatible: set()
Final missing wheels: set()
PythonPipBuilder:ResolveDependencies succeeded
Running PythonPipBuilder:CopySource
PythonPipBuilder:CopySource succeeded
2021-12-29 10:50:24,307 | Build inside container returned response {"jsonrpc": "2.0", "id": 1, "result": {"artifacts_dir": "/tmp/samcli/artifacts"}}
2021-12-29 10:50:24,308 | Build inside container was successful. Copying artifacts from container to host
2021-12-29 10:50:24,534 | Copying from container: /tmp/samcli/artifacts/. -> /Users/rehnc/Desktop/sam-app/.aws-sam/build/HelloWorldFunction
2021-12-29 10:50:29,944 | Build inside container succeeded

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {stack-name} --watch
[*] Deploy: sam deploy --guided

[Monal5031]

@hoffa x86_64 architecture

[hoffa]

In that case the comment above still applies; I'm unable to reproduce with the steps you provided:

sam init --name sam-app --runtime python3.9 --dependency-manager pip --app-template hello-world
cd sam-app
echo 'fastparquet==0.6.3' > hello_world/requirements.txt
sam build --use-container --template ./template.yaml --debug

Builds fine. Can you confirm that running the commands above work?

One difference I'm seeing is that you're using SAM CLI 1.28.0, while mine is 1.36.0. If the above steps don't work, you might want to try updating SAM CLI first.

[c2tarun]

@Monal5031 can you please confirm if your issue is resolved by latest version of aws-sam-cli?

[Monal5031]

Hey, Yes I tried with the latest version of aws-sam-cli and it works with it. Due to dependency in other packages it won't be possible for us to use the latest version. Is there any workaround that's possible? Thanks for help 😄

[c2tarun]

Thanks for confirming @Monal5031, I was able to reproduce your issue with latest version as well, actually it is not an issue with sam-cli but fastparquet's wheels.
TLDR; If you want to use older version of sam-cli (which we don't recommend), you can try building with --use-container and your build should work fine. Please let us know on this thread if you still see failure.

I would still recommend using latest version of sam-cli and would like to know more about the constraints where you are not able to use latest version of sam-cli.

Long explanation:
When I try to install fastparquet using pip I get following log:

Collecting fastparquet==0.6.3
  Using cached fastparquet-0.6.3.tar.gz (318 kB)
Requirement already satisfied: pandas>=1.1.0 in /Users/tarumall/.pyenv/versions/3.9.6/envs/venv39/lib/python3.9/site-packages (from fastparquet==0.6.3->-r hello_world/requirements.txt (line 1)) (1.3.4)
Requirement already satisfied: numpy>=1.11 in /Users/tarumall/.pyenv/versions/3.9.6/envs/venv39/lib/python3.9/site-packages (from fastparquet==0.6.3->-r hello_world/requirements.txt (line 1)) (1.21.3)
Collecting thrift>=0.11.0
  Using cached thrift-0.15.0.tar.gz (59 kB)
Collecting cramjam>=2.3.0
  Using cached cramjam-2.5.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (2.7 MB)
Collecting fsspec
  Using cached fsspec-2021.11.1-py3-none-any.whl (132 kB)
Requirement already satisfied: pytz>=2017.3 in /Users/tarumall/.pyenv/versions/3.9.6/envs/venv39/lib/python3.9/site-packages (from pandas>=1.1.0->fastparquet==0.6.3->-r hello_world/requirements.txt (line 1)) (2021.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /Users/tarumall/.pyenv/versions/3.9.6/envs/venv39/lib/python3.9/site-packages (from pandas>=1.1.0->fastparquet==0.6.3->-r hello_world/requirements.txt (line 1)) (2.8.2)
Requirement already satisfied: six>=1.5 in /Users/tarumall/.pyenv/versions/3.9.6/envs/venv39/lib/python3.9/site-packages (from python-dateutil>=2.7.3->pandas>=1.1.0->fastparquet==0.6.3->-r hello_world/requirements.txt (line 1)) (1.16.0)
>>>>>>>>>>>>>>>>>> Notice this
Building wheels for collected packages: fastparquet, thrift
  Building wheel for fastparquet (setup.py) ... done
  Created wheel for fastparquet: filename=fastparquet-0.6.3-cp39-cp39-macosx_11_0_x86_64.whl size=484428 sha256=3ef242bba1308efca9abd41bb98c53c1a31e5c1ddb96cf4b902362790bbe40a7
  Stored in directory: /Users/tarumall/Library/Caches/pip/wheels/5f/c3/83/d44367a87212d3113ebc644470af7887fdad3a1f7b47e05457
  Building wheel for thrift (setup.py) ... done
  Created wheel for thrift: filename=thrift-0.15.0-cp39-cp39-macosx_11_0_x86_64.whl size=177457 sha256=ec38e9c7d29bdf7f6693cbf708ec211b4cf9a49b8d514e431ea5737b8d46f008
  Stored in directory: /Users/tarumall/Library/Caches/pip/wheels/00/41/cd/f65d329f7cc147ce79cc833b11b94f7bea7329df0ad9d81629
Successfully built fastparquet thrift
<<<<<<<<<<<<<<<<<< Till here
Installing collected packages: thrift, fsspec, cramjam, fastparquet
Successfully installed cramjam-2.5.0 fastparquet-0.6.3 fsspec-2021.11.1 thrift-0.15.0

Notice that pip was also not able to find the required wheels and went ahead with building the dependency on local. This is because fastparquet's provide wheels only upto macos 10.9. I am not 100% sure why sam-cli's use of pip don't actually build the dependency but one reason can be security.

When you use --use-container flag then sam-cli builds your package inside a docker container which is a linux environment. In that case pip is able to find the corresponding wheel, build the package and copies build artifact to your local. I am resolving this ticket for now, please let me know if you have more questions.

[c2tarun]

Actually I looked more into this issue and it looks like an issue with how sam-cli triggers pip installation. After adding some logs to lambda-builders I see following lines before attempt to build fastparquet wheel fails:

          building \'fastparquet.speedups\' extension
          creating build/temp.macosx-11.4-x86_64-3.9
          creating build/temp.macosx-11.4-x86_64-3.9/fastparquet
          /var/false -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Users/tarumall/.pyenv/versions/venv39/include -I/Users/tarumall/.pyenv/versions/3.9.6/include/python3.9 -I/Users/tarumall/.pyenv/versions/venv39/lib/python3.9/site-packages/numpy/core/include -c fastparquet/speedups.c -o build/temp.macosx-11.4-x86_64-3.9/fastparquet/speedups.o
          error: command \'/var/false\' failed: No such file or directory
          ----------------------------------------
          ERROR: Failed building wheel for fastparquet
          ERROR: Failed to build one or more wheels

When try to install using pip, it also tries to build the wheel and the same command is below:

  building 'fastparquet.speedups' extension
  creating build/temp.macosx-11.4-x86_64-3.9
  creating build/temp.macosx-11.4-x86_64-3.9/fastparquet
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Users/tarumall/.pyenv/versions/3.9.6/envs/venv39/include -I/Users/tarumall/.pyenv/versions/3.9.6/include/python3.9 -I/Users/tarumall/.pyenv/versions/3.9.6/envs/venv39/lib/python3.9/site-packages/numpy/core/include -c fastparquet/speedups.c -o build/temp.macosx-11.4-x86_64-3.9/fastparquet/speedups.o
  In file included from fastparquet/speedups.c:625:

On looking closely sam-cli's invocation uses /var/false and pip's invocation uses clang. On manual inspection there is no executable on my machine with /var/false. I'll re-open the issue and try to identify the root cause.

[Monal5031]

Hey @c2tarun, Thanks a lot for looking into it.
An update from our side we were able to fix the dependency constraints and use the latest version of aws-sam-cli. The dependency issue was due to boto3 (botocore precisely) version mismatch between awscli and aws-sam-cli

[mndeveci]

Glad to hear that issue is fixed on your side @Monal5031

If you are installing SAM CLI through pip, we recommend using a separate virtual environment for that, for not facing any issues with dependency resolution.

Please comment or create a new issue, if you are still facing this or similar problem.