sam build does not recognise local nodejs projects outside CodeUri #2064

rfrad · 2020-06-22T13:55:11Z

Description

I have 2 Node.js projects: CommonT and Profile. Profile has a local dependency on CommonT.

When running the sam build -t template.yaml --debug command, it fails with the following error:

npm ERR! code ENOLOCAL
npm ERR! Could not install from "../CommonT" as it does not contain a package.json file.

Here is the project structure:

├── Backend
│   ├── CommonT
│   │   ├── package.json
│   │   └── src
│   └── Profile
│       ├── package.json
│       └── src
└── template.yaml

Here is the Profile/package.json:

{
  "main": "dist/src/index.js",
  ...
  "scripts": {
    "test": "ts-node ./node_modules/jasmine/bin/jasmine",
    "build": "tsc",
    "deploy": "npm run build && sls deploy"
  },
  "dependencies": {
    "@myteam/common": "file:../CommonT",
	...
  },
  "devDependencies": {
    "@types/aws-lambda": "^8.10.46",
    "@types/node": "^12.12.32",
    "aws-sdk": "^2.508.0",
    "ts-node": "^8.10.2",
    "typescript": "^3.8.3",
	...
  }
}

Here is the CommonT/package.json:

{
  "name": "@myteam/common",
  ...
  "main": "dist/src/index",
  "typings": "dist/src/index",
  "private": true,
  "scripts": {
    "test": "ts-node ./node_modules/jasmine/bin/jasmine",
    "build": "tsc",
    "deploy": "npm run build && sls deploy"
  },
  "dependencies": {
    ...
  },
  "devDependencies": {
    "@types/aws-lambda": "^8.10.46",
    "@types/node": "^12.12.32",
    "aws-sdk": "^2.508.0",
    "ts-node": "^8.10.2",
    "typescript": "^3.8.3",
    ...
  }
}

Here is my template.yaml:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  My APIs
 
Resources:
  Profile:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: Backend/Profile
      Handler: dist/src/index.handler
      Runtime: nodejs12.x
      Events:
        GetOneUser:
          Type: Api
          Properties:
            ...
  ...

This is an error I was facing with sam --version: 0.11.0 that disapeared when upgrading to sam --version: 0.52.0 using homebrew. I am currently moving CICD tool and I installed the same sam --version: 0.52.0 using pip3 as I cannot install homebrew on my CICD platform. So here it is:

npm build run manually: works fine
sam build (v 0.11.0) - Homebrew install: ERR! code ENOLOCAL
sam build (v 0.52.0) - Homebrew install: works fine
sam build (v 0.52.0) - pip3 install: ERR! code ENOLOCAL

Steps to reproduce

cd __project_root__

npm install --prefix Backend/CommonT
npm run build --prefix Backend/CommonT
npm run test --prefix Backend/CommonT

npm install --prefix Backend/Profile
npm run build --prefix Backend/Profile
npm run test --prefix Backend/Profile

# I tried both with and without this line, it doesn't make a difference
# npm install -q --no-audit --no-save --production --unsafe-perm --prefix %ROOT%/CommonT

# This is the line that fails
sam build -t template.yaml --debug

Observed result

Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
'build' command is called
Collected default values for parameters: {}
8 resources found in the template
Found Serverless function with name='Profile' and CodeUri='Backend/Profile'
Found Serverless function with name='###Something else###' and CodeUri='###'
Collected default values for parameters: {}
Building function 'Profile'
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Found workflow 'NodejsNpmBuilder' to support capabilities 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Running workflow 'NodejsNpmBuilder'
Running NodejsNpmBuilder:NpmPack
NODEJS packaging file:/my/directory/job/Backend/Profile to /my/directory/temp/buildTmp/XXXXX
executing NPM: ['npm', 'pack', '-q', 'file:/my/directory/job/Backend/Profile']
NODEJS packed to canaree-profile-1.0.0.tgz
NODEJS extracting to /my/directory/temp/buildTmp/XXXXX/unpacked
NodejsNpmBuilder:NpmPack succeeded
Running NodejsNpmBuilder:CopyNpmrc
.npmrc copying in: /my/directory/temp/buildTmp/XXXXX/unpacked/package
NodejsNpmBuilder:CopyNpmrc succeeded
Running NodejsNpmBuilder:CopySource
NodejsNpmBuilder:CopySource succeeded
Running NodejsNpmBuilder:NpmInstall
NODEJS installing in: /my/directory/job/.aws-sam/build/Profile
executing NPM: ['npm', 'install', '-q', '--no-audit', '--no-save', '--production', '--unsafe-perm']
NodejsNpmBuilder:NpmInstall failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/aws_lambda_builders/workflows/nodejs_npm/actions.py", line 108, in execute
    ["install", "-q", "--no-audit", "--no-save", "--production", "--unsafe-perm"], cwd=self.artifacts_dir
  File "/usr/local/lib/python3.6/dist-packages/aws_lambda_builders/workflows/nodejs_npm/npm.py", line 85, in run
    raise NpmExecutionError(message=err.decode("utf8").strip())
aws_lambda_builders.workflows.nodejs_npm.npm.NpmExecutionError: NPM Failed: npm WARN deprecated [email protected]: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated [email protected]: https://github.com/lydell/resolve-url#deprecated
npm ERR! code ENOLOCAL
npm ERR! Could not install from "../CommonT" as it does not contain a package.json file.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2020-06-22T12_03_11_067Z-debug.log

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/aws_lambda_builders/workflow.py", line 269, in run
    action.execute()
  File "/usr/local/lib/python3.6/dist-packages/aws_lambda_builders/workflows/nodejs_npm/actions.py", line 112, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: NPM Failed: npm WARN deprecated [email protected]: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated [email protected]: https://github.com/lydell/resolve-url#deprecated
npm ERR! code ENOLOCAL
npm ERR! Could not install from "../CommonT" as it does not contain a package.json file.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/XXXXXXXX-debug.log

Build Failed

npm ERR!     /root/.npm/_logs/XXXXXXXX-debug.log
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 2034, 'exitReason': 'WorkflowFailedError', 'exitCode': 1, 'requestId': '42b13729-2136-4df8-8442-a25c0ae2ce19', 'installationId': 'acac1f1a-some-long-id-b854a9c82c0b', 'sessionId': 'fb6bba5f-some-other-long-id-ed535022cafb', 'executionEnvironment': 'CLI', 'pyversion': '3.6.9', 'samcliVersion': '0.52.0'}}]}
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: NodejsNpmBuilder:NpmInstall - NPM Failed: npm WARN deprecated [email protected]: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated [email protected]: Please see https://github.com/lydell/urix#deprecated
npm ERR! code ENOLOCAL
npm ERR! Could not install from "../CommonT" as it does not contain a package.json file.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/XXXXXXXX-debug.log

Expected result

When manually running the builds/tests with npm, everything compiles and run properly. I expect the same behavour to happen with sam build.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: Ubuntu 18.04.4 LTS
sam --version: 0.52.0

The text was updated successfully, but these errors were encountered:

rfrad · 2020-07-01T08:27:06Z

@sriram-mv , sorry, I'm a bit new to github. Could you please explain what the added "type/feature" means? Does that mean this bug is being looked at?
Thanks a lot

rfrad · 2020-07-10T14:59:52Z

Here is a work around:

I did manage to "force" the creation of the CommonT package by adding a resource with the CommonT name in my template.yaml.

Here is my new file:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  My APIs
 
Resources:
  CommonT: # <= This NEEDS to have the same name as the CommonT Typescript project!
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: Backend/CommonT
      Handler: dist/src/index.handler
      Runtime: nodejs12.x
      Events:
        GetCommonData:
          Type: Api
          Properties:
            RestApiId: !Ref RestAPI
            Path: /some/config/data/for/example
            Method: get

  Profile:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: Backend/Profile
      Handler: dist/src/index.handler
      Runtime: nodejs12.x
      Events:
        GetOneUser:
          Type: Api
          Properties:
            ...
  ...

Then, when sam build creates the .aws-sam package, it starts by creating the folder CommonT for the resource with the same name. And it is then used as the common project as well.

It is dirty... but it allows to deploy for now.

EladHeller · 2020-08-13T13:08:28Z

This work around work only when the name of the directory is alphanumeric.

dsandor · 2020-08-19T13:40:37Z

Here is my work around. It is not pretty but it works.

First, use npm link or yarn link on your local node modules.

Next, set the dependencies section in your lambda's package.json to a path that is relative to the Lambda Function Code Folder under .aws-sam.

My folders look like this:

   |--.aws-sam
   |----build
   |------zipcodeFunction
   |--------node_modules
   |--local-modules
   |----data-layer
   |--src
   |----zipcodes

So I start by going into the local-modules/data-layer directory and running yarn link.
Next, go into the src/zipcodes (my Lambda function) directory and run yarn link data-layer.

Now a sam build will work and all your development tooling works also because the yarn link in the lambda directory adds the file in node modules. When you do a sam build it will bundle the local node module for you as you intend.

bingtimren · 2020-10-06T12:16:11Z

Here's my work around. Simply put - copy the dependency project into the current root.

My project has three parts - frontend, backend (SAM) and "proj-common", common artifacts for both front and backend.

I first put the dependency in package.json "dependencies" as "file:../proj-commons", pointing to the local folder. This works for the IDE, but sam build fails. The reason is SAM build in a container. In that container the mapped dir cannot access proj-commons. Check .aws-sam/build dir for the files actually available in SAM build.

I tried to put "../proj-commons" in "files" of "package.json" and it does not work. I tried to symbol link "../proj-commons" to a folder under the root, and refer to the folder in both "dependencies" and "files", it does not work.

So the final work around is - in my build script, first copy "../proj-commons" into a directory under the SAM project root, and also put it inside "files" in "project.json".

Here's how my package.json looks like. Note the build script.

    "dependencies": {
        "aws-sdk": "^2.437.0",
        "proj-commons": "file:./.dependency_by_copy/proj-commons"
    },
    "files": [
        "built/",
        ".dependency_by_copy/"
    ],
....
    "scripts": {
        "build": ". ./env.sh; rm -fr ./.dependency_by_copy; mkdir .dependency_by_copy; cp -r ../proj-commons ./.dependency_by_copy; tsc; sam build; announce \"Project Built\"",

fbn-roussel · 2020-11-08T19:38:30Z

I am trying to add some shared code to a layer but I am getting this error as well.

The yarn link did not worked for me. But @bingtimren workaround did, thank you !

Before :

{
  ...
  "scripts": {
    "build": "yarn install"
  },
  "dependencies": {
    "shared": "file:./../shared"
  }
}

After :

{
  ...
  "scripts": {
    "build": "mkdir -p dist && rsync -avr ../../shared dist --exclude node_modules --exclude src && yarn install"
  },
  "dependencies": {
    "shared": "file:./dist/shared"
  },
  "files": [
    "dist"
  ]
}

You can find the full example here : https://github.com/fbn-roussel/aws-sam-typescript-webpack-backend/blob/main/layers/application-layer/package.json

Native support of local path dependency would be greatly appreciated :)

mgrandis · 2021-02-27T02:19:29Z

We are currently reviewing a PR that should address this issue:
aws/aws-lambda-builders#215

kellerkind87 · 2021-08-12T08:13:34Z

hi,
are there any news reagrding that topic? we are highly interested in that beeing solved.

eladg · 2021-08-14T00:47:17Z

@cBiscuitSurprise thank you for the hard work building this critical functionality! 🙏

@mgrandis I and probably the entire AWS SAM community are wondering if there is any progress on this PR since Feb 26? are there any better workarounds we should be aware of? Is there any way we can help test this so it will move forward?

Thanks!

bingtimren · 2021-08-14T05:10:25Z

If you are willing to try CDK, NodejsFunction with esbuild works for me pretty well.

See

https://docs.aws.amazon.com/cdk/api/latest/docs/aws-lambda-nodejs-readme.html#local-bundling

https://docs.aws.amazon.com/cdk/api/latest/docs/aws-lambda-nodejs-readme.html#configuring-esbuild

cBiscuitSurprise · 2021-08-14T11:36:15Z

We've been limping along using 8c33ad6 custom-installed into our build environments. As much as I'd love to see @mgrandis and I's work merged into the product, my team has been slowly making proper JS packages out of all the instances where we're using this feature.

I'm happy to continue helping get this across the finish line. On the other hand, we can live without it (once we're fully converted to proper JS packages).

eladg · 2021-08-17T20:44:45Z

Thanks, @bingtimren & @cBiscuitSurprise, for the replies! Sadly we are not using CDK for various reasons.

my team has been slowly making proper JS packages out of all the instances where we're using this feature.

With that, do you mean a package that is published to the npm repository? Unfortunately, our internal library is private and needs to be resolved by our build system and other dev machines. Is there a way to get around this?

My goal is to share a library between all of my lambda functions. I honestly don't have a preference on how it's happening.

cBiscuitSurprise · 2021-08-18T13:57:32Z

@eladg Yep, we're slowly isolating our modules as needed into packages which we publish to our private npm package registry (we use GitLab's package registry -- it can be free, even for private registries) and then just installing them like anything else.

kellerkind87 · 2021-08-25T11:57:34Z

@cBiscuitSurprise that idea is quite the last option we should consider in our case. local modules are standard in npm so it should just work. the impact in introducing new registries for npm packages is quite big - credentials, build pipelines, local build etc. But at least it is a valid option.

what we have done so far, is to create a symlink for our local packages into the aws build dir. at least it is working, but still not a very good option, because there is no optimization within the package steps. and you have to put this peace of linking code to each function.

{
...

"scripts": {
    "preinstall": "node -e \"const fs = require('fs'); const path = require('path'); if (process.env.sam_build && !fs.existsSync('../../lib')) { fs.symlinkSync((process.env.CODEBUILD_SRC_DIR || '..') +'/lib/', path.resolve('../../lib/'), 'junction')}\""
  },
...
}

with that kind of workaround we can build and package using sam build on local machine and with cloudbuild.

In the end we need the support for local modules. :-)

torresxb1 · 2023-01-12T22:15:41Z

Closing so that we can track in the above issue (Feature request: Build in source). There's multiple open issues relating to similar problems, so we want to have a central place to track.

github-actions · 2023-01-12T22:16:04Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

github-actions · 2023-01-12T22:18:47Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

github-actions · 2023-01-12T22:22:05Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

sriram-mv added the type/feature Feature request label Jun 22, 2020

mribichich mentioned this issue Feb 24, 2021

[sam build/Feature] Turn off use of temp directory, allow to build in current folder #2655

Closed

aahung changed the title ~~sam build does not recognise local nodejs projects~~ sam build does not recognise local nodejs projects outside CodeUri Mar 4, 2021

aahung added the area/build sam build command label Mar 4, 2021

gojko mentioned this issue Jul 12, 2021

Esbuild nodejs builder aws/aws-lambda-builders#252

Closed

aws-sam-cli-bot added maintainer/need-followup and removed maintainer/need-followup labels Oct 7, 2021

mildaniel mentioned this issue Nov 9, 2021

Enable esbuild in the Node.js builder aws/aws-lambda-builders#307

Merged

torresxb1 mentioned this issue Jan 12, 2023

Feature request: Build in source #4571

Closed

torresxb1 closed this as completed Jan 12, 2023

torresxb1 closed this as not planned Won't fix, can't repro, duplicate, stale Jan 12, 2023

torresxb1 closed this as completed Jan 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sam build does not recognise local nodejs projects outside CodeUri #2064

sam build does not recognise local nodejs projects outside CodeUri #2064

rfrad commented Jun 22, 2020

rfrad commented Jul 1, 2020

rfrad commented Jul 10, 2020

EladHeller commented Aug 13, 2020

dsandor commented Aug 19, 2020

bingtimren commented Oct 6, 2020 •

edited

Loading

fbn-roussel commented Nov 8, 2020 •

edited

Loading

mgrandis commented Feb 27, 2021

kellerkind87 commented Aug 12, 2021

eladg commented Aug 14, 2021

bingtimren commented Aug 14, 2021 •

edited

Loading

cBiscuitSurprise commented Aug 14, 2021

eladg commented Aug 17, 2021

cBiscuitSurprise commented Aug 18, 2021 •

edited

Loading

kellerkind87 commented Aug 25, 2021

torresxb1 commented Jan 12, 2023 •

edited

Loading

github-actions bot commented Jan 12, 2023

github-actions bot commented Jan 12, 2023

github-actions bot commented Jan 12, 2023

sam build does not recognise local nodejs projects outside CodeUri #2064

sam build does not recognise local nodejs projects outside CodeUri #2064

Comments

rfrad commented Jun 22, 2020

Description

Steps to reproduce

Observed result

Expected result

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

rfrad commented Jul 1, 2020

rfrad commented Jul 10, 2020

EladHeller commented Aug 13, 2020

dsandor commented Aug 19, 2020

bingtimren commented Oct 6, 2020 • edited Loading

fbn-roussel commented Nov 8, 2020 • edited Loading

mgrandis commented Feb 27, 2021

kellerkind87 commented Aug 12, 2021

eladg commented Aug 14, 2021

bingtimren commented Aug 14, 2021 • edited Loading

cBiscuitSurprise commented Aug 14, 2021

eladg commented Aug 17, 2021

cBiscuitSurprise commented Aug 18, 2021 • edited Loading

kellerkind87 commented Aug 25, 2021

torresxb1 commented Jan 12, 2023 • edited Loading

github-actions bot commented Jan 12, 2023

⚠️COMMENT VISIBILITY WARNING⚠️

github-actions bot commented Jan 12, 2023

⚠️COMMENT VISIBILITY WARNING⚠️

github-actions bot commented Jan 12, 2023

⚠️COMMENT VISIBILITY WARNING⚠️

bingtimren commented Oct 6, 2020 •

edited

Loading

fbn-roussel commented Nov 8, 2020 •

edited

Loading

bingtimren commented Aug 14, 2021 •

edited

Loading

cBiscuitSurprise commented Aug 18, 2021 •

edited

Loading

torresxb1 commented Jan 12, 2023 •

edited

Loading