SAM build fails on 'npm prune'

[josep11]

Description:

I am using a custom Makefile to copy only some artifacts to the .aws-sam folder and to perform some tasks.
I am running sam build and it seems that there is a permission problem with npm prune --production when it tries to remove the unnecessary npm dev packages.

If I run npm prune --production from the terminal it works perfectly.

Steps to reproduce:

Have a node.js with some npm dependencies, src, config folders and handler.js on the root

This is the Makefile:

build-RssFetchNewJobs:
	npm install
	npm test
	npm prune --production
ifeq ($(OS),Windows_NT)
	Xcopy /E * $(ARTIFACTS_DIR) 
else
	cp -r handler.js src node_modules config $(ARTIFACTS_DIR)
endif

Even being a really bad practice I changed the conflicting line with sudo npm prune --production and it run without problems.

Observed result:

If the modules to be removed are installed using sam command (in the build) then everything works properly, but if they were installed by my OS user (straight from the terminal) then it doesn't work. The output is:

Build Failed
Error: CustomMakeBuilder:MakeBuild - Make Failed: internal/modules/cjs/loader.js:968
throw err;
^

Error: Cannot find module '../lib/cli/options'
Require stack:

• /private/var/folders/ck/r6n33qs13zx7j2w3trsnvl6c0000gn/T/tmp6gklk5wn/node_modules/.bin/mocha
  at Function.Module._resolveFilename (internal/modules/cjs/loader.js:965:15)
  at Function.Module._load (internal/modules/cjs/loader.js:841:27)
  at Module.require (internal/modules/cjs/loader.js:1025:19)
  at require (internal/modules/cjs/helpers.js:72:18)
  at Object. (/private/var/folders/ck/r6n33qs13zx7j2w3trsnvl6c0000gn/T/tmp6gklk5wn/node_modules/.bin/mocha:13:23)
  at Module._compile (internal/modules/cjs/loader.js:1137:30)
  at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)
  at Module.load (internal/modules/cjs/loader.js:985:32)
  at Function.Module._load (internal/modules/cjs/loader.js:878:14)
  at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
  '/private/var/folders/ck/r6n33qs13zx7j2w3trsnvl6c0000gn/T/tmp6gklk5wn/node_modules/.bin/mocha'
  ]
  }
  npm ERR! Test failed. See above for more details.
  make: *** [build-RssFetchNewJobs] Error 1

Expected result:

the folders under node_modules to be deleted by sam

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Mac OS 10.14.6
2. sam --version: 1.31.0
3. AWS region: eu-west-3

Those are my dependencies in package.json:

"dependencies": {
    "axios": "^0.23.0",
    "dotenv": "^10.0.0",
    "fast-xml-parser": "^3.20.3",
    "fs": "0.0.1-security",
    "luxon": "^2.0.2"
  },
  "devDependencies": {
    "chai": "^4.3.4",
    "eslint": "^8.0.0",
    "mocha": "^9.1.2",
    "mocha-junit-reporter": "^2.0.2"
  }

[hawflau]

@josep11 Thanks for raising the issue.

• You mentioned "If the modules to be removed are installed using sam command (in the build) then everything works properly" - that makes sense to me because what CustomMakeBuilder did was to copy your source to a scratch dir, then run what defined in the Makefile
• "if they were installed by my OS user (straight from the terminal) then it doesn't work." - Are the user running install straight from terminal the same user running sam build? If they are different, it could be because the user running sam build doesn't have permission to remove files created by the user running install. Can you help confirm that?

[josep11]

@josep11 Thanks for raising the issue.

• You mentioned "If the modules to be removed are installed using sam command (in the build) then everything works properly" - that makes sense to me because what CustomMakeBuilder did was to copy your source to a scratch dir, then run what defined in the Makefile
• "if they were installed by my OS user (straight from the terminal) then it doesn't work." - Are the user running install straight from terminal the same user running sam build? If they are different, it could be because the user running sam build doesn't have permission to remove files created by the user running install. Can you help confirm that?

Hello @hawflau, about the 2nd question: yes, it is the same user running npm and sam build

[hawflau]

@josep11 Thanks for the information. I could reproduce the same as you saw. I found the error came from npm test instead of npm prune, as the stack trace shows it was calling node_modules/.bin/mocha, which is a symlink to mocha/bin/mocha under node_modules. Because Lambda Builder does not preserve links when copying source code to a scratch directory before building, you end up with the actual mocha file in the .bin dir, breaking the require().

Can you help us understand more about your use case? like, is there any reason that you must run npm install outside of sam build?
I'm also wondering if this PR will address your use case.

[hawflau]

Also, moving to Lambda Builder since it's a behavior in Lambda Builder

[josep11]

@josep11 Thanks for the information. I could reproduce the same as you saw. I found the error came from npm test instead of npm prune, as the stack trace shows it was calling node_modules/.bin/mocha, which is a symlink to mocha/bin/mocha under node_modules. Because Lambda Builder does not preserve links when copying source code to a scratch directory before building, you end up with the actual mocha file in the .bin dir, breaking the require().

Can you help us understand more about your use case? like, is there any reason that you must run npm install outside of sam build? I'm also wondering if this PR will address your use case.

The idea is to run the tests when I do the build so that if they don't pass the build fails. That is one of the reasons why I use a custom Makefile (attached in the first message)

[mndeveci]

@josep11 looking at the problem, it seems like it is caused by relative symlinks. Is there a way to generate those symlinks with absolute path, which should work on the same machine?

[lucashuy]

Thanks for creating this issue. Closing due to inactivity, please feel free to reopen this issue if there are any findings or create a new issue if you have any other feedback.