npm link causes sam local to error

[paulsson]

Hello! I'm using sam local to rapidly develop and test nodejs lambda functions.

SAM version: SAM CLI, version 0.6.1
MacOS 10.13.6

I have common code in separate node modules that I am trying to use 'npm link' with to create symlinks for in my lambda functions node_modules directory. This would be a great way to allow editing the common code in directories outside of my lambda function directories and just have those changes immediately available to my lambda functions where they are linked in the lamdba function node_modules directories. Otherwise, every time I change the common code I would have to go through extra steps to copy it to every lambda functions node_modules directory for the lambdas that need to use that common code.

The advantages of this is explained very well here:
https://forums.aws.amazon.com/thread.jspa?threadID=214756#jive-message-671727
and here:
https://stackoverflow.com/questions/33511652/including-local-dependencies-in-deployment-to-lambda

I can see that my symlinks are created properly and I can 'ls' the symlink to see that it lists the expected files back in the linked directory. However, when I run my code and try to test when using these symlinks I get:

START RequestId: f453f83b-58e5-1a7f-acd9-1af0f9a251c5 Version: $LATEST
Unable to import module 'index': Error
    at Function.Module._resolveFilename (module.js:547:15)
    at Function.Module._load (module.js:474:25)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/var/task/index.js:3:13)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
END RequestId: f453f83b-58e5-1a7f-acd9-1af0f9a251c5

If I copy my common node modules directly into my lambda function node_modules directories everything works perfectly.

Any suggestions? Does it sound like I am doing something wrong? Is this a known issue?
If I can copy my node module directly into my lamda node_modules directory rather than using a symlink it seems like there might be some issue preventing this. Having this capability would greatly help developing node lambda functions to be much faster.

Thanks!

[jceloi]

Same exact problem for some local developments and dependencies loaded via a local "file:" path which result in a symlink too.
It's pretty annoying !

[jfuss]

@paulsson Thanks for submitting the issue. This sounds like something that sam build for node will solve. This is being added to aws-lambda-builders. @sanathkr and @gojko, do you guys things build will help in resolving this?

[gojko]

this will be resolved with the new builder. for now, you can use claudia pack to produce a zip (you don't have to use claudia to deploy, just use the pack command to produce a zip that SAM will deploy). I'm porting that code over to SAM now, so this should be also available soon.

[ericallam]

@jfuss the post sam build workflow doesn't allow for hot-reloading code, so it would still be good if symlinks could work in sam local so we can make changes to symlinked shared code.

[jfuss]

@ericallam It does support hot-reloading but you need to build in another terminal window. We will eventually introduce a sam build --watch command that will make this even easier. This remains consistent with the CX of the CLI before build.

Symlinks and docker do not play well together: https://stackoverflow.com/questions/38485607/mount-host-directory-with-a-symbolic-link-inside-in-docker-container. So yes we understand this will help, but if docker doesn't support symlinks, it hard for us to support it through mounting of the code. Therefore need some other process (I am thinking build) to help solve that.

[ericallam]

@jfuss ah I didn't know about the plan for sam build --watch, thanks for the clarification.

[braidn]

I am also building an application with Node that's constructed in much the same way. I am using lerna to manage symlinks from packages outside of the main handlers. This is juggleable now by releasing minor patch versions that my handlers can consume but, would love the freedom to be able to work with the symlinks.

Also totally understand that docker and symlinks don't play super nice together. Excited to see if this build watch for the node 8.10 runtime fixes this issue. Thank you all.

[jfuss]

@paulsson We have released build support for npm. Can you give this a try by using sam build on your functions that require this?

[paulsson]

@jfuss this sounds great! I will be testing this out today and get back to you with any questions or feedback. Thanks!

[paulsson]

@jfuss does this work when using 'sam local start-api' for simulating api-gateway locally?
I am getting the same error still. I tried running 'sam build' before and after running 'sam local start-api'.

I see the output of 'sam build' does not list 'sam local start-api' as a next command:

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Package: sam package --s3-bucket <yourbucket>

Do I have to run 'sam local start-api' from the newly created .aws-sam/build directory where the new
.aws-sam/build/template.yaml is located?
Also, the node_modules dir for my function under the generated build artifact directory under .aws-sam/build/<function>/node_modules does not contain my npm linked module.
Any documentation or steps for getting this working would be great! I am looking forward to using this new feature.
Thanks!

[gojko]

@paulsson symlinks aren't supported in sam build for node yet, I plan to work on it once all the pending PRs for the node builder get merged to avoid conflicts.

[mmcglone]

@gojko Any sense of when symlinks will be supported in sam build for node? This will be awesome!

[abbottdev]

+1 - symlinks, or at least, npm packages with "file://../another_folder/" would be a massive help for code-reuse.

[gojko]

@abbottdev we have the code ready, just need to add a few more tests. should be within the next few days hopefully

[braidn]

Thanks all. Exited to give it a whirl

[gojko]

just a quick update on this, the code is ready, but it seems that it's not likely going to be merged, so you won't be able to use SAM with local deps (aws/aws-lambda-builders#106). My suggestion is to use claudia pack as a preprocessor and not use sam build in this case. claudia pack will deal with local dependencies correctly and produce a deployable zip, and you can link to the zip from the SAM/cloudformation template.

[mmcglone]

@gojko Bummer. But thanks very much for the update!

[jfuss]

This is something we want to support but we have a challenge due to how we currently build functions, which is what is holding this up not because we don't want it merged.

@thesriram Made a comment on the PR: aws/aws-lambda-builders#106 (comment) but will rephrase here.

Currently SAM CLI mounts code into the container for each function. For the local dependency case (in the container), there is no way to build with local dependencies unless the code is mounted. To support this we need to change how we mount code into the container, most likely mount the project instead and execute sam build in the container. This fundamentally changes the current architecture. Now this could work in the non-container build but comes with a usability concern. That is customers who need this kind of building would fail if they try to do --use-container and cause further confusion on "this build works in this context but not the other". To truly support building with local dependencies, we want to enable both use-cases.

[fcasals]

in my case, i'm using npm pack command and linking on my main project the dependency as "file:..", but sam doesn't pack the dependency... :(
I tried claudia pack and works as expected.
of course, once the symlinks gets support, i'll move to that strategy... i

[dtelaroli]

+1

I'm testing git submodule and it's working well, but npm is better to this use case.

[jashanbhullar]

+1
Need to make the code reusable

[rh-danderson]

Hi!

First off, I love the vscode integration. Launching a debugger from within the code file by clicking an annotation is genius.

I'd love to start using sam, but without being able to set a breakpoint in vscode and step into a local dependency's symlinked (via "file:" and npm link) code, I just can't start using it for everyday development work.

I'm hoping there are others out there like me who are writing lambdas that depend on local libraries, and want to set a breakpoint in a dependent library while debugging things? It seems like a common real-world node development thing. (Especially with the introduction of yarn workspaces, which takes this to the next level.)

/functions/loader/app.js
const bl = require("@my-org/business");

/functions/loader/package.json
"@my-org/business": "file: ../../libraries/business"

/libraries/business/package.json
"name": "@my-org/business"

It's been suggested to package the dependencies up on build. This is a little iffy, because now the code that you are debugging is not the real code in your working directory. I've been here, and you wind up flipping between files a lot, and accidentally making changes to the ephemeral file and losing them on the next build. You should be able to make changes to the file that it's showing you while you debug. It's not a great developer experience.

Support for this would be so amazing. It really was a "so close!" moment for me.

Any others out there struggling with this? Say heyyy

[jashanbhullar]

My current solution is to include the package in devDependencies so it only installs for dev environment.
In production after sam build I use

find build-directory -maxdepth 1 -mindepth 1 -type d -exec npm i --prefix {} /path/to.package \;

This commands

• Finds all the folder at 1st level build-directory
• Installs the common package in every node_modules folder.( prefix flag)

This ensures the dependencies of the common package are properly installed. sam package works okay with this even if the packages are sym-linked.

Problems with this approach:

• Will install in all the folders. That can be fixed by looking for the node_modules but I am working for node only so it doesn't matter for me
• If a dev installs it as a normal dependency it will work on dev machine but will fail the production and I am not how to enforce this as a dev dependency package only.

[mgrandis]

It looks like this PR will help:
aws/aws-lambda-builders#215

Will it fix the issue or do you need other features to work with local packages?

[MatteoGioioso]

@gojko I have tried claudia pack, but unfortunately is not working either with symlink, I have inspected the zip but the linked module is not present =(

[alphonse92]

That's happening because the default esbuild pipeline in build phase. Of course, when sam clit run: Running NodejsNpmEsbuildBuilder:CopySourceit does not update the package.json dependency to point where the local package is. It just copies and pastes the package.json. Then it executes Running NodejsNpmEsbuildBuilder:NpmInstall, of course npm try to install a local dependency that does not exist, and set a invalid symlink as u can see here in this image: