SAM debugging fails (using colima)

[zachlipton]

Problem

When using the built-in Hello World nodejs Lambda template, invoking the default launch configuration results in a JavaScript ImportModule Error: Error: Cannot find module 'app' while manually running sam local invoke in the terminal works fine.

Steps to reproduce the issue

1. Create new workspace
2. Cmd+P, >AWS: Create Lambda SAM Application
3. nodejs20.x (or nodejs22.x)
4. arm64
5. AWS SAM Hello World
6. Select current folder
7. return
8. Select Run and Debug from the menu and run the default launch configuration

Actual behavior

2024-12-05 01:04:36.659 [info] 2024-12-05T09:04:36.727Z undefined ERROR Uncaught Exception {"errorType":"Runtime.ImportModuleError","errorMessage":"Error: Cannot find module 'app'\nRequire stack:\n- /var/runtime/index.mjs","stack":["Runtime.ImportModuleError: Error: Cannot find module 'app'","Require stack:","- /var/runtime/index.mjs"," at _loadUserApp (file:///var/runtime/index.mjs:1087:17)"," at async Object.load (file:///var/runtime/index.mjs:1119:21)"," at async start (file:///var/runtime/index.mjs:1282:23)"," at async file:///var/runtime/index.mjs:1288:1"]}

The full log output by Run and Debug is attached: vscode.txt

Expected behavior

The function is invoked locally.

The odd thing is that if I switch to the terminal tab and run cd lambda-nodejs20.x && sam local invoke, it works as expected and outputs the hello world response sucuessfully. It works just running the sam-cli directly in the terminal, but always fails to find the module when invoked through the vscode launch configuration. I get the same behavior in my actual project, but simplified it down to the built-in hello world template for the purpose of this report.

System details (run AWS: About and/or Amazon Q: About)

OS: Darwin arm64 24.1.0
Visual Studio Code extension host:  1.95.3
AWS Toolkit:  3.37.0
node: 20.18.0
electron: 32.2.1

[jonife]

Hello @zachlipton Thank you for raise this issue.

I wasn't able to reproduce the issue you are experiencing for version 3.36, 3.37

Test for X86

2024-12-05 11:29:26.990 [info] Debugger attached.
2024-12-05 11:29:27.704 [info] END RequestId: 55179370-56b9-47c3-b13a-f4cd476bc139
REPORT RequestId: 55179370-56b9-47c3-b13a-f4cd476bc139	Init Duration: 0.51 ms	Duration: 1282.36 ms	Billed Duration: 1283 ms	Memory Size: 128 MB	Max Memory Used: 128 MB
2024-12-05 11:29:27.705 [info] {"statusCode": 200, "body": "{\"message\":\"hello world\"}"}

Test for arm64

2024-12-05 12:50:08.423 [info] 2024-12-05 12:50:08,421 | Local image is up-to-date
2024-12-05 12:50:08.437 [info] 2024-12-05 12:50:08,436 | Checking free port on 127.0.0.1:7640
2024-12-05 12:50:08.445 [info] 2024-12-05 12:50:08,445 | Using local image: public.ecr.aws/lambda/nodejs:20-rapid-arm64.
2024-12-05 12:50:08.445 [info] 2024-12-05 12:50:08,445 | Mounting /tmp/aws-toolkit-vscode/vsctk92417656/output/HelloWorldFunction as /var/task:ro,delegated, inside runtime container
2024-12-05 12:50:08.613 [info] 2024-12-05 12:50:08,612 | Setting up SIGTERM interrupt handler
2024-12-05 12:50:08.718 [info] 2024-12-05 12:50:08,717 | Getting lock for the key localhost-7640
2024-12-05 12:50:08.719 [info] 2024-12-05 12:50:08,717 | Waiting to retrieve the lock (localhost-7640) to start invocation
2024-12-05 12:50:08.723 [info] START RequestId: a45c1739-5f1b-49bf-b20b-58fcada41b3a Version: $LATEST
2024-12-05 12:50:08.732 [info] Debugger listening on ws://0.0.0.0:5858/f1e3bc9a-9766-41fe-97ec-fccadf8df45e
For help, see: https://nodejs.org/en/docs/inspector
2024-12-05 12:50:08.799 [info] Debugger attached.
2024-12-05 12:50:08.972 [info] END RequestId: 679bb003-767f-47e0-80bb-706c46ceb4c4
REPORT RequestId: 679bb003-767f-47e0-80bb-706c46ceb4c4	Init Duration: 0.05 ms	Duration: 249.22 ms	Billed Duration: 250 ms	Memory Size: 128 MB	Max Memory Used: 128 MB
2024-12-05 12:50:08.972 [info] 
2024-12-05 12:50:08.972 [info] {"statusCode": 200, "body": "{\"message\":\"hello world\"}"}

I used sam init via the terminal to generate an Hello world function of runtime nodejs20 with zip package type and then clicked on invoke via the app-builder. Is this consistent with the steps you took to get this error? If yes

Upon closer examination of the error log, it's evident that we're dealing with a module import issue within the Lambda runtime environment. The specific error message, "Error: Cannot find module 'app'", is particularly telling and warrants further investigation. Could this be that

a) The file 'app.js' or 'app.mjs' doesn't exist in the project.
b) There's a typo in the import statement or the filename.
c) The 'app' module is not in a location where Node.js can find it.
d) Given that the error stack mentions 'index.mjs', can you confirm that the 'app' module is also using the appropriate module system and file extension (.mjs for ESM).

If No: Do you mind pointing out are divergence in the our steps

[zachlipton]

Thanks @jonife!

app.mjs exists in the project. It's created by the template, and for the sake of this testcase, I didn't modify the code or anything from the template in any way; I just used it the way "AWS: Create Lambda SAM Application" created it. When I do that, sam local invoke works fine from the terminal, but attempting to invoke the function through vscode (using the launch configuration created by aws-toolkit-vscode) fails.

If it's helpful, here's a zip of the entire project as it was created by "AWS: Create Lambda SAM Application" (this one is nodejs22.x but the behavior is the same with nodejs20.x as well).

It sounds like you created the Hello World function with sam init in the terminal instead of the "AWS: Create Lambda SAM Application" vscode command. As a test, I went ahead and created a new project using sam init, Hello World, nodejs22, zip package. If I run sam local invoke on it in the terminal, it works fine. But if I open that same new project (the one created with sam init) in vscode, go to Application Builder->sam-app->HelloWorldFunction, click on the run button, and click invoke, I get Error: Cannot find module 'app'. So the issue is exactly the same whether the project is created with sam init or "AWS: Create Lambda SAM Application."

The main distinction I can see is that when I run sam local invoke in the terminal, it mounts the task volume in the container directly from the source dir:

Mounting /Users/zach/sam/sam-app/hello-world as /var/task:ro,delegated, inside runtime container

But when I invoke the function within the vscode UI, it does two things differently. It attaches the debugger, and it mounts the task's code from a directory in /tmp:

Mounting /tmp/aws-toolkit-vscode/vsctk0ff54b82/output/HelloWorldFunction as /var/task:ro,delegated, inside runtime container

I have no idea if that's at all relevant to why it works in the terminal but not when running from inside vscode.

[zachlipton]

Ok I figured out what's going on. I copied the project into /tmp and tried running sam local invoke directly from there, and it failed with the same error.

Which made me realize the issue isn't exactly with invoking the function from vscode; it's that it works when the code is in my homedir but invoking the function fails when aws-toolkit-vscode copies it to /tmp/aws-toolkit-vscode.

The workaround that fixex it for me was to run colima stop and then colima start --vm-type=vz --mount-type=virtiofs --mount /tmp/aws-toolkit-vscode:w, explicitly allowing the docker runtime to be able to mount directories from /tmp/aws-toolkit-vscode (if you had an old configuration like me, you might have to rm -rf ~/.colima to get it to switch to the virtualization framework from qemu). Once I did that, invoking the function in vscode worked perfectly as expected and I'm able to use the debugger.

So I suppose this isn't exactly an aws-toolkit-vscode bug. Fundamentally, it's a docker/colima configuration issue, but an extremely non-obvious one with little in the way of debug messages or online resources to aid diagnosis. It turns out this situation is documented by colima, but because it fails silently, you'd have to already know what's wrong to have any hope of finding that documentation.

I suppose it would be ideal if the toolkit put the output somewhere under the project directory rather than /tmp/aws-toolkit-vscode to avoid this. Or at least detected this condition and provided a suggestion on how to fix it. Up to you if you want to close the issue as out of scope or if there's a way the toolkit or aws-sam-cli could help people out who run into this error, since it's triggered by the fact the toolkit copies everything to /tmp/aws-toolkit-vscode. Either way, at least I hope this info helps others.

[justinmk3]

I suppose it would be ideal if the toolkit put the output somewhere under the project directory

Agreed! Tracked in #2050

Thanks for that very helpful debugging info.