Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: recursively copies artifact dir when it's in source dir #382

Closed
israelg99 opened this issue Aug 13, 2022 · 5 comments
Closed

Bug: recursively copies artifact dir when it's in source dir #382

israelg99 opened this issue Aug 13, 2022 · 5 comments

Comments

@israelg99
Copy link

Recursively copies artifact dir when it's in source dir until it errors out:

JSON-RPC input:

{
...
"source_dir":    "path/to/source",
"artifacts_dir":  "path/to/source/.build/artifact",
...
}

Then running lambda-builders <JSON INPUT>

Results in

PythonPipBuilder:CopySource - [Errno 63] File name too long: '/path/to/source/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/.build/artifact/python_dateutil-2.8.2.dist-info/top_level.txt'

Setting artifact_dir outside the source_dir solves this issue.


Is this a bug, or is this expected? Perhaps we should add ignore: {} interface? I can contribute a PR for this if you guys can point me in the right direction.


Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: Mac
  2. If using SAM CLI, sam --version:
  3. AWS region: us-east-1 (shouldn't matter)

Add --debug flag to any SAM CLI commands you are running

@israelg99 israelg99 added stage/needs-triage Automatically applied to new issues and PRs, indicating they haven't been looked at. type/bug labels Aug 13, 2022
@jfuss
Copy link
Contributor

jfuss commented Aug 16, 2022

@israelg99 artifact_dir and source_dir should be different directories unless you wanted to embed (and copy them) for some reason. To my knowledge, we don't want to restrict the behavior here so we don't want to fail the run. Maybe we should? But not sure what that could break now, if we do.

On the ignore interface, I am not sure. It seems like a good idea and something we might be able to leverage in SAM CLI to ignore different directories or files. I can check with the team on the support. My concern is more about making sure we have the right scope and not locking ourselves into something we couldn't expand. For example, we would need to make sure its something we can support in SAM CLI, as I know customers have wanted support for ignoring files in their source. Your ask seems slightly different (maybe it isn't) but my point is we just need to really think through all the use-cases for something like this.

In the mean time, my suggestion is to use separate directories for each as this was how Lambda Builders (and SAM CLI) was designed.

@jfuss jfuss added type/feature Feature request and removed type/bug maintainer/need-response stage/needs-triage Automatically applied to new issues and PRs, indicating they haven't been looked at. labels Aug 16, 2022
@pkit
Copy link

pkit commented Jan 14, 2023

@jfuss

In the mean time, my suggestion is to use separate directories for each as this was how Lambda Builders (and SAM CLI) was designed.

Err... how exactly it should be done?
Copying all the files three times or what?
Let's think about a typical project:

/home/user/my_project/
/home/user/my_project/main.py
/home/user/my_project/requirements.txt
/home/user/my_project/my_module/
/home/user/my_project/my_module/module.py

What's the proposed solution for building this one?
Typical solution:

/home/user/my_project/.build

will fail miserably because of that strange behavior.


The solution that I would do will have something like:

{
  ....
  ignore_file: ".ignore"
}

Where:

cat .ignore
.build/

@pkit
Copy link

pkit commented Jan 14, 2023

Ah, this one is so cute!





So, aws-sam does suffer from the same bug, but you've just hardcoded the escape hatch...
Nice!

@israelg99
Copy link
Author

@pkit Agreed. I think exposing this escape hatch with some ignore: interface would be helpful.

@torresxb1
Copy link
Contributor

We have a few issues asking for a similar thing in ignore: or a .samignore file. Closing so that we can track here: #185

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants