"Legacy" Lambda Layers keep failing deployments

[hisham]

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

20.12.1

Amplify CLI Version

12.12.4

What operating system are you using?

Mac

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

no

Describe the bug

amplify push fails from time to time with error Resource is not in the state stackUpdateComplete. When I look at the error detail in cloudformation, the error is:
The following resource(s) failed to create: [LambdaLayerVersion675d0075, LambdaLayerPermissionAwsAccounts914282159778e0e2307fLegacy406].

and Resource handler returned message: "1 validation error detected: Value '914282159778e0e2307f' at 'principal' failed to satisfy constraint: Member must satisfy regular expression pattern: \d{12}|\*|arn:(aws[a-zA-Z-]*):iam::\d{12}:root (Service: AWSLambdaInternal; Status Code: 400; Error Code: ValidationException; Request ID: f11c4ba0-5e26-4eca-a520-39c4ed933b61; Proxy: null)" (RequestToken: b59897b3-cc95-1d5c-3e8a-7c499e5b1214, HandlerErrorCode: GeneralServiceException)

The issue is these "legacy" layers in my *Layer-awscloudformation-template.json file:

    "LambdaLayerPermissionAwsAccounts914282159778e0e2307fLegacy406": {
      "Type": "AWS::Lambda::LayerVersionPermission",
      "Properties": {
        "Action": "lambda:GetLayerVersion",
        "LayerVersionArn": "arn:aws:lambda:us-east-1:914282159778:layer:essappCliLambdaLayer-hishamdev:406",
        "Principal": "914282159778e0e2307f"
      }
    },

But these layers are not legacy at all. They've been deployed in the last few months. I workaround this issue by deleting the layer manually and removing the "Legacy" entries in the cloudformation file like the one above. However, this error is now affecting the latest layer I have, so if I delete it, my lambdas will not function correctly.

Expected behavior

amplify push should just work and update layers

Reproduction steps

Unclear what the repro steps are, but this issue has been happening now in pretty much every push in one specific environment in my stack.

Project Identifier

3def1a2bd59d61900f734d59f169a578

Log output

# Put your logs below this line

Additional information

This issue is also discussed in #8525 but the resolution there does not work.

Before submitting, please confirm:

• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• I have removed any sensitive information from my code snippets and submission.

[hisham]

I worked around this issue btw via amplify update function and re-confirming the permissions for the troublesome layer version...

[hisham]

this keeps happening pretty much anytime I push a new layer version....

[irothenbaum]

I just noticed this happening for us shortly after we upgraded to node 20x. What was happening (I think) was the cloudformation file was keeping a reference to the previous "Legacy" layer version running on node 16x. But in reality, we had done a few updates using node20 since. Somehow I gues the cloudformation file didn't update to reflect that it was actually already on node20 (or maybe we neglected to commit it?) and therefore didn't need to create a "Legacy" layer at all.

For example, the code below was how our cloudformation file looked before the change. Note that it's referencing version 55 which was a version still using node16. However, in reality our latest layer version was 58 - using node20. By changing the 55 (in bold) to 58, I was able to deploy without issue.

    "LambdaLayerPermissionAwsAccountsXXXXXX8cc9bc8c": {
      "Type": "AWS::Lambda::LayerVersionPermission",
      "Properties": {
        "Action": "lambda:GetLayerVersion",
        "LayerVersionArn": "arn:aws:lambda:us-east-1:XXXXX:layer:backendDependencies-dev:55", <----- changed 55 to 58
        "Principal": "XXXX"
      }
    },
    "LambdaLayerPermissionPrivate8cc9bc8c": {
      "Type": "AWS::Lambda::LayerVersionPermission",
      "Properties": {
        "Action": "lambda:GetLayerVersion",
        "LayerVersionArn": "arn:aws:lambda:us-east-1:XXXXX:layer:backendDependencies-dev:55", <----- changed 55 to 58
        "Principal": {
          "Ref": "AWS::AccountId"
        }
      }
    },

Hopefully that helps others experiencing this problem

[ykethan]

Have not been able to reproduce this behavior but this does appear to be related to #12916