Change in Ref'd Parameter Values do not trigger Function Alias update via AutoPublishAlias

[mokele]

Description:

The auto generation of Logical Resource Ids for AWS::Lambda::Version resources created do not take into account the Value of Parameters Ref'd within the Environment Variables block, but rather the Ref block configuration itself (e.g. {"Environment":{"Variables":{"MyEnv":"MyNewValue"}}}. The end result being that a new Version is not created and the AutoPublishAlias feature does not update the alias for the function to the new version that is created, and only the unqualified function version has its configuration updated.

The example below is shortened, but once DeploymentPreferences are introduced this can result in very unexpected behaviour whereby you think your code is updated to new configuration, but all existing usage of the alias will not be invoking the newly configured function. The only solution outside of this would be to simply no longer use Environment Variables for passing through parameterised configuration, but rather use SSM or AppConfig and write code to utilise these instead from within the Lambda implementation, or to increment or add another hardcoded value within every variable block to force an update.

Environment variables are simply not usable for passing through parameterised configuration (that's expected to be able to change) to lambdas when using AutoPublishAlias opening up the use of DeploymentPreferences

Steps to reproduce the issue:

1. Deploy a stack with the following template

# template.yml
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31

Parameters:
  MyParam:
    Type: String
    Default: MyDefaultValue

Resources:
  Function:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: ./
      Handler: index.handler
      Runtime: nodejs12.x
      AutoPublishAlias: live
      Environment:
        Variables:
          MyEnv: !Ref MyParam

// index.js
export const handler = event => {
    console.log('MyEnv', process.env.MyEnv)
}

2. Deploy for a 2nd time with added parameter overrides --parameter-overrides MyParam=MyNewValue

Observed result:

• $LATEST unqualified version has environment variable MyEnv=MyNewValue
• live Alias'd version not updated and has environment variable MyEnv=MyDefaultValue

Expected result:

• $LATEST unqualified version has environment variable MyEnv=MyNewValue
• live Alias'd version updated and has environment variable MyEnv=MyNewValue

[mokele]

This also include's usage of CloudFormation SSM Parameter types:
e.g.

Parameters:
  MyParam:
    Type: AWS::SSM::Parameter::Value<String>
    Default: /my/ssm/parameter/name

meaning that any deployment after a value is changed will not pick up the new value, defeating the object of using this format vs {{resolve:ssm...} syntax, and again AppConfig might be the only option and fully adopting a split between configuration deployments and code deployments wholesale for all SAM projects that may want to use this type of feature

[sforcier]

I have run into this, or something like it, except in my experience $LATEST isn't updated either. So if I took your template, deployed it once, then I updated the template by changing MyParam to the following:

  MyParam:
    Type: String
    Default: TotallyNewValue

I then performed a sam build and sam deploy and the environment variable was not updated in either $LATEST or live.

[mokele]

I have run into this, or something like it, except in my experience $LATEST isn't updated either. So if I took your template, deployed it once, then I updated the template by changing MyParam to the following:

  MyParam:
    Type: String
    Default: TotallyNewValue

I then performed a sam build and sam deploy and the environment variable was not updated in either $LATEST or live.

that is the correct behaviour – Default does not change an existing value, it only populates when either the parameter or stack is new and no parameter override was provided at that time

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html

A value of the appropriate type for the template to use if no value is specified when a stack is created.

[mokele]

I'm incredibly surprised this isn't being noticed by a lot more people – I'm fairly confident that this bug has gone under the radar for a lot of people and their aliases have not been updated the way they expect and are still using old versions.

Either that or a lot of people simply aren't using deployment preferences yet due to the barrier to entry being pretty confusing – for example having to have CodeDeployHook_ as the prefix for your traffic hook name or else code deploy can't call your function which is difficult to find out

[mokele]

I'm incredibly surprised this isn't being noticed by a lot more people – I'm fairly confident that this issue has gone under the radar for a lot of people and their aliases have not been updated the way they expect and are still using old versions.

Either that or a lot of people simply aren't using deployment preferences with SAM yet due to the barrier to entry being pretty confusing – for example having to have CodeDeployHook_ as the prefix for your traffic hook name or else code deploy can't call your function which is difficult to find out amongst other nuances

[cunymatthieu]

I have the same concern. It happens when I change the parameter value in --parameter-overrides.

SAM does not detect any change in the template file. And the AutoPublishAlias feature, fails to create a new version and change the alias. However created correctly the $LATEST version.

[ferdinandsalis]

I am have experienced the same issue but regarding with layers as mentioned in #1777

[jfalkenstein]

This issue is really hampering my plans and completely defeats the purpose of using AutoPublishAlias. We use layers to store a lot of the core logic of our lambdas. Please, can this issue be prioritized?

[npalethorpe]

Appreciate this isn't a solution but as there are no work-arounds above I felt I should offer one!

In my project I have a deployment script which deploys my project using a template.json file using SAM CLI. I was hitting this issue whenever I would update + deploy only a Layer function change - I'd then find that any unchanged lambdas utilising the new Layer would appear to use an older version of the Layer - as stated above.

The workaround was as simple as writing a little function to iterate through my lambda folder and update all Lambdas package.json files to include a new 'buildNumber' property which I increment with each deploy. I keep a 'base' buildNumber property in my projects root package.json file which I use to increment each time. Basically any change to a Lambda seems to trigger an update so we just exploit that here.

So my solution is:

1. In my root package.json I create a new 'buildNumber' value as "0".
2. I run a function on each deploy which reads the root buildNumber, increments it and stores the new buildNumber in all package.json files in my Lambda + Layers folder (and my root package.json otherwise we'd never increment!).
3. Then I push the deploy to AWS.

In case it helps anyone; the function I use to iterate through my Lambda folder and update the buildNumber values is as follows: (./api is the folder that contains my Lambdas' and my 'Layers' folder)

const updateLambdaBuildNumber = (buildNumber) => {
    return new Promise((resolve, reject) => {
        const recurse = (dir) => {
            const content = fs.readdirSync(dir);
            for (const c of content) {
                const stats = fs.statSync(`${dir}/${c}`);
                if (stats.isDirectory() && c !== "node_modules") {
                    recurse(`${dir}/${c}`);
                } else if (stats.isFile() && c === "package.json") {
                    let package_json = JSON.parse(fs.readFileSync(`${dir}/${c}`));
                    package_json.buildNumber = (buildNumber || 0).toString();
                    fs.writeFileSync(`${dir}/${c}`, JSON.stringify(package_json, null, 4));
                }
            }
        };
        recurse("./api");
        resolve();
    });
};

[AndreaCatalucciTractable]

hi, this is blocking more than one of our teams, and the workarounds suggested (bumping version on every deploy or adding a listener for the new layer publish event) feel quite cumbersome. Since the issue has been open for almost two years now, I wonder whether there are there any cleaner workarounds suggested or plans to fix? thanks!

[ljayatRabbit]

Does using the cloudformation resources AWS::Lambda::Alias and AWS::Lambda::Version solves the problem?

[hoffa]

You might be able to get this to work by adding AWS::LanguageExtensions to Transform as such:

Transform:
  - AWS::LanguageExtensions
  - AWS::Serverless-2016-10-31

AWS::LanguageExtensions resolves intrinsic functions if the value is known when Transforms are run.

See #2533 for more information.

[hoffa]

Closing in favor of #2533.