sam build dotnet core sends normal output to STDERR

[abelperezok]

Description

When running sam build on a dotnet core lambda project, most of the output is sent to STDERR as opposed to STDOUT. This can be particularly confusing when running this command as part of a CI pipeline such as Azure DevOps as it can fail the step for no good reason.

Steps to reproduce

Run sam build and at glance all the output seems ok, but it's sending a big chunk of output to STDERR.

Observed result

Following a trick found in another thread azure-pipelines-tasks/issues/6840 I can see that most of the output is actually sent to STDERR.

The trick is to add 2> >(while read line; do (>&2 echo "STDERROR: $line"); done) at the end of the command and it will prepend "STDERROR: " to each line sent to STDERR.

$ sam build 2> >(while read line; do (>&2 echo "STDERROR: $line"); done)
STDERROR: Building function 'EmailSenderFunction'
STDERROR: Running DotnetCliPackageBuilder:GlobalToolInstall
STDERROR: 
STDERROR: Tool 'amazon.lambda.tools' was reinstalled with the latest stable version (version '4.1.0').
STDERROR: Running DotnetCliPackageBuilder:RunPackageAction
STDERROR: Amazon Lambda Tools for .NET Core applications (4.1.0)
STDERROR: Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet
STDERROR: 
STDERROR: Executing publish command
STDERROR: Deleted previous publish folder
STDERROR: ... invoking 'dotnet publish', working folder '/home/abel/Documents/email-sender-lambda/src/EmailSender/bin/Release/netcoreapp3.1/publish'
STDERROR: ... dotnet publish --output "/home/abel/Documents/email-sender-lambda/src/EmailSender/bin/Release/netcoreapp3.1/publish" --configuration "Release" --framework "netcoreapp3.1" /p:GenerateRuntimeConfigurationFiles=true --runtime linux-x64 --self-contained false
STDERROR: ... publish: Microsoft (R) Build Engine version 16.7.0-preview-20360-03+188921e2f for .NET
STDERROR: ... publish: Copyright (C) Microsoft Corporation. All rights reserved.
STDERROR: ... publish:   Determining projects to restore...
STDERROR: ... publish:   Restored /home/abel/Documents/email-sender-lambda/src/EmailSender/EmailSender.csproj (in 502 ms).
STDERROR: ... publish:   EmailSender -> /home/abel/Documents/email-sender-lambda/src/EmailSender/bin/Release/netcoreapp3.1/linux-x64/EmailSender.dll
STDERROR: ... publish:   EmailSender -> /home/abel/Documents/email-sender-lambda/src/EmailSender/bin/Release/netcoreapp3.1/publish/
STDERROR: Changed permissions on published file (chmod +rx AWSSDK.Extensions.NETCore.Setup.dll).
STDERROR: Changed permissions on published file (chmod +rx Microsoft.Extensions.Configuration.dll).
STDERROR: Changed permissions on published file (chmod +rx EmailSender.pdb).
STDERROR: Changed permissions on published file (chmod +rx EmailSender.deps.json).
STDERROR: Changed permissions on published file (chmod +rx AWSSDK.Core.dll).
STDERROR: Changed permissions on published file (chmod +rx Amazon.Extensions.Configuration.SystemsManager.dll).
STDERROR: Changed permissions on published file (chmod +rx EmailSender.runtimeconfig.json).
STDERROR: Changed permissions on published file (chmod +rx Amazon.Lambda.Core.dll).
STDERROR: Changed permissions on published file (chmod +rx Microsoft.Extensions.Primitives.dll).
STDERROR: Changed permissions on published file (chmod +rx Microsoft.Extensions.Configuration.Abstractions.dll).
STDERROR: Changed permissions on published file (chmod +rx EmailSender.dll).
STDERROR: Changed permissions on published file (chmod +rx Microsoft.Extensions.DependencyInjection.Abstractions.dll).
STDERROR: Changed permissions on published file (chmod +rx Amazon.Lambda.Serialization.SystemTextJson.dll).
STDERROR: Changed permissions on published file (chmod +rx Amazon.Lambda.SQSEvents.dll).
STDERROR: Changed permissions on published file (chmod +rx Microsoft.Extensions.Logging.Abstractions.dll).
STDERROR: Changed permissions on published file (chmod +rx AWSSDK.SimpleSystemsManagement.dll).
STDERROR: Changed permissions on published file (chmod +rx Newtonsoft.Json.dll).
STDERROR: Zipping publish folder /home/abel/Documents/email-sender-lambda/src/EmailSender/bin/Release/netcoreapp3.1/publish to /home/abel/Documents/email-sender-lambda/.aws-sam/build/EmailSenderFunction/EmailSender.zip
STDERROR: Creating directory /home/abel/Documents/email-sender-lambda/.aws-sam/build/EmailSenderFunction

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided
    
STDERROR: ... zipping:   adding: AWSSDK.Extensions.NETCore.Setup.dll (deflated 55%)
STDERROR: ... zipping:   adding: Microsoft.Extensions.Configuration.dll (deflated 46%)
STDERROR: ... zipping:   adding: EmailSender.pdb (deflated 42%)
STDERROR: ... zipping:   adding: EmailSender.deps.json (deflated 84%)
STDERROR: ... zipping:   adding: AWSSDK.Core.dll (deflated 67%)
STDERROR: ... zipping:   adding: Amazon.Extensions.Configuration.SystemsManager.dll (deflated 51%)
STDERROR: ... zipping:   adding: EmailSender.runtimeconfig.json (deflated 23%)
STDERROR: ... zipping:   adding: Amazon.Lambda.Core.dll (deflated 46%)
STDERROR: ... zipping:   adding: Microsoft.Extensions.Primitives.dll (deflated 50%)
STDERROR: ... zipping:   adding: Microsoft.Extensions.Configuration.Abstractions.dll (deflated 47%)
STDERROR: ... zipping:   adding: EmailSender.dll (deflated 57%)
STDERROR: ... zipping:   adding: Microsoft.Extensions.DependencyInjection.Abstractions.dll (deflated 54%)
STDERROR: ... zipping:   adding: Amazon.Lambda.Serialization.SystemTextJson.dll (deflated 47%)
STDERROR: ... zipping:   adding: Amazon.Lambda.SQSEvents.dll (deflated 49%)
STDERROR: ... zipping:   adding: Microsoft.Extensions.Logging.Abstractions.dll (deflated 55%)
STDERROR: ... zipping:   adding: AWSSDK.SimpleSystemsManagement.dll (deflated 73%)
STDERROR: ... zipping:   adding: Newtonsoft.Json.dll (deflated 63%)
STDERROR: Created publish archive (/home/abel/Documents/email-sender-lambda/.aws-sam/build/EmailSenderFunction/EmailSender.zip).
STDERROR: Lambda project successfully packaged: /home/abel/Documents/email-sender-lambda/.aws-sam/build/EmailSenderFunction/EmailSender.zip

Another simpler way to test it is to get rid of the error log by redirecting to null:

$ sam build 2>/dev/null

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided
    

Only a few lines are displayed as normal standard output.

Expected result

No output sent to STDERR unless it's actually an error with the command.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS:

$ uname -a
Linux Debian-Bullseye 5.7.0-3-amd64 aws/aws-sam-cli#1 SMP Debian 5.7.17-1 (2020-08-23) x86_64 GNU/Linux

2. sam --version:

SAM CLI, version 1.3.2

[major-mayer]

Don't know if this is related, but when you execute sam local start-lambda all the container output is also sent to stderr.
This makes it difficult to differentiate between real error output and standard logs when using the local lambda endpoint to test the lambda functions.

[sriram-mv]

@abelperezok Thanks for raising this! would you consider a general rule that all workflows would need to get all their output over to stdout?

[abelperezok]

@sriram-mv Yes, I'd consider a general rule to get all workloads output to stdout, I'm not sure how complex it is internally but ideally only errors should go to stderr as it can confuse the caller process about the invocation outcome.

[mndeveci]

Hi @abelperezok ,

When I checked for dotnet workflow, I saw that we are piping the output of the dotnet CLI that we run for the build.

aws-lambda-builders/aws_lambda_builders/workflows/dotnet_clipackage/dotnetcli.py

Line 57 in 8d817fd

p = self.os_utils.popen(invoke_dotnet, stdout=self.os_utils.pipe, stderr=self.os_utils.pipe, cwd=cwd)

Since we are just piping whatever output comes from the underlying CLI that we run, could this be coming from dotnet CLI?