Refactor helper binaries to save 161MB of disk space when the agent is installed and reduce RPM by 48MB #1454
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…-agent to save 114M on disk and 35MB on the RPM
chadpatel
commented
Dec 3, 2024
cmd/config-translator/translator.go
Outdated
| @@ -1,137 +1,61 @@ | |||
| // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. | |||
| // SPDX-License-Identifier: MIT | |||
There was a problem hiding this comment.
I can bring this back, lost it when moving code around
chadpatel
commented
Dec 3, 2024
| @@ -95,6 +96,16 @@ var fRunAsConsole = flag.Bool("console", false, "run as console application (win | |||
| var fSetEnv = flag.String("setenv", "", "set an env in the configuration file in the format of KEY=VALUE") | |||
| var fStartUpErrorFile = flag.String("startup-error-file", "", "file to touch if agent can't start") | |||
|
|
|||
| // config-translator | |||
| var fConfigTranslator = flag.Bool("config-translator", false, "run in config-translator mode") | |||
There was a problem hiding this comment.
I spent some time trying to reduce the copy/paste aspect. Some of these constants are defined twice. There are some things to do to clean it up but they all end up less readable and add a lot of indirection. Open to ideas tho.
If I was forced to make one change I would put the descriptions in a constant
chadpatel
commented
Dec 3, 2024
| @@ -38,3 +40,195 @@ func TestTranslateJsonMapToEnvConfigFile(t *testing.T) { | |||
| assert.Equal(t, expectedJson[envconfig.CWAGENT_LOG_LEVEL], actualJson[envconfig.CWAGENT_LOG_LEVEL]) | |||
| assert.Equal(t, expectedJson[envconfig.AWS_SDK_LOG_LEVEL], actualJson[envconfig.AWS_SDK_LOG_LEVEL]) | |||
| } | |||
|
|
|||
| func TestAgentConfig(t *testing.T) { | |||
There was a problem hiding this comment.
this code was moved from translator_test.go
chadpatel
commented
Dec 3, 2024
translator/cmdutil/translator.go
Outdated
| ctx *context.Context | ||
| } | ||
|
|
||
| func NewConfigTranslator(inputOs, inputJsonFile, inputJsonDir, inputTomlFile, inputMode, inputConfig, multiConfig string) (*ConfigTranslator, error) { |
There was a problem hiding this comment.
this code was moved from the old config-translator/translator.go
|
This PR was marked stale due to lack of activity. |
…urther decrease deployment size
chadpatel
changed the title
chadpatel
changed the title
|
This PR was marked stale due to lack of activity. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of the issue
The cloudwatch-agent has 3 helper binaries that use an excessive amount of disk
The reason is they are directly and indirectly pulling dependencies from cloudwatch-agent. To solve this problem, I updated config-downloader, config-translator and the wizard to be shims that just redirect to amazon-cloudwatch-agent binary
This works fine, the main risk is these binaries are no longer "portable", they depend on finding the path to amazon-cloudwatch-agent at runtime. I am using the same method as
start-amazon-cloudwatch-agentfor finding the pathDescription of changes
High level the approach is to maintain the same argument interface for the existing 3 commands and seamlessly move the logic in to the main binary. To do this the old commands need to keep the same args but we prefix the args when we call CWA so that there are no duplicate args.
The general approach I took was to create a new
cmdwrapperwhich offers two methods.flagAPI for pulling command line args. I considered using subcommands but things got too complicated and ugly, prefixing was simpleramazon-cloudwatch-agentand it calls it with the new flags. It remaps stdin/stdout/stderr so it appears seemless. Things like the wizard which rely on stdin still workI moved the flags in to their own separate
flagspackages which have NO dependencies (keeping the binaries small). And then the old binaries and amazon-cloudwatch-agent pull in the flags and the commands and link everything together. For the wizard I had to move a few other common constants like the inlinuxMigration.goandwindows_migration.goI added and updated unit tests wherever possible. The old translator_test.go was moved in to translatorutil_test.go as that is primarily what those tests were testing
Note: inline diff is VERY hard to follow because of all the moved code, I recommend
splitdiff or we can do a code walk-throughLicense
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Tests
Manual testing to confirm cloudwatch agent still starts/loads
There were no prior tests that actually tested
config-translatebinary... the old tests basically just testedcmdutilso I moved those in to thecmdutiltests. We could write more tests for the shim and fortranslate.go. Tricky tests to write due to the OS coupling.Before:
total 136M
-rwxr-xr-x 1 patchad amazon 129M Dec 16 20:34 amazon-cloudwatch-agent
-rwxr-xr-x 1 patchad amazon 1.7M Dec 16 20:34 amazon-cloudwatch-agent-config-wizard
-rwxr-xr-x 1 patchad amazon 1.7M Dec 16 20:34 config-downloader
-rwxr-xr-x 1 patchad amazon 1.7M Dec 16 20:34 config-translator
-rwxr-xr-x 1 patchad amazon 2.1M Dec 16 20:34 start-amazon-cloudwatch-agent
-rw-r--r-- 1 patchad amazon 65M Dec 16 20:35 /local/home/patchad/workplace/cwa/amazon-cloudwatch-agent/build/bin/linux/amd64/amazon-cloudwatch-agent.rpm
2024/12/17 21:54:39 Starting config-wizard, this will map back to a call to amazon-cloudwatch-agent
2024/12/17 21:54:39 Executing /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent with arguments: [-config-wizard -config-wizard-is-non-interactive-windows-migration false -config-wizard-use-parameter-store false -config-wizard-is-non-interactive-linux-migration false -config-wizard-traces-only false -config-wizard-non-interactive-xray-migration false]
= Welcome to the Amazon CloudWatch Agent Configuration Manager =
= =
= CloudWatch Agent allows you to collect metrics and logs from =
= your host and send them to CloudWatch. Additional CloudWatch =
= charges may apply. =
On which OS are you planning to use the agent?
default choice: [1]:
1
Trying to fetch the default region based on ec2 metadata...
I! imds retry client will retry 1 timesD! should retry false for imds error : RequestCanceled: EC2 IMDS access disabled via AWS_EC2_METADATA_DISABLED env varW! could not get region from ec2 metadata... EC2MetadataRequestError: failed to get EC2 instance identity document
caused by: RequestCanceled: EC2 IMDS access disabled via AWS_EC2_METADATA_DISABLED env varAre you using EC2 or On-Premises hosts?
default choice: [2]:
➜ amazon-cloudwatch-agent git:(patchad-config-translate-refactor) ✗ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
****** processing amazon-cloudwatch-agent ******
2024/12/17 21:56:21 Starting config-downloader, this will map back to a call to amazon-cloudwatch-agent
2024/12/17 21:56:21 Executing /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent with arguments: [-config-downloader -config-downloader-config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml -config-downloader-multi-config default -config-downloader-mode ec2 -config-downloader-download-source file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -config-downloader-output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d]
I! Trying to detect region from ec2 D! [EC2] Found active network interface I! imds retry client will retry 1 times
Start configuration validation...
2024/12/17 21:56:21 Starting config-translator, this will map back to a call to amazon-cloudwatch-agent
2024/12/17 21:56:21 Executing /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent with arguments: [-config-translator -config-translator-multi-config default -config-translator-input /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -config-translator-input-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d -config-translator-output /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml -config-translator-mode ec2 -config-translator-config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml]
2024-12-17T21:56:21Z I! Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_amazon-cloudwatch-agent.json.tmp ...
2024-12-17T21:56:21Z I! Valid Json input schema.
2024-12-17T21:56:21Z I! Configuration validation first phase succeeded
I! Trying to detect region from ec2
D! [EC2] Found active network interface
I! imds retry client will retry 1 times
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
Configuration validation second phase succeeded
Configuration validation succeeded
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
****** processing amazon-cloudwatch-agent ******
2024/12/17 21:57:13 Starting config-downloader, this will map back to a call to amazon-cloudwatch-agent
2024/12/17 21:57:13 Executing /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent with arguments: [-config-downloader -config-downloader-mode ec2 -config-downloader-download-source file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -config-downloader-output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d -config-downloader-config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml -config-downloader-multi-config default]
2024-12-17T21:57:13Z E! Failed to initialize config downloader: fail to fetch/remove json config: open /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json: no such file or directory
2024/12/17 21:57:13 E! Translation process exited with non-zero status: 1, err: exit status 1
panic: E! Translation process exited with non-zero status: 1, err: exit status 1
goroutine 1 [running]:
log.Panicf({0x4e8cc3?, 0xc0000c8020?}, {0xc0000c3de0?, 0xc000098038?, 0x3?})
log/log.go:439 +0x65
github.com/aws/amazon-cloudwatch-agent/tool/cmdwrapper.ExecuteAgentCommand({0x4e165b, 0x11}, 0xc0000a4120)
github.com/aws/amazon-cloudwatch-agent/tool/cmdwrapper/cmdwrapper.go:59 +0x5df
main.main()
github.com/aws/amazon-cloudwatch-agent/cmd/config-downloader/downloader.go:20 +0xc8
➜ amazon-cloudwatch-agent git:(patchad-config-translate-refactor) ✗ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
****** processing amazon-cloudwatch-agent ******
2024/12/17 21:58:07 E! Fail to fetch/remove json config: open /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json: no such file or directory