Metadata error logs in Cloudwatch agent - 404 EC2MetadataError: failed to make EC2Metadata request #1435

lesterxy · 2024-11-20T09:44:56Z

Describe the bug

cloudwatch agent logs throw repeated errors.
status code: 404, request id: D! should retry true for imds error : EC2MetadataError: failed to make EC2Metadata request

This only happened on v.1300049.
On v.1.300048.1b904, agent don't have error logs.

Steps to reproduce

Create an Ubuntu 22.04.5 LTS EC2 instance.
Create and attach Role to EC2 with "CloudWatchAgentServerPolicy" managed policy.
Install the cloudwatch agent using the documentation (root user):
3.1. wget https://amazoncloudwatch-agent.s3.amazonaws.com/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
3.3. dpkg -i -E ./amazon-cloudwatch-agent.deb
3.4. /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
3.5. /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
Check the agent status: systemctl status amazon-cloudwatch-agent

What did you expect to see?
No errors on the status logs.

What did you see instead?

● amazon-cloudwatch-agent.service - Amazon CloudWatch Agent
     Loaded: loaded (/etc/systemd/system/amazon-cloudwatch-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-11-20 09:34:55 UTC; 55s ago
   Main PID: 1456 (amazon-cloudwat)
      Tasks: 7 (limit: 1078)
     Memory: 21.0M
        CPU: 389ms
     CGroup: /system.slice/amazon-cloudwatch-agent.service
             └─1456 /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -config /opt/aws/amazon-cloudwatch-agent/etc/amazon>

Nov 20 09:34:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
Nov 20 09:34:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]:                  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transi>
Nov 20 09:34:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]: <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang=">
Nov 20 09:34:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]:  <head>
Nov 20 09:34:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]:   <title>404 - Not Found</title>
Nov 20 09:34:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]:  </head>
Nov 20 09:34:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]:  <body>
Nov 20 09:34:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]:   <h1>404 - Not Found</h1>
Nov 20 09:34:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]:  </body>
Nov 20 09:34:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]: </html>

from journalctl, this log keeps repeating

Nov 20 09:39:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]:         status code: 404, request id: D! should retry true for imds error : EC2MetadataError: failed to make EC2Metadata request
Nov 20 09:39:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]: <?xml version="1.0" encoding="iso-8859-1"?>
Nov 20 09:39:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
Nov 20 09:39:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]:                  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
Nov 20 09:39:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]: <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
Nov 20 09:39:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]:  <head>
Nov 20 09:39:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]:   <title>404 - Not Found</title>
Nov 20 09:39:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]:  </head>
Nov 20 09:39:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]:  <body>
Nov 20 09:39:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]:   <h1>404 - Not Found</h1>
Nov 20 09:39:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]:  </body>
Nov 20 09:39:55 ip-172-31-20-11 start-amazon-cloudwatch-agent[1456]: </html>

What version did you use?
v1.300049.1b929

What config did you use?
Reference agent config output:

{
        "agent": {
                "metrics_collection_interval": 60,
                "run_as_user": "cwagent"
        },
        "metrics": {
                "aggregation_dimensions": [
                        [
                                "InstanceId"
                        ]
                ],
                "append_dimensions": {
                        "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
                        "ImageId": "${aws:ImageId}",
                        "InstanceId": "${aws:InstanceId}",
                        "InstanceType": "${aws:InstanceType}"
                },
                "metrics_collected": {
                        "disk": {
                                "measurement": [
                                        "used_percent"
                                ],
                                "metrics_collection_interval": 60,
                                "resources": [
                                        "*"
                                ]
                        },
                        "mem": {
                                "measurement": [
                                        "mem_used_percent"
                                ],
                                "metrics_collection_interval": 60
                        }
                }
        }
}

Environment
OS version: Ubuntu 22.04.5 LTS

Additional context
This only happened on v.1300049.
On v.1.300048.1b904, agent don't have error logs.

The text was updated successfully, but these errors were encountered:

debu99 · 2024-11-24T01:23:58Z

how to download v.1.300048.1b904 binary?

tdtm · 2024-11-28T23:40:55Z

I'm also encountering this in v1.300049.1b929.

EDIT: additional details from #1440 indicates:

In version 1.300049.0 and above, the agent will log the above message regardless of log levels on a defined interval. This is because we recently enabled instance tags by default to retrieve ASG name and instance tag name for entity service names. This becomes an issue when instance metadata tags is not enabled which can be majority case since instance metadata tags is an opt-in feature. The issue is especially apparent in EKS since EKS does not support instance metadata tags

Instance metadata tags instructions: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/work-with-tags-in-IMDS.html#allow-access-to-tags-in-IMDS

martinvocast · 2024-12-03T10:09:11Z

I'm also encountering this in v1.300049.1b929.

EDIT: additional details from #1440 indicates:

In version 1.300049.0 and above, the agent will log the above message regardless of log levels on a defined interval. This is because we recently enabled instance tags by default to retrieve ASG name and instance tag name for entity service names. This becomes an issue when instance metadata tags is not enabled which can be majority case since instance metadata tags is an opt-in feature. The issue is especially apparent in EKS since EKS does not support instance metadata tags

Instance metadata tags instructions: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/work-with-tags-in-IMDS.html#allow-access-to-tags-in-IMDS

This fixed it for me, thank you! I was wondering why my CloudWatch metrics stopped working yesterday - enabling the tags access on the EC2 did it.

motiumansky · 2024-12-03T15:15:29Z

Reached out to AWS support per this. This is a bug that will be resolved in the next version. They advice against allowing access to tags in instance metadata. In the meanwhile, you can use 1.300048.1b904.

ibaraki-douji · 2024-12-04T11:06:35Z

Yes, downgrading to 1.300048.1b904 works (to use it, replace the latest keyword on the CW Agent download link with the version)

I also tried to keep the latest and enable the metadata on the EC2, the 404 error was gone but no logs where uploaded to CW.
So ye downgrading the version was the only option to make it work again for me.

iaon · 2024-12-06T18:39:48Z

how to download v.1.300048.1b904 binary?

https://amazoncloudwatch-agent.s3.amazonaws.com/amazon_linux/arm64/1.300048.1b904/amazon-cloudwatch-agent.rpm

6lick · 2024-12-13T05:43:03Z

How do you get this to work from .ebextensions in an elastic beanstalk environment? I cannot get it to work.

files:
  "/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json":
    mode: "000644"
    owner: root 
    group: root
    content: |
      {
        "agent": {
          "metrics_collection_interval": 60,
          "run_as_user": "root",
          "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log"
        },
        "metrics": {
          "namespace": "System/Linux",
          "append_dimensions": {
            "InstanceId": "${aws:InstanceId}",
            "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
            "ImageId": "${aws:ImageId}",
            "InstanceType": "${aws:InstanceType}"
          },
          "metrics_collected": {
            "mem": {
              "measurement": [
                "mem_used_percent",
                "mem_used",
                "mem_total"
              ],
              "metrics_collection_interval": 60
            },
            "disk": {
              "measurement": [
                "disk_used_percent"
              ],
              "resources": [
                "/"
              ]
            }
          }
        },
        "logs": {
          "logs_collected": {
            "files": {
              "collect_list": [
                {
                  "file_path": "/var/log/messages",
                  "log_group_name": "/aws/elasticbeanstalk/{instance_id}/var/log/messages",
                  "log_stream_name": "{instance_id}",
                  "timezone": "UTC"
                }
              ]
            }
          }
        }
      }

commands:
  "01_stop_service":
    command: |
      if systemctl is-active amazon-cloudwatch-agent; then
        systemctl stop amazon-cloudwatch-agent
      fi

  "02_download_cloudwatch_agent":
    command: |
      wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/1.300048.1b904/amazon-cloudwatch-agent.rpm -O /tmp/amazon-cloudwatch-agent.rpm
      rpm -U /tmp/amazon-cloudwatch-agent.rpm
      rm -f /tmp/amazon-cloudwatch-agent.rpm

container_commands:
  "01_setup_agent":
    command: |
      mkdir -p /opt/aws/amazon-cloudwatch-agent/etc
      cp /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json.bak || true
      chmod 644 /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

  "02_start_agent":
    command: |
      /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
      systemctl enable amazon-cloudwatch-agent
      systemctl restart amazon-cloudwatch-agent```

jparsonstf · 2024-12-14T01:33:32Z

This is happening with the RPM version as well for AL2 / amzn2
amazon-cloudwatch-agent.x86_64 1.300049.1b929-1

wilfredt · 2024-12-18T02:50:49Z

Same problem in Rocky 9 with amazon-cloudwatch-agent 1.300049.1b929. "Access to tags in instance metadata" already enabled. Both IMDSv2 set to optional / required still same problem.

EarningsCall · 2024-12-22T17:16:36Z

I'm also encountering this in v1.300049.1b929.
EDIT: additional details from #1440 indicates:

In version 1.300049.0 and above, the agent will log the above message regardless of log levels on a defined interval. This is because we recently enabled instance tags by default to retrieve ASG name and instance tag name for entity service names. This becomes an issue when instance metadata tags is not enabled which can be majority case since instance metadata tags is an opt-in feature. The issue is especially apparent in EKS since EKS does not support instance metadata tags

Instance metadata tags instructions: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/work-with-tags-in-IMDS.html#allow-access-to-tags-in-IMDS

This fixed it for me, thank you! I was wondering why my CloudWatch metrics stopped working yesterday - enabling the tags access on the EC2 did it.

This also fixed it for me. I allowed instance metadata in my launch template like so:

zhihonl mentioned this issue Nov 21, 2024

[Bug Fix] Fix Excessive IMDS related error logging #1440

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Metadata error logs in Cloudwatch agent - 404 EC2MetadataError: failed to make EC2Metadata request #1435

Metadata error logs in Cloudwatch agent - 404 EC2MetadataError: failed to make EC2Metadata request #1435

lesterxy commented Nov 20, 2024

debu99 commented Nov 24, 2024

tdtm commented Nov 28, 2024 •

edited

Loading

martinvocast commented Dec 3, 2024

motiumansky commented Dec 3, 2024 •

edited

Loading

ibaraki-douji commented Dec 4, 2024

iaon commented Dec 6, 2024

6lick commented Dec 13, 2024 •

edited

Loading

jparsonstf commented Dec 14, 2024

wilfredt commented Dec 18, 2024

EarningsCall commented Dec 22, 2024 •

edited

Loading

Metadata error logs in Cloudwatch agent - 404 EC2MetadataError: failed to make EC2Metadata request #1435

Metadata error logs in Cloudwatch agent - 404 EC2MetadataError: failed to make EC2Metadata request #1435

Comments

lesterxy commented Nov 20, 2024

debu99 commented Nov 24, 2024

tdtm commented Nov 28, 2024 • edited Loading

martinvocast commented Dec 3, 2024

motiumansky commented Dec 3, 2024 • edited Loading

ibaraki-douji commented Dec 4, 2024

iaon commented Dec 6, 2024

6lick commented Dec 13, 2024 • edited Loading

jparsonstf commented Dec 14, 2024

wilfredt commented Dec 18, 2024

EarningsCall commented Dec 22, 2024 • edited Loading

tdtm commented Nov 28, 2024 •

edited

Loading

motiumansky commented Dec 3, 2024 •

edited

Loading

6lick commented Dec 13, 2024 •

edited

Loading

EarningsCall commented Dec 22, 2024 •

edited

Loading