Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: adjusting container instrumentation infrastructure monitoring to highlight assignment of container logs #16709

Closed
wants to merge 19 commits into from
Closed
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
---
title: Container logs configuration for infrastructure monitoring
tags:
- Infrastructure
- Install the infrastructure agent
- Linux installation
metaDescription: New Relic's infrastructure monitoring agent automatically instruments container runtimes to collect container metrics and metadata.
freshnessValidatedDate: never
---

import infrastructureAddNamesLogs from 'images/infrastructure_screenshot-crop_add-names-logs.webp'

import infrastructureAddContainersLogPath from 'images/infrastructure_screenshot-crop_add-containers-log-path.webp'

import logsParsingUi from 'images/infrastructure_screenshot-full_logs-parsing-ui.webp'

Our [infrastructure agent](/docs/infrastructure/infrastructure-monitoring/get-started/get-started-infrastructure-monitoring) can gather logs from your containers. When you enable logs from the infrastructure agent, the agent will automatically assign those logs to the hosts running Docker Compose.

## Requirements [#requirements]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do customers need to have instrumented their containers? Should we instead link them to the other containers dock as a prerequisite? I imagine they can't collect logs about containers w/o having instrumentation first.

@harrykimpel

Copy link
Author

@harrykimpel harrykimpel Apr 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@akristen yes, that is correct. I guess it makes sense to link them to the other containers doc.


To collect logs from your containers, you need to have:

* Infrastructure agent [1.8.32](/docs/release-notes/infrastructure-release-notes/infrastructure-agent-release-notes/new-relic-infrastructure-agent-1832) or higher running on Linux
* CentOS version 6.0 or higher, if you're using CentOS
* Docker with engine from [v1.12](https://docs.docker.com/engine/release-notes/prior-releases/#1120-2016-07-28) or other [`containerd`-based container runtimes](https://kubernetes.io/docs/setup/production-environment/container-runtimes)

## Collect logs from your containers [#collect-logs]

1. Open the configuration file for the infrastructure agent and add this snippet:

```yml
- name: containers
file: /var/lib/docker/containers/*/*.log
```

<img
title="Add the containers path to your snippet"
alt="A screenshot that shows where to add this snippet in context of your yaml file"
src={infrastructureAddContainersLogPath}
/>

* You may find this file at this path: `/etc/newrelic-infra/logging.d/logging.yml`.
* This section directs the agent to collect logs from your containers

2. Add a tag section to your `docker-compose.yml` that passes in the default `{{.Name}}` attribute:

```yml
version: '3.9'
x-default-logging: &logging
driver: "json-file"
options:
max-size: "5m"
max-file: "2"
tag: "{{.Name}}"
```

<img
title="Configuration file in context"
alt="Screenshot that shows this snippet in context of a configuration file"
src={infrastructureAddNamesLogs}
/>

This adds a `attrs.tag` attribute to your logs, which looks like this snippet:

```json
{
agentName:Infrastructure
agentVersion:1.50.0
**attrs.tag:cart-service**
coreCount:1
...
}
```

3. Once the `attrs.tag` appears in your host-level logs, you can enable container logs by going to <DoNotTranslate>**[one.newrelic.com](https://one.newrelic.com) > Infrastructure </DoNotTranslate>, then click <DoNotTranslate>**HERE**</DoNotTranslate>.

4. Create a log parsing rule by applying these configurations:

* Name: container name
* Field to parse: attrs.tag
* Filter logs based on NRQL: attrs.tag is not null
* Parsing rule: %{NOTSPACE:containerName}

Your modal should look like this:

<img
title="Log parsing rules"
alt="Screenshot of log parsing in UI"
src={logsParsingUi}
/>
<figcaption>

</figcaption>

5. Generate some data, wait a few minutes, and you'll see container logs assigned to your containers running on your host, rather than assigned to the host itself.
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
---
title: Container instrumentation for infrastructure monitoring
harrykimpel marked this conversation as resolved.
Show resolved Hide resolved
tags:
- Infrastructure
- Install the infrastructure agent
- Linux installation
- Infrastructure
- Install the infrastructure agent
- Linux installation
metaDescription: New Relic's infrastructure monitoring agent automatically instruments container runtimes to collect container metrics and metadata.
redirects:
- /docs/infrastructure-docker-integration
- /docs/infrastructure-docker-instrumentation
- /docs/infrastructure/install-configure-manage-infrastructure/docker-installation/docker-container-monitoring-infrastructure
- /docs/infrastructure/new-relic-infrastructure/data-instrumentation/docker-instrumentation-infrastructure
- /docs/infrastructure/install-configure-manage-infrastructure/docker-installation/docker-instrumentation-infrastructure
- /docs/infrastructure/install-configure-manage-infrastructure-agent/linux-installation/docker-instrumentation-infrastructure
- /docs/infrastructure/install-infrastructure-agent/linux-installation/docker-instrumentation-infrastructure-monitoring
- /docs/infrastructure/install-infrastructure-agent/linux-installation/docker-instrumentation-infrastructure
- /docs/infrastructure-docker-integration
- /docs/infrastructure-docker-instrumentation
- /docs/infrastructure/install-configure-manage-infrastructure/docker-installation/docker-container-monitoring-infrastructure
- /docs/infrastructure/new-relic-infrastructure/data-instrumentation/docker-instrumentation-infrastructure
- /docs/infrastructure/install-configure-manage-infrastructure/docker-installation/docker-instrumentation-infrastructure
- /docs/infrastructure/install-configure-manage-infrastructure-agent/linux-installation/docker-instrumentation-infrastructure
- /docs/infrastructure/install-infrastructure-agent/linux-installation/docker-instrumentation-infrastructure-monitoring
- /docs/infrastructure/install-infrastructure-agent/linux-installation/docker-instrumentation-infrastructure
freshnessValidatedDate: never
---

Expand All @@ -24,39 +24,39 @@ Our [infrastructure agent](/docs/welcome-new-relic-infrastructure) automatically
* Link to related entities that may be affected by issues with the container.
* Set [container-related alert conditions](#alerts).

## Requirements
This doc walks you through basic configurations for your instrumented containers.

Requirement details for automatic container monitoring for New Relic's infrastructure agent:
## Requirements [#requirements]

Review our requirements for monitoring your containers with the infrastructure agent:

* A New Relic account. Don't have one? [Sign up for free!](https://newrelic.com/signup). No credit card required.
* Infrastructure agent [1.8.32](/docs/release-notes/infrastructure-release-notes/infrastructure-agent-release-notes/new-relic-infrastructure-agent-1832) or higher running on Linux
* If using CentOS, you must have CentOS version 6.0 or higher
* Docker with engine from [v1.12](https://docs.docker.com/engine/release-notes/prior-releases/#1120-2016-07-28) or other [`containerd`-based container runtimes](https://kubernetes.io/docs/setup/production-environment/container-runtimes).
* CentOS version 6.0 or higher if using CentOS
* Docker with engine from [v1.12](https://docs.docker.com/engine/release-notes/prior-releases/#1120-2016-07-28) or other [`containerd`-based container runtimes](https://kubernetes.io/docs/setup/production-environment/container-runtimes)

<Callout variant="important">
Support for Operating systems using Control Group v2 is included from infrastructure agent v.1.26.0 and nri-docker v1.7.0.
<Callout variant="tip">
Support for operating systems using Control Group v2 is included from infrastructure agent v.1.26.0 and nri-docker v1.7.0.
</Callout>

## Enable container monitoring [#enable]
## Enable container monitoring [#enable-container-monitoring]

If you meet the [requirements](#requirements) and have installed the correct infrastructure monitoring agent, there are no additional steps to enable container monitoring. If a container runtime is running, data will automatically be reported.

You can also use a container image containing the infrastructure monitoring agent. For more information, see [Container for infrastructure monitoring](/docs/infrastructure/install-infrastructure-agent/linux-installation/container-infrastructure-monitoring).

<InstallFeedback />

## View your container data [#find-data]

To view your container data in the New Relic UI, use either of these options:

* Go to <DoNotTranslate>**[one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Infrastructure > Hosts > Containers**</DoNotTranslate>.

OR
OR

* Go to <DoNotTranslate>**[one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Infrastructure > Third-party services**</DoNotTranslate>, and select Docker-related links.

For more information, see [Query your data](/docs/using-new-relic/data/understand-data/query-new-relic-data).

## container attributes [#container-data]
## Container attributes [#container-data]

Container attributes (metrics and metadata) are attached to the [`ContainerSample`](/attribute-dictionary/?event=ContainerSample) event. Here's an example of a query to find out how many containers are associated with each container image:

Expand Down Expand Up @@ -87,7 +87,8 @@ To create container-related alert conditions, use either of these options:
2. Create a new [alert condition](/docs/alerts/new-relic-alerts/defining-conditions/create-alert-conditions).
3. For the condition type, select <DoNotTranslate>**Container metrics**</DoNotTranslate>.

## Enable container metrics collection from Docker API
## Enable container metrics collection from Docker API [#enable-metrics-docker-api]

The nri-docker integration, by default, employs the Docker API in conjunction with the /proc filesystem to extract container metrics. As of version v1.51.0 of the infrastructure agent, you can reconfigure the integration to solely source metrics from the Docker API.

This Docker API-only collection strategy applies only when the Docker Engine Cgroup operates under version V2. To enforce the Docker API-only collection, adapt the docker-config.yml configuration file located in the integrations.d directory to resemble the following:
Expand All @@ -107,6 +108,7 @@ integrations:
Metrics variations between metrics sources are detailed in the [`ContainerSample`](/attribute-dictionary/?event=ContainerSample) attributes.
</Callout>

## Disabling container monitoring
## Disable container monitoring [#disable-container-monitoring]

To disable this capability simply delete the `docker-config.yml` configuration file from the `integrations.d` folder.

Binary file not shown.
Binary file not shown.
Binary file not shown.
4 changes: 3 additions & 1 deletion src/nav/infrastructure.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,10 @@ pages:
path: /docs/infrastructure/install-infrastructure-agent/linux-installation/install-infrastructure-monitoring-agent-linux
- title: Configuration management tools
path: /docs/infrastructure/install-infrastructure-agent/linux-installation/install-configuration-management-tools
- title: Container instrumentation
akristen marked this conversation as resolved.
Show resolved Hide resolved
- title: Instrument your host containers
path: /docs/infrastructure/install-infrastructure-agent/linux-installation/container-instrumentation-infrastructure-monitoring
- title: Enable logs for instrumented Containers
path: /docs/infrastructure/install-infrastructure-agent/linux-installation/container-instrumentation-infrastructure-monitoring-logs/
- title: Azure extensions for Infrastructure
path: /docs/infrastructure/install-infrastructure-agent/linux-installation/azure-extensions-infrastructure
- title: Install as a container
Expand Down
Loading