Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

metrics: Reduce CPU cycles spent in Prometheus metric collection #3414

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

metrics: Reduce CPU cycles spent in Prometheus metric collection #3414

wants to merge 2 commits into from

Commits on Oct 13, 2023

  1. metrics: Add benchmark for container metric collection

    @brancz
    brancz committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    ed5cc47 View commit details
    Browse the repository at this point in the history

  2. metrics: Reuse descriptions when label names are unchanged 

    Prior to this change, Prometheus description objects were re-created on
every collection cycle, even though label names change extremely rarely
in practice (perhaps even never on a given machine). This means that we
were unnecessarily causing CPU cycles. Using the benchmark in the
previous commit, we determine we're making collection at least 23%
faster, but based on production profiling data, we estimate this is
improving the Kubelet's baseline CPU usage by ~1.3% (potentially even
more since it also relieves the GC). This is because the benchmark shows
that all calls to `NewDesc` are completely disappearing from profiling
data, therefore stastically speaking reducing the kubelet's CPU time by
that much.

```
benchstat old.txt new.txt
goos: linux
goarch: amd64
pkg: github.com/google/cadvisor/metrics
cpu: AMD Ryzen 5 3400GE with Radeon Vega Graphics
                      │   old.txt    │               new.txt               │
                      │    sec/op    │    sec/op     vs base               │
PrometheusCollector-8   614.4µ ± ∞ ¹   470.8µ ± ∞ ¹  -23.37% (p=0.008 n=5)
¹ need >= 6 samples for confidence interval at level 0.95
```
    @brancz
    brancz committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    6eefc42 View commit details
    Browse the repository at this point in the history