Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cni-metrics-helper metrics: do type assertion before type casting #3152

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

dshehbaj
Copy link
Member

@dshehbaj dshehbaj commented Dec 12, 2024

What type of PR is this?

Bug fix

Which issue does this PR fix?:

#3144

Testing done on this change:

Added unit tests for code path for publishing CloudWatch and Prometheus Metrics and also pasted debug logs to confirm that no panic happens.

Built metrics helper image successfully using make docker-metrics.

Unit tests passing


// Cloud watch unit test

$> go test -timeout 30s -run ^TestProduceCloudWatchMetrics$ github.com/aws/amazon-vpc-cni-k8s/cmd/cni-metrics-helper/metrics

ok  	github.com/aws/amazon-vpc-cni-k8s/cmd/cni-metrics-helper/metrics


// Prometheus Unit Test

$> go test -timeout 30s -run ^TestProducePrometheusMetrics$ github.com/aws/amazon-vpc-cni-k8s/cmd/cni-metrics-helper/metrics

ok  	github.com/aws/amazon-vpc-cni-k8s/cmd/cni-metrics-helper/metrics

Debug logs before the change

// LIST MAINTAINED BY CNI METRICS HELPER

{"msg":"Printing out all the Prometheus Mappings"}
{"msg":"Metric Name: awscni_eni_max, Metric Type: *prometheus.gauge"}
{"msg":"Metric Name: awscni_del_ip_req_count, Metric Type: *prometheus.CounterVec"}
{"msg":"Metric Name: awscni_total_ip_addresses, Metric Type: *prometheus.gauge"}
{"msg":"Metric Name: awscni_assigned_ip_addresses, Metric Type: *prometheus.gauge"}
{"msg":"Metric Name: awscni_no_available_ip_addresses, Metric Type: *prometheus.counter"}
{"msg":"Metric Name: awscni_total_ipv4_prefixes, Metric Type: *prometheus.gauge"}
{"msg":"Metric Name: awscni_ip_max, Metric Type: *prometheus.gauge"}
{"msg":"Metric Name: awscni_add_ip_req_count, Metric Type: *prometheus.counter"}
{"msg":"Metric Name: awscni_eni_allocated, Metric Type: *prometheus.gauge"}
{"msg":"Metric Name: awscni_force_removed_enis, Metric Type: *prometheus.counter"}
{"msg":"Metric Name: awscni_force_removed_ips, Metric Type: *prometheus.counter"}

// LIST RECIEVED FROM CNI

{"msg":"Printing out all the families recieved"}
{"msg":"Metric Name: awscni_ipamd_action_inprogress, Metric Type: GAUGE"}
{"msg":"Metric Name: awscni_reconcile_count, Metric Type: COUNTER"}
{"msg":"Metric Name: awscni_assigned_ip_per_cidr, Metric Type: GAUGE"}
{"msg":"Metric Name: awscni_ip_max, Metric Type: GAUGE"}
{"msg":"Metric Name: awscni_aws_api_latency_ms, Metric Type: SUMMARY"}
{"msg":"Metric Name: awscni_force_removed_enis, Metric Type: GAUGE"}
panic: interface conversion: *prometheus.counter is not prometheus.Gauge: missing method Dec

goroutine 1 [running]:
github.com/aws/amazon-vpc-cni-k8s/cmd/cni-metrics-helper/metrics.producePrometheusMetrics({0x35e8190, 0xc0004a6960}, 0xc0004a2150, 0xc0001aea50)
        /go/src/github.com/aws/amazon-vpc-cni-k8s/cmd/cni-metrics-helper/metrics/metrics.go:393 +0xce5
github.com/aws/amazon-vpc-cni-k8s/cmd/cni-metrics-helper/metrics.Handler({0x35e2690?, 0xc0001d7090?}, {0x35e8190, 0xc0004a6960})
        /go/src/github.com/aws/amazon-vpc-cni-k8s/cmd/cni-metrics-helper/metrics/metrics.go:479 +0x16a
main.main()
        /go/src/github.com/aws/amazon-vpc-cni-k8s/cmd/cni-metrics-helper/main.go:176 +0xf7c

Debug logs after the change

// LIST MAINTAINED BY CNI METRICS HELPER

{"msg":"Printing out all the Prometheus Mappings"}
{"msg":"Metric Name: awscni_eni_max, Metric Type: *prometheus.gauge"}
{"msg":"Metric Name: awscni_total_ip_addresses, Metric Type: *prometheus.gauge"}
{"msg":"Metric Name: awscni_force_removed_ips, Metric Type: *prometheus.counter"}
{"msg":"Metric Name: awscni_total_ipv4_prefixes, Metric Type: *prometheus.gauge"}
{"msg":"Metric Name: awscni_ip_max, Metric Type: *prometheus.gauge"}
{"msg":"Metric Name: awscni_add_ip_req_count, Metric Type: *prometheus.counter"}
{"msg":"Metric Name: awscni_del_ip_req_count, Metric Type: *prometheus.CounterVec"}
{"msg":"Metric Name: awscni_eni_allocated, Metric Type: *prometheus.gauge"}
{"msg":"Metric Name: awscni_assigned_ip_addresses, Metric Type: *prometheus.gauge"}
{"msg":"Metric Name: awscni_force_removed_enis, Metric Type: *prometheus.counter"}
{"msg":"Metric Name: awscni_no_available_ip_addresses, Metric Type: *prometheus.counter"}
{"msg":"\n\n"}

// LIST RECIEVED FROM CNI

{"msg":"Printing out all the families recieved"}
{"msg":"Metric Name: awscni_eni_max, Metric Type: GAUGE"}
{"msg":"Metric Name: awscni_assigned_ip_per_cidr, Metric Type: GAUGE"}
{"msg":"Metric Name: awscni_ipamd_action_inprogress, Metric Type: GAUGE"}
{"msg":"Metric Name: awscni_assigned_ip_addresses, Metric Type: GAUGE"}
{"msg":"Metric Name: awscni_reconcile_count, Metric Type: COUNTER"}
{"msg":"Metric Name: awscni_eni_allocated, Metric Type: GAUGE"}
{"msg":"Metric Name: awscni_del_ip_req_count, Metric Type: COUNTER"}
{"msg":"Metric Name: awscni_ec2api_req_count, Metric Type: COUNTER"}
{"msg":"Metric Name: awscni_ec2api_error_count, Metric Type: COUNTER"}

// TYPE ASSERTION CHECK
{"msg":"Metric Name: awscni_force_removed_ips, Metric Type: GAUGE"}
{"msg":"Metric awscni_force_removed_ips is not a Gauge type, skipping"}

// TYPE ASSERTION CHECK
{"msg":"Metric Name: awscni_force_removed_enis, Metric Type: GAUGE"}
{"msg":"Metric awscni_force_removed_enis is not a Gauge type, skipping"}

{"msg":"Metric Name: awscni_ip_max, Metric Type: GAUGE"}
{"msg":"Metric Name: awscni_aws_api_latency_ms, Metric Type: SUMMARY"}
{"msg":"Metric Name: awscni_aws_api_error_count, Metric Type: GAUGE"}

// TYPE ASSERTION CHECK
{"msg":"Metric Name: awscni_add_ip_req_count, Metric Type: GAUGE"}
{"msg":"Metric awscni_add_ip_req_count is not a Gauge type, skipping"}

{"msg":"Metric Name: awscni_total_ipv4_prefixes, Metric Type: GAUGE"}
{"msg":"Metric Name: awscni_total_ip_addresses, Metric Type: GAUGE"}


// NO PANIC HAPPENS

@dshehbaj dshehbaj marked this pull request as ready for review December 12, 2024 20:29
@dshehbaj dshehbaj requested a review from a team as a code owner December 12, 2024 20:29
@dshehbaj dshehbaj force-pushed the shehbaj/metrics-helper-incompatibility-fix branch 2 times, most recently from 5394d02 to b6be2b9 Compare December 19, 2024 23:10
@dshehbaj dshehbaj requested a review from orsenthil December 19, 2024 23:14
@dshehbaj dshehbaj force-pushed the shehbaj/metrics-helper-incompatibility-fix branch from 76e366a to 2ae0160 Compare December 19, 2024 23:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant