Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic in reconciler causing segmentation violation #150

Open
matthewgoodman13 opened this issue Jun 23, 2022 · 0 comments
Open

Panic in reconciler causing segmentation violation #150

matthewgoodman13 opened this issue Jun 23, 2022 · 0 comments
Labels
bug Something isn't working
Milestone

Comments

@matthewgoodman13
Copy link
Contributor

During benchmark testing, controller completely halted with a panic due to the attempt of accessing an invalid memory address or nil pointer dereference causing a segmentation violation.

Here are the last logs from the manager:

{"level":"info","ts":1655941251.2284162,"logger":"controllers.Cloudmap","msg":"CalculateChanges_ES_Plan","elapsed":9}
{"level":"info","ts":1655941251.228578,"logger":"controllers.Cloudmap","msg":"CalculateChanges_ES_Plan","elapsed":10}
{"level":"info","ts":1655941252.7608504,"logger":"controllers.ServiceExport","msg":"updating Cloud Map service","namespace":"demov2","name":"nginx-benchmark-service-61"}
{"level":"info","ts":1655941252.7608776,"logger":"cloudmap","msg":"fetching a service","namespace":"demov2","name":"nginx-benchmark-service-61"}
{"level":"info","ts":1655941252.76091,"logger":"cloudmap","msg":"creating a new service","namespace":"demov2","name":"nginx-benchmark-service-61"}
{"level":"info","ts":1655941252.8043206,"logger":"cloudmap","msg":"service created","namespace":"demov2","name":"nginx-benchmark-service-61","id":"srv-vvx3saukvkvkytng"}
{"level":"info","ts":1655941252.8043518,"logger":"cloudmap","msg":"fetching a service","namespace":"demov2","name":"nginx-benchmark-service-61"}
{"level":"info","ts":1655941252.8277564,"msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"serviceexport","controllerGroup":"multicluster.x-k8s.io","controllerKind":"ServiceExport","serviceExport":{"name":"nginx-benchmark-service-61","namespace":"demov2"},"namespace":"demov2","name":"nginx-benchmark-service-61","reconcileID":"f1c0f919-fdb1-4e63-be72-6aa2f4df35e7"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x15204f7]

goroutine 221 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118 +0x1f4
panic({0x16819e0, 0x271b0e0})
	/usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/aws/aws-cloud-map-mcs-controller-for-k8s/pkg/controllers.(*ServiceExportReconciler).handleUpdate(0xc000461e40, {0x1aad4b0, 0xc0007644c0}, 0xc00032be00, 0xc000d08000)
	/workspace/pkg/controllers/serviceexport_controller.go:136 +0x597
github.com/aws/aws-cloud-map-mcs-controller-for-k8s/pkg/controllers.(*ServiceExportReconciler).Reconcile(0xc000461e40, {0x1aad558, 0xc000b44540}, {{{0xc000d0b876, 0x18}, {0xc000b68780, 0x40d3a7}}})
	/workspace/pkg/controllers/serviceexport_controller.go:93 +0x7d5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1aad4b0, {0x1aad558, 0xc000b44540}, {{{0xc000d0b876, 0x17a4d00}, {0xc000b68780, 0x40ee1d}}})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121 +0xd1
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00031b7c0, {0x1aad4b0, 0xc000461c40}, {0x16e0800, 0xc00098b8c0})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320 +0x33c
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00031b7c0, {0x1aad4b0, 0xc000461c40})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:230 +0x36f

Note that the code lines do not line up exactly with the Controller's code.
serviceexport_controller.go:136 points to the Current: cmService.Endpoints line
and serviceexport_controller.go:93 to the return r.handleUpdate(ctx, &serviceExport, &service)

@matthewgoodman13 matthewgoodman13 added the bug Something isn't working label Jun 23, 2022
@runakash runakash added this to the 0.4.0 milestone Aug 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: 📋 Backlog
Development

No branches or pull requests

2 participants