Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

v1.34.0 Segfault on ReadRelationships before WriteSchema when using memstore #1985

Closed
theronic opened this issue Jul 17, 2024 · 9 comments
Closed

v1.34.0 Segfault on ReadRelationships before WriteSchema when using memstore #1985

theronic opened this issue Jul 17, 2024 · 9 comments
Labels
kind/bug Something is broken or regressed

Comments

@theronic
Copy link

theronic commented Jul 17, 2024 •
edited
Loading

What platforms are affected?

macos

What architectures are affected?

amd64

What SpiceDB version are you using?

v1.34.0

Steps to Reproduce

Docker image sha256:5f81b3ea0702907c847e11cb20a6a58d60921811628da6776336ce2da7c60d68.

I can provide a Docker image and code if you send me an Image registry I can push to. I stopped a SpiceDB Docker container, and then restarted it with docker start -a <container-id> and attempted to ReadRelationships for some non-existent schema. The FailedPrecondition seems to be the cause of the segfault.

Expected Result

Spice should not segfault. ReadRelationships should return an error.

Actual Result

Segmentation Fault. Logs:

{"level":"warn","protocol":"grpc","grpc.component":"server","grpc.service":"authzed.api.v1.PermissionsService","grpc.method":"ReadRelationships","grpc.method_type":"server_stream","requestID":"edd969d0685a5dda472a2c43b57d41d0","peer.address":"192.168.65.1:60543","grpc.start_time":"2024-07-17T13:27:29Z","grpc.code":"FailedPrecondition","grpc.error":"object definition `team` not found","grpc.time_ms":0,"time":"2024-07-17T13:27:29Z","message":"finished call"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xe55780]

goroutine 360 [running]:
github.com/authzed/spicedb/internal/namespace.CheckNamespaceAndRelations({0x2db1268, 0x4000bec1b0}, {0x4000e7e238, 0x2, 0x40008bf1d0?}, {0x2dbdaa0, 0x4000bec240})
	/home/runner/actions-runner/_work/spicedb/spicedb/internal/namespace/util.go:72 +0x2c0
github.com/authzed/spicedb/internal/services/v1.(*permissionServer).CheckPermission(0x40008ca870, {0x2db1268, 0x4000bec1b0}, 0x4000c7d260)
	/home/runner/actions-runner/_work/spicedb/spicedb/internal/services/v1/permissions.go:53 +0x154
github.com/authzed/authzed-go/proto/authzed/api/v1._PermissionsService_CheckPermission_Handler.func1({0x2db1268, 0x4000bec1b0}, {0x25cd1c0?, 0x4000c7d260})
	/home/runner/go/pkg/mod/github.com/authzed/[email protected]/proto/authzed/api/v1/permission_service_grpc.pb.go:342 +0x74
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.UnaryServerInterceptor.UnaryServerInterceptor.func4({0x2db1268, 0x4000bec0c0}, {0x25cd1c0, 0x4000c7d260}, 0x4000e7e601?, 0x4000c9a8e8)
	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/server.go:22 +0x1f0
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainUnaryServer.func1.1.1({0x2db1268?, 0x4000bec0c0?}, {0x25cd1c0?, 0x4000c7d260?})
	/home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:23 +0x40
github.com/authzed/spicedb/internal/middleware/handwrittenvalidation.UnaryServerInterceptor({0x2db1268, 0x4000bec0c0}, {0x25cd1c0, 0x4000c7d260}, 0xffff5aa3fca8?, 0x4000754dc0)
	/home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/handwrittenvalidation/handwrittenvalidation.go:26 +0xc0
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainUnaryServer.func1.1.1({0x2db1268?, 0x4000bec0c0?}, {0x25cd1c0?, 0x4000c7d260?})
	/home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:23 +0x40
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/validator.UnaryServerInterceptor.func1({0x2db1268, 0x4000bec0c0}, {0x25cd1c0, 0x4000c7d260}, 0x30?, 0x4000754e00)
	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/validator/interceptors.go:23 +0x7c
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainUnaryServer.func1.1.1({0x2db1268?, 0x4000bec0c0?}, {0x25cd1c0?, 0x4000c7d260?})
	/home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:23 +0x40
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainUnaryServer.func1({0x2db1268, 0x4000bec0c0}, {0x25cd1c0, 0x4000c7d260}, 0x4000e7e7f8?, 0x12c3528?)
	/home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:32 +0xb8
github.com/authzed/spicedb/internal/middleware/servicespecific.UnaryServerInterceptor({0x2db1268, 0x4000bec0c0}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20, 0x4000c9a8e8)
	/home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/servicespecific/servicespecific.go:25 +0x84
google.golang.org/grpc.getChainUnaryHandler.func1({0x2db1268, 0x4000bec0c0}, {0x25cd1c0, 0x4000c7d260})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1205 +0xa0
github.com/authzed/spicedb/pkg/cmd/server.DefaultUnaryMiddleware.UnaryServerInterceptor.func15({0x2db1268, 0x4000bec090}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20, 0x4000b518c0)
	/home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/consistency/consistency.go:191 +0x138
google.golang.org/grpc.getChainUnaryHandler.func1({0x2db1268, 0x4000bec090}, {0x25cd1c0, 0x4000c7d260})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1205 +0xa0
github.com/authzed/spicedb/pkg/cmd/server.DefaultUnaryMiddleware.UnaryServerInterceptor.func13({0x2db1268?, 0x4000bec060?}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20?, 0x4000b51880)
	/home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/datastore/datastore.go:70 +0x90
google.golang.org/grpc.getChainUnaryHandler.func1({0x2db1268, 0x4000bec060}, {0x25cd1c0, 0x4000c7d260})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1205 +0xa0
github.com/authzed/spicedb/pkg/cmd/server.DefaultUnaryMiddleware.UnaryServerInterceptor.func11({0x2db1268?, 0x4000969fb0?}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20?, 0x4000b51840)
	/home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/dispatcher/dispatcher.go:66 +0x90
google.golang.org/grpc.getChainUnaryHandler.func1({0x2db1268, 0x4000969fb0}, {0x25cd1c0, 0x4000c7d260})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1205 +0xa0
github.com/authzed/spicedb/pkg/cmd/server.DefaultUnaryMiddleware.UnaryServerInterceptor.UnaryServerInterceptor.func20({0x2db1268, 0x4000969fb0}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20?, 0x4000b51800)
	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/server.go:22 +0x1f0
google.golang.org/grpc.getChainUnaryHandler.func1({0x2db1268, 0x4000969fb0}, {0x25cd1c0, 0x4000c7d260})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1205 +0xa0
github.com/authzed/spicedb/pkg/cmd/server.DefaultUnaryMiddleware.UnaryServerInterceptor.func8({0x2db1268, 0x4000969fb0}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20, 0x4000b517c0)
	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/auth/auth.go:48 +0xbc
google.golang.org/grpc.getChainUnaryHandler.func1({0x2db1268, 0x4000969fb0}, {0x25cd1c0, 0x4000c7d260})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1205 +0xa0
github.com/grpc-ecosystem/go-grpc-prometheus.init.(*ServerMetrics).UnaryServerInterceptor.func3({0x2db1268, 0x4000969fb0}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20?, 0x4000b51780)
	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:107 +0x74
google.golang.org/grpc.getChainUnaryHandler.func1({0x2db1268, 0x4000969fb0}, {0x25cd1c0, 0x4000c7d260})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1205 +0xa0
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x2db1268, 0x4000969d10}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20, 0x4000b51700)
	/home/runner/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:326 +0x3f0
google.golang.org/grpc.getChainUnaryHandler.func1({0x2db1268, 0x4000969d10}, {0x25cd1c0, 0x4000c7d260})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1205 +0xa0
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/logging.UnaryServerInterceptor.UnaryServerInterceptor.func2({0x2db1268, 0x4000969ce0}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20?, 0x4000b516c0)
	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/server.go:22 +0x1f0
google.golang.org/grpc.getChainUnaryHandler.func1({0x2db1268, 0x4000969ce0}, {0x25cd1c0, 0x4000c7d260})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1205 +0xa0
github.com/authzed/spicedb/pkg/cmd/server.DefaultUnaryMiddleware.UnaryServerInterceptor.UnaryServerInterceptor.func18({0x2db1268, 0x4000969800}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20?, 0x4000b51680)
	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/server.go:22 +0x1f0
google.golang.org/grpc.getChainUnaryHandler.func1({0x2db1268, 0x4000969800}, {0x25cd1c0, 0x4000c7d260})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1205 +0xa0
github.com/authzed/spicedb/pkg/middleware/requestid.UnaryServerInterceptor.UnaryServerInterceptor.func1({0x2db1268, 0x4000969770}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20?, 0x4000b51640)
	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/server.go:22 +0x1f0
google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0x2db1268, 0x4000969770}, {0x25cd1c0, 0x4000c7d260}, 0x4001303988?, 0x536b98?)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1196 +0x88
github.com/authzed/authzed-go/proto/authzed/api/v1._PermissionsService_CheckPermission_Handler({0x24f9e40?, 0x40008ca870}, {0x2db1268, 0x4000969770}, 0x4000b4eb00, 0x4000754940)
	/home/runner/go/pkg/mod/github.com/authzed/[email protected]/proto/authzed/api/v1/permission_service_grpc.pb.go:344 +0x12c
google.golang.org/grpc.(*Server).processUnaryRPC(0x40005dc800, {0x2db1268, 0x40009696e0}, {0x2dbf358, 0x4001350000}, 0x40012406c0, 0x4000f16f30, 0x47d97d0, 0x0)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1385 +0xb8c
google.golang.org/grpc.(*Server).handleStream(0x40005dc800, {0x2dbf358, 0x4001350000}, 0x40012406c0)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1796 +0xc4c
google.golang.org/grpc.(*Server).serveStreams.func2.1()
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1029 +0x8c
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 261
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1040 +0x150
@theronic theronic added the kind/bug Something is broken or regressed label Jul 17, 2024
@josephschorr
Copy link
Member

@theronic Are you using an in-memory datastore?

@josephschorr
Copy link
Member

Please reopen if you can reproduce. If you're using an in-memory datastore and resuming the container, it likely won't function.

@theronic
Copy link
Author

Another segfault on ReadRelationships when running SpiceDB v1.31.0 on CircleCI, when no schema exists:

8:31AM INF telemetry reporter scheduled endpoint=https://telemetry.authzed.com interval=1h0m0s next=5m3s
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `server` not found" grpc.method=LookupResources grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:56Z grpc.time_ms=0 peer.address=127.0.0.1:46884 protocol=grpc requestID=cqu6ne3tgl0dnppd3rig
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `account` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:58Z grpc.time_ms=0 peer.address=127.0.0.1:46900 protocol=grpc requestID=cqu6nejtgl0dnppd3rj0
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `account` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:58Z grpc.time_ms=0 peer.address=127.0.0.1:46900 protocol=grpc requestID=cqu6nejtgl0dnppd3rjg
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `account` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:58Z grpc.time_ms=0 peer.address=127.0.0.1:46900 protocol=grpc requestID=cqu6nejtgl0dnppd3rk0
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `server` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:58Z grpc.time_ms=0 peer.address=127.0.0.1:46900 protocol=grpc requestID=cqu6nejtgl0dnppd3rkg
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `server` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:58Z grpc.time_ms=0 peer.address=127.0.0.1:46900 protocol=grpc requestID=cqu6nejtgl0dnppd3rl0
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `server` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:58Z grpc.time_ms=0 peer.address=127.0.0.1:46900 protocol=grpc requestID=cqu6nejtgl0dnppd3rlg
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x13dc74d]

goroutine 283 [running]:
github.com/authzed/spicedb/internal/namespace.CheckNamespaceAndRelations({0x3604fb0, 0xc0005acc80}, {0xc000e02088, 0x2, 0x3617300?}, {0x3613a70, 0xc000a4d110})
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/namespace/util.go:71 +0x34d
github.com/authzed/spicedb/internal/services/v1.(*permissionServer).LookupResources(0xc001512a80, 0xc0001c1110, {0x3611720, 0xc000ea90f0})
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/services/v1/permissions.go:381 +0x191
github.com/authzed/authzed-go/proto/authzed/api/v1._PermissionsService_LookupResources_Handler({0x2bf0460, 0xc001512a80}, {0x360d868, 0xc000a4cfc0})
        /home/runner/go/pkg/mod/github.com/authzed/[email protected]/proto/authzed/api/v1/permission_service_grpc.pb.go:407 +0x107
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.MustStreamServerInterceptor.func2({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cf60}, 0x2b5c620?, 0x2f32638)
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/streamtimeout/streamtimeout.go:29 +0x136
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainStreamServer.func3.1.1({0x2bf0460?, 0xc001512a80?}, {0x360ca68?, 0xc000a4cf60?})
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:47 +0x34
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.StreamServerInterceptor.StreamServerInterceptor.func5({0x2bf0460, 0xc001512a80}, {0x360e4d0, 0xc000ea8fd0}, 0x2b32820?, 0xc000d19140)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/server.go:35 +0x2bb
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainStreamServer.func3.1.1({0x2bf0460?, 0xc001512a80?}, {0x360e4d0?, 0xc000ea8fd0?})
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:47 +0x34
github.com/authzed/spicedb/internal/middleware/handwrittenvalidation.StreamServerInterceptor({0x2bf0460, 0xc001512a80}, {0x360e488, 0xc000b55698}, 0x2b963a0?, 0xc000d19160)
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/handwrittenvalidation/handwrittenvalidation.go:33 +0x7a
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainStreamServer.func3.1.1({0x2bf0460?, 0xc001512a80?}, {0x360e488?, 0xc000b55698?})
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:47 +0x34
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/validator.StreamServerInterceptor.func1({0x2bf0460, 0xc001512a80}, {0x360cab0, 0xc000d19120}, 0x20?, 0xc000d19180)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/validator/interceptors.go:58 +0x9e
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainStreamServer.func3.1.1({0x2bf0460?, 0xc001512a80?}, {0x360cab0?, 0xc000d19120?})
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:47 +0x34
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainStreamServer.func3({0x2bf0460, 0xc001512a80}, {0x360cab0, 0xc000d19120}, 0x1?, 0x7f9db5fba8e8?)
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:56 +0xb8
github.com/authzed/spicedb/internal/middleware/servicespecific.StreamServerInterceptor({0x2bf0460, 0xc001512a80}, {0x360cab0, 0xc000d19120}, 0xc000b555f0, 0x2f32638)
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/servicespecific/servicespecific.go:35 +0x83
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360cab0, 0xc000d19120})
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.func21({0x2bf0460, 0xc001512a80}, {0x360c990, 0xc000d19100}, 0xc000b555f0, 0xc0009ecb40)
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/consistency/consistency.go:205 +0x1a2
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360c990, 0xc000d19100})
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.func19({0x2bf0460, 0xc001512a80}, {0x360c990?, 0xc000d19100?}, 0xc000b555f0?, 0xc0009ecb00)
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/datastore/datastore.go:83 +0xbd
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360c990, 0xc000d19100})
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.func17({0x2bf0460, 0xc001512a80}, {0x360ca68?, 0xc000a4ce70?}, 0xc000b555f0?, 0xc0009ecac0)
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/dispatcher/dispatcher.go:79 +0xbd
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4ce70})
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.StreamServerInterceptor.func27({0x2bf0460, 0xc001512a80}, {0x360c990, 0xc000d190e0}, 0xc000b555f0?, 0xc0009eca80)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/server.go:35 +0x2bb
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360c990, 0xc000d190e0})
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.func14({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cde0}, 0xc000b555f0, 0xc0009eca40)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/auth/auth.go:68 +0x126
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cde0})
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.createServerMetrics.(*ServerMetrics).StreamServerInterceptor.StreamServerInterceptor.func6({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cdb0}, 0xc000b555f0?, 0xc0009eca00)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/server.go:35 +0x2bb
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cdb0})
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1532 +0xb2
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/logging.StreamServerInterceptor.StreamServerInterceptor.func2({0x2bf0460, 0xc001512a80}, {0x360d5f0, 0xc0009ec900}, 0xc000e76941?, 0xc0009ec980)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/server.go:35 +0x2bb
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.func11({0x2bf0460, 0xc001512a80}, {0x360d5f0, 0xc0009ec900}, 0xc000b555f0, 0xc0009ec980)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/selector/selector.go:50 +0x1c4
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360d5f0, 0xc0009ec900})
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.func7({0x2bf0460, 0xc001512a80}, {0x360d5f0, 0xc0009ec900}, 0xc000b555f0, 0xc0009ec940)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/selector/selector.go:52 +0x1fa
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360d5f0, 0xc0009ec900})
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1532 +0xb2
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.StreamServerInterceptor.func1({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cc60}, 0xc000b555f0, 0xc0009ec880)
        /home/runner/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:446 +0x5bb
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cc60})
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.StreamServerInterceptor.func24({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cba0}, 0xc000b555f0?, 0xc0009ec840)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/server.go:35 +0x2bb
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cba0})
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/middleware/requestid.StreamServerInterceptor.StreamServerInterceptor.func1({0x2bf0460, 0xc001512a80}, {0x360d4d0, 0xc000d5c4b0}, 0xc000b555f0?, 0xc0009ec800)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/server.go:35 +0x2bb
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x2bf0460, 0xc001512a80}, {0x360d4d0, 0xc000d5c4b0}, 0xc000b555f0, 0xc000ea8aa0?)
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1523 +0x85
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00051f400, {0x3604f78, 0xc000a4c660}, {0x3615060, 0xc000e26000}, 0xc00086fe60, 0xc00097e0f0, 0x51643c0, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1687 +0x11e7
google.golang.org/grpc.(*Server).handleStream(0xc00051f400, {0x3615060, 0xc000e26000}, 0xc00086fe60)
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1801 +0xe36
google.golang.org/grpc.(*Server).serveStreams.func2.1()
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1027 +0x8b
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 262
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1038 +0x125

Exited with code 2

@theronic
Copy link
Author

theronic commented Aug 14, 2024 •
edited
Loading

@josephschorr using in-memory container for tests, but no container resumption in the latest segfault case.

@theronic
Copy link
Author

theronic commented Aug 14, 2024 •
edited
Loading

Bumped SpiceDB version to 1.35.2 and same LookupResources segfault on empty schema with in-memory data store. We can switch to Postgres but this is inconvenient for tests.

Have you guys considered using Sqlite as a backing store?

@tstirrat15
Copy link
Contributor

@theronic can you explain the use case further? Are you using spicedb serve or spicedb serve-testing when running the container? Is this for running integration tests? What kinds of assertions are the tests making?

I agree that a segfault isn't the best behavior. What behavior are you expecting in this case? Should it be erroring or are you expecting that the schema should be present at this point in your workflow?

@josephschorr
Copy link
Member

@theronic We've tried to reproduce to no avail and, unless we're missing something, a nil issue is not possible with the in-memory datastore, on the reported line of code in the stack trace.

At this point, we need exact reproduction instructions if possible.

@theronic
Copy link
Author

theronic commented Aug 15, 2024 •
edited
Loading

@tstirrat15 running spicedb serve using SpiceDB 1.34.0 via Docker:

 docker run --rm -it \
    --name spicedb \
    -p 50051:50051 \
    authzed/spicedb \
    serve \
    --grpc-preshared-key "spicedb"

The behaviour I expect is that ReadSchema or ReadRelationships should never crash, even if no schema exists. Should return empty string for schema or empty set of relationships on ReadRelationships when blank schema. Segfault concerning for us because it implies under certain conditions if we Read from Spice, it will bring down Spice in prod, but hopefully that won't happen when using Postgres (only memstore).

@josephschorr I am attempting to reproduce a minimal example for you. It seems to be related to ReadSchema before any schema was written to in-memory store, followed by ReadRelationships (args are just resource_type = "server") and possibly also DeleteRelationships. I suspect there is some initial state that is null when no schema has been written, that is fixed after writing schema. This happens when running SpiceDB in Docker on macOS, not just on CircleCI:

11:52AM INF grpc server started serving addr=:50051 insecure=true network=tcp service=grpc workers=0
11:52AM INF running server datastore=*schemacaching.definitionCachingProxy
11:52AM INF telemetry reporter scheduled endpoint=https://telemetry.authzed.com interval=1h0m0s next=1m13s
11:52AM INF http server started serving addr=:9090 insecure=true service=metrics
11:52AM INF finished call grpc.code=NotFound grpc.component=server grpc.error="rpc error: code = NotFound desc = No schema has been defined; please call WriteSchema to start" grpc.method=ReadSchema grpc.method_type=unary grpc.service=authzed.api.v1.SchemaService grpc.start_time=2024-08-15T11:52:26Z grpc.time_ms=0 peer.address=192.168.65.1:47865 protocol=grpc requestID=cquunumaahes73cq3km0
11:52AM INF finished call grpc.code=NotFound grpc.component=server grpc.error="rpc error: code = NotFound desc = No schema has been defined; please call WriteSchema to start" grpc.method=ReadSchema grpc.method_type=unary grpc.service=authzed.api.v1.SchemaService grpc.start_time=2024-08-15T11:52:26Z grpc.time_ms=0 peer.address=192.168.65.1:47865 protocol=grpc requestID=cquunumaahes73cq3kmg
11:52AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `server` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-15T11:52:26Z grpc.time_ms=0 peer.address=192.168.65.1:47865 protocol=grpc requestID=cquunumaahes73cq3kn0
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x10c86c8]

goroutine 249 [running]:
github.com/authzed/spicedb/internal/namespace.CheckNamespaceAndRelations({0x33a0718, 0x40012e2750}, {0x4001041e90, 0x2, 0x4001041df8?}, {0x33b1fa8, 0x40012e27e0})
	/home/runner/actions-runner/_work/spicedb/spicedb/internal/namespace/util.go:71 +0x288
github.com/authzed/spicedb/internal/services/v1.(*permissionServer).CheckPermission(0x4000b70850, {0x33a0718, 0x40012e2750}, 0x40009cb980)
	/home/runner/actions-runner/_work/spicedb/spicedb/internal/services/v1/permissions.go:58 +0x154
github.com/authzed/authzed-go/proto/authzed/api/v1._PermissionsService_CheckPermission_Handler.func1({0x33a0718?, 0x40012e2750?}, {0x2a27f00?, 0x40009cb980?})
	/home/runner/go/pkg/mod/github.com/authzed/[email protected]/proto/authzed/api/v1/permission_service_grpc.pb.go:375 +0xd0
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.UnaryServerInterceptor.UnaryServerInterceptor.func4({0x33a0718, 0x40012e2660}, {0x2a27f00, 0x40009cb980}, 0x4001042101?, 0x40012bc660)
	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/server.go:22 +0x1f0
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainUnaryServer.func1.1.1({0x33a0718?, 0x40012e2660?}, {0x2a27f00?, 0x40009cb980?})
...

Note I am calling this via Clojure using gRPC libraries, but SpiceDB actually crashes in my Docker, so I don't believe this is related to any gRPC impl. bugs.

@theronic
Copy link
Author

theronic commented Aug 15, 2024 •
edited
Loading

I'm not a Go guy, but possibly one of these error values causing it when attempting ReadRelationships before writing any schema.

These are guesses, but maybe r.txSource() or tx.Lowerbound(tableNamespace, indexID) before indexID = "id" is written to memstore (pre schema write).

I am confident there is a memstore bug when reading before writing schema.

Seems to be intermittent on my side, which hints at cache-related, or memory allocation related. I cannot consistently reproduce it locally, but can do it consistently on CircleCI on our prod stack, which I'm trying to reduce down to a minimal example.

@theronic theronic changed the title Segmentation Fault on ReadRelationships v1.34.0 Docker Image v1.34.0 Segfault on ReadRelationships before WriteSchema when using memstore Aug 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Something is broken or regressed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@theronic @tstirrat15 @josephschorr