Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sentry: replica_write.go:186: log.Fatal: replica is corrupted: replica corruption (processed=true): × (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(... #132032

Open
cockroach-sentry opened this issue Oct 5, 2024 · 0 comments
Open

Sentry: replica_write.go:186: log.Fatal: replica is corrupted: replica corruption (processed=true): × (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(... #132032

cockroach-sentry opened this issue Oct 5, 2024 · 0 comments
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented Oct 5, 2024 •
edited by cockroach-jira-scripts
Loading

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5957387155/?referrer=webhooks_plugin

Panic Message:

replica_write.go:186: log.Fatal: replica is corrupted: replica corruption (processed=true): ×
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:186
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:495
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:185
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:206
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1190
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1274
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:841
  | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:96
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:952
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:276
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:952
  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:952
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:243
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:322
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:241
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:952
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:851
  | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:227
  | github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1036
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:921
  | github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1044
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:210
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:189
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2168
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1694
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1265
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendParallelCommit
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1065
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:884
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:82
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:47
Wraps: (2) secondary error attachment
  | replica corruption (processed=true): ×
  | (1) replica corruption (processed=true): ×
  | Error types: (1) *kvpb.ReplicaCorruptionError
Wraps: (3) log.Fatal: replica is corrupted: replica corruption (processed=true): ×
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: replica is corrupted: replica corruption (processed=true): ×
*secondary.withSecondaryError
replica_write.go:186: *withstack.withStack (top exception)
Stacktrace (expand for inline code snippets):

cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go

Lines 46 to 48 in 2c1d4af

br, pErr := m.wrapped.SendLocked(ctx, ba)
if pErr != nil {

cockroach/pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go

Lines 81 to 83 in 2c1d4af

defer gs.mu.Lock()
return gs.wrapped.Send(ctx, ba)
}

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 883 to 885 in 2c1d4af

if withParallelCommit {
rpl, pErr = ds.divideAndSendParallelCommit(ctx, ba, rs, isReverse, 0 /* batchIdx */)
} else {

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 1064 to 1066 in 2c1d4af

// isReverse flag.
br, pErr = ds.divideAndSendBatchToRanges(ctx, ba, rs, isReverse, true /* withCommit */, batchIdx)

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 1264 to 1266 in 2c1d4af

if !ri.NeedAnother(rs) {
resp := ds.sendPartialBatch(
ctx, ba, rs, isReverse, withCommit, batchIdx, ri.Token(), nil, /* positions */

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 1693 to 1695 in 2c1d4af

prevTok = routingTok
reply, err = ds.sendToReplicas(ctx, ba, routingTok, withCommit)

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 2167 to 2169 in 2c1d4af

}
br, err = transport.SendNext(ctx, ba)
ds.maybeIncrementErrCounters(br, err)

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 188 to 190 in 2c1d4af

}
return gt.sendBatch(ctx, r.NodeID, iface, ba)
}

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 209 to 211 in 2c1d4af

}
reply, err := iface.Batch(ctx, ba)
// If we queried a remote node, perform extra validation.

https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/rpc/pkg/rpc/context.go#L1043-L1045
https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/rpc/pkg/rpc/context.go#L920-L922
https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/rpc/pkg/rpc/context.go#L1035-L1037

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 226 to 228 in 2c1d4af

if localRequest {
return invoker(ctx, method, req, resp, cc, opts...)
}

https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/rpc/pkg/rpc/context.go#L850-L852
https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/rpc/pkg/rpc/context.go#L951-L953
https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/rpc/pkg/rpc/context.go#L240-L242

cockroach/pkg/util/stop/stopper.go

Lines 321 to 323 in 2c1d4af

return f(ctx)
}

https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/rpc/pkg/rpc/context.go#L242-L244
https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/rpc/pkg/rpc/context.go#L951-L953
https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/rpc/pkg/rpc/auth.go#L104-L106
https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/rpc/pkg/rpc/context.go#L951-L953
https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/rpc/pkg/rpc/context.go#L275-L277
https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/rpc/pkg/rpc/context.go#L951-L953

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 95 to 97 in 2c1d4af

if methodExcludedFromTracing(info.FullMethod) {
return handler(ctx, req)
}

https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/rpc/pkg/rpc/context.go#L840-L842

cockroach/pkg/server/node.go

Lines 1273 to 1275 in 2c1d4af

br, err := n.batchInternal(ctx, tenantID, args)

cockroach/pkg/server/node.go

Lines 1189 to 1191 in 2c1d4af

var pErr *kvpb.Error
br, writeBytes, pErr = n.stores.SendWithWriteBytes(ctx, args)
if pErr != nil {

https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L202-L204
https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L205-L207
https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L184-L186
https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L494-L496
https://github.com/cockroachdb/cockroach/blob/2c1d4afd2c2de091a00a4dba4392ceeaf8504a95/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go#L185-L187

pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go in pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked at line 47
pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go in pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked at line 82
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).Send at line 884
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendParallelCommit at line 1065
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges at line 1265
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch at line 1694
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas at line 2168
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext at line 189
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch at line 210
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 1044
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 921
pkg/rpc/pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 1036
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2 at line 227
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 851
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 952
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 241
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 322
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 243
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 952
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 105
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 952
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 276
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 952
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1 at line 96
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 841
pkg/server/node.go in pkg/server.(*Node).Batch at line 1274
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1190
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 203
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 206
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 185
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 495
pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).executeWriteBatch at line 186

Tags

Tag Value
Command server
Environment v23.1.8
Go Version go1.19.10
Platform linux amd64
Distribution CCL
Cockroach Release v23.1.8
Cockroach SHA 2c1d4af
# of CPUs 8
# of Goroutines 556

Jira issue: CRDB-42794
@cockroach-sentry cockroach-sentry added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Oct 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cockroach-sentry